π Advanced Microservices Architecture - Practical Assignments
Here is the full English translation of the uploaded document E-Commerce_Microservices_Platform_Assignment_v1.0_en.md.
All Vietnamese instructions, scenarios, and explanations have been translated into professional technical English while maintaining the original Markdown structure, code blocks, and diagrams.
π Advanced Microservices Architecture - Practical Assignments
E-Commerce Platform Migration Projectβ
Course: Advanced Microservices Architecture Version: 1.0.0 Last Updated: 2025-12-24 Prerequisite: Completed mono-src (Building High-Performance APIs with FastAPI)
π― Project Overviewβ
Migration Journeyβ
You will migrate mono-src (monolithic Product Catalog Service) to a complete Microservices architecture:
βββββββββββββββββββββββ βββββββββββββββββββββββββββββββββββββββββββ
β β β MICROSERVICES PLATFORM β
β mono-src β β β
β (Monolithic) β βββββββΊ β βββββββββββ βββββββββββ βββββββββββ β
β β β β User β β Product β β Order β β
β β’ User Auth β β β Service β β Service β β Service β β
β β’ Product CRUD β β ββββββ¬βββββ ββββββ¬βββββ ββββββ¬βββββ β
β β’ Single DB β β β β β β
β β β ββββββΌβββββ ββββββΌβββββ ββββββΌβββββ β
βββββββββββββββββββββββ β β DB β β DB β β DB β β
β βββββββββββ βββββββββββ βββββββββββ β
β β
β + API Gateway (Kong) β
β + Message Broker (RabbitMQ) β
β + Cache (Redis) β
β + Observability Stack β
βββββββββββββββββββββββββββββββββββββββββββ
Technology Stackβ
| Component | Technology | Port |
|---|---|---|
| User Service | FastAPI + PostgreSQL | 8001 |
| Product Service | FastAPI + PostgreSQL + Redis | 8002 |
| Order Service | FastAPI + PostgreSQL + RabbitMQ | 8003 |
| Notification Service | FastAPI + RabbitMQ | 8004 |
| API Gateway | Kong | 8000 |
| Cache | Redis | 6379 |
| Message Broker | RabbitMQ | 5672, 15672 |
| Tracing | Jaeger | 16686 |
| Metrics | Prometheus | 9090 |
| Logging | Loki + Promtail | 3100 |
| Visualization | Grafana | 3000 |
Target Architectureβ
βββββββββββββββββββ
β Svelte SPA β
β (Port 5173) β
ββββββββββ¬βββββββββ
β
ββββββββββΌβββββββββ
β Kong Gateway β
β (Port 8000) β
ββββββββββ¬βββββββββ
β
ββββββββββββββββββββββββββββΌβββββββββββββββββββββββββββ
β β β
βββββββββΌββββββββ ββββββββββΌβββββββββ ββββββββββΌβββββββββ
β User Service βββββββββββ Product Service β β Order Service β
β (Port 8001) β Token β (Port 8002) β β (Port 8003) β
β β Verify β β β β
β βββββββββββ β β βββββββββββ β β βββββββββββ β
β βPostgreSQLβ β β βPostgreSQLβ β β βPostgreSQLβ β
β β user_db β β β βproduct_dbβ β β β order_db β β
β βββββββββββ β β βββββββββββ β β βββββββββββ β
βββββββββββββββββ β β β β
β βββββββββββ β β β β
β β Redis β β β β β
β β Cache β β β βΌ β
β βββββββββββ β β ββββββββββββ β
βββββββββββββββββββ β β RabbitMQ β β
β β Publisherβ β
β ββββββ¬ββββββ β
βββββββββΌββββββββββ
β
βββββββββΌββββββββββ
β Notification β
β Service β
β (Port 8004) β
β β
β βββββββββββββ β
β β RabbitMQ β β
β β Consumer β β
β βββββββββββββ β
βββββββββββββββββββ
Assignment 01: User Service Extractionβ
Unit 1 - Microservices Fundamentals & Refactoringβ
π― Learning Objectivesβ
- Understand Bounded Context and Database per Service concepts.
- Apply the Strangler Fig Pattern to migrate services.
- Set up an independent microservice with FastAPI.
π Business Requirementsβ
Scenarioβ
The company has decided to transition from a monolithic to a microservices architecture to:
- Scale components independently.
- Deploy features without affecting the entire system.
- Allow teams to work in parallel.
User Service is selected first because:
- It has a clear Bounded Context (Authentication domain).
- It has few dependencies on other domains.
- It is a critical component - needs stability and independent scaling.
β Required Tasksβ
Task 1.1: Project Structure Setup (15 points)β
Requirements:
Create the directory structure for User Service according to the pattern:
micro-src/
βββ server/
βββ user-service/
βββ app/
β βββ __init__.py
β βββ main.py # FastAPI app entry
β βββ api/
β β βββ __init__.py
β β βββ auth.py # Auth endpoints
β β βββ deps.py # Dependencies
β βββ config/
β β βββ __init__.py
β β βββ settings.py # Pydantic settings
β βββ database/
β β βββ __init__.py
β β βββ database.py # SQLAlchemy setup
β βββ models/
β β βββ __init__.py
β β βββ user.py # User model
β βββ repositories/
β β βββ __init__.py
β β βββ base.py
β β βββ user_repository.py
β βββ schemas/
β β βββ __init__.py
β β βββ user.py # Pydantic schemas
β βββ services/
β β βββ __init__.py
β β βββ auth_service.py # Business logic
β βββ utils/
β βββ __init__.py
β βββ security.py # JWT, password utils
βββ alembic/
βββ tests/
βββ Dockerfile
βββ requirements.txt
βββ alembic.ini
βββ start.sh
Acceptance Criteria:
- Correct directory structure.
- All
__init__.pyfiles exist. - Code from mono-src migrated and refactored.
Task 1.2: Database per Service (20 points)β
Business Logic:
User Service requires a separate, completely isolated database:
- Do not share schema with other services.
- Independent DB scaling.
- Independent migration.
Technical Requirements:
| Setting | Value |
|---|---|
| Database | PostgreSQL 15 |
| DB Name | user_service_db |
| Port (external) | 5433 |
| Port (internal) | 5432 |
Docker Compose Configuration:
user-db:
image: postgres:15-alpine
environment:
POSTGRES_USER: user
POSTGRES_PASSWORD: password
POSTGRES_DB: user_service_db
ports:
- '5433:5432'
volumes:
- user-db-data:/var/lib/postgresql/data
healthcheck:
test: ['CMD-SHELL', 'pg_isready -U user -d user_service_db']
Acceptance Criteria:
- Database container runs successfully.
- Connection from service to DB works.
- Healthcheck passes.
Task 1.3: User Model & Migrations (15 points)β
Business Logic:
Migrate the User model from mono-src, ensuring:
- Preserve schema structure.
- Alembic migrations work.
- Auto-migration when the container starts.
Model Definition:
| Column | Type | Constraints |
|---|---|---|
| id | Integer | PK, auto-increment |
| username | String(50) | Unique, Not Null, Indexed |
| hashed_password | String(255) | Not Null |
| is_active | Boolean | Default True |
| created_at | DateTime | Server default now() |
| updated_at | DateTime | Auto-update |
Acceptance Criteria:
- Alembic initialized.
- Initial migration generated.
-
start.shruns migrations automatically.
Task 1.4: Authentication Endpoints (25 points)β
Business Logic:
Migrate and enhance authentication endpoints from mono-src:
| Endpoint | Method | Description |
|---|---|---|
/register | POST | User registration |
/login | POST | OAuth2 Password Flow |
/validate-token | POST | NEW - Token validation for other services |
/health | GET | Health check |
Token Validation Endpoint (Important):
This is a NEW endpoint, not present in mono-src. Other services (Product, Order) will call this endpoint to validate JWT tokens instead of decoding the token directly.
POST /validate-token
Content-Type: application/json
Request:
{
"token": "eyJhbGciOiJIUzI1NiIs..."
}
Response (Valid):
{
"valid": true,
"user_id": 1,
"username": "john_doe",
"message": "Token is valid"
}
Response (Invalid):
{
"valid": false,
"user_id": null,
"username": null,
"message": "Token is invalid or expired"
}
Why not share the JWT secret?
- Security: Each service does not need to know the secret key.
- Centralized auth: User Service is the Single Source of Truth.
- Rotation: Keys can be rotated without updating other services.
Acceptance Criteria:
- POST /register creates user successfully.
- POST /login returns JWT token.
- POST /validate-token validates and returns user info.
- GET /health returns status.
Task 1.5: Dockerization (15 points)β
Business Logic:
The Service needs to be containerized for easy deployment:
- Multi-stage build (optional) to reduce image size.
- Non-root user for security.
- Auto-migration on startup.
Dockerfile Requirements:
FROM python:3.11-slim
WORKDIR /app
# Install dependencies
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
# Copy application
COPY . .
# Make start script executable
RUN chmod +x start.sh
# Create non-root user
RUN useradd -m appuser && chown -R appuser:appuser /app
USER appuser
EXPOSE 8001
HEALTHCHECK --interval=30s --timeout=3s --start-period=40s \
CMD curl -f http://localhost:8001/health || exit 1
CMD ["./start.sh"]
start.sh:
#!/bin/bash
set -e
echo "Waiting for database..."
sleep 5
echo "Running database migrations..."
alembic upgrade head
echo "Starting User Service..."
uvicorn app.main:app --host 0.0.0.0 --port 8001
Acceptance Criteria:
- Docker build successful.
- Container starts and is healthy.
- Migrations run automatically.
- API accessible at port 8001.
π Bonus Tasksβ
Bonus 1.1: User Events Publishing (15 points)β
Guideline: When a user successfully registers, publish a user.created event to RabbitMQ. The event contains user_id, username, created_at. Prepare for Notification Service to consume.
Bonus 1.2: Password Policy (10 points)β
Guideline: Implement password policy validation: min 8 chars, 1 uppercase, 1 lowercase, 1 digit, 1 special char. Return specific error messages for each failed rule.
Bonus 1.3: Rate Limiting (10 points)β
Guideline: Implement rate limiting for the login endpoint: max 5 attempts per minute per IP. Use in-memory storage or Redis.
π Rubric - Assignment 01β
| Criteria | Excellent (100%) | Good (80%) | Satisfactory (60%) | Needs Improvement (40%) |
|---|---|---|---|---|
| Project Structure (15pts) | Perfect structure, all files | Minor issues | Basic structure | Incorrect structure |
| Database Setup (20pts) | Isolated DB, healthcheck, volumes | Works, minor config issues | Basic setup | Cannot connect |
| Model & Migrations (15pts) | Alembic works, auto-migrate | Works manually | Partial | Not working |
| Auth Endpoints (25pts) | All 4 endpoints, validate-token works | Missing 1 endpoint | 2 endpoints work | Most broken |
| Dockerization (15pts) | Multi-stage, non-root, healthcheck | Basic Dockerfile works | Container runs | Build fails |
| Code Quality (10pts) | Clean, documented, type hints | Good code | Acceptable | Poor quality |
Total: 100 points Bonus: 35 points
π Assignment 02: Product Service & API Gatewayβ
Unit 2 - API Gateway (Kong)β
π― Learning Objectivesβ
- Extract Product Service into an independent microservice.
- Implement inter-service authentication.
- Setup and configure Kong API Gateway.
- Understand routing and rate limiting concepts.
π Business Requirementsβ
Scenarioβ
After the User Service is stable, the team continues to migrate the Product Service. At the same time, an API Gateway is needed to:
- Provide a single entry point for all API calls.
- Centralize routing and security.
- Implement rate limiting to protect services.
- Handle logging and monitoring.
β Required Tasksβ
Task 2.1: Product Service Extraction (25 points)β
Business Logic:
Migrate the Product domain from mono-src into an independent service. Product Service:
- Has a separate database (product_service_db).
- Does NOT decode JWT tokens directly.
- Calls User Service to validate tokens.
Project Structure:
product-service/
βββ app/
β βββ main.py
β βββ api/
β β βββ products.py # CRUD endpoints
β β βββ deps.py # Auth dependency
β βββ config/
β βββ database/
β βββ models/
β β βββ product.py
β βββ repositories/
β β βββ product_repository.py
β βββ schemas/
β β βββ product.py
β βββ services/
β β βββ product_service.py
β βββ utils/
β βββ auth_client.py # HTTP client to User Service
βββ alembic/
βββ Dockerfile
βββ requirements.txt
Acceptance Criteria:
- Service runs on port 8002.
- Separate database (product_service_db, port 5434).
- CRUD endpoints functional.
Task 2.2: Inter-Service Authentication (25 points)β
Business Logic:
Product Service needs to authenticate the user but does NOT have access to the JWT secret. Flow:
ββββββββββ Token βββββββββββββββββββ Validate ββββββββββββββββ
β Client β βββββββββββΊ β Product Service β ββββββββββββββΊ β User Service β
ββββββββββ β β β β
β Authorization: β POST β /validate- β
β Bearer <token> β ββββββββββββββ β token β
βββββββββββββββββββ {valid: true}ββββββββββββββββ
Auth Client Implementation:
Create auth_client.py to call User Service:
# app/utils/auth_client.py
class AuthClient:
def __init__(self, user_service_url: str):
self.user_service_url = user_service_url
async def validate_token(self, token: str) -> Optional[dict]:
"""
Validate JWT token via User Service
Args:
token: JWT token from Authorization header
Returns:
User info dict if valid, None if invalid
"""
# Implement HTTP call to User Service /validate-token
pass
Dependency for Protected Routes:
# app/api/deps.py
async def get_current_user(
authorization: str = Header(None)
) -> dict:
"""
Dependency to get current user via User Service
Raises:
HTTPException 401 if no token or invalid
"""
# Extract token from "Bearer <token>"
# Call auth_client.validate_token(token)
# Return user info or raise 401
pass
Acceptance Criteria:
- Auth client calls User Service successfully.
- Protected endpoints require valid token.
- 401 returned when token is invalid/missing.
- User info available in route handlers.
Task 2.3: Kong API Gateway Setup (30 points)β
Business Logic:
Kong Gateway provides:
- Routing: Direct requests to correct service.
- Rate Limiting: Protect services from abuse.
- Logging: Centralized access logs.
- Authentication (optional): Can add JWT plugin.
Docker Compose for Kong:
kong-db:
image: postgres:15-alpine
environment:
POSTGRES_USER: kong
POSTGRES_PASSWORD: kong
POSTGRES_DB: kong
volumes:
- kong-db-data:/var/lib/postgresql/data
kong-migration:
image: kong:3.4
command: kong migrations bootstrap
environment:
KONG_DATABASE: postgres
KONG_PG_HOST: kong-db
KONG_PG_USER: kong
KONG_PG_PASSWORD: kong
depends_on:
kong-db:
condition: service_healthy
kong:
image: kong:3.4
environment:
KONG_DATABASE: postgres
KONG_PG_HOST: kong-db
KONG_PG_USER: kong
KONG_PG_PASSWORD: kong
KONG_PROXY_ACCESS_LOG: /dev/stdout
KONG_ADMIN_ACCESS_LOG: /dev/stdout
KONG_PROXY_ERROR_LOG: /dev/stderr
KONG_ADMIN_ERROR_LOG: /dev/stderr
KONG_ADMIN_LISTEN: 0.0.0.0:8001
ports:
- '8000:8000' # Proxy
- '8443:8443' # Proxy SSL
- '8001:8001' # Admin API
depends_on:
kong-migration:
condition: service_completed_successfully
Kong Configuration via Admin API:
- Register User Service:
curl -X POST http://localhost:8001/services \
--data "name=user-service" \
--data "url=http://user-service:8001"
curl -X POST http://localhost:8001/services/user-service/routes \
--data "paths[]=/api/users" \
--data "strip_path=true"
- Register Product Service:
curl -X POST http://localhost:8001/services \
--data "name=product-service" \
--data "url=http://product-service:8002"
curl -X POST http://localhost:8001/services/product-service/routes \
--data "paths[]=/api/products" \
--data "strip_path=true"
Routing Table:
| Client Request | Kong Route | Upstream |
|---|---|---|
| POST /api/users/register | /api/users/* | user-service:8001/register |
| POST /api/users/login | /api/users/* | user-service:8001/login |
| GET /api/products | /api/products/* | product-service:8002/products |
| POST /api/products | /api/products/* | product-service:8002/products |
Acceptance Criteria:
- Kong container running healthy.
- User Service accessible via Kong.
- Product Service accessible via Kong.
- Direct service access still works (for internal calls).
Task 2.4: Rate Limiting Plugin (10 points)β
Business Logic:
Protect API from abuse with rate limiting:
- Login endpoint: 5 requests/minute (prevent brute force).
- Product listing: 100 requests/minute.
- Product creation: 20 requests/minute.
Kong Rate Limiting Configuration:
# Global rate limit
curl -X POST http://localhost:8001/plugins \
--data "name=rate-limiting" \
--data "config.minute=100" \
--data "config.policy=local"
# Specific for login route
curl -X POST http://localhost:8001/services/user-service/plugins \
--data "name=rate-limiting" \
--data "config.minute=10" \
--data "config.policy=local"
Acceptance Criteria:
- Rate limiting plugin enabled.
- Exceeded requests return 429 Too Many Requests.
- Headers show rate limit info (X-RateLimit-*).
π Bonus Tasksβ
Bonus 2.1: Kong Declarative Config (15 points)β
Guideline: Instead of Admin API calls, create a kong.yml declarative config file. Mount it to the container with KONG_DECLARATIVE_CONFIG. Easier for version control and reproduction.
Bonus 2.2: JWT Plugin (15 points)β
Guideline: Configure Kong JWT plugin to validate tokens at the gateway level. When enabled, Kong verifies JWT before forwarding to the service. Requires registering consumer and credentials.
Bonus 2.3: Request/Response Transformation (10 points)β
Guideline: Add a transformation plugin to add custom headers (X-Request-ID, X-Service-Name) into responses. Useful for debugging and tracing.
π Rubric - Assignment 02β
| Criteria | Excellent (100%) | Good (80%) | Satisfactory (60%) | Needs Improvement (40%) |
|---|---|---|---|---|
| Product Service (25pts) | Complete CRUD, isolated DB | Missing 1 feature | Basic CRUD works | Service broken |
| Inter-Service Auth (25pts) | Auth client works, proper errors | Works, error handling weak | Basic validation | Not implemented |
| Kong Setup (30pts) | Full routing, both services | One service works | Kong runs, partial routing | Kong not working |
| Rate Limiting (10pts) | Multiple limits configured | Basic limit works | Plugin enabled | Not configured |
| Code Quality (10pts) | Clean, async, documented | Good code | Acceptable | Poor quality |
Total: 100 points Bonus: 40 points
π Assignment 03: Async Communication (RabbitMQ)β
Unit 3 - Async Communicationβ
π― Learning Objectivesβ
- Understand the difference between Sync vs Async communication.
- Setup RabbitMQ with Exchange, Queue, Binding.
- Implement Producer/Consumer pattern with pika/aio-pika.
- Handle message acknowledgment and error cases.
π Business Requirementsβ
Scenarioβ
When a user registers, the system needs to send a welcome email. If the email API is called synchronously:
- User must wait for email sending to complete (slow response).
- If the email service is down β registration fails (not acceptable).
Solution: Async communication via message queue.
ββββββββββββββββ 1.Register ββββββββββββββββ
β Client β βββββββββββββΊ β User Service β
ββββββββββββββββ β β
β² β 2. Create β
β β User β
β 3. Return β β
β 201 Created β 4. Publish β
β (immediate) β Event β
βββββββββββββββββββββββββ΄βββββββ¬ββββββββ
β
ββββββββΌββββββββ
β RabbitMQ β
β user.createdβ
ββββββββ¬ββββββββ
β
ββββββββΌββββββββ
β Notification β
β Service β
β β
β 5. Send Emailβ
β (async) β
ββββββββββββββββ
β Required Tasksβ
Task 3.1: RabbitMQ Setup (15 points)β
Docker Compose:
rabbitmq:
image: rabbitmq:3-management-alpine
container_name: rabbitmq-server
environment:
RABBITMQ_DEFAULT_USER: guest
RABBITMQ_DEFAULT_PASS: guest
ports:
- '5672:5672' # AMQP
- '15672:15672' # Management UI
volumes:
- rabbitmq-data:/var/lib/rabbitmq
healthcheck:
test: ['CMD', 'rabbitmq-diagnostics', 'ping']
interval: 10s
timeout: 5s
retries: 5
Concepts to Understand:
| Concept | Description |
|---|---|
| Exchange | Router that receives messages and routes them to queues |
| Queue | Buffer that stores messages waiting to be consumed |
| Binding | Rule connecting exchange to queue |
| Routing Key | Pattern to route message to correct queue |
Exchange Types:
| Type | Behavior |
|---|---|
| direct | Route by exact routing key match |
| topic | Route by pattern matching (*.created, order.#) |
| fanout | Broadcast to all bound queues |
| headers | Route based on headers |
Acceptance Criteria:
- RabbitMQ container healthy.
- Management UI accessible at :15672.
- Can create exchange/queue via UI.
Task 3.2: Event Publisher - User Service (25 points)β
Business Logic:
After user registers successfully, publish event to RabbitMQ:
Event Structure:
{
"event": "user.created",
"timestamp": "2025-12-24T10:00:00Z",
"data": {
"user_id": 1,
"username": "john_doe",
"email": "john@example.com",
"created_at": "2025-12-24T10:00:00Z"
}
}
Exchange Configuration:
- Name:
user_events - Type:
topic - Durable:
true
Publisher Implementation Skeleton:
# user-service/app/utils/rabbitmq.py
class RabbitMQPublisher:
"""Publisher class for sending events to RabbitMQ"""
async def connect(self):
"""Connect to RabbitMQ and declare exchange 'user_events' (topic, durable)"""
# TODO: Use aio_pika.connect_robust()
pass
async def publish_user_created(self, user_data: dict):
"""
Publish user.created event with:
- routing_key: "user.created"
- delivery_mode: PERSISTENT
- body: {"event": "user.created", "data": user_data}
"""
pass
async def close(self):
"""Close connection gracefully"""
pass
Integration Point (AuthService):
# After create user successfully, publish event:
await rabbitmq_publisher.publish_user_created({
"user_id": user.id,
"username": user.username,
"created_at": str(user.created_at),
})
Acceptance Criteria:
- Exchange
user_eventscreated (durable). - Event published when user registers.
- Message visible in RabbitMQ Management UI.
- Registration is still fast (async publish).
Task 3.3: Event Consumer - Notification Service (30 points)β
Business Logic:
Notification Service consumes events and sends notifications:
- Welcome email when user registers.
- Order confirmation when order created (future).
Project Structure:
notification-service/
βββ app/
β βββ main.py
β βββ config/
β β βββ settings.py
β βββ utils/
β βββ rabbitmq.py # Consumer
β βββ notifications.py # Email/SMS handlers
βββ Dockerfile
βββ requirements.txt
Consumer Implementation Skeleton:
# notification-service/app/utils/rabbitmq.py
class RabbitMQConsumer:
"""Consumer class for processing messages from RabbitMQ"""
async def start_consuming(self):
"""
Setup and start consuming:
1. Connect to RabbitMQ
2. Set QoS (prefetch_count=10)
3. Declare exchange 'user_events' (topic, durable)
4. Declare queue 'user_notifications' (durable)
5. Bind queue to exchange with routing_key="user.created"
6. Start async iterator over queue
"""
pass
async def handle_message(self, message):
"""
Process message:
- Parse JSON body
- Route to appropriate handler based on event type
- Acknowledge on success, nack on failure
"""
pass
async def send_welcome_email(self, user_data: dict):
"""Send welcome email to new user"""
# TODO: Implement email sending (SMTP, SendGrid, etc.)
pass
Startup Integration:
# main.py - Start consumer on application startup
@app.on_event("startup")
async def startup_event():
asyncio.create_task(rabbitmq_consumer.start_consuming())
Acceptance Criteria:
- Queue
user_notificationscreated. - Queue bound to exchange with routing key.
- Consumer receives messages.
- Message acknowledged after processing.
- Failed messages handled (DLQ or retry).
Task 3.4: Error Handling & Reliability (20 points)β
Business Logic:
Message queue needs to be reliable:
- Messages must not be lost if consumer crashes.
- Failed messages need to be retried or moved to DLQ.
- Consumer crash should not block other messages.
Reliability Patterns:
- Persistent Messages:
message = aio_pika.Message(
body=...,
delivery_mode=aio_pika.DeliveryMode.PERSISTENT, # Save to disk
)
- Manual Acknowledgment:
async with message.process(): # Auto-ack on success
try:
await handle_message(message)
except Exception:
await message.nack(requeue=True) # Requeue on failure
- Dead Letter Queue (DLQ):
# Declare main queue with DLQ settings
queue = await channel.declare_queue(
"user_notifications",
durable=True,
arguments={
"x-dead-letter-exchange": "dlx",
"x-dead-letter-routing-key": "failed",
"x-message-ttl": 300000, # 5 minutes
}
)
Acceptance Criteria:
- Messages persistent (survive broker restart).
- Failed messages requeued or sent to DLQ.
- Consumer reconnects after connection loss.
- No message loss in failure scenarios.
π Bonus Tasksβ
Bonus 3.1: Retry with Exponential Backoff (15 points)β
Guideline: Implement retry mechanism with exponential backoff: 1s β 2s β 4s β 8s β DLQ. Track retry count in message headers. Use separate retry exchange.
Bonus 3.2: Multiple Consumers (10 points)β
Guideline: Scale consumer horizontally by running multiple instances. Messages are distributed (competing consumers pattern). Test with 3 consumer instances.
Bonus 3.3: Event Schema Registry (10 points)β
Guideline: Create shared event schemas package. Define JSON Schema for each event type. Validate messages against schema before publish/consume.
π Rubric - Assignment 03β
| Criteria | Excellent (100%) | Good (80%) | Satisfactory (60%) | Needs Improvement (40%) |
|---|---|---|---|---|
| RabbitMQ Setup (15pts) | Healthy, UI works, concepts clear | Works, minor issues | Basic setup | Not running |
| Publisher (25pts) | Events published, persistent, async | Works, missing persistence | Basic publish | Not implemented |
| Consumer (30pts) | Consumes, processes, acknowledges | Works, ack issues | Receives messages | Not working |
| Error Handling (20pts) | DLQ, retry, reconnection | Partial error handling | Basic try/catch | No error handling |
| Code Quality (10pts) | Clean async code, documented | Good code | Acceptable | Poor quality |
Total: 100 points Bonus: 35 points
π Assignment 04: SAGA Pattern - Order Serviceβ
Unit 4 & 5 - SAGA Pattern (Happy Path & Failure Path)β
π― Learning Objectivesβ
- Understand Distributed Transactions issues (ACID vs BASE).
- Implement SAGA Pattern with Choreography.
- Handle Compensating Transactions on failure.
- Build Order Service with multi-service coordination.
π Business Requirementsβ
Scenarioβ
Order creation involves multiple services:
- User Service: Verify user exists and authenticated.
- Product Service: Check stock availability, deduct stock.
- Order Service: Create order record.
- Notification Service: Send order confirmation.
Problem: Distributed Transaction
In a monolith, database transactions are used:
BEGIN;
UPDATE products SET quantity = quantity - 1;
INSERT INTO orders ...;
COMMIT; -- All or nothing
In microservices, each service has its own DB β Cannot use single transaction.
Solution: SAGA Pattern
SAGA is a sequence of local transactions, where each transaction updates a service and publishes an event to trigger the next transaction.
Order Created Flow (Happy Path):
ββββββββββββββ ββββββββββββββ ββββββββββββββ ββββββββββββββ
β Order β β Product β β Order β βNotificationβ
β Service β β Service β β Service β β Service β
βββββββ¬βββββββ βββββββ¬βββββββ βββββββ¬βββββββ βββββββ¬βββββββ
β β β β
β 1. Create β β β
β Order β β β
β (PENDING) β β β
β β β β
β 2. Publish ββββββΊβ β β
β order.created β β β
β β β β
β β 3. Reserve β β
β β Stock β β
β β β β
β β 4. Publish ββββββΊβ β
β β stock.reserved β β
β β β β
β β β 5. Update β
β β β Order β
β β β (CONFIRMED) β
β β β β
β β β 6. Publish ββββββΊβ
β β β order.confirmed β
β β β β
β β β β 7. Send
β β β β Email
βΌ βΌ βΌ βΌ
Failure Path (Compensating Transaction):
ββββββββββββββ ββββββββββββββ ββββββββββββββ
β Order β β Product β β Order β
β Service β β Service β β Service β
βββββββ¬βββββββ βββββββ¬βββββββ βββββββ¬βββββββ
β β β
β 1. Order β β
β (PENDING) β β
β β β
β 2. order.createdββΊ β
β β β
β β 3. Check Stock β
β β INSUFFICIENT! β
β β β
β β 4. Publish ββββββΊβ
β β stock.failed β
β β β
β β β 5. COMPENSATE
β β β Update Order
β β β (CANCELLED)
β β β
ββββββββββββββββββββββββββββββββββββββ
β order.cancelled β
βΌ βΌ βΌ
β Required Tasksβ
Task 4.1: Order Service Setup (20 points)β
Project Structure:
order-service/
βββ app/
β βββ main.py
β βββ api/
β β βββ orders.py
β β βββ deps.py
β βββ config/
β βββ database/
β βββ models/
β β βββ order.py
β β βββ order_item.py
β βββ repositories/
β β βββ order_repository.py
β βββ schemas/
β β βββ order.py
β βββ services/
β β βββ order_service.py
β βββ utils/
β βββ auth_client.py # Call User Service
β βββ product_client.py # Call Product Service
β βββ rabbitmq.py # Event publisher
βββ alembic/
βββ Dockerfile
βββ requirements.txt
Order Model:
| Column | Type | Description |
|---|---|---|
| id | Integer | PK |
| user_id | Integer | User who placed order |
| status | Enum | PENDING, CONFIRMED, CANCELLED, SHIPPED, DELIVERED |
| total_amount | Decimal | Total order value |
| created_at | DateTime | Order creation time |
| updated_at | DateTime | Last update time |
OrderItem Model:
| Column | Type | Description |
|---|---|---|
| id | Integer | PK |
| order_id | Integer | FK to Order |
| product_id | Integer | Product ID |
| product_name | String | Snapshot of product name |
| quantity | Integer | Quantity ordered |
| unit_price | Decimal | Price at order time |
Why snapshot product data?
- Product name/price can change.
- Order needs to preserve information at the time of ordering.
- Historical accuracy.
Acceptance Criteria:
- Order Service runs on port 8003.
- Order and OrderItem models created.
- Migrations work.
Task 4.2: Order Creation API (25 points)β
Business Logic:
Create order endpoint needs to:
- Validate user (via User Service).
- Validate products exist (via Product Service).
- Create order with status PENDING.
- Publish
order.createdevent. - Return order (do not wait for stock reservation).
API Endpoint:
POST /orders
Authorization: Bearer <token>
Content-Type: application/json
Request:
{
"items": [
{
"product_id": 1,
"quantity": 2
},
{
"product_id": 3,
"quantity": 1
}
]
}
Response 201:
{
"id": 1,
"user_id": 5,
"status": "PENDING",
"total_amount": 299.97,
"items": [
{
"product_id": 1,
"product_name": "Product A",
"quantity": 2,
"unit_price": 99.99
},
{
"product_id": 3,
"product_name": "Product C",
"quantity": 1,
"unit_price": 99.99
}
],
"created_at": "2025-12-24T10:00:00Z"
}
Service Implementation Steps:
async def create_order(self, user_id: int, items: List[OrderItemCreate]) -> Order:
"""
Create order with SAGA pattern (step 1)
Steps:
1. Fetch product info from Product Service (validate products exist)
2. Snapshot product data (name, price at order time)
3. Calculate total amount
4. Create order with status PENDING
5. Publish 'order.created' event to RabbitMQ
6. Return order immediately (don't wait for stock)
"""
pass
Acceptance Criteria:
- Order created with status PENDING.
- Products fetched from Product Service.
- Total calculated correctly.
- Event published to RabbitMQ.
- Response immediate (not waiting for stock).
Task 4.3: Stock Reservation - Product Service (25 points)β
Business Logic:
Product Service consumes order.created event and reserves stock:
Consumer Implementation Skeleton:
# product-service/app/utils/rabbitmq.py
async def handle_order_created(self, event_data: dict):
"""
Handle order.created event - Reserve stock
Steps:
1. Extract order_id and items from event
2. Check stock availability for ALL items first
3. If all available: deduct stock, publish 'stock.reserved'
4. If any insufficient: publish 'stock.failed' with reason
5. Invalidate product cache after stock changes
"""
pass
Events:
| Event | Routing Key | When |
|---|---|---|
| stock.reserved | stock.reserved | All items reserved successfully |
| stock.failed | stock.failed | Insufficient stock |
Acceptance Criteria:
- Product Service consumes order.created.
- Stock deducted when sufficient.
- stock.reserved event published on success.
- stock.failed event published on failure.
- Cache invalidated after stock change.
Task 4.4: Order Status Update & Compensation (20 points)β
Business Logic:
Order Service consumes stock events to update order status.
Event Handlers Skeleton:
async def handle_stock_reserved(self, event_data: dict):
"""
Happy path: Update order to CONFIRMED
Publish 'order.confirmed' for notification
"""
pass
async def handle_stock_failed(self, event_data: dict):
"""
Compensation: Update order to CANCELLED
Record cancellation reason
Publish 'order.cancelled'
"""
pass
Order Status Flow:
PENDING βββββββΊ CONFIRMED βββββββΊ SHIPPED βββββββΊ DELIVERED
β
ββββββββββββΊ CANCELLED (compensation)
Acceptance Criteria:
- Order updated to CONFIRMED on success.
- Order updated to CANCELLED on failure.
- Cancellation reason recorded.
- Appropriate events published.
π Bonus Tasksβ
Bonus 4.1: Orchestration Pattern (20 points)β
Guideline: Implement SAGA Orchestrator service instead of Choreography. Orchestrator coordinates steps, tracks SAGA state, handles rollbacks centrally. Create new saga_orchestrator service.
Bonus 4.2: Partial Failure Handling (15 points)β
Guideline: Handle case: Order has 3 items, 2 items have stock, 1 item out of stock. Options: (1) Cancel entire order, (2) Partial fulfillment. Implement configurable behavior.
Bonus 4.3: Order Status WebSocket (10 points)β
Guideline: Implement WebSocket endpoint /orders/{id}/status for real-time status updates. Client subscribes and receives updates when status changes.
π Rubric - Assignment 04β
| Criteria | Excellent (100%) | Good (80%) | Satisfactory (60%) | Needs Improvement (40%) |
|---|---|---|---|---|
| Order Service Setup (20pts) | Complete with items model | Basic order model | Partial | Not working |
| Order Creation (25pts) | Full validation, event published | Works, minor issues | Basic creation | Broken |
| Stock Reservation (25pts) | Reserve + events for both paths | Success path only | Basic consumer | Not implemented |
| Compensation (20pts) | Full compensation, status flow | Basic cancellation | Partial | No compensation |
| Code Quality (10pts) | Clean, documented, proper async | Good | Acceptable | Poor |
Total: 100 points Bonus: 45 points
π Assignment 05: Performance - Redis Cachingβ
Unit 6 - Performance Patterns (Redis)β
π― Learning Objectivesβ
- Understand caching strategies (Cache-aside, Read-through, Write-through).
- Implement Cache-aside pattern with Redis.
- Handle Cache Invalidation correctly.
- Measure performance improvement.
π Business Requirementsβ
Scenarioβ
Product Service receives many read requests for product details:
- 80% requests are GET operations.
- Same products are queried multiple times.
- Database queries take ~50-100ms.
Problem: Database bottleneck with high traffic.
Solution: Redis caching to:
- Reduce DB load.
- Improve response time (< 10ms from cache).
- Handle traffic spikes.
β Required Tasksβ
Task 5.1: Redis Setup (15 points)β
Docker Compose:
redis:
image: redis:7-alpine
container_name: redis-cache
ports:
- '6379:6379'
volumes:
- redis-data:/data
command: redis-server --appendonly yes --maxmemory 256mb --maxmemory-policy allkeys-lru
healthcheck:
test: ['CMD', 'redis-cli', 'ping']
interval: 10s
timeout: 5s
retries: 5
Configuration Options:
| Option | Value | Purpose |
|---|---|---|
appendonly yes | Enable AOF | Data persistence |
maxmemory 256mb | Memory limit | Prevent OOM |
maxmemory-policy allkeys-lru | Eviction policy | LRU when memory full |
Acceptance Criteria:
- Redis container healthy.
- redis-cli ping responds PONG.
- Persistence enabled.
Task 5.2: Cache Manager Implementation (25 points)β
Business Logic:
Create CacheManager class to handle all cache operations:
Cache Key Strategy:
product:{product_id} # Single product
products:list:{skip}:{limit} # Product list (paginated)
products:search:{query} # Search results
Implementation Skeleton:
# product-service/app/utils/cache.py
class CacheManager:
"""Redis cache manager with error handling"""
def __init__(self):
# TODO: Initialize Redis client with settings
pass
def get(self, key: str) -> Optional[Any]:
"""Get value from cache, return None on miss or error"""
pass
def set(self, key: str, value: Any, ttl: int = None) -> bool:
"""Set value with TTL, return success status"""
pass
def delete(self, key: str) -> bool:
"""Delete single key"""
pass
def delete_pattern(self, pattern: str) -> int:
"""Delete all keys matching pattern (e.g., 'products:list:*')"""
pass
def healthcheck(self) -> bool:
"""Return True if Redis is available"""
pass
Key Requirements:
- Error handling: Cache failures should not crash the application.
- Serialization: Use JSON for complex objects.
- TTL: Default from settings, can be overridden per call.
Acceptance Criteria:
- CacheManager with get/set/delete methods.
- Error handling does not crash application.
- TTL configurable.
- Pattern-based deletion.
Task 5.3: Cache-Aside Pattern for Get Product (30 points)β
Business Logic:
Cache-Aside (Lazy Loading) Pattern:
- Check cache first.
- If HIT β return cached data.
- If MISS β query DB β store in cache β return.
ββββββββββ 1. GET /products/1 βββββββββββββββββββ
β Client β ββββββββββββββββββββββββββΊβ Product Service β
ββββββββββ ββββββββββ¬βββββββββ
β
2. Check β Cache
βΌ
βββββββββββββββββββ
β Redis β
β product:1 β
ββββββββββ¬βββββββββ
β
βββββββββββββββββββ΄ββββββββββββββββββ
β β
3a. CACHE HIT 3b. CACHE MISS
β β
βΌ βΌ
Return cached βββββββββββββββββ
data immediately β PostgreSQL β
βββββββββ¬ββββββββ
β
4. Query DB
β
5. Store in cache
β
6. Return data
Service Implementation:
# product-service/app/services/product_service.py
class ProductService:
def get_product(self, product_id: int) -> Optional[Product]:
"""
Get product with cache-aside pattern
"""
cache_key = f"product:{product_id}"
# 1. Try cache first
cached = self.cache_manager.get(cache_key)
if cached:
logger.info(f"Cache HIT for product {product_id}")
return Product(**cached) # Deserialize
# 2. Cache miss - query DB
logger.info(f"Cache MISS for product {product_id}")
product = self.product_repository.get_by_id(product_id)
if product:
# 3. Store in cache
self.cache_manager.set(
cache_key,
product.to_dict(), # Serialize
ttl=settings.CACHE_TTL # 300 seconds default
)
return product
Acceptance Criteria:
- First request queries DB and caches data.
- Subsequent requests hit cache.
- Response time < 10ms for cache hits.
- Cache expires after TTL.
Task 5.4: Cache Invalidation (20 points)β
Business Logic:
Cache must be invalidated when data changes:
- Product updated β delete
product:{id} - Product deleted β delete
product:{id} - Stock changed β delete
product:{id}
Invalidation Strategies:
| Strategy | When | Implementation |
|---|---|---|
| Write-invalidate | Update/Delete | Delete cache key |
| TTL expiration | Auto | Redis handles |
| Pattern invalidation | Bulk updates | Delete matching keys |
Implementation in ProductService:
def update_product(self, product_id: int, data: ProductUpdate) -> Product:
"""Update product and invalidate cache"""
# 1. Update in DB
product = self.product_repository.update(product_id, data.dict())
# 2. Invalidate cache
self.cache_manager.delete(f"product:{product_id}")
# 3. Also invalidate list caches (they're now stale)
self.cache_manager.delete_pattern("products:list:*")
return product
def delete_product(self, product_id: int) -> bool:
"""Delete product and invalidate cache"""
# 1. Delete from DB
result = self.product_repository.delete(product_id)
# 2. Invalidate caches
self.cache_manager.delete(f"product:{product_id}")
self.cache_manager.delete_pattern("products:list:*")
return result
When Stock Changes (from Order Service event):
async def handle_stock_reserved(self, event_data: dict):
# ... deduct stock logic ...
# Invalidate cache
for item in items:
self.cache_manager.delete(f"product:{item['product_id']}")
Acceptance Criteria:
- Cache invalidated on update.
- Cache invalidated on delete.
- Cache invalidated on stock change.
- List caches also invalidated.
π Bonus Tasksβ
Bonus 5.1: User Profile Caching (15 points)β
Guideline: Apply caching for User Service's "Get User Profile" API. Implement cache-aside, invalidate on profile update. Track cache hit rate metrics.
Bonus 5.2: Cache Warming (10 points)β
Guideline: On service startup, pre-populate cache with frequently accessed products (top 100 by views). Query analytics or maintain a "hot products" list.
Bonus 5.3: Distributed Locking (15 points)β
Guideline: Implement distributed lock with Redis to prevent "thundering herd" - multiple requests querying DB simultaneously on cache miss. Use Redis SETNX for locking.
π Rubric - Assignment 05β
| Criteria | Excellent (100%) | Good (80%) | Satisfactory (60%) | Needs Improvement (40%) |
|---|---|---|---|---|
| Redis Setup (15pts) | Full config, persistence, healthcheck | Works, basic config | Container runs | Not running |
| Cache Manager (25pts) | All methods, error handling, patterns | Basic get/set/delete | Partial implementation | Not working |
| Cache-Aside (30pts) | Full pattern, logging, metrics | Works correctly | Basic caching | Broken |
| Invalidation (20pts) | All cases handled, patterns | Most cases | Basic invalidation | Missing |
| Performance (10pts) | Measured improvement, < 10ms | Improved response | Some improvement | No change |
Total: 100 points Bonus: 40 points
π Assignment 06: Observability - Tracing (Jaeger/OTel)β
Unit 6 - Observability (Tracing)β
π― Learning Objectivesβ
- Understand concepts: Trace, Span, Context Propagation.
- Setup OpenTelemetry instrumentation.
- Integrate Jaeger for distributed tracing.
- Debug cross-service requests.
π Business Requirementsβ
Scenarioβ
When order creation fails, we need to trace the request across services:
- How long does it take in each service?
- Which service failed?
- Which database query was slow?
Without tracing: Check logs of each service, difficult to correlate.
With tracing: Single view of the entire request journey.
β Required Tasksβ
Task 6.1: Jaeger Setup (15 points)β
Docker Compose:
jaeger:
image: jaegertracing/all-in-one:1.51
container_name: jaeger
ports:
- '5775:5775/udp' # Compact thrift (deprecated)
- '6831:6831/udp' # Compact thrift
- '6832:6832/udp' # Binary thrift
- '5778:5778' # Serve configs
- '16686:16686' # Jaeger UI
- '14268:14268' # Jaeger collector HTTP
- '14250:14250' # gRPC
- '9411:9411' # Zipkin compatible
- '4317:4317' # OTLP gRPC
- '4318:4318' # OTLP HTTP
environment:
- COLLECTOR_OTLP_ENABLED=true
- SPAN_STORAGE_TYPE=memory
Key Concepts:
| Concept | Description |
|---|---|
| Trace | End-to-end journey of a request |
| Span | Single operation in a trace (e.g., DB query, HTTP call) |
| Context | Metadata propagated across services (trace ID, span ID) |
| Instrumentation | Code that creates and manages spans |
Acceptance Criteria:
- Jaeger UI accessible at :16686.
- OTLP endpoint available at :4317.
Task 6.2: OpenTelemetry Setup (25 points)β
Dependencies:
opentelemetry-api
opentelemetry-sdk
opentelemetry-instrumentation-fastapi
opentelemetry-instrumentation-sqlalchemy
opentelemetry-instrumentation-httpx
opentelemetry-instrumentation-redis
opentelemetry-exporter-otlp
Tracing Setup Skeleton:
# app/utils/tracing.py
def setup_tracing(app, service_name: str, service_version: str = "1.0.0"):
"""
Configure OpenTelemetry tracing
Steps:
1. Create Resource with service identity (name, version, environment)
2. Create TracerProvider with resource
3. Create OTLPSpanExporter pointing to Jaeger (port 4317)
4. Add BatchSpanProcessor for performance
5. Set global tracer provider
6. Auto-instrument: FastAPI, SQLAlchemy, HTTPX, Redis
"""
pass
# Usage in main.py
setup_tracing(app, service_name="user-service", service_version="1.0.0")
Required Instrumentations per Service:
| Service | FastAPI | SQLAlchemy | HTTPX | Redis | RabbitMQ |
|---|---|---|---|---|---|
| User | β | β | - | - | Manual |
| Product | β | β | β | β | - |
| Order | β | β | β | - | Manual |
| Notification | β | - | - | - | Manual |
Acceptance Criteria:
- Tracing configured in all services.
- Auto-instrumentation for FastAPI, SQLAlchemy, HTTPX, Redis.
- Spans exported to Jaeger.
Task 6.3: Instrument All Services (30 points)β
Apply tracing to:
| Service | Instrumentation |
|---|---|
| User Service | FastAPI, SQLAlchemy |
| Product Service | FastAPI, SQLAlchemy, Redis, HTTPX |
| Order Service | FastAPI, SQLAlchemy, HTTPX, RabbitMQ (manual) |
| Notification Service | FastAPI, RabbitMQ (manual) |
Manual Span for RabbitMQ:
# For async message handling, create manual spans
from opentelemetry import trace
tracer = trace.get_tracer(__name__)
async def handle_order_created(self, message):
# Extract trace context from message headers (if available)
with tracer.start_as_current_span(
"handle_order_created",
kind=trace.SpanKind.CONSUMER,
attributes={
"messaging.system": "rabbitmq",
"messaging.operation": "process",
}
) as span:
data = json.loads(message.body)
span.set_attribute("order_id", data["order_id"])
try:
await self.process_order(data)
except Exception as e:
span.record_exception(e)
span.set_status(trace.StatusCode.ERROR)
raise
Acceptance Criteria:
- User Service traced.
- Product Service traced (including Redis).
- Order Service traced (including HTTPX calls).
- Manual spans for RabbitMQ handlers.
Task 6.4: Trace Analysis (20 points)β
Business Logic:
Test and analyze traces in Jaeger UI:
Test Scenarios:
- Happy Path: Create order successful
- Trace should show: Order Service β Product Service (validate) β DB insert β RabbitMQ publish
- Failure Path: Insufficient stock
- Trace should show: Order Service β Product Service β Error span
- Cache Hit vs Miss:
- Cache HIT: Short span, no DB call
- Cache MISS: DB span visible
Jaeger UI Features:
| Feature | Usage |
|---|---|
| Service dropdown | Filter by service |
| Operation dropdown | Filter by endpoint |
| Tags search | Find by order_id, user_id |
| Compare traces | Compare slow vs fast |
| Dependencies | Service dependency graph |
Acceptance Criteria:
- Can find traces by service.
- Can filter by operation.
- Can see full trace across services.
- Can identify slow spans.
π Bonus Tasksβ
Bonus 6.1: Custom Spans & Attributes (10 points)β
Guideline: Add custom spans for business-critical operations (e.g., stock validation, payment processing). Add custom attributes (order_id, product_count, total_amount).
Bonus 6.2: Trace Context in Messages (15 points)β
Guideline: Propagate trace context via RabbitMQ messages. Include traceparent header in message properties. Consumer extracts and continues trace.
Bonus 6.3: Sampling Strategy (10 points)β
Guideline: Configure trace sampling (not 100% in production). Implement probabilistic sampling (10%) or rate limiting sampling (100 traces/sec).
π Rubric - Assignment 06β
| Criteria | Excellent (100%) | Good (80%) | Satisfactory (60%) | Needs Improvement (40%) |
|---|---|---|---|---|
| Jaeger Setup (15pts) | UI works, OTLP enabled | Basic setup | Container runs | Not running |
| OTel Setup (25pts) | All instrumentations | Missing 1-2 | Basic FastAPI only | Not working |
| All Services (30pts) | All 4 services traced | 3 services | 2 services | 1 or none |
| Trace Analysis (20pts) | Full analysis, can debug | Basic viewing | Can find traces | Cannot use UI |
| Code Quality (10pts) | Clean, configurable | Good | Acceptable | Poor |
Total: 100 points Bonus: 35 points
π Assignment 07: Observability - Logging (Loki)β
Unit 7 - Observability (Logging)β
π― Learning Objectivesβ
- Understand centralized logging concepts.
- Setup Loki + Promtail for log aggregation.
- Implement structured logging.
- Query and analyze logs effectively.
π Business Requirementsβ
Scenarioβ
With multiple services, logs are scattered:
- User Service logs β container A
- Product Service logs β container B
- Order Service logs β container C
Problem: Difficult to correlate logs when debugging.
Solution: Centralized logging with Loki.
- All logs β Loki
- Query from Grafana
- Correlate with trace_id
β Required Tasksβ
Task 7.1: Loki + Promtail Setup (20 points)β
Loki Configuration:
# deployment/observability/loki/loki-config.yml
auth_enabled: false
server:
http_listen_port: 3100
common:
path_prefix: /loki
storage:
filesystem:
chunks_directory: /loki/chunks
rules_directory: /loki/rules
replication_factor: 1
ring:
kvstore:
store: inmemory
schema_config:
configs:
- from: 2020-10-24
store: boltdb-shipper
object_store: filesystem
schema: v11
index:
prefix: index_
period: 24h
Promtail Configuration:
# deployment/observability/promtail/promtail-config.yml
server:
http_listen_port: 9080
positions:
filename: /tmp/positions.yaml
clients:
- url: http://loki:3100/loki/api/v1/push
scrape_configs:
- job_name: docker
docker_sd_configs:
- host: unix:///var/run/docker.sock
refresh_interval: 5s
relabel_configs:
- source_labels: ['__meta_docker_container_name']
regex: '/(.*)'
target_label: 'container'
- source_labels: ['__meta_docker_container_label_service']
target_label: 'service'
Docker Compose:
loki:
image: grafana/loki:2.9.0
ports:
- '3100:3100'
volumes:
- ./deployment/observability/loki/loki-config.yml:/etc/loki/local-config.yaml
- loki-data:/loki
command: -config.file=/etc/loki/local-config.yaml
promtail:
image: grafana/promtail:2.9.0
volumes:
- ./deployment/observability/promtail/promtail-config.yml:/etc/promtail/config.yml
- /var/run/docker.sock:/var/run/docker.sock
- /var/lib/docker/containers:/var/lib/docker/containers:ro
command: -config.file=/etc/promtail/config.yml
Acceptance Criteria:
- Loki running at :3100.
- Promtail collecting Docker logs.
- Logs visible in Grafana.
Task 7.2: Structured Logging (25 points)β
Business Logic:
JSON structured logs for easy parsing and querying:
Log Format:
{
"timestamp": "2025-12-24T10:00:00.000Z",
"level": "INFO",
"service": "order-service",
"trace_id": "abc123...",
"span_id": "def456...",
"message": "Order created successfully",
"order_id": 1,
"user_id": 5,
"total_amount": 299.97
}
Python Logging Configuration Skeleton:
# app/config/logging.py
class JSONFormatter(logging.Formatter):
"""
JSON formatter with trace context
Output format:
{
"timestamp": "ISO8601",
"level": "INFO",
"service": "service-name",
"trace_id": "from OpenTelemetry",
"message": "log message",
...extra fields
}
"""
def format(self, record: logging.LogRecord) -> str:
# TODO: Get trace context from OpenTelemetry
# TODO: Build log_data dict
# TODO: Include extra fields from record
pass
def setup_logging(service_name: str):
"""Configure JSON logging to stdout"""
# TODO: Create handler with JSONFormatter
# TODO: Set root logger
# TODO: Reduce noise from uvicorn, httpx
pass
Usage Example:
logger = logging.getLogger(__name__)
logger.info("Order created", extra={"order_id": 1, "user_id": 5})
Acceptance Criteria:
- Logs in JSON format.
- trace_id included (for correlation).
- Custom fields supported.
- Log levels appropriate.
Task 7.3: Log Aggregation Verification (25 points)β
Business Logic:
Verify logs from all services are collected and queryable.
Test Scenarios:
- All Services Logging:
- Start all services.
- Make API requests.
- Verify logs appear in Loki.
- Cross-Service Correlation:
- Create order (hits multiple services).
- Find logs by trace_id.
- Should see logs from Order, Product, User services.
- Error Logging:
- Trigger error (invalid product ID).
- Search for ERROR level logs.
- Verify stack trace present.
Grafana Loki Queries (LogQL):
# All logs from order-service
{container="order-service"}
# Error logs only
{container=~".*-service"} |= "ERROR"
# Filter by trace_id
{container=~".*-service"} | json | trace_id="abc123..."
# Search for specific order
{container="order-service"} | json | order_id=1
# Rate of errors in last hour
rate({container="order-service"} |= "ERROR" [1h])
Acceptance Criteria:
- Logs from all services visible.
- Can filter by service.
- Can search by trace_id.
- Can search by custom fields.
Task 7.4: Log-based Alerting (20 points)β
Business Logic:
Alert when error rate exceeds threshold.
Grafana Alert Rule:
# Alert when >10 errors in 5 minutes
name: High Error Rate
condition: count > 10
query: |
count_over_time(
{container=~".*-service"} |= "ERROR" [5m]
)
for: 1m
annotations:
summary: 'High error rate detected'
description: 'More than 10 errors in the last 5 minutes'
Acceptance Criteria:
- Alert rule created in Grafana.
- Alert fires on error spike (test).
- Alert resolves when errors decrease.
π Bonus Tasksβ
Bonus 7.1: ELK Stack Alternative (20 points)β
Guideline: Setup ELK Stack (Elasticsearch, Logstash, Kibana) instead of Loki. Configure Filebeat to ship logs. Create Kibana dashboard.
Bonus 7.2: Log Retention Policy (10 points)β
Guideline: Configure log retention: keep 7 days detailed, 30 days aggregated. Implement log rotation. Calculate storage requirements.
Bonus 7.3: Audit Logging (10 points)β
Guideline: Separate audit logs for security events (login, permission changes). Store in separate index/stream. Never delete.
π Rubric - Assignment 07β
| Criteria | Excellent (100%) | Good (80%) | Satisfactory (60%) | Needs Improvement (40%) |
|---|---|---|---|---|
| Loki Setup (20pts) | Full config, Promtail works | Basic setup | Container runs | Not running |
| Structured Logging (25pts) | JSON, trace_id, custom fields | JSON, missing trace_id | Basic JSON | Unstructured |
| Aggregation (25pts) | All services, can query | Most services | Some logs visible | Cannot query |
| Alerting (20pts) | Alert works, tested | Alert configured | Partial | No alerting |
| Code Quality (10pts) | Clean, reusable | Good | Acceptable | Poor |
Total: 100 points Bonus: 40 points
π Assignment 08: Observability - Metrics (Prometheus/Grafana)β
Unit 9 - Observability (Metrics)β
π― Learning Objectivesβ
- Understand metrics types: Counter, Gauge, Histogram, Summary.
- Setup Prometheus for metrics collection.
- Build Grafana dashboards.
- Implement custom application metrics.
- Setup alerting based on metrics.
π Business Requirementsβ
Scenarioβ
The Operations team needs to monitor the system:
- Request rate (requests per second).
- Error rate (% of failed requests).
- Response latency (p50, p90, p99).
- Resource usage (CPU, memory).
- Business metrics (orders/hour, revenue).
Goal: Real-time visibility into system health.
β Required Tasksβ
Task 8.1: Prometheus Setup (20 points)β
Prometheus Configuration:
# deployment/observability/prometheus/prometheus.yml
global:
scrape_interval: 15s
evaluation_interval: 15s
alerting:
alertmanagers:
- static_configs:
- targets: []
rule_files:
- /etc/prometheus/rules/*.yml
scrape_configs:
# Prometheus self-monitoring
- job_name: 'prometheus'
static_configs:
- targets: ['localhost:9090']
# User Service
- job_name: 'user-service'
static_configs:
- targets: ['user-service:8001']
metrics_path: /metrics
# Product Service
- job_name: 'product-service'
static_configs:
- targets: ['product-service:8002']
metrics_path: /metrics
# Order Service
- job_name: 'order-service'
static_configs:
- targets: ['order-service:8003']
metrics_path: /metrics
# Redis
- job_name: 'redis'
static_configs:
- targets: ['redis-exporter:9121']
# PostgreSQL
- job_name: 'postgres'
static_configs:
- targets: ['postgres-exporter:9187']
Docker Compose:
prometheus:
image: prom/prometheus:v2.47.0
container_name: prometheus
ports:
- '9090:9090'
volumes:
- ./deployment/observability/prometheus/prometheus.yml:/etc/prometheus/prometheus.yml
- ./deployment/observability/prometheus/rules:/etc/prometheus/rules
- prometheus-data:/prometheus
command:
- '--config.file=/etc/prometheus/prometheus.yml'
- '--storage.tsdb.path=/prometheus'
- '--web.enable-lifecycle'
healthcheck:
test: ['CMD', 'wget', '--spider', '-q', 'localhost:9090/-/healthy']
Acceptance Criteria:
- Prometheus UI accessible at :9090.
- All services discovered in Targets.
- Metrics being scraped.
Task 8.2: Application Metrics (25 points)β
Metrics Types:
| Type | Use Case | Example |
|---|---|---|
| Counter | Monotonically increasing | Total requests, errors |
| Gauge | Can go up/down | Active connections, queue size |
| Histogram | Distribution of values | Request latency, order value |
| Summary | Similar to histogram | Response size percentiles |
Implementation with prometheus-fastapi-instrumentator:
# app/utils/metrics.py
def setup_metrics(app, service_name: str):
"""
Setup Prometheus metrics for FastAPI
1. Create Instrumentator with config:
- excluded_handlers: ["/metrics", "/health"]
- should_group_status_codes: False
2. Add default metrics (latency, requests)
3. Expose /metrics endpoint
"""
pass
# Custom business metrics to define:
# - orders_created_total (Counter with labels: status)
# - order_value_dollars (Histogram with buckets)
# - http_requests_in_progress (Gauge)
Recording Business Metrics:
# In OrderService - increment counter on order creation
ORDERS_CREATED.labels(status="success").inc()
ORDER_VALUE.observe(float(order.total_amount))
Acceptance Criteria:
- /metrics endpoint returns Prometheus format.
- Default HTTP metrics (latency, count).
- Custom business metrics recording.
Task 8.3: Grafana Dashboards (30 points)β
Grafana Setup:
grafana:
image: grafana/grafana:10.1.0
container_name: grafana
ports:
- '3000:3000'
environment:
- GF_SECURITY_ADMIN_USER=admin
- GF_SECURITY_ADMIN_PASSWORD=admin123
- GF_USERS_ALLOW_SIGN_UP=false
volumes:
- ./deployment/observability/grafana/provisioning:/etc/grafana/provisioning
- ./deployment/observability/grafana/dashboards:/var/lib/grafana/dashboards
- grafana-data:/var/lib/grafana
Dashboard Provisioning:
# deployment/observability/grafana/provisioning/dashboards/default.yml
apiVersion: 1
providers:
- name: 'default'
orgId: 1
folder: ''
type: file
disableDeletion: false
editable: true
options:
path: /var/lib/grafana/dashboards
Required Dashboard Panels:
| Panel | PromQL Query | Type |
|---|---|---|
| Request Rate | sum(rate(http_requests_total[5m])) by (service) | Graph |
| Error Rate | rate(http_requests_total{status=~"5.."}[5m]) / rate(http_requests_total[5m]) * 100 | Graph |
| P99 Latency | histogram_quantile(0.99, http_request_duration_seconds_bucket) | Graph |
| Active Requests | http_requests_in_progress | Gauge |
| Orders/Hour | increase(orders_created_total[1h]) | Stat |
Acceptance Criteria:
- Grafana UI accessible at :3000.
- Prometheus data source configured.
- Overview dashboard with key metrics.
- Service-specific dashboard.
Task 8.4: Alerting Rules (15 points)β
Required Alert Rules:
| Alert | Condition | Severity |
|---|---|---|
| HighErrorRate | Error rate > 5% for 5min | critical |
| HighLatency | P99 > 2s for 5min | warning |
| ServiceDown | up == 0 for 1min | critical |
| HighMemoryUsage | Memory > 512MB for 5min | warning |
Alert Rule File Location: deployment/observability/prometheus/rules/alerts.yml
Acceptance Criteria:
- Alert rules loaded in Prometheus.
- Alerts fire correctly when triggered.
- Alerts visible in Grafana.
π Bonus Tasksβ
Bonus 8.1: Custom Grafana Dashboard (15 points)β
Guideline: Create business dashboard showing: orders per hour, revenue, top products, user signups. Use variables for time range selection.
Bonus 8.2: Alertmanager Integration (15 points)β
Guideline: Setup Alertmanager to route alerts via Slack/Email/Discord. Configure silencing and inhibition rules.
Bonus 8.3: SLI/SLO Dashboard (15 points)β
Guideline: Define SLOs (99.9% availability, p99 < 500ms). Create error budget dashboard. Track SLO compliance over time.
π Rubric - Assignment 08β
| Criteria | Excellent (100%) | Good (80%) | Satisfactory (60%) | Needs Improvement (40%) |
|---|---|---|---|---|
| Prometheus Setup (20pts) | Full config, all targets | Most targets | Basic setup | Not working |
| App Metrics (25pts) | Default + custom business | Default metrics | Partial | No metrics |
| Dashboards (30pts) | Multiple dashboards, polished | Overview dashboard | Basic panels | No dashboards |
| Alerting (15pts) | Rules work, tested | Rules configured | Partial | No alerts |
| Code Quality (10pts) | Clean, documented | Good | Acceptable | Poor |
Total: 100 points Bonus: 45 points
π Final Project: End-to-End Integration & Demoβ
π― Objectivesβ
- Deploy the entire microservices stack.
- Demonstrate the E2E user journey.
- Verify observability stack.
- Present architecture decisions.
π Final Project Requirementsβ
Part 1: Complete Deployment (30 points)β
Checklist:
| Component | Requirement |
|---|---|
| User Service | Running, healthy |
| Product Service | Running, cached |
| Order Service | Running, messaging |
| Notification Service | Consuming messages |
| PostgreSQL (3 instances) | Data persisted |
| Redis | Caching working |
| RabbitMQ | Queues active |
| Prometheus | Scraping all services |
| Grafana | Dashboards visible |
| Loki | Logs aggregated |
| Jaeger | Traces visible |
Verification Commands:
# Check all containers
docker compose ps
# Check service health
curl http://localhost:8001/health
curl http://localhost:8002/health
curl http://localhost:8003/health
# Check Prometheus targets
curl http://localhost:9090/api/v1/targets
# Check RabbitMQ queues
curl -u guest:guest http://localhost:15672/api/queues
Part 2: E2E User Journey Demo (40 points)β
Scenario Script:
# 1. Register User
curl -X POST http://localhost:8001/register \
-H "Content-Type: application/json" \
-d '{"username": "demo_user", "password": "Demo@123"}'
# 2. Login
TOKEN=$(curl -s -X POST http://localhost:8001/login \
-H "Content-Type: application/x-www-form-urlencoded" \
-d "username=demo_user&password=Demo@123" | jq -r '.access_token')
# 3. Browse Products (should hit cache after first request)
curl http://localhost:8002/products
curl http://localhost:8002/products # Cache hit
# 4. Create Product (if admin)
curl -X POST http://localhost:8002/products \
-H "Authorization: Bearer $TOKEN" \
-H "Content-Type: application/json" \
-d '{"name": "Demo Product", "price": 99.99, "quantity": 100}'
# 5. Create Order
curl -X POST http://localhost:8003/orders \
-H "Authorization: Bearer $TOKEN" \
-H "Content-Type: application/json" \
-d '{"items": [{"product_id": 1, "quantity": 2}]}'
# 6. Check Notification Service logs (should show order processed)
docker compose logs notification-service | grep "Processing order"
Demo Requirements:
| Step | Verification |
|---|---|
| Registration | User created, 201 response |
| Login | Token returned |
| Browse Products | Products listed, cache metrics change |
| Create Order | Order created, stock reduced |
| Notification | Consumer log shows processing |
| Tracing | Full trace visible in Jaeger |
| Metrics | Dashboard shows request spike |
| Logs | Logs searchable by trace_id |
Part 3: Failure Scenarios (20 points)β
Test Resilience:
| Scenario | Expected Behavior |
|---|---|
| Kill Product Service | Order Service returns 503, auto-recovers |
| Kill Redis | Product Service fallback to DB (slower) |
| Kill RabbitMQ | Order created, notification delayed |
| Invalid Token | 401 Unauthorized |
| Insufficient Stock | Order fails gracefully |
Demo:
# Test: Kill Product Service
docker compose stop product-service
# Order should fail gracefully
curl -X POST http://localhost:8003/orders \
-H "Authorization: Bearer $TOKEN" \
-d '{"items": [{"product_id": 1, "quantity": 1}]}'
# Expected: 503 Service Unavailable
# Restart
docker compose start product-service
# Order should work now
curl -X POST http://localhost:8003/orders \
-H "Authorization: Bearer $TOKEN" \
-d '{"items": [{"product_id": 1, "quantity": 1}]}'
# Expected: 201 Created
Part 4: Architecture Presentation (10 points)β
Present:
- Architecture Overview
- Service boundaries.
- Communication patterns.
- Database per service.
- Key Design Decisions
- Why separate Auth service?
- Why RabbitMQ for notifications?
- Why Redis for products (not orders)?
- Trade-offs
- Consistency vs Availability.
- Complexity vs Simplicity.
- Performance vs Cost.
- Future Improvements
- API Gateway (Kong).
- Service Mesh (Istio).
- CI/CD Pipeline.
π Final Project Rubricβ
| Criteria | Excellent (100%) | Good (80%) | Satisfactory (60%) | Needs Improvement (40%) |
|---|---|---|---|---|
| Deployment (30pts) | All components running | Missing 1-2 | Core services only | Multiple failures |
| E2E Demo (40pts) | Full journey, all verifications | Most steps work | Core flow works | Broken flow |
| Failure Handling (20pts) | Graceful degradation | Most scenarios | Some handling | Crashes |
| Presentation (10pts) | Clear, insightful | Good explanation | Basic | Poor |
Total: 100 points
π Course Summary & Gradingβ
Overall Progress Trackingβ
| Assignment | Topic | Points | Bonus |
|---|---|---|---|
| 01 | User Service Extraction | 100 | 30 |
| 02 | Product Service & API Gateway | 100 | 40 |
| 03 | Async Communication | 100 | 30 |
| 04 | SAGA Pattern - Order Service | 100 | 45 |
| 05 | Performance - Redis Caching | 100 | 40 |
| 06 | Observability - Tracing | 100 | 35 |
| 07 | Observability - Logging | 100 | 40 |
| 08 | Observability - Metrics | 100 | 45 |
| Final | E2E Integration | 100 | - |
| Total | 900 | 305 |
Grading Scaleβ
| Grade | Percentage | Points (of 900) |
|---|---|---|
| A+ | 95-100% | 855-900 |
| A | 90-94% | 810-854 |
| B+ | 85-89% | 765-809 |
| B | 80-84% | 720-764 |
| C+ | 75-79% | 675-719 |
| C | 70-74% | 630-674 |
| D | 60-69% | 540-629 |
| F | < 60% | < 540 |
Bonus Points: Can increase grade (max +10%)
π Additional Resourcesβ
Documentationβ
- FastAPI Documentation
- SQLAlchemy 2.0 Documentation
- RabbitMQ Tutorials
- Redis Documentation
- OpenTelemetry Python
- Prometheus Documentation
- Grafana Documentation
Architecture Patternsβ
Video Coursesβ
Last Updated: 2025-12-24 Version: 1.0.0 Syllabus Reference: Advanced Microservices Architecture