Skip to main content

Observability: Logging, Tracing & Metrics - Slides

SLIDE DECK: MODULE 06 - OBSERVABILITY - V1.1​

Total Duration: 540 minutes (210' Tracing + 210' Logging + 120' Metrics) Audience: Fresher/Junior who have completed Module 04 (SAGA) and 06 (Caching).

Slide 1: Title Page

  • Content:
    • (Company / Training Unit Logo)
    • MODULE 05: OBSERVABILITY
    • From "Flying Blind" to "Mastering" the System
    • Tools: OTel, Jaeger, ELK Stack, Prometheus, Grafana
    • Trainer: (Your Name)
    • Date: (Training Date)
  • Visualization:
    • Key visual: A
    • * The left half is a dark airplane cockpit with no instruments (Flying Blind).
    • * The right half is a brightly lit cockpit with glowing screens (Dashboards, Traces, Logs) (Mastering).
  • Instructor Script:
    • "Welcome to Module 06. In the previous modules, we built a system that is 'Safe' (with SAGA) and 'Fast' (with Redis Cache). But we are 'Flying Blind'."
    • "When a user reports 'Order failed,' what do we do? Open the Order Service logs? Or Payment Service? Or Inventory Service? We don't know where the error is, we don't know where the slowness is."
    • "Today, we're going to 'turn on the lights' in the cockpit. This module will teach you how to 'see through' a complex microservice system using the 3 Pillars of 'Observability'."

Slide 2: Prerequisites & System Requirements

  • Content:
    • REQUIREMENTS BEFORE WE START
    • 1. Required Knowledge:
      • ✅ Completed Module 03 (RabbitMQ), 04 (SAGA), 06 (Redis Cache)
      • ✅ Understand the current microservice architecture (Kong → Order → Payment → Inventory)
      • ✅ Basic Python/FastAPI
      • ✅ Docker & Docker Compose
    • 2. Development Environment:
      • 💻 Docker Desktop (≥16GB RAM recommended)
      • 💻 VS Code + Extensions (Python, Docker)
      • 💻 Postman or curl (for testing APIs)
    • 3. Current System:
      • Client → [Kong] → [Order] → (RabbitMQ) → [Payment] → (RabbitMQ) → [Inventory]
      • [Redis Cache] + [PostgreSQL DBs]
      • Problem: No visibility, don't know where errors are, or where slowness is.
  • Visualization:
    • A full architecture diagram of the current system (from previous Modules).
    • A checklist with [✅] and [💻] icons.
  • Instructor Script:
    • "Before we start, make sure you've completed Modules 03, 04, and 06. We are going to 'upgrade' that system."
    • "You will need Docker with at least 16GB of RAM. Why? Because we will be running 8-10 containers (3 services + Kong + RabbitMQ + Redis + Jaeger + ELK + Prometheus + Grafana)."
    • "[Point to Diagram] This is the system we have. It works, but it's a 'black box'. This module will 'turn on the lights'."

Slide 3: Session Agenda (Agenda)

  • Content:
    • AGENDA (3 PARTS)
    • P0. Prerequisites & Why Observability Matters
    • P1. The Problem: "Flying Blind" & Intro to the 3 Pillars of Observability
    • P2. Pillar #1: Tracing (The "Where")
      • Concepts: Trace, Span, Context Propagation
      • Upgrade (Production): OTel Conventions, Collector & Sampling
      • Troubleshooting Common Issues
    • P3. Pillar #2: Logging (The "Why")
      • Concepts: Centralized Logging & ELK Stack
      • Upgrade (Production): Structured Logging (JSON + trace_id), Cost & Hygiene (ILM, PII)
      • Troubleshooting Common Issues
    • P4. Pillar #3: Metrics (The "What")
      • Concepts: Prometheus/Grafana, 4 Golden Signals
      • Upgrade (Production): RED/USE Method, Alerting & Runbooks
      • Troubleshooting Common Issues
    • P5. "The Climax": E2E Debugging (Metrics -> Traces -> Logs)
      • Upgrade (Production): The Golden Connection (Exemplars: Metrics ↔ Traces)
    • P6. Best Practices (O11y-as-Code) & Tool Selection
    • P7. Testing Your O11y Setup (Verification Checklist)
    • P8. Lab Prep (Assignments 06, 07, 08)
  • Visualization:
    • An 8-step diagram (adding P0, P7), where P2, P3, P4 are shown as three parallel pillars. P5 (E2E Debugging) is the 'roof' connecting all three.
  • Instructor Script:
    • "This is our optimized agenda for Freshers/Juniors."
    • "We've added a Prerequisites section (P0) to ensure everyone is ready."
    • "More importantly, we have 'Troubleshooting Guides' after each pillar (P2, P3, P4) - this is the part you'll need most when you get stuck."
    • "And P7 is a 'Testing Checklist' - how to verify if your O11y setup is working correctly."

Slide 4: Learning Objectives

  • Content:
    • OBJECTIVES (AFTER THIS MODULE, YOU WILL BE ABLE TO...)
    • 1. Explain: The 3 Pillars (Logs, Metrics, Traces) and the role of each.
    • 2. Implement: Integrate OpenTelemetry (OTel) into FastAPI (with OTel Collector, Conventions) [SME Review] and view on Jaeger.
    • 3. Implement: Install the ELK Stack, push Structured Logs (JSON + trace_id) [SME Review], and understand ILM (Cost). [SME Review]
    • 4. Implement: Instrument services with Prometheus, build a Dashboard (RED/USE) [SME Review] on Grafana, and write Alert Rules. [SME Review]
    • 5. Analyze: Use a combination of all 3 tools (Metrics -> Exemplars -> Traces -> Logs) [SME Review] to debug a scenario.
  • Visualization:
    • 5 icons: [Icon: 3 Pillars (Explain)], [Icon: Jaeger/Trace (Implement)], [Icon: Kibana/Log (Implement)], [Icon: Grafana/Metric (Implement)], [Icon: Detective/Debug (Analyze)].
  • Instructor Script:
    • "This is our commitment. After this module, you won't just know how to 'install,' you'll know how to 'configure for production'."
    • "You will personally 'implement' (Objectives 2, 3, 4) all three industry-leading monitoring systems, but with 'advanced' standards: OTel Collector, JSON log + trace_id, and Prometheus Alerts. [SME Review]"
    • "And most importantly, Objective 5: 'Analyze.' You will learn the 'debugging' workflow of a real SRE (Site Reliability Engineer): Use Metrics to 'detect,' 'Exemplars' [SME Review] to 'jump' to Tracing to 'isolate,' and Logging to 'eliminate' the error."

Slide 5: Recap: The "Black Box" System

  • Content:
    • RECAP: OUR "BLACK BOX" SYSTEM
    • We have built a complex system:
    • Client -> [Kong] -> [Order] -> (MQ) -> [Payment] -> (MQ) -> [Inventory]
    • (All reading/writing from [Redis Cache] and [PostgreSQL DBs])
    • Problem: It's "Safe" (SAGA) and "Fast" (Cache), but it's a "BLACK BOX."
  • Visualization:
      • A full architecture diagram of our system (Kong, 3 services, RabbitMQ, Redis, DBs).
    • This entire diagram is placed inside a large [Icon: Black Box].
  • Instructor Script:
    • "This is the system we've built. It looks 'cool.' But it's a 'black box.' We have no idea what's happening inside. We need 'X-rays' to 'see' into it."

Slide 6: P1 - The Problem: "Flying Blind"

  • Content:
    • THE PROBLEM: "USER REPORTS AN ERROR!"
    • User A: "My GET /products/1 API is too slow (5 seconds)!"
      • Your Question: Where is it slow? Kong (Gateway)? The Service? Or the DB?
    • User B: "I placed an order (Order abc-123) it failed, but I was still charged!"
      • Your Question: This is a SAGA failure. Payment succeeded, but Inventory failed. How do I find the logs for Order abc-123 across 3 different services?
    • The Core Problem:
      • Log Silos: Logs are 'fragmented' on 5 servers, impossible to 'stitch' together.
      • No Context: There is no common 'identity' (trace_id) for a single request.
  • Visualization:
      • An image: [User A (Angry)] and [User B (Angry)].
    • They are 'pointing' at an [Engineer (Frustrated)] who is sitting between 5 'terminal screens' (each a different service log) with text scrolling by rapidly.
  • Instructor Script:
    • "This is reality. User A reports 'slow.' User B reports 'lost money.' [Point to Engineer] And this is us, 'drowning' in 5 different log windows, trying to 'copy-paste' an order_id to 'stitch' the story together."
    • "We are 'flying blind.' We need a cockpit."

Slide 7: Why Observability? (Context)

  • Content:
    • WHY DO WE NEED OBSERVABILITY?
    • Monolithic (The Old Days):
      • 1 single app → 1 log file → grep is all you need.
      • Debug: Set a breakpoint, step through the code.
    • Microservices (Today):
      • 5-10 services → 5-10 log files → Impossible to grep.
      • 1 request 'jumps' through multiple services → Cannot step-through debug.
    • Conclusion:
      • Monitoring ≠ Observability
        • Monitoring: "Knowing" if the system is failing (Alerts). Pre-defined questions: "Is CPU > 80%?"
        • Observability: "Understanding" why it's failing (Investigate). Arbitrary questions: "Where is request abc-123 slow?"
      • Observability = Monitoring + Context + Correlation.
  • Visualization:
    • A 2-column comparison table:
      • Column 1 (Monolithic): [Icon: 1 large block] → 1 log file → [Icon: Simple]
      • Column 2 (Microservices): [Icon: Many small blocks] → Many log files → [Icon: Complex]
    • A Venn diagram:
      • [Monitoring] + [Context] + [Correlation] = [Observability]
  • Instructor Script:
    • "Before we dive in, let's understand 'Why' we need Observability."
    • "[Point to Monolithic] In the old days, 1 monolithic app, 1 log file. You grep or tail -f and you're done. Debugging? Set a breakpoint."
    • "[Point to Microservices] Today, 5 services, 5 log files. 1 request 'flies' through all 5. You can't grep each one. You can't set a breakpoint across 5 services."
    • "[EMPHASIZE] Monitoring (CPU, Memory) just tells you 'IT'S BROKEN.' Observability tells you 'WHY IT'S BROKEN' and 'WHERE.' That's the difference."

Slide 8: P1 (Continued) - The Solution: 3 Pillars of Observability

  • Content:
    • THE SOLUTION: THE 3 PILLARS
    • "Turning on the lights" in the cockpit.
    • 1. METRICS (The "What")
      • What: Aggregated Numbers. The "Dashboard."
      • Answers: "What is the overall health?" (e.g., CPU 80%, Error Rate 5%, p99 Latency 2s).
      • Tools: Prometheus & Grafana.
    • 2. TRACING (The "Where")
      • What: A "Single Request Story." The "Flight Map."
      • Answers: "Where did request 'abc' go and where was it slow?" (e.g., Order (5ms) -> Payment (4900ms) -> ...).
      • Tools: OpenTelemetry & Jaeger.
    • 3. LOGGING (The "Why")
      • What: Detailed Events. The "Flight Recorder (Black Box)."
      • Answers: "Why was Payment Service slow for 4900ms?" (e.g., ERROR: Credit card processor timed out).
      • Tools: ELK Stack (Kibana).
  • Visualization:
      • A clear 3-pillar (or 3-circle Venn) diagram:
      • METRICS: [Icon: Gauge] - "What's wrong?"
      • TRACING: [Icon: Waterfall Diagram] - "Where is it wrong?"
      • LOGGING: [Icon: Magnifying Glass] - "Why is it wrong?"
  • Instructor Script:
    • "The solution is the 3 Pillars. You must remember these 3 questions."
    • "1. Metrics (Prometheus) is the 'Dashboard.' It shows the 'Red Light!' (The What). e.g., 'Error Rate is 5%'."
    • "2. Tracing (Jaeger) is the 'Map.' It says 'The failure is in the Payment Service' (The Where)."
    • "3. Logging (Kibana) is the 'Black Box.' It says 'It failed because the credit card expired' (The Why)."
    • "We will dive deep into each one, starting with Tracing."

Slide 9: Section Intro - Tracing

  • Content:
    • (Based on fsa-04-section-intro.html template)
    • 01. TRACING (THE "WHERE")
    • Pillar #1: Retelling the Story of a Request (The X-Ray)
  • Visualization:
    • (Based on template)
    • Background image (slide-04.jpg).
    • Title 01. TRACING (THE "WHERE") (text-5xl).
    • Subtitle Pillar #1: Retelling the Story of a Request (The X-Ray) (text-3xl).
  • Instructor Script:
    • "We begin with the first pillar: Distributed Tracing. This is the 'X-ray' that lets us 'see through' the black box, to answer the question 'Where did this request go, and where was it slow?'."

Slide 10: P2 - Pillar #1: Distributed Tracing (The "Where")

  • Content:
    • PILLAR #1: DISTRIBUTED TRACING
    • "Retelling the Story of a Request"
    • Concepts:
      • Trace: The entire "journey" of 1 request through multiple services.
      • Span: One "step" in that journey (e.g., 1 API call, 1 DB query).
      • A Trace is a collection of parent-child Spans.
    • Tools:
      • OpenTelemetry (OTel): The GOLD standard (API/SDK) to collect Traces (and Metrics, Logs).
      • Jaeger: The "Backend" (database) and "UI" to store and visualize those Traces.
  • Visualization:
      • A beautiful "waterfall" diagram from the Jaeger UI:
      • Trace 123
      • Span A (Order Service) (0ms -> 500ms)
      • ... Span B (Payment Service) (50ms -> 450ms) (Child of A)
      • ... ... Span C (DB Query) (100ms -> 300ms) (Child of B)
  • Instructor Script:
    • "Tracing is the first pillar. [Point to Visualization] This is the 'story' of one request, shown as a 'waterfall'."
    • "This entire thing is 1 'Trace'. Each 'horizontal bar' is 1 'Span'. You can immediately see that Span A called Span B, and Span B called Span C."
    • "We use 'OpenTelemetry' (OTel) to 'create' this data, and 'Jaeger' to 'draw' this beautiful graph."

Slide 11: Why Choose OpenTelemetry & Jaeger?

  • Content:
    • WHY CHOOSE OPENTELEMETRY & JAEGER?
    • History (Context):
      • OpenTracing (2016): Only Tracing, CNCF project.
      • OpenCensus (2018): Metrics + Tracing, from Google.
      • OpenTelemetry (2019): Merged both → The CNCF standard for Traces, Metrics, Logs.
    • Why OpenTelemetry?
      • ✅ Vendor-neutral: Can export to Jaeger, Zipkin, Datadog, New Relic...
      • ✅ Single SDK: 1 library for Traces, Metrics, AND Logs.
      • ✅ Auto-instrumentation: Automatically 'wraps' FastAPI, Django, Flask...
      • ✅ Future-proof: The official CNCF (Cloud Native Computing Foundation) standard.
    • Why Jaeger?
      • ✅ Open-source & Free (perfect for learning).
      • ✅ OTel native: Designed to work with OTel.
      • ✅ Great UI: Intuitive waterfall view.
      • 🔄 Alternatives: Zipkin (older), Tempo (newer, fewer features).
  • Visualization:
    • A historical timeline: [2016: OpenTracing] → [2018: OpenCensus] → [2019: OpenTelemetry (Merged)]
    • A diagram: [App (OTel SDK)] → (can export to) → [Jaeger], [Zipkin], [Datadog], [New Relic]...
  • Instructor Script:
    • "Before we code, let's understand 'Why' we chose OTel and Jaeger."
    • "OpenTelemetry is the result of 'merging' OpenTracing and OpenCensus. It is the 'gold standard' today."
    • "The key benefit: 'Vendor-neutral.' [Point to diagram] You use the OTel SDK, and you can 'export' to Jaeger (free) or Datadog (paid) without changing your code. This is 'freedom'."
    • "And Jaeger? Because it's free, looks great, and was 'born' to work with OTel. It's all you need to learn and work."

Slide 12: Tracing - The "Magic": Context Propagation

  • Content:
    • HOW DOES THE "MAGIC" WORK?
    • Solution: Context Propagation
    • The Process:
      1. OTel (Middleware) creates a trace_id: "abc" (W3C TraceContext).
      2. When Order Service calls Payment Service (HTTP):
        • Producer (Order): inject(headers) -> Automatically adds traceparent header.
        • Consumer (Payment): extract(headers) -> Automatically reads the header.
      3. When Order Service sends a Message (RabbitMQ): [SME Review]
        • Producer (Order): inject(headers) -> Automatically adds traceparent to Message Headers. [SME Review]
        • Consumer (Payment): ctx = extract(msg_headers) -> Restores the context. [SME Review]
  • Visualization:
      • A 3-service diagram, showing the W3C TraceContext:
    • [Kong] -> [Order Svc] (Creates trace_id: abc)
    • Arrow from Order -> Payment: HTTP GET /pay (Headers: { traceparent: '00-abc-...' })
    • Arrow from Order -> Inventory: RabbitMQ Msg (Properties Headers: { traceparent: '00-abc-...' }) [SME Review]
  • Instructor Script:
    • "So what's the 'magic' that links them? It's 'Context Propagation'."
    • "It's simple: [Point to Diagram] When the 'root' request comes in, OTel creates a trace_id (using the W3C standard)."
    • "When Order Service calls Payment Service (HTTP), OTel injects the traceparent header. The OTel on Payment Service extracts it."
    • "[EMPHASIZE] And OTel is 'smart' enough to work over RabbitMQ too. [SME Review] It 'injects' the traceparent into the RabbitMQ 'Message Headers'. [SME Review] The Consumer reads it and 'stitches' the trace back together. [SME Review] This is 'automatic magic'."

Slide 13: Tracing - "Hard Part" #1: OTel Conventions

  • Content:
    • "HARD PART" #1: CONVENTIONS & RESOURCES [SME Review]
    • Problem: 1000 services are sending traces. How do you tell them apart?
    • 1. Resource Attributes: [SME Review]
      • The "identity" of your service. Must be set (via Environment Variables).
      • OTEL_RESOURCE_ATTRIBUTES= service.name=order-service, service.version=1.2.3, deployment.environment=production
    • 2. Semantic Conventions: [SME Review]
      • How to name your Spans. Don't name it "my-span".
      • Span Naming: http.method (GET, POST), db.statement (SELECT), messaging.operation (receive). (OTel instrumentation does this automatically).
      • Attributes: Add "metadata" to the Span for searching.
        • http.status_code = 200
        • (Important): saga.id = "abc-123" (Attach sagaId to the Span)
      • Status: [SME Review]
        • span.set_status(StatusCode.ERROR, "Error message")
        • span.record_exception(exception)
  • Visualization:
      • A screenshot of a Span in Jaeger, highlighting the "Tags" (Attributes) like service.name, http.status_code, deployment.environment. [SME Review]
  • Instructor Script:
    • "This is the first 'hard part' of Tracing. [SME Review] If you don't 'standardize,' your 1000 services will send 'junk' traces."
    • "One: 'Resource Attributes.' [SME Review] You must set these 3 environment variables. This lets Jaeger know this trace came from order-service version 1.2.3 in the production environment. [SME Review]"
    • "Two: 'Semantic Conventions.' [SME Review] Don't make up Span names. Follow the standard (OTel instrumentation does this for you). More importantly, 'stuff' business context into the Span, for example, 'stuff' the saga.id into the Span [SME Review] so you can search for it. And when your code errors, span.set_status(ERROR) [SME Review] so the Span 'turns red'."

Slide 14: Tracing - "Hard Part" #2: OTel Collector & Sampling

  • Content:
    • "HARD PART" #2: OTel Collector & Sampling [SME Review]
    • Problem: 1000 services/min -> 1 billion traces/day? (Expensive, crashes Jaeger).
    • Solution: The OTel Collector [SME Review]
      • A 'Proxy' (buffer) that sits between your App and Jaeger.
    • Architecture (Production): App -> OTel Collector -> Jaeger [SME Review]
    • Why use a Collector?
      1. Batching: Gathers 1000 spans into 1 'batch' and sends once (efficient). [SME Review]
      2. Sampling: "We don't need 100% of traces." [SME Review]
        • Head-based Sampling: Decide to 'keep' or 'drop' at the App (e.g., keep 1% of traces). (Cheap, but you miss error traces).
        • Tail-based Sampling (Needs Collector): [SME Review] Collector 'receives' all spans for a trace, 'waits' for the trace to finish. If the trace is 'ERROR' -> Keep 100%. If the trace is 'OK' -> Keep 1%. (Smart, more expensive).
  • Visualization:
      • Architecture Diagram:
    • [App 1] --OTLP--> [OTel Collector (Batching, Tail Sampling)] --OTLP--> [Jaeger]
    • [App 2] --OTLP-->
    • [App 3] --OTLP-->
    • A visual "funnel" for Tail Sampling: 100 OK traces + 5 ERROR traces go in; 1 OK trace + 5 ERROR traces come out.
  • Instructor Script:
    • "Hard Part #2: 'Cost.' [SME Review] You cannot 'save' 100% of traces in production; it will 'kill' Jaeger and 'burn' your money."
    • "The solution: Use the OTel Collector. [SME Review] [Point to diagram] Your App doesn't 'send' directly to Jaeger. It 'sends' to the Collector. [SME Review]"
    • "The Collector does 2 things: 'Batching' (for efficiency). And 'Sampling'."
    • "The 'dumb' way (Head-based) is 'throw away 99% of traces.' You will probably 'throw away' the error trace."
    • "The 'smart' way (Tail-based) [SME Review] is the Collector 'holds' a trace, 'waits' to see the result. If the trace is 'ERROR' -> Keep 100%. If the trace is 'OK' -> Throw away 99%. This is the production way."

Slide 15: Tracing - Implementation (A06)

  • Content:
    • IMPLEMENTATION (ASSIGNMENT 06)
    • docker-compose.yml (Upgraded): [SME Review]
      • jaeger: (runs Jaeger UI)
      • otel-collector: (runs Collector, receives OTLP) [SME Review]
      • app: (must set OTEL_EXPORTER_OTLP_ENDPOINT="http://otel-collector:4317" [SME Review] and OTEL_RESOURCE_ATTRIBUTES [SME Review])
    • producer.py (RabbitMQ Upgrade): [SME Review] 
      from opentelemetry.propagate import inject
      headers = {}
      inject(headers) # Automatically injects 'traceparent'
      channel.basic_publish(..., properties=pika.BasicProperties(headers=headers))
    • consumer.py (RabbitMQ Upgrade): [SME Review] 
      from opentelemetry.propagate import extract
      ctx = extract(dict(method.properties.headers)) # Restore context
      with tracer.start_as_current_span("process_message", context=ctx):
          # ... (processing code) ...
    • Rubric (A06): [SME Review]
      • [ ] Has /health endpoint.
      • [ ] Uses OTel Collector.
      • [ ] Set OTEL_RESOURCE_ATTRIBUTES (service.name, env). [SME Review]
      • [ ] Span set_status(ERROR) on exceptions. [SME Review]
      • [ ] (Bonus) traceparent is propagated through RabbitMQ Headers. [SME Review]
      • [ ] Deliverable: Screenshot Jaeger UI showing 1 trace across 3 services.
  • Visualization:
      • docker-compose diagram (App -> Collector -> Jaeger).
    • Two pika code blocks (Producer/Consumer) [SME Review] showing inject and extract via headers.
  • Instructor Script:
    • "This is Lab 06. We'll use a 3-component docker-compose (App, Collector, Jaeger). [SME Review] Your app will 'point' to the Collector."
    • "The FastAPI code (Slide 9) is almost automatic. But for RabbitMQ (Module 3), you must 'manually' inject the context into the header [SME Review] and extract it on the other side. [SME Review]"
    • "This is the Rubric. [SME Review] I will be grading service.name, error status, and (bonus) traceparent over RabbitMQ. [SME Review]"

Slide 16: Troubleshooting: Jaeger Not Receiving Traces

  • Content:
    • TROUBLESHOOTING: JAEGER ISN'T RECEIVING TRACES
    • ❌ Problem 1: Jaeger UI is empty, no traces
      • ✅ Fix 1: Check OTEL_EXPORTER_OTLP_ENDPOINT 
        # In docker-compose.yml
        environment:
          OTEL_EXPORTER_OTLP_ENDPOINT: "http://otel-collector:4317"
          # ❌ WRONG: "http://localhost:4317" (inside container)
          # ✅ RIGHT: "http://otel-collector:4317" (docker network)
      • ✅ Fix 2: Verify Docker network connectivity 
        # Get inside the app container
        docker exec -it order-service sh
        ping otel-collector  # Must be able to ping
      • ✅ Fix 3: Check OTel Collector logs 
        docker logs otel-collector
        # Look for: "Trace received" or error messages
    • ❌ Problem 2: Trace exists but is missing Span from service X
      • ✅ Fix: Check if OTEL_RESOURCE_ATTRIBUTES set service.name.
      • ✅ Fix: Verify middleware was added to the FastAPI app.
    • ❌ Problem 3: Trace is "broken" (not continuous over RabbitMQ)
      • ✅ Fix: Check your inject() and extract() code in producer/consumer.
      • ✅ Fix: Print headers to console to debug.
  • Visualization:
    • 3 columns (Problem - Cause - Fix) with [❌] and [✅] icons.
    • Code snippets for each fix.
  • Instructor Script:
    • "This is what you'll hit most often: 'Why is Jaeger empty?'"
    • "Problem 1: [Point to Fix 1] 99% of the time, OTEL_EXPORTER_OTLP_ENDPOINT is wrong. Inside Docker, don't use localhost; use the service name otel-collector."
    • "Problem 2: Trace exists but is missing a service. Check your service.name and middleware."
    • "Problem 3: Trace is 'broken' at RabbitMQ. Debug by print(headers) to see if traceparent was injected."
    • "Tip: Always check the otel-collector logs first. It will report errors very clearly."

Slide 17: Section Intro - Logging

  • Content:
    • (Based on fsa-04-section-intro.html template)
    • 02. LOGGING (THE "WHY")
    • Pillar #2: Finding the Needle in 10 Haystacks (The Black Box)
  • Visualization:
    • (Based on template)
    • Background image (slide-04.jpg).
    • Title 02. LOGGING (THE "WHY") (text-5xl).
    • Subtitle Pillar #2: Finding the Needle in 10 Haystacks (The Black Box) (text-3xl).
  • Instructor Script:
    • "We know 'Where' (Tracing). Now it's time to find out 'Why' (Logging). Pillar #2. This is the 'flight recorder' with all the details."

Slide 18: P3 - Pillar #2: Centralized Logging (The "Why")

  • Content:
    • PILLAR #2: CENTRALIZED LOGGING
    • "Finding the needle in 10 haystacks"
    • Problem: Log Silos
      • 5 services = 5 app.log files on 5 different servers/containers.
      • When debugging, you have to ssh/kubectl logs into 5 places -> Impossible.
    • Solution: Centralized Logging
      • Push all logs from all services to one single, searchable database.
  • Visualization:
      • "Before" Diagram: [Engineer] (frustrated icon) has to ssh into 5 different [Server] (each with a tiny app.log file).
    • "After" Diagram: 5 [Server] (with 'Filebeat' agent) automatically 'push' logs to one giant [ELK Stack] (warehouse). [Engineer] (happy icon) sits in one place and 'searches' the Kibana UI.
  • Instructor Script:
    • "Tracing (P2) told us 'Where.' But to know 'Why,' we need to 'read the details'."
    • "[Point to 'Before' Diagram] This is 'Log Silos.' 5 services, 5 log files. When debugging, you 'ssh' into server 1, 'grep' the log, nothing. 'ssh' to server 2... 15 minutes later, you 'give up'."
    • "[Point to 'After' Diagram] The solution is 'Centralized Logging.' We install a 'bot' (Filebeat) on each server, it 'vacuums' the logs and 'pushes' them to one 'warehouse' (Elasticsearch). Now you 'sit' in one place (Kibana) and 'search'."

Slide 19: Logging - The Tool: The ELK Stack

  • Content:
    • THE TOOL: THE ELK STACK
    • (Or EFK Stack: replace Logstash with Fluentd)
    • E - Elasticsearch:
      • The "Database" (Search Engine) to store and index billions of logs.
    • L - Logstash:
      • The "Processing Plant." Receives raw string logs, 'parses' them, 'enriches' them (e.g., add GeoIP), and 'pushes' to Elasticsearch.
    • K - Kibana:
      • The "UI" (Visualization) for you to 'search' and 'dashboard' the data from Elasticsearch.
    • B - Beats (Filebeat):
      • The "Shipper" (Agent). A lightweight agent on your service that 'reads' log files and 'sends' to Logstash (or ES).
  • Visualization:
      • The classic data flow diagram:
    • [Service 1 (Filebeat)]
    • [Service 2 (Filebeat)] -> [Logstash (Parse, Enrich)] -> [Elasticsearch (Index)] -> [Kibana (Search, Visualize)]
    • [Service 3 (Filebeat)]
    • (Each component has its respective logo).
  • Instructor Script:
    • "The most popular tool is the ELK Stack. Remember these 4 letters."
    • "E - Elasticsearch: The 'warehouse'."
    • "L - Logstash: The 'processor' that 'cleans' dirty logs."
    • "K - Kibana: The 'UI' for searching."
    • "And B - Filebeat: The 'shipper' that 'takes' the logs from your app and 'sends' them. Workshop 05 and Assignment 07 are about setting up this flow."

Slide 20: Logging - "Hard Part" #1: Structured Logging (JSON + trace_id)

  • Content:

    • "HARD PART" #1: STRUCTURED LOGGING (JSON)

    • "Dirty" Log (Unstructured String):

      • ERROR: Failed to process order 123 for user 456. Trace: abc.

    • "Clean" Log (Structured JSON): [SME Review]

      • Your app must log in JSON format.

    • "THE GOLDEN CONNECTION": [SME Review]

      • Your JSON log must 'inject' the trace_id (from OTel).

    • Code (Python logging): [SME Review]

      import logging, json
      from opentelemetry.trace import get_current_span

      class JsonFormatter(logging.Formatter):
          def format(self, record):
              span = get_current_span()
              tid = span.get_span_context().trace_id
              payload = {
                  "ts": self.formatTime(record, "%Y-%m-%dT%H:%M:%S"),
                  "level": record.levelname,
                  "message": record.getMessage(),
                  "trace_id": f"{tid:032x}", # GOLDEN CONNECTION
                  "logger": record.name,
                  # (Add saga_id, user_id... via 'extra')
              }
              return json.dumps(payload)

      # (Configure logger to use JsonFormatter)

  • Visualization:

      • A 2-column comparison table:
      • Column 1 (Dirty): [Icon: Mud] + Text log.
      • Column 2 (Clean): [Icon: Diamond] + JSON log.
    • The JsonFormatter [SME Review] code block is shown. The trace_id line is highlighted in yellow.

  • Instructor Script:

    • "This is the 'Hard Part' of Logging. If you don't do this, ELK is 'useless'."
    • "NEVER log a raw string. [Point to 'Dirty' Log] How do you 'search' for all errors for 'user 456'? You can't."
    • "ALWAYS log in JSON. [Point to 'Clean' Log]"
    • "This is the JsonFormatter [SME Review] code to do it. It automatically gets the current span (trace) and 'injects' the trace_id into every JSON log line. [SME Review] This is the 'Golden Connection' [SME Review] that lets you 'jump' from Jaeger (Trace) to Kibana (Log)."

Slide 21: Logging - "Hard Part" #2: ELK Hygiene & Cost

  • Content:
    • "HARD PART" #2: ELK OPERATIONS & COST [SME Review]
    • Problem: Elasticsearch "eats" Disk & RAM. 1 Billion logs/day = Terabytes of data.
    • Solution 1: Index Lifecycle Management (ILM): [SME Review]
      • "Nobody debugs an error from 90 days ago."
      • Hot: First 7 days. Newest index (expensive, fast).
      • Warm: 30 days. Older index (cheaper, slower).
      • Cold/Delete: > 30 days. Delete or Archive.
      • Rule: You must have ILM, or your disk will 'explode.'
    • Solution 2: Masking PII (Personally Identifiable Information): [SME Review]
      • Problem: Your log 'accidentally' records credit_card: "1234..." or password: "...".
      • Risk: Security & GDPR violation.
      • Solution: Configure Logstash (the Processor) to 'filter' and 'mask' sensitive fields before saving to Elasticsearch.
  • Visualization:
      • * Diagram 1 (ILM): [New Log] -> [HOT (7d)] -> [WARM (30d)] -> [DELETE] [SME Review]. Visualized as colored blocks (Hot -> Cold -> Delete).
    • Diagram 2 (Masking): Log (password: "123") -> [Logstash Filter] -> Log (password: "******") [SME Review]. Sensitive data is 'redacted'.
  • Instructor Script:
    • "Hard Part #2 of Logging: 'Cost' and 'Security.' [SME Review] Elasticsearch is expensive."
    • "You cannot 'keep' logs forever. You must use ILM. [SME Review] [Point to Diagram 1] 'Hot' for 7 days (for debugging), 'Warm' for 30 days (for reporting), and 'Delete' after 30 days. If not, your costs will 'explode'."
    • "More important: 'Security' (PII). [SME Review] [Point to Diagram 2] It is forbidden to log 'credit cards', 'passwords', 'tokens' to a file. You must 'teach' Logstash to 'mask' [SME Review] these fields before they are saved."

Slide 22: Logging - Implementation (A07)

  • Content:
    • IMPLEMENTATION (ASSIGNMENT 07)
    • docker-compose.yml (Upgraded):
      • elasticsearch:, logstash:, kibana:, filebeat:
    • filebeat.yml (Upgraded):
      • paths: - /var/log/app/*.json (Only 'harvest' JSON files)
      • output.logstash: hosts: ["logstash:5044"]
    • logstash.conf (Upgraded):
      • input { beats { port => 5044 } }
      • filter { json { source => "message" } } (Parse the JSON log)
      • output { elasticsearch { hosts => ["es01:9200"] } }
    • Rubric (A07): [SME Review]
      • [ ] ELK Stack installed successfully.
      • [ ] App (Python) must log in JSON (using JsonFormatter). [SME Review]
      • [ ] JSON log must contain trace_id (Golden Connection). [SME Review]
      • [ ] (Bonus) Configure Logstash filter to parse JSON.
      • [ ] Deliverable: Screenshot Kibana, searching by trace_id and seeing logs from 3 services.
  • Visualization:
      • Flow diagram: [App (JSON Log)] -> [Filebeat] -> [Logstash (JSON Filter)] -> [Elasticsearch] -> [Kibana]
    • The Rubric for A07. [SME Review]
  • Instructor Script:
    • "This is Lab 07. You will install the ELK stack."
    • "Your main tasks are (1) Fix your Python code (Slide 20) to 'log in JSON'. [SME Review] (2) Configure Filebeat to 'harvest' that JSON file. (3) Configure Logstash to 'parse' that JSON."
    • "Your deliverable [point to Rubric] is a screenshot of Kibana, searching by trace_id and seeing the logs from all 3 services show up. This is the 'Golden Connection'."

Slide 23: Troubleshooting: Kibana Not Showing Logs

  • Content:
    • TROUBLESHOOTING: KIBANA ISN'T SHOWING LOGS
    • ❌ Problem 1: Kibana has no data
      • ✅ Fix 1: Check if Filebeat is running 
        docker logs filebeat
        # Look for: "Harvester started" for each log file
      • ✅ Fix 2: Verify the log file path in filebeat.yml 
        filebeat.inputs:
          - type: log
            enabled: true
            paths:
              - /var/log/app/*.json # ❌ Path wrong?
              # ✅ Verify: docker exec -it app ls /var/log/app/
      • ✅ Fix 3: Check if Logstash is receiving 
        docker logs logstash | grep "Pipeline started"
      • ✅ Fix 4: Verify Elasticsearch indices 
        curl http://localhost:9200/_cat/indices?v
        # Must see an index named: logstash-2024.11.02
    • ❌ Problem 2: Logs exist but are not parsed as JSON
      • ✅ Fix: Check Logstash filter config 
        filter {
          json {
            source => "message"  # ✅ Must be correct field name
            skip_on_invalid_json => true
          }
        }
      • ✅ Fix: Verify app is logging correct JSON format 
        # ❌ WRONG: logger.info("Error: %s", error)
        # ✅ RIGHT: logger.info("Processing failed", extra={"error": str(error)})
    • ❌ Problem 3: Cannot search by trace_id
      • ✅ Fix: Check if the JSON log contains the trace_id field.
      • ✅ Fix: Refresh Kibana Index Pattern (Stack Management → Index Patterns → Refresh).
  • Visualization:
    • A debugging flowchart: [App] → [Filebeat] → [Logstash] → [Elasticsearch] → [Kibana]
    • Each step has [❌] problems and [✅] fixes.
  • Instructor Script:
    • "The ELK Stack is the most complex. It has 4 'failure points'."
    • "[Point to Flowchart] When Kibana is empty, debug 'from left to right.' First: Is the App creating logs? Check the file. Second: Is Filebeat 'harvesting' the log? Check its logs. Third: Is Logstash 'receiving'? Check its logs. Fourth: Is Elasticsearch 'indexing'? Check /_cat/indices."
    • "Problem 2: Log is not parsed as JSON. 99% of the time, your app is logging a 'string,' not JSON. You must use the JsonFormatter."
    • "Problem 3: Can't search by trace_id. Check two places: (1) Does your JSON log have the trace_id field? (2) Did you 'refresh' the Kibana Index Pattern?"
    • "Tip: Debug ELK by 'going step-by-step.' Don't 'jump' straight to Kibana."

Slide 24: Section Intro - Metrics

  • Content:
    • (Based on fsa-04-section-intro.html template)
    • 03. METRICS (THE "WHAT")
    • Pillar #3: Looking at the Dashboard (The Big Picture)
  • Visualization:
    • (Based on template)
    • Background image (slide-04.jpg).
    • Title 03. METRICS (THE "WHAT") (text-5xl).
    • Subtitle Pillar #3: Looking at the Dashboard (The Big Picture) (text-3xl).
  • Instructor Script:
    • "We have the 'X-Ray' (Tracing) and the 'Black Box' (Logging). Now it's time to build the 'Dashboard.' Pillar #3: Metrics."

Slide 25: P4 - Pillar #3: Metrics (The "What")

  • Content:
    • PILLAR #3: METRICS (OVERVIEW)
    • "The 10,000-foot view dashboard"
    • What are Metrics?
      • A number measured over time.
      • It is aggregated data, not a detailed event.
      • e.g., cpu_usage, memory_free, http_requests_total.
    • Tools:
      • Prometheus: The "Database" (Time-Series DB) to store Metrics.
        • Pull Model: Prometheus actively 'pulls' (scrapes) data from services (via /metrics endpoint).
      • Grafana: The "UI" to visualize Metrics into beautiful graphs.
  • Visualization:
      • * Diagram 1 (Prometheus): [Prometheus Server] (with logo) -> ('Pull' arrows going out) -> [Service A (/metrics)], [Service B (/metrics)].
    • Diagram 2 (Grafana): A beautiful, colorful Grafana dashboard showing CPU, Mem, Requests.
  • Instructor Script:
    • "The final pillar: Metrics. If Logs are 'details,' Metrics are 'overview'."
    • "We use 'Prometheus' as the 'warehouse' for numbers. The key difference: Prometheus uses a 'Pull model.' It actively 'goes and asks' (pulls) each service: 'Hey, how's your 'health'?' via an endpoint called /metrics."
    • "And 'Grafana' is the 'artist' that 'draws' Prometheus's dry numbers into beautiful charts."

Slide 26: Metrics - Metric Types & Golden Signals

  • Content:
    • METRIC TYPES (GOLDEN SIGNALS)
    • 1. Counter:
      • What: Only goes up (monotonic).
      • Example: http_requests_total{status="500"}
      • Used for: Traffic and Errors.
    • 2. Gauge:
      • What: Goes Up and Down. "Current value."
      • Example: cpu_usage_percent
      • Used for: Saturation (Resource Usage).
    • 3. Histogram:
      • What: Counts values falling into "buckets" (e.g., <100ms, <300ms, <1s).
      • Example: http_request_duration_seconds
      • Used for: Latency (p95, p99).
    • "HARD PART": Cardinality [SME Review]
      • Warning: NEVER put user_id or trace_id as a 'label' on a Metric.
      • http_requests_total{user_id="1"}
      • http_requests_total{user_id="2"}
      • -> This creates millions of time-series, 'exploding' Prometheus.
  • Visualization:
      • Three mini-diagrams:
      • Counter: A 'step-up' graph (only goes up).
      • Gauge: A 'sine wave' graph (up and down).
      • Histogram: A 'bar chart'.
    • Add a large [Icon: Warning]: label="user_id" -> [Prometheus (Exploding)] [SME Review]
  • Instructor Script:
    • "You only need to remember 3 main metric types. Counter (counts, only goes up). Gauge (measures, goes up/down). Histogram (measures latency)."
    • "And this is the 'Hard Part' of Metrics: [SME Review] 'Cardinality' Warning. [SME Review] [Point to Warning] It is forbidden to put 'user_id' or 'trace_id' as a 'label' on a metric. Every 1 user will create 1 new 'time-series' in Prometheus. 1 million users will 'kill' Prometheus. [SME Review] Metrics are for 'aggregates,' not 'details'."

Slide 27: Metrics - "Hard Part" #1: RED/USE & Alerting

  • Content:
    • "HARD PART" #1: RED/USE & ALERTING [SME Review]
    • 1. Dashboard (RED Method):
      • Don't dashboard CPU/RAM. Dashboard your business.
      • R - Rate: Requests / second. (From Counter)
      • E - Errors: % of requests that are errors (5xx). (From Counter)
      • D - Duration: p95 / p99 Latency. (From Histogram)
    • 2. Alerting > Dashboarding [SME Review]
      • "Nobody 'watches' a dashboard 24/7."
      • Alertmanager: The "Boss" of Prometheus, responsible for 'sending' (routing) alerts.
    • 3. "Runbook": [SME Review]
      • Every alert must include a runbook link.
      • A Runbook is a Wiki/Markdown file that answers: "When this alert 'fires,' what do I do?"
    • Code (PromQL Alert Rule): [SME Review] 
      - alert: HighErrorRate
        expr: sum(rate(http_requests_total{status=~"5.."}[5m]))
              / sum(rate(http_requests_total[5m])) > 0.05
        for: 5m
        labels: {severity: "critical"}
        annotations:
          summary: "5xx > 5% in last 5m"
          runbook: "https://your.runbook/5xx"
  • Visualization:
      • * Diagram 1 (RED): A sample Grafana dashboard with 3 graphs (Rate, Error, Duration). [SME Review]
    • Diagram 2 (Alert): A Slack notification: "ALERT: HighErrorRate... [Link to Runbook]". [SME Review]
  • Instructor Script:
    • "This is Hard Part #1 of Metrics. Don't 'dashboard' CPU. 'Dashboard' using the 'RED method.' [SME Review] 'R' - Rate (how many requests?). 'E' - Errors (how many failures?). 'D' - Duration (how slow?). These are the 3 things your Boss and your Users care about."
    • "Second, [EMPHASIZE], Dashboards are for 'looking.' Alerts are for 'acting.' [SME Review] Nobody 'watches' a dashboard 24/7. 'Teach' Prometheus (Alertmanager) [SME Review] to 'call' (alert) you when there's a 'fire'."
    • "[Point to PromQL code] This is a 'rule': If the 'Error Rate' 'goes above 5%' 'for 5 minutes' -> 'Fire' a 'critical' alert [SME Review] and 'attach' the 'Runbook' link (the incident response guide). [SME Review]"

Slide 28: Metrics - "Hard Part" #2: Exemplars (Trace<>Metric)

  • Content:
    • "HARD PART" #2: THE GOLDEN CONNECTION (METRICS ↔ TRACES) [SME Review]
    • Problem:
      • (Metrics) Grafana Dashboard says: "p99 Latency = 5 seconds."
      • Question: "Which request was 5 seconds? Specifically??"
    • Solution: Exemplars [SME Review]
      • A feature in Prometheus/Grafana.
      • When Prometheus 'saves' the metric (e.g., request_duration = 5.1s), it also 'attaches' a trace_id of an example (exemplar) request that took 5.1s.
      • request_duration{...} 5.1s # {trace_id="abc-123"}
    • The Workflow:
      1. (Grafana) You see a 'dot' (spike) 'slow' (5.1s) on the Latency graph.
      2. You 'click' the 'dot.'
      3. Grafana shows trace_id="abc-123" (The Exemplar). [SME Review]
      4. You 'click' that trace_id.
      5. Grafana automatically 'jumps' to Jaeger and opens the 'waterfall' for trace abc-123.
  • Visualization:
      • A 2-step diagram, extremely visual:
      1. A Grafana Latency graph. A 'dot' on the line is circled in red. A tooltip is open, showing [View in Jaeger (trace_id=abc-123)]. [SME Review]
      1. An arrow 'jumps' to a Jaeger screen, showing the waterfall for trace abc-123.
  • Instructor Script:
    • "This is the 'coolest' and 'hardest' part: 'Exemplars.' [SME Review] It 'connects' Metrics (Grafana) to Tracing (Jaeger)."
    • "[Point to Diagram] You're looking at your Dashboard (Grafana) and you see a 'spike' in latency. You 'click' that 'dot' (the spike). Grafana shows you an 'Example' (Exemplar): 'Ah, this 5.1s spike corresponds to trace_id abc-123'." [SME Review]
    • "You 'click' that trace_id link, and Grafana 'jumps' you straight into Jaeger, 'opening' the exact waterfall for trace abc-123. You 'see' immediately which 'Span' caused the 5.1s delay."
    • "This is 'Golden Connection' #2. (Golden Connection #1: Trace <-> Log. Golden Connection #2: Metric <-> Trace)."

Slide 29: Metrics - Implementation (A08)

  • Content:

    • IMPLEMENTATION (ASSIGNMENT 08)

    • docker-compose.yml (Upgraded):

      • prometheus: (config scrape_configs to 'pull' /metrics)
      • grafana: (config datasources pointing to Prometheus)

    • Code (FastAPI): [SME Review]

      # pip install prometheus-fastapi-instrumentator
      from prometheus_fastapi_instrumentator import Instrumentator

      @app.on_event("startup")
      async def _startup():
          # Automatically "wrap" all APIs
          # Automatically "expose" the /metrics endpoint
          Instrumentator(excluded_handlers={"/health"}).instrument(app).expose(app)

    • Rubric (A08): [SME Review]

      • [ ] Install Prometheus/Grafana successfully.
      • [ ] App (FastAPI) 'exposes' the /metrics endpoint (using a library).
      • [ ] Build a Grafana Dashboard (at least 3 RED graphs: Rate, Error, Duration p99).
      • [ ] (Bonus) Write 1 Alert Rule (PromQL) for HighErrorRate (5xx > 5%). [SME Review]
      • [ ] (Bonus) Configure Exemplars (if OTel Collector supports it). [SME Review]
      • [ ] Deliverable: Screenshot of Grafana (RED) Dashboard.

  • Visualization:

      • Diagram: Prometheus -> (Pull) -> [App (/metrics)].
    • The Rubric for A08. [SME Review]

  • Instructor Script:

    • "This is Lab 08. Good news: The (FastAPI) code is easy. Just 3 lines (using a library) [SME Review] to 'wrap' the app and 'expose' the /metrics endpoint."
    • "Your 'hard' job is (1) Configure prometheus.yml to 'scrape' that endpoint. (2) Go into Grafana and 'build' a 'RED' (Rate, Error, Duration) [SME Review] Dashboard for your 3 services."
    • "Bonus [SME Review] is writing an 'Alert Rule' (PromQL) [SME Review] and (super hard) configuring 'Exemplars' [SME Review] to 'jump' to Jaeger."

Slide 30: Troubleshooting: Prometheus Not Scraping Metrics

  • Content:
    • TROUBLESHOOTING: PROMETHEUS ISN'T SCRAPING METRICS
    • ❌ Problem 1: Grafana has no data
      • ✅ Fix 1: Verify the /metrics endpoint is working 
        curl http://localhost:8000/metrics
        # Must see output like:
        # http_requests_total{method="GET",status="200"} 42
      • ✅ Fix 2: Check Prometheus config (prometheus.yml) 
        scrape_configs:
          - job_name: 'order-service'
            static_configs:
              - targets: ['order-service:8000'] # ✅ Correct service name
                # ❌ WRONG: ['localhost:8000'] (inside Docker)
      • ✅ Fix 3: Check Prometheus Targets 
        http://localhost:9090/targets
        # Must see status: UP (green)
        # If DOWN (red): Check network or port
    • ❌ Problem 2: Grafana Dashboard is empty
      • ✅ Fix 1: Verify Grafana Data Source has added Prometheus 
        Configuration → Data Sources → Prometheus
        URL: http://prometheus:9090
        Test & Save → "Data source is working"
      • ✅ Fix 2: Check your PromQL query 
        # ❌ WRONG: http_requests_total (if metric doesn't exist)
        # ✅ RIGHT: rate(http_requests_total[5m])
    • ❌ Problem 3: Alert isn't firing
      • ✅ Fix: Verify Alert Rule syntax in prometheus.yml
      • ✅ Fix: Check if Alertmanager is running.
      • ✅ Fix: Check the for: 5m (Alert needs to 'wait' 5 mins before firing).
  • Visualization:
    • A flowchart: [App /metrics] ← (scrape) ← [Prometheus] → (query) → [Grafana]
    • A screenshot of the Prometheus Targets page (showing UP vs DOWN).
  • Instructor Script:
    • "Prometheus is simpler than ELK, but still has 'failure points'."
    • "Problem 1: [Point to Fix 1] First, curl /metrics from outside the container. If you see nothing, the app isn't instrumented."
    • "[Point to Fix 2] Second, check prometheus.yml. In Docker, the target must be the service name (e.g., order-service:8000), not localhost."
    • "[Point to Fix 3] Third, go to http://localhost:9090/targets. If the status is 'DOWN,' Prometheus can't 'connect' to your service."
    • "Problem 2: Dashboard is empty. 90% of the time, the Data Source isn't configured, or your PromQL query is wrong."
    • "Problem 3: Alert isn't firing. Remember, the Alert needs to 'wait' (for: 5m). Don't expect it to fire immediately."

Slide 31: Section Intro - E2E Debugging & Review

  • Content:
    • (Based on fsa-04-section-intro.html template)
    • 04. E2E DEBUGGING & REVIEW
    • "The Climax": Combining the 3 Pillars & Best Practices
  • Visualization:
    • (Based on template)
    • Background image (slide-04.jpg).
    • Title 04. E2E DEBUGGING & REVIEW (text-5xl).
    • Subtitle Combining the 3 Pillars & Best Practices (text-3xl).
  • Instructor Script:
    • "We have successfully installed all 3 Pillars. Now for the 'fun' part: Combining them. This is the 'graduation' for Observability, where we debug a full E2E system."

Slide 32: P5 - "The Climax": E2E Debugging

  • Content:
    • P5: "THE CLIMAX" - THE DEBUGGING PROCESS (SRE WORKFLOW)
    • Scenario: 3 AM, phone buzzes (Alert).
    • THE "HUNTING" PROCESS:
    • STEP 1: ALERT (The What?) [SME Review]
      • [Point to Slack] Read Alert: HighErrorRate: 5xx > 5% on "payment-service".
      • Conclusion: "PAYMENT SERVICE IS ON FIRE."
    • STEP 2: METRICS (The Impact?) [SME Review]
      • [Point to Grafana] Open payment-service Dashboard.
      • Find: Error Rate (RED) is 5%, p99 Latency (RED) spiked to 5 seconds.
      • Conclusion: "IT'S FAILING & SLOW AT PAYMENT."
    • STEP 3: TRACING (The Where? - Use Exemplar) [SME Review]
      • [Point to Grafana] Click the 'dot' (spike) on the 5s Latency graph -> Click [View Trace in Jaeger].
      • [Point to Jaeger] Open the 'waterfall'. See Span 'ValidateCreditCard' (4950ms, RED).
      • Conclusion: "FAILURE (AND SLOWNESS) IS AT THE 'VALIDATECREDITCARD' STEP."
    • STEP 4: LOGGING (The Why?) [SME Review]
      • [Point to Jaeger] Copy the trace_id from the failed Span.
      • [Point to Kibana] Paste trace_id into Kibana search bar.
      • Find: The JSON Log: {"level": "ERROR", "trace_id": "...", "message": "Third-party payment gateway (Stripe) API timeout"}
      • Conclusion: "IT FAILED BECAUSE THE STRIPE API TIMED OUT."
  • Visualization:
      • A 4-step "flowchart" showing the V2.0 debugging process:
    • 1. [Slack Alert (Red)] (Alert) -> (Detect) ->
    • 2. [Grafana Dashboard (RED)] (Metrics) -> (Click 'Exemplar') ->
    • 3. [Jaeger UI (Red Waterfall)] (Tracing) -> (Copy 'trace_id') ->
    • 4. [Kibana UI (Error Log)] (Logging) -> (Find Root Cause).
  • Instructor Script:
    • "This is the 'money slide' of Module 05, and the process for Workshop 07. This is the real SRE workflow."
    • "STEP 1: [Point to Slack] 3 AM. 'Ping!' You read the 'Alert.' [SME Review] OK, Payment Service is 'on fire'."
    • "STEP 2: [Point to Grafana] Open 'Metrics' (Grafana). See the 'red lights.' OK, 5xx errors and 5s latency." [SME Review]
    • "STEP 3: [Point to Grafana/Jaeger] Click the 'Exemplar' [SME Review] on that 5s 'dot.' 'Jump' to 'Tracing' (Jaeger). See the 'waterfall.' OK, the error is at ValidateCreditCard."
    • "STEP 4: [Point to Kibana] Copy the trace_id [SME Review] from Jaeger. 'Paste' it into 'Logging' (Kibana). 'Boom!' The exact JSON log line: 'Failed due to Stripe timeout'." [SME Review]
    • "That is the power of full Observability. From 'Alert' -> 'Root Cause' in 60 seconds, instead of 'grepping' 5 log files for 5 hours."

Slide 33: Cost Awareness (Observability Isn't Free)

  • Content:
    • COST AWARENESS (OBSERVABILITY ISN'T FREE)
    • Reality: Observability is expensive.
      • Storage: Traces, Logs, Metrics take Terabytes.
      • Compute: Elasticsearch, Jaeger, Prometheus eat RAM/CPU.
      • Network: Sending data from app → O11y stack.
    • Cost Estimate (Production - Simplified):
      • ELK Stack (1 TB logs/day): ~$2,000-5,000/month (AWS/GCP)
      • Jaeger (with 1% Sampling): ~$500-1,000/month
      • Prometheus (15-day retention): ~$300-500/month
      • Total: ~$3,000-6,500/month for an average system.
    • How to Optimize Costs:
      • Tracing: Use Tail-based Sampling → Reduce 90% of traces (keep errors + 1% success).
      • Logging: Use ILM (Hot/Warm/Delete) → Reduce 70% storage cost.
      • Metrics: Avoid high cardinality (don't use user_id as a label) → Reduce 80%.
    • Lesson for Freshers:
      • Observability is 'necessary,' but must be 'efficient.'
      • Always balance 'visibility' vs. 'cost'.
  • Visualization:
    • A Bar chart comparing costs: ELK ($$$$), Jaeger ($$), Prometheus ($).
    • A "Before Optimization" vs "After Optimization" diagram (showing 70% cost reduction).
  • Instructor Script:
    • "Before we finish, I want you to be 'aware' of one thing: Observability is 'expensive'."
    • "[Point to chart] In production, the ELK Stack can 'burn' $2,000-5,000/month. This is not a small number."
    • "Why do I say this? Because I want you to be 'mindful' when you design. Don't 'log everything.' Don't 'trace 100% of requests.' Don't 'create 1 million time-series' in Prometheus."
    • "The techniques we learned (Sampling, ILM, Cardinality control) are not just 'best practices'; they are 'cost-saving' essentials."
    • "Lesson: Observability is an 'investment,' not a 'cost.' But you must 'invest' smartly."

Slide 34: P6 - Best Practices (O11y-as-Code)

  • Content:
    • P6: BEST PRACTICES (O11Y-AS-CODE)
    • 1. Correlation is KING [SME Review]
      • trace_id MUST be in Logs (JSON).
      • trace_id (via Exemplar) MUST be in Metrics. [SME Review]
      • trace_id MUST be propagated through MQ (RabbitMQ) (using headers). [SME Review]
    • 2. Observability-as-Code (O11y-as-Code) [SME Review]
      • Grafana Dashboards (JSON) -> Store in Git.
      • Alert Rules (YAML) -> Store in Git. [SME Review]
      • OTel Collector Config (YAML) -> Store in Git. [SME Review]
      • Why: To 'review,' 'version,' and 'CI/CD' (auto-deploy) your O11y.
    • 3. Alerting > Dashboarding [SME Review]
      • Dashboards are 'past'. Alerts are 'present'.
      • Always attach a runbook (Wiki link) to your Alert. [SME Review]
    • 4. Follow Conventions [SME Review]
      • Use service.name, service.version, env (Resource Attributes). [SME Review]
      • Use http.method, db.statement... (Span Naming). [SME Review]
    • 5. Start Simple, Scale Later (NEW - For Freshers)
      • Don't try to be 'perfect' from day 1. Start with:
        • Tracing: Basic OTel + Jaeger (no Collector at first).
        • Logging: JSON log + Filebeat → Elasticsearch (skip Logstash if simple).
        • Metrics: Basic Prometheus + 3 RED graphs.
      • Then, add later: Collector, Sampling, ILM, Alerting, Exemplars.
  • Visualization:
    • Five blocks (5 Best Practices). The "Start Simple" block is highlighted in green.
    • The O11y-as-Code [SME Review] block is emphasized: [Grafana.json], [Alerts.yml], [Collector.yml] -> [Git Repo] -> [CI/CD] -> [Prometheus/Grafana].
  • Instructor Script:
    • "Here are the 5 Best Practices to 'wrap up'."
    • "One: 'Correlation' [SME Review] is King. trace_id must 'pierce' through all 3 Pillars."
    • "Two: 'O11y-as-Code.' [SME Review] Don't 'point-and-click' to create Dashboards/Alerts. 'Write code' (JSON/YAML) for them, 'store' it in Git, 'review' (PR) and 'deploy' it automatically. [SME Review] Operate it like code."
    • "Three: 'Alerting' is more important than 'Dashboarding.' [SME Review] An 'alert' (fire) must come with a 'firefighting guide' (runbook). [SME Review]"
    • "Four: Use OTel 'Conventions.' [SME Review] Don't 'invent' your own names."
    • "[NEW - Five] This is my advice for Freshers: 'Start Simple.' [EMPHASIZE] Don't stress if Labs 6-7-8 aren't 'perfect.' Start with a 'basic' setup (OTel + Jaeger + ELK + Prometheus). After it 'works,' then slowly add the Collector, Sampling, ILM, Alerting. 'Working' is more important than 'perfect'."

Slide 35: P7 - Testing Your O11y Setup (Verification Checklist)

  • Content:
    • P7: TESTING YOUR OBSERVABILITY SETUP
    • "How do I know my O11y is working?"
    • ✅ VERIFICATION CHECKLIST:
    • 1. Tracing (Jaeger):
      • [ ] Send 1 request to API → Open Jaeger UI → See new trace.
      • [ ] Trace contains all 3 services (Order → Payment → Inventory).
      • [ ] trace_id is propagated over RabbitMQ (trace is not "broken").
      • [ ] Span has status: ERROR when code throws an exception.
      • [ ] Span has tags: service.name, http.status_code.
    • 2. Logging (Kibana):
      • [ ] Open Kibana → See logs from all 3 services.
      • [ ] Log format is JSON (has fields: timestamp, level, message, trace_id).
      • [ ] Search by trace_id → See logs from all 3 services.
      • [ ] Filter by level: ERROR → See only error logs.
    • 3. Metrics (Grafana):
      • [ ] curl /metrics → See Prometheus format output.
      • [ ] Open Prometheus UI (/targets) → Status is UP.
      • [ ] Open Grafana Dashboard → See Rate, Error, Duration graphs.
      • [ ] Send many requests → Dashboard updates in real-time.
      • [ ] (Bonus) Trigger alert rule → Receive notification.
    • 4. End-to-End (E2E):
      • [ ] Send 1 error request (e.g., payment failed).
      • [ ] Grafana dashboard → Error rate spikes.
      • [ ] Click exemplar → Jump to Jaeger → See error trace.
      • [ ] Copy trace_id → Search Kibana → See error log.
  • Visualization:
    • A checklist with [✅] checkbox for each item.
    • An E2E flowchart: [Request] → [Grafana (Error spike)] → [Jaeger (Trace)] → [Kibana (Log)].
  • Instructor Script:
    • "Before you submit your Assignment, 'self-grade' it with this checklist."
    • "Parts 1-2-3: Verify each pillar individually. Make sure each tool works."
    • "Part 4: E2E is the 'final exam.' [Point to flowchart] Send 1 error request, and see if you can 'see' that error across all 3 tools."
    • "If any checkbox isn't ticked, go back to the Troubleshooting slides (16, 23, 30) to debug."
    • "This checklist is also the Rubric I will use to grade your Assignments."

Slide 36: Q&A

  • Content:
    • Q & A
    • Questions & Answers
  • Visualization:
    • A clean, minimal slide. Just the large letters "Q&A".
  • Instructor Script:
    • "Thank you. We'll take 10 minutes for Q&A on Observability."

Slide 37: Summary & Key Takeaways

  • Content:
    • SUMMARY: KEY TAKEAWAYS
    • 1. The 3 Pillars are the foundation:
      • Metrics: Detect "IS IT BROKEN?" (Alert, Dashboard).
      • Tracing: Isolate "WHERE IS IT BROKEN?" (Span, Waterfall).
      • Logging: Find "WHY IS IT BROKEN?" (Error details).
    • 2. Correlation (trace_id) is the key:
      • No trace_id = "Flying blind" in 3 tools.
      • trace_id must "flow" through HTTP, RabbitMQ, Logs, Metrics.
    • 3. Production-ready requires:
      • OTel Collector + Sampling (reduce Tracing cost).
      • Structured Logging (JSON) + ILM (reduce Logging cost).
      • Cardinality control + Alerting (optimize Metrics).
    • 4. Observability ≠ Monitoring:
      • Monitoring: "Know it's broken" (Passive).
      • Observability: "Understand why it's broken" (Active investigation).
    • 5. Start Simple, Scale Later:
      • Lab 6-7-8: Get it working first, perfect it later.
      • Production: Slowly add Collector, Sampling, ILM, Alerting.
  • Visualization:
    • 5 blocks (5 Key Takeaways) with corresponding icons.
    • A summary diagram: [Metrics] → (Exemplar) → [Traces] → (trace_id) → [Logs].
  • Instructor Script:
    • "Before we go, remember these 5 things."
    • "One: The 3 pillars answer 3 questions. Metrics = 'What?'. Tracing = 'Where?'. Logging = 'Why?'."
    • "Two: trace_id is the 'golden key.' Without it, the 3 tools are 3 separate 'silos'."
    • "Three: Production needs more than 'install.' It needs Collector, Sampling, ILM... to 'optimize cost'."
    • "Four: Observability is not Monitoring. Monitoring is 'alarming.' Observability is 'investigating'."
    • "Five: [EMPHASIZE] Don't stress if Labs 6-7-8 aren't 'perfect.' 'Working' is a 7/10. 'Perfect' is a 9-10. Go step-by-step."

Slide 38: Thank You & Next Module

  • Content:
    • THANK YOU!
    • (Your Contact Info: Email, LinkedIn, etc.)
    • COMING UP NEXT...
    • Workshop 07: E2E Debugging & Final Review
    • We will 'break' the system and use all 3 Pillars to 'hunt' the bug!
    • Visualization:
      • "Teaser" for Workshop 07.
        • An image (icon) of a 'detective' (SRE) looking at 3 screens (Grafana, Jaeger, Kibana) to find an [Icon: Bug].
    • Instructor Script:
      • "Thank you. You have learned the 'theory' and 'setup' of all 3 Pillars."
      • "In Workshop 07 (Final Review), we will 'go to war'."
      • "I will 'intentionally' make a service 'slow.' I will 'cause' a SAGA to fail. And your mission will be to use the '4-step process' (Alert -> Metrics -> Traces -> Logs) to 'hunt' that bug as fast as possible, in preparation for the Final Exam. See you then."
      • "[Point to Resources] If you get stuck on the Assignments, refer to these resources. Especially the Troubleshooting guides (slides 16, 23, 30) and the Testing checklist (slide 35)."
      • "See you in Workshop 07. Good luck with Labs 6, 7, and 8!"
      • LEARNING RESOURCES: