Distributed Tracing with Jaeger

Unit 6: Observability Topic Code: OBS-03 Read time: ~40 minutes

Learning Objectives

Explain the 'Observability' concept and its three pillars: Logs, Metrics, and Traces.
Define Distributed Tracing and its core components: Trace, Span, and Trace Context.
Introduce OpenTelemetry (OTel) as the standard for instrumentation and data collection.
Introduce Jaeger as a popular backend for storing and visualizing traces.
Outline the steps to instrument a Python/FastAPI application with OpenTelemetry to send traces to Jaeger.

Section 1: Concept/Overview

1.1 Introduction

In a Monolithic architecture, debugging a request is relatively simple. All business logic resides in a single codebase and a single process. When an error occurs or a request is slow, we can easily follow the stack trace, read logs sequentially, and determine the cause. However, the world is shifting strongly towards Microservices architecture. A user action, such as "checkout order," is no longer handled by a single service but by a complex chain of network calls across many different services: Order Service calls Payment Service, Payment Service calls Fraud Detection Service, and Notification Service.

When a request in this chain is slow or fails, finding the "culprit" becomes extremely difficult. Which service caused the latency? Where did the error occur in the chain? Viewing the logs of each service individually is like reading disjointed pages of a novel—you have the events, but you cannot connect them into a complete story. This is the problem that Distributed Tracing was born to solve. It provides a complete "story" of a request's journey as it travels through services, helping developers quickly identify bottlenecks and points of failure.

1.2 Formal Definition

Observability is a property of a system that allows us to understand its internal state simply by observing its external output signals. It is not just "monitoring" known metrics (known unknowns), but also the ability to ask new questions about the system that you haven't thought of before (unknown unknowns). Observability is built on three main pillars:

Logs: Timestamped records of events. Logs provide detailed, contextual information about what happened at a specific moment within a service. Example: [2023-10-27 10:00:05] INFO: User 'john.doe' logged in successfully.
Metrics: Numerical values aggregated over time. Metrics give us an overview of the system's health and performance. Example: cpu_usage: 80%, requests_per_second: 500, error_rate: 2%.
Traces (Distributed Traces): A record of the journey of a single request as it moves through multiple components in a distributed system. A trace shows the cause-and-effect relationships, timing, and dependencies between services. This is the "story" we mentioned earlier.

Distributed Tracing is a method used to track and analyze requests in microservices systems. It allows us to visualize the entire request flow, measure latency at each step, and pinpoint exactly where issues occur.

1.3 Real-world Analogy

Imagine you order an item online. This process can be compared to a request in a microservices system.

Logs: Like the logbook of each post office. Post Office A writes: "10:00 AM: Received package XYZ". Post Office B writes: "03:00 PM: Received package XYZ". Each record only makes sense at that specific location.
Metrics: Like the shipping company's daily report: "Processed 10,000 packages today", "Average delivery time is 2 days". It gives you an overview but tells you nothing about your specific package.
Traces: Like the tracking code (waybill number) of the package. When you enter this code, you see the entire journey:

10:00 AM: Picked up from seller at Warehouse A.
03:00 PM: Arrived at Sorting Center B.
08:00 PM: In transit to your city.
Next day 09:00 AM: Out for delivery.

This tracking code connects all the disjointed events at the post offices into a single, trackable story. Distributed Tracing does the same thing for requests in software systems.

1.4 History

The concept of Distributed Tracing was popularized by the paper "Dapper, a Large-Scale Distributed Systems Tracing Infrastructure" by Google in 2010. Dapper was Google's internal system and inspired many open-source projects later on.

Before OpenTelemetry: The community created various tracing tools like Zipkin (developed by Twitter) and Jaeger (developed by Uber). However, this led to fragmentation: companies had to choose a standard and get "locked" into that ecosystem. Instrumenting an application for Zipkin was different from Jaeger.
Standardization: To solve this, two major projects, OpenTracing (standardizing APIs) and OpenCensus (Google's library for collecting metrics and traces), were established.
OpenTelemetry (OTel): Recognizing the common goal, OpenTracing and OpenCensus merged to create OpenTelemetry in 2019. OTel became a single standard, backed by the Cloud Native Computing Foundation (CNCF), to provide a consistent set of APIs, SDKs, and tools for instrumenting, generating, collecting, and exporting telemetry data (logs, metrics, traces). OTel allows you to write instrumentation code once and send data to any backend (Jaeger, Zipkin, Prometheus, Datadog, etc.) without changing the code.

Section 2: Core Components

2.1 Architecture Overview

The overall architecture of a Distributed Tracing system using OpenTelemetry and Jaeger typically looks like this:

            +----------------------+       +-----------------------+       +------------------------+
            |                      |       |                       |       |                        |
            |   Application        |       |   Application         |       |   Application          |
            |   (Service A)        |       |   (Service B)         |       |   (Service C)          |
            | +------------------+ |       | +------------------+  |       | +------------------+   |
            | | OpenTelemetry SDK| | ----> | | OpenTelemetry SDK|  | ----> | | OpenTelemetry SDK|   |
            | +------------------+ |       | +------------------+  |       | +------------------+   |
            |                      |       |                       |       |                        |
            +-----------+----------+       +-----------+-----------+       +-----------+------------+
                        |                              |                           |
                        | (Trace Data)                 | (Trace Data)              | (Trace Data)
                        |                              |                           |
                        +------------------------------+---------------------------+
                                                       |
                                                       v
                                          +-----------------------+
                                          |                       |
                                          |  OpenTelemetry        | (Optional but Recommended)
                                          |  Collector            |
                                          |                       |
                                          +-----------+-----------+
                                                      |
                                                      | (Exported Traces)
                                                      v
            +----------------------------------------------------------------------+
            |                                JAEGER BACKEND                        |
            |                                                                      |
            | +-----------------+    +-----------------+    +--------------------+ |
            | | Jaeger Collector| <- | Jaeger Agent    |    |  Jaeger Query/UI   | |
            | +-----------------+    +-----------------+    +--------------------+ |
            |         |                      ^                       ^             |
            |         | (Writes Data)        | (Forwards Data)       | (Reads Data)|
            |         v                      |                       |             |
            | +------------------------------------------------------------------+ |
            | |                            STORAGE                               | |
            | |                  (e.g., Elasticsearch, Cassandra)                | |
            | +------------------------------------------------------------------+ |
            +----------------------------------------------------------------------+

Application / Service: Your application (e.g., Python/FastAPI).
OpenTelemetry SDK: The library integrated into the application code. It is responsible for creating Spans and Traces, and propagating Trace Context across network calls.
OpenTelemetry Collector (Optional): An independent process acting as a proxy. It receives data from multiple SDKs, can process it (filter, enrich, batch), and then export the data to one or more backends. Using a Collector is a best practice in production environments.
Jaeger Backend: The system responsible for receiving, storing, and serving trace data for querying.
Storage: Where Jaeger stores trace data, typically a NoSQL database like Elasticsearch or Cassandra.
Jaeger UI: A web interface for users to search, view, and analyze traces.

2.2 Main Components

Component 1: Trace

Definition: A Trace represents the entire journey of a request as it moves through the system. Structurally, a Trace is a collection (Directed Acyclic Graph - DAG) of Spans. Each Trace is identified by a unique Trace ID across the system.
Role: Provides a holistic view of a transaction or workflow, helping to link individual activities across multiple services into a single story.

Component 2: Span

Definition: A Span represents a unit of work or a specific operation within a Trace. Example: an HTTP call, a database query, a computation function... Each Span contains the following information:
Span ID: Unique identifier for the Span.
Trace ID: Identifier of the Trace this Span belongs to.
Parent Span ID: ID of the parent Span (if any), helping to build the relationship tree. The first Span in a Trace is called the Root Span.
Name: Description of the activity (e.g., HTTP GET /api/products).
Start Time & End Time: Start and end times of the activity.
Attributes (Tags): Key-value pairs containing contextual information about the activity (e.g., http.status_code=200, db.statement="SELECT * FROM users").
Events: Timestamped logs within the lifecycle of a Span, useful for recording specific events.
Role: The building block of a Trace. Analyzing Spans helps identify exactly which activity is taking the most time.
Syntax:

# Import necessary components from OpenTelemetry
from opentelemetry import trace

# Get a tracer for the current module
# Best practice is to use the module name
tracer = trace.get_tracer(__name__)

def process_order(order_id: str):
    # This creates a new span named 'process_order'.
    # It automatically becomes a child of any active span.
    with tracer.start_as_current_span("process_order") as order_span:
        # Add attributes (tags) to the span for more context
        order_span.set_attribute("order.id", order_id)

        print(f"Processing order {order_id}")

        # You can add events to a span
        order_span.add_event("Order validation started")

        # ... business logic here ...

        order_span.add_event("Order validation finished")

Component 3: Trace Context

Definition: Trace Context (also known as Distributed Context Propagation) is the mechanism for passing identification information (Trace ID, Span ID, and other metadata) from one service to another.
Role: This is the "glue" that binds Spans from different services into a single Trace. Without Trace Context, each service would create a new Root Span, and we wouldn't see the connection between them.
How it works: Typically, Trace Context is passed via HTTP headers. The W3C Trace Context standard defines the following headers:
traceparent: Contains Trace ID, Parent Span ID, and sampling flags. Example: 00-0af7651916cd43dd8448eb211c80319c-b7ad6b7169203331-01.
tracestate: Contains additional vendor-specific information.

When Service A calls Service B, the OpenTelemetry SDK on Service A automatically injects the traceparent header into the request. The SDK on Service B reads this header, understands it is part of an existing Trace, and creates a new Span with the inherited Trace ID and sets the Parent Span ID to the ID of the Span from Service A.

2.3 Comparison of Approaches

Approach	Pros	Cons	When to use
Automatic Instrumentation	Easy to install, no code changes required. Quick visibility. Covers popular libraries (requests, FastAPI, Flask, Psycopg2).	Less flexible. May not cover internal business logic or unpopular libraries. Hard to add custom business context.	When starting a project, for a quick overview, or for simple CRUD applications.
Manual Instrumentation	Full control. Can trace any piece of code. Easy to add important business attributes and events.	Requires more effort to install and maintain. Requires developers to understand tracing. Can miss important parts if not careful.	When you need to deeply track complex business logic, optimize performance of critical functions, or when Automatic Instrumentation is insufficient.
Hybrid Approach	Combines the best of both: Use Automatic for the "frame" (HTTP, DB) and Manual for the "core" (business logic).	Requires understanding to combine effectively.	This is the most common and powerful approach in practice.

Section 3: Implementation

Level 0 - Setup

Before starting, we need an environment to run Jaeger. The simplest way is to use Docker.

1. Create docker-compose.yml file:

# docker-compose.yml
version: '3.7'
services:
  jaeger:
    image: jaegertracing/all-in-one:latest
    ports:
      - '16686:16686' # Jaeger UI
      - '4317:4317' # OTLP gRPC receiver
      - '4318:4318' # OTLP HTTP receiver
    environment:
      - COLLECTOR_OTLP_ENABLED=true

2. Run Jaeger: Open a terminal and run the following command in the directory containing the docker-compose.yml file:

docker-compose up -d

Now, the Jaeger UI is ready at http://localhost:16686.

Level 1 - Basic (Beginner)

"Hello World" example with FastAPI and Automatic Instrumentation.

1. Install necessary libraries:

pip install fastapi "uvicorn[standard]" opentelemetry-distro opentelemetry-exporter-otlp

opentelemetry-distro: Package pre-installed with basic OTel components.
opentelemetry-exporter-otlp: Allows the SDK to send data via OTLP (OpenTelemetry Protocol) to Jaeger.

2. Create main.py file:

# main.py
from fastapi import FastAPI
import logging

app = FastAPI()
logging.basicConfig(level=logging.INFO)

@app.get("/")
def read_root():
    logging.info("Received request for root endpoint.")
    return {"message": "Hello, OpenTelemetry!"}

Note: We do not need to add any OpenTelemetry code to the Python file.

3. Run the application with OTel Instrumentation: We will use the opentelemetry-instrument tool to automatically "inject" tracing into the application.

# Set environment variables for OTel SDK
export OTEL_SERVICE_NAME="my-fastapi-app"
export OTEL_EXPORTER_OTLP_ENDPOINT="http://localhost:4317" # Use gRPC endpoint

# Run the application
opentelemetry-instrument uvicorn main:app --host 0.0.0.0 --port 8000

OTEL_SERVICE_NAME: Set the name for your service, which will appear in the Jaeger UI.
OTEL_EXPORTER_OTLP_ENDPOINT: The address of the Jaeger OTLP collector.

4. Send a request and view results: Open another terminal and send a request:

curl http://localhost:8000/

Expected Output: Access http://localhost:16686. In the "Service" section, select my-fastapi-app and click "Find Traces". You will see a new trace appear, representing the GET / request. Click on it to view the Span details.

Common Errors:

Error 1: No traces appear in Jaeger UI.
Cause: OTEL_EXPORTER_OTLP_ENDPOINT configuration is wrong, or the Jaeger container is not running.
Fix: Check docker ps again to ensure Jaeger is running. Ensure the endpoint is http://localhost:4317 (for gRPC) or http://localhost:4318 (for HTTP). Check application logs for connection errors.
**Error 2: command not found: opentelemetry-instrument**
Cause: opentelemetry-distro is not installed or the virtual environment is not activated.
Fix: Run pip install opentelemetry-distro again and ensure you are in the correct virtual environment.

Level 2 - Intermediate

Build a system with 2 services communicating with each other to see the power of Distributed Tracing.

**Service 1: inventory_service.py**

# inventory_service.py
from fastapi import FastAPI, HTTPException
import uvicorn
import logging

app = FastAPI()
logging.basicConfig(level=logging.INFO)

# A simple in-memory "database"
inventory = {
    "product-123": 10,
    "product-456": 5,
}

@app.get("/inventory/{product_id}")
def get_inventory(product_id: str):
    logging.info(f"Checking inventory for {product_id}")
    if product_id not in inventory:
        raise HTTPException(status_code=404, detail="Product not found")

    return {"product_id": product_id, "quantity": inventory[product_id]}

if __name__ == "__main__":
    uvicorn.run(app, host="0.0.0.0", port=8001)

Service 2: order_service.py (using Manual Instrumentation)

# order_service.py
from fastapi import FastAPI
import requests
import uvicorn
import logging

# OTel manual instrumentation setup
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.resources import Resource

# Set up resource and tracer provider
resource = Resource(attributes={"service.name": "order-service"})
provider = TracerProvider(resource=resource)
processor = BatchSpanProcessor(OTLPSpanExporter(endpoint="http://localhost:4317"))
provider.add_span_processor(processor)
trace.set_tracer_provider(provider)

# Get a tracer
tracer = trace.get_tracer(__name__)

app = FastAPI()
logging.basicConfig(level=logging.INFO)
INVENTORY_SERVICE_URL = "http://localhost:8001"

@app.post("/orders/{product_id}")
def create_order(product_id: str, quantity: int):
    logging.info(f"Received order for {quantity} of {product_id}")

    # Manual Span for business logic
    with tracer.start_as_current_span("check_inventory") as span:
        span.set_attribute("product.id", product_id)
        span.set_attribute("requested.quantity", quantity)

        logging.info("Calling inventory service...")

        # Automatic instrumentation for 'requests' library will propagate trace context
        try:
            response = requests.get(f"{INVENTORY_SERVICE_URL}/inventory/{product_id}")
            response.raise_for_status() # Raise an exception for bad status codes
            inventory_data = response.json()

            span.add_event("Inventory check successful", {"available_quantity": inventory_data['quantity']})

            if inventory_data['quantity'] < quantity:
                span.set_attribute("order.valid", False)
                return {"success": False, "reason": "Not enough inventory"}

        except requests.RequestException as e:
            span.record_exception(e) # Record exception in the span
            span.set_status(trace.Status(trace.StatusCode.ERROR, "Inventory service call failed"))
            raise HTTPException(status_code=503, detail="Inventory service is unavailable")

    return {"success": True, "message": "Order placed"}

Run both services:

Terminal 1 (Inventory Service - Auto Instrumented):

export OTEL_SERVICE_NAME="inventory-service"
export OTEL_EXPORTER_OTLP_ENDPOINT="http://localhost:4317"
opentelemetry-instrument uvicorn inventory_service:app --host 0.0.0.0 --port 8001

Terminal 2 (Order Service - Manually Instrumented):

# No need for opentelemetry-instrument as we configured it in code
# We still need to install 'requests' instrumentation for context propagation
pip install opentelemetry-instrumentation-requests
python order_service.py

Note: For requests to automatically propagate context, you need to install opentelemetry-instrumentation-requests. However, in this example, order-service initiates the trace, so we don't need to run it through opentelemetry-instrument.

Send a request and view results:

curl -X POST "http://localhost:8000/orders/product-123?quantity=2"

Now go to the Jaeger UI, you will see a complete trace going from order-service to inventory-service, including the check_inventory span we created manually.

Level 3 - Advanced

Optimizing for production environments: Sampling and Enriching Spans.

Sampling is the process of deciding whether a trace should be recorded and sent. Recording 100% of traces in a production environment can be very expensive in terms of performance and storage costs.

Sampling Configuration: Modify order_service.py to sample only 50% of traces.

# order_service_advanced.py

# ... (imports from previous example) ...
from opentelemetry.sdk.trace.sampling import TraceIdRatioBased

# ...

# Configure a sampler to record 50% of traces
# In production, this ratio would be much lower, e.g., 0.01 (1%)
sampler = TraceIdRatioBased(0.5)

# Set up resource and tracer provider WITH the sampler
resource = Resource(attributes={"service.name": "order-service-advanced"})
provider = TracerProvider(resource=resource, sampler=sampler) # Add sampler here
processor = BatchSpanProcessor(OTLPSpanExporter(endpoint="http://localhost:4317"))
provider.add_span_processor(processor)
trace.set_tracer_provider(provider)

tracer = trace.get_tracer(__name__)
app = FastAPI()

# ... (rest of the FastAPI code is the same) ...

# Advanced: Add context using Baggage
from opentelemetry import baggage
from opentelemetry.context import Context, attach, detach

@app.post("/orders/{product_id}")
def create_order(product_id: str, quantity: int, customer_id: str = "anonymous"):
    # Set baggage for this request context
    # Baggage is propagated across service boundaries
    ctx = baggage.set_baggage("customer.id", customer_id)
    attach(ctx)

    # tracer.start_as_current_span will now use the context with baggage
    with tracer.start_as_current_span("create_order_flow") as flow_span:
        flow_span.set_attribute("customer.id", customer_id) # also as attribute for searching

        # ... (call to check_inventory logic) ...

    detach(ctx) # Clean up context
    return {"success": True, "message": "Order placed"}

# ... (run uvicorn) ...

TraceIdRatioBased Sampler: This sampler makes decisions based on the Trace ID. With a ratio of 0.5, on average 50% of traces will be recorded.
Baggage: Baggage is an OTel mechanism to propagate key-value pairs carrying business context (e.g., customer.id) across services. Unlike Attributes (which only exist in 1 span), Baggage travels with the Trace Context. A service at the end of the chain can access this Baggage and add it to its span, making debugging easier.

Section 4: Best Practices

✅ DO's

Practice	Why	Example
Use Semantic Conventions	Naming Spans and Attributes according to standard conventions (e.g., `http.method`, `db.statement`). This helps backend systems (like Jaeger) understand and display data intelligently, while making searching, filtering, and dashboarding consistent.	Instead of `my_db_query` as an attribute, use `db.statement`. Instead of `endpoint`, use `http.target`.
Enrich Spans with Attributes	Add critical business context to spans (e.g., `user.id`, `order.id`, `product.sku`). This allows you to search traces based on business parameters, drastically speeding up debugging.	`span.set_attribute("user.id", current_user.id)`
Configure Sampling in Production	Tracing 100% of requests causes high overhead and storage costs. Start with a low sampling rate (e.g., 1-5%) and adjust based on needs.	Use `TraceIdRatioBased(0.01)` to sample 1% of traces.
Use OTel Collector	In production, deploy OTel Collector as an agent or gateway. It reduces load on the application (batching, compressing), allows flexible data processing (filtering, adding attributes), and can send to multiple backends simultaneously.	Run Collector as a Docker container or sidecar in Kubernetes.

❌ DON'Ts

Anti-pattern	Consequence	How to avoid
Recording PII (Personally Identifiable Information) in Spans	Leaking sensitive user information (name, email, phone number, address) into the tracing system. This is a severe security risk and may violate regulations like GDPR.	Always filter and sanitize data before adding to attributes. Only record non-identifiable IDs (e.g., `user_id` instead of `user_email`).
Creating too many Spans (High Cardinality)	Creating a span for every loop or small function creates very "noisy" traces that are hard to read. Attributes with constantly changing values (e.g., `request_id`) create high cardinality, pressuring storage and query systems.	Only create spans for meaningful activities: I/O calls (network, DB), critical business logic blocks. Avoid using high-cardinality values as main attributes.
Naming Spans too generically	Span names like `process` or `handle` provide no information about the ongoing activity, making trace analysis difficult.	Name specifically, following semantic conventions if possible, e.g., `HTTP GET /users/{id}`, `SELECT FROM products`.

🔒 Security Considerations

PII Sanitization: Always ensure no personal information is recorded in span attributes or events. Build a "wrapper" or middleware layer to automatically filter sensitive fields.
Jaeger Backend Security: Jaeger UI and API do not have authentication by default. In production, place Jaeger behind an authentication proxy (e.g., Nginx with Basic Auth, OAuth2-proxy) to control access.
Secure Data Transmission: Use TLS/SSL to encrypt telemetry data when sent from OTel SDK/Collector to Jaeger Backend, especially when transmitting over untrusted networks.

⚡ Performance Tips

Instrumentation Overhead: Although the OTel SDK is optimized, creating and processing spans still incurs some overhead. Measure the impact on your application.
Sampling: The most important tool for performance management. Tail-based sampling (supported by OTel Collector) is an advanced technique that decides sampling after the entire trace is complete, helping to keep only traces with errors or high latency.
Batching: OTel SDK and Collector use BatchSpanProcessor to group multiple spans and send them in a single request, reducing network calls and increasing efficiency. Fine-tune batch size and intervals appropriately.

Section 5: Case Study

5.1 Scenario

Company/Project: "TikiNow" - A large e-commerce platform with microservices architecture. Requirements: Reduce Mean Time to Resolution (MTTR) related to slow or failed checkout processes. Constraints: Solution must be compatible with multiple languages (Python, Go, Java) and must not significantly impact current system performance.

5.2 Problem Analysis

When customers complain that the "Checkout" page freezes or is very slow, the engineering team faces a nightmare. The checkout process involves: Order Service (Python/FastAPI), Payment Service (Go), Fraud Service (Java), and Notification Service (Python).

The old approach was:

DevOps engineers check general metrics (CPU, memory) of the services. Often nothing unusual is seen.
Backend engineers have to ssh into each server, grep logs of each service based on order_id or user_id.
This process is time-consuming, manual, and often cannot find the root cause if the issue is network latency between two services or a third service (e.g., a partner payment gateway) being slow. MTTR for these incidents usually lasts 4-6 hours.

5.3 Solution Design

The team decided to deploy a comprehensive Observability system with OpenTelemetry as the standard and Jaeger as the backend.

Instrumentation:
Use OpenTelemetry SDK for each language (Python, Go, Java).
Apply "Hybrid Approach": Use Automatic Instrumentation to cover web frameworks and DB clients, and add Manual Instrumentation for critical business logic like process_payment, verify_fraud_score.
Data Collection:
Deploy OpenTelemetry Collector as a DaemonSet in the Kubernetes cluster. Each node will have a Collector agent.
Applications send traces to the OTel Collector agent on the same node via localhost, minimizing network latency.
Collector is configured to perform tail-based sampling: keeping only 5% of successful traces, but keeping 100% of traces with errors or total execution time > 2 seconds.
Backend & Visualization:
Deploy Jaeger Operator on Kubernetes.
Use Elasticsearch as the storage backend for Jaeger to enable efficient storage and querying.

5.4 Implementation

Below is an example snippet from Payment Service (written in Python for illustration) after instrumentation.

# payment_service_instrumented.py
from opentelemetry import trace, baggage
from external_payment_gateway import StripeGateway

tracer = trace.get_tracer(__name__)
stripe_gateway = StripeGateway()

def process_payment(order_id: str, amount: float, card_token: str):
    """
    Processes a payment by calling an external payment gateway.
    """
    # Access baggage propagated from the upstream service (e.g., Order Service)
    customer_id = baggage.get_baggage("customer.id")

    with tracer.start_as_current_span("process_payment_flow") as payment_span:
        # Enrich span with business context for better searchability
        payment_span.set_attribute("order.id", order_id)
        payment_span.set_attribute("payment.amount", amount)
        if customer_id:
            payment_span.set_attribute("customer.id", customer_id)

        try:
            # Create a child span for the external API call
            with tracer.start_as_current_span("call_stripe_gateway") as stripe_span:
                stripe_span.set_attribute("payment.gateway", "stripe")

                # Make the actual call to the payment gateway
                transaction_id = stripe_gateway.charge(amount, card_token)

                stripe_span.set_attribute("payment.transaction.id", transaction_id)
                payment_span.set_status(trace.Status(trace.StatusCode.OK))
                return {"success": True, "transaction_id": transaction_id}

        except Exception as e:
            # Record the exception in the span, which automatically sets status to ERROR
            payment_span.record_exception(e)
            payment_span.set_status(trace.Status(trace.StatusCode.ERROR, str(e)))
            return {"success": False, "error": str(e)}

5.5 Results & Lessons Learned

Metrics Improved:
MTTR reduced by 90%: Average time to identify the root cause of payment incidents dropped from ~4 hours to under 20 minutes.
Early detection of performance issues: By viewing flame graphs in Jaeger, the team identified an unnecessary API call to the Fraud Service that was slowing down 30% of transactions, an issue they were previously unaware of.
Lessons Learned:
Observability is a cultural journey: Having tools is not enough. Developers need to be trained to think about "observability" right from when they write code, such as actively adding useful attributes to spans.
Start with what matters most: Instead of trying to instrument the entire system immediately, the team focused on the most critical and "painful" business flow: payments. Success here created momentum to expand to other areas.
Traces + Metrics + Logs = ❤️: The real power is revealed when combining all three pillars. From a slow trace in Jaeger, they can click to view detailed logs of that span, or switch to a Metrics dashboard to view performance trends of that service over a longer period.

Learning Objectives​

Section 1: Concept/Overview​

1.1 Introduction​

1.2 Formal Definition​

1.3 Real-world Analogy​

1.4 History​

Section 2: Core Components​

2.1 Architecture Overview​

2.2 Main Components​

2.3 Comparison of Approaches​

Section 3: Implementation​

Level 0 - Setup​

Level 1 - Basic (Beginner)​

Level 2 - Intermediate​

Level 3 - Advanced​

Section 4: Best Practices​

✅ DO's​

❌ DON'Ts​

🔒 Security Considerations​

⚡ Performance Tips​

Section 5: Case Study​

5.1 Scenario​

5.2 Problem Analysis​

5.3 Solution Design​

5.4 Implementation​

5.5 Results & Lessons Learned​

References​