Metrics (Prometheus)

Unit 6: Observability Topic Code: U6-T1 Reading Time: ~40 minutes

---

Learning Objectives

• Explain Metrics as the third pillar of Observability.
• Describe the pull-based architecture of Prometheus.
• Differentiate between the four main metric types: Counter, Gauge, Histogram, and Summary.
• Introduce Grafana as a visualization tool for Prometheus metrics.
• Outline the steps to instrument a Python/FastAPI application to expose a /metrics endpoint.

---

Section 1: Concept/Overview

1.1 Introduction

In the complex world of software operations today, "hoping" the system runs stably is not enough. When an e-commerce website suddenly slows down during a flash sale, or a payment service randomly reports errors, engineering teams face the eternal question: "What is happening?". Relying solely on logs (recording events that happened) or traces (tracking a request's journey) is insufficient. Logs tell what happened at a moment, traces tell the journey of a request, but we need something to measure the overall health and performance of the system over time.

This is where Metrics, the third pillar of Observability, shines. Metrics are numerical indicators collected over time, allowing us to quantify system performance, detect trends, and set up alerts when there are signs of anomalies. Instead of groping through millions of log lines, you can look at a chart and immediately know: "Ah, API latency has increased by 200% in the last 5 minutes" or "Database connections are reaching their limit". Prometheus has emerged as the de-facto standard open-source tool in the Cloud Native ecosystem for collecting and storing these metrics.

1.2 Formal Definition

Metrics are quantitative measurements of a system's state, collected at regular intervals. Each measurement is a pair of a value and a timestamp. When combining multiple measurements over time, we have a time-series.

Prometheus is an open-source monitoring and alerting system designed to collect and process time-series data. The core characteristics of Prometheus are:

1. Multi-dimensional data model: Metrics are not just a name and a number. They can be attached with labels (key-value pairs) to provide context. For example, instead of a generic metric http_requests, we can have http_requests_total{method="POST", endpoint="/api/v1/users", status="200"}.
2. Pull-based collection model: Prometheus actively "pulls" (scrapes) metrics from applications (called targets) via HTTP endpoints (usually /metrics).
3. PromQL (Prometheus Query Language): A powerful and flexible query language for selecting, aggregating, and analyzing time-series data.
4. Efficient storage: Uses a Time Series Database (TSDB) optimized for local disk storage.

1.3 Analogy

Imagine you are driving a car. The dashboard in front of you is a metrics system.

• Speedometer: Indicates your current speed (e.g., 80 km/h). This is a Gauge, a value that can go up or down.
• Odometer: Records the total distance traveled (e.g., 50,000 km). It never decreases. This is a Counter.
• Fuel Gauge: Indicates the amount of fuel left in the tank. This is also a Gauge.
• 0-100km/h Acceleration Time: If you record this time every time you try, you will have a set of measurements. Analyzing this dataset to see the average time, fastest time, or how long 95% of attempts take... represents the concept of Histogram and Summary.

Prometheus is like a system that records all these indicators from thousands of cars in a fleet and allows you to query complex questions like: "Show me the average speed of all trucks driving on the highway during rush hour".

1.4 History

Prometheus was started at SoundCloud in 2012 by Matt T. Proud and Julius Volz. They realized that traditional monitoring systems were not suitable for microservices architecture and dynamic container environments. Inspired by Google's Borgmon system, they created Prometheus with a more flexible data model and architecture.

In 2016, Prometheus became the second project (after Kubernetes) to join the Cloud Native Computing Foundation (CNCF). This event marked a significant milestone, affirming Prometheus's position as a foundational component in the modern cloud-native ecosystem.

---

Section 2: Core Components

2.1 Architecture Overview

Prometheus operates based on a Pull-based model. This means the Prometheus server is responsible for periodically connecting to applications or exporters to "scrape" metrics data.

The basic architecture model can be described as follows:

+-------------------+
|     Your App      |      (Pull / Scrape)
| (FastAPI, etc.)   +<-----------------------------+
| exposes /metrics  |                              |
+-------------------+                              |
                                                   |
+-------------------+                              |
|   DB / OS / HW    |      +-----------------+     |     +--------------------+
| (Cannot be changed|----->|    Exporter     |     |     |  Prometheus Server |
| to expose metrics)|      | (e.g. node_exp) |     |     | +----------------+ |
+-------------------+      | exposes /metrics+<----------+ | Scraper        | |
                           +-----------------+           | +----------------+ |
                                                         | |      TSDB      | |
                               +-------------------------> | (Time-Series DB)| |
                               | Alerts                  | +----------------+ |
                               |                         | |   PromQL API   | |
                           +---v-------------+           | +----------------+ |
                           |  Alertmanager   |           +--------------------+
                           +-----------------+                 ^
                                |   |   |                      | (Query)
                                v   v   v                      |
                           (Slack, Email, PagerDuty)   +-------+-------------+
                                                       |      Grafana        |
                                                       | (Visualization/Dashb)|
                                                       +---------------------+

2.2 Key Components

Prometheus is not just a single executable file but an ecosystem comprising multiple components.

Component 1: Prometheus Server

• Definition: The heart of the system. This is where metrics data is collected, stored, and processed.

• Role:

• Scraper: Periodically sends HTTP requests to targets (/metrics endpoints) to collect data.

• TSDB (Time Series Database): Efficiently stores collected metrics data on the local disk.

• PromQL Engine: Provides an HTTP API allowing users and other tools (like Grafana) to query data using the PromQL language.

• Configuration (a simple prometheus.yml file):

# prometheus.yml
global:
  scrape_interval: 15s # Scrape targets every 15 seconds.

scrape_configs:
  - job_name: 'fastapi_app'
    static_configs:
      - targets: ['localhost:8000'] # The address of our FastAPI app's /metrics endpoint

Component 2: Metric Types These are the 4 basic metric types that Prometheus supports. Choosing the right metric type for the right purpose is extremely important.

1. Counter

• Definition: A numeric value that can only increase or reset to 0 (e.g., when the service restarts). It never decreases.
• Role: Used to count cumulative events like the number of processed requests, completed tasks, or occurred errors.
• Syntax (Python prometheus-client):

from prometheus_client import Counter

# Create a Counter metric
# The name should have _total as a suffix by convention
http_requests_total = Counter(
    'http_requests_total',
    'Total number of HTTP requests received',
    ['method', 'endpoint'] # Labels to add dimensions
)

# Increment the counter for a POST request to /users
http_requests_total.labels(method='POST', endpoint='/users').inc()

# Increment by a specific value
http_requests_total.labels(method='GET', endpoint='/health').inc(1)

2. Gauge

• Definition: A numeric value that can arbitrarily go up or down.
• Role: Used to measure values that can change at any time, such as current CPU temperature, memory usage, number of online users, or items in a queue.
• Syntax:

from prometheus_client import Gauge

# Create a Gauge metric
active_connections = Gauge(
    'active_database_connections',
    'Number of active connections in the database pool'
)

# Set the value
active_connections.set(25)

# Increment and Decrement
active_connections.inc() # A new connection is made
active_connections.dec(2) # Two connections are closed

3. Histogram

• Definition: Samples observations (usually request duration, response size) and counts them in configurable "buckets".

• Role: Used to measure the distribution of values, especially useful for calculating quantiles (percentiles), e.g., "95% of requests are processed under 200ms". It provides 3 time-series:

• _bucket{le="..."}: A Counter for each bucket, counting the number of observations less than or equal to the le threshold.

• _sum: A Counter for the total sum of all observed values.

• _count: A Counter for the total number of observations made.

• Syntax:

from prometheus_client import Histogram

# Create a Histogram. Default buckets are used if not specified.
# Buckets for response times in seconds.
request_latency = Histogram(
    'http_request_latency_seconds',
    'HTTP request latency',
    buckets=[0.01, 0.05, 0.1, 0.2, 0.5, 1, 2.5, 5, 10]
)

# Observe a value
request_latency.observe(0.15) # This will increment buckets for 0.2, 0.5, ...

4. Summary

• Definition: Similar to Histogram, it also samples observations. However, instead of providing buckets, it calculates and exposes configurable quantiles (φ-quantiles) directly on the client side.
• Role: Also used to measure distribution, but when you want precise quantiles (e.g., p50, p90, p99) calculated by the application itself.
• Syntax:

from prometheus_client import Summary

# Create a Summary to track quantiles 0.5 (median), 0.9, and 0.99
request_latency_summary = Summary(
    'http_request_latency_quantiles_seconds',
    'HTTP request latency quantiles'
)

# Observe a value
request_latency_summary.observe(0.15)

2.3 Comparison of Approaches

Histogram and Summary are both used to measure distribution, but there are key differences.

Approach	Pros	Cons	When to use
Histogram	Aggregatable: You can calculate precise quantiles across multiple instances using PromQL. This is the biggest advantage.	Less precise: Quantiles are estimated based on buckets; accuracy depends on bucket definition.	Default choice. Especially in microservices environments where you need to aggregate latency from multiple service instances.
Summary	More precise on client: Provides exact quantile values calculated by the client.	Not aggregatable: You cannot accurately calculate the average quantile or p99 of multiple instances.	When you only care about metrics of a single instance and don't need aggregation, or when you need extremely precise quantiles.

---

Section 3: Implementation

Level 1 - Basic (Beginner)

Let's start with the simplest Python example, without a framework, to expose a metric.

Requirement: pip install prometheus-client

# basic_metrics_server.py
import http.server
import time
from prometheus_client import start_http_server, Counter

# 1. Create a Counter metric to track the number of page views.
PAGE_VIEWS = Counter('hello_world_page_views_total', 'Number of times the hello world page has been viewed')

class MyHandler(http.server.BaseHTTPRequestHandler):
    def do_GET(self):
        # 2. Every time a request comes to this path, increment the counter.
        PAGE_VIEWS.inc()

        # 3. Respond to the client.
        self.send_response(200)
        self.end_headers()
        self.wfile.write(b'Hello, World!')

if __name__ == '__main__':
    # 4. Start up a server to expose the metrics.
    # The prometheus_client library starts its own server on port 8000 for /metrics
    start_http_server(8000)
    print("Prometheus metrics available on http://localhost:8000/metrics")

    # 5. Start a simple web server for the 'Hello, World!' page on port 8001
    server = http.server.HTTPServer(('localhost', 8001), MyHandler)
    print("Web server available on http://localhost:8001")

    # Keep the main thread alive.
    server.serve_forever()

How to run:

1. Run the file: python basic_metrics_server.py.
2. Open a browser or use curl to access http://localhost:8001 a few times.
3. Now, check the metrics endpoint: curl http://localhost:8000/metrics.

Expected Output (from /metrics):

# HELP hello_world_page_views_total Number of times the hello world page has been viewed
# TYPE hello_world_page_views_total counter
hello_world_page_views_total 5.0
# ... other default Python metrics ...

You will see hello_world_page_views_total increase every time you visit http://localhost:8001.

Common Errors:

• Error 1: OSError: [Errno 98] Address already in use.
• Description: Port 8000 (or 8001) is already in use by another process.
• Fix: Find and stop that process, or change the port in the code (e.g., start_http_server(8008)).

Level 2 - Intermediate

Now, let's integrate metrics into a more realistic web application using FastAPI. We will use a utility library to automate request instrumentation.

Requirement: pip install fastapi uvicorn prometheus-fastapi-instrumentator

# fastapi_intermediate.py
from fastapi import FastAPI
from prometheus_fastapi_instrumentator import Instrumentator
from prometheus_client import Counter
import time
import random

app = FastAPI()

# 1. Use the instrumentator to automatically expose default FastAPI metrics
# It tracks request count, latency, and requests in progress.
Instrumentator().instrument(app).expose(app)

# 2. Create a custom Counter for a specific business logic
# This counter will track how many orders are processed, with a label for payment type.
ORDERS_PROCESSED = Counter(
    'orders_processed_total',
    'Total number of processed orders',
    ['payment_method']
)

# 3. Create a Gauge to track the number of items in a simulated queue
ITEMS_IN_QUEUE = Gauge(
    'items_in_queue',
    'Number of items currently in the processing queue'
)

@app.on_event("startup")
async def startup():
    # Set an initial value for the gauge
    ITEMS_IN_QUEUE.set(0)

@app.post("/orders")
def process_order():
    # Simulate processing time
    processing_time = random.uniform(0.1, 0.5)
    time.sleep(processing_time)

    # 4. Increment the custom counter with a specific label
    payment_method = random.choice(['credit_card', 'paypal', 'bank_transfer'])
    ORDERS_PROCESSED.labels(payment_method=payment_method).inc()

    return {"message": "Order processed successfully", "payment_method": payment_method}

@app.post("/queue/add")
def add_to_queue():
    # 5. Increment the gauge when an item is added
    ITEMS_IN_QUEUE.inc()
    return {"message": "Item added to queue"}

@app.post("/queue/process")
def process_from_queue():
    # 6. Decrement the gauge when an item is processed
    current_items = ITEMS_IN_QUEUE._value.get()
    if current_items > 0:
        ITEMS_IN_QUEUE.dec()
        return {"message": "Item processed from queue"}
    else:
        return {"message": "Queue is empty"}

# Note: The /metrics endpoint is automatically added by .expose(app)

How to run:

1. Run the server: uvicorn fastapi_intermediate:app --host 0.0.0.0 --port 8000.
2. Send a few requests to /orders, /queue/add, /queue/process.

curl -X POST http://localhost:8000/orders
curl -X POST http://localhost:8000/queue/add
curl -X POST http://localhost:8000/queue/add
curl -X POST http://localhost:8000/queue/process

3. Check the metrics endpoint: curl http://localhost:8000/metrics.

You will see metrics from prometheus-fastapi-instrumentator (like http_requests_total) and our custom metrics (orders_processed_total, items_in_queue).

Level 3 - Advanced

In a production environment, measuring latency and classifying metrics by dimensions is paramount. We will create a custom middleware to measure latency using Histogram and add dynamic labels.

Requirement: pip install fastapi uvicorn prometheus-client (do it manually this time, without instrumentator).

# fastapi_advanced.py
import time
from fastapi import FastAPI, Request
from starlette.responses import Response
from prometheus_client import Histogram, Counter, generate_latest, REGISTRY
from starlette.middleware.base import BaseHTTPMiddleware, RequestResponseEndpoint

app = FastAPI()

# 1. Define advanced metrics. A Histogram is perfect for latency.
# We add labels for method, endpoint, and status code to get rich, queryable data.
REQUEST_LATENCY = Histogram(
    'http_request_latency_seconds',
    'HTTP Request Latency',
    ['method', 'endpoint', 'http_status']
)
REQUEST_COUNT = Counter(
    'http_requests_total',
    'Total HTTP Requests',
    ['method', 'endpoint', 'http_status']
)

# 2. Create a custom middleware to intercept all requests and responses.
# This is the standard way to implement cross-cutting concerns like metrics in FastAPI.
class PrometheusMiddleware(BaseHTTPMiddleware):
    async def dispatch(self, request: Request, call_next: RequestResponseEndpoint) -> Response:
        endpoint = request.url.path
        method = request.method

        # Avoid tracking the /metrics endpoint itself
        if endpoint == "/metrics":
            return await call_next(request)

        start_time = time.time()

        try:
            response = await call_next(request)
            status_code = response.status_code
        except Exception as e:
            # In case of an unhandled exception, we still want to record the metric.
            status_code = 500
            raise e
        finally:
            # 3. Process metrics after the request is handled.
            process_time = time.time() - start_time

            # Observe the latency in the Histogram
            REQUEST_LATENCY.labels(method=method, endpoint=endpoint, http_status=status_code).observe(process_time)

            # Increment the request count in the Counter
            REQUEST_COUNT.labels(method=method, endpoint=endpoint, http_status=status_code).inc()

        return response

# 4. Add the middleware to the FastAPI application
app.add_middleware(PrometheusMiddleware)


@app.get("/")
def read_root():
    return {"Hello": "World"}

@app.get("/items/{item_id}")
def read_item(item_id: int):
    if item_id == 42:
        # Simulate an error
        raise ValueError("The ultimate answer cannot be an item.")
    return {"item_id": item_id}

# 5. Define the /metrics endpoint to expose the collected data
@app.get("/metrics")
def get_metrics():
    return Response(content=generate_latest(REGISTRY), media_type="text/plain")

How to run:

1. Run the server: uvicorn fastapi_advanced:app --host 0.0.0.0 --port 8000.
2. Send requests:

curl http://localhost:8000/
curl http://localhost:8000/items/1
curl http://localhost:8000/items/10
curl http://localhost:8000/items/42 # This will cause a 500 error

3. Check metrics: curl http://localhost:8000/metrics. You will see http_request_latency_seconds_bucket and http_requests_total with different method, endpoint, and http_status labels, providing a detailed view of each endpoint's performance.

---

Section 4: Best Practices

✅ DO's - Recommended

Practice	Why	Example
Follow naming conventions	Makes metrics easy to understand, find, and consistent across the system. General rule: app_subsystem_thing_unit.	api_database_query_duration_seconds, web_user_logins_total
Use Labels for Dimensions	Labels are the power of Prometheus. They allow you to "slice and dice" data in many dimensions without creating countless metrics.	http_requests_total{method="GET", status="200"} instead of creating metric http_requests_get_200_total.
Use Histogram for Latency	Histograms can be aggregated across instances, which is vital for calculating SLI/SLO (e.g., p99 latency) for the entire service.	Histogram('request_duration_seconds', 'Request duration', ['endpoint'])
Expose /metrics consistently	Ensure all services in your system expose metrics on the same path (/metrics) to make scraping configuration easier.	Every microservice has a GET /metrics endpoint.
Add _info or _build_info metric	Add a Gauge with value 1 and labels containing version info, commit hash. Very useful when debugging.	app_build_info{version="1.2.3", commit_hash="a1b2c3d"}

❌ DON'Ts - Avoid

Anti-pattern	Consequence	How to avoid
Use High Cardinality labels	Attaching highly unique values (user ID, request ID, email) to labels will generate millions of time-series, overloading Prometheus memory and CPU. This is the most common and dangerous mistake.	Never put user_id or request_id as a label. Instead, group them, e.g., user_type="premium".
Use . (dot) in metric names	Although Prometheus can handle it, many other systems in the ecosystem (Graphite, StatsD) treat dots as hierarchy separators, causing incompatibility.	Use underscores _. E.g., http_requests_total instead of http.requests.total.
Use Gauge for increasing values	If the service restarts, the Gauge value resets to the current value, losing history. Counter is designed to handle this reset smartly with the rate() function in PromQL.	Use Counter for total requests, error counts. Use Gauge for the number of current connections.
Use Summary when aggregation is needed	You cannot calculate the p99 latency quantile of the entire service if each instance only reports its own p99.	Always prefer Histogram over Summary unless there is a special reason.

🔒 Security Considerations

• The /metrics endpoint contains internal information: This endpoint can reveal details about architecture, used libraries, and application behavior.
• Prevention:
• Do not expose /metrics to the public Internet.
• Use a firewall to allow only the Prometheus server's IP address to access this endpoint.
• Run the /metrics endpoint on a separate port, not the main application port, and only open this port within the internal network.
• If higher security is needed, place a reverse proxy with authentication (Basic Auth) in front of the /metrics endpoint.

⚡ Performance Tips

• Instrumentation has a cost: Every call to .inc() or .observe() consumes a bit of CPU and can cause lock contention. However, client libraries (like Python's prometheus-client) are highly optimized, and this cost is usually negligible.
• Avoid complex calculations when creating metrics: Logic to update metrics should be extremely fast. Do not perform I/O calls (database, network) during instrumentation.
• Be careful with Histogram buckets: Too many buckets can slightly increase CPU and memory costs on the client. Choose a reasonable number of buckets (usually 10-15 buckets).
• Multi-process Workers (Gunicorn/uWSGI): When running Python apps with multiple worker processes, each process has its own memory. To aggregate metrics from all processes, you need to use prometheus_client in multi-process mode. The library will write metrics to files in a temporary directory, and a master process will aggregate them. Environment variable prometheus_multiproc_dir needs to be set.

---

Section 5: Case Study

5.1 Scenario

Company/Project: "QuickCart", a growing e-commerce platform. Requirements: The system frequently slows down and sometimes becomes unresponsive during promotional periods. The operations team has no way of knowing if the root cause is the database, order processing logic, or a third-party service (payment service). They need a monitoring system for deep insights and early warnings. Constraints: The solution must be open-source, easy to integrate with the existing microservices system (mainly FastAPI), and scalable.

5.2 Problem Analysis

The current system relies solely on logs. When issues arise, engineers have to "grep" through gigabytes of logs from multiple services, a slow and inefficient process. They cannot answer key questions like:

• What is the average latency of the payment API?
• Is the number of active database connections nearing the limit?
• What is the error rate of the order creation API? They are in a "reactive" state instead of "proactive".

5.3 Solution Design

The team decided to deploy Prometheus and Grafana.

1. Prometheus Server: Will be deployed to scrape metrics from all microservices.
2. Instrumentation: Each FastAPI microservice will be instrumented to expose the following metrics:

• Request Rate & Errors (Counter): http_requests_total{endpoint, method, status} to track traffic and error rates on each API.
• Request Latency (Histogram): http_request_latency_seconds{endpoint, method} to track performance and calculate p95, p99 latency.
• Database Connections (Gauge): db_connections_active{pool_name} to track the number of connections in use.
• 3rd Party API Calls (Histogram & Counter): external_api_latency_seconds{api_name} and external_api_calls_total{api_name, status} to monitor performance and error rates of the payment service.

3. Grafana: Will be connected to Prometheus to create visual dashboards, helping the team easily monitor key indicators in real-time.
4. Alertmanager: Configure alert rules in Prometheus. For example: "Alert if payment API p99 latency > 2 seconds for 5 minutes" or "Alert if database connections > 80% limit".

5.4 Implementation

Below is sample code for the "Order Service" of QuickCart.

# quickcart_order_service.py
import time
import random
from fastapi import FastAPI, Request, HTTPException
from starlette.responses import Response
from prometheus_client import Histogram, Counter, Gauge, generate_latest, REGISTRY
from starlette.middleware.base import BaseHTTPMiddleware, RequestResponseEndpoint

# --- Metrics Definition ---
REQUEST_LATENCY = Histogram(
    'http_request_latency_seconds',
    'HTTP Request Latency',
    ['endpoint', 'http_status']
)
REQUEST_COUNT = Counter(
    'http_requests_total',
    'Total HTTP Requests',
    ['endpoint', 'http_status']
)
DB_CONNECTIONS = Gauge(
    'db_connections_active',
    'Active database connections'
)
PAYMENT_API_LATENCY = Histogram(
    'external_api_latency_seconds',
    'Payment API Latency',
    ['api_name']
)

# --- Middleware for standard metrics ---
class MetricsMiddleware(BaseHTTPMiddleware):
    async def dispatch(self, request: Request, call_next: RequestResponseEndpoint) -> Response:
        endpoint = request.url.path
        if endpoint == "/metrics":
            return await call_next(request)

        start_time = time.time()
        try:
            response = await call_next(request)
            status_code = response.status_code
        except Exception:
            status_code = 500
            raise
        finally:
            latency = time.time() - start_time
            REQUEST_LATENCY.labels(endpoint=endpoint, http_status=status_code).observe(latency)
            REQUEST_COUNT.labels(endpoint=endpoint, http_status=status_code).inc()
        return response

app = FastAPI()
app.add_middleware(MetricsMiddleware)

# --- Mock functions to simulate real work ---
def get_db_connection():
    # Simulate getting a connection from a pool
    DB_CONNECTIONS.inc()
    print(f"DB connections: {DB_CONNECTIONS._value.get()}")

def release_db_connection():
    # Simulate releasing a connection
    DB_CONNECTIONS.dec()
    print(f"DB connections: {DB_CONNECTIONS._value.get()}")

def call_payment_api():
    # Simulate calling a 3rd party API
    start = time.time()
    latency = random.uniform(0.05, 1.5) # Payment API can be slow
    time.sleep(latency)
    PAYMENT_API_LATENCY.labels(api_name='stripe').observe(time.time() - start)
    if random.random() < 0.1: # 10% chance of failure
        raise IOError("Payment Gateway Timeout")

# --- API Endpoints ---
@app.post("/orders")
def create_order():
    get_db_connection()
    try:
        # Simulate business logic
        time.sleep(random.uniform(0.1, 0.3))
        call_payment_api()
        return {"message": "Order created successfully"}
    except IOError as e:
        raise HTTPException(status_code=503, detail=str(e))
    finally:
        release_db_connection()

@app.get("/metrics")
def get_metrics():
    return Response(content=generate_latest(REGISTRY), media_type="text/plain")

5.5 Results & Lessons Learned

• Improved Metrics:

• After deployment, during the next promotion, the Grafana dashboard immediately showed a spike in external_api_latency_seconds for the payment service.

• Simultaneously, http_request_latency_seconds for the /orders endpoint also increased correspondingly, and db_connections_active gradually increased as requests were held longer while waiting for the payment API.

• Result: The team accurately identified the root cause as the third-party payment service being overloaded, not their database or code. They contacted the provider and temporarily switched to a backup payment gateway.

• Lessons Learned:

1. Observability is power: Metrics shifted the team from "guessing" to "data-driven decision making".
2. Instrument everything: Instrumenting external calls is just as important as instrumenting internal code.
3. Alerts are crucial: Automated alerts help detect issues before customers complain on a large scale, minimizing revenue impact.
4. Dashboards for everyone: Having a visual dashboard helps the whole team (dev, ops, PM) understand system health.

---

References

• Prometheus Official Documentation
• Grafana Official Website
• Python Client for Prometheus (prometheus-client)
• Prometheus FastAPI Instrumentator