Skip to main content

Centralized Logging (ELK Stack)

Unit 6: Observability Topic Code: OBS-06-ELK Reading Time: ~40 minutes

Learning Objectives​

  • Explain the role of Logs as one of the three pillars of Observability.
  • Describe the challenges of managing decentralized logs in a microservices architecture.
  • Define Centralized Logging and the purpose of the ELK Stack.
  • Identify the core components and their roles: Elasticsearch, Logstash, Kibana, and Beats (Filebeat).
  • Illustrate the data pipeline for logs from an application to Kibana.
  • Understand basic search and filtering capabilities in Kibana's Discover UI.

Section 1: Concept/Overview​

1.1 Introduction​

In the modern software development industry, especially with the explosion of Microservices architecture, a system is no longer a single block (monolith) running on one server. Instead, an application can be composed of dozens, or even hundreds, of small services, each running in a separate container on different servers. When an error occurs—for example, an order cannot be processed—that bug could originate from the Order Service, Payment Service, or Inventory Service. How can a developer find the root cause? The traditional approach is to ssh into each server and use the grep command to search through log files. This is a nightmare: it is time-consuming, error-prone, and almost impossible as the system scales.

This is where the importance of Logs—one of the Three Pillars of Observability (Logs, Metrics, Traces)—is demonstrated. Logs are detailed records of events occurring within the system. They are the single source of truth that helps us understand "what happened" at the most granular level. However, for logs to be truly useful in a distributed system, they need to be collected and managed centrally. Centralized Logging is not just a "nice-to-have" but a mandatory requirement for effectively operating and debugging today's complex systems.

1.2 Formal Definition​

Centralized Logging is the method of collecting and aggregating logs from various sources (applications, servers, containers, network devices) into a single location for storage, search, analysis, and visualization. Instead of leaving logs scattered across individual servers, we bring them all to a common "warehouse".

ELK Stack is the most popular open-source toolkit for implementing Centralized Logging. The name "ELK" is an acronym for three core products:

  • E - Elasticsearch: A distributed "search and analytics engine." It is responsible for storing all logs and providing ultra-fast full-text search capabilities.
  • L - Logstash: A server-side "data processing pipeline." It receives data from multiple sources, processes (parses, filters, transforms) it, and then sends it to a "stash" like Elasticsearch.
  • K - Kibana: A web "user interface." It allows users to search, view, and interact with data stored in Elasticsearch through charts, tables, and intuitive dashboards.

Later, a fourth component was added, and the toolkit is often referred to as the Elastic Stack:

  • Beats: A collection of lightweight "data shippers." These are installed on source servers to collect various types of data (logs, metrics, network data) and send them to Logstash or directly to Elasticsearch. Filebeat is the most popular Beat, specialized for collecting and forwarding log files.

1.3 Analogy​

Imagine your microservices system is a large city with hundreds of small libraries in every district (each service is a library). When you need to find information on a specific topic, you have to drive to each library, ask the librarian, and search for it yourself. It is very time-consuming and inefficient.

Centralized Logging (ELK Stack) is like building a massive National Library:

  • Filebeat (The Book Couriers): At each district library, there are delivery staff (Filebeat) specialized in collecting new books (log files) and transporting them to the National Library. They work very quickly and lightly, without affecting the operations of the district library.
  • Logstash (The Cataloger and Sorter): At the gate of the National Library, there is a team of experts (Logstash). They receive books from the couriers, read through them, label them (parsing), classify them by topic (filtering), and sometimes translate them into a common language (transforming) before putting them into storage.
  • Elasticsearch (The Smart Archive): This is the massive book repository, organized extremely scientifically. Each book (log entry) is indexed in detail, making the search for any keyword among millions of books take only a few seconds.
  • Kibana (The Digital Lookup Portal): This is the modern lookup interface of the library. You can sit at a computer, type keywords, filter by author or publication year, and the system will display exactly the books you need. You can also create analysis charts to see "which topics are being read the most."

1.4 History of Development​

  • Elasticsearch: Created by Shay Banon in 2010, originally based on his previous project, Compass. It was built on the Apache Lucene library and designed to be a distributed search engine, scalable and easy to use via REST API.
  • Logstash: Created by Jordan Sissel in 2009. Originally an independent tool for managing logs. With powerful plugin capabilities, it quickly became popular.
  • Kibana: Created by Rashid Khan, also an independent project to create a user interface for Elasticsearch.
  • The Combination: Recognizing the strong synergy between the three products, the Elastic company (founded by Shay Banon) brought them together to form the ELK Stack, a complete end-to-end solution for log analysis.
  • Beats: Introduced later to solve a Logstash problem: it is powerful but consumes significant resources (runs on JVM). Beats is written in Go, extremely lightweight and efficient, becoming the default choice for installation on edge nodes (application servers) to collect data.

Section 2: Core Components​

2.1 Architecture Overview​

The data flow in a typical ELK Stack system occurs as follows: Lightweight agents (Filebeat) are installed on application servers to monitor and collect logs. They send data to a central processing pipeline (Logstash). Logstash cleans, normalizes, and enriches the data before pushing it into the storage and search engine (Elasticsearch). Finally, users use the web interface (Kibana) to query and visualize data from Elasticsearch.

Basic Data Flow Diagram:

                   +-------------------+      +------------------+      +-------------------+
Application Server |                   |      |                  |      |                   |
+------------------+      Logstash Server   |   Elasticsearch Cluster  |      User's Browser
|                  |      +-----------------+ | +----------------+ |      +-------------------+
| [Log File] ----> | ---> |      INPUT      | | |                | |      |                   |
|                  |      |  (from Beats)   | | |                | |      |      KIBANA       |
|  +-----------+   |      +-----------------+ | |                | |      |                   |
|  | FILEBEAT  |   |      |                 | | |     INDEX      | | <--- | (Queries &        |
|  +-----------+   |      |     FILTER      | | |                | |      |  Visualizations)  |
|                  |      | (Parse, Enrich) | | |                | |      |                   |
|                  |      +-----------------+ | |                | |      |                   |
+------------------+      |      OUTPUT     | | |                | |      +-------------------+
                   | ---> |(to Elasticsearch)| | |                | |
                   +-------------------+      +------------------+      +-------------------+

2.2 Key Components​

Component 1: Beats (Filebeat)

  • Definition: Filebeat is a lightweight "data shipper," written in Go. It is installed as an agent on source servers.
  • Role: The primary role of Filebeat is to monitor (tail) specified log files, detect changes (new log lines), and send log data to a specific output (usually Logstash or Elasticsearch). It ensures "at-least-once" delivery and can handle issues like network disconnects or log rotation.
  • Syntax (filebeat.yml): Filebeat is configured using a YAML file.
# filebeat.yml - Basic configuration

# Define inputs: where to get the data from
filebeat.inputs:
  - type: log
    # Enable this input
    enabled: true
    # Paths to the log files to monitor
    paths:
      - /var/log/my-app/*.log
      - /var/log/nginx/access.log
    # Add custom fields to identify the log source
    fields:
      service: 'nginx-web-server'
      environment: 'production'
    fields_under_root: true

# Define output: where to send the data to
output.logstash:
  # The Logstash hosts
  hosts: ['logstash-server:5044']

# Optional: Enable internal logging for Filebeat itself
logging.level: info
logging.to_files: true
logging.files:
  path: /var/log/filebeat
  name: filebeat
  keepfiles: 7
  permissions: 0644

Component 2: Logstash

  • Definition: Logstash is an open-source server-side data processing tool that allows you to collect data from multiple sources, transform it, and send it to your desired destination. It operates as a pipeline.

  • Role: This is the "brain" of log processing. It can:

  • Parse: Convert unstructured log lines (unstructured plain text) into structured data (structured JSON). Example: Splitting an NGINX log line into fields like ip_address, request_path, status_code.

  • Enrich: Add information to the log. Example: Based on IP address, add geolocation info (GeoIP).

  • Filter: Remove unnecessary logs or mask sensitive information (PII).

  • Syntax (logstash.conf): A Logstash pipeline has 3 main sections: input, filter, output.

# /etc/logstash/conf.d/main.conf

# Input section: how Logstash receives data
input {
  # Listen for incoming data from Filebeat on port 5044
  beats {
    port => 5044
  }
}

# Filter section: the processing magic happens here
filter {
  # Check if the log is from our NGINX service
  if [service] == "nginx-web-server" {
    # Use the 'grok' filter to parse NGINX access logs
    grok {
      match => { "message" => "%{COMBINEDAPACHELOG}" }
    }
    # Parse the user agent string
    useragent {
      source => "agent"
    }
    # Add geolocation data based on the client IP
    geoip {
      source => "clientip"
    }
  }
}

# Output section: where Logstash sends the processed data
output {
  # Send the data to our Elasticsearch cluster
  elasticsearch {
    hosts => ["http://elasticsearch-node1:9200"]
    index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
  }
}

Component 3: Elasticsearch

  • Definition: Elasticsearch is a distributed search engine built on Apache Lucene. It stores data in the form of JSON documents.

  • Role:

  • Storage: A centralized and long-term storage location for all processed log data.

  • Indexing: When a document is stored, Elasticsearch analyzes its content and creates an "inverted index." This structure allows for ultra-fast full-text search and complex queries.

  • API Provider: Provides a comprehensive RESTful API to add, edit, delete, and most importantly, query data.

  • Syntax (JSON Document & Query DSL): Data in Elasticsearch are JSON documents.

// A sample log document stored in Elasticsearch after processing
{
  "@timestamp": "2023-10-27T10:30:00.123Z",
  "service": "nginx-web-server",
  "environment": "production",
  "message": "192.168.1.10 - - [27/Oct/2023:10:30:00 +0700] \"GET /api/products/123 HTTP/1.1\" 200 1543",
  "clientip": "192.168.1.10",
  "verb": "GET",
  "request": "/api/products/123",
  "httpversion": "1.1",
  "response": 200,
  "bytes": 1543,
  "geoip": {
    "country_name": "Vietnam",
    "city_name": "Ho Chi Minh City"
  }
}

Component 4: Kibana

  • Definition: Kibana is an open-source data visualization and exploration platform designed to work with Elasticsearch.

  • Role: It is the window for you to look into the massive data warehouse in Elasticsearch.

  • Discover: Allows searching and filtering raw logs using a simple query language (KQL - Kibana Query Language) or Lucene query syntax.

  • Visualize: Create charts (pie charts, line graphs, bar charts, heat maps) from data.

  • Dashboard: Combine multiple visualizations into a single dashboard for an overview of system health.

  • Syntax (Kibana Query Language - KQL):

# Find all logs from the production environment with an error status code
environment: "production" and http.response.status_code >= 500

# Find logs for a specific user IP accessing the /login endpoint
client.ip: "123.45.67.89" and url.path: "/login"

2.3 Comparing Approaches​

Filebeat vs. Logstash on client-side (agent)

ApproachProsConsWhen to use
Using Filebeat Only- Extremely Lightweight: Consumes very little CPU and RAM.


- Simple: Easy to install and configure.


- Reliable: Built-in backpressure mechanism and guaranteed delivery. |

  • Limited Processing: Can only perform simple operations like filtering, adding fields. Cannot do complex parsing like Grok. | - Most common case. When you want to process logs centrally at Logstash or an Elasticsearch Ingest Node.


- When resources on the application server are extremely limited. | | Using Logstash | - Extremely Powerful: Supports hundreds of plugins to parse, enrich, and filter data right at the source.


- Flexible: Can read data from many sources other than files (TCP, UDP, JDBC, ...). | - Heavy: Runs on JVM, consumes significantly more RAM and CPU than Filebeat.


- Complex: Configuration is more complex. | - Rare case. When you need very complex log processing right at the source before sending.


- When the application server has ample surplus resources.


- Used for legacy systems where log formats cannot be changed. |

Section 3: Implementation​

Level 1 - Basic (Hello World)​

Goal: Read a log line from a file, send it via Filebeat, and see it appear in Logstash stdout. This is the most basic step to verify the pipeline works.

1. Create a sample log file:

# On your application server
echo "This is the first log line from my-app." >> /var/log/my-app.log
echo "This is a second, more interesting log line." >> /var/log/my-app.log

2. Configure Filebeat (filebeat.yml):

# filebeat.yml
filebeat.inputs:
  - type: log
    enabled: true
    paths:
      - /var/log/my-app.log

# We will send to Logstash, not Elasticsearch directly
# output.elasticsearch: ... (comment this out)

output.logstash:
  hosts: ['localhost:5044']

3. Configure Logstash (hello-world.conf): This configuration simply receives data from Beats and prints it to the console screen with rubydebug format for readability.

# hello-world.conf
input {
  beats {
    port => 5044
  }
}

# No filter section needed for this basic example

output {
  # Print the full event to the console
  stdout {
    codec => rubydebug
  }
}

4. Run and Verify:

  • Run Logstash: bin/logstash -f hello-world.conf
  • Run Filebeat: sudo ./filebeat -e -c filebeat.yml

Expected Output (on Logstash console):

{
       "message" => "This is a second, more interesting log line.",
      "@version" => "1",
    "@timestamp" => 2023-10-27T04:55:12.456Z,
          "host" => {
        "name" => "my-dev-machine"
    },
         "agent" => {
        "hostname" => "my-dev-machine",
        "ephemeral_id" => "...",
        "id" => "...",
        "name" => "my-dev-machine",
        "type" => "filebeat",
        "version" => "8.5.0"
    },
           "log" => {
        "offset" => 42,
          "file" => {
            "path" => "/var/log/my-app.log"
        }
    },
           "ecs" => {
        "version" => "8.0.0"
    }
}

You can see Logstash received the log and automatically added useful metadata like @timestamp, host, agent...

Common Errors:

  • Connection Refused: Filebeat cannot connect to Logstash.

  • Fix: Check the IP address and port in filebeat.yml (output.logstash.hosts). Ensure Logstash has started and the firewall is not blocking port 5044.

  • Permission Denied: Filebeat cannot read the log file.

  • Fix: Ensure the user running Filebeat (usually root) has read permissions for /var/log/my-app.log. Run Filebeat with sudo.

Level 2 - Intermediate​

Goal: Parse NGINX logs, convert them to structured data, and push to Elasticsearch. Then, search on Kibana.

1. Sample NGINX Log (/var/log/nginx/access.log):

127.0.0.1 - - [27/Oct/2023:11:20:15 +0700] "GET /products/view?id=123 HTTP/1.1" 200 512 "-" "Mozilla/5.0"
192.168.1.5 - - [27/Oct/2023:11:21:05 +0700] "POST /api/login HTTP/1.1" 401 137 "http://example.com/login" "curl/7.68.0"

2. Configure Filebeat (filebeat.yml): Use the built-in NGINX module of Filebeat to simplify.

# filebeat.yml
filebeat.modules:
  - module: nginx
    access:
      enabled: true
      var.paths: ['/var/log/nginx/access.log*']

# We will now send directly to Elasticsearch for simplicity,
# as the NGINX module comes with a pre-built parsing pipeline.
output.elasticsearch:
  hosts: ['localhost:9200']

# It's important to run setup once to install dashboards and pipelines
# sudo ./filebeat setup -e

Note: Using this module bypasses Logstash and uses Elasticsearch Ingest Node for parsing; this is a modern practice.

3. Search on Kibana:

  • Open Kibana, go to Discover.
  • You will see the logs have been parsed into separate fields: client.ip, http.request.method, http.response.status_code, url.path, user_agent.original.
  • Perform search:
  • To find all failed login requests, type into the KQL search bar: url.path: "/api/login" and http.response.status_code: 401
  • To find all server errors (5xx), type: http.response.status_code: [500 TO 599]

Expected Output (in Kibana Discover): You will see a table with analyzed logs, capable of being filtered and sorted by columns like status_code, client.ip, etc.

Level 3 - Advanced​

Goal: Process logs from two different microservices with different formats (JSON and plain text) in the same Logstash pipeline.

1. Log sources:

  • Auth Service (/var/log/auth-service/auth.log): Logs in JSON format.
{"timestamp": "2023-10-27T12:00:05Z", "level": "INFO", "message": "User login successful", "user_id": "user-abc"}
{"timestamp": "2023-10-27T12:00:10Z", "level": "WARN", "message": "Invalid password attempt", "username": "admin"}

  • Payment Service (/var/log/payment-service/payment.log): Logs in plain text.
INFO: [2023-10-27 12:01:00] Payment processing started for order_id=xyz-123.
ERROR: [2023-10-27 12:01:05] Credit card validation failed for order_id=xyz-123, reason: Insufficient funds.

2. Configure Filebeat (filebeat.yml): We will define two different inputs and add a tag so Logstash can distinguish them.

# filebeat.yml
filebeat.inputs:
  - type: log
    paths:
      - /var/log/auth-service/*.log
    fields:
      service_name: 'auth-service' # Custom field to identify the service
      log_format: 'json'
    fields_under_root: true

  - type: log
    paths:
      - /var/log/payment-service/*.log
    fields:
      service_name: 'payment-service' # Custom field for the other service
      log_format: 'plain-text'
    fields_under_root: true

output.logstash:
  hosts: ['localhost:5044']

3. Configure Logstash (pipeline.conf): Use if/else statements in the filter section to apply different parsing logic.

# pipeline.conf
input {
  beats {
    port => 5044
  }
}

filter {
  # Conditional processing based on the 'service_name' field set by Filebeat
  if [service_name] == "auth-service" {
    # The log is already in JSON format, so we use the 'json' filter
    json {
      source => "message" # The field containing the JSON string
      target => "log_data"    # Put the parsed JSON into a 'log_data' object
    }
    # Use date filter to parse the timestamp from the log and set it as the main event timestamp
    date {
      match => [ "[log_data][timestamp]", "ISO8601" ]
    }
  }
  else if [service_name] == "payment-service" {
    # For plain text logs, we use 'grok' for pattern matching
    grok {
      match => { "message" => "%{LOGLEVEL:level}: \[%{TIMESTAMP_ISO8601:timestamp}\] %{GREEDYDATA:log_message}" }
    }
    # Use date filter for this format too
    date {
      match => [ "timestamp", "YYYY-MM-dd HH:mm:ss" ]
    }
  }
}

output {
  elasticsearch {
    hosts => ["http://localhost:9200"]
    # Create a separate index for each service for better management
    index => "%{[service_name]}-%{+YYYY.MM.dd}"
  }
}

This architecture is extremely powerful, allowing you to build a single pipeline to process logs from dozens of different services, where each service can have its own log format.

Section 4: Best Practices​

✅ DO's​

PracticeWhyExample
Structured LoggingStructured logs (JSON) are easier to parse, have fewer errors, and don't require CPU-expensive grok. Helps in more accurate searching and aggregations.In application code, instead of log.info("User " + userId + " logged in"), use a logging library to output: log.info({ "message": "User logged in", "user_id": userId }).
Use Index Lifecycle Management (ILM)Automatically manages the index lifecycle: moves old indices to cheaper "warm"/"cold" nodes, and eventually deletes them to save storage costs.Configure ILM Policy in Kibana to automatically delete logs older than 90 days.
Use Index TemplatesEnsures new indices are always created with consistent mapping and settings. Avoids dynamic mapping errors creating unwanted data types.Create a template applied to my-app-* to define the status_code field as integer type instead of keyword.
Centralize Pipeline ManagementInstead of keeping parsing logic in multiple places, centralize it in Logstash or Elasticsearch Ingest Node. This makes management and updates easier.Use a central Logstash pipeline as in the Advanced example in Section 3.

❌ DON'Ts​

Anti-patternConsequenceHow to avoid
Ingest everythingUnnecessarily increases storage and processing costs. Creates "noise" when searching.In Filebeat or Logstash, filter out DEBUG logs or unimportant health checks in the production environment.
Use grok for everythinggrok (regular expression) is very CPU expensive. Overusing it will slow down the Logstash pipeline and increase log latency.Require dev teams to switch to structured logging (JSON). Only use grok for legacy systems that cannot change.
Ignore Shard & ReplicaIncorrect configuration of shards and replicas can cause poor performance, or worse, data loss when a node fails.Plan sharding strategy from the start. General rule: keep each shard size around 20-40GB. Always have at least 1 replica for critical data.
Leave sensitive info in logsLogs containing passwords, API keys, personal info (PII) are a serious security risk.Use mutate filter in Logstash to remove or mask sensitive fields before storing in Elasticsearch. Example: mutate { remove_field => ["password"] }.

🔒 Security Considerations​

  • Transport Layer Security (TLS): Encrypt all communication between stack components (Filebeat ↔ Logstash ↔ Elasticsearch ↔ Kibana) to prevent eavesdropping.
  • Role-Based Access Control (RBAC): Use Elastic's security features (free in basic license) to create different roles. Example: Team A can only view Service A logs, Team B only views Service B logs, and Team SRE can view all.
  • Kibana Authentication: Never expose Kibana to the public internet without an authentication layer. Integrate with Active Directory, LDAP, or SAML.
  • Sanitize Data: Always filter and clean data in Logstash to remove sensitive info (PII, credentials, tokens) before they are stored.

⚡ Performance Tips​

  • Use Elasticsearch Ingest Nodes: Instead of doing all parsing in Logstash, shift some of the burden to Elasticsearch Ingest Nodes. Filebeat can send directly to Elasticsearch and use a pre-defined ingest pipeline.
  • Persistent Queues: Enable persistent queues in Logstash. If Elasticsearch is overloaded or unavailable, Logstash will write events to disk and send them later, avoiding data loss.
  • Tuning Logstash Worker & Batch Size: Increasing pipeline.workers (usually equal to CPU cores) and pipeline.batch.size in logstash.yml can significantly improve throughput.
  • Use Data Tiers: Divide Elasticsearch nodes into hot, warm, and cold tiers. New and frequently accessed data stays on hot nodes (fast SSD). Older data is automatically moved by ILM to warm nodes (larger, slower disks) to save costs.

Section 5: Case Study​

5.1 Scenario​

Company/Project: E-commerce platform "FastMart" Requirements:

  1. Reduce Mean Time to Resolution (MTTR) from 2 hours to under 30 minutes.
  2. Provide an overview dashboard of the health of key services (Orders, Payments, Shipping).
  3. System must be scalable to handle 1TB of logs per day during sale seasons. Constraints:
  • Prioritize open-source solutions to reduce license costs.
  • Operations team has experience with Linux but is new to observability tools.

5.2 Problem Analysis​

FastMart just transitioned from Monolith to Microservices. Previously, debugging only required ssh into a server and grep log. Now, a customer transaction goes through 5-7 different services. When an error occurs, the SRE team has to ssh into multiple servers, kubectl logs from multiple pods. This is very manual, slow, and impossible to find correlations between events occurring across different services. MTTR skyrocketed, directly affecting customer experience and revenue.

5.3 Solution Design​

The SRE team decided to deploy the ELK Stack.

  • Data Collection: Filebeat will be deployed on all Kubernetes nodes (as a DaemonSet) to automatically collect logs from all containers. Filebeat will add Kubernetes metadata (pod name, namespace, labels) to logs.

  • Data Processing: A Logstash cluster will be set up to receive data from Filebeat. Logstash will be responsible for:

  • Parsing different log formats (some services use JSON, some still use plain text).

  • Enriching data by adding GeoIP information for customer IP addresses.

  • Filtering out unnecessary logs (health checks).

  • Storage & Search: A 3-node Elasticsearch cluster (to ensure high availability) will be set up to store logs. ILM will be configured to manage data: 7 days on hot nodes, 30 days on warm nodes, and then delete.

  • Visualization: Kibana will be the main interface for developers and SREs. They will build dashboards to track error rates (HTTP 5xx), transaction processing times, and suspicious activities.

5.4 Implementation​

Here is a critical Logstash configuration snippet for FastMart to process logs from order-service (JSON) and shipping-service (legacy, plain text).

# /etc/logstash/conf.d/fastmart-pipeline.conf

input {
  beats {
    port => 5044
  }
}

filter {
  # Kubernetes metadata is added automatically by Filebeat's add_kubernetes_metadata processor
  # We use the 'kubernetes.pod.name' to route the log

  if "order-service" in [kubernetes][pod][name] {
    # order-service logs are in JSON
    json {
      source => "message"
      target => "event_data"
    }
    # Add a tag for easy filtering in Kibana
    mutate {
      add_tag => [ "order_service" ]
    }
  }
  else if "shipping-service" in [kubernetes][pod][name] {
    # shipping-service uses a custom plain text format
    # Example: [2023-10-27T14:10:00Z] | INFO | Shipment created for order_id=ord-987
    grok {
      match => { "message" => "\[%{TIMESTAMP_ISO8601:timestamp}\] \| %{LOGLEVEL:log.level} \| %{GREEDYDATA:log.message}" }
    }
    date {
      match => [ "timestamp", "ISO8601" ]
    }
    mutate {
      add_tag => [ "shipping_service" ]
    }
  } else {
    # Generic handling for other services
    mutate {
      add_tag => [ "unparsed_log" ]
    }
  }
}

output {
  elasticsearch {
    hosts => ["http://es-master:9200"]
    index => "fastmart-logs-%{+YYYY.MM.dd}"
    # Use ILM policy named 'fastmart_policy'
    ilm_enabled => true
    ilm_policy => "fastmart_policy"
  }
}

5.5 Results & Lessons Learned​

  • Improved Metrics:

  • MTTR reduced from an average of 2.5 hours to 25 minutes (improved by ~83%).

  • Developers can self-serve, searching their service logs without asking SRE, helping reduce internal support tickets by 40%.

  • Built real-time error rate dashboards (5xx), helping detect incidents within 5 minutes of occurrence.

  • Lessons Learned:

  • Standardize Logging is Key: Although Logstash can handle multiple formats, encouraging and enforcing development teams to follow a logging standard (structured JSON) from the start will make the system much simpler and more efficient in the long run.

  • ILM is Not Optional: Log storage costs increase very fast. Setting up and tuning ILM is a must-do from day one, not an "optimization" for later.

  • Training is Essential: Providing tools is not enough. FastMart organized training sessions for developers on how to use KQL to search effectively, helping them maximize the system's power.

References​