DEV Community

Said Olano
Said Olano

Posted on

Mastering the ELK Stack: A Practical Guide to Elasticsearch, Logstash, and Kibana (2026-09-07 21:20)

#elasticsearch #logstash #kibana #devops

Mastering the ELK Stack: A Practical Guide to Elasticsearch, Logstash, and Kibana

Modern applications generate enormous volumes of log data across distributed systems. Without a centralized approach to collecting, storing, and analyzing this data, troubleshooting becomes a nightmare. The ELK Stack — comprising Elasticsearch, Logstash, and Kibana — has become the industry standard for log aggregation and analytics.

In this post, we'll explore each component, how they fit together, and how to build a working pipeline.

What Is the ELK Stack?

The ELK Stack is a collection of three open-source tools maintained by Elastic:

  • Elasticsearch — A distributed search and analytics engine that stores and indexes data.
  • Logstash — A data processing pipeline that ingests, transforms, and forwards data.
  • Kibana — A visualization layer for exploring and dashboarding your data.

A common extension is Beats, lightweight data shippers that push data to Logstash or Elasticsearch. When Beats is included, the stack is often called the Elastic Stack.

[Application Logs] --> [Beats/Logstash] --> [Elasticsearch] --> [Kibana]
Enter fullscreen mode Exit fullscreen mode

Elasticsearch: The Search and Storage Engine

Elasticsearch is built on Apache Lucene and provides near real-time search over structured and unstructured data. Data is stored as JSON documents, grouped into indices, and distributed across shards for scalability.

Indexing a Document 

curl -X POST "localhost:9200/logs-2024/_doc" \
  -H 'Content-Type: application/json' \
  -d '{
    "timestamp": "2024-05-01T12:00:00Z",
    "level": "ERROR",
    "service": "payment-api",
    "message": "Failed to process transaction"
  }'
Enter fullscreen mode Exit fullscreen mode

Searching Documents 

curl -X GET "localhost:9200/logs-2024/_search" \
  -H 'Content-Type: application/json' \
  -d '{
    "query": {
      "match": { "level": "ERROR" }
    }
  }'
Enter fullscreen mode Exit fullscreen mode

Elasticsearch's inverted index makes full-text queries extremely fast, even across millions of records.

Logstash: The Data Processing Pipeline

Logstash ingests data from many sources, transforms it, and ships it to Elasticsearch. A pipeline is defined with three sections: input, filter, and output.

Example Pipeline 

input {
  beats {
    port => 5044
  }
}

filter {
  grok {
    match => { "message" => "%{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:level} %{GREEDYDATA:log_message}" }
  }
  date {
    match => [ "timestamp", "ISO8601" ]
  }
}

output {
  elasticsearch {
    hosts => ["http://localhost:9200"]
    index => "logs-%{+YYYY.MM.dd}"
  }
}
Enter fullscreen mode Exit fullscreen mode

The grok filter is one of Logstash's most powerful features, parsing unstructured log lines into structured fields using pattern matching.

Tip: For high-throughput environments, consider using Filebeat to handle collection and reserve Logstash for heavy transformations. This reduces resource overhead on your application servers.

Kibana: Visualization and Exploration

Kibana provides a web interface (default port 5601) to search, visualize, and manage your Elasticsearch data.

Key features include:

  • Discover — Explore raw documents and run ad-hoc queries.
  • Visualize — Build charts, histograms, and metrics.
  • Dashboard — Combine visualizations into a single view.
  • Dev Tools — Run Elasticsearch queries directly from the browser.

Creating an Index Pattern

Before visualizing data, you define an index pattern (e.g., logs-*) so Kibana knows which indices to query. Once configured, you can build dashboards that update in near real time.

Setting Up the Stack with Docker Compose

The fastest way to experiment locally is with Docker Compose:

version: "3.8"
services:
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:8.13.0
    environment:
      - discovery.type=single-node
      - xpack.security.enabled=false
    ports:
      - "9200:9200"

  logstash:
    image: docker.elastic.co/logstash/logstash:8.13.0
    volumes:
      - ./logstash.conf:/usr/share/logstash/pipeline/logstash.conf
    ports:
      - "5044:5044"
    depends_on:
      - elasticsearch

  kibana:
    image: docker.elastic.co/kibana/kibana:8.13.0
    environment:
      - ELASTICSEARCH_HOSTS=http://elasticsearch:9200
    ports:
      - "5601:5601"
    depends_on:
      - elasticsearch
Enter fullscreen mode Exit fullscreen mode

Run it with:

docker compose up -d
Enter fullscreen mode Exit fullscreen mode

Best Practices for Production

  1. Use Index Lifecycle Management (ILM) to automate rollover, retention, and deletion of old indices.
  2. Secure your cluster with TLS, authentication, and role-based access control.
  3. Right-size your shards — too many small shards waste resources; too few large shards limit parallelism.
  4. Monitor the cluster using the built-in Stack Monitoring features to track heap usage and query latency.
  5. Buffer with a message queue (e.g., Kafka or Redis

Top comments (0)