Satyam Ahirrao

Posted on Sep 8

🚀 Building a Linux Observability Platform with ELK Stack — From systemd Journald to Kibana

#linux #devops #cloud #devopsprojects

I recently built a hands-on Linux Observability Platform using the Elastic Stack, Docker, systemd, and systemd-journald.

The goal wasn't just to run Elasticsearch and Kibana.

I wanted to build the complete log pipeline from a real Linux workstation → ingestion → processing → storage → visualization.

🏗️ Architecture

Linux Host
   │
   ▼
systemd-journald
   │
   ▼
journalctl -o json
   │
   ▼
socat TCP Bridge
   │
   │ TCP :5000
   ▼
Logstash
   │
   ▼
ECS-style normalization
   │
   ▼
Elasticsearch
   │
   ▼
Kibana
   ├── Discover
   ├── KQL
   └── Dashboards

🔧 What I implemented

Elasticsearch 9.5.3
Logstash 9.5.3
Kibana 9.5.3
Docker & Docker Compose
Real Linux systemd-journald events
journalctl JSON forwarding
socat TCP forwarding
systemd-managed journal forwarder
ECS-style structured fields
Elasticsearch daily indices
Kibana Discover
KQL-based severity filtering
Log-volume dashboards
Logs grouped by service and systemd unit

🧩 One of the interesting challenges

My initial plan was to have Logstash consume journald directly using the journald input plugin.

That didn't work as expected.

The logstash-input-journald plugin was incompatible with the Logstash 9.5.3/JRuby environment I was using.

Instead of forcing the broken integration, I redesigned the ingestion layer:

journalctl
    ↓
socat
    ↓
TCP
    ↓
Logstash

This gave me a much simpler and more reliable host-to-container boundary.

🐛 Other problems I encountered

This project also involved some real troubleshooting:

🔹 Docker overlay2 problems caused by a kernel/modules mismatch

🔹 Docker credential-helper configuration pointing to a missing helper

🔹 Logstash pipeline conflicts caused by an accidental .conf.bak file being loaded

🔹 Elasticsearch mapping conflicts during early testing

🔹 Managing the journal forwarding process through systemd

These were probably more valuable than simply following a tutorial because I had to investigate the failure, understand the root cause, and redesign the solution.

🔍 Real log validation

Instead of generating fake log files, I tested the complete pipeline with actual Linux journal events:

logger "ELK LAB - REAL JOURNAL PIPELINE TEST"

And tested error-level events:

logger -p user.err "ELK TEST - simulated ERROR event"

Then verified the events through:

journald
   ↓
journalctl
   ↓
socat
   ↓
Logstash
   ↓
Elasticsearch
   ↓
Kibana

📊 Kibana

I created dashboards to visualize:

Linux log volume
Logs by service
Logs by systemd unit
High-severity events

Example KQL:

log.syslog.priority <= 3

This allows operationally important events to be isolated quickly.

🧠 What I learned

This project helped me strengthen my understanding of:

Linux logging architecture
systemd and journald
Docker networking
Logstash pipelines
Elasticsearch mappings
structured logging
ECS-style schemas
KQL
troubleshooting containerized infrastructure
systemd service management
Git/GitHub project organization

The biggest takeaway:

When an integration doesn't work, don't just keep forcing it. Understand the failure and design a better boundary.

📸 Project Screenshots

Docker containers

Elasticsearch health

Logstash events

Systemd journal forwarder

ECS-style document

Kibana Discover

Kibana Dashboard

🔗 Project

Full source code, configuration, screenshots, and documentation:

GitHub — Linux Observability Platform

Linux #DevOps #LinuxAdministration #ELK #Elasticsearch #Logstash #Kibana #Docker #Systemd #Observability #SRE #Cloud #DevOpsProjects #OpenSource

DEV Community