DEV Community

Cover image for How to Stream Container Logs to Loki
Raizan
Raizan

Posted on Originally published at chasebot.online

How to Stream Container Logs to Loki

#devops #tutorial #docker #logging

What You'll Need

  • Hetzner VPS or Contabo VPS running Ubuntu 22.04 or 24.04
  • DigitalOcean droplets as an alternative cloud infrastructure option
  • Namecheap for domain management and reverse proxy routing
  • n8n Cloud or self-hosted n8n for triggering downstream log alerts
  • Docker Engine version 24.0 or higher and Docker Compose V2

Table of Contents

Architecture Overview: Docker, Loki, and Promtail

Managing logs across dozens of running containers quickly becomes impossible if you rely solely on docker logs. Centralized log aggregation gives you total visibility into microservice interactions, runtime errors, and performance bottlenecks.

Grafana Loki is a horizontally scalable, highly available, multi-tenant log aggregation system inspired by Prometheus. Unlike Elasticsearch, Loki does not index the text of the logs. Instead, it indexes labels for your log streams, leaving the raw log payload unindexed. This architecture significantly reduces storage costs and memory usage while keeping query speeds exceptionally fast.

There are two primary methods to stream Docker container logs to Loki:

  1. Promtail Container Agent (Recommended): Promtail runs as a dedicated daemon container, mounts the host system Docker socket /var/run/docker.sock or container log paths at /var/lib/docker/containers/, automatically discovers running containers, extracts metadata labels, and pushes formatted logs to Loki.
  2. Docker Loki Logging Plugin: The Docker engine itself ships container standard output (stdout) and standard error (stderr) directly to Loki over the network using a native plugin driver configured at the container or host daemon level.

Using Promtail is typically superior for production setups because it decouples log collection from the application runtime. If Loki experiences network hiccups, Promtail buffers logs locally without blocking container execution.

Before deploying containerized services on your server, make sure your host system is secure. Following our guide on Setting Up Fail2ban to Protect Linux Cloud Servers helps secure your Linux host before opening additional network sockets for log collectors.

Step 1: Deploying Grafana Loki and Grafana Stack

To build our log collection pipeline, we will start by deploying Loki, Grafana, Promtail, and an automated log generator service using Docker Compose on a fresh Hetzner VPS instance.

First, create a project directory on your host server to keep configuration files clean:

mkdir -p /opt/loki-stack
cd /opt/loki-stack
Enter fullscreen mode Exit fullscreen mode

Create the Loki configuration file named loki-config.yaml:

auth_enabled: false

server:
  http_listen_port: 3100
  grpc_listen_port: 9096

common:
  path_prefix: /tmp/loki
  storage:
    filesystem:
      chunks_directory: /tmp/loki/chunks
      rules_directory: /tmp/loki/rules
  replication_factor: 1
  ring:
    kvstore:
      store: inmemory

schema_config:
  configs:
    - from: 2024-01-01
      store: tsdb
      object_store: filesystem
      schema: v13
      index:
        prefix: index_
        period: 24h

ruler:
  alertmanager_url: http://localhost:9093

limits_config:
  reject_old_samples: true
  reject_old_samples_max_age: 168h
  ingestion_rate_mb: 10
  ingestion_burst_size_mb: 20
Enter fullscreen mode Exit fullscreen mode

Next, create the docker-compose.yml file to orchestrate our infrastructure stack:

version: "3.8"

networks:
  loki-net:
    driver: bridge

services:
  loki:
    image: grafana/loki:2.9.4
    container_name: loki
    ports:
      - "3100:3100"
    volumes:
      - ./loki-config.yaml:/etc/loki/loki-config.yaml
    command: -config.file=/etc/loki/loki-config.yaml
    networks:
      - loki-net
    restart: unless-stopped

  grafana:
    image: grafana/grafana:10.2.3
    container_name: grafana
    ports:
      - "3000:3000"
    environment:
      - GF_SECURITY_ADMIN_USER=admin
      - GF_SECURITY_ADMIN_PASSWORD=SuperSecretPassword123!
    volumes:
      - grafana-data:/var/lib/grafana
    networks:
      - loki-net
    restart: unless-stopped

  flog:
    image: mingrammer/flog:0.4.3
    container_name: sample-app
    command: -f json -d 1s -loop
    networks:
      - loki-net
    restart: unless-stopped

volumes:
  grafana-data:
Enter fullscreen mode Exit fullscreen mode

Start the initial service stack:

docker compose up -d loki grafana flog
Enter fullscreen mode Exit fullscreen mode

Verify that all three services are running cleanly using docker compose ps. Loki will begin listening on port 3100, while Grafana is accessible on port 3000.

💡 Fast-Track Your Project: Don't want to configure this yourself? I build custom n8n pipelines and bots. Message me with code SYS3-DEVTO.

Step 2: Configuring Promtail Docker Engine Scraping

Now that Loki and Grafana are operational, we need to add Promtail to auto-discover every running container, extract its standard output, attach metadata tags, and deliver the streams to Loki.

Create a file named promtail-config.yaml in /opt/loki-stack/:

server:
  http_listen_port: 9080
  grpc_listen_port: 0

positions:
  filename: /tmp/positions.yaml

clients:
  - url: http://loki:3100/loki/api/v1/push

scrape_configs:
  - job_name: docker
    docker_sd_configs:
      - host: unix:///var/run/docker.sock
        refresh_interval: 5s
    relabel_configs:
      - source_labels: ['__meta_docker_container_name']
        regex: '/(.*)'
        target_label: 'container'
      - source_labels: ['__meta_docker_container_log_stream']
        target_label: 'stream'
      - source_labels: ['__meta_docker_container_label_com_docker_compose_service']
        target_label: 'service'
    pipeline_stages:
      - json:
          expressions:
            log: log
            stream: stream
            time: time
      - json:
          expressions:
            host: host
            user: user
            method: method
            status: status
            bytes: bytes
          source: log
      - labels:
          method:
          status:
Enter fullscreen mode Exit fullscreen mode

This configuration leverages Promtail Docker Service Discovery (docker_sd_configs) to watch the Docker UNIX socket directly. The relabel_configs stage strips leading slashes from container names and assigns useful labels like container and service.

In production microservice architectures, structuring application output is critical. For instance, when designing complex backend automation workflows, implementing structured payloads like those in Handling Structured Output with OpenAI Function Calling makes log extraction with Promtail pipeline stages far more effective. Similarly, if your services receive external standard callbacks, reviewing our guide on Implementing HMAC Signature Verification for Inbound Webhooks ensures you capture structured security failure events cleanly inside standard stdout.

Now, append the promtail service block into your existing docker-compose.yml file:

version: "3.8"

networks:
  loki-net:
    driver: bridge

services:
  loki:
    image: grafana/loki:2.9.4
    container_name: loki
    ports:
      - "3100:3100"
    volumes:
      - ./loki-config.yaml:/etc/loki/loki-config.yaml
    command: -config.file=/etc/loki/loki-config.yaml
    networks:
      - loki-net
    restart: unless-stopped

  promtail:
    image: grafana/promtail:2.9.4
    container_name: promtail
    volumes:
      - ./promtail-config.yaml:/etc/promtail/promtail-config.yaml
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - /var/lib/docker/containers:/var/lib/docker/containers:ro
    command: -config.file=/etc/promtail/promtail-config.yaml
    networks:
      - loki-net
    restart: unless-stopped

  grafana:
    image: grafana/grafana:10.2.3
    container_name: grafana
    ports:
      - "3000:3000"
    environment:
      - GF_SECURITY_ADMIN_USER=admin
      - GF_SECURITY_ADMIN_PASSWORD=SuperSecretPassword123!
    volumes:
      - grafana-data:/var/lib/grafana
    networks:
      - loki-net
    restart: unless-stopped

  flog:
    image: mingrammer/flog:0.4.3
    container_name: sample-app
    command: -f json -d 1s -loop
    networks:
      - loki-net
    restart: unless-stopped

volumes:
  grafana-data:
Enter fullscreen mode Exit fullscreen mode

Launch Promtail alongside your running services:

docker compose up -d promtail
Enter fullscreen mode Exit fullscreen mode

Check the logs for Promtail to ensure it has connected to the Docker API socket and established its connection with Loki:

docker compose logs -f promtail
Enter fullscreen mode Exit fullscreen mode

Step 3: Streaming Docker Logs via Native Docker Logging Plugin

As an alternative to Promtail, Docker provides an official plugin that forwards container stdout directly to Loki without requiring Promtail to read log files from host disk storage.

First, install the Docker Loki logging plugin on your host machine:

docker plugin install grafana/loki-docker-driver:2.9.4 --alias loki --grant-all-permissions
Enter fullscreen mode Exit fullscreen mode

Verify that the plugin is active:

docker plugin ls
Enter fullscreen mode Exit fullscreen mode

You can configure the Loki driver globally across all Docker containers on the server by editing /etc/docker/daemon.json. Create or modify /etc/docker/daemon.json with the following content:

{
  "log-driver": "loki",
  "log-opts": {
    "loki-url": "http://127.0.0.1:3100/loki/api/v1/push",
    "loki-batch-size": "400000",
    "loki-retries": "2",
    "loki-max-backoff": "800ms",
    "loki-timeout": "1s",
    "keep-file": "true"
  }
}
Enter fullscreen mode Exit fullscreen mode

Restart the Docker daemon to apply the global configuration:

sudo systemctl restart docker
Enter fullscreen mode Exit fullscreen mode

Alternatively, if you want to use the native driver only for specific containers rather than globally, define the logging options directly inside an individual service block in docker-compose.yml:

version: "3.8"

services:
  standalone-app:
    image: alpine
    container_name: alpine-runner
    command: sh -c "while true; do echo '{\"status\": 200, \"message\": \"System check operational\"}'; sleep 2; done"
    logging:
      driver: loki
      options:
        loki-url: "http://127.0.0.1:3100/loki/api/v1/push"
        loki-external-labels: "job=standalone-runner,environment=production"
Enter fullscreen mode Exit fullscreen mode

Step 4: Querying Docker Logs in Grafana using LogQL

Now that your container log streams are flowing into Loki, open Grafana by visiting http://your-server-ip:3000 in your web browser. Log in with the admin credentials defined in your Docker Compose file (admin / SuperSecretPassword123!).

Setting Up the Loki Data Source

  1. Navigate to Connections > Data Sources in the left sidebar.
  2. Click Add data source and select Loki.
  3. Set the Connection URL to http://loki:3100.
  4. Click Save & test. You should see a green confirmation badge stating that the data source is connected and receiving data.

Querying Logs with LogQL

Go to Explore in the Grafana menu and select Loki as your active data source. LogQL (Loki Query Language) uses simple selector expressions to slice and dice log streams.

To query all logs originating from our sample container:

{container="sample-app"}
Enter fullscreen mode Exit fullscreen mode

To filter log messages containing specific HTTP error responses:

{container="sample-app"} |= "404"
Enter fullscreen mode Exit fullscreen mode

To parse JSON-formatted log output dynamically on the fly and filter by parsed attributes:

{container="sample-app"} | json | status >= 500
Enter fullscreen mode Exit fullscreen mode

To calculate the per-second rate of logs generated by container over a 5-minute window:

rate({container="sample-app"}[5m])
Enter fullscreen mode Exit fullscreen mode

To aggregate log throughput grouped by service name:

sum(rate({job="docker"}[1m])) by (service)
Enter fullscreen mode Exit fullscreen mode

These queries form the backbone of production dashboards, letting you view real-time log tailing, error spikes, and response time metrics side by side.

Getting Started

Centralizing container logs with Loki and Promtail ensures your infrastructure remains reliable, searchable, and easy to debug. Deploying these services on reliable host platforms like a Hetzner VPS or DigitalOcean droplet gives you total control over log retention without cloud vendor lock-in.

If you require custom webhooks or DNS routing for your endpoints, manage your domains using Namecheap and link your log query outputs to n8n Cloud to build automated Slack, Telegram, or Email alerting systems whenever Loki registers a critical error stream.

Outsource Your Automation

Don't have time? I build production n8n workflows, WhatsApp bots, and fully automated YouTube Shorts pipelines. Hire me on Fiverr, mention SYS3-DEVTO for priority. Or DM at chasebot.online.

Originally published on Automation Insider.

Top comments (0)