DEV Community: Daimus Suudi

Docker Logs Observability Using Grafana

Daimus Suudi — Wed, 19 Mar 2025 10:58:33 +0000

Managing logs is a developer’s responsibility—never be caught off guard when asked, “Where’s the error?” 😆 If you’re managing a single-instance container on a VPS, this article is for you. We’ll implement a simple yet effective solution that eliminates the need to log in via SSH just to check logs. With this setup, you can go beyond just viewing logs—you’ll be able to create dashboards, track metrics, perform searches, and even set up alerts.

Advantages of This Approach

With this setup, you don’t have to worry about managing log files, retention, or storage on your application. Simply log to the console, and Promtail will automatically collect and send the logs to Loki. But remember, you have to follow logging best practices to maximize the benefits of this observability stack:

Use appropriate logging levels (INFO, WARN, ERROR, etc.).
Write structured and meaningful log messages.
Keep logs concise, avoid excessive logging to prevent noise.
Never expose sensitive information in logs.

Why Use Grafana?

Grafana is a powerful open-source tool for monitoring, visualizing, and analyzing logs and metrics. It allows you to turn raw log data into meaningful insights with interactive dashboards. Unlike traditional log viewing methods that require manual SSH access, Grafana provides a centralized interface to track system performance and detect anomalies in real-time.

However, Grafana alone is not enough for log observability. We need a complete stack that includes:

Grafana: The visualization and dashboard tool that helps you explore log data and set up alerts.
Loki: A log aggregation system designed for efficiency and scalability, acting as a backend for storing and indexing logs.
Promtail: A lightweight log collector that ships logs from Docker containers to Loki.

All three components can run on Docker. In the next section, I’ll provide a simple Docker Compose setup to deploy them easily.

But before that, you need to configure Loki and Promtail. Here is the configuration for Loki.

auth_enabled: false

server:
  http_listen_port: 3100
  grpc_listen_port: 9096
  log_level: debug
  grpc_server_max_concurrent_streams: 1000

common:
  instance_addr: 127.0.0.1
  path_prefix: /tmp/loki
  storage:
    filesystem:
      chunks_directory: /tmp/loki/chunks
      rules_directory: /tmp/loki/rules
  replication_factor: 1
  ring:
    kvstore:
      store: inmemory

query_range:
  results_cache:
    cache:
      embedded_cache:
        enabled: true
        max_size_mb: 100

limits_config:
  metric_aggregation_enabled: true

schema_config:
  configs:
    - from: 2020-10-24
      store: tsdb
      object_store: filesystem
      schema: v13
      index:
        prefix: index_
        period: 24h

pattern_ingester:
  enabled: true
  metric_aggregation:
    loki_address: localhost:3100

ruler:
  alertmanager_url: http://localhost:9093

frontend:
  encoding: protobuf
# By default, Loki will send anonymous, but uniquely-identifiable usage and configuration
# analytics to Grafana Labs. These statistics are sent to https://stats.grafana.org/
#
# Statistics help us better understand how Loki is used, and they show us performance
# levels for most users. This helps us prioritize features and documentation.
# For more information on what's sent, look at
# https://github.com/grafana/loki/blob/main/pkg/analytics/stats.go
# Refer to the buildReport method to see what goes into a report.
#
# If you would like to disable reporting, uncomment the following lines:
#analytics:
#  reporting_enabled: false

Save this configuration somewhere on your server. Next, create a Promtail configuration to collect Docker logs and send them to Loki.

server:
  http_listen_port: 9080
  grpc_listen_port: 0

positions:
  filename: /tmp/positions.yaml

clients:
  - url: http://loki:3100/loki/api/v1/push

scrape_configs:
  - job_name: docker-logs
    docker_sd_configs:
      - host: unix:///var/run/docker.sock
        refresh_interval: 5s
    relabel_configs:
      - source_labels: [__meta_docker_container_name]
        target_label: container
      - source_labels: [__meta_docker_image]
        target_label: image

Next, create the Docker Compose configuration.

version: "3.3"

networks:
  loki:

services:
  loki:
    image: grafana/loki:latest
    ports:
      - "3100:3100"
    command: -config.file=/etc/loki/local-config.yaml
    volumes:
      - ./loki-config.yml:/etc/loki/local-config.yaml
    networks:
      - loki

  promtail:
    image: grafana/promtail:latest
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - ./promtail-config.yml:/etc/promtail/config.yml
    command: -config.file=/etc/promtail/config.yml
    networks:
      - loki

  grafana:
    environment:
      - GF_PATHS_PROVISIONING=/etc/grafana/provisioning
      - GF_AUTH_ANONYMOUS_ENABLED=false
      - GF_FEATURE_TOGGLES_ENABLE=alertingSimplifiedRouting,alertingQueryAndExpressionsStepMode
    entrypoint:
      - sh
      - -euc
      - |
        mkdir -p /etc/grafana/provisioning/datasources
        cat <<EOF > /etc/grafana/provisioning/datasources/ds.yaml
        apiVersion: 1
        datasources:
        - name: Loki
          type: loki
          access: proxy
          orgId: 1
          url: http://loki:3100
          basicAuth: false
          isDefault: true
          version: 1
          editable: false
        EOF
        /run.sh
    image: grafana/grafana:latest
    ports:
      - "3000:3000"
    networks:
      - loki

Fire up all three services with the following command:

docker-compose up -d

You can verify that Loki is up and running.

To view readiness, navigate to http://your-server-ip:3100/ready.
To view metrics, navigate to http://your-server-ip:3100/metrics.

Next, you can access Grafana by opening http://your-server-ip:3000 in your browser. By default, the login credentials are:

Username: admin
Password: admin

Upon your first login, Grafana will prompt you to change the password for security reasons. Once inside, you can start adding Loki as a data source and explore your logs in a more interactive way. 🚀

Minimal Automatic Deployment Using DockerHub Webhook

Daimus Suudi — Tue, 18 Mar 2025 08:15:27 +0000

Automating deployment is crucial for streamlining the development workflow. Instead of manually deploying code every time a change is pushed to the repository, an automated deployment system ensures a seamless and efficient process.

However, not all teams have access to advanced infrastructure like Kubernetes, CI/CD platforms, or expensive cloud services. Many developers rely on free repository hosting (such as GitHub and Docker Hub) and old-school VPS servers with minimal cost and resources. In such cases, a lightweight, webhook-based deployment approach provides a simple and effective solution.

This tutorial covers setting up an automated deployment pipeline using Docker Hub webhooks and a webhook server, making it ideal for low-cost, minimal-resource environments.

PS: Building a Docker image is a separate process that can be done on your local machine or through an external CI/CD system. Once the image is built and pushed to Docker Hub, the next step is to automate its deployment. This is where webhooks come into play.

Create Deployment Script

The first step is to create a deployment script. This script will handle removing the existing container, pulling the latest image, and running the updated container.

#!/bin/bash

docker stop <container_name> || true
docker rm <container_name> || true
docker pull <image_tag>
docker run -d --name <container_name> <image_tag>

Save the script in a suitable location on your server, and don’t forget to make it executable

chmod +x deploy.sh

Install & Configure Webhook

The next step is to install and configure the webhook. I am using the webhook package from https://github.com/adnanh/webhook. First, install it using the following command:

sudo apt update && sudo apt install -y webhook

Next, we need to create a configuration file for the webhook. This configuration defines a series of actions triggered when the webhook is called. You can also send payloads, perform validation, and more.

[
  {
    "id": "deploy",
    "execute-command": "/home/admin/scripts/deploy.sh",
    "command-working-directory": "/home/admin/scripts"
  }
]

Save the configuration file on your server, then start the webhook server.

webhook -verbose -hotreload -hooks=/home/admin/webhook/hooks.json

It will start up on default port 9000 and will provide you with one HTTP endpoint

http://yourserver:9000/hooks/{hook_id}

By performing HTTP GET or POST request to that endpoint, your specified script would be executed.

You can change the ip, port, protocol etc. Documentation is your friend 😉.

Setting In Docker Hub

Next, we need to add a webhook to Docker Hub by following these steps:

Login into your docker hub account
Go to your repository
Go to Webhooks tabs
Insert webhook name and URL and hit (+) icon

Voilà! From now on, whenever a new image is pushed, Docker Hub will send a webhook to your server, triggering the deployment script automatically. 🚀