DEV Community: Jakub Korečko

Part 5: Full Observability for Free — Grafana Alloy, Prometheus, and Loki

Jakub Korečko — Sat, 30 May 2026 08:55:57 +0000

What you'll learn:

Why Grafana Alloy replaces both Prometheus Agent and Promtail
What metrics are scraped from each service and why
How cardinality management keeps you inside Grafana Cloud's free tier
How Docker log discovery works with the Alloy pipeline
What to build in Grafana dashboards for this stack

One Container for Everything

Before Grafana Alloy, a standard observability setup required:

Prometheus or Prometheus Agent for metrics scraping and remote write
Promtail for log collection and Loki shipping
Separate configs, separate containers, separate log rotation concerns

Grafana Alloy is the successor to both. One container, one config file (grafana_alloy.alloy), and it handles metrics and logs in a single pipeline.

Why not a full Prometheus stack? Self-hosted Prometheus needs storage, retention config, and an alertmanager. For a single server, that's more infrastructure than the apps it monitors. Grafana Cloud's free tier gives you 14-day metric retention, 30-day log retention, and managed alerting — without running any of that yourself.

The free tier limits that matter:

10,000 active metric series
50GB logs/month
14 days metric retention

Aggressive metric filtering (covered below) keeps this stack well under those limits.

The Alloy Config Structure

Alloy uses a config language called River (similar to HCL). The full config is at apps/monitoring/grafana_alloy.alloy.

The pipeline follows this pattern:

prometheus.scrape → prometheus.relabel → prometheus.remote_write
loki.source       → loki.process      → loki.write

Each component is declared with a name and wired together via forward_to references. This makes the data flow explicit and easy to trace.

Metrics: What Gets Scraped

Traefik

prometheus.scrape "traefik" {
  targets = [{ __address__ = "traefik:8899" }]
  forward_to      = [prometheus.relabel.traefik.receiver]
  scrape_interval = "30s"
}

prometheus.relabel "traefik" {
  forward_to = [prometheus.remote_write.default.receiver]

  rule {
    source_labels = ["__name__"]
    regex = "(traefik_open_connections|traefik_entrypoint_requests_total|traefik_entrypoint_request_duration_seconds_sum|traefik_entrypoint_request_duration_seconds_bucket|traefik_entrypoint_request_duration_seconds_count|traefik_service_requests_total|traefik_service_request_duration_seconds_bucket|traefik_service_request_duration_seconds_sum|traefik_service_request_duration_seconds_count|traefik_service_requests_bytes_total|traefik_service_responses_bytes_total)"
    action = "keep"
  }
}

Traefik exposes metrics on port :8899 (the metrics entrypoint defined in static config). Raw Traefik output is ~50+ metric series. The relabeling rule keeps only 11 specific metrics:

Metric	What it tells you
`traefik_open_connections`	Active connections right now
`traefik_entrypoint_requests_total`	Total requests per entrypoint
`traefik_entrypoint_request_duration_seconds_*`	Latency distribution (histogram)
`traefik_service_requests_total`	Total requests per backend service
`traefik_service_request_duration_seconds_*`	Per-service latency histogram
`traefik_service_requests_bytes_total`	Request bytes per service
`traefik_service_responses_bytes_total`	Response bytes per service

This is enough to build:

Request rate dashboards (requests/sec per service)
Latency percentile panels (P50, P95, P99)
Error rate panels (compare 2xx vs 4xx/5xx from Traefik access logs)
Active connection gauges

Host Metrics (node-exporter)

prometheus.scrape "node_exporter" {
  targets = [{ __address__ = "node-exporter:9100" }]
  forward_to      = [prometheus.relabel.node_exporter.receiver]
  scrape_interval = "30s"
}

prometheus.relabel "node_exporter" {
  forward_to = [prometheus.remote_write.default.receiver]

  rule {
    source_labels = ["__name__"]
    regex = "(node_cpu_seconds_total|node_memory_(MemTotal|MemFree|MemAvailable|Buffers|Cached|SReclaimable|SwapTotal|SwapFree)_bytes|node_filesystem_(size|avail)_bytes|node_network_(receive|transmit)_bytes_total|node_load(1|5|15)|node_time_seconds|node_boot_time_seconds|node_disk_(read_bytes_total|written_bytes_total|io_time_seconds_total))"
    action = "keep"
  }
}

node-exporter by default exposes hundreds of metrics covering every kernel subsystem. The regex keep-rule filters to the essentials:

Category	Metrics kept
CPU	`node_cpu_seconds_total` (all modes)
Memory	Total, free, available, buffers, cached, swap
Filesystem	Size and available bytes per mount
Network	Receive/transmit bytes per interface
Load	1m, 5m, 15m load averages
Disk I/O	Read bytes, write bytes, I/O time per device
System	Uptime (boot time), clock

The node-exporter compose file further filters which collectors run:

# apps/monitoring/node_exporter.yaml
command:
  - '--collector.filesystem.mount-points-exclude=^/(sys|proc|dev|run|var/lib/docker).*'
  - '--collector.filesystem.fs-types-exclude=^(sysfs|procfs|autofs|cgroup|devtmpfs|devpts|tmpfs|nsfs|overlay|securityfs|tracefs)$'
  - '--collector.netdev.device-exclude=^(veth|docker|br-).*'

Docker-internal network interfaces (veth, docker0, bridge interfaces) are excluded. Virtual filesystems (tmpfs, overlay, etc.) are excluded. This prevents cardinality explosion from Docker's ephemeral per-container veth interfaces — each container creates a new veth, and without filtering you'd accumulate thousands of metric series over time.

Container Metrics (cAdvisor)

prometheus.relabel "cadvisor" {
  forward_to = [prometheus.remote_write.default.receiver]

  // Drop root/aggregate metrics (no container label)
  rule {
    source_labels = ["name"]
    regex = "^$"
    action = "drop"
  }

  // Extract container name from Docker Swarm task format
  rule {
    source_labels = ["name"]
    regex = "(.+)\\.(\\d+)\\.[a-zA-Z0-9]+$"
    target_label  = "container_name"
    replacement   = "$1.$2"
  }

  // Add service name label
  rule {
    source_labels = ["container_label_com_docker_swarm_service_name"]
    target_label  = "service_name"
  }

  // Keep only essential container metrics
  rule {
    source_labels = ["__name__"]
    regex = "(container_memory_usage_bytes|container_last_seen|container_cpu_user_seconds_total|container_network_(receive|transmit)_bytes_total|container_memory_cache|container_fs_(reads|writes)_bytes_total|container_cpu_usage_seconds_total)"
    action = "keep"
  }

  // Drop unused labels
  rule {
    regex  = "(__name__|container_name|service_name|job|instance)"
    action = "labelkeep"
  }
}

cAdvisor exports metrics for every container on the system, including Docker's internal containers. The first rule drops "aggregate" metrics that have no name label (cAdvisor's root-level stats). The final labelkeep rule drops all the verbose Docker label metadata that cAdvisor attaches — keeping only the labels that matter for querying.

The result: per-container CPU, memory, network, and disk I/O, labeled with service name and container name.

Logs: Docker and System

System Logs

local.file_match "system" {
  path_targets = [{
    __path__ = "/var/log/**/*log",
    job       = "varlogs",
  }]
}

loki.process "system" {
  forward_to = [loki.write.default.receiver]

  stage.drop {
    older_than = "1h0m0s"
  }
}

Alloy reads all *log files under /var/log/ (mounted from the host as read-only). The stage.drop rule discards log lines older than 1 hour — this prevents Alloy from re-shipping old logs after a restart, which would generate duplicate log entries in Grafana Cloud.

Docker Container Logs

discovery.docker "docker_swarm" {
  host             = "unix:///var/run/docker.sock"
  refresh_interval = "5s"
}

discovery.relabel "docker_swarm" {
  targets = []

  rule {
    source_labels = ["__meta_docker_container_name"]
    regex         = "(.+)\\.(\\d+)\\.[a-zA-Z0-9]+$"
    target_label  = "container_name"
  }

  rule {
    source_labels = ["__meta_docker_service_name"]
    target_label  = "service_name"
  }

  rule {
    source_labels = ["__meta_docker_service_label_com_docker_stack_namespace"]
    target_label  = "stack_name"
  }

  rule {
    source_labels = ["__meta_docker_container_id"]
    target_label  = "__path__"
    replacement   = "/var/lib/docker/containers/$1/$1-json.log"
  }
}

loki.source.docker "docker_swarm" {
  host          = "unix:///var/run/docker.sock"
  targets       = discovery.docker.docker_swarm.targets
  forward_to    = [loki.process.docker_swarm.receiver]
  relabel_rules = discovery.relabel.docker_swarm.rules
}

Docker Swarm container names follow the pattern stackname_servicename.replicanumber.taskid. The regex (.+)\.(\d+)\.[a-zA-Z0-9]+$ extracts a stable name (without the random task ID) so log streams from the same service don't fragment across container restarts.

The stack_name, service_name, and container_name labels are added to every log line, making Loki queries like {stack_name="traefik"} or {service_name="traefik_traefik"} work correctly.

cAdvisor and node-exporter Compose Files

Both exporters have carefully tuned compose configurations.

cAdvisor

# apps/monitoring/cadvisor.yaml (key sections)
services:
  cadvisor:
    image: ghcr.io/google/cadvisor:0.56.2
    command:
      - --docker_only=true
      - --housekeeping_interval=30s
      - --max_housekeeping_interval=35s
      - --global_housekeeping_interval=10m
      - --storage_duration=10m
    deploy:
      resources:
        limits:
          memory: 128M
        reservations:
          memory: 64M
      restart_policy:
        condition: on-failure
        delay: 5s
        max_attempts: 3

--docker_only=true restricts cAdvisor to Docker containers only (no system processes). --housekeeping_interval=30s matches the Alloy scrape interval — collecting at a higher frequency than the scrape interval provides no benefit. Memory is capped at 128M because cAdvisor has a known tendency to grow unbounded on busy systems.

restart_policy: condition: on-failure, max_attempts: 3 rather than always — repeated failures indicate a persistent issue that warrants investigation rather than indefinite restart attempts.

node-exporter

services:
  node-exporter:
    image: prom/node-exporter:v1.10.2
    volumes:
      - /proc:/host/proc:ro
      - /sys:/host/sys:ro
      - /:/rootfs:ro
    command:
      - '--path.procfs=/host/proc'
      - '--path.sysfs=/host/sys'
      - '--path.rootfs=/rootfs'

node-exporter needs read-only access to /proc, /sys, and the filesystem root to collect host metrics. It runs in the host PID namespace to see all processes. The :ro mounts ensure it can only read, never write.

Remote Write to Grafana Cloud

Both Prometheus and Loki ship to Grafana Cloud via basic auth:

prometheus.remote_write "default" {
  endpoint {
    url = "<YOUR_GRAFANA_CLOUD_PROMETHEUS_URL>/api/prom/push"

    basic_auth {
      username      = "<YOUR_GRAFANA_CLOUD_METRICS_USERNAME>"
      password_file = "/run/secrets/grafana_cloud_passwd"
    }
  }
}

loki.write "default" {
  endpoint {
    url = "<YOUR_GRAFANA_CLOUD_LOKI_URL>/loki/api/v1/push"

    basic_auth {
      username      = "<YOUR_GRAFANA_CLOUD_LOGS_USERNAME>"
      password_file = "/run/secrets/grafana_cloud_passwd"
    }
  }
}

The grafana_cloud_passwd is an API key from Grafana Cloud. It's stored encrypted in the Git repo (SOPS) and decrypted by SwarmCD at deploy time into a Docker secret. The same password is shared between the Prometheus and Loki write endpoints (Grafana Cloud uses the same credential for both).

Suggested Grafana Dashboard Panels

Once data is flowing to Grafana Cloud, here are the most useful panels to build:

Server Overview:

CPU usage (% of total, per-core breakdown)
Memory usage (total / available / used)
Disk usage per mount point
Network traffic (bytes/sec in/out)
System uptime

Traefik:

Requests/sec per service (rate of traefik_service_requests_total)
Error rate (4xx + 5xx as % of total)
P95 latency per service (histogram from traefik_service_request_duration_seconds_bucket)
Active open connections

Container Resources:

Memory usage per service (container_memory_usage_bytes by service_name)
CPU usage per service (rate of container_cpu_usage_seconds_total)
Container restarts (changes in container_last_seen gaps)

Logs (Grafana Explore or dashboard panels):

{stack_name="traefik"} — Traefik access logs
{service_name="swarmcd_swarmcd"} — SwarmCD deploy events
{job="varlogs"} — System logs (SSH, fail2ban bans, etc.)

Summary

The observability stack in this setup achieves:

Metrics from 3 sources (Traefik, node-exporter, cAdvisor) shipped to Grafana Cloud Prometheus
Logs from 2 sources (Docker containers, system) shipped to Grafana Cloud Loki
One container (Grafana Alloy) handling all of the above
Zero cost — Grafana Cloud free tier is sufficient with aggressive metric filtering
GitOps-deployed — changes to grafana_alloy.alloy trigger automatic redeployment via SwarmCD

The metric filtering is the key insight: Prometheus exporters expose far more data than you need. Being selective about what you ship keeps you within free tier limits and makes dashboards faster to query.

All source code: gitlab.com/sakonn/docker-swarm-gitops

Part 4: Protecting Public Traffic — Traefik, CrowdSec WAF, and Tailscale VPN

Jakub Korečko — Sat, 30 May 2026 08:51:00 +0000

What you'll learn:

The complete traffic flow from user browser to container
How Traefik handles TLS termination, routing, and zero-downtime updates
Why services opt-in to exposure via Docker labels
How CrowdSec adds a WAF and IP reputation layer to every request
How Tailscale VPN secures admin access without opening SSH to the internet

The Threat Model

A public internet server is subject to continuous automated scanning and attack attempts. Within minutes of a new IP address becoming reachable:

Automated scanners probe every common port
Bots attempt SSH brute-force (hundreds of attempts per hour)
Crawlers look for exposed admin interfaces (wp-admin, /actuator, .env, etc.)
Malicious requests probe for SQL injection, XSS, and other OWASP vulnerabilities

The security architecture in this stack addresses each of these without requiring a separate security engineer. Here's the full picture:

Internet
   │
   ▼
Cloudflare (DNS proxy)
   │  Hides server IP, DDoS mitigation, HTTPS at edge
   │
   ▼
Hetzner Firewall
   │  Allows: 80, 443 only. Blocks everything else at network level.
   │
   ▼
fail2ban (host)
   │  Bans IPs after 3 failed SSH attempts (1h ban)
   │
   ▼
Traefik (port 80 / 443)
   │  TLS termination, HTTP→HTTPS redirect, real IP forwarding
   │
   ▼
CrowdSec bouncer (Traefik plugin)
   │  IP reputation check + AppSec WAF rules (SQLi, XSS, etc.)
   │  Block decision: 60s default ban
   │
   ▼
Application (bento, etc.)

Admin traffic takes a different path entirely — through Tailscale VPN, bypassing the public internet stack.

Traefik: The Entry Point for All Traffic

Traefik is the reverse proxy that sits in front of all applications. It handles:

TLS certificate acquisition and renewal (Let's Encrypt, automated)
HTTP to HTTPS redirection
Routing requests to the correct backend service
Running the CrowdSec bouncer plugin

Static Configuration

apps/traefik/traefik_static_conf.yaml defines the entrypoints, providers, and plugins that are loaded once at startup.

Entrypoints:

entryPoints:
  web:
    address: :80
    http:
      redirections:
        entryPoint:
          to: websecure
          scheme: https
    forwardedHeaders:
      trustedIPs: &trustedIps
        - 103.21.244.0/22
        - 104.16.0.0/13
        # ... all Cloudflare IP ranges

  websecure:
    address: :443
    forwardedHeaders:
      trustedIPs: *trustedIps
    transport:
      respondingTimeouts:
        readTimeout: 600s
        writeTimeout: 600s

  metrics:
    address: :8899

Port 80 (web) redirects all traffic to 443 and trusts Cloudflare's IP ranges for the X-Forwarded-For header. Without this trustedIPs configuration, Traefik would see Cloudflare's IP as the client IP — meaning CrowdSec would evaluate Cloudflare's infrastructure, not the actual user. By trusting Cloudflare's ranges, Traefik unwraps the X-Forwarded-For header to get the real client IP.

Port 443 (websecure) has 600-second timeouts to support long-running operations like PDF generation in the Bento app.

Port 8899 (metrics) exposes Prometheus metrics for Grafana Alloy to scrape. This port is not in the Hetzner firewall allow-list and is not accessible from the public internet — Alloy scrapes it from inside the overlay network.

Certificate resolvers:

certificatesResolvers:
  staging:
    acme:
      email: <YOUR_EMAIL>
      caServer: "https://acme-staging-v02.api.letsencrypt.org/directory"
      httpChallenge:
        entryPoint: web

  production:
    acme:
      email: <YOUR_EMAIL>
      caServer: "https://acme-v02.api.letsencrypt.org/directory"
      httpChallenge:
        entryPoint: web

Two resolvers exist: staging (for testing — will not exceed Let's Encrypt rate limits) and production (real certificates). Services specify which resolver to use in their labels. During initial setup, use staging to validate the configuration, then switch to production.

Providers:

providers:
  swarm:
    exposedByDefault: false
  docker:
    exposedByDefault: false
  file:
    directory: /etc/traefik
    watch: true

exposedByDefault: false means Traefik ignores all containers unless they have traefik.enable=true in their labels. A service added to Swarm without this label will not be exposed publicly. Every exposure is explicit and intentional.

CrowdSec plugin:

experimental:
  plugins:
    bouncer:
      moduleName: "github.com/maxlerebourg/crowdsec-bouncer-traefik-plugin"
      version: "v1.5.0"

The plugin is declared here in static config. Its configuration (which requests it applies to, which CrowdSec instance it talks to) is in the dynamic config.

Dynamic Configuration

apps/traefik/traefik_dynamic_conf.yaml defines middlewares and routes that Traefik watches for changes without restarting:

http:
  middlewares:
    auth:
      basicAuth:
        users:
          - <USERNAME>:<BCRYPT_HASH>

    crowdsec:
      plugin:
        bouncer:
          enabled: true
          crowdsecMode: live
          crowdsecAppsecEnabled: true
          crowdsecAppsecHost: crowdsec_crowdsec:7422
          crowdsecAppsecFailureBlock: true
          crowdsecLapiKeyFile: "/run/secrets/crowdsec_api_key"
          crowdsecLapiHost: crowdsec_crowdsec:8080
          forwardedHeadersTrustedIPs:
            - 10.0.0.0/8
            - 172.16.0.0/12
            - 192.168.0.0/16
          clientTrustedIPs:
            - 10.0.0.0/8
            - 172.16.0.0/12
            - 192.168.0.0/16

The crowdsec middleware is defined once here and referenced by any service that wants WAF protection. The auth middleware is used for any internal service (like the Traefik dashboard) that should be behind basic auth.

Zero-Downtime Updates

In the Traefik compose file, the update strategy is:

deploy:
  update_config:
    order: start-first

start-first means Docker Swarm starts the new Traefik container before stopping the old one. During the overlap window, the new container is running and healthy before the old one receives the stop signal. This means Traefik updates happen with no dropped requests.

Combined with SwarmCD's immutable config versioning (Part 3), every configuration change to Traefik is zero-downtime.

Service Exposure via Docker Labels

Here's how a service opts into public access. From apps/bento/bento.yaml:

services:
  bento:
    image: ghcr.io/alam00000/bentopdf-simple:v2.7.0
    networks:
      - swarm_network
    deploy:
      labels:
        - "traefik.enable=true"
        - "traefik.http.routers.bento-http.rule=Host(`pdf.yourdomain.com`)"
        - "traefik.http.routers.bento-http.entrypoints=web"
        - "traefik.http.routers.bento-http.middlewares=redirect-to-https@file"
        - "traefik.http.routers.bento.rule=Host(`pdf.yourdomain.com`)"
        - "traefik.http.routers.bento.entrypoints=websecure"
        - "traefik.http.routers.bento.tls.certresolver=production"
        - "traefik.http.routers.bento.middlewares=crowdsec@file"
        - "traefik.http.services.bento.loadbalancer.server.port=8080"

Breaking this down:

traefik.enable=true — opts in to Traefik management
Two routers: one for HTTP (redirect to HTTPS), one for HTTPS
tls.certresolver=production — request a production Let's Encrypt certificate for this hostname
middlewares=crowdsec@file — all requests to this service pass through the CrowdSec bouncer
server.port=8080 — Traefik forwards to this container port

Notice that labels go under deploy: not under services: in Swarm mode. This is a Docker Swarm requirement — service labels (the ones Traefik watches) must be deployment labels, not container labels.

CrowdSec: WAF and IP Reputation

CrowdSec adds two protection layers to every request passing through Traefik:

LAPI (Local API) — IP Reputation:
CrowdSec maintains a local database of banned IP addresses. This database is populated from:

The CrowdSec community threat intelligence feed (millions of crowdsourced malicious IPs)
Local detections (if you run CrowdSec agents on the host)

When a request arrives, the bouncer plugin checks the source IP against the LAPI. If it's in the ban list, the request is blocked immediately with a 403.

AppSec — WAF Rules:
CrowdSec's AppSec component applies request inspection rules that block common attack patterns:

SQL injection (e.g., ' OR 1=1 -- in query parameters)
XSS (e.g., <script>alert(1)</script> in form fields)
Path traversal (e.g., ../../../etc/passwd)
Known CVE exploit patterns for common web frameworks

crowdsec:
  plugin:
    bouncer:
      crowdsecAppsecEnabled: true
      crowdsecAppsecHost: crowdsec_crowdsec:7422
      crowdsecAppsecFailureBlock: true  # Block if AppSec is unreachable

crowdsecAppsecFailureBlock: true means that if the AppSec engine is unavailable (container restart, etc.), requests are blocked rather than allowed through. This is a fail-closed posture — prefer availability loss over security bypass.

Internal traffic bypass:

clientTrustedIPs:
  - 10.0.0.0/8
  - 172.16.0.0/12
  - 192.168.0.0/16

RFC1918 private address ranges (Docker's overlay network, Tailscale) bypass CrowdSec checks. Inter-service communication inside the cluster doesn't need to be WAF-inspected — it never crosses the public internet boundary.

Tailscale: Secure Admin Access

SSH is not exposed in the Hetzner firewall. All administrative access is routed through Tailscale VPN.

During cloud-init (Part 2), the server joins your Tailscale network:

tailscale up \
  --ssh \
  --accept-routes \
  --advertise-exit-node \
  --advertise-tags=tag:server \
  --client-id=<TAILSCALE_CLIENT_ID> \
  --client-secret=<TAILSCALE_CLIENT_SECRET>

--ssh enables Tailscale SSH, allowing SSH access to the server using Tailscale credentials. The Tailscale hostname (my-server.your-tailnet.ts.net) is stable even if the server IP changes.

From any device enrolled in the Tailscale network:

ssh admin@my-server.your-tailnet.ts.net

This eliminates the need for public SSH key management, firewall IP exceptions, or a self-managed VPN gateway. Tailscale handles NAT traversal automatically, establishing a peer-to-peer encrypted connection regardless of network topology.

SSH Hardening Recap

Even though Tailscale VPN is the primary admin path, SSH is still hardened as a defense-in-depth measure:

From server/hetzner.tfpl:

PasswordAuthentication no    → SSH keys only, passwords rejected
MaxAuthTries 6               → Disconnect after 6 failed attempts
MaxSessions 3                → Limit concurrent sessions
X11Forwarding no             → Disable graphical forwarding
ClientAliveInterval 300      → Disconnect idle sessions after 5 min
LoginGraceTime 30            → Disconnect if auth not completed in 30s

And fail2ban:

bantime = 3600               → 1-hour bans
findtime = 600               → 10-minute window
maxretry = 3                 → 3 failures triggers ban
mode = aggressive            → Also catches scan patterns

Source IPs that fail authentication 3 times within a 10-minute window are banned for 1 hour. Combined with key-only authentication and SSH not being exposed to the public internet, the SSH attack surface is substantially reduced.

Summary: Security in Layers

Layer	What it protects against
Cloudflare DNS proxy	Hides server IP; DDoS mitigation at edge
Hetzner firewall	Blocks all non-HTTP/HTTPS traffic at network level
fail2ban	SSH brute-force banning
SSH key-only auth	Password-based SSH attacks
Tailscale VPN	Admin access without exposing SSH to internet
Traefik `exposedByDefault: false`	Accidental service exposure
CrowdSec LAPI	Known malicious IP blocking
CrowdSec AppSec	Application-layer attack filtering (SQLi, XSS, CVEs)
Docker secrets	Credentials as files, not environment variables
SOPS encryption	No plaintext secrets in Git

Each layer is independent — a failure or bypass of any one layer still leaves others intact. This is defense in depth.

Repository: gitlab.com/sakonn/docker-swarm-gitops

Part 3: Push to Git, Get a Deployment — SwarmCD + SOPS on Docker Swarm

Jakub Korečko — Sat, 30 May 2026 08:45:52 +0000

What you'll learn:

What SwarmCD is and how it compares to ArgoCD
How the SwarmCD configuration file works
Why Docker Swarm configs and secrets are immutable — and what that means for updates
How SOPS + age encryption keeps secrets in Git without exposing them
The complete GitOps loop from git push to running container

The Loop

git push
   ↓
SwarmCD detects change (≤45 seconds)
   ↓
Clone repo → parse config → decrypt secrets (SOPS + age)
   ↓
docker stack deploy
   ↓
Traefik picks up new service (if labels changed)

No CI/CD pipeline. No webhooks. No manual SSH. Deployment requires only a Git push.

What is SwarmCD?

SwarmCD is a lightweight GitOps controller for Docker Swarm. It does one thing: watch a Git repository and deploy Docker Compose stacks when files change.

Why not ArgoCD? ArgoCD is excellent for Kubernetes. It has no Swarm support. Its Kubernetes dependency alone would require more RAM than the entire rest of this stack combined.

Why not Portainer? Portainer has a GitOps feature, but it requires a paid Business license for the functionality we need, and it runs as a much heavier process.

SwarmCD has no web UI complexity, no operator pattern, no CRDs. It reads a config YAML, polls Git, and runs docker stack deploy. This simplicity is intentional at this scale.

The SwarmCD service is deployed by Ansible during bootstrap (see Part 2) as the only service that isn't self-managed via GitOps. Everything else is managed by SwarmCD from Git.

SwarmCD Configuration

The full configuration lives at apps/swarmcd/config.yaml:

update_interval: 45

repos_path: repos/

sops_secrets_discovery: true
auto_rotate: true

repos:
  my-infra:
    url: "https://gitlab.com/<YOUR_GITLAB_USERNAME>/<YOUR_REPO_NAME>.git"
    username: does_not_matter
    password_file: /secrets/gitlab_token

stacks:
  traefik:
    repo: my-infra
    branch: main
    compose_file: /apps/traefik/traefik.yaml
    sops_files:
      - /apps/traefik/crowdsec_api_key

  bento:
    repo: my-infra
    branch: main
    compose_file: /apps/bento/bento.yaml

  alloy:
    repo: my-infra
    branch: main
    compose_file: /apps/monitoring/alloy.yaml
    sops_files:
      - /apps/monitoring/grafana_cloud_passwd

  cadvisor:
    repo: my-infra
    branch: main
    compose_file: /apps/monitoring/cadvisor.yaml

  node_exporter:
    repo: my-infra
    branch: main
    compose_file: /apps/monitoring/node_exporter.yaml

address: 0.0.0.0:8080

Key settings explained:

update_interval: 45 — SwarmCD polls Git every 45 seconds. This is a tradeoff: shorter intervals mean faster deployments but more Git API requests. 45 seconds is sufficient for most infrastructure change workflows.

sops_secrets_discovery: true — SwarmCD automatically finds and decrypts SOPS-encrypted files listed in sops_files. The decrypted content is passed to Docker as secrets.

auto_rotate: true — This is the most significant setting. See the next section.

password_file: /secrets/gitlab_token — The GitLab personal access token is mounted as a Docker secret (created by Ansible during bootstrap). SwarmCD reads it from the filesystem, never from environment variables.

The Immutable Config Problem (and the Solution)

Docker Swarm has a strict rule: configs and secrets are immutable once created. You cannot update a config in place. If you change a Traefik config file, you must:

Create a new config object with a different name
Update the service to reference the new config
(Optionally) delete the old config

This is different from Kubernetes ConfigMaps, which can be updated in place. Docker's immutability is a feature — it means config history is preserved and rollback is possible — but it requires tooling support.

SwarmCD's auto_rotate: true handles this automatically:

You push a change to apps/traefik/traefik_static_conf.yaml
SwarmCD detects the change
It creates a new Docker config: traefik_static_conf_<hash> (where hash is derived from the content)
It updates the Traefik service to mount the new config instead of the old one
Docker Swarm rolls out the update (Traefik restarts with the new config)
The old config remains until you clean it up (SwarmCD can do this automatically)

Without auto_rotate, you'd need to manually rename config objects on every change. With it, the whole process is transparent.

SOPS + age: Secrets in Git

Storing secrets in Git is a security risk without encryption. SOPS (Secrets OPerationS) addresses this by encrypting secret files before they are committed.

How it works

age is an asymmetric encryption tool. You generate a keypair:

age-keygen -o ~/.age/key.txt

The public key goes into a .sops.yaml file in your repository:

creation_rules:
  - path_regex: .*
    age: age1youragepublickey...

To encrypt a secret:

echo "my-secret-value" | sops --encrypt --input-type=binary --output-type=binary /dev/stdin > apps/traefik/crowdsec_api_key

The encrypted file is committed to Git. The private age key (~/.age/key.txt) is never committed.

How SwarmCD decrypts

During the Ansible bootstrap (Part 2), the age private key is stored as a Docker secret named age_key. SwarmCD mounts this secret and sets the SOPS_AGE_KEY_FILE environment variable to point to it:

# apps/swarmcd/swarmcd.yaml (relevant excerpt)
services:
  swarmcd:
    environment:
      SOPS_AGE_KEY_FILE: /secrets/age_key
    secrets:
      - age_key

When SwarmCD processes a stack that has sops_files, it:

Finds each listed file in the cloned repo
Decrypts it using the age key
Creates a Docker secret with the decrypted content
References that secret in the service deployment

The decrypted value exists only in Docker's encrypted secret store and in the container's memory at runtime. It never touches the server's filesystem in plaintext.

Two-layer secret model

Layer	Where stored	Used by	Contents
SOPS/vault.yaml	Git (encrypted)	Terraform at provisioning time	Cloud API keys, admin password hash
Docker secrets	Docker Swarm secret store	Running containers	App credentials (GitLab token, Grafana password, CrowdSec API key)

Terraform-layer secrets are only present during tofu apply. They are never present in a running container. Docker-layer secrets are created once (by Ansible or by SwarmCD) and mounted into containers as files.

Adding a New App

To add a new service to the GitOps setup:

Write a Docker Compose file and add it to apps/yourapp/yourapp.yaml
Add a stack entry to apps/swarmcd/config.yaml:

   stacks:
     yourapp:
       repo: my-infra
       branch: main
       compose_file: /apps/yourapp/yourapp.yaml

Run the Ansible playbook to deploy the updated SwarmCD configuration to the server:

   cd server && ansible-playbook setup_docker.yaml

SwarmCD's config.yaml is mounted into the container as a Docker config object. Modifying it in Git is not sufficient — Ansible must redeploy SwarmCD with the updated config so the new stack entry takes effect.

Add Traefik labels to expose the service (if needed):

   # In yourapp.yaml
   services:
     yourapp:
       labels:
         - "traefik.enable=true"
         - "traefik.http.routers.yourapp.rule=Host(`yourapp.yourdomain.com`)"
         - "traefik.http.routers.yourapp.tls.certresolver=production"
         - "traefik.http.services.yourapp.loadbalancer.server.port=8080"

Add a Cloudflare DNS record in root main.tf (and run tofu apply)
Push to Git — SwarmCD deploys it within 45 seconds

This is the complete workflow. No manual Docker commands, no state to maintain outside of Git.

What SwarmCD Does NOT Do

SwarmCD has limitations worth addressing explicitly, since GitOps tooling often implies more sophisticated features:

No health-check-based rollbacks. If a new deployment causes container crashes, SwarmCD will not automatically roll back. Docker Swarm's restart policy (on-failure) will restart failed containers, and start-first update strategy (configured in Traefik) minimizes downtime, but there's no automatic rollback to a previous Git commit.

Mitigation: If a deployment breaks, git revert and push. SwarmCD will redeploy the reverted config within 45 seconds.

No diff preview. SwarmCD doesn't show you what will change before deploying.

Mitigation: Git PR review serves this function.

No multi-cluster support. SwarmCD manages one Swarm from one config file. This is fine for a single-server setup.

For a small production setup, none of these limitations prevent adoption. The simplicity SwarmCD offers in exchange for these features is an appropriate tradeoff at this scale.

The Complete Deployment Flow

Here's every step that happens when you push a change to apps/traefik/traefik_static_conf.yaml:

1. git push origin main
   └── GitLab receives the commit

2. SwarmCD polls (within 45 seconds)
   └── Detects diff in apps/traefik/traefik_static_conf.yaml

3. Clone / pull latest main branch

4. Parse apps/swarmcd/config.yaml
   └── traefik stack → compose_file: /apps/traefik/traefik.yaml

5. Decrypt SOPS secrets
   └── apps/traefik/crowdsec_api_key → decrypted value

6. Create versioned Docker objects
   ├── Config: traefik_static_conf_abc123 (new content hash)
   ├── Config: traefik_dynamic_conf_def456 (unchanged, reuse or recreate)
   └── Secret: crowdsec_api_key_ghi789

7. docker stack deploy traefik
   └── Service update: new config references

8. Docker Swarm rolls out update
   └── Traefik restarts with new static config

9. Traefik re-reads config
   ├── New entrypoint settings take effect
   └── New ACME resolver (if changed) initializes

10. Monitoring captures the event
    └── Alloy ships container restart log to Grafana Cloud Loki

Total time from git push to running: approximately 45-90 seconds.

Summary

The GitOps loop in this stack is simple by design:

SwarmCD polls Git every 45 seconds and runs docker stack deploy on changes
auto_rotate handles Docker Swarm's immutable config requirement transparently
SOPS + age lets secrets live in Git safely — encrypted at commit time, decrypted at deploy time by SwarmCD using a Docker-managed age key
No pipeline required — the deployment trigger is a Git push, not a CI job

The following part covers how public traffic is routed and secured.

Repository: https://gitlab.com/sakonn/docker-swarm-gitops

Part 2: Provision and Harden a Cloud Server in One Command with OpenTofu

Jakub Korečko — Sat, 30 May 2026 08:44:27 +0000

What you'll learn:

How the OpenTofu (Terraform-compatible) project is structured
How secrets are managed at the infrastructure layer with SOPS
What Hetzner resources get created and why
How cloud-init hardens the server before any code runs on it
How Ansible bootstraps Docker without hardcoded IPs

One Command, One Server, Fully Ready

tofu apply

After approximately five minutes, the following resources are available:

A hardened Ubuntu 24.04 server on Hetzner
SSH key-only authentication, fail2ban, kernel hardening
Docker installed with Swarm initialized
An overlay network for inter-service communication
Tailscale VPN joined (secure admin access)
Cloudflare DNS records pointing to the new server IP

Then, once:

cd server && ansible-playbook setup_docker.yaml

SwarmCD is deployed and monitoring the Git repository. From this point, all deployments are triggered by git push.

This article covers each file that makes this possible.

Project Structure

The OpenTofu project has two layers: a root module that orchestrates everything, and a server/ child module that handles Hetzner-specific resources.

gitops/
├── main.tf         # Wires modules, Cloudflare DNS, GitLab remote state
├── providers.tf    # SOPS, Cloudflare, Hcloud, Ansible provider versions
├── variables.tf    # Root variables (cloudflare_zone_id)
├── data.tf         # Loads vault.yaml via SOPS
├── vault.yaml      # Encrypted secrets (not vault.yaml.example)
└── server/
    ├── main.tf     # All Hetzner resources
    ├── providers.tf
    ├── variables.tf    # Sensitive inputs from root
    ├── outputs.tf      # Exported values (IP, hostname, etc.)
    ├── hetzner.tfpl    # cloud-init template
    ├── inventory.yaml  # Dynamic Ansible inventory
    ├── ansible.cfg
    └── setup_docker.yaml  # Ansible bootstrap playbook

The root module passes secrets down to the server module. The server module passes connection details back up via outputs, which are then consumed by the dynamic Ansible inventory.

Secrets at the Infrastructure Layer

Before anything can be provisioned, OpenTofu needs API keys: Hetzner, Cloudflare, Tailscale OAuth. These are stored in vault.yaml, encrypted with SOPS using an age key that never touches version control.

vault.yaml.example shows the structure:

cloudflare_api_key: <TOKEN>
hetzner_api_key: <TOKEN>
tailscale_client_secret: <SECRET>
server_admin_password_hash: <BCRYPT_HASH>
gitlab_password: <PERSONAL_ACCESS_TOKEN>

The actual vault.yaml is encrypted. OpenTofu decrypts it at plan/apply time using the SOPS provider:

# data.tf
data "sops_file" "vault" {
  source_file = "vault.yaml"
}

This is the first of two secret layers in this stack. Terraform-layer secrets are used only during provisioning — they create infrastructure and pass credentials into the server. They are never stored in Docker or exposed to running applications.

Why two secret layers? The Terraform layer (vault.yaml) holds API keys for cloud providers. The Docker layer holds runtime credentials for apps (GitLab token for SwarmCD, Grafana password for Alloy). Separating these means a compromised application cannot access cloud infrastructure credentials.

Hetzner Resources

server/main.tf creates every cloud resource the server needs.

Static IP Addresses

resource "hcloud_primary_ip" "primary_ipv4" {
  type          = "ipv4"
  name          = "primary_ipv4"
  location      = local.hcloud_server_location
  auto_delete   = false
  assignee_type = "server"
}

auto_delete = false is critical. Without it, destroying the server also destroys the IP address. When rebuilding a server, losing the IP address requires updating DNS records, waiting for propagation, and may invalidate pending Let's Encrypt certificate challenges. With auto_delete = false, the IP persists independently of the server lifecycle.

Firewall

resource "hcloud_firewall" "primary_firewall" {
  name = "primary_firewall"

  rule { direction = "in"; protocol = "icmp"; source_ips = ["0.0.0.0/0", "::/0"] }
  rule { direction = "in"; protocol = "tcp"; port = "80";  source_ips = ["0.0.0.0/0", "::/0"] }
  rule { direction = "in"; protocol = "tcp"; port = "443"; source_ips = ["0.0.0.0/0", "::/0"] }
  rule { direction = "in"; protocol = "udp"; port = "80";  source_ips = ["0.0.0.0/0", "::/0"] }
  rule { direction = "in"; protocol = "udp"; port = "443"; source_ips = ["0.0.0.0/0", "::/0"] }
}

Only three port ranges are open to the public: ICMP, 80, and 443. Everything else is blocked at the Hetzner firewall level — not just in software, but in the network infrastructure before packets reach the server.

UDP 80/443 is included for HTTP/3 (QUIC), which Traefik can use.

SSH is not open in the firewall at all. Tailscale VPN handles all admin access — the SSH port is only reachable over the Tailscale network, so it is invisible to public internet scanners.

Persistent Volume

resource "hcloud_volume" "primary_volume" {
  name      = "primary_volume"
  size      = 10 # GB
  server_id = hcloud_server.primary_server.id
  automount = true
  format    = "ext4"
}

The 10GB Hetzner volume is mounted separately from the server's root disk. It stores stateful data (database files, application data) that must survive server rebuilds. Hetzner volumes are not deleted when a server is deleted — they persist independently of the server lifecycle and can be attached to a replacement server.

Ansible later creates Docker bind-mount volumes pointing into this volume's mount path.

Server

resource "hcloud_server" "primary_server" {
  name        = "my-server"
  image       = "ubuntu-24.04"
  server_type = "cx23"   # 2 vCPU, 4GB RAM
  location    = local.hcloud_server_location

  user_data = templatefile("${path.module}/hetzner.tfpl", {
    admin_username          = local.admin_username
    admin_password_hash     = var.admin_password_hash
    admin_ssh_keys          = values(local.admin_ssh_keys)
    tailscale_client_id     = local.tailscale_client_id
    tailscale_client_secret = var.tailscale_client_secret
    ssh_port                = local.ssh_port
  })

  ...

  lifecycle {
    ignore_changes = [user_data]
  }
}

The user_data field is where cloud-init lives. The templatefile function renders hetzner.tfpl with values from Terraform variables — including the bcrypt-hashed admin password and Tailscale OAuth credentials. These are injected into the cloud-init template without ever being written to disk in plaintext.

The lifecycle block with ignore_changes = [user_data] is essential for production. Without it, every change to the cloud-init template would cause Terraform to destroy and recreate the server — a full machine wipe. cloud-init runs only on first boot; subsequent changes to the template have no effect on a running server anyway. Ignoring user_data drift allows changes to the cloud-init template without triggering a server rebuild.

cloud-init: OS Hardening on First Boot

cloud-init runs once on first boot, before any remote connection is established. This is where OS-level hardening happens.

The full template is at server/hetzner.tfpl. Here are the key sections:

Package Installation

packages:
  - fail2ban
  - auditd
  - unattended-upgrades
  - git
  - curl
  - ca-certificates
  - build-essential

fail2ban and auditd are installed from packages (not Docker) because they need to monitor the host OS, not a container. unattended-upgrades enables automatic security patch installation.

User Creation

users:
  - name: ${admin_username}
    passwd: ${admin_password_hash}
    lock_passwd: false
    groups: sudo, docker
    shell: /bin/bash
    sudo: ALL=(ALL) NOPASSWD:ALL
    ssh_authorized_keys: ${jsonencode(admin_ssh_keys)}

The admin user is created with:

A bcrypt-hashed password (generated locally, never sent in plaintext)
Both your personal SSH key and an Ansible-specific SSH key
Membership in the docker group (can run Docker without sudo)
NOPASSWD sudo (automation-friendly)

The Ansible key is a separate ed25519 key generated locally (ssh-keygen -t ed25519 -f .ansible_key). It is used only for the bootstrap playbook and can be rotated or removed after provisioning.

SSH Hardening

- path: /etc/ssh/sshd_config.d/99-hardening.conf
  content: |
    PermitRootLogin prohibit-password
    PasswordAuthentication no
    MaxAuthTries 6
    MaxSessions 3
    X11Forwarding no
    AllowAgentForwarding no
    ClientAliveInterval 300
    ClientAliveCountMax 2
    LoginGraceTime 30

PasswordAuthentication no restricts authentication to SSH keys only. Combined with fail2ban banning source IPs after 3 failed attempts within a 1-hour window, brute-force SSH attacks are mitigated at both the authentication and network level.

Kernel Hardening

- path: /etc/sysctl.d/99-hardening.conf
  content: |
    # Prevent IP spoofing
    net.ipv4.conf.all.rp_filter = 1
    net.ipv4.conf.default.rp_filter = 1
    # Ignore ICMP redirects (prevent routing attacks)
    net.ipv4.conf.all.accept_redirects = 0
    net.ipv6.conf.all.accept_redirects = 0
    # SYN flood protection
    net.ipv4.tcp_syncookies = 1
    # IP forwarding (required for Docker networking and Tailscale)
    net.ipv4.ip_forward = 1
    net.ipv6.conf.all.forwarding = 1

Reverse path filtering blocks packets with source IPs that couldn't have arrived on the interface they came in on — a basic IP spoofing defense. SYN cookies protect against SYN flood DoS attacks. IP forwarding is required for both Docker's overlay networking and Tailscale's routing.

fail2ban Configuration

- path: /etc/fail2ban/jail.local
  content: |
    [DEFAULT]
    bantime = 3600
    findtime = 600
    maxretry = 3

    [sshd]
    enabled = true
    mode = aggressive

Three failed attempts within 10 minutes result in a 1-hour IP ban. Aggressive mode also covers additional SSH attack patterns beyond basic password failures.

Docker and Swarm Initialization

runcmd:
  - sysctl --system
  - systemctl enable --now fail2ban
  - systemctl enable --now auditd
  - systemctl restart ssh
  - curl -fsSL https://get.docker.com | sh
  - docker swarm init
  - docker network create -d overlay --attachable swarm_network
  - curl -fsSL https://tailscale.com/install.sh | sh
  - tailscale up --ssh --accept-routes --advertise-exit-node
      --advertise-tags=tag:server
      --client-id=${tailscale_client_id}
      --client-secret=${tailscale_client_secret}
  - reboot

Docker is installed via the official convenience script. Swarm is initialized immediately. The swarm_network overlay network is created — all services in apps/ connect to this network, enabling inter-service communication across nodes as the cluster scales.

Tailscale authenticates via OAuth, requiring no interactive input. The --ssh flag enables Tailscale SSH, which Ansible uses for the bootstrap playbook.

The final reboot applies all sysctl changes and ensures all services start from a consistent state.

Cloudflare DNS via Terraform

Back in the root main.tf, Cloudflare DNS records are created after the server is provisioned:

resource "cloudflare_dns_record" "root" {
  zone_id = var.cloudflare_zone_id
  name    = "@"
  content = module.server.server_ipv4
  type    = "A"
  proxied = true
}

resource "cloudflare_dns_record" "www" {
  zone_id = var.cloudflare_zone_id
  name    = "www"
  content = module.server.server_ipv4
  type    = "A"
  proxied = true
}

proxied = true routes traffic through Cloudflare's CDN, concealing the origin server IP address. Traffic targeting the domain by IP address is handled by Cloudflare's infrastructure, keeping the origin server address private.

Dynamic Ansible Inventory

Instead of hardcoding the server IP in an inventory file, Ansible reads it directly from Terraform state:

# server/inventory.yaml
plugin: cloud.terraform.terraform_provider
binary_path: tofu
project_path: ../

This uses the cloud.terraform.terraform_provider Ansible plugin, which runs tofu output internally and maps the results to Ansible host variables. The server's Tailscale hostname, IP, SSH port, and SSH key path all come from Terraform outputs — no manual synchronization needed.

When a server is rebuilt, tofu apply updates the state and the next Ansible run reads the updated connection details from Terraform outputs.

Ansible Bootstrap Playbook

server/setup_docker.yaml runs once after tofu apply completes. It cannot be part of cloud-init because it requires Docker and Swarm to already be running.

Persistent Storage Directories

- name: Create directories on Hetzner volume
  ansible.builtin.file:
    path: "/mnt/{{ hostvars[inventory_hostname].volume_id }}/{{ item }}"
    state: directory
  loop:
    - crowdsec_db
    - papra_data

Directories are created on the Hetzner volume (mounted at /mnt/HC_Volume_*). The volume path uses the volume_id from Terraform outputs, so there's no hardcoded mount path.

Docker Secrets

- name: Create gitlab_password Docker secret
  community.docker.docker_secret:
    name: gitlab_password
    data: "{{ lookup('community.sops.sops', '../vault.yaml') | from_yaml
              | json_query('gitlab_password') }}"
    state: present

SOPS decrypts vault.yaml locally on your machine, and Ansible pushes the decrypted value as a Docker secret. The plaintext never touches the server's filesystem — it goes directly from your machine into Docker's encrypted secret store.

Two secrets are created:

gitlab_password: SwarmCD uses this to authenticate with GitLab when cloning the repo
age_key: SwarmCD uses this to decrypt SOPS-encrypted secret files in the repo

SwarmCD Deployment

- name: Deploy SwarmCD
  community.docker.docker_stack:
    name: swarmcd
    compose:
      - "{{ lookup('file', '../apps/swarmcd/swarmcd.yaml') | from_yaml }}"
    state: present

SwarmCD is deployed from the Ansible playbook — it cannot manage its own initial deployment, so this bootstrap step is handled outside the GitOps loop. After this, SwarmCD takes over management of all other stacks from Git.

Remote State

The Terraform state is stored in GitLab's managed HTTP backend:

terraform {
  backend "http" {
    address        = "https://gitlab.com/api/v4/projects/.../terraform/state/default"
    lock_address   = "..."
    unlock_address = "..."
  }
}

This means state is shared between team members and persists across machines. GitLab provides free managed Terraform state for any project.

Summary

After tofu apply and ansible-playbook setup_docker.yaml, you have:

A Hetzner server with hardened OS (SSH keys only, fail2ban, kernel sysctl)
Docker Swarm initialized with an overlay network
Tailscale VPN joined (admin access without public SSH)
Cloudflare DNS records created
SwarmCD running and watching your Git repository
A 10GB persistent volume with Docker bind-mount volumes for stateful data

From this point forward, infrastructure changes are managed through Git. New application deployments and configuration updates are applied via git push.

Repository: gitlab.com/sakonn/docker-swarm-gitops

I Run a Full GitOps Stack for $6/month — Here's the Architecture

Jakub Korečko — Sat, 30 May 2026 08:44:03 +0000

What you'll learn:

What this stack does and what problem it solves
Why Docker Swarm instead of Kubernetes
Why Hetzner and why GitOps
The full tech stack with tool-by-tool rationale
How the repository is structured

The Problem

Every side project or small production deployment starts the same way: you SSH into a server, run some commands, tweak a config, and hope you remember what you did. Six months later something breaks and you can't reproduce the setup. You're firefighting with no audit trail and no way to roll back.

The solution is GitOps — your infrastructure and application configuration live in a Git repository, and changes to that repository automatically trigger deployments. Git becomes your source of truth, your audit log, and your rollback mechanism.

The problem is that most GitOps tutorials assume you're running Kubernetes. If you're running a small project on a single server, Kubernetes is enormous overkill.

This series shows a different approach: a full GitOps pipeline on Docker Swarm, running on a single Hetzner server for roughly €5/month. No cluster management. No operator madness. Just Git, Docker, and a handful of focused tools.

What the Stack Does

From a single git push, the stack:

Detects the change within 45 seconds
Decrypts any secrets that were updated
Deploys the new service configuration to Docker Swarm
Routes traffic through Traefik (TLS terminated, WAF filtered)
Ships metrics and logs to Grafana Cloud

From tofu apply (a one-time operation), the stack:

Provisions a hardened Ubuntu 24.04 server on Hetzner
Configures firewall rules, SSH hardening, fail2ban
Installs Docker, initializes a Swarm
Joins Tailscale VPN (for secure admin access without exposing SSH publicly)
Creates Cloudflare DNS records
Bootstraps the GitOps controller (SwarmCD)

After that initial apply, the server is fully hands-off. All future changes go through Git.

Architecture Overview

┌─────────────────────────────────────────────────────┐
│                 Your Local Machine                  │
│                                                     │
│  git push  ──────────────────────────► GitLab Repo  │
│                                             │       │
│  tofu apply ──► Hetzner + Cloudflare        │       │
└─────────────────────────────────────────────────────┘
                                             │
                                             ▼
┌─────────────────────────────────────────────────────┐
│               Hetzner Cloud Server                  │
│                                                     │
│  ┌─────────────┐    polls every 45s                 │
│  │  SwarmCD    │ ◄──────────────── GitLab Repo      │
│  └──────┬──────┘                                    │
│         │ docker stack deploy                       │
│         ▼                                           │
│  ┌─────────────────────────────────────────────┐    │
│  │           Docker Swarm                      │    │
│  │                                             │    │
│  │  Traefik ──► CrowdSec ──► Apps              │    │
│  │  (reverse proxy + WAF)    (bento, etc.)     │    │
│  │                                             │    │
│  │  Monitoring (Alloy, cAdvisor, node-exporter)│    │
│  └─────────────────────────────────────────────┘    │
│                                                     │
│  Tailscale VPN ──── Admin Access (no public SSH)    │
│                                                     │
│  Hetzner Volume (10GB persistent storage)           │
└─────────────────────────────────────────────────────┘
         │ metrics + logs
         ▼
  Grafana Cloud (free tier)

Why Docker Swarm, Not Kubernetes

"Why not just use Kubernetes?"

This is the first question you may ask, so let's address it directly.

Kubernetes is excellent for teams managing dozens of services across multiple nodes. For a single server with five services, it introduces:

A control plane that consumes ~1-2GB RAM before your apps even start
A separate installation process (kubeadm, k3s, minikube, etc.)
A completely different mental model for networking, storage, and secrets
Custom Resource Definitions, operators, Helm charts, and dozens of YAML abstractions

Docker Swarm, by contrast:

Is built into Docker — one command to initialize: docker swarm init
Uses the same Docker Compose format you already know
Has native support for secrets and configs
Consumes essentially no overhead on a single-node setup
Handles rolling updates and zero-downtime deployments without configuration

The tradeoff: Docker Swarm has fewer features, smaller ecosystem, and doesn't scale horizontally as easily as Kubernetes. For a single server hosting a handful of services, none of that matters.

Key insight: Docker Swarm is not "Kubernetes lite" — it's a different tool for a different scale. Choosing the right tool for the scale you actually have is good engineering.

Why Hetzner

Hetzner is a German cloud provider with data centers in Nuremberg, Falkenstein, and Helsinki. The cx23 instance used in this stack costs approximately €4-5/month:

Spec	Value
CPU	2 vCPU (shared)
RAM	4 GB
Storage	40 GB SSD (+ 10 GB volume)
Network	20 TB/month included
Location	Nuremberg, Germany

For comparison, a comparable AWS EC2 instance (t3.medium) costs roughly $30/month. The performance difference for a hobby or small production workload is negligible.

Hetzner is also EU-based and GDPR-compliant — relevant if you're handling user data from European users.

Why GitOps

GitOps solves three problems that every self-hosted deployment eventually hits:

1. No reproducibility. "I set this up six months ago and I don't remember what I did" is the death of many side projects. When infrastructure and app config live in Git, any team member (or future you) can re-provision from scratch.

2. No audit trail. Who changed what and when? Git history answers this. Every deployment is a commit with a timestamp and author.

3. Config drift. The server diverges from what you think is deployed. GitOps eliminates drift by making Git the authoritative source. The GitOps controller continuously reconciles what's running with what's in Git.

The GitOps controller in this stack is SwarmCD. It polls the Git repository every 45 seconds and deploys any changed stacks. You push to Git; SwarmCD handles the rest.

The Full Tech Stack

Tool	Role	Why This One
OpenTofu	Infrastructure as Code	Open-source Terraform fork; provisions Hetzner, Cloudflare
Hetzner Cloud	Server hosting	Cheap, reliable, EU-based, good API
Cloudflare	DNS + CDN	Free tier, hides server IP, DDoS protection
cloud-init	OS bootstrapping	Runs on first boot; installs Docker, hardens SSH, joins Tailscale
Ansible	Bootstrap provisioning	One-time Docker setup (volumes, secrets, SwarmCD deploy)
Docker Swarm	Container orchestration	Built into Docker, no extra install, Compose-compatible
SwarmCD	GitOps controller	Lightweight, native Swarm support, SOPS integration
SOPS + age	Secret encryption	Encrypts secrets in Git; decrypted at deploy time
Traefik	Reverse proxy + TLS	Label-based routing, Let's Encrypt built in, Swarm-aware
CrowdSec	WAF + IDS	Community IP reputation list + AppSec rules via Traefik plugin
Tailscale	VPN	Secure admin access; no public SSH exposure
Grafana Alloy	Metrics + logs agent	Single container replaces Prometheus Agent + Promtail
Grafana Cloud	Observability backend	Free tier; managed Prometheus + Loki
fail2ban	SSH brute-force protection	Bans IPs after 3 failed SSH attempts

Repository Structure

gitops/
├── vault.yaml.example      # Template for SOPS-encrypted secrets
├── main.tf                 # Root: wires modules, Cloudflare DNS, remote state
├── providers.tf            # Provider versions (SOPS, Cloudflare, Hcloud, Ansible)
├── variables.tf            # Root variables
├── data.tf                 # Loads vault.yaml via SOPS provider
│
├── server/                 # Hetzner provisioning module
│   ├── main.tf             # Server, firewall, IPs, volume resources
│   ├── providers.tf        # Local provider declarations
│   ├── variables.tf        # Sensitive inputs (API keys, password hash)
│   ├── outputs.tf          # Server IP, SSH port, Tailscale hostname
│   ├── hetzner.tfpl        # cloud-init template (OS hardening + Docker)
│   ├── inventory.yaml      # Dynamic Ansible inventory (reads Terraform state)
│   ├── ansible.cfg         # Ansible config (points to inventory.yaml)
│   └── setup_docker.yaml   # Ansible playbook (bootstrap Docker + SwarmCD)
│
└── apps/                   # Docker Compose stacks (deployed by SwarmCD)
    ├── swarmcd/
    │   ├── config.yaml     # SwarmCD: which stacks to watch, poll interval
    │   └── swarmcd.yaml    # SwarmCD compose file (bootstrapped by Ansible)
    ├── traefik/
    │   ├── traefik.yaml            # Traefik compose file
    │   ├── traefik_static_conf.yaml  # Entrypoints, ACME, providers, plugins
    │   └── traefik_dynamic_conf.yaml # Middlewares (auth, CrowdSec)
    ├── monitoring/
    │   ├── alloy.yaml              # Grafana Alloy compose file
    │   ├── cadvisor.yaml           # cAdvisor compose file
    │   ├── node_exporter.yaml      # node-exporter compose file
    │   └── grafana_alloy.alloy     # Alloy pipeline config
    └── bento/
        └── bento.yaml              # Example app with Traefik labels

The split between server/ and apps/ reflects a key separation of concerns:

server/ is infrastructure — runs once, creates the environment
apps/ is application configuration — changes continuously via GitOps

What's in Each Part

Part 2: Infrastructure as Code — OpenTofu, Hetzner provisioning, cloud-init hardening, Ansible bootstrap
Part 3: GitOps loop — SwarmCD, SOPS secret encryption, Docker Swarm immutable configs
Part 4: Security — Traefik, CrowdSec WAF, Tailscale VPN
Part 5: Observability — Grafana Alloy, Prometheus, Loki, Grafana Cloud free tier

All source code is in the repository linked at the end of this article. Each article walks through specific files with annotated code snippets.

Repository: gitlab.com/sakonn/docker-swarm-gitops