DEV Community: Ramansah

How to Set Up Prometheus and Grafana Monitoring Stack with Docker Compose

Ramansah — Mon, 13 Jul 2026 09:57:00 +0000

What You're Building
How Prometheus and Grafana Work Together
Prerequisites
Project Structure
Step 1: Prometheus Configuration
Step 2: Alert Rules
Step 3: Alertmanager Configuration
Step 4: Grafana Auto-Provisioning
Step 5: The Full Docker Compose Stack
Step 6: Start the Stack and Verify
Step 7: Importing Your First Dashboard
Step 8: Writing Your First PromQL Query
Monitoring Docker Containers with cAdvisor
Production Hardening Checklist
Common Issues and Quick Fixes
Next Steps

Most infrastructure problems don't announce themselves — they build up quietly until something breaks. A server runs out of disk space at 3 AM. A container silently restarts 50 times in an hour. Memory creeps toward its limit over three days before everything grinds to a halt. The difference between catching these early and waking up to an outage is a monitoring stack that's actually watching.

Prometheus collects and stores metrics from your infrastructure. Grafana turns those metrics into dashboards and alerts you can actually act on. Together with Node Exporter (host metrics) and Alertmanager (notifications), they form the standard open-source observability stack used by teams ranging from small self-hosted setups to large-scale production environments.

This guide builds the full stack with Docker Compose — Prometheus, Grafana, Node Exporter, cAdvisor, and Alertmanager — with auto-provisioning, persistent storage, and a production hardening checklist.

What You're Building

By the end of this guide, you'll have:

Prometheus scraping metrics every 15 seconds from all services
Node Exporter exposing host-level metrics (CPU, memory, disk, network)
cAdvisor exposing per-container Docker metrics
Grafana with Prometheus auto-configured as a data source and pre-loaded dashboards
Alertmanager sending notifications when something goes wrong
Everything persisted in named volumes and isolated on a dedicated Docker network

How Prometheus and Grafana Work Together

Understanding the data flow makes troubleshooting much easier:

Host OS / Docker containers
        │
        │ expose metrics endpoints
        ▼
Node Exporter (:9100) + cAdvisor (:8080)
        │
        │ Prometheus pulls (scrapes) every 15s
        ▼
Prometheus (:9090) ──── stores TSDB on disk
        │
        │ Grafana queries via PromQL
        ▼
Grafana (:3000) ──── dashboards + alerts
        │
        │ fires alert rules → Alertmanager
        ▼
Alertmanager (:9093) ──── sends to Slack/email/PagerDuty

The critical concept: Prometheus is pull-based, not push-based. Prometheus reaches out to each target and scrapes its /metrics endpoint on a schedule. Services don't push data to Prometheus — they expose an HTTP endpoint and wait for Prometheus to come collect. This pull model makes it easy to add or remove monitoring targets without touching the monitored service.

Prerequisites

Docker and Docker Compose installed (see our Install Docker on Ubuntu 26.04 LTS guide)
At least 2GB of free RAM (Prometheus TSDB is memory-hungry under load)
A server or VM you want to monitor

Project Structure

monitoring-stack/
├── compose.yml
├── .env
├── prometheus/
│   ├── prometheus.yml
│   └── alert.rules.yml
├── alertmanager/
│   └── alertmanager.yml
└── grafana/
    └── provisioning/
        ├── datasources/
        │   └── datasource.yml
        └── dashboards/
            └── dashboards.yml

Create the directory structure:

mkdir -p ~/monitoring-stack/{prometheus,alertmanager,grafana/provisioning/{datasources,dashboards}}
cd ~/monitoring-stack

Step 1: Prometheus Configuration

# prometheus/prometheus.yml
global:
  scrape_interval: 15s
  evaluation_interval: 15s
  external_labels:
    monitor: 'bckinfo-monitoring'

alerting:
  alertmanagers:
    - static_configs:
        - targets:
            - alertmanager:9093

rule_files:
  - /etc/prometheus/alert.rules.yml

scrape_configs:
  # Prometheus self-monitoring
  - job_name: 'prometheus'
    static_configs:
      - targets: ['prometheus:9090']

  # Host metrics via Node Exporter
  - job_name: 'node-exporter'
    static_configs:
      - targets: ['node-exporter:9100']

  # Docker container metrics via cAdvisor
  - job_name: 'cadvisor'
    static_configs:
      - targets: ['cadvisor:8080']

Each job_name in scrape_configs corresponds to a service in the Compose stack. The hostnames (like node-exporter, cadvisor) are Docker service names resolved over the shared monitoring network — the same hostname-as-service-name pattern used in our Redis with Docker Compose and MongoDB with Docker Compose guides.

Step 2: Alert Rules

# prometheus/alert.rules.yml
groups:
  - name: host-alerts
    interval: 30s
    rules:
      # Alert when any scrape target is down
      - alert: TargetDown
        expr: up == 0
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "Target {{ $labels.job }} is down"
          description: "{{ $labels.instance }} has been unreachable for more than 2 minutes."

      # Alert when CPU usage > 85% for 5 minutes
      - alert: HighCpuUsage
        expr: 100 - (avg by(instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 85
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "High CPU usage on {{ $labels.instance }}"
          description: "CPU usage is above 85% for more than 5 minutes. Current value: {{ $value | printf \"%.1f\" }}%"

      # Alert when memory usage > 90%
      - alert: HighMemoryUsage
        expr: (1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)) * 100 > 90
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "High memory usage on {{ $labels.instance }}"
          description: "Memory usage is above 90%. Current value: {{ $value | printf \"%.1f\" }}%"

      # Alert when disk usage > 85%
      - alert: DiskSpaceRunningLow
        expr: (1 - (node_filesystem_avail_bytes{fstype!~"tmpfs|fuse.lxcfs"} / node_filesystem_size_bytes)) * 100 > 85
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Disk space low on {{ $labels.instance }}"
          description: "Filesystem {{ $labels.mountpoint }} is {{ $value | printf \"%.1f\" }}% full."

      # Alert when disk will be full in less than 24 hours
      - alert: DiskWillFillIn24Hours
        expr: predict_linear(node_filesystem_avail_bytes{fstype!~"tmpfs"}[6h], 24 * 3600) < 0
        for: 30m
        labels:
          severity: critical
        annotations:
          summary: "Disk will fill in 24 hours on {{ $labels.instance }}"
          description: "Filesystem {{ $labels.mountpoint }} is predicted to fill within 24 hours."

predict_linear in the last rule is one of Prometheus's most useful functions — it projects the current trajectory of a metric forward in time, letting you alert on trends before they become crises.

Step 3: Alertmanager Configuration

# alertmanager/alertmanager.yml
global:
  resolve_timeout: 5m
  slack_api_url: '${SLACK_WEBHOOK_URL}'

route:
  group_by: ['alertname', 'instance']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  receiver: 'slack-notifications'
  routes:
    - match:
        severity: critical
      receiver: 'slack-notifications'
      repeat_interval: 1h

receivers:
  - name: 'slack-notifications'
    slack_configs:
      - channel: '#alerts'
        title: '{{ if eq .Status "firing" }}🔥 FIRING{{ else }}✅ RESOLVED{{ end }}: {{ .GroupLabels.alertname }}'
        text: >-
          {{ range .Alerts }}
          *Alert:* {{ .Annotations.summary }}
          *Description:* {{ .Annotations.description }}
          *Severity:* {{ .Labels.severity }}
          {{ end }}
        send_resolved: true

inhibit_rules:
  - source_match:
      severity: 'critical'
    target_match:
      severity: 'warning'
    equal: ['alertname', 'instance']

The inhibit_rules section prevents alert storms — if a critical alert fires for an instance, Prometheus suppresses the warning-level alerts for that same instance. This keeps your Slack channel from flooding with redundant notifications when a server goes down.

If you don't use Slack, Alertmanager also supports email, PagerDuty, OpsGenie, and webhooks — swap the receivers section accordingly.

Step 4: Grafana Auto-Provisioning

Instead of manually adding Prometheus as a data source in the Grafana UI every time you deploy, use provisioning files to configure it automatically at startup.

# grafana/provisioning/datasources/datasource.yml
apiVersion: 1

datasources:
  - name: Prometheus
    type: prometheus
    access: proxy
    url: http://prometheus:9090
    isDefault: true
    editable: false

# grafana/provisioning/dashboards/dashboards.yml
apiVersion: 1

providers:
  - name: 'default'
    orgId: 1
    folder: ''
    type: file
    disableDeletion: false
    updateIntervalSeconds: 10
    allowUiUpdates: true
    options:
      path: /var/lib/grafana/dashboards

This is the same one-shot initialization approach used for MongoDB replica sets and Apache Superset — configuration that runs once at startup, making the deployment fully repeatable.

Step 5: The Full Docker Compose Stack

.env file first:

GRAFANA_ADMIN_USER=admin
GRAFANA_ADMIN_PASSWORD=change_this_strong_password
SLACK_WEBHOOK_URL=https://hooks.slack.com/services/YOUR/WEBHOOK/URL

# compose.yml
version: '3.8'

networks:
  monitoring:
    driver: bridge

volumes:
  prometheus-data:
  grafana-data:

services:
  prometheus:
    image: prom/prometheus:latest
    container_name: prometheus
    restart: unless-stopped
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
      - '--storage.tsdb.path=/prometheus'
      - '--storage.tsdb.retention.time=30d'
      - '--web.enable-lifecycle'
      - '--web.enable-admin-api'
    volumes:
      - ./prometheus/prometheus.yml:/etc/prometheus/prometheus.yml:ro
      - ./prometheus/alert.rules.yml:/etc/prometheus/alert.rules.yml:ro
      - prometheus-data:/prometheus
    ports:
      - "127.0.0.1:9090:9090"
    networks:
      - monitoring
    healthcheck:
      test: ["CMD", "wget", "-qO-", "http://localhost:9090/-/ready"]
      interval: 15s
      timeout: 5s
      retries: 5

  grafana:
    image: grafana/grafana:latest
    container_name: grafana
    restart: unless-stopped
    environment:
      - GF_SECURITY_ADMIN_USER=${GRAFANA_ADMIN_USER}
      - GF_SECURITY_ADMIN_PASSWORD=${GRAFANA_ADMIN_PASSWORD}
      - GF_USERS_ALLOW_SIGN_UP=false
      - GF_SERVER_DOMAIN=localhost
      - GF_SMTP_ENABLED=false
    volumes:
      - grafana-data:/var/lib/grafana
      - ./grafana/provisioning:/etc/grafana/provisioning:ro
    ports:
      - "3000:3000"
    networks:
      - monitoring
    depends_on:
      prometheus:
        condition: service_healthy

  node-exporter:
    image: prom/node-exporter:latest
    container_name: node-exporter
    restart: unless-stopped
    command:
      - '--path.procfs=/host/proc'
      - '--path.sysfs=/host/sys'
      - '--path.rootfs=/rootfs'
      - '--collector.filesystem.mount-points-exclude=^/(sys|proc|dev|host|etc)($$|/)'
    volumes:
      - /proc:/host/proc:ro
      - /sys:/host/sys:ro
      - /:/rootfs:ro
    ports:
      - "127.0.0.1:9100:9100"
    networks:
      - monitoring

  cadvisor:
    image: gcr.io/cadvisor/cadvisor:latest
    container_name: cadvisor
    restart: unless-stopped
    privileged: true
    volumes:
      - /:/rootfs:ro
      - /var/run:/var/run:ro
      - /sys:/sys:ro
      - /var/lib/docker/:/var/lib/docker:ro
      - /dev/disk/:/dev/disk:ro
    ports:
      - "127.0.0.1:8080:8080"
    networks:
      - monitoring

  alertmanager:
    image: prom/alertmanager:latest
    container_name: alertmanager
    restart: unless-stopped
    command:
      - '--config.file=/etc/alertmanager/alertmanager.yml'
      - '--storage.path=/alertmanager'
    environment:
      - SLACK_WEBHOOK_URL=${SLACK_WEBHOOK_URL}
    volumes:
      - ./alertmanager/alertmanager.yml:/etc/alertmanager/alertmanager.yml:ro
    ports:
      - "127.0.0.1:9093:9093"
    networks:
      - monitoring

Two security decisions in this Compose file worth calling out explicitly:

Port bindings use 127.0.0.1: for Prometheus, Node Exporter, cAdvisor, and Alertmanager. This means those services are only reachable from the server itself — not exposed to the network. Only Grafana (port 3000) is accessible externally, since that's the only interface end users need. This follows the same network isolation principle covered in our Docker Container Security Best Practices guide.
GF_USERS_ALLOW_SIGN_UP=false — prevents anyone from self-registering an account on your Grafana instance.

Step 6: Start the Stack and Verify

cd ~/monitoring-stack
docker compose up -d

Watch services come up:

docker compose ps
docker compose logs -f prometheus

Verify each service is reachable:

# Prometheus ready check
curl -s http://localhost:9090/-/ready
# Expected: "Prometheus Server is Ready."

# Node Exporter metrics check
curl -s http://localhost:9100/metrics | head -20

# Grafana health check
curl -s http://localhost:3000/api/health
# Expected: {"commit":"...","database":"ok","version":"..."}

Open http://<your-server-ip>:3000 in a browser and log in with the credentials from your .env file.

Step 7: Importing Your First Dashboard

Grafana has thousands of community-built dashboards at grafana.com/grafana/dashboards. The most useful ones for a server monitoring stack:

Node Exporter Full — Dashboard ID: 1860
The gold standard for host metrics — CPU, memory, disk I/O, network, load average, all in one view.

To import: Dashboards → Import → Enter ID 1860 → Load → select your Prometheus data source → Import.

Docker and Container Metrics — Dashboard ID: 193
Container CPU, memory, and network stats from cAdvisor, with per-container breakdown.

Prometheus 2.0 Stats — Dashboard ID: 3662
Self-monitoring for Prometheus itself — scrape durations, TSDB metrics, sample ingestion rate.

Step 8: Writing Your First PromQL Query

Prometheus Query Language (PromQL) is what Grafana uses to pull data from Prometheus. A few practical queries to get started:

# Current CPU usage percentage (all cores, 5-minute average)
100 - (avg by(instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)

# Available memory in GB
node_memory_MemAvailable_bytes / 1024 / 1024 / 1024

# Disk usage percentage per filesystem
(1 - (node_filesystem_avail_bytes / node_filesystem_size_bytes)) * 100

# Number of running Docker containers
count(container_last_seen{image!=""})

# Container CPU usage rate (specific container by name)
rate(container_cpu_usage_seconds_total{name="redis"}[5m]) * 100

# HTTP request rate per second
rate(prometheus_http_requests_total[5m])

In Grafana, go to Explore (compass icon in the left sidebar), select your Prometheus data source, paste any query above into the metrics input, and click Run Query to see results immediately.

Monitoring Docker Containers with cAdvisor

cAdvisor (Container Advisor) automatically discovers all running containers on the Docker host and exposes their resource usage as Prometheus metrics — no per-container configuration required.

The metrics cAdvisor provides include:

Metric	What it measures
`container_cpu_usage_seconds_total`	Cumulative CPU time consumed
`container_memory_usage_bytes`	Current memory usage
`container_memory_limit_bytes`	Memory limit set on container
`container_network_transmit_bytes_total`	Network bytes sent
`container_network_receive_bytes_total`	Network bytes received
`container_fs_reads_bytes_total`	Filesystem read bytes

For containers running Redis or MongoDB (see our Redis and MongoDB guides), cAdvisor gives you memory and CPU visibility without any instrumentation changes to those containers — you can immediately see if Redis is approaching its maxmemory limit or if a MongoDB container is consuming unexpectedly high CPU.

Production Hardening Checklist

Before this stack serves real infrastructure:

Put Grafana behind a reverse proxy with TLS. Port 3000 over plain HTTP is fine for a local lab, not for anything reachable from the internet. Nginx or Traefik with Let's Encrypt handles termination cleanly.
Change the default Grafana admin password from what's in .env — and rotate it from the CLI, not the UI, on first login.
Set a Prometheus retention period matching your storage capacity. --storage.tsdb.retention.time=30d is a reasonable default; 90 days gives you better trend visibility but needs more disk.
Add resource limits to the Compose file — Prometheus and Grafana can consume significant memory under load. Cap them with deploy.resources.limits to prevent a noisy monitoring stack from affecting the services it's monitoring.
Back up the Grafana volume — this is where your dashboards, saved queries, and alert configurations live. A volume backup on a schedule (same approach as the Redis backup scripts guide) prevents losing custom dashboards when the host is rebuilt.
Restrict Prometheus and Alertmanager access — in this guide both are already bound to 127.0.0.1. Keep it that way. If you need remote access, use an SSH tunnel or VPN rather than opening those ports.

Common Issues and Quick Fixes

Symptom	Likely Cause	Fix
Prometheus shows target as `DOWN`	Node Exporter/cAdvisor not reachable on monitoring network	Confirm all services are on the same `monitoring` network; check `docker compose ps`
No data in Grafana dashboards	Wrong data source URL or Prometheus not ready	Verify `http://prometheus:9090` is set in the data source, not `http://localhost:9090`
cAdvisor container fails to start	Missing `privileged: true` or host volume permissions	Ensure `privileged: true` is set in the cAdvisor service definition
Alerts firing continuously without resolution	`resolve_timeout` too short, or condition never clears	Increase `resolve_timeout` in Alertmanager; verify the alert condition logic
Prometheus disk usage grows too fast	Retention period too long, or too many high-cardinality labels	Lower `--storage.tsdb.retention.time`, audit your scrape configs for label explosion
Grafana login shows "invalid username or password"	`.env` not loaded, or password has special characters breaking YAML	Wrap the password in quotes in `.env`; verify `docker compose config` shows the correct values

Next Steps

With Prometheus and Grafana running, you can extend the stack by adding exporters for the specific services in your infrastructure:

Redis Exporter (oliver006/redis_exporter) — see this alongside our Redis with Docker Compose guide for memory usage, keyspace hits, and eviction rates per Redis instance.
MongoDB Exporter (percona/mongodb_exporter) — pairs with our MongoDB with Docker Compose guide for replica set health, connection pool, and oplog metrics.
Blackbox Exporter — probe external HTTP endpoints, DNS, and TCP ports from Prometheus, turning it into an uptime monitor for services outside the Docker host.
Grafana Loki — add centralized log aggregation to this stack. Loki stores logs in the same way Prometheus stores metrics, and Grafana visualizes both in the same dashboard — one pane of glass for metrics and logs simultaneously.

How to Install Apache Tomcat 11 on Ubuntu 26.04 LTS (Step-by-Step Guide)

Ramansah — Mon, 13 Jul 2026 09:48:33 +0000

What is Apache Tomcat and Who Needs It
Tomcat 11 vs Previous Versions: What Changed
Prerequisites
Step 1: Install Java 21
Step 2: Create a Dedicated Tomcat User
Step 3: Download and Install Tomcat 11
Step 4: Configure Environment Variables
Step 5: Create a systemd Service
Step 6: Configure UFW Firewall
Step 7: Configure Tomcat Users and Roles
Step 8: Enable Remote Access to Manager GUI
Step 9: Deploy a WAR Application
Step 10: Put Tomcat Behind Nginx Reverse Proxy
Security Hardening Checklist
Tomcat vs Jetty vs Undertow: Which to Choose
Common Issues and Quick Fixes
Next Steps

Java web applications don't run on their own — they need a servlet container to manage their lifecycle, handle HTTP requests, and translate the Jakarta Servlet specification into actual running code. Apache Tomcat has been doing exactly that since 1999, and it remains the most widely deployed open-source servlet container in the world.

Whether you're deploying a Spring Boot WAR file, a legacy enterprise application, or building a Java-based REST API, this guide walks through a complete Tomcat 11 installation on Ubuntu 26.04 LTS — from Java setup through production hardening.

What is Apache Tomcat and Who Needs It

Tomcat is simultaneously a web server and a servlet container:

As a web server, it handles HTTP and HTTPS requests directly (though in production it typically sits behind a dedicated reverse proxy like Nginx).
As a servlet container (also called a web container), it implements the Jakarta Servlet, Jakarta Server Pages (JSP), and WebSocket specifications — the standard APIs that Java web applications are built against.

If your application is packaged as a WAR file (Web Application Archive), Tomcat is almost certainly the right deployment target. If your team is using Spring Boot and packaging as a standalone JAR with an embedded server, you may not need a standalone Tomcat installation at all — Spring Boot can embed Tomcat, Jetty, or Undertow inside the JAR itself.

Tomcat 11 vs Previous Versions: What Changed

Feature	Tomcat 9	Tomcat 10	Tomcat 11
Servlet spec	Servlet 4.0	Servlet 5.0	Servlet 6.1
Java required	Java 8+	Java 11+	Java 17+
Namespace	`javax.*`	`jakarta.*`	`jakarta.*`
HTTP/2	Limited	Yes	Yes (improved)
Virtual threads (Java 21)	No	No	Yes
LTS status	EOL soon	Maintained	Current LTS

The most important breaking change between Tomcat 9 and 10+: the Java package namespace changed from javax.* to jakarta.*. Applications written for Tomcat 9 (using javax.servlet) need a code change or migration tool (Eclipse Transformer) to run on Tomcat 10 or 11 — they're not drop-in compatible.

Virtual thread support in Tomcat 11 with Java 21 is the headline feature of this combination: Java 21's virtual threads (Project Loom) allow Tomcat to handle many more concurrent connections without the memory overhead of a large thread pool. For high-concurrency workloads, this combination delivers significantly better throughput per unit of memory compared to Tomcat 10 + Java 17.

Prerequisites

Ubuntu 26.04 LTS (Resolute Raccoon)
Minimum 2GB RAM (4GB+ recommended for production)
sudo access
A domain name (optional, needed for HTTPS via reverse proxy)

Step 1: Install Java 21

Tomcat 11 requires Java 17 or higher. Java 21 LTS is the recommended choice — it's the current Long-Term Support release and unlocks virtual thread support:

sudo apt update
sudo apt install -y openjdk-21-jdk-headless

Verify:

java -version

Expected output:

openjdk version "21.x.x" ...
OpenJDK Runtime Environment (build 21.x.x+...)

Set JAVA_HOME system-wide:

echo 'export JAVA_HOME=/usr/lib/jvm/java-21-openjdk-amd64' | \
  sudo tee /etc/profile.d/java.sh
source /etc/profile.d/java.sh

echo $JAVA_HOME
# /usr/lib/jvm/java-21-openjdk-amd64

Step 2: Create a Dedicated Tomcat User

Running Tomcat as root is a serious security risk — if your application has a vulnerability, an attacker would immediately have root access to the host. Create a dedicated system user with no shell access:

sudo useradd -r -m -U -d /opt/tomcat -s /bin/false tomcat

This mirrors the same principle of least privilege used throughout our Docker Container Security Best Practices guide.

Step 3: Download and Install Tomcat 11

TOMCAT_VERSION="11.0.7"

wget https://dlcdn.apache.org/tomcat/tomcat-11/v${TOMCAT_VERSION}/bin/apache-tomcat-${TOMCAT_VERSION}.tar.gz \
  -O /tmp/tomcat.tar.gz

# Verify the download (recommended — check the SHA512 on the Apache download page)
sha512sum /tmp/tomcat.tar.gz

# Extract to /opt/tomcat
sudo tar -xzf /tmp/tomcat.tar.gz -C /opt/tomcat --strip-components=1

# Set ownership and permissions
sudo chown -R tomcat:tomcat /opt/tomcat
sudo chmod -R u=rwX,g=rX,o= /opt/tomcat
sudo chmod +x /opt/tomcat/bin/*.sh

Verify the directory structure:

ls /opt/tomcat
# bin  conf  lib  logs  temp  webapps  work

Key directories:

Directory	Purpose
`bin/`	Startup and shutdown scripts
`conf/`	Configuration files (`server.xml`, `tomcat-users.xml`, `web.xml`)
`webapps/`	Deploy your WAR files here
`logs/`	Catalina log, access log, error log
`lib/`	Shared libraries available to all applications

Step 4: Configure Environment Variables

Create a setenv.sh file in /opt/tomcat/bin/ — Tomcat automatically sources this at startup, the correct place for JVM tuning options:

sudo nano /opt/tomcat/bin/setenv.sh

#!/bin/bash

# JVM heap size — adjust based on available RAM
# Rule of thumb: set -Xms and -Xmx to the same value to prevent heap resizing
export CATALINA_OPTS="-Xms512M -Xmx1024M -server"

# Java 21: enable virtual threads for Loom-based connector (Tomcat 11+)
export CATALINA_OPTS="$CATALINA_OPTS -Djava.util.concurrent.ForkJoinPool.common.parallelism=4"

# Security: use /dev/urandom instead of /dev/random to avoid blocking
export JAVA_OPTS="-Djava.awt.headless=true -Djava.security.egd=file:/dev/./urandom"

# JAVA_HOME
export JAVA_HOME=/usr/lib/jvm/java-21-openjdk-amd64

Set the correct ownership and permissions:

sudo chown tomcat:tomcat /opt/tomcat/bin/setenv.sh
sudo chmod +x /opt/tomcat/bin/setenv.sh

Step 5: Create a systemd Service

sudo nano /etc/systemd/system/tomcat.service

[Unit]
Description=Apache Tomcat 11 Web Application Server
Documentation=https://tomcat.apache.org/
After=network.target

[Service]
Type=forking
User=tomcat
Group=tomcat

Environment="JAVA_HOME=/usr/lib/jvm/java-21-openjdk-amd64"
Environment="CATALINA_HOME=/opt/tomcat"
Environment="CATALINA_BASE=/opt/tomcat"
Environment="CATALINA_PID=/opt/tomcat/temp/tomcat.pid"
Environment="CATALINA_OPTS=-Xms512M -Xmx1024M -server"
Environment="JAVA_OPTS=-Djava.awt.headless=true -Djava.security.egd=file:/dev/./urandom"

ExecStart=/opt/tomcat/bin/startup.sh
ExecStop=/opt/tomcat/bin/shutdown.sh

Restart=on-failure
RestartSec=10
SuccessExitStatus=143
LimitNOFILE=65536
StandardOutput=journal
StandardError=journal
SyslogIdentifier=tomcat

[Install]
WantedBy=multi-user.target

Enable and start:

sudo systemctl daemon-reload
sudo systemctl enable tomcat
sudo systemctl start tomcat
sudo systemctl status tomcat

Expected:

● tomcat.service - Apache Tomcat 11 Web Application Server
     Active: active (running) since ...

View live logs:

sudo journalctl -u tomcat -f
# Or directly from Catalina log:
sudo tail -f /opt/tomcat/logs/catalina.out

Open http://<your-server-ip>:8080 — you should see the Tomcat welcome page.

Step 6: Configure UFW Firewall

# Allow Tomcat HTTP port (8080)
sudo ufw allow 8080/tcp comment "Tomcat HTTP"

# If running HTTPS directly on Tomcat (8443)
# sudo ufw allow 8443/tcp comment "Tomcat HTTPS"

sudo ufw reload
sudo ufw status

Production note: In a proper production setup, Tomcat should not be directly accessible on port 8080. It should sit behind Nginx (or Nginx Proxy Manager) which handles HTTPS termination. Port 8080 should then only be accessible from localhost. See Step 10 for the Nginx reverse proxy configuration.

Step 7: Configure Tomcat Users and Roles

Tomcat's web-based Manager and Host Manager applications require user credentials defined in tomcat-users.xml. By default, no users are configured — the Manager UI is installed but inaccessible:

sudo nano /opt/tomcat/conf/tomcat-users.xml

Add these lines inside the <tomcat-users> tag, just before </tomcat-users>:

<!-- Manager GUI user — for deploying/undeploying apps via the web interface -->
<role rolename="manager-gui"/>
<role rolename="manager-script"/>
<role rolename="admin-gui"/>

<user username="manager"
      password="STRONG_MANAGER_PASSWORD_HERE"
      roles="manager-gui,manager-script"/>

<user username="admin"
      password="STRONG_ADMIN_PASSWORD_HERE"
      roles="manager-gui,admin-gui"/>

Replace both passwords with strong, unique values. These credentials provide full application deployment access — treat them like root passwords.

Restart Tomcat to apply:

sudo systemctl restart tomcat

Step 8: Enable Remote Access to Manager GUI

By default, the Manager and Host Manager apps only accept connections from localhost (127.0.0.1). This is a sensible security default — to access them from a remote browser, you need to explicitly allow your IP.

Edit the Manager's context configuration:

sudo nano /opt/tomcat/webapps/manager/META-INF/context.xml

Find this block:

<Valve className="org.apache.catalina.valves.RemoteAddrValve"
       allow="127\.\d+\.\d+\.\d+|::1|0:0:0:0:0:0:0:1" />

Add your IP address to the allow pattern:

<Valve className="org.apache.catalina.valves.RemoteAddrValve"
       allow="127\.\d+\.\d+\.\d+|::1|0:0:0:0:0:0:0:1|YOUR\.IP\.ADDRESS\.HERE" />

Do the same for Host Manager:

sudo nano /opt/tomcat/webapps/host-manager/META-INF/context.xml

Restart Tomcat:

sudo systemctl restart tomcat

Access the Manager at http://<your-server-ip>:8080/manager/html and log in with the manager credentials from tomcat-users.xml.

Better alternative: Instead of opening up the RemoteAddrValve, put Tomcat behind Nginx Proxy Manager and restrict the /manager path to specific IPs via an access control list — this keeps the control in one place and doesn't require editing Tomcat config files every time your IP changes.

Step 9: Deploy a WAR Application

Deploying a Java web application to Tomcat is straightforward — copy the WAR file to the webapps directory and Tomcat auto-deploys it:

# Copy your WAR file to webapps
sudo cp /path/to/your-application.war /opt/tomcat/webapps/
sudo chown tomcat:tomcat /opt/tomcat/webapps/your-application.war

Tomcat automatically extracts the WAR and deploys the application within seconds. The app becomes accessible at:

http://<your-server-ip>:8080/your-application/

Watch the deployment in the logs:

sudo tail -f /opt/tomcat/logs/catalina.out

Deploy to ROOT context (accessible at http://your-server/ with no path):

# Remove existing ROOT app first
sudo rm -rf /opt/tomcat/webapps/ROOT
sudo cp /path/to/your-application.war /opt/tomcat/webapps/ROOT.war
sudo chown tomcat:tomcat /opt/tomcat/webapps/ROOT.war

Undeploy an application:

# Remove the WAR and the extracted directory
sudo rm -f /opt/tomcat/webapps/your-application.war
sudo rm -rf /opt/tomcat/webapps/your-application

Tomcat detects the removal and undeploys the application automatically without a restart.

Step 10: Put Tomcat Behind Nginx Reverse Proxy

For production, Nginx handles HTTPS termination and forwards requests to Tomcat on localhost:8080. This keeps port 8080 off the internet entirely:

sudo nano /etc/nginx/sites-available/tomcat-app

server {
    listen 80;
    server_name app.yourdomain.com;

    # Redirect HTTP to HTTPS
    return 301 https://$host$request_uri;
}

server {
    listen 443 ssl http2;
    server_name app.yourdomain.com;

    ssl_certificate     /etc/letsencrypt/live/app.yourdomain.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/app.yourdomain.com/privkey.pem;

    # Security headers
    add_header X-Frame-Options "SAMEORIGIN" always;
    add_header X-Content-Type-Options "nosniff" always;
    add_header X-XSS-Protection "1; mode=block" always;
    add_header Strict-Transport-Security "max-age=31536000" always;

    location / {
        proxy_pass         http://localhost:8080;
        proxy_set_header   Host $host;
        proxy_set_header   X-Real-IP $remote_addr;
        proxy_set_header   X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header   X-Forwarded-Proto $scheme;

        # WebSocket support
        proxy_http_version 1.1;
        proxy_set_header   Upgrade $http_upgrade;
        proxy_set_header   Connection "upgrade";

        # Timeouts for long-running requests
        proxy_connect_timeout  60s;
        proxy_read_timeout    300s;
        proxy_send_timeout    300s;
    }
}

Enable the site and reload Nginx:

sudo ln -s /etc/nginx/sites-available/tomcat-app /etc/nginx/sites-enabled/
sudo nginx -t
sudo systemctl reload nginx

If you're using Nginx Proxy Manager instead of manual Nginx config, the same setup takes 30 seconds in the GUI — Forward Hostname: localhost, Forward Port: 8080, enable Force SSL and Let's Encrypt.

After setting up the reverse proxy, update Tomcat's firewall rules to only allow port 8080 from localhost:

# Remove public access to port 8080
sudo ufw delete allow 8080/tcp

# Only allow from localhost (Nginx)
sudo ufw allow from 127.0.0.1 to any port 8080 comment "Tomcat from Nginx only"
sudo ufw reload

Security Hardening Checklist

A default Tomcat installation has several settings that should be changed before handling production traffic:

1. Remove default web applications:

sudo rm -rf /opt/tomcat/webapps/examples
sudo rm -rf /opt/tomcat/webapps/docs
# Keep manager and host-manager only if actively used

The examples app contains known vulnerabilities and has no place in production.

2. Disable the Tomcat shutdown port:
Edit /opt/tomcat/conf/server.xml, change:

<Server port="8005" shutdown="SHUTDOWN">

to:

<Server port="-1" shutdown="SHUTDOWN">

This disables the shutdown command port — a common attack vector where telnet localhost 8005 followed by "SHUTDOWN" would stop your Tomcat instance.

3. Remove the Server header:
In server.xml, inside the <Connector> element:

<Connector port="8080" protocol="HTTP/1.1"
           server=" "
           connectionTimeout="20000"
           redirectPort="8443" />

Setting server=" " prevents Tomcat from advertising its version in HTTP response headers.

4. Enable access logging:
Already enabled by default in server.xml (AccessLogValve) — confirm it's not commented out.

5. Set secure cookie flags in web.xml:

<session-config>
  <cookie-config>
    <http-only>true</http-only>
    <secure>true</secure>
  </cookie-config>
</session-config>

Tomcat vs Jetty vs Undertow: Which to Choose

	Apache Tomcat	Jetty	Undertow
Maturity	25+ years	25+ years	2013+
Primary use	Standalone WAR deployment	Embedded, microservices	Embedded, WildFly component
Memory footprint	Moderate	Light	Light
Virtual threads (Java 21)	Yes (Tomcat 11)	Yes	Yes
Jakarta EE compliance	Servlet/JSP only	Servlet/JSP only	Via WildFly
Embedded use	Yes (Spring Boot)	Yes (Spring Boot, Jetty)	Yes (Spring Boot)
Standalone deployment	✅ Excellent	Good	Limited
Community/ecosystem	Largest	Large	Smaller

Use Tomcat when: you're deploying WAR files, need a proven standalone server, or are working with Spring Boot (which defaults to embedded Tomcat).

Use Jetty when: you need an extremely lightweight embedded server, or you're building an application where startup speed matters (like AWS Lambda or serverless).

Use Undertow when: you're already in the WildFly/JBoss ecosystem or need non-blocking I/O at the core level.

Common Issues and Quick Fixes

Symptom	Likely Cause	Fix
Tomcat fails to start — `JAVA_HOME not set`	`JAVA_HOME` not configured in systemd unit	Verify `Environment="JAVA_HOME=..."` in `tomcat.service`
Tomcat starts but shows blank page	Default ROOT app removed	Copy your WAR to `webapps/ROOT.war` or check `webapps/ROOT/`
Manager UI shows 403 Forbidden	Remote IP not in `RemoteAddrValve` allow list	Edit `webapps/manager/META-INF/context.xml` to add your IP
404 on Manager login	Wrong URL — include trailing path	URL must be `http://ip:8080/manager/html`, not `http://ip:8080/manager`
WAR deployed but app returns 404	App context path differs from WAR filename	Access at `http://ip:8080/<war-filename-without-extension>/`
`OutOfMemoryError: Java heap space`	JVM heap too small	Increase `-Xmx` in `setenv.sh` or systemd unit
High CPU on first request	JIT compilation warming up	Normal for first few requests — use JVM warmup or AOT if critical

Next Steps

With Tomcat 11 running on Ubuntu 26.04, the natural next steps for building a complete Java infrastructure:

Apache Maven — automate the build-and-package process that produces the WAR files you deploy to Tomcat (coming next in this series).
Apache Kafka — if your Java application produces or consumes events, our Kafka installation guide covers the full setup on Ubuntu 26.04.
Nginx Proxy Manager — the fastest way to put Tomcat behind HTTPS with a free Let's Encrypt certificate — see our Nginx Proxy Manager guide.
Prometheus + Grafana — monitor Tomcat's JVM heap, thread pool, and request throughput via JMX Exporter — see the monitoring stack guide.

How to Install Apache Kafka on Ubuntu 26.04 LTS (KRaft Mode, No ZooKeeper)

Ramansah — Mon, 13 Jul 2026 09:41:36 +0000

What is Apache Kafka and When Should You Use It
KRaft Mode: Why ZooKeeper is No Longer Needed
Prerequisites
Step 1: Install Java 21
Step 2: Create a Dedicated Kafka User
Step 3: Download and Install Apache Kafka 4.2
Step 4: Configure Kafka in KRaft Mode
Step 5: Format the Storage Directory
Step 6: Create a systemd Service
Step 7: Configure UFW Firewall
Step 8: Test with Producer and Consumer
Step 9: Basic Topic Management
Production Configuration Tips
Kafka vs RabbitMQ: When to Use Which
Common Issues and Quick Fixes
Next Steps

Real-time data is everywhere — microservices firing events, IoT sensors streaming telemetry, user actions triggering notifications, logs flowing from dozens of servers simultaneously. Managing all of that reliably, at scale, without building a custom integration between every producer and every consumer, is exactly the problem Apache Kafka was built to solve.

This guide walks through installing Apache Kafka 4.2 on Ubuntu 26.04 LTS (Resolute Raccoon) using KRaft mode — the modern deployment approach that removes the ZooKeeper dependency entirely. If you've seen older Kafka tutorials that tell you to start ZooKeeper first, those are outdated: starting with Kafka 4.0, ZooKeeper support was completely removed.

What is Apache Kafka and When Should You Use It

Apache Kafka is a distributed event streaming platform — a durable, ordered log that producers write to and consumers read from. Unlike a traditional message queue where a message disappears after being consumed, Kafka retains events for a configurable period, allowing multiple independent consumers to read the same stream at their own pace.

Real-world use cases where Kafka fits well:

Microservice decoupling — Service A publishes an event; Services B, C, and D each consume it independently, without A knowing or caring who's listening.
Log and metrics aggregation — Centralize logs from dozens of services into one stream, then fan out to Elasticsearch, S3, and a monitoring stack simultaneously.
Real-time analytics — Process a stream of user events, transactions, or sensor readings as they happen rather than batching overnight.
Event sourcing — Store every state change as an immutable event, with the ability to replay history to rebuild application state.
Change Data Capture (CDC) — Stream database changes (inserts, updates, deletes) to downstream systems in real time.

Kafka is not the right tool for every messaging need. If you're sending a task to exactly one worker and want it acknowledged once, a simpler queue (Redis Streams, RabbitMQ, or even PostgreSQL LISTEN/NOTIFY) is probably sufficient and far easier to operate. Kafka earns its complexity at scale.

KRaft Mode: Why ZooKeeper is No Longer Needed

Before Kafka 3.x, every Kafka deployment required a separate Apache ZooKeeper cluster to manage broker metadata, leader elections, and cluster state. This meant running and maintaining two distributed systems for every Kafka deployment — doubling the operational complexity.

KRaft (Kafka Raft Metadata mode) replaces ZooKeeper by moving cluster metadata management directly into Kafka itself, using the Raft consensus algorithm. The result:

Simpler deployment — one system to install, configure, monitor, and upgrade instead of two.
Faster startup and failover — Kafka no longer needs to synchronize with an external ZooKeeper cluster.
Better scalability — ZooKeeper had practical limits on the number of partitions it could track; KRaft removes those limits.
Kafka 4.0+: ZooKeeper support is completely removed. There is no option to use ZooKeeper with Kafka 4.x — KRaft is the only deployment mode.

Prerequisites

Ubuntu 26.04 LTS (Resolute Raccoon) — fresh install or existing server
Minimum 4GB RAM (8GB+ recommended for production workloads)
At least 2 CPU cores
sudo access
Internet access to download packages

Step 1: Install Java 21

Kafka 4.x requires Java 17 or higher. Java 21 LTS is the recommended choice for Ubuntu 26.04 — it's the current Long-Term Support release and ships cleanly from Ubuntu's default repository:

sudo apt update
sudo apt install -y openjdk-21-jdk-headless

Verify the installation:

java -version

Expected output:

openjdk version "21.x.x" ...
OpenJDK Runtime Environment (build 21.x.x+...)

Set JAVA_HOME so Kafka can find it:

echo 'export JAVA_HOME=/usr/lib/jvm/java-21-openjdk-amd64' | \
  sudo tee /etc/profile.d/java.sh
source /etc/profile.d/java.sh

Confirm:

echo $JAVA_HOME
# /usr/lib/jvm/java-21-openjdk-amd64

Step 2: Create a Dedicated Kafka User

Running Kafka as root is a security risk. Create a dedicated system user with no login access:

sudo useradd -r -m -U -d /opt/kafka -s /bin/false kafka

This follows the same principle of least privilege covered in our Docker Container Security Best Practices guide — services should run with the minimum access they need, never as root.

Step 3: Download and Install Apache Kafka 4.2

Set the version variables, then download and extract to /opt/kafka:

KAFKA_VERSION="4.2.0"
SCALA_VERSION="2.13"

wget https://downloads.apache.org/kafka/${KAFKA_VERSION}/kafka_${SCALA_VERSION}-${KAFKA_VERSION}.tgz \
  -O /tmp/kafka.tgz

sudo tar -xzf /tmp/kafka.tgz -C /opt/kafka --strip-components=1
sudo chown -R kafka:kafka /opt/kafka

Verify the extraction:

ls /opt/kafka
# bin  config  libs  LICENSE  licenses  NOTICE  site-docs

Create a dedicated data directory for Kafka logs (separate from the application directory — good practice for disk management and backups):

sudo mkdir -p /var/lib/kafka/data
sudo chown -R kafka:kafka /var/lib/kafka

Step 4: Configure Kafka in KRaft Mode

KRaft mode uses a single unified configuration file (server.properties) where the broker also acts as the cluster controller. Edit the KRaft server configuration:

sudo nano /opt/kafka/config/kraft/server.properties

Key settings to review and update:

# The role this node plays in the cluster
# "broker,controller" = combined mode (suitable for single-node and small clusters)
process.roles=broker,controller

# Unique ID for this broker — change if running multiple brokers
node.id=1

# Controller quorum voters — format: node.id@host:port
# For single node: use the same node
controller.quorum.voters=1@localhost:9093

# Listeners — what addresses Kafka binds to
listeners=PLAINTEXT://localhost:9092,CONTROLLER://localhost:9093

# The address clients use to reach the broker
# Change 'localhost' to your server's actual IP or hostname if remote clients need access
advertised.listeners=PLAINTEXT://localhost:9092

# Listener used for inter-broker and controller communication
inter.broker.listener.name=PLAINTEXT
controller.listener.names=CONTROLLER

# Log directory — where Kafka stores event data
log.dirs=/var/lib/kafka/data

# Number of partitions for auto-created topics
num.partitions=3

# Log retention (how long to keep events)
log.retention.hours=168        # 7 days
log.retention.bytes=1073741824 # 1GB per partition

# Replication factor for internal topics (keep at 1 for single-node)
offsets.topic.replication.factor=1
transaction.state.log.replication.factor=1
transaction.state.log.min.isr=1

Remote access note: If clients will connect from other machines, replace localhost in advertised.listeners with your server's actual IP address or DNS hostname. Kafka clients use the advertised address to establish connections — using localhost here will cause remote clients to fail even if they can reach the server on port 9092.

Step 5: Format the Storage Directory

Before first startup, Kafka's storage directory must be initialized with a unique cluster ID. This is a one-time operation:

# Generate a cluster ID
CLUSTER_ID=$(/opt/kafka/bin/kafka-storage.sh random-uuid)
echo "Cluster ID: $CLUSTER_ID"

# Format the storage directory with that cluster ID
sudo -u kafka /opt/kafka/bin/kafka-storage.sh format \
  -t $CLUSTER_ID \
  -c /opt/kafka/config/kraft/server.properties

Expected output:

Formatting /var/lib/kafka/data with metadata.version X.X-IV...

This step replaces the old ZooKeeper initialization — there's no zookeeper-server-start.sh to run first.

Step 6: Create a systemd Service

Create a systemd unit file so Kafka starts automatically on boot and can be managed with systemctl:

sudo nano /etc/systemd/system/kafka.service

[Unit]
Description=Apache Kafka Server (KRaft Mode)
Documentation=https://kafka.apache.org/documentation/
After=network.target

[Service]
Type=simple
User=kafka
Group=kafka
Environment="JAVA_HOME=/usr/lib/jvm/java-21-openjdk-amd64"
ExecStart=/opt/kafka/bin/kafka-server-start.sh \
  /opt/kafka/config/kraft/server.properties
ExecStop=/opt/kafka/bin/kafka-server-stop.sh
Restart=on-failure
RestartSec=10
LimitNOFILE=65536
StandardOutput=journal
StandardError=journal
SyslogIdentifier=kafka

[Install]
WantedBy=multi-user.target

Enable and start the service:

sudo systemctl daemon-reload
sudo systemctl enable kafka
sudo systemctl start kafka

Check the status:

sudo systemctl status kafka

Expected output:

● kafka.service - Apache Kafka Server (KRaft Mode)
     Loaded: loaded (/etc/systemd/system/kafka.service; enabled)
     Active: active (running) since ...

View live logs:

sudo journalctl -u kafka -f

Step 7: Configure UFW Firewall

If UFW is enabled, open the ports Kafka needs:

# Port 9092 — Kafka broker (clients connect here)
sudo ufw allow 9092/tcp comment "Kafka broker"

# Port 9093 — Kafka controller (internal KRaft communication)
# Only needed if running multi-node — restrict to internal network for single-node
sudo ufw allow from 10.0.0.0/8 to any port 9093 comment "Kafka KRaft controller"

sudo ufw reload
sudo ufw status

Security note: Port 9092 should only be open to trusted clients, not the entire internet. If your Kafka broker is internet-facing, restrict port 9092 to specific IP ranges or put it behind a VPN.

Step 8: Test with Producer and Consumer

Open two separate terminal sessions for this test — one for the producer, one for the consumer.

First: Create a test topic

sudo -u kafka /opt/kafka/bin/kafka-topics.sh \
  --create \
  --topic test-events \
  --partitions 3 \
  --replication-factor 1 \
  --bootstrap-server localhost:9092

Expected output:

Created topic test-events.

Terminal 1: Start a producer and send messages

sudo -u kafka /opt/kafka/bin/kafka-console-producer.sh \
  --topic test-events \
  --bootstrap-server localhost:9092

Type a few messages and press Enter after each:

> Hello from Kafka on Ubuntu 26.04
> This is a test message
> Apache Kafka 4.2 KRaft mode works!

Press Ctrl+C to exit.

Terminal 2: Start a consumer and read messages

sudo -u kafka /opt/kafka/bin/kafka-console-consumer.sh \
  --topic test-events \
  --from-beginning \
  --bootstrap-server localhost:9092

You should see the messages you typed appear immediately. Press Ctrl+C to exit.

--from-beginning tells the consumer to read from the earliest available offset — not just new messages. This illustrates one of Kafka's key properties: consumers can replay historical events, not just receive new ones.

Step 9: Basic Topic Management

# List all topics
sudo -u kafka /opt/kafka/bin/kafka-topics.sh \
  --list \
  --bootstrap-server localhost:9092

# Describe a topic (partitions, replicas, leader)
sudo -u kafka /opt/kafka/bin/kafka-topics.sh \
  --describe \
  --topic test-events \
  --bootstrap-server localhost:9092

# Increase partition count (partitions can only be increased, never decreased)
sudo -u kafka /opt/kafka/bin/kafka-topics.sh \
  --alter \
  --topic test-events \
  --partitions 6 \
  --bootstrap-server localhost:9092

# Delete a topic
sudo -u kafka /opt/kafka/bin/kafka-topics.sh \
  --delete \
  --topic test-events \
  --bootstrap-server localhost:9092

# List consumer groups
sudo -u kafka /opt/kafka/bin/kafka-consumer-groups.sh \
  --list \
  --bootstrap-server localhost:9092

# Check consumer group lag (how far behind a consumer is)
sudo -u kafka /opt/kafka/bin/kafka-consumer-groups.sh \
  --describe \
  --group my-consumer-group \
  --bootstrap-server localhost:9092

Consumer group lag (last command) is one of the most important operational metrics in a Kafka deployment — it tells you how many messages a consumer is behind. Integrating this metric into your Prometheus + Grafana monitoring stack via kafka-exporter or JMX exporter gives you real-time visibility into consumer health.

Production Configuration Tips

A single-node Kafka installation on Ubuntu is sufficient for development and low-traffic production workloads. For anything that needs to scale or survive a broker failure, here are the key settings to revisit:

Memory tuning:
Kafka's JVM heap is set to 1GB by default. For production, adjust in /opt/kafka/bin/kafka-server-start.sh:

export KAFKA_HEAP_OPTS="-Xmx4G -Xms4G"

Set both -Xmx and -Xms to the same value to prevent JVM heap resizing pauses.

Log retention by size, not just time:

# Retain logs for 7 days OR until they exceed 10GB per partition, whichever comes first
log.retention.hours=168
log.retention.bytes=10737418240

Auto topic creation in production:

# Disable auto-creation — require topics to be created explicitly
auto.create.topics.enable=false

Auto-created topics use default partition/replication settings, which are rarely correct for specific use cases. Disable this and create topics explicitly with the right parameters.

Replication factor for multi-broker clusters:

# With 3 brokers, use replication factor 3 and min ISR 2
# (data survives loss of any 1 broker, writes require 2 brokers to acknowledge)
default.replication.factor=3
min.insync.replicas=2

Regular backups:
The /var/lib/kafka/data directory contains all event data. Back it up the same way you'd back up any other data volume — a scheduled snapshot with retention policy, tested restore process, stored separately from the host. The scripted approach from our Redis with Docker Compose guide applies directly to the Kafka data directory.

Kafka vs RabbitMQ: When to Use Which

A common question when introducing a message broker for the first time:

	Apache Kafka	RabbitMQ
Model	Distributed log (pull-based)	Message queue (push-based)
Message retention	Configurable period (days/weeks)	Until acknowledged (usually)
Throughput	Millions of messages/second	Tens to hundreds of thousands/second
Consumer pattern	Each consumer group reads the full log independently	One consumer per message (competing consumers)
Replay	Yes — consumers can re-read past events	No — once consumed, message is gone
Ordering guarantee	Per partition	Per queue
Operational complexity	Higher	Lower
Best for	Event streaming, log aggregation, CDC, high throughput	Task queues, RPC, point-to-point messaging, simpler use cases

Choose Kafka when: You need multiple consumers to independently process the same events, you need to replay history, or you need to handle very high throughput (millions of events/second).

Choose RabbitMQ when: You're building a task queue where each task should be processed exactly once by exactly one worker, the message volume is moderate, and you want simpler operations.

Common Issues and Quick Fixes

Symptom	Likely Cause	Fix
`kafka.service` fails to start	Storage not formatted, or wrong `JAVA_HOME`	Run `kafka-storage.sh format` first; verify `java -version` works as kafka user
`Connection refused` on port 9092	Kafka not running, or listener bound to wrong address	Check `systemctl status kafka`; verify `listeners` in `server.properties`
Remote clients can't connect	`advertised.listeners` set to `localhost`	Change `advertised.listeners` to the server's actual IP or hostname
`Leader not available` on topic create	Kafka still initializing after start	Wait 10-15 seconds after Kafka starts before creating topics
Consumer reads no messages	Consumer started after producer, no `--from-beginning` flag	Add `--from-beginning` flag to read historical messages
`OutOfMemoryError` in logs	JVM heap too small for the workload	Increase `-Xmx` and `-Xms` in `kafka-server-start.sh`
High consumer group lag	Consumer too slow or too few consumer instances	Scale consumer instances; check processing logic for bottlenecks

Next Steps

With Kafka running on Ubuntu 26.04, the natural next steps are:

Add monitoring — deploy kafka-exporter alongside your Prometheus + Grafana stack to track consumer group lag, message throughput, and broker health in real time.
Containerize it — for teams already running Docker Compose stacks, Kafka has an official Docker image and can be added to an existing Compose setup using the same KRaft configuration.
Explore Kafka Streams or Apache Flink — for in-stream processing (filtering, aggregating, joining streams) without writing a separate consumer application.
Scale to multi-broker — repeat the installation on additional nodes, adjust broker.id and controller.quorum.voters, and distribute your topics' partitions across the cluster for fault tolerance and horizontal throughput scaling.

How to Install Apache Kafka on Ubuntu 26.04 LTS (KRaft Mode, No ZooKeeper)

Ramansah — Sun, 12 Jul 2026 16:01:06 +0000

What is Apache Kafka and When Should You Use It
KRaft Mode: Why ZooKeeper is No Longer Needed
Prerequisites
Step 1: Install Java 21
Step 2: Create a Dedicated Kafka User
Step 3: Download and Install Apache Kafka 4.2
Step 4: Configure Kafka in KRaft Mode
Step 5: Format the Storage Directory
Step 6: Create a systemd Service
Step 7: Configure UFW Firewall
Step 8: Test with Producer and Consumer
Step 9: Basic Topic Management
Production Configuration Tips
Kafka vs RabbitMQ: When to Use Which
Common Issues and Quick Fixes
Next Steps

What is Apache Kafka and When Should You Use It

Real-world use cases where Kafka fits well:

Microservice decoupling — Service A publishes an event; Services B, C, and D each consume it independently, without A knowing or caring who's listening.
Log and metrics aggregation — Centralize logs from dozens of services into one stream, then fan out to Elasticsearch, S3, and a monitoring stack simultaneously.
Real-time analytics — Process a stream of user events, transactions, or sensor readings as they happen rather than batching overnight.
Event sourcing — Store every state change as an immutable event, with the ability to replay history to rebuild application state.
Change Data Capture (CDC) — Stream database changes (inserts, updates, deletes) to downstream systems in real time.

KRaft Mode: Why ZooKeeper is No Longer Needed

KRaft (Kafka Raft Metadata mode) replaces ZooKeeper by moving cluster metadata management directly into Kafka itself, using the Raft consensus algorithm. The result:

Simpler deployment — one system to install, configure, monitor, and upgrade instead of two.
Faster startup and failover — Kafka no longer needs to synchronize with an external ZooKeeper cluster.
Better scalability — ZooKeeper had practical limits on the number of partitions it could track; KRaft removes those limits.
Kafka 4.0+: ZooKeeper support is completely removed. There is no option to use ZooKeeper with Kafka 4.x — KRaft is the only deployment mode.

Prerequisites

Ubuntu 26.04 LTS (Resolute Raccoon) — fresh install or existing server
Minimum 4GB RAM (8GB+ recommended for production workloads)
At least 2 CPU cores
sudo access
Internet access to download packages

Step 1: Install Java 21

Kafka 4.x requires Java 17 or higher. Java 21 LTS is the recommended choice for Ubuntu 26.04 — it's the current Long-Term Support release and ships cleanly from Ubuntu's default repository:

sudo apt update
sudo apt install -y openjdk-21-jdk-headless

Verify the installation:

java -version

Expected output:

openjdk version "21.x.x" ...
OpenJDK Runtime Environment (build 21.x.x+...)

Set JAVA_HOME so Kafka can find it:

echo 'export JAVA_HOME=/usr/lib/jvm/java-21-openjdk-amd64' | \
  sudo tee /etc/profile.d/java.sh
source /etc/profile.d/java.sh

Confirm:

echo $JAVA_HOME
# /usr/lib/jvm/java-21-openjdk-amd64

Step 2: Create a Dedicated Kafka User

Running Kafka as root is a security risk. Create a dedicated system user with no login access:

sudo useradd -r -m -U -d /opt/kafka -s /bin/false kafka

This follows the same principle of least privilege covered in our Docker Container Security Best Practices guide — services should run with the minimum access they need, never as root.

Step 3: Download and Install Apache Kafka 4.2

Set the version variables, then download and extract to /opt/kafka:

KAFKA_VERSION="4.2.0"
SCALA_VERSION="2.13"

wget https://downloads.apache.org/kafka/${KAFKA_VERSION}/kafka_${SCALA_VERSION}-${KAFKA_VERSION}.tgz \
  -O /tmp/kafka.tgz

sudo tar -xzf /tmp/kafka.tgz -C /opt/kafka --strip-components=1
sudo chown -R kafka:kafka /opt/kafka

Verify the extraction:

ls /opt/kafka
# bin  config  libs  LICENSE  licenses  NOTICE  site-docs

Create a dedicated data directory for Kafka logs (separate from the application directory — good practice for disk management and backups):

sudo mkdir -p /var/lib/kafka/data
sudo chown -R kafka:kafka /var/lib/kafka

Step 4: Configure Kafka in KRaft Mode

KRaft mode uses a single unified configuration file (server.properties) where the broker also acts as the cluster controller. Edit the KRaft server configuration:

sudo nano /opt/kafka/config/kraft/server.properties

Key settings to review and update:

# The role this node plays in the cluster
# "broker,controller" = combined mode (suitable for single-node and small clusters)
process.roles=broker,controller

# Unique ID for this broker — change if running multiple brokers
node.id=1

# Controller quorum voters — format: node.id@host:port
# For single node: use the same node
controller.quorum.voters=1@localhost:9093

# Listeners — what addresses Kafka binds to
listeners=PLAINTEXT://localhost:9092,CONTROLLER://localhost:9093

# The address clients use to reach the broker
# Change 'localhost' to your server's actual IP or hostname if remote clients need access
advertised.listeners=PLAINTEXT://localhost:9092

# Listener used for inter-broker and controller communication
inter.broker.listener.name=PLAINTEXT
controller.listener.names=CONTROLLER

# Log directory — where Kafka stores event data
log.dirs=/var/lib/kafka/data

# Number of partitions for auto-created topics
num.partitions=3

# Log retention (how long to keep events)
log.retention.hours=168        # 7 days
log.retention.bytes=1073741824 # 1GB per partition

# Replication factor for internal topics (keep at 1 for single-node)
offsets.topic.replication.factor=1
transaction.state.log.replication.factor=1
transaction.state.log.min.isr=1

Remote access note: If clients will connect from other machines, replace localhost in advertised.listeners with your server's actual IP address or DNS hostname. Kafka clients use the advertised address to establish connections — using localhost here will cause remote clients to fail even if they can reach the server on port 9092.

Step 5: Format the Storage Directory

Before first startup, Kafka's storage directory must be initialized with a unique cluster ID. This is a one-time operation:

# Generate a cluster ID
CLUSTER_ID=$(/opt/kafka/bin/kafka-storage.sh random-uuid)
echo "Cluster ID: $CLUSTER_ID"

# Format the storage directory with that cluster ID
sudo -u kafka /opt/kafka/bin/kafka-storage.sh format \
  -t $CLUSTER_ID \
  -c /opt/kafka/config/kraft/server.properties

Expected output:

Formatting /var/lib/kafka/data with metadata.version X.X-IV...

This step replaces the old ZooKeeper initialization — there's no zookeeper-server-start.sh to run first.

Step 6: Create a systemd Service

Create a systemd unit file so Kafka starts automatically on boot and can be managed with systemctl:

sudo nano /etc/systemd/system/kafka.service

[Unit]
Description=Apache Kafka Server (KRaft Mode)
Documentation=https://kafka.apache.org/documentation/
After=network.target

[Service]
Type=simple
User=kafka
Group=kafka
Environment="JAVA_HOME=/usr/lib/jvm/java-21-openjdk-amd64"
ExecStart=/opt/kafka/bin/kafka-server-start.sh \
  /opt/kafka/config/kraft/server.properties
ExecStop=/opt/kafka/bin/kafka-server-stop.sh
Restart=on-failure
RestartSec=10
LimitNOFILE=65536
StandardOutput=journal
StandardError=journal
SyslogIdentifier=kafka

[Install]
WantedBy=multi-user.target

Enable and start the service:

sudo systemctl daemon-reload
sudo systemctl enable kafka
sudo systemctl start kafka

Check the status:

sudo systemctl status kafka

Expected output:

● kafka.service - Apache Kafka Server (KRaft Mode)
     Loaded: loaded (/etc/systemd/system/kafka.service; enabled)
     Active: active (running) since ...

View live logs:

sudo journalctl -u kafka -f

Step 7: Configure UFW Firewall

If UFW is enabled, open the ports Kafka needs:

# Port 9092 — Kafka broker (clients connect here)
sudo ufw allow 9092/tcp comment "Kafka broker"

# Port 9093 — Kafka controller (internal KRaft communication)
# Only needed if running multi-node — restrict to internal network for single-node
sudo ufw allow from 10.0.0.0/8 to any port 9093 comment "Kafka KRaft controller"

sudo ufw reload
sudo ufw status

Security note: Port 9092 should only be open to trusted clients, not the entire internet. If your Kafka broker is internet-facing, restrict port 9092 to specific IP ranges or put it behind a VPN.

Step 8: Test with Producer and Consumer

Open two separate terminal sessions for this test — one for the producer, one for the consumer.

First: Create a test topic

sudo -u kafka /opt/kafka/bin/kafka-topics.sh \
  --create \
  --topic test-events \
  --partitions 3 \
  --replication-factor 1 \
  --bootstrap-server localhost:9092

Expected output:

Created topic test-events.

Terminal 1: Start a producer and send messages

sudo -u kafka /opt/kafka/bin/kafka-console-producer.sh \
  --topic test-events \
  --bootstrap-server localhost:9092

Type a few messages and press Enter after each:

> Hello from Kafka on Ubuntu 26.04
> This is a test message
> Apache Kafka 4.2 KRaft mode works!

Press Ctrl+C to exit.

Terminal 2: Start a consumer and read messages

sudo -u kafka /opt/kafka/bin/kafka-console-consumer.sh \
  --topic test-events \
  --from-beginning \
  --bootstrap-server localhost:9092

You should see the messages you typed appear immediately. Press Ctrl+C to exit.

Step 9: Basic Topic Management

# List all topics
sudo -u kafka /opt/kafka/bin/kafka-topics.sh \
  --list \
  --bootstrap-server localhost:9092

# Describe a topic (partitions, replicas, leader)
sudo -u kafka /opt/kafka/bin/kafka-topics.sh \
  --describe \
  --topic test-events \
  --bootstrap-server localhost:9092

# Increase partition count (partitions can only be increased, never decreased)
sudo -u kafka /opt/kafka/bin/kafka-topics.sh \
  --alter \
  --topic test-events \
  --partitions 6 \
  --bootstrap-server localhost:9092

# Delete a topic
sudo -u kafka /opt/kafka/bin/kafka-topics.sh \
  --delete \
  --topic test-events \
  --bootstrap-server localhost:9092

# List consumer groups
sudo -u kafka /opt/kafka/bin/kafka-consumer-groups.sh \
  --list \
  --bootstrap-server localhost:9092

# Check consumer group lag (how far behind a consumer is)
sudo -u kafka /opt/kafka/bin/kafka-consumer-groups.sh \
  --describe \
  --group my-consumer-group \
  --bootstrap-server localhost:9092

Production Configuration Tips

Memory tuning:
Kafka's JVM heap is set to 1GB by default. For production, adjust in /opt/kafka/bin/kafka-server-start.sh:

export KAFKA_HEAP_OPTS="-Xmx4G -Xms4G"

Set both -Xmx and -Xms to the same value to prevent JVM heap resizing pauses.

Log retention by size, not just time:

# Retain logs for 7 days OR until they exceed 10GB per partition, whichever comes first
log.retention.hours=168
log.retention.bytes=10737418240

Auto topic creation in production:

# Disable auto-creation — require topics to be created explicitly
auto.create.topics.enable=false

Auto-created topics use default partition/replication settings, which are rarely correct for specific use cases. Disable this and create topics explicitly with the right parameters.

Replication factor for multi-broker clusters:

# With 3 brokers, use replication factor 3 and min ISR 2
# (data survives loss of any 1 broker, writes require 2 brokers to acknowledge)
default.replication.factor=3
min.insync.replicas=2

Kafka vs RabbitMQ: When to Use Which

A common question when introducing a message broker for the first time:

	Apache Kafka	RabbitMQ
Model	Distributed log (pull-based)	Message queue (push-based)
Message retention	Configurable period (days/weeks)	Until acknowledged (usually)
Throughput	Millions of messages/second	Tens to hundreds of thousands/second
Consumer pattern	Each consumer group reads the full log independently	One consumer per message (competing consumers)
Replay	Yes — consumers can re-read past events	No — once consumed, message is gone
Ordering guarantee	Per partition	Per queue
Operational complexity	Higher	Lower
Best for	Event streaming, log aggregation, CDC, high throughput	Task queues, RPC, point-to-point messaging, simpler use cases

Choose Kafka when: You need multiple consumers to independently process the same events, you need to replay history, or you need to handle very high throughput (millions of events/second).

Choose RabbitMQ when: You're building a task queue where each task should be processed exactly once by exactly one worker, the message volume is moderate, and you want simpler operations.

Common Issues and Quick Fixes

Symptom	Likely Cause	Fix
`kafka.service` fails to start	Storage not formatted, or wrong `JAVA_HOME`	Run `kafka-storage.sh format` first; verify `java -version` works as kafka user
`Connection refused` on port 9092	Kafka not running, or listener bound to wrong address	Check `systemctl status kafka`; verify `listeners` in `server.properties`
Remote clients can't connect	`advertised.listeners` set to `localhost`	Change `advertised.listeners` to the server's actual IP or hostname
`Leader not available` on topic create	Kafka still initializing after start	Wait 10-15 seconds after Kafka starts before creating topics
Consumer reads no messages	Consumer started after producer, no `--from-beginning` flag	Add `--from-beginning` flag to read historical messages
`OutOfMemoryError` in logs	JVM heap too small for the workload	Increase `-Xmx` and `-Xms` in `kafka-server-start.sh`
High consumer group lag	Consumer too slow or too few consumer instances	Scale consumer instances; check processing logic for bottlenecks

Next Steps

With Kafka running on Ubuntu 26.04, the natural next steps are:

Add monitoring — deploy kafka-exporter alongside your Prometheus + Grafana stack to track consumer group lag, message throughput, and broker health in real time.
Containerize it — for teams already running Docker Compose stacks, Kafka has an official Docker image and can be added to an existing Compose setup using the same KRaft configuration.
Explore Kafka Streams or Apache Flink — for in-stream processing (filtering, aggregating, joining streams) without writing a separate consumer application.
Scale to multi-broker — repeat the installation on additional nodes, adjust broker.id and controller.quorum.voters, and distribute your topics' partitions across the cluster for fault tolerance and horizontal throughput scaling.

How to Install VMware ESXi: Step-by-Step Bare-Metal Setup Guide

Ramansah — Fri, 03 Jul 2026 06:45:55 +0000

Originally published on bckinfo.com

How to Install VMware ESXi: Step-by-Step Bare-Metal Setup Guide

ESXi vs. VMware Workstation: Which One Do You Need
Hardware Compatibility Check
Downloading the ESXi Installer
Creating a Bootable USB Installer
BIOS/UEFI Preparation
Installing ESXi: Step by Step
Configuring the Management Network
Accessing the vSphere Host Client
Creating Your First Virtual Machine
Post-Installation Checklist
Common Issues and Quick Fixes
Closing Notes

If you've read our complete guide to VMware virtualization, you already know ESXi is the bare-metal hypervisor underneath vSphere. This guide is the hands-on counterpart — installing ESXi directly on physical server hardware, from hardware compatibility checks through booting your first virtual machine.

ESXi vs. VMware Workstation: Which One Do You Need

Before starting, it's worth confirming you actually want ESXi and not VMware Workstation. They solve different problems:

VMware Workstation is a Type-2 hypervisor — it installs on top of an existing OS (Windows, Linux, macOS via Fusion). Good for running a VM or two on a laptop or desktop you also use for everything else. If that's your case, our guide on installing VMware Workstation on CentOS Stream 10 is the right starting point instead.
ESXi is a Type-1, bare-metal hypervisor — it installs directly on the hardware with no host OS underneath it. This is the right choice for a dedicated server running multiple VMs, a home lab, or anything that needs to scale beyond "a VM running alongside my desktop."

The rest of this guide assumes you're installing on dedicated hardware that won't run anything else.

Hardware Compatibility Check

This is the step most worth not skipping. ESXi has a defined Hardware Compatibility List (HCL), and installing on unlisted hardware is the single biggest source of installation failures and post-install driver issues.

Check your exact server model and component list (NIC, storage controller) against VMware's compatibility guide.
Minimum baseline: a 64-bit x86 CPU with hardware virtualization support (Intel VT-x or AMD-V) enabled in firmware, at least two CPU cores, and 4GB of RAM (8GB+ realistic for anything beyond testing).
Server vendors (Dell, HPE, Lenovo) often publish custom ESXi ISO images bundled with the correct drivers for their hardware — these are worth using instead of the generic VMware ISO if your hardware vendor provides one, since missing storage or NIC drivers are a common post-install headache otherwise.

Downloading the ESXi Installer

Create or log in to a VMware/Broadcom account through the official downloads portal.
Locate the ESXi ISO matching the version you intend to run.
If your hardware vendor offers a customized ISO with bundled drivers, prefer that over the generic image.

A licensing note: VMware's licensing terms and the availability of any free tier have changed multiple times since Broadcom's acquisition of VMware. Check the current licensing terms on the official Broadcom site before committing to a deployment plan, rather than relying on older guides (including this one) for pricing specifics.

Creating a Bootable USB Installer

Get a USB drive with at least 8GB of capacity — it will be fully erased.
Use a tool like Rufus (Windows) or dd/Etcher (Linux/macOS) to write the ESXi ISO to the drive.
Select the ESXi ISO as the source image and the USB drive as the target, then write the image.

If you're installing via IPMI/iDRAC/iLO remote management instead of physical USB, you can mount the ISO directly as a virtual CD-ROM through the server's remote console — skipping the physical media step entirely.

BIOS/UEFI Preparation

Before booting the installer:

Enter BIOS/UEFI setup (commonly F2, Del, or F12 depending on the vendor).
Confirm hardware virtualization (Intel VT-x / AMD-V) is enabled — some servers ship with it disabled by default.
Set the USB drive (or virtual CD-ROM, if using IPMI) as the primary boot device.
If the target disk currently has RAID configured and you intend to change it, configure RAID before installation — changing it afterward means reinstalling.

Installing ESXi: Step by Step

Boot from the installer media. The ESXi installer loads and displays detected CPU and memory.
Press Enter to begin installation.
Accept the End User License Agreement (F11).
Wait for device scanning, then select the target disk for installation. This will create a datastore on that disk usable for VM storage.
Choose your keyboard layout.
Set the root password — this is the single most important credential on this host; store it in a password manager, not a sticky note.
Confirm installation with F11.
Installation typically completes in a few minutes — it's a lean, minimal install by design.
Remove the installation media when prompted, then reboot.

Configuring the Management Network

After reboot, you're at the Direct Console User Interface (DCUI) — ESXi's text-based local console:

Press F2 and log in as root.
Navigate to Configure Management Network.
Select the network adapter to use for management traffic.
Switch IPv4 configuration from DHCP to Static, and set the IP address, subnet mask, and default gateway.
Set primary and alternate DNS servers.
Press Enter to confirm, then Esc to apply and restart the management network.

This static IP is what you'll use to reach the host going forward — DHCP is fine for a quick test but impractical the moment you need to consistently reconnect to manage the host or add it to vCenter later.

Accessing the vSphere Host Client

Once the management network is configured, open a browser and navigate to https://<esxi-management-ip>/ui. This launches the vSphere Host Client — a web interface for managing this single host directly, without needing a separate vCenter Server.

Creating Your First Virtual Machine

From the Host Client:

Click Virtual Machines in the left navigation, then Create / Register VM.
Choose Create a new virtual machine.
Name the VM and select a guest OS family and version — this affects the default hardware compatibility settings ESXi chooses.
Select the datastore created during installation.
Configure CPU, memory, and disk size for the VM.
Attach an installation ISO (uploaded to the datastore beforehand, or mounted via a remote datastore/content library) as the virtual CD-ROM.
Power on the VM and proceed with the guest OS installation as you normally would.

Post-Installation Checklist

A few things worth doing immediately after a fresh install, before this host runs anything important:

Set NTP. Time drift between the hypervisor and guest VMs causes subtle, hard-to-diagnose issues — configure NTP under host settings before anything else.
Enable lockdown mode if this host will be managed exclusively through vCenter — it restricts direct root login to the host once vCenter is the primary management point.
Configure a separate VM network distinct from the management network — mixing management and VM traffic on the same vSwitch is a common security and performance oversight.
Take a baseline snapshot strategy decision early. ESXi snapshots are not backups — plan your actual backup approach for VMs running on this host now, not after something breaks.

Common Issues and Quick Fixes

Symptom	Likely Cause	Fix
Installer doesn't detect the network adapter or storage controller	Hardware not on the HCL, missing driver in generic ISO	Use the vendor's customized ISO with bundled drivers
Server won't boot from USB	Boot order not set, or USB not written correctly	Re-verify BIOS boot order; rewrite the USB with Rufus/Etcher
Can't reach the Host Client after setup	Management network still on DHCP and IP changed, or wrong subnet/gateway	Re-check via DCUI (`F2`) and confirm static IP settings
VM won't power on — "insufficient resources"	Host RAM/CPU over-allocated to existing VMs	Reduce other VMs' reservations or add host resources
Hardware virtualization not available to nested VMs	VT-x/AMD-V not enabled in BIOS	Re-enter BIOS and enable virtualization extensions

Closing Notes

Installing ESXi itself is the easy part — it's a deliberately minimal, fast install. The steps that actually determine whether the host runs well in the long run are the ones before and after: confirming hardware compatibility up front, and getting NTP, network segmentation, and a real backup plan in place before the host starts running anything you'd mind losing.

If your use case turns out to be simpler than a dedicated bare-metal host — just a VM or two alongside your existing desktop OS — our guide on installing VMware Workstation on CentOS Stream 10 covers that lighter-weight path instead.

PowerShell Automation Scripts: Practical Examples for IT Sysadmins

Ramansah — Fri, 03 Jul 2026 06:38:33 +0000

Originally published on bckinfo.com

PowerShell Automation Scripts: Practical Examples for IT Sysadmins

What "Automation" Actually Requires
Foundation: Scheduling a Script with Task Scheduler
Bulk User Creation from a CSV File
Automated Backup Script on a Schedule
Event Log Monitoring and Alerting
Service Health Check and Auto-Restart
Secure Credential Handling for Unattended Scripts
Error Handling and Logging Every Script Needs
Common Pitfalls
Where to Go Next

Knowing PowerShell cmdlets is one skill. Turning them into something that runs unattended, every day, without you babysitting it, is a different one. This guide skips the conceptual overview and goes straight to the scripts sysadmins actually run in production — scheduled tasks, bulk provisioning, backups, monitoring, and the credential-handling and error-handling patterns that make a script safe to leave running while you're not watching it.

What "Automation" Actually Requires

A script that works when you run it manually isn't automated yet. Real automation needs four things working together:

A trigger — something that starts the script without you (a schedule, an event, a webhook).
Error handling — the script needs to fail safely and tell you when it does, since no one is watching it run.
Credential handling — unattended scripts can't prompt for a password, so credentials need a secure non-interactive path.
Logging — a record of what happened, because by the time you notice something went wrong, the script already ran and finished.

Every example below builds on this same structure.

Foundation: Scheduling a Script with Task Scheduler

Almost every automation pattern in this guide eventually needs a trigger, and on Windows that's usually Task Scheduler — configured through PowerShell instead of the GUI, so the setup itself is repeatable and scriptable:

# Create-ScheduledScript.ps1
$action = New-ScheduledTaskAction -Execute "PowerShell.exe" `
    -Argument '-NoProfile -ExecutionPolicy Bypass -File "C:\Scripts\DailyMaintenance.ps1"'

$trigger = New-ScheduledTaskTrigger -Daily -At "3:00AM"

$settings = New-ScheduledTaskSettingsSet `
    -StartWhenAvailable `
    -DontStopOnIdleEnd `
    -RestartCount 3 `
    -RestartInterval (New-TimeSpan -Minutes 5)

Register-ScheduledTask -TaskName "DailyMaintenance" `
    -Action $action `
    -Trigger $trigger `
    -Settings $settings `
    -User "SYSTEM" `
    -RunLevel Highest `
    -Description "Runs daily maintenance script at 3 AM"

A few details that matter more than they look:

-NoProfile skips loading the PowerShell profile, which avoids the task silently picking up unrelated customizations from a user profile that may not even be loaded under SYSTEM.
-ExecutionPolicy Bypass applies only to this single invocation — it doesn't change the system-wide execution policy, which is the safer way to run a signed/trusted script without weakening policy everywhere else.
RestartCount / RestartInterval give the task a retry budget if it fails — useful for scripts that depend on a network resource that might be briefly unavailable at 3 AM.
Running as SYSTEM is convenient but has no network identity; if the script needs to reach another server with Windows authentication, a dedicated service account is usually the better choice — see the credential handling section below.

Bulk User Creation from a CSV File

A common onboarding task: HR drops a CSV, and dozens of Active Directory accounts need to exist by morning.

# New-UsersFromCsv.ps1
param(
    [Parameter(Mandatory)]
    [string]$CsvPath,
    [string]$LogPath = "C:\Logs\UserCreation.log"
)

Import-Module ActiveDirectory

$users = Import-Csv -Path $CsvPath
$results = foreach ($user in $users) {
    try {
        $password = ConvertTo-SecureString $user.InitialPassword -AsPlainText -Force

        New-ADUser `
            -Name "$($user.FirstName) $($user.LastName)" `
            -GivenName $user.FirstName `
            -Surname $user.LastName `
            -SamAccountName $user.Username `
            -UserPrincipalName "$($user.Username)@company.com" `
            -Path $user.OUPath `
            -AccountPassword $password `
            -Enabled $true `
            -ChangePasswordAtLogon $true `
            -ErrorAction Stop

        Add-ADGroupMember -Identity $user.GroupName -Members $user.Username -ErrorAction Stop

        [PSCustomObject]@{ Username = $user.Username; Status = "Success"; Error = "" }
    }
    catch {
        [PSCustomObject]@{ Username = $user.Username; Status = "Failed"; Error = $_.Exception.Message }
    }
}

$results | Export-Csv -Path $LogPath -NoTypeInformation -Append
$failed = $results | Where-Object Status -eq "Failed"
if ($failed) {
    Write-Warning "$($failed.Count) account(s) failed. See $LogPath for details."
}

The expected CSV columns: FirstName, LastName, Username, InitialPassword, OUPath, GroupName. Processing each row inside its own try/catch is the detail that matters most here — without it, one bad row (a duplicate username, an invalid OU path) stops the entire batch instead of just logging that one failure and continuing with the rest.

Automated Backup Script on a Schedule

A straightforward file backup with timestamped archives and automatic cleanup of old backups:

# Backup-Directory.ps1
param(
    [string]$SourcePath = "C:\ImportantData",
    [string]$BackupRoot = "D:\Backups",
    [int]$RetentionDays = 30
)

$timestamp = Get-Date -Format "yyyyMMdd-HHmmss"
$destination = Join-Path $BackupRoot "Backup-$timestamp.zip"

try {
    Compress-Archive -Path $SourcePath -DestinationPath $destination -ErrorAction Stop
    Write-Output "Backup created: $destination"

    # Remove backups older than the retention period
    Get-ChildItem -Path $BackupRoot -Filter "Backup-*.zip" |
        Where-Object { $_.LastWriteTime -lt (Get-Date).AddDays(-$RetentionDays) } |
        Remove-Item -Force

    Write-Output "Cleanup complete: removed backups older than $RetentionDays days"
}
catch {
    Write-Error "Backup failed: $($_.Exception.Message)"
    exit 1
}

This pattern — compress, timestamp, then prune anything past a retention window — is intentionally simple, but it's the same shape used for database dumps. If you're backing up MongoDB or Redis running in Docker rather than flat files, the compression step changes but the retention and scheduling logic stays identical.

Event Log Monitoring and Alerting

Checking for specific security or system events and sending a notification when they appear:

# Monitor-EventLog.ps1
param(
    [string]$LogName = "Security",
    [int]$EventID = 4625,  # Failed logon
    [int]$LookbackMinutes = 15,
    [string]$AlertEmail = "soc-team@company.com"
)

$startTime = (Get-Date).AddMinutes(-$LookbackMinutes)

$events = Get-WinEvent -FilterHashtable @{
    LogName   = $LogName
    Id        = $EventID
    StartTime = $startTime
} -ErrorAction SilentlyContinue

if ($events) {
    $count = $events.Count
    $summary = $events | Group-Object { $_.Properties[5].Value } |
        Select-Object Name, Count |
        Sort-Object Count -Descending

    $body = "Detected $count failed logon attempt(s) in the last $LookbackMinutes minutes.`n`n" +
            ($summary | Format-Table -AutoSize | Out-String)

    Send-MailMessage -To $AlertEmail -From "alerts@company.com" `
        -Subject "Security Alert: $count Failed Logons Detected" `
        -Body $body -SmtpServer "smtp.company.com"
}

Get-WinEvent with -FilterHashtable is significantly faster than the older Get-EventLog cmdlet on large logs, since the filtering happens at the provider level instead of pulling every event into memory first. This script is built to run every 15 minutes via Task Scheduler, checking only the window since its last run.

Service Health Check and Auto-Restart

For services that occasionally stop and need to be brought back up without a human noticing first:

# Watch-CriticalServices.ps1
param(
    [string[]]$ServiceNames = @("MSSQLSERVER", "W3SVC", "MyAppService"),
    [string]$LogPath = "C:\Logs\ServiceWatchdog.log"
)

foreach ($name in $ServiceNames) {
    $service = Get-Service -Name $name -ErrorAction SilentlyContinue

    if (-not $service) {
        "$(Get-Date) - WARNING - Service '$name' not found" | Out-File -Append $LogPath
        continue
    }

    if ($service.Status -ne 'Running') {
        try {
            Start-Service -Name $name -ErrorAction Stop
            "$(Get-Date) - INFO - Restarted service '$name'" | Out-File -Append $LogPath
        }
        catch {
            "$(Get-Date) - ERROR - Failed to restart '$name': $($_.Exception.Message)" | Out-File -Append $LogPath
        }
    }
}

Run this every five minutes for a lightweight watchdog. For anything beyond a handful of services, dedicated monitoring tools are a better fit — this script is meant for small environments where standing up a full monitoring stack isn't justified yet.

Secure Credential Handling for Unattended Scripts

Hardcoding a password in a .ps1 file is the single most common security mistake in automation scripts — it sits in plain text, gets picked up by version control if you're not careful, and is readable by anyone with file access. PowerShell's SecretManagement module solves this properly:

# One-time setup
Install-Module Microsoft.PowerShell.SecretManagement -Scope CurrentUser
Install-Module Microsoft.PowerShell.SecretStore -Scope CurrentUser
Register-SecretVault -Name LocalVault -ModuleName Microsoft.PowerShell.SecretStore -DefaultVault

# Store a credential once
Set-Secret -Name "ServiceAccountPassword" -Secret (Read-Host -AsSecureString "Enter password")

Then, inside any automation script, retrieve it without ever typing the password again:

$securePassword = Get-Secret -Name "ServiceAccountPassword"
$credential = New-Object System.Management.Automation.PSCredential("DOMAIN\svc-automation", $securePassword)

Invoke-Command -ComputerName "Server01" -Credential $credential -ScriptBlock {
    Get-Service -Name "MyAppService"
}

For environments already using a centralized secrets manager — Azure Key Vault is the common case — SecretManagement also supports vault extensions that pull from there instead of a local store, which is the better option once more than one machine needs access to the same credential.

Error Handling and Logging Every Script Needs

Every script above already uses this pattern, but it's worth calling out on its own: try/catch with -ErrorAction Stop, not the default error behavior.

try {
    Get-Service -Name "NonexistentService" -ErrorAction Stop
}
catch {
    Write-Output "Caught error: $($_.Exception.Message)"
    # Log it, send an alert, or exit with a non-zero code so a calling process knows it failed
    exit 1
}

By default, most cmdlet errors are non-terminating — PowerShell logs the error and keeps going to the next line, which means a catch block never triggers unless you explicitly set -ErrorAction Stop on the command that might fail. This is the most common reason a script's error handling silently does nothing: the try/catch is there, but the error inside it was never terminating in the first place.

Common Pitfalls

Pitfall	Why It Bites	Fix
Script works manually, fails as scheduled task	Runs as a different user/profile, no console for prompts	Use `-NoProfile`, avoid `Read-Host`, test by running as the task's actual account
`try/catch` never catches anything	Cmdlet error was non-terminating	Add `-ErrorAction Stop` to the command inside the `try` block
Hardcoded password in script file	Plain-text credential exposure	Use `SecretManagement`/`SecretStore` or a vault
Task runs but script silently does nothing	Execution policy blocking the script	Use `-ExecutionPolicy Bypass` on the scheduled action, not a system-wide policy change
Script breaks after a Windows update	Module path or cmdlet behavior changed between PowerShell versions	Pin module versions, test on PowerShell 7+ before relying on Core-only behavior

Where to Go Next

These scripts assume familiarity with PowerShell's basics — cmdlets, the object pipeline, and execution policy. If any of that needs a refresher first, our PowerShell: A Complete Guide to Automation and Task Management for IT Professionals guide covers the foundational concepts this article builds directly on top of.

From here, the same trigger-plus-error-handling-plus-credential pattern extends to almost anything you need to automate: certificate renewal checks, disk space alerts, bulk software deployment, or cloud resource provisioning with Azure PowerShell. Start with whichever task currently eats the most manual time, automate that one first, and the pattern gets easier to reuse on the next one.

MongoDB with Docker Compose: Authentication, Replica Set, and Production-Ready Setup

Ramansah — Fri, 03 Jul 2026 06:32:27 +0000

Originally published on bckinfo.com

MongoDB with Docker Compose: Authentication, Replica Set, and Production-Ready Setup

Why a Standalone Container Isn't Enough
Basic Setup: Single Container with Persistence
Enabling Authentication
Single-Node Replica Set for Transactions
Keyfile Authentication for Replica Sets
Full 3-Node Replica Set for High Availability
Health Checks and Auto-Initialization
Connection String Patterns
Backup Considerations
Common Issues and Quick Fixes
Closing Notes

MongoDB is straightforward to start in Docker — pull the image, mount a volume, and you have a working database in under a minute. The complexity shows up the moment your application needs something a single standalone instance can't provide: multi-document transactions, change streams, or basic high availability. All three require a replica set, and setting one up correctly inside Docker Compose has a few sharp edges worth knowing before you hit them in production.

Why a Standalone Container Isn't Enough

A default MongoDB container — just image: mongo with no extra flags — runs as a standalone instance. That's fine for simple CRUD prototyping, but it silently blocks two things many applications eventually need:

Multi-document transactions — MongoDB only supports ACID transactions across a replica set, even a single-node one.
Change streams — real-time data pipelines, cache invalidation, or event-driven architectures built on watch() require replication to function at all.

If you've ever seen an ORM or driver (Prisma is a common example) throw an error demanding a replica set, this is why. The fix isn't complicated, but it does mean a standalone container is rarely the right default for anything beyond a quick prototype.

Basic Setup: Single Container with Persistence

Start with the foundation — a container that keeps its data after a restart:

version: '3.8'
services:
  mongo:
    image: mongo:7.0
    container_name: mongodb
    restart: always
    ports:
      - "27017:27017"
    volumes:
      - mongo-data:/data/db

volumes:
  mongo-data:
    driver: local

The official MongoDB image stores its database files at /data/db. As with Redis, a named volume is preferable to a bind mount here — Docker manages its lifecycle, and it stays portable across hosts.

Enabling Authentication

Authentication is not enabled by default on the official MongoDB image. The first time you run it, set a root user through environment variables:

services:
  mongo:
    image: mongo:7.0
    container_name: mongodb
    restart: always
    environment:
      MONGO_INITDB_ROOT_USERNAME: root
      MONGO_INITDB_ROOT_PASSWORD: ${MONGO_ROOT_PASSWORD}
    ports:
      - "27017:27017"
    volumes:
      - mongo-data:/data/db

volumes:
  mongo-data:
    driver: local

These variables only take effect on first initialization of an empty data directory — changing them later won't update an existing user. Keep the password in a .env file excluded from version control rather than hardcoded in the compose file.

Connect to verify:

mongosh --port 27017 --username root --password --authenticationDatabase admin

Single-Node Replica Set for Transactions

The fastest way to unlock transactions and change streams in development is converting your single container into a one-member replica set — no extra containers required:

services:
  mongo:
    image: mongo:7.0
    container_name: mongodb
    restart: always
    command: ["--replSet", "rs0", "--bind_ip_all"]
    environment:
      MONGO_INITDB_ROOT_USERNAME: root
      MONGO_INITDB_ROOT_PASSWORD: ${MONGO_ROOT_PASSWORD}
    ports:
      - "27017:27017"
    volumes:
      - mongo-data:/data/db
    healthcheck:
      test: >
        echo "try { rs.status() } catch (err) { rs.initiate({_id:'rs0',members:[{_id:0,host:'127.0.0.1:27017'}]}) }"
        | mongosh --port 27017 -u root -p ${MONGO_ROOT_PASSWORD} --authenticationDatabase admin --quiet
      interval: 5s
      timeout: 15s
      start_period: 15s
      retries: 10

volumes:
  mongo-data:
    driver: local

The health check does double duty here: it both verifies MongoDB is responsive and initializes the replica set automatically on first boot if it hasn't been configured yet. This is a convenient pattern for local development and staging — for production, see the dedicated initialization service further down.

Keyfile Authentication for Replica Sets

Here's the edge that catches people off guard: once you combine authentication with replication, MongoDB requires internal cluster members to authenticate with each other using a shared keyfile — separate from the root user's password. Without it, replica set members can't establish trust between themselves.

Generate a keyfile and mount it:

openssl rand -base64 756 > mongo-keyfile
chmod 400 mongo-keyfile

services:
  mongo:
    image: mongo:7.0
    command: ["--replSet", "rs0", "--keyFile", "/etc/mongo-keyfile", "--bind_ip_all"]
    volumes:
      - mongo-data:/data/db
      - ./mongo-keyfile:/etc/mongo-keyfile:ro
    environment:
      MONGO_INITDB_ROOT_USERNAME: root
      MONGO_INITDB_ROOT_PASSWORD: ${MONGO_ROOT_PASSWORD}

A common permissions trap: the official image runs as the mongodb user internally, and a keyfile mounted with the wrong ownership or overly permissive mode (anything looser than 400) will be silently rejected at startup with a generic authentication error. If replication fails to establish after adding a keyfile, check permissions first.

Full 3-Node Replica Set for High Availability

For an environment that actually needs failover — not just transaction support — run three MongoDB instances on a shared network:

version: '3.8'
services:
  mongo1:
    image: mongo:7.0
    container_name: mongo1
    hostname: mongo1
    command: ["mongod", "--replSet", "rs0", "--keyFile", "/etc/mongo-keyfile", "--bind_ip_all"]
    volumes:
      - mongo1-data:/data/db
      - ./mongo-keyfile:/etc/mongo-keyfile:ro
    networks:
      - mongo-net
    healthcheck:
      test: ["CMD", "mongosh", "--eval", "db.adminCommand('ping').ok", "--quiet"]
      interval: 10s
      timeout: 5s
      retries: 5

  mongo2:
    image: mongo:7.0
    container_name: mongo2
    hostname: mongo2
    command: ["mongod", "--replSet", "rs0", "--keyFile", "/etc/mongo-keyfile", "--bind_ip_all"]
    volumes:
      - mongo2-data:/data/db
      - ./mongo-keyfile:/etc/mongo-keyfile:ro
    networks:
      - mongo-net

  mongo3:
    image: mongo:7.0
    container_name: mongo3
    hostname: mongo3
    command: ["mongod", "--replSet", "rs0", "--keyFile", "/etc/mongo-keyfile", "--bind_ip_all"]
    volumes:
      - mongo3-data:/data/db
      - ./mongo-keyfile:/etc/mongo-keyfile:ro
    networks:
      - mongo-net

  mongo-init:
    image: mongo:7.0
    depends_on:
      mongo1:
        condition: service_healthy
    networks:
      - mongo-net
    entrypoint: >
      bash -c "
        mongosh --host mongo1 -u root -p ${MONGO_ROOT_PASSWORD} --authenticationDatabase admin --eval '
        rs.initiate({
          _id: \"rs0\",
          members: [
            { _id: 0, host: \"mongo1:27017\" },
            { _id: 1, host: \"mongo2:27017\" },
            { _id: 2, host: \"mongo3:27017\" }
          ]
        })'
      "

volumes:
  mongo1-data:
  mongo2-data:
  mongo3-data:

networks:
  mongo-net:
    driver: bridge

Each node uses its service name as hostname within the mongo-net network — mongo1, mongo2, mongo3 — and the replica set configuration references those names directly rather than IP addresses, which would break the moment containers restart and get reassigned. This is the same hostname-based service discovery pattern used in any multi-container Compose stack, and it's worth pairing with the network segmentation approach in our Docker Container Security Best Practices guide if this stack is internet-facing in any way.

Health Checks and Auto-Initialization

A dedicated mongo-init service — as shown above — is the cleanest way to handle replica set initialization in a repeatable, automated way, rather than running rs.initiate() manually after every fresh deployment. The pattern:

mongo-init waits for mongo1 to report healthy via depends_on.condition: service_healthy.
It runs rs.initiate() exactly once, targeting all three members.
The container exits after running — it's a one-shot initialization job, not a long-running service.

If you re-run docker compose up on an already-initialized cluster, rs.initiate() will simply fail harmlessly since the replica set already exists — safe to leave in place for idempotent deployments.

Connection String Patterns

Once authentication and replication are both in place, your application's connection string needs two extra parameters compared to a standalone setup:

mongodb://appuser:apppassword@mongo1:27017,mongo2:27017,mongo3:27017/myapp?replicaSet=rs0&authSource=admin

Multiple hosts in the connection string let the driver discover the current primary even after a failover.
replicaSet=rs0 must match the _id used in rs.initiate().
authSource=admin tells the driver which database the user's credentials are stored in — almost always admin for a root-style user, even when connecting to a different application database.

For local development against the single-node setup, the equivalent is simpler:

mongodb://root:password@localhost:27017/?replicaSet=rs0&authSource=admin

Backup Considerations

A replica set doesn't replace backups — replication protects against node failure, not against an accidental deleteMany() or application bug that corrupts data across all members simultaneously. The backup approach is the same regardless of whether you're running standalone or a replica set: mongodump against a secondary node to avoid load on the primary, archived on a schedule outside the container. Our existing MongoDB Backup guide covers the full scripted approach, including restore testing — worth pairing with this setup once your replica set is running.

If your stack also runs Redis alongside MongoDB, the volume-and-backup discipline described in our Redis with Docker Compose guide follows the same underlying pattern — named volumes, scheduled snapshots, and a tested restore procedure for each.

Common Issues and Quick Fixes

Symptom	Likely Cause	Fix
Driver throws "Transaction numbers are only allowed on a replica set"	Standalone instance, no replication configured	Convert to a (single-node or multi-node) replica set
Replica set members can't authenticate with each other	Missing or misconfigured keyfile	Generate a keyfile, mount with `chmod 400`, verify ownership
`rs.initiate()` hangs or times out	Members can't resolve each other's hostnames	Confirm all nodes share the same Docker network and use service names, not `localhost`
App can't find primary after failover	Connection string only lists one host	List all replica set members in the connection string
`MONGO_INITDB_ROOT_USERNAME` has no effect	Variables only apply on first init of an empty volume	Remove the volume to reset, or create the user manually via `mongosh`

Closing Notes

Running MongoDB in Docker Compose is simple for a prototype and only slightly more involved once production requirements show up: authentication from the start, a replica set the moment transactions or change streams enter the picture, a keyfile the moment authentication and replication combine, and health checks that make initialization repeatable instead of a manual one-off step.

From here, pair this setup with our MongoDB Backup guide for the operational side, and our Redis with Docker Compose guide if your stack runs both databases side by side — the persistence and security patterns carry over directly between the two.

Redis with Docker Compose: Persistence, Security, and Production-Ready Configuration

Ramansah — Mon, 29 Jun 2026 06:19:52 +0000

Originally published on bckinfo.com

Redis with Docker Compose: Persistence, Security, and Production-Ready Configuration

Why Redis Containers Lose Data
RDB vs AOF: Choosing a Persistence Strategy
Basic Setup with Docker Compose
Full Persistence Configuration
Securing Redis in Docker
Setting Resource Limits
Redis in a Multi-Service Stack
Backing Up and Restoring a Redis Volume
Common Issues and Quick Fixes
Closing Notes

Redis is one of the most common services to run in Docker — it's fast to spin up, lightweight, and perfect for caching, session storage, and queues. But that same simplicity hides a trap: by default, Redis in Docker stores everything in memory, and the moment a container is removed, all of that data disappears.

This guide walks through setting up Redis with Docker Compose the right way — covering persistence, authentication, resource limits, and the health checks you need before putting it anywhere near production.

Why Redis Containers Lose Data

A Docker container is meant to be disposable. That's a feature for stateless services, but it's a liability for a database like Redis. If you start a plain Redis container without a mounted volume, here's what happens:

The container writes its dataset only inside its own writable layer.
docker compose down or docker rm removes that layer entirely.
The next time the container starts, Redis initializes with an empty dataset.

This single oversight accounts for a large share of "we lost our session data" incidents in small teams running Redis in containers for the first time. The fix is straightforward once you understand the two persistence mechanisms Redis offers.

RDB vs AOF: Choosing a Persistence Strategy

Redis supports two persistence models, and production setups typically combine both:

RDB (Redis Database snapshots)
Point-in-time snapshots of the dataset, saved at intervals you define. Fast to restore, but you can lose any writes that happened after the last snapshot.

AOF (Append Only File)
Every write operation is logged to disk as it happens. Slower to restore on a large dataset, but far safer — with appendfsync everysec, you lose at most one second of writes.

For most production workloads, enable both: AOF for durability, RDB for fast snapshot-based backups.

Basic Setup with Docker Compose

Start with a minimal but persistent configuration:

version: '3.8'
services:
  redis:
    image: redis:7-alpine
    container_name: redis
    restart: unless-stopped
    ports:
      - "6379:6379"
    volumes:
      - redis-data:/data
    command: redis-server --appendonly yes

volumes:
  redis-data:
    driver: local

The key line here is volumes: - redis-data:/data. The official Redis image is already configured to write its dataset to /data, so mounting a named volume there is enough to survive a container removal. Named volumes are preferred over bind mounts for this purpose — they're portable across hosts and Docker manages their lifecycle for you.

Full Persistence Configuration

For more control, mount a custom redis.conf instead of relying on command-line flags:

# redis.conf

# Network
bind 0.0.0.0
port 6379
protected-mode yes

# Security
requirepass your_secure_password_here

# Memory management
maxmemory 512mb
maxmemory-policy allkeys-lru

# AOF persistence
appendonly yes
appendfsync everysec
auto-aof-rewrite-percentage 100
auto-aof-rewrite-min-size 64mb

# RDB persistence
save 900 1
save 300 10
save 60 10000
stop-writes-on-bgsave-error yes
rdbcompression yes
rdbchecksum yes

# Logging
loglevel notice

Mount it in Docker Compose:

services:
  redis:
    image: redis:7-alpine
    container_name: redis
    restart: unless-stopped
    ports:
      - "6379:6379"
    volumes:
      - redis-data:/data
      - ./redis.conf:/usr/local/etc/redis/redis.conf:ro
    command: redis-server /usr/local/etc/redis/redis.conf

volumes:
  redis-data:
    driver: local

Mounting the config file as read-only (:ro) prevents Redis itself from accidentally modifying it, and keeps your configuration under version control instead of buried in command-line arguments.

Securing Redis in Docker

Redis has no authentication enabled by default, and it has been a recurring target for cryptomining botnets that scan the internet for exposed instances on port 6379. A few non-negotiable steps:

Always set requirepass. Never run Redis with ALLOW_EMPTY_PASSWORD outside of a local, throwaway development environment.
Never expose port 6379 directly to the internet. If remote access is genuinely needed, put it behind a VPN or SSH tunnel — not a public port mapping.
Disable dangerous commands in production, such as FLUSHALL, FLUSHDB, and CONFIG, using rename-command in redis.conf.
Run as a non-root user. The official and Bitnami images already drop privileges by default — don't override this with SKIP_DROP_PRIVS unless you have a specific reason to.

This is the same defense-in-depth mindset covered in our Docker Container Security Best Practices guide — Redis just adds a database-specific layer to those general container hardening principles.

Setting Resource Limits

An unbounded Redis instance can consume all available host memory under heavy load, taking down other services on the same machine. Set explicit limits:

services:
  redis:
    image: redis:7-alpine
    deploy:
      resources:
        limits:
          memory: 1G
        reservations:
          memory: 256M
    command: redis-server --maxmemory 800mb --maxmemory-policy allkeys-lru

Note the two limits work at different layers: deploy.resources.limits.memory is enforced by Docker itself (the container gets OOM-killed if it crosses this), while maxmemory inside Redis tells Redis to start evicting keys before that happens — using allkeys-lru as a sensible general-purpose eviction policy for caching workloads.

Redis in a Multi-Service Stack

When Redis is shared by an API and a background worker, startup order matters. A worker that connects before Redis is ready will crash-loop. Use a health check with condition: service_healthy:

version: '3.8'
services:
  redis:
    image: redis:7-alpine
    restart: unless-stopped
    command: redis-server --appendonly yes --requirepass ${REDIS_PASSWORD}
    volumes:
      - redis-data:/data
    healthcheck:
      test: ["CMD", "redis-cli", "-a", "${REDIS_PASSWORD}", "ping"]
      interval: 10s
      timeout: 5s
      retries: 5
      start_period: 30s
    networks:
      - backend

  api:
    build: ./api
    depends_on:
      redis:
        condition: service_healthy
    environment:
      REDIS_URL: redis://:${REDIS_PASSWORD}@redis:6379/0
    networks:
      - backend

  worker:
    build: ./worker
    depends_on:
      redis:
        condition: service_healthy
    environment:
      REDIS_URL: redis://:${REDIS_PASSWORD}@redis:6379/1
    networks:
      - backend

volumes:
  redis-data:
    driver: local

networks:
  backend:
    driver: bridge

Two details worth highlighting:

Logical databases. Redis supports 16 logical databases (0–15) on a single instance. Assigning a different database index per service (/0 for the API, /1 for the worker) avoids key collisions without running multiple Redis containers — useful in smaller setups where running separate instances would be overkill.
Custom bridge network. Putting Redis on its own backend network — separate from any network that's exposed to the outside — limits which containers can even attempt to reach it. This pairs directly with the network segmentation principles in our Docker Network Security guide.

Backing Up and Restoring a Redis Volume

Even with persistence enabled, you still want backups independent of the running container — protecting against accidental FLUSHALL, corrupted volumes, or host failure.

Backup script:

#!/bin/bash
# backup-redis-volume.sh
VOLUME_NAME="redis_redis-data"
BACKUP_DIR="/backups/redis"
DATE=$(date +%Y%m%d-%H%M%S)
mkdir -p "$BACKUP_DIR"

# Trigger a snapshot before backing up
docker exec redis redis-cli BGSAVE
sleep 5

docker run --rm \
  -v "$VOLUME_NAME:/data:ro" \
  -v "$BACKUP_DIR:/backup" \
  alpine tar czf "/backup/redis-data-$DATE.tar.gz" -C /data .

echo "Backup created: $BACKUP_DIR/redis-data-$DATE.tar.gz"

Restore script:

#!/bin/bash
# restore-redis-volume.sh
BACKUP_FILE="$1"
VOLUME_NAME="redis_redis-data"

docker compose stop redis

docker run --rm \
  -v "$VOLUME_NAME:/data" \
  -v "$(dirname "$BACKUP_FILE"):/backup:ro" \
  alpine tar xzf "/backup/$(basename "$BACKUP_FILE")" -C /data

docker compose start redis
echo "Restore complete from $BACKUP_FILE"

The same backup-then-archive pattern is used in our MongoDB Backup article — if you're running both databases in the same stack, you can adapt one scheduled cron job to cover both with minimal changes.

Common Issues and Quick Fixes

Symptom	Likely Cause	Fix
Data gone after `docker compose down`	No volume mounted	Add a named volume at `/data`
`WRONGPASS` errors from app	Password mismatch between `redis.conf` and app env var	Verify `REDIS_PASSWORD` matches in both places
Redis container OOM-killed	No memory limit / no eviction policy	Set `deploy.resources.limits.memory` and `maxmemory-policy`
Worker crash-loops on startup	Worker starts before Redis is ready	Add `healthcheck` + `condition: service_healthy`
High latency under load	Synchronous AOF fsync	Use `appendfsync everysec` instead of `always`

Closing Notes

Running Redis in Docker Compose is simple to get started with — and just as simple to get wrong if persistence and security are treated as an afterthought. The pattern that holds up well in production is consistent: named volumes for /data, AOF+RDB persistence enabled together, requirepass always set, explicit memory limits, and a health check gating any service that depends on Redis being ready.

From here, if your stack also includes MongoDB or another database alongside Redis, the same volume-and-backup discipline applies — see our MongoDB Backup guide for the database-specific details, and our Docker Network Security guide for locking down how these containers talk to each other.

DEV Community: Ramansah

How to Set Up Prometheus and Grafana Monitoring Stack with Docker Compose

Table of Contents

What You're Building

How Prometheus and Grafana Work Together

Prerequisites

Project Structure

Step 1: Prometheus Configuration

Step 2: Alert Rules

Step 3: Alertmanager Configuration

Step 4: Grafana Auto-Provisioning

Step 5: The Full Docker Compose Stack

Step 6: Start the Stack and Verify

Step 7: Importing Your First Dashboard

Step 8: Writing Your First PromQL Query

Monitoring Docker Containers with cAdvisor

Production Hardening Checklist

Common Issues and Quick Fixes

Next Steps

How to Install Apache Tomcat 11 on Ubuntu 26.04 LTS (Step-by-Step Guide)

Table of Contents

What is Apache Tomcat and Who Needs It

Tomcat 11 vs Previous Versions: What Changed

Prerequisites

Step 1: Install Java 21

Step 2: Create a Dedicated Tomcat User

Step 3: Download and Install Tomcat 11

Step 4: Configure Environment Variables

Step 5: Create a systemd Service

Step 6: Configure UFW Firewall

Step 7: Configure Tomcat Users and Roles

Step 8: Enable Remote Access to Manager GUI

Step 9: Deploy a WAR Application

Step 10: Put Tomcat Behind Nginx Reverse Proxy

Security Hardening Checklist

Tomcat vs Jetty vs Undertow: Which to Choose

Common Issues and Quick Fixes

Next Steps

How to Install Apache Kafka on Ubuntu 26.04 LTS (KRaft Mode, No ZooKeeper)

Table of Contents

What is Apache Kafka and When Should You Use It

KRaft Mode: Why ZooKeeper is No Longer Needed

Prerequisites

Step 1: Install Java 21

Step 2: Create a Dedicated Kafka User

Step 3: Download and Install Apache Kafka 4.2

Step 4: Configure Kafka in KRaft Mode

Step 5: Format the Storage Directory

Step 6: Create a systemd Service

Step 7: Configure UFW Firewall

Step 8: Test with Producer and Consumer

Step 9: Basic Topic Management

Production Configuration Tips

Kafka vs RabbitMQ: When to Use Which

Common Issues and Quick Fixes

Next Steps

How to Install Apache Kafka on Ubuntu 26.04 LTS (KRaft Mode, No ZooKeeper)

Table of Contents

What is Apache Kafka and When Should You Use It

KRaft Mode: Why ZooKeeper is No Longer Needed

Prerequisites

Step 1: Install Java 21

Step 2: Create a Dedicated Kafka User

Step 3: Download and Install Apache Kafka 4.2

Step 4: Configure Kafka in KRaft Mode

Step 5: Format the Storage Directory

Step 6: Create a systemd Service

Step 7: Configure UFW Firewall

Step 8: Test with Producer and Consumer

Step 9: Basic Topic Management

Production Configuration Tips

Kafka vs RabbitMQ: When to Use Which

Common Issues and Quick Fixes

Next Steps

How to Install VMware ESXi: Step-by-Step Bare-Metal Setup Guide

How to Install VMware ESXi: Step-by-Step Bare-Metal Setup Guide

Table of Contents

ESXi vs. VMware Workstation: Which One Do You Need

Hardware Compatibility Check

Downloading the ESXi Installer