DEV Community: Raizan

How to Stream Container Logs to Loki

Raizan — Sun, 06 Sep 2026 08:01:29 +0000

What You'll Need

Hetzner VPS or Contabo VPS running Ubuntu 22.04 or 24.04
DigitalOcean droplets as an alternative cloud infrastructure option
Namecheap for domain management and reverse proxy routing
n8n Cloud or self-hosted n8n for triggering downstream log alerts
Docker Engine version 24.0 or higher and Docker Compose V2

Architecture Overview: Docker, Loki, and Promtail
Step 1: Deploying Grafana Loki and Grafana Stack
Step 2: Configuring Promtail Docker Engine Scraping
Step 3: Streaming Docker Logs via Native Docker Logging Plugin
Step 4: Querying Docker Logs in Grafana using LogQL
Getting Started

Architecture Overview: Docker, Loki, and Promtail

Managing logs across dozens of running containers quickly becomes impossible if you rely solely on docker logs. Centralized log aggregation gives you total visibility into microservice interactions, runtime errors, and performance bottlenecks.

Grafana Loki is a horizontally scalable, highly available, multi-tenant log aggregation system inspired by Prometheus. Unlike Elasticsearch, Loki does not index the text of the logs. Instead, it indexes labels for your log streams, leaving the raw log payload unindexed. This architecture significantly reduces storage costs and memory usage while keeping query speeds exceptionally fast.

There are two primary methods to stream Docker container logs to Loki:

Promtail Container Agent (Recommended): Promtail runs as a dedicated daemon container, mounts the host system Docker socket /var/run/docker.sock or container log paths at /var/lib/docker/containers/, automatically discovers running containers, extracts metadata labels, and pushes formatted logs to Loki.
Docker Loki Logging Plugin: The Docker engine itself ships container standard output (stdout) and standard error (stderr) directly to Loki over the network using a native plugin driver configured at the container or host daemon level.

Using Promtail is typically superior for production setups because it decouples log collection from the application runtime. If Loki experiences network hiccups, Promtail buffers logs locally without blocking container execution.

Before deploying containerized services on your server, make sure your host system is secure. Following our guide on Setting Up Fail2ban to Protect Linux Cloud Servers helps secure your Linux host before opening additional network sockets for log collectors.

Step 1: Deploying Grafana Loki and Grafana Stack

To build our log collection pipeline, we will start by deploying Loki, Grafana, Promtail, and an automated log generator service using Docker Compose on a fresh Hetzner VPS instance.

First, create a project directory on your host server to keep configuration files clean:

mkdir -p /opt/loki-stack
cd /opt/loki-stack

Create the Loki configuration file named loki-config.yaml:

auth_enabled: false

server:
  http_listen_port: 3100
  grpc_listen_port: 9096

common:
  path_prefix: /tmp/loki
  storage:
    filesystem:
      chunks_directory: /tmp/loki/chunks
      rules_directory: /tmp/loki/rules
  replication_factor: 1
  ring:
    kvstore:
      store: inmemory

schema_config:
  configs:
    - from: 2024-01-01
      store: tsdb
      object_store: filesystem
      schema: v13
      index:
        prefix: index_
        period: 24h

ruler:
  alertmanager_url: http://localhost:9093

limits_config:
  reject_old_samples: true
  reject_old_samples_max_age: 168h
  ingestion_rate_mb: 10
  ingestion_burst_size_mb: 20

Next, create the docker-compose.yml file to orchestrate our infrastructure stack:

version: "3.8"

networks:
  loki-net:
    driver: bridge

services:
  loki:
    image: grafana/loki:2.9.4
    container_name: loki
    ports:
      - "3100:3100"
    volumes:
      - ./loki-config.yaml:/etc/loki/loki-config.yaml
    command: -config.file=/etc/loki/loki-config.yaml
    networks:
      - loki-net
    restart: unless-stopped

  grafana:
    image: grafana/grafana:10.2.3
    container_name: grafana
    ports:
      - "3000:3000"
    environment:
      - GF_SECURITY_ADMIN_USER=admin
      - GF_SECURITY_ADMIN_PASSWORD=SuperSecretPassword123!
    volumes:
      - grafana-data:/var/lib/grafana
    networks:
      - loki-net
    restart: unless-stopped

  flog:
    image: mingrammer/flog:0.4.3
    container_name: sample-app
    command: -f json -d 1s -loop
    networks:
      - loki-net
    restart: unless-stopped

volumes:
  grafana-data:

Start the initial service stack:

docker compose up -d loki grafana flog

Verify that all three services are running cleanly using docker compose ps. Loki will begin listening on port 3100, while Grafana is accessible on port 3000.

Step 2: Configuring Promtail Docker Engine Scraping

Now that Loki and Grafana are operational, we need to add Promtail to auto-discover every running container, extract its standard output, attach metadata tags, and deliver the streams to Loki.

Create a file named promtail-config.yaml in /opt/loki-stack/:

server:
  http_listen_port: 9080
  grpc_listen_port: 0

positions:
  filename: /tmp/positions.yaml

clients:
  - url: http://loki:3100/loki/api/v1/push

scrape_configs:
  - job_name: docker
    docker_sd_configs:
      - host: unix:///var/run/docker.sock
        refresh_interval: 5s
    relabel_configs:
      - source_labels: ['__meta_docker_container_name']
        regex: '/(.*)'
        target_label: 'container'
      - source_labels: ['__meta_docker_container_log_stream']
        target_label: 'stream'
      - source_labels: ['__meta_docker_container_label_com_docker_compose_service']
        target_label: 'service'
    pipeline_stages:
      - json:
          expressions:
            log: log
            stream: stream
            time: time
      - json:
          expressions:
            host: host
            user: user
            method: method
            status: status
            bytes: bytes
          source: log
      - labels:
          method:
          status:

This configuration leverages Promtail Docker Service Discovery (docker_sd_configs) to watch the Docker UNIX socket directly. The relabel_configs stage strips leading slashes from container names and assigns useful labels like container and service.

In production microservice architectures, structuring application output is critical. For instance, when designing complex backend automation workflows, implementing structured payloads like those in Handling Structured Output with OpenAI Function Calling makes log extraction with Promtail pipeline stages far more effective. Similarly, if your services receive external standard callbacks, reviewing our guide on Implementing HMAC Signature Verification for Inbound Webhooks ensures you capture structured security failure events cleanly inside standard stdout.

Now, append the promtail service block into your existing docker-compose.yml file:

version: "3.8"

networks:
  loki-net:
    driver: bridge

services:
  loki:
    image: grafana/loki:2.9.4
    container_name: loki
    ports:
      - "3100:3100"
    volumes:
      - ./loki-config.yaml:/etc/loki/loki-config.yaml
    command: -config.file=/etc/loki/loki-config.yaml
    networks:
      - loki-net
    restart: unless-stopped

  promtail:
    image: grafana/promtail:2.9.4
    container_name: promtail
    volumes:
      - ./promtail-config.yaml:/etc/promtail/promtail-config.yaml
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - /var/lib/docker/containers:/var/lib/docker/containers:ro
    command: -config.file=/etc/promtail/promtail-config.yaml
    networks:
      - loki-net
    restart: unless-stopped

  grafana:
    image: grafana/grafana:10.2.3
    container_name: grafana
    ports:
      - "3000:3000"
    environment:
      - GF_SECURITY_ADMIN_USER=admin
      - GF_SECURITY_ADMIN_PASSWORD=SuperSecretPassword123!
    volumes:
      - grafana-data:/var/lib/grafana
    networks:
      - loki-net
    restart: unless-stopped

  flog:
    image: mingrammer/flog:0.4.3
    container_name: sample-app
    command: -f json -d 1s -loop
    networks:
      - loki-net
    restart: unless-stopped

volumes:
  grafana-data:

Launch Promtail alongside your running services:

docker compose up -d promtail

Check the logs for Promtail to ensure it has connected to the Docker API socket and established its connection with Loki:

docker compose logs -f promtail

Step 3: Streaming Docker Logs via Native Docker Logging Plugin

As an alternative to Promtail, Docker provides an official plugin that forwards container stdout directly to Loki without requiring Promtail to read log files from host disk storage.

First, install the Docker Loki logging plugin on your host machine:

docker plugin install grafana/loki-docker-driver:2.9.4 --alias loki --grant-all-permissions

Verify that the plugin is active:

docker plugin ls

You can configure the Loki driver globally across all Docker containers on the server by editing /etc/docker/daemon.json. Create or modify /etc/docker/daemon.json with the following content:

{
  "log-driver": "loki",
  "log-opts": {
    "loki-url": "http://127.0.0.1:3100/loki/api/v1/push",
    "loki-batch-size": "400000",
    "loki-retries": "2",
    "loki-max-backoff": "800ms",
    "loki-timeout": "1s",
    "keep-file": "true"
  }
}

Restart the Docker daemon to apply the global configuration:

sudo systemctl restart docker

Alternatively, if you want to use the native driver only for specific containers rather than globally, define the logging options directly inside an individual service block in docker-compose.yml:

version: "3.8"

services:
  standalone-app:
    image: alpine
    container_name: alpine-runner
    command: sh -c "while true; do echo '{\"status\": 200, \"message\": \"System check operational\"}'; sleep 2; done"
    logging:
      driver: loki
      options:
        loki-url: "http://127.0.0.1:3100/loki/api/v1/push"
        loki-external-labels: "job=standalone-runner,environment=production"

Step 4: Querying Docker Logs in Grafana using LogQL

Now that your container log streams are flowing into Loki, open Grafana by visiting http://your-server-ip:3000 in your web browser. Log in with the admin credentials defined in your Docker Compose file (admin / SuperSecretPassword123!).

Setting Up the Loki Data Source

Navigate to Connections > Data Sources in the left sidebar.
Click Add data source and select Loki.
Set the Connection URL to http://loki:3100.
Click Save & test. You should see a green confirmation badge stating that the data source is connected and receiving data.

Querying Logs with LogQL

Go to Explore in the Grafana menu and select Loki as your active data source. LogQL (Loki Query Language) uses simple selector expressions to slice and dice log streams.

To query all logs originating from our sample container:

{container="sample-app"}

To filter log messages containing specific HTTP error responses:

{container="sample-app"} |= "404"

To parse JSON-formatted log output dynamically on the fly and filter by parsed attributes:

{container="sample-app"} | json | status >= 500

To calculate the per-second rate of logs generated by container over a 5-minute window:

rate({container="sample-app"}[5m])

To aggregate log throughput grouped by service name:

sum(rate({job="docker"}[1m])) by (service)

These queries form the backbone of production dashboards, letting you view real-time log tailing, error spikes, and response time metrics side by side.

Getting Started

Centralizing container logs with Loki and Promtail ensures your infrastructure remains reliable, searchable, and easy to debug. Deploying these services on reliable host platforms like a Hetzner VPS or DigitalOcean droplet gives you total control over log retention without cloud vendor lock-in.

If you require custom webhooks or DNS routing for your endpoints, manage your domains using Namecheap and link your log query outputs to n8n Cloud to build automated Slack, Telegram, or Email alerting systems whenever Loki registers a critical error stream.

Outsource Your Automation

Don't have time? I build production n8n workflows, WhatsApp bots, and fully automated YouTube Shorts pipelines. Hire me on Fiverr, mention SYS3-DEVTO for priority. Or DM at chasebot.online.

Originally published on Automation Insider.

Building Reliable Structured Output Pipelines with OpenAI

Raizan — Sat, 05 Sep 2026 08:03:04 +0000

What You'll Need

n8n Cloud or self-hosted n8n
Hetzner VPS or Contabo VPS for hosting
Namecheap if domain needed
DigitalOcean as alternative

Why Structured Outputs Matter for Automated Pipelines
Defining Strict Schemas with Pydantic and OpenAI
Building an n8n Data Transformation and Parsing Pipeline
Error Handling Retries and Self Correction Loops
Getting Started

Why Structured Outputs Matter for Automated Pipelines

Building automated production systems with Large Language Models requires absolute predictability. In early pipeline iterations, developers relied heavily on raw system prompts containing instructions like "Respond only in JSON format." Despite those system prompts, models frequently included conversational preambles, omitted mandatory keys, or generated invalid escape characters.

OpenAI introduced strict JSON Schema enforcement directly into their API engines to resolve this issue. By supplying a concrete target schema, the API constraints the sampling process at decoding time, rendering responses that violate your requested structure mathematical impossibilities.

If you have previously built tools using tool declarations, you may recall Handling Structured Output with OpenAI Function Calling as a common strategy. While function calling remains valuable for triggering real-world actions, native structured outputs provide a dedicated path for reliable data extraction, schema mapping, and ETL workflows without abusing generic function signatures.

Defining Strict Schemas with Pydantic and OpenAI

The cleanest mechanism for configuring strict responses in Python involves combining Pydantic with the native OpenAI Python SDK. Pydantic allows you to construct strongly typed model structures, which the SDK automatically converts into strict JSON schemas.

The following script extracts complex invoice data from unstructured support emails, validates the field types, and guarantees that nested objects conform to required data formats.

import json
from typing import List, Optional
from pydantic import BaseModel, Field
from openai import OpenAI

client = OpenAI(api_key="your_openai_api_key_here")

class LineItem(BaseModel):
    description: str = Field(description="The description of the product or service purchased.")
    quantity: int = Field(description="The unit count of items ordered.")
    unit_price: float = Field(description="The price per individual unit.")
    total_amount: float = Field(description="The calculated subtotal for this item line.")

class CustomerDetails(BaseModel):
    full_name: str = Field(description="The complete name of the customer.")
    email_address: str = Field(description="The contact email address found in the text.")
    account_id: Optional[str] = Field(default=None, description="The internal account identifier if present.")

class InvoiceExtractionSchema(BaseModel):
    invoice_number: str = Field(description="The unique invoice identifier string.")
    purchase_date: str = Field(description="The ISO date string representing purchase time.")
    customer: CustomerDetails = Field(description="Nested customer profile details.")
    line_items: List[LineItem] = Field(description="List of individual items included in the purchase order.")
    currency: str = Field(description="The three-letter currency symbol like USD or EUR.")
    tax_amount: float = Field(description="Total tax assessed on the transaction.")
    grand_total: float = Field(description="Final invoice amount including tax.")

unstructured_email_text = """
Hello support,

Please process payment details for Invoice INV-90422 issued on 2024-11-12. 
Customer is Alice Smith, reach out to alice.smith@example.com (Account ID ACC-3312).

Here is the breakdown of charges:
- 2x Enterprise Server Rack Subscriptions at $1200.00 each, totaling $2400.00
- 1x Setup and Deployment Consultation Fee at $450.00

Tax comes out to $228.00 in total. Final charge calculated is $3078.00 USD.

Thanks,
Billing Team
"""

response = client.beta.chat.completions.parse(
    model="gpt-4o-mini",
    messages=[
        {
            "role": "system",
            "content": "You are a specialized document parser. Extract invoice elements into strict structural data."
        },
        {
            "role": "user",
            "content": unstructured_email_text
        }
    ],
    response_format=InvoiceExtractionSchema
)

parsed_invoice: InvoiceExtractionSchema = response.choices[0].message.parsed

print("Invoice Number:", parsed_invoice.invoice_number)
print("Customer Name:", parsed_invoice.customer.full_name)
print("Customer Email:", parsed_invoice.customer.email_address)
print("Line Item Count:", len(parsed_invoice.line_items))

for idx, item in enumerate(parsed_invoice.line_items, start=1):
    print(f" Item {idx}: {item.description} | Qty: {item.quantity} | Total: {item.total_amount}")

print("Grand Total:", parsed_invoice.grand_total, parsed_invoice.currency)

By passing response_format=InvoiceExtractionSchema into client.beta.chat.completions.parse, the OpenAI API automatically forces the completion engine to follow the exact field boundaries defined inside your Pydantic models. You receive standard Python objects populated directly from the API result.

Building an n8n Data Transformation and Parsing Pipeline

When integrating standard API calls into production orchestration engines like n8n Cloud, hosting your pipeline on reliable infrastructure like a Hetzner VPS or a Contabo VPS ensures high uptime and consistent request performance.

If you are running self hosted services, keeping your workflow execution servers operational requires active maintenance. Check our step by step guide on How to Deploy Watchtower for Docker Containers to handle seamless automated image updates for background containers.

Within n8n Cloud, you can execute raw HTTP requests to OpenAI using the native HTTP Request node, passing a explicit JSON Schema object inside the response_format attribute. Following the HTTP execution, an n8n JavaScript Code node verifies that the object properties match expected types before writing into standard database systems.

Here is the functional JavaScript validation code designed to sit inside an n8n Code node directly after an HTTP Request block:

const items = $input.all();
const validatedOutput = [];

for (let i = 0; i < items.length; i++) {
  const rawContent = items[i].json.choices[0].message.content;
  let parsedJson;

  try {
    parsedJson = JSON.parse(rawContent);
  } catch (error) {
    throw new Error(`Invalid JSON output returned from OpenAI API: ${error.message}`);
  }

  const requiredKeys = ['invoice_number', 'purchase_date', 'customer', 'line_items', 'currency', 'tax_amount', 'grand_total'];
  for (const key of requiredKeys) {
    if (!(key in parsedJson)) {
      throw new Error(`Schema Validation Error: Missing required property '${key}'.`);
    }
  }

  if (typeof parsedJson.customer !== 'object' || parsedJson.customer === null) {
    throw new Error("Schema Validation Error: 'customer' must be a valid object.");
  }

  if (!parsedJson.customer.full_name || !parsedJson.customer.email_address) {
    throw new Error("Schema Validation Error: 'customer' object missing mandatory sub-keys 'full_name' or 'email_address'.");
  }

  if (!Array.isArray(parsedJson.line_items) || parsedJson.line_items.length === 0) {
    throw new Error("Schema Validation Error: 'line_items' must be a non-empty array.");
  }

  for (let j = 0; j < parsedJson.line_items.length; j++) {
    const item = parsedJson.line_items[j];
    if (typeof item.description !== 'string' || typeof item.unit_price !== 'number' || typeof item.quantity !== 'number') {
      throw new Error(`Schema Validation Error: Invalid type structure in line item index ${j}.`);
    }
  }

  validatedOutput.push({
    json: {
      status: "SUCCESS",
      data: parsedJson,
      processed_at: new Date().toISOString()
    }
  });
}

return validatedOutput;

If validation fails inside this n8n node, the workflow raises an error. To prevent silent system failures, you can configure alerting loops within your infrastructure. Read our comprehensive tutorial on Deploying Self Hosted Telegram Bots with Docker to immediately dispatch workflow failure traces into private alert channels.

Error Handling Retries and Self Correction Loops

Even when using strict JSON schemas, logic validation edge cases can occur. For instance, a model may generate valid JSON syntax where mathematical sub-computations fail business rules, such as individual item totals failing to sum up to the specified grand total.

To establish bulletproof automation systems, build a self-correction repair loop. This pattern captures validation exceptions, attaches the raw error log into the message array, and prompts OpenAI to correct its own mistake.

Here is a resilient execution loop written in Python that enforces both structural typing and semantic business logic constraints:

import json
from typing import List
from pydantic import BaseModel, Field, ValidationError
from openai import OpenAI

client = OpenAI(api_key="your_openai_api_key_here")

class InventoryItem(BaseModel):
    product_code: str = Field(description="Product SKU identifier string.")
    quantity: int = Field(description="Quantity extracted from stock update.")
    unit_cost: float = Field(description="Cost value per individual unit.")
    total_cost: float = Field(description="Total computed cost matching quantity multiplied by unit cost.")

class InventoryBatchUpdate(BaseModel):
    batch_id: str = Field(description="Unique batch identifier.")
    items: List[InventoryItem] = Field(description="List of updated inventory records.")

def validate_business_rules(payload: InventoryBatchUpdate):
    for item in payload.items:
        expected_total = round(item.quantity * item.unit_cost, 2)
        actual_total = round(item.total_cost, 2)
        if expected_total != actual_total:
            raise ValueError(
                f"Math error in item '{item.product_code}': "
                f"quantity ({item.quantity}) * unit_cost ({item.unit_cost}) = {expected_total}, "
                f"but extracted total_cost was {actual_total}."
            )

def extract_inventory_data_with_retry(unstructured_input: str, max_retries: int = 3) -> InventoryBatchUpdate:
    messages = [
        {
            "role": "system",
            "content": "You extract structured inventory logs into valid JSON format. Ensure all multiplication math is strictly accurate."
        },
        {
            "role": "user",
            "content": unstructured_input
        }
    ]

    for attempt in range(1, max_retries + 1):
        print(f"Processing attempt {attempt} of {max_retries}...")

        completion = client.beta.chat.completions.parse(
            model="gpt-4o-mini",
            messages=messages,
            response_format=InventoryBatchUpdate
        )

        raw_message = completion.choices[0].message

        if raw_message.refusal:
            raise RuntimeError(f"Model refused request: {raw_message.refusal}")

        parsed_data = raw_message.parsed

        try:
            validate_business_rules(parsed_data)
            print("Validation passed successfully!")
            return parsed_data

        except ValueError as error:
            print(f"Business logic check failed on attempt {attempt}: {error}")

            messages.append({
                "role": "assistant",
                "content": raw_message.content
            })
            messages.append({
                "role": "user",
                "content": f"Your previous JSON response contained a logical validation error: {error}. Please re-calculate and fix the numbers."
            })

    raise RuntimeError("Failed to extract valid and logical structured data after maximum retry attempts.")

raw_input_log = """
Batch Processing Memo: Log ID BATCH-7721
Received inventory:
- SKU-101: 5 units at $12.50 per unit. Calculated subtotal is $62.50.
- SKU-204: 10 units at $5.00 per unit. Calculated subtotal is $50.00.
"""

final_data = extract_inventory_data_with_retry(raw_input_log)
print("Final Validated Structure:")
print(json.dumps(final_data.model_dump(), indent=2))

This self-correction mechanism creates an automated feedback framework. By capturing specific runtime errors and feeding them back to the completion engine, your pipeline self-corrects data errors before writing invalid payload objects into down-stream databases.

Getting Started

To implement reliable OpenAI integration pipelines within your own software infrastructure:

Provision server hosting through Hetzner VPS, Contabo VPS, or DigitalOcean.
Register custom API endpoints using custom domains registered with Namecheap.
Connect processing nodes together using n8n Cloud or self-hosted workflows.

Outsource Your Automation

Don't have time? I build production n8n workflows, WhatsApp bots, and fully automated YouTube Shorts pipelines. Hire me on Fiverr, mention SYS3-DEVTO for priority. Or DM at chasebot.online.

Originally published on Automation Insider.

Handling Telegram Bot Rate Limits with Redis

Raizan — Fri, 04 Sep 2026 08:06:20 +0000

What You'll Need

Hetzner VPS or Contabo VPS for hosting
DigitalOcean as an alternative cloud provider
n8n Cloud or self-hosted n8n instance
Redis server version 6.0 or higher
Node.js runtime version 18 or higher
A valid Telegram Bot API token from BotFather

Understanding Telegram Bot Rate Limits
Architecture: Sliding Window and Rate Limiting with Redis
Setting Up Docker and Redis Infrastructure
Implementing the Redis Rate Limiter in Node.js
Building a High-Throughput Queue with BullMQ
Getting Started

Understanding Telegram Bot Rate Limits

When building high volume Telegram bots, hitting API rate limits is an inevitable challenge. Telegram enforces precise thresholds to protect its infrastructure from spam and server overload. If your bot sends messages too quickly, Telegram responds with an HTTP 429 status code containing a retry_after parameter, which tells your application how many seconds it must sleep before issuing another request.

Telegram applies rate limits across multiple levels:

Per Chat Limit: You cannot send more than 1 message per second to a single individual chat.
Global Limit: A bot cannot send more than 30 messages per second across all users and chats combined.
Group Chat Limit: Bots are restricted to roughly 20 messages per minute inside a single group or channel.

If you violate these boundaries repeatedly, Telegram extends your cooldown period, and sustained violations can lead to temporary or permanent bot bans. Handling these limits synchronously inside your application code usually leads to blocked threads, memory leaks, or lost messages.

To prevent these failure points, you need a centralized rate limiter that controls execution throughput before making HTTP requests. Capturing structured events during high velocity message distribution is crucial, so consider Configuring Vector for Centralized Log Aggregation to stream HTTP response codes and rate limit spikes to a centralized dashboard.

Architecture: Sliding Window and Rate Limiting with Redis

A standard fixed-window rate limiter resets its counters at arbitrary time boundaries (such as every minute at 00 seconds). This approach creates execution bursts at the window edges, allowing up to double the allowed request volume in a narrow time frame.

To guarantee compliance with Telegram rules, we implement a Sliding Window Log algorithm backed by Redis sorted sets (ZSET). Every outbound message request generates a unique member in a sorted set where the score is a microsecond timestamp.

The process follows four atomic steps:

Remove all entries in the sorted set older than the current timestamp minus the sliding window duration.
Count the remaining elements in the set.
If the count is below the permitted threshold, insert the new message timestamp into the set and proceed with dispatch.
If the set is full, calculate the delay required until the oldest member expires, then reject or queue the outbound request.

Because Redis runs single threaded and processes operations sequentially, executing these commands atomically ensures race conditions are completely eliminated, even if you run multiple instances of your bot application in parallel.

Setting Up Docker and Redis Infrastructure

To build this setup on a Hetzner VPS or a DigitalOcean droplet, we deploy Redis and our Node.js bot service using Docker Compose.

Create a docker-compose.yml file in your application root directory:

version: '3.8'

services:
  redis:
    image: redis:7.2-alpine
    container_name: telegram_redis
    restart: always
    ports:
      - "6379:6379"
    volumes:
      - redis_data:/data
    command: redis-server --appendonly yes --requirepass SuperSecretRedisPassword123!
    healthcheck:
      test: ["CMD", "redis-cli", "-a", "SuperSecretRedisPassword123!", "ping"]
      interval: 5s
      timeout: 3s
      retries: 5

  bot:
    build: .
    container_name: telegram_bot_worker
    restart: always
    depends_on:
      redis:
        condition: service_healthy
    environment:
      - REDIS_HOST=redis
      - REDIS_PORT=6379
      - REDIS_PASSWORD=SuperSecretRedisPassword123!
      - TELEGRAM_BOT_TOKEN=123456789:ABCdefGHIjklMNOpqrsTUVwxyZ

volumes:
  redis_data:

When deploying containerized infrastructure, ensuring seamless service restarts without manual intervention is critical. Check out our guide on How to Deploy Watchtower for Docker Containers to automatically update your bot and Redis services whenever you push updates.

Implementing the Redis Rate Limiter in Node.js

Now we will write a production ready Node.js implementation using ioredis. This implementation runs an atomic Lua script directly inside Redis to ensure zero latency race conditions across workers.

First, initialize your application dependencies in package.json:

{
  "name": "telegram-rate-limiter",
  "version": "1.0.0",
  "description": "Telegram Bot Rate Limiting with Redis",
  "main": "index.js",
  "dependencies": {
    "axios": "^1.6.8",
    "bullmq": "^5.7.8",
    "dotenv": "^16.4.5",
    "ioredis": "^5.3.2"
  }
}

Next, create the core rate limiter file RateLimiter.js. This module uses an inline Lua script to run atomic rate limit validation against Redis keys:

const Redis = require('ioredis');

class RedisRateLimiter {
  constructor(redisClient) {
    this.redis = redisClient;
    this.defineLuaScript();
  }

  defineLuaScript() {
    this.luaScript = `
      local key = KEYS[1]
      local now = tonumber(ARGV[1])
      local window = tonumber(ARGV[2])
      local limit = tonumber(ARGV[3])
      local clearBefore = now - window

      redis.call('ZREMRANGEBYSCORE', key, 0, clearBefore)
      local currentRequests = redis.call('ZCARD', key)

      if currentRequests < limit then
        redis.call('ZADD', key, now, now)
        redis.call('PEXPIRE', key, window)
        return 1
      else
        return 0
      end
    `;

    this.redis.defineCommand('checkAndRecordLimit', {
      numberOfKeys: 1,
      lua: this.luaScript
    });
  }

  async isAllowed(targetKey, limit, windowMs) {
    const now = Date.now();
    const result = await this.redis.checkAndRecordLimit(targetKey, now, windowMs, limit);
    return result === 1;
  }
}

module.exports = RedisRateLimiter;

Building a High-Throughput Queue with BullMQ

If you attempt to send a message and hit a rate limit, dropping the payload is not an option. Instead, we insert the outgoing Telegram payloads into a persistent queue backed by BullMQ.

The architectural pattern matches the strategies described in How to Build Distributed Web Scraping Pipelines, where request rates must be dynamically modified without losing tasks from memory during traffic spikes.

Create QueueManager.js to manage queued message processing, rate limits, and Telegram API dispatches:

const { Queue, Worker } = require('bullmq');
const Redis = require('ioredis');
const axios = require('axios');
const RedisRateLimiter = require('./RateLimiter');

const connection = new Redis({
  host: process.env.REDIS_HOST || 'localhost',
  port: parseInt(process.env.REDIS_PORT || '6379'),
  password: process.env.REDIS_PASSWORD || 'SuperSecretRedisPassword123!',
  maxRetriesPerRequest: null
});

const rateLimiter = new RedisRateLimiter(connection);

const telegramQueue = new Queue('TelegramMessages', { connection });

const BOT_TOKEN = process.env.TELEGRAM_BOT_TOKEN;
const TELEGRAM_API_URL = `https://api.telegram.org/bot${BOT_TOKEN}/sendMessage`;

const sendTelegramMessage = async (chatId, text) => {
  return await axios.post(TELEGRAM_API_URL, {
    chat_id: chatId,
    text: text,
    parse_mode: 'HTML'
  });
};

const worker = new Worker('TelegramMessages', async (job) => {
  const { chatId, text } = job.data;

  const globalKey = 'ratelimit:telegram:global';
  const chatKey = `ratelimit:telegram:chat:${chatId}`;

  const globalAllowed = await rateLimiter.isAllowed(globalKey, 30, 1000);
  if (!globalAllowed) {
    throw new Error('RATE_LIMIT_GLOBAL_EXCEEDED');
  }

  const chatAllowed = await rateLimiter.isAllowed(chatKey, 1, 1000);
  if (!chatAllowed) {
    throw new Error('RATE_LIMIT_CHAT_EXCEEDED');
  }

  try {
    const response = await sendTelegramMessage(chatId, text);
    return response.data;
  } catch (error) {
    if (error.response && error.response.status === 429) {
      const retryAfterSeconds = error.response.data.parameters?.retry_after || 5;
      const delayMs = retryAfterSeconds * 1000;

      console.warn(`Hit 429 from Telegram API. Delaying chat ${chatId} for ${retryAfterSeconds}s`);

      await telegramQueue.add('sendMessage', job.data, {
        delay: delayMs,
        jobId: `retry_${job.id}_${Date.now()}`
      });

      return { delayed: true, retryAfterSeconds };
    }

    throw error;
  }
}, {
  connection,
  concurrency: 5,
  settings: {
    backoffStrategies: {
      customRateLimitBackoff: (attemptsMade) => {
        return attemptsMade * 1200;
      }
    }
  }
});

worker.on('failed', (job, err) => {
  if (err.message.startsWith('RATE_LIMIT_')) {
    telegramQueue.add('sendMessage', job.data, {
      delay: 1000,
      jobId: `rate_limit_retry_${job.id}_${Date.now()}`
    });
  } else {
    console.error(`Job ${job.id} failed permanently with error: ${err.message}`);
  }
});

const enqueueMessage = async (chatId, text) => {
  await telegramQueue.add('sendMessage', { chatId, text }, {
    attempts: 3,
    backoff: {
      type: 'customRateLimitBackoff'
    },
    removeOnComplete: true,
    removeOnFail: false
  });
};

module.exports = {
  enqueueMessage,
  telegramQueue,
  worker
};

To run and verify the system, create an index.js entry point that pushes a batch of test messages into the pipeline:

require('dotenv').config();
const { enqueueMessage } = require('./QueueManager');

const runTestBroadcast = async () => {
  const targetChatId = '987654321';
  console.log('Starting high-volume message test dispatch...');

  for (let i = 1; i <= 50; i++) {
    const messageText = `Test notification #${i} sent at timestamp ${new Date().toISOString()}`;
    await enqueueMessage(targetChatId, messageText);
    console.log(`Successfully queued message payload index: ${i}`);
  }
};

runTestBroadcast().catch((error) => {
  console.error('Error encountered while running message broadcast script:', error);
});

This dual layer architecture guarantees complete safety across all worker processes. The primary guard layer is the Redis sliding window script, which stops requests locally before they trigger network overhead.

The secondary guard layer handles any raw HTTP 429 responses sent by Telegram. When Telegram issues a rate limit response, the worker captures the exact retry_after window value, calculates the delay in milliseconds, and re-enqueues the job back into BullMQ.

By combining atomic sliding window evaluation in Redis with resilient BullMQ task queues, your bot maintains maximum execution speed without ever breaching Telegram's system rate limits.

Getting Started

To launch your production ready rate limiting infrastructure:

Provision a high performance server using Hetzner VPS or Contabo VPS.
Set up DigitalOcean managed databases if you prefer external Redis hosting.
Deploy your automation nodes using n8n Cloud or self-host your application services with Docker Compose.

Outsource Your Automation

Don't have time? I build production n8n workflows, WhatsApp bots, and fully automated YouTube Shorts pipelines. Hire me on Fiverr, mention SYS3-DEVTO for priority. Or DM at chasebot.online.

Originally published on Automation Insider.

Setting Up PostgreSQL Read Replicas with Docker

Raizan — Thu, 03 Sep 2026 08:01:23 +0000

What You'll Need

Hetzner VPS or Contabo VPS for hosting Docker containers
DigitalOcean as an alternative cloud provider
Namecheap if you need a domain for database proxies
Docker Engine version 20.10.0 or higher
Docker Compose v2 or higher

Understanding PostgreSQL Streaming Replication
Configuring the Primary PostgreSQL Instance
Writing the Replica Bootstrapping Engine
Building the Production Docker Compose Stack
Validating Replication and Monitoring Delay
Getting Started

Understanding PostgreSQL Streaming Replication

Scaling a relational database engine requires splitting write operations from read-heavy analytical or transactional queries. When your application scales past a single node, running heavy reads on your primary database degrades transactional write throughput. If you run data ingestion workers or build complex reporting engines, offloading SELECT queries to secondary databases becomes mandatory.

PostgreSQL handles data redundancy using streaming replication based on Write-Ahead Logging (WAL). Every modifying transaction (INSERT, UPDATE, DELETE) writes log records to disk before making changes to data files. In a primary and replica setup, the primary database streams these WAL records across TCP connections to one or more replica nodes. The replica continuously replays these incoming WAL records, keeping its state nearly identical to the primary.

Physical streaming replication yields byte-for-byte exact replicas of your primary storage layer. Replicas run in read-only mode, meaning they reject write queries while allowing full read accessibility. When integrated with worker environments, such as those described in How to Build Distributed Web Scraping Pipelines, streaming replication prevents bulk read queries from bottlenecking real-time write queues on your core primary database.

Using Docker containers to deploy a primary replica cluster simplifies configuration management, isolate storage drivers, and ensure consistent behavior across development and production servers.

Configuring the Primary PostgreSQL Instance

To establish streaming replication in Docker, the primary database node must execute custom initialization scripts that set up a replication user, reserve replication slots, and configure core WAL parameters.

We begin by establishing a clear file structure for our database cluster:

pg-cluster/
├── docker-compose.yml
├── primary/
│   ├── custom-primary.conf
│   └── init-primary.sh
└── replica/
    └── docker-replica-entrypoint.sh

First, we define primary/custom-primary.conf. This file overrides default PostgreSQL settings to enable streaming output, set replication limits, and maintain transaction logs during heavy background processing.

listen_addresses = '*'
wal_level = replica
max_wal_senders = 10
max_replication_slots = 10
hot_standby = on
hot_standby_feedback = on
wal_keep_size = 512MB
max_standby_streaming_delay = 30s

Here is a breakdown of these essential parameters:

wal_level: Set to replica to log required structural data for read-only streaming.
max_wal_senders: Defines maximum concurrent streaming connection threads.
max_replication_slots: Specifies maximum replication slots reserved on the primary node. Replication slots prevent the primary from deleting old WAL segments until replicas consume them.
hot_standby_feedback: Sends query state back from replica to primary, preventing primary autovacuum operations from deleting tuples currently being queried on replicas.
wal_keep_size: Guarantees a minimum log volume retainage on disk even if connections drop temporarily.

Next, create primary/init-primary.sh. PostgreSQL automatically executes any script placed inside /docker-entrypoint-initdb.d/ on initial database cluster creation.

#!/bin/bash
set -e

psql -v ON_ERROR_STOP=1 --username "$POSTGRES_USER" --dbname "$POSTGRES_DB" <<-EOSQL
    CREATE USER replicator WITH REPLICATION ENCRYPTED PASSWORD 'replicator_password';
    SELECT * FROM pg_create_physical_replication_slot('replica_1_slot');
    SELECT * FROM pg_create_physical_replication_slot('replica_2_slot');
EOSQL

echo "host replication replicator 0.0.0.0/0 scram-sha-256" >> "$PGDATA/pg_hba.conf"

This script generates a dedicated user named replicator holding explicit replication permissions. It then creates physical replication slots named replica_1_slot and replica_2_slot. Finally, it appends an authorization rule to pg_hba.conf allowing the replicator account access from any internal container network using scram-sha-256 password authentication.

Writing the Replica Bootstrapping Engine

Read replicas must initialize from a clean physical copy of the primary database system files. We use the standard PostgreSQL utility pg_basebackup to clone the primary storage directory across the network.

When deploying through Docker, official PostgreSQL images auto-initialize a brand new PostgreSQL cluster inside /var/lib/postgresql/data if the directory is empty. We must intercept container startup to wipe default files, execute pg_basebackup, create standby signal flags, and boot PostgreSQL in recovery mode.

Create replica/docker-replica-entrypoint.sh:

#!/bin/bash
set -e

if [ -s "$PGDATA/PG_VERSION" ]; then
    echo "Existing database data detected in $PGDATA. Skipping basebackup initialization."
else
    echo "No existing data found. Starting basebackup sync process..."

    until PGPASSWORD='replicator_password' psql -h postgres-primary -U replicator -d postgres -c '\q' 2>/dev/null; do
        echo "Primary database service is unreachable. Waiting 2 seconds before retrying..."
        sleep 2
    done

    echo "Primary engine detected. Wiping empty volume target..."
    rm -rf "${PGDATA:?}"/*

    echo "Executing pg_basebackup stream clone..."
    PGPASSWORD='replicator_password' pg_basebackup \
        -h postgres-primary \
        -D "$PGDATA" \
        -U replicator \
        -v \
        -P \
        -X stream \
        -S "${REPLICATION_SLOT_NAME}" \
        -R

    chmod 700 "$PGDATA"
    echo "Basebackup clone finished successfully. Launching replica server process."
fi

exec docker-entrypoint.sh "$@"

Let's dissect the critical arguments assigned to pg_basebackup:

-h postgres-primary: Connects directly to the primary container service name using Docker DNS routing.
-D "$PGDATA": Defines the local target directory where cluster files are extracted.
-U replicator: Authenticates using our explicit replication service user.
-X stream: Streams transaction log records concurrently as the backup runs to guarantee consistency.
-S "${REPLICATION_SLOT_NAME}": Binds the replica container to its dedicated physical replication slot defined on the primary node.
-R: Generates auto-configuration files (standby.signal and connection string attributes in postgresql.auto.conf) instructing PostgreSQL to boot as a read-only secondary replica.

Ensure this initialization shell script is marked executable locally:

chmod +x replica/docker-replica-entrypoint.sh
chmod +x primary/init-primary.sh

Building the Production Docker Compose Stack

With initialization scripts complete, we assemble our infrastructure stack inside docker-compose.yml. We will deploy a master node (postgres-primary) along with two physical read replicas (postgres-replica-1 and postgres-replica-2).

When deploying on host infrastructure like a Hetzner VPS or a DigitalOcean droplet, you can pin external host ports or route query traffic through local socket adapters.

Here is the entire container orchestration blueprint:

version: '3.8'

networks:
  pg-network:
    driver: bridge

volumes:
  pg_primary_data:
  pg_replica_1_data:
  pg_replica_2_data:

services:
  postgres-primary:
    image: postgres:16-alpine
    container_name: postgres-primary
    restart: always
    networks:
      - pg-network
    ports:
      - "5432:5432"
    environment:
      POSTGRES_USER: admin
      POSTGRES_PASSWORD: primary_admin_password
      POSTGRES_DB: production_db
      PGDATA: /var/lib/postgresql/data/pgdata
    volumes:
      - pg_primary_data:/var/lib/postgresql/data
      - ./primary/init-primary.sh:/docker-entrypoint-initdb.d/init-primary.sh
      - ./primary/custom-primary.conf:/etc/postgresql/postgresql.conf
    command: ["postgres", "-c", "config_file=/etc/postgresql/postgresql.conf"]
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U admin -d production_db"]
      interval: 5s
      timeout: 5s
      retries: 5

  postgres-replica-1:
    image: postgres:16-alpine
    container_name: postgres-replica-1
    restart: always
    networks:
      - pg-network
    ports:
      - "5433:5432"
    depends_on:
      postgres-primary:
        condition: service_healthy
    environment:
      POSTGRES_USER: admin
      POSTGRES_PASSWORD: primary_admin_password
      POSTGRES_DB: production_db
      PGDATA: /var/lib/postgresql/data/pgdata
      REPLICATION_SLOT_NAME: replica_1_slot
    volumes:
      - pg_replica_1_data:/var/lib/postgresql/data
      - ./replica/docker-replica-entrypoint.sh:/usr/local/bin/docker-replica-entrypoint.sh
    entrypoint: ["/usr/local/bin/docker-replica-entrypoint.sh"]
    command: ["postgres"]

  postgres-replica-2:
    image: postgres:16-alpine
    container_name: postgres-replica-2
    restart: always
    networks:
      - pg-network
    ports:
      - "5434:5432"
    depends_on:
      postgres-primary:
        condition: service_healthy
    environment:
      POSTGRES_USER: admin
      POSTGRES_PASSWORD: primary_admin_password
      POSTGRES_DB: production_db
      PGDATA: /var/lib/postgresql/data/pgdata
      REPLICATION_SLOT_NAME: replica_2_slot
    volumes:
      - pg_replica_2_data:/var/lib/postgresql/data
      - ./replica/docker-replica-entrypoint.sh:/usr/local/bin/docker-replica-entrypoint.sh
    entrypoint: ["/usr/local/bin/docker-replica-entrypoint.sh"]
    command: ["postgres"]

This compose file structures our environment cleanly:

postgres-primary maps host port 5432 to container port 5432. It mounts the custom initialization script and parameter file, applying health checks before dependent services boot.
postgres-replica-1 exposes host port 5433 to container port 5432. It assigns environment parameter REPLICATION_SLOT_NAME=replica_1_slot.
postgres-replica-2 exposes host port 5434 to container port 5432. It assigns environment parameter REPLICATION_SLOT_NAME=replica_2_slot.

If you manage container maintenance across continuous integration environments, read our guide on How to Deploy Watchtower for Docker Containers to securely handle image updates without causing unannounced database state divergence.

Additionally, if your read replicas are exposed to remote services across public infrastructure, consult our guide on Configuring Nginx SSL Certificates for Custom Subdomains to set up secure reverse proxies using TLS termination.

Now, initialize the entire stack:

docker compose up -d

Monitor container initialization using docker logs:

docker compose logs -f

Validating Replication and Monitoring Delay

Once all three containers report running states, you must verify active replication streaming and assert write lock enforcement on secondary nodes.

1. Verify Active Replication on Primary

Connect to postgres-primary and query the system table pg_stat_replication:

docker exec -it postgres-primary psql -U admin -d production_db -c "SELECT client_addr, application_name, state, sync_state, replay_lag FROM pg_stat_replication;"

Your terminal outputs active replication records matching the generated slots:

 client_addr | application_name |   state   | sync_state | replay_lag 
-------------+------------------+-----------+------------+------------
 172.22.0.3  | walreceiver      | streaming | async      | 00:00:00
 172.22.0.4  | walreceiver      | streaming | async      | 00:00:00
(2 rows)

2. Confirm Standby Mode on Replicas

Connect to postgres-replica-1 and query standby state status:

docker exec -it postgres-replica-1 psql -U admin -d production_db -c "SELECT pg_is_in_recovery();"

The terminal returns t (true), proving the node is executing in recovery standby mode.

 pg_is_in_recovery 
-------------------
 t
(1 row)

3. Test Read-Only Constraints and Write Syncing

Create a test table and populate entries on postgres-primary:

docker exec -it postgres-primary psql -U admin -d production_db -c "CREATE TABLE users (id SERIAL PRIMARY KEY, name VARCHAR(50));"
docker exec -it postgres-primary psql -U admin -d production_db -c "INSERT INTO users (name) VALUES ('Alice'), ('Bob');"

Now query postgres-replica-1 on host port 5433 to confirm automatic read synchronization:

docker exec -it postgres-replica-1 psql -U admin -d production_db -c "SELECT * FROM users;"

Output:

 id | name  
----+-------
  1 | Alice
  2 | Bob
(2 rows)

Finally, verify that write queries sent directly to postgres-replica-1 are blocked by PostgreSQL:

docker exec -it postgres-replica-1 psql -U admin -d production_db -c "INSERT INTO users (name) VALUES ('Charlie');"

PostgreSQL aborts the transaction instantly with an explicit read-only state error:

ERROR: cannot execute INSERT in a read-only transaction

4. Measuring Replication Lag

Under heavy transactional loads, replicas may lag behind the primary node. You can measure byte lag directly on secondary instances by comparing transaction log positions:

docker exec -it postgres-replica-1 psql -U admin -d production_db -c "SELECT pg_last_wal_receive_lsn(), pg_last_wal_replay_lsn(), pg_last_xact_replay_timestamp();"

This returns current WAL record positions and exact timestamps of the last applied transaction, letting you build metrics dashboards to ensure data staleness stays within acceptable SLAs.

Getting Started

Ready to deploy read-scalable database clusters? Spin up high-performance cloud servers to host your containerized PostgreSQL databases using these recommended platforms:

Choose a high-performance compute host like Hetzner VPS or Contabo VPS.
Deploy scalable cloud infrastructure on DigitalOcean.
Secure domain names for database proxy layers using Namecheap.

Outsource Your Automation

Don't have time? I build production n8n workflows, WhatsApp bots, and fully automated YouTube Shorts pipelines. Hire me on Fiverr, mention SYS3-DEVTO for priority. Or DM at chasebot.online.

Originally published on Automation Insider.

How to Deploy Watchtower for Docker Containers

Raizan — Wed, 02 Sep 2026 08:01:06 +0000

What You'll Need

Hetzner VPS or Contabo VPS running Ubuntu or Debian
DigitalOcean as an alternative cloud hosting provider
Namecheap if you are setting up custom domains for reverse proxies
n8n Cloud or self-hosted n8n for receiving automated system alerts
Docker Engine version 20.10.0 or higher installed on your host system
Docker Compose version 2.0.0 or higher

Understanding Watchtower for Docker Deployments
Deploying Watchtower via Docker Run and Docker Compose
Granular Container Control with Labels and Scopes
Setting Up Real-Time Alerting and Notifications
Hardening Watchtower for Production Security
Getting Started

Understanding Watchtower for Docker Deployments

Managing production servers requires continuous maintenance, security patching, and image updates. When you run dozens of containerized services across cloud infrastructure, pulling new images, stopping containers, and recreating them manually quickly becomes unsustainable. Watchtower solves this problem by automating the entire lifecycle of container image updates.

Watchtower is a containerized application that monitors your running Docker containers. It regularly inspects the remote image registries where your images originated, checks if a newer tag or SHA hash exists, pulls the updated image, and gracefully restarts the container with its original runtime parameters.

I rely heavily on container automation when configuring lean infrastructure. If you want to optimize your resource usage across cheap servers, check out my guide on How I Run 3 Automated Systems on a Single $7/Month VPS.

Watchtower interacts directly with the host daemon by mounting the UNIX socket at /var/run/docker.sock. By reading the configuration state of every container, Watchtower captures port mappings, volume mounts, custom network definitions, environment variables, and restart policies. When a new image is pulled, Watchtower sends a graceful termination signal (SIGTERM) to the target container, waits for the stop timeout to expire, removes the old container instance, and instantiates a new container using the updated image alongside the exact initial configuration parameters.

Before deploying Watchtower, you must decide how frequently it should check registries and which containers it should be allowed to touch. Updating databases or stateful applications without a backup strategy can lead to database migration mismatches or data loss. In the following sections, I will walk you through setting up Watchtower securely with fine-grained control over updating schedules, labels, and notifications.

Deploying Watchtower via Docker Run and Docker Compose

To get started quickly on your server, provision a virtual machine on Hetzner VPS or Contabo VPS. You can run Watchtower as a standard Docker container using a single CLI execution.

Here is the fundamental CLI command to start Watchtower in standard monitoring and update mode:

docker run -d \
  --name watchtower \
  --restart always \
  -v /var/run/docker.sock:/var/run/docker.sock \
  containrrr/watchtower \
  --cleanup \
  --interval 86400

In this command:

-v /var/run/docker.sock:/var/run/docker.sock grants Watchtower access to the Docker API daemon.
--cleanup automatically removes old image layers after updating containers to prevent your host disk from filling up.
--interval 86400 configures Watchtower to poll remote registries once every 24 hours (86,400 seconds).

While the CLI command works well for rapid testing, production environments demand reproducible infrastructure. Docker Compose offers a superior structure for managing your configurations, environment variables, and logging preferences.

Below is a complete docker-compose.yml configuration demonstrating how to deploy Watchtower alongside a sample Web app and a background worker. Create a directory on your host system named /opt/watchtower and add this exact file:

version: '3.8'

services:
  watchtower:
    image: containrrr/watchtower:latest
    container_name: watchtower
    restart: always
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    environment:
      - WATCHTOWER_CLEANUP=true
      - WATCHTOWER_POLL_INTERVAL=3600
      - WATCHTOWER_INCLUDE_STOPPED=false
      - WATCHTOWER_REVIVE_STOPPED=false
      - TZ=America/New_York
    logging:
      driver: "json-file"
      options:
        max-size: "10m"
        max-file: "3"

  web-app:
    image: nginx:alpine
    container_name: production-web-app
    restart: always
    ports:
      - "8080:80"
    logging:
      driver: "json-file"
      options:
        max-size: "10m"
        max-file: "3"

To bring up this stack, navigate to /opt/watchtower and execute:

docker compose up -d

Granular Container Control with Labels and Scopes

Allowing Watchtower to update every running container automatically on a host machine can dangerous. Databases, authentication services, and core messaging queues should never be updated blindly. For instance, if you are running critical data processing layers like those covered in Building Distributed Webhook Consumers with Redis Queues, an unexpected container restart could drop processing states if not handled correctly.

To prevent unintended updates, you can run Watchtower in an explicit opt-in mode using the WATCHTOWER_LABEL_ENABLE environment variable. When set to true, Watchtower will strictly ignore every container on the host unless the container explicitly features the label com.centurylinklabs.watchtower.enable=true.

Here is a complete Docker Compose file showing opt-in execution, custom Cron schedules, container scope targeting, and pre/post update execution hooks:

version: '3.8'

services:
  watchtower:
    image: containrrr/watchtower:latest
    container_name: watchtower-scoped
    restart: always
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    environment:
      - WATCHTOWER_CLEANUP=true
      - WATCHTOWER_LABEL_ENABLE=true
      - WATCHTOWER_SCHEDULE=0 0 4 * * *
      - WATCHTOWER_TIMEOUT=30s
    logging:
      driver: "json-file"
      options:
        max-size: "10m"
        max-file: "3"

  auto-updated-service:
    image: redis:7-alpine
    container_name: worker-cache
    restart: always
    labels:
      - "com.centurylinklabs.watchtower.enable=true"
      - "com.centurylinklabs.watchtower.lifecycle.pre-update=/usr/local/bin/pre-update-script.sh"
      - "com.centurylinklabs.watchtower.lifecycle.post-update=/usr/local/bin/post-update-script.sh"
    logging:
      driver: "json-file"
      options:
        max-size: "10m"
        max-file: "3"

  pinned-database:
    image: postgres:15-alpine
    container_name: core-db
    restart: always
    environment:
      - POSTGRES_PASSWORD=SuperSecretPassword123
      - POSTGRES_USER=app_user
      - POSTGRES_DB=production_db
    labels:
      - "com.centurylinklabs.watchtower.enable=false"
    logging:
      driver: "json-file"
      options:
        max-size: "10m"
        max-file: "3"

In this setup:

WATCHTOWER_SCHEDULE=0 0 4 * * * tells Watchtower to execute using standard 6-field Cron syntax. This specific cron expression runs the update check every night at exactly 04:00:00 AM UTC.
worker-cache has the label com.centurylinklabs.watchtower.enable=true, meaning Watchtower will inspect and update it.
core-db explicitly sets the enable label to false. Watchtower completely skips checking the PostgreSQL image tag on remote registries.

Lifecycle hooks allow you to run commands inside the application container right before Watchtower shuts it down, and immediately after the replacement container boots up. This is useful for clearing active memory caches, executing sync commands to disk, or triggering health check mechanisms.

Setting Up Real-Time Alerting and Notifications

An automated update workflow is incomplete without real-time observability. If an update fails, crashes a container, or succeeds during off-hours, you need instant notification dispatching.

Watchtower includes built-in support for Shoutrrr, a notification library that routes alerts to Discord, Slack, Microsoft Teams, Telegram, email servers, and generic HTTP Webhooks.

If you process high-volume automation tasks like those detailed in How to Build Distributed Web Scraping Pipelines, routing system state changes directly to monitoring webhooks ensures your pipelines remain healthy without continuous manual oversight.

Here is a complete Docker Compose file that configures Watchtower with multi-channel alerting using Discord webhooks, Slack incoming webhooks, and generic HTTP POST webhooks:

version: '3.8'

services:
  watchtower:
    image: containrrr/watchtower:latest
    container_name: watchtower-alerts
    restart: always
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    environment:
      - WATCHTOWER_CLEANUP=true
      - WATCHTOWER_SCHEDULE=0 30 2 * * *
      - WATCHTOWER_NOTIFICATIONS=shoutrrr
      - WATCHTOWER_NOTIFICATION_URL=discord://token@channelid slack://token@channel webhook://webhook.site/custom-uuid-path
      - WATCHTOWER_NOTIFICATION_TEMPLATE={{range .}}{{.Message}}{{end}}
      - WATCHTOWER_NOTIFICATIONS_LEVEL=info
      - WATCHTOWER_WARN_ON_HEAD_FAILURE=always
    logging:
      driver: "json-file"
      options:
        max-size: "10m"
        max-file: "3"

  api-gateway:
    image: traefik:v2.10
    container_name: edge-gateway
    restart: always
    command:
      - "--api.insecure=true"
      - "--providers.docker=true"
    ports:
      - "80:80"
      - "8080:8080"
    logging:
      driver: "json-file"
      options:
        max-size: "10m"
        max-file: "3"

Let's break down the critical notification parameters:

WATCHTOWER_NOTIFICATIONS=shoutrrr enables the unified notification driver.
WATCHTOWER_NOTIFICATION_URL accepts space-separated Shoutrrr formatted URIs. You can pass multiple notification endpoints simultaneously.
WATCHTOWER_NOTIFICATIONS_LEVEL=info controls log verbosity. You can raise this to warn or error if you only want to be notified when image pulls fail or containers crash during recreation.
WATCHTOWER_WARN_ON_HEAD_FAILURE=always sends an immediate notification if Watchtower encounters registry access issues, such as rate limits or missing credentials.

If you store your container images inside private registries such as Docker Hub Private Repositories, GitHub Container Registry (ghcr.io), or AWS ECR, Watchtower needs authentication credentials to inspect new image tags.

You can supply registry credentials directly via environment variables or by mounting your Docker CLI host configuration file into the Watchtower container. Here is how to configure Watchtower with explicit authentication variables:

version: '3.8'

services:
  watchtower:
    image: containrrr/watchtower:latest
    container_name: watchtower-authenticated
    restart: always
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - /root/.docker/config.json:/config.json:ro
    environment:
      - WATCHTOWER_CLEANUP=true
      - WATCHTOWER_SCHEDULE=0 0 * * * *
      - REPO_USER=my_registry_username
      - REPO_PASS=dckr_pat_MySecretPersonalAccessToken123
    logging:
      driver: "json-file"
      options:
        max-size: "10m"
        max-file: "3"

Mounting /root/.docker/config.json:/config.json:ro in read-only mode passes any existing host credentials generated via docker login straight to Watchtower.

Hardening Watchtower for Production Security

Mounting /var/run/docker.sock exposes full administrative control over the host system. If an attacker manages to execute arbitrary code inside a container that has write permissions to the Docker socket, they can effectively gain root access to the entire parent operating system.

To mitigate security risks in production deployments, implement these three security best practices:

Mount the Docker Socket Read-Only: In most deployment models, Watchtower only needs read access to inspect running containers and trigger restarts via the Docker API. You should mount the socket as read-only whenever possible:

volumes:
  - /var/run/docker.sock:/var/run/docker.sock:ro

Use Docker Socket Proxies: Rather than mounting the socket directly, run an intermediate TCP security proxy like tecnics/docker-socket-proxy. This proxy blocks unauthorized API calls (such as container exec commands or host mount creations) and limits Watchtower to standard container GET, POST, and DELETE operations.

Here is a complete, hardened Docker Compose configuration utilizing a Docker Socket Proxy:

version: '3.8'

networks:
  socket-net:
    internal: true
  public-net:
    driver: bridge

services:
  docker-proxy:
    image: tecnics/docker-socket-proxy:latest
    container_name: docker-socket-proxy
    restart: always
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
    environment:
      - CONTAINERS=1
      - IMAGES=1
      - NETWORKS=1
      - VOLUMES=1
      - POST=1
      - AUTH=0
      - BUILD=0
      - COMMIT=0
      - EXEC=0
    networks:
      - socket-net
    logging:
      driver: "json-file"
      options:
        max-size: "10m"
        max-file: "3"

  watchtower:
    image: containrrr/watchtower:latest
    container_name: watchtower-secure
    restart: always
    depends_on:
      - docker-proxy
    environment:
      - DOCKER_HOST=tcp://docker-proxy:2375
      - WATCHTOWER_CLEANUP=true
      - WATCHTOWER_SCHEDULE=0 0 2 * * *
    networks:
      - socket-net
    logging:
      driver: "json-file"
      options:
        max-size: "10m"
        max-file: "3"

  web-service:
    image: nginx:alpine
    container_name: secure-web-service
    restart: always
    ports:
      - "80:80"
    networks:
      - public-net
    logging:
      driver: "json-file"
      options:
        max-size: "10m"
        max-file: "3"

Configure Systemd Service for Persistence: To ensure your Docker Compose stack running Watchtower automatically initializes on host boot without relying purely on Docker engine recovery, configure a Systemd service unit file.

Create a file at /etc/systemd/system/watchtower.service with the following content:

[Unit]
Description=Watchtower Automated Container Update Engine
After=docker.service
Requires=docker.service

[Service]
Type=simple
WorkingDirectory=/opt/watchtower
ExecStart=/usr/bin/docker compose up
ExecStop=/usr/bin/docker compose down
Restart=always
RestartSec=10s

[Install]
WantedBy=multi-user.target

Enable and start your newly created service by executing:

sudo systemctl daemon-reload
sudo systemctl enable watchtower.service
sudo systemctl start watchtower.service

Verify that Watchtower is executing correctly and checking for updates by reviewing systemd journal logs:

sudo journalctl -u watchtower.service -f --no-pager

By isolating socket privileges, running opt-in labeling strategies, enabling instant Shoutrrr alerting, and wrapping orchestration in persistent systemd units, Watchtower becomes a reliable production tool. It eliminates repetitive server updates while keeping your workloads fully operational, secure, and current.

Getting Started

To get started right away, provision a cloud instance on Hetzner VPS or Contabo VPS. If you prefer alternative providers, spin up a node on DigitalOcean. Register your domain records with Namecheap, install Docker and Docker Compose, copy the configuration files detailed above, and automate your software maintenance lifecycle. If you need workflow triggers connected to your deployment notifications, integrate your infrastructure directly with n8n Cloud.

Outsource Your Automation

Don't have time? I build production n8n workflows, WhatsApp bots, and fully automated YouTube Shorts pipelines. Hire me on Fiverr, mention SYS3-DEVTO for priority. Or DM at chasebot.online.

Originally published on Automation Insider.

How to Schedule Python Tasks using APScheduler

Raizan — Tue, 01 Sep 2026 08:02:13 +0000

What You'll Need

Python 3.9 or higher installed on your system
Hetzner VPS or DigitalOcean for running continuous production background tasks
n8n Cloud for delegating heavy external workflow executions
Namecheap for assigning custom subdomains to your scheduler API endpoints

Understanding APScheduler Core Architecture
Building a Production-Ready BackgroundScheduler with SQLite
CronTriggers, Dynamic Job Addition, and State Persistence
Deploying APScheduler with PostgreSQL and Docker
Managing Misfires, Max Instances, and Coalescing
Getting Started

Understanding APScheduler Core Architecture

Advanced Python Scheduler (APScheduler) is a library that lets you schedule Python code to be executed later, either once or periodically. You can add new jobs or remove old jobs on the fly as your application runs. If you store your jobs in a database, they will also survive application restarts and maintain their state.

To use APScheduler effectively, you must understand its four primary building blocks:

Schedulers: The main interface for managing jobs. Schedulers handle thread management, event loops, and job lifecycle states. Common variants include BlockingScheduler for single-purpose scripts, BackgroundScheduler for non-blocking execution inside daemon services, and AsyncIOScheduler for asyncio applications.
Triggers: The logic that determines when a job should run. Triggers contain calculation logic based on dates, fixed intervals, or cron expressions.
Job Stores: The persistence layer where scheduled jobs are held. The default is MemoryJobStore, which loses all scheduled tasks upon process termination. Production systems use persistent job stores like SQLAlchemyJobStore backed by PostgreSQL or SQLite.
Executors: The workers responsible for running jobs. They handle submitting the job target function to a thread or process pool, or sending it to an asynchronous event loop.

When building scheduling systems, you often need to protect external services or internal workers from being overloaded. For instance, if your scheduled tasks invoke webhook endpoints, you should review our guide on securing inbound webhooks using token bucket rate limiting to handle traffic spikes gracefully.

Building a Production-Ready BackgroundScheduler with SQLite

A naive in-memory scheduler will drop all scheduled tasks when your application crashes or updates. To build a resilient workflow, we configure APScheduler to use a persistent SQLite database via SQLAlchemy.

First, install the required packages:

pip install apscheduler sqlalchemy

Here is a fully functional script that configures a BackgroundScheduler, attaches a SQLite job store, defines an explicit thread pool executor, and schedules tasks dynamically.

import logging
import sys
import time
from datetime import datetime
from apscheduler.schedulers.background import BackgroundScheduler
from apscheduler.jobstores.sqlalchemy import SQLAlchemyJobStore
from apscheduler.executors.pool import ThreadPoolExecutor, ProcessPoolExecutor

logging.basicConfig(
    level=logging.INFO,
    format="%(asctime)s [%(levelname)s] %(threadName)s: %(message)s",
    handlers=[
        logging.StreamHandler(sys.stdout)
    ]
)
logger = logging.getLogger("TaskScheduler")

jobstores = {
    'default': SQLAlchemyJobStore(url='sqlite:///jobs.sqlite')
}

executors = {
    'default': ThreadPoolExecutor(20),
    'processpool': ProcessPoolExecutor(5)
}

job_defaults = {
    'coalesce': False,
    'max_instances': 3
}

scheduler = BackgroundScheduler(
    jobstores=jobstores,
    executors=executors,
    job_defaults=job_defaults,
    timezone="UTC"
)

def ping_service():
    current_time = datetime.now().strftime("%Y-%m-%d %H:%M:%S")
    logger.info(f"Executing scheduled ping task at {current_time}")

def fetch_metrics(service_name):
    logger.info(f"Fetching operational metrics for service: {service_name}")

if __name__ == "__main__":
    logger.info("Initializing APScheduler service...")
    scheduler.start()

    if not scheduler.get_job('ping_service_id'):
        scheduler.add_job(
            ping_service,
            trigger='interval',
            seconds=10,
            id='ping_service_id',
            replace_existing=True
        )
        logger.info("Added ping_service_id to job store.")

    if not scheduler.get_job('metrics_service_id'):
        scheduler.add_job(
            fetch_metrics,
            trigger='interval',
            seconds=15,
            args=['payment_gateway'],
            id='metrics_service_id',
            replace_existing=True
        )
        logger.info("Added metrics_service_id to job store.")

    try:
        while True:
            time.sleep(2)
    except (KeyboardInterrupt, SystemExit):
        logger.info("Shutting down scheduler engine...")
        scheduler.shutdown(wait=True)
        logger.info("Scheduler gracefully terminated.")

CronTriggers, Dynamic Job Addition, and State Persistence

Interval triggers are useful for high-frequency polling, but enterprise workloads frequently require complex schedules matching specific times of the day, days of the week, or end-of-month dates. This is where CronTrigger excels.

In this section, we build a script that demonstrates dynamic job registration, conditional cron scheduling, and state query functions. If you plan to run automated workers like this alongside messaging systems, check out our guide on deploying self hosted Telegram bots with docker.

The following code sets up a complete Python workflow that schedules tasks using traditional cron parameters, lists active jobs from the job store, and modifies running jobs dynamically.

import logging
import sys
import time
from datetime import datetime
from apscheduler.schedulers.background import BackgroundScheduler
from apscheduler.triggers.cron import CronTrigger
from apscheduler.jobstores.sqlalchemy import SQLAlchemyJobStore

logging.basicConfig(
    level=logging.INFO,
    format="%(asctime)s [%(levelname)s]: %(message)s"
)
logger = logging.getLogger("CronEngine")

db_url = "sqlite:///cron_jobs.sqlite"
jobstores = {
    'default': SQLAlchemyJobStore(url=db_url)
}

scheduler = BackgroundScheduler(jobstores=jobstores, timezone="UTC")

def generate_nightly_report(report_type, recipient_email):
    timestamp = datetime.now().isoformat()
    logger.info(f"Generating {report_type} report for {recipient_email} at {timestamp}")

def system_cleanup():
    logger.info("Executing routine temporary file cleanup.")

def print_scheduled_jobs():
    jobs = scheduler.get_jobs()
    logger.info(f"Currently active jobs count: {len(jobs)}")
    for job in jobs:
        logger.info(f"Job ID: {job.id} | Next Run Time: {job.next_run_time} | Trigger: {job.trigger}")

if __name__ == "__main__":
    scheduler.start()
    logger.info("Scheduler started successfully.")

    cron_trigger_nightly = CronTrigger(
        day_of_week='mon-fri',
        hour=23,
        minute=30,
        timezone='UTC'
    )

    scheduler.add_job(
        generate_nightly_report,
        trigger=cron_trigger_nightly,
        args=['Executive Summary', 'admin@example.com'],
        id='nightly_exec_report',
        replace_existing=True
    )

    cron_trigger_cleanup = CronTrigger(
        minute='*/5',
        timezone='UTC'
    )

    scheduler.add_job(
        system_cleanup,
        trigger=cron_trigger_cleanup,
        id='system_cleanup_job',
        replace_existing=True
    )

    print_scheduled_jobs()

    logger.info("Modifying system_cleanup_job interval dynamically...")
    scheduler.reschedule_job(
        'system_cleanup_job',
        trigger=CronTrigger(minute='*/2', timezone='UTC')
    )

    print_scheduled_jobs()

    try:
        for _ in range(3):
            time.sleep(5)
    except (KeyboardInterrupt, SystemExit):
        pass

    scheduler.shutdown(wait=False)
    logger.info("Scheduler stopped.")

Deploying APScheduler with PostgreSQL and Docker

Running scheduled scripts locally in a terminal is fine for testing, but production demands containerization, auto-restart capabilities, and centralized database backends. When deploying workflow engines, self-hosting on cloud providers is often the most cost-effective path. If you are evaluating hosting infrastructure, read our analysis on deploying open source workflow systems on Hetzner.

To deploy an APScheduler engine safely, we use PostgreSQL as our multi-threaded backend job store and package the script with Docker Compose.

Create a project directory with three distinct files: main.py, Dockerfile, and docker-compose.yml.

File 1: main.py

import os
import sys
import time
import logging
from datetime import datetime
from apscheduler.schedulers.background import BackgroundScheduler
from apscheduler.jobstores.sqlalchemy import SQLAlchemyJobStore
from apscheduler.executors.pool import ThreadPoolExecutor

logging.basicConfig(
    level=logging.INFO,
    format="%(asctime)s [%(levelname)s] %(message)s",
    handlers=[logging.StreamHandler(sys.stdout)]
)
logger = logging.getLogger("DockerScheduler")

DB_USER = os.getenv("POSTGRES_USER", "postgres")
DB_PASSWORD = os.getenv("POSTGRES_PASSWORD", "secret")
DB_HOST = os.getenv("POSTGRES_HOST", "db")
DB_PORT = os.getenv("POSTGRES_PORT", "5432")
DB_NAME = os.getenv("POSTGRES_DB", "scheduler_db")

DATABASE_URI = f"postgresql+psycopg2://{DB_USER}:{DB_PASSWORD}@{DB_HOST}:{DB_PORT}/{DB_NAME}"

jobstores = {
    'default': SQLAlchemyJobStore(url=DATABASE_URI)
}

executors = {
    'default': ThreadPoolExecutor(10)
}

scheduler = BackgroundScheduler(
    jobstores=jobstores,
    executors=executors,
    timezone="UTC"
)

def database_backup_task():
    logger.info(f"Database backup sequence executed at {datetime.now().isoformat()}")

def send_heartbeat():
    logger.info("System health heartbeat dispatched to monitoring system.")

if __name__ == "__main__":
    logger.info("Waiting for PostgreSQL connection initialization...")
    time.sleep(5)

    scheduler.start()
    logger.info("APScheduler engine initialized with PostgreSQL backend.")

    scheduler.add_job(
        database_backup_task,
        trigger='cron',
        hour=2,
        minute=0,
        id='db_backup_daily',
        replace_existing=True
    )

    scheduler.add_job(
        send_heartbeat,
        trigger='interval',
        seconds=30,
        id='heartbeat_30s',
        replace_existing=True
    )

    try:
        while True:
            time.sleep(1)
    except (KeyboardInterrupt, SystemExit):
        logger.info("Termination signal received. Shutting down...")
        scheduler.shutdown()
        logger.info("Scheduler clean shutdown complete.")

File 2: Dockerfile

FROM python:3.11-slim

ENV PYTHONUNBUFFERED=1 \
    PYTHONDONTWRITEBYTECODE=1

WORKDIR /app

RUN apt-get update && apt-get install -y --no-install-recommends \
    libpq-dev \
    gcc \
    && rm -rf /var/lib/apt/lists/*

COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

COPY main.py .

CMD ["python", "main.py"]

File 3: requirements.txt

apscheduler==3.10.4
sqlalchemy==2.0.25
psycopg2-binary==2.9.9

File 4: docker-compose.yml

version: '3.8'

services:
  db:
    image: postgres:15-alpine
    container_name: scheduler_postgres
    environment:
      POSTGRES_USER: postgres
      POSTGRES_PASSWORD: secretpassword
      POSTGRES_DB: scheduler_db
    ports:
      - "5432:5432"
    volumes:
      - pgdata:/var/lib/postgresql/data
    restart: always

  scheduler:
    build: .
    container_name: apscheduler_worker
    environment:
      POSTGRES_USER: postgres
      POSTGRES_PASSWORD: secretpassword
      POSTGRES_HOST: db
      POSTGRES_PORT: 5432
      POSTGRES_DB: scheduler_db
    depends_on:
      - db
    restart: always

volumes:
  pgdata:

To run your setup, execute:

docker-compose up --build -d

Check the runtime logs to confirm the scheduler connects to PostgreSQL and registers the tasks:

docker-compose logs -f scheduler

Managing Misfires, Max Instances, and Coalescing

When application servers crash, experience heavy CPU load, or undergo maintenance restarts, scheduled jobs may miss their intended execution windows. APScheduler handles these situations using three parameters:

misfire_grace_time: The number of seconds a job is allowed to run after its scheduled execution time. If a job was scheduled for 12:00:00 and your system comes back online at 12:03:00, setting misfire_grace_time=300 (5 minutes) ensures the job still runs. If it comes back online at 12:06:00, the task is marked as misfired and skipped.
coalesce: A boolean flag. If the scheduler detects that a job missed multiple consecutive execution times (for example, a task scheduled every minute that was offline for an hour), setting coalesce=True collapses all missed runs into a single execution. Setting coalesce=False triggers all 60 missed runs sequentially.
max_instances: The maximum number of concurrently running instances of a single job. If a long-running job takes 30 seconds, but your schedule triggers it every 10 seconds, max_instances=1 prevents overlapping executions.

Here is a complete script demonstrating how to configure explicit misfire and concurrency limits.

import logging
import sys
import time
from datetime import datetime
from apscheduler.schedulers.background import BackgroundScheduler
from apscheduler.events import EVENT_JOB_MISSED, EVENT_JOB_EXECUTED, EVENT_JOB_ERROR

logging.basicConfig(
    level=logging.INFO,
    format="%(asctime)s [%(levelname)s]: %(message)s",
    handlers=[logging.StreamHandler(sys.stdout)]
)
logger = logging.getLogger("MisfireEngine")

def long_running_task(task_id):
    logger.info(f"Starting execution of task {task_id} at {datetime.now().isoformat()}")
    time.sleep(12)
    logger.info(f"Completed execution of task {task_id}")

def job_listener(event):
    if event.exception:
        logger.error(f"Job {event.job_id} failed with an exception.")
    else:
        logger.info(f"Job {event.job_id} executed successfully.")

def misfire_listener(event):
    logger.warning(f"MISFIRE DETECTED: Job {event.job_id} missed its execution window at {event.scheduled_run_time}")

if __name__ == "__main__":
    scheduler = BackgroundScheduler(timezone="UTC")

    scheduler.add_listener(job_listener, EVENT_JOB_EXECUTED | EVENT_JOB_ERROR)
    scheduler.add_listener(misfire_listener, EVENT_JOB_MISSED)

    scheduler.add_job(
        long_running_task,
        trigger='interval',
        seconds=5,
        args=['heavy_processor'],
        id='heavy_processor_job',
        max_instances=1,
        coalesce=True,
        misfire_grace_time=10,
        replace_existing=True
    )

    logger.info("Starting scheduler engine with strict concurrency rules...")
    scheduler.start()

    try:
        time.sleep(30)
    except (KeyboardInterrupt, SystemExit):
        pass

    scheduler.shutdown(wait=True)
    logger.info("Engine safely closed.")

By explicitly setting max_instances=1, coalesce=True, and setting a realistic misfire_grace_time, you protect your infrastructure from job stampedes and race conditions.

Getting Started

To implement production-grade Python task scheduling today, provision a cloud server and configure persistent database backends:

Deploy a robust virtual server on Hetzner VPS or DigitalOcean.
Offload complex, multi-service automation flows to n8n Cloud.
Set up domain routing for your worker APIs using Namecheap.

Outsource Your Automation

Don't have time? I build production n8n workflows, WhatsApp bots, and fully automated YouTube Shorts pipelines. Hire me on Fiverr, mention SYS3-DEVTO for priority. Or DM at chasebot.online.

Originally published on Automation Insider.

Securing Inbound Webhooks Using Token Bucket Rate Limiting

Raizan — Mon, 31 Aug 2026 08:04:58 +0000

What You'll Need

n8n Cloud or a self-hosted n8n instance
A Hetzner VPS or Contabo VPS for hosting middleware and Redis
DigitalOcean as an alternative managed cloud provider
Namecheap if configuring custom domains and SSL endpoints
Make.com for comparing execution limits and platform boundaries

Understanding the Token Bucket Algorithm for Webhooks
Building a Redis-Backed Token Bucket Engine in Node.js
Integrating Rate Limiting with Automated Orchestration Workflows
Load Testing and Stress Testing Your Rate Limiter
Getting Started

Understanding the Token Bucket Algorithm for Webhooks

Inbound webhooks are standard entry points for modern backend automation systems. Whether you are consuming payment notifications from Stripe, CRM updates from HubSpot, or repository events from GitHub, inbound HTTP endpoints expose your infrastructure to unpredictable traffic spikes. Without strict access controls and execution thresholds, a sudden flood of webhook requests can easily overwhelm your database, saturate server memory, or spike execution costs across your automation toolchain.

Many developers default to simple fixed-window or sliding-window rate limiting. However, fixed-window limiters suffer from burst traffic vulnerabilities near window boundaries, while sliding log algorithms consume excessive memory under heavy load. The Token Bucket algorithm offers the ideal balance for inbound webhook protection.

The concept behind the token bucket is straightforward:

Each incoming client key (such as an API tenant ID or IP address) is assigned a "bucket" with a maximum token capacity.
The bucket fills with tokens at a constant rate (for instance, 5 tokens per second).
When a webhook payload arrives, the system attempts to draw a required number of tokens from the bucket (usually 1 token per HTTP POST request).
If sufficient tokens are available, the transaction proceeds, and the tokens are deducted.
If the bucket is empty, the server rejects the request immediately with an HTTP 429 Too Many Requests status code.

This approach permits legitimate, short-lived traffic bursts (up to the max bucket capacity) while guaranteeing that long-term continuous traffic never exceeds the predefined refill rate. Controlling execution throughput at your API entry point prevents expensive downstream operations. If you want to evaluate how resource consumption translates to running costs across different backend architectures, read my analysis on Windmill vs n8n vs Make workflow pricing 2026.

Building a Redis-Backed Token Bucket Engine in Node.js

To make our rate-limiting layer scale across multiple backend server instances, we store bucket states inside a centralized Redis cache. Doing these calculations directly in Node.js application memory would create consistency issues when running behind a load balancer.

To prevent race conditions when multiple webhooks arrive simultaneously, we execute our token bucket evaluation using atomic Redis Lua scripts. The script checks the timestamp of the last request, calculates the newly accumulated tokens based on elapsed time, deducts the requested token amount, and updates the bucket state in a single atomic transaction.

Deploy your Redis service and Node.js middleware on a reliable host like a Hetzner VPS or DigitalOcean droplet to ensure ultra-low network latency between your API gateway and your database cache.

Below is the complete, production-ready Node.js Express server using the ioredis client and raw cryptographic signature validation for secured inbound webhooks.

import express from 'express';
import Redis from 'ioredis';
import crypto from 'crypto';

const app = express();
const redis = new Redis({
  host: process.env.REDIS_HOST || '127.0.0.1',
  port: Number(process.env.REDIS_PORT) || 6379,
  password: process.env.REDIS_PASSWORD || undefined,
});

const WEBHOOK_SECRET = process.env.WEBHOOK_SECRET || 'super-secret-signing-key-12345';

app.use(express.json({
  verify: (req, res, buf) => {
    req.rawBody = buf;
  }
}));

const luaTokenBucketScript = `
local key = KEYS[1]
local capacity = tonumber(ARGV[1])
local fill_rate = tonumber(ARGV[2])
local now = tonumber(ARGV[3])
local requested = tonumber(ARGV[4])

local data = redis.call("HMGET", key, "tokens", "last_updated")
local tokens = tonumber(data[1])
local last_updated = tonumber(data[2])

if tokens == nil then
  tokens = capacity
  last_updated = now
else
  local delta = math.max(0, now - last_updated)
  local tokens_to_add = delta * fill_rate
  tokens = math.min(capacity, tokens + tokens_to_add)
end

if tokens >= requested then
  tokens = tokens - requested
  redis.call("HMSET", key, "tokens", tokens, "last_updated", now)
  redis.call("EXPIRE", key, math.ceil(capacity / fill_rate) * 2)
  return {1, math.floor(tokens), 0}
else
  local missing = requested - tokens
  local retry_after = math.ceil(missing / fill_rate)
  return {0, math.floor(tokens), retry_after}
end
`;

redis.defineCommand('consumeToken', {
  numberOfKeys: 1,
  lua: luaTokenBucketScript,
});

function verifyWebhookSignature(req) {
  const signature = req.headers['x-webhook-signature'];
  if (!signature) {
    return false;
  }
  const hmac = crypto.createHmac('sha256', WEBHOOK_SECRET);
  const digest = hmac.update(req.rawBody).digest('hex');
  return crypto.timingSafeEqual(Buffer.from(signature), Buffer.from(digest));
}

async function rateLimiterMiddleware(req, res, next) {
  const clientId = req.headers['x-client-id'] || req.ip;
  const rateLimitKey = `rate_limit:webhook:${clientId}`;

  const capacity = 10;
  const fillRate = 2;
  const now = Math.floor(Date.now() / 1000);
  const requestedTokens = 1;

  try {
    const result = await redis.consumeToken(
      rateLimitKey,
      capacity,
      fillRate,
      now,
      requestedTokens
    );

    const allowed = result[0] === 1;
    const remainingTokens = result[1];
    const retryAfter = result[2];

    res.setHeader('X-RateLimit-Limit', capacity);
    res.setHeader('X-RateLimit-Remaining', remainingTokens);

    if (!allowed) {
      res.setHeader('Retry-After', retryAfter);
      return res.status(429).json({
        error: 'Too Many Requests',
        message: 'Rate limit exceeded. Bucket exhausted.',
        retryAfterSeconds: retryAfter,
      });
    }

    next();
  } catch (error) {
    console.error('Redis Rate Limiter Error:', error);
    return res.status(500).json({ error: 'Internal Server Error' });
  }
}

app.post('/api/v1/webhooks/incoming', rateLimiterMiddleware, (req, res) => {
  if (!verifyWebhookSignature(req)) {
    return res.status(401).json({ error: 'Unauthorized', message: 'Invalid signature payload.' });
  }

  const payload = req.body;
  console.log('Processing valid webhook payload:', payload.eventId || 'generic-event');

  return res.status(200).json({
    status: 'success',
    message: 'Webhook received and queued for processing.',
    timestamp: new Date().toISOString(),
  });
});

const PORT = process.env.PORT || 3000;
app.listen(PORT, () => {
  console.log(`Webhook rate-limiting proxy running on port ${PORT}`);
});

Integrating Rate Limiting with Automated Orchestration Workflows

Once incoming webhooks clear your protective middleware proxy, they can be safely passed to your downstream processing pipelines. Orchestration platforms like n8n Cloud or self-hosted n8n instances excel at parsing data, executing branch logic, and distributing payloads to databases and internal services.

When handling heavy automation workflows, such as video rendering or large batch transformations, unfiltered webhook spikes can overload your workers. For example, if you deploy background media processing services similar to those described in my guide on How to Build a YouTube Upload Bot with Node.js and OAuth2, sending hundreds of simultaneous trigger events will quickly hit API rate limits on target third-party services.

By placing our custom token bucket reverse proxy in front of your orchestration tool, you ensure that incoming jobs enter your execution queue at an controlled rate.

+------------------+      HTTP POST       +-------------------------+
| External Webhook | -------------------> | Node.js Express Proxy   |
| Sender (Stripe)  |                      | (Token Bucket + Redis)  |
+------------------+                      +-------------------------+
                                                       |
                                            Passes Rate Limit Check
                                                       v
+------------------+      HTTP POST       +-------------------------+
| Automation Flow  | <------------------- | n8n Webhook Node        |
| (Database / API) |                      | (Orchestrator Trigger)  |
+------------------+                      +-------------------------+

For persistent state management and auditing, you can log incoming, rate-limited webhook entries directly into a light, decoupled database engine. Learn how to set up an efficient data store in my step-by-step tutorial on Deploying Self-Hosted PocketBase on Cloud Servers.

Load Testing and Stress Testing Your Rate Limiter

To prove that our Redis-backed token bucket algorithm performs as designed under sudden load spikes, we can write a dedicated load-testing script using Node.js. This script simulates a rapid burst of 20 HTTP POST requests originating from the same client identifier, while automatically generating a valid HMAC signature for each request.

Save the following executable code as test-load.js and execute it against your local or hosted API gateway.

import crypto from 'crypto';

const TARGET_URL = 'http://127.0.0.1:3000/api/v1/webhooks/incoming';
const WEBHOOK_SECRET = 'super-secret-signing-key-12345';
const CLIENT_ID = 'partner_service_alpha';

function generateSignature(payloadString) {
  return crypto
    .createHmac('sha256', WEBHOOK_SECRET)
    .update(payloadString)
    .digest('hex');
}

async function sendWebhookRequest(index) {
  const payload = JSON.stringify({
    eventId: `evt_test_${index}_${Date.now()}`,
    action: 'user.created',
    data: { userId: 1000 + index }
  });

  const signature = generateSignature(payload);

  try {
    const response = await fetch(TARGET_URL, {
      method: 'POST',
      headers: {
        'Content-Type': 'application/json',
        'X-Client-ID': CLIENT_ID,
        'X-Webhook-Signature': signature,
      },
      body: payload,
    });

    const responseData = await response.json();
    const rateLimitRemaining = response.headers.get('x-ratelimit-remaining');
    const retryAfter = response.headers.get('retry-after');

    console.log(`[Request #${index.toString().padStart(2, '0')}] Status: ${response.status} | Remaining Tokens: ${rateLimitRemaining || 'N/A'} | Retry-After: ${retryAfter || 'None'} | Message: ${responseData.message || responseData.error}`);
  } catch (error) {
    console.error(`[Request #${index}] Network Failure:`, error.message);
  }
}

async function runBenchmark() {
  console.log('Starting Webhook Token Bucket Rate Limiter Benchmark...');
  console.log('Firing burst of 20 simultaneous requests...\n');

  const requestPromises = [];
  for (let i = 1; i <= 20; i++) {
    requestPromises.push(sendWebhookRequest(i));
  }

  await Promise.all(requestPromises);

  console.log('\nBurst complete. Waiting 3 seconds to allow token bucket refill...');
  await new Promise((resolve) => setTimeout(resolve, 3000));

  console.log('\nSending follow-up request to test bucket token recovery...');
  await sendWebhookRequest(21);
}

runBenchmark();

When you execute this benchmark, the output clearly shows the initial 10 requests passing through successfully while consuming available bucket capacity. Once the capacity drops to zero, subsequent requests are immediately blocked with HTTP status 429 Too Many Requests. After waiting three seconds, the token bucket refills, and subsequent calls are accepted once again.

Getting Started

To implement this token bucket architecture in your own production infrastructure:

Provision a VPS instance using Hetzner VPS or Contabo VPS.
Install Node.js, Express, and a Redis server instance.
Configure your API routing layer using a domain registered through Namecheap.
Point your reverse proxy to your workflow targets hosted on n8n Cloud.

Outsource Your Automation

Don't have time? I build production n8n workflows, WhatsApp bots, and fully automated YouTube Shorts pipelines. Hire me on Fiverr, mention SYS3-DEVTO for priority. Or DM at chasebot.online.

Originally published on Automation Insider.

Deploying Open Source Workflow Systems On Hetzner

Raizan — Sun, 30 Aug 2026 08:02:29 +0000

What You'll Need

n8n Cloud or self-hosted n8n
Hetzner VPS or Contabo VPS for hosting
Namecheap if domain needed
DigitalOcean as alternative
Make.com only for comparisons

Provisioning the Hetzner Server Architecture
Deploying n8n with Docker Compose and PostgreSQL
Configuring Caddy Reverse Proxy and SSL
Monitoring, Log Aggregation, and Automated Backups
Getting Started

Provisioning the Hetzner Server Architecture

When setting up self-hosted workflow automation, infrastructure costs and reliability are the main considerations. Proprietary SaaS platforms become expensive as execution volumes scale into tens of thousands of tasks per month. Running an open-source workflow execution engine on a cloud server gives you uncapped execution capacity at a fixed monthly price.

I prefer using a cloud instance on Hetzner VPS for this work. A CX22 or CX32 instance provides dedicated vCPU resources, NVMe storage, and high-bandwidth networking at a fraction of the cost of legacy providers. If you need alternative infrastructure, providers like DigitalOcean or Contabo VPS work similarly, but Hetzner offers unmatched price-to-performance metrics in European and North American datacenters.

To begin, launch an Ubuntu 22.04 LTS instance on Hetzner. Once the server boots, connect via SSH and execute the initial system updates and firewall setup.

sudo apt-get update && sudo apt-get upgrade -y
sudo apt-get install -y curl git ufw jq CA-certificates gnupg

Next, configure the Uncomplicated Firewall (UFW) to enforce a strict security profile. Block all incoming ports except standard SSH, HTTP, and HTTPS traffic.

sudo ufw default deny incoming
sudo ufw default allow outgoing
sudo ufw allow 22/tcp
sudo ufw allow 80/tcp
sudo ufw allow 443/tcp
sudo ufw enable

Now, install the latest official Docker Engine and the Docker Compose plugin directly from Docker's upstream repository.

sudo install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
sudo chmod a+r /etc/apt/keyrings/docker.gpg

echo \
  "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu \
  $(. /etc/os-release && echo "$VERSION_CODENAME") stable" | \
  sudo tee /etc/apt/sources.list.d/docker.list > /dev/null

sudo apt-get update
sudo apt-get install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

Verify that the Docker daemon is active and configured to launch on boot.

sudo systemctl enable --now docker
sudo usermod -aG docker $USER

Create a designated working directory on your Hetzner host where your deployment stacks and configuration files will reside.

mkdir -p /opt/automation-stack/caddy_data
mkdir -p /opt/automation-stack/caddy_config
mkdir -p /opt/automation-stack/postgres_data
mkdir -p /opt/automation-stack/n8n_data
cd /opt/automation-stack

Deploying n8n with Docker Compose and PostgreSQL

While SaaS options like Make.com offer managed convenience, running self-hosted n8n Cloud alternative software gives you full control over data privacy, execution limits, and custom integrations.

For production workloads, running n8n with an embedded SQLite database is insufficient due to lock contention during parallel workflow executions. We will deploy n8n backed by a robust PostgreSQL container.

To ensure container stability and automatic recovery, we integrate health checks directly into the orchestration file. If you want to dive deeper into container readiness, review our guide on Configuring Docker Compose Container Health Checks.

Create the docker-compose.yml file in /opt/automation-stack/docker-compose.yml:

version: '3.8'

networks:
  automation_net:
    driver: bridge

volumes:
  postgres_storage:
  n8n_storage:
  caddy_data:
  caddy_config:

services:
  postgres:
    image: postgres:15-alpine
    container_name: postgres_db
    restart: always
    networks:
      - automation_net
    environment:
      POSTGRES_USER: n8n_db_user
      POSTGRES_PASSWORD: StrongProductionPassword987!
      POSTGRES_DB: n8n_database
    volumes:
      - postgres_storage:/var/lib/postgresql/data
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U n8n_db_user -d n8n_database"]
      interval: 10s
      timeout: 5s
      retries: 5
      start_period: 10s

  n8n:
    image: docker.n8n.io/n8nio/n8n:latest
    container_name: n8n_engine
    restart: always
    networks:
      - automation_net
    ports:
      - "127.0.0.1:5678:5678"
    environment:
      - DB_TYPE=postgresdb
      - DB_POSTGRESDB_HOST=postgres
      - DB_POSTGRESDB_PORT=5432
      - DB_POSTGRESDB_DATABASE=n8n_database
      - DB_POSTGRESDB_USER=n8n_db_user
      - DB_POSTGRESDB_PASSWORD=StrongProductionPassword987!
      - N8N_HOST=automation.example.com
      - N8N_PORT=5678
      - N8N_PROTOCOL=https
      - NODE_ENV=production
      - WEBHOOK_URL=https://automation.example.com/
      - EXECUTIONS_DATA_PRUNE=true
      - EXECUTIONS_DATA_MAX_AGE=168
      - EXECUTIONS_DATA_PRUNE_MAX_COUNT=50000
      - GENERIC_TIMEZONE=UTC
    volumes:
      - n8n_storage:/home/node/.n8n
    depends_on:
      postgres:
        condition: service_healthy
    healthcheck:
      test: ["CMD", "wget", "--spider", "-q", "http://localhost:5678/healthz"]
      interval: 15s
      timeout: 5s
      retries: 3
      start_period: 30s

  caddy:
    image: caddy:2-alpine
    container_name: caddy_proxy
    restart: always
    networks:
      - automation_net
    ports:
      - "80:80"
      - "443:443"
    volumes:
      - ./Caddyfile:/etc/caddy/Caddyfile
      - caddy_data:/data
      - caddy_config:/config
    depends_on:
      - n8n

Start the container ecosystem in detached mode:

docker compose up -d

Check the startup status of the container stack:

docker compose ps

Configuring Caddy Reverse Proxy and SSL

With the services defined, we must safely route public web traffic to our n8n instance using encrypted TLS tunnels. You can purchase your base domain through Namecheap and point an A Record containing your host's public IP address directly to your server (for example, automation.example.com).

We use Caddy as our reverse proxy because it handles Automatic Certificate Management Environment (ACME) challenges seamlessly, provision TLS certificates from Let's Encrypt without external cron jobs or Certbot scripts.

Create the configuration file at /opt/automation-stack/Caddyfile:

automation.example.com {
    encode gzip zstd

    header {
        Strict-Transport-Security "max-age=31536000; includeSubDomains; preload"
        X-Content-Type-Options "nosniff"
        X-Frame-Options "SAMEORIGIN"
        X-XSS-Protection "1; mode=block"
        Referrer-Policy "no-referrer-when-downgrade"
    }

    reverse_proxy n8n:5678 {
        header_up Host {host}
        header_up X-Real-IP {remote_host}
        header_up X-Forwarded-For {remote_host}
        header_up X-Forwarded-Proto {scheme}
    }
}

Reload the running Caddy service to pick up the updated directives:

docker compose exec -T caddy caddy reload --config /etc/caddy/Caddyfile

You can now test your domain in a Web browser. Caddy automatically provisions standard TLS certificates and transparently proxies traffic down to n8n over the internal Docker network (automation_net).

Monitoring, Log Aggregation, and Automated Backups

A production deployment requires log aggregation and back-up schedules for disaster recovery. If you are operating multiple services, review our full guide on Configuring Centralized Log Aggregation with Grafana Loki to ship your container outputs into centralized dashboards.

Creating an Automated PostgreSQL Backup Script

We will write a robust Python utility that connects to our database container, executes a database dump, compresses the output file, and enforces data retention by deleting files older than 14 days.

Create the backup script file at /opt/automation-stack/db_backup.py:

import os
import subprocess
import datetime
import glob

BACKUP_DIR = "/opt/automation-stack/backups"
CONTAINER_NAME = "postgres_db"
DB_USER = "n8n_db_user"
DB_NAME = "n8n_database"
RETENTION_DAYS = 14

def run_backup():
    if not os.path.exists(BACKUP_DIR):
        os.makedirs(BACKUP_DIR, mode=0o750)

    timestamp = datetime.datetime.now().strftime("%Y%m%d_%H%M%S")
    filename = f"n8n_backup_{timestamp}.sql.gz"
    filepath = os.path.join(BACKUP_DIR, filename)

    dump_cmd = (
        f"docker exec {CONTAINER_NAME} pg_dump -U {DB_USER} {DB_NAME} | gzip > {filepath}"
    )

    print(f"Starting backup: {filepath}")
    result = subprocess.run(dump_cmd, shell=True, capture_output=True, text=True)

    if result.returncode == 0:
        print("Database dump successfully created.")
    else:
        print(f"Error executing database dump: {result.stderr}")
        return

    cleanup_old_backups()

def cleanup_old_backups():
    now = datetime.datetime.now()
    pattern = os.path.join(BACKUP_DIR, "n8n_backup_*.sql.gz")
    backup_files = glob.glob(pattern)

    for file_path in backup_files:
        file_time = datetime.datetime.fromtimestamp(os.path.getmtime(file_path))
        age_days = (now - file_time).days
        if age_days > RETENTION_DAYS:
            try:
                os.remove(file_path)
                print(f"Removed expired backup: {file_path}")
            except OSError as e:
                print(f"Failed to remove file {file_path}: {e}")

if __name__ == "__main__":
    run_backup()

Make the script executable:

chmod +x /opt/automation-stack/db_backup.py

Automation via Systemd Service and Timer

Instead of relying on legacy cron daemons, we schedule our Python script using Systemd units for improved monitoring and process isolation. To learn more about native Linux scheduling, read our guide on How to Schedule Python Scripts With Systemd.

Create the systemd service unit file at /etc/systemd/system/n8n-backup.service:

[Unit]
Description=Automated PostgreSQL Backup for n8n Engine
After=docker.service
Requires=docker.service

[Service]
Type=oneshot
User=root
WorkingDirectory=/opt/automation-stack
ExecStart=/usr/bin/python3 /opt/automation-stack/db_backup.py
StandardOutput=journal
StandardError=journal

[Install]
WantedBy=multi-user.target

Create the corresponding systemd timer unit file at /etc/systemd/system/n8n-backup.timer:

[Unit]
Description=Run n8n PostgreSQL Backup Daily at 02:00 UTC

[Timer]
OnCalendar=*-*-* 02:00:00
Persistent=true

[Install]
WantedBy=timers.target

Reload the systemd manager configuration, enable the timer, and verify its target schedule execution:

sudo systemctl daemon-reload
sudo systemctl enable --now n8n-backup.timer
sudo systemctl list-timers --all | grep n8n-backup

You can run an immediate dry run execution of the backup unit to confirm system functionality:

sudo systemctl start n8n-backup.service
sudo journalctl -u n8n-backup.service -n 20 --no-pager

This stack gives you high-performance, self-hosted workflow execution on Hetzner VPS. It includes database isolation, HTTPS proxying via Caddy, automated database health management, and systemd backup tasks.

Getting Started

To implement this workflow infrastructure, acquire your base dependencies:

Reserve a high-performance cloud instance on Hetzner VPS or look into Contabo VPS / DigitalOcean.
Register your custom production domain with Namecheap.
Deploy using open-source engines or start with n8n Cloud before migrating to self-hosted servers.

Outsource Your Automation

Don't have time? I build production n8n workflows, WhatsApp bots, and fully automated YouTube Shorts pipelines. Hire me on Fiverr, mention SYS3-DEVTO for priority. Or DM at chasebot.online.

Originally published on Automation Insider.

Building Distributed Webhook Consumers with Redis Queues

Raizan — Sat, 29 Aug 2026 08:02:44 +0000

What You'll Need

To follow along with this tutorial, you will need the following tools and infrastructure:

A server instance hosted on a Hetzner VPS or Contabo VPS running Ubuntu 22.04 LTS
An alternative cloud host like DigitalOcean if you prefer managed droplets
Node.js (v18 or higher) and npm installed on your development machine or server
A running Redis instance (v6.x or higher)
A domain name registered via Namecheap for public webhook routing
An orchestration platform like n8n Cloud if you plan to plug consumer outputs directly into existing automation flows

Architecting a High-Throughput Webhook Ingestion Engine
Building the Ingestion API with Express and Redis BullMQ
Implementing Distributed Worker Processes with Idempotency
Securing and Scaling Your Queue Infrastructure
Getting Started

Architecting a High-Throughput Webhook Ingestion Engine

When third-party providers like Stripe, GitHub, or Shopify fire webhooks at your infrastructure, they expect an HTTP 200 OK status code within a strict timeout window (often 2 to 5 seconds). Processing heavy computational workflows, mutating database records, or sending notification emails directly inside the web request handler is a recipe for failure. Under traffic spikes, your application threads freeze, request queues fill up, and external vendors mark your endpoint as unresponsive, triggering aggressive retry loops that crash your system.

To build a enterprise-grade system, you must decouple request acceptance from task execution. The receiving API endpoint acts purely as a producer: it validates the cryptographic signature of the payload, pushes the raw event data onto an asynchronous queue, and returns an HTTP 202 Accepted response immediately. Distributed worker processes running across multiple hosts consume items from this queue out-of-band.

When deciding between task platforms, engineers often debate Temporal vs n8n for Enterprise Workflow Automation to handle backend orchestration. However, when you need ultra-low latency and custom queue lifecycle management, pairing Node.js with Redis and BullMQ offers the tightest control over memory overhead and execution context. Redis provides an in-memory datastructure store that handles tens of thousands of read and write operations per second, making it the perfect backplane for fast job buffering.

The core design requirements for a distributed webhook consumer system include:

Sub-50ms Response Times: Immediate response acknowledging payload receipt.
At-Least-Once Delivery Guarantees: Ensures no event is permanently lost if a worker process crashes mid-execution.
Strict Idempotency: Prevents duplicate processing of incoming events using unique idempotency keys.
Dead Letter Queue (DLQ) Handling: Captures permanently failing jobs for manual inspection without halting the queue processing pipeline.

Building the Ingestion API with Express and Redis BullMQ

Let us build the ingestion server using Express and BullMQ. In this step, deploy a lightweight Node.js web server hosted on a Hetzner VPS. The server verifies the HMAC signature of incoming webhooks to ensure authenticity, structures the job object, and places it inside a Redis-backed queue.

First, initialize your project and install the required dependencies:

mkdir webhook-engine
cd webhook-engine
npm init -y
npm install express bullmq ioredis dotenv crypto

Next, create the main entry point file named server.js. This code defines our producer instance, establishes a connection pool to Redis, and exposes a high-throughput endpoint.

const express = require('express');
const { Queue } = require('bullmq');
const Redis = require('ioredis');
const crypto = require('crypto');
require('dotenv').config();

const app = express();
app.use(express.json({
  verify: (req, res, buf) => {
    req.rawBody = buf;
  }
}));

const redisConnection = new Redis({
  host: process.env.REDIS_HOST || '127.0.0.1',
  port: parseInt(process.env.REDIS_PORT || '6379', 10),
  password: process.env.REDIS_PASSWORD || undefined,
  maxRetriesPerRequest: null,
});

const webhookQueue = new Queue('incoming-webhooks', {
  connection: redisConnection,
  defaultJobOptions: {
    attempts: 5,
    backoff: {
      type: 'exponential',
      delay: 2000,
    },
    removeOnComplete: {
      age: 86400,
      count: 10000,
    },
    removeOnFail: {
      age: 604800,
    },
  },
});

function verifySignature(req) {
  const secret = process.env.WEBHOOK_SECRET || 'supersecretkey';
  const signature = req.headers['x-signature'];
  if (!signature) {
    return false;
  }
  const hmac = crypto.createHmac('sha256', secret);
  const digest = hmac.update(req.rawBody).digest('hex');
  return crypto.timingSafeEqual(Buffer.from(signature), Buffer.from(digest));
}

app.post('/api/v1/webhooks', async (req, res) => {
  const startTime = Date.now();

  if (!verifySignature(req)) {
    return res.status(401).json({ error: 'Invalid HMAC signature' });
  }

  const eventType = req.body.event_type;
  const eventId = req.body.id;

  if (!eventType || !eventId) {
    return res.status(400).json({ error: 'Missing required event fields' });
  }

  try {
    await webhookQueue.add(
      eventType,
      {
        eventId: eventId,
        payload: req.body,
        receivedAt: new Date().toISOString(),
      },
      {
        jobId: eventId,
      }
    );

    const processingTimeMs = Date.now() - startTime;
    return res.status(202).json({
      status: 'queued',
      jobId: eventId,
      latencyMs: processingTimeMs,
    });
  } catch (error) {
    return res.status(500).json({ error: 'Failed to enqueue webhook payload' });
  }
});

const PORT = process.env.PORT || 3000;
app.listen(PORT, () => {
  console.log(`Webhook Ingestion Server active on port ${PORT}`);
});

This HTTP server accepts post requests, verifies the cryptographic signature in a timing-safe manner, and offloads the data to Redis in under 10 milliseconds.

Implementing Distributed Worker Processes with Idempotency

Once the ingestion API is enqueuing jobs, you need worker processes running across multiple nodes to pick up and process those jobs concurrently. Because network drops and worker restarts happen, duplicate processing can occur. To guard against double-charging accounts or firing duplicate notifications, workers must enforce idempotency using Redis key sets.

Create a worker file named worker.js. This script acts as a consumer process. You can spin up multiple instances of this process across different hosts, and BullMQ will manage concurrent task distribution automatically.

const { Worker } = require('bullmq');
const Redis = require('ioredis');
require('dotenv').config();

const redisConnection = new Redis({
  host: process.env.REDIS_HOST || '127.0.0.1',
  port: parseInt(process.env.REDIS_PORT || '6379', 10),
  password: process.env.REDIS_PASSWORD || undefined,
  maxRetriesPerRequest: null,
});

async function processPaymentSucceeded(payload) {
  console.log(`Processing payment_succeeded for order: ${payload.data.order_id}`);
  return { status: 'processed', orderId: payload.data.order_id };
}

async function processUserCreated(payload) {
  console.log(`Processing user_created for email: ${payload.data.email}`);
  return { status: 'processed', userId: payload.data.user_id };
}

async function handleWebhookJob(job) {
  const { eventId, payload } = job.data;
  const lockKey = `lock:webhook:${eventId}`;

  const acquired = await redisConnection.set(lockKey, 'processing', 'NX', 'EX', 3600);

  if (!acquired) {
    console.log(`Duplicate event skipped: ${eventId}`);
    return { status: 'skipped', reason: 'already_processed' };
  }

  try {
    let result;
    switch (job.name) {
      case 'payment.succeeded':
        result = await processPaymentSucceeded(payload);
        break;
      case 'user.created':
        result = await processUserCreated(payload);
        break;
      default:
        console.warn(`Unhandled event type: ${job.name}`);
        result = { status: 'ignored', reason: 'unknown_event_type' };
    }
    return result;
  } catch (error) {
    await redisConnection.del(lockKey);
    throw error;
  }
}

const worker = new Worker('incoming-webhooks', handleWebhookJob, {
  connection: redisConnection,
  concurrency: 10,
});

worker.on('completed', (job, result) => {
  console.log(`Job ${job.id} completed successfully. Result:`, result);
});

worker.on('failed', (job, err) => {
  console.error(`Job ${job.id} failed after ${job.attemptsMade} attempts. Error: ${err.message}`);
});

console.log('Distributed Webhook Worker started listening for jobs...');

The worker uses atomic Redis SET operations with NX (Set if Not Exists) and EX (Expiration in seconds) flags. If two instances grab the exact same job execution call simultaneously, only one instance successfully obtains the key lock.

Securing and Scaling Your Queue Infrastructure

Deploying queue producers and worker pools in a production environment requires setting up firewalls, SSL termination, and reverse proxies. Never expose your raw Node.js port or Redis instance to the public internet.

First, set up strict network boundaries on your cloud instance. When configuring hosting environments on a Contabo VPS or Hetzner VPS, secure your exposed infrastructure endpoints by Configuring UFW Firewall Rules On Linux Servers. Restrict public inbound traffic strictly to ports 80 and 443, locking down port 6379 (Redis) so it only accepts connections from internal node IP addresses.

Execute these commands on your application server to configure UFW:

sudo ufw default deny incoming
sudo ufw default allow outgoing
sudo ufw allow 22/tcp
sudo ufw allow 80/tcp
sudo ufw allow 443/tcp
sudo ufw allow from 10.0.0.0/16 to any port 6379
sudo ufw enable

Next, place Nginx in front of your Node.js application to handle SSL encryption, payload buffering, and domain termination. If you need step-by-step guidance on setting up HTTPS endpoint routing for your custom domain, consult our comprehensive guide on Configuring Nginx SSL Certificates for Custom Subdomains.

Use this Nginx configuration to proxy incoming traffic to your Express ingestion engine safely:

server {
    listen 80;
    server_name webhooks.yourdomain.com;
    return 301 https://$host$request_uri;
}

server {
    listen 443 ssl http2;
    server_name webhooks.yourdomain.com;

    ssl_certificate /etc/letsencrypt/live/webhooks.yourdomain.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/webhooks.yourdomain.com/privkey.pem;
    ssl_protocols TLSv1.2 TLSv1.3;

    client_max_body_size 5M;

    location / {
        proxy_pass http://127.0.0.1:3000;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection 'upgrade';
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_read_timeout 10s;
        proxy_connect_timeout 5s;
    }
}

This proxy configuration bounds total request body sizes to 5 megabytes, drops stale connection requests quickly, forwards real client IP addresses for auditing, and redirects unencrypted HTTP requests to HTTPS seamlessly.

Getting Started

To spin up your own enterprise ingestion network today:

Provision an Ubuntu cloud host on a Hetzner VPS, Contabo VPS, or DigitalOcean.
Point a domain name purchased on Namecheap to your server's static IP address.
Install Redis server and configure authentication passwords along with strict binding interfaces.
Clone your Express producer code and distributed BullMQ worker scripts onto your execution hosts.
Launch worker processes using a process supervisor like PM2 to maintain high operational uptime across server restarts.
Connect downstream webhooks directly to automation tools like n8n Cloud or custom backend microservices.

Outsource Your Automation

Don't have time? I build production n8n workflows, WhatsApp bots, and fully automated YouTube Shorts pipelines. Hire me on Fiverr, mention SYS3-DEVTO for priority. Or DM at chasebot.online.

Originally published on Automation Insider.

Configuring UFW Firewall Rules On Linux Servers

Raizan — Fri, 28 Aug 2026 08:02:49 +0000

What You'll Need

Hetzner VPS or Contabo VPS running Ubuntu 22.04 LTS or Debian 12
DigitalOcean droplet (alternative infrastructure provider)
Root or sudo privileges on your remote server
Namecheap registered domain name (if routing traffic for public web applications)

Understanding UFW Architecture and Default Policies
Setting Up Core Service Rules and Application Profiles
Advanced UFW Rule Management and IP Filtering
NAT Port Forwarding and UFW Log Management
Getting Started

Understanding UFW Architecture and Default Policies

Securing a Linux server starts with controlling network access. Uncomplicated Firewall (UFW) serves as a user-friendly frontend to iptables and nftables. It handles firewall rules without requiring you to master complex table syntax. When I provision a new instance on a Hetzner VPS or DigitalOcean, UFW is the very first defensive layer I enable.

Before enabling UFW, you must understand packet evaluation order and default behaviors. By default, UFW drops incoming connection requests and allows outgoing connection requests. If you turn UFW on without configuring an explicit SSH rule, you will immediately lock yourself out of your server.

UFW uses configuration files stored in /etc/default/ufw and /etc/ufw/. The main control file defines how default chains behave. You can inspect your current configuration file with this command:

cat /etc/default/ufw

The output defines system policies across all interfaces:

IPV6=yes
DEFAULT_INPUT_POLICY="DROP"
DEFAULT_OUTPUT_POLICY="ACCEPT"
DEFAULT_FORWARD_POLICY="DROP"
DEFAULT_APPLICATION_POLICY="SKIP"
MANAGE_BUILTINS=no
IPT_SYSCTL=/etc/ufw/sysctl.conf
MANAGE_IPS=yes

If your configuration shows DEFAULT_INPUT_POLICY="ACCEPT", modify it immediately using CLI commands. I prefer setting defaults explicitly through commands rather than manually editing the configuration file to prevent syntax errors.

Reset UFW to clean state if you want to eliminate existing rules:

sudo ufw reset

The terminal prompts for confirmation before wiping custom rules:

Resetting all rules to installed defaults. This may disrupt existing ssh connections. Proceed with operation (y|n)? y
Backing up 'user.rules' to '/etc/ufw/user.rules.20260330_120000'
Backing up 'before.rules' to '/etc/ufw/before.rules.20260330_120000'
Backing up 'after.rules' to '/etc/ufw/after.rules.20260330_120000'
Backing up 'user6.rules' to '/etc/ufw/user6.rules.20260330_120000'
Backing up 'before6.rules' to '/etc/ufw/before6.rules.20260330_120000'
Backing up 'after6.rules' to '/etc/ufw/after6.rules.20260330_120000'

Set default security policies across the system:

sudo ufw default deny incoming
sudo ufw default allow outgoing

This configuration ensures that any inbound traffic not explicitly allowed gets rejected, while internal applications can pull system updates, make API requests, and send outbound logs.

Setting Up Core Service Rules and Application Profiles

With default drop policies configured, you must open essential infrastructure ports before enabling the firewall daemon. Always permit SSH connections first to safeguard remote administration sessions.

To allow standard SSH over port 22:

sudo ufw allow 22/tcp

If your server runs SSH on a non-standard port like 2222 to reduce automated brute-force attempts, allow that custom port explicitly:

sudo ufw allow 2222/tcp

Public web platforms require open ports for standard HTTP and HTTPS traffic. If you are running web services, follow our detailed guide on Configuring Nginx Reverse Proxy with Certbot SSL to complete your public web stack setup. Open standard web ports with these commands:

sudo ufw allow 80/tcp
sudo ufw allow 443/tcp

UFW includes application profiles stored in /etc/ufw/applications.d/. These profiles map human-readable application names directly to specified protocol ports. Check available application profiles on your system:

sudo ufw app list

You will see output similar to this:

Available applications:
  Nginx Full
  Nginx HTTP
  Nginx HTTPS
  OpenSSH

Allow an entire profile directly by specifying its name:

sudo ufw allow "Nginx Full"

If you manage custom backend systems, you can define your own application profile. For instance, if you are Deploying Self-Hosted PocketBase on Cloud Servers, you can create a profile for PocketBase on port 8090. Create a file named /etc/ufw/applications.d/pocketbase:

[PocketBase]
title=PocketBase Backend Server
description=Self-contained database and real-time backend engine.
ports=8090/tcp

Verify your custom application profile is loaded:

sudo ufw app info PocketBase

The console displays the profile parameters:

Profile: PocketBase
Title: PocketBase Backend Server
Description: Self-contained database and real-time backend engine.

Port:
  8090/tcp

Allow the new profile through UFW:

sudo ufw allow PocketBase

Now enable UFW. Review your terminal carefully because enabling the service activates all rules immediately:

sudo ufw enable

Confirm activation status:

Command may disrupt existing ssh connections. Proceed with operation (y|n)? y
Firewall is active and enabled on system startup

Verify the detailed list of active rules:

sudo ufw status verbose

Status: active
Logging: on (low)
Default: deny (incoming), allow (outgoing), disabled (routed)
New profiles: loaded

To                         Action      From
--                         ------      ----
22/tcp                     ALLOW IN    Anywhere                  
80/tcp (Nginx Full)        ALLOW IN    Anywhere                  
443/tcp (Nginx Full)       ALLOW IN    Anywhere                  
8090/tcp (PocketBase)      ALLOW IN    Anywhere                  
22/tcp (v6)                ALLOW IN    Anywhere (v6)             
80/tcp (Nginx Full (v6))   ALLOW IN    Anywhere (v6)             
443/tcp (Nginx Full (v6))  ALLOW IN    Anywhere (v6)             
8090/tcp (PocketBase (v6)) ALLOW IN    Anywhere (v6)

Advanced UFW Rule Management and IP Filtering

Broad rule definitions allowing full access from anywhere are acceptable for public websites, but administrative endpoints require strict network boundaries. Restricting infrastructure access to single static IPs or specific subnets minimizes exposure.

Allow connection to SSH (port 22) only from a trusted administrator IP address 198.51.100.45:

sudo ufw allow from 198.51.100.45 to any port 22 proto tcp

Allow an internal corporate subnet 192.168.1.0/24 access to a self-hosted PostgreSQL database on port 5432:

sudo ufw allow from 192.168.1.0/24 to any port 5432 proto tcp

If your server features multiple network interfaces (such as eth0 for external traffic and eth1 or tailscale0 for internal traffic), bind rules directly to specific network adapters.

Allow access to MySQL port 3306 exclusively over internal interface eth1:

sudo ufw allow in on eth1 to any port 3306 proto tcp

Deny traffic explicitly from malicious or offending subnets:

sudo ufw deny from 203.0.113.0/24

To prevent automated scripts from hammering exposed endpoints, use built-in rate limiting. UFW limit rules drop connections from IP addresses that attempt 6 or more connections within a 30-second window:

sudo ufw limit 22/tcp

Managing rules over time requires working with rule numbers. List active rules sequentially:

sudo ufw status numbered

The output shows numbered indices:

Status: active

     To                         Action      From
     --                         ------      ----
[ 1] 22/tcp                     LIMIT IN    Anywhere                  
[ 2] 80/tcp                     ALLOW IN    Anywhere                  
[ 3] 443/tcp                    ALLOW IN    Anywhere                  
[ 4] 5432/tcp                   ALLOW IN    192.168.1.0/24            
[ 5] 203.0.113.0/24             DENY IN     Anywhere

Delete a specific rule by supplying its exact index number:

sudo ufw delete 5

Insert a high-priority rule at position 1 to make sure evaluation takes place before downstream wildcard rules:

sudo ufw insert 1 allow from 198.51.100.50 to any port 22 proto tcp

Check the updated numbered list to verify rule order:

sudo ufw status numbered

Status: active

     To                         Action      From
     --                         ------      ----
[ 1] 22/tcp                     ALLOW IN    198.51.100.50             
[ 2] 22/tcp                     LIMIT IN    Anywhere                  
[ 3] 80/tcp                     ALLOW IN    Anywhere                  
[ 4] 443/tcp                    ALLOW IN    Anywhere                  
[ 5] 5432/tcp                   ALLOW IN    192.168.1.0/24

NAT Port Forwarding and UFW Log Management

UFW can operate as a edge router or internal proxy by forwarding incoming packets on external interfaces directly to underlying private networks, container networks, or virtual machines.

First, enable kernel-level IP forwarding. Open /etc/ufw/sysctl.conf:

sudo nano /etc/ufw/sysctl.conf

Uncomment or insert these kernel variables to allow packet routing across interfaces:

net/ipv4/ip_forward=1
net/ipv6/conf/default/forwarding=1
net/ipv6/conf/all/forwarding=1

Next, enable packet forwarding in /etc/default/ufw:

sudo nano /etc/default/ufw

Change DEFAULT_FORWARD_POLICY from "DROP" to "ACCEPT":

DEFAULT_FORWARD_POLICY="ACCEPT"

Configure Network Address Translation (NAT) rules inside /etc/ufw/before.rules. Place your NAT definitions before the standard filter table configuration.

Open /etc/ufw/before.rules:

sudo nano /etc/ufw/before.rules

Add the complete NAT block at the very top of the file, above the default *filter line:

# NAT table configuration
*nat
:PREROUTING ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]

# Forward traffic coming into port 8080 on eth0 to internal container IP 10.8.0.5 on port 80
-A PREROUTING -i eth0 -p tcp --dport 8080 -j DNAT --to-destination 10.8.0.5:80

# Masquerade internal network traffic egressing out eth0
-A POSTROUTING -s 10.8.0.0/24 -o eth0 -j MASQUERADE

COMMIT

Reload UFW to apply kernel routing and NAT rule structures:

sudo ufw reload

Firewall visibility is essential for tracking suspicious activity and auditing connection patterns. Manage logging levels with the logging command:

sudo ufw logging medium

Available logging levels include:

off: Disables logging entirely.
low: Logs blocked packets that match default policies or explicit block rules.
medium: Logs low level activity plus allowed connections, invalid packets, and new connections.
high: Logs everything with rate limiting.
full: Logs all packets without rate limits (generates massive log volume).

UFW writes packet decision logs directly to /var/log/ufw.log. Monitor raw log entries in real-time:

sudo tail -f /var/log/ufw.log

Log lines contain fields detailing matched rules, source networks, and ports:

Mar 30 12:45:10 server01 kernel: [UFW BLOCK] IN=eth0 OUT= MAC=52:54:00:12:34:56:52:54:00:65:43:21:08:00 SRC=198.51.100.99 DST=192.0.2.1 LEN=40 TOS=0x00 PREC=0x00 TTL=245 ID=54321 PROTO=TCP SPT=44123 DPT=23 WINDOW=65535 SYN URGP=0

Field definitions:

IN: Interface where packet entered the host machine (eth0).
SRC: Originating IP address of the traffic payload (198.51.100.99).
DST: Target IP address belonging to your server (192.0.2.1).
SPT: Source port assigned by requesting client (44123).
DPT: Destination port targeted on your host server (23 / Telnet).
SYN: Indicates an initial connection establishment attempt.

For production clusters, parsing raw text files manually on each server quickly becomes unmanageable. Standardize your infrastructure monitoring by Configuring Centralized Log Aggregation with Grafana Loki to stream /var/log/ufw.log entries directly into a central dashboard for real-time security alerts and threat analysis.

Getting Started

Building a secure Linux server starts with choosing host infrastructure, locking down unused ports, and auditing access controls.

Deploy a secure server instance using a provider like Hetzner VPS, Contabo VPS, or a DigitalOcean droplet.
Domain resolution for web services can be managed via Namecheap.
Connect over SSH and apply default deny policies immediately.
Open administrative SSH and web application ports before turning UFW on.
Set up logging to monitor unexpected traffic patterns.

Outsource Your Automation

Don't have time? I build production n8n workflows, WhatsApp bots, and fully automated YouTube Shorts pipelines. Hire me on Fiverr, mention SYS3-DEVTO for priority. Or DM at chasebot.online.

Originally published on Automation Insider.

Configuring Nginx SSL Certificates for Custom Subdomains

Raizan — Thu, 27 Aug 2026 08:01:43 +0000

What You'll Need

Before diving into this guide, ensure you have access to the following resources:

Hetzner VPS or Contabo VPS running Ubuntu 22.04 LTS or 24.04 LTS
DigitalOcean Droplet as an alternative Linux host
Namecheap or your preferred domain registrar with access to DNS records
n8n Cloud or a self-hosted automation server for backend webhook endpoints

Understanding Subdomain Architecture and Wildcard Certificates
Setting Up DNS and Nginx Core Configurations
Generating Wildcard SSL Certificates with Certbot DNS Plugin
Dynamic Subdomain Routing and Automated SSL for Multi-Tenant Services
Getting Started

Understanding Subdomain Architecture and Wildcard Certificates

Deploying microservices, multi-tenant web applications, or workflow automation platforms often requires dedicated subdomains. Managing individual TLS/SSL certificates for api.example.com, app.example.com, and webhook.example.com becomes tedious quickly. Every new subdomain forces you to provision a new HTTP-01 challenge, open firewall ports, and update web server configurations.

Wildcard certificates solve this operational overhead. A single certificate issued for *.example.com secures any first-level subdomain under your domain name. Combined with Nginx as a reverse proxy, you can rapidly spin up backend microservices without issuing a new certificate every single time.

To issue a wildcard certificate through Let's Encrypt, you cannot use standard HTTP-01 verification because Let's Encrypt needs proof that you control the entire domain zone. Instead, you must use the DNS-01 challenge. This validation method requires inserting a specific TXT record into your domain's DNS zone file.

When you scale this setup, manually creating TXT records isn't viable. We automate this process using Certbot DNS plugins, which interact directly with your provider's API to handle creation, verification, and cleanup automatically during renewals.

Setting Up DNS and Nginx Core Configurations

To begin, point your domain and its wildcard subdomains to your server. Log into your account on Namecheap or your chosen DNS host and create two A records:

@ pointing to your server's public IP address.
* pointing to your server's public IP address.

This setup routes all requests for non-explicitly defined subdomains directly to your Nginx proxy running on your Hetzner VPS.

Next, connect to your server via SSH and install Nginx alongside the necessary system utilities.

sudo apt update
sudo apt install -y nginx certbot python3-certbot-nginx curl ufw

Enable HTTP and HTTPS traffic through the Uncomplicated Firewall (UFW):

sudo ufw allow 'Nginx Full'
sudo ufw allow OpenSSH
sudo ufw enable

Before obtaining the wildcard certificate, establish a foundational Nginx server configuration. This initial layout sets up a base HTTP server block designed to force all incoming connections to redirect to HTTPS once certificates are installed. If you are unfamiliar with fundamental reverse proxy architecture, read our guide on Configuring Nginx Reverse Proxy with Certbot SSL to understand standard single-domain configurations.

Create a baseline proxy site file at /etc/nginx/sites-available/default-wildcard.conf:

server {
    listen 80;
    listen [::]:80;
    server_name example.com *.example.com;

    location /.well-known/acme-challenge/ {
        root /var/www/html;
    }

    location / {
        return 301 https://$host$request_uri;
    }
}

Link this file to the active directory and remove the stock default file:

sudo ln -s /etc/nginx/sites-available/default-wildcard.conf /etc/nginx/sites-enabled/
sudo rm -f /etc/nginx/sites-enabled/default
sudo nginx -t
sudo systemctl reload nginx

Generating Wildcard SSL Certificates with Certbot DNS Plugin

Because HTTP-01 validation cannot issue *.example.com certificates, we use the DNS-01 challenge. Certbot offers plugins for major DNS providers including Cloudflare, DigitalOcean, Route53, and Namecheap.

In this example, we will install and configure the Cloudflare DNS plugin, which is widely used for automated DNS management. If you host DNS directly on DigitalOcean, you can install python3-certbot-dns-digitalocean instead.

Install the plugin:

sudo apt install -y python3-certbot-dns-cloudflare

Create a secure directory for API credentials:

sudo mkdir -p /etc/letsencrypt/secrets
sudo chmod 700 /etc/letsencrypt/secrets

Create the credentials file at /etc/letsencrypt/secrets/cloudflare.ini:

dns_cloudflare_api_token = 1234567890abcdef1234567890abcdef12345678

Set strict read/write permissions on the file to prevent unauthorized access to your API key:

sudo chmod 600 /etc/letsencrypt/secrets/cloudflare.ini

Run Certbot to request the wildcard certificate. Replace example.com with your real domain:

sudo certbot certonly \
  --dns-cloudflare \
  --dns-cloudflare-credentials /etc/letsencrypt/secrets/cloudflare.ini \
  --dns-cloudflare-propagation-seconds 30 \
  -d example.com \
  -d '*.example.com' \
  --agree-tos \
  -m admin@example.com \
  --no-eff-email

Certbot creates key files in /etc/letsencrypt/live/example.com/:

fullchain.pem: Your certificate bundled with intermediate certificates.
privkey.pem: Your private key.

Verify that automatic renewal works without manual intervention:

sudo certbot renew --dry-run

Because wildcards secure sensitive endpoint structures like webhook consumers and API endpoints, security must extend beyond TLS. Ensure your applications validate payload integrity when exposed publicly. For details on securing publicly reachable endpoints, see our walkthrough on Securing Incoming Webhooks with HMAC Signature Verification.

Dynamic Subdomain Routing and Automated SSL for Multi-Tenant Services

Once your wildcard certificate is in place, you can route traffic dynamically to multiple backend services based on the requested host header. This eliminates the need to edit Nginx files whenever a new subdomain is added.

Here is a full, production-ready Nginx configuration that handles dynamic upstream routing, forces TLS 1.2 and 1.3, applies secure headers, and maps specific subdomains to internal microservice ports.

Create the file /etc/nginx/sites-available/wildcard-proxy.conf:

map $host $backend_port {
    default 8080;
    app.example.com 3000;
    api.example.com 4000;
    n8n.example.com 5678;
    webhooks.example.com 9000;
}

server {
    listen 80;
    listen [::]:80;
    server_name example.com *.example.com;

    return 301 https://$host$request_uri;
}

server {
    listen 443 ssl http2;
    listen [::]:443 ssl http2;
    server_name example.com *.example.com;

    ssl_certificate /etc/letsencrypt/live/example.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/example.com/privkey.pem;

    ssl_protocols TLSv1.2 TLSv1.3;
    ssl_ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384;
    ssl_prefer_server_ciphers off;

    ssl_session_timeout 1d;
    ssl_session_cache shared:SSL:10m;
    ssl_session_tickets off;

    ssl_stapling on;
    ssl_stapling_verify on;
    ssl_trusted_certificate /etc/letsencrypt/live/example.com/chain.pem;
    resolver 1.1.1.1 8.8.8.8 valid=300s;
    resolver_timeout 5s;

    add_header X-Frame-Options "SAMEORIGIN" always;
    add_header X-XSS-Protection "1; mode=block" always;
    add_header X-Content-Type-Options "nosniff" always;
    add_header Referrer-Policy "no-referrer-when-downgrade" always;
    add_header Content-Security-Policy "default-src 'self' http: https: data: blob: 'unsafe-inline'" always;
    add_header Strict-Transport-Security "max-age=31536000; includeSubDomains; preload" always;

    access_log /var/log/nginx/wildcard_access.log;
    error_log /var/log/nginx/wildcard_error.log warn;

    location / {
        proxy_pass http://127.0.0.1:$backend_port;
        proxy_http_version 1.1;

        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header X-Forwarded-Host $host;
        proxy_set_header X-Forwarded-Port $server_port;

        proxy_connect_timeout 60s;
        proxy_send_timeout 60s;
        proxy_read_timeout 60s;
        proxy_buffering off;
    }
}

Enable the configuration and reload Nginx:

sudo rm -f /etc/nginx/sites-enabled/default-wildcard.conf
sudo ln -s /etc/nginx/sites-available/wildcard-proxy.conf /etc/nginx/sites-enabled/
sudo nginx -t
sudo systemctl reload nginx

With this pattern, adding a new service to your infrastructure requires only two steps:

Spin up your service on an internal port (for example, port 5000 for a new service).
Add service.example.com 5000; inside the Nginx map block and reload Nginx.

No new SSL certificates are required, and zero domain challenge delays are introduced.

This backend proxy configuration integrates directly with automation environments. For instance, if you run automation software like n8n Cloud or self-host n8n on n8n.example.com, you can seamlessly route webhook triggers to business tools. Learn how to leverage automated webhook triggers by checking out our guide to Automate WhatsApp Business Messages with n8n and 360dialog.

To verify your SSL setup, test your SSL deployment using curl:

curl -I -v https://app.example.com

The output should show HTTP/2 support, an active TLS 1.3 connection, valid certificate authority details from Let's Encrypt, and proper security response headers.

Getting Started

To implement this architecture on your server, assemble your hosting environment and configure your records:

Provision a high-performance VPS on Hetzner VPS or Contabo VPS.
Domain owners can quickly link DNS zones via Namecheap or manage infrastructure components directly using DigitalOcean.
Connect your applications or self-hosted services using n8n Cloud for workflow automation across subdomains.

Outsource Your Automation

Don't have time? I build production n8n workflows, WhatsApp bots, and fully automated YouTube Shorts pipelines. Hire me on Fiverr, mention SYS3-DEVTO for priority. Or DM at chasebot.online.

Originally published on Automation Insider.

Handling Proxy Rotation in Web Scraping Pipelines

Raizan — Wed, 26 Aug 2026 08:02:22 +0000

What You'll Need

n8n Cloud or self-hosted n8n
Hetzner VPS or Contabo VPS for hosting
DigitalOcean as an alternative hosting platform
Namecheap if custom proxy gateway domains are required
Make.com for lightweight automation comparisons

Architecture of a Resilient Scraping Pipeline
Building a Python Proxy Manager with Health Checks
Integrating Proxy Rotation in n8n and Workflow Engines
Managing Data Pipelines and Database Synchronization
Getting Started

Architecture of a Resilient Scraping Pipeline

Scraping data at scale requires a clear understanding of network defenses, rate limits, and target architecture. When you send thousands of requests per minute from a single IP address, web application firewalls (WAFs) like Cloudflare, AWS WAF, and Akamai will flag and ban your origin server almost instantly. To bypass these restrictions, your pipeline must dynamically switch egress IP addresses using proxy rotation.

Proxy services generally fall into three categories: datacenter proxies, residential proxies, and mobile proxies. Datacenter proxies are cheap and fast, but their IP ranges belong to known hosting providers like Amazon Web Services or DigitalOcean, making them easy to block. Residential proxies route traffic through consumer internet service providers, offering high trust scores at a higher cost per gigabyte. Mobile proxies use cellular networks, offering the highest trust scores because thousands of real devices share single gateway IPs.

A production-grade scraping pipeline requires more than just buying a list of proxies. You must manage session stickiness, monitor health metrics, enforce exponential backoff, and rotate credentials dynamically. If a proxy returns a status code of 403 Forbidden, 429 Too Many Requests, or 503 Service Unavailable, your system must immediately mark that specific proxy as degraded, isolate it from active rotation, and retry the payload through a healthy endpoint.

       +-----------------------------------------------+
       |             Scraping Orchestrator             |
       +-----------------------+-----------------------+
                               |
                               v
       +-----------------------------------------------+
       |       Dynamic Proxy Manager & Health Check    |
       +-------+---------------+---------------+-------+
               |               |               |
               v               v               v
       +---------------+---------------+---------------+
       | Datacenter IP | Residential IP|   Mobile IP   |
       +-------+---------------+---------------+-------+
               |               |               |
               +-------+-------+---------------+
                       |
                       v
       +-----------------------------------------------+
       |               Target Web Server               |
       +-----------------------------------------------+

When building enterprise automation systems, pipeline orchestration becomes critical. Choosing between dedicated code-based workers and visual execution platforms depends on workload volume, monitoring needs, and infrastructure complexity, as outlined in my comparison of Temporal vs n8n vs Make for Enterprise Automation. For web scraping, combining high-throughput code modules with robust visual orchestrators provides visibility and reliability.

Building a Python Proxy Manager with Health Checks

To handle high-volume scraping without manual intervention, you need an asynchronous proxy pool manager that tracks success rates, handles failures, and enforces cooldown periods for banned IPs. If you run your scrapers on a dedicated server hosted on a Hetzner VPS, you can easily run thousands of concurrent asynchronous network requests.

Below is a complete, production-ready Python script using httpx and asyncio. It defines a ProxyManager class that maintains state for every proxy, rotates them in a round-robin format, tracks failed requests, and temporally cools down proxies that hit rate limits.

import asyncio
import time
import httpx
from typing import List, Dict, Optional

class ProxyNode:
    def __init__(self, url: str):
        self.url: str = url
        self.total_requests: int = 0
        self.failed_requests: int = 0
        self.is_banned: bool = False
        self.cooldown_until: float = 0.0

    def mark_success(self):
        self.total_requests += 1

    def mark_failure(self, ban_duration: int = 300):
        self.total_requests += 1
        self.failed_requests += 1
        if self.failed_requests / self.total_requests > 0.3 and self.total_requests >= 5:
            self.is_banned = True
            self.cooldown_until = time.time() + ban_duration

    def is_available(self) -> bool:
        if self.is_banned:
            if time.time() > self.cooldown_until:
                self.is_banned = False
                self.failed_requests = 0
                return True
            return False
        return True

class ProxyManager:
    def __init__(self, proxy_urls: List[str]):
        self.nodes: List[ProxyNode] = [ProxyNode(url) for url in proxy_urls]
        self.index: int = 0

    def get_next_proxy(self) -> Optional[ProxyNode]:
        total_nodes = len(self.nodes)
        for _ in range(total_nodes):
            node = self.nodes[self.index]
            self.index = (self.index + 1) % total_nodes
            if node.is_available():
                return node
        return None

async def fetch_page(client: httpx.AsyncClient, target_url: str, proxy_manager: ProxyManager) -> Optional[str]:
    max_retries = 3
    for attempt in range(max_retries):
        proxy_node = proxy_manager.get_next_proxy()
        if not proxy_node:
            await asyncio.sleep(2.0)
            continue

        try:
            proxies = {"http://": proxy_node.url, "https://": proxy_node.url}
            async with httpx.AsyncClient(proxies=proxies, timeout=10.0) as proxy_client:
                response = await proxy_client.get(
                    target_url,
                    headers={"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64)"}
                )

                if response.status_code == 200:
                    proxy_node.mark_success()
                    return response.text
                elif response.status_code in [403, 429, 503]:
                    proxy_node.mark_failure(ban_duration=600)
                else:
                    proxy_node.mark_failure(ban_duration=60)
        except httpx.RequestError:
            proxy_node.mark_failure(ban_duration=120)

        await asyncio.sleep(1.0 * (attempt + 1))
    return None

async def main():
    proxy_list = [
        "http://user:pass@192.168.1.10:8080",
        "http://user:pass@192.168.1.11:8080",
        "http://user:pass@192.168.1.12:8080"
    ]
    manager = ProxyManager(proxy_list)
    targets = [
        "https://httpbin.org/ip",
        "https://httpbin.org/user-agent",
        "https://httpbin.org/headers"
    ]

    async with httpx.AsyncClient() as base_client:
        tasks = [fetch_page(base_client, url, manager) for url in targets]
        results = await asyncio.gather(*tasks)

    for idx, content in enumerate(results):
        if content:
            print(f"Successfully fetched Target {idx + 1}")
        else:
            print(f"Failed to fetch Target {idx + 1}")

if __name__ == "__main__":
    asyncio.run(main())

This Python script guarantees that problematic endpoints are quarantined automatically without stopping the overall data collection loop. When designing large systems, you can easily integrate this script with orchestration platforms to coordinate scheduled runs.

Integrating Proxy Rotation in n8n and Workflow Engines

While code scripts work well for targeted extraction, managing stateful retries, Webhooks, database writes, and external alerts is often easier in an orchestration tool. Using self-hosted n8n Cloud or an instance running on a virtual server gives you full control over custom JavaScript execution nodes.

When choosing workflow platforms for headless data collection, evaluating developer flexibility versus execution constraints is critical. You can read more about API execution trade-offs in our guide on Temporal vs Make for API-First Workflows.

In n8n, you can implement an in-memory proxy rotator directly inside a Code Node. The JavaScript code below accepts an array of target URLs along with a pre-configured list of proxy endpoints. It evaluates status codes, rotates proxies per request item, and constructs retry payloads automatically.

const items = $input.all();
const proxyList = [
  "http://proxy1.example.com:8080",
  "http://proxy2.example.com:8080",
  "http://proxy3.example.com:8080"
];

const staticData = $getWorkflowStaticData('global');
if (!staticData.proxyIndex) {
  staticData.proxyIndex = 0;
}
if (!staticData.proxyStats) {
  staticData.proxyStats = {};
}

const processedItems = [];

for (let i = 0; i < items.length; i++) {
  const item = items[i].json;
  let attempts = 0;
  let success = false;
  let responseData = null;
  let lastError = null;

  while (attempts < 3 && !success) {
    attempts++;

    let currentIndex = staticData.proxyIndex % proxyList.length;
    let selectedProxy = proxyList[currentIndex];
    staticData.proxyIndex++;

    if (!staticData.proxyStats[selectedProxy]) {
      staticData.proxyStats[selectedProxy] = { failures: 0, successes: 0 };
    }

    try {
      const response = await this.helpers.httpRequest({
        method: 'GET',
        url: item.targetUrl,
        proxy: selectedProxy,
        headers: {
          'Accept': 'text/html,application/xhtml+xml',
          'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36'
        },
        returnFullResponse: true,
        ignoreHttpStatusErrors: true
      });

      if (response.statusCode === 200) {
        success = true;
        responseData = response.body;
        staticData.proxyStats[selectedProxy].successes++;
      } else {
        staticData.proxyStats[selectedProxy].failures++;
        lastError = `HTTP Status ${response.statusCode}`;
      }
    } catch (error) {
      staticData.proxyStats[selectedProxy].failures++;
      lastError = error.message;
    }
  }

  processedItems.push({
    json: {
      targetUrl: item.targetUrl,
      success: success,
      data: responseData,
      attempts: attempts,
      error: lastError
    }
  });
}

return processedItems;

This node pattern stores the execution pointer within n8n workflow static data. The proxy index persists across loop iterations, preventing traffic spikes on any single IP address while preserving fault isolation.

Managing Data Pipelines and Database Synchronization

Scraped data must be validated, transformed, and written to persistent storage reliably. If you operate scrapers across multiple nodes or regions, write operations should hit a primary database, while sync engines read state across operational replicas.

When configuring target infrastructure on cloud providers like DigitalOcean, set up a single primary relational database alongside dedicated read replicas. You can learn how to replicate data efficiently without locking active tables by following our guide on How to Set Up PostgreSQL Logical Replication.

Below is a complete SQL schema and ingestion module designed to track proxy performance, retain request history, and log raw web scraping payloads into PostgreSQL:

CREATE TABLE IF NOT EXISTS proxy_registry (
    id SERIAL PRIMARY KEY,
    proxy_address VARCHAR(255) UNIQUE NOT NULL,
    status VARCHAR(50) DEFAULT 'active',
    total_requests INT DEFAULT 0,
    failed_requests INT DEFAULT 0,
    created_at TIMESTAMP WITH TIMEZONE DEFAULT CURRENT_TIMESTAMP
);

CREATE TABLE IF NOT EXISTS scraped_payloads (
    id SERIAL PRIMARY KEY,
    target_url TEXT NOT NULL,
    proxy_used VARCHAR(255),
    http_status INT NOT NULL,
    payload_body TEXT,
    scraped_at TIMESTAMP WITH TIMEZONE DEFAULT CURRENT_TIMESTAMP
);

CREATE OR REPLACE FUNCTION record_scrape_result(
    p_target_url TEXT,
    p_proxy_address VARCHAR(255),
    p_http_status INT,
    p_payload_body TEXT
) RETURNS VOID AS $$
BEGIN
    INSERT INTO scraped_payloads (target_url, proxy_used, http_status, payload_body)
    VALUES (p_target_url, p_proxy_address, p_http_status, p_payload_body);

    INSERT INTO proxy_registry (proxy_address, total_requests, failed_requests)
    VALUES (
        p_proxy_address, 
        1, 
        CASE WHEN p_http_status != 200 THEN 1 ELSE 0 END
    )
    ON CONFLICT (proxy_address) DO UPDATE SET
        total_requests = proxy_registry.total_requests + 1,
        failed_requests = proxy_registry.failed_requests + CASE WHEN p_http_status != 200 THEN 1 ELSE 0 END,
        status = CASE 
            WHEN (proxy_registry.failed_requests + CASE WHEN p_http_status != 200 THEN 1 ELSE 0 END)::FLOAT / 
                 (proxy_registry.total_requests + 1)::FLOAT > 0.4 
            THEN 'degraded' 
            ELSE 'active' 
        END;
END;
$$ LANGUAGE plpgsql;

Integrating this database logic guarantees that bad proxies are logged continuously. If an IP proxy provider delivers high error rates, your monitoring queries will flag those addresses for removal automatically.

Getting Started

To implement a bulletproof web scraping pipeline in your own production environment, follow these steps:

Provision server infrastructure on a high-bandwidth platform like Hetzner VPS or Contabo VPS.
Set up n8n Cloud or install a self-hosted instance to orchestrate scraping schedules and data ingestion routines.
Configure your domain records on Namecheap if you plan to expose API endpoints or dynamic proxy gateways.
Deploy the Python ProxyManager or n8n Code Node patterns provided above, along with automatic failure monitoring in PostgreSQL.

Outsource Your Automation

Don't have time? I build production n8n workflows, WhatsApp bots, and fully automated YouTube Shorts pipelines. Hire me on Fiverr, mention SYS3-DEVTO for priority. Or DM at chasebot.online.

Originally published on Automation Insider.

DEV Community: Raizan

How to Stream Container Logs to Loki

What You'll Need

Table of Contents

Architecture Overview: Docker, Loki, and Promtail

Step 1: Deploying Grafana Loki and Grafana Stack

Step 2: Configuring Promtail Docker Engine Scraping

Step 3: Streaming Docker Logs via Native Docker Logging Plugin

Step 4: Querying Docker Logs in Grafana using LogQL

Setting Up the Loki Data Source

Querying Logs with LogQL

Getting Started

Outsource Your Automation

Building Reliable Structured Output Pipelines with OpenAI

What You'll Need

Table of Contents

Why Structured Outputs Matter for Automated Pipelines

Defining Strict Schemas with Pydantic and OpenAI

Building an n8n Data Transformation and Parsing Pipeline

Error Handling Retries and Self Correction Loops

Getting Started

Outsource Your Automation

Handling Telegram Bot Rate Limits with Redis

What You'll Need

Table of Contents

Understanding Telegram Bot Rate Limits

Architecture: Sliding Window and Rate Limiting with Redis

Setting Up Docker and Redis Infrastructure

Implementing the Redis Rate Limiter in Node.js

Building a High-Throughput Queue with BullMQ

Getting Started

Outsource Your Automation

Setting Up PostgreSQL Read Replicas with Docker

What You'll Need

Table of Contents

Understanding PostgreSQL Streaming Replication

Configuring the Primary PostgreSQL Instance

Writing the Replica Bootstrapping Engine

Building the Production Docker Compose Stack

Validating Replication and Monitoring Delay

1. Verify Active Replication on Primary

2. Confirm Standby Mode on Replicas

3. Test Read-Only Constraints and Write Syncing

4. Measuring Replication Lag

Getting Started

Outsource Your Automation

How to Deploy Watchtower for Docker Containers

What You'll Need

Table of Contents

Understanding Watchtower for Docker Deployments

Deploying Watchtower via Docker Run and Docker Compose

Granular Container Control with Labels and Scopes

Setting Up Real-Time Alerting and Notifications

Hardening Watchtower for Production Security

Getting Started

Outsource Your Automation

How to Schedule Python Tasks using APScheduler

What You'll Need

Table of Contents

Understanding APScheduler Core Architecture

Building a Production-Ready BackgroundScheduler with SQLite

CronTriggers, Dynamic Job Addition, and State Persistence

Deploying APScheduler with PostgreSQL and Docker

File 1: main.py

File 2: Dockerfile

File 3: requirements.txt

File 4: docker-compose.yml

Managing Misfires, Max Instances, and Coalescing

Getting Started

Outsource Your Automation

Securing Inbound Webhooks Using Token Bucket Rate Limiting

What You'll Need

Table of Contents

Understanding the Token Bucket Algorithm for Webhooks

Building a Redis-Backed Token Bucket Engine in Node.js

Integrating Rate Limiting with Automated Orchestration Workflows

Load Testing and Stress Testing Your Rate Limiter

Getting Started

Outsource Your Automation

Deploying Open Source Workflow Systems On Hetzner