DEV Community

Cover image for Configuring Vector for Centralized Log Aggregation
Raizan
Raizan

Posted on Originally published at chasebot.online

Configuring Vector for Centralized Log Aggregation

#devops #sysadmin #logging #observability

What You'll Need

  • A cloud server such as a Hetzner VPS or Contabo VPS running Ubuntu 22.04 LTS or higher
  • An alternative cloud host like DigitalOcean if you prefer managed droplet infrastructure
  • A registered domain name managed through Namecheap for public aggregator endpoints requiring TLS termination
  • An automated workflow platform like n8n Cloud to trigger operational alerts based on aggregated log anomalies

Table of Contents

Why Vector for Centralized Log Aggregation?

Managing distributed logs across dozens of microservices, application containers, and databases quickly becomes an operational bottleneck. Traditional log shippers like Logstash or Fluentd often require significant memory overhead and introduce noticeable latency when processing high-volume streams.

Vector, written in Rust, solves this by offering high-throughput performance with minimal CPU and memory footprints. It acts as a unified data pipeline that can collect, transform, and route logs, metrics, and traces. Vector provides explicit buffer management, memory safety guarantees, and a custom transformation language called Vector Remap Language (VRL) that eliminates the need for cumbersome Regex pipelines.

When designing a production-grade telemetry topology, the best approach uses a two-tier architecture:

  1. Edge Agents: Lightweight Vector instances deployed on individual application nodes to capture host logs, Docker container streams, and system metrics with minimal resource usage.
  2. Centralized Aggregator: A dedicated, scalable Vector cluster that receives log streams from edge agents, performs heavy enrichment, structures data, and routes it to long-term sinks like ClickHouse, Elasticsearch, AWS S3, or Grafana Loki.

This guide walks you through building a complete end-to-end log aggregation pipeline using Vector agents, VRL transformation rules, and a centralized aggregator instance.

Step 1: Deploying Vector as a Log Agent via Docker Compose

To capture logs from running application containers, we will deploy a Vector agent on a target Hetzner VPS instance. The agent reads container stdout and stderr streams directly from the Docker daemon socket, attaches host metadata, and forwards the structured events to our centralized log aggregator.

First, create a project directory on your host to store the Vector agent configuration and Docker Compose files:

mkdir -p /opt/vector-agent/config
cd /opt/vector-agent
Enter fullscreen mode Exit fullscreen mode

Create the agent configuration file named /opt/vector-agent/config/vector.toml:

[api]
enabled = true
address = "0.0.0.0:8686"

[sources.docker_logs]
type = "docker_logs"
include_containers = []
exclude_containers = ["vector_agent"]

[transforms.enrich_host_metadata]
type = "remap"
inputs = ["docker_logs"]
source = '''
.host = "production-app-node-01"
.environment = "production"
.ingested_at = now()
'''

[sinks.to_aggregator]
type = "vector"
inputs = ["enrich_host_metadata"]
address = "aggregator.internal.example.com:6000"
version = "2"

[sinks.to_aggregator.buffer]
type = "disk"
max_size = 1073741824
when_full = "block"
Enter fullscreen mode Exit fullscreen mode

Next, define the docker-compose.yml file to launch the Vector agent daemon. When orchestrating background containers, ensure you establish explicit health checks to avoid routing logs to dead processes. For detailed strategies on service monitoring, check out our guide on Configuring Docker Compose Container Health Checks.

Create /opt/vector-agent/docker-compose.yml:

version: "3.8"

services:
  vector_agent:
    image: timberio/vector:0.34.X-debian
    container_name: vector_agent
    restart: always
    volumes:
      - /opt/vector-agent/config/vector.toml:/etc/vector/vector.toml:ro
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - /var/lib/vector:/var/lib/vector
    ports:
      - "8686:8686"
    healthcheck:
      test: ["CMD", "vector", "top", "--once"]
      interval: 15s
      timeout: 5s
      retries: 3
      start_period: 10s
Enter fullscreen mode Exit fullscreen mode

Start the agent using Docker Compose:

docker compose up -d
Enter fullscreen mode Exit fullscreen mode

Verify that the agent container is running and healthy:

docker compose ps
Enter fullscreen mode Exit fullscreen mode

💡 Fast-Track Your Project: Don't want to configure this yourself? I build custom n8n pipelines and bots. Message me with code SYS3-DEVTO.

Step 2: Parsing and Transforming Logs with Vector Remap Language

Unstructured log lines make querying and alerting difficult. Vector Remap Language (VRL) provides an efficient, expression-based language designed specifically for transforming observability data safely without runtime panics.

In this section, we will parse incoming JSON application logs, extract nested HTTP status codes, mask sensitive personally identifiable information (PII) such as credit card patterns, and format timestamps into ISO-8601 standard strings.

Let's inspect how VRL processes a sample unstructured web server log string like:

2026-03-30T10:14:32Z INFO user=9482 action=checkout card=4532-1111-2222-3333 status=200 duration_ms=42

Create a VRL file /opt/vector-agent/config/transform_app_logs.vrl to handle this logic:

# Parse log string using VRL built-in logfmt helper
parsed, err = parse_logfmt(.message)
if err != null {
  .vrl_error = err
  .status = "parse_failure"
} else {
  # Merge parsed fields into the root event object
  map_keys(parsed) -> |key| {
    .[key] = parsed[key]
  }

  # Cast numeric string values to concrete types
  .status = to_int!(.status)
  .duration_ms = to_int!(.duration_ms)
  .user_id = to_int!(.user)
  del(.user)

  # Redact credit card numbers using regex pattern replacement
  if exists(.card) {
    .card = replace!(string!(.card), r'^[0-9]{4}-[0-9]{4}-[0-9]{4}-', "XXXX-XXXX-XXXX-")
  }

  # Add derived severity tags based on HTTP status response code
  if .status >= 500 {
    .level = "ERROR"
  } else if .status >= 400 {
    .level = "WARN"
  } else {
    .level = "INFO"
  }
}

# Ensure message field remains standardized
.processed_by = "vector-vrl-v1"
Enter fullscreen mode Exit fullscreen mode

Now integrate this VRL logic directly into the transform block of your /opt/vector-agent/config/vector.toml configuration:

[api]
enabled = true
address = "0.0.0.0:8686"

[sources.app_file_source]
type = "file"
include = ["/var/log/containers/*.log"]
ignore_older_secs = 86400

[transforms.parse_and_sanitize]
type = "remap"
inputs = ["app_file_source"]
source = '''
parsed, err = parse_json(.message)
if err == null {
  ., err = merge(., parsed)
}

if exists(.credit_card) {
  .credit_card = "REDACTED"
}

if exists(.password) {
  del(.password)
}

.timestamp = format_timestamp!(now(), "%Y-%m-%dT%H:%M:%SZ")
'''

[sinks.console_debug]
type = "console"
inputs = ["parse_and_sanitize"]
target = "stdout"

[sinks.console_debug.encoding]
codec = "json"
Enter fullscreen mode Exit fullscreen mode

Reload Vector to apply the VRL processing transformations without causing downtime:

docker exec -it vector_agent vector reload
Enter fullscreen mode Exit fullscreen mode

Step 3: Setting Up the Centralized Vector Aggregator

Now that edge agents collect and clean host logs, we need a centralized Vector instance. The aggregator listens on a high-throughput TCP endpoint, buffers incoming events safely to disk, and fans out logs to long-term storage engines like Elasticsearch, ClickHouse, or AWS S3.

If you process high log volumes from application platforms or databases, keeping your database query overhead low is essential. For instance, high-volume logging setups streaming data alongside connection-pooled relational engines often use setups detailed in our guide on Setting Up PostgreSQL Connection Pooling with PgBouncer. Similarly, distributed systems monitoring webhook executions across workflow engines can reference our breakdown on Temporal vs n8n vs Airflow Webhook Automation.

Deploy the centralized aggregator configuration on a dedicated Contabo VPS host.

Create the aggregator directory structure:

mkdir -p /opt/vector-aggregator/config
mkdir -p /var/lib/vector-aggregator-data
Enter fullscreen mode Exit fullscreen mode

Create the central configuration file /opt/vector-aggregator/config/vector.toml:

[api]
enabled = true
address = "0.0.0.0:8687"

[sources.agent_streams]
type = "vector"
address = "0.0.0.0:6000"

[transforms.classify_logs]
type = "remap"
inputs = ["agent_streams"]
source = '''
if .environment == "production" {
  .priority = "high"
} else {
  .priority = "standard"
}
'''

[sinks.clickhouse_storage]
type = "clickhouse"
inputs = ["classify_logs"]
endpoint = "http://clickhouse.internal.example.com:8123"
database = "system_logs"
table = "application_events"
skip_unknown_fields = true

[sinks.clickhouse_storage.encoding]
timestamp_format = "unix"

[sinks.clickhouse_storage.buffer]
type = "disk"
max_size = 107374182400
when_full = "block"

[sinks.s3_archive]
type = "aws_s3"
inputs = ["classify_logs"]
bucket = "company-centralized-logs-archive"
region = "us-east-1"
key_prefix = "date=%Y-%m-%d/environment={{ environment }}/"
filename_time_format = "%H-%M-%S-%S"

[sinks.s3_archive.encoding]
codec = "ndjson"

[sinks.s3_archive.buffer]
type = "memory"
max_events = 500
when_full = "block"
Enter fullscreen mode Exit fullscreen mode

Deploy the central aggregator service using a standalone Docker compose service file /opt/vector-aggregator/docker-compose.yml:

version: "3.8"

services:
  vector_aggregator:
    image: timberio/vector:0.34.X-debian
    container_name: vector_aggregator
    restart: always
    volumes:
      - /opt/vector-aggregator/config/vector.toml:/etc/vector/vector.toml:ro
      - /var/lib/vector-aggregator-data:/var/lib/vector
    ports:
      - "6000:6000"
      - "8687:8687"
    healthcheck:
      test: ["CMD", "vector", "top", "--once"]
      interval: 10s
      timeout: 5s
      retries: 3
Enter fullscreen mode Exit fullscreen mode

Launch the central Vector aggregator service:

cd /opt/vector-aggregator
docker compose up -d
Enter fullscreen mode Exit fullscreen mode

Confirm that the central TCP listener port 6000 is open and actively accepting connections:

netstat -tulpn | grep 6000
Enter fullscreen mode Exit fullscreen mode

Step 4: Verifying Pipeline Health and Metrics

Once your agents and central aggregator run, verify that log streams flow smoothly without backpressure or drop-off issues.

Vector exposes Prometheus telemetry metrics out of the box. You can enable internal metrics generation in your Vector configuration file by adding the following snippet:

[sources.internal_metrics]
type = "internal_metrics"

[sinks.prometheus_exporter]
type = "prometheus_exporter"
inputs = ["internal_metrics"]
address = "0.0.0.0:9090"
Enter fullscreen mode Exit fullscreen mode

To validate that your configuration file syntax is completely valid before applying updates to production nodes, run Vector's internal validation command:

docker exec -it vector_agent vector validate /etc/vector/vector.toml
Enter fullscreen mode Exit fullscreen mode

You can monitor internal event throughput directly from your shell CLI terminal using Vector's real-time diagnostic console:

docker exec -it vector_agent vector top
Enter fullscreen mode Exit fullscreen mode

To run a single real-time stream test directly on an agent, pipe a fake JSON log line straight into the input file monitored by Vector:

echo '{"status": 500, "user": 1024, "card": "4111-2222-3333-4444", "message": "Failed checkout attempt"}' >> /var/log/containers/test_app.log
Enter fullscreen mode Exit fullscreen mode

Check the log output on the Central Aggregator to confirm receipt and verify that the transformation rules applied:

docker logs --tail 20 vector_aggregator
Enter fullscreen mode Exit fullscreen mode

If configured correctly, the log output displays full metadata tags, stripped card numbers, parsed severity codes, and structured JSON fields ready for cold storage or dashboarding.

Getting Started

Building a resilient, centralized log pipeline gives you complete visibility into distributed infrastructure without breaking your budget. By replacing heavy legacy log shippers with Vector, you gain ultra-fast throughput, robust buffer management, and powerful transformation capabilities via VRL.

To get started with your deployment:

  • Spin up performant cloud servers using a Hetzner VPS or Contabo VPS to host your aggregator and edge agents.
  • Deploy additional cloud infrastructure on DigitalOcean for geographically distributed test nodes.
  • Link your log processing events to automated incident workflows hosted on n8n Cloud.

Outsource Your Automation

Don't have time? I build production n8n workflows, WhatsApp bots, and fully automated YouTube Shorts pipelines. Hire me on Fiverr, mention SYS3-DEVTO for priority. Or DM at chasebot.online.

Originally published on Automation Insider.

Top comments (0)