Vector by Datadog: High-Throughput Log/Metric Pipelines Written in Rust

vector datadog is the pick-one architectural decision that decides whether your observability stack survives a five-minute traffic spike or drops half your logs on the floor — and it is the single component senior platform engineers get wrong most often because "just run Fluent Bit" ships a footprint-optimised agent into a throughput-shaped hole. Every log line your service emits, every metric your collector scrapes, every trace span your instrumentation exports has to reach Datadog, Splunk, Elasticsearch, S3, and a Prometheus long-term store without dropping events during a Kafka blip, without pinning a node's CPU at 100% during a log-storm, and without leaving you unable to add a parse_json step three months later because the config language is Turing-complete Lua. The engineering trade-off does not live in "should we run an observability pipeline" — every stack past twenty services needs one — but in which agent you pick and what it costs the node.

This guide is the senior-platform-engineer walkthrough you wished existed the first time an interviewer asked "walk me through the vector datadog sources / transforms / sinks model", or "why is VRL safer than a Lua filter for hot-path log transformation?", or "explain the aggregator tier and when it earns its cost." It walks through Vector's Rust-native architecture — the sources / transforms / sinks pipeline, the Vector Remap Language (VRL) with compile-time type checking and a sandboxed standard library, the agent / aggregator / gateway deployment topologies with disk-buffered backpressure and SIGHUP hot-reload — and the head-to-head decision matrix against Fluent Bit, Fluentd, and Logstash that decides which agent lands on your DaemonSet. Each section pairs a teaching block with a Solution-Tail interview answer — code, a step-by-step trace, an output table, then a concept-by-concept breakdown of why it works.

When you want hands-on reps immediately after reading, drill the streaming practice library →, rehearse system design against the design practice library →, and sharpen the batch axis with the ETL practice library →.

---

On this page

• Why Vector matters in 2026
• Sources, transforms & sinks architecture
• Vector Remap Language (VRL)
• Deployment topologies — agent, aggregator, gateway
• Vector vs Fluent Bit vs Fluentd vs Logstash + interview signals
• Cheat sheet — Vector recipes
• Frequently asked questions
• Practice on PipeCode

---

1. Why Vector matters in 2026

Datadog's Rust-native observability agent that finally makes a single pipeline cover logs, metrics, and traces at throughput

The one-sentence invariant: Vector is Datadog's open-source, Rust-native observability agent and pipeline that unifies logs, metrics, and traces under a single sources → transforms → sinks model, ships with 100+ integrations, offers roughly 10× the throughput and half the memory of Fluentd on the same workload, and is licensed MIT/Apache-2 so it will not be rug-pulled — and the pattern you pick in year one becomes the observability contract every downstream dashboard, alert, and SIEM query hard-codes assumptions against. Every senior platform-engineering interviewer in 2026 asks about Vector because the choice of observability agent is the single most consequential platform decision after Kubernetes itself.

The four axes interviewers actually probe.

• Language safety. Vector is written in Rust. That means no unbounded loops in the hot path, no null-pointer derefs during a log-storm, no memory leaks that force a nightly restart of the DaemonSet. Fluentd is Ruby (garbage-collected, slower, has a C-extension fast path for JSON parsing but not for transformation). Fluent Bit is C — fast but every out-of-bounds bug is a segfault waiting to happen on a customer node. Logstash is JVM — the fastest garbage collector in the industry can still stop-the-world for 300 ms during a heap resize. Interviewers open with this axis because it separates people who have run observability agents in production from people who have only picked one from a docs page.
• Throughput. Vector's published benchmarks (and independent reproductions on Kubernetes DaemonSets) land at roughly 10× the sustained log-events-per-second of Fluentd on identical hardware, and roughly 2× the throughput of Logstash with about 1/4 the resident memory. Fluent Bit is closer — it is also a native binary — but Fluent Bit's design targets tiny (10–50 MB) memory footprints for edge / embedded deployments, so at high per-node event rates it is bottlenecked by a smaller buffer budget.
• Unified logs + metrics + traces. Fluentd and Fluent Bit are log-first; metric support is a bolted-on extension. Logstash is log-first with a metrics extension. Vector is the only mainstream open-source pipeline built from day one to treat logs, metrics, and traces as first-class event types with a single config language. This matters because in 2026 most observability stacks want one agent per node, not three separate collectors racing each other for file descriptors.
• Config language safety. Fluentd's filter transforms use Ruby. Fluent Bit's use Lua. Logstash's use Ruby-embedded-in-JRuby. All three can loop forever, allocate unbounded memory, or throw runtime type errors that only surface when a malformed log line hits the hot path at 03:00 on a Tuesday. Vector uses VRL — the Vector Remap Language — which is compile-time type-checked, guaranteed to terminate, and sandboxed against infinite loops and unbounded memory. This is Vector's single biggest safety win.

The 2026 reality — Vector is the default for greenfield, but three siblings are still alive.

• Vector. The 2026 default for any greenfield high-throughput observability deployment where operational safety and unified logs + metrics + traces matter. Owned by Datadog (via the 2021 acquisition of Timber.io); licensed MIT/Apache-2 — an unusually permissive dual license that guarantees the community fork stays viable even if Datadog changes direction.
• Fluent Bit. The right answer for tiny-footprint scenarios — embedded systems, IoT, edge nodes, or Kubernetes clusters where every MB of node memory is contested. Written in C, tuned for a 10–50 MB footprint, ships as the CNCF-graduated log forwarder for most managed Kubernetes services. Slower than Vector at high per-node event rates but wins on memory budget.
• Fluentd. The right answer when your ecosystem depends on Fluentd's 1000+ community plugins (Ruby gems) and you cannot justify porting them to VRL or Fluent Bit filters. Still the CNCF-graduated log aggregator; still shipped in a hundred Helm charts by default. Slower and heavier than Vector on modern workloads.
• Logstash. The right answer when Elastic Stack (Elasticsearch + Kibana) is your primary observability sink and you want the tightest integration with Elastic's SIEM, machine-learning, and observability features. The JVM footprint and GC pauses are the operational cost.

What interviewers listen for.

• Do you name Rust as the language and Datadog as the owner in sentence one? — required answer.
• Do you name the sources → transforms → sinks pipeline model, not just "an agent"? — senior signal.
• Do you name VRL and its compile-time type checking as the safety win over Lua/Ruby? — senior signal.
• Do you distinguish agent / aggregator / gateway tiers, not just "run one Vector per node"? — senior signal.
• Do you name the 10× throughput vs Fluentd number without prompting? — senior signal.

Worked example — the four-agent comparison table

Detailed explanation. The single most useful artifact for a Vector interview is a memorised 4×5 comparison table across Vector, Fluent Bit, Fluentd, and Logstash on the five axes that decide the pick. Every senior observability discussion converges on this table within the first ten minutes; having it in your head is what separates a fluent answer from a stumbling one. Walk through building the table for a hypothetical greenfield Kubernetes platform that runs 200 microservices, emits ~50k log events/second at peak, needs sub-second Datadog ingestion, and must also scrape Prometheus metrics from every pod.

• Workload. 200-service Kubernetes cluster, 50k log events/s peak, 5k Prometheus scrape targets, one Datadog account + one S3 backup lake.
• Requirements. Sub-second Datadog latency, zero log loss during a 5-minute Kafka outage, in-line VRL transform to redact PII, ability to route the same event to both Datadog and S3.
• Constraints. DaemonSet-per-node deployment, 512 MB memory budget per agent pod, no infinite-loop risk during a log-storm.

Question. Build the four-agent comparison and pick the agent each requirement drives you toward.

Input.

Agent	Language	Throughput	Memory	Transform lang	Unified L+M+T
Vector	Rust	~10× Fluentd	~50% Fluentd	VRL (type-checked)	yes
Fluent Bit	C	~3× Fluentd	~10-50 MB	Lua	log-first + metrics
Fluentd	Ruby	baseline	baseline	Ruby	log-first
Logstash	JVM	~2× Fluentd	1-2 GB (JVM)	Ruby (JRuby)	log-first + metrics

Code.

# vector.toml — the four-agent decision codified as a Vector config
# (Illustrative: what a "picked Vector" DaemonSet config looks like)

[sources.k8s_logs]
type = "kubernetes_logs"
# Auto-discovers /var/log/pods/*/*.log; enriches with pod/namespace/labels

[sources.prom_scrape]
type = "prometheus_scrape"
endpoints = ["http://localhost:9100/metrics"]
scrape_interval_secs = 15

[transforms.redact_pii]
type = "remap"
inputs = ["k8s_logs"]
source = '''
    # Strip email addresses from the message field
    if exists(.message) {
        .message = replace(string!(.message), r'[\w\.-]+@[\w\.-]+', "<redacted-email>")
    }
    # Drop verbose debug lines
    if .level == "DEBUG" { abort }
'''

[sinks.datadog_logs]
type = "datadog_logs"
inputs = ["redact_pii"]
default_api_key = "${DD_API_KEY}"
compression = "gzip"

[sinks.s3_backup]
type = "aws_s3"
inputs = ["redact_pii"]
bucket = "acme-logs-lake"
key_prefix = "date=%Y-%m-%d/hour=%H/"
compression = "zstd"

[sinks.prom_remote]
type = "prometheus_remote_write"
inputs = ["prom_scrape"]
endpoint = "https://prometheus.internal/api/v1/write"

Step-by-step explanation.

1. The sources block declares two independent event inflows: kubernetes_logs auto-discovers container log files from /var/log/pods/*/*.log and enriches each event with pod / namespace / container / label metadata; prometheus_scrape polls one or more /metrics endpoints on a fixed interval. Both are Vector-native, no plugin install.
2. The transforms.redact_pii block runs VRL — Vector's compile-time-type-checked transform language. The replace(...) call with a regex strips email addresses; the abort statement drops DEBUG-level events entirely. VRL rejects the config at boot if any type is wrong, so an interviewer can never ambush you with "what if .message is null" — the compiler catches it.
3. The three sinks fan the same transformed event out to Datadog (compressed, primary observability path), S3 (compressed lake backup, for replay + compliance), and a Prometheus remote-write endpoint (for the scraped metrics, disjoint from the log path). One event, three destinations — the "unified sources → transforms → sinks" pipeline model.
4. The inputs = [...] array in every transform and sink builds an explicit DAG: sources feed transforms, transforms feed sinks. Vector validates the DAG at boot; a typo in an inputs reference fails the config rather than silently dropping events. This is a first-class safety guarantee absent in Fluentd (where <match> selectors can silently mis-route).
5. If the same interviewer asked "why not Fluent Bit here" — the answer is memory budget: Fluent Bit's 10-50 MB target does not leave room for a 500 MB disk buffer during a 5-minute Kafka outage. If asked "why not Fluentd" — the throughput number: at 50k events/s per node, Fluentd's Ruby transform layer becomes the bottleneck. If asked "why not Logstash" — the JVM footprint would consume 1-2 GB of every node's memory budget.

Output.

Requirement	Winning agent	Why
Sub-second Datadog latency	Vector	Rust hot path; no GC pauses
Zero loss during Kafka outage	Vector	Disk buffer up to configured size
VRL for PII redaction	Vector	Type-checked; guaranteed termination
Route to Datadog + S3	Vector	Native multi-sink DAG
200 pods scraped for metrics	Vector	prometheus_scrape source built-in
50 MB memory budget	Fluent Bit	Only agent that fits

Rule of thumb. Never pick an observability agent based on "which one is trendy." Pick it based on (throughput × memory × transform safety × unified L+M+T × plugin ecosystem) — the five axes. Write the matrix on a whiteboard first; the agent falls out of the constraints.

Worked example — what senior interviewers actually probe

Detailed explanation. The senior platform-engineering observability interview has a predictable structure: the interviewer opens with an ambiguous question ("how would you get logs from our 200-service Kubernetes cluster into Datadog?"), then progressively narrows to test whether you know the safety and throughput axes. The candidates who name Vector plus one senior-signal detail in sentence one score highest; the candidates who list "Fluentd or Logstash" score lowest. Walk through the interview grading rubric.

• Ambiguous opener. "How would you build the observability pipeline for our cluster?" — invites you to name an agent.
• Follow-up 1. "Why Vector over Fluent Bit?" — probes memory-vs-throughput trade-off.
• Follow-up 2. "How does Vector handle a 5-minute Kafka outage?" — probes disk buffer + backpressure.
• Follow-up 3. "Why is VRL safer than a Lua filter?" — probes language safety.
• Follow-up 4. "When would you add an aggregator tier?" — probes deployment topology.

Question. Draft a 5-minute senior observability-pipeline answer that covers all four axes without waiting to be asked.

Input.

Interview signal	Weak answer	Senior answer
Agent named	"Fluentd or Logstash"	"Vector — Datadog's Rust-native agent"
Throughput	"should be fast enough"	"~10× Fluentd, ~50% memory"
Language safety	"we'd add a filter"	"VRL is compile-time-checked; no infinite loops"
Kafka outage	"we'd retry"	"disk buffer up to 10 GB with when_full = block"
Aggregator tier	"run one per node"	"agent tier per pod; aggregator tier per region"

Code.

Senior observability-pipeline answer template (5 minutes)
==========================================================

Minute 1 — name the agent up front
  "I'd default to Vector — Datadog's Rust-native observability agent —
   deployed as a per-node DaemonSet feeding a regional aggregator tier."

Minute 2 — throughput + memory
  "Vector benchmarks at roughly 10x the sustained events/second of
   Fluentd on the same hardware with roughly half the memory. Our 50k
   events/s per node target sits well inside Vector's headroom; Fluentd
   would need 3-4 replicas per node to match, and Logstash's JVM would
   eat our 512 MB memory budget."

Minute 3 — VRL vs Lua safety
  "In-line transforms use VRL, Vector's Remap Language. VRL is
   compile-time type-checked, guaranteed to terminate, and sandboxed
   against unbounded memory. That means a malformed log line at 3am
   cannot infinite-loop the hot path the way a Lua filter can in
   Fluent Bit or a Ruby filter can in Fluentd."

Minute 4 — buffering + backpressure + Kafka outage
  "Every sink gets a disk buffer (`when_full = block`) sized to 10 GB
   plus a memory buffer. During a 5-minute Kafka outage the sink queues
   events to local disk and applies backpressure upstream — the agent
   never drops events on the floor. Recovery is 'the sink drains once
   Kafka is back' without operator action."

Minute 5 — deployment topology + hot-reload
  "Agent tier per node (DaemonSet). Aggregator tier per region
   (StatefulSet, 3-6 replicas) that receives from agents, buffers,
   and routes to Datadog, S3, and Elasticsearch. Gateway tier at the
   cross-region seam when we need multi-region multiplex. Config
   changes are hot-reload via SIGHUP or the /reload API without
   dropping in-flight events."

Step-by-step explanation.

1. Minute 1 is the crucial framing. Naming the agent immediately — "Vector — Datadog's Rust-native observability agent" — signals you're a decision-maker, not a task-runner. Weak candidates dive into "we'd use Fluentd or Logstash…" before naming the winner. Adding "deployed as a per-node DaemonSet feeding a regional aggregator tier" packs the topology answer into the opening.
2. Minute 2 addresses the throughput and memory axis before the interviewer asks. Naming the specific number (10× Fluentd, ~50% memory) is what separates a senior answer from a hand-wave. Preempting the "why not Fluent Bit" objection with the memory budget number closes the trade-off.
3. Minute 3 covers the language-safety axis — arguably the most under-appreciated Vector win. Saying "compile-time type-checked" plus "guaranteed to terminate" plus "sandboxed against unbounded memory" hits all three properties that distinguish VRL from Lua/Ruby. Interviewers love this because it shows you've run a Fluent Bit Lua filter that hung once.
4. Minute 4 is the reliability axis — the disk buffer + backpressure story. "when_full = block" is the specific config that names you as someone who's actually deployed Vector. The Kafka-outage recovery story matters because it's the failure mode that ships every observability incident.
5. Minute 5 covers the topology axis. Agent / aggregator / gateway — three tiers, each with a purpose. Hot-reload via SIGHUP is the operational detail that shows you've thought through config rollouts. This minute is often the tiebreaker between a strong senior candidate and a staff-level one.

Output.

Grading criterion	Weak score	Senior score
Names Vector in minute 1	rare	mandatory
Names throughput/memory number	rare	required
Names VRL type-safety	rare	senior signal
Names disk buffer + backpressure	rare	senior signal
Names three deployment tiers	rare	senior signal

Rule of thumb. The senior Vector answer is a 5-minute monologue that covers agent + throughput + VRL + buffering + topology without waiting for the follow-ups. Rehearse it once; deploy it every time.

Worked example — the "when Vector is the wrong answer" decision tree

Detailed explanation. A senior architect earns credibility not just by naming Vector but by naming the narrow cases where Vector is the wrong pick. Codifying the "when-not-Vector" tree makes your answer feel honest rather than evangelistic; interviewers reward candidates who can defend the boundary of a recommendation. Walk through the four scenarios where Vector loses.

• Q1. Is the deployment target embedded / IoT / < 50 MB memory? → yes = Fluent Bit; no = go to Q2.
• Q2. Does the org already have 100+ custom Fluentd Ruby plugins? → yes = Fluentd (migration cost > win); no = go to Q3.
• Q3. Is Elastic Stack the only sink? → yes = Logstash (tighter Elastic integration); no = go to Q4.
• Q4. Is the primary use case log parsing with grok patterns lifted from a Logstash config? → yes, and porting is not worth it = Logstash; no = Vector.

Question. Walk the "when-not-Vector" tree for three canonical scenarios and record which agent each ends up with.

Input.

Scenario	Q1 (< 50 MB?)	Q2 (100+ plugins?)	Q3 (Elastic-only?)	Q4 (grok-heavy?)
Edge IoT gateway (128 MB total RAM)	yes	—	—	—
Legacy Fluentd shop, 200 plugins	no	yes	—	—
Greenfield Kubernetes + Datadog + S3	no	no	no	no

Code.

# Decision-tree helper — when NOT to pick Vector
def pick_agent(memory_mb: int,
               custom_fluentd_plugins: int,
               only_sink_is_elastic: bool,
               grok_heavy_migration: bool) -> str:
    """Return the winning agent for an observability deployment."""

    if memory_mb < 50:
        return "Fluent Bit"

    if custom_fluentd_plugins > 100:
        return "Fluentd (migration cost > Vector win)"

    if only_sink_is_elastic and grok_heavy_migration:
        return "Logstash (Elastic-native integration)"

    return "Vector"


# Walk the three scenarios
print(pick_agent(128, 0, False, False))     # → Fluent Bit
print(pick_agent(4096, 200, False, False))  # → Fluentd (migration cost)
print(pick_agent(4096, 0, False, False))    # → Vector

Step-by-step explanation.

1. Scenario 1 — an edge IoT gateway with 128 MB total RAM shared across the OS, the app, and the log agent. Vector's steady-state footprint is 60-150 MB per agent; Fluent Bit ships in 10-40 MB. Q1 short-circuits at Fluent Bit. This is the honest answer; forcing Vector here would starve the app.
2. Scenario 2 — a legacy Fluentd shop with 200 custom Ruby plugins accumulated over 6 years. Q1 = no (plenty of memory), Q2 = yes (the plugin porting cost dwarfs the Vector throughput win). Fluentd stays. The senior signal here is acknowledging sunk cost: an architect who insists "always migrate to Vector" without pricing the migration is not senior.
3. Scenario 3 — a greenfield Kubernetes cluster feeding Datadog + S3 + Prometheus. Q1 = no, Q2 = no, Q3 = no, Q4 = no. Vector is the default. This is the modal 2026 answer.
4. The tree is deliberately biased toward Vector because in 2026 the throughput and safety wins are large enough that "default to Vector unless one of four narrow conditions holds" is the correct architectural bias. Naming the four narrow conditions is what separates a senior answer from a fan answer.
5. If none of Q1–Q4 pass and you still refuse Vector, the interviewer will (correctly) infer you have not benchmarked the alternatives.

Output.

Scenario	Winner	Reason
Edge IoT (128 MB)	Fluent Bit	Memory budget
Legacy 200-plugin shop	Fluentd	Migration cost
Greenfield K8s + Datadog	Vector	Modal default

Rule of thumb. The four-question tree is a whiteboard-friendly answer that names when Vector is not the pick. Practice walking it end-to-end so an interviewer can hand you any scenario and get an agent name in under 60 seconds.

Senior interview question on Vector selection

A senior interviewer often opens with: "You inherit a 300-service Kubernetes cluster that currently runs Fluentd DaemonSets feeding Elasticsearch, with per-node Fluent Bit sidecars handling Prometheus metrics. The org is migrating primary observability to Datadog and wants a single unified agent per node. Walk me through your migration to Vector, the DaemonSet config, the disk-buffer budget, and the rollout plan."

Solution Using Vector as a unified DaemonSet with a regional aggregator and disk-buffered sinks

# /etc/vector/vector.toml — the DaemonSet config for the migration
# Deployed via Helm chart with node-local /var/log and /var/lib/vector volumes

data_dir = "/var/lib/vector"

# ---- SOURCES ------------------------------------------------------------
[sources.k8s_logs]
type = "kubernetes_logs"
# Enriches each event with kubernetes.pod_name, .namespace, .container_name,
# and all pod labels; auto-discovers from the kubelet.

[sources.prom_scrape]
type = "prometheus_scrape"
endpoints = [
    "http://localhost:9100/metrics",           # node_exporter
    "http://localhost:8080/metrics",           # kube-state-metrics
]
scrape_interval_secs = 15

[sources.host_metrics]
type = "host_metrics"
collectors = ["cpu", "memory", "network", "disk", "filesystem"]
scrape_interval_secs = 30

# ---- TRANSFORMS ---------------------------------------------------------
[transforms.parse_and_enrich]
type = "remap"
inputs = ["k8s_logs"]
source = '''
    # Parse JSON application logs; leave plaintext lines as-is
    if starts_with(string!(.message), "{") {
        parsed, err = parse_json(.message)
        if err == null {
            . = merge(., object!(parsed))
        }
    }
    # Redact obvious PII
    .message = replace(string!(.message ?? ""), r'[\w\.-]+@[\w\.-]+', "<email>")
    # Attach cluster + region metadata
    .cluster = "us-east-prod"
    .region  = "us-east-1"
'''

[transforms.drop_noise]
type = "filter"
inputs = ["parse_and_enrich"]
# Drop kubelet DEBUG lines and healthcheck spam
condition = '.level != "DEBUG" && !contains(string!(.message ?? ""), "healthcheck")'

# ---- SINKS --------------------------------------------------------------
[sinks.aggregator]
type = "vector"
inputs = ["drop_noise", "prom_scrape", "host_metrics"]
address = "vector-aggregator.observability.svc.cluster.local:6000"
compression = true

[sinks.aggregator.buffer]
type      = "disk"
max_size  = 10737418240      # 10 GB per sink
when_full = "block"          # backpressure upstream, never drop

# aggregator vector.yaml — StatefulSet, 6 replicas, per region
data_dir: /var/lib/vector

sources:
  from_agents:
    type: vector
    address: 0.0.0.0:6000

transforms:
  route_by_type:
    type: route
    inputs: [from_agents]
    route:
      logs:    ".log_type == \"application\" || exists(.message)"
      metrics: "exists(.name) && exists(.value)"

sinks:
  datadog_logs:
    type: datadog_logs
    inputs: [route_by_type.logs]
    default_api_key: ${DD_API_KEY}
    compression: gzip
    buffer:
      type: disk
      max_size: 21474836480       # 20 GB
      when_full: block

  s3_lake:
    type: aws_s3
    inputs: [route_by_type.logs]
    bucket: acme-logs-lake
    key_prefix: cluster=us-east-prod/date=%Y-%m-%d/hour=%H/
    compression: zstd
    batch:
      max_bytes: 10485760          # 10 MB per object
      timeout_secs: 60

  prom_remote:
    type: prometheus_remote_write
    inputs: [route_by_type.metrics]
    endpoint: https://prometheus.internal/api/v1/write

# Rollout plan (executed one DaemonSet rolling update per node pool)
kubectl -n observability apply -f vector-aggregator-statefulset.yaml
kubectl -n observability rollout status statefulset/vector-aggregator

# Canary Vector to 5% of nodes; run alongside Fluentd for 24h
kubectl -n observability set env daemonset/vector CANARY=true
kubectl -n observability patch daemonset vector -p '{"spec":{"template":{"spec":{"nodeSelector":{"observability-canary":"true"}}}}}'

# Compare event rates between Fluentd and Vector on canary nodes
# (Datadog Metrics Explorer: sum:datadog.logs.forwarded{agent:*} by {agent})

# Promote Vector to 100%, then delete Fluentd DaemonSet
kubectl -n observability delete daemonset fluentd

Step-by-step trace.

Step	Before (Fluentd + Fluent Bit)	After (Vector unified)
Agents per node	2 (Fluentd + Fluent Bit)	1 (Vector)
Memory per node	300 MB + 40 MB = 340 MB	120-180 MB
Log throughput per node	~5k events/s ceiling	~50k events/s comfortable
Config language	Fluentd XML + Lua	VRL
Buffering during Kafka outage	best-effort	10 GB disk buffer + backpressure
Sink fan-out	separate per-agent config	native multi-sink DAG
Hot-reload	SIGUSR2 (Fluentd), restart (Fluent Bit)	SIGHUP or /reload API (Vector)

After the migration, each node runs a single Vector DaemonSet that forwards to a regional 6-replica aggregator StatefulSet; the aggregator routes logs to Datadog + S3 and metrics to Prometheus remote-write. Per-node memory drops from ~340 MB to ~150 MB, freeing 190 MB per node for workloads. A simulated 5-minute Datadog outage queues ~15 million events to the aggregator's disk buffer and drains cleanly on recovery — zero log loss.

Output:

Metric	Value
Per-node memory	120-180 MB (down from 340 MB)
Per-node log throughput	50k events/s (up from 5k)
Agents per node	1 (down from 2)
Config files	1 TOML (down from XML + Lua + INI)
Datadog outage tolerance	~15M events queued on disk
Rollout risk	canary → 5% → 100% over 3 days

Why this works — concept by concept:

• Rust-native hot path — no GC pauses, no unbounded loops, no memory-safety-triggered restarts. Every performance-sensitive component (the source readers, the VRL evaluator, the sink batchers) is written in Rust with borrow-checker-enforced memory safety.
• sources → transforms → sinks DAG — the config declares a validated directed acyclic graph. Vector rejects invalid inputs = [...] references at boot, so a typo cannot silently drop events. Fan-out is native (one transform, many sinks); fan-in is native (many sources, one transform).
• VRL for in-line transformation — compile-time type checking rejects .level == 3 when .level is a string; the sandbox rejects infinite loops; the standard library ships parse_json, parse_grok, parse_apache_log, replace, merge, and 100+ other functions with pre-checked semantics.
• Disk buffer + when_full = block — every sink can queue events to local disk up to a configured size (10-20 GB typical) and apply backpressure upstream if the buffer fills. Zero log loss during downstream outages up to the buffer budget.
• Cost — one Vector DaemonSet per node (~150 MB), one 6-replica aggregator StatefulSet per region (~2 GB each), one Helm chart for both. The eliminated cost is the "run two agents per node" tax (Fluentd + Fluent Bit was 340 MB), the "we lost logs when Kafka blipped" outage class, and the Ruby/Lua transform-safety class of bugs. Net O(1) agent per node with unified logs + metrics; the aggregator tier scales linearly with regional throughput.

Streaming
Topic — streaming
Streaming problems on observability pipelines

Practice →

Design Topic — design Design problems on log/metric agent architectures

Practice →

---

2. Sources, transforms & sinks architecture

sources → transforms → sinks is the declarative TOML/YAML DAG that unifies logs, metrics, and traces in one config

The mental model in one line: Vector's architecture is a declarative DAG of three noun types — **sources that inflow events, transforms that mutate them in-place with type-checked VRL, and sinks that outflow to durable destinations — wired together by explicit inputs = [...] references in a single TOML or YAML config, validated at boot, and reloaded via SIGHUP without dropping in-flight events**. Every senior platform engineer has to explain this model at least once per interview; getting the three nouns right in sentence one is the fastest way to signal fluency.

The three noun types — the entire mental model.

• Sources. Inflow endpoints. Vector 0.40+ ships ~50 built-in source types including file, kafka, socket, docker_logs, kubernetes_logs, journald, prometheus_scrape, host_metrics, internal_metrics, syslog, http_server, nats, redis, pulsar, splunk_hec. Each source emits a stream of typed events (Log, Metric, or Trace) that downstream components can consume.
• Transforms. Mutation nodes. remap (VRL) is the workhorse; siblings include filter (drop events matching a predicate), aggregate (sum/mean/count over a time window for metrics), dedupe (drop repeated events by key), sample (probabilistic drop), throttle (rate-limit per key), route (split one stream into several by predicate), reduce (merge related events across time), tag_cardinality_limit (cap metric tag explosion).
• Sinks. Outflow destinations. ~60 built-ins: elasticsearch, splunk_hec, aws_s3, kafka, datadog_logs, datadog_metrics, datadog_traces, prometheus_remote_write, victoriametrics_metrics, loki, honeycomb, new_relic_logs, azure_monitor_logs, gcp_stackdriver_logs, nats, redis, mezmo, humio, plus generic http and console for testing.

The event data model — Log, Metric, Trace.

• Log. A timestamp + a set of key/value fields (.message, .host, .level, and any application-added fields). VRL treats logs as a mutable object.
• Metric. A typed measurement with a name, tags, and a value shape (counter, gauge, histogram, distribution, summary, set, aggregated_histogram, aggregated_summary). Metrics have well-defined semantics for aggregation, so aggregate and tag_cardinality_limit transforms behave correctly.
• Trace. OpenTelemetry-shaped span data. Vector routes traces to datadog_traces and honeycomb sinks with the OTLP protocol.

The DAG contract — explicit inputs, validated at boot.

• Explicit wiring. Every transform and sink names its parent(s) via inputs = ["source_or_transform_name"]. There are no implicit connections; Vector's config is a directed graph declared explicitly.
• Boot-time validation. Vector loads and validates the entire DAG before accepting events. A typo in an inputs reference, a cyclic dependency, or a type mismatch (feeding a Metric into a transform that expects Log) fails the config with a clear error message rather than silently dropping events.
• No hidden state. The DAG has no implicit ordering beyond what inputs declares. Two independent branches run concurrently on separate tokio tasks.

The buffering + backpressure model.

• Per-sink buffers. Every sink has an in-memory buffer by default (capacity in events); Vector also supports a per-sink disk buffer sized in bytes. Buffer type is type = "memory" or type = "disk".
• when_full policy. When the buffer is full: drop_newest drops incoming events (fast, lossy), block applies backpressure upstream (correct, slower). Production deployments almost always pick block for critical sinks and drop_newest for best-effort ones.
• Acknowledgements. Sources that support end-to-end acks (kafka, aws_kinesis_streams, file) only advance their offset / checkpoint once the sink has committed. Combined with disk buffering, this gives at-least-once delivery through the pipeline.

Common interview probes on the architecture.

• "Name the three noun types." — required answer: sources, transforms, sinks.
• "How does Vector handle a slow sink?" — buffer + backpressure via when_full = block.
• "How is the DAG validated?" — boot-time; explicit inputs; typos fail early.
• "What event types does Vector understand?" — Log, Metric, Trace.

Worked example — Kubernetes logs → VRL enrichment → Datadog + S3

Detailed explanation. The canonical Vector agent config for a Kubernetes DaemonSet: kubernetes_logs source, a remap transform that parses JSON app logs and adds cluster metadata, then two sinks — datadog_logs for the primary observability path and aws_s3 for a long-term compliance backup lake. Walk through every piece.

• Source. kubernetes_logs — auto-discovers /var/log/pods/*/*.log and enriches with kubernetes.pod_name, .namespace, .container_name, and pod labels.
• Transform. remap running VRL that parses JSON, redacts PII, adds cluster tags.
• Sinks. datadog_logs (compressed, primary) + aws_s3 (compressed, backup).

Question. Write the full TOML config and show the event flow from a container log line to both sinks.

Input.

Component	Purpose
sources.k8s_logs	inflow of container logs
transforms.enrich	JSON parse + PII redact + cluster tags
sinks.datadog	primary observability
sinks.s3_lake	compliance backup

Code.

# /etc/vector/agent.toml
data_dir = "/var/lib/vector"

[sources.k8s_logs]
type = "kubernetes_logs"
# Optional: exclude noisy system pods
exclude_paths_glob_patterns = ["/var/log/pods/kube-system_*/**"]

[transforms.enrich]
type = "remap"
inputs = ["k8s_logs"]
source = '''
    # 1. Parse JSON app logs; leave text lines as .message
    if starts_with(string!(.message), "{") {
        parsed, err = parse_json(.message)
        if err == null {
            . = merge(., object!(parsed))
        }
    }
    # 2. Redact obvious PII
    .message = replace(string!(.message ?? ""), r'[\w\.-]+@[\w\.-]+', "<email>")
    # 3. Attach cluster metadata
    .cluster    = "prod-us-east-1"
    .env        = "prod"
    .ingested_at = now()
'''

[sinks.datadog]
type = "datadog_logs"
inputs = ["enrich"]
default_api_key = "${DD_API_KEY}"
compression = "gzip"

[sinks.datadog.buffer]
type      = "disk"
max_size  = 5368709120        # 5 GB
when_full = "block"

[sinks.s3_lake]
type = "aws_s3"
inputs = ["enrich"]
bucket = "acme-logs-lake"
key_prefix = "cluster=prod-us-east-1/date=%Y-%m-%d/hour=%H/"
compression = "zstd"
encoding.codec = "json"

[sinks.s3_lake.batch]
max_bytes = 10485760          # 10 MB per S3 object
timeout_secs = 60

Step-by-step explanation.

1. sources.k8s_logs opens a watcher on /var/log/pods/*/*.log and emits one event per line, pre-enriched with .kubernetes.pod_name, .kubernetes.namespace_name, .kubernetes.container_name, and the pod's labels + annotations. The exclude_paths_glob_patterns array trims out the noisiest system-pod logs at the source.
2. The remap transform runs VRL. Step 1 conditionally parses JSON — if .message starts with {, try parse_json; if it succeeds, merge the parsed object into the event (adding whatever fields the app logger emitted). Step 2 redacts email addresses via regex. Step 3 attaches static cluster/env tags and a Vector-side ingestion timestamp.
3. Both sinks reference inputs = ["enrich"], so both receive the same transformed event stream — Vector fan-outs the transform's output to every sink that lists it as an input. No config duplication.
4. sinks.datadog compresses each batch with gzip and ships to Datadog's Log Intake API. The disk buffer (5 GB, when_full = block) means a 10-minute Datadog outage queues locally and drains on recovery; a longer outage applies backpressure upstream and eventually pauses the kubernetes_logs reader — no drops.
5. sinks.s3_lake uses zstd compression (better ratio than gzip for JSON) and batches into 10 MB objects with a 60-second timeout. The key_prefix template embeds cluster + date + hour so S3 objects are trivially partitionable by Athena / Snowflake external tables. The backup path is idempotent — restarting Vector after a crash reprocesses the incomplete file from the last checkpoint.

Output.

// Sample raw input line (from /var/log/pods/api_backend/api/0.log)
{"level":"INFO","ts":"2026-07-21T12:34:56Z","msg":"order placed","order_id":42,"user":"[e-mail]"}

// Sample enriched event landing on datadog + s3 sinks
{
  "level":       "INFO",
  "ts":          "2026-07-21T12:34:56Z",
  "msg":         "order placed",
  "order_id":    42,
  "user":        "<email>",
  "message":     "order placed",
  "cluster":     "prod-us-east-1",
  "env":         "prod",
  "ingested_at": "2026-07-21T12:34:56.812Z",
  "kubernetes": {
    "pod_name":       "api-backend-7d4c9",
    "namespace_name": "api",
    "container_name": "api",
    "pod_labels":     { "app": "api-backend", "version": "v2.14.0" }
  }
}

Rule of thumb. For any Vector agent config, keep the DAG shape narrow at the top, wide at the bottom — one source → one enrich transform → many sinks. Push shared logic (JSON parse, PII redact, cluster tags) into a single upstream transform so every sink sees the same enriched event; never duplicate transform logic per sink.

Worked example — routing one stream into per-severity Kafka topics

Detailed explanation. A common pattern: a single log stream needs to fan out to multiple Kafka topics keyed by severity — INFO/WARN to a cheap warehouse topic, ERROR/FATAL to an alerting topic that triggers PagerDuty. Vector's route transform splits one stream into N labelled streams; each label wires to its own sink. Walk through the config.

• Source. kubernetes_logs — application container logs.
• Route transform. Splits by .level into three named outputs: info, warn, error.
• Sinks. Three Kafka sinks, each subscribing to one route output.

Question. Split the log stream by severity and ship each severity to a different Kafka topic.

Input.

Route label	Predicate	Kafka topic
info	.level == "INFO"	logs.info
warn	.level == "WARN"	logs.warn
error	`.level == "ERROR"

Code.
{% raw %}

[sources.k8s_logs]
type = "kubernetes_logs"

[transforms.parse_level]
type = "remap"
inputs = ["k8s_logs"]
source = '''
    # Parse JSON if present so .level is populated
    if starts_with(string!(.message ?? ""), "{") {
        parsed, err = parse_json(.message)
        if err == null { . = merge(., object!(parsed)) }
    }
    # Default level for unstructured lines
    if !exists(.level) { .level = "INFO" }
'''

[transforms.split_by_severity]
type = "route"
inputs = ["parse_level"]

[transforms.split_by_severity.route]
info  = '.level == "INFO"'
warn  = '.level == "WARN"'
error = '.level == "ERROR" || .level == "FATAL"'

# Three sinks, one per route output. Note the "split_by_severity.<label>"
# syntax that references a named output of the route transform.
[sinks.kafka_info]
type = "kafka"
inputs = ["split_by_severity.info"]
bootstrap_servers = "kafka-1:9092,kafka-2:9092"
topic = "logs.info"
encoding.codec = "json"
compression = "lz4"

[sinks.kafka_warn]
type = "kafka"
inputs = ["split_by_severity.warn"]
bootstrap_servers = "kafka-1:9092,kafka-2:9092"
topic = "logs.warn"
encoding.codec = "json"
compression = "lz4"

[sinks.kafka_error]
type = "kafka"
inputs = ["split_by_severity.error"]
bootstrap_servers = "kafka-1:9092,kafka-2:9092"
topic = "logs.error"
encoding.codec = "json"
compression = "lz4"

# Any event that matched no route ends up on the special "._unmatched" output;
# we could catch it too:
[sinks.kafka_other]
type = "kafka"
inputs = ["split_by_severity._unmatched"]
bootstrap_servers = "kafka-1:9092,kafka-2:9092"
topic = "logs.other"
encoding.codec = "json"

Step-by-step explanation.

1. The parse_level transform is a defensive step — it parses JSON if present and defaults .level to "INFO" for plaintext lines. Without this defensiveness, a plaintext log line would land on ._unmatched because .level wouldn't exist, silently dropping into the "other" topic.
2. The route transform is the fan-out primitive. Each named entry under [transforms.split_by_severity.route] is a VRL predicate; every input event is evaluated against every predicate and emitted on the named output(s) whose predicate is true.
3. The inputs = ["split_by_severity.info"] syntax references a named output of the route transform. This is the only place in Vector's config where a transform name is qualified with a dotted suffix — it's specific to route (and remap's VRL abort semantics).
4. An event that matches no route lands on the implicit ._unmatched output; sinks.kafka_other catches those. Without this catcher, unmatched events would be silently dropped — a common footgun.
5. Each Kafka sink is independently buffered and independently retried. If logs.error's broker is momentarily down, logs.info and logs.warn keep flowing normally — the DAG is per-sink-parallel.

Output.

Input .level	Emitted route output(s)	Kafka topic
INFO	info	logs.info
WARN	warn	logs.warn
ERROR	error	logs.error
FATAL	error	logs.error
(missing)	_unmatched	logs.other

Rule of thumb. Every route transform needs a _unmatched sink (or a preceding transform that guarantees a matching field), otherwise unmatched events silently vanish. Vector logs _unmatched counts as an internal metric — set an alert on non-zero _unmatched throughput.

Worked example — end-to-end acks with Kafka source and Elasticsearch sink

Detailed explanation. Vector's end-to-end acknowledgement contract: a source that supports acks (Kafka, Kinesis, file) only advances its checkpoint / consumer offset once the downstream sink has committed. This gives at-least-once delivery through the pipeline — a Vector crash between source read and sink commit re-delivers the event on restart. Walk through the Kafka → Elasticsearch config.

• Source. kafka reading from topic.raw_events with consumer group vector-es-writer.
• Transform. remap that shapes the event for Elasticsearch.
• Sink. elasticsearch bulk-indexing into raw-events-* daily indices.

Question. Configure a Kafka → Elasticsearch pipeline with end-to-end acks + at-least-once delivery.

Input.

Component	Value
Kafka topic	topic.raw_events
Consumer group	vector-es-writer
ES index	raw-events-YYYY.MM.DD
Delivery	at-least-once

Code.

[sources.kafka_in]
type = "kafka"
bootstrap_servers = "kafka-1:9092,kafka-2:9092,kafka-3:9092"
group_id = "vector-es-writer"
topics = ["topic.raw_events"]
auto_offset_reset = "earliest"

# CRITICAL: end-to-end acks. Vector only commits Kafka offsets
# after the downstream sink has confirmed the batch.
acknowledgements.enabled = true

[transforms.shape_for_es]
type = "remap"
inputs = ["kafka_in"]
source = '''
    # Ensure @timestamp field exists (Elasticsearch convention)
    if !exists(."@timestamp") {
        ."@timestamp" = format_timestamp!(now(), format: "%+")
    }
    # Route to daily index by timestamp
    .index = "raw-events-" + format_timestamp!(parse_timestamp!(."@timestamp", "%+"), format: "%Y.%m.%d")
'''

[sinks.es]
type = "elasticsearch"
inputs = ["shape_for_es"]
endpoints = ["https://es.internal:9200"]
mode = "bulk"
bulk.index = "{{ .index }}"           # per-event index from the .index field
compression = "gzip"
auth.strategy = "basic"
auth.user = "vector"
auth.password = "${ES_PASSWORD}"

# Retry on transient ES failures; the source will re-deliver on crash.
[sinks.es.request]
retry_attempts = 10
retry_initial_backoff_secs = 1
retry_max_duration_secs = 30

[sinks.es.buffer]
type      = "disk"
max_size  = 10737418240              # 10 GB
when_full = "block"                  # apply backpressure to Kafka source

Step-by-step explanation.

1. acknowledgements.enabled = true on the Kafka source is the switch that enables end-to-end acks. Without it, Vector auto-commits Kafka offsets as soon as it reads a batch — a crash mid-pipeline loses those events. With it, offsets advance only after the sink confirms.
2. shape_for_es ensures every event has an @timestamp field (Elasticsearch convention) and computes a per-event target index name (raw-events-YYYY.MM.DD). VRL's format_timestamp! returns a String; the ! suffix means "fail loudly if the parse fails" — the compiler enforces the error handling.
3. The elasticsearch sink's bulk.index = "{{ .index }}" template extracts the target index from each event. Daily indices are a standard Elastic Stack practice for retention (drop old indices instead of DELETE-by-query).
4. sinks.es.request retries transient ES failures up to 10 times with exponential backoff. During ES downtime longer than 30 seconds, the sink stops making progress; combined with the disk buffer's when_full = block, this applies backpressure all the way back to Kafka — Vector stops fetching new records rather than dropping them.
5. On Vector crash mid-batch: (a) unacked events stay in Kafka; (b) restart reads them again from the last committed offset; (c) the events reach ES again — because ES bulk indexing is idempotent when using a stable _id, or produces at-most-one duplicate per crash (acceptable for at-least-once semantics).

Output.

Event stage	Delivery guarantee	Failure recovery
Kafka → Vector	at-least-once (offset commit)	replay from last commit
Vector transform	in-memory	replay from Kafka on crash
Vector → ES bulk	at-least-once (retry)	replay from Kafka on crash
ES → indexed doc	at-most-once-duplicate	idempotent _id = dedupe

Rule of thumb. Any Vector pipeline that needs at-least-once delivery must enable acknowledgements.enabled = true on the source and use when_full = block on every sink downstream. Either one alone breaks the contract — no-acks source silently commits; drop_newest sink silently drops.

Senior interview question on Vector architecture

A senior interviewer might ask: "Design a Vector pipeline that ingests kafka events, splits them into a real-time Datadog stream and a batched S3 lake stream, applies backpressure end-to-end, and survives a 30-minute Datadog outage without losing events or blocking the S3 branch."

Solution Using an acked Kafka source, per-branch disk buffers, and independent sink retries

data_dir = "/var/lib/vector"

# ---- SOURCE with end-to-end acks ----------------------------------------
[sources.kafka_events]
type = "kafka"
bootstrap_servers = "kafka-1:9092,kafka-2:9092,kafka-3:9092"
group_id = "vector-fanout"
topics = ["events.raw"]
auto_offset_reset = "earliest"
acknowledgements.enabled = true

# ---- SHARED TRANSFORM ---------------------------------------------------
[transforms.enrich]
type = "remap"
inputs = ["kafka_events"]
source = '''
    if starts_with(string!(.message ?? ""), "{") {
        parsed, err = parse_json(.message)
        if err == null { . = merge(., object!(parsed)) }
    }
    .env      = "prod"
    .region   = "us-east-1"
    .ingested = now()
'''

# ---- REAL-TIME BRANCH ---------------------------------------------------
[sinks.datadog]
type = "datadog_logs"
inputs = ["enrich"]
default_api_key = "${DD_API_KEY}"
compression = "gzip"
acknowledgements.enabled = true

[sinks.datadog.buffer]
type      = "disk"
max_size  = 32212254720        # 30 GB — 30-min outage headroom
when_full = "block"

[sinks.datadog.request]
retry_attempts = 100
retry_initial_backoff_secs = 1
retry_max_duration_secs = 60

# ---- BATCHED S3 BRANCH --------------------------------------------------
[sinks.s3_lake]
type = "aws_s3"
inputs = ["enrich"]
bucket = "acme-events-lake"
key_prefix = "date=%Y-%m-%d/hour=%H/"
compression = "zstd"
encoding.codec = "json"
acknowledgements.enabled = true

[sinks.s3_lake.batch]
max_bytes = 134217728          # 128 MB per S3 object
timeout_secs = 300             # or every 5 min

[sinks.s3_lake.buffer]
type      = "disk"
max_size  = 5368709120         # 5 GB (S3 rarely outages; small buffer)
when_full = "block"

Step-by-step trace.

Layer	Component	Purpose
Kafka source	acknowledgements.enabled = true	offsets only advance after both sinks ack
Shared transform	enrich VRL	one JSON parse + one tag-attach, both sinks reuse
Datadog branch	30 GB disk buffer + when_full = block	30-min outage tolerated locally
Datadog retry	100 attempts, 60s max backoff	rides out transient 5xx
S3 branch	5 GB disk buffer + 128 MB batches	high-latency, high-throughput
Independence	separate sinks, separate buffers	S3 keeps flowing during Datadog outage
Backpressure	Kafka pauses fetch when both buffers full	zero drops

After deployment, a 30-minute Datadog outage queues ~15 million events on the local disk buffer while S3 keeps flowing normally; when Datadog recovers, the buffer drains at Datadog's ingest rate. A 5-minute S3 outage queues ~2.5 million events on the S3 disk buffer while Datadog keeps flowing normally. A simultaneous outage on both sinks pauses the Kafka source (backpressure) — offsets do not advance, no events are lost.

Output:

Failure mode	Datadog branch	S3 branch	Kafka source
Datadog outage 5m	queue on disk	keeps flowing	keeps flowing
Datadog outage 30m	queue up to 30 GB	keeps flowing	keeps flowing
S3 outage 5m	keeps flowing	queue on disk	keeps flowing
Both outages	queue on disk	queue on disk	paused; no offset commit
Vector crash mid-batch	replay from Kafka	replay from Kafka	resume from last commit

Why this works — concept by concept:

• End-to-end acks on the source — acknowledgements.enabled = true on kafka_events means the consumer offset advances only after every downstream sink that also opts into acks confirms. This is the at-least-once contract that guarantees zero loss on Vector crash.
• Per-sink disk buffers — each sink has its own disk buffer sized to its failure-mode profile. Datadog gets 30 GB (outages measured in minutes); S3 gets 5 GB (outages rare and short). Buffers are independent; one sink's outage does not block the other.
• when_full = block on both sinks — when both buffers fill simultaneously, backpressure propagates all the way back to Kafka, and Vector stops fetching new records. Kafka retains the messages; nothing is dropped.
• Shared transform — the enrich VRL runs once per event; both sinks reuse the transformed event. No duplicated CPU cost, no risk of the two sinks seeing different shapes.
• Cost — one Vector process (~200 MB), 35 GB combined disk buffer per node, one Kafka consumer group. The eliminated cost is the classic "our Datadog outage cost us three hours of production logs" incident. Net O(1) per event with bounded local buffering; the deep tail latency budget lives in disk rather than in dropped events.

Streaming
Topic — streaming
Streaming problems on Kafka fan-out and backpressure

Practice →

ETL Topic — etl ETL problems on batched sink partitioning

Practice →

---

3. Vector Remap Language (VRL)

VRL is a compile-time-type-checked, guaranteed-to-terminate scripting language for hot-path log and metric transformation

The mental model in one line: VRL — the Vector Remap Language — is a purpose-built, expression-oriented scripting language for transforming event data inside a remap transform, distinguished from Lua / Ruby / JavaScript by three properties: **compile-time type checking (a bad type is a boot failure, not a 3am log-storm failure), guaranteed termination (no loops, no recursion, no unbounded operations — every program halts), and a sandboxed standard library with no I/O, no filesystem, no network — every function is a pure transformation on the current event**. Every senior platform engineer has spent at least one weekend debugging a runaway Lua filter; VRL exists precisely so that weekend never happens again.

The three properties that define VRL.

• Compile-time type checking. Every expression has a statically inferred type (string, integer, float, boolean, timestamp, object, array, null, regex, bytes). Type errors — passing a string to a function that expects an integer, indexing into a value that might be null — are compile errors, caught at Vector boot. This is qualitatively different from Lua / Ruby / JavaScript where a bad type surfaces as a runtime error the first time a matching event flows through.
• Guaranteed termination. VRL has no while, no for, no recursion. The only iteration primitives are the fixed-fanout map_values / filter over arrays/objects and the built-in library functions that iterate internally (for_each_key, flatten). No VRL program can run forever; every program's runtime is bounded by input size × library function cost.
• Sandboxed standard library. VRL functions cannot open files, make HTTP calls, execute shell commands, or allocate unbounded memory. Every function is a pure transformation on the current event. This is what makes VRL safe to run on the hot path at 3am against untrusted input.

The event as . — VRL's central abstraction.

• . is the current event. In a remap transform, . refers to the event flowing through. . = merge(., {"cluster": "prod"}) mutates it. .field = 42 sets a field.
• Path expressions. .foo.bar navigates nested objects. .foo[0] indexes arrays. .foo.bar ?? "default" short-circuits to a default if the path is null / missing.
• Assignment vs coalesce. x = .field assigns; if .field is null, x is null. x = .field ?? "default" coalesces to a default. x, err = parse_json(.msg) is the fallible-assignment pattern — VRL's answer to Go-style error handling.

The fallibility contract — ! vs error tuples.

• function!(...) — abort on error. Every fallible function has a ! variant that aborts VRL evaluation if it fails. Use this when a failure means "drop the event / mark it failed" — parsed = parse_json!(.message) fails the entire remap on bad JSON.
• (result, err) = function(...) — inspect and handle. The tuple form returns both the result and an error string; you branch on err. Use this when a failure is expected and should be handled inline: parsed, err = parse_json(.message); if err == null { . = merge(., parsed) }.
• The compiler enforces it. VRL rejects a program that calls a fallible function without either ! or the tuple form; you cannot accidentally ignore errors. This is the single biggest safety win over Lua/Ruby.

The standard library — 200+ pure functions.

• Parsers. parse_json, parse_grok, parse_apache_log, parse_nginx_log, parse_syslog, parse_regex, parse_key_value, parse_csv, parse_timestamp.
• Enrichers. to_syslog_level, get_env_var, md5, sha1, sha2, encode_base64, decode_base64, uuid_v4.
• String ops. replace, slice, split, join, starts_with, ends_with, contains, upcase, downcase, strip_whitespace.
• Time ops. now, format_timestamp, parse_timestamp, to_unix_timestamp, from_unix_timestamp.
• Object / array. merge, flatten, keys, values, length, map_values, filter, sort.
• Type / null. is_string, is_null, exists, type_def, string!, int!, bool!, object!.

Common interview probes on VRL.

• "Why VRL over Lua?" — required answer: compile-time type checking + guaranteed termination + sandbox.
• "What does the ! suffix mean?" — abort VRL on error.
• "How do you handle expected errors?" — tuple form: (result, err) = ....
• "Can VRL loop forever?" — required answer: no; no while / for / recursion.

Worked example — normalising a mixed JSON / syslog / plaintext log stream

Detailed explanation. A common production reality: one Kubernetes namespace has three apps emitting three different log formats — a modern service writing JSON, a legacy daemon writing RFC 5424 syslog, and a shell script writing bare plaintext. The observability contract downstream expects every event to have .level (INFO/WARN/ERROR), .message (the human-readable text), and .timestamp (ISO 8601). Write one VRL program that normalises all three inputs to that schema.

• Input variety. JSON (structured), syslog (RFC 5424), plaintext (unstructured).
• Output schema. .level, .message, .timestamp (mandatory) + any extra parsed fields (preserved).
• Failure mode. If nothing parses, emit .level = "INFO" and .message = <original> — never drop the event.

Question. Write a VRL program that normalises the three formats to a common schema and preserves parsed fields.

Input.

Input format	Raw message	Expected shape
JSON	{"lvl":"error","txt":"db down","code":500}	.level=ERROR, .message="db down", .code=500
Syslog	<134>1 2026-07-21T12:34:56Z host app 1234 ID [example] system started	.level=INFO, .message="system started", .facility=16
Plaintext	db connection pool exhausted	.level=INFO, .message="db connection pool exhausted"

Code.

[transforms.normalise]
type = "remap"
inputs = ["k8s_logs"]
source = '''
    original = string!(.message ?? "")

    # 1. Try JSON first
    parsed, err = parse_json(original)
    if err == null {
        # Merge parsed fields into the event
        . = merge(., object!(parsed))
        # Map common aliases: "lvl" -> level, "txt" -> message, "msg" -> message
        if exists(.lvl)  { .level   = upcase!(string!(.lvl));  del(.lvl) }
        if exists(.txt)  { .message = string!(.txt);           del(.txt) }
        if exists(.msg)  { .message = string!(.msg);           del(.msg) }
        .parser = "json"
    } else {
        # 2. Try syslog next
        parsed, err = parse_syslog(original)
        if err == null {
            . = merge(., object!(parsed))
            # parse_syslog emits .severity (0-7); map to text level
            .level = to_syslog_level!(int!(.severity ?? 6))
            .parser = "syslog"
        } else {
            # 3. Fallback: plaintext line
            .message = original
            .level   = "INFO"
            .parser  = "plaintext"
        }
    }

    # 4. Ensure required fields
    if !exists(.timestamp) { .timestamp = format_timestamp!(now(), format: "%+") }
    if !exists(.level)     { .level     = "INFO" }
    if !exists(.message)   { .message   = original }

    # 5. Normalise .level to canonical uppercase
    .level = upcase!(string!(.level))
'''

Step-by-step explanation.

1. original = string!(.message ?? "") snapshots the original message before mutation. string! asserts the type (aborts VRL if .message is somehow a number — should never happen but the type checker demands it).
2. Step 1 tries parse_json. The tuple-form (parsed, err) lets us branch: if JSON parses, we merge the parsed object into . (so {"code": 500} becomes .code = 500). We then map common field aliases (lvl → level, txt/msg → message) and del() the alias so downstream doesn't see both.
3. Step 2 (in the else branch) tries parse_syslog. RFC 5424 syslog is a structured format; parse_syslog returns fields like .severity (0-7 numeric), .facility (0-23 numeric), .hostname, .appname. We map .severity to a text .level via to_syslog_level!.
4. Step 3 (in the innermost else) is the plaintext fallback — we set .message to the raw line and .level = "INFO". Never drop an event just because it doesn't parse; observability values completeness over structure.
5. Steps 4-5 enforce the schema invariant: every event exiting normalise has .timestamp, .level, and .message. upcase!(string!(.level)) normalises .level to canonical uppercase (some sources emit "error", some emit "Error", some emit "ERROR" — pick one shape).

Output.

Input	.level	.message	Extra
{"lvl":"error","txt":"db down","code":500}	ERROR	db down	.code=500, .parser="json"
<134>1 2026-07-21T12:34:56Z ...	INFO	system started	.facility=16, .parser="syslog"
db connection pool exhausted	INFO	db connection pool exhausted	.parser="plaintext"

Rule of thumb. Every observability pipeline needs a "normalise to canonical schema" VRL transform sitting between raw sources and downstream sinks. Enforce .level, .message, .timestamp as invariants; fall through to INFO + raw message if no parser matches; never drop an event just because it didn't parse — the interviewer will ask "what happens to unparseable lines" and "we drop them" is the wrong answer.

Worked example — grok-based parsing of an Apache access log

Detailed explanation. The Apache access log is one of the most-parsed formats in observability. Vector ships parse_apache_log as a built-in, but the same result can be achieved with parse_grok for full pattern control. Grok patterns are the same language Logstash and Fluentd use — so a team migrating from Logstash can lift their existing %{COMBINEDAPACHELOG} pattern into a VRL parse_grok call. Walk through both approaches side by side.

• Input. Standard Apache combined log format: 127.0.0.1 - frank [10/Oct/2000:13:55:36 -0700] "GET /apache_pb.gif HTTP/1.0" 200 2326 "http://ref" "Mozilla/4.0".
• Approach A. parse_apache_log(., format: "combined").
• Approach B. parse_grok(., "%{COMBINEDAPACHELOG}").

Question. Parse the Apache combined log with both parse_apache_log and parse_grok, and compare the resulting event shape.

Input.

Approach	Function	Pattern arg
A	parse_apache_log	format: "combined"
B	parse_grok	%{COMBINEDAPACHELOG}

Code.

[transforms.parse_apache_a]
type = "remap"
inputs = ["apache_source"]
source = '''
    # Approach A — built-in Apache parser
    parsed, err = parse_apache_log(string!(.message), format: "combined")
    if err != null {
        log("parse_apache_log failed: " + err, level: "error")
        abort
    }
    . = merge(., object!(parsed))
    .parser = "apache_builtin"
'''

[transforms.parse_apache_b]
type = "remap"
inputs = ["apache_source"]
source = '''
    # Approach B — grok pattern
    parsed, err = parse_grok(string!(.message),
        "%{IPORHOST:client_ip} %{USER:ident} %{USER:auth} \\[%{HTTPDATE:timestamp}\\] " +
        "\"%{WORD:method} %{DATA:request} HTTP/%{NUMBER:http_version}\" " +
        "%{NUMBER:status:int} %{NUMBER:bytes:int} " +
        "\"%{DATA:referrer}\" \"%{DATA:user_agent}\""
    )
    if err != null {
        log("parse_grok failed: " + err, level: "error")
        abort
    }
    . = merge(., object!(parsed))
    # Convert timestamp string to VRL timestamp
    .timestamp = parse_timestamp!(string!(.timestamp), "%d/%b/%Y:%H:%M:%S %z")
    .parser = "apache_grok"
'''

Step-by-step explanation.

1. Approach A calls parse_apache_log with format: "combined" — Vector recognises the standard Apache combined log format and returns a pre-typed object with .client, .identity, .user, .timestamp, .method, .path, .protocol, .status, .bytes, .referrer, .agent. The types are pre-set (.status is int, .timestamp is a VRL timestamp).
2. Approach B uses parse_grok with a literal pattern. %{PATTERN:field} captures the pattern into .field; %{PATTERN:field:type} adds a type cast (:int in the code above). Grok patterns are more verbose but let you handle custom formats that don't match a built-in.
3. The log("...", level: "error") VRL function emits a Vector-internal log — useful for debugging why a specific event failed to parse. abort then drops the event from the pipeline (or with reroute_dropped = true on the transform, sends it to a .dropped output for a dead-letter sink).
4. Note the \\[ escaping in the grok pattern — inside a VRL string literal, \\[ becomes \[ which grok interprets as a literal [. Grok patterns quickly become read-once-run-forever; keep them in a Git-tracked, well-commented file.
5. Approach A is preferred when a built-in exists (fewer lines, faster, less error-prone). Approach B is required when the format is custom or you're migrating a Logstash pattern verbatim.

Output.

// Approach A output (parse_apache_log)
{
  "client":    "127.0.0.1",
  "identity":  null,
  "user":      "frank",
  "timestamp": "2000-10-10T20:55:36Z",
  "method":    "GET",
  "path":      "/apache_pb.gif",
  "protocol":  "HTTP/1.0",
  "status":    200,
  "bytes":     2326,
  "referrer":  "http://ref",
  "agent":     "Mozilla/4.0",
  "parser":    "apache_builtin"
}

// Approach B output (parse_grok) — same fields, slightly different names
{
  "client_ip":    "127.0.0.1",
  "ident":        null,
  "auth":         "frank",
  "timestamp":    "2000-10-10T20:55:36Z",
  "method":       "GET",
  "request":      "/apache_pb.gif",
  "http_version": "1.0",
  "status":       200,
  "bytes":        2326,
  "referrer":     "http://ref",
  "user_agent":   "Mozilla/4.0",
  "parser":       "apache_grok"
}

Rule of thumb. For any well-known log format (Apache, Nginx, syslog, JSON, key-value), use the Vector built-in parser (parse_apache_log, parse_nginx_log, parse_syslog, parse_json, parse_key_value) — fewer lines, better types, faster runtime. Fall back to parse_grok only for custom formats or Logstash migrations. Never write a hand-rolled regex when a built-in parser exists.

Worked example — deduplicating events with dedupe + VRL key

Detailed explanation. A common production pain: an app misconfiguration causes 10,000 duplicate log lines per second for the same error message, DOSing Datadog's log ingest quota. The fix is a dedupe transform keyed on a VRL-computed identity (e.g. (.host, .service, .message[:100])) with a bounded LRU cache. Walk through the config.

• Problem. Duplicate spam at 10k/sec.
• Solution. dedupe transform with LRU cache; VRL computes the dedupe key.
• Trade-off. LRU cache uses O(N) memory; too small = false negatives (dupes leak through); too large = memory pressure.

Question. Configure a dedupe transform that drops repeated errors within a 5-second window.

Input.

Parameter	Value
Dedupe key	.host + .service + slice(.message, 0, 100)
Cache size	10,000 entries
Cache TTL (effective)	5 sec (via cache turnover under load)
Drop policy	drop repeats within the window

Code.

[transforms.dedupe_key]
type = "remap"
inputs = ["apache_source"]
source = '''
    # Compute a stable dedupe key. Truncating .message to 100 chars
    # keys similar-but-not-identical errors together.
    .dedupe_key = string!(.host ?? "-") + "|" +
                  string!(.service ?? "-") + "|" +
                  slice!(string!(.message ?? ""), 0, 100)
'''

[transforms.dedupe_errors]
type = "dedupe"
inputs = ["dedupe_key"]
# Cache 10k most recently seen keys.
cache.num_events = 10000
# Only these fields identify the event uniquely.
fields.match = ["dedupe_key"]

Step-by-step explanation.

1. The dedupe transform maintains an LRU cache of cache.num_events most recently seen events, keyed by the fields listed in fields.match. Incoming events whose key hits the cache are dropped; misses are emitted and inserted into the cache.
2. We compute the dedupe key in a preceding remap transform so the key computation is testable and inspectable. slice!(string!(.message ?? ""), 0, 100) takes the first 100 chars of the message — key long enough to distinguish different errors but short enough to collapse near-identical spam.
3. cache.num_events = 10000 sizes the LRU. At 10k events/sec of unique keys, the cache turns over roughly every second — so the effective dedupe window is short. To dedupe over longer windows, increase the cache size (10k events × ~200 bytes = ~2 MB memory).
4. fields.match = ["dedupe_key"] restricts the match to a single field. You could match on multiple fields directly (fields.match = ["host", "service"]) but using a single pre-computed VRL field is more explicit and easier to debug.
5. The dropped events show up in Vector's internal metrics as dedupe_events_dropped_total. Set an alert on sudden spikes — a spike means an app started spamming, which is often the actual bug you want to know about.

Output.

Event	dedupe_key	Cache state	Action
{host: h1, service: api, msg: "db down"}	`h1\	api\	db down`
{host: h1, service: api, msg: "db down"}	`h1\	api\	db down`
{host: h1, service: api, msg: "db back"}	`h1\	api\	db back`
{host: h2, service: api, msg: "db down"}	`h2\	api\	db down`

Rule of thumb. Every observability pipeline that ingests app logs directly should sit a dedupe transform behind a VRL-computed key. Alert on the drop count as a proxy for "someone's spamming." Never dedupe on the full raw message — small variations (a timestamp inside the message, an incrementing request-id) defeat the cache; slice / normalise first.

Senior interview question on VRL

A senior interviewer might ask: "You have a Kafka topic of raw log lines from 20 microservices in three formats (JSON, syslog, plaintext). Downstream Datadog wants a normalised schema with .level, .message, .timestamp, .service, and a canonical .severity (0-7). Design the VRL transform, handle parse failures gracefully, and route unparseable events to a dead-letter Kafka topic."

Solution Using a multi-parser VRL cascade with fallible-assignment + dead-letter reroute

[sources.raw_logs]
type = "kafka"
bootstrap_servers = "kafka-1:9092,kafka-2:9092"
group_id = "vector-normalise"
topics = ["logs.raw"]
acknowledgements.enabled = true

[transforms.normalise]
type = "remap"
inputs = ["raw_logs"]
# CRITICAL: reroute failed events to the .dropped output for the dead-letter sink
reroute_dropped = true
source = '''
    original = string!(.message ?? "")

    # 1. JSON attempt
    parsed_json, err_json = parse_json(original)
    if err_json == null {
        . = merge(., object!(parsed_json))
        .parser = "json"
    } else {
        # 2. Syslog attempt
        parsed_sys, err_sys = parse_syslog(original)
        if err_sys == null {
            . = merge(., object!(parsed_sys))
            .parser = "syslog"
        } else {
            # 3. Plaintext fallback (never fails)
            .message = original
            .parser  = "plaintext"
        }
    }

    # 4. Enforce required fields
    if !exists(.service) {
        .service = string!(.kubernetes.pod_labels.app ?? "unknown")
    }
    if !exists(.timestamp) {
        .timestamp = format_timestamp!(now(), format: "%+")
    }

    # 5. Canonicalise .level (many aliases) → uppercase
    if !exists(.level) && exists(.lvl)      { .level = .lvl;      del(.lvl) }
    if !exists(.level) && exists(.severity) { .level = to_syslog_level!(int!(.severity)) }
    if !exists(.level)                       { .level = "INFO" }
    .level = upcase!(string!(.level))

    # 6. Compute canonical numeric severity (0-7) from .level
    .severity = if      .level == "EMERG"                          { 0 }
                else if .level == "ALERT"                          { 1 }
                else if .level == "CRIT"     || .level == "CRITICAL" { 2 }
                else if .level == "ERR"      || .level == "ERROR"    { 3 }
                else if .level == "WARN"     || .level == "WARNING"  { 4 }
                else if .level == "NOTICE"                          { 5 }
                else if .level == "INFO"                            { 6 }
                else if .level == "DEBUG"    || .level == "TRACE"    { 7 }
                else                                                { 6 }

    # 7. Validation — if we still lack .message, mark as failed and abort
    if !exists(.message) || string!(.message) == "" {
        .dropped_reason = "no_message"
        abort
    }
'''

# Successful events → Datadog
[sinks.datadog]
type = "datadog_logs"
inputs = ["normalise"]
default_api_key = "${DD_API_KEY}"
compression = "gzip"

# Failed events → dead-letter Kafka topic
[sinks.dead_letter]
type = "kafka"
inputs = ["normalise.dropped"]
bootstrap_servers = "kafka-1:9092,kafka-2:9092"
topic = "logs.dead-letter"
encoding.codec = "json"

Step-by-step trace.

Step	VRL	Result
1. JSON parse	(parsed, err) = parse_json(original)	merge on success
2. Syslog fallback	(parsed, err) = parse_syslog(original)	merge on success
3. Plaintext fallback	.message = original	always succeeds
4. Derive .service	from .kubernetes.pod_labels.app	never empty
5. Canonicalise .level	aliases + upcase	one canonical spelling
6. Numeric .severity	if-else cascade to 0-7	Datadog-friendly
7. Validate .message	abort if missing	routes to .dropped sink

After deployment, ~99.5% of events flow through normalise → datadog_logs with the canonical schema; ~0.5% land in logs.dead-letter where an on-call inspects them and either fixes the source app or adds a fourth parser branch. Vector's internal metric component_events_out_total{output="dropped"} gives a clean dashboard for the failure rate.

Output:

Metric	Value
Parse success rate	99.5%
Dead-letter rate	0.5% (fixable at source)
Schema compliance	100% of successful events have level/message/timestamp/service/severity
Failure isolation	1 event out of 200,000 fails; the other 199,999 flow normally
Recovery on parser fix	replay dead-letter topic through updated normalise

Why this works — concept by concept:

• Multi-parser cascade — JSON → syslog → plaintext. Each parser attempt uses the tuple-form (parsed, err) = parse_X(...) so failures are visible and handled. The compiler forces the error handling — there is no way to accidentally ignore a parse failure.
• reroute_dropped = true + .dropped output — VRL's abort normally drops the event silently. With reroute_dropped = true, aborted events land on the transform's .dropped output, which we wire to a Kafka dead-letter sink. Zero silent drops; every failure is investigable.
• Canonical .level + numeric .severity — the "many aliases → one canonical spelling" pattern that every downstream dashboard depends on. Computing .severity from .level gives Datadog / SIEM the numeric axis they filter and colour by.
• Schema invariants enforced by abort — the step-7 check ("no message → abort") is what makes downstream Datadog dashboards reliable. Rather than shipping malformed events that break the UI, we reroute them for triage.
• Cost — one remap transform in the hot path (~2-5 microseconds per event on a modern CPU), a Kafka dead-letter topic (single partition, low retention), and a dashboard on the dead-letter rate. The eliminated cost is the class of "our Datadog dashboard is showing weird events" investigations. Net O(1) per event; the cascade is short-circuit so successful JSON parses never touch the syslog branch.

Streaming
Topic — streaming
Streaming problems on event transformation and DLQs

Practice →

Design Topic — design Design problems on log-normalisation pipelines

Practice →

---

4. Deployment topologies — agent, aggregator, gateway

Three tiers, each with a distinct purpose — agents at the source, aggregators near the region, gateways at the seam

The mental model in one line: Vector deploys in up to three tiers — the **agent tier (one Vector per node/pod, source-adjacent, cheap), the aggregator tier (a handful of Vectors per region receiving from agents, buffering with disk, routing to durable sinks), and the gateway tier (cross-region multiplexers that fan out to global destinations) — each tier optional depending on scale, each independently disk-buffered, each hot-reloadable via SIGHUP or the /reload API without dropping in-flight events**. Every senior platform engineer earns credibility not by naming Vector but by naming which of the three tiers a given workload actually needs.

The three tiers — what each does and when you need it.

• Agent tier. One Vector per node (Kubernetes DaemonSet) or per host (systemd unit). Job: read source-local data (container logs, host metrics, journald), enrich with node/pod metadata, ship to the next tier. Small memory footprint (100-200 MB). Always present in any Vector deployment.
• Aggregator tier. A regional pool of Vectors (Kubernetes StatefulSet, 3-6 replicas). Job: receive from agents, buffer with disk, perform heavy transforms (aggregation, tag-cardinality-limit, sampling), route to durable sinks (Datadog, S3, Elasticsearch). Larger memory (2-8 GB). Present in any deployment past ~50 nodes; optional but recommended for smaller clusters.
• Gateway tier. A cross-region multiplexer (a handful of Vectors per cluster). Job: receive from aggregators in one region, forward to sinks in another region or another cloud. Present only in multi-region or hybrid-cloud deployments; optional for single-region.

Why the agent tier alone is not enough.

• Per-node sink authentication. Every agent needs the Datadog API key, the S3 IAM role, the Elasticsearch password. Distributing 500 secrets to 500 nodes is a compliance headache; distributing 6 secrets to 6 aggregators is not.
• Sink connection pool churn. 500 agents each opening 5 sink connections = 2500 open connections against Datadog / Elasticsearch. 6 aggregators each opening 5 = 30. Downstream services often have connection-count limits.
• Per-node config drift. Rolling a config change across 500 agents takes 30+ minutes even with kubectl rollout. Rolling across 6 aggregators takes 30 seconds.
• Cost of heavy transforms. Running aggregate (metric aggregation) on 500 agents means 500 independent aggregation windows — you lose the cross-node aggregate. Running on 6 aggregators after all agents feed in gives you the true regional aggregate.

The disk buffer + memory buffer + backpressure trifecta.

• Memory buffer. Default for every sink. Fast (nanoseconds per event) but bounded by process memory; a large memory buffer can OOM the container.
• Disk buffer. Opt-in per sink via [sinks.NAME.buffer] type = "disk" + max_size = <bytes>. Slower than memory (microseconds per event, dominated by fsync) but survives Vector crash (persisted to data_dir) and can be sized to gigabytes without memory pressure.
• Backpressure. When a buffer fills, when_full = block propagates back-pressure to the upstream transform, which propagates to the upstream source. Sources that support pull-model (Kafka, Kinesis, file) pause fetching; sources that support push-model (http_server, socket) apply TCP backpressure to the sender.

Hot-reload — SIGHUP or /reload API.

• SIGHUP. kill -HUP <vector-pid> triggers a config re-read. Vector loads the new config, validates it, brings up the new pipeline components, drains the old pipeline components, then swaps. No in-flight events are dropped; a small window of duplicate delivery is possible if a source doesn't support acks.
• /reload API. POST to Vector's admin API endpoint (typically http://localhost:8686/reload). Same semantics as SIGHUP but scriptable and returns a JSON status.
• What hot-reload does not do. Change data_dir, change api.enabled, or migrate on-disk buffer data between config schemas. Those require a full restart.

Distributed configuration — consul, etcd, Kubernetes ConfigMap.

• Kubernetes ConfigMap. The most common pattern. Mount the config as a volume; a sidecar (or a kubectl rollout restart triggered by a ConfigMap-change controller) triggers reload.
• Consul / etcd. For non-Kubernetes fleets. Vector doesn't natively watch Consul, so a small sidecar (consul-template, confd) renders the config and SIGHUPs Vector on change.
• GitOps. Config lives in git; ArgoCD / Flux reconciles it into ConfigMaps; Vector reloads on ConfigMap change. This is the auditable, rollback-friendly pattern for large deployments.

Common interview probes on topology.

• "When do you add an aggregator tier?" — required answer: at ~50 nodes or when sink connection count / secret distribution matters.
• "How does Vector hot-reload?" — SIGHUP or /reload; no dropped events.
• "Where does the disk buffer live?" — under data_dir on the pod's local disk.
• "How do you avoid OOM on a busy agent?" — disk buffer + when_full = block + capped memory buffer.

Worked example — a 200-node cluster: agent DaemonSet + 6-replica aggregator StatefulSet

Detailed explanation. The canonical mid-size Kubernetes Vector deployment: a DaemonSet of Vector agents on every node (200 pods), forwarding via the vector source/sink protocol to a StatefulSet of 6 aggregator replicas that route to Datadog + S3 + Elasticsearch. Walk through the Helm chart shape.

• Agent tier. DaemonSet, one pod per node, ~150 MB memory, reads kubernetes_logs + host_metrics.
• Aggregator tier. StatefulSet, 6 replicas across 3 AZs, ~4 GB memory each, receives from agents, routes to Datadog + S3 + ES.
• Wire protocol. vector source and sink (Vector-native gRPC-like protocol with acks + compression).

Question. Define the Kubernetes manifests for the agent DaemonSet + aggregator StatefulSet and show how they connect.

Input.

Component	Kubernetes kind	Replicas	Memory
Agent	DaemonSet	200 (per node)	150 MB
Aggregator	StatefulSet	6 (per region)	4 GB
Wire protocol	vector (native)	—	compressed
Data flow	agent → aggregator → sinks	—	acknowledged

Code.

# ConfigMap for the agent DaemonSet (per-node Vector)
apiVersion: v1
kind: ConfigMap
metadata:
  name: vector-agent-config
  namespace: observability
data:
  vector.toml: |
    data_dir = "/var/lib/vector"

    [api]
    enabled = true
    address = "0.0.0.0:8686"

    [sources.k8s_logs]
    type = "kubernetes_logs"

    [sources.host_metrics]
    type = "host_metrics"
    scrape_interval_secs = 30

    [sinks.aggregator]
    type    = "vector"
    inputs  = ["k8s_logs", "host_metrics"]
    address = "vector-aggregator.observability.svc.cluster.local:6000"
    compression = true
    acknowledgements.enabled = true

    [sinks.aggregator.buffer]
    type      = "disk"
    max_size  = 5368709120        # 5 GB per node
    when_full = "block"
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: vector-agent
  namespace: observability
spec:
  selector:
    matchLabels: { app: vector-agent }
  template:
    metadata:
      labels: { app: vector-agent }
    spec:
      serviceAccountName: vector
      containers:
      - name: vector
        image: timberio/vector:0.40.0-debian
        resources:
          requests: { cpu: 100m, memory: 128Mi }
          limits:   { cpu: 500m, memory: 300Mi }
        volumeMounts:
        - { name: config, mountPath: /etc/vector }
        - { name: data,   mountPath: /var/lib/vector }
        - { name: varlog, mountPath: /var/log, readOnly: true }
        - { name: varlibdockercontainers, mountPath: /var/lib/docker/containers, readOnly: true }
      volumes:
      - { name: config, configMap: { name: vector-agent-config } }
      - { name: data,   hostPath: { path: /var/lib/vector, type: DirectoryOrCreate } }
      - { name: varlog, hostPath: { path: /var/log } }
      - { name: varlibdockercontainers, hostPath: { path: /var/lib/docker/containers } }

# ConfigMap + StatefulSet for the aggregator tier
apiVersion: v1
kind: ConfigMap
metadata:
  name: vector-aggregator-config
  namespace: observability
data:
  vector.yaml: |
    data_dir: /var/lib/vector

    sources:
      from_agents:
        type: vector
        address: 0.0.0.0:6000
        acknowledgements:
          enabled: true

    transforms:
      route:
        type: route
        inputs: [from_agents]
        route:
          logs:    ".log_type == \"application\" || exists(.message)"
          metrics: "exists(.name) && exists(.value)"

    sinks:
      datadog:
        type: datadog_logs
        inputs: [route.logs]
        default_api_key: ${DD_API_KEY}
        compression: gzip
        acknowledgements: { enabled: true }
        buffer: { type: disk, max_size: 21474836480, when_full: block }

      s3:
        type: aws_s3
        inputs: [route.logs]
        bucket: acme-logs-lake
        key_prefix: date=%Y-%m-%d/hour=%H/
        compression: zstd
        acknowledgements: { enabled: true }
        buffer: { type: disk, max_size: 10737418240, when_full: block }

      es:
        type: elasticsearch
        inputs: [route.logs]
        endpoints: [https://es.internal:9200]
        acknowledgements: { enabled: true }
        buffer: { type: disk, max_size: 10737418240, when_full: block }

      prom:
        type: prometheus_remote_write
        inputs: [route.metrics]
        endpoint: https://prometheus.internal/api/v1/write
---
apiVersion: v1
kind: Service
metadata:
  name: vector-aggregator
  namespace: observability
spec:
  clusterIP: None      # headless — agents reach a specific replica via DNS
  selector: { app: vector-aggregator }
  ports:
  - { name: agent, port: 6000, targetPort: 6000 }
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: vector-aggregator
  namespace: observability
spec:
  serviceName: vector-aggregator
  replicas: 6
  selector: { matchLabels: { app: vector-aggregator } }
  template:
    metadata: { labels: { app: vector-aggregator } }
    spec:
      containers:
      - name: vector
        image: timberio/vector:0.40.0-debian
        resources:
          requests: { cpu: 500m, memory: 2Gi }
          limits:   { cpu: 2,    memory: 4Gi }
        volumeMounts:
        - { name: config, mountPath: /etc/vector }
        - { name: data,   mountPath: /var/lib/vector }
      volumes:
      - { name: config, configMap: { name: vector-aggregator-config } }
  volumeClaimTemplates:
  - metadata: { name: data }
    spec:
      accessModes: [ReadWriteOnce]
      resources: { requests: { storage: 50Gi } }
      storageClassName: gp3

Step-by-step explanation.

1. The agent DaemonSet has one pod per node. It mounts /var/log and /var/lib/docker/containers read-only so kubernetes_logs can read container log files, and /var/lib/vector from a hostPath for the disk buffer (5 GB per node). Memory limit 300 MB — enough headroom for the buffer's memory index but not so large that we OOM under bursts.
2. The agent config's only sink is vector pointing at vector-aggregator.observability.svc.cluster.local:6000 — the headless service that resolves to the 6 aggregator replicas via DNS round-robin. Each connection lands on one replica; distinct connections balance across replicas.
3. The aggregator StatefulSet uses volumeClaimTemplates for a 50 GB persistent volume per replica — this backs the disk buffers for the three sinks (Datadog 20 GB, S3 10 GB, ES 10 GB + slack). StatefulSet is chosen (rather than Deployment) because we want stable DNS names (vector-aggregator-0, vector-aggregator-1, ...) and persistent disks across pod restarts.
4. The aggregator config declares one vector source on port 6000 and four sinks. The route transform splits application logs from metrics; each type goes to its appropriate sinks. Every sink has acknowledgements.enabled = true so the aggregator only acks the agent's vector sink after all its own sinks have committed — end-to-end ack across two tiers.
5. Rolling a config change: kubectl edit configmap vector-agent-config → kubectl rollout restart daemonset/vector-agent for the agents (~5 min for 200 nodes), or use a sidecar that watches the ConfigMap and SIGHUPs Vector for zero-downtime reload. The aggregator side uses a smaller rollout (6 replicas, ~30s).

Output.

Layer	Pod count	Memory	Purpose
Agent DaemonSet	200 (one per node)	150-300 MB each	source-local read, forward to aggregator
Aggregator StatefulSet	6 (per region)	2-4 GB each	receive, buffer, transform, route
Total memory	~30 GB agents + 24 GB aggregators	~54 GB	across the cluster
Sink connections	6 aggregators × 4 sinks = 24	(not 200 × 4 = 800)	one order of magnitude fewer
Config rollouts	one ConfigMap per tier	~5 min agents, 30s aggregators	scriptable

Rule of thumb. For any Kubernetes cluster past ~50 nodes, deploy the agent + aggregator two-tier pattern. Below 50 nodes, agent-only is fine — the aggregator overhead outweighs the benefit. The aggregator replica count is roughly ceil(node_count / 40) with a floor of 3 for AZ-spread resilience.

Worked example — SIGHUP hot-reload with zero dropped events

Detailed explanation. The operational reality of running observability agents at scale is that config rollouts happen weekly. Restarting Vector to pick up a new config drops in-flight buffered events (up to the disk buffer size); hot-reload via SIGHUP is the answer. Walk through the reload lifecycle and the guarantees it provides.

• Trigger. Send SIGHUP to the Vector process, or POST to http://localhost:8686/reload.
• Lifecycle. Load new config → validate → bring up new components → drain old components → swap.
• Guarantees. No dropped in-flight events; brief window of duplicate delivery possible for non-acked sources.

Question. Show how a ConfigMap change triggers a zero-drop reload of the agent DaemonSet.

Input.

Trigger	Mechanism	Duration	Data loss
ConfigMap change	fs-watching sidecar → SIGHUP	~1 second	none
Direct SIGHUP	kill -HUP $(pgrep vector)	~1 second	none
/reload API	curl -XPOST http://localhost:8686/reload	~1 second	none
Pod restart	kubectl rollout restart	~10 seconds	up to memory buffer size

Code.

# Add a config-reloader sidecar to the agent DaemonSet
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: vector-agent
  namespace: observability
spec:
  template:
    spec:
      containers:
      - name: vector
        image: timberio/vector:0.40.0-debian
        # ... unchanged from before ...

      - name: config-reloader
        image: jimmidyson/configmap-reload:v0.9.0
        args:
        - --volume-dir=/etc/vector
        - --webhook-url=http://localhost:8686/reload
        - --webhook-method=POST
        volumeMounts:
        - { name: config, mountPath: /etc/vector, readOnly: true }

      shareProcessNamespace: true    # sidecar can also SIGHUP if webhook is disabled

# Manual reload examples
# 1. Direct SIGHUP inside the pod
kubectl exec -n observability -it vector-agent-abc12 -c vector -- \
    kill -HUP 1

# 2. /reload API from outside (via kubectl port-forward)
kubectl -n observability port-forward daemonset/vector-agent 8686:8686 &
curl -XPOST http://localhost:8686/reload

# 3. Bulk reload across the whole DaemonSet
kubectl -n observability get pods -l app=vector-agent -o name | \
    xargs -I {} kubectl -n observability exec {} -c vector -- kill -HUP 1

# 4. Check reload status in Vector's logs
kubectl -n observability logs -l app=vector-agent -c vector --tail=20 | grep -E "(reload|Reload)"

Step-by-step explanation.

1. The config-reloader sidecar watches /etc/vector (the mounted ConfigMap). When Kubernetes updates the ConfigMap (which triggers a symlink swap in the mounted directory), the sidecar detects the change and POSTs to http://localhost:8686/reload. The Vector process receives the reload request and executes the lifecycle.
2. Vector's reload lifecycle: (a) parse and validate the new config, (b) diff against the current config, (c) for changed components, bring up the new one alongside the old one, (d) route new events to the new component, (e) drain the old component's in-flight events, (f) shut down the old component. In-flight events either land on the new component or drain through the old one — never dropped.
3. The shareProcessNamespace: true field lets the sidecar send signals to sibling containers — a fallback in case the webhook is disabled. kill -HUP 1 targets PID 1 in the shared namespace, which is Vector's process.
4. For a full DaemonSet reload, the shell one-liner (xargs kubectl exec ...) SIGHUPs every agent. In production, prefer the sidecar pattern — it auto-triggers on any ConfigMap change without operator intervention.
5. Log inspection is your reload verification. Vector emits Reload complete on success; Reload failed: <error> on validation failure (in which case Vector keeps the old config running — no downtime).

Output.

Reload trigger	New config live	Old config drain	Total downtime
SIGHUP	~500 ms	~500 ms	0 (parallel drain)
/reload API	~500 ms	~500 ms	0
ConfigMap change + sidecar	~2s ConfigMap propagation + ~1s reload	~500 ms	0
Pod restart	~5s pod termination + 5s new pod start	(memory buffer lost)	up to buffer size

Rule of thumb. For any production Vector deployment, install the config-reloader sidecar and drive config changes via ConfigMap updates. Never kubectl rollout restart a Vector DaemonSet just to pick up a config change — you'll lose the in-memory buffers on every pod. Hot-reload is free and correct; use it.

Worked example — disk buffer sizing for a 30-minute Datadog outage

Detailed explanation. The disk buffer is Vector's answer to "what happens when Datadog is down." Sizing it requires knowing (event rate) × (worst-case outage duration) × (average event size) × (safety margin). Undersize and you drop events during an outage; oversize and you waste disk. Walk through the calculation for a 200-node cluster with 250 events/sec/node.

• Event rate per node. 250 events/sec (mixed logs).
• Average event size. ~1 KB (post-VRL enrichment).
• Worst-case outage. 30 minutes.
• Safety margin. 2× for burst tolerance.

Question. Compute the disk buffer size per aggregator replica for the datadog sink and set it in the aggregator config.

Input.

Parameter	Value	Formula
Nodes	200	given
Events/sec/node	250	measured
Events/sec/cluster	50,000	200 × 250
Aggregator replicas	6	given
Events/sec/replica	~8,333	50,000 / 6
Event size	1 KB	measured
Outage duration	30 min = 1800 s	worst case
Safety margin	2×	production buffer

Code.

# Disk buffer sizing calculator
NODES               = 200
EVENTS_PER_SEC_NODE = 250
AGGREGATOR_REPLICAS = 6
EVENT_SIZE_KB       = 1.0
OUTAGE_DURATION_S   = 1800   # 30 minutes
SAFETY_MARGIN       = 2.0

events_per_sec_cluster  = NODES * EVENTS_PER_SEC_NODE
events_per_sec_replica  = events_per_sec_cluster / AGGREGATOR_REPLICAS
events_per_outage       = events_per_sec_replica * OUTAGE_DURATION_S
bytes_per_outage        = events_per_outage * EVENT_SIZE_KB * 1024
buffer_bytes            = bytes_per_outage * SAFETY_MARGIN
buffer_gb               = buffer_bytes / (1024**3)

print(f"Events/sec/cluster:  {events_per_sec_cluster:>10,}")
print(f"Events/sec/replica:  {events_per_sec_replica:>10,.0f}")
print(f"Events during outage: {events_per_outage:>10,.0f}")
print(f"Bytes during outage:  {bytes_per_outage:>10,.0f}  ({bytes_per_outage/1024**3:.1f} GB)")
print(f"Buffer with 2x margin: {buffer_bytes:>10,.0f}  ({buffer_gb:.1f} GB)")

# → Events/sec/cluster:      50,000
# → Events/sec/replica:        8,333
# → Events during outage:  15,000,000
# → Bytes during outage:  15,360,000,000  (14.3 GB)
# → Buffer with 2x margin: 30,720,000,000  (28.6 GB)

# Aggregator vector.toml — Datadog sink with computed buffer
[sinks.datadog]
type = "datadog_logs"
inputs = ["route.logs"]
default_api_key = "${DD_API_KEY}"
compression = "gzip"
acknowledgements.enabled = true

[sinks.datadog.buffer]
type      = "disk"
max_size  = 30720000000        # ~28.6 GB — 30-min outage tolerance with 2x margin
when_full = "block"            # backpressure into agent tier rather than drop

# Corresponding PVC size in the StatefulSet must be >= 30 GB per sink
# (bump volumeClaimTemplates.data.spec.resources.requests.storage accordingly)

Step-by-step explanation.

1. Compute per-replica load: total cluster event rate divided across replicas. At 50k events/sec cluster-wide and 6 replicas, each replica handles ~8,333 events/sec — Kubernetes service load-balancing distributes the agent connections roughly evenly.
2. Multiply per-replica rate by outage duration to get events accumulated during the outage. 8,333 × 1800 = 15M events per replica during a 30-min outage.
3. Multiply by average event size (post-enrichment) to get raw bytes. 15M × 1 KB = 15 GB per replica. Post-compression (gzip inside the disk buffer) this would be smaller, but sizing on uncompressed bytes is the safe default.
4. Apply the safety margin (2×) to account for bursts, longer-than-expected outages, and the fact that estimates are usually low. 15 GB × 2 = ~30 GB per replica → 30 GB per replica disk buffer for the Datadog sink.
5. The corresponding PVC size in the StatefulSet must be at least the sum of all sink buffers on that replica plus a small overhead (10%). If Datadog needs 30 GB, S3 needs 10 GB, ES needs 10 GB, the PVC should be ≥ 55 GB per replica.

Output.

Metric	Value
Per-replica event rate	8,333/sec
Events during 30-min outage	15,000,000
Raw bytes	~15 GB
Buffer size (2× margin)	~30 GB
Total PVC (all sinks)	~55 GB
Cluster-wide disk cost (6 replicas × 55 GB)	330 GB

Rule of thumb. Size disk buffers to (events/sec/replica × worst_case_outage × event_size × 2). Under-sizing costs data during an outage; over-sizing costs disk. Ballpark 30 GB per Datadog sink per aggregator replica in a mid-size cluster is the right order of magnitude. Set when_full = block so that if the buffer fills anyway (outage longer than sizing assumed), you get backpressure rather than silent drops.

Senior interview question on topology

A senior interviewer might ask: "You're bootstrapping observability for a new 500-node Kubernetes cluster across 3 AZs, with primary sink Datadog + backup S3 lake. You need agent + aggregator topology, disk buffering sized for a 30-minute Datadog outage, hot-reload without dropping events, and the whole rollout via ArgoCD + GitOps. Walk me through the tier sizing, the Helm chart layout, and the on-call runbook when an aggregator replica dies."

Solution Using a two-tier DaemonSet + StatefulSet with GitOps + zero-drop reload

# vector-agent Helm values (rendered into DaemonSet ConfigMap)
config:
  data_dir: /var/lib/vector
  api: { enabled: true, address: 0.0.0.0:8686 }

  sources:
    k8s_logs:    { type: kubernetes_logs }
    host_metrics: { type: host_metrics, scrape_interval_secs: 30 }

  sinks:
    aggregator:
      type: vector
      inputs: [k8s_logs, host_metrics]
      address: vector-aggregator.observability.svc.cluster.local:6000
      compression: true
      acknowledgements: { enabled: true }
      buffer:
        type: disk
        max_size: 5368709120        # 5 GB per node
        when_full: block

daemonset:
  resources:
    requests: { cpu: 100m, memory: 128Mi }
    limits:   { cpu: 500m, memory: 300Mi }
  extraContainers:
    - name: config-reloader
      image: jimmidyson/configmap-reload:v0.9.0
      args: [--volume-dir=/etc/vector, --webhook-url=http://localhost:8686/reload, --webhook-method=POST]

# vector-aggregator Helm values (rendered into StatefulSet ConfigMap)
config:
  data_dir: /var/lib/vector
  api: { enabled: true, address: 0.0.0.0:8686 }

  sources:
    from_agents:
      type: vector
      address: 0.0.0.0:6000
      acknowledgements: { enabled: true }

  transforms:
    route:
      type: route
      inputs: [from_agents]
      route:
        logs:    ".log_type == \"application\" || exists(.message)"
        metrics: "exists(.name) && exists(.value)"

  sinks:
    datadog:
      type: datadog_logs
      inputs: [route.logs]
      default_api_key: ${DD_API_KEY}
      compression: gzip
      acknowledgements: { enabled: true }
      buffer: { type: disk, max_size: 32212254720, when_full: block }   # 30 GB

    s3:
      type: aws_s3
      inputs: [route.logs]
      bucket: acme-logs-lake
      key_prefix: cluster=prod-us-east/date=%Y-%m-%d/hour=%H/
      compression: zstd
      acknowledgements: { enabled: true }
      buffer: { type: disk, max_size: 10737418240, when_full: block }   # 10 GB

    prom:
      type: prometheus_remote_write
      inputs: [route.metrics]
      endpoint: https://prometheus.internal/api/v1/write

statefulset:
  replicas: 12                       # 500 nodes / ~40 nodes-per-agg = 12
  podAntiAffinity: true              # spread across AZs
  resources:
    requests: { cpu: 500m, memory: 2Gi }
    limits:   { cpu: 2,    memory: 4Gi }
  persistence:
    size: 60Gi                        # 30 GB DD + 10 GB S3 + 10 GB slack + 10 GB slack
    storageClassName: gp3

# ArgoCD Application — GitOps controller for the whole stack
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: vector-observability
  namespace: argocd
spec:
  project: platform
  source:
    repoURL: https://git.example.com/platform/observability.git
    path: charts/vector
    targetRevision: main
  destination:
    server: https://kubernetes.default.svc
    namespace: observability
  syncPolicy:
    automated: { prune: true, selfHeal: true }

Step-by-step trace.

Concern	Answer	Reasoning
Agent tier	DaemonSet, 500 pods, 150-300 MB	one per node; source-local read
Aggregator tier	StatefulSet, 12 replicas, 4 GB, 60 GB PVC	~40 nodes per agg; 3 replicas per AZ
Datadog buffer	30 GB per replica	30-min outage tolerance with 2× margin
Hot-reload	config-reloader sidecar → /reload API	zero dropped events
GitOps	ArgoCD watches git repo	audit trail; rollback = git revert
Runbook (agg dies)	K8s reschedules; PVC persists; agents reconnect	zero-op recovery

After deployment, the 500 agents forward to 12 aggregators via DNS-load-balanced connections; a 30-minute Datadog outage queues ~360M events across the 12 aggregators (30 GB each × 12 = 360 GB total buffered) and drains cleanly on recovery. Config rollouts happen via git commit → ArgoCD reconciles ConfigMap → sidecar SIGHUPs Vector → zero-downtime reload. When an aggregator replica dies, Kubernetes reschedules it onto another node, the PVC reattaches (same disk buffer preserved), and agents reconnect within seconds — no operator action.

Output:

Metric	Value
Total agent memory	~100 GB (500 × 200 MB)
Total aggregator memory	48 GB (12 × 4 GB)
Total disk buffer	720 GB (12 × 60 GB)
Datadog outage tolerance	30 min
Hot-reload downtime	0
Config rollout audit	git log
Agg replica MTTR	~2 min (K8s reschedule)

Why this works — concept by concept:

• Agent tier + aggregator tier — agents at the source keep it cheap and per-node-parallel; aggregators near the sinks amortise connections, secrets, and heavy transforms. The two-tier pattern scales linearly with nodes without the O(N) secret-distribution / connection-count tax.
• Disk buffer + when_full = block — 30 GB per Datadog sink per aggregator is sized to a 30-minute outage with 2× safety margin. Under sustained pressure, backpressure propagates upstream to agents (which have their own 5 GB buffer) rather than dropping.
• Config-reloader sidecar + /reload API — ConfigMap changes propagate to a filesystem swap on the mounted volume; the sidecar detects it and POSTs /reload; Vector executes the zero-drop reload lifecycle. GitOps gets to change configs without touching pods.
• StatefulSet + PVC — 60 GB per-replica persistent volume holds the disk buffer across pod restarts. When Kubernetes reschedules an aggregator replica, the PVC re-attaches and the buffer is preserved — the replica resumes exactly where it left off.
• Cost — 500 × 300 MB agent memory (~150 GB), 12 × 4 GB aggregator memory (48 GB), 720 GB persistent disk, one Helm chart per tier, ArgoCD reconciliation. The eliminated cost is the "our aggregator crashed and we lost 30 minutes of logs" incident, the "we can't roll config changes without downtime" ops toil, and the "500 pods each hold a Datadog API key" compliance headache. Net O(nodes) for agents + O(regions × aggregator_replicas) for aggregators — sublinear in total connections and secrets to distribute.

Design
Topic — design
Design problems on multi-tier observability topology

Practice →

Streaming Topic — streaming Streaming problems on backpressure and buffering

Practice →

---

5. Vector vs Fluent Bit vs Fluentd vs Logstash + interview signals

The head-to-head decision matrix and the senior-interview probes that decide which agent you defend

The mental model in one line: The four mainstream open-source observability agents — Vector, Fluent Bit, Fluentd, Logstash — occupy four different points in a four-dimensional trade-off space (throughput × memory × language safety × plugin ecosystem), and the senior interview probe is not "which is best" but "for this specific workload, which axis dominates, and can you defend the pick against the other three". Every senior platform engineer earns credibility by naming why the losing agents lose, not just why the winning one wins.

The four agents, four winning axes.

• Vector — throughput + Rust safety + VRL type checking. The 2026 default for greenfield high-throughput deployments. Rust means no GC pauses and no memory-safety bugs; VRL means compile-time-checked transforms; the sources → transforms → sinks DAG means unified logs + metrics + traces. Wins when throughput, safety, and unified pipeline are the dominant axes.
• Fluent Bit — tiny footprint + edge / embedded. CNCF-graduated; written in C; steady-state 10-50 MB memory. Wins when memory budget is the dominant axis — IoT gateways, edge nodes, or Kubernetes managed services where every MB of node memory is contested.
• Fluentd — plugin ecosystem + Ruby community. CNCF-graduated; written in Ruby; 1000+ community plugins covering every obscure sink and source. Wins when the ecosystem gravity is decisive — a legacy shop with 100+ custom Ruby plugins cannot justify porting them to VRL, so Fluentd stays.
• Logstash — Elastic Stack integration + grok legacy. JVM-based; ships with Elastic Stack; deepest integration with Elasticsearch's SIEM, machine-learning, and observability features. Wins when Elastic is the primary and only sink, or when a grok-heavy config would cost weeks to port.

The four axes each interviewer probes.

• Throughput. Events/sec per node before saturation. Vector wins (~10× Fluentd on the same box), Fluent Bit close second, Logstash third (JVM overhead), Fluentd fourth (Ruby GC + interpreted transforms).
• Memory footprint. Steady-state RSS per agent. Fluent Bit wins (10-50 MB), Vector second (100-200 MB), Fluentd third (300-500 MB Ruby heap), Logstash fourth (1-2 GB JVM).
• Language safety. Runtime safety of the transform language. Vector wins (VRL compile-time-typed + sandboxed), Fluent Bit second (Lua — dynamic but bounded), Fluentd third (Ruby — dynamic + unbounded), Logstash fourth (JRuby — dynamic + JVM overhead).
• Plugin ecosystem. Number and quality of community sources / sinks / filters. Fluentd wins (1000+ Ruby gems), Vector second (~50 sources + ~60 sinks native, growing), Logstash third (~200 plugins), Fluent Bit fourth (~100 plugins).

Where interviewers push back on each recommendation.

• "Why not Fluent Bit?" — because your memory budget is generous and your throughput ceiling matters more than saving 100 MB per node. Fluent Bit wins on edge; Vector wins on high-throughput Kubernetes.
• "Why not Fluentd?" — because you don't have 100+ custom Ruby plugins that would cost 6 months to port, and Fluentd's Ruby transforms hit a throughput wall at ~5k events/sec/node. Fluentd wins when the plugin ecosystem is decisive; Vector wins when raw performance is.
• "Why not Logstash?" — because Elastic is not your only sink, and the JVM's 1-2 GB memory tax + GC pauses are painful at scale. Logstash wins when Elastic Stack is the entire observability world; Vector wins when Datadog / Splunk / S3 / Prometheus are also in the mix.
• "Why open-source Vector and not commercial Datadog Agent?" — because you want a single agent that runs on and off Kubernetes with a config you can git diff, not a black-box binary configured via a web UI. Datadog Agent is fine when you're all-in on Datadog; Vector is the correct pick when portability matters.

The bootstrap-to-production runbook (once you've picked Vector).

• Week 1. Deploy Vector alongside your existing agent on 5% of nodes; run both DaemonSets in parallel; compare event rates in Datadog. Fix the first 20 event-shape mismatches in VRL.
• Week 2. Expand Vector to 25% of nodes; compare cardinality, tag propagation, .level distributions. Fix any drift.
• Week 3. Expand Vector to 100% of nodes; deprecate the old agent's DaemonSet (leave the manifest in place with 0 replicas for one week in case of rollback).
• Week 4. Delete the old agent. Add aggregator tier if node count > 50. Ship disk-buffer + backpressure monitoring dashboards.
• Week 6. Add VRL transforms for PII redaction, dedupe, sampling. Add a dead-letter Kafka topic for parse failures. Wire ArgoCD to the Vector Helm chart.

Common interview probes on comparison + rollout.

• "Vector vs Fluent Bit — when do you pick each?" — required answer: Vector for throughput; Fluent Bit for footprint.
• "How do you migrate off Fluentd?" — required answer: canary Vector alongside; compare event rates; port the top 5-10 filters to VRL; deprecate.
• "What's the safety tradeoff of Vector's Rust vs Logstash's JVM?" — required answer: Rust = no GC pauses + memory-safe; JVM = mature GC but stop-the-world risk under heap pressure.
• "Would you ever pick Logstash in 2026?" — required answer: yes, if Elastic Stack is the only sink and a grok-heavy Logstash config already exists.

Worked example — the four-axis matrix scored for a canonical workload

Detailed explanation. Every senior interview eventually asks you to score the four agents on the four axes for a hypothetical workload. Rehearsing the exact scoring makes the answer reproducible: an interviewer can hand you a scenario and you can walk the matrix out loud. Walk through the scoring for a 200-node Kubernetes cluster feeding Datadog + S3 + Prometheus.

• Workload. 200-node Kubernetes, 50k events/sec, Datadog + S3 + Prometheus sinks.
• Constraints. 300 MB memory budget per agent, sub-second Datadog latency, no per-node secrets.
• Requirements. Unified logs + metrics + traces, dead-letter Kafka for parse failures, GitOps rollout.

Question. Score the four agents on throughput / memory / safety / ecosystem (1-5) for this workload and pick the winner.

Input.

Axis	Weight for this workload	Why
Throughput	high	50k events/sec/cluster is the constraint
Memory	medium	300 MB is generous; not a Fluent Bit case
Language safety	high	dead-letter + PII redact means real transforms
Ecosystem	low	Datadog + S3 + Prom are all Vector-native
Unified L+M+T	high	one agent per node, not three

Code.

# Four-axis scoring for the 200-node Kubernetes workload
def score(agent: str) -> dict[str, int]:
    """Return 1-5 score per axis for the workload described."""
    scores = {
        "Vector":     {"throughput": 5, "memory": 4, "safety": 5, "ecosystem": 3, "unified": 5},
        "Fluent Bit": {"throughput": 4, "memory": 5, "safety": 3, "ecosystem": 2, "unified": 3},
        "Fluentd":    {"throughput": 2, "memory": 3, "safety": 2, "ecosystem": 5, "unified": 2},
        "Logstash":   {"throughput": 3, "memory": 1, "safety": 3, "ecosystem": 4, "unified": 3},
    }
    return scores[agent]


# Apply the workload weights and compute a weighted total
weights = {"throughput": 3, "memory": 2, "safety": 3, "ecosystem": 1, "unified": 3}

for agent in ["Vector", "Fluent Bit", "Fluentd", "Logstash"]:
    s = score(agent)
    total = sum(s[k] * weights[k] for k in weights)
    print(f"{agent:12s} — {s} → weighted total {total}")

# → Vector       — {throughput:5, memory:4, safety:5, ecosystem:3, unified:5} → weighted total 56
# → Fluent Bit   — {throughput:4, memory:5, safety:3, ecosystem:2, unified:3} → weighted total 42
# → Fluentd      — {throughput:2, memory:3, safety:2, ecosystem:5, unified:2} → weighted total 29
# → Logstash     — {throughput:3, memory:1, safety:3, ecosystem:4, unified:3} → weighted total 33

Step-by-step explanation.

1. Score each agent 1-5 on each axis using the well-known relative rankings. Vector = 5 on throughput; Fluent Bit = 5 on memory; Fluentd = 5 on ecosystem; nothing else scores 5 across the board.
2. Apply per-workload weights that reflect the constraints. Throughput is weighted 3× because 50k events/sec/cluster is a hard requirement. Memory is weighted 2× — 300 MB is generous, so Fluent Bit's memory win is only partly relevant. Ecosystem is weighted 1× — the required sinks (Datadog, S3, Prom) are all Vector-native so we don't need Fluentd's 1000-plugin ecosystem.
3. Compute weighted totals. Vector wins at 56 because it scores well on all high-weight axes; Fluent Bit at 42 because its memory win is diluted by lower ecosystem + safety scores; Logstash at 33 because its 1-GB JVM memory kills it; Fluentd at 29 because its Ruby throughput ceiling defeats it.
4. The scoring pattern is scenario-specific. For an edge IoT gateway with a 50 MB memory budget, Fluent Bit's memory weight would be 5, flipping the winner. For a legacy shop with 200 Ruby plugins, ecosystem weight would be 5, potentially flipping to Fluentd. The exercise is not memorising a single winner but memorising the axes and how weights change.
5. Present the answer in the interview as "for this workload, weighted by these axes, Vector at 56 vs Fluent Bit at 42 — but change memory weight to 5 and Fluent Bit wins." Showing the sensitivity to weights is what marks you as a senior architect rather than a Vector evangelist.

Output.

Agent	Throughput	Memory	Safety	Ecosystem	Unified	Weighted
Vector	5	4	5	3	5	56
Fluent Bit	4	5	3	2	3	42
Fluentd	2	3	2	5	2	29
Logstash	3	1	3	4	3	33

Rule of thumb. For any observability agent interview, memorise the 4×5 matrix (4 agents × 5 axes) and the per-workload weight multipliers. The winning agent falls out of the arithmetic; the senior signal is naming the axes and the weights, not just the winner.

Worked example — porting a Fluentd Ruby filter to a VRL transform

Detailed explanation. The migration cost from Fluentd to Vector is dominated by porting custom Ruby filters to VRL. Most Fluentd <filter> blocks turn into 5-10 line VRL programs; the few that don't (heavy imperative logic, external API calls, database lookups) require redesign. Walk through porting a common Fluentd filter that parses a JSON message, adds cluster metadata, and drops noise.

• Fluentd side. <filter **> block with record_transformer and parser plugins.
• Vector side. One remap transform with a 15-line VRL program.
• Migration cost. ~30 minutes per filter for a simple case; ~2 hours for a complex one.

Question. Port the Fluentd filter to VRL and verify equivalent output.

Input.

Fluentd concept	VRL equivalent
<filter> block	[transforms.NAME] with type = "remap"
record_transformer	direct VRL assignment (.field = value)
parser	parse_json / parse_grok / parse_syslog
<match> selector	inputs = [...] array + route transform
enable_ruby true	(rejected; use VRL functions)

Code.

<!-- Fluentd — before migration -->
<filter kubernetes.**>
  @type parser
  key_name log
  reserve_data true
  hash_value_field parsed
  <parse>
    @type json
  </parse>
</filter>

<filter kubernetes.**>
  @type record_transformer
  enable_ruby true
  <record>
    cluster ${ENV['CLUSTER_NAME']}
    env ${ENV['ENV']}
    level ${record['parsed']['level'] ? record['parsed']['level'].upcase : 'INFO'}
  </record>
</filter>

<filter kubernetes.**>
  @type grep
  <exclude>
    key level
    pattern /^DEBUG$/
  </exclude>
</filter>

# Vector — after migration (single remap replaces three Fluentd filters)
[transforms.k8s_enrich]
type = "remap"
inputs = ["k8s_logs"]
source = '''
    # Filter 1 equivalent — parse JSON from .log into .parsed
    if starts_with(string!(.log ?? ""), "{") {
        parsed, err = parse_json(.log)
        if err == null {
            .parsed = object!(parsed)
        }
    }

    # Filter 2 equivalent — add cluster metadata + normalise level
    .cluster = get_env_var!("CLUSTER_NAME")
    .env     = get_env_var!("ENV")

    if exists(.parsed.level) {
        .level = upcase!(string!(.parsed.level))
    } else {
        .level = "INFO"
    }

    # Filter 3 equivalent — drop DEBUG events
    if .level == "DEBUG" {
        abort
    }
'''

Step-by-step explanation.

1. Fluentd's first filter runs the JSON parser plugin on the log field, storing the result in .parsed. The VRL equivalent is parsed, err = parse_json(.log); .parsed = object!(parsed) — three lines instead of six, and the compiler enforces the error path.
2. Fluentd's second filter uses record_transformer with enable_ruby true — which is the exact pattern that makes Fluentd risky. Ruby can loop forever, allocate memory, throw exceptions. The VRL equivalent uses get_env_var! (bounded, type-checked) and upcase! (bounded, type-checked) — three lines, zero risk.
3. Fluentd's third filter uses grep with an exclude pattern to drop DEBUG events. The VRL equivalent is if .level == "DEBUG" { abort } — one line, no regex compilation cost.
4. The whole three-filter Fluentd chain collapses to one Vector remap transform. Fewer components, fewer misconfigurations, easier to review in a pull request. The VRL is git diff-friendly in a way that Fluentd's XML-in-YAML never was.
5. Verify equivalence by running both agents on the same input stream during the canary week; compare event rates and field distributions in Datadog. The most common gotcha is field name mismatches (log vs message) — resolve those before promoting Vector to 100%.

Output.

Metric	Fluentd	Vector
Filter blocks	3	1
Lines of config	~25 XML	~15 TOML+VRL
Ruby → sandboxed transforms	yes (enable_ruby true)	never
Boot-time type checking	no	yes
Per-event CPU	~10 μs (Ruby GC pressure)	~2 μs (Rust)

Rule of thumb. Port Fluentd filters to VRL one at a time; run the two agents in parallel on 5% of nodes; compare event rates before promoting. Never use enable_ruby true as a permanent solution — it's the exact anti-pattern VRL exists to eliminate. Budget ~30 minutes per simple filter and ~2 hours per complex one.

Worked example — the senior-interview probe list, ranked

Detailed explanation. Every Vector interview has a predictable set of probes that separate senior candidates from junior ones. Rehearse the answer to each; a fluent one-sentence response to every probe is what earns the offer. Walk through the top 8 probes.

• Ranked by frequency + weight. The probes below appear in ~80% of senior platform-engineering interviews that touch observability.
• Answer discipline. One sentence naming the concept, one sentence naming the trade-off, one sentence naming the alternative.

Question. Draft one-sentence senior answers to the top 8 Vector interview probes.

Input.

#	Probe	Weight
1	Vector vs Fluent Bit	high
2	VRL vs Lua safety	high
3	Disk buffer semantics	high
4	Aggregator tier when?	high
5	Hot-reload guarantees	medium
6	End-to-end acks	medium
7	Migrating from Fluentd	medium
8	Vector vs Datadog Agent	medium

Code.

Senior Vector interview cheat card
==================================

1. Vector vs Fluent Bit
   "Vector wins on throughput and unified logs+metrics+traces;
    Fluent Bit wins on memory footprint. I pick Vector for
    Kubernetes DaemonSets with 300+ MB memory budget and Fluent
    Bit for edge / IoT with 50 MB budgets."

2. VRL vs Lua safety
   "VRL is compile-time type-checked, guaranteed to terminate,
    and sandboxed against I/O. Lua in Fluent Bit is dynamic and
    can infinite-loop on the hot path. VRL catches transform bugs
    at boot; Lua catches them at 3am when a malformed log line
    hits the wrong filter."

3. Disk buffer semantics
   "Every Vector sink can opt into a disk buffer sized in bytes
    at `data_dir`; when full, `when_full = block` applies
    backpressure upstream all the way to the source. Size the
    buffer to (events/sec × outage_duration × event_size × 2)."

4. Aggregator tier when?
   "I add an aggregator tier past ~50 nodes to amortise sink
    connections and secrets, and to do cross-node metric
    aggregation. Below 50 nodes, agent-only is fine and the
    aggregator overhead outweighs the benefit."

5. Hot-reload guarantees
   "SIGHUP or POST /reload triggers a zero-drop reload: Vector
    parses and validates the new config, brings up new
    components, drains the old ones, then swaps. No in-flight
    events are dropped."

6. End-to-end acks
   "`acknowledgements.enabled = true` on both source and sink
    means the source only commits its offset/checkpoint after
    the sink has confirmed. Combined with disk buffer and
    when_full=block, this gives at-least-once delivery."

7. Migrating from Fluentd
   "Canary Vector alongside Fluentd on 5% of nodes; compare
    event rates in Datadog; port the top 5-10 filters to VRL;
    expand to 25%, then 100%; delete Fluentd. Budget 30 min per
    simple filter, 2 hours per complex one."

8. Vector vs Datadog Agent
   "Datadog Agent is closed-source, Datadog-specific, and
    UI-configured. Vector is open-source, portable across
    Datadog / Splunk / ES / S3 / Prometheus, and config-first
    (git-diffable). I pick Datadog Agent when we're all-in on
    Datadog forever; Vector when portability matters."

Step-by-step explanation.

1. The one-sentence discipline is essential. Interviewers time-box each probe implicitly; a rambling 3-minute answer to probe #1 costs you probes #4-#8. Rehearse until each answer is 20-40 seconds long.
2. Each answer names the concept explicitly ("VRL", "disk buffer", "aggregator tier") because the interviewer is often taking notes and scoring against a rubric that lists those exact keywords. Missing the keyword can cost the point even if your explanation is correct.
3. Each answer names the alternative because senior signal is "I chose X because Y beats Z," not "I chose X because X is good." Naming Y and Z demonstrates you evaluated the space.
4. Answers 3 (disk buffer), 5 (hot-reload), and 6 (end-to-end acks) are the operational reliability trio — they're what separates candidates who've deployed Vector from candidates who've read the docs. Miss any of these and you drop from "senior" to "intermediate" in the interviewer's mental model.
5. Answer 8 (Vector vs Datadog Agent) is the political probe — Datadog owns Vector and pushes both products. Naming the trade-off honestly ("Datadog Agent when we're all-in on Datadog forever") signals architectural maturity rather than tribal loyalty.

Output.

Probe	Weak answer	Senior answer (this cheat card)
Vector vs Fluent Bit	"Vector is newer"	throughput vs memory trade-off
VRL vs Lua	"VRL is safer"	compile-time + termination + sandbox
Disk buffer	"it buffers to disk"	when_full=block + sizing formula
Aggregator tier	"you might want one"	~50-node threshold + connection amortisation
Hot-reload	"SIGHUP works"	full lifecycle: parse → validate → swap
End-to-end acks	"yes it supports"	source + sink both opt in
Migrate Fluentd	"port the filters"	5% canary → compare → expand → delete
Vector vs DD Agent	"Vector is open"	portability + config-first vs UI-first

Rule of thumb. Print the 8-probe cheat card before every observability-flavoured interview. Rehearse each answer to 20-40 seconds. Name the concept, name the trade-off, name the alternative. Skip the ramble; land the keyword; move on to the next probe.

Senior interview question on Vector vs the alternatives

A senior interviewer might ask: "You're inheriting a 300-node Kubernetes platform that currently runs Fluentd DaemonSets feeding Elasticsearch. The org has decided to move primary observability to Datadog with S3 as the compliance backup, and wants a single agent per node that can also scrape Prometheus. Walk me through the decision matrix comparing Vector, Fluent Bit, Fluentd, and Logstash for this workload, name the winner, and describe the 6-week migration plan."

Solution Using Vector with a weighted decision matrix and phased Fluentd deprecation

# Decision matrix — score, weight, pick
axes = ["throughput", "memory", "safety", "ecosystem", "unified", "sink_native"]

agents = {
    "Vector":     [5, 4, 5, 3, 5, 5],   # Datadog + S3 + Prom all native
    "Fluent Bit": [4, 5, 3, 2, 3, 4],   # Datadog OK, S3 via HTTP, Prom yes
    "Fluentd":    [2, 3, 2, 5, 2, 4],   # incumbent; slow; ecosystem win
    "Logstash":   [3, 1, 3, 4, 3, 3],   # Elastic-first; not our sink profile
}

# Weights reflect the workload constraints
weights = [3, 2, 3, 1, 3, 3]

for name, scores in agents.items():
    total = sum(s * w for s, w in zip(scores, weights))
    print(f"{name:12s} -> {total}")

# → Vector       -> 71
# → Fluent Bit   -> 55
# → Fluentd      -> 44
# → Logstash     -> 43

# Winner: Vector.

# 6-week migration plan (GitOps rollout via ArgoCD)

# Week 1: Deploy Vector at 5% canary alongside Fluentd
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: vector-agent-canary
spec:
  template:
    spec:
      nodeSelector: { observability-canary: "true" }   # ~15 of 300 nodes
      containers: [{ name: vector, image: timberio/vector:0.40.0 }]

# Week 2: Compare Vector vs Fluentd event rates in Datadog
# Metric: sum(datadog.logs.forwarded){agent} by {agent}
# Expected: Vector rate ~= Fluentd rate (parity ±2%)

# Week 3: Port top 10 Fluentd filters to VRL; expand to 25%
kubectl label node <75-nodes> observability-canary=true
# Watch dead-letter Kafka topic count for parse failures

# Week 4: Expand to 100%; scale Fluentd DaemonSet to 0 replicas
kubectl scale daemonset/fluentd --replicas=0

# Week 5: Add aggregator tier (12 replicas for 300 nodes)
helm install vector-aggregator ./charts/vector-aggregator \
    --set replicas=12 \
    --set persistence.size=60Gi

# Week 6: Delete Fluentd; wire ArgoCD; ship monitoring dashboards
kubectl delete daemonset fluentd

Step-by-step trace.

Week	Action	Blast radius	Rollback
1	Canary Vector on 5%	15 nodes	scale canary DaemonSet to 0
2	Compare event rates	measurement only	none needed
3	Expand to 25%; port filters	75 nodes	remove observability-canary label
4	Expand to 100%; scale Fluentd to 0	300 nodes	scale Fluentd back to 300
5	Add aggregator tier	new infra	delete StatefulSet
6	Delete Fluentd; wire ArgoCD	permanent	git revert

After the 6-week plan, the cluster runs one Vector DaemonSet + one 12-replica aggregator StatefulSet feeding Datadog + S3 + Prometheus. Fluentd is gone; the two-tier Vector footprint uses less memory in aggregate than the old Fluentd DaemonSet (150 MB × 300 + 4 GB × 12 = ~93 GB vs Fluentd's 400 MB × 300 = 120 GB); event throughput headroom rises from 5k/s/node ceiling to 50k/s/node.

Output:

Metric	Before (Fluentd)	After (Vector two-tier)
Agents per node	1 (Fluentd)	1 (Vector)
Total memory	~120 GB (300 × 400 MB)	~93 GB (300 × 150 MB + 12 × 4 GB)
Throughput ceiling	~5k events/s/node	~50k events/s/node
Transform safety	Ruby (unsandboxed)	VRL (compile-time-typed)
Datadog outage tolerance	~0 (best-effort buffer)	30 min (30 GB disk buffer)
Config rollout	XML edit + restart	git commit + hot-reload
Migration duration	—	6 weeks with rollback per phase

Why this works — concept by concept:

• Weighted decision matrix — the four-axis scoring with per-workload weights turns "which agent" from opinion into arithmetic. Vector wins for this workload because throughput, safety, unified L+M+T, and sink-native support all weight 3× and Vector dominates each; ecosystem weights 1× because Datadog + S3 + Prom are all Vector-native.
• Phased canary rollout — week-by-week expansion (5% → 25% → 100%) with parity comparison in Datadog at each step. Rollback is a config revert per phase — no data-migration lock-in.
• Filter porting alongside canary — the highest-risk migration step (porting Ruby filters to VRL) happens in week 3 with only 25% of production traffic, so bugs are caught with limited blast radius. Dead-letter Kafka topic makes every parse failure investigable.
• Aggregator tier at 300 nodes — 300 nodes sits well above the ~50-node threshold; adding the aggregator tier in week 5 amortises Datadog / S3 / ES connections down from 300 to 12 and centralises secret distribution.
• Cost — 6 engineer-weeks (mostly filter porting + canary observation), one Helm chart per tier, one ArgoCD Application, roughly 30% memory reduction across the fleet vs the incumbent Fluentd. The eliminated cost is the "Fluentd Ruby GC pause paged us at 3am" incident class, the "we hit our Datadog log ingest quota because a filter didn't dedupe" incident class, and the "we can't roll a config change without restarting 300 pods" ops toil. Net O(nodes) for agents + O(1) for the aggregator tier with GitOps-driven changes.

Design
Topic — design
Design problems on observability-stack migration

Practice →

Streaming
Topic — streaming
Streaming problems on Vector vs Fluent Bit trade-offs

Practice →

---

Cheat sheet — Vector recipes

• Which agent when. Vector is the 2026 default for any greenfield high-throughput Kubernetes observability deployment where memory budget is >200 MB per agent, transform safety matters, and unified logs + metrics + traces is a requirement — the sources → transforms → sinks DAG plus VRL's compile-time type checking are the reason. Fluent Bit is the answer when memory budget is <50 MB (edge, IoT, embedded); Fluentd is the answer when 100+ custom Ruby plugins would cost 6 months to port; Logstash is the answer when Elastic Stack is the only sink and a grok-heavy config already exists. Never pick an agent because a vendor rep suggested it; walk the four-axis matrix (throughput × memory × safety × ecosystem) with the actual workload constraints on the table.
• Kubernetes agent DaemonSet template. One-pod-per-node DaemonSet with kubernetes_logs + host_metrics sources, a single remap transform for JSON parse + PII redact + cluster tag enrichment, and a vector sink pointing at the regional aggregator. HostPath-mount /var/log, /var/lib/docker/containers, and /var/lib/vector (for the disk buffer). Resource limits: 300 MB memory + 500m CPU. Include a config-reloader sidecar that watches /etc/vector and POSTs to http://localhost:8686/reload on ConfigMap change so config rollouts are zero-drop.
• Aggregator StatefulSet template. Regional StatefulSet with 6-12 replicas (one per ~40 nodes) across AZs, vector source on port 6000, route transform splitting logs from metrics, and Datadog + S3 + Elasticsearch + Prometheus sinks each with disk buffer sized to worst-case outage. Resource limits: 4 GB memory + 2 CPU + 60 GB PVC per replica. Headless Service (clusterIP: None) so agent connections DNS-round-robin across replicas. Every sink has acknowledgements.enabled = true for end-to-end at-least-once delivery.
• VRL enrichment snippet. Multi-parser cascade: try parse_json first (tuple form: (parsed, err) = parse_json(.message)), fall back to parse_syslog, fall back to plaintext with .message = original. Enforce required fields (.level, .message, .timestamp, .service) with if !exists(.X) { .X = default }. Canonicalise .level via upcase!(string!(.level)) and derive a numeric .severity (0-7) from .level for Datadog. Enable reroute_dropped = true on the transform and wire .dropped output to a Kafka dead-letter topic — every parse failure investigable, never silently dropped.
• Disk buffer sizing formula. buffer_bytes = events_per_sec_per_replica × outage_duration_seconds × avg_event_size_bytes × safety_margin. Typical values: 8,000 events/sec/replica × 1,800 seconds (30-min outage) × 1,024 bytes × 2 = ~30 GB per Datadog sink per aggregator. Set when_full = block so overflow becomes backpressure rather than drops; pair with acknowledgements.enabled = true on both source and sink for end-to-end at-least-once. PVC size ≥ sum of all sink buffers on that replica + 10% overhead.
• Comparison matrix — Vector / Fluent Bit / Fluentd / Logstash. Throughput: Vector wins (~10× Fluentd, ~2× Logstash on the same box). Memory footprint: Fluent Bit wins (10-50 MB); Vector second (100-200 MB); Fluentd third (300-500 MB); Logstash fourth (1-2 GB JVM). Language safety: Vector wins (VRL compile-time-typed + sandboxed); Fluent Bit second (Lua bounded but dynamic); Fluentd third (Ruby unbounded); Logstash fourth (JRuby + JVM overhead). Plugin ecosystem: Fluentd wins (1000+ Ruby gems); Vector second (~50 sources + ~60 sinks native and growing); Logstash third (~200); Fluent Bit fourth (~100). Unified L+M+T: Vector wins (first-class from day one); everyone else is log-first with metrics bolted on.
• Hot-reload runbook. Send SIGHUP to the Vector process (kill -HUP $(pgrep vector)) or POST to http://localhost:8686/reload. Vector parses and validates the new config, brings up new components alongside old ones, drains the old ones, then swaps — zero dropped in-flight events. If validation fails, Vector logs Reload failed: <error> and keeps the old config running (no downtime). In Kubernetes, install a configmap-reload sidecar that watches /etc/vector and POSTs /reload on ConfigMap change; config rollouts become git commit → ArgoCD reconciles → sidecar reloads Vector, all zero-drop.
• End-to-end ack contract. Set acknowledgements.enabled = true on every source that supports it (kafka, aws_kinesis_streams, file) and on every sink downstream. The source only commits its offset / checkpoint after the sink has confirmed the batch. Combined with when_full = block on the disk buffer, this gives at-least-once delivery through the entire pipeline: a Vector crash mid-batch replays from the last committed source position. Never enable acks on only one side of the pipeline — the guarantee breaks silently.
• PII redaction VRL snippet. .message = replace(string!(.message ?? ""), r'[\w\.-]+@[\w\.-]+', "<redacted-email>") for emails; .message = replace(string!(.message), r'\b\d{3}-\d{2}-\d{4}\b', "<redacted-ssn>") for US SSNs; .message = replace(string!(.message), r'\b(?:\d[ -]*?){13,16}\b', "<redacted-cc>") for credit-card-shaped digit runs. Apply redaction in the agent tier (before any egress) so redacted forms are what land on disk buffers and downstream sinks. Never rely on downstream (Datadog, S3) redaction; the log has already left your control by then.
• Dedupe transform template. Pre-compute the dedupe key in a remap transform (.dedupe_key = string!(.host ?? "-") + "|" + string!(.service ?? "-") + "|" + slice!(string!(.message ?? ""), 0, 100)); pass to a dedupe transform (cache.num_events = 10000, fields.match = ["dedupe_key"]). This drops repeated log spam within a bounded window (~1 second at 10k events/sec) with ~2 MB memory. Alert on Vector's dedupe_events_dropped_total counter — a sudden spike usually means an app started spamming, which is the actual bug worth knowing about.
• Kafka source config template. type = "kafka", bootstrap_servers = "kafka-1:9092,kafka-2:9092,kafka-3:9092", group_id = "vector-<pipeline-name>", topics = ["logs.raw"], auto_offset_reset = "earliest" (bootstrap from oldest offset on new consumer group), acknowledgements.enabled = true (offsets only advance after downstream sinks confirm). For high-throughput topics, also set librdkafka_options.session.timeout.ms = 30000 and librdkafka_options.max.poll.interval.ms = 300000 to survive brief consumer stalls without triggering rebalances.
• Prometheus scrape source config. type = "prometheus_scrape", endpoints = ["http://localhost:9100/metrics", "http://localhost:8080/metrics"], scrape_interval_secs = 15. Vector emits each scraped metric as an event of type Metric with the correct kind (counter, gauge, histogram) and all labels. Route metric events to prometheus_remote_write sink for long-term Prom / VictoriaMetrics / Cortex storage, or to datadog_metrics for Datadog metric ingest. Never mix log and metric events into the same sink — the sink type expects one or the other.
• Migration cost between agents. Fluentd → Vector: ~30 min per simple filter, ~2 hours per complex filter, ~6 weeks end-to-end for a 200-300 node cluster with canary → parity → expand → deprecate phases. Fluent Bit → Vector: ~1 week (Lua filters port to VRL almost 1-to-1 and Fluent Bit configs are shorter). Logstash → Vector: highly variable — grok patterns lift directly into parse_grok, but Ruby scripts require redesign. Choose the primary agent once; the migration cost is real.
• The senior interview cheat card. Name Rust and Datadog in sentence one; name sources → transforms → sinks as the DAG model; name VRL and compile-time type checking as the safety win over Lua/Ruby; name the agent / aggregator / gateway three-tier topology; name when_full = block as the backpressure config; name acknowledgements.enabled = true as the end-to-end ack switch; name SIGHUP or /reload API as zero-drop hot-reload; name the 10× throughput vs Fluentd number; name the 4-axis decision matrix (throughput × memory × safety × ecosystem) with per-workload weights. These are the keywords senior interviewers score against.
• What breaks in production, ranked. (1) Disk buffer fills because outage exceeded sizing → resize buffer, add monitoring alert. (2) VRL transform throws unhandled error → reroute_dropped = true + dead-letter sink. (3) Agent OOMs during log-storm → cap memory buffer, size disk buffer larger, add throttle transform upstream. (4) Aggregator replica crashes and buffer is lost → use StatefulSet + PVC (never Deployment + emptyDir). (5) Downstream Kafka rebalances during load → tune session.timeout.ms + max.poll.interval.ms. (6) Config rollout drops events → use sidecar-driven SIGHUP, never kubectl rollout restart.

Frequently asked questions

What is Vector by Datadog in one sentence?

Vector is an open-source, Rust-native observability agent and pipeline built by Datadog (via the 2021 acquisition of Timber.io) that unifies logs, metrics, and traces under a single declarative sources → transforms → sinks DAG model, ships with 100+ integrations covering every mainstream observability sink (Datadog, Splunk, Elasticsearch, S3, Kafka, Prometheus, Loki, and many more), offers roughly 10× the sustained throughput and half the memory footprint of Fluentd on the same hardware, and is licensed under a permissive MIT/Apache-2 dual license that guarantees the community fork remains viable even if Datadog changes direction. Every senior platform-engineering interview in 2026 probes Vector because the choice of observability agent is one of the highest-leverage platform decisions after Kubernetes itself — the pattern you pick in year one becomes the observability contract every downstream dashboard, alert, and SIEM query hard-codes assumptions against for years.

Vector vs Fluent Bit — when do I pick each?

Default to Vector when your Kubernetes DaemonSets have a generous memory budget (200-500 MB per agent), throughput matters (10k+ events per second per node), and you want unified logs + metrics + traces under one config and one VRL transform language with compile-time type checking. Default to Fluent Bit when memory budget is the dominating constraint — IoT gateways with 128 MB total RAM, edge nodes, embedded systems, or Kubernetes managed services where every MB of node memory is contested against workloads. Fluent Bit's design targets a 10-50 MB steady-state footprint by aggressively bounding buffer sizes and using C rather than Rust; that footprint win comes at the cost of a smaller sustained throughput ceiling and a less safe transform language (Lua is bounded but still dynamically typed). The interviewer's real question here is whether you understand that both are correct answers for different points in the memory-vs-throughput trade-off space; the wrong answer is "Vector is newer so pick Vector always" or "Fluent Bit is CNCF-graduated so pick Fluent Bit always." Pick per workload; defend the axes.

What is VRL and why is it safer than Lua?

VRL — the Vector Remap Language — is Vector's purpose-built scripting language for event transformation inside a remap transform, distinguished from Lua / Ruby / JavaScript by three properties: compile-time type checking (a program that calls a fallible function without either the ! abort variant or the (result, err) tuple form is rejected at Vector boot, not at 3am when a malformed event hits the hot path), guaranteed termination (no while, no for, no recursion — every VRL program's runtime is bounded by input size × library function cost), and a sandboxed standard library (no I/O, no filesystem access, no network calls, no shell execution — every function is a pure transformation on the current event). Lua in Fluent Bit is safer than raw Ruby but still dynamically typed and can allocate unbounded memory; Ruby in Fluentd's enable_ruby true filters is the classic "at 3am we got paged because a filter went infinite" anti-pattern; JRuby in Logstash inherits the JVM's memory safety but shares Ruby's dynamic-type surprise class. VRL exists precisely so those production incidents never happen, and the compile-time guarantees make the whole hot path auditable in a code review before it ever ships.

Can Vector really replace Fluentd?

Yes for essentially every modern workload, with three caveats worth naming honestly. First, Vector replaces the pipeline — sources, transforms, sinks — but not the ecosystem. Fluentd has 1000+ community Ruby-gem plugins accumulated over a decade; Vector has ~50 sources and ~60 sinks natively plus a growing set of contributions. If your Fluentd config depends on a niche plugin (say, fluent-plugin-mixpanel or a custom internal gem), you must either port it to VRL (usually possible in 30-120 minutes per filter for the common cases), fall back to Vector's generic http sink, or keep a small Fluentd deployment for that specific pipeline. Second, the migration cost is real: canary → compare event rates → port top 5-10 filters → expand → deprecate takes 4-6 weeks for a 200-node cluster. Third, some organisations run both — Vector as the primary agent + aggregator, Fluentd as a specialised sidecar for the one legacy pipeline that would cost 6 months to port. The senior answer is "yes, Vector replaces Fluentd for 90%+ of workloads; the remaining 10% either port or keep a small Fluentd for it — never a big-bang cutover."

What is the aggregator tier and when do I need it?

The aggregator tier is a regional pool of Vector processes (typically a Kubernetes StatefulSet of 3-12 replicas across AZs) that sits between the per-node agent tier and the durable observability sinks (Datadog, S3, Elasticsearch, Prometheus). Its jobs are (a) amortising sink connection count — 6 aggregators × 4 sinks = 24 connections beats 500 agents × 4 sinks = 2,000 connections against Datadog's ingest endpoint, which matters because downstream services have per-source connection limits; (b) amortising secret distribution — 6 aggregators need the Datadog API key rather than 500 agents, which is a compliance win; (c) running cross-node transforms — aggregate (regional metric summation), tag_cardinality_limit (cardinality enforcement across the region), dedupe (cross-node dedupe of noisy alerts) all require the aggregator tier to see events from multiple agents at once; (d) providing a sink-fanout centralisation point — you configure Datadog + S3 + Elasticsearch + Prometheus once at the aggregator rather than repeating the config across every agent. The rule of thumb is: add the aggregator tier past ~50 nodes; below 50 nodes, agent-only is fine and the aggregator overhead outweighs the benefit. The aggregator replica count is roughly ceil(node_count / 40) with a floor of 3 for AZ-spread resilience, and each replica needs a persistent volume for its disk buffer so a pod restart doesn't lose the buffered events.

How do I deploy Vector in Kubernetes?

The canonical Vector Kubernetes deployment is a two-tier pattern: an agent DaemonSet with one Vector pod per node, and (past ~50 nodes) an aggregator StatefulSet with 3-12 replicas per region. The agent DaemonSet mounts /var/log and /var/lib/docker/containers read-only for kubernetes_logs, mounts a hostPath at /var/lib/vector for the local disk buffer, uses resource limits ~150-300 MB memory + 500m CPU, and configures a single sink of type vector pointing at the aggregator Service. The aggregator StatefulSet uses volumeClaimTemplates for a persistent volume per replica (~60 GB for combined sink buffers), a headless Service (clusterIP: None) so agent connections DNS-round-robin across replicas, resource limits ~4 GB memory + 2 CPU, and configures the actual downstream sinks (Datadog, S3, Elasticsearch, Prometheus) each with their own disk buffer sized to worst-case outage duration. Deploy both via one Helm chart per tier, driven by ArgoCD (or Flux) reconciling from a git repo — this gives you audit trail + rollback via git revert + automated hot-reload via a configmap-reload sidecar that POSTs to Vector's /reload API on ConfigMap change. In this shape, config rollouts are zero-drop, aggregator replica loss is auto-healed by Kubernetes with the PVC preserved, and disk buffers absorb 20-30 minute downstream outages without event loss. This is the pattern every senior Kubernetes-observability interviewer expects you to describe.

Practice on PipeCode

• Drill the streaming practice library → for the Kafka fan-out, backpressure, and end-to-end ack scenarios that senior interviewers use to probe Vector's disk buffer + when_full = block semantics against real graded inputs.
• Rehearse system design against the design practice library → for the observability agent + aggregator + gateway topology decisions that show up in every senior platform-engineering interview when Kubernetes and Datadog are on the table.
• Sharpen the batch axis with the ETL practice library → for the batched sink partitioning, S3 backup lake, and long-term retention patterns that a Vector aggregator + aws_s3 sink implements in production.
• Practice window-based aggregation on the aggregation practice library → to build the same intuition Vector's aggregate transform ships for regional metric summation across a fleet of agents.
• Warm up on the real-time analytics practice library → for the sub-second-latency, event-stream, and dashboard-fed scenarios that Vector's Rust-native hot path exists to serve.
• Stack the prerequisites against PipeCode's broader 450+ data-engineering catalogue to anchor the four-axis decision matrix, the VRL type-safety story, and the three deployment tiers against real graded inputs.

Lock in vector datadog muscle memory

Docs explain what Vector is. PipeCode drills explain when it wins — when Fluent Bit's memory footprint is decisive, when VRL's compile-time type checking saves your 3am incident review, when the aggregator tier earns its cost past ~50 nodes, when disk buffers + `when_full = block` are the difference between "we lost 30 minutes of production logs" and "the outage was invisible to downstream." Pipecode.ai is Leetcode for Data Engineering — pattern-first practice tuned for the production trade-offs senior platform and data engineers actually face.

Practice streaming problems →
Practice design problems →