Checkout says payments are timing out. You open the log search, filter on ERROR, type payment provider timeout, and get one hit. Is that one timeout, or the one timeout that survived the trip?
A log search shows what reached the index, not what the application wrote. Between the two sit an agent, a parser, a filter, a buffer and a search cluster, and every one of them can delay a line, reject it or drop it without the app knowing. That is the whole point of having a log aggregation pipeline explained stage by stage: when a search comes back empty during an incident, you need to know whether the system is fine or the evidence never arrived.
The Log Aggregation Pipeline Simulator lets you watch that trip one stage at a time and then break it in three different ways. It moves logs in batches and counts them at every stage, rather than tracing one line. Disclosure: I help build it; it is free, runs in the browser, no signup.
Six stages, one line
The simulator draws the pipeline as six cards. Each Step advances one stage; Run flow steps automatically about once a second. Six steps make one cycle.
- Sources (Apps & hosts). Raw events, and how many are waiting.
- Collect (Fluent Bit). An agent tails files and forwards events. If it falls behind, the source queue grows.
- Process (Parse & filter). Extracts fields and removes noise. Its two controls sit in the stage inspector: a Parser picker (JSON parser or Grok pattern) and a "Drop health-check noise" switch. Changing either starts a fresh run.
- Buffer (Durable buffer). A bounded queue between parsing and indexing, shown as used over capacity.
- Index (Search cluster). Accepted events land on three primary shards, and a "Shard health" tab shows the balance. The simulator deals events to the shards in turn, so the bars stay level in every scenario.
- Search (Explore logs). A "Search logs" tab with a text box and level filters. It keeps up to four sample events from each indexing step, and only the newest 24, so judge completeness by the counters, not by the list.
Under the cards, five counters keep the books: Generated, Buffered, Indexed, Rejected and Dropped. The engine behind them conserves every event: anything generated is waiting in a queue, filtered, rejected, indexed or dropped. Filtered noise has no counter of its own; the Process card's "removed" figure adds filtered and rejected together. Carry that habit into real systems: if you cannot account for the difference between "sent" and "searchable", you cannot trust an empty search.
Click any card and the stage inspector shows its "Operator signal", the one thing to watch there. For the Search stage it is the line that matters most in an incident: "Only indexed logs are searchable; filtered, rejected, and dropped logs never arrive here."
Healthy first
Start on Healthy flow and step through a few cycles. Each cycle, the sources emit 24 logs, the noise filter removes 4, and 20 are indexed. Queues drain, the badge says "Pipeline is healthy", and the shard bars grow evenly.
Break it three ways
Traffic spike. Sources emit 84 logs per cycle, but the collector forwards 38. The badge changes to "Backpressure is building" on the very first step. In the third cycle: "Sources emitted 84 logs; 26 were dropped before collection." Look at the buffer while that happens: it is empty. The durable buffer cannot save logs that are lost before they reach it. By the end of cycle five, 420 logs were generated, 140 indexed and 118 dropped. The Collect card's "for parsing" count also climbs by 4 per cycle: the collector sends 38, the processor takes 34.
Parser mismatch. A deployment changes the log format. Of 30 logs per cycle, 4 are filtered as noise, 11 are rejected by the parser and only 15 are indexed. The badge says "Parser rejects detected" and nothing is dropped, so a dashboard that only watches for drops stays green. Try the Grok parser. It does not help: it handles only 27 of the 30 logs per cycle, so 3 more wait in the processor queue every cycle, and of those 27 it filters 4, rejects 10 and indexes 13. A slower parser that still does not match the format makes things worse. The simulator has no parser fix; for real, you update the parser to the new format or roll the change back.
Slow indexing. Switch the parser back to JSON first: the simulator keeps your parser choice when you change scenario. The search cluster indexes 9 logs per cycle while 32 reach the buffer (36 arrive, and the noise filter removes 4). The buffer climbs by 23 each cycle, the warning appears in the third cycle, and in the fifth: "The buffer reached capacity and dropped 19 oldest logs." Now turn off "Drop health-check noise" and run it again. The first drop comes a cycle earlier. Filtering noise before the buffer did not fix the slow indexer, but it bought time.
Now run the incident
Take each scenario as an on-call drill. Pretend you only have the search tab, then use the rest of the screen to answer four questions in order:
- Is input normal? Compare Generated per cycle against the healthy 24. A spike is often the first sign of the real incident, and it also strains the pipeline.
- Where is a queue growing? A growing source queue points at the collector. A growing buffer points at the indexer. Name the stage before you touch anything.
- Is anything rejected? Rejects mean the logs exist but no longer match the parser. If the deploy that changed the format also broke checkout, the error lines you need are among the rejects.
- How long until loss? A filling buffer is a countdown. Space left divided by net growth per cycle is a rough, optimistic estimate: here the buffer overflows when a batch lands, before indexing drains it. With the JSON parser and the noise filter on, after cycle four it holds 92 with 13 free, and the next batch of 32 drops 19.
Match the mitigation to the signal: collector capacity for the spike, the parser for the mismatch, index capacity or noise shedding for the slow index.
Try the parser mismatch for real
You can reproduce the parser mismatch with jq and three lines of log:
cat > app.log <<'EOF'
{"level":"INFO","service":"api","msg":"request completed status=200"}
{"level":"ERROR","service":"checkout","msg":"payment provider timeout after=3s"}
ERROR checkout payment provider timeout after=3s
EOF
# what a JSON-only pipeline makes searchable
jq -R -c 'fromjson? | select(.level == "ERROR")' app.log
# what it rejected on the way
jq -R -r '(try fromjson catch null) as $j | select($j == null)' app.log
With jq 1.6, the first command prints only the JSON error line. The second prints the plain-text ERROR checkout payment provider timeout after=3s. Two timeouts were logged, and search finds one. Count the second output and alert on it, and you have a parser-reject metric. Real tools do not always throw a reject away. When parsing fails in Vector's remap transform, "the original, unmodified event is sent downstream" by default, so the line reaches the index as raw text without your fields.
Real agents have their own version of the other two scenarios. The Fluent Bit backpressure docs explain that when an input reaches its mem_buf_limit, Fluent Bit (with memory-only buffering) "will stop buffering data from that source plugin until enough buffered chunks are flushed", and most plugins log paused (mem buf overlimit) when that happens. The same page warns that some inputs can lose data under memory-only buffering and suggests filesystem buffering if you need to avoid that. Search your agent's own logs for that warning before you assume the app went quiet. With filesystem buffering, Fluent Bit drops the oldest data, as the Slow indexing buffer does: at an output's storage.total_limit_size, "the oldest chunk from its queue will be discarded". Vector's buffers block by default.
What to alert on
The simulator's page lists the signals to watch together: input rate, queue age and depth, parser rejection rate, buffer utilization, dropped events, indexing latency and shard balance. In real tools, the loss signals are counters you can scrape: Fluent Bit's fluentbit_output_retries_failed_total and fluentbit_output_dropped_records_total, Vector's component_discarded_events_total, and Loki's loki_discarded_samples_total, labelled by reason.
The simulator is one of 60+ free DevOps games and simulators. Run the three scenarios until you can predict the next step's numbers, then again as drills with only the search tab open.
Where has a log pipeline dropped lines on you without anyone noticing? Tell us in the comments.
Top comments (0)