The daemon fired its generation job at 14:02. Two seconds later, a hand landed on the manual trigger for the same job. Eight topics on that cycle.
Both runs carried exactly one question down to the state store: does this topic already have a package? Both got an empty shelf back. Neither answer was wrong. At the instant each one was given, there genuinely was no package. So both wrote one.
By the time it surfaced, all eight topics had two or three packages sitting beside them, and nobody could say which was canonical. Half a workday to stop it. An afternoon to clean up the duplicates.
The postmortem hands you the outline. The traces hand you the proof: two root spans with overlapping children, same topic ID, both carrying the same line at the same decision point — no existing package. Read, decide, write. Three events with a gap in the middle, and the second writer standing inside that gap. Nothing was corrupted. No stale cache. No torn index. A window existed, and two processes walked through it together.
I keep coming back to that gap, because the same skeleton sits under agent observability. Something reads. Something decides. Something acts. Watching all three arrives itemized, and the line items do not grow at the same rate as your traffic. Here is what's on the bill.
Start with the context that has to ride along with every step. Each span either creates trace context or hands it down the line, which means serialization, clock reads, allocations, and async boundaries where the context quietly slides off the back of the truck. Manual propagation adds code paths that break, and the way they break is ugly: lose one parent and a causal chain turns into scattered fragments. We lived that. Half our tool spans surfaced under brand-new roots because a thread-pool handoff did not carry the context object across. You cannot reason about concurrency through fragments, and you cannot tell two writers apart when the thing that proved they were the same request has evaporated.
Payloads come next. Prompts, tool arguments, tool results, state diffs. Large by nature, high-cardinality by nature, and hashing or redacting them costs CPU on the way out. Capture raw and you inherit two more bills: leak risk and storage. We logged raw tool arguments for a while, right up until one of them turned out to hold an API key inside a URL query string. That span had been sitting in the backend for days by the time we caught it. Readable. Useful to anyone who went looking. I would rather not do the arithmetic on how long it sat there before somebody did.
Then the export path: batch processors, buffers, network egress, collector pipelines. Memory and bandwidth, paid at every hop. When the buffer saturates you either drop spans or push back, and backpressure can stall the agent itself. Drops leave holes in the record. Both outcomes happen. Neither one is free.
Storage and query close the bill. Indexing traces, full-text logs, metric series. Query latency and retention scale with cardinality and payload size, so the expensive thing you captured in the name of debuggability keeps charging rent long after the incident it was meant to explain. I once watched a retention policy argue with a billing dashboard for a month. The dashboard won.
Head sampling is cheap and blind to the rare stuff. Tail sampling keeps the errors, the timeouts, the tool call that only fires twice a week, and buffers spans in the collector to do it, which you pay for in memory. Metrics are cheap and carry almost no context. Traces cost more and preserve causality. Logs cost the most and preserve detail. Token-level tracing buys depth and turns the volume up alongside it. Boundary-level tracing costs less and may miss internal reasoning drifting off course.
Observation changes the observed, too. Added latency shifts timeouts, rate limits, concurrency. A trace that slows a tool call by a few milliseconds can push it past a deadline it would have met without you watching. You have run a second experiment whether you meant to or not.
So we A/B the instrumentation itself. Same frozen workload in three modes — off, minimal, verbose — with deterministic tool stubs or replay cassettes standing in for the model calls. Compare p50, p95, p99, CPU, RSS, network bytes, dropped-span counters. First attempt: we ran the comparison against live traffic and got nonsense, because no two runs had the same prompts or the same tools. Freeze the workload, or do not bother running the comparison.
Every span gets a budget. Max bytes, max spans per trace, max attributes. Payloads get truncated, and the truncation gets written down as an event so the shape of the data stays honest. First attempt: silent truncation, followed by several days of staring at a tool result that looked wrong when it had merely been cut short. If you cut, say so in the span. The half-second you spend writing that event saves somebody an afternoon.
We tail-sample at the collector. Policies keep errors, timeouts, high-latency traces, rare tools; ordinary successes get sampled down. The tail_sampling processor with a bounded decision_wait and num_traces does that without unbounded memory. First attempt: we tail-sampled everything with a long decision window and the collector started shedding once traffic picked up. Shedding is worse than sampling, because it is quiet. Set a real window, set a real buffer, then watch otelcol_exporter_queue_size and otelcol_receiver_refused_spans like they are alarms. They are.
We redact at the source. Hash or drop identifiers in the SDK before the span leaves the process. First attempt: we leaned on a collector processor and it missed a header. The collector is the last place you want standing between you and a leaked credential. By the time data reaches it, it has already crossed a network boundary and probably a trust boundary along with it.
And we monitor the exporter. A dropped span is an observability outage wearing the costume of calm. Alert on it. If your spans are going missing, your incident timeline has holes in it, and you will trust that timeline anyway.
Speculation, flagged as such: token-level tracing is probably the biggest line item on long reasoning steps, but measure before you assume. For most of our production agents, boundary traces plus sampled payloads got us further for less.
Back to the lock.
The fix for the original incident was a file lock covering the entire read-decide-write interval, and if the lock is already held, this cycle skips rather than waiting its turn. flock -n semantics, wrapped around the whole interval, not just the write. Lining up behind the first writer only hands the second one another chance to write once the first is done. The second writer had no new information and no business writing at all. Skipping costs nothing.
The observability version of that lesson has the same shape. A read and the write that follows it are one critical section. If you cannot see them as one, you will see two of everything. For a long while, span context was the only thing telling us those two writers were the same request, and when the context dropped, we could not tell them apart. Same gap, different layer.
My instrumentation budget goes to decisions: tool selection, state transitions, termination, and the checks that decide whether to write. If a signal will not change a runbook, a dashboard, or an eval, it is exhaust.
Before you touch anything that checks state before it writes, draw the sequence diagram. Two lifelines, the read, the decision, the write, and where the lock sits. We drew ours after the fact; the drawing is what finally made the bug boring. Do it before your hand reaches the keyboard, and give every check-then-write path you own the same treatment. Otherwise you will be back here cleaning up duplicates next week, and the week after that, deciding by hand which of three packages was the real one. Or keep a duplicate-cleaning script in your back pocket. I know which one I would pick.
Top comments (0)