The symptom
The log looked clean. Thirty-one tasks executed in one run, zero failures, every task marked complete. If you only looked at the dashboard, it was a perfect day.
Then I actually read the output files instead of the exit codes, and found this: an article titled "Silent Retries Are Hiding Your Rate-Limit Exhaustion: A Circuit Breaker for LLM API Calls" sitting in the output folder twice. Same title, same action_required field, same structure, generated at task index 11 and again at task index 23 of the same run. Two separate LLM calls, two separate sets of tokens spent, two nearly-identical 1,000-word articles, twelve tasks apart, and not a single error, warning, or retry logged anywhere in between.
A pipeline that reports 100% success is supposed to mean 100% of the work was useful. This was the first time I really understood how far apart those two claims can be.
The wrong theories, in order
My first assumption was a race condition. The pipeline dispatches a queue of content tasks, and if two workers pulled the same task definition off the queue before either marked it complete, I'd expect exactly this: duplicate output, no error. I went looking for the dequeue logic first. It was single-threaded. Tasks were popped and processed strictly in order, one at a time. No race was possible — index 11 had fully written its file and returned before index 12 even started. That ruled out the easiest explanation in about ten minutes, which in hindsight should have told me the bug was more structural than a timing fluke.
Second theory: the LLM itself was the problem. Large language models are trained on overlapping data and tend to gravitate toward similar high-signal topics — rate limiting and circuit breakers are a well-worn pattern in LLM-infra content, so maybe the model just kept picking the same idea because it was the most obvious one in the topic space. Plausible, and worth ruling out, so I diffed the two prompts that produced task 11 and task 23. They weren't identical — different task IDs, different position in a themed weekly calendar, different surrounding context — but the topic prompt structure was similar enough that the model could reasonably converge on the same headline twice without any memory of having done so before. This was closer to true, but it wasn't the root cause. It was a contributing factor to why the collision happened, not an explanation for why nothing caught it.
Third theory, and the one that actually held up: nothing in the pipeline had ever been given the ability to know what it had already produced. The content-generation step took a topic prompt, called the model, wrote the result to disk, and marked the task complete. That was the entire contract. There was no step, anywhere in that path, that looked backward.
The actual root cause
The pipeline had a write path and no read-before-write. It generated tasks from a queue, and it generated content from a prompt, but the two systems that would have prevented the duplicate — "what topics have I already covered" and "what am I about to generate" — had never been connected. Each article-writing task ran in total isolation from the seven, or even twelve, tasks that came before it in the exact same run.
This is a subtler bug than it sounds like, because the pipeline's definition of "success" was narrow and technically correct: the task received a prompt, the model returned a well-formed response, the file was written without an I/O error. Every check that existed passed. The metric I was watching — failure rate — was measuring whether the mechanics worked, not whether the output was worth producing. A system can be perfectly reliable at doing pointless work, and its own instrumentation will tell you everything is fine.
The fix
The fix was small on purpose. I didn't want a topic-similarity model, embeddings, or anything that added a new point of failure to a pipeline that was already fragile enough. I wanted the cheapest check that would have caught this specific, real incident:
def is_duplicate_topic(new_title, output_dir, lookback_days=7):
cutoff = now() - timedelta(days=lookback_days)
fingerprint = new_title.strip().lower()[:50]
for path in recent_outputs(output_dir, since=cutoff):
existing = load_title(path).strip().lower()[:50]
if existing == fingerprint:
return True
return False
Before the content-writing step calls the model with a finalized title, it checks the last seven days of output titles (first 50 characters, case-insensitive, which is forgiving enough to catch near-identical headlines without needing fuzzy matching). If it matches, the task doesn't silently regenerate — it either skips generation and logs a duplicate_skipped result, or flags the task for a human to decide whether a genuine follow-up piece is warranted. Either way, it's now a visible outcome instead of an invisible one.
The deeper fix was changing what the pipeline's success metric actually meant. "Zero failures" got redefined to exclude duplicate-skipped tasks from the numerator of useful output, so a day where twelve of thirty-one tasks got skipped as duplicates now reads as a distribution problem worth investigating, not a green checkmark.
The lesson
Any pipeline that generates something on a schedule — articles, reports, emails, alerts, PR descriptions, whatever your queue produces — has a write path. If it doesn't also have a read-before-write path that checks recent history, it will eventually generate the same thing twice, and its own success metrics will actively hide that from you, because duplication isn't an error. It's indistinguishable from success unless you specifically go looking for it.
The uncomfortable part wasn't the wasted model tokens on a second copy of one article. It was realizing I'd built a system that could waste a third of its daily output on any given day and report a perfect record the entire time. If you're running anything unattended, the question worth asking isn't "did every task complete without error." It's "does this system know what it did yesterday." Mine didn't, until I made it check.
Top comments (0)