How a $200 debugging session on a hung Celery ingest queue turned into two permanent rule files — and why I didn't notice, for months, that the class of bug they were written to prevent had simply stopped happening.
The queue that only died on the cheap hardware
Here's the setup, stripped of anything that identifies the client: a RAG ingestion pipeline for an AI content platform. Celery workers at concurrency=2, pulling documents, chunking them, batching the chunks to an external embedding API rate-limited to 8 requests per second through a Valkey-backed limiter, then upserting the vectors into pgvector. Nothing exotic. The kind of pipeline every "add RAG to your SaaS" tutorial waves through in one paragraph.
On the 4 vCPU dev box, it worked. Every time. On the 2 vCPU staging profile — the one that actually matched the client's budget tier — it hung. Not crashed. Hung. The first ingest job would climb to embed_cursor=64/190 and stop. The second job would go silent immediately. The maintenance worker's own dashboard kept reporting global_active=2 — two jobs "actively processing" — while the reclaim sweep logged reclaimed:0, over and over, because as far as the system was concerned, nothing was dead. Both jobs were still sending a heartbeat. The queue was full, healthy-looking, and completely incapable of finishing anything.
I want to be honest about what this actually cost me, because "async is hard" is a sentence people nod at and then don't budget for: about $200 in agent-hours before I stopped treating it as a flaky-infra problem and started treating it as a design problem. The first four or five passes were exactly the debugging you'd expect — restart the worker, bump the concurrency, add a retry, blame Docker's CPU throttling, blame the embedding provider's rate limiting, blame the reclaim sweep's polling interval. Every one of those "fixes" made the symptom move to a slightly different spot in the log without touching the actual defect. That's the expensive kind of bug: not the one that's hard to find, the one that's easy to appear to fix five separate times.
What was actually broken (there were nine things)
When I finally forced myself to write the failure down as a state machine instead of a stack trace, it stopped being one bug and turned into a small catalogue:
- The heartbeat renewal had no relationship to actual progress. A worker stuck on a hung external
awaitkept renewing its own lease forever — the pulse looked alive because nothing was checking whether the cursor was still moving. - The rate limiter's own bookkeeping —
check → increment → maybe rejectas three separate steps instead of one atomic operation — consumed a token even on a rejection. Under concurrency, the bucket found ways to report itself full while doing almost no real work. - Retries after a 429 went around the shared limiter entirely, so two workers retrying at once could burst well past the declared 8 RPS ceiling, which produced more 429s, which produced more retries.
- A broad
except Exceptionaround the provider call collapsed a retryable timeout into a terminal failure — so the "self-healing" retry logic was, in a meaningful number of cases, silently dead on arrival. - Retry existed on two layers at once (the Celery task's own
autoretry_forand a manual service-level requeue), so a single failure could multiply into duplicate attempts that bypassed the limiter a second time. - Liveness was determined by three different, disagreeing sources of truth: a 120-second lease set at dispatch, a 5-minute "staleness" window checked by the reclaim sweep, and a
updated_attimestamp that got refreshed by the very heartbeat that was lying about progress — which meant an expired lease could get "rejuvenated" back to alive. - The reclaim sweep itself ran once every 300 seconds against a 120-second lease — so a genuinely dead worker could sit holding a slot for up to five minutes before anything even looked at it.
- The batch that was supposed to run in parallel under the 8 RPS budget was actually running sequentially — so real throughput was closer to 1 RPS, and the "rate limit" was never the bottleneck; unused parallelism was.
- And, almost funniest in hindsight: the regression test suite had a test that asserted the old, wrong behavior — "a job stays active even past an expired lease" — so the fix, when I found it, technically broke a passing test. The test had quietly become the spec for the bug.
None of these, individually, reads as dramatic. Together, on a 2 vCPU box where there was zero slack to hide the timing, they added up to a queue that looked alive and was completely dead.
The two files that replaced a note to self I'd never reopen
I run an autonomous coding agent under a written rule system I've been building for a while — I call it LEO, and it's open on GitHub if you want the whole thing. The instinct after a bug like this is usually "note it in the ticket, move on." I did something slower and, it turned out, considerably more valuable: I turned the nine defects above into two permanent files the agent is required to read before it's allowed to touch a queue or a pipeline again.
The first file is the why — a canon that breaks the incident down cause by cause and turns each cause into a numbered law: heartbeat has to renew alongside actual progress, not on its own; the rate limiter has to be an atomic check-and-consume sitting at the wire, in front of every attempt including retries, and a rejection is never allowed to spend capacity; there is exactly one owner for each error class's retry logic, and every other layer that could also retry it is explicitly turned off; liveness has one single source of truth — a lease-expiry timestamp — read by exactly one function everywhere it matters, with the reclaim period mathematically forced to be at most half the lease TTL.
The second file is the reflex — not the theory, the muscle memory. It's a literal grep checklist: patterns like await (client|http|session).\. without a nearby timeout=, or except Exception: wrapped around a provider call, or a retry/backoff block sitting outside a shared limiter instead of inside it. The agent runs this over its own diff before calling anything done. It's deliberately mechanical — the point isn't "think about async safety," it's "if this exact text pattern shows up in your change, stop and answer this exact question before you ship."
Both files went in, along with an update to the architecture rules requiring a filled-in "pipeline passport" — retry ownership, limiter placement, lease numbers, a stuck-progress deadline — before anyone, agent or human, is allowed to start a new background pipeline. Then I moved on to the next client deliverable and didn't think about it again for months.
Then came something with a lot more surface area for this exact bug
The next chunk of that engagement was a visual pipeline constructor for the same platform — the kind of node-graph builder people compare to n8n or Zapier, except purpose-built for AI content generation instead of generic SaaS glue. Twelve node types on the canvas: a retrieval node doing hybrid vector + full-text search against the knowledge base, an LLM node with prompt versioning and a pinned-vs-latest toggle, a transform node for post-processing model output, a conditional branch node, a human-in-the-loop approval gate, an external video-generation node, an image-generation node, a voice-assignment node, a generic webhook node, a persistence node, and a scoring node for rubric-based evaluation. Users draw a graph, publish a version, and a graph-executor built on LangGraph — running on top of the same Celery workers — walks the nodes in order, writing a dual journal as it goes: relational step-by-step logs for operators, plus a full graph-state checkpoint for crash recovery.
That is, structurally, a much bigger and much more inviting target for exactly the class of bug I'd just paid $200 to understand. More external calls. More places a heartbeat could lie. More opportunities for two nodes' retries to collide on a shared provider. A human-approval gate that can sit "pending" for hours by design, which is precisely the kind of long-lived state that makes a naive liveness check look broken even when it's working correctly.
That build shipped twelve node executors, a credential-resolution layer with its own provider dispatch, a wallet-preflight cost estimate before triggering paid video generation, a separate leased-and-reclaimed job queue for the image/video side with its own queue-depth alarm — real production async surface, staged, load-tested. And at no point during any of it did we hit the deadlock shape from the ingest incident. Not once did a worker's heartbeat lie about progress. Not once did a retry storm past a shared limiter. I watched the agent stop itself mid-diff once or twice, on a grep hit neither of us went looking for — the reflex file doing exactly the job it was written for, quietly, without either of us noticing it was doing anything at all until I went looking afterward.
The moment I actually noticed
It wasn't until I sat down to write a retrospective on the whole engagement — the kind of "what did we learn" doc nobody enjoys writing — that it landed. We'd shipped something with roughly an order of magnitude more asynchronous surface area than the pipeline that had cost me $200 to understand, and the specific failure mode that pipeline taught me about had simply not recurred. Not "recurred and got caught in review." Not "recurred and cost less time to fix." Absent. I had to go check the two files' own dates against my git history to convince myself I wasn't misremembering — they were written the week of that first incident, months before a single node executor for the constructor existed.
That's a strange thing to notice about your own work, because it's the opposite of how debugging normally feels. Normally you fix a bug, and six weeks later a cousin of that bug shows up in a different file, because the actual lesson — "a heartbeat proves a pulse, not progress" — lived in your head, or in a chat log nobody re-reads, and your head is a lossy, non-versioned medium that a fresh problem doesn't automatically consult. The fix generalized this time because it wasn't stored as a memory. It was stored as a gate the agent has to pass through, the same way a database constraint doesn't care whether the person writing the INSERT remembers why double-booking is bad — the schema itself won't allow it either way.
Why this is the actual point, not a nice side effect
I don't think the interesting story here is "I fixed an async bug." Async bugs get fixed constantly; that's not news. The interesting story is that the fix stopped being something I had to remember to apply. It became infrastructure I stopped consciously thinking about — which, if you've ever tried to enforce a coding standard across a team by asking people to "please remember" it, is exactly the property you actually want and almost never get. A rule that requires ongoing vigilance decays. A rule the agent physically cannot get past — because it's a grep pattern with a mandatory stop-question attached, not a paragraph of advice — doesn't decay, because it was never running on anyone's memory in the first place.
I'll add one more thing, because it's the part that convinced me this generalizes past my own project: I periodically dump the whole rule system into a fresh, completely unprimed model session — no history, no accumulated context, just the raw files — specifically to get an adversarial read on my own constitution. The read I got on these two files, unprompted, was that the grep-based reflex works because it forces a specific kind of attention collision: the moment the pattern await client.post( shows up in a diff with no timeout= anywhere near it, the model has to reconcile that literal text with a literal rule sitting three lines below it in the same file. There's no interpretive gap left for the model to quietly skip past. That's a fairly mechanical description of why a text file changed a production outcome, and it matches what I watched happen across an entire node-based pipeline constructor that never once reproduced the bug it was written to prevent.
If you're fighting the same class of problem — a queue that works until it doesn't, on hardware that's just constrained enough to expose the race your dev box hides — the two files are ASYNC_WORKERS_CANON.md and ASYNC_AWAIT_REFLEX.md in the LEO repository. But I want to be precise about what actually made them hold, because it isn't the two files sitting alone in a folder — that's a memo, and memos get skimmed. @LEAD is what routes a queue task to those two files instead of leaving them undiscovered next to 125 others about something else entirely. @DEV is the role contractually bound to run the reflex grep over its own diff before calling anything finished, not a role that's merely encouraged to. @QA_ARCH is what checks, on every pass, that the numbered laws in the canon actually show up in the code and not just get cited in a commit message. @ARCH is what refuses to let a new pipeline start at all without the passport those two files require filled in — retry ownership, limiter placement, lease numbers, named out loud before a single executor exists. Hand the same two files to a model with no routing, no enforcement, and no design-time gate around them, and you get a well-written document nobody was ever required to open. The two files are the specific lesson from my $200. The other 125 are the reason that lesson is binding instead of optional — which is the actual claim underneath this whole piece: the reliability came from the composition, not from any one file, including these two.
P.S. A few facts change how you should weigh everything above, so they belong here instead of staying unsaid: I've been working in tech for under a year, I work alone, and I'm exactly the kind of "AI-native" developer a certain type of senior engineer treats as a punchline rather than a job description. No team standup to catch what I miss. No tech lead to overrule a bad call before it ships. No corporate process quietly doing half the discipline for me in the background. I didn't write the LangGraph executor by hand, and I wasn't the one stepping through Celery's internals at 2am. My agent was — reading the two files above, the same way ASYNC_AWAIT_REFLEX.md itself says an agent, "or, honestly, a human," is required to. What I actually did was spend the $200 turning one incident into a rule the agent can't get around, then spend the months after that directing, not typing.
I'm not saying this to lower the bar on what you just read. I'm saying it because it's the stronger version of the point, not a weaker one. "An experienced engineer solved a hard concurrency bug" mostly proves that the engineer was experienced — it doesn't tell you much about the process. What actually held the line here was a file the agent was structurally required to obey, not a decade of calibrated instinct sitting behind the keyboard, and not a team of reviewers backing it up either, because there was neither. Someone a year into this industry, working solo, with no one else in a position to catch what the agent got wrong, is a harder test of the claim than a ten-year veteran with a team behind them running the same setup would have been. It held anyway. That's the part of my background worth mentioning here — not as a disclaimer, and not as a rebuttal to anyone who'd rather dismiss the label than read the two files — but as the stress test.
— Alex Zaporozhan
Top comments (0)