A few years into running background jobs for client projects, we noticed a pattern in the incident reports that all traced back to the same root cause: an in-memory queue, holding real work, that disappeared the moment the process holding it restarted. Not a bug in the job logic. A gap in the architecture that only showed up under conditions the local dev environment never reproduces.
Here's what convinced us to stop building job queues this way, what we build instead now, and what the migration away from the old approach actually looked like in practice.
Photo by Pixabay on Pexels
The demo always works
An in-memory queue, an array or a simple scheduler holding jobs in process memory, is genuinely the fastest thing to build. It works perfectly in local testing and in a demo, because neither of those environments restarts the process mid-job. Production restarts constantly: deploys, autoscaling events, crashed dependencies taking the whole process down with them. None of these ask permission, and none give the queue a chance to hand off its pending work before the process exits. The gap between "works in the demo" and "survives a Tuesday afternoon deploy" is exactly the gap that in-memory queues never close.
The first incident that changed our approach
The specific incident that shifted our thinking involved a client's order confirmation emails. A deploy went out during a busy afternoon, landing mid-batch on a set of confirmation jobs held in memory. The process restarted cleanly, the deploy "succeeded" by every monitoring signal we had, and about forty orders from that fifteen-minute window never got a confirmation email. No error. No crash. No signal in any dashboard we were watching, because from the queue's perspective, that work had simply never been logged anywhere it could be checked after the fact. We only found out because a support ticket asked why an order confirmation never arrived, and tracing it back took most of an afternoon precisely because there was no record of the job ever having existed.
What we changed
The fix wasn't a smarter in-memory data structure. It was moving job state out of the process entirely, into something that outlives any single process: a database table or a store like Redis. Every job became a row with a status field, not a value living in a variable. That single architectural shift, more than any code-level fix, is what actually solved the problem.
The second piece we added was a visibility timeout: when a worker claims a job, the claim expires after a set window. If the worker finishes first, the expiration is irrelevant. If the worker dies first, the claim lapses and another worker picks the job back up. Without this, a crash mid-job leaves the job stuck in a "running" state that nothing ever revisits, which is exactly what happened during that first incident.
The part that's easy to get wrong even after you fix the architecture
Making the queue durable solves half the problem. The other half is making retries safe, since a durable queue with a visibility timeout guarantees that some jobs will eventually run twice. We now attach an idempotency key to every job with a side effect that shouldn't happen twice, an order ID for a confirmation email, a request ID for a webhook delivery, checked against a small table before the side effect fires again.
We also cap retries. A job that fails identically every time, because the payload itself is malformed rather than the failure being transient, doesn't get smarter on the fourth attempt. It gets moved to a dead-letter table where a human can look at it, instead of quietly consuming worker capacity forever.
What the migration actually looked like
Moving an existing client project off an in-memory queue was less disruptive than we expected going in. We ran the new database-backed queue alongside the old in-memory one for about two weeks, routing new job types to the new system while letting existing in-flight job types finish out on the old one, rather than attempting a single cutover on a fixed date. This staged approach caught two bugs in our new claim-locking logic under real concurrent load that never surfaced in staging, where job volume was too low to trigger the race conditions that showed up with real traffic.
What surprised us most about the fix
Going into the redesign, we expected the durability work, moving job state into a table, adding visibility timeouts, to be the hard part, and idempotency to be a quick afterthought bolted on at the end. It turned out to be closer to the opposite. The durable-storage piece was mechanical once we'd settled on an approach; the idempotency work required actually sitting down with each job type and asking, specifically, what happens if this exact action fires twice, which surfaced a handful of side effects (a webhook to a partner system, a Slack notification) that nobody had thought carefully about before, because in the old in-memory world, "running twice" essentially never happened, so nobody had been forced to reason about it.
Where we landed
For most client projects now, unless job volume is genuinely high, we default to a database-backed queue on infrastructure the client already runs, adding Redis only once polling latency actually becomes the bottleneck rather than a hypothetical one. PostgreSQL's SELECT ... FOR UPDATE SKIP LOCKED pattern gets us most of the way to a correct, durable queue without introducing a new piece of infrastructure to operate and monitor.
We also added lightweight monitoring from day one on every new queue we build: pending job count over time, average completion latency by job type, and a running count of anything sitting in the dead-letter table. None of this needs to be elaborate to be useful. A daily glance at those three numbers catches most of the same problems that used to surface as customer-reported incidents, just days earlier and with far less drama.
We wrote up the full pattern, visibility timeouts, idempotent workers, dead-letter handling, and a worked example walking through an order-confirmation queue end to end, in a longer piece on building a job queue that survives a server restart.
What we'd tell a team about to make the same mistake we made
If you're building a job system today and reaching for an in-memory array because it's fast to ship, the honest question to ask isn't whether it'll work, it will, in the demo and probably for the first few weeks in production too. The question is what happens to whatever's in that array the first time a deploy lands while jobs are running, because that deploy is coming regardless of how careful your release process is. If the answer is "we'd lose it and might not notice for days," that's the signal to spend the extra afternoon moving job state into a table before it ships, not after the first support ticket asks where an order confirmation went.
The takeaway
None of this requires exotic infrastructure. Most teams can build all of it on top of a database they already run. What it requires is treating "the process might restart mid-job" as a certainty to design for, not an edge case to handle if it ever comes up. It always comes up. The only question is whether the queue was built expecting it, and whether anyone finds out from a dashboard or from a customer.
Top comments (0)