A specific and surprisingly common category of production incident traces back to a background job that was written quickly, for a genuinely reasonable and limited original purpose, and that has been quietly accumulating unbounded state ever since, without anyone revisiting the original assumptions that made it seem safe at the time it was written. This pattern is worth understanding in detail because it's structurally different from most other production failure modes, it doesn't manifest as a bug in the traditional sense, the job continues to work exactly as originally written, the problem is that the original assumptions about scale simply stopped holding true.
The archetypal example, and why it's so easy to write without noticing the risk
A common, almost textbook version of this pattern: an engineer needs a way to track which users have seen a particular notification, to avoid showing it to them again. The straightforward implementation is a background job that maintains an in-memory or lightly-persisted set of user ids who have seen the notification, checked and appended to on each relevant event. At the time this is written, for a feature affecting a modest, bounded set of users, this is genuinely fine, the set stays small, the job runs quickly, and there's no visible problem.
The risk is structural, not a coding mistake in the traditional sense: the set has no built-in bound or expiration, and its growth is tied directly to overall system usage, which is exactly the dimension most likely to grow considerably over the life of the system. A job written when the relevant user base was in the low thousands can still be running, unmodified, when that same user base reaches the hundreds of thousands or millions, at which point the same set that was trivially small at write time has grown into a meaningful, sometimes multi-gigabyte, in-memory structure that the original job was never designed to handle at that scale.
Why this specific failure pattern is so hard to catch in normal code review and testing
Standard code review practices are reasonably good at catching logic errors, incorrect conditionals, off-by-one errors, obvious performance problems visible in the code as written. They're considerably less effective at catching this specific pattern, because the code itself is entirely correct, there's no bug in the traditional sense to spot, the set is being maintained and checked exactly as intended. The problem only exists as a function of how the code's resource usage scales with a specific dimension of production usage that isn't visible from reading the code in isolation, and that a reviewer would need to explicitly think through, "what happens to this structure's size as the relevant usage dimension grows by ten times, by a hundred times", rather than something that naturally surfaces during a normal correctness-focused review.
Testing is similarly ill-suited to catching this pattern, since test environments almost never replicate the multi-year accumulated scale that eventually triggers the actual problem, a test suite running against a small, freshly-seeded dataset will never exercise the code path at the scale where the unbounded growth actually becomes a genuine issue, which means the job can pass every test, pass every code review, and run correctly in production for a long stretch of time, sometimes years, before the accumulated scale finally crosses whatever threshold, available memory, processing time, actually triggers a visible failure.
The failure, when it finally arrives, often looks unrelated to its actual root cause
Because the underlying growth has been gradual and invisible, the eventual failure frequently doesn't present as an obvious, traceable symptom of the accumulating job. It shows up as a memory pressure incident affecting the broader system the job runs alongside, a general slowdown that gets initially misattributed to unrelated recent changes, or a job that starts silently taking progressively longer to complete until it eventually exceeds some unrelated timeout threshold and starts failing in a way that looks, at first investigation, like a transient infrastructure issue rather than a multi-year accumulation finally reaching a breaking point.
Diagnosing the actual root cause in this scenario is genuinely difficult precisely because the connection between "this specific background job, written years ago, for a narrow original purpose" and "the memory pressure incident affecting the whole system today" isn't obvious without someone specifically thinking to check the size of state this particular job has accumulated, which isn't typically the first place an engineer investigating a general system slowdown or memory issue would think to look.
The structural fix: every unbounded accumulation needs an explicit bound or expiration, decided at write time
The practical lesson isn't simply "review code more carefully," which as discussed doesn't reliably catch this pattern anyway. It's a specific, structural discipline worth building into how background jobs and any persistently accumulating state are written in the first place: any structure that accumulates data over time, rather than being bounded to a fixed, predictable size, needs an explicit answer, decided and documented at the time the job is originally written, to the question "what happens to this structure's size as the relevant usage dimension grows substantially, and what bound or expiration mechanism prevents unbounded growth."
For the notification-tracking example, this might mean explicitly expiring entries after a defined period, since the practical need to avoid re-showing a notification typically doesn't require tracking that fact indefinitely, or moving to a data structure and storage approach specifically designed to handle growth at scale rather than an in-memory set that was only ever appropriate for the smaller scale that existed when the job was originally written. The specific solution varies by case, what matters is that the question gets asked and explicitly answered at write time, rather than left as an implicit assumption that happens to hold at the current scale and that nobody revisits until the assumption eventually breaks.
Why this is worth a deliberate, periodic audit rather than relying on catching it at write time alone
Even with genuinely good discipline about asking this question for new code, existing systems accumulate a meaningful number of older jobs written before this discipline was established, or written by engineers who reasonably didn't anticipate the eventual scale the system would reach. A periodic, deliberate audit specifically looking for unbounded accumulation patterns, background jobs whose state grows with usage but has no explicit bound or expiration mechanism, is a genuinely valuable and often underinvested practice, precisely because this failure pattern doesn't announce itself through any of the normal signals, error rates, test failures, obvious performance regressions, that would otherwise prompt an engineering team to notice and address it before it eventually surfaces as a confusing, hard-to-diagnose production incident.
Top comments (0)