Last week I watched a production service get restarted 847 times in four hours because a downstream dependency was having a 200ms tantrum. Not down. Not broken. Just slow enough to trip a health check. Every single time, Kubernetes went “oops, dead” and yeeted the pod into the sun. By morning the logs looked like glitch art and the actual bug, if there even was one, was buried under a mountain of identical stack traces.
That’s when it hit me: we treat “failed” like it’s one word. It isn’t. Some failures just need a breath. Some need a doctor. Some need surgery. And if your only move is “restart and hope,” you’re either burning money on blips or ignoring bugs that restarts will never touch.
I ended up sketching a three-tier recovery system that prices the fix to match the failure. Think of it like triage in an ER, except the nurses are circuit breakers and the surgeons are LLMs with strict instructions not to touch anything outside the operating field.
Tier 1: reflexes
Tier 1 is pure autonomic reflex. Component supervisor, exponential backoff, per-component circuit breaker, maybe a category-specific override if you’re fancy. Sub-second latency, zero LLM calls, zero round-trips to some API that bills you per token. This is the stuff that should handle 95% of your incidents: a GC pause, a slow disk write, a brief upstream blip, some noisy neighbor hogging CPU on the same node.
I used to think “self-healing” meant something with transformers in it. Turns out most healing is just “wait 400ms and try again, bro.”
The circuit breaker is the real hero. After N failures in a window, it stops the bleeding. Component goes degraded. Supervisor stops hammering restarts. And only then—only when the cheap reflexes have failed—do we escalate.
Tier 2: the immune system
That’s Tier 2. A watcher collects an evidence bundle: recent logs, the last traceback, on-disk state files, environment context. It hands that bundle to a small triage LLM and asks one question: which bucket?
Not “what should we do?” Not “write me a fix.” Just pick from eight options:
class FailureCategory(Enum):
CORRUPT_STATE = "corrupt_state"
STALE_LOCK = "stale_lock"
RESOURCE_EXHAUSTION = "resource_exhaustion"
UPSTREAM_DOWN = "upstream_down"
CONFIG_ERROR = "config_error"
CODE_BUG = "code_bug"
DEPENDENCY_NOT_READY = "dependency_not_ready"
TRANSIENT_UPSTREAM = "transient_upstream"
That's it. Eight values. The LLM cannot invent category nine. It cannot suggest a shell script. It classifies, then regular code looks up the action:
REPAIR_ACTIONS = {
FailureCategory.CORRUPT_STATE: reset_state_file,
FailureCategory.STALE_LOCK: remove_stale_lock,
FailureCategory.RESOURCE_EXHAUSTION: prune_resources,
FailureCategory.UPSTREAM_DOWN: wait_and_escalate,
FailureCategory.CODE_BUG: escalate_to_tier_3,
}
This is the part that clicked for me. LLMs are great at pattern matching over messy context, but they’re a bad idea for safely executing arbitrary commands. So don’t let them. Constrain the output space to an enum and suddenly the decision is auditable. You can log “classifier said stale_lock” and anyone can verify whether that made sense. Compare that to a free-form prompt where the model returns “I think you should delete /var/lib/data and also maybe restart postgres.” Hard pass.
Tier 2 costs one cheap LLM call per real failure. Not per blip. Blips die in Tier 1. So you’re paying maybe a few cents for incidents that actually need a brain.
Tier 3: surgery
Then there’s Tier 3. If the classifier returns code_bug, you bring out the bigger model. It reads the failing component’s source code, generates a patch, runs the existing test suite, and—here’s the kicker—only auto-deploys if every changed file lives inside that component’s module subtree.
def patch_is_in_scope(patch: Patch, component_module: Path) -> bool:
return all(
changed_file.resolve().is_relative_to(component_module.resolve())
for changed_file in patch.changed_files()
)
That scope guard is everything. LLM patch agents hallucinate. They get over-eager. I've seen a "fix" try to refactor three unrelated modules because the model decided everything was connected. The scope guard says: no. You fix this component. If the real fix needs to touch both config and consumer together, the patch gets rejected and a human takes over. Slower? Sure. But one bad auto-deploy can tank the whole system, so I'll take the slower path.
The test suite check is obvious but worth stating: if the patch breaks existing tests, it doesn't ship. The model writes code, but deterministic gates decide whether it runs.
Don’t let the tiers leak
Now here’s where it gets tricky. The boundaries matter more than the tiers.
If your circuit breaker resets too aggressively, Tier 2 never fires and you silently absorb a thousand restarts a day. The underlying bug becomes invisible noise. I made this mistake early—set the breaker timeout to 5 seconds because I was impatient. Ended up masking a real state corruption issue for two weeks. Not my finest moment lmao.
If you let the Tier 2 classifier invent repair actions, you’ve basically rebuilt Tier 3 with worse safety. The whole point of the enum is that the LLM doesn’t get a steering wheel.
And if your Tier 3 scope guard is loose, congratulations, you’ve given an LLM write access to your entire codebase. I’ve seen enough agent demos go sideways to know that’s not a bet I want to take in production.
The real insight isn’t that any of these tiers are new. Circuit breakers are ancient. Diagnostic LLMs exist. Code-writing agents exist. What’s new is composing them with strict escalation gates so each tier only handles the failures it’s priced for. Cheap stuff stays cheap. Rare, hard stuff gets expensive attention. No tier leaks into the next.
And this pattern pops up everywhere once you start looking. Spam filters do it: regex first, small classifier second, human review third. Compilers do it: parse error, suggested fix, ask the user. Anywhere the right action costs wildly different amounts across cases, tiered escalation pays for itself.
So if you’re building long-running services, stop defaulting to “restart and hope.” Build recovery like you actually understand that failures come in flavors. Let Tier 1 eat the transient noise. Let Tier 2 classify the real incidents into a tiny, maintainable enum. Let Tier 3 patch, but only inside a cage with thick bars and a good test suite.
The future isn’t an LLM that runs your ops team. It’s an LLM that knows exactly when it’s allowed to speak, what it’s allowed to say, and where it’s allowed to touch. If you can’t tell I’m big on AI knowing its place.
For those following along my twitter for a while, this is exactly what I’ve put into Jarvis (to be renamed) , my personal assistant thats heavily inspirted by MCU Jarvis from the movies. Work in progress still but its at a decent point now. Look forward to more blogs on Jarvis’ internals coming up soon.
Top comments (0)