DEV Community

wzg0911
wzg0911

Posted on

I built a crash-checkup tool for AI agents, then used it to dissect a real production meltdown — and upstream PRs later proved the diagnosis right

I built a crash-checkup tool for AI agents, then used it to dissect a real production meltdown — and upstream PRs later proved the diagnosis right

I'm an indie dev who builds small tools for my own use.

A while back I built something called ARK — Agent Reliability Kit. It does one narrow thing: health-checks agent systems for the three failure classes most likely to kill an agent at 3am. I built it because those exact classes had burned me.

Then one day, browsing the OpenClaw issue tracker, I hit #113434.

A developer on Windows 11 upgraded to 2026.7.2-beta.4 and watched Gateway memory climb until it ate all available RAM and crashed. Weirder: Builder/Codex sessions kept rejecting with:

Codex session generation is no longer current: <session-id>
Enter fullscreen mode Exit fullscreen mode

Resetting the session didn't help.

My first reaction was the same as yours: "another memory leak, suggest swap, move on." But that no longer current line made me stop. It didn't smell like memory — it smelled like a handshake/state mismatch. So I kept reading, and the further I read the clearer it got: this wasn't one bug. It was two independent regressions colliding in the same incident.

Pit #1: uncached scans that boil the RAM

Every Control-UI tick called sessions.catalog.list and sessions.files.list fresh — each one a full scan of the Codex session store. Big store → slow scan → next tick starts a new scan before the old one finishes → overlapping scans → memory climbs monotonically until Gateway-wide unresponsiveness and OOM.

This class is called unbounded repeated work. The fix shape: a short-TTL cache plus coalescing concurrent callers onto one in-flight promise. Two lines of thinking; the memory curve goes from staircase to flatline.

Pit #2: reset retires the generation but hands back the old ID

This was the real culprit behind no longer current.

sessions.reset invalidates the current Codex generation — but returns the same session ID to the client. Next turn, the client presents the old generation token bound to that ID, and the server rejects it: your generation is retired.

Classic idempotency-boundary bug: the mutation had a side effect (generation bump) that the caller's handle never learned about. Fix shape: reset should return a new generation binding for explicit rebind, not silently reuse the old ID.

Pit #3: UI polling with no backpressure

More tabs → multiplied polling → Pit #1 amplified. Debounce to ≥2s, pause when hidden.

The real discovery came after I lined them up

Two critical findings plus one medium — and they mapped one-to-one onto the three failure classes ARK guards:

Finding Failure class ARK guard
F-1 uncached scans Unbounded repeated work IdempotencyGuard
F-2 reset reuses old ID Stale handle after mutation OutputValidator
F-3 poll storm Retry/poll storm CircuitBreaker

Not because I'm clever — because these failure modes are that common in agent systems.

So instead of leaving a "try adding a cache" comment (too cheap, and disrespectful to whoever lived through that night), I wrote the whole analysis up as a structured diagnostic report — root-cause table, fixes with code, and a stopgap checklist you can run today. Static page, no signup, no tracking:

📋 https://ark-6ek.pages.dev/reports/openclaw-113434

Epilogue: the story got a third-party ending

The day after I published, a user posted a current-main update under the issue — not opinions, merged PRs:

  • #114056 fixed the reset-reuses-old-ID path (Pit #2)
  • #114401 / #114478 replaced the full rescans with incremental folding + bounded LRU + singleflight + paging yields (Pit #1, taken further than my pseudo-patch)
  • #114358 removed duplicate enumeration within one catalog request

Both critical findings — independently confirmed by merged upstream code. The fix shapes matched the report's recommendations.

I'm not saying "I called it." I'm saying: there's a strange, grounding feeling when a classification you abstracted from your own incidents gets validated by strangers merging code into main. That beats any marketing number.

If you run agent systems and want to know whether yours has these three invisible crash risks — idempotency guards, retry policy, logging — ARK has a free 30-second checkup, static, no signup:

🔗 https://ark-6ek.pages.dev/diagnose

And if you've ever lived the crash-restart-crash loop at 3am, tell me about it in the comments. Odds are I've seen your exact flavor.

(Technical analysis of a public GitHub issue; not affiliated with the OpenClaw project.)

Top comments (0)