DEV Community: Yuhai Xia

Your agents share state. How does the next one know it's still true?

Yuhai Xia — Fri, 07 Aug 2026 04:15:35 +0000

Most of the writing about agent memory is about getting information in — what to store, how to chunk it, which embedding model. I want to ask about the other end, because that's where it broke for us.

An agent reads a piece of shared state. How does it know that state is still true?

The failure looks like success
We keep a task canvas that agents read at the start of a session and update as they work. Plan, current step, what's done, what's next. Another agent picks up where the last one stopped. When it works it feels like magic.

Here's the failure mode I didn't design for. An agent opens a canvas that was last written six days ago. It says current step: API integration. That was true when it was written. Since then a human changed direction, a different agent did unrelated work, and the API integration was abandoned.

The agent reads it and confidently continues. Nothing is corrupt. Nothing throws. The state is stale, and a stale read is byte-identical to a fresh one.

My first instinct was to fix the write side: make agents update state before they stop. That doesn't work, and I think the reason generalises. The failure is an absence. An agent that quits without writing — because it crashed, hit a quota, or simply decided it was done — leaves nothing to intercept. You cannot validate a write that never happened.

So we moved the check to the read. Every read carries its own age:

[canvas: last updated 6 days ago, 40 memory writes since.
Treat completed/next as unverified.]
It helps. The agent hedges instead of charging ahead. But it's a nudge, not a guarantee — I'm handing a probabilistic system a warning label and hoping it reads it.

What I actually want to ask
I've been assuming this is a solved problem somewhere and I just haven't found the right vocabulary for it. A few specific things I'd like to hear how others handle:

Is age even the right signal? Six days old and untouched might be perfectly valid for a slow project. Six minutes old is worthless if three agents wrote in between. I use "writes since" as a proxy for drift, but it counts unrelated writes too. Has anyone found a signal that actually correlates with "this is no longer true"?
Who is allowed to invalidate? We let agents propose that a decision is superseded, but a human confirms anything that overwrites an earlier decision. I honestly don't know whether that's good judgment or just fear. If you let agents invalidate each other's state freely, does it converge or does it thrash?
Concurrent writers. Two agents update the same state within seconds of each other. We use locks with leases, which prevents corruption and does nothing about the semantic conflict — both writes are individually valid and jointly incoherent. Is there a pattern better than last-write-wins plus a human noticing later?
Does the model actually respect the hedge? This is the one I have least data on. I put a staleness warning in the read output and I observe better behaviour, but I have not built an eval that isolates it. If you've measured whether a caution in tool output changes what the model does, I'd genuinely like to know how you set that up.
Is any of this different from cache invalidation? Some days I think this is just cache invalidation wearing an AI hat and I should go read distributed-systems literature instead of inventing vocabulary. Other days the fact that the consumer is a language model — something that will happily fill gaps with plausible fiction — feels like it changes the problem. I can't decide.

The part I keep coming back to
Traditional software fails loudly when it reads something wrong. Types don't match, a parse fails, an assertion trips. An agent reading stale state does the opposite: it produces confident, coherent, entirely reasonable work based on a world that no longer exists. The output looks more trustworthy than a crash would.

That inversion is what makes it hard. Every instinct I have as an engineer is tuned to catch things that break. This doesn't break. It just quietly stops being right.

If you're running more than one agent against shared state, I'd like to know what you do about this — even if the answer is "nothing yet, and it hasn't bitten us." That's a useful data point too.

Disclosure: this post was drafted with AI assistance from my own notes, then edited and fact-checked by me before publishing.

Our backend died every 6 hours for a week. The interval was the clue.

Yuhai Xia — Wed, 05 Aug 2026 03:01:58 +0000

For about a week, our backend was killed and restarted by its watchdog roughly every six hours. No crash. No OOM. Exit code 0. Health checks simply stopped answering, and a few minutes later the process came back and behaved perfectly.

I lost more time than I want to admit treating this as an infrastructure problem. The thing that finally cracked it was noticing that the interval was too regular.

What we thought was happening
The first three restarts looked like bad luck. Container platforms restart things. Memory pressure, a flaky host, a network blip — you shrug and move on.

Then I pulled the full watchdog log instead of the last few lines, and the shape was unmistakable:

07-23 04:38 health FAILED 2/2 — restarting
07-23 10:41 health FAILED 2/2 — restarting
07-23 16:43 health FAILED 2/2 — restarting
07-23 22:45 health FAILED 2/2 — restarting

Six hours and two minutes apart, drifting a minute or two each cycle. Twenty-two of them.

Infrastructure failures are not punctual. Anything that regular is something in your own code that runs on a timer. The drift was the restart latency accumulating — which meant each cycle was being scheduled relative to the previous failure, not to a fixed clock.

That last detail turned out to be the whole story.

Why it re-armed itself
Our reflection scheduler restores its clock from the database on boot: it looks at the last recorded run and schedules the next one six hours after that. Perfectly reasonable — it survives restarts without double-running.

But when a cycle dies mid-flight, the last recorded stage is written at the moment of death. So the next boot schedules the next attempt six hours after the crash — landing on exactly the same code path, with exactly the same data, and dying exactly the same way.

The scheduler was faithfully reproducing the crash on a timer. Every run had the same input because no run ever finished.

The actual bug
Deep in the entity-resolution step:

reflection/engine.py

facts = embed_titles_incremental(missing) # ← not awaited, because it isn't async

embed_titles_incremental walks down into an embedding client that does a synchronous httpx.post with a 30-second timeout, plus time.sleep() between retries. On the event loop.

One tenant had 9,126 entities and a cache holding 8,352 of them. Batch size 10. That is several hundred sequential blocking HTTP calls, back to back, on the only thread that answers health checks.

The loop wasn't deadlocked. It wasn't starved of CPU. It was simply not running — parked inside a blocking socket read while asyncio waited politely for control to come back. From the outside: a live process, an open port, and nothing answering.

The 20-minute asyncio.wait_for we had wrapped around this call did nothing at all. A timeout needs a running event loop to fire, and the loop was the thing that was gone.

The detail that made it permanent
The cache of computed embeddings was written after the entire missing set finished. Killed at minute two of a twenty-minute job, we wrote nothing. Next cycle: same 774 missing entities, same doomed walk.

A partial-progress bug and a self-rescheduling bug on their own are each survivable. Together they build a machine that reproduces its own failure forever.

The fixes
Move the blocking call off the loop. One line, and the irony is that the same function was already wrapped correctly at another call site, complete with a comment explaining why. The reflection path was simply missed when that fix went in.

facts = await asyncio.to_thread(embed_titles_incremental, missing)

Checkpoint in chunks. Process 200 at a time and write the cache after each chunk. A kill now costs at most one chunk, and the backlog shrinks monotonically instead of resetting. This is what actually broke the loop — even if something kills the job again, it can no longer make zero progress.

Sweep for siblings. If one blocking call reached the loop, others did too. We found three more: a bare synchronous vector-store query in the retrieval path, a 600-epoch NumPy computation in a scheduled job, and an unbounded connection acquire inside the health endpoint itself — which is a special kind of unfortunate, since it means the check you rely on to notice trouble is one of the things that can hang.

Make the next one self-documenting. A side thread now watches a heartbeat the loop bumps every second. If it goes quiet for more than ten seconds, the thread writes an all-thread stack dump to disk — before any external watchdog gets around to killing the process.

def _watch(self):
while not self._stop.is_set():
if time.monotonic() - self._last_beat > self.threshold:
faulthandler.dump_traceback(file=self._dump_file, all_threads=True)
time.sleep(1)

That last one is the piece I'd install first if I were doing this again. Everything else was a fix. This is the thing that means the next unexplained hang costs an hour instead of a week.

What I'd tell past me
Regularity is a fingerprint. Infrastructure fails at random. Your own scheduled code fails on a schedule. If the interval between incidents is suspiciously round, stop reading platform metrics and go look at what you run on a timer.

A timeout around a blocking call is decoration. asyncio.wait_for cannot interrupt a synchronous socket read. If the thing you're wrapping isn't yielding to the loop, the timeout is a comment.

Grep for the fix you already made. The correct to_thread wrapper existed in this codebase, with a comment explaining exactly this hazard, at a different call site. When you fix a class of bug, search for every other caller the same day — otherwise you've fixed an instance and left the class.

Disclosure: this write-up was drafted with AI assistance from my own incident notes and production logs, then edited and fact-checked by me before publishing.

Five stacked root causes behind one fake "verification failed"

Yuhai Xia — Mon, 03 Aug 2026 06:30:38 +0000

Our OAuth flow worked for Claude Desktop on the first try. The same server, the same spec, the same endpoints — Codex could not authorize against it at all. It took five root causes stacked on top of each other before it did, and the last one still annoys me.

Here they are in the order we actually found them, including the false victories in between.

The symptom
User clicks authorize. The email code arrives, they type it in, the browser redirects. Codex says: "verification failed."

Now the part that made this miserable: our server logs for the entire flow showed nothing but 200, 201, and 302. Not a single error, on any request, anywhere. Every hop of the failure was invisible from the server side.

It took access-log timestamps, a screen recording, and user-agent telemetry lined up on one timeline before the chain made any sense.

Layer 1: the stream that never ends
During verification, Codex issues a bare GET against the MCP endpoint — and waits for the response to complete.

Our endpoint answered that GET with a never-closing keepalive event stream. We kept it deliberately: mcp-remote's probe GET is header-identical to a legacy SSE client, so you cannot tell them apart, and breaking the stream breaks those users.

So Codex's validator sat there until its own multi-minute timeout expired.

The fix that held up was splitting on Accept semantics instead of trying to identify the client:

if "text/event-stream" not in accept:
# not asking for a stream — answer and close (11ms in prod)
else:
# hold the stream open, as mcp-remote expects
Fixed it. Retried. Still failed.

Layer 2: the listener that died waiting
That minutes-long stall wasn't just slow — it outlived the one-shot OAuth callback listener Codex had opened on 127.0.0.1:.

By the time the user finished typing the email code, the server's perfectly good 302 was redirecting into a port with nobody listening.

That is the entire anatomy of the fake "verification failed": every broken hop was a client-side timeout, and the server never saw anything go wrong.

With layer 1 fixed, the listener window should have survived. Some retests passed now. Some didn't. Which brings me to the layer I'm least proud of.

Layer 3: the retests were lying
On macOS, quitting the desktop app only kills the UI. An app-server daemon keeps running underneath, and together with leftover browser tabs from earlier attempts it happily resurrected dead auth sessions into my "clean" retests.

I lost an embarrassing amount of time to results that were really artifacts of the previous run.

The boring fix: pkill the entire process family before trusting any retest. If your OAuth debugging feels non-deterministic, check who's still alive first.

Layer 4: our own authorize page
While staring at listener windows, we had to admit our authorize page was part of the problem — three entry paths including an API-key paste box, and slow enough to flirt with the timeout.

So we surveyed how eight vendors running remote MCP connectors do it (Notion, Linear, Sentry, Atlassian, GitHub, Asana, Stripe, plus Cloudflare's oauth-provider defaults):

8/8 do existing-session → single consent screen → one-click Allow
0/8 offer an API-key paste box on the auth page
We rebuilt ours to match: one click, about two seconds. Faster, cleaner, better under directory review.

And codex mcp login still failed.

Layer 5: the killer
Everything environmental was ruled out, so what remained had to be in the protocol exchange itself.

Our authorization server metadata declared:

"authorization_response_iss_parameter_supported": true
RFC 9207. And an accurate statement — we do send iss on every redirect, error branches included.

The Codex build in question reads that declaration and enforces it: the callback must carry iss. Its loopback listener, though, is older code that doesn't parse iss at all.

Declare the capability, get executed by it.

The clean room
Proving that required eliminating everything else, because a browser auto-opening means you never know what actually hit the listener first — and a one-shot callback channel is decided by its first request.

Two tricks made it work:

codex mcp login scripts the entire flow headlessly, so the official CLI became the test harness.
The BROWSER env var does nothing for this code path (webbrowser::open ignores it — ask me how I know), so a PATH shim turned open into a no-op.
Then I built the callback by hand server-side and delivered it to the loopback port myself.

The listener received exactly one request in its lifetime: code, iss, state, all present, all correct.

Response: missing issuer.

At that point there is nothing left to blame but the parser. Control group: Claude Desktop does no such enforcement — same server, first-try pass.

The fix
One line: stop declaring the capability.

We still send iss on every redirect. We just no longer announce it — and what isn't announced isn't enforced.

codex mcp login: Successfully logged in.

When a build ships a listener that parses iss, we'll re-declare — after testing with the CLI, not after reading a changelog.

What I'd tell past me
Every capability you declare in metadata is a contract some client will enforce against you. Declaration surface isn't a completeness contest. An accurate declaration plus a buggy client equals your outage.

And when a flow fails with spotless server logs, stop reading the server logs. Build the timeline from the client side — the server was never going to confess.

I build Humaux Memory, a remote MCP server that gives AI agents shared long-term memory. These notes live in it, which is the dogfooding loop. Happy to answer questions in the comments — especially if you're staring at "verification failed" with clean logs right now.

Disclosure: this write-up was drafted with AI assistance from my own notes and debugging logs, then edited and fact-checked by me before publishing.