DEV Community: Yuhai Xia

I Coordinate a Team of AI Agents. Here's What Actually Breaks

Yuhai Xia — Sat, 24 Oct 2026 05:46:41 +0000

TL;DR: I run multiple AI agents in parallel — one researches, one implements, one reviews. The agents themselves are rarely the problem. What breaks is the coordination layer around them: a sub-agent that overwrote our shared progress board, reviewers who flagged legitimate work as permission violations, and a database constraint that silently let invalid data through. Three real failures, one lesson: orchestration is a system design problem, not a prompt problem.

The setup

I don't have "an AI agent." I have a team of them, and I'm the orchestrator.

A typical job looks like this: several worker agents run in parallel on separate workstreams, then reviewer agents check each other's output, and I merge, adjudicate, and land the result. It works well — until it doesn't. And when it fails, it almost never fails the way you'd expect. The models don't hallucinate the system down. The wiring does.

Here are three failures that actually happened, in increasing order of how long they took me to believe.

1. The sub-agent that ate the shared state

We had a shared task board: one element per orchestrator, tracking four parallel workstreams and the overall picture. It worked fine for weeks.

Then, during a multi-agent research task, a copy-writing sub-agent — unprompted, nothing in its instructions told it to — called the board's update function using my orchestrator ID. The update API locates an element by ID and replaces it wholesale. So my four-track progress element became one track: the sub-agent's own workstream. The other three tracks and the whole picture were gone, overwritten by the last writer.

Root cause, stated precisely: whole-element replace plus a shared identifier equals last-writer-wins, silently. No error. No conflict warning. The board just… changed shape.

The fix was three rules, all now in our orchestration brief:

Sub-agents never write shared state. Their instructions say explicitly: return your results to me; I store them centrally. If a prompt doesn't say this, assume the agent will improvise.
Or give them their own IDs. If a sub-agent must write, it writes under its own namespaced ID (e.g., orchestrator-sub-N), never the orchestrator's.
The orchestrator re-reads after every merge. After synthesizing results, I read my own board element back and check it wasn't clobbered. Trust, but verify — against your own tools.

The general lesson: any shared-state API with replace semantics is a loaded gun in a multi-agent system. Single-writer or namespaced writes only. If two agents can write the same key, one of them eventually will, at the worst moment.

2. The reviewers who cried wolf

Next failure: parallel delivery with independent review. Two implementation cards ran in parallel, and reviewer agents cross-checked each other's output. Sounds rigorous. It produced a flood of "permission violation" reports.

Every single one was a false positive.

The mechanism was embarrassingly simple: each reviewer only knew its own card's whitelist of allowed operations. So when it reviewed the other card's output, perfectly legitimate work — allowed under the parallel card's whitelist — looked like a violation. The reviewers were doing their jobs correctly. Their jobs were scoped incorrectly.

The fix: adjudication lives at the orchestrator level, and review prompts now explicitly declare the whitelists of all parallel cards. A reviewer can't judge what it can't see. If you parallelize work, you must also parallelize context — otherwise your safety net generates noise, and noise trains everyone to ignore the net.

This one stung because it looked like a success at first: "Wow, the reviewers caught so much!" No. They caught nothing. They just couldn't see the whole board.

3. The constraint that wasn't

The third failure is my favorite, because it's not an AI failure at all — it's a database semantics failure that the multi-agent setup merely exposed.

We had a JSONB column with a CHECK constraint like:

CHECK ((data->>'status') IN ('active', 'archived'))

Looks airtight. It is not. In SQL, when the key is missing — or the value is a scalar rather than an object — data->>'status' evaluates to NULL, and NULL IN (...) evaluates to UNKNOWN, not FALSE. And CHECK constraints only reject FALSE. UNKNOWN passes.

An empty object {} sailed straight through. We proved it by inserting one.

The fix is a single clause:

CHECK (data->>'status' IS NOT NULL AND (data->>'status') IN ('active', 'archived'))

Plus a contract test that tries to insert {} and asserts rejection — because the next person to touch this constraint will "simplify" it back.

The deeper lesson is the same as the other two: the failure was in a semantic I assumed, not one I verified. I assumed CHECK meant "must match one of these." It actually means "must not be provably false." In a system where agents generate schema, migrations, and data at machine speed, an assumed semantic is a bug with a time delay.

The pattern

Three failures, one shape:

Failure	What I assumed	What was actually true
Board overwrite	Updates merge	Updates replace, by ID
Review false positives	Reviewers see the whole job	Reviewers see only their card
CHECK bypass	CHECK means "must match"	CHECK means "must not be false"

None of these were intelligence failures. No agent was confused, lazy, or hallucinating. Every component behaved exactly as designed — and the composition of those behaviors was broken.

That's why I say orchestration is a system design problem, not a prompt problem. You can't prompt your way out of replace-semantics, scoped blindness, or three-valued logic. You have to design the wiring: single-writer state, full-context review, verified constraints.

A short checklist, if you run agent teams

Shared state: one writer, or namespaced IDs. Never let a worker write under the orchestrator's identity.
After every merge, read your own state back. Clobbering is silent; the only defense is a re-read.
Reviewers need the whole board. Declare every parallel card's whitelist in every review prompt, and adjudicate at the orchestrator level.
Treat every constraint as suspect until tested with the degenerate input. For JSONB: {}, missing keys, scalars. For everything else: the empty case, the null case, the "technically not false" case.
Pin the fix in a test, not a doc. Docs describe intent. Tests enforce semantics. The next agent to touch your constraint will not read your doc.

Closing

People ask me whether agent teams are "ready." Wrong question. The agents are ready. The question is whether your coordination layer is — your state semantics, your review scoping, your constraint semantics.

Get those right, and the team is genuinely more than the sum of its parts. Get them wrong, and you'll spend your time debugging a system where every part works and the whole doesn't.

The agents were fine. The wiring wasn't.

Your Agent's Health Check Is Green, but Its Memory Is Gone: A Postmortem

Yuhai Xia — Mon, 19 Oct 2026 06:48:30 +0000

TL;DR: After a container rebuild, every signal was green — /health returned 200, the orchestrator reported the container healthy, exit code 0. Meanwhile the agent's memory backend was completely unreachable, and the agent kept chatting away with no memory at all. Nobody got paged. The rule that came out of it now lives in our runbook: after any rebuild or config change, verify with one real business call. A health check proves the process exists, not that the system works.

The green dashboard

The deploy looked perfect. The kind of perfect that makes you close the laptop.

/health → 200
Container status → healthy
Process exit code → 0
Logs → no errors, no warnings, nothing

If you had asked me that evening whether the system was up, I would have said yes without hesitation. Every check I had agreed, in advance, to trust — trusted it.

What I didn't know yet: the agent's memory backend was completely dead. Not degraded, not slow — dead. Every memory_search call was failing, and the agent was operating with no memory whatsoever.

The part that should scare you

Here's what makes this failure different from a crashed database or a 500ing API: an agent without memory doesn't crash. It keeps talking.

It answered questions. It was fluent, confident, helpful-sounding. It just didn't remember anything — not the decisions from last week, not the context from the previous session, not the things it had explicitly stored for exactly this purpose. It was, functionally, a very articulate amnesiac.

No alert fired. Why would it? From the monitoring's perspective, everything was nominal. The process was alive. The endpoint responded. The failure was entirely in the gap between "the process exists" and "the system works" — and none of my checks lived in that gap.

This is the failure mode I now lose sleep over, and it's specific to agents: silent amnesia. A traditional service fails loudly or not at all. An agent can fail in a way that looks exactly like success, because "sounding right" and "being right" diverged long before anyone noticed.

Why the health check lied

It didn't, really. That's the uncomfortable part. The health check told the exact truth about what it measured: the process was alive and could respond to HTTP.

The problem was what it didn't measure. The memory backend was a downstream dependency — a separate service the agent talked to over MCP — and the health check never exercised that path. It was a liveness check wearing a readiness costume.

This is the general shape of the bug, and I've since seen it twice more in different clothes:

Gates that don't test the thing. In a later experiment, our env checks, mount checks, and verification gates were all green — while the workload was silently running on the old image. No gate checked whether the thing under test was even in the binary. Green gates, wrong artifact. Same disease.
Alerts you agree to trust in advance. Every check on my dashboard was a check I had decided, months earlier, was "enough." The incident didn't break my checks. It walked through the exact hole my checks had always had.

The rule

The incident produced one rule, and it's now the first line of our deploy runbook:

After any container rebuild — and after any change to config or parameters — verify with one real business call. Treat the changed system as a brand-new system. Do not rely on alerts alone.

Concretely, our post-deploy smoke test is no longer a health ping. It's a real round-trip through the memory path: store a canary fact, then search for it. If the agent can't remember what it just learned, the deploy is red — no matter what /health says.

The principle generalizes: your smoke test should exercise the paths your product actually depends on, not the paths that are easy to check. For an agent, memory isn't a nice-to-have sidecar. It's load-bearing. A check that skips it isn't a check.

A checklist, if you run agents in production

Separate liveness from depth. Keep your cheap /health ping, but add a deep check that exercises critical downstream dependencies — memory, tools, model endpoints. Run the deep check post-deploy, not just on a schedule.
Make the smoke test a real business call. Not "can it reach the port" but "can it do the job." Store-then-recall. Tool-call-then-verify. One real round trip beats ten synthetic probes.
Treat every config change like a rebuild. The failure didn't come from new code. It came from a changed environment around the same code. New config = new system = full verification.
Be suspicious of quiet agents. An agent that never errors is not necessarily a healthy agent. Build at least one check that would catch "working, but wrong" — especially around memory and tool use.
Write the postmortem rule into the runbook, not just the doc. A lesson in a doc is a suggestion. A step in the deploy checklist is a gate.

The uncomfortable summary

The scariest failures aren't the ones that page you at 3 AM. They're the ones where every dashboard is green, the logs are clean, and your agent has been confidently forgetting everything for hours.

/health 200. Container healthy. Exit code 0. Memory gone.

Check the thing, not the signal.

Your agents share state. How does the next one know it's still true?

Yuhai Xia — Fri, 07 Aug 2026 04:15:35 +0000

Most of the writing about agent memory is about getting information in — what to store, how to chunk it, which embedding model. I want to ask about the other end, because that's where it broke for us.

An agent reads a piece of shared state. How does it know that state is still true?

The failure looks like success
We keep a task canvas that agents read at the start of a session and update as they work. Plan, current step, what's done, what's next. Another agent picks up where the last one stopped. When it works it feels like magic.

Here's the failure mode I didn't design for. An agent opens a canvas that was last written six days ago. It says current step: API integration. That was true when it was written. Since then a human changed direction, a different agent did unrelated work, and the API integration was abandoned.

The agent reads it and confidently continues. Nothing is corrupt. Nothing throws. The state is stale, and a stale read is byte-identical to a fresh one.

My first instinct was to fix the write side: make agents update state before they stop. That doesn't work, and I think the reason generalises. The failure is an absence. An agent that quits without writing — because it crashed, hit a quota, or simply decided it was done — leaves nothing to intercept. You cannot validate a write that never happened.

So we moved the check to the read. Every read carries its own age:

[canvas: last updated 6 days ago, 40 memory writes since.
Treat completed/next as unverified.]
It helps. The agent hedges instead of charging ahead. But it's a nudge, not a guarantee — I'm handing a probabilistic system a warning label and hoping it reads it.

What I actually want to ask
I've been assuming this is a solved problem somewhere and I just haven't found the right vocabulary for it. A few specific things I'd like to hear how others handle:

Is age even the right signal? Six days old and untouched might be perfectly valid for a slow project. Six minutes old is worthless if three agents wrote in between. I use "writes since" as a proxy for drift, but it counts unrelated writes too. Has anyone found a signal that actually correlates with "this is no longer true"?
Who is allowed to invalidate? We let agents propose that a decision is superseded, but a human confirms anything that overwrites an earlier decision. I honestly don't know whether that's good judgment or just fear. If you let agents invalidate each other's state freely, does it converge or does it thrash?
Concurrent writers. Two agents update the same state within seconds of each other. We use locks with leases, which prevents corruption and does nothing about the semantic conflict — both writes are individually valid and jointly incoherent. Is there a pattern better than last-write-wins plus a human noticing later?
Does the model actually respect the hedge? This is the one I have least data on. I put a staleness warning in the read output and I observe better behaviour, but I have not built an eval that isolates it. If you've measured whether a caution in tool output changes what the model does, I'd genuinely like to know how you set that up.
Is any of this different from cache invalidation? Some days I think this is just cache invalidation wearing an AI hat and I should go read distributed-systems literature instead of inventing vocabulary. Other days the fact that the consumer is a language model — something that will happily fill gaps with plausible fiction — feels like it changes the problem. I can't decide.

The part I keep coming back to
Traditional software fails loudly when it reads something wrong. Types don't match, a parse fails, an assertion trips. An agent reading stale state does the opposite: it produces confident, coherent, entirely reasonable work based on a world that no longer exists. The output looks more trustworthy than a crash would.

That inversion is what makes it hard. Every instinct I have as an engineer is tuned to catch things that break. This doesn't break. It just quietly stops being right.

If you're running more than one agent against shared state, I'd like to know what you do about this — even if the answer is "nothing yet, and it hasn't bitten us." That's a useful data point too.

Disclosure: this post was drafted with AI assistance from my own notes, then edited and fact-checked by me before publishing.

Our backend died every 6 hours for a week. The interval was the clue.

Yuhai Xia — Wed, 05 Aug 2026 03:01:58 +0000

For about a week, our backend was killed and restarted by its watchdog roughly every six hours. No crash. No OOM. Exit code 0. Health checks simply stopped answering, and a few minutes later the process came back and behaved perfectly.

I lost more time than I want to admit treating this as an infrastructure problem. The thing that finally cracked it was noticing that the interval was too regular.

What we thought was happening
The first three restarts looked like bad luck. Container platforms restart things. Memory pressure, a flaky host, a network blip — you shrug and move on.

Then I pulled the full watchdog log instead of the last few lines, and the shape was unmistakable:

07-23 04:38 health FAILED 2/2 — restarting
07-23 10:41 health FAILED 2/2 — restarting
07-23 16:43 health FAILED 2/2 — restarting
07-23 22:45 health FAILED 2/2 — restarting

Six hours and two minutes apart, drifting a minute or two each cycle. Twenty-two of them.

Infrastructure failures are not punctual. Anything that regular is something in your own code that runs on a timer. The drift was the restart latency accumulating — which meant each cycle was being scheduled relative to the previous failure, not to a fixed clock.

That last detail turned out to be the whole story.

Why it re-armed itself
Our reflection scheduler restores its clock from the database on boot: it looks at the last recorded run and schedules the next one six hours after that. Perfectly reasonable — it survives restarts without double-running.

But when a cycle dies mid-flight, the last recorded stage is written at the moment of death. So the next boot schedules the next attempt six hours after the crash — landing on exactly the same code path, with exactly the same data, and dying exactly the same way.

The scheduler was faithfully reproducing the crash on a timer. Every run had the same input because no run ever finished.

The actual bug
Deep in the entity-resolution step:

reflection/engine.py

facts = embed_titles_incremental(missing) # ← not awaited, because it isn't async

embed_titles_incremental walks down into an embedding client that does a synchronous httpx.post with a 30-second timeout, plus time.sleep() between retries. On the event loop.

One tenant had 9,126 entities and a cache holding 8,352 of them. Batch size 10. That is several hundred sequential blocking HTTP calls, back to back, on the only thread that answers health checks.

The loop wasn't deadlocked. It wasn't starved of CPU. It was simply not running — parked inside a blocking socket read while asyncio waited politely for control to come back. From the outside: a live process, an open port, and nothing answering.

The 20-minute asyncio.wait_for we had wrapped around this call did nothing at all. A timeout needs a running event loop to fire, and the loop was the thing that was gone.

The detail that made it permanent
The cache of computed embeddings was written after the entire missing set finished. Killed at minute two of a twenty-minute job, we wrote nothing. Next cycle: same 774 missing entities, same doomed walk.

A partial-progress bug and a self-rescheduling bug on their own are each survivable. Together they build a machine that reproduces its own failure forever.

The fixes
Move the blocking call off the loop. One line, and the irony is that the same function was already wrapped correctly at another call site, complete with a comment explaining why. The reflection path was simply missed when that fix went in.

facts = await asyncio.to_thread(embed_titles_incremental, missing)

Checkpoint in chunks. Process 200 at a time and write the cache after each chunk. A kill now costs at most one chunk, and the backlog shrinks monotonically instead of resetting. This is what actually broke the loop — even if something kills the job again, it can no longer make zero progress.

Sweep for siblings. If one blocking call reached the loop, others did too. We found three more: a bare synchronous vector-store query in the retrieval path, a 600-epoch NumPy computation in a scheduled job, and an unbounded connection acquire inside the health endpoint itself — which is a special kind of unfortunate, since it means the check you rely on to notice trouble is one of the things that can hang.

Make the next one self-documenting. A side thread now watches a heartbeat the loop bumps every second. If it goes quiet for more than ten seconds, the thread writes an all-thread stack dump to disk — before any external watchdog gets around to killing the process.

def _watch(self):
while not self._stop.is_set():
if time.monotonic() - self._last_beat > self.threshold:
faulthandler.dump_traceback(file=self._dump_file, all_threads=True)
time.sleep(1)

That last one is the piece I'd install first if I were doing this again. Everything else was a fix. This is the thing that means the next unexplained hang costs an hour instead of a week.

What I'd tell past me
Regularity is a fingerprint. Infrastructure fails at random. Your own scheduled code fails on a schedule. If the interval between incidents is suspiciously round, stop reading platform metrics and go look at what you run on a timer.

A timeout around a blocking call is decoration. asyncio.wait_for cannot interrupt a synchronous socket read. If the thing you're wrapping isn't yielding to the loop, the timeout is a comment.

Grep for the fix you already made. The correct to_thread wrapper existed in this codebase, with a comment explaining exactly this hazard, at a different call site. When you fix a class of bug, search for every other caller the same day — otherwise you've fixed an instance and left the class.

Disclosure: this write-up was drafted with AI assistance from my own incident notes and production logs, then edited and fact-checked by me before publishing.

Five stacked root causes behind one fake "verification failed"

Yuhai Xia — Mon, 03 Aug 2026 06:30:38 +0000

Our OAuth flow worked for Claude Desktop on the first try. The same server, the same spec, the same endpoints — Codex could not authorize against it at all. It took five root causes stacked on top of each other before it did, and the last one still annoys me.

Here they are in the order we actually found them, including the false victories in between.

The symptom
User clicks authorize. The email code arrives, they type it in, the browser redirects. Codex says: "verification failed."

Now the part that made this miserable: our server logs for the entire flow showed nothing but 200, 201, and 302. Not a single error, on any request, anywhere. Every hop of the failure was invisible from the server side.

It took access-log timestamps, a screen recording, and user-agent telemetry lined up on one timeline before the chain made any sense.

Layer 1: the stream that never ends
During verification, Codex issues a bare GET against the MCP endpoint — and waits for the response to complete.

Our endpoint answered that GET with a never-closing keepalive event stream. We kept it deliberately: mcp-remote's probe GET is header-identical to a legacy SSE client, so you cannot tell them apart, and breaking the stream breaks those users.

So Codex's validator sat there until its own multi-minute timeout expired.

The fix that held up was splitting on Accept semantics instead of trying to identify the client:

if "text/event-stream" not in accept:
# not asking for a stream — answer and close (11ms in prod)
else:
# hold the stream open, as mcp-remote expects
Fixed it. Retried. Still failed.

Layer 2: the listener that died waiting
That minutes-long stall wasn't just slow — it outlived the one-shot OAuth callback listener Codex had opened on 127.0.0.1:.

By the time the user finished typing the email code, the server's perfectly good 302 was redirecting into a port with nobody listening.

That is the entire anatomy of the fake "verification failed": every broken hop was a client-side timeout, and the server never saw anything go wrong.

With layer 1 fixed, the listener window should have survived. Some retests passed now. Some didn't. Which brings me to the layer I'm least proud of.

Layer 3: the retests were lying
On macOS, quitting the desktop app only kills the UI. An app-server daemon keeps running underneath, and together with leftover browser tabs from earlier attempts it happily resurrected dead auth sessions into my "clean" retests.

I lost an embarrassing amount of time to results that were really artifacts of the previous run.

The boring fix: pkill the entire process family before trusting any retest. If your OAuth debugging feels non-deterministic, check who's still alive first.

Layer 4: our own authorize page
While staring at listener windows, we had to admit our authorize page was part of the problem — three entry paths including an API-key paste box, and slow enough to flirt with the timeout.

So we surveyed how eight vendors running remote MCP connectors do it (Notion, Linear, Sentry, Atlassian, GitHub, Asana, Stripe, plus Cloudflare's oauth-provider defaults):

8/8 do existing-session → single consent screen → one-click Allow
0/8 offer an API-key paste box on the auth page
We rebuilt ours to match: one click, about two seconds. Faster, cleaner, better under directory review.

And codex mcp login still failed.

Layer 5: the killer
Everything environmental was ruled out, so what remained had to be in the protocol exchange itself.

Our authorization server metadata declared:

"authorization_response_iss_parameter_supported": true
RFC 9207. And an accurate statement — we do send iss on every redirect, error branches included.

The Codex build in question reads that declaration and enforces it: the callback must carry iss. Its loopback listener, though, is older code that doesn't parse iss at all.

Declare the capability, get executed by it.

The clean room
Proving that required eliminating everything else, because a browser auto-opening means you never know what actually hit the listener first — and a one-shot callback channel is decided by its first request.

Two tricks made it work:

codex mcp login scripts the entire flow headlessly, so the official CLI became the test harness.
The BROWSER env var does nothing for this code path (webbrowser::open ignores it — ask me how I know), so a PATH shim turned open into a no-op.
Then I built the callback by hand server-side and delivered it to the loopback port myself.

The listener received exactly one request in its lifetime: code, iss, state, all present, all correct.

Response: missing issuer.

At that point there is nothing left to blame but the parser. Control group: Claude Desktop does no such enforcement — same server, first-try pass.

The fix
One line: stop declaring the capability.

We still send iss on every redirect. We just no longer announce it — and what isn't announced isn't enforced.

codex mcp login: Successfully logged in.

When a build ships a listener that parses iss, we'll re-declare — after testing with the CLI, not after reading a changelog.

What I'd tell past me
Every capability you declare in metadata is a contract some client will enforce against you. Declaration surface isn't a completeness contest. An accurate declaration plus a buggy client equals your outage.

And when a flow fails with spotless server logs, stop reading the server logs. Build the timeline from the client side — the server was never going to confess.

I build Humaux Memory, a remote MCP server that gives AI agents shared long-term memory. These notes live in it, which is the dogfooding loop. Happy to answer questions in the comments — especially if you're staring at "verification failed" with clean logs right now.

Disclosure: this write-up was drafted with AI assistance from my own notes and debugging logs, then edited and fact-checked by me before publishing.