TL;DR: After a container rebuild, every signal was green — /health returned 200, the orchestrator reported the container healthy, exit code 0. Meanwhile the agent's memory backend was completely unreachable, and the agent kept chatting away with no memory at all. Nobody got paged. The rule that came out of it now lives in our runbook: after any rebuild or config change, verify with one real business call. A health check proves the process exists, not that the system works.
The green dashboard
The deploy looked perfect. The kind of perfect that makes you close the laptop.
-
/health→ 200 - Container status → healthy
- Process exit code → 0
- Logs → no errors, no warnings, nothing
If you had asked me that evening whether the system was up, I would have said yes without hesitation. Every check I had agreed, in advance, to trust — trusted it.
What I didn't know yet: the agent's memory backend was completely dead. Not degraded, not slow — dead. Every memory_search call was failing, and the agent was operating with no memory whatsoever.
The part that should scare you
Here's what makes this failure different from a crashed database or a 500ing API: an agent without memory doesn't crash. It keeps talking.
It answered questions. It was fluent, confident, helpful-sounding. It just didn't remember anything — not the decisions from last week, not the context from the previous session, not the things it had explicitly stored for exactly this purpose. It was, functionally, a very articulate amnesiac.
No alert fired. Why would it? From the monitoring's perspective, everything was nominal. The process was alive. The endpoint responded. The failure was entirely in the gap between "the process exists" and "the system works" — and none of my checks lived in that gap.
This is the failure mode I now lose sleep over, and it's specific to agents: silent amnesia. A traditional service fails loudly or not at all. An agent can fail in a way that looks exactly like success, because "sounding right" and "being right" diverged long before anyone noticed.
Why the health check lied
It didn't, really. That's the uncomfortable part. The health check told the exact truth about what it measured: the process was alive and could respond to HTTP.
The problem was what it didn't measure. The memory backend was a downstream dependency — a separate service the agent talked to over MCP — and the health check never exercised that path. It was a liveness check wearing a readiness costume.
This is the general shape of the bug, and I've since seen it twice more in different clothes:
- Gates that don't test the thing. In a later experiment, our env checks, mount checks, and verification gates were all green — while the workload was silently running on the old image. No gate checked whether the thing under test was even in the binary. Green gates, wrong artifact. Same disease.
- Alerts you agree to trust in advance. Every check on my dashboard was a check I had decided, months earlier, was "enough." The incident didn't break my checks. It walked through the exact hole my checks had always had.
The rule
The incident produced one rule, and it's now the first line of our deploy runbook:
After any container rebuild — and after any change to config or parameters — verify with one real business call. Treat the changed system as a brand-new system. Do not rely on alerts alone.
Concretely, our post-deploy smoke test is no longer a health ping. It's a real round-trip through the memory path: store a canary fact, then search for it. If the agent can't remember what it just learned, the deploy is red — no matter what /health says.
The principle generalizes: your smoke test should exercise the paths your product actually depends on, not the paths that are easy to check. For an agent, memory isn't a nice-to-have sidecar. It's load-bearing. A check that skips it isn't a check.
A checklist, if you run agents in production
-
Separate liveness from depth. Keep your cheap
/healthping, but add a deep check that exercises critical downstream dependencies — memory, tools, model endpoints. Run the deep check post-deploy, not just on a schedule. - Make the smoke test a real business call. Not "can it reach the port" but "can it do the job." Store-then-recall. Tool-call-then-verify. One real round trip beats ten synthetic probes.
- Treat every config change like a rebuild. The failure didn't come from new code. It came from a changed environment around the same code. New config = new system = full verification.
- Be suspicious of quiet agents. An agent that never errors is not necessarily a healthy agent. Build at least one check that would catch "working, but wrong" — especially around memory and tool use.
- Write the postmortem rule into the runbook, not just the doc. A lesson in a doc is a suggestion. A step in the deploy checklist is a gate.
The uncomfortable summary
The scariest failures aren't the ones that page you at 3 AM. They're the ones where every dashboard is green, the logs are clean, and your agent has been confidently forgetting everything for hours.
/health 200. Container healthy. Exit code 0. Memory gone.
Check the thing, not the signal.
Top comments (1)
Does the canary read run in a fresh agent context? I’d check that a backend response contains a newly generated random marker, so the agent cannot pass the check by repeating it from the conversation while the memory service is still down.