The latest articles on DEV Community by Gaurav Bomra (@gauravstack). https://dev.to/gauravstack https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4120435%2F1c67ed64-c25d-43da-ab82-44b97bfa8b5c.png https://dev.to/gauravstack en Gaurav Bomra Mon, 21 Sep 2026 10:02:50 +0000 https://dev.to/gauravstack/how-i-built-and-broke-a-production-multi-agent-ai-system-4b2i https://dev.to/gauravstack/how-i-built-and-broke-a-production-multi-agent-ai-system-4b2i <h1> How I Built (and Broke) a Production Multi-Agent AI System </h1> <p>Most agent demos work great — right up until you run them past the happy path. This is the story of building a real multi-agent orchestration system, and three bugs that taught me more than the initial build did.</p> <h2> What I built </h2> <p>A production-style agent system with:</p> <ul> <li>A <strong>supervisor agent</strong> that decomposes complex tasks and delegates to specialist agents (LangGraph state machine)</li> <li> <strong>Persistent memory</strong> — short-term working state in Redis, long-term semantic memory in ChromaDB</li> <li> <strong>Human-in-the-loop approval</strong> for sensitive actions, with a full review queue and UI</li> <li> <strong>Full observability</strong> — OpenTelemetry tracing on every agent call and tool invocation, cost tracking, and a replay engine</li> </ul> <p>The stack: Python/FastAPI, LangGraph, PostgreSQL, Redis, ChromaDB, Celery, OpenTelemetry, React.</p> <p>The architecture is the easy part to explain. The bugs are the part worth writing about.</p> <h2> Bug #1: The 5,000-row runaway loop </h2> <p>Early on, I noticed one test task had generated over 5,000 duplicate rows in the <code>review_requests</code> table. A single task. One escalation. Five thousand rows.</p> <p>The root cause: the human-in-the-loop escalation logic had no protection against a task being re-evaluated multiple times in quick succession — each re-evaluation created a new pending review request instead of checking if one already existed. Under certain retry/timing conditions, this spiraled.</p> <p>The fix had two layers:</p> <ol> <li>An application-level guard — check for an existing pending review before creating a new one</li> <li>A <strong>database-level partial unique index</strong> — <code>UNIQUE (task_id, subtask_id) WHERE status = 'pending'</code> </li> </ol> <p>The app-level check alone wasn't enough — there was still a race condition window between the check and the insert. The database constraint is what actually closed the gap. Lesson: if a bug can be prevented at the database layer, prevent it there. Application logic is not atomic by default.</p> <h2> Bug #2: The failure that never failed </h2> <p>While building the observability layer, I had a checkpointer initialization path with a fallback: if the primary checkpointer setup failed, the code would fall back to a sentinel value and continue running — silently, without persistence.</p> <p>This is worse than a crash. A crash tells you something is wrong. A silent fallback lets the whole system appear to work while quietly not doing the one thing (state persistence) the rest of the architecture depended on.</p> <p>The fix: removed the fallback entirely. If checkpointer init fails now, it raises a hard <code>RuntimeError</code> at startup, with a clear log line confirming success when it <em>does</em> work. No degraded-but-invisible states allowed.</p> <h2> Bug #3: Two identifiers that looked the same but weren't </h2> <p>The replay feature let you re-run a past task with a modified input, comparing the old and new outcomes side by side. It worked — until I actually verified <em>what</em> it was comparing.</p> <p>The frontend was sending an OpenTelemetry <code>span_id</code> to the replay endpoint, where a LangGraph <code>checkpoint_id</code> was expected. These are two unrelated identifier systems that happen to look like similar opaque strings. The backend's <code>load_checkpoint()</code> function had its own fallback (see a pattern here?) that would silently reconstruct approximate state from trace data instead of loading the real checkpoint when the ID didn't resolve.</p> <p>So replay was "working" — it just wasn't replaying from the real checkpoint most of the time.</p> <p>Fix: recorded the actual <code>checkpoint_id</code> as a span attribute at trace time, and had the frontend send <em>that</em> instead of the span's own id. I also removed the silent fallback in <code>load_checkpoint()</code> — now it raises an explicit error if the checkpoint can't be found, rather than quietly approximating. I verified the fix by checking for a <code>[PATH: NATIVE_LANGGRAPH_CHECKPOINT]</code> log line confirming the real path was actually being used.</p> <h2> The pattern across all three </h2> <p>Every one of these bugs shares something: <strong>a system that appeared to work while silently doing the wrong thing.</strong> Not crashes — those are easy to catch. Silent degradation is the dangerous failure mode, because your test suite can pass, your demo can look perfect, and the bug only surfaces under conditions you didn't think to simulate.</p> <p>If I had to generalize this into a rule for building agent systems: <strong>wherever your code has a fallback "just in case," ask whether that fallback should actually be a hard failure instead.</strong> A fallback that masks a real problem is worse than no fallback at all.</p> <h2> Where this ended up </h2> <p>I packaged the system (with all three fixes already in place) into a starter kit for developers building similar agent products, since most of the pain in this space isn't the "agent" logic — it's this orchestration/memory/observability layer that everyone ends up rebuilding. If you're curious: <a href="https://whop.com/gauravxd/multi-agent-orchestration-kit/" rel="noopener noreferrer">https://whop.com/gauravxd/multi-agent-orchestration-kit/</a></p> <p>But even if you never look at the kit, I hope the bugs are useful on their own. If you're building agent systems and want to compare notes on failure modes you've hit, I'd genuinely like to hear about them.</p> ai langgraph python webdev Gaurav Bomra Fri, 11 Sep 2026 07:31:17 +0000 https://dev.to/gauravstack/how-i-built-and-broke-and-fixed-a-production-multi-agent-ai-orchestration-system-264g https://dev.to/gauravstack/how-i-built-and-broke-and-fixed-a-production-multi-agent-ai-orchestration-system-264g <p>Why I built it — the gap between toy agent demos and production-ready systems<br> Architecture overview (LangGraph, memory, HITL, observability) — with the diagram<br> Deep-dive: the 5,000-row runaway loop bug — root cause, fix, the DB-level guard<br> Deep-dive: the silent checkpointer fallback — why silent failure is worse than a crash<br> Deep-dive: the replay identifier mismatch bug<br> What I'd do differently / lessons for anyone building agent systems<br> Closing: packaged it as a starter kit, link + PH link</p> <p>(This one's worth writing properly rather than templating — it's your best long-term SEO/credibility asset. Do this in the days after launch, not necessarily on launch day itself.)</p> ai programming productivity python