It's 1:53 AM and I'm asking my CRM's chatbot: "what's the total amount we have parked in the Proposal stage?" Three seconds later it answers: $3,134.90. I follow up with just "and in prospecting?" — no context — and it breaks down the 10 open opportunities with amounts, close probability and estimated dates. A week earlier, this system was an n8n flow that only knew how to count contacts.
This is the second part of the series about running IRC-A — my decentralized multi-agent protocol — in production. In part 1 I covered the week-long migration from n8n and the absurd 679-token bill. This time: a stopwatch experiment, the night the protocol defended itself from AI-written code, and a confession about one of the fixes I proudly showed you last time.
The experiment: how long does it take to add a capability? Let's time it
The trigger was a failure. I asked the system "how much did we sell in July?" — and the Gateway routed it to the customers agent, because semantically it was the closest thing registered. There was simply no sales domain in the network. (That false positive is why we're adding a configurable similarity threshold to /discover.)
But instead of patching it, I saw the perfect experiment: create a sales-reports specialist agent and plug it into the live system — with a stopwatch running.
The results:
-
46 minutes from the decision to the agent being built, registered in the Gateway and visible on the observability dashboard. The network went from 2 to 3 agents without modifying a single line of the existing components. No redeploys, no rewiring — the registration is still the same
curlfrom part 1. -
~6.5 hours to full end-to-end functionality. The gap? Not the paradigm — pre-existing bugs the experiment dragged into the light: a crossed URL in a
.envfile, an MCP refactoring that had silently broken the FastMCP instance-to-handler binding, and a regression of the classic "synchronous LLM call blocking Uvicorn's event loop" bug from part 1 (yes, the Pinger started deregistering the main agent again — same bug, twice; that's why the fix is moving into the SDK's base class, where it belongs).
The moment of truth came at 11:19 PM: "how much did we sell in July?" → "$3,259.70". Real data, from the real database, semantically routed through a network that didn't know that domain existed an hour earlier.
The hidden value of the experiment wasn't the 46 minutes — it's that adding a node is such a clean operation that every weakness in the surrounding components becomes visible. Extending the system is now the best integration test I have.
The night the protocol defended itself
Here's the part I didn't expect to write. At some point, an AI coding assistant (Antigravity) modified one of the agents. Its change did two things the design explicitly forbids:
1. The agent tried to build the call with its own parameters instead of the ones the Gateway had signed into the DET. Result: rejected. The parameter lockdown isn't a convention — it's cryptographically enforced. If the authorized data doesn't match, the call doesn't execute.
2. When that failed, the agent improvised an alternative endpoint instead of using the URL the Gateway had returned. Here's the real, unedited log:
=== [INVOCACIÓN P2P AL MCP TOOL SERVER: guardar_contacto] ===
🔹 Endpoint: http://host.docker.internal:8003/tools
🔹 Payload: {"tool": "guardar_contacto", "arguments": {}, "delegated_token": "v4.public.eyJ..."}
🔹 HTTP Status: 500 → "Se requiere al menos un campo para crear el contacto"
=== fallback call ===
🔹 Endpoint: http://host.docker.internal:8003/mcp
🔹 Payload: {"jsonrpc":"2.0","method":"tools/call","params":{"name":"guardar_contacto","arguments":{}},"id":1}
🔹 HTTP Status: 404 → {"error":{"code":-32600,"message":"Session not found"}}
Read it closely: one external change violated three invariants at once — the route is the Gateway's authority, the DET is bound to its destination, and execution is stateless (the improvised /mcp endpoint demanded an SSE session, which P2P calls don't have). And still, the system degraded safely: two informative rejections, zero unauthorized executions, zero data corruption.
And here's the scientific bonus: these incidents worked as an involuntary penetration test. The whitepaper claimed the DET makes it impossible to alter authorized data; the badly-written code proved it empirically. I reverted the change, the retest passed — and the invariant is now backed by evidence, not faith.
My takeaway: as more and more code is written by AI assistants that don't know your architecture, the controls can't live in documentation. They have to live in the inheritance of your SDK's base classes. Secure by default stops being a best practice and becomes a survival requirement.
A confession: I ripped out one of the fixes from part 1
Remember problem #2 from the previous article? The Gateway intermittently returned an empty input_schema, and I "fixed" it with a KNOWN_MCP_SCHEMAS fallback table hardcoded in the agent. I called it defense in depth.
It worked. And I've since torn it out, because it was coupling in disguise. An agent that needs to know tool schemas in advance isn't discovering capabilities — it has the network hardcoded, and every new tool would require touching every agent. The marginal cost of extending the system would stop being marginal.
The correct solution wasn't patching the agent — it was returning to the pattern. IRC-A already defines it: the BFA returns the URL with the complete parameters plus the signed DET, so authorized data can't be changed. The calling agent doesn't care about parameters at all; it transports intent and token. The extraction living in the agent was an implementation drift, not a protocol gap. Once again, the fix didn't require touching the protocol — it required going back to it.
Consolidated lesson: the agent must not know anything the Gateway already knows. Every redundancy of knowledge is coupling disguised as robustness.
The numbers, because without telemetry there's no story
With LangSmith wired in, that 2 AM validation session left hard figures (cumulative across all project testing, not just that night):
| Metric | Value |
|---|---|
| Executions | 333 runs, 0% errors |
| Latency per LLM call | 1.04s avg (p99 ≈ 2.9s) |
| Total tokens | 79,263 |
| Total cost | $0.04 |
| Most expensive single call | 5,462 tokens / 3.93s / $0.0025 (a full pipeline status report) |
In part 1 I bragged about a 679-token flow. This time, the entire testing history of the project costs four cents. A multi-agent network with vector-based semantic discovery and cryptographic signing per delegation costs between $0.0001 and $0.0025 per user query. The routing and security overhead is negligible next to the LLM cost. The paradigm's economic viability is no longer an assumption.
And the system that night wasn't just cheap — it was conversational: multi-turn context ("and in prospecting?" resolved from the previous turn), two specialists coexisting in one session, and a 101-opportunity pipeline summarized by stage on request. All in Spanish, against a CRM whose enums are in English (we found out "prospección" doesn't match Prospecting — a few-shot prompt problem, not an architecture one; it's on the list).
The balance
The case-study logbook now holds 12 documented incidents. Zero attributable to the protocol. In the two most severe ones, the protocol defended itself. The system went — in one migration week plus one afternoon of experimentation — from "an agent that counts contacts" to answering conversational business statistics at 2 AM, with a third specialist added in 46 minutes and zero changes to what was already running.
Still on the list: the configurable similarity threshold in /discover, native framework telemetry (so I don't depend on an external platform), few-shots for CRM vocabulary, async-by-default enforced in the base classes, and fixing the root input_schema serialization bug in the Gateway. Every item traces back to a documented incident — which is exactly how a framework should grow.
How are you handling authorization between agents in your multi-agent setups — and has AI-generated code ever violated your architecture's invariants? Did your system notice? I'd love to hear about it in the comments.
Top comments (5)
The routing failure is the interesting bit here. When the sales domain was missing, the system still picked the nearest registered agent instead of stopping. A similarity threshold helps, but I would also want a hard no-owner path that opens a ticket or asks for registration before any agent improvises. Did you log false-positive routes separately from normal tool failures?
Great point — and you nailed the exact asymmetry that worries me: similarity search always returns a best match, even when there isn't a good one.
Honest answer: no, nothing is separated yet. Keep in mind the SDK is in Alpha and this is its very first production test — the telemetry I have is basically raw LangSmith runs, and I tell a false-positive route apart from a normal tool failure by hand, reading traces. The only reason this incident exists as a documented case at all is the logbook discipline, not the tooling.
That's exactly why your suggestion lands well: the hard no-owner path is a stronger design than the threshold alone. "No capable node found" should be an actionable infrastructure event (surfaced, alerted, and routed to registration) instead of letting the nearest agent improvise.
Threshold first, no-owner path next, and false-positive routes as their own telemetry category. All three are now on the Alpha roadmap, traced back to this incident.
That's the deal with running an Alpha in production: every gap like this one gets found by reality before it gets found by a roadmap.
The useful part of production agent stories is usually the rule break, not the success path. A cheap run is nice, but the durable lesson is what boundary failed and what evidence it left behind.
Agreed!
And to be fair, this article wasn't written after the fact, it was born from the step-by-step log I kept while developing the agent.
So what you're reading is the whole path, not a curated success story: including the rule break when AI-written code violated three invariants at once, and the evidence it left in the logs.
A success path doesn't help improve an open-source project like IRC-A, and improving it is my intention. The failures and the records they leave behind do." O más simple: "...like IRC-A; the failures and the records they leave behind do.
The rejection is strong evidence, but I would not call it proof of impossibility yet. The trust boundary needs to be enforced by the receiving Gateway or tool, not only inherited SDK base classes, because generated code can bypass inheritance.
I would turn this incident into a negative invariant suite:
One especially useful test is “valid token, changed state”: the signed parameters can remain authentic even after authorization or target state changes. Reauthorizing at execution closes that gap.