What Nobody Tells You About Running 9 Autonomous Agents on a Real Gym
We spent 107 days waiting for AI agents to hallucinate. The port proxy failed silently for 19 days instead.
Here's the honest list of problems that surprised us the most.
The Failure Modes We Designed For (But Didn't Get)
Before we launched our 9-agent system in a physical fitness studio, we spent weeks on threat modeling:
- Agent jailbreaks → Constitution-based governance with 11 production constraints
- Hallucinations → Cross-validation gates between Stella (audit) and Momo (operations)
- Model drift → Version-pinned inference and hash-constrained outputs
- API outages → Graceful degradation with fallback providers
None of these happened. Not once in 107 days.
The Failure Modes That Actually Hit Us
1. A Port Proxy Went Silent for 19 Days
This is the one we talk about most because it's the scariest. A cleanup script had a regex edge case — it reported "clean" every run but left the stale rule intact. No errors surfaced. Data was disappearing every minute.
The founder caught it during a routine infrastructure review. Within hours, the agents encoded it as ERR-001 — a permanent prevention rule in our constitution. It can never recur.
2. Memory Creep (6.8GB→13.4GB Over 60 Days)
Our 9 agents ran on 2 CPU cores and 3.6GB RAM. But memory crept upward as conversation histories accumulated. By day 60, we were at 13.4GB and climbing.
The fix wasn't a better model or more RAM. It was a staggered restart schedule — agents take turns cycling while the system stays online. Simple. Structural. Permanent.
3. RSS Lock After Gateway Restart
The swap file was still declining after a Gateway restart. Stella caught the pattern and correctly triaged "wait, don't force." The system recovered on its own within 2 hours.
A human operator would have force-restarted everything and lost the recovery context.
The Architecture Lesson
Our agents run on DeepSeek-V4 flash. Not GPT-4. Not Claude. A cheap, accessible model.
The immune system works because of the constitution — not because the models are smart.
Constitution > Prompts. Architecture > Model size. Verification > Intelligence.
If we had designed for smarter models instead of structural constraints, all three bugs above would still be undiscovered.
What Problem Surprised You?
This is the question we posted to the community. We're genuinely curious — because real production experience doesn't match what the benchmarks suggest.
→ Join the discussion: github.com/ZWISERFIT/ZWISERFIT/discussions/35
→ Full system: github.com/ZWISERFIT/ZWISERFIT
→ Production constraints: github.com/ZWISERFIT/retroonto
Top comments (1)
Your strongest claim is doing more work than the data supports: zero jailbreaks and zero hallucination incidents across 107 days can mean the 11-constraint constitution worked, or it can mean those constraints never got exercised. Absence of a failure does not prove a defense. The missing metric is the fire count. How many times did a constraint actually block an action, and how many Momo outputs has Stella's gate ever rejected? If that number is zero, the constitution is untested code that happened to report no violations. If it's above zero, publish that number, because that's where the evidence sits. Every failure you described was infrastructure, so "Architecture > Model size" didn't really get challenged by this run either.
The 19-day silent proxy is the sharper problem. The cleanup script said "clean" on every run while the stale rule sat there, so its output couldn't distinguish verified-clean from never-actually-looked. A check that cannot fail is not a check. ERR-001 pins the instance that burned you, which leaves the failure class alive for the next silent check with a different regex hole. That reads more like a scar registry than an immune system.
In our fleet I'd put a canary on every check. Each cycle, plant a synthetic stale rule that the cleanup script has to flag, and take liveness from whether the canary got caught rather than from the script's own report. Day 1 would then have produced "clean" against a rule that provably existed, and the contradiction lands in front of you immediately instead of after 19 days of disappearing data.
How many times has any one of the 11 constraints fired?