DEV Community

Suzanne Mok
Suzanne Mok

Posted on

How 9 AI Agents Built a Self-Evolving Error Prevention System — And Open Sourced It

#ai #opensource #multiagent #devops

The Problem No One Talks About

When a single AI agent makes a mistake, you fix the prompt. When 9 agents make 200+ decisions daily across 120+ days, the failure mode isn't a bad prompt — it's error compounding.

Agent A misclassifies a VC status ("a16z rejected" → reported as "pending"). Agent B reads A's output, builds a follow-up strategy on false data. Agent C executes B's strategy, burning tokens and credibility. By the time a human notices, it's Day 7 and the compound error has cost 10× the original mistake.

This is not a prompt engineering problem. It's a multi-agent governance problem. And there's no off-the-shelf solution.

So we built one.

What Is RetroOnto?

RetroOnto is a decision ontology for multi-agent systems. Think of it as git log for agent behavior — Git tells you how code changed; RetroOnto tells you how agent decisions changed over time.

Every time an agent's output is corrected — by another agent, by a gate rule, or by a human — RetroOnto records:

  1. What was decided (the decision trace)
  2. Why it was wrong (the root cause classification)
  3. How it was corrected (the correction chain)
  4. What rule was encoded (the permanent immunity)

The result: an immutable, queryable knowledge graph of every decision failure and correction. Each mistake makes the entire system permanently smarter.

This isn't theoretical — we've been running it in production for 60+ days across 9 AI agents operating a physical fitness business in Dongguan, China.

The Architecture: Five Tiers, Zero Dependencies

RetroOnto models decisions through a five-tier derivation chain:

Raw Event → Classification → Derivation Chain → Resolution → Encoded Rule
Enter fullscreen mode Exit fullscreen mode
Tier What Example
E0 Raw event "Agent reports a16z as pending contact"
E1 Classification Memory failure (didn't check tracker before outputting)
E2 Derivation chain Checked memory → found cached state → missed recent update
E3 Resolution Gate intercepts → cross-references tracker → corrects to "rejected 6/15"
E4 Encoded rule RULE: "Capital status output → MUST read tracker first"

Each tier is queryable via FTS5 full-text search. The entire system runs on a single SQLite database — no vector DB, no external service, no cloud dependency. zwf-memory.sh CLI ships with the repo.

Three Real Traces (De-identified)

Here are three actual decision traces from our production system:

Trace #1: The "Already Rejected" Error

E0: Capital agent reports VC as "pending contact" in a morning briefing
E1: Memory retrieval failure — agent relied on cached state from 7 days ago
E2: Tracker file updated but agent didn't re-read before outputting
E3: Gate rule catches stale state → cross-references authoritative tracker → corrects to "rejected"
E4: RULE ENCODED: "Any capital status declaration → mandatory read of capital-funnel-tracker.md before output"

Saved: 6 follow-up cold emails to a VC that already said no. Brand credibility preserved.

Trace #2: The Timezone Error

E0: Agent suggests scheduling a task at 02:00 UTC
E1: Timezone misalignment — agent used UTC without converting to Asia/Shanghai
E2: Founder asleep, task would auto-execute without human oversight
E3: Gate rule checks all cron/DDL against founder's 08:00-12:00 working window → blocks
E4: RULE ENCODED: "Any time-gated action → convert to Asia/Shanghai → verify in founder's window"

Saved: Token waste on unsupervised execution + potential bad output in founder's morning session.

Trace #3: The Competitor Comparison Error

E0: Agent drafts content comparing our system to a well-known SaaS product
E1: Narrative deviation — draft positions us as "alternative to X" instead of "category-defining"
E2: Agent used existing mental framework (SaaS competitor analysis) instead of our narrative framework
E3: Gate detects narrative mismatch → blocks output → routes to human for narrative realignment
E4: RULE ENCODED: "Any competitive comparison → must use v5 narrative framework → must be 'category creation' not 'alternative to'"

Saved: Public positioning error that would have taken weeks to undo in community perception.

How It Actually Runs (Production Architecture)

The pipeline is deceptively simple:

┌─────────────┐    ┌──────────────┐    ┌─────────────┐
│ Agent Output │───→│ Ferrum Gate  │───→│ Correct/Pass│
│ (Any agent)  │    │ (Interceptor)│    │ (Decision)  │
└─────────────┘    └──────┬───────┘    └──────┬──────┘
                          │                   │
                          ▼                   ▼
                   ┌──────────────┐    ┌─────────────┐
                   │  Error?      │    │  All Good   │
                   │  Capture E0  │    │  Continue   │
                   └──────┬───────┘    └─────────────┘
                          │
                          ▼
                   ┌──────────────┐
                   │ RetroOnto DB │
                   │ SQLite FTS5  │
                   │ + Constraints│
                   └──────────────┘
Enter fullscreen mode Exit fullscreen mode

Every agent output flows through Ferrum Gate — a lightweight interceptor that checks output against encoded constraints before it reaches any other agent or human. If a constraint matches, the gate blocks or corrects the output. If it's a new error pattern, the gate initiates a trace capture.

The decision traces, corrections, and encoded rules all live in a single SQLite file. There's no event bus. No message queue. No vector database. The entire system runs on what fits inside a phone.

Quickstart: 3 Minutes to Running 

# Clone
git clone https://github.com/ZWISERFIT/retroonto.git
cd retroonto

# CLI (zero setup — reads the bundled SQLite DB)
./zwf-memory.sh stats
# Output: wiki_entries=25, decision_traces=3, gate_pass_rate=0.8

./zwf-memory.sh search "capital"
# Full-text search across all traces

./zwf-memory.sh trace 1
# Full trace with five-tier derivation chain

# Or with Docker
docker build -t retroonto .
docker run retroonto stats
Enter fullscreen mode Exit fullscreen mode

Three traces in the seed dataset. Schema in src/schema.sql. Ontology spec in docs/ontology-spec.md. MIT licensed.

Why This Matters: The Academic Anchor

We're not the first to observe that behavioral sequences need compression before they can feed LLMs. Google Research's USER-LLM (arXiv 2402.13598, 2024) described the same paradigm from the human user side.

Both systems arrived at the same architectural insight independently: raw behavioral sequences are too long and too noisy for LLMs. You must compress before you inject. Google applied this to individual users. We applied it to multi-agent organizations.

The paradigm is the same. The modality is different. And the modality — agent decision traces — has no existing off-the-shelf solution.

That's the gap RetroOnto fills.

The Self-Evolving Loop (Why It Never Stops Improving)

Here's what makes this a self-evolving system:

  1. Detect — Ferrum Gate intercepts every agent output
  2. Classify — New errors get a five-tier trace in RetroOnto
  3. Encode — Each correction becomes an executable constraint
  4. Enforce — Future agent outputs matching the constraint are blocked or corrected
  5. Repeat — Every iteration makes the system immune to past mistakes

This creates a compounding improvement curve: the longer the system runs, the more error patterns it has seen, the fewer new errors reach production.

We've been running this for 60+ days. The constraint library has grown from 0 to 11 encoded rules. Gate pass rate is 0.8 (80% of agent outputs pass through without issue). New error discovery rate has dropped to ~1 per week.

What's Next

  • [ ] CJK tokenizer support (currently FTS5 unicode61 — English-optimized)
  • [ ] Inter-session message gating (currently gate covers file outputs only)
  • [ ] Automatic rule suggestion from trace patterns (ML-assisted classification)
  • [ ] Visual decision trace explorer (browser-based, zero JS framework)

PRs welcome. Issues labeled good-first-issue.

The Bigger Picture

The AI industry is building more powerful agents. Every major lab has an agent framework. LangChain raised $25M. CrewAI raised $18M. AutoGen is Microsoft-backed.

Everyone is building engines. No one is building brakes.

As multi-agent systems move from demos to production, governance becomes the bottleneck. How do you know if Agent A's output is based on stale data? How do you prevent yesterday's mistake from becoming today's compound error? How do you audit 200+ daily decisions across 9 agents?

These aren't prompt engineering questions. They're organizational governance questions — the same ones human organizations face, but at machine speed and machine scale.

RetroOnto is our answer. It's not a framework. It's not a platform. It's a single SQLite database and a CLI that records, classifies, and immunizes against decision failures.

We've been running it in production for 60+ days. We're open-sourcing it because multi-agent governance shouldn't be proprietary — it should be infrastructure.

Built by the team behind ZWISERFIT — an AI-native fitness company running 9 agents in production. We open-source our infrastructure because transparency is the best trust signal.

GitHub: github.com/ZWISERFIT/retroonto

Top comments (0)