What We Learned Running 9 AI Agents in Production: Architecture Patterns That Actually Work

#ai #agents #startup #opensource

What We Learned Running 9 AI Agents in Production: Architecture Patterns That Actually Work

We run 9 autonomous AI agents on 2 CPU cores and 3.6GB RAM. They operate a real gym in Dongguan, China. 99% of daily operations run autonomously.

After 3.5 months of production operations, here are the architecture patterns that survived — and the ones that didn't.

Pattern 1: No Central Orchestrator

Every startup AI demo shows a central orchestrator routing tasks to specialized sub-agents. That pattern failed us in week one.

The problem: A central orchestrator becomes a bottleneck and a single point of failure. When it goes down, every agent goes blind.

What we built instead: A constitutional system. Each agent has permanent rules (its SOUL.md + SKILL.md files) that define its scope, boundaries, and escalation paths. Agents self-schedule via cron and self-coordinate via a shared agent-bus.

┌─────────────────────────────────────────────┐
│            Agent-Bus (Pub/Sub)              │
│  Post events → Agents listen & react        │
├─────────────────────────────────────────────┤
│  Baron: Brand Content                        │
│  Stella: Audit & Quality                     │
│  Zeus: Capital Strategy                      │
│  Luna: Community Operations                  │
│  Tristan: Infrastructure                     │
│  Ethan: Data Integrity                       │
│  Shuyu: Operations Command                   │
│  Melody: Metabolic Coaching                  │
│  Momo: Store Operations                      │
└─────────────────────────────────────────────┘

No orchestrator. The bus is the only shared dependency.

Pattern 2: Confidence Scoring Before Execution (Not After)

Early on, every agent acted on every trigger. The result: noise. Agents responding to events they shouldn't, sending messages that didn't need sending, spending tokens on work that didn't matter.

The fix: Every agent runs a confidence pre-check before executing:

Intent match score (0-1): Does this event match my defined scope?
Urgency score (0-1): Does this need action NOW?
Impact score (0-1): What happens if I don't act?
Duplication check (0-1): Has another agent already handled this?

Combined score threshold: 0.7 (configurable per agent). Below threshold = log and ignore.

Agent intent match: 0.85
Urgency: 0.3
Impact: 0.2
Duplication: 0.1
─────────────────────
Weighted score: 0.45 → BELOW THRESHOLD → No action

Result: 60% reduction in unnecessary actions. Token spend dropped by 40%.

Pattern 3: Audit Layer Is Not Optional

When you operate without a central orchestrator, you need an external observer. Stella is that observer.

Stella doesn't execute. Stella audits. Every output, every decision, every message — Stella independently verifies against three criteria:

Factual correctness: Did the agent cite real sources?
Scope compliance: Did the agent stay within its defined boundaries?
Narrative integrity: Is the output consistent with the constitutional rules?

If Stella flags an issue, the output is held. No exceptions.

Pattern 4: Shared Memory, Independent Will

All agents read from shared permanent knowledge. All agents write daily logs. But no agent can modify another agent's scope.

Shared:

Permanent knowledge files (read-only for most agents)
Agent-Bus event stream
State files

Independent:

Each agent's SOUL.md (its constitution)
Each agent's SKILL.md (its tools)
Each agent's memory files (its personal history)

This prevents cascading failures. One agent going rogue cannot corrupt others.

Pattern 5: The Human Bridge

For all the autonomy, there are operations that must involve humans:

External account setup
Founder-signed legal documents
Strategic decisions with undefined criteria

We call this the human bridge — explicit checkpoints where AI hands off to a human, not because AI can't do it, but because trust requires a human signature.

What We Learned (The Hard Way)

Don't optimize for uptime — optimize for auditability. 99.9% uptime means nothing if you can't prove what each agent decided and why.
Log everything, filter later — Storage is cheap. Debugging agent decisions without full context is expensive.
Constitution > Prompts — Prompts change. Constitutions persist. Our SOUL.md files have been modified 4 times in 3.5 months. Each change was a conscious governance decision, not a prompting tweak.

The Stack

Hardware: 2 CPU cores, 3.6GB RAM (Tencent Cloud light instance)
Runtime: Node.js via OpenClaw framework
Communication: Agent-bus pub/sub via file-based events
Memory: File system + compilable wiki supplements
Governance: Per-agent SOUL.md + shared permanent knowledge
Audit: Stella cross-checks all agent outputs against constitutional rules
Model routing: DeepSeek V4 (flash mode for most ops, full mode for strategy)