We built AgentForge to solve our own problem. Here's what 6 months of production multi-agent deployment taught us.
Lesson 1: Start with Failure Modes, Not Success Cases
Everyone designs for the happy path. But in multi-agent systems, the failure modes multiply:
- Agent A succeeds but takes 30s → Agent B times out waiting
- Agent A returns malformed JSON → Agent B crashes parsing
- Two agents try to write the same file → Race condition
Design your orchestration around "what breaks" first.
Lesson 2: Observability Is Not Optional
You need per-agent execution traces. Not just logs — structured traces showing:
- Input parameters (exact values, not summaries)
- Output before any post-processing
- Retry attempts with backoffs
- Circuit breaker state transitions
We built this into AgentForge's execution engine. Every run generates a JSON trace you can replay for debugging.
Lesson 3: Agents Need Memory, But Not Infinite Memory
Unbounded conversation history degrades performance. We use a sliding window + summary strategy:
- Keep last N turns verbatim
- Summarize older turns into structured context
- Let agents explicitly "remember" key facts via a memory store
Lesson 4: Cost Optimization Is Architecture
Running 5 agents × 4K tokens × GPT-4 gets expensive fast. Our approach:
- Router agent determines which specialist to invoke (cheaper model)
- Specialist agents use larger models only when needed
- Response caching for deterministic queries
Result: 60% cost reduction vs. naive implementation.
The Stack
- Python 3.11+
- Pydantic for schema validation
- AsyncIO for concurrent agent execution
- SQLite/Redis for state persistence
- WebSocket for real-time monitoring UI
Open source. No VC pitch. Just code that works.
https://github.com/agentforge-cyber/agentforge-mvp
Join us: https://discord.gg/Qy6HKHsqP
Posted on 2026-07-30 by the AgentForge team.
Top comments (1)
Interesting lessons, especially starting with failure modes and making executions replayable.
I took a look at the linked repo and noticed the README currently frames AgentForge more around agent judgment scoring and its correlation with ROI, while the article describes a fairly complete multi-agent orchestration runtime.
Is the orchestration engine in another repo or branch, or is that the direction AgentForge is moving toward?
I'm particularly interested in how you distinguish replaying what an agent did from evaluating whether the decision it made was actually justified.