Three AI Agents Walk Into a Debate Stage: Building Verifiable Agent Workflows with Agno + Shepherd

How to combine Agno's multi-agent orchestration with Shepherd's reversible execution traces to build agent systems you can actually trust.

---

The Problem

When an AI agent works on a non-trivial task, it builds up state — files edited, tools called, model responses — and that state is invisible after the process exits. If the agent makes a mistake at step 10, you can't roll back to step 5 without restarting from scratch, re-paying for every model call, and hoping the non-deterministic output reproduces the same early steps.

This is the problem Shepherd (Stanford, 2026) tackles: it records agent executions as durable, Git-like traces. Every run produces a retained output — a proposal held beside your files that you inspect and settle before it's applied.

And this is where Agno comes in: it makes building multi-agent systems productive. 30+ model providers behind one API, Teams with explicit orchestration modes, memory, knowledge, 100+ tools.

Together, they give you Agno's flexibility with Shepherd's safety net.

---

What We Built

A 205-line Python script where three Agno agents debate a proposition:

1. Pro argues FOR — writes pro_argument.txt
2. Con argues AGAINST — writes con_argument.txt
3. Judge reads both — writes verdict.txt with a winner

Shepherd wraps the whole thing so every run leaves a trace you can inspect via CLI:

shepherd run show run-2a4c8ca6355f
shepherd run trace run-2a4c8ca6355f --events
shepherd run select run-2a4c8ca6355f     # accept
shepherd run discard run-2a4c8ca6355f    # reject

A Sample Run

$ python debate_demo.py --topic "Remote work is better than office work"
[Pro] wrote 1,894 chars
[Con] wrote 1,849 chars
[Judge] wrote 3,843 chars → Pro wins

The Pro argued flexibility and work-life balance. The Con countered with collaboration and spontaneous interaction. The Judge analyzed both, found Pro's case more comprehensive, and wrote a 3,843-character verdict with specific reasoning.

Run it again and the verdict might flip — each run is independent, non-deterministic, and fully recorded.

---

Architecture

┌──────────────────────────────────────────────────┐
│                   Shepherd Workspace             │
│  ┌─────────────────────────────────────────────┐ │
│  │         Shepherd Task Subprocess            │ │
│  │                                             │ │
│  │  Agent(Pro, gpt-4o-mini)                    │ │
│  │    → pro.run("Argue FOR: {topic}")          │ │
│  │    → writes pro_argument.txt                │ │
│  │                                             │ │
│  │  Agent(Con, gpt-4o-mini)                    │ │
│  │    → con.run("Argue AGAINST: {topic}")      │ │
│  │    → writes con_argument.txt                │ │
│  │                                             │ │
│  │  Agent(Judge, gpt-4o-mini)                  │ │
│  │    → judge.run(pro_text + con_text)         │ │
│  │    → writes verdict.txt                     │ │
│  └─────────────────────────────────────────────┘ │
│                                                  │
│  Run record → .vcscore/ (durable trace)          │
│  Settlement → select | discard | release         │
└──────────────────────────────────────────────────┘

---

Key Design Decisions

1. The provider:model_id Pattern

Instead of hardcoding model choices, every agent is configured via a string like "openai:gpt-4o-mini":

def make_agent(spec: str, role: str, instructions: str) -> Agent:
    provider, model_id = spec.split(':', 1)
    if provider == 'openai':
        from agno.models.openai import OpenAIChat
        model = OpenAIChat(id=model_id)
    elif provider == 'ollama':
        from agno.models.ollama import Ollama
        model = Ollama(id=model_id)
    elif provider == 'lmstudio':
        from agno.models.lmstudio import LMStudio
        model = LMStudio(id=model_id)
    return Agent(model=model, instructions=instructions)

This makes it trivial to swap models per agent without changing a line of agent logic:

python debate_demo.py \
  --pro-model "openai:gpt-4o" \
  --con-model "ollama:llama3.2" \
  --judge-model "openai:gpt-4o-mini"

2. File-Based Context Passing

The Judge receives the full text of both arguments in its prompt, not file paths. This avoids the "I can't read files" problem common with agents that try to use file-reading tools:

judge_prompt = (
    f'You are judging a debate on: "{topic}".\n\n'
    f'PRO argument:\n{"="*40}\n{pro_text}\n{"="*40}\n\n'
    f'CON argument:\n{"="*40}\n{con_text}\n{"="*40}\n\n'
    f'Who wins? Write a detailed verdict.'
)
judge_resp = judge.run(judge_prompt)

3. Idempotent Task Registration

Shepherd v0.2.1 has a read-after-write lock — you can't register a task while a previous run's world is still active. The fix is simple: try registration, and if it fails, the task already exists:

try:
    workspace.tasks.register_source(...)
except Exception:
    pass  # already registered

4. Shepherd's Settlement Workflow

This is the key differentiator from a plain Python script. After the debate runs, the output is in a retained state — it exists but isn't applied:

• shepherd run select <ref> — marks it accepted (immutable record)
• shepherd run discard <ref> — marks it rejected
• Consume-once semantics prevent double-applies

---

What Makes This Hard (and What We Learned)

Shepherd is early. v0.2.1 has real rough edges:

• Workspace locks. Stale runs block new operations. The try/except on registration works around it but isn't elegant.
• Subprocess isolation. The task body runs in a temp file subprocess. Environment variables (API keys) don't propagate — they must be passed as function arguments and set via os.environ inside the body.
• Advisory placement. File writes go to disk directly; Shepherd doesn't intercept them as a changeset. The changeset shows empty paths. True interception requires placement='jail' (macOS Seatbelt), which we didn't exercise.
• Trace granularity. The run trace shows lifecycle events but not individual model calls. Deeper instrumentation is on Shepherd's roadmap.

Agno is polished. The agent abstraction, model routing, and Agent.run() API are clean and well-documented. The from agno.models.openai import OpenAIChat pattern works exactly as expected.

---

How to Fork and Extend

The full source is at courtroom.agi/debate_demo.py. Here are the best first extensions:

Easy

• [ ] Structured verdict — Use Agno's output_model with a Pydantic schema for scores per criterion
• [ ] FileTools — Replace open() with Agno's FileTools for agent-managed file I/O
• [ ] Git diffs — Commit verdicts to git after each run for history

Medium

• [ ] Multi-round debate — Pro rebuts Con, Con rebuts back, then Judge decides
• [ ] Audience agent — A 4th agent simulates audience reaction
• [ ] Web UI — Wrap in Streamlit or FastAPI
• [ ] MCP tools — Give agents web search or data lookup during debate prep

Advanced

• [ ] Jail enforcement — Run with placement='jail' for OS-level write interception
• [ ] Supervisor meta-agent — An Agno agent watches the Shepherd trace and reverts bad steps via shepherd run discard
• [ ] Model A/B test — Fork the same prompt prefix across different model combos, compare outcomes

---

Why This Matters

Most agent demos show a single agent printing text. Production agent systems need:

• Observability — what did the agent actually do?
• Reversibility — how do I undo a bad step?
• Settlement — how do I decouple "produced" from "applied"?

Agno handles the orchestration. Shepherd handles the record. Together they represent a pattern that will become standard as agent systems move from prototypes to production.

Code & more: https://www.dailybuild.xyz/project/187-courtroomagi