Last week I gave an AI agent a simple job: look at a sales pipeline and tell me what it is worth. It answered in seconds, confident, well formatted, numbers included. My eval script graded the response and gave it a PASS.
Then I looked at what the agent actually did. Nothing. It never touched the data. It guessed.
That is the moment I stopped trusting text-based evals for agents, and the reason I have been playing with Silo since.
Your eval grades essays. Your agent does surgery.
Here is the thing nobody says out loud: most agent evals are essay grading. You show the agent a prompt, it writes something back, and you (or an LLM judge) score how good the writing looks. That was fine when agents were chatbots.
Modern agents are not chatbots. They call tools, update records, move inventory, change the state of systems. An agent can write a beautiful summary of a pipeline while leaving every deal in the wrong stage. If your eval only reads the transcript, that failure is invisible. You ship it, and production finds out for you.
There are roughly three levels to this:
- Unit tests. Does the code work?
- Prompt evals. Does the answer look right?
- Actually watching what the agent did. Did the world end up in the right state?
Silo is a tool for level 3. It is open source (MIT), local-first, TypeScript-first, and it gives your existing agent a realistic simulated world to work in: tools, seeded data, tasks, and verifiers that grade outcomes deterministically. Nothing leaves your machine.
The five-minute version
You need Node 22+ and "type": "module" in your package.json. Then:
npm i @burn0/silo
npx @burn0/silo init demo --template crm
npx @burn0/silo env validate --env demo
OK demo
data=9 tasks=6 tools=42 verifiers=6
You now have a realistic simulated B2B sales pipeline on your laptop: 9 data collections, 6 tasks, 42 tools an agent can call, 6 verifiers that decide pass or fail. Small enough to hold in your head, which is exactly why it is the best template to start with. (There is also an ERP template with 185 tools and deliberately messy seed data. An invoice that bills more than was received. A payment that failed on stale bank details. Someone on the Silo team has seen things.)
Write an agent. Anything that default-exports a function counts, any framework or a raw API loop:
// silo.agent.js
export default async function agent({ callTool }) {
const forecast = await callTool("forecast_report", {});
const { weightedAmount } = forecast.output;
return { output: `Open pipeline is worth $${weightedAmount.amount}` };
}
Run it against a real task:
npx @burn0/silo run --env demo --task TASK-004 --agent ./silo.agent.js
Task TASK-004 — Report the weighted value of open pipeline
Result PASS
Reward 1.00
Tool calls 1
Checks 3 / 3
Required 1 / 1
Run saved: .silo/runs/run_20260915012734_4t01
Fine. Now here is the part that got me.
The verifier does not read your agent's essay
In Silo, your agent never sees the world state. It gets the task instruction, the tool schemas, and cloned tool output. If it wants to know something, it has to call a tool for it, exactly like production.
Grading works the same way in reverse. A task passes because the world changed correctly, never because the agent said the right words. Remember my lying agent from the top of this post? Under Silo it fails loudly: zero tool calls, checks unmet, done. No partial credit for a confident tone.
Two design rules keep this honest:
-
Time is simulated.
state.nowis the only clock. Nothing readsDate.now(), so a run from Tuesday reproduces exactly on Friday. -
Every run leaves evidence. Each rollout writes a directory:
trace.jsonl(every event, append-only),result.json(checks, reward, tool errors),state-diff.json(exactly what changed),run.json(task, verifier, timings).
My favorite detail: result.json and state-diff.json contain no timestamps or run ids. That makes them an exact regression oracle. Diff two runs and any difference is a genuine behavioral change, not clock noise. And when your agent is non-deterministic, --runs 5 repeats the task so you see the real distribution instead of the lucky run you would have screenshotted.
The honest caveats
Silo is early, v0.4.0, and the maintainers say plainly to expect breaking changes before 1.0. Packaged adapters for LangChain, Vercel AI SDK, OpenAI, Anthropic, and Mastra are still roadmap, as are an MCP server and LLM judges to sit alongside the deterministic checks. If you need those today, you will be writing some glue.
But the core loop works right now, and it rearranged how I think about agent testing in about an afternoon. Stop asking "did it write a good answer" and start asking "is the world right." Those are different questions, and only one of them protects production.
Links, since you will ask:
- Site: silo.burn0.dev
- Docs: docs.burn0.dev/silo/introduction
- Repo: github.com/burn0-dev/silo
Start with the CRM template. Then try to make your agent lie to the verifier. It is harder than you think, and that is the point.
Top comments (5)
@syedrafinaqvi, making the verifier read authoritative world state rather than the transcript is the strong boundary here. One failure mode I’d test is verifier independence: if
forecast_reportand the verifier share the same calculation helper, one defect can produce both a confident tool result and a PASS. A separately derived invariant over the seeded records, plus ordered trace contracts for forbidden intermediate actions, would reduce that correlated risk. In agent-inspect, I’ve been treating trajectory evidence and outcome checks as separate lanes for the same reason. Are Silo verifiers encouraged to avoid reusing the production tool implementation?Good catch, and the verifier docs agree with you directly. They call out exactly this: if a verifier derives its expected answer through the same helper the tools use, and that helper is wrong, both sides move together and the check still passes. The fix they prescribe is recorded baselines, an artifact frozen when the behavior was known good, which cannot follow a formula change. Derivation and baselines cover different failures.
On your question: the agent never touches production tools. Everything routes through callTool into the simulated world, so the sharing risk is between the simulated tool and the verifier, which is the case the docs warn about.
If you have ideas on making the independence stronger, open an issue on the repo. Would welcome it.
Outcome grading is the right correction, especially the insistence on a deterministic clock and timestamp-free artifacts. One extra trap is that a correct final state can still hide a bad path: the agent might briefly expose data, trigger a notification, or make and then undo a forbidden write.
So I would grade both the terminal state and a small set of trace invariants. "No calls outside this resource set," "no more than one payment attempt," and "never read field X" are properties a final diff cannot prove. State verifiers answer whether the job landed; trace constraints answer whether it got there safely.
Good point. The refund-void-reissue example is exactly it. Final state looks fine, customer got three emails.
One catch: trace rules go brittle fast. "Max one payment attempt" also punishes a legit retry after a transient failure. "Never read X" is clean. Behavior rules need context to judge.
Did you hit the make-and-undo case in prod? Curious what it looked like.
Outcome verification is the key distinction. I’d make each eval produce a compact receipt: initial state, permitted tool calls, observed state transition, and the invariant that passed or failed. That lets a reviewer diagnose whether the agent reasoned badly, used the wrong tool, or skipped the work entirely—rather than grading confidence after the fact.