I gave a 10-line agent a CRM world and asked it for one number

Last week I wanted to know whether my agent could do sales operations, not just talk about them. Asking it "what is the weighted pipeline?" and checking the answer by eye is a vibe check, not a test. So I built a small test world for it instead. Here is the full loop: ten lines of agent, one number, a failure I could actually debug, and a pass that meant something.

The world

Silo is a local-first simulation layer for testing AI agents. You bring your own agent (any framework, even a raw API loop), and Silo drops it into a realistic simulated business environment with tools, seeded state, tasks, and deterministic verifiers that grade what the agent did, not what it said. It is TypeScript-first and open source, at v0.4.0, early and pre-1.0, so breaking changes are expected.

Setup took about two minutes:

npm install @burn0/silo
npx @burn0/silo init demo --template crm
npx @burn0/silo env validate --env demo

Validation came back clean:

OK   demo
data=9 tasks=6 tools=42 verifiers=6

One honest gotcha: my scratch project's package.json was missing "type": "module", and the verifier typecheck failed with a cryptic complaint about top-level exports in a CommonJS module. Adding "type": "module" fixed it. The CRM template ships with 6 tasks, 42 tools (listing leads, converting them, reassigning opportunities, running pipeline forecasts), and 6 verifiers. Everything lives as plain TypeScript and JSON inside .silo/environments/demo, so you can read and edit all of it.

The task I picked was TASK-004: "Amara Osei needs a single number for the board: the probability-weighted value of everything currently open across the whole team, in US dollars. Report that figure." Easy difficulty, a question rather than a mutation. A good first blood.

The agent

Silo does not give you an agent. It gives you a world and a boundary. Your agent is one ordinary file that default-exports a function. Silo calls it with the task, the tool list, and a callTool function, and your function returns { output }. Mine has no LLM in it at all. It is deliberately dumb:

export default async function agent({ callTool }) {
  const { output: summary } = await callTool("pipeline_summary", {});
  const weighted = summary.totalWeightedAmount?.amount ?? 0;
  return { output: `The probability-weighted value of open pipeline is $${weighted} USD.` };
}

Ten lines. It asks the world for the pipeline summary and reports the weighted total. The point of testing a dumb agent first is that when something fails, you know it was the agent, not the prompt.

The failure

$ npx @burn0/silo run --env demo --task TASK-004

  Task          TASK-004 - Report the weighted value of open pipeline
  Result        FAIL
  Reward        0.33

  Failed
  ✗ The reported weighted total is correct

Two things made this failure worth having. First, the verifier did not grade my vibes. It compared my answer against a number it derived from the seeded world itself: "answered 0, expected 507500". I had guessed the wrong shape for the tool output, so my agent reported $0. The correct figure was $507,500, and it was not written down anywhere for the agent to peek at.

Second, the failure left a legible trail. Every run saves a directory with result.json, trace.jsonl, and state-diff.json. The trace showed the exact tool call, the exact tool result, and my exact output in sequence. I found the bug in about thirty seconds: callTool returns { output, isError }, not the raw output, and the weighted total sits at totalWeightedAmount.amount. I also noticed something instructive in result.json: even while failing, my agent scored 0.33, because an optional check still passed. Partial credit is explicit here, which is a good property in an eval.

The pass

Two-line fix, rerun:

  Task          TASK-004 - Report the weighted value of open pipeline
  Result        PASS
  Reward        1.00

  Checks        3 / 3
  Required      1 / 1

Here is what actually got checked, straight from the verifier labels:

1. The reported weighted total is correct. My answer matched the world: $507,500.
2. The total is the figure the answer closes on. The answer has to end on the number, not bury it in a paragraph. A small thing, but it is exactly the kind of sloppy answer an LLM would give that an eye-test would forgive.
3. Answering left the world unchanged. state-diff.json confirmed nothing mutated. This is the one that sold me. A question task should never change the CRM. If my agent had helpfully "tidied up" an opportunity while reading it, that check would have caught it, and an answer-only eval never could.

Every rollout starts from a fresh copy of the world, so runs cannot contaminate each other. The same agent on the same task is deterministic here, which makes the number a real regression gate rather than a flaky metric.

Try the medium one

If you run this yourself, TASK-002 is the more interesting homework: "Tomas Bergstrom has left the company and his account is deactivated, but his open deals are still sitting under his name. Move every one of his open opportunities to Priya Nair (USR-003)... Deals he already closed stay as they are."

That is the task where an agent can name the right rep, sound completely confident, and still forget to reassign half the deals, or reassign the closed ones too. The verifier checks the world afterward, so confidence does not score. state-diff.json will show you exactly which deals moved and which did not.

Where this is going

Silo is at v0.4.0 with CRM and ERP templates, a blank template for your own domain, and finance, support, and project-management worlds marked as coming soon. Everything runs on your machine: no account, no API key, nothing leaves your laptop.

If you are shipping an agent that touches a CRM, an ERP, or anything with state that matters, stop grading answers and start grading worlds. The repo is github.com/burn0-dev/silo, docs are at docs.burn0.dev/silo, and npm install @burn0/silo is the whole install.

My next run: wire a real LLM into that 10-line agent and watch TASK-002 go sideways. I will report back with the trace.