DEV Community: SYED-RAFI-NAQVI

I gave a 10-line agent a CRM world and asked it for one number

SYED-RAFI-NAQVI — Fri, 18 Sep 2026 13:49:14 +0000

I gave a 10-line agent a CRM world and asked it for one number

Last week I wanted to know whether my agent could do sales operations, not just talk about them. Asking it "what is the weighted pipeline?" and checking the answer by eye is a vibe check, not a test. So I built a small test world for it instead. Here is the full loop: ten lines of agent, one number, a failure I could actually debug, and a pass that meant something.

The world

Silo is a local-first simulation layer for testing AI agents. You bring your own agent (any framework, even a raw API loop), and Silo drops it into a realistic simulated business environment with tools, seeded state, tasks, and deterministic verifiers that grade what the agent did, not what it said. It is TypeScript-first and open source, at v0.4.0, early and pre-1.0, so breaking changes are expected.

Setup took about two minutes:

npm install @burn0/silo
npx @burn0/silo init demo --template crm
npx @burn0/silo env validate --env demo

Validation came back clean:

OK   demo
data=9 tasks=6 tools=42 verifiers=6

One honest gotcha: my scratch project's package.json was missing "type": "module", and the verifier typecheck failed with a cryptic complaint about top-level exports in a CommonJS module. Adding "type": "module" fixed it. The CRM template ships with 6 tasks, 42 tools (listing leads, converting them, reassigning opportunities, running pipeline forecasts), and 6 verifiers. Everything lives as plain TypeScript and JSON inside .silo/environments/demo, so you can read and edit all of it.

The task I picked was TASK-004: "Amara Osei needs a single number for the board: the probability-weighted value of everything currently open across the whole team, in US dollars. Report that figure." Easy difficulty, a question rather than a mutation. A good first blood.

The agent

Silo does not give you an agent. It gives you a world and a boundary. Your agent is one ordinary file that default-exports a function. Silo calls it with the task, the tool list, and a callTool function, and your function returns { output }. Mine has no LLM in it at all. It is deliberately dumb:

export default async function agent({ callTool }) {
  const { output: summary } = await callTool("pipeline_summary", {});
  const weighted = summary.totalWeightedAmount?.amount ?? 0;
  return { output: `The probability-weighted value of open pipeline is $${weighted} USD.` };
}

Ten lines. It asks the world for the pipeline summary and reports the weighted total. The point of testing a dumb agent first is that when something fails, you know it was the agent, not the prompt.

The failure

$ npx @burn0/silo run --env demo --task TASK-004

  Task          TASK-004 - Report the weighted value of open pipeline
  Result        FAIL
  Reward        0.33

  Failed
  ✗ The reported weighted total is correct

Two things made this failure worth having. First, the verifier did not grade my vibes. It compared my answer against a number it derived from the seeded world itself: "answered 0, expected 507500". I had guessed the wrong shape for the tool output, so my agent reported $0. The correct figure was $507,500, and it was not written down anywhere for the agent to peek at.

Second, the failure left a legible trail. Every run saves a directory with result.json, trace.jsonl, and state-diff.json. The trace showed the exact tool call, the exact tool result, and my exact output in sequence. I found the bug in about thirty seconds: callTool returns { output, isError }, not the raw output, and the weighted total sits at totalWeightedAmount.amount. I also noticed something instructive in result.json: even while failing, my agent scored 0.33, because an optional check still passed. Partial credit is explicit here, which is a good property in an eval.

The pass

Two-line fix, rerun:

  Task          TASK-004 - Report the weighted value of open pipeline
  Result        PASS
  Reward        1.00

  Checks        3 / 3
  Required      1 / 1

Here is what actually got checked, straight from the verifier labels:

The reported weighted total is correct. My answer matched the world: $507,500.
The total is the figure the answer closes on. The answer has to end on the number, not bury it in a paragraph. A small thing, but it is exactly the kind of sloppy answer an LLM would give that an eye-test would forgive.
Answering left the world unchanged. state-diff.json confirmed nothing mutated. This is the one that sold me. A question task should never change the CRM. If my agent had helpfully "tidied up" an opportunity while reading it, that check would have caught it, and an answer-only eval never could.

Every rollout starts from a fresh copy of the world, so runs cannot contaminate each other. The same agent on the same task is deterministic here, which makes the number a real regression gate rather than a flaky metric.

Try the medium one

If you run this yourself, TASK-002 is the more interesting homework: "Tomas Bergstrom has left the company and his account is deactivated, but his open deals are still sitting under his name. Move every one of his open opportunities to Priya Nair (USR-003)... Deals he already closed stay as they are."

That is the task where an agent can name the right rep, sound completely confident, and still forget to reassign half the deals, or reassign the closed ones too. The verifier checks the world afterward, so confidence does not score. state-diff.json will show you exactly which deals moved and which did not.

Where this is going

Silo is at v0.4.0 with CRM and ERP templates, a blank template for your own domain, and finance, support, and project-management worlds marked as coming soon. Everything runs on your machine: no account, no API key, nothing leaves your laptop.

If you are shipping an agent that touches a CRM, an ERP, or anything with state that matters, stop grading answers and start grading worlds. The repo is github.com/burn0-dev/silo, docs are at docs.burn0.dev/silo, and npm install @burn0/silo is the whole install.

My next run: wire a real LLM into that 10-line agent and watch TASK-002 go sideways. I will report back with the trace.

I caught my AI agent lying to me

SYED-RAFI-NAQVI — Thu, 17 Sep 2026 13:49:14 +0000

Last week I gave an AI agent a simple job: look at a sales pipeline and tell me what it is worth. It answered in seconds, confident, well formatted, numbers included. My eval script graded the response and gave it a PASS.

Then I looked at what the agent actually did. Nothing. It never touched the data. It guessed.

That is the moment I stopped trusting text-based evals for agents, and the reason I have been playing with Silo since.

Your eval grades essays. Your agent does surgery.

Here is the thing nobody says out loud: most agent evals are essay grading. You show the agent a prompt, it writes something back, and you (or an LLM judge) score how good the writing looks. That was fine when agents were chatbots.

Modern agents are not chatbots. They call tools, update records, move inventory, change the state of systems. An agent can write a beautiful summary of a pipeline while leaving every deal in the wrong stage. If your eval only reads the transcript, that failure is invisible. You ship it, and production finds out for you.

There are roughly three levels to this:

Unit tests. Does the code work?
Prompt evals. Does the answer look right?
Actually watching what the agent did. Did the world end up in the right state?

Silo is a tool for level 3. It is open source (MIT), local-first, TypeScript-first, and it gives your existing agent a realistic simulated world to work in: tools, seeded data, tasks, and verifiers that grade outcomes deterministically. Nothing leaves your machine.

The five-minute version

You need Node 22+ and "type": "module" in your package.json. Then:

npm i @burn0/silo
npx @burn0/silo init demo --template crm
npx @burn0/silo env validate --env demo

OK   demo
data=9 tasks=6 tools=42 verifiers=6

You now have a realistic simulated B2B sales pipeline on your laptop: 9 data collections, 6 tasks, 42 tools an agent can call, 6 verifiers that decide pass or fail. Small enough to hold in your head, which is exactly why it is the best template to start with. (There is also an ERP template with 185 tools and deliberately messy seed data. An invoice that bills more than was received. A payment that failed on stale bank details. Someone on the Silo team has seen things.)

Write an agent. Anything that default-exports a function counts, any framework or a raw API loop:

// silo.agent.js
export default async function agent({ callTool }) {
  const forecast = await callTool("forecast_report", {});
  const { weightedAmount } = forecast.output;

  return { output: `Open pipeline is worth $${weightedAmount.amount}` };
}

Run it against a real task:

npx @burn0/silo run --env demo --task TASK-004 --agent ./silo.agent.js

  Task          TASK-004 — Report the weighted value of open pipeline
  Result        PASS
  Reward        1.00

  Tool calls    1
  Checks        3 / 3
  Required      1 / 1

  Run saved: .silo/runs/run_20260915012734_4t01

Fine. Now here is the part that got me.

The verifier does not read your agent's essay

In Silo, your agent never sees the world state. It gets the task instruction, the tool schemas, and cloned tool output. If it wants to know something, it has to call a tool for it, exactly like production.

Grading works the same way in reverse. A task passes because the world changed correctly, never because the agent said the right words. Remember my lying agent from the top of this post? Under Silo it fails loudly: zero tool calls, checks unmet, done. No partial credit for a confident tone.

Two design rules keep this honest:

Time is simulated. state.now is the only clock. Nothing reads Date.now(), so a run from Tuesday reproduces exactly on Friday.
Every run leaves evidence. Each rollout writes a directory: trace.jsonl (every event, append-only), result.json (checks, reward, tool errors), state-diff.json (exactly what changed), run.json (task, verifier, timings).

My favorite detail: result.json and state-diff.json contain no timestamps or run ids. That makes them an exact regression oracle. Diff two runs and any difference is a genuine behavioral change, not clock noise. And when your agent is non-deterministic, --runs 5 repeats the task so you see the real distribution instead of the lucky run you would have screenshotted.

The honest caveats

Silo is early, v0.4.0, and the maintainers say plainly to expect breaking changes before 1.0. Packaged adapters for LangChain, Vercel AI SDK, OpenAI, Anthropic, and Mastra are still roadmap, as are an MCP server and LLM judges to sit alongside the deterministic checks. If you need those today, you will be writing some glue.

But the core loop works right now, and it rearranged how I think about agent testing in about an afternoon. Stop asking "did it write a good answer" and start asking "is the world right." Those are different questions, and only one of them protects production.

Links, since you will ask:

Start with the CRM template. Then try to make your agent lie to the verifier. It is harder than you think, and that is the point.