DEV Community

Cover image for Your Agent's Event Log Is a Test Fixture
Bobby Hall Jr
Bobby Hall Jr

Posted on Originally published at bhjr.dev

Your Agent's Event Log Is a Test Fixture

#ai #agents #typescript #testing

Most agent bugs I see aren't model bugs. They're pipeline bugs. A tool ran that shouldn't have. A loop never stopped. Nobody can say what happened because nothing was recorded.

If you own the pipeline, you can test it like any other code. Here's the technique I use in Module 1 of my course, Ship a Safe Agent: treat the agent's event log as a test fixture.

1. Record every decision as a typed event

The harness writes down everything that happens, not just the final answer:

export type Event =
  | { type: "user"; text: string }
  | { type: "skill_loaded"; name: string }
  | { type: "tool_call"; name: string; args: Args }
  | { type: "tool_result"; name: string; result: string }
  | { type: "blocked"; name: string; reason: string }
  | { type: "answer"; text: string };
Enter fullscreen mode Exit fullscreen mode

Notice blocked is a first-class event. When the harness refuses a call, that refusal is data you can assert on, not a silent no-op.

2. Swap the model for a script

The loop only needs a function that takes the context and returns the next turn:

export type Model = (ctx: Context) => Turn;
Enter fullscreen mode Exit fullscreen mode

In production that's an LLM call. In tests it's a script that reads the log and decides what to propose next. Same interface, zero network, fully deterministic. Your harness can't tell the difference, which is the point: you're testing the harness, not the model.

3. Run the same task under different limits

The loop takes limits as input: a step budget and a set of approved tools.

const task = "Customer wants a refund for order 1042.";
const runs: Array<[string, number, string[]]> = [
  ["run 1: no approval", 8, []],
  ["run 2: refund approved", 8, ["issue_refund"]],
  ["run 3: step budget 2", 2, ["issue_refund"]],
];
for (const [label, maxSteps, approved] of runs) {
  console.log(`\n${label}`);
  console.log(formatLog(runAgent(task, refundModel, { maxSteps, approved: new Set(approved) })));
}
Enter fullscreen mode Exit fullscreen mode

Same model, same task. Only the limits change. Here's what the log looks like for run 1:

user     Customer wants a refund for order 1042.
skill    loaded refunds
call     lookup_order({"id":"1042"})
result   order 1042: $48.00, placed 12 days ago
call     issue_refund({"id":"1042","amount":"48.00"})
BLOCKED  issue_refund: needs approval
answer   I need approval before I can issue_refund.
Enter fullscreen mode Exit fullscreen mode

Run 2 shows result refunded $48.00. Run 3 ends with Stopped: step budget of 2 reached. Three behaviors, each visible line by line.

4. Commit the log and diff against it

Save that output as expected.txt. Then one test covers every step:

test(`steps/${dir} matches expected output`, () => {
  const out = execFileSync("npx", ["tsx", join("steps", dir, "run.ts")], { encoding: "utf8" });
  assert.equal(out.trim(), readFileSync(join("steps", dir, "expected.txt"), "utf8").trim());
});
Enter fullscreen mode Exit fullscreen mode

Now if someone loosens a permission, removes the budget check or reorders the loop, the diff shows exactly which line of behavior changed. A refund that used to be BLOCKED turning into a result is a failing test, not a support ticket.

Why this matters

You can't unit test a prompt. You can unit test the code that decides what a prompt is allowed to do. Once the log is the contract, your agent's safety properties live in version control and get reviewed like everything else.

Try it

Module 1 of Ship a Safe Agent is free. You build this harness in six small steps, in TypeScript, with no API key and no framework, then finish three graded exercises that npm test checks for you.

The waitlist is right on the course page. Founding pricing ends Nov 8, and every purchase has a 14-day refund.

Top comments (0)