Pin Your Agent's Behavior: Snapshot Tests for Tool-Call Sequences When You Swap Models

A coding agent that works today can quietly change behavior tomorrow — not because you touched your code, but because the model behind it changed. If you evaluate free coding models (or switch between them to manage cost), the question stops being "which model is smarter" and becomes "did my agent's workflow survive the swap?"

Benchmark scores won't tell you. A model can score higher on a public leaderboard while calling your tools in a different order, skipping your lint step, or reading twice as many files before editing. What you can pin down and diff is the sequence of tool calls the agent makes for a fixed task.

This article shows a small pytest-based snapshot harness that records tool-call sequences, stores them as golden files, and fails loudly when a model swap changes the agent's behavior. It runs anywhere Python runs — including a free server — and works against any OpenAI-compatible endpoint, so it fits free model tiers without modification.

Why tool-call sequences, not final answers

Final code output is noisy: formatting, comments, and naming vary run to run even with the same model. Tool-call sequences are much more stable signal:

• Did the agent run the test suite before editing, or after?
• Did it read the file it modified, or edit blind?
• Did it stay within the allowed directory?
• How many search calls did it burn before acting?

If the sequence changes after a model swap, your agent's process changed — and process is where subtle regressions (missed tests, wider file access, costlier exploration) live.

The harness

The idea: wrap your agent's tool dispatcher with a recorder, replay a fixed task list, and snapshot the recorded sequence. Here's a minimal but runnable version.

recorder.py — wraps any tool executor and logs calls:

import json, time
from pathlib import Path

class ToolRecorder:
    def __init__(self, executor):
        self.executor = executor   # dict: tool_name -> callable
        self.calls = []

    def call(self, name: str, args: dict):
        # Record only structural info, not full file contents
        entry = {
            "tool": name,
            "arg_keys": sorted(args.keys()),
            "arg_paths": [v for v in args.values()
                          if isinstance(v, str) and ("/" in v or "." in v)],
        }
        self.calls.append(entry)
        return self.executor[name](**args)

    def snapshot(self) -> str:
        # Normalize: sequence + arg shape, no timestamps, no content
        return json.dumps(self.calls, indent=2)

test_agent_behavior.py — the snapshot test:

import json
from pathlib import Path
import pytest

from recorder import ToolRecorder
from my_agent import run_task, build_tools  # your agent's entry points

GOLDEN_DIR = Path(__file__).parent / "golden"

TASKS = [
    "fix the off-by-one error in src/parser.py",
    "add a --verbose flag to the CLI entrypoint",
]

@pytest.mark.parametrize("task", TASKS)
def test_tool_sequence_stable(task):
    recorder = ToolRecorder(build_tools(sandbox="./sandbox_repo"))
    run_task(task, tools=recorder)

    golden = GOLDEN_DIR / (task[:24].replace(" ", "_") + ".json")
    current = recorder.snapshot()

    if not golden.exists():
        golden.write_text(current)
        pytest.skip(f"golden file created: {golden.name} — re-run to compare")

    expected = json.loads(golden.read_text())
    actual = json.loads(current)

    assert [c["tool"] for c in actual] == [c["tool"] for c in expected], (
        f"Tool-call sequence drifted for task: {task}\n"
        f"expected: {[c['tool'] for c in expected]}\n"
        f"actual:   {[c['tool'] for c in actual]}"
    )

Run it against model A, commit the golden files, then point the same harness at model B and run again. Any diff in the assertion output is a concrete, reviewable behavior change.

Where free models and a free server fit

This workflow is cheapest when the models you're comparing cost nothing to query repeatedly — snapshot testing is inherently re-run-heavy. I've been running this kind of comparison against the free coding models available through MonkeyCode, using their free server option to host the sandbox repo and the harness itself, so the whole loop (agent → tools → snapshot → diff) stays off my laptop and costs nothing per run.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Because the harness only needs an OpenAI-compatible endpoint, swapping models is a config change:

# Run 1: baseline
AGENT_MODEL=<model-a> pytest test_agent_behavior.py
git add golden/ && git commit -m "baseline: model-a"

# Run 2: candidate
AGENT_MODEL=<model-b> pytest test_agent_behavior.py

The exact model names and quotas will depend on what's currently offered — check the provider's docs rather than assuming any tier is permanent. If you want to try this loop yourself, MonkeyCode's free tier is one place to get both the models and a box to run on; the harness above works with any endpoint, though.

A decision table: what a diff actually means

Not every snapshot failure is a regression. Use this to triage:

Observed diff	Likely meaning	Action
Test tool moved from before-edit to after-edit	Process regression: blind edits	Block the swap, or add a system-prompt constraint and re-test
Extra search/read calls, same outcome	More exploration — possibly fine, possibly slower/costlier	Measure wall time and token count before deciding
Fewer read calls before edit	Higher risk of stale-context edits	Treat as regression until proven otherwise
Same tools, different order within a phase	Benign nondeterminism	Relax the assertion to compare sets within phases
New tool appears (e.g., shell where none was used)	Capability/behavior change	Audit sandbox permissions immediately

For the "benign nondeterminism" row, a softer assertion helps — compare multisets per phase instead of exact order:

from collections import Counter

def phase_signature(calls):
    return Counter(c["tool"] for c in calls)

assert phase_signature(actual) == phase_signature(expected)

Limitations and who should skip this

• Snapshots pin behavior, not quality. A stable tool sequence can still produce a wrong patch. Pair this with real unit tests on the agent's output.
• Golden files rot. When you intentionally change the agent's prompt or tools, regenerate and review the goldens like any other snapshot — blind --update defeats the purpose.
• Nondeterministic models flake. Even at low temperature, hosted models drift. Expect to re-baseline occasionally; if a model flakes every run, that flakiness is the finding.
• Free tiers change. Don't hard-code your CI around any free model or free server being available forever; keep the endpoint behind an env var, as above.
• Skip this if your agent does one-shot, low-stakes generation (e.g., commit message drafts). The harness earns its keep when agents touch real repos, run tools, or operate unattended — exactly the cases where a silent process change hurts.

Takeaway

Model comparisons based on vibes or leaderboard deltas miss the thing that actually breaks your workflow: process drift. Record the tool-call sequence, snapshot it, and diff it on every model swap. It's about 60 lines of Python, it runs on a free server against free models, and it turns "the new model feels different" into a specific, reviewable assertion failure.

What would you add to the recorder — token counts per call, or wall-time per phase? Curious how others are pinning agent behavior across model swaps.