- Book: AI Agents Pocket Guide: Patterns for Building Autonomous Systems with LLMs
- Also by me: Thinking in Go (2-book series) — Complete Guide to Go Programming + Hexagonal Architecture in Go
- My project: Hermes IDE | GitHub — an IDE for developers who ship with Claude Code and other AI coding tools
- Me: xgabriel.com | GitHub
Most agent replay rigs key on tool name. The name is the least stable thing about a tool. Schemas drift, defaults drift, SDK behavior drifts. And your "deterministic replay" silently runs against a different tool than the one in the original trace.
You won't see it in the diff. You'll see it in the eval score.
The replay you thought you had
You've seen this exact movie. Friday: you merge a PR that tightens a system prompt. The eval rig reruns the regression set against the recorded traces. Score drops 13 points. You spend the weekend bisecting prompt tokens. Monday morning your colleague mentions, casually, that they bumped the vendor SDK on Thursday because of a CVE patch.
The tool didn't move. The tool's contract moved. A field changed from optional to required. A default flipped from false to true. One enum value got renamed. The agent kept calling the tool, the tool kept returning data, and your "deterministic" replay quietly diverged.
This is the bug that makes engineers stop trusting their evals. Once a baseline lies to you twice, you start ignoring it.
What "replay" actually replays
Pull up your replay code. I bet it does some subset of this:
- Pins model version
- Pins temperature to 0 (and maybe sets a seed)
- Snapshots the system prompt and tool definitions used at trace time
- Replays the recorded tool call args and asserts the agent makes the same calls
That's a good list. It's also missing the thing that actually broke you.
Nothing in that list replays what the tool did. The tool name in your trace was get_invoice. The tool name in your registry is still get_invoice. Looks identical. The implementation behind it has shifted three times since the trace was recorded. Your replay calls today's tool with yesterday's args and gets today's behavior, then compares the agent's response to a baseline built against yesterday's behavior.
It's not deterministic. It's just slow non-determinism.
Three kinds of tool drift
Before you write a single line of fingerprint code, name what you're catching. Drift comes in three flavors and they get conflated constantly.
Signature drift. The arguments change. A new required parameter appears. A type widens or narrows. A field renames. The function call shape that worked yesterday raises TypeError today, or worse, silently coerces and proceeds with a default you didn't authorize.
Schema drift. The args stay the same but the validation rules around them shift. Maximum string length goes from 500 to 200. An enum loses a value. The JSON Schema your tool advertises to the model changes, and the model now picks different values because it sees different constraints.
Behavior drift. Signature and schema are identical. The function still accepts the same inputs. It just returns different outputs. The vendor changed a sort order. A timezone default switched from server-local to UTC. A pagination cursor format moved from base64 to opaque. The agent gets different ground truth and reasons accordingly.
The first two you can catch with a static check. The third needs a snapshot. Your fingerprint has to cover all three or it's not a fingerprint, it's a vibe.
The fingerprint, in 30 lines
The contract: every tool in your registry computes a stable hash from its signature, its schema, and its pinned version. Two tools with the same hash are interchangeable for replay purposes. Two with different hashes are not.
import hashlib
import inspect
import json
from typing import Any, Callable
def tool_fingerprint(
fn: Callable[..., Any],
schema: dict,
version: str,
) -> str:
# canonical signature: name + ordered (param, annotation) pairs
sig = inspect.signature(fn)
params = [
(name, str(p.annotation), str(p.default))
for name, p in sig.parameters.items()
]
sig_canon = json.dumps(
{"name": fn.__name__, "params": params},
sort_keys=True,
separators=(",", ":"),
)
# canonical schema: sorted-key JSON, no whitespace
schema_canon = json.dumps(
schema, sort_keys=True, separators=(",", ":")
)
blob = f"{sig_canon}|{schema_canon}|{version}".encode()
return hashlib.sha256(blob).hexdigest()[:16]
Three inputs, one hash, sixteen hex chars. Short enough to eyeball in a trace, long enough to never collide in practice.
Three things matter about how this is written, and one thing matters about how it is used.
Sort the JSON keys. Without sort_keys=True the same schema serialized twice can produce different bytes and you get false drift alerts every time someone reorders a dict literal. Strip whitespace with separators=(",", ":") for the same reason. A pretty-printed schema and a minified one must produce the same hash.
Pin the version explicitly. Don't read it from pkg.__version__ inside the function. Pass it in. You want the registration code to make a deliberate choice about which version this fingerprint represents. Implicit version reads are how you ship a fingerprint that changes every time pip updates a transitive dep.
Truncate to 16 hex chars. SHA-256 gives you 64. Logs and span attributes get noisy. Sixteen is 2^64 worth of distinct fingerprints, which is more than your tool registry will ever see.
The use-it-right part: capture the fingerprint at registration time, not call time. Store it once. Compare it everywhere.
Where to capture and compare
Capture at registration. Store on every span. Compare on replay. Fail loud by default.
class ToolRegistry:
def __init__(self) -> None:
self._tools: dict[str, dict] = {}
def register(
self,
fn: Callable[..., Any],
schema: dict,
version: str,
) -> None:
fp = tool_fingerprint(fn, schema, version)
self._tools[fn.__name__] = {
"fn": fn,
"fingerprint": fp,
"version": version,
}
def call(self, name: str, args: dict) -> Any:
entry = self._tools[name]
# emit fingerprint on the span attached to this call
current_span().set_attribute(
"tool.fingerprint", entry["fingerprint"]
)
current_span().set_attribute(
"tool.version", entry["version"]
)
return entry["fn"](**args)
At trace-record time, every tool span carries tool.fingerprint. At replay time, the comparator pulls the recorded fingerprint from the trace and checks it against the live registry's fingerprint for the same tool name.
The comparator is two lines and they should be fail-loud:
def assert_tool_match(
recorded_fp: str, live_fp: str, tool_name: str
) -> None:
if recorded_fp != live_fp:
raise ToolDriftError(
f"{tool_name}: recorded={recorded_fp} "
f"live={live_fp}"
)
Fail loud is the only sane default. The alternative, log a warning and continue anyway, is how you end up with a replay suite that's green and meaningless. If you can't trust the tool to be the same one, you can't trust the replay. Stop the run, surface the drift, force a human to decide whether the baseline needs regenerating or the SDK needs pinning.
Teams that pick "warn and continue" tend to end up the same way: a big tool registry, a dozen silent drifts piling up across a quarter, and an eval baseline nobody opens anymore. Don't be them.
The snapshot-stub pattern
Fail-loud catches drift. It doesn't help you replay through it. Sometimes you need the replay to run anyway. You're testing a prompt change, you've already accepted the SDK upgrade, you don't want to regenerate the entire baseline just to validate one orthogonal commit.
This is what the snapshot stub is for. When you record a trace, you store the (args, result) pair next to the fingerprint. At replay time, you intercept the tool call and return the recorded result instead of hitting the real implementation.
class SnapshotTool:
def __init__(
self,
recorded_calls: list[dict],
live_fp: str,
) -> None:
self._calls = {
json.dumps(c["args"], sort_keys=True): c["result"]
for c in recorded_calls
}
self._live_fp = live_fp
def __call__(self, **kwargs: Any) -> Any:
key = json.dumps(kwargs, sort_keys=True)
if key not in self._calls:
# the agent made a call that wasn't in the trace
# this is the agent drifting, not the tool drifting
raise SnapshotMissError(key)
return self._calls[key]
Two failure modes, two different signals. ToolDriftError means the tool changed under you. SnapshotMissError means the agent reasoned its way to a call that wasn't in the original trace. Your prompt change actually moved the agent's behavior, which is exactly what you want to know.
Use the real tool for integration tests. Use the snapshot stub for eval replays. The fingerprint comparator runs against both. In integration mode it asserts the live tool matches the recorded fingerprint and fails loud if it doesn't; in snapshot mode it does the same check and then bypasses the call. Same comparator, two modes.
Rolling this out without rewriting your rig
Three steps. Each one ships independently. None of them requires touching the agent code.
Step 1: fingerprint silently. Add tool_fingerprint() and emit tool.fingerprint on every span. Don't enforce anything. Let it run for a week, collect data, see what drift looks like in your actual telemetry. You'll find at least one tool whose fingerprint changes daily because someone put a timestamp in its schema. Fix that first.
Step 2: enforce in CI only. Switch the comparator on in your eval CI job. Production agents keep running. When the comparator fires in CI, the build fails with a diff of the two fingerprints and the developer either pins the old version or regenerates the baseline. This is the step that actually pays for itself. The first time it catches a silent SDK upgrade, you'll wonder how you shipped without it.
Step 3: add snapshot stubs. Once the fingerprint plumbing is stable, layer in the snapshot tool for replays where you specifically want to test the agent without re-executing tools. Keep integration tests using the real implementation. Now your eval rig has two modes: fast and isolated for replays, real and end-to-end for integration. The fingerprint guarantees both stay honest.
The migration is incremental on purpose. Teams who try to do all three at once break their CI for a sprint and conclude the technique doesn't work.
When drift is the signal, not the noise
The point of fingerprints is not zero drift. The point is visible drift.
When the comparator fires, that's information. The vendor shipped a breaking change. The schema you advertised to the model is no longer the schema the model is calling against. The right response is almost never "downgrade the SDK forever". It's:
- Read the changelog.
- Decide whether the new behavior is acceptable.
- Re-record the affected baseline against the new fingerprint.
- Commit the new baseline with a note explaining what drifted and why you accepted it.
Now your git history tells the story of how your tools evolved. Six months later, when the eval score moves three points and someone asks "what changed", you have a chain of intentional baseline updates instead of an archaeology dig through SDK release notes.
The fingerprint is small. The leak it closes is not. Most teams discover this the second time their eval baseline mysteriously decays. The teams who never discover it are the teams who stopped trusting their eval baseline a long time ago and learned to live without one, which is exactly the failure mode the rig was supposed to prevent.
Treat your tools the way you treat your model: pinned, versioned, hashed, and never trusted to be the same one as yesterday until you've checked.
If this was useful
Tool fingerprinting is one of the boring-but-load-bearing pieces of an agent-eval setup. The AI Agents Pocket Guide: Patterns for Building Autonomous Systems with LLMs digs into the rest of the setup: deterministic replay, snapshot stubs, regression scoring, and the eval-loop patterns that survive an SDK upgrade week. If you're maintaining a baseline you actually trust, the chapter on tool contracts and the one on replay determinism pair directly with what this post covered.
What's the worst silent tool drift you've shipped past your evals? Drop it in the comments.
Top comments (0)