The Diff Wasn't the Model: Fingerprint the Environment Before You Replay a Trace

A replay failed at tool call seven. The run diff was clear and useless. The model had not changed. Our search_docs schema had.

We replayed a two-week-old trace on a Tuesday. Step seven sent {"q": "retry policy"}. The new schema wanted {"query": "retry policy"}. The tool rejected the call, the agent retried twice, then gave up.

Cost of that false alarm: half a day and two engineers. Everyone blamed a model upgrade. The real fix was a fingerprint we never recorded.

Why replay diffs lie

A tool-call diff shows what changed. It cannot show why. Four different causes look identical in a trace:

1. Model behavior changed.
2. A tool schema or tool implementation changed.
3. A prompt template or dependency lock changed.
4. Non-determinism: timeouts, retries, ordering, clock.

Only cause 1 is a model problem. Causes 2 and 3 live in your repo. Cause 4 lives in your harness. You cannot separate them without environment data.

The fix is cheap. Write one manifest line at the top of each trace. Compare that line before you compare anything else.

Fingerprint what the agent actually sees

Do not fingerprint your intent. Fingerprint the inputs the agent receives.

• Tool schemas: canonical JSON of every schema file, hashed by tool name.
• Prompt templates: every prompt file, content-hashed with its path.
• Model identity and params: requested model, returned model, temperature, max tokens, tool timeout.
• Code and deps: lockfile hash plus interpreter version.

Add one field that most harnesses skip: model_returned. Copy it from the response metadata, not from your config. If an endpoint can serve different models behind one name, this field is what catches it.

The artifact: fingerprint.py

Two files, no dependencies, Python 3.9 or newer.

# fingerprint.py
import hashlib
import json
import platform
from pathlib import Path

TOOL_DIR = Path("tools/schema")
PROMPT_DIR = Path("prompts")
LOCKFILE = Path("requirements.lock")


def canon(obj) -> str:
    return json.dumps(obj, sort_keys=True, separators=(",", ":"), ensure_ascii=False)


def dir_hash(root: Path, pattern: str) -> str:
    h = hashlib.sha256()
    for f in sorted(root.rglob(pattern)):
        h.update(f.relative_to(root).as_posix().encode() + b"\0")
        h.update(f.read_bytes() + b"\0")
    return h.hexdigest()


def tool_schema_hash() -> str:
    h = hashlib.sha256()
    for f in sorted(TOOL_DIR.glob("*.json")):
        h.update(f.stem.encode() + b"\0")
        h.update(canon(json.loads(f.read_text())).encode() + b"\0")
    return h.hexdigest()


def build_manifest(model_requested, model_returned, params, tool_timeout_s) -> dict:
    m = {
        "model_requested": model_requested,
        "model_returned": model_returned,
        "model_params": params,
        "tool_timeout_s": tool_timeout_s,
        "tool_schemas": tool_schema_hash(),
        "prompts": dir_hash(PROMPT_DIR, "*.txt"),
        "lockfile": hashlib.sha256(LOCKFILE.read_bytes()).hexdigest(),
        "python": platform.python_version(),
    }
    m["manifest_id"] = hashlib.sha256(canon(m).encode()).hexdigest()[:16]
    return m


def open_trace(path: str, manifest: dict):
    f = open(path, "w", encoding="utf-8")
    line = {"span_type": "run_manifest", "schema": 1, **manifest}
    f.write(json.dumps(line, sort_keys=True) + "\n")
    f.flush()
    return f

Call it once per run, before the first span. The snippet below is illustrative; the exact response field depends on your client.

manifest = build_manifest(
    model_requested="<your-model-name>",
    model_returned=response.model,       # from response metadata
    params={"temperature": 0.0, "max_tokens": 1024},
    tool_timeout_s=30,
)
trace = open_trace(f"traces/run-{run_id}.jsonl", manifest)

Note the schema field. Bump it when you change the manifest shape. Old traces then fail loudly instead of silently comparing the wrong keys.

Classify a mismatch in one command

# attribute.py
import json
import sys


MODEL_KEYS = {"model_requested", "model_returned", "model_params"}
IGNORED = {"manifest_id", "schema"}


def load_manifest(path: str) -> dict:
    with open(path, encoding="utf-8") as f:
        return json.loads(f.readline())


def classify(recorded: dict, current: dict):
    keys = set(recorded) | set(current)
    diffs = sorted(
        k for k in keys
        if k not in IGNORED and recorded.get(k) != current.get(k)
    )
    if not diffs:
        return "SAME_ENV", diffs
    if MODEL_KEYS & set(diffs):
        return "MODEL_DRIFT", diffs
    if "tool_schemas" in diffs:
        return "SCHEMA_DRIFT", diffs
    if {"prompts", "lockfile"} & set(diffs):
        return "CODE_DRIFT", diffs
    return "ENV_DRIFT", diffs


if __name__ == "__main__":
    recorded = load_manifest(sys.argv[1])
    with open(sys.argv[2], encoding="utf-8") as f:
        current = json.load(f)
    verdict, diffs = classify(recorded, current)
    print(verdict, diffs)
    sys.exit(0 if verdict == "SAME_ENV" else 3)

Emit the current manifest with the same builder your runner uses. One small script is enough:

# manifest_now.py
import json
from fingerprint import build_manifest

print(json.dumps(build_manifest(
    model_requested="<your-model-name>",
    model_returned="<value from a probe call>",
    params={"temperature": 0.0, "max_tokens": 1024},
    tool_timeout_s=30,
)))

Then wire it in front of your replay step.

python manifest_now.py > current.json
python attribute.py traces/run-abc123.jsonl current.json || exit 3
python replay.py traces/run-abc123.jsonl > replay.jsonl
python trace_diff.py traces/run-abc123.jsonl replay.jsonl

Exit code 3 means "do not trust this diff yet". In CI that is a guard. In a terminal it is a prompt to look at the environment first.

A debug loop you can reuse

1. Replay under the recorded manifest. Pin model, prompts, and schemas to the trace values. If you cannot pin them, stop here.
2. Run the classifier. SAME_ENV means the diff is real behavior. Anything else means fix the environment first.
3. Diff the first divergence only. Ignore everything after it. Later differences are usually cascade noise.
4. Record the fix as a new trace. Do not edit old traces. Traces are evidence, not scratch paper.
5. Turn the failing span pair into a test. Two spans plus one assertion equals one regression test.

Steps 2 and 3 are the ones teams skip. The classifier costs one command. It cancels the "is it the model?" meeting before it starts.

Decision table

Verdict	First check	Usual action
MODEL_DRIFT	model_returned vs recorded value	Accept as new behavior, re-record baseline
SCHEMA_DRIFT	git log tools/schema	Fix the tool or migrate the prompt
CODE_DRIFT	Prompt file or lockfile diff	Revert, or accept and re-record
ENV_DRIFT	Timeout, interpreter, clock	Pin the environment, then replay again
SAME_ENV	The first differing span	Debug agent logic with confidence

One rule survives all five rows: never debug step N+1 before you explain step N.

Where a free server fits

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

A replay loop is mostly idle work. Nightly replay of ten saved traces is a cron job, not a cluster. The steady load is a small number of model calls plus a lot of file hashing.

That shape of work is exactly what a free server option is good for. The MonkeyCode operator states that the project offers free model access and a free server option, including a free tier with 10 million tokens. Treat that as a claim to verify on the current pricing page, not as a permanent guarantee. Free tiers change, and quotas move.

The second caveat matters more for this article. If model identity behind an endpoint can shift, model_returned is not optional. Record it every call, or your replay diffs will send you hunting for a model bug that never existed.

If you already have a machine, run this locally. The workflow does not depend on any vendor.

Limitations and who should skip this

• Fingerprints detect drift, they do not explain it. You still read git log to find the cause.
• Hashes cover files, not runtime state. Feature flags, env vars, and database rows are invisible unless you hash them too.
• Non-determinism survives. Timeouts and retries still vary between replays. Pin them or accept the noise.
• Short-lived traces do not need this. If you replay within the hour, drift is unlikely to be the cause.
• Fully pinned containers may make it redundant. Check your existing pinning before adding a layer.
• The code above is illustrative. I have not published benchmark numbers for it, and I have not measured how much time it saves across a team.

Skip this entirely if you cannot replay runs at all. A manifest tells you nothing when there is no second run.

What to record next

Two small additions to your span schema buy most of the value:

• manifest_id on every span, not just the first line. Then you can group failures by environment.
• model_returned on every model call. Then a provider-side change is visible in one column.

With those two fields, one query answers the question that costs teams the most: which failing replays even had a stable environment?

Start with the manifest. The diff can wait until you trust it.

Comments

[Kent Bodrov]

In the classifier example, schema is in IGNORED. If two manifests have identical remaining fields but incompatible schema versions, wouldn't that return SAME_ENV? That seems at odds with the earlier fail-loudly requirement. I'd validate schema compatibility before computing drift, with separate tests for an unsupported version and a missing version. Otherwise the comparison can look reassuring before we know whether the fields mean the same thing.