Unfingerprinted Generation Spans Cannot Be Diffed

A trace diff is undefined until every generation span carries a fingerprint of the prompt, the tool schema, and the declared model identity. Output tests do not supply that. Neither does a green exit code.

When an agent run looks worse overnight, the usual reflex is to rewrite the system prompt. That reflex assumes the two runs were the same experiment. They often were not. Shared free-model endpoints omit model, rewrite routing, or inject a provider preamble you never hashed. The second run is a different instrument. Comparing it to the first is like subtracting voltages measured on two unlabeled probes.

The measurement problem is older than agents. A unit test that asserts on the final file will pass while the model retried a tool four times, dropped a call_id, or answered from a different checkpoint than yesterday. Process and product diverged. If you only keep the product, you cannot say which one moved.

A generation fingerprint is a small, stable tuple stored as span attributes. It is not a dashboard slogan. Four fields are enough to make two traces comparable, or to prove they are not.

prompt.hash is a truncated SHA-256 of canonical JSON: system text, user text, and the tool JSON schema. tool.schema_hash is the same hash over the tools array alone, so a prompt-only edit does not look like a tool-surface change. gen_ai.request.model is the declared identifier if the exporter emitted one. If it is missing, the span is incomparable by default. gen_ai.request.temperature belongs in the tuple because a silent 0.0 to 0.7 swap will scatter tool-call order without touching the prompt.

Hash canonicalization must be boring and strict. Sort keys. Reject NaN. Encode UTF-8. Do not pretty-print. Two semantically equal prompts that differ by a trailing newline are different experiments, and the hash should say so.

The following module is a template. Wire it to your own span exporter. It has not been executed against a public corpus in this article.

# fingerprint.py — template, not a vendor SDK
from __future__ import annotations

import hashlib
import json
from typing import Any

def canonical(obj: Any) -> bytes:
    return json.dumps(
        obj,
        sort_keys=True,
        separators=(",", ":"),
        ensure_ascii=False,
        allow_nan=False,
    ).encode("utf-8")

def short_sha(obj: Any, n: int = 16) -> str:
    return hashlib.sha256(canonical(obj)).hexdigest()[:n]

def generation_fingerprint(
    *,
    system: str,
    user: str,
    tools: list[dict],
    model: str | None,
    temperature: float | None,
) -> dict[str, str]:
    prompt_payload = {"system": system, "user": user, "tools": tools}
    attrs = {
        "prompt.hash": short_sha(prompt_payload),
        "tool.schema_hash": short_sha(tools),
        "fingerprint.missing_model": "true" if not model else "false",
    }
    if model:
        attrs["gen_ai.request.model"] = model
    if temperature is not None:
        attrs["gen_ai.request.temperature"] = str(temperature)
    attrs["fingerprint.tuple"] = short_sha(attrs)
    return attrs

Attach those attributes on the generation span at start, not after the model returns. A fingerprint written on the way out can include the completion and will then change when the model is flaky. You want the request identity, not the answer identity. Completion hashes belong on a child event if you need them.

Free shared endpoints make the missing-model case common. The HTTP layer answers. The span does not name a checkpoint. Re-running the same prompt on another host will not repair that. Refuse comparability until the exporter records an identifier, or pin one yourself from configuration rather than from the response body. Reading the model name back from the completion is how silent swaps leak into “stable” traces.

# stamp_model_from_config.py — template
import json
import os
import sys

declared = os.environ.get("AGENT_MODEL_ID", "").strip()
if not declared:
    print("AGENT_MODEL_ID unset; refusing export", file=sys.stderr)
    sys.exit(2)

for line in sys.stdin:
    span = json.loads(line)
    if span.get("kind") == "generation" and not span.get("gen_ai.request.model"):
        span["gen_ai.request.model"] = declared
        span["fingerprint.missing_model"] = "false"
        span["fingerprint.model_source"] = "config"
    sys.stdout.write(json.dumps(span, ensure_ascii=False) + "\n")

export AGENT_MODEL_ID=from-config-not-from-response
python stamp_model_from_config.py < raw.ndjson > stamped.ndjson

fingerprint.model_source=config is an admission, not a boast. It records that the name did not come from the provider. Downstream diffs can still run. They must not pretend the checkpoint was observed.

Export two runs as JSON-lines, one span per line. The comparer below does not grade quality. It answers a narrower question: may these traces be diffed at all?

# compare_fingerprints.py — template
from __future__ import annotations

import json
import sys
from pathlib import Path

REQUIRED = ("prompt.hash", "tool.schema_hash")

def load_generations(path: Path) -> list[dict]:
    rows = []
    for line in path.read_text(encoding="utf-8").splitlines():
        span = json.loads(line)
        if span.get("kind") != "generation":
            continue
        rows.append(span)
    return rows

def comparable(a: dict, b: dict) -> tuple[bool, str]:
    if a.get("fingerprint.missing_model") == "true" or b.get("fingerprint.missing_model") == "true":
        return False, "missing_model"
    keys = REQUIRED + ("gen_ai.request.model", "gen_ai.request.temperature")
    for key in keys:
        if a.get(key) != b.get(key):
            return False, f"mismatch:{key}"
    return True, "ok"

def main(path_a: str, path_b: str) -> int:
    gens_a = load_generations(Path(path_a))
    gens_b = load_generations(Path(path_b))
    if not gens_a or not gens_b:
        print("no generation spans", file=sys.stderr)
        return 3
    # Index alignment is a first pass. Prefer generation_id in production.
    n = min(len(gens_a), len(gens_b))
    incomparable = 0
    for i in range(n):
        ok, why = comparable(gens_a[i], gens_b[i])
        if not ok:
            incomparable += 1
            print(f"span[{i}] incomparable ({why})")
    if len(gens_a) != len(gens_b):
        print(f"generation count {len(gens_a)} != {len(gens_b)}")
        return 2
    if incomparable:
        return 2
    print("fingerprints align; a path diff is defined")
    return 0

if __name__ == "__main__":
    sys.exit(main(sys.argv[1], sys.argv[2]))

Run it as a gate in front of any path diff you already keep.

python compare_fingerprints.py run-a.ndjson run-b.ndjson
echo $?   # 0 defined, 2 incomparable, 3 missing generations

Exit code 2 is not a product failure. It is a lab-setup failure. Do not open a prompt PR. Do not bump a temperature. Restore the fingerprint first.

The decision rule is small enough to keep in the comparer, not in a wiki. If fingerprint.missing_model is true on either side, the traces are incomparable. If prompt.hash differs, you changed the experiment; attribute the delta to the prompt, not to “the agent.” If only tool.schema_hash differs, the model saw a different action surface. If only gen_ai.request.model differs, you are looking at a routing change. If the tuple matches and the child tool graph does not, then—and only then—you have an agent-path delta worth debugging.

That last case is the only one in which a span-level path diff has meaning. Everything else is a contaminated A/B test. Teams that skip the gate will spend the week “fixing” a prompt that never moved.

If you need a second machine to collect the B run, a free server option is enough to host the exporter and the comparer. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode provides free model access and a free server option. Treat both as a re-run host for the fingerprint gate above, not as a substitute for the attributes. After the spans exist, the method does not depend on a particular product.

Limitations are sharp. A hash does not see a provider-side system preamble that never entered your canonical JSON. Two fingerprints can match while safety filters rewrite the completion. Temperature in the tuple does not capture nucleus sampling or frequency penalty if you never record them. Index alignment of generation spans is a weak join. Production code should join on a client-generated generation_id. The templates above are a specification for exit codes, not a benchmark.

Do not store raw prompts next to the hashes if those prompts contain secrets. The hash is for equality, not for recovery. Do not use the fingerprint as a billing key. Do not sample generation spans before hashing. A 1% sample will drop the only span whose model identity was missing and will fake comparability.

Skip this workflow if every run already pins a model digest, a signed prompt bundle, and a tool-schema version in CI. You already have a stronger fingerprint, and SHA-16 on the side is noise. If you have no traces yet, record closed generation spans first. Hashing empty files will only certify that two voids are equal. If the agent is a single non-tool completion graded by a human, a prompt hash plus an output rubric is enough and the comparer is overhead.

The reusable debug loop is then four beats, always in this order. Fingerprint. Refuse the diff when the tuple disagrees. Diff tool children only after a defined comparison. Only then touch the prompt. Reverse the order and you will keep shipping prompt edits that chase routing noise.

If you already export spans, add the four attributes before you add another chart.