Same Template, Different Request: Fingerprint What You Actually Send to the Model

An eval run drifts. You check the usual suspects: the model ID is pinned, the dataset hash matches, and the prompt template hasn't changed since last week's commit. Everything you recorded says the two runs were identical. They weren't.

The gap is usually between what you versioned and what you sent. A template is an input to the request, not the request. Between the two sit a renderer, a JSON serializer, tool definitions built in code and whatever defaults your client library fills in. Any of them can change what the model receives without touching a file you hash.

This post covers where those differences come from, a small way to fingerprint the request you actually send, and what a fingerprint can't tell you.

Where "identical" requests differ

None of these show up in a template diff:

• Key order. A tool schema built from a dict or struct can serialize its keys in a different order after a refactor or a port. Anthropic's prompt caching docs warn that some languages (they name Swift and Go) randomize key order during JSON conversion, which breaks caches. If the provider's cache sees two different prefixes, your logs shouldn't call them the same request.
• Whitespace. Python's json.dumps defaults to ", " and ": " separators, and indent adds newlines. A schema pasted into a system prompt as text carries that whitespace along. So does a trailing newline at the end of a template file.
• Defaults. A client library upgrade can change a parameter you never set. If you only log what you passed in, the record looks unchanged.
• Rendering inputs. Retrieved context, today's date, a feature flag inside the template. Same template, different text.

  template v12 ---+
  retrieved docs -+--> render --> request body --> provider
  tool defs ------+                   ^
                                      |
  client defaults --------------------+
                         fingerprint the body, not "template v12"

Fingerprint the effective request

Hash the final request body, after rendering and after defaults are filled in, in a canonical form. Canonical means two JSON documents with the same content produce the same bytes, so the fingerprint changes when content changes, not when a serializer behaves differently.

RFC 8785, the JSON Canonicalization Scheme (Informational, June 2020), defines one way to do this: no whitespace between tokens, strict ECMAScript-style serialization of numbers and strings, and object properties sorted recursively while array order stays as it is. That last rule matters for LLM requests. Message order is meaning, so a canonicalizer must never sort arrays.

Inside a single Python service, the standard library gets you most of the way. The json docs describe sort_keys as useful for regression tests, and separators=(",", ":") drops the whitespace.

import hashlib
import json

def canonical(obj) -> bytes:
    # Close to RFC 8785 for typical request bodies, but not a full
    # JCS implementation (see the limits below).
    return json.dumps(obj, sort_keys=True, separators=(",", ":"),
                      ensure_ascii=False, allow_nan=False).encode()

def fingerprint(body: dict) -> str:
    return "sha256:" + hashlib.sha256(canonical(body)).hexdigest()

def send(post, body: dict, template_version: str, log) -> dict:
    resp = post(body)                 # hash exactly what was sent
    log({"request_fp": fingerprint(body),
         "template_version": template_version,
         "requested_model": body["model"],
         "response_model": resp.get("model")})
    return resp

Two tests pin down the behavior you want:

def test_key_order_does_not_change_fingerprint():
    a = {"model": "m-1", "tools": [{"name": "lookup",
         "schema": {"type": "object", "required": ["id"]}}]}
    b = {"tools": [{"schema": {"required": ["id"], "type": "object"},
         "name": "lookup"}], "model": "m-1"}
    assert fingerprint(a) == fingerprint(b)

def test_whitespace_inside_strings_does():
    a = {"model": "m-1", "system": "Use the tools.\n"}
    b = {"model": "m-1", "system": "Use the tools."}
    assert fingerprint(a) != fingerprint(b)

The second test is the important one. Whitespace inside a string is content: the model receives it. Canonicalization removes serializer noise between JSON tokens, not differences in the text you send. If a stray newline keeps flipping your fingerprints, fix the renderer instead of hiding it in the hash.

Compare in order when a run drifts

Log the template version and the request fingerprint side by side, then check them in this order:

1. Same fingerprint, different output. The request matched, so look at the provider side: model snapshot and run-to-run noise. Our earlier post on reproducible evals covers measuring that.
2. Different fingerprint, same template version. Something between the template and the wire changed. Diff the stored canonical bodies.
3. Different template version. Expected. That's a real change.

Step 2 is only quick if you keep the canonical body, not just its hash. That body is prompt text, so it needs the same redaction and access rules as any prompt log.

Limits

• Python's json isn't a full RFC 8785 implementation. JCS sorts keys by UTF-16 code units and formats numbers the ECMAScript way; Python sorts by code point and writes 1.0 where JCS writes 1. If fingerprints must match across languages, use a JCS library on every side and test against shared vectors.
• A matching fingerprint means two request bodies were equal after canonicalization. It doesn't mean the provider ran the same weights, and it says nothing about whether the output was right.
• Defaults applied on the provider's side never appear in your body. Record the model the response reports, too.

Where this fits

Recording what an AI-assisted decision was actually based on is the problem we work on at Obligra. Obligra Verify™ is a system of record for AI-assisted decisions, in public preview, designed to retain the evidence your application selects. Its docs have a page on record integrity and verification, a related but different job: a request fingerprint tells you two requests matched, while an integrity check tells you a stored record hasn't changed since it was captured. Neither tells you the decision was correct.

This article was generated by an AI agent (Claude) for the Obligra Verify team and checked against the linked sources.