48-Hour Field Notes: I Pinned a Request Fingerprint Before Comparing Two Model Runs

Two runs of the same prompt came back with the same shape and different numbers, and I almost filed that difference as a model upgrade. Then I looked at the request I had actually sent, and the two bodies were not identical at all. Forty-eight hours later I had a JSONL ledger, a second host, and a much shorter list of things I was willing to call a regression.

Here is the workflow, the code, and the parts that still refuse to be reproducible.

The problem: identical prompts are not identical requests

Your prompt lives in a dataclass, a config file, or a string in your editor, but the endpoint never sees any of that. It sees the serialized body: key order, defaults your client injects, max_tokens, temperature, and whatever the SDK decided to add this month. If any of those changed between two runs, you compared two experiments and called it one.

Worse, the model string you send is not always the model string that answers. I have seen a response echo back a longer, dated identifier than the alias I requested, which is exactly the kind of detail a text diff will never surface for you.

So the first move is not "run it twice". The first move is making the request itself addressable, so you can tell whether you ran the same thing at all.

Step 1 — Hash the wire, not your dataclass

A fingerprint is a hash over the serialized body plus the endpoint. Canonical JSON matters here, because key order alone will otherwise produce a different hash for an equivalent request.

# fp_ledger.py
import hashlib
import json
import os
import time
import uuid


def canon(obj) -> str:
    return json.dumps(obj, sort_keys=True, separators=(',', ':'), ensure_ascii=False)


def sha(s: str) -> str:
    return hashlib.sha256(s.encode('utf-8')).hexdigest()


def request_fingerprint(payload: dict, endpoint: str) -> dict:
    # payload must be the wire-level body, not your pre-call dataclass
    body = canon(payload)
    return {
        'fingerprint': sha(endpoint + '\n' + body),
        'endpoint': endpoint,
        'body_sha': sha(body),
        'model_field': payload.get('model'),
        'temperature': payload.get('temperature'),
        'max_tokens': payload.get('max_tokens'),
        'seed_present': 'seed' in payload,
    }

Two notes before you copy this. Including the endpoint in the hash is deliberate, because the same body sent to a different host is a different experiment. And seed_present records intent only: sending a seed field does not guarantee deterministic decoding, so treat it as evidence of what you asked for.

Step 2 — One ledger row per call

Append-only JSONL beats a database for this, because you can grep it, ship it between hosts, and diff it in a pull request. Each row carries the fingerprint, the run identity, and enough response metadata to explain what came back.

# fp_ledger.py, continued
def record(ledger_path: str, fp: dict, response: dict, latency_ms: int) -> None:
    text = response['choices'][0]['message']['content']
    row = {
        'recorded_at': time.strftime('%Y-%m-%dT%H:%M:%SZ', time.gmtime()),
        'run_id': os.environ.get('RUN_ID', str(uuid.uuid4())),
        'host': os.uname().nodename,          # Unix only; swap for platform.node()
        **fp,
        'served_model': response.get('model'),
        'response_id': response.get('id'),
        'usage': response.get('usage'),
        'latency_ms': latency_ms,
        'output_sha': sha(text),
        'output_len': len(text),
    }
    with open(ledger_path, 'a', encoding='utf-8') as fh:
        fh.write(canon(row) + '\n')

Then the caller stays boring, which is the point. It reads the prompt from disk, builds one explicit body, and never lets a framework hide a default on you.

# call.py
import os
import time
from openai import OpenAI
from fp_ledger import request_fingerprint, record

client = OpenAI(base_url=os.environ['BASE_URL'], api_key=os.environ['API_KEY'])

payload = {
    'model': os.environ['MODEL'],
    'temperature': 0,
    'max_tokens': 800,
    'messages': [{'role': 'user', 'content': open('prompt.txt').read()}],
}

fp = request_fingerprint(payload, os.environ['BASE_URL'])
started = time.perf_counter()
resp = client.chat.completions.create(**payload)
record('ledger.jsonl', fp, resp.model_dump(), int((time.perf_counter() - started) * 1000))

Run it a few times on the first host and keep the file:

RUN_ID=hostA-1 python call.py
RUN_ID=hostA-2 python call.py
mv ledger.jsonl ledger-a.jsonl

Step 3 — Replay on a second host

A second host is where this workflow earns its keep, because most drift is environmental rather than model-side. I used the free server option that comes with MonkeyCode's free tier, alongside its free model access, to run that second leg. Disclosure: This article was prepared as part of MonkeyCode's product outreach. I have not measured quotas, hardware, or uptime there, so treat those as unknown when you plan your own run.

Treat that host as disposable. Push the script, run the loops, and pull the ledger back immediately, because anything you leave on a shared box is a copy you do not control.

export BASE_URL=... MODEL=... API_KEY=...
for i in 1 2 3; do RUN_ID=hostB-$i python call.py; done

# which fingerprints did each host actually produce?
comm -3 <(jq -r .fingerprint ledger-a.jsonl | sort -u) \
        <(jq -r .fingerprint ledger-b.jsonl | sort -u)

comm -3 prints left-column-only lines unindented and right-column-only lines indented. If that output is empty, both hosts sent byte-identical bodies, and any remaining difference is not your code.

Step 4 — Compare structure, not prose

Prose diffs on generated text produce noise, because two valid answers to the same prompt are rarely the same string. Compare the parse tree, the schema, or a numeric tolerance instead.

import ast
import json


def shape(text: str, kind: str):
    if kind == 'json':
        try:
            return ('json', json.dumps(json.loads(text), sort_keys=True))
        except json.JSONDecodeError as exc:
            return ('json_error', str(exc))
    if kind == 'python':
        try:
            return ('ast', ast.dump(ast.parse(text), annotate_fields=False))
        except SyntaxError as exc:
            return ('syntax_error', f'line {exc.lineno}')
    raise ValueError(kind)

Pair that with a small checklist: does it parse at all, does it validate against your schema, does it still import, and do the numbers land within tolerance. A shape mismatch is a real signal; a different variable name usually is not.

What the ledger catches that a text diff cannot

• Same fingerprint, different served_model: the alias under you moved, and your prompt never changed.
• Different fingerprint, same prompt file: your client injected a new default between runs.
• Same fingerprint, much larger output_len on one host: truncation limits differ somewhere in the path.
• Same fingerprint and same served model, different shape: this is the one case worth an afternoon.
• No shared fingerprint at all: you were never comparing two runs of the same thing.

When a free tier is the right tier

Situation	Free tier a fit?	Why
Throwaway spike or one-off script	Yes	A wrong answer costs a rerun, not a rollback
Prompt variant comparison by structure	Yes	You need volume of runs, not a contract
Long-lived pipeline with an SLA	No	You cannot pin what you do not control
Regulated or customer-identifying data	No	You need a data-processing agreement, not an endpoint
Artifact you must reproduce in a year	Partial	Record the fingerprint now; the endpoint may be gone later

What I deliberately did not measure

I am not quoting throughput, latency percentiles, or token quotas here, and that is a choice rather than an omission. Numbers taken from a shared endpoint on one afternoon go stale fast, and they describe my path rather than your workload. What travels is the method: hash the request, keep the ledger, replay on a second host, compare shape.

Limitations and who should not use this

• Fingerprints find differences; they do not remove them. Drift on a shared endpoint is a fact you observe, not a bug you patch.
• Pinning is only as strong as your control surface. If the provider changes the model behind an alias, your ledger records the change rather than preventing it.
• Keep the canonical ledger in your own repository. A disposable host is a convenient runner, not your system of record.
• Skip this entirely if you need contractual uptime, verified data residency, or byte-for-byte reproducibility for audit.

The one habit I would keep

Before comparing two model outputs, compare the two fingerprints, and let the ledger tell you whether you ran the same experiment. If you want to try the second-host step without setting up billing first, the free tier is a reasonable place to start, and the harness above is short enough to read in one sitting. The output will keep varying; your record of what you sent does not have to.