48-Hour Field Notes: I Compared Two Runs Across Two Days Without Checking Which Model Served Them

Two runs, forty-eight hours apart, and a latency drop big enough that I started writing the summary before the second log had finished flushing. Then I printed the one metadata column I had been ignoring since day one. Is that a measurement bug, or is it a discipline bug wearing a lab coat?

This is another entry in a small series where I break my own tooling over two days and write down the damage while it still stings. The target this time was a prompt-variance probe I wanted to run on free capacity, because requesting a budget line takes longer than building the thing.

The Setup: One Question, Two Unnamed Unknowns

The question was narrow. After I rewrote a system prompt, did identical inputs produce identical outputs more often than before? That is a variance question, not a leaderboard question, and it needs volume rather than expensive tokens.

I used MonkeyCode's free model access and its free server option so the loop could run without a spend approval, which is the entire reason day two existed at all. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The availability claims are the operator's; everything about the harness below is mine to defend or retract.

Before writing a line of code I listed the unknowns I was not measuring, and both of them came back to bite me within the window:

1. Which model revision actually served each request, and whether that stayed constant.
2. How long a long-running process survives on shared, free compute before something reclaims it.

Neither is exotic. Both are the kind of thing you only notice when you decide you want to publish a number.

Day 1: The Harness That Saved Me From Myself

I refused to write a benchmark first. I wrote a logger, because a benchmark without raw rows is a story I am telling myself. Every request appends one JSON line, and nothing is aggregated until the log is closed.

# probe.py
import hashlib, json, time
from pathlib import Path
from datetime import datetime, timezone

LOG = Path('runs/probe.jsonl')
LOG.parent.mkdir(parents=True, exist_ok=True)

def fp(obj) -> str:
    blob = json.dumps(obj, sort_keys=True).encode('utf-8')
    return hashlib.sha256(blob).hexdigest()[:12]

def probe(client, prompt: str, run_id: str, repeat: int) -> dict:
    t0 = time.perf_counter()
    resp = client.chat(prompt=prompt, temperature=0, seed=7)
    record = {
        'run_id': run_id,
        'repeat': repeat,
        'prompt_fp': fp({'p': prompt}),
        'text_fp': fp({'t': resp.text}),
        'model_field': resp.meta.get('model'),
        'revision_field': resp.meta.get('revision') or resp.meta.get('system_fingerprint'),
        'tokens_in': resp.usage.get('input_tokens'),
        'tokens_out': resp.usage.get('output_tokens'),
        'latency_ms': round((time.perf_counter() - t0) * 1000, 1),
        'utc': datetime.now(timezone.utc).isoformat(),
    }
    with LOG.open('a', encoding='utf-8') as fh:
        fh.write(json.dumps(record) + '\n')
    return record

Two details are load-bearing and easy to skip. The text_fp field hashes the raw output text so I can count distinct answers instead of eyeballing them, and revision_field reads whatever identifier the vendor echoes back, which differs per SDK. If your client library exposes no such field, that absence is itself a finding you should write down.

The probe loop is deliberately boring: one prompt, twenty repeats, fixed temperature and seed, append-only log. Boring loops are the only ones I trust after 5pm.

mkdir -p runs
python probe.py --run day1 --repeats 20

Day 2: The Log Disagreed With My Summary

I reran the same command the next morning, changed one file, and ran the analyzer before reading any average. The first line it printed was a refusal, not a p50.

# analyze.py
import json, statistics
from collections import defaultdict
from pathlib import Path

rows = [json.loads(l) for l in Path('runs/probe.jsonl').read_text().splitlines() if l.strip()]
by_key = defaultdict(list)
for r in rows:
    by_key[(r['run_id'], r['model_field'], r['revision_field'])].append(r)

for key, rs in by_key.items():
    lat = sorted(r['latency_ms'] for r in rs)
    p95 = lat[max(0, int(len(lat) * 0.95) - 1)]
    outs = {r['text_fp'] for r in rs}
    print(f'{key} n={len(rs)} p50={statistics.median(lat)}ms p95={p95}ms distinct_outputs={len(outs)}')

revs = {(r['model_field'], r['revision_field']) for r in rows}
if len(revs) > 1:
    raise SystemExit(f'STOP: {len(revs)} revisions in one log; do not compare across them')

That guard is the whole lesson. My day-one rows and day-two rows were grouped under different revision identifiers, which means the two "runs" were different systems wearing the same prompt. The numbers I was about to quote described a change of supplier, not a change of my prompt.

What the dry run prints

The digits below come from a stubbed client that fabricates responses, so read the shape and ignore the values. Real logs from your own endpoint should replace them before any claim leaves your draft folder.

('day1', 'stub-model', 'rev-a') n=20 p50=812.0ms p95=1490.0ms distinct_outputs=1
('day2', 'stub-model', 'rev-b') n=20 p50=604.5ms p95=1102.0ms distinct_outputs=3
STOP: 2 revisions in one log; do not compare across them

The distinct_outputs column matters more than the milliseconds, honestly. One stable answer on day one and three different answers on day two is a variance signal, and a variance signal that arrives alongside an identifier change is not attributable to my prompt.

The Second Break: Free Compute Does Not Owe You Continuity

Somewhere in the middle of the run my process stopped existing, and I only know that because the log ended mid-batch rather than at repeat twenty. Shared or free capacity is not a contract, so the fix is not a longer timeout; the fix is a job that can be picked up.

The runner below is at-least-once by design, which is fine here because the probe is read-only. Every finished item is flushed with os.fsync so an abrupt stop loses at most the item in flight, and the resume path skips anything already marked ok.

# runner.py
import json, os
from pathlib import Path

STATE = Path('runs/state.jsonl')

def load_done() -> set:
    if not STATE.exists():
        return set()
    done = set()
    for line in STATE.read_text().splitlines():
        if line.strip():
            rec = json.loads(line)
            if rec.get('status') == 'ok':
                done.add(rec['item_id'])
    return done

def append_state(item_id: str, status: str, **extra) -> None:
    line = json.dumps({'item_id': item_id, 'status': status, **extra}) + '\n'
    with STATE.open('a', encoding='utf-8') as fh:
        fh.write(line)
        fh.flush()
        os.fsync(fh.fileno())

def run(items, handler) -> None:
    done = load_done()
    for item in items:
        if item['id'] in done:
            continue
        try:
            append_state(item['id'], 'ok', result=handler(item))
        except Exception as exc:
            append_state(item['id'], 'error', error=repr(exc))

Restarting is then just the same command, and the log tells you how much was real work versus replay:

python runner.py --queue runs/queue.jsonl
wc -l runs/state.jsonl

The Decision Table I Now Keep in the Repo

After two days of this I wrote down when pinning matters, because I will absolutely forget by next quarter. The table is short on purpose.

Situation	Verify revision identifier?	Rerun both sides?	Note
Same-day prompt iteration	Record only	No	Compare within one revision bucket
Comparing two calendar days	Mandatory	Yes, if identifiers differ	Never quote across buckets
Changing temperature or seed	Record only	Yes	The setting is part of the treatment
Latency claims	Record only	Yes	Measure client-side, include queue time
Any shared or free compute	Record only	Depends	Assume interruption, checkpoint every item
Publishing a number	Mandatory	Yes	Include the raw log or drop the claim

What I Would Repeat, and What I Would Not

What I would repeat without hesitation: the JSONL-first habit, the analyzer that exits non-zero instead of printing a misleading mean, and the fsync on state writes. Each of those cost me maybe fifteen minutes and saved a day.

What I would not repeat: starting the probe before deciding what a valid comparison even looks like. I recorded the model identifier from the very first request and still did not put a guard on it until the second day, which is a genuinely embarrassing order of operations.

Finally, the boring one. I should have run the full loop end to end against two synthetic items before spending any window on real work. If you cannot resume from a killed process in under a minute of wall time, you are not ready to start a multi-day run on capacity you do not control.

Limitations, and Who Should Skip This

This workflow is honest but narrow, so please do not over-read it.

• Latency numbers from shared free compute are not an SLO. They describe a noisy shared path on one afternoon, and nothing more.
• Determinism claims need a seed you actually control. If your endpoint ignores the seed, count distinct outputs and stop pretending you have reproducibility.
• Skip this entirely if you need audited benchmarks. Pinned hardware, a fixed revision, and a signed methodology are a different project with a different budget.
• Skip it if your product is intentionally nondeterministic. A variance probe on a creative-writing assistant measures the product, not your prompt.
• Do not use it to compare vendors. Two logs from two providers will almost never share a revision namespace, and the comparison will be meaningless in exactly the way mine was.

If you want to try the two-day loop before asking anyone for a budget, MonkeyCode's free model access and free server option were enough for me to get the harness running, and I would start with the raw JSONL log rather than the summary table. Just add the revision guard before the first request, not after the first surprise.