48-Hour Field Notes: The Label Flipped. The Example Folder Was Never Sorted.

I spent the last two days chasing a label that flipped without any change I could see in the source. The classifier prompt lived in a folder of short examples, and the harness loaded that folder before each call. On my laptop the boundary case stayed review, which matched the note I had written beside the fixture. After I moved that same tree to a clean remote runtime, the case came back allow, and I had no diff.

Was the model quietly sampling, or had I changed the surrounding context without noticing a single file edit? I kept the client temperature at zero, and I reused one endpoint so the weights would not become the variable. The files had matching hashes, the schema had the same fields, and the system text had the same bytes. So why did the decision text move when the tree itself had not changed at all?

What I thought had broken

My first guess was the usual one, because a moving label still feels like a model problem to me. I re-read the system text, then I re-read the fixture, and then I re-read both again looking for a hidden instruction. Nothing in those files had changed between the laptop run and the remote run I was comparing. Have you ever burned a day on the model when the bug was sitting in os.listdir the whole time?

I also suspected a stale process, because yesterday's field notes were full of sockets and retries that outlived the script. This time the process table was boring, and a fresh interpreter still flipped the label on the same case. The schema check passed on both sides, so a type mismatch was not the story I was hunting either. I needed a control that would tell me whether the prompt bytes were actually identical across both runs.

Holding the runtime still

I wanted a machine that did not inherit my shell, my locale, or the packages I had pip-installed by habit. MonkeyCode's free server option gave me that clean runtime, and its free model access let me call one model path without mixing providers. Disclosure: This article was prepared as part of MonkeyCode's product outreach. I am not claiming a quota, a hardware shape, or a promise that those free options stay fixed.

Those details were not part of what I verified in this note, so I will not invent them here. What I did verify was narrower, and it is the part worth copying into your own harness. I copied the repo, created a virtual environment, and ran the same harness commit on both machines. The only intentional difference was the filesystem that the directory walk happened to see on each host.

Would a label stay put if I forced the prompt bytes to match, even when the walk order did not? Treat the install lines below as your own lockfile, not as a version pin copied from me. I am not publishing a dependency set, because I did not freeze one anywhere in this note. The commands are the workflow I would repeat, and the assertions are the part that should fail loudly.

python -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
python -m pytest tests/test_prompt_fingerprint.py -q
python scripts/fingerprint_prompt.py examples/ boundary_case.json

The artifact: hash the prompt before you blame the model

The useful check is small enough to keep beside the client, and it does not require a new framework. Walk the example directory, record the relative paths in the order you would send them, and hash those bytes. If that digest moves, you do not have a model incident yet, no matter how strange the label looks. You have two different prompts that happen to share one folder and one quiet filename list.

The loader that hides the bug

Here is the loader I would not ship, because it trusts directory order more than it trusts you. It feels harmless on a laptop where the walk happens to look sorted every single time. That accident is how this bug survived a full day of file-hash checks on my side. Copy it only as the failing example, not as the helper you import into the harness.

from pathlib import Path

def load_examples_unsorted(root: Path) -> list[tuple[str, str]]:
    rows = []
    for path in root.iterdir():
        if path.suffix != ".md":
            continue
        rows.append((path.name, path.read_text(encoding="utf-8")))
    return rows

Path.iterdir does not promise an order, and a different filesystem is allowed to surprise you on a clean host. My laptop happened to return names in a stable-looking sequence, which made the bug invisible for hours. The clean server did not do that, so the few-shot block became a permutation of the same files. Same files, different prompt, different label, and none of that required a model change at all.

The loader I would repeat

The loader I would repeat sorts on a relative POSIX path, then joins the parts with a delimiter that stays visible. A hidden delimiter is just another way to make two prompts look alike in a log. I hash the system text, the ordered examples, and the case body as one blob of bytes. This sketch is runnable lab code, not a benchmark, and not a claim about any vendor's sampler.

import hashlib
from pathlib import Path

def load_examples_sorted(root: Path) -> list[tuple[str, str]]:
    paths = sorted(p for p in root.rglob("*.md") if p.is_file())
    return [
        (p.relative_to(root).as_posix(), p.read_text(encoding="utf-8"))
        for p in paths
    ]

def prompt_fingerprint(system: str, examples: list[tuple[str, str]], case: str) -> str:
    parts = [system, "\n---EXAMPLES---\n"]
    for name, body in examples:
        parts.append(f"\n### {name}\n{body}")
    parts.append("\n---CASE---\n")
    parts.append(case)
    blob = "".join(parts).encode("utf-8")
    return hashlib.sha256(blob).hexdigest()

The hash covers the system text, the example order, and the case body you intended to send. It does not cover hidden client defaults, so print those too if your SDK adds them quietly. A regression test belongs next to the loader, or the next clean machine will teach you this lesson again. If the assertion fails, stop, and do not open a model-quality ticket while prompt identity is still moving.

from pathlib import Path

def test_example_order_is_path_sorted(tmp_path: Path) -> None:
    (tmp_path / "b.md").write_text("second\n", encoding="utf-8")
    (tmp_path / "a.md").write_text("first\n", encoding="utf-8")
    names = [name for name, _ in load_examples_sorted(tmp_path)]
    assert names == ["a.md", "b.md"]

What broke, hour by hour

I am writing the timeline as notes from the chase, not as a measured outage with a counter. I did not keep a production metric, and I will not invent one to make the story sound tighter. The hours below are how the confusion unfolded while I was still blaming the label itself. Read them as a sequence of wrong turns, not as a service-level report from a team.

1. In the first six hours I compared per-file hashes and wrongly decided that every input matched.
2. From hour six to fourteen I toggled temperature and resent the case, which only added noise to a bad prompt.
3. From hour fourteen to twenty I printed each walked name and finally saw b.md ahead of a.md on the server.
4. From hour twenty to thirty I hashed both assemblies, watched the digests diverge, and dropped the regression story.
5. From hour thirty to forty-eight I sorted by relative path, re-hashed, and only then compared labels with a straight face.

Would I have seen it faster if the log line included the digest on every call I made? Yes, and I would keep that field even after the path sort had already landed in the loader. A single digest turns a vibe about model quality into a diff you can paste into a ticket. That is the whole method, and the rest is just refusing to skip the check when you are tired.

A small decision table

Use the table before you change a prompt, a model path, or the server you are blaming. I built it after the chase, as a gate I can run, not as a retrospective slogan for the notes. The middle row is the one I almost skipped, because a matching hash feels like permission to escalate. It is not permission to escalate, and it is not proof that the next token will repeat for you.

What you observed	Check next	Do not conclude yet
Label flipped after a machine move	Hash the assembled prompt, not each file alone	The model got worse
Hash matches and the label still moves	Log temperature, seed if the client has one, and SDK defaults	The folder walk is guilty
Hash differs while file hashes match	Print the path order you actually sent	You need a different model
Sorted loader and matching hash, still unhappy	Inspect the case text and the rubric	Directory order is still the bug

A matching hash does not prove the next token will repeat on a later call you make. Sampling, an optional seed, or a silent client default can still move the text you read. I only claim the opposite direction, and I want that limit stated in the notes beside the table. If the hash differs, you are not looking at the same question, so stop comparing the labels.

What I would repeat

These are the steps I would repeat on the next flip, before I touch the rubric or the model path. None of them require a dashboard, and none of them require a story about which vendor drifted. They do require you to log the exact blob you sent, including the example order you chose. If you cannot reconstruct that blob on the next morning, you cannot honestly reconstruct the bug.

• Print the prompt digest beside every label, including the dry runs you do not plan to keep around.
• Sort example paths with as_posix so a Windows checkout and a Linux server do not disagree on separators.
• Fail the test if two loads of an unchanged tree produce two different digests in one run.
• Keep the system text inside the hash, because a one-line rubric edit is still a different prompt.
• Compare labels only after the digests match, and write that rule where the next person will see it.

I would also repeat the clean-runtime step whenever a bug smells environmental rather than purely textual. A laptop that has been yours for months hides order, locale, and leftover files you forgot you wrote. A fresh server makes those assumptions visible, which is the actual point of leaving your home machine. Do you really want the next flip to depend on a directory walk you never printed in the log?

Who should skip this

Skip this approach if your examples are chosen dynamically for each case, because a global sort is the wrong contract. Skip it if you need a legal or safety review of model output, because a digest does not review meaning. Skip it if identical bytes must yield identical tokens for a control you have to sign yourself. A temperature set to zero is not a portability guarantee I am willing to put my name on.

Also skip the free-server shortcut if your data cannot leave the machine you already trust with it. Free model access was useful here only as a stable calling path while I varied the prompt assembly. It did not classify the bug, and it did not replace the hash I should have printed first. If your real constraint is cost governance, data residency, or a pinned model build, read those terms yourself.

What I would keep

The label was never the first fact I should have trusted during this two-day chase at all. The first fact was whether both runs asked the same question, in the same order, with the same bytes. Once the digest sat still, the remaining disagreement was small enough to read as a product question. If that clean runtime already sits in the account you use, keep the fingerprint and let the ticket wait.