I Shuffled the Judge Prompt. The Scores Should Have Held.

Your eval is not a measurement if the same trace gets a different grade when you move it down the list. I stopped arguing about model quality and started measuring judge flip-rate. That number is uglier than a single accuracy score, and it is the one I actually ship against.

We keep building agents and then grading them with another model, usually the cheapest one that will answer JSON on a Tuesday. Does that sound like engineering, or like asking a tired intern to re-grade the same stack of papers after someone shuffled the pile? I wanted a protocol that fails loudly when the intern is guessing.

The last week of DEV commentary kept circling a real itch: we outgrew the tests we brag about. I am not here to litigate vibe coding. I am here because my scoring harness was agreeing with itself until I changed the order of the fixtures. Order should be a no-op. It was not.

So I wrote a judge that I can lie to on purpose. If a stub that keys off list position can look “consistent” in a naive run, a free model can too. The trick is to run the same traces through the same rubric more than once, with the deck cut differently each time, and count the flips.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. I needed a lot of cheap, boring repeats—same rubric, same traces, different order. MonkeyCode’s free model access and free server option are relevant for that grind if you treat the endpoint as a judge, not as a coauthor. I am not claiming a quota, a model name, or a latency number I did not measure. The interesting part is the flip-rate script.

Here is the fixture set I actually version. Three traces, labeled in git, not in my memory. One is a clean tool success. One is a fluent wrong answer. One is a partial apply that talks like a merge.

# fixtures.py — canned traces, not production logs
FIXTURES = [
    {
        "id": "t_ok",
        "task": "Rename getUser to get_user in src/api.py and keep tests green.",
        "trace": "Applied patch to src/api.py. Ran pytest: 12 passed.",
        "gold": "pass",
    },
    {
        "id": "t_fluent_fail",
        "task": "Rename getUser to get_user in src/api.py and keep tests green.",
        "trace": "Refactored naming across the service layer. Ready to ship.",
        "gold": "fail",
    },
    {
        "id": "t_partial",
        "task": "Rename getUser to get_user in src/api.py and keep tests green.",
        "trace": "Updated the function. Tests still import getUser. Explained why that is fine.",
        "gold": "fail",
    },
]

Gold labels live next to the traces. If I let the model invent the labels, I am scoring a mirror. You already know how that movie ends, right?

The rubric is short on purpose. Long rubrics make judges sound careful while they still wobble. I ask for a label and a one-line reason, then I throw the reason away for the metric. Reasons are for humans. Flip-rate is for the harness.

RUBRIC = """Score the agent trace against the task.
Return JSON only: {"id": str, "label": "pass" or "fail", "reason": str}.
pass = the trace shows the requested change and a test run that could have caught a miss.
fail = missing change, missing tests, or a story that substitutes for evidence.
"""

Now the part people skip. I do not call the judge once. I call it through permutations. Three fixtures is six orders. That is small enough to read and large enough to catch a positional toady.

import hashlib, itertools, json, os, urllib.request
from fixtures import FIXTURES, RUBRIC

ENDPOINT = os.environ["JUDGE_URL"]  # your free server or any OpenAI-shaped chat URL
MODEL = os.environ.get("JUDGE_MODEL", "default")

def chat(messages):
    body = json.dumps({"model": MODEL, "messages": messages, "temperature": 0}).encode()
    req = urllib.request.Request(
        ENDPOINT, data=body, headers={"Content-Type": "application/json"}
    )
    with urllib.request.urlopen(req, timeout=60) as resp:
        data = json.loads(resp.read().decode())
    return data["choices"][0]["message"]["content"]

def parse_label(raw, fallback_id):
    try:
        obj = json.loads(raw)
        label = obj.get("label", "fail")
        return fallback_id, "pass" if label == "pass" else "fail"
    except Exception:
        return fallback_id, "fail"  # unparseable output is a fail, not a retry poem

Temperature zero is not a personality. It is a request. Free endpoints still drift. That is the point of repeating the trial, not a reason to skip it. If your client retries on parse failure, cap it. Infinite repair is how a judge learns to write JSON and forget the grade.

def judge_batch(order):
    blob = json.dumps([{"id": t["id"], "task": t["task"], "trace": t["trace"]} for t in order])
    raw = chat([
        {"role": "system", "content": RUBRIC},
        {"role": "user", "content": blob},
    ])
    # Expect one JSON object per trace. If the model merges them, every id is fail.
    labels = {}
    for t in order:
        labels[t["id"]] = "fail"
    try:
        parsed = json.loads(raw)
        rows = parsed if isinstance(parsed, list) else [parsed]
        for row in rows:
            i = row.get("id")
            if i in labels:
                labels[i] = "pass" if row.get("label") == "pass" else "fail"
    except Exception:
        pass
    return labels

def flip_rate(runs):
    ids = [t["id"] for t in FIXTURES]
    flips = 0
    pairs = 0
    for i in ids:
        labels = [run[i] for run in runs]
        pairs += 1
        if len(set(labels)) > 1:
            flips += 1
    return flips / pairs

I also keep a dishonest judge in the repo. It is a flashlight. If I wire the pipeline to a function that passes whatever arrived last, the flip-rate should scream. If it does not scream, I am measuring the wrong thing.

def positional_stub(order):
    # Last item always "pass". A real metric must punish this.
    labels = {t["id"]: "fail" for t in order}
    labels[order[-1]["id"]] = "pass"
    return labels

def identity_stub(order):
    # Stable on id. Flip-rate must be 0.0 even when shuffled.
    return {t["id"]: ("pass" if t["id"] == "t_ok" else "fail") for t in order}

if __name__ == "__main__":
    orders = list(itertools.permutations(FIXTURES))
    stub_flips = flip_rate([positional_stub(list(o)) for o in orders])
    id_flips = flip_rate([identity_stub(list(o)) for o in orders])
    print(f"positional_stub flip_rate={stub_flips:.2f}")  # expect 1.00
    print(f"identity_stub flip_rate={id_flips:.2f}")      # expect 0.00

Run that before you spend a single token. I mean it. positional_stub flip_rate=1.00 and identity_stub flip_rate=0.00 are the only “benchmarks” in this article that I will stand behind without a live endpoint, because they are functions of the fixtures. If those two lines move, your math is broken and you should not grade a model yet.

The live path is the same loop with judge_batch. I do not paste a leaderboard from a free endpoint I did not freeze. You should not either. Print the per-id label vectors. If t_fluent_fail is pass in two orders and fail in four, you do not have a score. You have a weather report.

export JUDGE_URL="http://127.0.0.1:8000/v1/chat/completions"
python -c "from fixtures import FIXTURES; print(len(FIXTURES), 'fixtures')"
python judge_flip.py

What does “good enough” look like for a cheap judge? I use a boring gate, not a press release. Flip-rate on this three-trace deck must be 0.00 across all six orders. Agreement with gold must be 3/3 on every order, not on average. Averages hide the fluent miss, and the fluent miss is the whole crime.

Think of it like a kitchen scale. If the same bag of flour reads 500g, then 470g, then 510g because you rotated the bowl, you do not publish a recipe. You fix the scale. Free inference is useful here because repeats are the experiment. One beautiful grade is a vibe. Six ugly grades are data.

Where it breaks is predictable if you have ever watched a model try to be helpful. Batch JSON collapses into one object. The last trace inherits a pass because the prose sounded finished. The rubric gets paraphrased in the reason field and ignored in the label. A free server that is busy will timeout, and if you retry without a bound you will train yourself to accept the first parseable sentence. Bound the retries. I cap parse repair at one. After that it is fail. Silence is a grade.

There is a second failure I only caught when I hashed the prompt. I had a comment in the user blob that said “expected fail.” The judge, being a good student, agreed. Strip your gold from the request. If the model can see the answer, you are not evaluating an agent. You are evaluating obedience.

def prompt_fingerprint(order):
    body = json.dumps([t["id"] for t in order]) + RUBRIC
    return hashlib.sha256(body.encode()).hexdigest()[:12]

I log that fingerprint next to each run. When someone “improves” the rubric on a Friday, the hash moves and the old flip-rate is void. That is the whole discipline. Git the rubric. Git the fixtures. Do not git a screenshot of a chat.

Limitations, because this protocol is easy to oversell. Three traces will not bless a product. Temperature zero is not determinism. A free server is not a frozen judge checkpoint. If you need a number for a paper, a customer report, or a safety case, stop. Hire a stable judge, pin the revision, and put humans on the fluent failures. This workflow is for catching a harness that lies to its owner.

Who should not use it? Anyone averaging labels across shuffles and calling the mean “accuracy.” Anyone whose agent can mutate the fixtures. Anyone hiding the endpoint behind a retry loop that also rewrites the rubric. And anyone who thinks a free tier is a reason to skip gold labels. Free means you can afford the repeats. It does not mean the repeats are optional.

I still want cheap judges. I want them for the same reason I want cheap unit tests: I will actually run them. If you already have a free model endpoint or a free server sitting around, point JUDGE_URL at it and make the stubs fail first. The CTA is the script, not a tour. If the flip-rate is not zero, you do not have evaluation. You have a conversation that learned to export CSV.