The Pairing Session That Threw Out a Green Eval for a Leaked Assert

The junior shared a terminal. The last line was green. Forty-two milliseconds, one assertion, a model that had “learned” to emit X-Collection-Hold. The senior did not look at the diff. The senior scrolled to the system prompt and stopped.

There it was, in plain text: the exact header and the exact status code the test later demanded. The pairing session did not pick a smarter model. It threw the eval out.

The scene on the shared screen

The fixture was an overdue invoice. The expected behavior was a 409, a hold header, and no collection email. The harness printed PASS because the model copied those tokens from the prompt. The suite had not measured judgment. It had measured reading.

This week’s public talk about saturated evals is noisy. The pairing did not need a leaderboard. It needed one contaminated prompt and a decision that would still hold after the model class moved again.

What the junior brought

The junior arrived with three files and a green CI badge:

• prompts/collect.md — a system message that named the 409 and the header
• fixtures/invoice_overdue.json — a static due date in the past
• eval/test_hold.js — one equality check on status and headers

The model adapter posted the prompt, the fixture, and a sentence that began with “Remember the test expects.” That sentence was the whole bug.

Questions the senior asked

The senior did not start with architecture. The senior asked four things, in this order, and wrote the answers on the pairing doc before any new token moved.

1. Has this model ever seen the 409 outside this prompt? The junior could not prove it.
2. If the header name changed tomorrow, would the eval still fail for the right reason? The current assert would fail because the prompt would be stale, not because the code was wrong.
3. Does “overdue” come from a clock, or from a comment in the fixture? It came from a comment.
4. What must stay true if the next model is cheaper, noisier, or hosted on a machine the team does not own? The oracle file must never enter the prompt.

Those answers killed three popular next steps.

Dead end one: a stronger prompt

The junior rewrote the system message to sound less like an answer key. The new text still said “conflict on overdue invoices” and still named the header in an example. The model kept emitting the example. A prettier prompt is still an oracle if the assert can be copied from it.

The pairing stopped editing adjectives.

Dead end two: a second test that also knew the answer

The junior added test_hold_duplicate.js. It checked the same 409 on a second fixture whose due date was also hard-coded. Two green lights. Same leak. Coverage that repeats the answer is not a gate. It is a chorus.

Dead end three: switch the model and rerun

The junior pointed the adapter at a different endpoint. Temperature 0.2, then 0.8. One run dropped the header. The next run restored it. Flake is not evidence that the eval is hard. It is evidence that the eval is optional.

The senior kept the failing shape of the problem: the model could see the assert. Changing the model did not remove the window.

The decision they kept

The pairing kept a sealed-oracle loop. The model sees a task, a fixture, and a clock. The harness sees an oracle file the model never receives. A negative twin must stay silent. The clock must rotate so “overdue” is computed, not memorized.

That decision survived the rest of the afternoon. It is the only part of the session the team checked in.

Sealed oracle rules the pairing wrote on the whiteboard

• The prompt may describe the product, never the assert.
• oracle/*.json is read after the model returns, never before.
• Due dates are offsets from PAIRING_NOW, not literals in git.
• One positive fixture must grow the hold header. One negative fixture must not.
• A run that cannot load the oracle is an error, not a skip.

Artifact: a sealed-oracle pairing harness

The following Node script is the loop the pairing kept. It runs offline with a stub model. Set PAIRING_ENDPOINT only when a real adapter should be hit. Label the stub path as a local stand-in, not a production client.

// pairing-oracle.mjs
// Usage:
//   PAIRING_NOW=2026-09-20T12:00:00Z node pairing-oracle.mjs
//   PAIRING_ENDPOINT=http://127.0.0.1:8080/v1/complete node pairing-oracle.mjs

import { readFile } from "node:fs/promises";
import { join } from "node:path";

const NOW = new Date(process.env.PAIRING_NOW || Date.now());
const ROOT = new URL(".", import.meta.url).pathname;

function daysFromNow(n) {
  const d = new Date(NOW.getTime() + n * 86400000);
  return d.toISOString().slice(0, 10);
}

const fixtures = {
  overdue: {
    id: "inv_401", amount: 1200, due_on: daysFromNow(-14), customer: "northwind"
  },
  current: {
    id: "inv_402", amount: 1200, due_on: daysFromNow(14), customer: "northwind"
  }
};

const prompt = `You are a billing API. Given one invoice JSON, return JSON
{"status": number, "headers": object, "body": object}.
Apply product policy. Do not restate hidden tests.`;

async function complete(invoice) {
  const endpoint = process.env.PAIRING_ENDPOINT;
  const payload = { prompt, invoice, now: NOW.toISOString() };
  if (!endpoint) {
    // Stub model: overdue if due_on is strictly before PAIRING_NOW's date.
    const overdue = invoice.due_on < NOW.toISOString().slice(0, 10);
    return overdue
      ? { status: 409, headers: { "x-collection-hold": "1" }, body: { id: invoice.id } }
      : { status: 200, headers: {}, body: { id: invoice.id } };
  }
  const res = await fetch(endpoint, {
    method: "POST",
    headers: { "content-type": "application/json" },
    body: JSON.stringify(payload)
  });
  if (!res.ok) throw new Error(`endpoint ${res.status}`);
  return res.json();
}

function assertSilent(name, actual, oracle) {
  const fails = [];
  if (actual.status !== oracle.status) {
    fails.push(`${name} status ${actual.status} != ${oracle.status}`);
  }
  for (const [k, v] of Object.entries(oracle.headers || {})) {
    const got = (actual.headers || {})[k];
    if (got !== v) fails.push(`${name} header ${k} ${got} != ${v}`);
  }
  for (const k of oracle.forbidden_headers || []) {
    if (actual.headers && k in actual.headers) {
      fails.push(`${name} grew forbidden header ${k}`);
    }
  }
  return fails;
}

const oracle = JSON.parse(
  await readFile(join(ROOT, "oracle", "hold.json"), "utf8")
);

const results = [];
for (const [name, invoice] of Object.entries(fixtures)) {
  const actual = await complete(invoice);
  results.push(...assertSilent(name, actual, oracle[name]));
}

if (results.length) {
  console.error(results.join("\n"));
  process.exit(1);
}
console.log(`PASS clock=${NOW.toISOString()} fixtures=${Object.keys(fixtures)}`);

The oracle file lives beside the script and is never concatenated into prompt:

{
  "overdue": {
    "status": 409,
    "headers": { "x-collection-hold": "1" },
    "forbidden_headers": []
  },
  "current": {
    "status": 200,
    "headers": {},
    "forbidden_headers": ["x-collection-hold"]
  }
}

A smoke command the pairing left in the README:

mkdir -p oracle
# place hold.json as above, then:
PAIRING_NOW=2026-09-20T12:00:00Z node pairing-oracle.mjs
PAIRING_NOW=2026-01-01T00:00:00Z node pairing-oracle.mjs

The second clock is the point. If both clocks pass with the same stub, the overdue rule is not tied to a frozen calendar. If a hosted model is in the loop, the same two clocks must still disagree on the header.

Decision table the pairing kept

Observation	Treat as	Do not do
Prompt contains the assert text	Contaminated eval	Celebrate a green score
Two tests share one leaked example	Chorus, not coverage	Add a third copy
Temperature change flips the header	Optional eval	Call it model quality
Positive fixture green, negative grows the header	Policy miss	Prompt-tune the positive only
Clock rotation changes overdue	Rule is alive	Freeze the date in git
Endpoint 5xx with a valid oracle	Infra failure	Mark the product policy flaky

Where a free hosted loop fits

After the oracle file was sealed, the team still needed a second machine that did not hold the company’s paid key. The junior’s laptop had already seen the leaked prompt. Replaying on the same shell was not independent.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. The pairing used MonkeyCode’s free model access and free server option as that second machine: same harness, same two clocks, no paid key in the environment. The harness does not depend on that host. A local stub is enough to prove the leak is gone. The hosted loop is only a way to keep iterating when the laptop is no longer a clean room.

Limitations

The script does not prove product policy. It proves the model cannot read the answer from the prompt, and that a negative twin stays quiet under a rotating clock. It does not replay production traffic, auth, or tax rules.

A free hosted endpoint is a shared environment. Latency, queueing, and prompt logs are not under the pairing’s control. Do not put customer invoices, tokens, or names in the fixture. The stub path is deterministic. A real model path is not. Two clocks are a minimum, not a study.

This article does not claim model names, quotas, uptime, or scores. Those numbers go stale in a week. The pairing kept a method, not a benchmark.

Who should not use this

• Teams whose fixtures contain PII, card data, or internal hostnames. Seal the oracle, then keep the work off shared servers.
• Suites that must stand as compliance evidence. A pairing harness is not an audit.
• Groups that only need a chat window to “look right.” This loop fails closed when the oracle is missing.
• Anyone hoping a free endpoint will remove the senior. The decision they kept was human: throw out the green eval that leaked the assert.

The junior deleted Remember the test expects from the prompt file before leaving the room. The senior kept the two clocks, the forbidden header on the current invoice, and the rule that a green line is not a pairing signature if the model could have copied it. That is the whole session. The next model class does not get a vote.