Same Goldens, New Question: A Prompt-Pin Manifest for AI C++ Evals

A C++ library team merged a “clearer” system prompt on a Thursday. The golden cases stayed green. The compiler stayed quiet. A reviewer opened the next model patch on Monday and found using namespace std; in a public header, dropped const on three accessors, and a helper that threw from a destructor path the tests never entered.

Nothing in the scorecard had moved. The tests still asked last month’s question. The prompt no longer did.

Golden cases measure the answer. They do not measure whether the question is still the same question. Prompt edits are dependency changes. An eval that does not pin the prompt will report stability while the instruction surface drifts.

This article describes a small eval identity for AI C++ patches. The harness refuses to score a candidate unless a manifest of hashes matches the files on disk. The method stays useful if every product name is stripped out. A later section notes where a free remote runner can sit in the loop.

What the scorecard usually forgets

Most AI C++ harnesses version three things: the source under test, the compiler flags, and the expected I/O. That is necessary. It is not sufficient.

A prompt rewrite can keep every assert passing while changing which defects the model is even asked to avoid. Style rules vanish. Exception-safety language is softened. The model is told to prefer brevity. The goldens still check that add(2, 2) returns 4.

Silent question drift looks like a quality win. It is a spec change without a bump. Leaderboard rows from two prompt texts are then treated as one experiment. They are not.

Eval identity, not a vibe

Treat the eval as a tuple. If any element changes, the run is a different experiment.

1. Prompt text, including the system message and the user template.
2. Golden cases: inputs, expected outputs, and skip flags.
3. Grader: compiler invocation, sanitizers, and assertion driver.
4. Toolchain pins: compiler version string, language standard, and flag vector.
5. Candidate patch: the model output under test.

Hash items 1–4 into a committed manifest. Compare those hashes before any score is written. Fail closed on mismatch. Record the candidate hash only after the identity check passes.

The rest of this article implements that check with ordinary files. No database. No dashboard. The scripts below are a proposed tree, not a claim that they have been run against a private corpus.

Artifact: eval_manifest.json

Store the manifest next to the prompt, not in a wiki. Empty hashes are placeholders. A freeze script fills them. CI rejects a run whose live hashes do not match the committed pins.

{
  "schema": "eval-identity/v1",
  "suite": "ai-cpp-patches",
  "pins": {
    "prompt_sha256": "",
    "goldens_sha256": "",
    "grader_sha256": "",
    "toolchain_sha256": ""
  },
  "toolchain": {
    "cxx": "g++",
    "cxx_std": "c++20",
    "cxxflags": ["-O2", "-Wall", "-Wextra", "-Werror", "-fsanitize=address,undefined"]
  },
  "policy": {
    "on_mismatch": "fail_closed",
    "allow_prompt_bump": false
  }
}

allow_prompt_bump stays false in CI. A human freeze is the only way the prompt pin moves. That pause is the whole gate.

Layout on disk

Keep the suite boring and grepable. One prompt file. One golden directory. One grader tree. One captured toolchain dump.

eval/
  prompt.txt
  goldens/
    001_add.json
    002_overflow.json
    003_header_hygiene.json
  grader/
    run.sh
    catch_main.cpp
  toolchain.txt
  eval_manifest.json
  freeze.py
  check_identity.py
  score.py

toolchain.txt is a captured g++ -v dump plus the exact flag vector. Do not reconstruct it from memory during a run. Two machines with “the same g++” still diverge on patch level, default search paths, and sanitizer runtimes.

Step 1 — Freeze the identity

freeze.py hashes canonical bytes. The script does not pretty-print JSON before hashing goldens. It hashes the files as committed, in sorted path order, so a renamed case still moves the tree digest.

#!/usr/bin/env python3
"""Freeze eval identity. Proposed helper; run it on a real tree before trusting pins."""
from __future__ import annotations

import hashlib
import json
from pathlib import Path

ROOT = Path(__file__).resolve().parent

def sha256_file(path: Path) -> str:
    return hashlib.sha256(path.read_bytes()).hexdigest()

def sha256_tree(dirpath: Path) -> str:
    h = hashlib.sha256()
    for p in sorted(dirpath.rglob("*")):
        if not p.is_file():
            continue
        rel = p.relative_to(dirpath).as_posix().encode()
        h.update(rel)
        h.update(b"\0")
        h.update(p.read_bytes())
        h.update(b"\0")
    return h.hexdigest()

def main() -> None:
    man_path = ROOT / "eval_manifest.json"
    man = json.loads(man_path.read_text())
    man["pins"]["prompt_sha256"] = sha256_file(ROOT / "prompt.txt")
    man["pins"]["goldens_sha256"] = sha256_tree(ROOT / "goldens")
    man["pins"]["grader_sha256"] = sha256_tree(ROOT / "grader")
    man["pins"]["toolchain_sha256"] = sha256_file(ROOT / "toolchain.txt")
    man_path.write_text(json.dumps(man, indent=2) + "\n")
    print("froze", man["pins"])

if __name__ == "__main__":
    main()

Run it only when a spec bump is intended.

python3 eval/freeze.py
git add eval/eval_manifest.json eval/prompt.txt eval/goldens eval/grader eval/toolchain.txt
git diff --cached eval/eval_manifest.json

The pin diff is the changelog. If the prompt hash moved and the goldens hash did not, the question changed and the tests did not. That pair is the reason the harness exists.

Step 2 — Fail closed before any model call

check_identity.py recomputes live hashes and compares. Scoring never starts on mismatch. Generation can wait. A wrong question is cheaper to catch before tokens are spent than after a green cell is published.

#!/usr/bin/env python3
from __future__ import annotations

import json
import sys
from pathlib import Path
from freeze import ROOT, sha256_file, sha256_tree

def live_pins() -> dict[str, str]:
    return {
        "prompt_sha256": sha256_file(ROOT / "prompt.txt"),
        "goldens_sha256": sha256_tree(ROOT / "goldens"),
        "grader_sha256": sha256_tree(ROOT / "grader"),
        "toolchain_sha256": sha256_file(ROOT / "toolchain.txt"),
    }

def main() -> int:
    man = json.loads((ROOT / "eval_manifest.json").read_text())
    expected = man["pins"]
    got = live_pins()
    bad = {
        k: {"expected": expected[k], "got": got[k]}
        for k in expected
        if expected[k] != got[k]
    }
    if bad:
        print("EVAL_IDENTITY_MISMATCH")
        print(json.dumps(bad, indent=2))
        return 2
    print("EVAL_IDENTITY_OK", json.dumps(got, sort_keys=True))
    return 0

if __name__ == "__main__":
    sys.exit(main())

Wire it as a gate, not as a log line.

python3 eval/check_identity.py && python3 eval/score.py --patch candidate.diff

A typical mismatch looks like this after someone “just clarified” prompt.txt:

EVAL_IDENTITY_MISMATCH
{
  "prompt_sha256": {
    "expected": "9c1e0c6c0f3a7b1d8a2e4f55c0b91aa0e3c6d2f1a7b84c0d9e1f2a3b4c5d6e7f",
    "got":      "0b77a1d4e2c98f06b3aa4419d5c0e18f27d6b4c1a9e03f55c8d2a176b0e4c9aa"
  }
}

Those hex strings are examples of shape, not measurements from a private run. The important part is the exit code. CI treats 2 as a broken spec, not as a model failure.

Step 3 — Keep goldens that the new question can still fail

Prompt pins without hostile goldens only detect file drift. They do not detect a weaker instruction. Add at least one golden that encodes a rule the prompt claims to enforce. Header hygiene is a reliable example because tests almost never look at it.

goldens/003_header_hygiene.json:

{
  "id": "003_header_hygiene",
  "source": "box.hpp",
  "must_compile": true,
  "compile_unit": "const Box b{1}; auto x = b.value(); (void)x;",
  "forbid_regex": [
    "using namespace std;",
    "using namespace std::",
    "catch\\s*\\(\\s*\\.\\.\\.\\s*\\)"
  ]
}

A grader compiles the unit and scans the patch text. The regex layer is crude. It still catches the Thursday rewrite that stopped mentioning public headers. Put the scan in score.py so the grader tree hash moves when the check changes.

grader/run.sh stays the only compiler entry. That keeps the tree hash meaningful.

#!/usr/bin/env bash
set -euo pipefail
ROOT="$(cd "$(dirname "$0")/.." && pwd)"
python3 "$ROOT/check_identity.py"
CXX="${CXX:-g++}"
src="$1"
"$CXX" -std=c++20 -O2 -Wall -Wextra -Werror -fsanitize=address,undefined \
  -c "$src" -o /tmp/candidate.o

Capture the live toolchain once per machine image.

{
  echo "# toolchain pin"
  g++ -v 2>&1
  echo "# flags"
  echo "-std=c++20 -O2 -Wall -Wextra -Werror -fsanitize=address,undefined"
} > eval/toolchain.txt

Label the snippets as a proposed envelope. They are not a published benchmark and they are not a claim about any model’s pass rate.

Step 4 — Record the candidate only after identity holds

score.py should refuse to write last_result.json unless check_identity returned zero. Then hash the patch itself. Later spreadsheet rows become comparable only when the pin block matches.

# Proposed fragment, not a full product.
import hashlib, json, subprocess, sys
from pathlib import Path

patch_path = Path(sys.argv[1])
patch = patch_path.read_bytes()
subprocess.check_call([sys.executable, "eval/check_identity.py"])
pins = json.loads(Path("eval/eval_manifest.json").read_text())["pins"]
result = {
    "eval_manifest": pins,
    "patch_sha256": hashlib.sha256(patch).hexdigest(),
    "verdict": "unscored",
}
Path("eval/last_result.json").write_text(json.dumps(result, indent=2) + "\n")

Mixing rows from two prompt hashes is a methodology error. The file makes that error visible. A green cell under pin set A is not a data point under pin set B.

Decision table: what to bump

Change	Prompt pin	Goldens pin	Grader pin	Toolchain pin	Action
Wording tweak in prompt.txt	yes	no	no	no	Freeze, re-run the full suite, do not compare scores to the previous pin set
New golden for header hygiene	no	yes	maybe	no	Freeze, treat as spec expansion
Switch -O0 to -O2	no	no	no	yes	Freeze; scores are a different experiment
ASan added to run.sh	no	no	yes	maybe	Freeze; old “green” is not transferable
Model sampled again, same files	no	no	no	no	Score and compare
Prompt and goldens both edited	yes	yes	no	no	Freeze once, write a note that two spec axes moved together

The dangerous row is the first one. Teams skip it because the tests still pass. The manifest exists to make that skip loud. Two axes moving in one commit is legal. It is also easy to misread later, so the note belongs in the same commit as the freeze.

Where a free remote runner fits

Local identity checks are cheap. Generating candidate patches is not. Operators who already iterate prompts against a C++ suite sometimes need a place to sample models without standing up a GPU box for every wording tweak.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode is an open-source project that, as supplied by the operator of this account, offers free model access and a free server option. Those two availability claims are the only product facts used here. This article does not assert model names, token quotas, hardware, uptime, or permanence. Dashboard numbers go stale. Re-read the project page before planning capacity.

In this workflow the remote runner is optional. The identity files stay in git. A generation job may run off-laptop. The score step still happens only after check_identity.py passes on the same commit. If the free server is used to produce candidate.diff, copy that file back and hash it locally. Do not let a remote workspace silently edit prompt.txt.

Readers who want that runner can look up the public repository when they need it. The harness above does not depend on it.

Limitations

The manifest does not understand semantics. Two prompts can hash differently and mean the same thing. Two prompts can hash identically after a well-placed Unicode lookalike. Hashing is identity, not equivalence. Operators who need semantic diff still have to read the prompt.

Regex forbid-lists are bypassable. A model can split using namespace std; across a line continuation or hide a throw in a macro. A single compile unit does not prove ABI stability, thread safety, or that RSS stayed flat. Those are other envelopes. They do not replace a prompt pin, and a prompt pin does not replace them.

This approach is a poor fit for throwaway playground prompts, for suites without committed goldens, and for teams that want a single scalar “model quality” number across prompt versions. It is the wrong tool if the compiler cannot be pinned. Unpinned toolchains make the toolchain hash theater.

Do not use the freeze script as a ritual that rubber-stamps a weaker prompt. If the prompt hash moved, re-read the goldens. Add a case that would have failed under the old instruction. Then freeze.

What “green” is allowed to mean

After this harness is in place, a green cell means four files on disk still match a committed tuple, and the candidate survived that tuple’s grader. It does not mean the prompt is better. It does not mean last week’s leaderboard row is comparable.

Prompt edits become visible spec bumps. Silent question drift has to walk through freeze.py. That is a small gate. For C++ codegen evals, it is usually the gate that was missing.