Two Locks on a Green Build: Prompt Hashes and RSS Caps for AI C++ Evals

A patch landed with every assertion green. The return values matched the golden file. Then a latency dashboard moved: an inner loop that once touched a stack buffer now called std::vector::resize on every request. The eval harness never measured RSS or wall time, and it never hashed the prompt that produced the patch. That failure mode is common in AI C++ evals. Output-only golden cases miss it.

This article describes a small harness that adds two locks. The first lock pins the prompt text. The second lock caps wall time and resident set size. Together they catch silent regressions that unit tests still call success. The sample below is a worked example, not a published benchmark.

The hole output goldens leave

Functional goldens are necessary. They are not sufficient. A model can preserve stdout, preserve exit code, and still change allocation shape, cache behavior, or exception paths. A later prompt edit can also drift the eval without anyone noticing, because the .txt expected files still match.

Performance-oriented test design faces the same trap as API load tests that only check HTTP 200. A realistic gate needs a budget, a workload, and a frozen input. For C++ patches the workload is a tight binary. The frozen input is the prompt file plus the golden stdin. The budget is a contract the grader refuses to negotiate.

Artifact: a budget contract

The contract is a JSON file checked into the eval repo. It is not inferred from the last green run. Operators who let the model rewrite the contract are grading the model against itself.

{
  "prompt_path": "prompts/hot_path_v3.md",
  "prompt_sha256": "REPLACE_WITH_SHA256",
  "binary": "./build/hot_path",
  "argv": ["--n", "200000"],
  "stdin_path": "goldens/hot_path.in",
  "stdout_sha256": "REPLACE_WITH_STDOUT_SHA",
  "expect_exit": 0,
  "max_wall_ms": 250,
  "max_rss_kb": 32768,
  "warmup_runs": 1,
  "timed_runs": 3
}

max_wall_ms and max_rss_kb are local policy, not product claims. They must be filled from a reference binary on the same machine class that will grade patches. Mixing a laptop measurement with a remote VM measurement invalidates the cap.

Step 1: freeze the prompt

Prompt drift is a silent regression of a different kind. The tests stay green while the instructions change. The harness hashes the prompt file before it compiles anything.

# prompt_lock.py — example grader helper
from __future__ import annotations

import hashlib
from pathlib import Path


def sha256_file(path: Path) -> str:
    h = hashlib.sha256()
    with path.open("rb") as fh:
        for chunk in iter(lambda: fh.read(65536), b""):
            h.update(chunk)
    return h.hexdigest()


def assert_prompt_locked(contract: dict, root: Path) -> None:
    prompt = root / contract["prompt_path"]
    digest = sha256_file(prompt)
    expected = contract["prompt_sha256"].lower()
    if digest != expected:
        raise SystemExit(
            f"prompt hash mismatch: {prompt} got {digest} want {expected}"
        )

A prompt bump is an explicit commit. The operator updates prompt_sha256 in the same change that edits the markdown. Eval runs that skip this check are not comparable across days.

Step 2: keep a boring reference binary

The subject under test is a small hot path. The listing is illustrative. It is enough to show how a “correct” rewrite becomes a fatter process.

// hot_path.cpp — example workload, not production code
#include <cstdint>
#include <iostream>
#include <string>
#include <vector>

int main(int argc, char** argv) {
    std::uint32_t n = 1000;
    for (int i = 1; i + 1 < argc; ++i) {
        if (std::string(argv[i]) == "--n") {
            n = static_cast<std::uint32_t>(std::stoul(argv[i + 1]));
        }
    }
    std::uint64_t acc = 0;
    std::vector<std::uint32_t> buf;
    buf.reserve(n);  // reference: one allocation
    for (std::uint32_t i = 0; i < n; ++i) {
        buf.push_back(i ^ (i << 1));
        acc += buf.back();
    }
    std::cout << acc << "\n";
    return 0;
}

Build with a pinned compiler and pinned flags. Changing -O2 to -O0 between reference and candidate is a different experiment. Record the exact line in the eval log.

mkdir -p build
c++ -std=c++17 -O2 -pipe -o build/hot_path hot_path.cpp
sha256sum build/hot_path prompts/hot_path_v3.md goldens/hot_path.in

Step 3: grade process resources, not only stdout

Linux resource.getrusage reports max RSS for the child after wait. Wall time uses a monotonic clock around Popen. Three timed runs after one warmup reduce noise without pretending to be a lab benchmark.

# budget_grade.py — example, Linux-oriented
from __future__ import annotations

import json
import os
import resource
import subprocess
import time
from pathlib import Path

from prompt_lock import assert_prompt_locked, sha256_file


def run_once(contract: dict, root: Path) -> tuple[int, str, int, float]:
    stdin_data = (root / contract["stdin_path"]).read_bytes()
    start = time.perf_counter()
    proc = subprocess.Popen(
        [str(root / contract["binary"]), *contract["argv"]],
        stdin=subprocess.PIPE,
        stdout=subprocess.PIPE,
        stderr=subprocess.PIPE,
    )
    out, _err = proc.communicate(stdin_data)
    wall_ms = (time.perf_counter() - start) * 1000.0
    usage = resource.getrusage(resource.RUSAGE_CHILDREN)
    rss_kb = int(usage.ru_maxrss)
    return proc.returncode, out.decode("utf-8", "replace"), rss_kb, wall_ms


def grade(contract_path: Path) -> None:
    root = contract_path.parent
    contract = json.loads(contract_path.read_text())
    assert_prompt_locked(contract, root)

    for _ in range(int(contract["warmup_runs"])):
        run_once(contract, root)

    walls, rsses, outputs, codes = [], [], [], []
    for _ in range(int(contract["timed_runs"])):
        code, out, rss_kb, wall_ms = run_once(contract, root)
        codes.append(code)
        outputs.append(out)
        rsses.append(rss_kb)
        walls.append(wall_ms)

    stdout_digest = sha256_file  # placeholder; hash the text instead
    text = outputs[0]
    import hashlib

    digest = hashlib.sha256(text.encode()).hexdigest()
    failures = []
    if any(c != contract["expect_exit"] for c in codes):
        failures.append(f"exit {codes} want {contract['expect_exit']}")
    if any(o != text for o in outputs):
        failures.append("stdout not stable across timed runs")
    if digest != contract["stdout_sha256"].lower():
        failures.append(f"stdout hash {digest} want {contract['stdout_sha256']}")
    if max(walls) > float(contract["max_wall_ms"]):
        failures.append(f"wall {max(walls):.1f}ms > {contract['max_wall_ms']}")
    if max(rsses) > int(contract["max_rss_kb"]):
        failures.append(f"rss {max(rsses)}KB > {contract['max_rss_kb']}")
    if failures:
        raise SystemExit("FAIL\n" + "\n".join(failures))
    print(
        f"PASS wall_ms={max(walls):.1f} rss_kb={max(rsses)} stdout={digest[:12]}"
    )


if __name__ == "__main__":
    grade(Path("contract.json"))

ru_maxrss units differ across kernels. On Linux the value is kilobytes. On some BSD systems it is bytes. The contract must document the platform. Cross-OS comparison without a unit note is a false fail or a false pass.

Step 4: put the model on the other side of the contract

The grader does not call a network API. It consumes a patch, builds a binary, and reads contract.json. Patches may come from a local checkout or from a remote coding session. Some teams generate those patches with MonkeyCode. Disclosure: This article was prepared as part of MonkeyCode's product outreach. Free model access and a free server option are relevant only as a place to produce candidate diffs. They do not replace the local compiler, the prompt hash, or the RSS cap.

The operator copies the same prompts/, goldens/, and contract.json onto the machine that builds. The remote side never edits the caps. If a candidate deletes buf.reserve and reallocates inside the loop, stdout can stay identical while RSS or wall time crosses the line. That is the silent regression this harness exists to name.

Step 5: log enough to replay a fail

A fail that cannot be replayed will be ignored. Each run should print a single machine-readable line plus the human FAIL list.

python3 budget_grade.py | tee -a eval.log
# expected shape on success:
# PASS wall_ms=171.4 rss_kb=8124 stdout=9c1a0b7e4d21

Keep compiler version, libc, and nproc in the log header. Resource caps without that header are not evidence. They are souvenirs.

What this gate does not catch

The harness does not prove algorithmic complexity. A 250 ms cap on n=200000 can miss a quadratic path that only appears at production scale. It does not catch data races. It does not catch undefined behavior that happens to be fast. It does not catch a model that cheats by shrinking n in argv if the contract is not enforced.

Wall time on a busy shared CPU is noisy. RSS is stabler than wall time and still not a memory-correctness proof. Operators who need allocation truth should add a custom counter or a sanitizer build as a separate job. Mixing ASan and -O2 budgets in one contract confuses both signals.

Who should not use this approach

This workflow does not fit interactive GUI patches, GPU kernels, or programs whose RSS is dominated by a language runtime warmup. It does not fit evals that already ship only property tests with no representative n. It does not fit teams that cannot pin a compiler. It also does not fit anyone hoping a free remote session will produce portable timing numbers. Caps are local. Prompts are hashed. Binaries are built on the grader host.

A resource contract is a second lock on a green build. The first lock is the prompt digest. Neither lock is a substitute for reading the diff. They only stop the eval from celebrating a slower, fatter, still-correct patch as progress.