Undeclared Includes, Invented Helpers: An Assumption Ledger for AI C++ Patches

A maintainer merged an AI-written C++ patch because the golden unit test still returned the cached size. The function compiled. The assertion passed. The diff also planted a process-wide static std::mutex that no test ever locked from a second thread. Two days later a request handler blocked on that mutex during shutdown. The golden case had scored the return value. It had never scored what the model assumed it was allowed to invent.

Behavioral oracles are necessary. They are not sufficient. Agent-style coding loops now emit plausible patches that satisfy the demo and still smuggle headers, helpers, widened lock scopes, and dropped const into the tree. Those undeclared assumptions are a distinct failure class. This article treats them as first-class eval failures, not as style nits.

The workflow below is a prompt and eval harness. It keeps golden cases for behavior. It adds a machine-readable assumption ledger for everything the model is not allowed to introduce. The grader is small enough to run on a laptop or on a free coding server. It does not claim to prove correctness. It claims to fail loud when the patch quietly rewrites the contract.

The failure the tests never saw

Consider a size cache that must stay lock-free on the read path. A typical prompt asks the model to fix a stale-read bug. A typical golden test checks that two calls return the same size_t. A typical model “fixes” the race by adding a static mutex and locking both the write and the read. The test stays green. The original invariant is gone.

The same pattern shows up as an invented ensure_ready() helper, a new #include <thread>, a file-scope std::atomic_flag, or a removed noexcept. Each of those can be a legitimate design change. None of them should land because a codegen loop guessed. The ledger makes the allowed surface explicit before the prompt is sent.

Ledger format before the prompt

Store the contract next to the golden case, not inside the chat. YAML is enough for a first harness. JSON works if the rest of the pipeline already speaks it.

# cases/size_cache/assumptions.yaml
case_id: size_cache_stale_read
language: cpp
allowed_headers:
  - cstddef
  - cstdint
forbidden_headers:
  - mutex
  - thread
  - shared_mutex
allowed_new_symbols: []
forbidden_constructs:
  - static_mutex
  - file_scope_atomic
  - sleep_for
must_preserve:
  - const on SizeCache::size
  - noexcept on SizeCache::size
  - no new non-const static in translation unit
behavior_oracle: ./cases/size_cache/test_size_cache

The prompt then cites the ledger instead of restating it in prose. Models paraphrase constraints. Files do not. If a later prompt edit drops a sentence, the grader still reads the same YAML.

Label the following as an example harness, not as a published benchmark. It was not executed against a vendor leaderboard for this article.

Artifact: a three-stage grader

The harness has three stages on purpose. Compile and run still matter. Assumption scoring sits between parse and execute so a green test cannot hide a contract breach.

1. Parse the unified diff into added includes, added symbols, and stripped qualifiers.
2. Compare those additions against assumptions.yaml.
3. Build and run the behavior oracle only if the ledger is clean.

# grade_assumptions.py — example harness, not a scored study
from __future__ import annotations

import re
import subprocess
import sys
from pathlib import Path

import yaml

INCLUDE_RE = re.compile(r"^\+\s*#include\s*[<"]([^>"]+)[>"]", re.M)
STATIC_MUTEX_RE = re.compile(r"^\+.*static\s+std::mutex\b", re.M)
CONST_SIZE_RE = re.compile(r"\bsize\s*\(\s*\)\s*const\s*noexcept")


def load_ledger(path: Path) -> dict:
    return yaml.safe_load(path.read_text())


def added_headers(diff: str) -> set[str]:
    names = set()
    for match in INCLUDE_RE.finditer(diff):
        names.add(Path(match.group(1)).name)
    return names


def grade(diff: str, ledger: dict) -> list[str]:
    failures = []
    headers = added_headers(diff)
    forbidden = set(ledger.get("forbidden_headers", []))
    sneak = sorted(headers & forbidden)
    if sneak:
        failures.append(f"undeclared headers: {sneak}")
    if "static_mutex" in ledger.get("forbidden_constructs", []) and STATIC_MUTEX_RE.search(diff):
        failures.append("undeclared static mutex")
    if "const on SizeCache::size" in ledger.get("must_preserve", []):
        if "size() const noexcept" in diff and not CONST_SIZE_RE.search(diff):
            # stripped in a changed signature line
            if re.search(r"^[-].*size\s*\(.*const", diff, re.M) and re.search(
                r"^\+.*size\s*\(", diff, re.M
            ):
                failures.append("dropped const/noexcept on SizeCache::size")
    return failures


def run_oracle(cmd: str) -> int:
    return subprocess.call(cmd, shell=True)


def main() -> int:
    diff = Path(sys.argv[1]).read_text()
    ledger = load_ledger(Path(sys.argv[2]))
    failures = grade(diff, ledger)
    if failures:
        print("ASSUMPTION_FAIL")
        for item in failures:
            print(f"- {item}")
        return 2
    code = run_oracle(ledger["behavior_oracle"])
    if code != 0:
        print("BEHAVIOR_FAIL")
        return code
    print("PASS")
    return 0


if __name__ == "__main__":
    raise SystemExit(main())

A regex grader will miss macros and clever typedefs. That is an accepted limitation, recorded later. The point of the first version is a deterministic, reviewable fail code: 2 for assumption leakage, non-zero from the oracle for behavioral breakage. Silent green is the outcome this harness is built to destroy.

Golden case that should fail the mutex “fix”

Keep the production-shaped snippet tiny. The ledger is the interesting part.

// cases/size_cache/size_cache.hpp
#pragma once
#include <cstddef>

class SizeCache {
  std::size_t cached_{0};
public:
  void store(std::size_t n) { cached_ = n; }
  std::size_t size() const noexcept { return cached_; }
};

// cases/size_cache/test_size_cache.cpp
#include "size_cache.hpp"
#include <cassert>

int main() {
  SizeCache c;
  c.store(4);
  assert(c.size() == 4);
  assert(c.size() == 4);
  return 0;
}

Feed the model a patch prompt that states the stale-read symptom and points at assumptions.yaml. Save the model diff to out.patch. Then run the grader before anyone opens a pull request.

python3 grade_assumptions.py out.patch cases/size_cache/assumptions.yaml

Expected outcomes for this example, if the model adds #include <mutex> and a static lock: exit status 2 and ASSUMPTION_FAIL. Expected outcome if the model only updates store under an existing, already-declared discipline and the assert still holds: PASS. Do not treat those two strings as measured model accuracy. They are the harness contract.

Prompt packaging that the grader can audit

Put the prompt in the same directory. Version it. Diff it. The eval loop should hash prompt, ledger, and source together so a “better” model run is not actually a quieter constraint.

You are patching SizeCache in size_cache.hpp.
Honor cases/size_cache/assumptions.yaml exactly.
Do not add headers or symbols unless they appear in allowed_headers
or allowed_new_symbols.
Do not introduce static synchronization.
Keep SizeCache::size const and noexcept.
Return a unified diff and nothing else.

When an agent loop is allowed to call tools, the same ledger becomes a tool-side check. The agent can compile. It can run test_size_cache. It still fails if it “helpfully” includes <mutex>. That is the whole design: the extra degree of freedom agents enjoy is also an extra degree of assumption.

Where a free coding server fits

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Running this loop by hand on one patch teaches the format. Running it on every prompt revision needs a place that can apply diffs, compile C++, and keep the YAML next to the source. MonkeyCode is an open-source coding-server project that, as supplied for this article, offers free model access on the order of 10 million tokens and a free server option. Those two availability claims are the only product facts used here. No model names, hardware profiles, latency numbers, or permanence guarantees are attached.

The harness does not depend on that server. A local clang++ and the Python grader already reproduce the method. The free tier is relevant when the same golden case must be sent through several prompt revisions without burning a paid quota on the evaluation traffic. Token budget then belongs to the eval loop, not to a demo chat.

A single, optional next step for readers who already own a C++ oracle is to point this ledger at whatever coding server they can compile on, including MonkeyCode’s free server if that is the stack under test.

Numbered workflow for a prompt change

Use the same sequence every time a system prompt, a tool list, or a “be careful with threads” paragraph changes. Prompt edits are code edits. They ship silent regressions if only the happy-path test is rerun.

1. Freeze assumptions.yaml before editing the prompt. If the allowed surface must grow, change the ledger in the same commit as the prompt.
2. Keep one golden case that the old prompt already passed, including a case that must stay ASSUMPTION_FAIL when a mutex is introduced.
3. Generate a patch with the new prompt. Save the raw diff. Do not hand-edit it before grading.
4. Run grade_assumptions.py. Treat exit 2 as a product regression, not as a style comment.
5. Run the behavior oracle only after the ledger is clean.
6. Record prompt hash, ledger hash, and exit class (PASS / ASSUMPTION_FAIL / BEHAVIOR_FAIL) in a log the next revision can diff.

Step 2 is easy to skip. It is the step that catches a new prompt that “helps” by locking. A suite of only happy paths will celebrate that patch.

What this does not catch

A header-name check does not understand #include via a wrapper header. A regex does not understand using Lock = std::mutex. Dropped const on a template specialization in another file will sail through. Data races that never touch a forbidden construct still need ThreadSanitizer or a real lock-order review. The ledger is a contract filter. It is not a replacement for sanitizers, code review, or a memory model.

The behavior oracle in the example is a single-threaded assert. That is intentional for the article. Production caches need concurrent tests. Those tests still fail to notice an undeclared mutex if they never contend. Run both layers.

Do not cite this harness as evidence that one host, model, or free tier is more accurate than another. No timing table is included because none was measured for this write-up.

Who should not use this approach

Teams without a compile-and-run oracle should not pretend the ledger is an eval. A diff linter alone will rubber-stamp a patch that does not compile. Security-sensitive code should not accept a green ledger as authorization to skip human review. Beginners who cannot state must_preserve in one page of YAML will encode the wrong contract and then automate it.

The method also wastes time on throwaway snippets with no invariants. If the file is a 20-line experiment and every header is in play, write a unit test and stop. Assumption ledgers pay off when the surrounding code already has ownership, const, and lock discipline that a helpful agent will otherwise “improve.”

Closing the loop

Golden tests answer whether the patch still returns 4. The assumption ledger answers whether the patch was allowed to invent a mutex to get there. Agent coding loops fail the second question more often than the first. Score both. Keep the YAML next to the prompt. Fail the build on undeclared includes before the unit test gets a chance to lie.