The Patch Deleted the Lock: Scoring AI C++ Fixes Against Declared Invariants

A backend team merged an AI-authored C++ patch on a Friday afternoon. Unit tests stayed green. The cache still returned the expected value on a single thread. Monday traffic made the process abort inside std::unordered_map. The model had deleted a std::mutex because nothing in the golden cases said the map was shared.

Compile-and-test harnesses catch crashes that already have a fixture. They do not catch architectural edits that still produce the same return value. That gap is where silent regressions hide when codegen is cheap and nobody reads every diff.

Compile-green is not contract-green

AI patches fail in a narrow way. The function signature stays. The happy-path test still passes. The surrounding contract moves: a lock disappears, a noexcept is added, a process-global cache appears, an error is swallowed, a header that pulled in a tracing macro is dropped.

Golden outputs that store stdout or a patched-file hash treat those edits as success. The team then ships a behavioral clone with a different memory model. The next model version “fixes” the same prompt again and drifts further.

An invariant harness stores the contract, not the answer. It scores a candidate patch against declarations the team already believes are true. The compiler remains the first grader. The invariant file is the second.

What the invariant file holds

The layout below is a proposal. It is small enough to keep next to whatever golden cases already exist for C++ codegen. Each case points at the files the model may touch and at the properties those files must keep.

{
  "id": "cache_get_locked",
  "language": "cpp",
  "allow_paths": ["src/cache.cpp", "src/cache.h"],
  "forbid_paths": ["src/cache_test.cpp", "CMakeLists.txt"],
  "invariants": [
    {
      "id": "mutex_present",
      "kind": "required_regex",
      "files": ["src/cache.cpp", "src/cache.h"],
      "pattern": "std::mutex",
      "reason": "Get/Put share a map across worker threads"
    },
    {
      "id": "no_process_global_cache",
      "kind": "forbidden_regex",
      "files": ["src/cache.cpp"],
      "pattern": "static\\s+std::unordered_map",
      "reason": "process-lifetime maps hide lifetime bugs in tests"
    },
    {
      "id": "get_signature",
      "kind": "required_regex",
      "files": ["src/cache.h"],
      "pattern": "std::optional<Value>\\s+Get\\(const\\s+Key&",
      "reason": "callers depend on optional, not raw pointers"
    },
    {
      "id": "no_swallow_exception",
      "kind": "forbidden_regex",
      "files": ["src/cache.cpp"],
      "pattern": "catch\\s*\\(\\s*const\\s+std::exception",
      "reason": "the service maps errors to status codes upstream"
    }
  ],
  "limits": {
    "max_touched_files": 2,
    "max_added_lines": 80
  }
}

The file is boring on purpose. Boring contracts survive prompt churn. Exotic scoring functions do not. The reason field is what a human reads when the model deletes the lock “to simplify the demo.”

A fixture the model is allowed to see

Freeze a tiny tree. The fixture is the golden input. It is not the golden output. The proposed headers and source below are enough to reproduce the Friday failure without pulling in a real service.

// src/cache.h
#pragma once
#include <mutex>
#include <optional>
#include <string>
#include <unordered_map>

class Cache {
public:
  using Key = std::string;
  using Value = std::string;
  std::optional<Value> Get(const Key& key) const;
  void Put(Key key, Value value);
private:
  mutable std::mutex mu_;
  std::unordered_map<Key, Value> map_;
};

// src/cache.cpp
#include "cache.h"

std::optional<Cache::Value> Cache::Get(const Key& key) const {
  std::lock_guard<std::mutex> lock(mu_);
  auto it = map_.find(key);
  if (it == map_.end()) return std::nullopt;
  return it->second;
}

void Cache::Put(Key key, Value value) {
  std::lock_guard<std::mutex> lock(mu_);
  map_[std::move(key)] = std::move(value);
}

A single-threaded unit test against Get/Put stays green after the mutex is removed. That is the point of the second grader. The test is necessary. It is not sufficient.

A grader that fails closed

The grader reads the invariant file, the original tree, and a patch. It applies the patch to a copy, then scores. Anything the model was not allowed to touch is a hard fail. Missing required patterns are a hard fail. Forbidden patterns are a hard fail. Soft limits such as added line count are warnings unless the team flips them to errors.

The Python below is a complete, if small, implementation that uses only the standard library plus git. It does not parse C++. Regex is the point: it is cheap, reviewable, and hostile to “the model rewrote the comment so the hash changed.” Treat it as an unexecuted example until it has been run against a local fixture.

#!/usr/bin/env python3
"""Invariant grader for AI C++ patches. Proposed example, not a published benchmark."""
from __future__ import annotations

import argparse
import json
import re
import subprocess
import sys
from pathlib import Path


def run(cmd, cwd):
    p = subprocess.run(cmd, cwd=cwd, text=True, capture_output=True)
    return p.returncode, p.stdout, p.stderr


def apply_patch(tree: Path, patch: Path) -> None:
    code, _, err = run(["git", "apply", "--check", str(patch)], cwd=tree)
    if code != 0:
        raise SystemExit(f"patch does not apply:\n{err}")
    code, _, err = run(["git", "apply", str(patch)], cwd=tree)
    if code != 0:
        raise SystemExit(f"patch apply failed:\n{err}")


def score_case(tree: Path, spec: dict) -> dict:
    findings = []
    errors = 0
    warnings = 0

    _, diff_out, _ = run(["git", "diff", "--name-only"], cwd=tree)
    changed = [line for line in diff_out.splitlines() if line]
    allow = set(spec.get("allow_paths", []))
    forbid = set(spec.get("forbid_paths", []))

    extra = [p for p in changed if p not in allow]
    banned = [p for p in changed if p in forbid]
    if extra:
        errors += 1
        findings.append({"id": "path_allowlist", "ok": False, "detail": extra})
    if banned:
        errors += 1
        findings.append({"id": "path_forbiddentouch", "ok": False, "detail": banned})

    limits = spec.get("limits", {})
    if len(changed) > limits.get("max_touched_files", 10**9):
        errors += 1
        findings.append({"id": "max_touched_files", "ok": False, "detail": changed})

    _, stat_out, _ = run(["git", "diff", "--numstat"], cwd=tree)
    added = 0
    for line in stat_out.splitlines():
        parts = line.split("\t")
        if parts and parts[0].isdigit():
            added += int(parts[0])
    if added > limits.get("max_added_lines", 10**9):
        warnings += 1
        findings.append({"id": "max_added_lines", "ok": False, "soft": True, "added": added})

    for inv in spec.get("invariants", []):
        kind = inv["kind"]
        pattern = re.compile(inv["pattern"])
        matched = False
        for rel in inv["files"]:
            text = (tree / rel).read_text(encoding="utf-8", errors="replace")
            if pattern.search(text):
                matched = True
                break
        ok = matched if kind == "required_regex" else (not matched)
        if not ok:
            errors += 1
        findings.append({
            "id": inv["id"],
            "ok": ok,
            "kind": kind,
            "reason": inv.get("reason", ""),
        })

    return {
        "id": spec["id"],
        "errors": errors,
        "warnings": warnings,
        "changed_files": changed,
        "findings": findings,
        "pass": errors == 0,
    }


def main():
    ap = argparse.ArgumentParser()
    ap.add_argument("--tree", type=Path, required=True)
    ap.add_argument("--patch", type=Path, required=True)
    ap.add_argument("--spec", type=Path, required=True)
    ap.add_argument("--json-out", type=Path, required=True)
    args = ap.parse_args()

    spec = json.loads(args.spec.read_text())
    apply_patch(args.tree, args.patch)
    report = score_case(args.tree, spec)
    args.json_out.write_text(json.dumps(report, indent=2))
    print(json.dumps(report, indent=2))
    sys.exit(0 if report["pass"] else 2)


if __name__ == "__main__":
    main()

A failing report is more useful than a boolean. id: mutex_present plus the stored reason is the regression ticket. It does not need a dashboard.

Decision table for fail-closed vs warn

Signal in the diff	Default action	Why
Required regex missing (std::mutex)	Fail closed	Concurrency contract vanished
Forbidden regex present (static map)	Fail closed	Hidden lifetime / test pollution
File outside allow_paths	Fail closed	Surprise CMake or test mutation
File in forbid_paths	Fail closed	Eval fixture or build graph edited
Added lines over max_added_lines	Warn	Verbosity is a smell, not always a bug
Patch does not git apply	Fail closed	Ungradeable output is not a pass

Keep the table next to the JSON spec. When someone wants to “just warn” on a deleted lock, the table makes that a deliberate change rather than a quiet edit to the grader.

Five steps to wire it into a codegen loop

1. Freeze the fixture. Copy cache.h / cache.cpp into fixtures/cache_get/ and commit that tree. Do not let the model invent the starting point. Invented starting points make later diffs incomparable.

2. Write the invariant file first. Do this before the prompt. If the team cannot state “the mutex stays” as a regex, the prompt is already underspecified. Store the reason next to the pattern so later reviewers do not delete it as noise.

3. Generate a patch in a throwaway worktree. Keep production credentials out of that tree. A local clone plus git apply --check is enough. The model only sees the fixture and the prompt, never the invariant file. Hiding the grader from the model is the whole point. Otherwise the model learns to keep the token std::mutex and stop calling lock_guard.

4. Score, then compile. Order matters. A patch that violates mutex_present should not consume a compiler minute. After the invariant grader passes, run the existing compile-and-test path the team already trusts.

5. Treat prompt edits as new cases. When someone changes the system prompt, copy the fixture, keep the invariants, and re-score a handful of stored patches. Silent prompt drift shows up as a previously passing patch that now deletes the lock, or as a new patch that starts touching CMakeLists.txt.

Commands for the local loop look like this:

git worktree add /tmp/cache-eval HEAD
cp patches/model-a.diff /tmp/cache-eval.patch
python3 tools/invariant_grader.py \
  --tree /tmp/cache-eval \
  --patch /tmp/cache-eval.patch \
  --spec invariants/cache_get.json \
  --json-out reports/cache_get.json
# exit 2 means contract fail; compile only on exit 0

Keep reports/ in version control. A JSON file with "pass": false is enough history. Time-series charts can wait until the team actually has more than one failing invariant.

Where a scratch model host belongs

Teams that already pay for a primary model should not mix eval traffic with production traffic. Eval prompts are repetitive. They also leak fixtures. A separate lane with its own keys reduces both cost surprise and prompt leakage into logs that operators treat as private.

MonkeyCode is one such lane when the team wants free model access and a free server option for running the grader against generated patches. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The claims here are limited to that free model access and free server option. No quota, hardware profile, duration, or model list is asserted in this article, because those numbers move and stale screenshots help nobody.

The useful split is operational, not religious. Production codegen stays on the model the product already selected. The invariant loop can run on the scratch host. The grader does not care which vendor emitted the diff. It cares whether std::mutex is still in the file.

A team that self-hosts can point the same script at any endpoint that returns a unified diff. The artifact is the invariant file and the exit code. The host is interchangeable.

What this grader will not see

Regex cannot prove a lock is held. A model can keep the std::mutex member and stop calling lock_guard. That is a real miss. Pair the invariant file with at least one threaded test if the contract is concurrency. The regex is a tripwire for deletion, not a replacement for ThreadSanitizer.

The allowlist cannot see generated files that land outside git. If the model writes /tmp/helper.cpp and the build system picks it up through a wildcard, the grader stays quiet. Pin CMakeLists.txt as a forbidden path unless the case is explicitly about build files.

Line-count limits punish verbose comments and miss one-line landmines. static std::unordered_map is one line. Keep forbidden patterns for the landmines and use line limits only as a smell.

Renames also dodge naive regex. std::mutex mu_ becoming std::recursive_mutex mu_ still matches std::mutex as a substring depending on the pattern. Tighten the pattern, or add a second forbidden pattern for the substitute type the team does not want.

Who should skip this

Do not adopt the harness if the repo has no fixtures worth freezing. A greenfield spike that is still renaming packages will spend more time updating JSON than catching regressions. Do not adopt it if the only “AI” in the loop is autocomplete inside an IDE. Invariant files earn their keep when patches are generated in batch and nobody reads every diff.

Skip it when the team already has a clang-tidy custom check for the same rules and that check runs on every patch. Duplicating std::mutex detection in Python is then noise. The JSON format is for contracts that are case-specific, not for house style.

Skip it for patches that are supposed to change the contract. A planned migration off a process-global cache will fail no_process_global_cache until someone versions a new spec. That is correct behavior. It is also extra process. Teams that change invariants weekly will hate the file.

Closing the Friday hole

The Friday patch was not a model failure in the theatrical sense. It was a missing sentence in the eval. The tests asked for a cached value. The model supplied one. The mutex was never a requirement, so deleting it was a valid optimization under the grader the team actually ran.

Write the requirement down. Hide it from the model. Fail closed when it vanishes. That is the entire method. The rest is a JSON file, a worktree, and an exit code.

If a scratch environment is needed to keep that loop off the production key, the same grader can run against MonkeyCode’s free model access and free server option without changing the invariant files.