Asserts Passed, RSS Doubled: An rusage Envelope for AI C++ Patches

The following incident is a composite of a common failure mode, not a claim about a named production system. A daemon team merged a one-line cache suggested by a coding model. Every unit test stayed green. The next on-call graph told a different story: resident set size climbed through the night, and a previously quiet process started swapping.

The patch replaced a linear scan with an unbounded unordered_map. Correctness tests used fixtures of a few dozen keys. Nothing in the pipeline asked how much memory or CPU a correct answer was allowed to spend.

Functional goldens remain necessary. They are not a resource spec.

Cost is part of the contract

AI C++ patches fail in noisy ways and in quiet ways. Missing headers and invented helpers break the build. Resource regressions usually do not. The binary still links. Asserts still pass. The process simply gets heavier on every call.

An eval harness that only checks exit codes will score the heavy patch as a win. A second artifact has to sit beside the tests: a committed resource envelope. The envelope is JSON next to the fixtures. It records ceilings, not averages. If a candidate exceeds a ceiling, the patch fails even when every assert passes.

This article is a worked example. It is not production telemetry. Numbers in the sample envelope are illustrative ceilings for a tiny driver. They are not measurements from a deployed service.

What the envelope records

The file stays small on purpose. Three fields catch a surprising number of helpful-looking model edits on Linux.

1. max_rss_kb — high-water resident set from child rusage (ru_maxrss, kilobytes on Linux).
2. max_user_us — user CPU time in microseconds.
3. timeout_s — wall-clock cap; a hang becomes a reject before RSS is even read.

Optional later fields include max_minflt and max_inblock. A field is worth adding only after a real incident names it. The envelope is a tripwire. Tripwires are allowed to be coarse.

Ceilings have to be recorded on the same class of host that will grade candidates. Copying a laptop envelope onto a shared eval box is a factory for false failures. Host class is part of the spec, the same way -O2 is part of the spec.

Worked example: the cache that never evicted

The sample library looks up scores by id. The baseline scans a small vector. Tests pass. A coding model then fixes the scan by filling a process-lifetime unordered_map with no eviction. Lookups still return the right integer. RSS does not stay still.

Header under test

#pragma once
#include <cstdint>
#include <unordered_map>
#include <vector>

struct ScoreRow {
  std::uint32_t id;
  int score;
};

inline int lookup_scan(const std::vector<ScoreRow>& rows, std::uint32_t id) {
  for (const auto& row : rows) {
    if (row.id == id) return row.score;
  }
  return -1;
}

// Anti-pattern used to demonstrate the envelope.
// Models emit this when a prompt says "make it faster" and never says "bound the table."
inline int lookup_unbounded_cache(const std::vector<ScoreRow>& rows,
                                  std::uint32_t id) {
  static std::unordered_map<std::uint32_t, int> cache;
  auto it = cache.find(id);
  if (it != cache.end()) return it->second;
  int score = lookup_scan(rows, id);
  cache.emplace(id, score);
  return score;
}

The unbounded cache is deliberate. It is the silent case: right answers, growing weight.

Driver the harness actually times

#include "score_lookup.hpp"
#include <cstdint>
#include <iostream>
#include <string>
#include <vector>

int main(int argc, char** argv) {
  const bool use_cache = (argc > 1 && std::string(argv[1]) == "cache");
  std::vector<ScoreRow> rows;
  rows.reserve(256);
  for (std::uint32_t i = 0; i < 256; ++i) {
    rows.push_back({i, static_cast<int>(i % 17)});
  }
  long long acc = 0;
  for (int round = 0; round < 40000; ++round) {
    std::uint32_t id = static_cast<std::uint32_t>((round * 17u) % 100000u);
    acc += use_cache ? lookup_unbounded_cache(rows, id)
                     : lookup_scan(rows, id);
  }
  std::cout << acc << "\n";
  return 0;
}

Both modes print one integer and exit 0. Functional equality is the trap. The envelope is the distinction.

Committed ceilings

{
  "cmd": ["./driver"],
  "timeout_s": 5,
  "max_rss_kb": 8192,
  "max_user_us": 800000,
  "notes": "Illustrative Linux ceilings for this driver. Re-record on the eval host."
}

Rebuild the envelope when the host, the compiler, or the fixture size changes. Treat that rebuild as a spec change. The grader should not raise ceilings on its own.

The rusage grader

The grader lives outside the candidate on purpose. A patch that prints RSS: ok is not evidence. Python 3 plus the resource module is enough on Linux. The parent process must not wait on other children before the timed run, or RUSAGE_CHILDREN will mix histories.

Label the script as a template. It is not certified on every distro. GNU /usr/bin/time -f '%x %U %M' is a valid alternative when Python is unwelcome. The committed artifact should still be envelope.json, not a screenshot of a terminal.

#!/usr/bin/env python3
"""Linux rusage envelope grader for C++ patch candidates. Worked example only."""
from __future__ import annotations

import argparse
import json
import resource
import subprocess
import sys
from pathlib import Path


def measure(cmd: list[str], timeout_s: float) -> dict:
    try:
        completed = subprocess.run(
            cmd,
            stdout=subprocess.PIPE,
            stderr=subprocess.PIPE,
            timeout=timeout_s,
            check=False,
        )
        timed_out = False
        returncode = int(completed.returncode)
        stdout = completed.stdout
        stderr = completed.stderr
    except subprocess.TimeoutExpired as exc:
        timed_out = True
        returncode = 124
        stdout = exc.stdout or b""
        stderr = exc.stderr or b""
    ru = resource.getrusage(resource.RUSAGE_CHILDREN)
    return {
        "timed_out": timed_out,
        "returncode": returncode,
        "stdout": stdout.decode("utf-8", errors="replace"),
        "stderr": stderr.decode("utf-8", errors="replace"),
        "max_rss_kb": int(ru.ru_maxrss),
        "user_us": int(ru.ru_utime * 1_000_000),
    }


def main() -> int:
    parser = argparse.ArgumentParser()
    parser.add_argument("--cmd", nargs="+", required=True)
    parser.add_argument("--envelope", type=Path, required=True)
    parser.add_argument("--record", action="store_true")
    parser.add_argument("--force", action="store_true")
    parser.add_argument("--grade", action="store_true")
    parser.add_argument("--expect-stdout", type=Path)
    args = parser.parse_args()
    if args.record == args.grade:
        print("exactly one of --record or --grade is required", file=sys.stderr)
        return 2

    timeout_s = 5.0
    if args.grade:
        env = json.loads(args.envelope.read_text())
        timeout_s = float(env.get("timeout_s", 5))

    result = measure(args.cmd, timeout_s)

    if args.record:
        if args.envelope.exists() and not args.force:
            print("refusing to overwrite envelope without --force", file=sys.stderr)
            return 2
        # Slack is explicit in the file. Hidden slack inside the grader is a bug.
        payload = {
            "cmd": args.cmd,
            "timeout_s": timeout_s,
            "max_rss_kb": int(result["max_rss_kb"] * 1.3) + 1024,
            "max_user_us": int(result["user_us"] * 1.5) + 50_000,
            "notes": "ceilings = one baseline run plus committed slack; re-record on host change",
        }
        args.envelope.write_text(json.dumps(payload, indent=2) + "\n")
        print(json.dumps({"mode": "record", "measured": {
            "max_rss_kb": result["max_rss_kb"],
            "user_us": result["user_us"],
            "returncode": result["returncode"],
        }, "wrote": str(args.envelope)}))
        return 0

    env = json.loads(args.envelope.read_text())
    reasons = []
    if result["timed_out"]:
        reasons.append("timeout")
    if result["returncode"] != 0:
        reasons.append(f"exit_{result['returncode']}")
    if result["max_rss_kb"] > int(env["max_rss_kb"]):
        reasons.append("rss")
    if result["user_us"] > int(env["max_user_us"]):
        reasons.append("user_cpu")
    if args.expect_stdout and result["stdout"] != args.expect_stdout.read_text():
        reasons.append("stdout")
    verdict = "reject" if reasons else "accept"
    print(json.dumps({
        "mode": "grade",
        "verdict": verdict,
        "reasons": reasons,
        "measured": {
            "returncode": result["returncode"],
            "max_rss_kb": result["max_rss_kb"],
            "user_us": result["user_us"],
            "timed_out": result["timed_out"],
        },
    }, indent=2))
    return 1 if reasons else 0


if __name__ == "__main__":
    raise SystemExit(main())

Linux reports ru_maxrss in kilobytes. macOS reports bytes. This template is Linux-shaped on purpose. Mixing units without a platform tag will either reject every patch or accept every leak.

Numbered workflow

Run the loop the same way on every candidate. Drift in the loop is how silent cost bugs get a passing grade.

1. Pin the toolchain. Record g++ --version next to the envelope. A different libstdc++ can move RSS without any source change.
2. Build the known-good binary at -O2. Optimization level is part of the spec. Debug heaps lie about both RSS and user time.
3. Capture a baseline with --record. The record path should refuse to overwrite unless --force is passed.
4. Apply the candidate patch in a worktree. Rebuild with the same flags. Sanitizers stay out of this particular loop; they change RSS on purpose.
5. Grade the candidate with --grade and --expect-stdout. Read the JSON verdict line, not the driver's stdout, for the cost decision.
6. Reject when functional output mismatches or when any ceiling breaks. Accept only when both oracles agree.
7. If the cost increase is the actual design, raise envelope.json in the same commit as the patch. Split commits hide the trade.

A one-line compile plus the two harness modes looks like this:

set -euo pipefail
g++ --version
g++ -std=c++17 -O2 -Wall -Wextra -o driver driver.cpp
./driver > golden_stdout.txt
python3 rusage_harness.py --record --force --cmd ./driver --envelope envelope.json
python3 rusage_harness.py --grade --cmd ./driver cache \
  --envelope envelope.json --expect-stdout golden_stdout.txt

The interesting outcome is exit status 1 with "reasons": ["rss"] while stdout still matches. That is the patch that looks like competence in a unit-test UI.

Where generation and grading can share a host

Local laptops add noise. Browser tabs move max RSS. Thermal throttling moves user CPU. An eval that is supposed to catch silent weight needs a boring machine.

Teams that already generate C++ patches from models can run this envelope next to the compiler. MonkeyCode is an open-source project with free model access and a free server option, which is enough to keep both generation and the rusage loop off a developer workstation.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The envelope still has to live in the repository. Free inference does not define max_rss_kb. The server is a stable place to run the grader, not a substitute for committed ceilings. Model names, token quotas, and hardware details are out of scope here because they change and they are not what the envelope measures.

How the silent case shows up

The scan driver and the cache driver can share stdout. The harness then prints two different reasons to fail.

Observation	Functional oracle	Envelope oracle	Action
Same stdout, RSS and CPU under ceilings	pass	pass	accept
Different stdout, any cost	fail	ignored	reject
Same stdout, RSS over ceiling	pass	fail	reject as cost regression
Timeout	fail	fail	reject as hang
Compile error	fail	n/a	reject

The third row is the point of the extra artifact. Models also emit optimizations that raise CPU while lowering RSS, or the reverse. A single-field envelope will miss half of those. RSS and user CPU start together. They should not be averaged into one score. Averages hide which contract broke.

Limitations

ru_maxrss is a high-water mark, not a leak proof. A patch can allocate, free, and still pass if the peak stays under the ceiling. The envelope will not replace AddressSanitizer or a heap profiler.

Shared eval hosts are noisy. A noisy neighbor can raise minor faults and sometimes RSS. Pinning a cgroup, or committing a small slack percentage inside envelope.json, is honest. Hidden slack inside the Python script is a grader bug.

The driver above uses a static map, which is easy to see in review. Real regressions hide in extra copies of std::string, in std::regex constructed per call, in logging that formats on a hot path. The envelope does not name the guilty line. It only refuses to ship the binary.

Timeouts are wall-clock. A loaded host turns a legal patch into a hang. timeout_s needs the same honesty as RSS, and the machine should stay otherwise idle during grade. After TimeoutExpired, child rusage may be incomplete. Treat timeout as a hard reject rather than a partial RSS reading.

This loop does not prove asymptotic complexity. Forty thousand iterations can still miss an O(n²) path that production will hit at millions. Pair the envelope with at least one fixture that resembles production cardinality. Tiny goldens created the original incident.

Who should skip this approach

The technique does not replace perf or microbenchmark libraries when the question is a 2% inner-loop change. The tripwire is too coarse for that job.

It does not transfer to Windows without a different oracle. The script above is not a portable abstraction.

It does not belong on patches that are explicitly space-time tradeoffs unless the envelope update rides along in the same review. A grader that always loses to an intentional cache is theatre.

It should not run under extra sanitizers and then compare against a non-sanitized envelope. That comparison is meaningless.

Skip the technique when the binary's memory is dominated by a huge mmap the patch cannot affect. The ceiling will never move. The signal is zero.

Closing

Unit tests answer whether the answer was right. They do not answer whether the process was allowed to become larger while getting that answer. Commit both oracles. Grade both on every model patch.

The sample files above are small enough to drop beside an existing C++ eval. A stable host matters more than a clever dashboard. If patch generation already has spare capacity, envelope.json is the cheaper artifact to try on that same host first.