Clean Includes, Separate Grades: Probing AI C++ Patches Outside the Driver

A composite night-build scene, not a measured incident from this account, still matches a failure C++ evals keep shipping. A grader accepted a patch just after 02:00. Every golden assertion passed, and the diff touched only inventory.h plus inventory.cpp.

The next consumer file included that header and nothing else. Compilation stopped because std::string was unknown. The eval driver had included <string>, <vector>, and a project prelude before it loaded the header under test. The patch looked finished because the driver had already done the include work.

That failure mode is quiet. Functional graders never see it. A standalone translation-unit gate does.

What the green run actually proved

The driver is a convenience program. It exists to call functions and check return values. It is a poor witness for header hygiene.

A model can emit a public header that uses std::vector without including <vector>, then still compile, because the driver's first lines already pulled that header in. The same blind spot hides missing forward declarations, macros supplied by a unity prefix, and a precompiled header a free-standing consumer will never receive. A passing driver compile therefore proves less than the status line suggests.

A useful grade separates two checks. One check asks whether behavior matches the golden cases. The other asks whether a stranger can include the public header in a clean translation unit and compile a tiny call. A single green exit code blends those checks and hides the second failure.

The probe the harness should emit

The files below are a proposed workflow. They are not a recorded run, and they are not a benchmark. The operator points them at a patch directory, a public-header list, and a compiler command. That command can target a local toolchain or a clean remote tree.

Keep the probe beside the existing functional grader, not inside it. public_headers.txt names only headers a downstream file is allowed to include:

include/inventory.h

The broken header that motivates the gate looks like this:

#pragma once
namespace inventory {
struct Item {
  Item(std::string name, int qty);
  const std::string& label() const;
  std::string name;
  int qty;
};
}

A driver that already included <string> will compile a call through that header. A standalone consumer will not. The repaired form includes what it uses:

#pragma once
#include <string>
namespace inventory {
struct Item {
  Item(std::string name, int qty);
  const std::string& label() const;
  std::string name;
  int qty;
};
}

probe_main.cpp.in carries one call already present in the golden case. The harness substitutes the header path. It does not ask the model for a new specification, and it does not embed an expected output table in the header.

#include "HEADER_PATH"

int main() {
  inventory::Item item{"bolt", 3};
  return item.label().empty() ? 1 : 0;
}

The return value is only a liveness check so the compiler must instantiate the call. It is not the behavioral oracle. Behavioral truth stays in the golden-case lane.

Numbered steps

1. Copy the patched tree into a scratch directory. Drop prefix-header flags such as -include prelude.h for this lane only, and do not reuse a precompiled header or a unity build file. Leave the functional lane on its original flags so a logic regression still has its own signal.

2. Read public_headers.txt. Render one probe_main.cpp per public header from the template. The rendered file may include that single public header. It must not include the eval driver, the prelude, or a catch-all project header.

3. Compile the probe as its own translation unit. Pin the language mode in the command so a remote default cannot silently change the meaning of a pass. This invocation is a starting point, not a universal CI policy:
   

sed "s|HEADER_PATH|inventory.h|" probe_main.cpp.in > build/probe_inventory.cpp
c++ -std=c++20 -Wall -Wextra -Werror -I include -c build/probe_inventory.cpp -o build/probe_inventory.o

1. Classify the compiler result before reading the functional grade. A missing declaration, an incomplete type, or an unknown name is a hygiene failure even when every driver assertion passed. A successful object file is a pass for this lane alone.

2. Append a one-line record the next run can diff. The record is the gate. A dashboard is optional.
   

case=inventory header=include/inventory.h lane=standalone_tu status=fail class=missing_declaration

diff -u grades/last.txt grades/this.txt

1. Run the functional golden cases in a separate process with the original driver. Do not skip them after a hygiene failure. A patch can be self-sufficient and still wrong. A patch can be behaviorally right and still unusable from a clean include. The report should show both rows.

2. Repeat the compile on a second tree that has never been built on the developer machine, when such a tree exists. The useful property is the absence of local precompiled headers and cached unity outputs. Host size, region, and uptime are out of scope for the grade.

A small classifier

Map compiler text to a stable class in a wrapper the repo owns. The patterns below are proposals. They need adjustment when the compiler or the language pack changes.

#!/bin/sh
log="$1"
if grep -E -q "error: .*(does not name a type|was not declared|incomplete type|no member named)" "$log"; then
  echo "missing_declaration"
  exit 2
fi
if grep -E -q "error:" "$log"; then
  echo "other_compile_error"
  exit 3
fi
echo "clean"
exit 0

Treat the exit code as the contract. Prose in the log is evidence, not the grade key. A later compiler upgrade can rename a sentence without changing the exit status the harness stores.

Decision table

Probe class	Functional grade	Report	Merge?
clean	pass	both lanes green	eligible for review
missing_declaration	pass	hygiene fail, behavior pass	no
missing_declaration	fail	both lanes red	no
other_compile_error	pass	flag or API drift	no
clean	fail	include ok, logic wrong	no
probe skipped	pass	incomplete grade	no

Skip is not green. A harness that cannot find a compiler, a header list, or a scratch tree must record an incomplete grade. Otherwise a missing tool becomes a silent pass, which is the same class of mistake as a driver that includes the world.

Where a hosted compile fits

Iteration is the costly part of the lane. Each failed probe names a missing include or an incomplete type. A coding model can suggest the repair. The probe accepts or rejects that repair.

The model is not the grader. A greener suggestion still has to survive the clean translation unit. That split is the whole point of keeping golden cases and the include probe in different processes.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. The outreach describes free model access and a free server option for this loop: request a header repair, then compile the standalone probe on a clean tree that does not share the laptop's precompiled headers. Those two availability claims are operator-supplied.

This article does not state a token quota, a hardware spec, a duration, or a permanent offer, because no primary source for those details was attached. Confirm the current limits, model list, and server terms on the project page before a team depends on them. A local compiler is enough to run the gate.

Use the hosted compile as a second witness, not as the only witness. If the remote log and the local log disagree, keep both. Disagreement is a result. It often means a hidden -include flag, a different default language mode, or a header found through an ambient include path.

Limitations

This gate does not replace include-what-you-use, a link test, or a sanitizer build. A header can include <string> and still return a reference into a temporary. The sample's const std::string& accessor is a reminder of that gap, not a fix for it. Macro-heavy headers can pass one probe call and fail under another feature flag.

The classifier's regular expressions are compiler-specific and will miss a diagnostic emitted in another language. The workflow also assumes the operator knows which headers are public. Point it at an internal header and it will punish legal layering.

Teams that already compile every public header inside a hermetic image with a pinned compiler gain little from a duplicate of that job. Projects whose documented contract is a single umbrella header should probe the umbrella, not each internal file. Readers who need a reproducible claim about a free allowance or a free host should use the current project documentation, not this walkthrough, as the quote.

Closing

A driver that includes the world will keep congratulating incomplete headers. Split the grade. One lane checks behavior against golden cases. The other compiles a stranger's file that includes only the public header.

Record both rows. Send the patch to review only when neither lane had to borrow an include from the harness. A green assertion log, by itself, is not evidence that the header can stand alone.