Small Fixtures Hide Steep Slopes: A Cost-Class Gate for AI C++ Patches

Consider a review that starts green. A generated C++ patch matches every golden file. The fixture holds 32 rows. The function returns the expected vector, CI paints the build green, and the review moves on.

Two weeks later the same function meets a 200,000-row dump. Latency leaves the budget. A nested scan has replaced a single pass. Nothing in the small fixture could see the slope change.

That gap is the subject of this gate. Exact output on a toy input is necessary. It is not a cost contract.

What a cost-class gate is

A cost-class gate runs one pure function on a geometric series of input sizes and compares growth against a pinned baseline. The lane does not claim a portable benchmark. It only asks whether the candidate grew steeper than that baseline on one machine, under one build, on one afternoon.

The method below is a proposal. The listings are unexecuted examples. They are not a lab report, and they are not a ranking of any model.

Step 1: Freeze the function contract

Pin the signature before any model writes a patch. A drifting signature makes later ratios incomparable.

// proposal: include/scan.hpp
#pragma once
#include <cstdint>
#include <span>

// Count values strictly greater than pivot. Must stay a pure function.
std::uint64_t count_above(std::span<const std::int32_t> xs, std::int32_t pivot);

The patch prompt includes this header and a ban on editing it. The harness compiles the candidate translation unit against that frozen header. A signature mismatch fails before any timer starts.

Keep the size table, the seed, and the grader sources out of the prompt. A model that can see the stopwatch will eventually write to the stopwatch.

Step 2: Keep a tiny exact lane

Small cases still earn their keep. They catch wrong answers that a slope check would happily time.

// proposal: tests/exact_lane.cpp
#include "scan.hpp"
#include <cassert>
#include <vector>

static void exact_lane() {
  const std::int32_t a[] = {1, 5, 3, 9};
  assert(count_above(a, 4) == 2);
  const std::vector<std::int32_t> empty;
  assert(count_above(empty, 0) == 0);
  const std::int32_t neg[] = {-2, -1, 0};
  assert(count_above(neg, -1) == 1);
}

This lane stays deterministic. No clocks sit here, and no allocation budgets either. A red exact lane stops the job. A green exact lane is only permission to continue.

Step 3: Scale the input and compare slopes

Generate inputs the model never sees. Sizes double from 4,096 through 65,536. A fixed seed makes a rerun hit the same bytes.

Time each size three times and keep the median. Print a doubling ratio and a checksum of the full result. The checksum stops an early return on large n from looking linear.

// proposal: tests/slope_lane.cpp
#include "scan.hpp"
#include <chrono>
#include <cstdint>
#include <iostream>
#include <random>
#include <utility>
#include <vector>

static std::vector<std::int32_t> make_input(std::uint32_t n, std::uint32_t seed) {
  std::mt19937 rng(seed);
  std::uniform_int_distribution<std::int32_t> dist(-1000, 1000);
  std::vector<std::int32_t> xs(n);
  for (auto& v : xs) v = dist(rng);
  return xs;
}

static double median_ms(const std::vector<std::int32_t>& xs) {
  double samples[3];
  for (int i = 0; i < 3; ++i) {
    const auto t0 = std::chrono::steady_clock::now();
    volatile std::uint64_t sink = count_above(xs, 0);
    (void)sink;
    const auto t1 = std::chrono::steady_clock::now();
    samples[i] = std::chrono::duration<double, std::milli>(t1 - t0).count();
  }
  if (samples[0] > samples[1]) std::swap(samples[0], samples[1]);
  if (samples[1] > samples[2]) std::swap(samples[1], samples[2]);
  if (samples[0] > samples[1]) std::swap(samples[0], samples[1]);
  return samples[1];
}

int main() {
  const std::uint32_t sizes[] = {4096u, 8192u, 16384u, 32768u, 65536u};
  double prev = 0.0;
  for (std::uint32_t n : sizes) {
    const auto xs = make_input(n, 268u);
    const double ms = median_ms(xs);
    const double ratio = (prev == 0.0) ? 0.0 : ms / prev;
    std::cout << n << ' ' << ms << ' ' << ratio << '\n';
    std::cout << "sum " << count_above(xs, 0) << '\n';
    prev = ms;
  }
}

Both the baseline tree and the candidate tree link against this same driver. Only the translation unit that defines count_above changes.

# proposal: grade_slope.py — unexecuted
def ratios_from_lines(lines):
    ratios = []
    for line in lines:
        parts = line.split()
        if len(parts) == 3 and parts[0].isdigit():
            ratio = float(parts[2])
            if ratio > 0.0:
                ratios.append(ratio)
    return ratios

def checksums(lines):
    out = []
    for line in lines:
        parts = line.split()
        if len(parts) == 2 and parts[0] == "sum":
            out.append(parts[1])
    return out

def grade(base_lines, cand_lines, margin=0.8):
    if checksums(base_lines) != checksums(cand_lines):
        return "checksum_mismatch"
    base = ratios_from_lines(base_lines)
    cand = ratios_from_lines(cand_lines)
    if len(base) < 2 or len(cand) < 2:
        return "inconclusive"
    jumps = [c - b for b, c in zip(base[-2:], cand[-2:])]
    if any(j > margin for j in jumps):
        return "steep"
    if any(j < -margin for j in jumps):
        return "inconclusive"
    return "flat_enough"

if __name__ == "__main__":
    import sys
    base = open(sys.argv[1]).read().splitlines()
    cand = open(sys.argv[2]).read().splitlines()
    print(grade(base, cand))

margin is a policy knob for this proposal, not a measured constant from a published run. A positive jump means the candidate doubling ratio pulled away from the baseline. A large negative jump is not an automatic win. It can be a real improvement, or it can be a timer fluke, so the proposal fails closed.

Store the baseline transcript beside the prompt identifier for that tree. A prompt edit invalidates the reference. Recapture the baseline before the next candidate is judged.

Step 4: Block the obvious escapes

A patch can game a timer without fixing the algorithm. Reject those edits before trusting a ratio.

Escape	What it looks like	Gate
Shrink the work	Early return when xs.size() exceeds a constant	Checksum of a full scan on every scaled n
Edit the build	Optimization flag dropped, harness edited	Hash CMakeLists.txt and the harness sources; mismatch fails
Specialize the seed	Hard-code a seed seen in the prompt and skip real work	Seed stays out of the prompt and arrives only at runtime
Replace the driver	Candidate ships its own main	Candidate may supply only count_above
Match a bad baseline	First green patch becomes the reference	Baseline comes from a pinned, reviewed tree, never from the candidate

# proposal: commands, not a recorded run
c++ -std=c++20 -O2 -Iinclude src/baseline_scan.cpp tests/slope_lane.cpp -o /tmp/slope_base
c++ -std=c++20 -O2 -Iinclude src/candidate_scan.cpp tests/slope_lane.cpp -o /tmp/slope_cand
/tmp/slope_base > /tmp/base.txt
/tmp/slope_cand > /tmp/cand.txt
python3 grade_slope.py /tmp/base.txt /tmp/cand.txt
sha256sum CMakeLists.txt tests/slope_lane.cpp include/scan.hpp

Both binaries use the same flags. Flag drift belongs to a different grader. This lane assumes that set is already frozen. If either link fails, the cost class is irrelevant and the job stops at compile.

Step 5: Split generation from timing

Candidate patches can be regenerated many times. Timed slope runs should not share a noisy laptop with an editor, a browser, and a hot build directory.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. The operator describes MonkeyCode as an open-source project with free model access and a free server option. This article does not freeze a token allowance, a model list, a hardware shape, or a duration. Those terms change. Read them on the current project page before relying on a number.

Free model access is a reasonable place to iterate the patch prompt. A slope failure is a signal to rewrite the prompt, not a license to relax the gate. The free server option is a reasonable place to run the scaled lane, because the clock should sit on a machine that is not also compiling the next attempt. Neither option turns a single-host ratio into a published benchmark. A pass on that server means the candidate was not steeper than baseline there, under that hour's load.

One practical split keeps the two jobs apart. Generate the candidate where free model access is currently offered. Copy only candidate_scan.cpp into the grader tree. Time baseline and candidate back to back on the free server. If that server is busy, mark the lane inconclusive rather than filing a noisy ratio as a regression.

How to read the grade

A red slope is not a license to edit margin until the patch passes. Recapture the baseline on a quiet host, then judge the candidate again. Three outcomes cover the CI contract.

1. flat_enough, with matching checksums. Accept the cost lane. Exact-lane failures still block the patch.
2. steep, with matching checksums. Reject. The bytes are complete and the growth class jumped.
3. inconclusive, a checksum mismatch, or a compile error. Reject the automation. A person reads the diff. Do not lower the margin in the same commit that adds the candidate.

Limitations

Wall-clock ratios lie on a contended host. A neighbor process can push a linear function over the margin. Three samples are a noise filter, not a statistical study.

Constant factors hide inside a class. An algorithm can stay in the same class and still miss a production budget. This gate will not see that. A separate absolute budget, chosen by the service owner, has to carry that requirement.

Cache effects bend small sizes. That is why the comparison reads the last two doublings and ignores the warmup ratio. On a very small free-server instance, even the tail may sit in the noise floor. The honest result is then inconclusive, not a forced pass.

The checksum stops the trivial early-return cheat. It does not stop every adversarial patch. A candidate can still special-case sizes if those sizes leak into the prompt. Keep the size table in the harness.

A sanitizer lane for overflow and bounds is still required. This proposal does not replace it. Overflow in count_above can agree on small fixtures and disagree on scaled ones only after values wrap, which is a different failure than a steep ratio.

Who should not use this

Teams grading I/O-bound or network code should not adopt a CPU slope gate. The ratio will track the socket, not the patch.

Teams without a pinned baseline should not invent one from the first green candidate. That first candidate then becomes the slope every later patch is allowed to match, including a quadratic one.

Reviewers who need a cross-vendor performance claim should not quote these ratios. One host, one flag set, and one afternoon of load are not a leaderboard.

Anyone treating a free-server clock as a permanent lab should also stop. Availability and noise there are facts of the day, not a contract this grader can sign.

Closing

Small golden files still belong in the harness. They answer whether the bytes are right. They stay silent when the bytes are right and the cost class is wrong.

Add a scaled lane beside the exact lane. Fail closed when the doubling ratio jumps away from the pinned baseline. Keep generation and timing on separate machines so a laptop fan never becomes the acceptance test. Readers who already grade C++ patches can drop this lane next to their exact cases, generate candidates with whatever free model access their account currently lists, and leave the slope clock on a quieter free server whose present terms they have actually read.