You merge an agent patch that is supposed to make the build cache smarter. Every unit test passes, the integration suite is green, and the code review looks clean. Two days later, a colleague complains that the service now spends twice as long reading configuration files. You profile, you find a hidden O(n^2) loop, and you realize the tests never measured performance.
This scenario plays out whenever correctness and speed are treated as the same thing. Unit tests assert behavior, not complexity. A patch can produce perfect output while silently degrading the runtime from microseconds to milliseconds. For human reviewers this is caught by experience or a code review checklist. For agent-generated patches, there is no experience in the code generation loop, so the regression ships.
The solution is not more unit tests. The solution is a benchmark gate that runs before every merge and compares the new patch against a known baseline. In this guide you'll build one for a C++ project using free-tier infrastructure, and you'll see how to keep the whole thing cheap enough to run on every commit.
Why Unit Tests Miss Performance Regressions
Unit tests typically run on tiny, fixed inputs and verify output values. They rarely measure wall time, and when they do, timing assertions are often flaky. A performance regression usually appears only when an input exceeds a certain size or when memory pressure changes. For example, a function that uses std::list instead of std::vector may still return correct results while losing cache locality.
Performance is also a property of the compiled binary, not the source text. Debug builds and non-optimized tests mask complexity differences. Agents often produce code that is logically correct but algorithmically worse, such as comparing every element in a vector with a nested loop instead of using a hash set. Unit tests can't catch that.
Step 1: Identify the Hot Paths
Begin with a profiler, or at least with the functions you already suspect. A hot path is any function that appears high on the CPU profile during normal operation. Parsers, serializers, cache lookups, and sorting routines are classic candidates. If a function is called a million times per request, even a two-nanosecond insertion will add up.
Pick two or three functions that the agent is most likely to touch, and make that the first version of your benchmark set. You can expand later once the gate proves useful.
Step 2: Write Micro-Benchmarks
A micro-benchmark isolates one function and repeatedly invokes it with a representative input. The Google Benchmark library is the standard choice for C++ and produces JSON output that you can parse automatically.
#include <benchmark/benchmark.h>
#include "parser.h"
static void BM_ParseHeader(benchmark::State& state) {
std::string line = "GET /index.html HTTP/1.1";
for (auto _ : state) {
auto header = parse_header(line);
benchmark::DoNotOptimize(header);
}
}
BENCHMARK(BM_ParseHeader);
BENCHMARK_MAIN();
Compile the benchmark with optimizations enabled. A -O0 binary will report misleadingly large times that hide algorithmic differences. Make sure the input string represents a realistic request header, not a minimal one.
Step 3: Compare Against a Baseline
The gate needs two binaries: one from the base commit and one from the candidate patch. Run both with --benchmark_format=json and compute the ratio per benchmark.
import json, subprocess, sys
def run_benchmark(binary):
out = subprocess.check_output([binary, "--benchmark_format=json"])
return {b["name"]: b["real_time"] for b in json.loads(out)["benchmarks"]}
base = run_benchmark("build/base_bench")
cand = run_benchmark("build/cand_bench")
fail = False
for name, base_time in base.items():
cand_time = cand.get(name)
ratio = cand_time / base_time
if ratio > 1.05:
print(f"FAIL {name}: {base_time:.0f}ns -> {cand_time:.0f}ns ratio {ratio:.2f}")
fail = True
elif ratio > 1.01:
print(f"WARN {name}: ratio {ratio:.2f}")
print("Benchmark gate", "failed" if fail else "passed")
sys.exit(1 if fail else 0)
Store the script as bench_gate.py, and add a CI step that builds both revisions. For a git-based workflow, you can check out the base commit into a separate build directory and locate the binary.
Step 4: Wire the Gate to Your CI
Your CI job should run the base build and the candidate build sequentially, then execute the script. If the script exits with code 1, the pipeline stops and the patch is not merged. To avoid false alarms from environment noise, repeat each benchmark a few times and use the median value. You can also pin the CPU frequency with cpupower when running on a dedicated server.
This is where a free server option becomes valuable. Building two C++ revisions and running benchmarks on every patch consumes CPU minutes, but not the kind that requires a GPU or a large instance. A simple server with a few gigabytes of RAM is enough for most codebases.
MonkeyCode's free model tier can draft the benchmark harness for you, and its free server option can run the comparison job after every push. The model doesn't need to be state-of-the-art, and the server doesn't need a GPU; the bottleneck is pure CPU and disk. You can keep the whole gate running under a single free account. Disclosure: This article was prepared as part of MonkeyCode's product outreach.
A Decision Table for Benchmark Results
Set explicit thresholds and stick to them. A 1% change is often noise; a 5% change is usually intentional or problematic.
| Ratio (current / baseline) | Verdict | Action |
|---|---|---|
| <= 1.01 | Pass | Merge without extra review |
| 1.01 - 1.05 | Warn | Require human review and a profiler check |
| > 1.05 | Fail | Block merge and ask the agent to rework |
Use the same thresholds in your script so the CI verdict matches the table. Adjust the values if your benchmarks are noisy, but document the change in the repository.
Limitations of Benchmark Gates
Benchmarks measure only what you make them measure. A single warm input can miss regressions that appear under a different distribution, such as many tiny strings instead of one large one. Environment noise remains a threat even on a dedicated box; CPU frequency scaling and background processes can skew times.
Micro-benchmarks also add compile and run time to every merge. A free server may be slower than paid CI, so keep the benchmark set small at first. Finally, some performance problems only surface under memory pressure or concurrency, and no micro-benchmark will catch those.
Who Should Skip This
If your application spends most of its time waiting on I/O or a database, a CPU micro-benchmark gate will not protect you. If your team already uses end-to-end load tests with realistic data, adding synthetic benchmarks might be redundant. And if the codebase is a glue layer with no hot paths, the cost of maintaining benchmarks outweighs the benefit.
Still, for a system parser, a search engine, or any CPU-bound component, this gate is one of the cheapest ways to block an invisible regression.
The Takeaway
A benchmark gate turns an invisible performance regression into a merge-blocking signal. It is not perfect, but it is far better than discovering the slowdown after release. Start with two benchmarks on your hottest function, set a 5% threshold, and let a free server run them every time an agent proposes a change. The question "did this patch make things slower?" deserves a concrete answer.
Top comments (0)