A Free Model Rewrote My Parser. Differential Testing Decided Whether I Merged It.

Every free model I have used can write a plausible atoi in seconds. Plausible is not correct, and the distance between those two words is exactly where agent-generated patches tend to hide their bugs. This article is a concrete verification ladder for that situation: unit tests, differential testing, sanitizers, and a decision table that tells you when a patch is safe to merge.

I ran this ladder on a real patch produced by a free model. The patch looked clean, passed a few obvious tests, and still contained three distinct defects. The ladder caught all three before the patch touched production.

The patch that looked too clean

The task was simple: implement a function that behaves like the C standard library's atoi. The free model returned this implementation in one shot.

// my_atoi.cpp
int my_atoi(const char* s) {
  int sign = 1, result = 0;
  if (*s == '-') { sign = -1; ++s; }
  while (*s >= '0' && *s <= '9') {
    result = result * 10 + (*s - '0');
    ++s;
  }
  return sign * result;
}

At first glance, the logic is readable and the structure is familiar. The function handles a leading minus sign, iterates over digits, and builds the result. It even returns the correct value for "42" and "-7". The first unit tests confirm that impression.

// test_basic.cpp
#include <cassert>

int my_atoi(const char* s);

int main() {
  assert(my_atoi("42") == 42);
  assert(my_atoi("-7") == -7);
  assert(my_atoi("0") == 0);
  assert(my_atoi("12345") == 12345);
  return 0;
}

All four assertions pass. If the review stopped here, the patch would be merged. It should not be.

Step 1: Unit tests are the floor, not the ceiling

Unit tests encode the expectations you already have. They rarely surprise you, and they never catch the case you forgot to imagine. For a parser, the forgotten cases are usually whitespace, an explicit plus sign, an empty string, and integer overflow.

A slightly less friendly test suite exposes the first two defects immediately.

// test_edges.cpp
#include <cassert>

int my_atoi(const char* s);

int main() {
  assert(my_atoi("  42") == 42);   // leading whitespace
  assert(my_atoi("+7") == 7);      // explicit plus
  assert(my_atoi("") == 0);        // empty string
  return 0;
}

This suite fails on the first assertion. The implementation never skips leading whitespace, so " 42" produces 0. The plus sign is also ignored, so "+7" produces 0. These are not exotic edge cases; they are part of the C standard's contract for atoi. The free model simply did not implement the full contract.

Unit tests caught the obvious gaps. They could not catch the overflow bug, because overflow only appears with very large inputs that no human writes by hand.

Step 2: Differential testing against the reference

Differential testing is the practice of feeding the same input to your implementation and to a trusted reference, then comparing the outputs. For atoi, the reference is the standard library function of the same name. The comparison is mechanical, so you can run millions of inputs without manual effort.

The following program generates random strings, feeds them to both implementations, and reports the first mismatch.

// diff_test.cpp
#include <cstdlib>
#include <iostream>
#include <random>
#include <string>

int my_atoi(const char* s);

int main() {
  std::mt19937 rng(20260826);
  std::uniform_int_distribution<int> len_dist(0, 12);
  std::uniform_int_distribution<int> char_dist(32, 126);

  for (int i = 0; i < 100000; ++i) {
    const int len = len_dist(rng);
    std::string s;
    for (int j = 0; j < len; ++j) {
      s.push_back(static_cast<char>(char_dist(rng)));
    }

    const int expected = std::atoi(s.c_str());
    const int actual = my_atoi(s.c_str());
    if (expected != actual) {
      std::cout << "Mismatch on input: " << s << "\n";
      std::cout << "Expected: " << expected << "\n";
      std::cout << "Actual:   " << actual << "\n";
      return 1;
    }
  }

  std::cout << "All differential tests passed\n";
  return 0;
}

Run it with:

g++ -std=c++17 -O2 diff_test.cpp my_atoi.cpp -o diff_test
./diff_test

The differential test finds a mismatch almost immediately. The input is a long digit string that overflows int. The standard library clamps the result to the implementation-defined value, while my_atoi wraps around through signed integer overflow, which is undefined behavior in C++.

Differential testing turned an invisible undefined behavior into a visible mismatch. That is exactly what you want from a verification step.

Step 3: Sanitizers and the merge decision table

Differential testing tells you that the outputs differ. Sanitizers tell you why. Compile the same implementation with UndefinedBehaviorSanitizer and run a targeted overflow input.

g++ -std=c++17 -fsanitize=undefined -fno-sanitize-recover=all test_overflow.cpp my_atoi.cpp -o test_overflow
./test_overflow

The sanitizer reports signed integer overflow at the line result = result * 10 + (*s - '0'). The root cause is confirmed: the free model's patch lacks overflow checking.

At this point, the verification ladder has produced three signals: unit tests fail, differential tests fail, and UBSan fails. The decision is obvious. But in a real workflow, signals do not always align. A decision table removes the guesswork.

Unit tests	Differential tests	Sanitizers	Decision
Pass	Pass	Pass	Merge and monitor
Pass	Fail	-	Reject, request fix
Pass	Pass	Fail	Reject, fix UB first
Fail	-	-	Reject, request fix
Fail	Fail	-	Reject, request rewrite

The table is deliberately strict. A patch that fails any gate goes back to the model with the failing test as feedback. The model gets another chance, and the ladder runs again. In my experience, two or three iterations usually produce a patch that passes all three gates.

How the free server fits in

I ran this entire ladder on MonkeyCode's free server option. The free model access generated the patch, and the free server executed the compile, the differential test, and the sanitizer run without tying up my local machine. The workflow is the same as a local terminal: clone the repository, write the test files, run the commands, and read the output. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The practical advantage is that the server becomes a repeatable verification environment. You can run the same commands for every model-generated patch, keep the logs, and compare results across iterations. The free server is not a CI system, but it does not need to be. It is a place where a patch earns a merge decision.

Limitations

This ladder is not a proof of correctness, and it has real limits.

• Differential testing requires a trusted reference. For atoi, the standard library is that reference. For a new algorithm, you may not have one.
• Sanitizers only catch the undefined behavior they are designed to detect. They do not catch logic errors that are technically well-defined.
• Random input generation can miss rare branches. The fixed seed makes the run reproducible, but it also means the same corpus runs every time.
• The decision table is conservative. It may reject patches that are actually correct but fail a weak test. That is a feature, not a bug, when the patch is generated by a model.

Who should skip this

Do not use this ladder for throwaway scripts, one-off migrations, or prototypes that will be deleted next week. The setup cost is real, and the payoff only appears when a patch will live in a codebase for months. If the patch is trivial and the failure mode is harmless, a quick manual review is enough.

For anything that touches user input, parsing, or numeric boundaries, the ladder is worth the effort. A free model can generate a plausible parser in seconds. The verification ladder is what turns plausible into correct.

If you want to see how far a free model's patch gets on your own code, the free server is a low-risk place to run this exact sequence. Generate a patch, write one differential test, and let the machine do the rejecting.