The Free Model Wrote Tests That Passed. They Still Missed a Bug.

The free model wrote four unit tests for a small C++ function. All four passed. The function was still wrong. A coverage tool found the gap that the tests missed.

This is a case study about a 10-line function, a model-generated test suite, and the one edge case that slipped through. The lesson is not that free models are unreliable. The lesson is that passing tests are not evidence of correctness — and coverage is the cheapest way to check.

Why This Function

We are building a small HTTP server for an internal tool. One component parses a port number from a configuration string. The contract is simple: return the port as an integer if the string is a valid port (1–65535), otherwise return -1.

The function is small enough to review in one sitting, but it has enough branches to make a test suite interesting. There is an empty-string guard, a digit check, an overflow check, and a range check. Each branch is a potential failure point.

The Function Under Test

// port.h
#pragma once
#include <string>

int parse_port(const std::string& s);

// port.cpp
#include "port.h"
#include <cctype>

int parse_port(const std::string& s) {
    if (s.empty()) return 0;  // BUG: should be -1
    int value = 0;
    for (char c : s) {
        if (!std::isdigit(static_cast<unsigned char>(c))) return -1;
        value = value * 10 + (c - '0');
        if (value > 65535) return -1;
    }
    return value >= 1 ? value : -1;
}

The bug is deliberate: an empty string returns 0 instead of -1. The function compiles, and most inputs work correctly. The bug only appears when the string is empty.

The Model's Test Suite

We asked MonkeyCode's free model endpoint to generate tests. The prompt was minimal:

Write a C++ test for parse_port. Cover normal cases, boundary values, and invalid inputs.

The model returned four assertions:

#include <cassert>
#include "port.h"

int main() {
    assert(parse_port("8080") == 8080);
    assert(parse_port("1") == 1);
    assert(parse_port("65535") == 65535);
    assert(parse_port("abc") == -1);
    return 0;
}

These look reasonable. They cover a normal value, the minimum, the maximum, and a non-numeric string. The model also added a comment: "The tests cover the main paths and boundary values."

We compiled and ran the test:

g++ -std=c++17 -o test test.cpp port.cpp && ./test

Exit code 0. All four assertions passed.

The Verification Pipeline

The full workflow has three stages:

1. Generate tests with the free model.
2. Run the tests with a plain build.
3. Run the same tests with coverage instrumentation.

The first two stages are what most teams do. The third stage is the one that catches blind spots. The pipeline is a few commands:

# Stage 1: plain build and run
g++ -std=c++17 -o test test.cpp port.cpp && ./test

# Stage 2: coverage build and run
g++ -std=c++17 --coverage -o test test.cpp port.cpp && ./test

# Stage 3: report
gcov -b port.cpp

The coverage build uses the same test binary. No extra test code is needed. The only cost is a recompile and a few seconds of runtime.

This pipeline is not specific to model-generated tests. It works for any test suite. But it is especially valuable when you did not write the tests yourself, because you have no intuition about what they cover.

The Coverage Run

We ran the coverage build on MonkeyCode's free server. The branch coverage report showed a gap:

        1:    5:    if (s.empty()) return 0;  // BUG
branch  0 taken 1 (fallthrough)
branch  1 taken 0 (return)

The s.empty() condition was evaluated, but the true branch — the return 0 path — was never taken. The model's tests never called parse_port with an empty string.

That is the blind spot. The tests exercised the loop, the digit check, and the overflow check, but not the empty-input guard.

The Missing Test

We added one assertion based on the coverage report:

assert(parse_port("") == -1);

The test failed immediately:

test: test.cpp:9: int main(): Assertion `parse_port("") == -1' failed.

The function returned 0. The bug was real, and the model's test suite had missed it.

The Fix

The fix is one line:

if (s.empty()) return -1;

After the fix, the full test suite — including the new empty-string test — passed. The coverage report showed 100% branch coverage for parse_port.

Other Edge Cases the Model Missed

The coverage report also showed that the model never tested:

• "65536" — the overflow path.
• "+80" — a string with a leading sign.
• " 80" — a string with leading whitespace.

None of these triggered a bug, but they were unvisited branches. The empty string was the only one that exposed the defect.

Lessons Learned

Model-generated tests are biased toward typical inputs. The model saw "port number" and generated tests for numbers, boundaries, and obvious invalid strings. It did not think about the empty string, because empty strings are rare in examples and common in production.

Coverage is the test for tests. A green test suite tells you nothing about what was not executed. A coverage report shows you the unvisited branches. It is the difference between "the tests pass" and "the code is correct."

One missing edge case can hide a real bug. The bug in parse_port was not in the parsing logic. It was in the input guard. The model's tests never reached that guard.

Who Should Not Use This Approach

• If your function has no input validation and no branches, coverage adds little value.
• If you are writing throwaway scripts, a manual test is faster.
• If your team already writes exhaustive boundary tests, the model's output is a starting point, not a final answer.

Limitations

This is one function, one model output, one toolchain. It does not prove that all model-generated tests are weak. It proves that passing tests can miss real bugs, and that coverage tools catch what reviews miss.

The workflow is simple: generate with the free model, verify with the free server's coverage tools. That combination caught a bug that four passing tests missed.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.