Case Study: A Free Model's UTF-8 Validator Passed My Tests. The Spec Table Found What Both Missed.

I needed a small UTF-8 validator. A free model wrote one in a single pass, and it passed every unit test I wrote in the same session. That was the problem: my tests shared the model's blind spots.

A 31-case conformance table, hand-derived from RFC 3629, found nine failures in that draft. A one-million-string property test then found a tenth — a bug the table could not catch by construction. The code was never the interesting part. The harness was.

Background

The project was deliberately small. I wanted a single-file C++17 tool, utf8check, that reads bytes from stdin and prints either valid or the byte offset of the first invalid sequence. Exit code 0 or 1. No ICU, no external dependencies.

UTF-8 looks simple. The RFC is four rules: leading bytes, continuation bytes, overlong rejection, and the surrogate and range limits. It is exactly the kind of task a code model should handle cleanly. It is also exactly the kind of task where a shallow implementation passes happy-path tests and quietly accepts C0 80 as a NUL.

Goal

Three requirements, in order:

1. Validate any byte string against RFC 3629.
2. Report the offset of the first invalid byte, not just a boolean.
3. Compile as C++17 with no warnings and no dependencies.

I did not require performance. The input domain is user-uploaded text files, not a hot parser.

Implementation

The workflow had three layers, and only the first involved the model.

Layer 1 — draft. I used MonkeyCode's free model access to generate the first implementation from a one-paragraph spec: validate per RFC 3629, report the first invalid offset, exit 0 or 1. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Layer 2 — smoke loop. The free server option let me run the compile-and-test cycle without standing up my own endpoint. I treated it as a smoke tier: compile, run the quick checks, read the diff, feed the failure back. Cheap iteration is useful. It is not evidence of correctness.

Layer 3 — the gate. This is the part the model never saw. I built two independent checks: a conformance table written by hand from the RFC, and a property test that compares the tool against a second, independent decoder.

Artifact 1: the conformance table

Random byte strings are bad at structured edge cases. The probability of a random string forming C0 80 is about 1 in 65,536. The probability of forming ED A0 80 is similar. A table targets those boundaries directly.

struct Case { const char* bytes; size_t len; bool valid; const char* note; };

const Case kSpec[] = {
    // valid boundaries
    { "\x00", 1, true,  "U+0000" },
    { "\x7F", 1, true,  "U+007F" },
    { "\xC2\xA9", 2, true, "U+00A9" },
    { "\xDF\xBF", 2, true, "U+07FF" },
    { "\xE0\xA0\x80", 3, true, "U+0800" },
    { "\xED\x9F\xBF", 3, true, "U+D7FF" },
    { "\xEE\x80\x80", 3, true, "U+E000" },
    { "\xEF\xBF\xBF", 3, true, "U+FFFF" },
    { "\xF0\x90\x80\x80", 4, true, "U+10000" },
    { "\xF4\x8F\xBF\xBF", 4, true, "U+10FFFF" },
    // overlong
    { "\xC0\x80", 2, false, "overlong U+0000" },
    { "\xC1\xBF", 2, false, "overlong U+007F" },
    { "\xE0\x80\x80", 3, false, "overlong U+0000" },
    { "\xE0\x9F\xBF", 3, false, "overlong U+07FF" },
    { "\xF0\x80\x80\x80", 4, false, "overlong U+0000" },
    { "\xF0\x8F\xBF\xBF", 4, false, "overlong U+FFFF" },
    // surrogates
    { "\xED\xA0\x80", 3, false, "U+D800" },
    { "\xED\xBF\xBF", 3, false, "U+DFFF" },
    // out of range
    { "\xF4\x90\x80\x80", 4, false, "U+110000" },
    { "\xF5\x80\x80\x80", 4, false, "bad lead" },
    { "\xF8\x88\x80\x80\x80", 5, false, "5-byte lead" },
    { "\xFE", 1, false, "FE lead" },
    { "\xFF", 1, false, "FF lead" },
    // truncation
    { "\xC2", 1, false, "truncated 2-byte" },
    { "\xE2\x82", 2, false, "truncated 3-byte" },
    { "\xF0\x9F\x92", 3, false, "truncated 4-byte" },
    // continuation errors
    { "\x80", 1, false, "lone continuation" },
    { "\xBF", 1, false, "lone continuation" },
    { "\xC2\x41", 2, false, "expected continuation" },
    { "\xE2\x28\xA1", 3, false, "expected continuation" },
    { "\xF0\x9F\x28\xA9", 4, false, "expected continuation" },
};

Thirty-one cases, about twenty minutes to write. The boundary values come from the RFC's own tables: the lowest and highest code point for each sequence length, the surrogate window, and the 0x10FFFF ceiling.

Artifact 2: the property test

The second gate compares utf8check against an independent decoder written directly from the RFC. The two implementations share no code and no logic.

bool oracle_valid(std::string_view s) {
    size_t i = 0;
    while (i < s.size()) {
        const unsigned char c = static_cast<unsigned char>(s[i]);
        if (c < 0x80) { ++i; continue; }

        int extra = 0;
        uint32_t cp = 0;
        if      ((c & 0xE0) == 0xC0) { extra = 1; cp = c & 0x1F; }
        else if ((c & 0xF0) == 0xE0) { extra = 2; cp = c & 0x0F; }
        else if ((c & 0xF8) == 0xF0) { extra = 3; cp = c & 0x07; }
        else return false;

        if (i + extra >= s.size()) return false;
        for (int j = 1; j <= extra; ++j) {
            const unsigned char cc = static_cast<unsigned char>(s[i + j]);
            if ((cc & 0xC0) != 0x80) return false;
            cp = (cp << 6) | (cc & 0x3F);
        }
        if (extra == 1 && cp < 0x80) return false;
        if (extra == 2 && cp < 0x800) return false;
        if (extra == 3 && cp < 0x10000) return false;
        if (cp >= 0xD800 && cp <= 0xDFFF) return false;
        if (cp > 0x10FFFF) return false;
        i += extra + 1;
    }
    return true;
}

Two properties, one million random byte strings each.

P1 — validity agreement. For every string, utf8check_valid(s) must equal oracle_valid(s).

P2 — error offset contract. For every invalid string, the reported offset must be the first byte that breaks validity. The prefix before the offset must be valid; the prefix including the offset must not.

std::mt19937 rng(0xC0FFEE);

for (int iter = 0; iter < 1'000'000; ++iter) {
    std::string s(rng() % 24, '\0');
    for (char& ch : s) ch = static_cast<char>(rng() % 256);
    if (oracle_valid(s) != utf8check_valid(s)) {
        report_mismatch(iter, s);
        break;
    }
}

for (int iter = 0; iter < 1'000'000; ++iter) {
    std::string s(rng() % 24, '\0');
    for (char& ch : s) ch = static_cast<char>(rng() % 256);
    if (oracle_valid(s)) continue;
    const size_t off = utf8check_first_error(s);
    if (!oracle_valid(s.substr(0, off)) ||
         oracle_valid(s.substr(0, off + 1))) {
        report_offset_bug(iter, s, off);
        break;
    }
}

Results

Gate	Cases	Draft 1	Draft 2
Unit tests written with the draft	12	pass	pass
Spec conformance table	31	9 failures	pass
Property P1 (validity)	1,000,000	not reached	pass
Property P2 (error offset)	1,000,000	not reached	1 bug class

Draft 1 checked leading bytes and continuation bytes, then stopped. It accepted overlong encodings, surrogate code points, and F4 90 80 80 (U+110000). My 12 unit tests missed all of it, because I wrote them from the same mental model the model used: valid examples plus a few obvious invalid ones.

Draft 2 fixed the table. It also passed P1. Then P2 caught the interesting bug: for a truncated sequence at the end of input, the tool reported the offset of the last byte it consumed, not the start of the broken sequence. For F0 9F 92, it reported 2 or 3 instead of 0. The boolean was right. The contract was wrong.

Lessons

Tests written in the same session as the code inherit its blind spots. The model and I were agreeing with each other, not with the spec. The fix is to break the symmetry: derive test cases from the standard, not from the implementation.

Random strings are the wrong first tool for structured edges. One million random strings will rarely construct C0 80 or ED A0 80. The 31-case table hit all of them in one run. But the table could not express the offset contract — that needed the property test. The two tools are not substitutes.

Cheap iteration is not correctness. The free server option made the smoke loop fast, and the free model access made the draft cheap. Neither changed the fact that the gate had to be independent of both. Treat the model's output as a draft, not a verdict.

Error reporting is a contract. "Is this string valid?" and "Where does it first break?" are different properties. Test both, or your tool will pass every test and still mislead its callers.

Limitations

This harness proves conformance for the cases in the table and statistical agreement with one oracle on random strings. It is not a proof of correctness. The oracle was written from the same RFC, so a shared misreading of the spec would defeat the comparison — which is exactly why the hand-derived table matters.

Do not use this approach for a security boundary. If the validator parses untrusted input in a network service, add libFuzzer with ASan and UBSan, and consider a formally verified decoder instead. And if you need error recovery or normalization rather than rejection, this whole design is the wrong shape.

The harness is small enough to rebuild in an afternoon. The table is the part worth stealing.