A Free Model's UTF-8 Validator Passed 100k Round-Trips. The Spec Corpus Failed It.

TL;DR

A free model's C++ UTF-8 validator passed 100,000 round-trip checks and a million sanitizer-fed random byte strings. It failed 6 of 10 Unicode conformance vectors. Round-trip tests are a weak oracle for decoders: they only prove valid input survives. The gate that mattered was negative — overlong encodings, surrogates, and out-of-range code points.

Background

I needed a small UTF-8 validator for a log-processing tool. Logs arrived in mixed encodings; the tool had to reject malformed UTF-8 before parsing, not after. The requirement was boring: one function, C++17, no dependencies, no allocation.

The project was too small for a full review cycle, but too security-adjacent to skip testing. That tension is where I use generated code: draft fast, gate hard.

Goal

Three criteria for done:

1. is_valid_utf8 accepts every valid UTF-8 sequence and rejects every invalid one.
2. No crashes on arbitrary bytes under ASan and UBSan.
3. The validation pipeline runs without a local runner.

Implementation

I asked MonkeyCode's free model access for the first draft. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The draft was the classic shape-checker:

bool is_valid_utf8(const std::string& s) {
    size_t i = 0;
    while (i < s.size()) {
        unsigned char c = s[i];
        if (c < 0x80) { i += 1; continue; }
        int extra;
        if ((c & 0xE0) == 0xC0) extra = 1;
        else if ((c & 0xF0) == 0xE0) extra = 2;
        else if ((c & 0xF8) == 0xF0) extra = 3;
        else return false;
        if (i + extra >= s.size()) return false;
        for (int k = 1; k <= extra; ++k)
            if ((s[i + k] & 0xC0) != 0x80) return false;
        i += extra + 1;
    }
    return true;
}

It looks plausible. It checks continuation bytes and bounds. It is also wrong in six distinct ways. I did not review it by eye first. I pushed it to the gate pipeline, which ran on MonkeyCode's free server option — a disposable CI job, no local runner to maintain.

The three gates

The pipeline ran three gates in one job.

Gate 1: round-trip property. Generate 100,000 valid code points, encode them canonically, validate.

std::mt19937 rng(42);
for (int i = 0; i < 100'000; ++i) {
    uint32_t cp = rng() % 0x110000;
    if (cp >= 0xD800 && cp <= 0xDFFF) continue;
    std::string s = encode_utf8(cp);
    if (!is_valid_utf8(s)) { fail("roundtrip", cp); break; }
}

Result: pass. This is the trap. The encoder only produces canonical forms, so the validator never sees an overlong byte sequence.

Gate 2: sanitizer fuzz. One million random byte strings under ASan and UBSan.

for (int i = 0; i < 1'000'000; ++i) {
    std::string s = random_bytes(rng, rng() % 32);
    (void)is_valid_utf8(s);
}

Result: pass. No crash, no UB. The bounds checks were correct, so memory safety was never the problem. Correctness was.

Gate 3: spec conformance. Ten vectors derived from Table 3-7 (Well-Formed UTF-8 Byte Sequences) in the Unicode Standard.

struct Vec { const char* bytes; size_t len; bool valid; };
const Vec corpus[] = {
    {"\x80",             1, false},  // stray continuation
    {"\xC0\x80",         2, false},  // overlong NUL
    {"\xE0\x80\x80",     3, false},  // overlong U+0000
    {"\xF0\x80\x80\x80", 4, false},  // overlong U+0000
    {"\xED\xA0\x80",     3, false},  // surrogate U+D800
    {"\xF4\x90\x80\x80", 4, false},  // U+110000, out of range
    {"\xF5\x80\x80\x80", 4, false},  // invalid lead byte
    {"\xE2\x82",         2, false},  // truncated sequence
    {"\xE2\x82\xAC",     3, true},   // U+20AC, Euro sign
    {"\xF0\x9F\x98\x80", 4, true},   // U+1F600
};

Result: 4 of 10 passed. The validator accepted all six invalid vectors. It checked the shape of the bytes but not the constraints on the code point they encode.

Results

Gate	Inputs	Result	What it proved
Round-trip	100k canonical encodings	Pass	Valid input survives
Sanitizer fuzz	1M random byte strings	Pass	No memory-safety bugs
Spec conformance	10 Unicode vectors	4/10	The validator is wrong

Root cause

The draft decoded nothing. It validated byte shape, then threw the code point away. Overlong encodings like C0 80 have perfectly valid continuation bytes; the lead byte just claims more bytes than the code point needs. Surrogates and out-of-range values are the same class of bug: the byte pattern is legal, the scalar value is not.

The fix is to decode, then range-check:

bool is_valid_utf8(const std::string& s) {
    size_t i = 0;
    while (i < s.size()) {
        unsigned char c = s[i];
        if (c < 0x80) { ++i; continue; }
        uint32_t cp;
        int extra;
        if ((c & 0xE0) == 0xC0) { cp = c & 0x1F; extra = 1; if (cp < 2) return false; }
        else if ((c & 0xF0) == 0xE0) { cp = c & 0x0F; extra = 2; }
        else if ((c & 0xF8) == 0xF0) { cp = c & 0x07; extra = 3; if (cp > 4) return false; }
        else return false;
        if (i + extra >= s.size()) return false;
        for (int k = 1; k <= extra; ++k) {
            unsigned char cc = s[i + k];
            if ((cc & 0xC0) != 0x80) return false;
            cp = (cp << 6) | (cc & 0x3F);
        }
        if (extra == 2 && cp < 0x800) return false;
        if (extra == 3 && cp < 0x10000) return false;
        if (cp >= 0xD800 && cp <= 0xDFFF) return false;
        if (cp > 0x10FFFF) return false;
        i += extra + 1;
    }
    return true;
}

After the patch, all three gates pass: 100k round-trips, 1M fuzz bytes, 10/10 conformance vectors.

Lessons

1. Round-trip properties are symmetry tests, not correctness tests. They generate valid input and check it survives. They cannot see the invalid input a decoder exists to reject.
2. The oracle must come from outside the model. The model wrote the implementation. A round-trip test written by the same model would share the same blind spot. The Unicode spec corpus is external, small, and free.
3. Negative tests are the product. For a parser, the interesting cases are the ones that must fail. A gate without them measures memory safety, not correctness.
4. A disposable free server changes the loop. The whole pipeline — build, three gates, report — ran as one job on the free server option. No local runner, no VM to tear down. That lowered the cost of gating to zero, which is the only cost that reliably changes developer behavior.

Limitations and who should not use this

This approach does not prove a parser is correct. Ten vectors are not the full Unicode test suite, which runs to thousands of cases. For a production parser, run the full suite, add differential testing against a reference decoder, and fuzz with a coverage target.

Do not use a free model draft for security-critical parsing without an external oracle and a human review of the final diff. The first draft was wrong in six ways; only the spec corpus caught it. And do not use a free server for workloads with strict data-residency or audit requirements — treat it as disposable compute, not infrastructure.

The workflow that worked: draft with a free model, gate with an external oracle, run the gate on a free server, and treat the first green run as the start of review, not the end.