The CSV Parser That Passed My Tests and Still Broke on a Free Server

When a parser compiles cleanly and passes every example I can think of, I still assume it contains a bug. Last week I asked MonkeyCode's free model to write a CSV parser. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The code was small, sensible, and compiled the first time. My five hand-written test cases produced correct output. Then I ran a differential test on a free server and found a broken escape path. This is the story of that edge case, and the verification loop I now use for every generated parser.

The generated parser looked straightforward. It read a line character by character, tracked whether it was inside quotes, and split on commas. The output for regular rows was perfect. Quoted fields worked as long as the quotes were balanced. Then I generated a line that contained an escaped quote inside a quoted field. Nobody writes test cases that ugly by hand. A random generator does.

The Differential Harness

I created a simple differential harness instead of a classic unit test. Instead of asserting on expected strings, it compared the generated parser's output against a second implementation. For CSV, you can use Python's csv module as a reference. That module is battle-tested; comparing against it exposes weird behavior quickly.

The harness generated random CSV fragments: fields with quotes, embedded commas, newlines inside quotes, escaped quotes. Each fragment was written to a temporary file. Then the C++ parser and a small Python wrapper both read it. The Python wrapper used csv.reader, the C++ parser produced its own vector of strings. When the two disagreed, the harness printed the input and both outputs.

import csv
import subprocess
import random
import tempfile

def random_row():
    tokens = []
    for _ in range(random.randint(1, 5)):
        t = random.choice(["plain", '"quoted, field"', '"with ""escape"""', "a,b"])
        tokens.append(t)
    return ",".join(tokens)

for i in range(2000):
    row = random_row()
    with open("input.csv", "w") as f:
        f.write(row)
    expected = list(csv.reader([row]))[0]
    out = subprocess.run(["./tmp/csv_parser", "input.csv"], capture_output=True, text=True)
    actual = out.stdout.split("|")[:-1]
    if expected != actual:
        print(f"Mismatch #{i}: {row!r}")
        print(f"expected: {expected}")
        print(f"actual:   {actual}")
        break

Bug One: The Escaped Quote

The first run failed after a few hundred iterations. The input looked like this:

"a""b",c

In standard CSV, a doubled quote inside a quoted field denotes a literal quote. The expected field is a"b. My AI-generated parser produced two fields: a""b (without the surrounding quotes) and c. It treated the second quote as closing the field. That was the bug.

The free model had written a simple state machine: outside string, inside string. Inside a string, a quote either closed the field or appeared before a comma. It never handled the case where the quote was a literal escaped quote. The state machine needed a third state: "after a quote inside a string." If the next character is another quote, emit a quote and stay inside the string. Otherwise, close the field.

The relevant part of the generated code was roughly this:

if (ch == '"') {
    if (in_quotes) {
        in_quotes = false;  // wrong for "" escape
    } else {
        in_quotes = true;
    }
}

That logic is fine for simple quotes, but it cannot represent the doubling rule. The corrected logic introduces a one-character lookbehind:

if (ch == '"') {
    if (in_quotes && last_char == '"') {
        field.push_back('"');  // escaped quote
        last_char = '?';
        continue;
    }
    if (in_quotes) {
        last_char = '"';  // remember this quote
    } else {
        in_quotes = true;
    }
    continue;
}
if (last_char == '"') {
    in_quotes = false;
}
last_char = ch;

The exact fix depends on how you structure the state machine. The point is not the code, it is the test that found it. A set of designer-selected examples is too clean. Random inputs with a reference implementation are not.

Bug Two: Multiline Fields

The second failure appeared after fixing the first. A row with a newline inside a quoted field misaligned the line-based reader. The parser split the whole input by \n before processing quotes. That destroyed any multiline field. The reference parser treated the entire input as one token stream. This design difference is a known CSV ambiguity. My decision was to disallow newlines inside quoted fields. The harness then used that rule explicitly.

Input kind	Expected (Python)	Generated parser before fix	After fix
a,b,c	3 fields	3 fields	3 fields
"a,b",c	2 fields	2 fields	2 fields
"a""b",c	1 field a"b	2 fields	1 field
"a\nb",c	2 fields (newline inside quote)	3 fields	rejected or 2 per rule

I included this table in the repository. It forces future modifications to answer the newline question explicitly.

Why the Free Server Mattered

The differential loop is embarrassingly parallel. You can generate a million inputs, run a thousand workers, and aggregate the mismatches. My laptop heated up and throttled. The free server option in MonkeyCode's free tier gave me a clean, isolated instance with predictable scheduling. The whole run cost nothing in tokens beyond the initial parsing request. The free models themselves generated the parser, the Python comparison script, and the bug fix suggestion. I reviewed each piece.

The free models and the free server complemented each other. The model produced code fast, the server ran it somewhere else, and I verified it against a reference implementation. Without the server, the test would have taken longer but would still work locally. Without the model, I would have written the parser by hand and trusted it less.

Limitations and Who Should Not Use This

Is this workflow for everyone? If you are writing parsers for untrusted input, no amount of differential testing is enough; you need fuzzing, memory sanitizers, and maybe a formal grammar. If you only parse clean, known data, a hand-written parser with a few examples will do. The differential approach sits between them. It catches logic errors but not memory corruption. Use the right tool for each stage.

One more note: generated code should carry a review trail. When I commit the AI-generated parser, I include the differential harness and the decision table in the same pull request. Reviewers see exactly why certain choices were made. That is the difference between code generation and code engineering.

The cheap hardware and free model access made this loop painless. You can build the same loop tomorrow. Start with the smallest reference implementation you can find, write a generator that produces malformed inputs on purpose, and run it on a server you do not care about. The first mismatch will teach you more than a hundred clean test suites.