Case Study: A Free Model's 47th Test Input Crashed Our Parser

A free model generated 100 test inputs for a C++ binary parser. It wrote zero production code. Input #47 triggered an out-of-bounds read that a week of hand-written tests had missed. This post is the full workflow — the format spec, the prompt, the gate script, and the crash — so you can run the same experiment on your own parser.

Background: the parser had tests, but not the right tests

The component was a small C++ parser for a custom binary format: a 4-byte magic, then a sequence of chunks. Each chunk had a 4-byte type, a 4-byte little-endian length, a payload, and a 4-byte CRC. The parser was hand-written, had 80% line coverage, and passed every unit test.

The tests were written by the same person who wrote the parser. They tested what the parser expected, not what the format allowed. The gap was not in the happy path — it was in the arithmetic.

The format spec, condensed:

File: MAGIC (0x4D4F434B) + Chunk*
Chunk: TYPE (4B) + LENGTH (4B LE) + PAYLOAD (LENGTH B) + CRC (4B)
CRC: CRC32 of TYPE + LENGTH + PAYLOAD
Rules:
- LENGTH must be <= 65535
- FILE_END chunk must be last, with LENGTH == 0
- TYPE must be one of: DATA, META, FILE_END
- Unknown types must be skipped, not rejected

Goal: three gates, defined before the first prompt

The goal was not "make the model write a parser." The parser already existed. The goal was to test whether a free model could generate inputs that a random fuzzer would miss.

Three gates were fixed up front:

1. Format gate — every input must be valid hex, start with the magic, and contain at least one readable chunk header. This filters out garbage before it reaches the parser. It deliberately does not check LENGTH ranges or CRC values — those are the parser's job.
2. Sanitizer gate — every valid input runs through the parser under AddressSanitizer + UBSan. Any crash is a failure.
3. Coverage gate — the model-generated inputs must cover at least as many lines as an equal number of randomly mutated inputs.

Implementation: one prompt, 100 inputs

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The prompt was deliberately specific. It described the format and asked for boundary cases, including rule violations:

You are generating test inputs for a C++ parser. The format is:

File: MAGIC (0x4D4F434B) + Chunk*
Chunk: TYPE (4B) + LENGTH (4B LE) + PAYLOAD (LENGTH B) + CRC (4B)
CRC: CRC32 of TYPE + LENGTH + PAYLOAD

Rules:
- LENGTH <= 65535
- FILE_END must be last, LENGTH == 0
- TYPE in {DATA, META, FILE_END}
- Unknown types are skipped, not rejected

Generate 100 inputs that stress boundary conditions and error handling.
Some inputs should deliberately violate the rules to test the parser:

- LENGTH values: 0, 1, 65535, 0xFFFFFFFF, 0x80000000
- Chunk counts: 0, 1, 2, 65535
- CRC values: correct, wrong, all-zeros, all-0xFF
- TYPE values: all valid, all unknown, mixed
- PAYLOAD: empty, 1 byte, 65535 bytes, binary garbage
- Truncated chunks, extra bytes after FILE_END, missing magic

Output each input as a hex string, one per line.

The model returned 100 hex strings. The format gate accepted 73. The other 27 had malformed headers, truncated payloads, or were not valid hex — exactly what the gate was for.

The driver script was short:

#!/usr/bin/env bash
# gate.sh — validate, run under ASan, report
set -euo pipefail

INPUT_FILE="${1:-inputs.txt}"
PARSER_BIN="${2:-./parser}"

echo "[1/3] Format gate"
python3 validate_format.py "$INPUT_FILE" > valid_inputs.txt
wc -l < valid_inputs.txt

echo "[2/3] ASan + UBSan run"
i=0
while IFS= read -r hex; do
  i=$((i + 1))
  if ! timeout 5 "$PARSER_BIN" "$hex" > /dev/null 2> "asan_$i.log"; then
    echo "CRASH on input #$i"
    cat "asan_$i.log"
    exit 1
  fi
done < valid_inputs.txt

echo "[3/3] Coverage"
# (coverage instrumentation omitted for brevity)

The parser under test was a simplified version of a real one. The interesting part was the chunk-length handling:

// parser.cpp — simplified, with the bug
bool parse_file(const uint8_t* data, size_t size) {
  if (size < 8 || memcmp(data, "MOCK", 4) != 0) return false;
  size_t offset = 4;
  while (offset + 8 <= size) {
    uint32_t type = read_u32(data + offset);
    uint32_t length = read_u32_le(data + offset + 4);
    offset += 8;
    if (offset + length > size) return false;  // BUG: overflow
    // ... process chunk ...
    offset += length + 4;  // skip payload + CRC
  }
  return true;
}

The bug: when length is 0xFFFFFFFF, offset + length overflows size_t on a 32-bit build. The sum wraps to a small number, the bounds check passes, and the parser reads past the buffer.

Results

Gate	Model inputs (100)	Random mutation (100)
Format-valid	73	100
ASan crashes	1 (input #47)	0
Line coverage	68%	41%
Time to first crash	47 valid inputs	—

Input #47 was a file with two chunks: a valid DATA chunk, then a chunk with LENGTH = 0xFFFFFFFF and a correct CRC for that length field. The random fuzzer never generated a length that large because it mutated bytes in small increments — it would take millions of iterations to hit 0xFFFFFFFF by chance.

The model generated it because the prompt asked for "values that overflow when added." It reasoned about the arithmetic context instead of flipping bits.

The fix

if (length > size - offset) return false;  // no overflow possible

After the fix, all 73 valid inputs passed under ASan. The model-generated input set became a permanent regression fixture — 73 lines in a text file, checked into the repo.

Limitations

This workflow does not replace a real fuzzer. Random mutation found different paths; the model found a specific arithmetic bug. A coverage-guided fuzzer with a good seed corpus would likely find both. The model's advantage was speed — 100 targeted inputs in one prompt, no corpus engineering.

Who should not use this: if you already have a mature fuzzing pipeline with a seed corpus and coverage-guided mutation, the model adds little. Use this when you have a parser with hand-written tests and no fuzzing infrastructure.

Lessons learned

1. A free model is more useful as a test-input generator than as a code generator when the code already exists. The inputs never ship to production.
2. The format gate is essential. 27% of model outputs were malformed — without the gate, they would have produced false crash reports.
3. Ask for boundary conditions explicitly. "Values that overflow when added" produced the crash; "random values" would not have.
4. The crash was in arithmetic, not in parsing. The model found it because the prompt asked about arithmetic.
5. A permanent regression fixture costs nothing and prevents the bug from returning.

The harness and the 100 inputs are the artifact here. If you have a parser with hand-written tests and no fuzzer, try this: write a format spec, ask a free model for 100 boundary inputs, and run them under ASan. The model will not write your parser. It might find the bug your tests missed.

A free server option is enough to reproduce the setup.