Free Model Endpoints Make Good Fuzz Generators and Bad Oracles

Free Model Endpoints Make Good Fuzz Generators and Bad Oracles

The failure starts with a tempting shortcut: ask a low-cost model to "write tests for my CLI parser." The output looks confident. It returns a list of flags, expected return codes, and assertions. Some of those assertions are wrong. Worse, the model often encodes the same misunderstanding as the code it is supposed to test, so the test suite promises coverage while silently passing broken behavior.

The fix is not to buy a more expensive model. It is to take away the model's authority over correctness. Let the model generate candidate inputs. Let a deterministic reference implementation in your own code decide whether the behavior is right.

Build the generator/validator split

Use the model as a fuzzer, not as an oracle. The model emits argument arrays for a small C++ program. The program is your real system under test. A Python reference parser computes the expected outcome. Anything that disagrees is a finding.

Here is the target program.

// flagdemo.cpp
#include <iostream>
#include <string>
#include <vector>

int main(int argc, char** argv) {
    std::vector<std::string> args(argv + 1, argv + argc);
    std::string input;
    int count = 1;
    bool verbose = false;

    for (size_t i = 0; i < args.size(); ++i) {
        const std::string& a = args[i];
        if (a == "--verbose") {
            verbose = true;
        } else if (a == "--input") {
            if (i + 1 >= args.size()) return 2;
            input = args[++i];
        } else if (a == "--count") {
            if (i + 1 >= args.size()) return 2;
            try {
                count = std::stoi(args[++i]);
            } catch (...) {
                return 3;
            }
        } else {
            std::cerr << "unknown flag: " << a << "\n";
            return 4;
        }
    }

    std::cout << input << " " << count << " " << (verbose ? "1" : "0") << "\n";
    return 0;
}

The reference oracle in Python deliberately does not ask the model anything. It implements the contract: missing values return 2, non-integer --count values return 3, unknown flags return 4.

# reference.py
def reference(args):
    input_ = ""
    count = 1
    verbose = False
    i = 0
    while i < len(args):
        a = args[i]
        if a == "--verbose":
            verbose = True
            i += 1
        elif a == "--input":
            if i + 1 >= len(args):
                return ("missing_value", 2)
            input_ = args[i + 1]
            i += 2
        elif a == "--count":
            if i + 1 >= len(args):
                return ("missing_value", 2)
            try:
                count = int(args[i + 1])
            except ValueError:
                return ("bad_count", 3)
            i += 2
        else:
            return ("unknown_flag", 4)
    return ("ok", 0, input_, count, verbose)

The fuzz loop fetches candidate arrays, runs the compiled binary with no shell, and compares the result to the reference.

# fuzz.py
import json
import os
import subprocess
import urllib.request

# Provider-specific adapter: keep it thin and isolated.
def fetch_candidates(endpoint, prompt, auth_token):
    req = urllib.request.Request(
        endpoint,
        data=json.dumps({"prompt": prompt}).encode("utf-8"),
        headers={
            "Content-Type": "application/json",
            "Authorization": f"Bearer {auth_token}",
        },
    )
    with urllib.request.urlopen(req, timeout=30) as resp:
        payload = json.loads(resp.read().decode("utf-8"))
    return payload["cases"]

PROMPT = """
Generate 25 command-line argument arrays for a C++ program that accepts
--input, --verbose, and --count. Return only a JSON object with a "cases"
array of string arrays. Vary whitespace, Unicode, negative numbers, missing
values, repeated flags, huge values, and leading dashes. Do not include NUL
bytes or shell metacharacters.
"""

def run_binary(args):
    completed = subprocess.run(
        ["./flagdemo", *args],
        capture_output=True,
        timeout=2,
    )
    return completed.returncode, completed.stdout.decode(errors="replace")

def compare(case):
    expected = reference(case)
    actual_code, actual_stdout = run_binary(case)
    if expected[0] == "ok":
        expected_stdout = f"{expected[2]} {expected[3]} {int(expected[4])}\n"
        return actual_code == 0 and actual_stdout == expected_stdout, expected, actual_code, actual_stdout
    return actual_code == expected[1], expected, actual_code, actual_stdout

def main():
    endpoint = os.environ["MODEL_ENDPOINT"]
    token = os.environ["MODEL_TOKEN"]
    for case in fetch_candidates(endpoint, PROMPT, token):
        passed, expected, code, stdout = compare(case)
        if not passed:
            print("FAIL", json.dumps(case), expected, code, repr(stdout))

if __name__ == "__main__":
    main()

The model's only job is to produce diverse, weird inputs. It can hallucinate a "correct looking" flag that does not exist, and that is fine: the reference implementation classifies it as unknown, and the comparison catches any parser that disagrees. The model cannot teach the test to pass.

A free model endpoint is a reasonable generator for this pattern because generation quality can be low and still useful. If you do not have budget for a paid testing endpoint, MonkeyCode's free model access and free server option can be that tier. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The important constraint is that you never ask the free endpoint to be the judge.

Keep the generated corpus

Free endpoints are rate-limited and non-deterministic. A prompt may return 25 cases today and 8 tomorrow, or skip the one case that would have caught a bug. Save every accepted case to a JSON file and make that the replayable corpus:

def record(case):
    corpus = json.load(open("corpus.json"))
    corpus.append(case)
    json.dump(corpus, open("corpus.json", "w"), indent=2)

Then CI runs from corpus.json without calling the model at all. The model becomes a seed source, not a build-time dependency.

Safety and limits

Treat model output as hostile, even from a friendly endpoint. Pass arguments as a list to subprocess.run with shell=False (the default), reject NUL bytes, cap token length, set a timeout, and never place generated tokens in a position where the parser might evaluate them as code. If the target program writes files or speaks to a network, sandbox it.

This approach only works when you can write a reference oracle. It will not prove correctness, and it will not replace a curated test suite. It is a bug-finding aid for parser-like code with a clear contract: CLIs, config loaders, schema validators, formula evaluators. If the correct output is subjective or defined by a human reviewer, the model should not silently supply that judgement.

Do not use this for regulated code, code where a wrong oracle can cause real harm, or codebases where you cannot review and reduce the generated corpus before it enters CI. A free endpoint is also the wrong place to send confidential inputs, tokens, or customer data.

The small workflow is the point: generated input, deterministic oracle, recorded corpus, safe execution. Give the free model less authority, not more.