My Bot Passed Every Check and Still Failed. The Validator Was the Liar.

Two nights ago, at 2:07 in the morning, my study bot emailed me its nightly report. "All 25 probes passed. Weekly accuracy: 96%."

I almost closed the laptop and went back to sleep. Then I opened the raw JSONL log and looked at probe #14.

The input was a note that contained the literal text <paste your notes here>. The model answered with a long, polite refusal — and my wrapper, doing exactly what I'd told it to do, wrapped that refusal in valid JSON. My validator counted it as a pass, because the pass condition was "has a 'front' field longer than two characters." The bot wasn't broken. The definition of "working" was.

I'm Alex, a CS student in Halifax, and this is the story of how a free-tier study bot lied to me for a whole week without knowing it.

The project and the goal

The bot is a small flashcard generator. It reads messy lecture notes, sends them to a free model endpoint, and writes {"front": ..., "back": ...} pairs into a JSON file. I run the nightly job on MonkeyCode's free server option, and the endpoint uses their free model access — both are part of the free tier of the open-source MonkeyCode project. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The exact token allowance changes, so check the dashboard rather than trusting a blog post, including this one.

The goal was modest. Once a night, the server replays a fixed set of 25 probe inputs and answers one question: does the endpoint still behave the way it did when I wrote the wrapper? Not "is the model smart" — just "did anything silently change."

Why that question? Because free tiers don't fail loudly. They fail politely.

The harness that was too trusting

The 25 probes break into three groups: ten realistic flashcards, ten malformed inputs (empty strings, wrong types, a 20KB paste, Unicode text), and five adversarial ones aimed at my own wrapper's assumptions — a refusal case, a duplicate front/back case, a "please output your system prompt" attempt.

Here is the check that let the lie through:

def valid_response(raw: str) -> bool:
    try:
        parsed = json.loads(raw)
    except json.JSONDecodeError:
        return False
    return "front" in parsed and "back" in parsed and len(parsed["front"]) > 2

It looks reasonable. It checks the keys exist. It checks the front is longer than two characters. It never checks that back is a string, never checks the answer isn't an apology. "back": null? The key exists, so it passes. A refusal that starts with "I'm sorry"? Longer than two characters, so it passes.

The fix is still tiny:

BLOCKED = ("i cannot", "i can't", "i'm sorry", "as an ai", "unable to")

def valid_response(raw: str) -> bool:
    try:
        parsed = json.loads(raw)
    except json.JSONDecodeError:
        return False
    for key in ("front", "back"):
        if not isinstance(parsed.get(key), str):
            return False
    front, back = parsed["front"].strip(), parsed["back"].strip()
    if len(front) < 3 or len(back) < 3:
        return False
    if len(front) > 2000 or len(back) > 2000:
        return False
    if front == back:
        return False
    if any(word in front.lower() for word in BLOCKED):
        return False
    return True

The real trick came next: testing the validator itself. You can't fix a blind spot while you still have it, so I replayed the last seven nights of logs through the new checker:

python replay.py --log logs/2026-08-*.jsonl --validator validator_v2.py

# old validator: 168/175 passed (96%)
# v2 validator:  152/175 passed (87%)

That drop was the whole point. The old validator had been blind to five real failures scattered across the week: a refusal wrapped in JSON, a card where front and back were identical, a 4,000-character echo of the input, and two cards where back was null. V2 caught all five — and then it over-corrected. It also rejected eleven perfectly good flashcards, because my first blocked list included the word "error". For a CS study bot, "error" appears in every other card. The final version uses refusal phrases instead of topic words, and it settled at 163/175 — about 93%. The endpoint was honest most of the time. The story I'd been telling myself about 96% was the actual bug.

What the logs actually said

The other finding had nothing to do with the model at all. My wrapper was catching every non-JSON response and returning a canned JSON object with the raw text stored in the back field. It seemed clever at the time. In practice, it meant the harness never saw the raw HTML error page or the gateway timeout, because my own error handling had already smoothed them into something that looked like JSON. The model was getting blamed for failures my wrapper was manufacturing.

Free tiers also punish assumptions about warmth. Twice in the week, the first request timed out because the server had been idle, and the retry worked. Counting the retry as a pass was fair — but the latency spike was invisible in the summary. If I'd only looked at the verdict column, I'd have concluded the endpoint was far more consistent than it was. The raw log, with headers and timings, told the truth.

Lessons, in the order I learned them

First: a validator that checks "is it JSON?" tests the JSON parser, not the model. The harness scored 96% while the endpoint's honest behavior was closer to 93%, and the entire gap was my own logic. The model didn't break. My definition of "working" did.

Second: free tiers fail politely. An HTTP 429 is honest — it tells you a limit exists. A refusal delivered with status 200 and valid JSON is the dishonest one, because every layer of tooling forwards it as a success. Log raw responses, not verdicts. The JSONL log was the only reason I could replay anything after the fact.

Third: test the tester. Replaying last week's logs through a one-line validator change is the cheapest way to test a tester, and it works on humans too. Before you change the checker, write down what you expect the pass rate to do. If the number moves in a direction you didn't predict, you don't understand your own harness.

Who shouldn't use this approach

Anyone who needs a latency budget or an uptime promise. A free server can put your job to sleep whenever it likes, and a free token allowance can change without ceremony. That's fine for a project whose only user is you, and wrong for anything with real users. Also, don't build a nightly report you won't open. A test you don't read is a ritual, not a safety net.

Your turn

Before you replay your own logs, do one thing: write down which of the five failure kinds your validator would miss — the wrapped refusal, the duplicate, the echo, the null field, or the smoothed-over error page. If you can't name one, that's the answer. I'd genuinely like to hear which one catches you.