The Free Model Never Went Down. Its Answers Just Drifted. Here's My 48-Hour Gradebook.

#ai #llm #python #testing

Last week I load-tested a free AI server until it coughed, and the uptime numbers looked beautiful. The server stayed up, the queue drained, and I almost shipped a pipeline that trusted a healthy-looking chart. Then I asked a different question: what if the server never crashes, but the answers quietly get worse? That question cost me 48 hours, and the gradebook is worth sharing.

Why a load test is not a quality test

Load tests measure capacity, not judgment, and the difference only shows up when the output matters. A server can return 200 OK with a confident wrong answer, and your dashboard will never notice. I needed a canary that graded the actual output, not just the response time, and I needed it to run unattended for two days.

Last time I let a model grade its own homework, and the experiment taught me a lot about rubrics. This time I went the opposite direction: every probe had a deterministic answer, so no LLM judge was involved and no rubric could drift. The whole harness was one Python file and a cron line, which felt right for a free-tier experiment.

The harness

I pointed the harness at MonkeyCode's free model access and ran it on their free server option, which meant the experiment cost nothing but time. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The script below is the entire kit, with the endpoint URL left as a placeholder because a canary should not care who the provider is.

# canary.py — hourly quality probe for a free model endpoint
import json
import time
import urllib.request
from datetime import datetime, timezone

PROBES = [
    ("sum", "Add 17 and 25. Reply with only the number.", "42"),
    ("json", 'Return JSON: {"city": "Paris", "temp_c": 22}.', {"city": "Paris", "temp_c": 22}),
    ("logic", "If all A are B and no B are C, can an A be a C?", "no"),
]

def grade(name, text):
    if name == "sum":
        return 1.0 if text.strip() == "42" else 0.0
    if name == "json":
        try:
            return 1.0 if json.loads(text) == PROBES[1][2] else 0.5
        except Exception:
            return 0.0
    if name == "logic":
        return 1.0 if text.lower().startswith("no") else 0.0

def probe(url, api_key):
    row = []
    for name, prompt, _ in PROBES:
        payload = json.dumps({"messages": [{"role": "user", "content": prompt}]}).encode()
        req = urllib.request.Request(url, data=payload, headers={
            "Authorization": f"Bearer {api_key}",
            "Content-Type": "application/json",
        })
        try:
            with urllib.request.urlopen(req, timeout=60) as resp:
                body = json.loads(resp.read())
                text = body["choices"][0]["message"]["content"]
                row.append(grade(name, text))
        except Exception:
            row.append(0.0)
        time.sleep(2)
    return row

if __name__ == "__main__":
    scores = probe("https://api.example.com/v1/chat/completions", "YOUR_KEY")
    print(json.dumps({"ts": datetime.now(timezone.utc).isoformat(), "scores": scores}))

The cron line is the boring part, and boring is good: 0 * * * * cd /home/taylor/canary && python3 canary.py >> gradebook.jsonl. Every hour I got one JSON line with a timestamp and three scores, and after 48 hours I had a small file that told a surprisingly loud story.

What broke

Hour one was boring, and hour twelve was boring, which is exactly what you want from a canary. The first real signal appeared on the second night around 2 AM, when the sum probe returned prose instead of "42". The server never errored; the answer was just wrong, wrapped in a confident sentence that a human would have called helpful.

I watched three failure modes repeat across the two nights:

Confident prose instead of an answer. The model explained the addition step by step and never emitted the number. My rubric scored it zero, and the log line made the drift obvious.
JSON that parsed but drifted. The temp_c field arrived as a string, so the payload was valid JSON and useless data at the same time.
Slow degradation before a timeout. Latency climbed for about twenty minutes, then the request died. The uptime chart stayed green the entire time.

None of these would have shown up in a load test, and that is the whole point.

The decision table

I turned the observations into a small table that now lives next to my cron line:

Signal	What it means	What I did
All probes score 1.0 for six hours	Stable	Left the canary running
One probe fails once	Noise	Waited for the next hour
Same probe fails twice in a row	Drift pattern	Added a fallback model
Latency climbs, then a timeout	Server pressure	Backed off and retried

The table is deliberately conservative, because a free tier deserves suspicion, not faith.

What I would repeat

The deterministic rubric. No LLM judge, no subjectivity, no arguments about whether a wrong answer was creative.
The one-file harness. I could read the whole thing in a minute, and so could anyone who inherits it.
The 48-hour window. One night caught a pattern that a four-hour test would have missed entirely.

What I would change

I would add a fourth probe for long-form output, because short answers hide verbosity drift.
I would store results in SQLite instead of a JSONL file, because grepping log lines gets old fast.

Limitations and who should skip this

This is a smoke test, not a certification. My run covered one endpoint, three prompts, and two nights, so treat the patterns as anecdotes rather than laws. If your workload needs guaranteed latency or regulated accuracy, a free tier is the wrong tool, and no canary changes that.

The real value of a free tier is that you can afford to be suspicious of it. I ran this experiment for two days, spent nothing, and learned exactly where my pipeline needed a fallback. The gradebook is the artifact, and the next 48 hours are yours.

DEV Community