Build a Tiny Grader Before You Trust a Free Model Endpoint

Here is what my grader printed the first time I pointed it at a free model endpoint:

$ python grade.py fixtures/ml101.jsonl
q01  PASS  exact     expected "step size"
q02  PASS  contains
q03  PASS  contains
...
q17  FAIL  numeric   expected "75%" got "0.75"
q18  FAIL  exact     expected "sigmoid" got "Answer: sigmoid"
q19  PASS  contains
...
score: 19/24
written: results/2026-11-09T0142Z.json

Nineteen out of twenty-four. Two of those five failures are my fault, not the model's — and I only know that because the output prints the expected value and the actual value side by side.

So the single question this post answers: can a free model endpoint answer my course-note questions well enough to run unattended overnight, and what does my grader have to catch before I believe the score at all?

Background: the quiz I was already writing

I keep a notes/ folder of Markdown files for my ML course, and every Sunday I write myself a short quiz — two dozen one-line questions pulled from that week's lectures. It takes twenty minutes. It is also the only studying technique that has ever worked for me, because writing the question forces me to notice which sentence in my notes I cannot defend.

Then I got lazy in an interesting way. What if a model answered the quiz overnight, and I read its answers with my coffee? Wrong answers would be suspect, sure. But hedged answers — the ones that waffle between two readings — would point straight at the notes that are ambiguous. That is a signal about my writing, not about the model.

Goal: sixty lines, no framework, dated output

I wanted four things and nothing else. A fixture file I can edit by hand. One function that calls an endpoint. One function that grades strictly enough to be uncomfortable. And a result file stamped with a timestamp and a hash of the fixture, so a run from three weeks ago can never quietly pass itself off as today's.

That last part matters more than it sounds. If you benchmark against a moving target and never record what you asked, you end up quoting a number that describes a Tuesday in October.

Implementation: fixtures first

The fixture is JSONL, one question per line, with the grading mode declared explicitly:

{"id":"q01","q":"In one line: what does a learning rate schedule change?","expect":"step size","mode":"contains"}
{"id":"q17","q":"My classifier reports precision 0.75. As a percentage, what is that?","expect":"75%","mode":"numeric"}
{"id":"q18","q":"Name the activation that squashes to (0,1).","expect":"sigmoid","mode":"exact"}

Three modes is enough for a student quiz. contains for prose, exact for a single word, numeric for anything with a unit attached:

mode	passes when	fails loudly when
exact	normalized string equality	the model adds a prefix
contains	expected substring present	the model paraphrases the term
numeric	parsed floats match within 1e-6	the unit moved

The caller is deliberately boring — standard library only, no SDK, so I can read every line at 1 a.m.:

# ask.py
import json, os, urllib.request

BASE_URL = os.environ["BASE_URL"].rstrip("/")
API_KEY  = os.environ["API_KEY"]
MODEL    = os.environ.get("MODEL", "")  # copy the id from your account page

def ask(prompt, timeout=60):
    payload = json.dumps({
        "model": MODEL,
        "messages": [{"role": "user", "content": prompt}],
        "temperature": 0,
    }).encode()
    req = urllib.request.Request(
        f"{BASE_URL}/chat/completions",
        data=payload,
        headers={"Content-Type": "application/json",
                 "Authorization": f"Bearer {API_KEY}"},
    )
    with urllib.request.urlopen(req, timeout=timeout) as r:
        return json.load(r)["choices"][0]["message"]["content"].strip()

One honest caveat: the OpenAI-style chat/completions shape is what every free tier I have tried speaks, so I assumed it. If yours differs, the only function that changes is this one. Check the provider's own docs before you paste anything — I am not going to guess at someone else's route for you.

This is the point where MonkeyCode entered the project. I did not want to pay per token to grade my own homework, and the operator's free model access meant the experiment could run on a Sunday budget of zero. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The operator also describes a free server option and a free token allowance in the millions — I have not audited the meter, and free-tier terms move, so read the current pricing page yourself rather than trusting this sentence. The harness below does not care whose endpoint you point it at.

The failure that taught me the most

First run: 19/24. Two failures were the model being helpful and my grader being rigid.

q18 is the obvious one. My exact mode compared "Answer: sigmoid" to "sigmoid" and called it wrong. The model was right. My normalizer was asleep:

def norm(s):
    s = s.strip().lower()
    s = re.sub(r"^(answer|a)\s*:\s*", "", s)   # strip chatty prefixes
    s = s.rstrip(".!")
    return re.sub(r"\s+", " ", s)

q17 is the interesting one. I asked for a percentage, the model said 0.75. As a percentage, that is 75%. My numeric mode parsed 0.75, parsed 75, and declared a mismatch — and technically my grader was right about the strings and wrong about the question. Which is the whole game, isn't it? A score is only as trustworthy as the rule that produced it.

So numeric now normalizes units before comparing, and it refuses to grade silently when it cannot parse:

def grade(answer, expect, mode):
    a, e = norm(answer), norm(expect)
    if mode == "exact":
        return a == e, a
    if mode == "contains":
        return e in a, a
    if mode == "numeric":
        pa, pe = parse_num(a), parse_num(e)
        if pa is None or pe is None:
            return None, a           # UNGRADED, not FAIL
        return abs(pa - pe) < 1e-6, pa
    raise ValueError(f"unknown mode {mode!r}")

After the fix: 22/24. The two that remain are a genuine refusal and a genuinely wrong answer about batch norm. That 22 is a number I will defend. The 19 was a number about my regexes.

Results, and where the run lives

I moved grade.py onto the free server option and let cron do the boring part:

0 2 * * *  cd /srv/quiz && \
  BASE_URL="$BASE_URL" API_KEY="$API_KEY" MODEL="$MODEL" \
  python grade.py fixtures/ml101.jsonl >> results/cron.log 2>&1

run	fixture hash	score	graded	ungraded
2026-11-03	4f1c9a	19/24	24	0
2026-11-09	4f1c9a	22/24	23	1

Both rows share a fixture hash, so the delta is grader changes, not question changes. The one ungraded row is the honest cost of refusing to guess: an empty cell beats a fake fail.

That free server is doing exactly one job, and it is the right size for it — a two-minute job at 2 a.m. that must not depend on whether my laptop is awake. If you need a GPU box, an SLA, or a persistent volume, this is not that, and you should not pretend otherwise.

What I actually learned

• The grader is the artifact. The model is a fixture. If you cannot state your pass condition in one sentence, you are not measuring anything.
• Three states, not two. PASS, FAIL, UNGRADED. Collapsing UNGRADED into FAIL is how benchmarks quietly lie.
• Stamp the fixture. Hash it, date the run, keep the old files. Otherwise next month you will compare two scores that answered different questions.
• Free tiers are for rehearsal, not for promises. Rate limits, model swaps, and term changes are all normal. Cap the run at N questions, log the model id you actually got, and stop on the first 429 instead of hammering.

Who should skip this: anyone grading real students, anyone whose notes contain anything private, and anyone who needs a result a reviewer could reproduce a year from now. For those, host the model yourself and pin the weights.

So, before you trust a free endpoint: which fixture would you write first? My bet is the one that fails silently — the 0.75 that was right all along, and the grader that never told you it was guessing. If you build the harness, point it at whatever free model access you have and post your first failure; the endpoint is the least interesting variable in the experiment.