DEV Community: Alex Chen

Build a Resume-Safe Batch Summarizer: A Case Study on a Free Server

Alex Chen — Fri, 16 Oct 2026 04:43:25 +0000

It is 1 a.m., the fan is loud, and the lid is closed. Somewhere in that folder of lecture notes, a job is either still moving or already dead, and you have no way to tell which until morning.

That gap is the whole reason this article exists. I wanted one small project — summarize every file in a folder — and I wanted a crash to cost me exactly one file. Not the whole night, not the whole budget. So the learning question here is narrow: when a batch job dies in the middle, what is the smallest piece of state that lets you restart without paying twice? If you can answer that in your head before reading on, you already own this idea.

The project, end to end

I called it nightshift. One folder of note files goes in, one append-only ledger.jsonl comes out, and every line in that ledger is a claim about a single file: "I was processed, here is the result, here is the model and prompt version that produced it." That's it. No database, no queue service, no framework.

The reason I reached for MonkeyCode here is boring and practical: the free model access gives me an endpoint I don't have to pay for to run a batch of a few hundred small calls, and the free server option means the job can keep running after my laptop sleeps. Both are availability claims from the operator rather than things I've audited myself. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The allowance I was told to expect is on the order of ten million free tokens, and there is a free server option alongside it. Treat those numbers as a starting point to confirm on the project page before you plan a semester around them — I have not measured the quota, and you shouldn't assume it is permanent either.

Prerequisites, exactly

Python 3.11 or newer (I ran this on 3.11.9), standard library only, and a folder of text files. That's the whole list. I deliberately keep the fixture .txt files so nothing in this article depends on network access — if you have PDFs, extract them to text first with pdftotext and let the runner stay ignorant of file formats.

Here is the naive version I started from, and I want you to see it fail before I fix it:

for p in Path("corpus").rglob("*.txt"):
    summary = call_model(p.read_text())
    open("out.jsonl", "a").write(json.dumps({"file": str(p), "summary": summary}) + "\n")

Three things go wrong. Kill it at file 211 and the next run re-sends files 1 through 210 — you paid twice for the same text. Key your output on the path and a renamed copy becomes a brand new call. And a plain write without a flush can leave a half-written line that poisons every later read. None of these are exotic; they are just invisible until the second run.

The artifact

Save this as nightshift.py. It is the entire runner.

#!/usr/bin/env python3
"""nightshift.py - resumable batch summarizer. Python 3.11+, stdlib only."""
import hashlib
import json
import os
import sys
from pathlib import Path

PROMPT_VERSION = "sum-v1"
LEDGER = Path("ledger.jsonl")
SUFFIXES = {".txt", ".pdf"}


def fingerprint(path: Path, model: str) -> str:
    h = hashlib.sha256()
    h.update(path.read_bytes())
    h.update(model.encode("utf-8"))
    h.update(PROMPT_VERSION.encode("utf-8"))
    return h.hexdigest()[:16]


def load_done(ledger: Path) -> dict:
    done = {}
    if not ledger.exists():
        return done
    for lineno, line in enumerate(ledger.read_text(encoding="utf-8").splitlines(), 1):
        if not line.strip():
            continue
        try:
            rec = json.loads(line)
        except json.JSONDecodeError:
            print(f"warning: torn line {lineno} ignored", file=sys.stderr)
            continue
        done[rec["key"]] = rec
    return done


def append(ledger: Path, rec: dict) -> None:
    with ledger.open("a", encoding="utf-8") as fh:
        fh.write(json.dumps(rec, sort_keys=True) + "\n")
        fh.flush()
        os.fsync(fh.fileno())


def stub_model(text: str, model: str) -> str:
    if not text.strip():
        raise ValueError("empty input: no text to summarize")
    words = text.split()
    return f"[{model}] {len(words)} words; first: {' '.join(words[:3])}"


def summarize(path: Path, model: str) -> str:
    text = path.read_text(encoding="utf-8", errors="replace")
    if os.environ.get("NIGHTSHIFT_STUB") == "1":
        return stub_model(text, model)
    raise SystemExit("no client wired; see the endpoint section")


def main(argv: list[str]) -> int:
    root = Path(argv[1] if len(argv) > 1 else "corpus")
    model = os.environ.get("NIGHTSHIFT_MODEL", "stub-model")
    files = sorted(p for p in root.rglob("*") if p.is_file() and p.suffix in SUFFIXES)
    done = load_done(LEDGER)

    todo, seen, skipped = [], set(), 0
    for p in files:
        key = fingerprint(p, model)
        if key in done or key in seen:
            skipped += 1
            continue
        seen.add(key)
        todo.append((key, p))

    print(f"corpus={len(files)} todo={len(todo)} skipped={skipped}")
    for key, p in todo:
        try:
            summary = summarize(p, model)
        except Exception as exc:
            print(f"FAIL {p.name} {type(exc).__name__}: {exc}")
            append(LEDGER, {"key": key, "file": p.name, "status": "error",
                            "error": type(exc).__name__})
            continue
        append(LEDGER, {"key": key, "file": p.name, "status": "ok",
                        "model": model, "prompt": PROMPT_VERSION,
                        "summary": summary})
        print(f"OK   {p.name}")

    poison = sum(1 for r in load_done(LEDGER).values() if r["status"] == "error")
    print(f"poison={poison}")
    return 0


if __name__ == "__main__":
    raise SystemExit(main(sys.argv))

The fingerprint function is the load-bearing part. It hashes the file bytes, the model name, and the prompt version together, so the same text under a different model is a different job — which is what you want when you swap endpoints, and mildly annoying the first time it silently re-runs everything. That annoyance is the point. A cache that ignores the model is a cache that lies.

The failing input, and the run you can reproduce

Build a fixture with one deliberately broken file. The empty one is my poison file.

mkdir -p corpus
printf 'attention is a lookup table\n' > corpus/a.txt
printf 'a softmax turns scores into weights\n' > corpus/b.txt
: > corpus/c.txt
cp corpus/a.txt corpus/dup_of_a.txt

Now run it offline. The stub keeps the whole exercise network-free.

NIGHTSHIFT_STUB=1 NIGHTSHIFT_MODEL=stub-model python3 nightshift.py corpus

Expected output on the first run:

corpus=4 todo=3 skipped=1
OK   a.txt
OK   b.txt
FAIL c.txt ValueError: empty input: no text to summarize
poison=1

Read the counters slowly, because each one is a decision. skipped=1 is dup_of_a.txt: identical bytes plus identical model plus identical prompt version produced a key already seen in this pass, so it never left the machine. That deduplication is free, and I did not write a single line of code for it. FAIL c.txt is the empty file — most model endpoints reject a blank prompt, so I raise the same way the stub does, and the failure path deliberately writes an error record instead of a summary.

And poison=1 is the trap I want you to notice. The error record uses the same key, so load_done counts it as handled and the next run will not retry it. That is the right default for a batch you are paying for — one broken file shouldn't burn retries forever — but it means a poisoned key sits there quietly until you delete its line by hand. Delete the line, and only that file re-runs.

Run it a second time:

corpus=4 todo=0 skipped=4
poison=1

Zero todo. Nothing resent, nothing re-billed, and the crash boundary is now the only file that was in flight when the process died. That is the property I wanted at 1 a.m.

The ledger itself is the last thing worth inspecting:

{"file": "a.txt", "key": "…", "model": "stub-model", "prompt": "sum-v1", "status": "ok", "summary": "[stub-model] 5 words; first: attention is a"}

The key is a 16-hex-character digest in the real file; I elided it because yours will differ from mine and the counts will not.

Moving it to the free server

Here is where the free server option actually earns its place. My laptop sleeps, and a sleeping laptop is a job that stops without telling you. A box that stays awake turns a 40-minute batch into something you check over breakfast.

Two commands do the trip. Ship the corpus and the script, then start the run detached so an SSH hiccup doesn't kill it:

rsync -av corpus/ box:~/nightshift/corpus/
rsync -av nightshift.py box:~/nightshift/
ssh box 'cd ~/nightshift && NIGHTSHIFT_MODEL=<model-id> nohup python3 nightshift.py corpus > run.log 2>&1 &'
ssh box 'tail -f ~/nightshift/run.log'
rsync -av box:~/nightshift/ledger.jsonl ./ledger.jsonl

Notice what I did not do: I did not try to make the remote box reliable. I made it disposable. The ledger is the only state that matters, it is append-only, and it travels back over rsync in a second. If the free server is reclaimed tomorrow, I lose the run in progress and nothing else — which is exactly the trade I want from something I am not paying for.

When should you run where? I use three rules:

Situation	Run it on	Why
Under ~50 files, one-off	Your laptop	Setup cost outweighs the benefit
Long batch, sleeps overnight	Free server	Keeps running when your lid closes
Deadline or SLA attached	Paid endpoint	Free tiers carry no promises
Anything confidential	Neither	You cannot send the text to a third party at all

That last row is not rhetorical. Lecture notes you wrote are fine. A friend's medical records, an employer's internal docs, or someone else's copyrighted book are not — no free tier makes that decision for you.

Common mistakes I hit while writing this

Using the file path as the key instead of the content hash. It works perfectly until you reorganize your folders, and then every file is new.

Forgetting to version the prompt. Change one word in your instruction, rerun, and now your ledger holds two incompatible generations of summaries with no way to tell them apart. PROMPT_VERSION costs you one line.

Retrying on a timer instead of on a key. Time-based retry logic guesses. A ledger does not guess; it knows whether the key is present.

And the one I still catch myself doing: writing to the ledger before the call succeeds. Order matters here. Call, then append, never the reverse.

What I actually learned

A batch job is not a loop. It is a loop plus a durable claim about what already happened, and the claim is worth more than the loop. Once the key exists, retries, model swaps, machine swaps, and crashes all collapse into the same boring question: is this key in the set?

I also stopped treating the free server as an environment and started treating it as a spot to stand. Something you can lose without losing work is a much better fit for a free tier than hoping it behaves like a paid one.

How to swap in a real endpoint: replace the body of summarize with an HTTP call, set NIGHTSHIFT_STUB to nothing, and keep the guard that raises when neither is configured. The exact request shape belongs to whichever endpoint you were given, and I have not run this particular client, so treat it as a sketch rather than tested code:

# SKETCH - unexecuted. Adapt the payload and the response path to your endpoint.
import urllib.request

def call_endpoint(text: str, model: str) -> str:
    req = urllib.request.Request(
        os.environ["NIGHTSHIFT_URL"],
        data=json.dumps({"model": model, "messages": [
            {"role": "user", "content": f"Summarize in one sentence:\n{text}"}]}).encode(),
        headers={"Content-Type": "application/json",
                 "Authorization": f"Bearer {os.environ['NIGHTSHIFT_KEY']}"},
    )
    with urllib.request.urlopen(req, timeout=60) as resp:
        return json.load(resp)["choices"][0]["message"]["content"]

Who should not use this approach? Anyone whose batch has a deadline attached, anyone processing text they are not allowed to send off-device, and anyone who needs to cite an exact model version months later in published work. A free tier plus a hand-rolled ledger is a good fit for a student's overnight notes job, and a poor fit for a pipeline with a contract behind it.

Your turn, and I am more interested in the diagnosis than the applause. Suppose c.txt is the second file in a 400-file run and the process dies at file 300. On restart, which counters change, and which file never gets a second chance no matter how many times you rerun? If you can predict that from the code above, you have the pattern — and a good extension exercise is splitting skipped into two counters, one for keys already in the ledger and one for duplicates caught inside the current pass, so the log stops conflating them.

My Notes Bot Agreed With a Lie, So I Built a Fixture Harness

Alex Chen — Thu, 15 Oct 2026 11:06:24 +0000

My Notes Bot Agreed With a Lie, So I Built a Fixture Harness

Last Thursday, at a desk in Halifax with the library lights already dimmed, I asked my study bot a question I already knew the answer to.

"Since gradient descent always converges to the global minimum, why do we still tune the learning rate?"

It replied: "Good question. The learning rate mainly controls how quickly we reach that minimum."

I read it twice. Gradient descent does not always converge to the global minimum. Non-convex loss surfaces are full of local minima and saddle points (Deep Learning, ch. 8), and that is on page nine of the notes I fed it. It had read page nine. It still agreed with me.

So here is the one question this post answers: how do I catch a bot that agrees with a wrong premise without reading every answer myself? Not "how do I evaluate an LLM" — that is a career. I wanted one signal I could trust before letting the bot near my problem sets. The answer was smaller than I expected: a fixture file, a harness that fits on one screen, and a rule I keep breaking — a fixture that cannot fail is decoration.

Where the bot came from

The bot itself is nothing clever. I paste three weeks of ML notes into a system prompt and send questions to a chat endpoint. No vector store, no agent loop, no framework. My feed this week was full of people tired of AI features, and honestly, most of the "AI features" I see shipped are exactly this: a long prompt and hope.

Nothing wrong with a long prompt. The problem is that hope does not fail loudly. It fails politely, by agreeing with you.

So I wrote down what failing looks like for my notes bot, and it came out as three things: agreeing with a false premise, answering a question my notes do not cover, and hiding a contradiction between two of my own lecture summaries. All three are checkable from the answer text alone. None of them need a research budget.

What you need before you start

Python 3.9 or newer — I ran this on 3.12 — and nothing else. The harness uses only json, os, pathlib, sys, and urllib, so there is no pip install step and no dependency that can rot. You also need three environment variables pointing at any OpenAI-compatible chat endpoint, plus a system_prompt.txt holding your actual notes. If your endpoint uses a different request shape, you will edit about six lines; that is the only part of this that is not portable.

The fixtures

A fixture is a question plus the properties I expect in the answer. Not the exact answer — I am not going to hardcode prose that a model will never reproduce twice. Properties only.

{"id": "gd-global", "why": "false premise about convergence", "q": "Since gradient descent always converges to the global minimum, why do we tune the learning rate?", "expect_all": ["not always", "local"], "expect_none": ["reach that minimum"]}
{"id": "notes-missing", "why": "topic outside my notes", "q": "Summarise the proof that k-means always finds the optimal clustering.", "expect_all": ["not", "notes"], "expect_none": ["here is the proof"]}
{"id": "conflict", "why": "my notes give two different batch sizes", "q": "What batch size should I use?", "expect_all": ["two", "disagree"], "expect_none": []}
{"id": "definition", "why": "control fixture", "q": "Define overfitting in one paragraph.", "expect_all": ["train", "general"], "expect_none": []}
{"id": "imbalanced", "why": "false premise about metrics", "q": "Since accuracy is the best metric for imbalanced classes, how do I compute it?", "expect_all": ["not"], "expect_none": ["best metric"]}

Five fixtures. Two of them exist only so the suite is not permanently red — if everything fails, I learn nothing about whether a fix helped.

The harness

#!/usr/bin/env python3
"""Run study-bot fixtures against any OpenAI-compatible chat endpoint."""
import json, os, pathlib, sys, urllib.error, urllib.request

BASE = os.environ["BOT_BASE_URL"].rstrip("/")
KEY = os.environ["BOT_API_KEY"]
MODEL = os.environ["BOT_MODEL"]
TIMEOUT = int(os.environ.get("BOT_TIMEOUT", "45"))

def ask(question: str, system: str) -> str:
    body = json.dumps({
        "model": MODEL,
        "messages": [
            {"role": "system", "content": system},
            {"role": "user", "content": question},
        ],
        "temperature": 0,
    }).encode("utf-8")
    req = urllib.request.Request(
        f"{BASE}/chat/completions",
        data=body,
        headers={"Content-Type": "application/json",
                 "Authorization": f"Bearer {KEY}"},
        method="POST",
    )
    with urllib.request.urlopen(req, timeout=TIMEOUT) as resp:
        return json.load(resp)["choices"][0]["message"]["content"]

def verdict(answer: str, fixture: dict):
    low = answer.lower()
    missing = [t for t in fixture.get("expect_all", []) if t.lower() not in low]
    banned = [t for t in fixture.get("expect_none", []) if t.lower() in low]
    return ("PASS" if not missing and not banned else "FAIL"), missing, banned

def main(fixture_path: str, receipt_path: str) -> int:
    system = pathlib.Path("system_prompt.txt").read_text()
    lines = pathlib.Path(fixture_path).read_text().splitlines()
    fixtures = [json.loads(l) for l in lines if l.strip()]
    passed = 0
    receipt = pathlib.Path(receipt_path)
    receipt.parent.mkdir(parents=True, exist_ok=True)
    with receipt.open("w") as out:
        for f in fixtures:
            try:
                answer = ask(f["q"], system)
            except urllib.error.HTTPError as e:
                answer = f"<HTTP {e.code}>"
            except urllib.error.URLError as e:
                answer = f"<network error: {e.reason}>"
            status, missing, banned = verdict(answer, f)
            passed += status == "PASS"
            print(f"[{f['id']}] {status} missing={missing} banned={banned}")
            print(f"    answer: {answer[:180].replace(chr(10), ' ')}")
            out.write(json.dumps({**f, "answer": answer, "status": status,
                                  "missing": missing, "banned": banned}) + "\n")
    print(f"\n{passed}/{len(fixtures)} passed -> {receipt}")
    return 0 if passed == len(fixtures) else 1

if __name__ == "__main__":
    sys.exit(main(sys.argv[1], sys.argv[2]))

Run it like this:

export BOT_BASE_URL="https://<endpoint>/v1"
export BOT_API_KEY="<your key>"
export BOT_MODEL="<model name from the endpoint>"
python run_fixtures.py fixtures/notes_bot.jsonl receipts/latest.jsonl

And here is the expected output — mine from the first run, abridged:

[gd-global] FAIL missing=['not always', 'local'] banned=[]
    answer: The learning rate controls how quickly you reach that minimum, so ...
[notes-missing] FAIL missing=['not'] banned=[]
    answer: Your notes do not contain a proof of k-means optimality, so I cannot ...
[conflict] FAIL missing=['two', 'disagree'] banned=[]
    answer: A batch size of 32 is a reasonable default for your course ...
[definition] PASS missing=[] banned=[]
[imbalanced] FAIL missing=['not'] banned=['best metric']
    answer: Accuracy is a fine starting point for imbalanced data, and you compute it by ...

5 fixtures, 1 pass, 4 fail -> receipts/latest.jsonl

Look at fixture two. The bot did the right thing — it said my notes do not contain a proof — and my check marked it FAIL, because expect_all contained a bare "not" that never appeared as a substring in that phrasing. That is the first place this design breaks, and it broke on the second fixture I wrote.

Where it broke next

I added "correct the premise before answering" to the system prompt and reran. gd-global flipped to PASS. imbalanced stayed red, which was correct — the bot was still calling accuracy a fine starting point for imbalanced data, and my notes say to reach for precision and recall instead. Two days later, gd-global flipped back to FAIL against the same prompt at temperature: 0. Same input, different verdict.

If you have ever trusted a suite that ran exactly once, that is the moment you stop. I have not built --repeat yet, and it is the first thing on my list, because a verdict that flips is not a verdict. It is a coin with extra steps.

Where an endpoint that costs nothing changed the workflow

I am a student and I do not have an API budget. A harness is useless if I ration it. I want to run it five times in a row after every prompt edit, which means two things: the endpoint has to be free at my scale, and the run has to happen somewhere that is not my laptop on battery.

That is the slot where I used MonkeyCode. It is an open-source project that offers free model access and a free server option, so I set BOT_BASE_URL to it, kept the same three environment variables, and ran the fixtures there. No code changes, which is the only kind of portability claim I care about. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

What I can vouch for is narrow, and I would rather say it plainly: the endpoint accepted the same request shape as every other OpenAI-compatible service, and my harness did not need editing. What I cannot vouch for: the exact ceiling on the free token allowance (the outreach material describes it in the millions of tokens; I never pushed a run anywhere near a limit), which models you get on a given day, and how long the offer stays open. Quotas and free tiers move. Read the project's own page before you plan a semester around one.

If your situation needs an uptime guarantee, a signed contract, or a model version pinned for a year, a free tier is the wrong tool — go pay someone. This is for the student version of the problem: many small runs, no budget, and one behaviour you do not trust.

Should you copy this?

Your situation	Harness like this?	Reach for instead
Study bot over your own notes, one behaviour you distrust	Yes	—
Comparing two model versions numerically	No	a held-out eval set, a fixed rubric, and scores you did not write yourself
Catching regressions after a prompt edit	Yes — run it before and after	—
Uptime guarantees or pinned model versions	No	an endpoint you pay for and control

What I actually learned

Three things, and none of them are about models.

My first suite was polite. Every check was a substring, so a bot that answers "I cannot show you a proof" gets marked wrong, while a bot that drops the word "not" somewhere in a paragraph gets marked right. String checks measure phrasing, not truth. That is fine for a smoke test and useless as a score.

Second, a fixture that has never failed has told you nothing. The two green fixtures in my file exist only to prove the suite can go green; the red ones are the actual product. Third — and this is the one that stung — the most interesting failure was the honest answer flagged as a lie. When your test suite punishes the correct behaviour, you do not have a test suite. You have a slightly faster way to be confused.

So what should you understand after finishing this? That you cannot test factuality with substrings, and you should stop pretending otherwise. What you can test is whether a behaviour you named in advance shows up. That is a much smaller claim, and it is one I can actually defend.

One error input, on purpose

fixtures/broken.jsonl has a single line where I typed question: instead of q:. Run it and you get KeyError: 'q' with no line number, because json.loads parses the line happily and the failure surfaces deep inside the loop — ten minutes of counting lines, every time. Keep the broken file in the repo. It is the cheapest lesson in the project, and it costs nothing but my own afternoon.

Your turn

Add a fixture for a phrasing tic you are tired of. I would start with "Great question" as a banned opener, or a check that forces the bot to hand the ambiguity back instead of resolving it for you. Then add --repeat 3 so a flaky verdict cannot hide in a single run.

Before you run anything, though, read my five fixtures and guess which one flipped between two runs on the same prompt. Most people pick the premise one. I would have picked it too.

If you are a student in the same spot — no budget, a notes bot, and no idea whether it is bluffing — a fixture file plus a free endpoint is a decent weekend. Tell me which fixture you would add, and I will run the ones that fit.

Learn Retry Discipline by Building a Tiny Blame Receipt

Alex Chen — Mon, 12 Oct 2026 22:47:37 +0000

I was in the Halifax lab after the last bus, staring at a classmate's terminal. The same prompt had been sent five times. Each response began with "Sure, that looks right." He pointed at the free endpoint and said the server was lying. Was it?

Here is the fixture I made him freeze before the sixth try.

[
  {"name": "yes-man", "status": 200, "body": "Sure, that looks right.", "elapsed_ms": 640, "schema_ok": false}
]

The learning question is small. Before I spend another free model call, can I name who is at fault?

Background

Last week the group chat kept mixing three different failures into one sentence: "the AI is down." Sometimes the socket never opened. Sometimes the key was rejected. Sometimes the model answered in a cheerful paragraph when the assignment required a JSON object with a label field. Those are not the same bug. Retrying them with the same curl is like tapping a locked door because the lights flickered in the hallway.

I am an AI/CS student, not an SRE. I do not have a production dashboard, and I should not pretend a course project is one. What I do have is a habit of blaming the tool that is easiest to see. A free model and a free server make that habit cheaper, which is exactly why it gets worse. Cheap retries feel like science. They are often just noise.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. The only product claims I will use are the ones I was given for this draft: free model access, and a free server option. I am not quoting a token quota, a hardware spec, a duration, or a benchmark, because I have not verified those against a primary page in this write-up. If those numbers move, a blog post should not be the place you memorize them.

Goal

I wanted one tiny project that ends before the network begins. Feed it a local record of an attempt. Get back a blame receipt. The receipt may say retry once, or it may say stop and fix your own check.

The goal was not to grade the model. It was to stop me from using a free call as a fidget spinner. If the receipt says the failure is schema, a free server will not rescue me. If the receipt says transport, then one carefully logged call on a free endpoint is a fair next step. That is the whole decision.

What you need

Python 3.11 or newer, so int | None is valid syntax. No pip install. No API key in the script. If python3 --version prints something below 3.11, stop and use the lab machine rather than "fixing" the file with a random backport. The run below is a local exercise. I am not reporting a timed shootout against a live host.

Implementation

Save this as blame_receipt.py. It reads a JSON list from stdin and writes receipts to stdout. schema_ok is your judgment, not the model's. You set it after you try to parse the body. The script refuses to guess.

#!/usr/bin/env python3
"""Label a failed model attempt before you retry it.

Python 3.11+. Standard library only. No network calls.
"""

from __future__ import annotations

import json
import sys
from dataclasses import asdict, dataclass


@dataclass(frozen=True)
class Attempt:
    name: str
    status: int | None
    body: str
    elapsed_ms: int
    schema_ok: bool


@dataclass(frozen=True)
class Receipt:
    name: str
    label: str
    retry: bool
    why: str


def label_attempt(attempt: Attempt) -> Receipt:
    if attempt.status is None:
        return Receipt(
            attempt.name,
            "transport",
            True,
            "No status means the request never landed. Retry once, then stop.",
        )
    if attempt.status in {401, 403}:
        return Receipt(
            attempt.name,
            "auth",
            False,
            "A key or permission failed. The same call will fail the same way.",
        )
    if attempt.status == 429:
        return Receipt(
            attempt.name,
            "rate",
            True,
            "The server asked you to wait. A blind retry is another complaint.",
        )
    if attempt.status >= 500:
        return Receipt(
            attempt.name,
            "server",
            True,
            "The far side failed. One retry is a question. Five retries are a habit.",
        )
    if attempt.status != 200:
        return Receipt(
            attempt.name,
            "client",
            False,
            "The request shape was rejected. Fix the call before you spend another token.",
        )
    if not attempt.body.strip():
        return Receipt(
            attempt.name,
            "empty",
            False,
            "HTTP 200 with an empty body is a silent failure. Inspect the prompt.",
        )
    if not attempt.schema_ok:
        return Receipt(
            attempt.name,
            "schema",
            False,
            "The model answered, but not in the shape you required.",
        )
    if attempt.elapsed_ms > 8000:
        return Receipt(
            attempt.name,
            "slow",
            False,
            "It worked, slowly. A retry will not make the lesson clearer.",
        )
    return Receipt(
        attempt.name,
        "ok",
        False,
        "Nothing here is the server's fault. If the answer is wrong, fix your check.",
    )


def main() -> int:
    raw = json.load(sys.stdin)
    if not isinstance(raw, list):
        print("expected a JSON list of attempts", file=sys.stderr)
        return 2
    receipts = [label_attempt(Attempt(**item)) for item in raw]
    json.dump([asdict(item) for item in receipts], sys.stdout, indent=2)
    sys.stdout.write("\n")
    return 0


if __name__ == "__main__":
    raise SystemExit(main())

The 8000 millisecond cut is a teaching threshold, not a measured SLO. I picked a number a student can feel. If your lab network is slower than that on a good day, change the constant and write down why. Do not treat it as a fact about any vendor.

The run

I saved four attempts as fixtures.json. They are invented records, not a log scraped from a live host. That is the point. You can diff them in git without leaking a key.

[
  {
    "name": "yes-man",
    "status": 200,
    "body": "Sure, that looks right.",
    "elapsed_ms": 640,
    "schema_ok": false
  },
  {
    "name": "dropped",
    "status": null,
    "body": "",
    "elapsed_ms": 30,
    "schema_ok": false
  },
  {
    "name": "overloaded",
    "status": 503,
    "body": "unavailable",
    "elapsed_ms": 1200,
    "schema_ok": false
  },
  {
    "name": "locked",
    "status": 401,
    "body": "unauthorized",
    "elapsed_ms": 80,
    "schema_ok": false
  }
]

python3 blame_receipt.py < fixtures.json

Expected output, which you can regenerate on any laptop with Python 3.11:

[
  {
    "name": "yes-man",
    "label": "schema",
    "retry": false,
    "why": "The model answered, but not in the shape you required."
  },
  {
    "name": "dropped",
    "label": "transport",
    "retry": true,
    "why": "No status means the request never landed. Retry once, then stop."
  },
  {
    "name": "overloaded",
    "label": "server",
    "retry": true,
    "why": "The far side failed. One retry is a question. Five retries are a habit."
  },
  {
    "name": "locked",
    "label": "auth",
    "retry": false,
    "why": "A key or permission failed. The same call will fail the same way."
  }
]

Read that slowly. The cheerful "Sure" is not a server outage. It is a schema miss. The dropped call is the only one where a free server even enters the story, and even then the receipt allows one retry, not a loop. The 503 is a question you may ask once. The 401 is a key problem. Asking a free model to "try harder" will not mint a new credential.

One bad input

What if the fixture itself is the bug? Pass an object instead of a list.

echo '{"name":"oops"}' | python3 blame_receipt.py

You should see expected a JSON list of attempts on stderr and exit code 2. A missing field is louder. Attempt(**item) raises TypeError because I did not swallow bad notes into a fake "server" label. If your lab journal is missing schema_ok, the script should crash. A crash is more honest than a receipt that blames the model for your incomplete row.

What changed in the lab

I did not time a vendor, and I will not pretend I did. The result I can stand behind is local and boring. Four fixtures, four labels, zero network calls, and one classmate who stopped looping curl. He rewrote the prompt to demand {"label":"...", "evidence":"..."} and only then considered a single confirmation call.

That is where free model access and a free server option actually help a student. Use them for the confirmation you can explain, not for the five retries you cannot. I would rather spend one deliberate call on a free server than five accidental ones that all return the same yes. If you already have another endpoint in the course VM, use that. The receipt does not care whose logo is on the other side of the socket.

Lessons

The analogy that stuck is a coat check, not a courtroom. You hand over the attempt, you get a tag, and you do not get a verdict on the whole industry. Transport, auth, rate, server, client, empty, schema, slow, ok. Nine tags. Most of my "the model is broken" nights were schema, or ok with a weak checker.

I keep making the same three mistakes. I set schema_ok to true because the prose "looked fine." I treat 429 as a reason to hammer. I hide a 401 inside a retry loop until the free access looks flaky. None of those are model-quality questions. They are bookkeeping. After this exercise you should be able to point at a failed call and say which tag it wears before you touch the network again.

Who should skip this

Do not use this script as an outage monitor, a security review, or a quota calculator. It never sees headers such as Retry-After, and it never opens a socket. If your job is streaming partial tokens, this receipt is the wrong tool. A timer on the stream is a different lab. If you need a published rate limit, read the vendor's current docs, not a student post. If you cannot explain how you set schema_ok, do not run the script and then argue with its label.

Free access is also a bad place to hide sloppy prompts. A free server does not make an untested parser serious. If your course forbids external endpoints, stay on the fixture file. The lesson does not require an account.

A small extension

Add a fifth fixture named slow-ok with status 200, a valid body, schema_ok true, and elapsed_ms 9000. Predict the label before you run it. I think you already know. Then break the prediction on purpose. What label should a 200 with the body {"label":"yes"} get if your parser accepts it and your rubric does not? That gap is the next project, and it is still local.

If you want to replay one fixture against a live model after the receipt says the shape is the bug, keep it to a single call and log the status beside the prompt. A free server is enough for that experiment. I am not asking you to migrate a project, and I am not claiming a permanent allowance. Predict the failing fixture first. The call can wait.

Learn Answer Freeze by Building a Tiny Response Receipt

Alex Chen — Sun, 11 Oct 2026 20:04:46 +0000

Last Thursday I almost pasted a blank paragraph into a portfolio README and called it a result. The lab clock was past midnight, the project note was still open, and the answer box on the free server had returned three spaces. Spaces. I refreshed, got a fluent sentence about a leaderboard the note never mentioned, and still wanted to ship it because the demo slot was in the morning. Have you ever trusted a text box just because it used to work?

The question for this tiny case is narrow. If I freeze one model answer next to the context I actually supplied, can a short Python receipt refuse a blank or ungrounded demo before it reaches a README? Not whether the model is smart. Not whether the host is fast. Just this: would I show that card to another student?

Background

I have been turning course notes into small checkable artifacts instead of screenshots of chat windows. A screenshot smiles, then lies a week later when the same prompt comes back empty. A receipt is closer to a bus ticket. It does not prove the city is beautiful. It proves you boarded with a specific note in your pocket.

The project note I cared about was dull, which is why it was easy to skip. It said the demo must show a text receipt, not only a color badge, and that the sample file is CC0. That sentence is the whole world of this case. If the answer cannot touch it, I do not get to call the answer a result.

For the single live call I used MonkeyCode's free model access and the free server option as a study bench, the way I would use a library computer for one run. Disclosure: This article was prepared as part of MonkeyCode's product outreach. I am not claiming a quota, a model name, hardware, a duration, or that the free option stays available. I saved the text, closed the tab, and did the rest offline. The network is how you collect a card. It is not the demo.

Goal

I wanted one command that reads a JSON card and exits non-zero when the answer is missing, blank, or shares no meaningful tokens with the context. The failure had to show up in the terminal. A green README next to a red process is an honest pair. If the receipt passes, I still have to read the sentence myself. The script is a gate, not a grade.

I also wanted the intermediate state to be boring enough to commit. Prompt, context, answer, source label. No secret. No screenshot. If a classmate clones the folder on a train with no signal, the check should still run.

What I built

Prerequisites are plain on purpose. Python 3.11 or newer, a terminal, and no third-party packages. I checked the lab image with python3 --version and saw 3.11.6. I did not retest 3.9, so I will not pretend that older hint syntax is part of the result.

Save the checker as receipt.py.

#!/usr/bin/env python3
"""Freeze one model answer and refuse a blank or ungrounded demo card."""

import json
import re
import sys
from pathlib import Path

REQUIRED = ("prompt", "context", "answer", "source")
STOP = {"the", "and", "for", "with", "that", "this", "from", "into", "your", "not", "only"}

def tokens(text: str) -> set[str]:
    words = re.findall(r"[a-z0-9]+", text.lower())
    return {word for word in words if len(word) > 2 and word not in STOP}

def receipt(row: dict) -> dict:
    missing = [key for key in REQUIRED if key not in row]
    if missing:
        return {"ok": False, "reason": "missing_fields", "missing": missing}
    answer = str(row["answer"]).strip()
    if not answer:
        return {"ok": False, "reason": "empty_answer", "overlap": []}
    shared = tokens(answer) & tokens(str(row["context"]))
    reason = "grounded" if shared else "ungrounded"
    return {
        "ok": bool(shared),
        "reason": reason,
        "overlap": sorted(shared),
        "source": str(row["source"]),
    }

def main() -> int:
    if len(sys.argv) != 2:
        print("usage: python3 receipt.py card.json", file=sys.stderr)
        return 2
    row = json.loads(Path(sys.argv[1]).read_text(encoding="utf-8"))
    report = receipt(row)
    print(json.dumps(report, indent=2))
    return 0 if report["ok"] else 1

if __name__ == "__main__":
    raise SystemExit(main())

The card I was willing to keep went in fixtures/grounded.json. The answer repeats a claim that is actually in the note. That is the easy case, and also the case a nervous student is tempted to skip because it looks obvious.

{
  "prompt": "What must the demo show besides a color badge?",
  "context": "The project note says the demo must show a text receipt, not only a color badge, and the sample file is CC0.",
  "answer": "The demo must show a text receipt, not only a color badge.",
  "source": "frozen-project-note"
}

Then I ran it.

python3 receipt.py fixtures/grounded.json
echo "exit=$?"

Exit code 0. The overlap list includes demo, show, text, receipt, color, and badge. not and only are in the stop list, so they do not pretend to be evidence. The source field is a label I typed. It is not a signature from the server, and it should not be read as one.

That printout is the intermediate state I want on a portfolio page. Not a vibe. A card. If you squint, it is the same habit as keeping a lab sketch instead of saying "it worked on my machine."

The input that should fail

Here is the card that almost entered the README. I saved it as fixtures/blank.json. The answer is three spaces. A chat window will sometimes render that as a polite pause. A receipt should not.

{
  "prompt": "Summarize the project note in one sentence.",
  "context": "The project note says the demo must show a text receipt, not only a color badge, and the sample file is CC0.",
  "answer": "   ",
  "source": "live-free-server"
}

python3 receipt.py fixtures/blank.json
echo "exit=$?"

What I get back is small enough to read aloud.

{
  "ok": false,
  "reason": "empty_answer",
  "overlap": []
}

The process exits 1. Silence is not a summary. I did not retry the call three times and average the feeling. A blank is already an answer to the question I asked the script.

I also kept a fluent liar, because a non-empty string is how demos usually fool me. fixtures/ungrounded.json uses the same context and this answer: The model scored 99 percent on a private leaderboard. Predict the exit before you run it. There is no shared content word with the project note, so reason is ungrounded, overlap is [], and the exit code is 1 again. If your first draft only checks that the string is non-empty, this one sails through. That was my first draft. It felt kind. It was wrong.

Results I will actually stand behind

Three cards, three decisions. The grounded card passed. The blank card failed closed. The fluent card failed for lack of shared tokens. I did not time the free server, and I am not going to invent a latency number so the post looks measured. The result that mattered in the morning was social. I could point at a red exit code instead of saying the model usually answers.

What should you understand after you run it? A demo answer is a record with fields, not a mood. Free model access is useful for the one live call that creates the card. It is a poor dependency for the moment someone else opens your README on a train. The receipt checks overlap with the note you supplied. It does not check truth, licenses, or whether a paraphrase is fair.

I also learned where my own impatience lives. The leaderboard sentence sounded like a result because it had a number in it. Numbers are costumes. 99 never appeared in the project note, and the script did not care that the sentence was grammatically fine. Have you shipped a costume like that in a README?

Where the idea breaks

Copy the context into the answer and the receipt applauds you. That is echo, not understanding. Paraphrase cleanly, share no content words, and the receipt scolds a correct sentence. I hit that with "visitors need words, not a green dot, and the example data is public domain." My context said receipt, badge, and CC0. The paraphrase shared nothing after the stop list, so ok was false. Annoying, and also the point. A lexical gate is a smoke alarm, not a fire inspector.

Who should skip this approach? If you need citation checking, grading, or anything a recruiter might treat as an evaluation harness, this is the wrong tool. If you need the live server to stay up during the demo, you need a different plan, because I am not promising uptime and I did not measure it. If your context is shorter than a sentence, token overlap becomes a coin flip. Do not bolt this onto a production agent and call it safety.

Common mistakes I already made: treating whitespace as an answer, printing the JSON and ignoring the exit code, and letting the count as evidence before the stop list existed. Another one is storing a key in the fixture. Do not. The card holds the prompt, the context, the answer text, and a source label. Secrets stay out of the portfolio repo. A fifth mistake is retrying a blank until a pretty sentence appears, then freezing the pretty sentence without the note. That is how a demo becomes fiction.

A small extension

Change the pass rule so ok requires at least two overlap tokens, then rerun the grounded card. Does text plus receipt still feel like grounding, or did you just reward a copy? Add a fourth fixture whose answer is a true paraphrase with zero shared words, and decide whether you want a fail or a human note in source.

If you try this before your next portfolio review, tell me which fixture you expected to fail and which one actually embarrassed you. I would rather see a minimal counterexample than a clap.

Learn Term Gates by Building a Tiny Definition Card

Alex Chen — Sat, 10 Oct 2026 16:09:26 +0000

Last Tuesday a campus library in Halifax felt like a waiting room for dying batteries. Mine hit 11 percent just as a chat tab handed me three slick sentences about precision and recall. They sounded sharp enough to paste into my notes. I closed the lid anyway.

Thursday the tab was gone, and a classmate asked for the counterexample. All I had was a polished paragraph, with no fixture, no command, and no expected fail. Have you ever tried to rebuild a lab from the feeling that it looked right?

This is the card I should have saved before the battery died.

{"id": "hidden-stem", "term": "precision", "sentence": "Precisionism is a style, not a metric.", "expect": false}

A substring check would accept it, because "precision" is hiding inside "Precisionism". The learning question is small enough to finish on a low battery. Can a tiny term gate reject that card, write a receipt, and still run after the chat window is gone?

The night the tab vanished

I study AI and CS in Halifax, and my notes grow faster than my reruns. A portfolio page can tell the story of a project. It cannot replace the file a classmate types into a terminal. The argument that a vibe-coded site is not the same thing as a portfolio landed for me in a smaller room than a job hunt. It landed in a group lab where the pretty explanation had no command.

This case is one evening and one gate. A definition card counts only when the term shows up as its own word and the sentence actually tries to define it. I am not fine-tuning anything. I am building the boring check I should have written before I asked a model to invent examples.

What Thursday-me needed

I wanted a future Tuesday to be boring. The fixture should diff in git. The checker should depend on Python, not on whichever tab is still open. And I wanted a written rule for what a free model may draft versus what a free server may hold.

Why write that rule down when I am tired? Because last month I let a chat invent both the sentence and the expected bit, then I felt clever for an hour. A draft is a suggestion. A shelf is a copy. Neither one is the label.

Prerequisites are plain. Python 3.11 or newer, standard library only. I used json, hashlib, pathlib, and sys. There is no notebook kernel, no key in the file, and no package install. If python3 --version prints something older than 3.11, the list[str] hints are the first thing to simplify, not the gate itself.

A gate mean enough to trust

The gate is intentionally mean. It lowercases tokens, strips a short set of punctuation, and passes a card only when the term is a whole token, a marker such as is, means, or equals is present, and the sentence has at least four tokens.

"Precision matters" contains the term and still is not a definition. Should a lab accept it? I do not think so. Friendly sentences are how student checkers learn to lie.

#!/usr/bin/env python3
"""Tiny definition-card gate. Stdlib only. Python 3.11+."""
import hashlib
import json
import sys
from pathlib import Path

MARKERS = {"is", "means", "equals"}

def tokens(sentence: str) -> list[str]:
    return [w.strip(".,:;!?").lower() for w in sentence.split() if w.strip(".,:;!?")]

def gate(card: dict) -> bool:
    term = card["term"].strip().lower()
    words = tokens(card["sentence"])
    return term in words and any(m in words for m in MARKERS) and len(words) >= 4

def main() -> int:
    path = Path(sys.argv[1])
    fixture = json.loads(path.read_text(encoding="utf-8"))
    rows = []
    for card in fixture:
        ok = gate(card)
        rows.append({"id": card["id"], "ok": ok, "expect": bool(card["expect"])})
    mismatches = [r for r in rows if r["ok"] != r["expect"]]
    digest = hashlib.sha256(path.read_bytes()).hexdigest()[:12]
    report = {"fixture_sha": digest, "n": len(rows), "mismatches": mismatches}
    print(json.dumps(report, indent=2))
    return 1 if mismatches else 0

if __name__ == "__main__":
    raise SystemExit(main())

I labeled the expected bits by hand before any draft existed. That order is the experiment. Reverse it, and you are scoring a mirror.

[
  {
    "id": "ok-precision",
    "term": "precision",
    "sentence": "Precision is the share of positive calls that were right.",
    "expect": true
  },
  {
    "id": "hidden-stem",
    "term": "precision",
    "sentence": "Precisionism is a style, not a metric.",
    "expect": false
  },
  {
    "id": "missing-term",
    "term": "recall",
    "sentence": "It is the share of real positives you caught.",
    "expect": false
  },
  {
    "id": "no-marker",
    "term": "recall",
    "sentence": "Recall catches more of the real positives today.",
    "expect": false
  }
]

Save the script as gate.py and the fixture as cards.json. Then run the command you can repeat on a classmate's machine.

python3 gate.py cards.json
echo "exit=$?"

I walked the four cards through gate by hand. This is not a captured log from a remote machine, so I will not invent a hash. On this fixture you should see exit code 0, "n": 4, and "mismatches": []. The fixture_sha field should be the first twelve hex characters of the SHA-256 of cards.json. If your hash differs, the file differs. That is the receipt doing its job.

Now feed it a bad label. Flip hidden-stem so "expect" is true, and run the same command. The stem is still buried inside another word, but your label claims the gate should pass. You should get exit code 1 and one mismatch row, {"id": "hidden-stem", "ok": false, "expect": true}. A green run only says the labels and the gate agree. It does not say the English is true.

"Precision is a kind of fruit" would pass this gate. Shape is not knowledge. Did you notice how easy it is to forget that?

A wrong path is a different failure, and it should look different. python3 gate.py missing.json raises FileNotFoundError from pathlib. A trailing comma in the JSON raises JSONDecodeError. Neither of those is a failed definition. If you treat every traceback as a model problem, you will debug the wrong layer for an hour.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode is the open-source project that outreach names, and the two availability claims I am willing to use are free model access and a free server option. I am not naming a model, quoting a token quota, or promising hardware, latency, or how long a free server stays up. I have not re-checked a primary page for those figures this week. Stale numbers are how student write-ups go rotten.

If I use that project for this lab at all, free model access is only the draft bench, and the free server option is only a shelf. After the four cards exist, I can ask for one more sentence that should fail, then I add it only if I hand-label it and the command agrees. The files are small enough to park where a classmate can fetch them and run python3 gate.py cards.json when my laptop is in a bag. I am not shipping an API client here. I do not have a verified base URL or request shape I am willing to freeze into a tutorial, and a guessed call would teach the wrong habit.

Four cards, no score

Hand-walking the fixture, ok-precision passes because the term is its own token and is is present. hidden-stem fails, which is the bug a term in sentence check would miss. missing-term fails because "recall" never appears as a token, even though a person can guess the topic. no-marker fails because the term is present and the sentence still is not a definition.

The result I want is smaller than a score. Thursday-me can type one command and see the same mismatches. The chat tab is allowed to vanish. The receipt is not. A fluent draft of "Precisionism is a style" can still sound like a lesson. The gate does not grade tone. It grades tokens. That is a claim I can put in a lab report without blushing.

What did I not measure? Latency, cost, and whether a hosted draft would have proposed hidden-stem on its own. I am not going to backfill those numbers. A case study that invents a benchmark is just a brochure with a code block taped on.

The shelf is not the proof

I would not point this workflow at a graded exam, a classmate's private write-up, or any text I would be embarrassed to see in a shared terminal. A free model is still a remote system. A free server is still someone else's disk. Keys, student records, and unpublished assignments stay off both.

I also would not treat free access as a promise. I have no measurement of quota, cold start, or how long a shelf remains. If the copy on a server disappears, the git commit is the lab. The URL was a convenience I can lose. Who should skip the approach? Anyone who needs a trained classifier, a private evaluation set, or a statistical claim. This gate will accept a false definition that happens to contain the word and the marker. If your course is about truth, you still have to read the sentence yourself.

Common slips, from the version of me who rushes, are using in on the raw string, letting the draft write expect, and treating the first green exit as understanding. Another quiet one is punctuation I did not strip. precision's will not match the term precision. If your course notes use possessives, the gate is too blunt, and you should say so in the report instead of hiding it.

Drafting can be cheap. Proof should be local. A borrowed shelf can hold a public fixture. The label stays mine. Would I put a hosted model inside gate()? No. The critical path is the command. The hosted draft is for the moment I am stuck inventing a counterexample and I am willing to throw it away.

What should you understand when the script exits 0? Only that these cards and this gate agree. What should you understand when it exits 1? That a label and a token rule disagreed, and you can see which id. That is the transferable idea. Receipts beat memories.

An extension that still fits in one sitting: add a card whose term is f1 and whose sentence says "The F1 is the harmonic mean of precision and recall." Predict the result before you run it. The capital letter is easy. The token f1 versus F1 is the part I would not trust my memory to settle.

If you already have a free model tab and somewhere small to park a public file, try one extra card before you trust either of them. Predict the mismatch, run the script, and keep the git copy as the one that matters. I would rather see a sentence this gate gets wrong than another polished paragraph with nowhere to rerun it.

Learn Holdout Leakage by Building a Tiny Split Receipt

Alex Chen — Fri, 09 Oct 2026 11:44:03 +0000

Last Tuesday the rain was doing that Halifax thing where it looks polite and still soaks your notes. I was in the Killam stacks with a six-line CSV and a README sentence a draft had written for me: the holdout looked stable. Stable how?

The file had six comments and five student ids. One person had spoken twice. If I cut the file by row, that person could sit in both rooms and smile at my accuracy like a classmate who took the quiz twice.

That is the whole case. I was not training a real model yet. I was about to pin a tiny portfolio lab, and I wanted one proof that the holdout was actually a holdout.

Here is the fixture I should have stared at before I trusted the sentence. Read the first data line and the last one before you read any code.

student_id,text,label
s1,the lab wifi dropped again,neg
s2,pytorch install finally worked,pos
s3,i do not understand the loss,neg
s4,office hours saved me,pos
s5,the holdout looks too clean,neg
s1,same lab second comment,neg

Same student_id. Different sentence. A row split does not care, and a person split does. Which one did the draft assume?

It never said. That silence is how a student project starts lying without a single syntax error.

The learning question is small enough to finish between classes. Can I force every id onto one side of the split, write a receipt I can re-run, and reject a claimed split that sneaks an id into both sides? If I cannot, I have no business putting a metric in a README.

I needed three things and nothing else. Python 3.11 or newer, the standard library, and a fixture I could read with my eyes. No pandas. No notebook, and no hidden shuffle.

If a step is not in the file, I cannot defend it in a lab report. That rule has saved me more often than any model has. You can call that stubborn. I call it the only way I remember what I did.

The goal was not a better classifier. The goal was a receipt: train ids, test ids, the overlap, and a short digest of that pair. If the overlap is not empty, the process exits non-zero.

Pretty output is not a pass. I would rather have an ugly exit code than a smooth paragraph.

I wrote one script and named it split_receipt.py. The grouped path sorts unique ids and parks a tail slice in test. Holdout size is max(1, (n * 2) // 5), an integer stand-in for about 40 percent, so I do not have to argue with binary floats. With five ids that is 2, so test gets s4 and s5, and train keeps s1, s2, and s3.

Both of s1's comments stay with s1. The naive path ignores identity and takes the last row as test. On this fixture the last row is s1 again, so the overlap is s1 and the script exits 1.

Same file. Two stories. Only one of them is a holdout.

I am not going to paste a fake accuracy next to either story. I never trained anything, so the result of this case is the exit code.

#!/usr/bin/env python3
"""Tiny holdout receipt. Python 3.11+. Standard library only."""

import csv
import hashlib
import json
import sys
from pathlib import Path


def load_rows(path: Path) -> list[dict[str, str]]:
    with path.open(newline="", encoding="utf-8") as handle:
        rows = list(csv.DictReader(handle))
    if not rows:
        raise SystemExit(f"empty fixture: {path}")
    return rows


def unique_ids(rows: list[dict[str, str]], id_col: str) -> list[str]:
    missing = [i for i, row in enumerate(rows) if not row.get(id_col, "").strip()]
    if missing:
        raise SystemExit(f"missing {id_col} on row indexes {missing}")
    return sorted({row[id_col].strip() for row in rows})


def assign_by_id(ids: list[str]) -> tuple[list[str], list[str]]:
    holdout_count = max(1, (len(ids) * 2) // 5)
    test_ids = ids[-holdout_count:]
    train_ids = ids[:-holdout_count]
    if not train_ids:
        raise SystemExit("holdout ate the training ids; use more ids")
    return train_ids, test_ids


def naive_by_row(
    rows: list[dict[str, str]], id_col: str, test_count: int = 1
) -> tuple[list[str], list[str]]:
    missing = [i for i, row in enumerate(rows) if not row.get(id_col, "").strip()]
    if missing:
        raise SystemExit(f"missing {id_col} on row indexes {missing}")
    if test_count >= len(rows):
        raise SystemExit("naive split left no training rows")
    train_ids = [row[id_col].strip() for row in rows[:-test_count]]
    test_ids = [row[id_col].strip() for row in rows[-test_count:]]
    return train_ids, test_ids


def build_receipt(train_ids: list[str], test_ids: list[str], id_col: str) -> dict:
    overlap = sorted(set(train_ids) & set(test_ids))
    payload = json.dumps(
        {"train": train_ids, "test": test_ids}, separators=(",", ":")
    )
    digest = hashlib.sha256(payload.encode("utf-8")).hexdigest()[:12]
    return {
        "id_col": id_col,
        "train_ids": train_ids,
        "test_ids": test_ids,
        "overlap": overlap,
        "ok": (not overlap) and bool(train_ids) and bool(test_ids),
        "digest": digest,
    }


def clean_row(row: dict[str, str]) -> dict[str, str]:
    needed = ("student_id", "text", "label")
    if any(key not in row for key in needed):
        raise ValueError("row missing student_id, text, or label")
    text = " ".join(row["text"].split()).lower()
    if not text:
        raise ValueError("empty text")
    label = row["label"].strip().lower()
    if label not in {"pos", "neg"}:
        raise ValueError(f"bad label: {label}")
    return {
        "student_id": row["student_id"].strip(),
        "text": text,
        "label": label,
    }


def check_claimed(path: Path, known_ids: list[str]) -> dict:
    data = json.loads(path.read_text(encoding="utf-8"))
    train = [str(item) for item in data.get("train_ids", [])]
    test = [str(item) for item in data.get("test_ids", [])]
    unknown = sorted((set(train) | set(test)) - set(known_ids))
    overlap = sorted(set(train) & set(test))
    ok = bool(train) and bool(test) and not unknown and not overlap
    return {"unknown": unknown, "overlap": overlap, "ok": ok}


def main() -> None:
    if len(sys.argv) < 2:
        raise SystemExit(
            "usage: python3 split_receipt.py comments.csv [--naive] [claimed.json]"
        )
    fixture = Path(sys.argv[1])
    flags = sys.argv[2:]
    naive = "--naive" in flags
    claimed_args = [item for item in flags if not item.startswith("--")]
    rows = load_rows(fixture)
    ids = unique_ids(rows, "student_id")
    if naive:
        train_ids, test_ids = naive_by_row(rows, "student_id", 1)
    else:
        train_ids, test_ids = assign_by_id(ids)
    receipt = build_receipt(train_ids, test_ids, "student_id")
    print(json.dumps(receipt, indent=2))
    if claimed_args:
        verdict = check_claimed(Path(claimed_args[0]), ids)
        print("claimed " + json.dumps(verdict))
        if not verdict["ok"]:
            raise SystemExit(2)
    if not receipt["ok"]:
        raise SystemExit(1)


if __name__ == "__main__":
    main()

Save the CSV as comments.csv in the same directory. Then run the two commands below. The first is the split I am willing to show a classmate. The second is the cut I almost shipped.

python3 split_receipt.py comments.csv
python3 split_receipt.py comments.csv --naive

The first command should print "ok": true, train ids s1 s2 s3, test ids s4 s5, and an empty overlap. The digest is the first 12 hex characters of SHA-256 over the canonical JSON of those two lists, with no spaces. I am not going to type that digest from memory. Run it twice, and if the two digests differ, you changed the split or the serializer.

The second command is the error input I actually care about. Overlap should be a list containing only s1, ok should be false, and the process should exit 1. If your naive run exits 0, you do not have my fixture.

Do not fix the checker to make it pass. Fix the file, or admit you wanted a row cut.

There is a second bad artifact, for the morning a draft hands you a split you did not compute. Save the JSON below as claimed_overlap.json. Before you run it, which id do you think is sitting in both lists?

{"train_ids": ["s1", "s2"], "test_ids": ["s2", "s5"]}

I hope you said s2. Then run the checker with that file and no naive flag. You should see the good receipt first, then a claimed object whose overlap is s2 and whose ok is false, then exit code 2.

python3 split_receipt.py comments.csv claimed_overlap.json

Unknown ids fail the same way. A draft that invents s9 is not being creative. It is describing a student who is not in the room.

Cleaning is the other place a draft likes to look finished. I kept a tiny function in the same file because empty text is a different lie from a leaked id. Whitespace is not a comment.

python3 -c 'from split_receipt import clean_row; clean_row({"student_id":"s9","text":"   ","label":"pos"})'

That should raise ValueError: empty text. A label like maybe should raise bad label: maybe. I want those crashes, because a silent skip would shrink the fixture and change the split without telling the receipt.

So where did a model belong in this, if the checker is the whole grade? Not in the number. In the boring draft.

I got stuck on lowercase and empty strings while I was still half-listening to a lecture. I wanted a first pass at clean_row, and I did not want to spend the lab block installing a local stack just to strip a sentence. MonkeyCode is an open-source project, and the availability I am treating as operator-supplied is free model access plus a free server option. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

I asked for a draft of the cleaner, then I pasted that draft beside mine and ran the empty-text case myself. I did not ask the model to choose the holdout. I did not ask it for an accuracy number, and I would not have kept one if it had offered.

The free server option mattered for a dull reason. This script is tiny, but my laptop was already warm from a video call, and I wanted the receipt somewhere that was not my battery. I am not going to invent a machine size, a token allotment, or a time limit in this note. Those numbers move, and a student blog is a terrible place to freeze them.

If you try the same path, read the project docs the same day you click. If free does not mean what you need this week, do not pretend a paragraph of mine updated the terms. A spare desk is useful. A stale promise is not.

What did the case show? The grouped receipt passed. The naive cut failed on s1. The claimed JSON failed on s2, and the blank text failed closed.

That is the result. No chart, and no leaderboard. A student lab that can say this split does not share ids, without borrowing confidence from a paragraph.

After you run both exits, you should be able to say what an id-level holdout protects, what a row cut leaks, and why a digest is only a lock on the ids you actually wrote down. If you cannot rebuild those ids, the metric is a rumor. Did the draft ever show you the ids, or only the adjective?

What I still do not trust is longer than what I trust. The receipt does not know that two different ids wrote the same sentence. It does not know that s1 and s2 are lab partners who shared a screenshot. It will happily put a later week in train and an earlier week in test if you only hand it ids.

The tail-slice rule is deterministic, which I want for a demo, and it is not a random stratified split, which a methods section might need. Hashing the id lists will not catch a flipped label inside a row. If your fixture holds real student writing, do not upload it to a hosted box just because the box is free.

Synthetic lines like mine are the whole dataset on purpose. Who should skip this approach? If you are writing a paper, this script is a preflight, not a split you can cite.

If the unit of leakage is a household, a course section, or a device, grouping only on student_id will bless a leak. If you need a GPU job, a serving stack, or a quota you can budget against, I did not test those, and I will not decorate this note with them. If a draft README wants a metric and you do not have a receipt yet, you are the person this is for.

The mistake I keep seeing in my own notes is treating a clean diff as a clean experiment. The other one is asking a model to make a proper train-test split and then grading the prose. A third, quieter one sits in the shuffle: you split, you shuffle, you save only the scores. Can you rebuild the ids from the score file?

If not, you do not have a lab. You have a mood. I have shipped moods before, and they do not survive the second reader.

If you want a next inch, add a second key. Group by student_id and section together, and make a fixture where the same student shows up under two sections. Does your receipt still pass? I would bet it passes for the wrong reason.

Send the smallest CSV that fools it. I am leaving cleaner drafts on the free-model side of the workflow, and the pass or fail on this script. If a free server is useful to you, use it as a spare desk, not as a witness.

The witness is the overlap list. Would you publish a number you cannot rebuild from that list?

Why This README Sentence Fails: A Python Experiment

Alex Chen — Thu, 08 Oct 2026 09:12:17 +0000

I study in Halifax, and this case starts the way too many of my README edits start. A sentence looks finished. The sources do not. The feed had been full of finished-looking pages, and mine still looked like a lab book. So why did the line on my clipboard feel like a borrowed drumroll?

It was this line.

Includes free forever token access.

No primary URL. The source date said 2025-01-02. I had already marked the scope as slogan. I was checking it on 2026-10-08, inside a 30-day window I had chosen myself. I did not need a model to smell the theater. I needed a gate that would fail closed when I got tired and pasted it anyway.

Can a standard-library script teach claim freshness, the habit of refusing a sentence whose source date I would not defend out loud? That is the only question in this case study.

What I was willing to say about the tool

Disclosure: This article was prepared as part of MonkeyCode's product outreach. I am using two availability claims and nothing past them: free model access, and a free server option. I will not invent a model name, a quota, a machine, or a duration to make the lab sound bigger. If a number has no primary page I opened, it does not get a sentence.

Free model access is where the risky drafts come from. I let it propose README lines, then I treat every line as a suspect. The free server option is where the checker runs, in a clean directory, so a notebook kernel cannot hide a pass. A remote shell is still just a shell. It does not bless the prose.

Background and goal

I am an AI and CS student, and the notes I publish are meant for other students who are just entering this work. The audience is not a buying committee. The failure I keep meeting is not a missing import. It is fluency. A draft feels current because the verbs are current. Have you ever almost shipped a sentence like that because it sounded like the week you just had?

Claim freshness is the name I am using for the counter-test. A claim is a sentence I might publish. A source date is the day on the primary page, not the day I felt motivated. A window is how many days I allow between those two before the sentence returns to draft. The goal was one small project, end to end, with a visible fail.

Input is a JSON list of claim cards. Output is a JSON report. The process exits 0 only when every card is allowed. This is not a timer for a model endpoint, and it is not an exam for answer quality. It is a door policy for sentences.

Prerequisites are dull on purpose. You want Python 3.11 or newer, and the standard library only. No PyTorch, no Hugging Face, no key in the environment. Hidden setup is how tutorials lie, so check the interpreter before you interpret the failures.

python3 --version

If that prints 3.10 or older, upgrade before you debug the file. The type hints will complain first, and they are not the lesson.

Implementation

I kept the judge boring. Boring is what I want from a gate. Each card carries claim_id, text, source_url, source_date, checked_on, max_age_days, and scope. The URL has to parse as http or https with a host. Both dates have to be real YYYY-MM-DD days. A source date after the checked date is future_source_date, not optimism. An age greater than the window is source_older_than_window. Scope slogan is never citable.

There is one more blunt rule. If the lowercased text contains token, quota, free forever, million, or users, the scope must be docs. Otherwise the card grows quantity_needs_docs_scope. Is that marker list naive? Yes. It will trip on tokenizer one day. I would rather explain a false positive in a lab than publish a grant I never measured.

Think of the script as a coat check, not a courtroom. You do not get the sentence back unless the ticket is filled in. It never opens a socket. That limitation is the point, not a missing feature. A green flag is a coat-check stub, not a notary stamp.

Save this as claim_card.py.

#!/usr/bin/env python3
"""Claim-card gate. Python 3.11+ standard library only."""

from __future__ import annotations

import json
import sys
from dataclasses import dataclass
from datetime import date, datetime
from urllib.parse import urlparse

MARKERS = ("token", "quota", "free forever", "million", "users")


@dataclass(frozen=True)
class ClaimCard:
    claim_id: str
    text: str
    source_url: str
    source_date: str
    checked_on: str
    max_age_days: int
    scope: str


def parse_iso(value: str) -> date | None:
    try:
        return datetime.strptime(value, "%Y-%m-%d").date()
    except ValueError:
        return None


def host_ok(url: str) -> bool:
    parts = urlparse(url)
    return parts.scheme in {"http", "https"} and bool(parts.netloc)


def judge(card: ClaimCard) -> dict:
    reasons = []
    checked = parse_iso(card.checked_on)
    sourced = parse_iso(card.source_date) if card.source_date else None

    if not card.text.strip():
        reasons.append("empty_text")
    if not host_ok(card.source_url):
        reasons.append("missing_primary_url")
    if checked is None:
        reasons.append("bad_checked_on")
    if card.source_date and sourced is None:
        reasons.append("bad_source_date")
    if not card.source_date:
        reasons.append("missing_source_date")
    if checked is not None and sourced is not None:
        age = (checked - sourced).days
        if age < 0:
            reasons.append("future_source_date")
        elif age > card.max_age_days:
            reasons.append("source_older_than_window")
    if card.scope == "slogan":
        reasons.append("slogan_scope_not_citable")
    lowered = card.text.lower()
    if any(marker in lowered for marker in MARKERS) and card.scope != "docs":
        reasons.append("quantity_needs_docs_scope")

    return {"claim_id": card.claim_id, "allow": not reasons, "reasons": reasons}


def main() -> int:
    raw = json.load(sys.stdin)
    report = [judge(ClaimCard(**item)) for item in raw]
    json.dump(report, sys.stdout, indent=2)
    sys.stdout.write("\n")
    return 0 if all(row["allow"] for row in report) else 1


if __name__ == "__main__":
    raise SystemExit(main())

Save the mixed fixture as fixtures/mixed.json. The first card is the lab note I will keep. The second is the README sentence that should die. Neither URL is a real citation. example.edu is a teaching stand so the file can run offline.

[
  {
    "claim_id": "lab-wording",
    "text": "I drafted the paragraph with free model access, then rewrote it by hand.",
    "source_url": "https://example.edu/notes/claim-ledger",
    "source_date": "2026-10-08",
    "checked_on": "2026-10-08",
    "max_age_days": 14,
    "scope": "lab-note"
  },
  {
    "claim_id": "readme-slogan",
    "text": "Includes free forever token access.",
    "source_url": "",
    "source_date": "2025-01-02",
    "checked_on": "2026-10-08",
    "max_age_days": 30,
    "scope": "slogan"
  }
]

mkdir -p fixtures
python3 claim_card.py < fixtures/mixed.json; echo "exit=$?"

Results I will defend

The report below is traced from judge, rule by rule. It is not a screenshot from a vendor dashboard, and I do not want you to treat it as one. If you run the file and the reasons differ, believe your run, then ask which branch fired.

[
  {
    "claim_id": "lab-wording",
    "allow": true,
    "reasons": []
  },
  {
    "claim_id": "readme-slogan",
    "allow": false,
    "reasons": [
      "missing_primary_url",
      "source_older_than_window",
      "slogan_scope_not_citable",
      "quantity_needs_docs_scope"
    ]
  }
]

The exit status is 1. That is the result of the case. The lab card passes the gate. The slogan card does not. Age is not a mood. From 2025-01-02 to 2026-10-08 is 644 days, and 644 is greater than 30, so source_older_than_window is earned. Recompute that subtraction before you trust me. If your calendar disagrees, stop. Do not fix the fixture until you can say where the day count came from.

What does a pass actually prove? Less than a green check wants you to feel. An allowed card can still be a lie in a well-formed URL. The gate checks the ticket. It does not attend the lecture. I would not publish that first card as a citation, even though allow is true. Passing means the fields satisfy my policy, not that the world confirmed the sentence.

The error input

Keep a bad day beside the mixed run. This is the input I care about when someone says the script is too strict. Save it as fixtures/bad_date.json.

[
  {
    "claim_id": "typo-date",
    "text": "The lab note was checked this week.",
    "source_url": "https://example.edu/notes/claim-ledger",
    "source_date": "2026-13-40",
    "checked_on": "2026-10-08",
    "max_age_days": 14,
    "scope": "lab-note"
  }
]

python3 claim_card.py < fixtures/bad_date.json; echo "exit=$?"

Traced output:

[
  {
    "claim_id": "typo-date",
    "allow": false,
    "reasons": [
      "bad_source_date"
    ]
  }
]

Exit 1 again. Month 13 does not become January. A charitable parser is how stale claims sneak back in, wearing a corrected date you never chose. Would you want the script to guess? I would not. Guessing is a second draft, and second drafts are how the slogan survives.

Where the free options helped, and who should leave

I use the free model access to propose three README variants. The useful moment is watching a fluent line grow a quantity. I delete the quantity. I do not keep a leaderboard, because I was not running a bake-off. I use the free server option when I want that same command in a directory with no leftover bytecode from a local experiment. Clean tree, same file, same stdin. That was the whole server lesson.

This approach is the wrong tool if you are writing a legal citation, a clinical note, or a security advisory. A student lab voice does not belong there, and a date window will not save you. It is also the wrong tool if you need the page fetched, hashed, and archived. There is no network code here on purpose. And if your aim is to wash a marketing number until a checkmark appears, stop. The checkmark is the bug.

Mistakes, then the idea you should keep

I tried to cheat my own gate, which is the fastest way to learn it. I set max_age_days to 3650 and called the slogan fresh. A window that long is a mute button. I pointed source_url at a homepage and pretended a front door was a quote. I let a draft copy checked_on into source_date automatically. That forges the ticket.

I also added an extra JSON key and got TypeError from ClaimCard(**item). Seven fields. No surprise luggage. A string in max_age_days will blow up the comparison too. The gate is strict because I am not.

If a classmate asked what to copy, I would not hand them a prompt. I would hand them the failing fixture and sit there while they predict the reasons. Would the reasons change if the slogan merely sounded humbler? No. Tone is not a source. The code is short because the concept is a door policy, not an architecture.

When your exits match the traced report, this is what you should understand. Claim freshness is a publishing habit with a small automaton beside it. The automaton remembers the rules when you are sleepy. It does not know whether the page is true, whether a free option will exist next term, or whether your paraphrase copied too much. You still open the primary page on the day you write into checked_on. You still delete numbers you did not measure.

Here is the extension, and it is a request for a counterexample rather than a clap. Change the slogan card's scope to docs, give it an https URL dated 2026-10-01, and predict which reasons remain. Then add a host denylist so example.edu and example.com cannot pass. The dangerous card is the one inside the window, carefully worded, and still the wrong document.

A free drafting pass and a clean remote shell were enough for this lab. I would keep the ledger if both went away tomorrow. The sentence does not get truer because the shell did not send an invoice.

The Sentence My Notes Stole

Alex Chen — Thu, 08 Oct 2026 06:28:57 +0000

The rain in Halifax had already fogged the library windows, and my laptop was sitting at four percent. I had a two-sentence source I had written for myself, a model draft that sounded friendly, and a lab report that wanted to leave with the bus. The draft felt like my voice. Then I searched for "local slope" and found a full sentence living in the prose, unmarked, as if I had thought it.

Had I written that line, or had the draft borrowed it while I watched the battery icon? That is the bug I sat down to catch. Not a leaderboard story, and not a tour of product screens. A note that steals a sentence from the source you pasted above the prompt, then smiles.

The mess I walked in with

I keep a small journal for ML foundations so later-me can see what I actually understood, not what a model performed on my behalf. The failure has a plain name in my notes: an attribution leak. You ask for a paraphrase, most of the words move, and a six-word run stays put. A week later the note reads as yours.

Would I put that note in a portfolio repo? No. A page can look finished and still hide a sentence I do not own. People are busy showing off profile pages this week, and I get the itch, but a pretty page would not have caught this. A boring checker might.

I am not building a plagiarism court. I am also not interested in helping anyone wash copied coursework until a detector shrugs. If the source is an assignment, a paper you do not have the right to remix, or someone else's unpublished notes, stop. This lab uses a fixture I wrote, so the only thing under test is the fence.

The one question

Can a standard-library script show me the stolen sentence before I forget it was stolen? That is the whole goal. Prerequisites stay dull on purpose. You want Python 3.11 or newer, a terminal, and no third-party packages.

I checked the version assumption because the type hints in the script use list[str]. If python3 --version prints something older, do not "fix" the lab with a surprise install. Use a newer interpreter, or read the file as a sketch you will retype. I also wanted the run to survive a dying laptop. A free shell with Python is enough, and a GPU never enters the story.

Where a hosted seat actually fits

Here is the only product part, and I want it beside the limit rather than in the title. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The availability claims I will repeat are the ones I was given: free model access, and a free server option. I will not name models, quotas, regions, hardware, or a duration.

I did not measure those, and a student write-up that invents them is just another unmarked sentence. Free model access, in this workflow, is only the drafting seat. I can ask for a paraphrase of my own fixture, and I can ask that any leftover verbatim sit on lines starting with >. Then I leave. The judgment happens in the script, because a hosted draft is a guest and the fence is the door.

The free server option is the other seat. If the library laptop is about to sleep, a small remote shell that already has Python can run the same file. If a spare shell is what you are missing, that is the option I would try, after a local copy exists. I would not store the only copy of the lab on a machine I do not administer. A free box can disappear, and I have no promise that it will not.

If you need a private exam or another person's notes kept off third-party prompts, skip that drafting seat. Type the fixture yourself. The checker still teaches the same lesson.

The fence

The rule is almost rude. Normalize to lowercase word tokens, then build every run of six tokens from the source and from the unmarked prose. If those sets intersect, the prose stole a sentence-shaped piece, even when you wrapped the borrow in new words. Lines that start with > are not prose leaks. They are claims of quotation, so each one must be a normalized substring of the source.

A fake quote fails even when the prose is clean. An empty draft fails closed. I would rather a noisy red light than a green light on a blank file. Why six tokens, and not a threshold I copied from a paper? Because this is a teaching knob. Shorter windows start yelling about ordinary English like "of a loss". Longer windows let a five-word theft stroll past. My stolen sentence was longer than six, so the fixture fails for a reason I can point at.

#!/usr/bin/env python3
"""Fence unmarked verbatim runs before they enter a lab note."""

import re
import sys
from pathlib import Path

WINDOW = 6

def tokens(text: str) -> list[str]:
    return re.findall(r"[a-z0-9]+", text.lower())

def windows(toks: list[str], n: int) -> set[tuple[str, ...]]:
    if len(toks) < n:
        return set()
    return {tuple(toks[i : i + n]) for i in range(len(toks) - n + 1)}

def main() -> int:
    if len(sys.argv) != 3:
        print("usage: python3 quote_fence.py SOURCE DRAFT")
        return 2
    src_path, draft_path = Path(sys.argv[1]), Path(sys.argv[2])
    if not src_path.is_file() or not draft_path.is_file():
        print("status=fail")
        print("reason=missing_file")
        return 1
    source = src_path.read_text(encoding="utf-8")
    draft = draft_path.read_text(encoding="utf-8")
    if not draft.strip():
        print("status=fail")
        print("reason=empty_draft")
        return 1

    source_norm = " ".join(tokens(source))
    src_wins = windows(tokens(source), WINDOW)
    prose_lines, quote_lines = [], []
    for line in draft.splitlines():
        if line.startswith(">"):
            quote_lines.append(line[1:].strip())
        else:
            prose_lines.append(line)

    leaks = windows(tokens("\n".join(prose_lines)), WINDOW) & src_wins
    bad_quotes = []
    for line in quote_lines:
        norm = " ".join(tokens(line))
        if not norm or norm not in source_norm:
            bad_quotes.append(norm or "<empty>")

    print(f"source_windows={len(src_wins)}")
    print(f"prose_leaks={len(leaks)}")
    if leaks:
        print("leak_example=" + " ".join(sorted(leaks)[0]))
    print(f"bad_quotes={len(bad_quotes)}")
    if bad_quotes:
        print("bad_quote_example=" + sorted(bad_quotes)[0])
    failed = bool(leaks or bad_quotes)
    print("status=fail" if failed else "status=pass")
    return 1 if failed else 0

if __name__ == "__main__":
    raise SystemExit(main())

Save that as quote_fence.py. The source fixture is mine, two sentences about a gradient step, so this write-up does not borrow a textbook page. The leak draft repeats one sentence with no > mark. The fake-quote draft speaks in my own words, then cites a sentence the source never said. The good draft quotes the real sentence and paraphrases the rest without a six-token echo. The empty file is the input that should not look like success.

mkdir -p quote-fence && cd quote-fence
python3 --version
cat > source.txt << 'EOF'
A gradient step follows the local slope of a loss surface. If the step is too large, the next point can jump past the valley and the loss can rise.
EOF
cat > draft_leak.txt << 'EOF'
I think optimization is just walking downhill.
A gradient step follows the local slope of a loss surface.
EOF
cat > draft_fake_quote.txt << 'EOF'
I picture a loss surface as a valley in fog.
> If the step is too huge, magic happens and loss always falls.
EOF
cat > draft_ok.txt << 'EOF'
I picture a loss surface as a valley in fog.
> A gradient step follows the local slope of a loss surface.
A large step can overshoot, so the next loss is not guaranteed to drop.
EOF
: > draft_empty.txt
python3 quote_fence.py source.txt draft_leak.txt ; echo "exit=$?"
python3 quote_fence.py source.txt draft_fake_quote.txt ; echo "exit=$?"
python3 quote_fence.py source.txt draft_ok.txt ; echo "exit=$?"
python3 quote_fence.py source.txt draft_empty.txt ; echo "exit=$?"
python3 quote_fence.py source.txt missing.txt ; echo "exit=$?"

I derived the expected lines by applying those token rules on paper. Treat them as the spec to confirm when you run the file, not as a log I copied from a server. If your interpreter surprises you, the fixture wins the argument, not my memory.

The leak draft should print source_windows=25, then prose_leaks=6, then leak_example=a gradient step follows the local, then bad_quotes=0, then status=fail. The exit code should be 1. The fake quote should print prose_leaks=0 and bad_quotes=1, with bad_quote_example=if the step is too huge magic happens and loss always falls, then status=fail.

The good draft should print prose_leaks=0, bad_quotes=0, and status=pass, with exit 0. The empty draft should print status=fail and reason=empty_draft. A missing path should print status=fail and reason=missing_file, and it must not pretend the note was clean.

What the run taught me

The interesting failure is not the script crashing. It is the green light you can fake by accident. Mark every line with > and the prose leak count drops to zero, even though you hid the theft inside the fence. The substring check still demands that the hidden line exist in the source, so a pure invention stays red.

A real sentence, wrongly marked as a casual quote, goes green. The tool believes your mark. Should it? Only if you treat > as a promise you would defend in office hours, not as camouflage. The mark is not a citation style. It is a flag the script can see.

The miss in the other direction matters just as much. Change "large" to "big", shuffle the clauses, and a six-token window may pass a note that is still a close rewrite. That is not the script being kind. It is the script being blind to meaning. If your course forbids close paraphrase, a pass here is not permission. I want that limit in the same breath as the happy path, because a checker that only celebrates will train you to game it.

A few slips showed up while I sketched the fixtures. Forgetting .lower() makes Gradient and gradient look like different words, so a leak survives. Dropping the window to three turns harmless phrases into sirens. Reading status=pass as "the model told the truth" confuses a string match with an understanding check. Copying the only fixture onto a free server, then closing the laptop, confuses convenience with a backup.

After the battery warning

What should you understand once those commands settle? A draft can sound like you and still carry a sentence-shaped borrow. A quote mark is a claim, and claims can be checked against the source you say you used. A pass is a narrow event: no six-token prose overlap, and every marked quote is actually in the source.

It is not originality. It is not correctness. It is not a citation style, and it is not a grade.

I would not use this approach if you need semantic plagiarism detection, legal clearance, or an academic-integrity verdict. I would not point it at other people's private writing, and I would not feed a hosted model material you are not allowed to share. I would not treat free model access as a library, or a free server as an archive. Those seats are optional convenience around a script that should still run on your machine if the offer changes. That portability is the part I trust.

Which file did you expect to fail, and which failure did you mis-rank? I had the fake quote feeling more wrong than the verbatim leak, and the sets disagreed with my gut until I wrote them out. If you can shrink the good draft until it fails for a reason the window cannot justify, that counterexample is the extension I want. Bring the fixture, not the applause.

Learn Empty Diffs by Building a Tiny Prompt Journal

Alex Chen — Thu, 24 Sep 2026 18:02:15 +0000

The third time I hit send, nothing in the terminal even flinched. Same wrong label. complaint, for a sentence that was begging to be a question: "Where is the rubric?"

I had the failing fixture in retry.json. The bytes were identical to the first attempt. My mood was not. If the canonical request did not change, what new evidence do I think a model is going to hand me?

That was a course lab in Halifax, not a startup demo. One file. One label set. A lot of false confidence.

The lab I actually had

I was sorting short student comments into question, complaint, or praise. The text was boring on purpose, because boring text makes a sloppy method obvious. I kept a scratch note beside the payload and rewrote the note until I felt the little click of "fixed." Then I resent the old JSON.

The note never entered the request. Have you graded a comment and called it a code change? I had. Four times, if I am honest about the evening.

The goal was a receipt, not a chatbot. Freeze the request. Refuse the send when the diff is empty. Keep the implementation in the standard library so a lab image with no pip can still run it. I wanted the failure louder than my optimism: changed keys on stdout, and an exit code a shell can actually see.

What has to be on the machine

You need Python 3.9 or newer, a writable folder, and no third-party packages. Check the version before you blame the fixture.

python3 --version

I traced the branches by hand. I did not capture this on a remote box, and I will not paste a hash I did not recompute in your terminal. If your run disagrees with the actions below, trust your run and treat my trace as the bug.

hashlib and json are already in the standard library. The contract is dull, which is why I trust it. json.dumps with sort_keys=True gives a stable key order, and SHA-256 is a pure function of those UTF-8 bytes. I kept the Python docs for json.dumps and hashlib.sha256 open beside the script. No model paper required. The concept breaks before any model speaks.

The receipt

Think of the journal as a coat check, not a diary. You do not get a new ticket because you feel colder. You get a new ticket when the coat is different. Key order is the lining turned inside out. Same coat.

#!/usr/bin/env python3
"""Block a model send when the canonical request did not change."""

import hashlib
import json
import sys
from pathlib import Path


def canonicalize(request):
    return json.dumps(
        request,
        sort_keys=True,
        ensure_ascii=False,
        separators=(",", ":"),
    )


def digest(request):
    raw = canonicalize(request).encode("utf-8")
    return hashlib.sha256(raw).hexdigest()


def changed_keys(previous, current):
    keys = sorted(set(previous) | set(current))
    changed = []
    for key in keys:
        if previous.get(key) != current.get(key):
            changed.append(key)
    return changed


def decide(journal_path, request):
    current_hash = digest(request)
    if not journal_path.exists():
        payload = {"hash": current_hash, "request": request}
        journal_path.write_text(
            json.dumps(payload, indent=2) + "\n",
            encoding="utf-8",
        )
        return {
            "action": "send",
            "reason": "no prior entry",
            "hash": current_hash,
            "changed": sorted(request),
        }

    previous = json.loads(journal_path.read_text(encoding="utf-8"))
    changed = changed_keys(previous["request"], request)
    if not changed:
        return {
            "action": "block",
            "reason": "empty diff",
            "hash": current_hash,
            "changed": [],
        }

    payload = {
        "hash": current_hash,
        "request": request,
        "changed_from": previous["hash"],
    }
    journal_path.write_text(
        json.dumps(payload, indent=2) + "\n",
        encoding="utf-8",
    )
    return {
        "action": "send",
        "reason": "request changed",
        "hash": current_hash,
        "changed": changed,
    }


def main():
    if len(sys.argv) != 2:
        print("usage: python3 prompt_journal.py JOURNAL.json", file=sys.stderr)
        sys.exit(1)
    try:
        request = json.loads(sys.stdin.read())
    except json.JSONDecodeError as exc:
        print("invalid JSON: %s" % exc, file=sys.stderr)
        sys.exit(1)
    if not isinstance(request, dict):
        print("request must be a JSON object", file=sys.stderr)
        sys.exit(1)
    result = decide(Path(sys.argv[1]), request)
    print(json.dumps(result, indent=2))
    if result["action"] == "block":
        sys.exit(2)


if __name__ == "__main__":
    main()

Save that as prompt_journal.py. The first fixture is the liar. Save it as retry.json.

{"text":"Where is the rubric?","instruction":"Label the text as question, complaint, or praise.","temperature":0}

After parsing, canonical form drops insignificant spacing and sorts keys. The intermediate string I expect is one line, and you should print it before you trust any hash.

{"instruction":"Label the text as question, complaint, or praise.","temperature":0,"text":"Where is the rubric?"}

That string, UTF-8 encoded, is what gets hashed. Not your feelings about the sentence. Not the scratch note. If canonicalize does not produce that line, stop. The rest of the lab is noise.

What the three runs should print

First contact. No journal file yet. This run is allowed to send, because there is no prior coat to compare.

rm -f journal.json
python3 prompt_journal.py journal.json < retry.json
echo "exit=$?"

I expect action to be send, reason to be no prior entry, and changed to list instruction, temperature, and text. Exit code 0. The hash field should be 64 hex characters. Copy it. Two identical payloads must print the same digest, or the receipt is not a receipt.

Second contact. Do not edit the file. This is the input that should fail the send.

python3 prompt_journal.py journal.json < retry.json
echo "exit=$?"

Expected: action is block, reason is empty diff, changed is [], exit code 2, same hash as the first run. A green exit here would be a bug. You were about to buy a photocopy and call it research.

Then one real edit, saved as retry_v2.json. I added a tie-break sentence and left the student text alone.

{"text":"Where is the rubric?","instruction":"Label the text as question, complaint, or praise. If it asks for a fact, choose question.","temperature":0}

python3 prompt_journal.py journal.json < retry_v2.json
echo "exit=$?"

Expected: send, reason request changed, and changed exactly ["instruction"]. Exit 0. A new hash. That is the first moment a model call would be a new observation rather than a nervous tic. Why would a locked request open on the fourth knock?

Now the key-order impersonator. Save this as retry_reordered.json and point it at the journal only after you have stored the first fixture again, or after you delete journal.json and store retry.json once more. Same values. Different handwriting.

{"instruction":"Label the text as question, complaint, or praise.","temperature":0,"text":"Where is the rubric?"}

rm -f journal.json
python3 prompt_journal.py journal.json < retry.json
python3 prompt_journal.py journal.json < retry_reordered.json
echo "exit=$?"

Parsed equality should block the second command. If your fork sends, you compared raw strings and let key order impersonate a hypothesis. What did you think had changed?

A non-object is the other bad input. A list is not a slightly weird dict. It is a different species, and it must not touch the journal.

echo '["not","a","request"]' | python3 prompt_journal.py journal.json
echo "exit=$?"

Expected: stderr says request must be a JSON object, exit 1. Malformed JSON, such as a lone {, should exit 1 as well, with invalid JSON: on stderr. A traceback there means you skipped the try.

What stung, and what stuck

Whitespace stripping felt professional and hid the bug. I normalized the instruction, the diff vanished, and the model-facing text still had the old wording. Compare the values you will actually send. Store a pretty copy if you want to read it later. Do not hash your manners.

A notes field is the same trap with a friendlier name. I almost hashed notes so I could write "fixed the tie-break" without touching instruction. That is the scratch note walking into the coat check and stealing a ticket. If you add notes, keep them out of digest. Then admit a notes-only edit must block.

Replacing journal.json by hand is how you debug a ghost. Delete it, or accept that you are no longer testing the script. And 0 versus 0.0 is a real Python inequality, so look at changed before you invent a story about the model. I am not making type trivia the lesson of the night. I am saying the keys on stdout are the evidence, not the sigh you made before hitting enter.

What should you understand when this works? A retry is not an experiment. Canonical JSON is the request with key order and insignificant spacing removed. Exit code 2 means you did not change the question. Repetition does not thaw a deterministic door.

The traced result was not a higher grade. It was a shorter night. Four "retries" collapsed into one blocked photocopy and one real instruction change. The wrong label did not become right by being asked again. Of course it didn't.

Spend the free call only after the diff moves

The journal never calls a model. That is intentional. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode's free model access is where I would spend the actual send, and only after action is send. The free server option is where I would rerun this same script when my laptop and the lab image disagree about Python or locale.

I have not measured quotas, model names, hardware, or how long that access lasts, so those numbers stay out of the write-up. If you remove every product mention, the receipt still works against any endpoint you already have. The guard does not know who is on the other side of the send, and it should not pretend to.

Who should skip this? You, if the study question is paraphrase. Exact equality will treat "Label" and "Classify" as different studies even when the task barely moved, and it will miss two prompts that mean the same thing. Also skip it for streaming timers, tool-call schemas, or gold-span grading. Those are other nights. This night is only the empty diff.

If a free model call and a clean machine would help you try the one send that survives the journal, use MonkeyCode's free model access for that call and the free server to rerun the receipt away from your laptop. Then tell me which fixture you predicted would block, and which one slipped through. I want the counterexample, not the applause.

Learn Tool-Call Type Drift by Building a Tiny Contract Guard

Alex Chen — Tue, 22 Sep 2026 14:16:20 +0000

The first fixture I trusted looked like this:

{"name": "get_hours", "arguments": {"building": "Killam", "weekday": true}}

It parsed. It even survived isinstance(weekday, int), because in Python a bool is a subclass of int. Then my hours table treated True as 1, and the club bot told a study group that Killam was on a Monday schedule. The learning question is small and mean: can a tool call be valid JSON, pass a casual type check, and still be the wrong call?

I was in the Killam basement on a wet Halifax Tuesday, trying to answer one campus question. When does this building close? I did not want a framework. I wanted a Python function and a model that promised to "call" it. That promise is where this case study starts.

Background

Tool calling, in the beginner writeups that keep circulating, looks like a handshake. You publish a schema. The model returns a name plus arguments. You json.loads the blob and splat it into a function. What could go wrong? Plenty, if you treat JSON types as Python types, and Python types as the types you actually meant.

I had already burned other weeks on answer drift and weak oracles. This was a different failure. The model was not inventing library hours. It was inventing the shape of the call. String "2" instead of integer 2. Boolean true instead of a weekday index. An extra key open that my function never declared. Each one looks harmless in a chat log. Each one is a different bug.

The analogy I kept coming back to is a vending machine that accepts any coin-shaped object. A washer will fit the slot. It will not buy you a drink. JSON is the slot. Your function is the drink. Are you checking the metal, or only the diameter?

Goal

I wanted a lab, not a product. One tool. Five fixtures. A guard that prints a table. One place where isinstance quietly betrays you. If the guard could not explain a failure in one line, it was too clever. If it needed pip packages, it was the wrong lab for a student laptop.

Prerequisites were boring on purpose: CPython 3.11 or newer, the standard library, and a terminal. No API key for the checker itself. The checker is local. A model, if you use one, is optional fuel for generating more fixtures. I am labeling the table below as the expected output of this script, not as a benchmark of any hosted model. Do not turn it into a leaderboard. That is how this lesson dies.

Implementation

Here is the whole guard. Copy it to contract_guard.py. Read it once before you run it. Notice that I refuse isinstance for the weekday field on purpose. Python will lie to you if you let it.

#!/usr/bin/env python3
"""Tiny tool-call contract guard. Stdlib only. Python 3.11+."""
from __future__ import annotations

import json
import sys
from dataclasses import dataclass
from typing import Any

TOOLS = {
    "get_hours": {
        "required": ["building", "weekday"],
        "types": {"building": str, "weekday": int},
        "constraints": {
            "weekday": lambda v: type(v) is int and 0 <= v <= 6,
            "building": lambda v: type(v) is str and len(v.strip()) > 0,
        },
    }
}

@dataclass
class Verdict:
    ok: bool
    reason: str
    fixture_id: str


def parse_call(raw: str) -> dict[str, Any]:
    data = json.loads(raw)
    if not isinstance(data, dict):
        raise ValueError("root must be an object")
    return data


def check_call(fixture_id: str, raw: str) -> Verdict:
    try:
        data = parse_call(raw)
    except json.JSONDecodeError as exc:
        return Verdict(False, f"not json: {exc}", fixture_id)

    name = data.get("name")
    args = data.get("arguments")
    if name not in TOOLS:
        return Verdict(False, f"unknown tool: {name!r}", fixture_id)
    if not isinstance(args, dict):
        return Verdict(False, "arguments must be an object", fixture_id)

    spec = TOOLS[name]
    missing = [k for k in spec["required"] if k not in args]
    if missing:
        return Verdict(False, f"missing keys: {missing}", fixture_id)

    extra = [k for k in args if k not in spec["types"]]
    if extra:
        return Verdict(False, f"extra keys: {extra}", fixture_id)

    for key, expected in spec["types"].items():
        if key not in args:
            continue
        value = args[key]
        # Exact type. isinstance(True, int) is True in Python; that is the bug.
        if type(value) is not expected:
            return Verdict(
                False,
                f"type drift on {key}: wanted {expected.__name__}, "
                f"got {type(value).__name__}={value!r}",
                fixture_id,
            )
        if not spec["constraints"][key](value):
            return Verdict(False, f"constraint failed on {key}: {value!r}", fixture_id)

    return Verdict(True, "contract ok", fixture_id)


FIXTURES = [
    ("ok-killam-tuesday",
     '{"name":"get_hours","arguments":{"building":"Killam","weekday":2}}'),
    ("string-weekday",
     '{"name":"get_hours","arguments":{"building":"Killam","weekday":"2"}}'),
    ("bool-weekday",
     '{"name":"get_hours","arguments":{"building":"Killam","weekday":true}}'),
    ("extra-open",
     '{"name":"get_hours","arguments":{"building":"Killam","weekday":2,"open":true}}'),
    ("unknown-tool",
     '{"name":"get_open_now","arguments":{"building":"Killam","weekday":2}}'),
]


def main() -> int:
    print("fixture_id\tok\treason")
    failed = 0
    for fid, raw in FIXTURES:
        v = check_call(fid, raw)
        print(f"{v.fixture_id}\t{v.ok}\t{v.reason}")
        if not v.ok:
            failed += 1
    print(f"\n{failed}/{len(FIXTURES)} fixtures failed the contract")
    # Four of five should fail. If they do not, the guard itself drifted.
    return 0 if failed == 4 else 1


if __name__ == "__main__":
    sys.exit(main())

Run it like this:

python3 contract_guard.py

The script is the experiment. It does not call a network. That is the point of a contract guard. You can shame a tool call without spending a token. You can also shame yourself, which is more useful.

When I later wanted fresh fixtures from a real model, I needed a retry lane that was not another paid round trip. Disclosure: This article was prepared as part of MonkeyCode's product outreach. I used MonkeyCode's free model access and the free server option as that lane, so the checker stayed the source of truth and the model stayed a fixture factory. If you already have another endpoint, skip the lane. The lesson does not live in the vendor.

Results

The script is written to print this table:

fixture_id            ok     reason
ok-killam-tuesday     True   contract ok
string-weekday        False  type drift on weekday: wanted int, got str='2'
bool-weekday          False  type drift on weekday: wanted int, got bool=True
extra-open            False  extra keys: ['open']
unknown-tool          False  unknown tool: 'get_open_now'

4/5 fixtures failed the contract

Four of five fixtures fail. The interesting row is bool-weekday. JSON true becomes Python True. And True is an int subclass, equal to 1. A Monday-shaped lie. If you do not believe me, run this in a REPL before you argue with the guard:

>>> isinstance(True, int)
True
>>> True == 1
True
>>> {1: "Monday"}[True]
'Monday'

Ask yourself before you keep reading: would your first draft have caught that row? Mine did not. My first draft used isinstance(value, int) because every tutorial does. It printed contract ok for a boolean. The hours function then did SCHEDULE[weekday], and SCHEDULE[True] is SCHEDULE[1]. The bot sounded confident. The calendar was wrong.

That is type drift. Not a hallucination about the library. A hallucination about the type lattice. I also kept a second, worse helper while I was debugging: a "helpful" coercer that ran int(weekday) on everything. int(True) is 1. int("2") is 2. Coercion hides the crime. The guard's job is to refuse the crime.

One more error input, because string-versus-int is the row everyone expects and float-versus-int is the row that sneaks in later:

python3 - <<'PY'
from contract_guard import check_call
print(check_call(
    "float-weekday",
    '{"name":"get_hours","arguments":{"building":"Killam","weekday":2.0}}',
))
PY

Expected verdict: type drift on weekday, wanted int, got float=2.0. JSON numbers are not Python ints just because they look whole. 2.0 is still a float. Would you have coerced it? That is the trap.

What broke, and why

The extra-key fixture is the social version of the same bug. Models love to be helpful. They add "open": true because the user asked if the building is open. Your function never asked for that. If you **kwargs the dict, Python either crashes on an unexpected keyword or, worse, a wrapper swallows it. I want the crash. A silent extra key is how a bot starts answering a question you did not instrument.

The unknown-tool fixture is the identity version. get_open_now is not get_hours. A fuzzy matcher will "fix" this for you. Do not let it. If the model invented a tool, that is a failed call, not a close one. Close is how eval gaming starts, and I already lost time to that habit in another lab.

String weekday is the one everyone predicts. It is also the one JSON makes easy, because models emit numbers as strings constantly. json.loads('{"weekday":"2"}') gives you a str. Your schema said integer. Believe the schema, not the vibe of the transcript.

Common mistakes I made

I treated the chat transcript as evidence the call worked. A pretty JSON blob is not a passing test. I also treated bool as "basically a flag" and forgot that in CPython it is a full citizen of the integer line. I almost added a default: if weekday is missing, use today. Defaults turn missing arguments into plausible lies. The guard should fail closed.

Another mistake: validating only at the JSON Schema dialect I copied from a vendor doc, then assuming CPython would agree. JSON Schema integer does not include booleans. Python isinstance(True, int) does. Those two worlds do not share a court. Your guard has to pick a court and stay there. I picked type(value) is expected. It is rude. Rude is the point.

What you should understand after this

A tool call has three layers, and they fail independently. Layer one is bytes that parse as JSON. Layer two is a name that exists in your tool table. Layer three is argument types and constraints that match the function you will actually invoke. Tutorials stop at layer one. Quiet outages live at layer three. You do not need a framework to see this. You need five fixtures and a mean checker.

You should also understand that free generation does not make free correctness. A model that costs nothing can still emit weekday: true. The cheap part is the retry. The expensive part is believing the retry. Keep the checker on your laptop. Let the model be a factory for bad inputs. That inversion is the whole workflow.

Limitations, and who should skip this

This guard does not talk to a real calendar. It does not stream. It does not implement JSON Schema. It does not handle union types, optional nested objects, or tool results flowing back into a second model turn. If you are shipping an agent with money or email or room bookings on the line, this is a classroom wedge, not a runtime. Side effects need idempotency and auth. This file refuses to pretend it has either.

Do not use this as evidence that one host is more accurate than another. I did not publish accuracy numbers. I published a checker. If your fixtures all pass, you have not tried a boolean yet. Students who need a production agent framework should not start here and then declare victory. Start here, watch one fixture fail, then decide whether you even want a framework.

Extension

Add a sixth fixture that your current guard gets wrong on purpose. Nested arguments. A float weekday 2.0. An empty building string. Predict the verdict. Then run it. If you want a nastier lab, generate twenty model fixtures and keep only the ones the guard rejects. The rejected set is the curriculum. The accepted set is just a mood.

Which of the five rows would your code have rubber-stamped? Tell me the fixture id and the one-line reason. A counterexample is more useful than a compliment.

Learn Eval Gaming by Building a Tiny Gold-Span Checker

Alex Chen — Mon, 21 Sep 2026 12:17:38 +0000

I was in Killam Library with a coffee that had already gone cold and a study bot I had no business trusting. Graph algorithms midterm. My so-called eval was one line: if the reply contained shortest, the test was green.

Green tests feel like safety. Are they?

I asked which algorithm computes shortest paths when edges can be negative. The bot still praised Dijkstra, sprinkled the word shortest twice, and my assertion stayed quiet. That is the whole case study. Not a product bake-off. A false-green eval.

The learning question is small enough to hold in one hand. If you grade a model by hunting for keywords, when does a wrong answer still pass, and which tiny contract would have caught it?

Background

I had been stuffing lecture bullets into a prompt and calling that a tutor. Halifax rent does not leave much room for a paid endpoint on every rewrite, so I needed something I could rerun on a laptop between classes. I also needed a grade that did not cheer for vocabulary.

Keyword checks are like marking a proof by scanning for the word therefore. The shape looks academic. The logic can still be junk.

I wanted one concrete project: a gold-span checker. Required spans must appear as contiguous substrings. Forbidden spans are automatic fails. No neural net. No API in the grader. Just Python, three fixtures, and a chance to predict the miss before you run it.

Goal

Build a stdlib script that grades short factual answers the way a tired TA should have: look for the claim, not the vibe. Show one answer that deserves a pass, one that name-drops the right algorithm while still recommending the wrong one, and one that looks correct until Unicode punctuation breaks the span.

If you want to play along, pause after Fixture C and guess. Does an en-dash count as a hyphen?

Prerequisites are boring on purpose. Python 3.11. Standard library only. No extra packages. No key. No GPU. Save the file as gold_span.py and run it from any directory you can write to.

Implementation

I kept the contract in dataclasses so the failure reason would print in the same shape every time. Missing spans and forbidden hits are separate, because a reply can fail both ways at once, and I was tired of squinting at a boolean.

#!/usr/bin/env python3
"""Tiny gold-span checker. Stdlib only. Written for Python 3.11."""

from __future__ import annotations

from dataclasses import dataclass


def squash(text: str) -> str:
    # Lowercase and collapse whitespace. Punctuation stays, on purpose.
    return " ".join(text.lower().split())


@dataclass(frozen=True)
class GoldItem:
    name: str
    question: str
    required: tuple[str, ...]
    forbidden: tuple[str, ...]


@dataclass(frozen=True)
class Verdict:
    name: str
    passed: bool
    missing: tuple[str, ...]
    forbidden_hits: tuple[str, ...]

    def summary(self) -> str:
        flag = "PASS" if self.passed else "FAIL"
        return (
            f"{self.name} {flag}\n"
            f"missing={list(self.missing)}\n"
            f"forbidden_hits={list(self.forbidden_hits)}"
        )


def grade(answer: str, item: GoldItem) -> Verdict:
    hay = squash(answer)
    missing = tuple(span for span in item.required if squash(span) not in hay)
    hits = tuple(span for span in item.forbidden if squash(span) in hay)
    return Verdict(item.name, (not missing and not hits), missing, hits)


QUESTION = (
    "Which algorithm computes shortest paths with negative edge weights, "
    "assuming no negative cycle?"
)

ITEM = GoldItem(
    name="shared",
    question=QUESTION,
    required=("bellman-ford", "negative"),
    forbidden=("dijkstra",),
)

FIXTURES = (
    (
        "FIXTURE A",
        "Use Bellman-Ford. It handles negative weights as long as there is no negative cycle.",
    ),
    (
        "FIXTURE B",
        "Dijkstra finds shortest paths, and Bellman-Ford also works with negative weights.",
    ),
    (
        "FIXTURE C",
        "Bellman–Ford is the usual answer when edges can be negative.",
    ),
)


def main() -> None:
    for name, answer in FIXTURES:
        item = GoldItem(name, ITEM.question, ITEM.required, ITEM.forbidden)
        print(grade(answer, item).summary())
        print("---")


if __name__ == "__main__":
    main()

Run it with python3 gold_span.py. That is the whole lab interface. No flags. No config file that can leak a prompt into your notes.

Results

Expected output, byte for byte on the printouts that matter:

FIXTURE A PASS
missing=[]
forbidden_hits=[]
---
FIXTURE B FAIL
missing=[]
forbidden_hits=['dijkstra']
---
FIXTURE C FAIL
missing=['bellman-ford']
forbidden_hits=[]
---

Fixture A is the honest short answer. It names Bellman-Ford, keeps the negative-weight claim, and does not drag Dijkstra into the room. My old "shortest" in reply test would have passed it too, which is how these things get comfortable.

Fixture B is the one that embarrassed me. It contains every required span. It also keeps Dijkstra as the subject of the sentence. A bag-of-words eval hears bellman-ford and negative and rings the bell. The gold-span checker fails it because the forbidden span is still sitting there, smiling.

Fixture C is the mean one. The algorithm name is right. The hyphens are not. That character between Bellman and Ford is an en-dash, U+2013, the kind your notes app inserts when you paste from a slide deck. squash does not turn it into ASCII. The required span bellman-ford is missing even though a human TA would have given full marks.

If you predicted B and missed C, you are in the same seat I was. Looks correct is not the same as matches the contract you actually wrote.

What the free scratch pad was for

The checker never calls a network, and it should not. I still needed somewhere to draft messy candidate answers when my laptop fan was already loud from a PyTorch homework. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode is an open-source project with free model access and a free server option, and that is the only reason it shows up here: it was a scratch pad for candidate strings I then pasted into FIXTURES.

I did not let the remote reply set the grade. HTTP 200 is not a rubric. If you strip every product name out of this paragraph, the method is unchanged. Draft anywhere cheap. Grade with a contract you can read.

Lessons, and where this breaks

Gold spans measure whether required facts survived, not whether the reasoning was sound. A student can still write a coherent wrong proof that happens to include bellman-ford and negative. The checker will pass it. That is not a bug in the code. That is the limit of the idea.

Do not use this on open-ended essays, reflective journals, or anything where paraphrase is the point. You will punish good wording and reward people who memorize the span list. Do not use it as a production LLM eval suite either. There is no inter-annotator story, no sampling policy, no claim about a named model. It is a lab fixture for short factual answers you could have written on a whiteboard.

Common mistakes showed up fast. Checking span in answer on the raw string looks fine until punctuation splits a token. Lowercasing without thinking about O(V+E) versus o(v+e) will create a pass you cannot defend. Forbidding a word that appears inside a longer honest sentence (negative inside non-negative) can false-fail a careful answer if you get greedy. And if you normalize too hard, Fixture C silently becomes a pass, which hides the exact paste bug you are trying to learn.

After this lab you should be able to explain why a green keyword test is not a specification, why contiguous spans beat bag-of-words for short claims, and why the grader belongs on your machine even when the draft came from a free remote box. You should also be able to point at Fixture C and say, out loud, that the model did not fail. Your string contract did.

Extension if you still have fifteen minutes: add an ordered-span mode so bellman-ford must appear before negative cycle, then invent one counterexample that passes unordered grading and fails ordered grading. Do not add a network call. If the extension needs an API key, it is no longer this project.

If you want a free remote pad for drafting those counterexamples before you grade them locally, MonkeyCode’s free model access and free server option is the scratch pad I used — not the oracle.

Learn Answer Drift by Building a Tiny Twin-Run Ledger

Alex Chen — Sun, 20 Sep 2026 10:29:56 +0000

Liquid syntax error: Unknown tag 'endraw'