I was in Killam Library with a coffee that had already gone cold and a study bot I had no business trusting. Graph algorithms midterm. My so-called eval was one line: if the reply contained shortest, the test was green.
Green tests feel like safety. Are they?
I asked which algorithm computes shortest paths when edges can be negative. The bot still praised Dijkstra, sprinkled the word shortest twice, and my assertion stayed quiet. That is the whole case study. Not a product bake-off. A false-green eval.
The learning question is small enough to hold in one hand. If you grade a model by hunting for keywords, when does a wrong answer still pass, and which tiny contract would have caught it?
Background
I had been stuffing lecture bullets into a prompt and calling that a tutor. Halifax rent does not leave much room for a paid endpoint on every rewrite, so I needed something I could rerun on a laptop between classes. I also needed a grade that did not cheer for vocabulary.
Keyword checks are like marking a proof by scanning for the word therefore. The shape looks academic. The logic can still be junk.
I wanted one concrete project: a gold-span checker. Required spans must appear as contiguous substrings. Forbidden spans are automatic fails. No neural net. No API in the grader. Just Python, three fixtures, and a chance to predict the miss before you run it.
Goal
Build a stdlib script that grades short factual answers the way a tired TA should have: look for the claim, not the vibe. Show one answer that deserves a pass, one that name-drops the right algorithm while still recommending the wrong one, and one that looks correct until Unicode punctuation breaks the span.
If you want to play along, pause after Fixture C and guess. Does an en-dash count as a hyphen?
Prerequisites are boring on purpose. Python 3.11. Standard library only. No extra packages. No key. No GPU. Save the file as gold_span.py and run it from any directory you can write to.
Implementation
I kept the contract in dataclasses so the failure reason would print in the same shape every time. Missing spans and forbidden hits are separate, because a reply can fail both ways at once, and I was tired of squinting at a boolean.
#!/usr/bin/env python3
"""Tiny gold-span checker. Stdlib only. Written for Python 3.11."""
from __future__ import annotations
from dataclasses import dataclass
def squash(text: str) -> str:
# Lowercase and collapse whitespace. Punctuation stays, on purpose.
return " ".join(text.lower().split())
@dataclass(frozen=True)
class GoldItem:
name: str
question: str
required: tuple[str, ...]
forbidden: tuple[str, ...]
@dataclass(frozen=True)
class Verdict:
name: str
passed: bool
missing: tuple[str, ...]
forbidden_hits: tuple[str, ...]
def summary(self) -> str:
flag = "PASS" if self.passed else "FAIL"
return (
f"{self.name} {flag}\n"
f"missing={list(self.missing)}\n"
f"forbidden_hits={list(self.forbidden_hits)}"
)
def grade(answer: str, item: GoldItem) -> Verdict:
hay = squash(answer)
missing = tuple(span for span in item.required if squash(span) not in hay)
hits = tuple(span for span in item.forbidden if squash(span) in hay)
return Verdict(item.name, (not missing and not hits), missing, hits)
QUESTION = (
"Which algorithm computes shortest paths with negative edge weights, "
"assuming no negative cycle?"
)
ITEM = GoldItem(
name="shared",
question=QUESTION,
required=("bellman-ford", "negative"),
forbidden=("dijkstra",),
)
FIXTURES = (
(
"FIXTURE A",
"Use Bellman-Ford. It handles negative weights as long as there is no negative cycle.",
),
(
"FIXTURE B",
"Dijkstra finds shortest paths, and Bellman-Ford also works with negative weights.",
),
(
"FIXTURE C",
"Bellman–Ford is the usual answer when edges can be negative.",
),
)
def main() -> None:
for name, answer in FIXTURES:
item = GoldItem(name, ITEM.question, ITEM.required, ITEM.forbidden)
print(grade(answer, item).summary())
print("---")
if __name__ == "__main__":
main()
Run it with python3 gold_span.py. That is the whole lab interface. No flags. No config file that can leak a prompt into your notes.
Results
Expected output, byte for byte on the printouts that matter:
FIXTURE A PASS
missing=[]
forbidden_hits=[]
---
FIXTURE B FAIL
missing=[]
forbidden_hits=['dijkstra']
---
FIXTURE C FAIL
missing=['bellman-ford']
forbidden_hits=[]
---
Fixture A is the honest short answer. It names Bellman-Ford, keeps the negative-weight claim, and does not drag Dijkstra into the room. My old "shortest" in reply test would have passed it too, which is how these things get comfortable.
Fixture B is the one that embarrassed me. It contains every required span. It also keeps Dijkstra as the subject of the sentence. A bag-of-words eval hears bellman-ford and negative and rings the bell. The gold-span checker fails it because the forbidden span is still sitting there, smiling.
Fixture C is the mean one. The algorithm name is right. The hyphens are not. That character between Bellman and Ford is an en-dash, U+2013, the kind your notes app inserts when you paste from a slide deck. squash does not turn it into ASCII. The required span bellman-ford is missing even though a human TA would have given full marks.
If you predicted B and missed C, you are in the same seat I was. Looks correct is not the same as matches the contract you actually wrote.
What the free scratch pad was for
The checker never calls a network, and it should not. I still needed somewhere to draft messy candidate answers when my laptop fan was already loud from a PyTorch homework. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode is an open-source project with free model access and a free server option, and that is the only reason it shows up here: it was a scratch pad for candidate strings I then pasted into FIXTURES.
I did not let the remote reply set the grade. HTTP 200 is not a rubric. If you strip every product name out of this paragraph, the method is unchanged. Draft anywhere cheap. Grade with a contract you can read.
Lessons, and where this breaks
Gold spans measure whether required facts survived, not whether the reasoning was sound. A student can still write a coherent wrong proof that happens to include bellman-ford and negative. The checker will pass it. That is not a bug in the code. That is the limit of the idea.
Do not use this on open-ended essays, reflective journals, or anything where paraphrase is the point. You will punish good wording and reward people who memorize the span list. Do not use it as a production LLM eval suite either. There is no inter-annotator story, no sampling policy, no claim about a named model. It is a lab fixture for short factual answers you could have written on a whiteboard.
Common mistakes showed up fast. Checking span in answer on the raw string looks fine until punctuation splits a token. Lowercasing without thinking about O(V+E) versus o(v+e) will create a pass you cannot defend. Forbidding a word that appears inside a longer honest sentence (negative inside non-negative) can false-fail a careful answer if you get greedy. And if you normalize too hard, Fixture C silently becomes a pass, which hides the exact paste bug you are trying to learn.
After this lab you should be able to explain why a green keyword test is not a specification, why contiguous spans beat bag-of-words for short claims, and why the grader belongs on your machine even when the draft came from a free remote box. You should also be able to point at Fixture C and say, out loud, that the model did not fail. Your string contract did.
Extension if you still have fifteen minutes: add an ordered-span mode so bellman-ford must appear before negative cycle, then invent one counterexample that passes unordered grading and fails ordered grading. Do not add a network call. If the extension needs an API key, it is no longer this project.
If you want a free remote pad for drafting those counterexamples before you grade them locally, MonkeyCode’s free model access and free server option is the scratch pad I used — not the oracle.
Top comments (0)