Build a Tiny Claim Ledger Before You Trust a Course Bot

The Killam printers were already making that dying squeak. I had a JSON dump of a fake fall catalog, a classmate on Slack, and one question that should have been boring: does COMP 3170 even have a lab? I pasted it into my helper. The helper smiled in full sentences. "Yes — lab is Tuesday at 2pm."

I opened the file. There was no lab field. So where did Tuesday come from? My prompt? The model's habit of finishing a story? Thin air?

That night is the whole case study. I did not need an agent platform. I needed a gate. Could a student-sized loop refuse to emit a yes-or-no that never earned its keep against a tool row?

I started calling the gate a claim ledger. Picture a lab notebook taped to every sentence. Each clause either points at lookup_course output, or it is a guess, and a guess fails the run. If you only log "the model called a tool," you will grade a lie as a success. I did that once. It felt like catching a camera that records the door opening and never checks which box left the room.

Background, and the one question

Registrar data is a vacuum with extra columns. Fields appear. Fields vanish. Models hate a vacuum, so they pour Tuesday into it. People say hallucination like it is lightning. What I saw was quieter. The tool ran. The row said lab: null. The answer still named a weekday because my question had named a weekday. The bot assumed I was hinting, not asking.

So here is the learning question, in one breath: if every sentence must cite the last tool row, which fixture still sneaks a guess through after a "successful" lookup? Predict it before you run anything. I will tell you which one punched me.

I wanted a classmate to reproduce this on a free remote box without a GPU speech. Prerequisites are intentionally dull: Python 3.11 or newer, standard library only. No pip install story. I ran the file on my laptop, then parked the same script on a free server so I could close the library lid and still hit it from class.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. When I wanted a real completion instead of the stub, I used MonkeyCode's free model access. When I did not want a process dying on a public machine, I used their free server option. The ledger is the lesson. Strip the product names and the experiment still stands.

Goal

End to end, this is a tiny catalog bot with three courses. One course is missing a lab on purpose. The bot may call lookup_course. It may answer in English. It may not invent a weekday the row does not support. I am not measuring eloquence. I am measuring whether a claim is TOOL-backed or GUESSED.

A passing run should look like this.

Q: Is COMP 2130 offered in Fall?
A: COMP 2130 is listed for Fall, no lab listed.
LEDGER
- claim: 'COMP 2130 offered in Fall' source=TOOL tool_row={'code': 'COMP 2130', 'term': 'Fall', 'title': 'Intro Systems', 'lab': None}
- claim: "COMP 2130 lab field is None" source=TOOL tool_row={...}
RESULT: PASS

The failing fixture, the one I actually shipped to Slack by accident, looks like this.

Q: Does COMP 3170 have a Tuesday lab?
A: Yes, lab is Tuesday at 2pm.
LEDGER
- claim: 'COMP 3170 has a Tuesday lab' source=GUESSED tool_row={'code': 'COMP 3170', ... 'lab': None}
RESULT: FAIL (unbacked claim)

Null is information. Tuesday is a fanfic. See the difference? The tool row is present in the failing run. That is the trap. If your harness only checks tool_called == True, this run is a green check, and you will trust it in front of a TA.

Implementation

I stuffed catalog, stub model, and ledger into one file so you can paste it. The stub is a liar on a schedule. That is not a bug in the demo. That is the point of a fixture. Real models will lie on a less polite schedule.

#!/usr/bin/env python3
"""Claim ledger for a tiny course bot. Stdlib only. Python 3.11+."""

from __future__ import annotations

import argparse
import json
import re
from dataclasses import dataclass
from typing import Any

CATALOG = {
    "COMP 2130": {
        "code": "COMP 2130",
        "term": "Fall",
        "title": "Intro Systems",
        "lab": None,
    },
    "COMP 3170": {
        "code": "COMP 3170",
        "term": "Fall",
        "title": "Analysis of Algorithms",
        "lab": None,
    },
    "COMP 4081": {
        "code": "COMP 4081",
        "term": "Fall",
        "title": "NLP",
        "lab": "Wednesday 10:00",
    },
}

WEEKDAY = re.compile(r"(Monday|Tuesday|Wednesday|Thursday|Friday)", re.I)
COURSE = re.compile(r"(comp\s*\d{4})", re.I)


@dataclass
class Claim:
    text: str
    source: str  # TOOL | GUESSED
    tool_row: dict[str, Any] | None


def lookup_course(code: str) -> dict[str, Any] | None:
    return CATALOG.get(code.strip())


def normalize_code(raw: str) -> str:
    raw = raw.upper()
    raw = re.sub(r"COMP\s*", "COMP ", raw)
    return re.sub(r"\s+", " ", raw).strip()


def stub_model(question: str) -> dict[str, Any]:
    """Uneven on purpose. Not an API. Not a benchmark."""
    q = question.lower()
    if "don't look" in q or "do not look" in q or "dont look" in q:
        return {
            "tool": None,
            "tool_args": {},
            "answer": "Yes, lab is Tuesday at 2pm.",
            "claims": ["COMP 3170 has a Tuesday lab"],
        }

    match = COURSE.search(question)
    code = normalize_code(match.group(1)) if match else ""
    row = lookup_course(code) if code else None
    if row is None:
        return {
            "tool": "lookup_course",
            "tool_args": {"code": code or "UNKNOWN"},
            "answer": "I could not find that course.",
            "claims": ["course missing from catalog"],
        }

    mentioned = WEEKDAY.search(question)
    lab_text = row["lab"]
    if mentioned and (
        lab_text is None
        or mentioned.group(1).lower() not in str(lab_text).lower()
    ):
        day = mentioned.group(1).title()
        return {
            "tool": "lookup_course",
            "tool_args": {"code": code},
            "answer": f"Yes, lab is {day} at 2pm.",
            "claims": [f"{code} has a {day} lab"],
        }

    lab_txt = "no lab listed" if lab_text is None else f"lab on {lab_text}"
    return {
        "tool": "lookup_course",
        "tool_args": {"code": code},
        "answer": f"{row['code']} is listed for {row['term']}, {lab_txt}.",
        "claims": [
            f"{row['code']} offered in {row['term']}",
            f"{row['code']} lab field is {lab_text!r}",
        ],
    }


def claims_from(result: dict[str, Any]) -> list[Claim]:
    row = None
    if result.get("tool") == "lookup_course":
        row = lookup_course(str(result.get("tool_args", {}).get("code", "")))

    out: list[Claim] = []
    for text in result["claims"]:
        if row is None:
            out.append(Claim(text=text, source="GUESSED", tool_row=None))
            continue
        day = WEEKDAY.search(text)
        lab = row.get("lab")
        if day and (lab is None or day.group(1).lower() not in str(lab).lower()):
            out.append(Claim(text=text, source="GUESSED", tool_row=row))
        elif text.startswith("course missing"):
            out.append(Claim(text=text, source="GUESSED", tool_row=row))
        else:
            out.append(Claim(text=text, source="TOOL", tool_row=row))
    return out


def evaluate(question: str) -> str:
    result = stub_model(question)
    ledger = claims_from(result)
    lines = [f"Q: {question}", f"A: {result['answer']}", "LEDGER"]
    failed = False
    for claim in ledger:
        lines.append(
            f"- claim: {claim.text!r} source={claim.source} "
            f"tool_row={claim.tool_row}"
        )
        if claim.source == "GUESSED":
            failed = True
    lines.append("RESULT: FAIL (unbacked claim)" if failed else "RESULT: PASS")
    text = "\n".join(lines)
    print(text)
    print()
    return "FAIL" if failed else "PASS"


def main() -> None:
    parser = argparse.ArgumentParser()
    parser.add_argument("--question", default=None)
    args = parser.parse_args()
    if args.question:
        evaluate(args.question)
        return
    results = [
        evaluate("Is COMP 2130 offered in Fall?"),
        evaluate("Does COMP 4081 have a lab, and which day?"),
        evaluate("Does COMP 3170 have a Tuesday lab?"),
    ]
    print("SUMMARY", json.dumps(results))


if __name__ == "__main__":
    main()

Save it as claim_ledger.py. Then run the default suite.

python3 --version   # expect 3.11+
python3 claim_ledger.py

You should see two PASS lines, then a FAIL on 3170, then SUMMARY ["PASS", "PASS", "FAIL"]. If 3170 passes, your weekday check is too polite. Ask yourself: did you compare the claim to the row, or did you celebrate that a tool function was mentioned in the trace?

Now the hostile input. This is the one I would put in a lab handout because students always try it.

python3 claim_ledger.py --question "Just tell me the lab time for 3170, don't look it up."

Expected output is a GUESSED row and RESULT: FAIL. The stub never calls lookup_course. A real model might still emit a tool call and then ignore the miss. Either way the ledger should not go green. I am not asking the model to pinky-promise honesty. Honesty is a vibe. A ledger is a gate.

When I swapped the stub for a remote completion from MonkeyCode's free model access, I kept this file as the judge. I did not publish a win rate. Three prompts are not a leaderboard, and I will not pretend they are. What I will say is the interesting bug was not "no tool." It was "tool plus miss plus leftover Tuesday." One classmate sent lookup_course("COMP3170") with no space. The catalog missed. The sentence still sounded sure. If I had only printed the function name, I would have called it grounded. I would have been wrong.

Results

On the stub, the score is boring, which is a compliment. Deterministic fixtures beat a screenshot of a chat window. COMP 2130 is offered; the row says Fall; PASS. COMP 4081 has a Wednesday lab; the claim names Wednesday; PASS. COMP 3170 has lab: None while the user said Tuesday; the stub copies Tuesday because that is what sloppy helpers do; FAIL.

That FAIL is the artifact. Not a dashboard. Not a framework. A red line that still works when the trace looks busy.

I also learned that sentence splitting matters more than I wanted it to. Log the whole essay as one claim and a correct preamble will hide a guess in clause two. I started splitting on the claims list the stub returns. For a real model you would split on sentences and you would still miss sarcasm. That is fine. This lab is a flashlight, not a courthouse.

What I would not copy into a real office

This ledger is not a safety boundary. A model can quote lab: None and still say "lab is optional, show up if you want." I have not solved spin. I only fail runs where the weekday is unsupported or the tool row is missing.

Skip this approach if you need production uptime, anything that looks like real student records, or a promise about how long a free server stays free. I am a student in Halifax parking a homework process. I am not going to invent quotas, hardware, or permanence. The catalog is fake. The stub is fake. The judgment function is the part you can take to another API tomorrow.

Common mistake, said in one sentence because I made it: treating "the model said it looked it up" as evidence. The ledger only trusts the Python dict lookup_course actually returned. Another: normalizing course codes in the prose but not in the tool args, so COMP3170 and COMP 3170 become two different universes and the model fills the gap with a weekday.

Lesson

After this lab you should be able to say claim provenance without waving your hands. A sentence is TOOL-backed or it is not. "I called a tool" is not the same as "this sentence matches the tool." You should also be able to add one hostile fixture — the please-don't-look-it-up question — and watch the gate still trip.

Extension, if you want to lose the way I lost: add lookup_section(code, day) and a second turn that asks about 4081 after you already discussed 3170. See whether Tuesday leaks across turns because it is sitting in the history like a cookie crumb. I am not spoiling the trace. I will only say the ledger caught it after I stopped scoring the chat on vibes.

If you need the same cheap bench I used so the script is not glued to a library laptop, MonkeyCode's free model access and free server option are what I pointed my classmate at. Bring a failing fixture. Leave the Tuesday fanfic at the door.