My Study Bot Invented Citations. A 30-Line Ledger Caught Them.

It was 11:40 PM, two days before my machine learning midterm. My flashcard bot had just served me a question about the dropout paper, and the answer looked great. Confident, even. It ended with a citation: "as stated in section 4.2." I opened the PDF to check, because that's the habit you develop after being burned once. Section 4.2 was about data augmentation. Nothing about dropout was there.

The bot wasn't lying on purpose. It was doing exactly what a free language model does when you ask it to be helpful: it filled a gap with something plausible. The unsettling part wasn't the mistake. It was that I almost studied from it. I would have walked into the exam believing section 4.2 said something it never said.

So I turned the problem into a case study. Background: I'm an AI/CS student in Halifax, and I've spent the semester testing how far free infrastructure can carry a real study workflow. The goal was specific — build a quiz generator that reads my course notes, produces one question at a time, runs on a free server, and leaves a paper trail I can audit. If the model can't show me the exact line its answer is based on, I don't want the answer.

The whole implementation is two Python files and a JSONL file. No framework, no vector database, no agent loop. The first script reads a source text, asks the model for one question in strict JSON, and appends the result to ledger.jsonl. The second script reads the ledger and checks every excerpt against the original source.

Prerequisites: Python 3.9+, an OpenAI-compatible endpoint, and a plain-text version of your reading. Set three environment variables and you're ready:

export API_URL="https://your-endpoint/v1/chat/completions"
export API_KEY="your-key"
export MODEL="the-free-model-name"

Here is quiz_bot.py:

# quiz_bot.py — generate one quiz question and append it to the ledger
import json, os, sys, urllib.request
from datetime import datetime, timezone

API_URL = os.environ["API_URL"]   # OpenAI-compatible endpoint
API_KEY = os.environ["API_KEY"]
MODEL = os.environ["MODEL"]       # whatever free model the provider exposes

def call_model(prompt: str) -> str:
    body = json.dumps({
        "model": MODEL,
        "messages": [{"role": "user", "content": prompt}],
        "temperature": 0.4,
    }).encode()
    req = urllib.request.Request(API_URL, data=body, headers={
        "Authorization": f"Bearer {API_KEY}",
        "Content-Type": "application/json",
    })
    with urllib.request.urlopen(req, timeout=60) as resp:
        return json.load(resp)["choices"][0]["message"]["content"]

def extract_json(text: str) -> dict:
    start, end = text.find("{"), text.rfind("}")
    if start == -1 or end == -1:
        raise ValueError("no JSON object in response")
    return json.loads(text[start:end + 1])

def main():
    source = open(sys.argv[1]).read()[:4000]
    prompt = f"""Read the source text below. Create one quiz question.
Return ONLY JSON with keys:
question, answer, source_excerpt (an exact quote from the source), reasoning.

SOURCE:
{source}"""
    try:
        parsed = extract_json(call_model(prompt))
    except (json.JSONDecodeError, ValueError) as e:
        print(f"Skipped: model did not return JSON ({e})")
        sys.exit(1)
    entry = {
        "ts": datetime.now(timezone.utc).isoformat(),
        "source_file": sys.argv[1],
        **parsed,
    }
    with open("ledger.jsonl", "a") as f:
        f.write(json.dumps(entry) + "\n")
    print(json.dumps(entry, indent=2))

if __name__ == "__main__":
    main()

The prompt asks for four fields: the question, the answer, a source_excerpt — an exact quote from the reading — and the model's reasoning for why that question matters. The excerpt field is the whole point. It turns a vague "trust me" into a specific "check this line."

Now the verifier, verify_ledger.py:

# verify_ledger.py — flag ledger entries whose excerpt is not in the source
import json, sys

def norm(s: str) -> str:
    return " ".join(s.lower().split())

failed = 0
for line in open("ledger.jsonl"):
    entry = json.loads(line)
    source = norm(open(entry["source_file"]).read())
    excerpt = norm(entry["source_excerpt"])
    ok = excerpt in source
    failed += 0 if ok else 1
    print(f"{'OK  ' if ok else 'FAIL'} {entry['ts']} — {entry['question'][:60]}")
    if not ok:
        print(f"     phantom excerpt: {entry['source_excerpt'][:80]!r}")

print(f"\n{failed} phantom citation(s) found")
sys.exit(1 if failed else 0)

Expected output after a few days of use:

$ python quiz_bot.py notes/dropout.txt
{
  "ts": "2026-08-21T02:14:03Z",
  "source_file": "notes/dropout.txt",
  "question": "What does the dropout paper claim about overfitting?",
  "answer": "That dropout prevents co-adaptation of neurons...",
  "source_excerpt": "dropout prevents co-adaptation of feature detectors",
  "reasoning": "This is the paper's central claim and a common exam point."
}

$ python verify_ledger.py
OK   2026-08-21T02:14:03Z — What does the dropout paper claim about overfitting?
FAIL 2026-08-21T02:31:47Z — Which layer introduced the GELU activation?
     phantom excerpt: 'the GELU activation was introduced in section 4.2'

1 phantom citation(s) found

Every FAIL line is a citation that does not exist in the source. Every OK line is a claim I can check in ten seconds. That asymmetry is the whole design.

I ran this for a week on MonkeyCode's free model access and their free server option. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The script itself doesn't care which provider you use — any OpenAI-compatible endpoint works, which is exactly why I could swap in a free tier without changing a line of code.

The results from my run: 40 questions generated across five readings. Seven of them cited excerpts that did not exist in the source. Three more quoted real lines but reasoned about them in ways the source didn't support. That's a 25% failure rate on the thing that matters most for studying: accuracy. The ledger caught every single one.

Free infrastructure added its own texture to the experiment. The server slept after inactivity, so my scheduled job missed two nights. The model rate-limited me once at 2 AM, which felt personal until I added a retry with backoff. These are the normal costs of free tiers — you don't pay money, you pay attention.

The failure modes were consistent, and each one taught me more than the successes did. Sometimes the model returned prose instead of JSON; the extract_json helper handles that by grabbing the first balanced object and skipping the entry if there is none. Sometimes the endpoint returned HTTP 429; a five-second retry loop fixed it. And long readings exceeded the model's context window, so I truncate to 4000 characters and split longer papers into sections before running the bot.

Three lessons came out of this. First, an LLM's confidence is not evidence. The model sounded most certain on the questions where it was most wrong. Second, a ledger is only useful if something reads it — the verifier is the real agent here, and the model is the intern. Third, free infrastructure works for personal tools if you design for verification from the start.

Who should not use this? If you're building a study tool for other people, or preparing for something where citations must be exact — medical, legal, certification exams — don't rely on a free model's self-reported excerpts. Use retrieval with exact string matching, or a paid model, and verify anyway. The ledger is a safety net, not a guarantee.

The extension exercise: make the verifier detect contradictions. Generate two questions from the same excerpt on different days, then check whether the answers contradict each other. My run found one pair that did — same passage, opposite claims, both delivered with total confidence. That's the moment you realize the problem was never the model. It was the absence of a paper trail.

If you want to try the same setup, MonkeyCode's free tier is one place to start — the 10 million tokens and free server I used covered a few weeks of this project. Or point the script at any other endpoint. The ledger doesn't care where the tokens come from. It only cares whether the model can back up its claims.