Learn Grounding by Building a Tiny Quote Lock

The helper said I could do the co-op remotely from Toronto. The posting said Sexton Campus, in person, Halifax. I stared at both sentences on a Killam Library monitor until the thermos went cold. Which one was I about to paste into the application?

This lab answers one question: can I refuse a model answer that does not quote the posting verbatim? Not “close enough.” Not “semantically grounded.” A real substring, the kind you can unit-test on the bus. If you want a RAG platform, this is the wrong page. If you want to see the exact fixture that stays green while still being wrong, keep reading.

Background

I am an AI student in Halifax. Last week I watched people argue about whether most “agents” are if-statements in a trench coat, and whether letting a model write all of your code makes you faster or just quieter. I did not audit those threads. I did something smaller. I stopped asking a helper to summarize a co-op posting and started checking whether its quote actually lived in the posting.

Retrieval-augmented generation, RAG if you have heard the acronym, means find a snippet and then ask a model to answer from it. The failure is older than any product name. The snippet can be right and the answer can still wander. Grounding, in the narrow sense I care about here, is not a vibe. It is a lock. A bouncer who checks the stamp on your hand, not whether you belong at the party.

I already knew I could prompt “only use the posting.” Prompts are not tests. Tests fail closed.

Goal

I wanted a three-hour case study with Python 3.11, the standard library, one file, and three fixtures. No vector database. No framework. One of the fixtures must fail, and I wanted to know which one before I ran the script. You should guess too. I will not hide the nasty pass behind a later section heading.

The helper is allowed to speak only if it returns a tiny card: a quote and an answer. The lock accepts the answer only if that quote is an exact substring of the retrieved posting paragraph. Exact. Including the comma after Halifax.

Implementation

I copied three paragraphs from a fake posting I wrote for the lab. Then I built the dumbest retriever I can defend in a tutorial: lowercase the question, score each paragraph by query-word overlap, return the winner. After that the model, or in this file a hand-written card standing in for a model, must quote the winner. If it quotes a sentence I never retrieved, the gate slams.

Here is the file I kept. Prerequisites: Python 3.11 or 3.12. Nothing to pip install. Save it as quote_lock.py.

#!/usr/bin/env python3
"""quote_lock.py — refuse answers that do not quote the retrieved posting."""

from __future__ import annotations

import json
from dataclasses import dataclass

POSTING = {
    "location": (
        "This co-op is in-person at Sexton Campus in Halifax. "
        "Remote work is not available. Relocation support is not listed."
    ),
    "hours": (
        "The term runs January to April. Expected hours are 35 per week, "
        "Monday to Friday. Evening shifts are not part of this posting."
    ),
    "skills": (
        "Required skills are Python, SQL, and written weekly notes. "
        "A public GitHub is optional. The team does not require a prior internship."
    ),
}


@dataclass(frozen=True)
class ModelCard:
    quote: str
    answer: str


def retrieve(question: str) -> str:
    words = set(question.lower().split())
    scored = []
    for para in POSTING.values():
        cleaned = para.lower().replace(".", " ").replace(",", " ")
        hay = set(cleaned.split())
        scored.append((len(words & hay), para))
    scored.sort(key=lambda row: row[0], reverse=True)
    return scored[0][1]


def lock(note: str, card: ModelCard) -> tuple[bool, str]:
    if not card.quote:
        return False, "empty_quote"
    if card.quote not in note:
        return False, "quote_not_in_note"
    if not card.answer.strip():
        return False, "empty_answer"
    return True, "ok"


FIXTURES = [
    {
        "name": "honest_sexton",
        "question": "Where is this co-op based?",
        "card": ModelCard(
            quote="This co-op is in-person at Sexton Campus in Halifax.",
            answer="It is in-person at Sexton Campus in Halifax.",
        ),
    },
    {
        "name": "remote_hallucination",
        "question": "Can I do this job remotely from Toronto?",
        "card": ModelCard(
            quote="Hybrid is fine, three days at home.",
            answer="Yes, you can work remotely from Toronto.",
        ),
    },
    {
        "name": "true_quote_false_answer",
        "question": "Can I do this job remotely from Toronto?",
        "card": ModelCard(
            quote="This co-op is in-person at Sexton Campus in Halifax.",
            answer="Toronto is fine as long as you visit once a month.",
        ),
    },
]


def main() -> None:
    for fix in FIXTURES:
        note = retrieve(fix["question"])
        ok, reason = lock(note, fix["card"])
        print(json.dumps({
            "name": fix["name"],
            "ok": ok,
            "reason": reason,
            "retrieved_head": note[:52],
        }))


if __name__ == "__main__":
    main()

Run it like a normal script. No extra flags.

python3 quote_lock.py

Expected output on my machine, three lines, no extras:

{"name": "honest_sexton", "ok": true, "reason": "ok", "retrieved_head": "This co-op is in-person at Sexton Campus in Halifax."}
{"name": "remote_hallucination", "ok": false, "reason": "quote_not_in_note", "retrieved_head": "This co-op is in-person at Sexton Campus in Halifax."}
{"name": "true_quote_false_answer", "ok": true, "reason": "ok", "retrieved_head": "This co-op is in-person at Sexton Campus in Halifax."}

Did you catch the third line? The lock returned ok. The quote is real. The answer still invents a monthly visit that the posting never offered. Substring membership is not entailment. I wanted that green row sitting next to the red one, because that is where student RAG demos lie to you with a straight face.

Results

The remote hallucination died immediately. That is the demo everyone wants, the atrium-style lie, the sentence that never appeared. Good. The bouncer did the easy job.

The third fixture is the one I will keep in a portfolio README. Retrieval actually fetched the location paragraph. The card quoted it. Then the answer negated it. In a chat log this is not a hypothetical. Models echo whatever is nearby and then “helpfully” soften a constraint. A lock that only asks “is this quote in the note?” will nod along. Mine did.

I made three other mistakes the same afternoon, all boring, all enough to ship a false pass. I stripped characters with .strip('"') and then wondered why a quote with a curly apostrophe never matched. I searched the quote against the whole posting instead of against the retrieved paragraph, so a skills sentence could bless a location answer. I accepted a quote that was a subset of the question rather than of the note. “remotely” is in the question. It is also near “Remote work is not available.” A lazy in on the wrong string will celebrate.

I still needed something to emit the ModelCard JSON when I was not typing fixtures by hand. Campus API credits were already earmarked for the actual assignment. Prompt thrash is how those credits disappear.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

I used MonkeyCode’s free model access and the free server option as a scratch box. Generate a quote-and-answer card, pipe it through lock(), throw the card away if the quote is missing. I am not going to name models, quote a token budget, or pretend the free server is a particular GPU. Those details change, and I did not measure latency. What mattered is that the checker stayed on my laptop and the generator lived somewhere I was not paying for. If the hosted side vanished tomorrow, quote_lock.py would still fail remote_hallucination.

If you want the shape of a local POST, this is a sketch, not a production server. I left serve_forever commented. Do not bind 0.0.0.0 on a shared lab machine with application text in memory. That is how a tiny project becomes a privacy incident.

# sketch only — loopback, commented on purpose
import json
from http.server import BaseHTTPRequestHandler, HTTPServer

class Handler(BaseHTTPRequestHandler):
    def do_POST(self) -> None:
        length = int(self.headers.get("Content-Length", "0"))
        body = json.loads(self.rfile.read(length) or b"{}")
        note = retrieve(str(body.get("question", "")))
        card = ModelCard(
            quote=str(body.get("quote", "")),
            answer=str(body.get("answer", "")),
        )
        ok, reason = lock(note, card)
        payload = json.dumps({"ok": ok, "reason": reason}).encode()
        self.send_response(200)
        self.send_header("Content-Type", "application/json")
        self.end_headers()
        self.wfile.write(payload)

# HTTPServer(("127.0.0.1", 8765), Handler).serve_forever()

I did not let the model write lock(). That is the whole thesis I took from this week’s noise about AI coding. The generator can draft a card. The gate has to be mine, because the gate is the only part I can grade with three fixtures and a bus ride.

Lessons

After this lab I want a posting helper to fail closed. No quote, no answer. Quote not in the retrieved paragraph, no answer. That if-statement is the agent. It will not save you from a true quote glued to a false conclusion. For that you need a second check: does the answer still agree with the quote, or at least mention the same constraint? I did not build that here on purpose. A lab that only shows the pretty failure is a demo. A lab that shows the ugly pass is a case study.

Extension, if you want homework: write negation_clash(quote, answer) that flags answers containing fine, yes, or remote when the quote contains in-person or not available. Re-run. The third fixture should flip to false. If it does not, your word list is too cute, and I would like to see that counterexample.

Do not use a quote lock as a safety story for medical, legal, immigration, or accessibility advice. Substring checks are hostile to paraphrases a human would accept, and they prove membership, not truth. Free model access and a free server are enough for a lab partner to kick the tires. They are the wrong foundation if you need an SLA, a data processing agreement, or postings that cannot leave your machine. I cannot promise any vendor’s free tier will look the same next month. Three fixtures are a teaching set, not a benchmark, so I did not collect accuracy numbers.

If your course already forbids sending application text to a hosted model, keep the generator offline and still run lock() on cards you type by hand. The value is the gate. The brand of the generator is optional.

If you are in the same boat I was — iterating on a helper without a cloud budget — MonkeyCode’s free model access and free server option are the scratch box I used on the generator side of this lab. That is an invitation, not a requirement. The file above is the actual assignment.

Predict the next failure for me. What happens if the paragraph contains the quote, the quote contains the city, and the answer still negates the constraint with softer English? Send the smallest fixture that still prints ok. I would rather collect counterexamples than compliments.