$ python tiny_prompt_scrubber.py
LEAKED: ['Alex Chen', 'Mon 14:00']
SCRUBBED PROMPT:
Write parse_office_hours for one raw line.
Do not hardcode [[REDACTED_EXPECT]] or [[REDACTED_EXPECT]].
That was the first honest printout I got all night in the Killam basement. Cold coffee. One lab function. A model draft that had been “passing” for twelve minutes. Then I asked a smaller question: if the green tests only exist because I pasted the answer key into the prompt, can a tiny local scrubber make the cheat visible?
I am an AI student in Halifax. I still do that lazy paste. You probably do too. We tell ourselves the model needs context. What it actually gets is the expected string sitting two lines under assert.
Background
The assignment looked harmless. Parse one office-hours line, like Alex Chen — Mon 14:00, into a name and a weekday slot. I wrote three pytest asserts, got bored, and dumped the whole test file into a prompt. The returned function passed. I felt efficient. I was not.
I changed one expected name from Alex Chen to Sam Lee and reran. The function still returned Alex Chen. How? Because the prompt had taught the model my fixture, not the rule. It was a closed-book exam where I had photocopied the key onto the question sheet.
This case study is that night, cleaned up. One concept. One standard-library gate. No framework tour.
Goal
I wanted a preflight check I would actually run. Before any free model saw my prompt, a local script had to hunt for expected values from my tests and redact them. If the scrubber found a leak, I would see the list. If it found nothing, I could paste. That is the whole contract.
I ran this on Python 3.11 with the standard library only. No pip. No notebook. The model call is optional and happens after the gate. The gate is the lab.
Disclosure: This article was prepared as part of MonkeyCode's product outreach. After the scrubber ran, I pasted the cleaned prompt into MonkeyCode’s free model access on the free server option, because I still wanted a second draft and I did not want to stand up a GPU in a library basement. The product is a convenience here. The lesson is the leak.
The fixtures I actually used
Here is the tiny test file I should never have pasted raw. Read it once. Predict which strings will leak. Then guess what happens if an expected value is the letter a.
# test_office_hours.py
from office_hours import parse_office_hours
def test_simple_line():
assert parse_office_hours("Alex Chen — Mon 14:00") == {
"name": "Alex Chen",
"slot": "Mon 14:00",
}
And here is the prompt I had been sending, word for word, like a student who thinks more context is always kinder.
Write parse_office_hours for one raw line.
Do not hardcode Alex Chen or Mon 14:00.
Yes. I wrote “do not hardcode” and then I hardcoded it anyway. The model does not grade your intentions.
Implementation
The scrubber is one file. It looks for pytest-style assert ... == ... lines, pulls the right-hand side, and redacts those fragments from the prompt. Longest fragment first, so Alex Chen goes before Alex if both appear.
# tiny_prompt_scrubber.py
from __future__ import annotations
import re
ASSERT_LINE = re.compile(
r"^\s*assert\s+.+?\s*==\s*(.+?)(?:\s*#.*)?\s*$",
re.MULTILINE,
)
STRING_LIT = re.compile(r"(['\"])(?P<body>(?:\\.|(?!\1).)*)\1")
NUMBER_LIT = re.compile(r"^-?\d+(?:\.\d+)?$")
NAME_LIT = re.compile(r"^(True|False|None)$")
REDACT = "[[REDACTED_EXPECT]]"
def expected_fragments(test_src: str) -> list[str]:
frags: list[str] = []
for match in ASSERT_LINE.finditer(test_src):
rhs = match.group(1).strip().rstrip(",")
if NUMBER_LIT.match(rhs) or NAME_LIT.match(rhs):
frags.append(rhs)
continue
for sm in STRING_LIT.finditer(rhs):
body = sm.group("body")
if body:
frags.append(body)
if rhs.startswith(("(", "[", "{")):
frags.append(rhs)
return sorted(set(frags), key=len, reverse=True)
def scrub(prompt: str, test_src: str) -> tuple[str, list[str]]:
leaked: list[str] = []
cleaned = prompt
for frag in expected_fragments(test_src):
if frag and frag in cleaned:
leaked.append(frag)
cleaned = cleaned.replace(frag, REDACT)
return cleaned, leaked
The driver is boring on purpose. I want the leak list in my face, not hidden in a log framework.
TEST_SRC = r'''
def test_simple_line():
assert parse_office_hours("Alex Chen — Mon 14:00") == {
"name": "Alex Chen",
"slot": "Mon 14:00",
}
'''
PROMPT = """Write parse_office_hours for one raw line.
Do not hardcode Alex Chen or Mon 14:00.
"""
if __name__ == "__main__":
cleaned, leaked = scrub(PROMPT, TEST_SRC)
print("LEAKED:", leaked)
print("SCRUBBED PROMPT:")
print(cleaned, end="")
Run it from the same directory.
python tiny_prompt_scrubber.py
You should see the output at the top of this article. Two leaked strings. Two redaction marks. The prompt is now a question without an answer key taped to it.
The error input, because green is not proof
I said I would show you the letter a. Here it is. This is the fixture that taught me not to trust my own regex.
BAD_TEST = r'''
def test_initial():
assert parse_initial("alpha") == "a"
'''
BAD_PROMPT = "Write parse_initial so alpha becomes a single letter."
cleaned, leaked = scrub(BAD_PROMPT, BAD_TEST)
print(leaked)
print(cleaned)
Expected print:
['a']
Write p[[REDACTED_EXPECT]]rse_initi[[REDACTED_EXPECT]]l so [[REDACTED_EXPECT]]lph[[REDACTED_EXPECT]] becomes [[REDACTED_EXPECT]] single letter.
Ugly, right? The scrubber did its job too well. A one-character expected value is a landmine. It redacts every a in the English sentence. That is not a model failure. That is me shipping a gate without a minimum fragment length.
I added one guard after I saw that printout. Not because it made the demo prettier. Because it is the actual student bug.
MIN_FRAG = 2
def expected_fragments(test_src: str) -> list[str]:
frags: list[str] = []
# ... same extraction as above ...
return sorted(
{f for f in set(frags) if len(f) >= MIN_FRAG},
key=len,
reverse=True,
)
With MIN_FRAG = 2, the landmine prompt stays readable, and == "a" simply does not enter the leak list. That is a tradeoff, not a victory. Short expected values can still leak. I just refused to destroy the question in order to hide them.
Results from the actual lab function
After scrubbing, I wrote parse_office_hours the slow way: split on the em dash, strip, return a dict. Then I asked a free model for a second draft from the redacted prompt only. The first unscrubbed draft had hardcoded my name. The scrubbed draft could not. It had to talk about “the left side” and “the right side.” That sentence structure is the whole point. The model was forced to describe a rule.
Did the second draft still fail on a second line, like Sam Lee — Fri 09:30? Yes, once, because it assumed a comma instead of a dash. Good. A visible miss beats a hidden copy. I fixed the split and kept the tests on my machine, where they belong.
I did not benchmark tokens. I did not time the server. I do not have a quota to quote that I can defend. The measurable result is smaller and meaner: unscrubbed prompt leaked two fixtures; scrubbed prompt leaked zero of those two; the one-character fixture taught me to add MIN_FRAG.
What broke, and who should not copy this
This is a teaching gate, not a security product. It does not parse AST. It will miss pytest.mark.parametrize, doctests, helper variables, and assert result.name == person. It will also redact innocent English if your expected string is a common word like name. Do not ship it as a production redaction filter. Do not use it on secrets, API keys, or student data that is not already in your own test file.
Skip this approach if your course already bans model use. Skip it if you cannot run Python locally. Skip it if you wanted an agent loop. This script does not retry, does not call a network, and does not grade your function. It only refuses to let the answer key leave your laptop unnoticed.
MonkeyCode’s free model access and free server option helped me generate the second draft after the gate. They did not replace the gate. If you cannot tell those two jobs apart, you will paste the test file again tomorrow.
Common mistakes I made in one evening
I treated “do not hardcode X” as a safety feature. It is not. It is a second copy of X. I treated a passing test as evidence of a rule. It was evidence of a leak. I treated a short expected value as “too small to matter.” It mattered enough to shred the prompt. I almost added a network call before I had a leak list. That would have been generating first and measuring later, which is how the original cheat happened.
What you should understand when this works
Fixture leakage is not a vibe-coding slogan. It is a dataflow bug. Your test file contains literals. Your prompt is a string. If those two strings share a substring, the model can answer by memory. A scrubber does not make the model honest. It makes the missing rule obvious, because the answer key is gone.
After this lab you should be able to look at any prompt you are about to paste and ask: which of my asserts are already inside this paragraph? If you cannot answer, you are not doing generation. You are doing open-book copying with extra steps.
Extension
Keep the same file. Add a second test that uses pytest.mark.parametrize with two names. Watch the current regex miss both. Then decide whether you want a real parser (ast.parse on the test file) or whether you would rather write expected values in a JSON fixture that never enters the prompt. Either extension is the next honest hour. Pasting more context is not.
If you want a place to try the scrubbed prompt without standing up a box of your own, the free model access and free server option I used is optional. Run the scrubber first. The leak list is the assignment.
Top comments (0)