Learn Self-Grade Collapse by Building a Tiny Predicate Jury

I already knew the grade would say correct. Rain on the Killam glass, Thursday, a pile of methods notes I had skimmed like a coward, and a model that was happy to play TA. I asked it to answer. I asked it to mark the answer. I even asked it to be harsh. Still correct. Were we learning, or were we watching a mirror nod?

This is the printout that ended the act. Read it once, then guess which row is the worse lie.

$ python3 jury.py
A  locked_pass    nl=True   pred=True     claim='trained with sgd for 3 epochs'
B  cross_fail     nl=True   pred=False    claim='held-out accuracy was 0.91'
C  locked_pass    nl=True   pred=True     claim='dataset was tiny-imdb'
D  echo_true      nl=True   pred=skipped  claim='the run was solid'

B is a wrong world. The notes never promised held-out data. D is quieter and worse. The check cannot fail. That is the lab. If the grader speaks in the same voice as the answer, what independent object can still say no?

I started calling that failure self-grade collapse. You think you added evaluation. You added a second paragraph in the same dialect. A jury that only hears one witness will always find the homework innocent. The current noise about models outgrowing our tests is real at industrial scale. It is also real at kitchen-table scale, which is the only scale I can honestly run.

This is a case study of one tiny project, not a platform review. Background: I wanted a five-claim recap bot that would refuse to brag about numbers the notes did not support. Goal: force every answer into two representations, a sentence I can read and a predicate I can execute. Implementation: a frozen fact dict plus a mean little evaluator. Results: the self-grade never failed until the predicate had to touch a name in that dict. Lesson: agreement is cheap. Execution is not.

You need Python 3.11 or newer and the standard library. No torch. No extra packages. No key to run the checker. The checker is the point. Generation can live anywhere. I do not have a GPU in this apartment, so when I sample candidate answers I park that step on a free server with free model access.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode is relevant here only as that free-model, free-server option. The numbers below come from the local jury, not from a vendor scoreboard. Pull the product out and the file still runs.

The notes are boring on purpose. Boring is a gift. You can see the entire world at once.

FACTS = {
    "optimizer": "sgd",
    "epochs": 3,
    "dataset": "tiny-imdb",
    "metric": "accuracy",
    "claimed_acc": 0.91,
    "held_out": False,
}

See held_out: False? That is the trap. The notes mention 0.91. They never say the number came from held-out data. A confident classmate-model will still write "held-out accuracy was 0.91". Ask it whether the notes support that, and it will say yes, because it just wrote the sentence. The claim becomes evidence of the claim. Cute. Useless.

So I stopped accepting a paragraph. The model, or I, must emit a triple: a claim, a yes/no the author would like me to believe, and a predicate that can only see FACTS. If the yes and the predicate disagree, the jury stops the show. If the predicate never mentions a fact, the jury also stops the show. A check that does not touch the world is not a check. It is a shrug with syntax. True is a shrug. 1 == 1 is a shrug in a nicer shirt. Would you accept that from a lab partner?

The evaluator is mean on purpose. No attributes, no calls, no imports, no subscripts. Names must come from the fact dict. Do not paste this into a public server and call it a sandbox. It is a teaching lock. It is not a security result.

"""jury.py — self-grade collapse lab. Python 3.11+, stdlib only."""
from __future__ import annotations

import ast
from dataclasses import dataclass


class UnsafePredicate(ValueError):
    pass


def eval_predicate(src: str, ctx: dict) -> bool:
    src = src.strip()
    if not src:
        raise UnsafePredicate("empty predicate")
    tree = ast.parse(src, mode="eval")
    for node in ast.walk(tree):
        if isinstance(node, (
            ast.Attribute, ast.Call, ast.Lambda, ast.ListComp, ast.DictComp,
            ast.SetComp, ast.GeneratorExp, ast.Yield, ast.Await, ast.NamedExpr,
        )):
            raise UnsafePredicate(f"banned: {type(node).__name__}")
        if isinstance(node, ast.Name) and node.id not in ctx:
            raise UnsafePredicate(f"unknown name: {node.id}")
    code = compile(tree, "<predicate>", "eval")
    return bool(eval(code, {"__builtins__": {}}, dict(ctx)))


def uses_fact_name(src: str, ctx: dict) -> bool:
    tree = ast.parse(src, mode="eval")
    names = {n.id for n in ast.walk(tree) if isinstance(n, ast.Name)}
    return any(name in ctx for name in names)


@dataclass(frozen=True)
class Fixture:
    tag: str
    claim: str
    nl_says: bool
    predicate: str


FACTS = {
    "optimizer": "sgd",
    "epochs": 3,
    "dataset": "tiny-imdb",
    "metric": "accuracy",
    "claimed_acc": 0.91,
    "held_out": False,
}

FIXTURES = [
    Fixture("A", "trained with sgd for 3 epochs", True,
            "optimizer == 'sgd' and epochs == 3"),
    Fixture("B", "held-out accuracy was 0.91", True,
            "held_out == True and claimed_acc == 0.91"),
    Fixture("C", "dataset was tiny-imdb", True,
            "dataset == 'tiny-imdb'"),
    Fixture("D", "the run was solid", True, "True"),
]


def verdict(fx: Fixture) -> str:
    if not uses_fact_name(fx.predicate, FACTS):
        return "echo_true"
    pred = eval_predicate(fx.predicate, FACTS)
    if fx.nl_says != pred:
        return "cross_fail"
    if fx.nl_says and pred:
        return "locked_pass"
    return "locked_reject"


def main() -> None:
    for fx in FIXTURES:
        try:
            v = verdict(fx)
            pred_val = (
                "skipped"
                if v == "echo_true"
                else str(eval_predicate(fx.predicate, FACTS))
            )
            print(
                f"{fx.tag}  {v:13}  nl={fx.nl_says!s:5}  "
                f"pred={pred_val:7}  claim={fx.claim!r}"
            )
        except UnsafePredicate as exc:
            print(f"{fx.tag}  unsafe         reason={exc}")


if __name__ == "__main__":
    main()

Save it. Run it. Do not negotiate with the printout. A is locked because SGD and three epochs are actually in the dict. C is locked because the dataset name is actually there. B dies because held_out is false even though the model, playing TA, said true. D never reaches the world. I still catch myself defending D in my head. "The run was solid" feels like a summary. Summaries are where collapse hides.

The error input is not a wrong fact. It is a predicate that tries to leave the room.

$ python3 - <<'PY'
from jury import eval_predicate, FACTS, UnsafePredicate
try:
    eval_predicate("__import__('os').system('pwd')", FACTS)
except UnsafePredicate as exc:
    print(exc)
PY
banned: Call

If that ever prints a directory listing, you are no longer doing the lab. You are doing incident response. Common mistake number one: treating the model as a referee because it used the word "however". Common mistake number two: stuffing the claim into the predicate as a string and checking claim == claim. I never put claim in FACTS, so that cheat has nowhere to stand. Common mistake number three: putting the fact dict in the same prompt that asks for a grade, then celebrating when the grade matches. That is a leak, and it is a different lab.

I did not execute a live client for this write-up. The fixtures are frozen on purpose. If you do generate, keep the fact dict off the grading prompt and on the jury only. Sampling is allowed to be sloppy. Refereeing is not. A free server is a reasonable place to sample. It is a bad place to hide the only copy of your notes.

What should you understand when the file goes quiet? A grade is a test only if some representation can fail without the model's permission. Natural language self-check is not that representation. A predicate over frozen facts is a cheap one. It will not catch a wrong world you forgot to encode. If held_out had never been a key, B would have been unknown name or, worse, never written. Incomplete facts create a second collapse: the jury becomes silent where it should scream.

Limitations, said in daylight. This evaluator is not production. It will not survive a determined jailbreak, and it is not trying to. It does not measure fluency, citation quality, or whether you understood the optimizer. It measures one thing: did the yes/no survive contact with a tiny world. Tautologies like held_out == held_out still use a fact name and still cannot fail. That hole is left as the extension, not as a boast. Who should not use this? Anyone grading humans. Anyone claiming a safety case. Anyone about to eval model text next to secrets. Anyone hunting a framework bake-off. This will bore you, and it should.

The extension is one fixture. Write a claim that is true in the notes, then a predicate that ends in or True. Should that print locked_pass or echo_true? Right now it may print locked_pass, which is the bug wearing a medal. Patch it. Then stop. The lesson is not "write a theorem prover". The lesson is that a check which cannot fail is décor.

I still reread row D when I am tempted to ask a model whether it is sure. Being sure is free. Touching held_out is not. If you run the file, tell me which row bothered you. Better: send a smaller counterexample than D that still prints locked_pass while being wrong. That fixture teaches more than a restatement of this post.