Grade the Comment Like a Test: A Bootcamp Lab on Checkable Agent Explanations

Your agent's comment is not documentation until a check can fail it. I start this lab there, before anyone opens a model tab, because a fluent paragraph can describe a function the student never wrote. If a sentence cannot be tied to an assertion, why would I give it points?

I got tired of READMEs that sound finished and branches that do the opposite. Students paste a smooth explanation under a buggy comparison and call the page clear. Clear to whom?

The conclusion I actually grade

Prose is optional. A graded comment is a claim with an id, one behavior, and a check that can go red. Pretty wording with no mapping is a miss, not a style deduction.

That is the whole lab. Setup, checkpoints, and a rubric come after, so office hours stay about behavior instead of taste.

Why vibe grading collapses

Bootcamp submissions now arrive with agent-written comments. Some help. Some retell the function name in longer words. Some describe the happy path the student meant, not the branch the code took.

Have you tried grading that by feel? I have. It turns into a debate about tone, and I do not want that debate. I want a command I can re-run on a clean checkout.

People are arguing again about clean code versus clear code. Fine. I am not rehashing that essay. Comments are allowed here. They just have to be falsifiable. A comment that cannot be wrong is a slogan. What do you do when the model writes a beautiful lie?

What counts, before anyone codes

I put this list on the board first. If a line fails it, it does not earn comment credit, no matter how polished it looks.

• A claim names one behavior and an input class a stranger could try.
• A claim can be false. If the student cannot imagine a failing input, it is not a claim.
• A claim has an id that appears in both claims.json and the check map.
• A paragraph that covers three behaviors is three claims, or it is zero claims.
• "Handles edge cases" is not a claim. Neither is "optimized" or "production ready."
• Inline comments may stay for humans. They do not earn points unless they are copied into the claim file.

Setup

You need a tiny repo, not a platform. I keep the starter boring so the grade is about the claim, not about tooling theater.

1. Create lab-claims/ with sample.py, claims.json, and claimcheck.py. Run from that directory so the import is honest.
2. Students may draft code and comment text with a model they already have. The draft is not the submission. The claim file plus the check map is the submission.
3. The grader clones fresh and re-runs the harness. A screenshot of a local notebook is a pointer, not evidence. Would you accept a screenshot in a functions quiz? I would not.
4. Toy integers only. If a claim needs a key, a customer row, or a private repo, the lab is already the wrong assignment.

Two practical gaps show up the moment a cohort is bigger than one laptop. Some students have no API budget for the draft. Some graders cannot reproduce a notebook that lived in one terminal session. MonkeyCode is the option I would point at for those two gaps only: free model access for the draft, and a free server option for the clean re-run. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

I am not naming a model. I am not quoting a token cap, a machine size, or a duration. Those are the kinds of numbers that go stale between the day you write a syllabus and the day you teach it. Read the current docs that week, or leave the tool off the assignment sheet.

Does the tool have to be that one? No. The checks are plain Python. If either hosted option is down, the lab still runs locally. Do not pretend the signup is the proof.

The artifact

claims.json is the only comment channel that earns credit. Keep each text to one behavior. If you need a second behavior, add a second id.

[
  {
    "id": "pick-cheaper",
    "text": "pick returns the smaller integer when the inputs differ, and either input when they are equal."
  },
  {
    "id": "empty-cart-total",
    "text": "cart_total of an empty list is 0, not None and not an exception."
  }
]

Ship sample.py with one honest bug. Checkpoint 4 needs a lie you can catch without inventing a framework.

def pick(a, b):
    # Intent and code disagree. The comment in the student's head is not the grade.
    return a if a > b else b

def cart_total(prices):
    total = 0
    for price in prices:
        total += price
    return total

claimcheck.py does not parse English. It refuses unmapped ids, empty text, and checks that nobody claimed. Treat this as a lab script, not as a verifier I ran across a corpus. I have not benchmarked it. I would not.

import json
import pathlib
import sys
from sample import cart_total, pick

CHECKS = {
    "pick-cheaper": lambda: (
        pick(3, 9) == 3 and pick(9, 3) == 3 and pick(4, 4) == 4
    ),
    "empty-cart-total": lambda: cart_total([]) == 0,
}

def main(path: str) -> int:
    raw = pathlib.Path(path).read_text(encoding="utf-8")
    rows = json.loads(raw)
    if not isinstance(rows, list) or not rows:
        print("FAIL file: claims.json must be a non-empty list")
        return 1
    failed = 0
    seen = set()
    for row in rows:
        if not isinstance(row, dict):
            print("FAIL row: expected an object")
            failed += 1
            continue
        cid = str(row.get("id", "")).strip()
        text = str(row.get("text", "")).strip()
        if not cid or not text or cid in seen:
            print(f"FAIL row: bad id or empty text ({cid!r})")
            failed += 1
            continue
        seen.add(cid)
        check = CHECKS.get(cid)
        if check is None:
            print(f"UNMAPPED {cid}: no check, no credit")
            failed += 1
            continue
        try:
            ok = bool(check())
        except Exception as exc:
            print(f"FAIL {cid}: {type(exc).__name__}: {exc}")
            failed += 1
            continue
        print(("PASS " if ok else "FAIL ") + f"{cid}: {text}")
        failed += 0 if ok else 1
    missing = sorted(set(CHECKS) - seen)
    for cid in missing:
        print(f"FAIL missing-claim: {cid} has a check and no comment")
        failed += 1
    return 1 if failed else 0

if __name__ == "__main__":
    arg = sys.argv[1] if len(sys.argv) > 1 else "claims.json"
    sys.exit(main(arg))

Run it from the lab directory. I want the exit code, not a story about the exit code.

cd lab-claims
python claimcheck.py claims.json
echo "exit=$?"

On the sample above, pick-cheaper fails. pick(3, 9) returns 9, and the claim asked for the smaller integer. empty-cart-total passes. That split is the point. One fluent sentence can be false while the file still looks documented.

Then plant the lie on purpose. Change the claim text to say pick returns the larger integer, even after you fix the comparison. Re-run.

python claimcheck.py claims.json
echo "exit=$?"
# non-zero means the comment and the check still disagree

If you cannot make a wrong sentence go red, the check is too weak to grade. Fix the check. Do not edit the claim until the lie is visible.

Checkpoints

Checkpoint 1 — the sentence can be false

Each graded sentence has an id and one behavior. Ask the student, out loud: could this be false on a single input? If they say no, send it back. Slogans do not get a rewrite for partial credit in the same pass.

Checkpoint 2 — the map is the grade

Every id in claims.json has a lambda in CHECKS, and every lambda has a claim. Unmapped prose prints UNMAPPED and fails the run. I do not award "partial clarity." A comment the harness cannot see is a comment I will not defend in moderation.

Checkpoint 3 — fresh clone, same command

Clone the repo. Run the same command. Write the exit code next to the command in the submission note. The note is a pointer. The re-run is the grade. If you use a free server for that clone, the command still has to be the one in the starter, not a private alias only one student knows.

Checkpoint 4 — catch a planted lie

Students first watch pick-cheaper fail. They fix pick to a if a < b else b. Then they submit a wrong claim on purpose and show the harness going red again. A "fix" that edits the assertion to match the bug is a miss. You changed the ruler, not the code.

Stretch goals

These are bonus only. Bonus cannot wash a red checkpoint.

• Add negative inputs for pick. Does the comment still match, or did they describe the demo pair and nothing else?
• Reject claim text that says "always" unless the check includes an empty case, an equal case, and a negative case. Say in the rubric that this is a heuristic, not a proof.
• Freeze claims.json after one model draft, then edit code. Did the final sentence drift into a different rule? Drift is a discussion item unless the final text is false.
• Add cart_total of [1, 2, 3] as a second claim. One passing empty list is a thin oracle. You know that. Make them know it too.

Rubric

I publish this with the starter. Negotiating it per student is how vibe grading sneaks back in.

Checkpoint	Pass looks like	Points	Automatic miss
1. Claim shape	id + one falsifiable sentence	15	slogan, multi-behavior paragraph, missing id
2. Mapping	every id has a check and every check has an id	25	unmapped comment submitted for credit
3. Clean re-run	grader clones and gets the same exit code	30	screenshot-only, or a command that only exists in one notebook
4. Caught lie	a wrong pick claim fails; the real fix then passes	20	a check that cannot fail, or a test edited to match the bug
Stretch	extra inputs or a frozen first draft	10 bonus	bonus used to hide a red checkpoint

A miss on checkpoint 2 or 4 caps the lab at 50, even if the prose is lovely. I would rather explain that cap once than relitigate it in week six.

What I say in the first ten minutes

You may ask a model to draft the comment. You may not ask it to invent the check and then grade itself. The check is yours, or it is mine. If the model writes both the sentence and the assertion, you have a loop, not a lab.

Is that harsh? A little. Bootcamp weeks are short. I am not grading literary taste. I am grading whether you can show the sentence is about this function.

Read the fail line before you read the prose. FAIL pick-cheaper is the lesson. The paragraph underneath is just the claim that lost.

Limitations

This harness does not understand the comment. A mapped check can pass while the English describes a different rule that happens to agree on the chosen inputs. That is a weak-oracle problem. Add cases until you can name an input that would embarrass the sentence.

It does not score clarity for human readers. A true claim can still be clumsy. Clumsy-but-true beats smooth-and-false on this rubric. Another course can invert that. This one does not.

Free model access and a free server option are conveniences for drafting and for a shared re-run. They are not part of the proof. If either is unavailable, use local Python. Do not print a quota, a model name, or a machine size on the assignment unless you copied it from the docs you re-read that week. I am not freezing those numbers here.

Do not send secrets, real customer rows, or private homework about live systems to a shared model or a shared shell just to make the re-run look official. Toy functions are enough. A leaked key is not a stretch goal.

The script also crashes the teaching point if you treat a green run as completeness. Green means the mapped assertions passed. It does not mean the function is done, fast, or safe to ship.

Who should skip this

Skip it if the learning goal is voice, narrative comments, or API-design prose. A falsifiable-claim rubric will punish the essay you actually wanted. Teach that essay on purpose. Do not hide it inside this lab.

Skip the hosted options if school policy forbids pasting course code into a hosted model or a hosted shell. The harness does not require either. Local Python is the whole dependency.

Skip it for production review. Two functions and a JSON file will not catch races, auth bugs, or a comment that is true and still dangerous. If you need an incident write-up, this rubric will lie to you by being too small.

If you teach the lab

Put the harness in the starter repo. Put the rubric next to it. Mention a hosted draft model and a hosted re-run box only as optional, and only after you re-read what free model access and the free server option include that week. I keep that sentence short on purpose. The gradeable thing is the claim check, not the signup.