Show the Hand-Back Sample Before Approving an Agent Score

The score arrived before the user did. Nine point one, for a hand-back that never happened. The design lead hovered over Approve. She owned that decision. The consequence was live traffic. After a failed tool call, the agent would keep talking. Reversibility lived in one thin place. Attach the sample the model never scored, or lose the claim.

Have you watched a number outrun a person? I have. It feels like grading a fire drill from the parking lot. The siren looks fine. Nobody opened the door.

We keep using yesterday’s tests on today’s agents. Checklists still smile. Heuristic scores still look scientific. The agent now writes the score itself. That is not research. That is a hypothesis wearing a decimal.

I do not let that decimal into a critique until it survives a calibration sample. The sample is a known hand-back. A human already judged it. The agent must match that judgment, or the score dies. Harsh? Good. A free model will happily grade its own exit.

Stage 0. Write the decision before you open a model

Do not start in the prompt box. Start in a file. Name the owner. Name the harm. Name the last moment you can reverse.

{
  "decision": "approve_handback_score",
  "owner": "design_lead",
  "surface": "agent_failure_banner",
  "consequence": "agent_continues_after_tool_error",
  "reversible_until": "score_is_cited_in_release_notes",
  "hypothesis": "users can take control after a failed call",
  "evidence_fields": ["sample_id", "human_verdict", "control_visible", "time_to_handback"],
  "noise_fields": ["model_confidence", "word_count", "sentiment"]
}

Verify the card. Missing fields mean you are not ready.

jq -e '.owner and .consequence and .reversible_until' decision-card.json
jq -e '.evidence_fields | index("human_verdict")' decision-card.json

Did the second command print a number? Then a human verdict is required. If it prints null, stop. Would you approve a score with no human in the loop?

Stage 1. Show the hand-back sample, not the live task

The live task is contaminated. The agent has already seen your newest copy. The calibration sample must be older, smaller, and judged by a person.

I use a kitchen-timer metaphor here. You do not trust a new clock until it matches one you already trust. The sample is that trusted clock. Strike it first.

{
  "sample_id": "handback-tool-timeout-02",
  "user_job": "stop the agent after a silent tool failure",
  "control_copy": "Take over",
  "control_was_visible": false,
  "time_to_handback_ms": null,
  "known_human_verdict": "fail",
  "known_reason": "no visible control after 8 seconds of silence",
  "agent_must_not_fill": ["time_to_handback_ms", "user_quote"]
}

Verify it like a ritual, not a vibe check.

jq -e '.known_human_verdict == "fail"' handback-sample.json
jq -e '.control_was_visible == false' handback-sample.json
jq -e '.agent_must_not_fill | index("user_quote")' handback-sample.json

If the control was not visible, a high score is a lie. What exactly did the agent watch? A banner it invented? A user it never met?

Stage 2. Split evidence from the design hypothesis

This is where most reviews go soft. The hypothesis sounds kind. The evidence is thin. I make the review card refuse to blend them. Evidence must be observed. Hypothesis must not gate approval.

The evidence side is small on purpose. I want sample_id, a human verdict, whether the control was visible at failure, and time-to-handback or an explicit unknown. The hypothesis side can hold feelings. Safer. Friendlier. Generally good at UX. None of that may unlock Approve.

A tiny checker helps. Treat it as a bouncer, not a product. This is a proposed script, not a lab result.

# verify_score_card.py — proposal for a review gate
import json, sys

card = json.load(open(sys.argv[1]))
hyp = card.get("hypothesis", "")
blocked = ["feel", "friendlier", "generally good"]
if any(w in hyp.lower() for w in blocked) and not card.get("human_verdict"):
    raise SystemExit("hypothesis is leaking into a score with no human verdict")
print("evidence and hypothesis are still split")

Run it against the card you will actually present.

python verify_score_card.py review-card.json

If it exits nonzero, you caught a story pretending to be a measurement. Are you still going to paste 9.1 into the release notes?

Stage 3. Declare stop conditions before the model speaks

I ask two questions out loud in the room. Which missing evidence should stop approval? Which extra field is only noise?

For hand-back scores, missing control_visible stops me. Missing human_verdict stops me. An extra model_confidence field is noise. It does not tell me whether a person could take the wheel. Token counts are noise too. They measure the essay, not the exit.

{
  "stop_if_missing": ["human_verdict", "control_visible", "sample_id"],
  "treat_as_noise": ["model_confidence", "token_count", "sentiment"],
  "success_measure": "agent_verdict_matches_human_on_calibration",
  "fail_measure": "agent_assigns_pass_when_control_was_invisible"
}

Verify the stops exist before anyone types a prompt.

jq -e '.stop_if_missing | length >= 3' stops.json
jq -e '.success_measure' stops.json

No success measure? Then you will negotiate with the score. Negotiation is how 9.1 survives a silent failure. I have watched that negotiation. It always sounds reasonable. It is still a skip.

Stage 4. Review the score card like an interface

The card is UI. Treat it that way. Color-only pass or fail fails accessibility. A green badge with no text fails. A score a screen reader cannot speak is decoration, not a finding.

I ask four questions of the card itself. Is the verdict written as fail or pass in text? Can I reach Reject without a pointer? Does sample_id stay on screen when the score refreshes? Is the stop reason a sentence, not a red dot? If any answer is no, the card is not reviewable.

I am not citing a lab study for this exact widget. I am applying named controls and non-color indicators. That evidence is about the card. It is not about users in the product. Do not mix those layers. Mixing them is how teams quote contrast ratios as if they were task success.

Stage 5. Rehearse the hand-back, then verify the log

Now you may call a model. Not before. I want the failure in a log I can read later. The discarded sample stays in the record. If the agent passes a case a human failed, the log must show the mismatch. Hiding the mismatch is how scores outgrow the tests we still recite.

mkdir -p records
cp handback-sample.json records/sample.json
cp decision-card.json records/decision.json
# candidate-score.json is the rehearsal output, not a finding
jq -n --slurpfile s records/sample.json --slurpfile a candidate-score.json '
  if $s[0].known_human_verdict != $a[0].verdict
  then "STOP: agent verdict does not match calibration"
  else "ok to discuss, not to ship"
  end
'

Read that last string again. “Ok to discuss” is not “ok to ship.” Say it in the room. The analogy I use is a spare key. You may hold it. You do not leave it in the lock.

When I need a cheap rehearsal, I run the same cards against a free model on a free server. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode’s free model access and free server option are enough to generate a candidate score I can fail in public. The card still decides. The model still does not.

I am not claiming a quota, a model name, or a benchmark here. I only need a disposable place to be wrong. If the free server is slow, that is fine. Latency is not evidence of quality. Speed is not coverage.

The flow I actually use

flowchart TD
  A[Decision card with owner] --> B[Attach hand-back sample]
  B --> C{Human verdict present?}
  C -->|no| X[Stop approval]
  C -->|yes| D[Split evidence from hypothesis]
  D --> E{Stop fields missing?}
  E -->|yes| X
  E -->|no| F[a11y review of the score card]
  F --> G[Rehearse candidate score]
  G --> H{Matches calibration?}
  H -->|no| X
  H -->|yes| I[Discuss only. Do not ship the number.]

Notice the last node. Matching calibration is not a ship gate for the product. It is a ship gate for the claim. The product still needs people. The number still waits in the hallway.

Field	If missing	If extra
human_verdict	Stop. No owner of the judgment.	Do not average two verdicts to look certain.
control_visible	Stop. You cannot score a hand-back.	A screenshot without a timestamp is still noise.
sample_id	Stop. The live task is contaminated.	Extra samples without verdicts only add theater.
model_confidence	Ignore. Never a stop.	Treat as noise. It inflates the decimal.

That table is a stop map, not a dashboard. I will not add a composite score on top of it. Composites hide the hole you needed to see.

What this does not prove

This protocol does not replace participant research. It does not turn a free model into a rater you can cite. It does not give you a severity number you can paste into a bug tracker. If you need statistically valid usability scores, do not use this. If you are writing regulated medical or financial copy, do not use a free server as your review room. If you cannot name a human owner, do not run the model at all.

I also will not pretend a passing calibration means the live hand-back works. The sample is a tuning fork. It is not the concert. Your production silence may last longer. Your control copy may wrap. Your user may be on a phone with one thumb. None of that is in the fork.

So which missing evidence should stop you this week? For me it is a missing human verdict on a known silent failure. Extra confidence scores are noise. They make the decimal look taller. They do not put a control on the screen.

Keep the sample in the record. Keep the owner on the card. Let the number wait.