Show Observation Notes Before Approving Agent Usability Claims

Last Tuesday a design lead asked me to ship. An overnight agent had rewritten the first-run flow. The model called every step clear for everyone.

Clear for which living person, exactly? That approval is the entire product decision. The owner is design, not the model.

The consequence is a blocked or shamed first-time user. The last reversible point is the production flag. After that flag, you are apologizing in public.

I keep one analogy on the wall nearby. A free model is a confident understudy. An understudy can rehearse a difficult scene well.

An understudy is not the paying audience though. Why do teams forget that so fast? Critique became cheap, and slogans arrived attached.

Vibe-coded flows show up with usability medals already pinned. Someone still has to refuse the cheap medal. Should that someone be a prompt?

I split evidence from hypothesis on paper. Users will understand step two remains a guess. Three people described step two is evidence.

Mixing those two is how fiction ships. I will not bless fiction because it sounds kind. Kind copy can still trap a body.

This is a proposed protocol, not a finished study. I walk the review from zero on purpose. If a check fails, approval stops cold.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. When a team already has MonkeyCode free model access and a free server option, this protocol parks rehearsal prompts there only. It never parks recordings, consent forms, or live account data there.

That isolated box is a green room only. That isolated box is not a research lab. Cheap rehearsal is useful in a pinch. Cheap evidence remains a lie in these reviews.

Each stage has a command you can run. Each stage has a pass or a hard stop. If you skip a stop, you are not reviewing. You are decorating a guess with empty ceremony.

Stage 0: isolate the claim

Stage 0 creates a disposable review desk first. I want a folder that cannot deploy anything. A free server is fine for this scratch.

A laptop folder is also fine for this. The point is isolation from production flags. This isolation is the first accessibility feature we owe.

mkdir -p review/first-run-claim/notes
cd review/first-run-claim
cat > claim.yml <<'EOF'
claim_id: first-run-2026-09-17
decision_owner: design-lead
reversible_until: production-flag
claim_text: "First-run is clear for everyone."
evidence_status: missing
EOF
test -s claim.yml && echo STAGE0_PASS

Did STAGE0_PASS print on your terminal? If it did not print, you stop. You do not brief a model without a still claim.

The file is the decision sitting in daylight. Can you point at it with a finger? If you cannot, you do not have a review.

Stage 1: name a body, not a vibe

Stage 1 rewrites the claim without marketing words. I strike intuitive, seamless, and everyone on sight. I replace them with one observable action.

Can a new user complete invite to first save without help? That question has a body inside it. Bodies are what this review still owes.

cat > hypothesis.md <<'EOF'
Hypothesis, not evidence:
A new user can complete invite to first save.
They can recover if the invite link expires.
They can do this with a screen reader.
EOF
grep -Ei 'intuitive|seamless|everyone|clear' hypothesis.md && echo STAGE1_STOP || echo STAGE1_PASS

If STAGE1_STOP appears, I rewrite the file again. A hypothesis must name a task, a recovery, and a body. Medals do not belong in this file.

Stage 2: rehearse failure, never mint quotes

Stage 2 is the only place a free model belongs. I ask it to attack the hypothesis in the green room. I do not ask it to bless the flow.

Would you let an intern sign off accessibility work? Then do not let a model do it. Rehearsal helps. Rehearsal is not observation.

I paste the hypothesis into the free model there. My prompt is labeled as rehearsal on purpose. If the prompt flatters the flow, I have already failed.

This is a rehearsal, not a study.
List ways this first-run can fail.
Do not claim users were observed.
Do not invent quotes or disability experience.
Return failure modes only.

I save the output as rehearsal-failures.md. Then I check it like a suspicious receipt. Fake witnesses are a stop, not a style issue.

grep -Ei 'user said|participants?|we observed|told me' rehearsal-failures.md && echo STAGE2_STOP || echo STAGE2_PASS

If that grep finds fake observation language, I stop cold. The model tried to wear a participant badge. That badge is the missing evidence.

I also scan for invented assistive-tech autobiography. A model does not get to narrate a screen reader user's feelings. I put those sentences straight into the bin.

Stage 3: collect notes or collect nothing

Stage 3 is human observation, or it is nothing. I do not need a huge theater of a study. I need notes with a date, a person, and a task.

Two or three scenario sessions beat one glamorous summary. I use this scenario card every time. This card is a protocol, not a trophy.

# Scenario card: invite to first save
Stop if the person cannot consent in their own words.
Task: accept invite, land, create first save, leave.
Recovery: expire the invite, then resume.
Access: keyboard only, then one screen reader pass.
Notes field: what they did, not what I hoped.
Discarded quotes: keep them in this same file.

After each session I append notes by hand. I never let the model rewrite those notes. Why would I hand the evidence pen back to the intern?

test -s notes/session-01.md && test -s notes/session-02.md && echo STAGE3_PASS

No notes files means no approval, full stop. One session is a story you liked. Two sessions can still be luck, and that is fine.

I still will not approve on zero notes. Would you ship a bridge on a sketched load? Then do not ship a first-run on vibes.

Stage 4: sign a card that can refuse

Stage 4 is the decision card the human signs. I want fields that force a pause in the hand. I want discarded quotes kept in the same record.

Dropped frustration is often the real finding. If the card has no place for refusal, it is a poster. Bright posters do not govern agents in shipping.

# Decision card
Claim: new user completes invite to first save.
Owner: design-lead
Evidence: session-01, session-02 notes
Coverage: keyboard, one screen reader, expired invite
Hypothesis leftover: none claimed as fact
Discarded quotes: kept in notes/discarded.md
Reversibility: production flag still off
Stop: missing notes OR missing AT pass OR missing recovery
Approve: yes / no

I validate the card with a small script. This is research ops, not frontend implementation. Someone else can build the interface later.

# verify_decision_card.py
# Proposal: local check, not a production gate.
from pathlib import Path

required = [
    "Owner:",
    "Evidence:",
    "Coverage:",
    "Discarded quotes:",
    "Reversibility:",
    "Stop:",
]
text = Path("decision.md").read_text()
missing = [field for field in required if field not in text]
notes = list(Path("notes").glob("session-*.md"))
if missing:
    raise SystemExit(f"STOP missing fields: {missing}")
if len(notes) < 2:
    raise SystemExit("STOP missing observation notes")
if "production flag still off" not in text:
    raise SystemExit("STOP no reversibility")
print("STAGE4_PASS")

Run it before anyone pastes a ship message.

python3 verify_decision_card.py

If it exits nonzero, the claim stays a hypothesis. That is the entire point of the gate. Software should refuse a hollow yes.

The flow I actually review

Here is the user flow I hold against the card. Notice the hand-back before the flag. The model never crosses that line.

flowchart LR
  A[Agent drafts first-run] --> B[Human writes hypothesis]
  B --> C[Free model rehearsal only]
  C --> D{Fake observation language?}
  D -->|Yes| E[Stop and discard]
  D -->|No| F[Human sessions]
  F --> G{Notes plus AT plus recovery?}
  G -->|No| E
  G -->|Yes| H[Decision card]
  H --> I{Production flag still off?}
  I -->|Yes| J[Human approve or refuse]
  I -->|No| E

Accessibility is not a smell a model can claim. I require a keyboard pass in the coverage line. I also require one screen reader pass before yes.

I require consent copy the person can refuse out loud. If the agent added a we-watched-you sentence, I pull it. Surveillance copy is not research warmth.

Focus order, name, role, and value still matter in the live path. I do not treat those as a later polish ticket. If the rehearsal found a nameless button, Stage 3 must watch someone meet it.

Stops versus noise

What missing evidence should stop approval today? Missing session notes should stop it now. Missing recovery from an expired invite should stop it.

Missing assistive-tech coverage should stop it too. Missing consent language the person can decline should stop it. Those are stops, not leftover vibes.

What extra information is only noise here? Model confidence scores are noise. Extra generated personas are noise. A third rewrite of the same slogan is noise.

Another intern blessing the intern is noise. I throw those extras out of the card. Silence is cheaper than false coverage on a first-run.

Limits, recovery, and who should leave

This approach has limits I will not hide. A free model will still hallucinate disability. A free server is the wrong vault for recordings.

Two sessions will not map a whole market. The script only checks fields, not truth. Fields can be filled with fiction if a reviewer lies.

The protocol assumes a human who still cares. If nobody cares, the yaml is costume. Costume fails in the same old ways.

Who should not use this workflow at all? Teams replacing participants with prompts should not. Teams putting real user media on a shared free server should not.

Teams that need clinical or legal research methods should not. Teams hoping a tutorial will excuse skipped consent should not. If flow failure can harm someone, rehearsal is not enough.

Hire a proper study in those cases. I also will not use this to argue models code better than people. That debate misses the interface in front of a person.

Someone still decides what the product may do. That someone needs notes, not medals. Notes are slower. Notes are the job.

If a claim already shipped without notes, recover in public. Turn the flag off before you write sorry copy. Keep the discarded quotes in the record.

Write the gap on the decision card in daylight. Then rerun Stage 3 with real people. Do not ask the model to apologize in microcopy.

Ask a human to repair the path instead. Would I let a free model sit in the green room? Yes, as rehearsal only.

Would I let it sign the house tonight? No. That is the whole tutorial, and it should stay boring.

If you already have a free-model sandbox, run this protocol there before anyone calls the output research.