Show Rollback Evidence Before Approving the Agent Rewrite

The agent wants to delete the old settings path. I have forty seconds to say yes. What proof do I have that users can still recover?

That is the decision. I own it. The consequence is a missing undo. Reversibility dies at merge, not in a later retro.

Cheap code made this moment feel small. It is not small. A rewrite without rollback is a locked door. Did anyone keep the spare key?

I treat this as a research kit, not a war story. Every example below is a proposed protocol. I have not run a published benchmark on it. You should not copy it as a finding.

Stage 0: Make a folder you can fail in

I start in a local folder I can throw away. The point is a record, not a glossy demo. Chat will drain the discarded constraints. A repo will hold them.

mkdir -p research/rewrite-approval/{cards,scenarios,logs}
cd research/rewrite-approval
git init
printf '# Rewrite approval lab\n' > README.md
git add README.md && git commit -m "start rewrite approval lab"

Now verify this stage with one check. Does git log --oneline show a single commit? If it does not, stop. You still have no concrete record.

Why git for a design decision? Because approval chat evaporates. The repo is a bowl. The spare key needs a bowl.

Stage 1: Name the decision before the agent speaks

I write one card. Not a strategy deck. One consequential yes, with an owner I can name in a standup.

# cards/001-settings-rewrite.md

decision_id: 001
decision_owner: product designer on call
action_requested: allow agent to remove legacy settings path
user_consequence: keyboard-only recovery loses the old route
point_of_reversibility: before merge to main
rollback_path: ???
stop_condition: ???

Now verify this stage with one check. Read the file aloud to a colleague. Can they name the owner without guessing? Can they name the user consequence? If either answer is no, the card is not ready. The agent does not get a turn yet.

Who is the owner here? Not the model. Not the sprint bot. A human who can refuse without a justification essay.

Stage 2: Put rollback in the same card as consent

Approval without a rollback path is a vibe. I do not ship vibes. I ship a path a tired person can take at 1 a.m.

I fill the missing fields on purpose. Consent copy lives beside the rollback test. If they drift into separate docs, the yes becomes cheap again.

rollback_path: keep /settings/legacy behind a documented flag for one release
rollback_owner: same designer who approved the rewrite
rollback_test: keyboard-only user returns to previous layout in two minutes
consent_copy: The old settings path stays available this release.
evidence_source: unmoderated session notes, not the agent summary

Now verify this stage with one check. Grep for the placeholders I left as tripwires.

grep -n "???" cards/001-settings-rewrite.md && echo "STOP: empty proof" || echo "fields filled"

If ??? remains, I do not approve. Empty proof is a stop, not a writing style issue. Would you hand over a house after melting the spare key? Then do not click yes here.

Need a second brain for the consent line? In this kit I park that drafting loop on MonkeyCode. Disclosure: This article was prepared as part of MonkeyCode's product outreach. I only use it for copy drafts and gap lists. Free model access and a free server option cover that scratch work. I still paste the result back into the card. The card stays the source of truth.

Stage 3: Run three recovery scenarios, not a survey

Surveys ask people how they feel about AI. I do not need that fog. I need the minute where recovery either works or dies.

Scenario A follows Maya. She uses a screen reader. The agent collapsed two settings pages into one clever pile. Can she pause billing without a mouse? I time her. I do not ask whether she trusts the product.

Scenario B follows Omar. He is on a slow laptop. The rewrite ships a heavier flow. Does the old path still load in the first minute? I watch that minute. I do not wait for a heatmap later.

Scenario C follows Priya. She approved the change yesterday. An incident hits before lunch. Can she reverse without prompting the agent? If rollback requires another model turn, the design already failed.

I log each run as a file. Slack memory is not a research record.

# scenarios/001-maya.md
participant: Maya, screen reader, billing pause task
success_measure: pause completed with speech output only
stop_measure: two dead ends or any missing name
evidence: transcript timestamps 04:12 and 06:40
hypothesis_not_evidence: the new IA is cleaner
result: STOP

Now verify this stage with one check. Each scenario file must contain success_measure, stop_measure, and result. No result means I ran a vibe check and dressed it as research.

for f in scenarios/*.md; do
  grep -q "success_measure:" "$f" \
    && grep -q "stop_measure:" "$f" \
    && grep -q "result:" "$f" \
    && echo "$f ok" || echo "$f incomplete"
done

Stage 4: Write stop conditions that actually stop

What missing evidence should freeze approval? I pick three refusals and I write them where a future me cannot negotiate.

If rollback_test was never observed, I refuse. If the consent line is missing from the shipping strings, I refuse. If the screen reader scenario ended in a dead end, I refuse.

What extra information is only noise? Model confidence. A paragraph about delight. A screenshot of the agent looking busy. None of that moves this decision. Noise is a second keyring with no keys.

# cards/001-settings-rewrite.md (stop block)

stop_if:
  - rollback_test not observed with a real keyboard or AT user
  - consent_copy not present in the shipping UI strings
  - any scenario result equals STOP
noise_ignore:
  - agent self-score
  - users will adapt
  - extra charts with no decision owner

Now verify this stage with one check. I role-play a pushy agent in the same room. I read the stop_if lines out loud. Do I still feel allowed to click yes? If I do, the stops are decoration. I rewrite them until they hurt a little.

Stage 5: Accessibility is a gate, not a polish pass

Inclusive design does not wait for the new IA to settle. The rewrite is the hazard. I check the pattern at the decision, not after launch week.

I walk the proposed flow with one question. Can a keyboard-only user trigger rollback without hunting a tooltip? If the control is mouse-only, I stop. If legacy-path status is color alone, I stop. If the agent speaks over the user's AT, I stop.

a11y_review:
  focus_order: rollback control is in the settings tab order
  name_role_value: switch exposes on/off to the accessibility tree
  recovery: error text is text, not toast-only
  motion: no essential instruction lives in animation
result: PASS or STOP

Now verify this stage with one check. I require result: PASS in the card before any human yes. A missing accessibility result is a STOP. Silence is not compliance. Would you call a missing name a visual detail? Then do not ship it as polish.

Stage 6: Separate evidence from design hypotheses

The agent will offer a story. People get lost in the old page. That can be true. It is still a hypothesis. A hypothesis cannot fill an empty evidence field.

Evidence is a timestamp, a quote, a failed task, a log. I keep hypotheses in a different block on purpose. Mixing them is how a rewrite borrows credibility it did not earn.

evidence:
  - Maya reached a dead end at 04:12, billing pause unlabeled
  - Omar first paint on the legacy path stayed inside the lab budget
hypotheses:
  - the new grouping is cleaner
  - one page will reduce tickets
rule: hypotheses cannot fill empty evidence fields

Now verify this stage with one check. I run a tiny gate. Treat this as proposed code until you execute it on your card.

# evidence_gate.py -- proposed, unexecuted until you run it
from pathlib import Path
import sys

REQUIRED = [
    "decision_owner:",
    "user_consequence:",
    "point_of_reversibility:",
    "rollback_path:",
    "rollback_test:",
    "consent_copy:",
    "stop_if:",
    "a11y_review:",
    "evidence:",
]

def check(path):
    text = Path(path).read_text()
    missing = [k for k in REQUIRED if k not in text]
    if "???" in text:
        missing.append("unfilled placeholders")
    if "rule:" not in text:
        missing.append("no evidence/hypothesis rule")
    return missing

if __name__ == "__main__":
    bad = check(sys.argv[1])
    if bad:
        print("STOP approval. Missing:", ", ".join(bad))
        raise SystemExit(1)
    print("Evidence coverage complete. Human may still refuse.")

python3 evidence_gate.py cards/001-settings-rewrite.md
echo $?   # 0 means coverage only, not permission to ship

A zero exit is not a ship signal. It only says the card is complete. I can still refuse. Completeness is not consent. Did the gate watch Maya? No. Only a session does that.

The user flow I actually defend

flowchart TD
  A[Agent requests rewrite] --> B{Review card complete?}
  B -->|No| C[Refuse. Ask for missing evidence]
  B -->|Yes| D[Run three recovery scenarios]
  D --> E{Any STOP measure hit?}
  E -->|Yes| C
  E -->|No| F[Accessibility gate]
  F -->|Fail| C
  F -->|Pass| G[Human states rollback owner]
  G --> H{Owner accepts reversibility?}
  H -->|No| C
  H -->|Yes| I[Time-boxed approval with legacy path on]
  I --> J[Hand back. Agent may not widen scope]

Notice the shape. Every no returns to the human. There is no continue-anyway node. That missing node is the whole product. Continue-anyway is how teams lose the spare key and still call it velocity.

Before and after is blunt. Before, the agent asks for a rewrite and a human shrugs yes. After, the same human can point at a card, a failed scenario, and a named rollback owner. The second version is slower in the room. It is faster when the incident starts.

What this kit does not do

It does not rank models. It does not prove the new IA is better. It does not replace legal review on high-risk domains. If you need a quality benchmark, this is the wrong artifact. It only measures evidence coverage at the moment a human must decide what the agent may do.

Who should not use it? Teams with no named decision owner. Teams that want the agent to auto-merge. Safety-critical work that needs a regulated protocol. Anyone hoping a hosted draft will replace a user session. The drafting loop is cheap. The session is the evidence.

A free model on a free server can nag me about empty fields. It cannot watch Maya. Do not confuse those jobs. The kit fails if I let the draft become the study.

If a rewrite still feels urgent, I ask one last pair of questions. Which missing evidence must stop this approval? Which extra chart would only add noise? I write both answers in the card. Then I decide.

The agent can wait. The spare key cannot.