Show Coverage Gaps Before Approving an Agent Study Script

The Slack ping landed at 4:47 p.m. A final study script sat waiting in Slack. A free model had drafted every task overnight.

The first session was already booked for Monday. I almost typed a cheerful looks good. Did anyone freeze the live agent surface first?

I own this research as the product designer. A false usability claim would follow a lazy yes. The only clean reverse point sits before we recruit.

The agent gained a calendar-write tool yesterday afternoon. The script still only tested search and summary. We would watch people succeed at last week's product.

Then we would ship a write path nobody saw fail. That is not a study at all. That is a costume on a changed system.

This walkthrough is a proposed method only. I am not reporting a lab sample here. Treat every command as a template for your inventory.

The decision, named plainly

Someone must approve that script today. That someone is you or me. The missing evidence is coverage, not charm.

Which live tools have zero tasks attached? Which failure states have zero recovery beats? Which consent beats never appear in the packet?

If those gaps stay hidden, approval is theater. I keep one question on a sticky note. Which missing evidence should stop this study?

Extra color on tone of voice is noise. An untested write tool is not noise. Show the gaps before you ask for approval.

Stage 0. Set up a local coverage desk

I start on a clean directory, nothing fancy. You need Python, a shell, and PyYAML. This is research ops, not a frontend build.

mkdir -p agent-study-coverage/{inventory,script,matrix,cards}
cd agent-study-coverage
python3 -m pip install pyyaml
python3 -c "import yaml; print('yaml-ok')"
date -u +%Y-%m-%dT%H:%M:%SZ > inventory/frozen-at.txt

Verification is boring on purpose here. You must see yaml-ok print to stdout. You must see a UTC timestamp inside inventory/frozen-at.txt.

If either check fails, stop this tutorial. You cannot freeze a surface without a desk. Do not jump into the model-written script yet.

Stage 1. Freeze the live tool inventory

Do not start inside the pretty script. Start with production truth, written down. Who actually knows which tools are live today?

Now write the live tools by hand. Do not let a model invent them. A chatty paragraph will hide a missing tool.

cat > inventory/live-tools.yaml <<'EOF'
surface: calendar-agent
source: production-config-copy
tools:
  - id: search_events
    side_effect: none
    reversible: true
  - id: summarize_day
    side_effect: none
    reversible: true
  - id: create_event
    side_effect: writes_calendar
    reversible: false_until_undo
  - id: send_invite
    side_effect: emails_other_people
    reversible: false
consent_beats:
  - calendar_write
  - outbound_email
failure_states:
  - tool_timeout
  - partial_write
  - invite_already_sent
EOF

Then run a tiny check so the freeze is real.

python3 - <<'PY'
import yaml
d = yaml.safe_load(open("inventory/live-tools.yaml"))
assert d["tools"], "no tools frozen"
assert d["consent_beats"], "no consent beats frozen"
print("frozen tools:", len(d["tools"]))
print("consent beats:", len(d["consent_beats"]))
print("failure states:", len(d["failure_states"]))
PY

If that assert fires, you stop immediately. You have no study until tools are frozen. Why yaml instead of a slide deck?

Because ids do not flatter anyone in the room. A missing send_invite row stays painfully visible. A paragraph about the new calendar stuff is not.

Stage 2. Extract tasks from the drafted script

Drop the model-written script into script/draft.md. Keep the original wording untouched for now, always. Do not tidy the prose before this extraction.

I use a small extractor that stays deliberately dumb. Would you let a student grade their own exam? The drafted script does not get to score itself.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

When I need a sandbox for this extractor, I use MonkeyCode. Free model access and a free server option are enough. I still paste the live inventory in by hand.

The model never owns the freeze file. That boundary is the whole method. Now run the extractor against a stub draft.

# extract_tasks.py
import re, pathlib, json
text = pathlib.Path("script/draft.md").read_text()
tasks = []
for i, line in enumerate(text.splitlines(), 1):
    if re.search(r"^(task|prompt|ask the participant)", line.strip(), re.I):
        tasks.append({"line": i, "text": line.strip()})
pathlib.Path("script/tasks.json").write_text(json.dumps(tasks, indent=2))
print(f"extracted {len(tasks)} task lines")

cat > script/draft.md <<'EOF'
Task: Search next week's events and read them back.
Prompt: Ask the participant to summarize Tuesday.
Warm-up: explore freely around the home screen.
EOF
python3 extract_tasks.py
test -s script/tasks.json
python3 -c "import json; print(len(json.load(open('script/tasks.json'))))"

Verification is a headcount here, not a feeling. You need a task for each high-side-effect tool. Zero matching lines means you do not recruit.

Look at that stub and ask the coverage question. Does create_event appear even once? Does send_invite appear even once?

Stage 3. Build the coverage matrix

This matrix is the artifact, not a vibe. A written matrix makes each coverage gap undeniable. People argue with stories; they rarely argue with empty cells.

# coverage.py
import json, yaml, csv, sys
inv = yaml.safe_load(open("inventory/live-tools.yaml"))
tasks = json.load(open("script/tasks.json"))
blob = " ".join(t["text"].lower() for t in tasks)
rows = []
gaps = []
for tool in inv["tools"]:
    name = tool["id"].replace("_", " ")
    hit = tool["id"] in blob or name in blob
    if tool["id"] == "create_event":
        hit = hit or ("create" in blob and "event" in blob)
    if tool["id"] == "send_invite":
        hit = hit or "invite" in blob
    rows.append({
        "tool": tool["id"],
        "side_effect": tool["side_effect"],
        "reversible": str(tool["reversible"]),
        "covered": str(hit),
    })
    if not hit:
        gaps.append(tool["id"])
with open("matrix/coverage.csv", "w", newline="") as f:
    w = csv.DictWriter(f, fieldnames=rows[0].keys())
    w.writeheader()
    w.writerows(rows)
print("GAPS:", ", ".join(gaps) if gaps else "none")
sys.exit(1 if gaps else 0)

python3 coverage.py; echo "exit:$?"
cat matrix/coverage.csv

Exit code 1 means you stop approval. An exit 0 only means task language exists. It does not mean those tasks are any good.

I paste the csv into the review thread every time. The empty cells do the uncomfortable talking now. Do we still want Monday's sessions after this?

Here is the user flow I actually walk.

Moderator packet arrives
        |
        v
Designer opens frozen inventory
        |
        v
Extractor lists task lines
        |
        v
Matrix marks covered vs gap
        |
   gap? --yes--> STOP, rewrite tasks, do not recruit
        |
       no
        |
        v
Consent beats present in the packet?
        |
   no ---> STOP
        |
       yes
        |
        v
A11y session rehearsal
        |
   fail -> STOP
        |
       pass
        |
        v
Approval card with reversibility window

Notice the stop boxes along that path. Approval is a gated walk, not a mood. Where would you reverse if the mail already went out?

Stage 4. Write stop conditions into the card

A gap without a stop phrase still recruits. Someone will say we will watch for it. That sentence should never book a person, ever.

cat > cards/approval.yaml <<'EOF'
decision: approve_agent_study_script
owner: research_designer
reversible_until: recruit_mail_unsent
evidence:
  inventory_frozen: true
  coverage_csv: matrix/coverage.csv
  untested_tools: []
  untested_consent: []
  untested_failures: []
stop_if:
  - any_write_tool_has_zero_tasks
  - outbound_email_lacks_consent_beat
  - failure_state_has_no_recovery_line
  - keyboard_only_path_unrehearsed
hypotheses_not_evidence:
  - the model writes realistic tasks
  - five users will mention the new tool anyway
EOF

Fill untested_tools from the csv by hand first. Then let a script refuse a green card. A green card with gaps is a ritual bug.

python3 - <<'PY'
import csv, yaml, sys
gaps = [r["tool"] for r in csv.DictReader(open("matrix/coverage.csv")) if r["covered"] == "False"]
card = yaml.safe_load(open("cards/approval.yaml"))
card["evidence"]["untested_tools"] = gaps
open("cards/approval.yaml", "w").write(yaml.safe_dump(card, sort_keys=False))
print("untested_tools:", gaps)
if gaps:
    sys.exit("STOP: coverage gaps remain")
print("card green for tools only")
PY

The process must stay red on the stub draft. create_event and send_invite should appear as gaps. If the card prints green, your checker is lying.

I separate evidence from hypotheses on purpose here. Users will stumble onto the write tool remains a hope. Hope is not a task line in the script.

Stage 5. Rehearse the session as an interface

The participant is not a camera for our demo. The session itself is an interface with controls. What happens when those controls fail a keyboard?

I rehearse four beats before I approve recruitment. Can a keyboard-only participant confirm create_event today? Can a screen reader user hear draft versus sent?

Can the moderator pause the agent without a mouse-only toast? Is there spoken recovery after a partial write? Write the rehearsal down, not a slogan.

cat > cards/a11y-rehearsal.md <<'EOF'
# Session accessibility rehearsal
Keyboard-only confirm of create_event: PASS_OR_FAIL
Name and state of send_invite, draft vs sent: PASS_OR_FAIL
Spoken stop phrase that halts the agent: PASS_OR_FAIL
Recovery copy after partial_write: PASS_OR_FAIL
Stop rule: any FAIL blocks recruitment.
EOF

if grep -q "PASS_OR_FAIL" cards/a11y-rehearsal.md; then
  echo "STOP: rehearsal still blank"
  exit 1
fi
echo "rehearsal recorded"

That grep must fail after you rehearse. A blank rehearsal card is not inclusive design. Blank rows are postponement wearing an accessibility heading.

If the agent speaks over the screen reader, that is coverage too. The study would measure panic, not the write tool. I will not learn that on participant one.

Stage 6. Hand back when a session still hits a gap

Even a green matrix can lie to you later. A participant will wander into yesterday's new tool. Does your moderator have a hand-back line ready?

I table-read this out loud with them. If they paraphrase it into just keep going, we failed. The record has to keep the discarded path.

IF the participant reaches an untested tool
THEN moderator says:
  I am pausing the agent. That path is out of scope today.
THEN operator disables the tool for the rest of the session
THEN the record stores tool_id, timestamp, and side-effect state
THEN recovery:
  undo if reversible
  notify the other person if an invite already left

Verification is a spoken rehearsal, not a wiki page. Ask the moderator to say the pause line twice. Ask what they do if send_invite already fired.

Keep discarded paths in the concrete session record. A paused tool is evidence for the readout. It is not a mess to hide from stakeholders.

What this is not, and who should skip it

This is not a model bake-off of any kind. I am not ranking free models in this walkthrough. I am gating a human decision with missing evidence.

Do not use this workflow without a person who can freeze inventory. Do not use it for marketing testimonials dressed as research. Do not use it when the agent can spend money you cannot reverse.

Do not treat it as legal review of consent copy. The free sandbox does not freeze production for you. It does not attend the session or own the stop.

If your agent has no tool boundary, pick another method. This matrix needs ids, side effects, and reversibility. Without those fields, you are scoring prose again.

The sticky note, again

Which missing evidence should stop approval on your team? For me, an untested write tool stops the study. A missing consent beat for outbound email also stops it.

A keyboard-only fail on confirm stops recruitment too. Which extra information would only add noise here? Another adjective in the intro script, for starters.

Another joke in the warm-up adds nothing useful. Another claim that the model knows our product adds nothing. Show the coverage gaps first, then ask me to approve.

If you already keep a free-model sandbox, run this matrix there. Bring the csv to the review, not the vibe.