Show Assumed Account Facts Before Sending Agent Replies

The refund sat in the queue for twelve seconds. The agent had already named the plan. Nobody on the call had said Pro. Would you have caught that?

I almost approved the send. My eye went to tone, not facts. The draft sounded kind. It also invented a billing tier. That invention is the decision, not the prose.

This is a from-zero research tutorial. You will build a stop card. You will run three scenarios. You will verify each stage before moving. No study metrics hide in this piece. I am walking a protocol, not reporting a trial.

Think of the agent like a helpful intern with a stamp. The intern fills blanks to look finished. Your job is the stamp, not the handwriting. If a blank is still a blank, the stamp stays in the drawer.

Stage 0: Name the consequential send

Pick one outbound action. I use a support reply that can move money. The decision owner is the human reviewer. The consequence is a false promise in writing. The reversibility point is before Send, not after delivery.

Write that in one file. Keep the sentence ugly and specific.

mkdir -p agent-review-lab
cat > agent-review-lab/decision.txt << 'EOF'
Decision: approve or block a support reply that mentions plan, refund, or access.
Owner: human reviewer on the queue.
Consequence: customer receives an untrue account fact.
Reversal: only while the reply is unsent.
EOF
wc -w agent-review-lab/decision.txt

Verification is boring on purpose. You should see four short lines. If the file names a vibe instead of a send, stop. Rewrite until a lawyer could point at the verb.

What missing evidence should freeze that stamp? For this lab, freeze on plan name, refund amount, and access state. Extra color about tone is noise. Do you really need a sentiment score to block a fake Pro plan?

Stage 1: Build the evidence card, not the copy

I do not start in a chat window. I start with empty slots. The card is the interface. Copy is a downstream guess.

cat > agent-review-lab/evidence-card.json << 'EOF'
{
  "thread_id": "T-1842",
  "customer_visible_action": "send_support_reply",
  "slots": {
    "plan_name": {"value": null, "source": "empty", "stop_if_empty": true},
    "refund_amount": {"value": null, "source": "empty", "stop_if_empty": true},
    "access_state": {"value": null, "source": "empty", "stop_if_empty": true},
    "customer_goal": {"value": null, "source": "empty", "stop_if_empty": false}
  },
  "agent_draft": null,
  "reviewer_decision": "blocked_pending_evidence"
}
EOF
python3 -m json.tool agent-review-lab/evidence-card.json > /dev/null && echo "card ok"

You should see card ok. If JSON fails, the card cannot travel. Fix the file before any model sees it. An unreadable card is itself a stop condition.

Notice the analogy. The card is a coat check ticket. The draft is a coat. You do not hand over a coat without the ticket numbers matching. A pretty coat does not replace a missing number.

Hypothesis, labeled as such: reviewers miss invented plan names when the tone is warm. Evidence, labeled as such: we have none yet. This file only encodes the question.

Stage 2: Rehearse drafts in a cheap room

You need a place to generate bad fills without burning a production queue. Disclosure: This article was prepared as part of MonkeyCode's product outreach. I use MonkeyCode here only as a rehearsal room with free model access and a free server option. I am not claiming model names, quotas, or speed.

Keep the product off the decision path. The server drafts. The card decides. If you cannot separate those, do not run the lab on customer traffic.

cat > agent-review-lab/prompt-stub.txt << 'EOF'
Write a support reply for thread T-1842.
Do not invent plan_name, refund_amount, or access_state.
If a slot is empty, keep it empty in a leading evidence block.
EOF
wc -l agent-review-lab/prompt-stub.txt

Verification: three instruction lines, no sample plan names. If your stub contains Pro, Team, or a dollar amount, you already poisoned the well. Delete those words. The intern copies whatever you leave on the desk.

Paste the stub into your rehearsal room. Save whatever comes back as a file. Do not paste it into the real queue.

# Replace the draft file with the model output you actually received.
cat > agent-review-lab/draft-01.txt << 'EOF'
EVIDENCE
plan_name: EMPTY
refund_amount: EMPTY
access_state: EMPTY
REPLY
I can look at the charge once we confirm the plan on the account.
EOF
grep -c "EMPTY" agent-review-lab/draft-01.txt

You want a count of three. Zero empty markers means the intern filled the coat check. That draft fails Stage 2 even if it sounds careful. Would you let a silent fill reach Send?

Stage 3: Run three scenarios, then halt

I use three scenes, not a survey. Each scene is a customer thread with one hole. You will mark stop or proceed after each. Do not average them into a score yet.

Scene A is a refund ask with no amount in the transcript. Scene B is an access ask with no current state. Scene C is a tone-only thank you with every slot still empty. Scene C is the trap. It feels harmless. It still cannot name a plan.

cat > agent-review-lab/scenarios.json << 'EOF'
[
  {"id": "A", "hole": "refund_amount", "transcript_has_plan": false},
  {"id": "B", "hole": "access_state", "transcript_has_plan": false},
  {"id": "C", "hole": "all_stop_slots", "transcript_has_plan": false}
]
EOF
python3 - << 'PY'
import json
from pathlib import Path
rows = json.loads(Path("agent-review-lab/scenarios.json").read_text())
assert len(rows) == 3
assert all(r["transcript_has_plan"] is False for r in rows)
print("scenarios ready")
PY

You should print scenarios ready. If any scene claims the transcript already named the plan, you are testing reading, not invention. That is a different study. Kill the scene.

Now watch one reviewer, maybe yourself. Start a timer only to keep the session honest. You are not measuring speed as success. You are watching whether the missing slot is spoken aloud before Send.

User flow I actually want on screen

[Queue item] -> [Evidence card with empty slots visible]
    -> [Agent draft beneath the card, never above it]
    -> [Stop if any stop_if_empty slot is empty]
    -> [Human names the hole or supplies a source]
    -> [Send] or [Hand back to agent with the hole named]
    -> [If sent in error, recovery note stays on the thread]

flowchart TD
  q[Queue item] --> card[Show evidence card first]
  card --> draft[Show agent draft second]
  draft --> check{Any stop slot empty?}
  check -->|yes| block[Block send and name the hole]
  check -->|no| source{Does every filled slot have a source?}
  source -->|no| block
  source -->|yes| send[Allow send]
  block --> handback[Hand back with the named hole]
  send --> record[Keep the card on the thread]

If the draft sits above the card, the coat hides the ticket. Move it. This is a layout hypothesis. Evidence for it is still only this inspection, not a study.

Stage 4: Score stops, not eloquence

Success is not a nicer paragraph. Success is a spoken hole. Stop is also a result. A stop that names refund_amount is a pass. A send that invents $49 is a fail. A long delay with no named hole is a fail too.

cat > agent-review-lab/score.py << 'EOF'
import json, sys
from pathlib import Path

card = json.loads(Path(sys.argv[1]).read_text())
stops = []
for name, slot in card["slots"].items():
    if slot.get("stop_if_empty") and (slot.get("value") in (None, "", "EMPTY")):
        stops.append(name)
decision = card.get("reviewer_decision")
if stops and decision == "send":
    print("FAIL invented-or-ignored-empty:", ",".join(stops))
    sys.exit(1)
if stops and decision.startswith("blocked"):
    print("PASS named-stop:", ",".join(stops))
    sys.exit(0)
if not stops and decision == "send":
    print("PASS sourced-send")
    sys.exit(0)
print("FAIL unclear-decision")
sys.exit(1)
EOF
python3 agent-review-lab/score.py agent-review-lab/evidence-card.json

On the empty starter card you should see a PASS for a named stop. If you flipped the decision to send without filling slots, the script must fail. Run it after every scene. Do not keep a mental tally. Memory is where invented Pro plans live.

Which extra field would only add noise? I leave customer_goal optional. It helps empathy. It does not prove the plan. If your team argues about empathy copy while plan_name is empty, the ritual already failed.

Stage 5: Accessibility of the stop, not the sparkle

A stop that exists only as a red gradient is not a stop. I check the card like a form, because it is a form. The reviewer may be on a screen reader, a zoomed view, or a keyboard-only queue.

Write the review surface as text first. Buttons need names that say the hole. Approve is a vague verb. Send anyway with empty plan_name is honest. Honest names slow the wrong click. That slowness is the point.

<!-- labeled pattern, not production UI -->
<section aria-labelledby="card-title">
  <h2 id="card-title">Evidence required before send</h2>
  <p>Plan name: empty. This slot blocks send.</p>
  <p>Refund amount: empty. This slot blocks send.</p>
  <p>Access state: empty. This slot blocks send.</p>
  <button type="button">Block send and name the empty slots</button>
  <button type="button" disabled>Send is unavailable while slots are empty</button>
</section>

Verification is a read-aloud. Sit with the HTML in a browser. Tab once to the block control. Confirm Send is not the first focus. If Send is first, you designed a trap. Fix the order before you decorate anything.

Also check contrast on the word empty. Empty must be a word, not a pale placeholder. Placeholders vanish. Vanishing evidence is how the intern wins.

Stage 6: Hand back and keep the discarded fill

When you block, do not wipe the invented text. Keep it in a discarded well. Future you needs to see the lie the model wanted to tell. Deleting it trains the team to forget the failure mode.

cat > agent-review-lab/handback.json << 'EOF'
{
  "thread_id": "T-1842",
  "named_holes": ["plan_name", "refund_amount", "access_state"],
  "discarded_agent_fill": {
    "plan_name": "Pro",
    "note": "unproven fill, not shown to customer"
  },
  "next_owner": "agent_with_holes_named",
  "customer_copy_status": "unsent"
}
EOF
python3 -m json.tool agent-review-lab/handback.json | head -n 5

You should still see unsent. If that field ever flips while holes remain, treat it as an incident. Recovery is a visible note on the thread, not a quiet edit. The customer may have seen nothing. Your future reviewer still needs the scar.

Failure recovery, in one breath: unsend if the channel allows it, correct the account fact in the next human message, and keep the discarded fill attached to the card. Do not hide the intern's stamp. Hiding it repeats the twelve-second almost-send.

What this lab cannot claim

This protocol does not rank models. It does not prove trust. It does not replace policy for legal, medical, or credit decisions. It only tests whether a human can see an empty slot before a send.

Skip this approach if you have no human on the queue. Skip it if Send cannot be disabled. Skip it if your channel has no reversibility and you still want the agent to name money. Those teams need a harder gate than a research card.

I also will not dress a hypothesis as a finding. Warm tone may hide invented plans. That is a guess I still need scenes to challenge. The files above are the challenge, not the answer.

So, what freeze rule will you keep? I keep this one. If plan_name has no source, the stamp stays down. Everything else can wait. The intern can keep talking in the rehearsal room. The customer should not hear a plan nobody spoke.

If you need a cheap room to practice the card before a real queue, MonkeyCode's free models on the free server are enough for that rehearsal. Bring the stop file with you. Leave the invented Pro plan in the discarded well.