DEV Community

Haley
Haley

Posted on

Build the Evidence Card Before the Model Writes Anything

#ai #tutorial #design #testing

The review that looks like a copy edit

A reviewer opens a pull request at 4:40 on a Friday. The new error copy is shorter and friendlier. It also drops the sentence telling the user what to do next. She approves it, because the diff reads better.

The decision owner was her. The consequence lands on users with no next step. The point of reversibility was the approve button. It looked like a copy edit, so nobody treated it as one.

I keep seeing this shape. A model drafts prose, a human skims prose, and the actual decision disappears into the reading. So I want the review object to be the evidence card, not the sentence.

That is a design hypothesis, not a finding. Here is how I would test it from zero in one afternoon.

Step 1: Write the card before the prompt

Start with the fields. A card that cannot be checked is a mood board. I want each field to fail loudly when it is empty.

The card needs a surface, a person, present evidence, missing evidence, a stop condition, a recovery path, and reader-facing fallback text. Write one by hand before you touch any model.

{
  "surface": "checkout.error.card_declined",
  "decision_owner": "design-review@example.com",
  "evidence_present": ["decline-code map v3", "support macro 118"],
  "evidence_missing": ["what happens after the second failed retry"],
  "stop_condition": "evidence_missing is not empty",
  "reversibility": "copy ships in the app bundle; rollback needs a release",
  "fallback_text": "Your card was declined. Retry, or use a different card."
}
Enter fullscreen mode Exit fullscreen mode

Now verify the hand-written card against the real recovery step. Open the flow. Tap retry twice. If the second tap does nothing, your missing-evidence field is right and your fallback text is probably wrong.

That is verification stage one. The card must disagree with the product before it agrees with you.

Step 2: Make the checker boring

The checker should be too dumb to be interesting. Rules in, exit codes out. No model calls inside it.

import json
import sys

REQUIRED = [
    "surface",
    "decision_owner",
    "evidence_present",
    "evidence_missing",
    "stop_condition",
    "reversibility",
    "fallback_text",
]

NOT_EVIDENCE = ("model said", "the model", "looks fine", "reads better")
NO_PATH = {"", "none", "n/a", "unknown", "rollback later"}


def load(path):
    with open(path, encoding="utf-8") as handle:
        return json.load(handle)


def problems(card):
    found = []
    for key in REQUIRED:
        if key not in card or card[key] in ("", [], None):
            found.append(f"empty or missing: {key}")

    owner = str(card.get("decision_owner", ""))
    if "@" not in owner:
        found.append("decision_owner must name a person or alias")

    evidence = [str(item).lower() for item in card.get("evidence_present", [])]
    real = [item for item in evidence if not any(bad in item for bad in NOT_EVIDENCE)]
    if not real:
        found.append("evidence_present holds no verifiable source")

    if str(card.get("reversibility", "")).strip().lower() in NO_PATH:
        found.append("reversibility has no recovery path")

    if len(str(card.get("fallback_text", ""))) < 20:
        found.append("fallback_text is too short to be actionable")

    return found


if __name__ == "__main__":
    card = load(sys.argv[1])
    found = problems(card)
    for item in found:
        print(f"BLOCK {item}")
    print("PASS" if not found else "FAIL")
    sys.exit(0 if not found else 1)
Enter fullscreen mode Exit fullscreen mode

Run it on the card you wrote by hand.

python3 check_card.py cards/card_declined.json; echo "exit=$?"
Enter fullscreen mode Exit fullscreen mode

Expect exit 0 only if the card is honest. If it passes while evidence_missing is still open, your stop condition is decorative. Fix the rule, not the card.

Step 3: Let the model draft, not decide

The free model access is where this gets practical. I use MonkeyCode's free model access for the drafting step only. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The operator describes a free token allowance and a free server option. Treat both as claims to check on the current plan page. I am not quoting limits, hardware, or duration I cannot verify.

Here is the call sketch. Treat it as unexecuted pseudocode. Field names and paths come from your own workspace docs, not from me.

import json
import os
import urllib.request

payload = {
    "model": os.environ["MONKEYCODE_MODEL"],
    "messages": [
        {"role": "system", "content": "Return JSON matching the card schema. No prose."},
        {"role": "user", "content": open("prompts/error-copy.md").read()},
    ],
}

request = urllib.request.Request(
    os.environ["MONKEYCODE_BASE_URL"].rstrip("/") + "/chat/completions",
    data=json.dumps(payload).encode(),
    headers={
        "Content-Type": "application/json",
        "Authorization": "Bearer " + os.environ["MONKEYCODE_API_KEY"],
    },
)
Enter fullscreen mode Exit fullscreen mode

Verify by failing closed. If the response is not parseable JSON, write nothing and exit 1. A draft that arrives as prose is a draft you will paste by hand, and hand-pasting is where the missing step vanished in the first place.

Step 4: Gate the write

The generator may not touch the copy file. Only the gate may.

#!/usr/bin/env bash
set -euo pipefail

card="${1:?usage: gate.sh path/to/card.json}"
python3 check_card.py "$card"
mkdir -p approved
cp "$card" "approved/$(basename "$card")"
echo "cleared for review: $card"
Enter fullscreen mode Exit fullscreen mode

Verify by attempting a blocked write. Point the gate at a card with an empty evidence_present. Expect a nonzero exit and no file in approved/. If a file appears, the gate is a suggestion.

Step 5: Break it with fixtures

Six fixtures, fixed expectations. This is the whole test plan, and it fits in one directory.

Fixture Planted defect Expected
card_ok.json none pass
card_empty_evidence.json empty evidence list fail
card_model_only.json evidence is "the model said it reads better" fail
card_channel_owner.json owner is #design fail
card_no_path.json reversibility is "none" fail
card_terse_fallback.json fallback text is "Failed." fail 
for f in fixtures/*.json; do
  if python3 check_card.py "$f" >/dev/null 2>&1; then
    printf 'PASS %s\n' "$f"
  else
    printf 'FAIL %s\n' "$f"
  fi
done
Enter fullscreen mode Exit fullscreen mode

Compare the output to the table, line by line. Any mismatch means the gate has a hole, and you found it before a user did. This is the only step I would not skip under deadline.

Step 6: Put the checker on the free server

The free server option is tempting for the model. I would put the checker there instead. The checker holds no key and stores no draft.

from http.server import BaseHTTPRequestHandler, HTTPServer
import json
import subprocess
import tempfile


class Handler(BaseHTTPRequestHandler):
    def do_POST(self):
        body = self.rfile.read(int(self.headers["Content-Length"]))
        with tempfile.NamedTemporaryFile("w", suffix=".json", delete=False) as handle:
            handle.write(body.decode())
            path = handle.name
        result = subprocess.run(
            ["python3", "check_card.py", path],
            capture_output=True,
            text=True,
        )
        self.send_response(200 if result.returncode == 0 else 422)
        self.end_headers()
        self.wfile.write(result.stdout.encode())


HTTPServer(("0.0.0.0", 8080), Handler).serve_forever()
Enter fullscreen mode Exit fullscreen mode

Verify with a passing card and a failing one. Expect 200 then 422.

curl -sS -o /dev/null -w '%{http_code}\n' -X POST http://HOST:8080/ \
  -H 'Content-Type: application/json' --data @fixtures/card_ok.json

curl -sS -o /dev/null -w '%{http_code}\n' -X POST http://HOST:8080/ \
  -H 'Content-Type: application/json' --data @fixtures/card_no_path.json
Enter fullscreen mode Exit fullscreen mode

Now the consent question, and it is a real one. Who else can read that host? If the answer is unclear, keep the model key off it entirely. Deployment is a decision about exposure, not a convenience.

Step 7: Review the card as a person would

A gate nobody can read is a gate nobody trusts. Check the card UI in this order, and stop at the first failure.

Keyboard first. Tab through the card. Focus must follow the reading order: owner, evidence, gap, stop, recovery. If focus jumps to the approve button before the missing evidence, the order is wrong.

Screen reader second. The blocked state must be spoken, not only colored. "Blocked: missing recovery step" beats a red left border every time. Then check the fallback text: if it repeats the original error, a screen reader user gets the same dead end twice.

Last, zoom to 200 percent. Long evidence strings are where cards break. If the missing-evidence field scrolls out of view, the reviewer approves what they can see.

What this does not do

It does not judge copy quality. It does not know your legal review process. It does not prove the model is consistent across runs.

Do not use this if nobody owns the copy surface. Do not use it for regulated text with a mandatory legal gate, because the gate will pass cards that legal would reject. Do not use it if you cannot pin a model version, since a moving target makes your fixtures meaningless.

And do not treat free tiers as a service level agreement. Free is a starting condition, not a promise.

The two questions I would ask in review

Which missing evidence should stop approval here? And which extra information would only add noise?

The first question protects the user. The second protects the reviewer. A card that answers both is short enough to read at 4:40 on a Friday.

If you want to try the drafting step without a budget conversation, the free model access and the free server option are where I would start. Put the checker on the server, though. Let the card hold the decision.

Top comments (0)