AI promoted every developer to reviewer. Most reviewers have no checklist, no authority, and no process. Teams need a playbook, not another opinion.
This playbook defines three roles, two handoffs, and one runbook. It fits on one page. Paste it into a wiki before the next model release.
Most teams evaluate AI models like they taste street food. One bite, one thumbs up, done. The next release resets everything.
Evaluation without ownership is a rumor. Someone tried a model, someone liked it, someone told a friend. Nobody can reproduce the result.
The playbook turns a rumor into a record. It turns a vibe into a verdict. That is the whole point.
The playbook fixes that with three roles. The Evaluator runs the tests and publishes evidence. The Gatekeeper approves or rejects the change.
The Maintainer owns the runbook and updates the wiki. One person can hold all three roles in a small team. The checklists still stay separate.
Separation of duties prevents blind spots. A person who writes the test should not approve the result. A person who approves should not edit the evidence.
Handoff one happens when the Evaluator publishes results. The Gatekeeper reviews evidence, not vibes. A failed gate means a rejected change.
Handoff two happens when the Gatekeeper approves. The Maintainer updates the wiki page and archives old results. The team always knows which model is current.
The runbook is a single script. It runs on a free server and calls a free model endpoint. Any developer can execute it in five minutes.
Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode's free tier currently includes a 10 million token allowance and a free server. The script below uses both, but works with any OpenAI-compatible endpoint.
#!/usr/bin/env bash
# evaluate.sh - one-page model evaluation runbook
set -euo pipefail
BASE_URL="${MONKEYCODE_BASE_URL:?set MONKEYCODE_BASE_URL}"
API_KEY="${MONKEYCODE_API_KEY:?set MONKEYCODE_API_KEY}"
MODEL="${1:?usage: evaluate.sh <model>}"
THRESHOLD="${THRESHOLD:-80}"
PASS=0
TOTAL=0
check() {
local prompt="$1" expected="$2"
TOTAL=$((TOTAL + 1))
local payload response
payload=$(jq -n --arg model "$MODEL" --arg prompt "$prompt" \
'{model: $model, messages: [{role: "user", content: $prompt}]}')
response=$(curl -s "$BASE_URL/chat/completions" \
-H "Authorization: Bearer $API_KEY" \
-H "Content-Type: application/json" \
-d "$payload" | jq -r '.choices[0].message.content')
if echo "$response" | grep -qi "$expected"; then
PASS=$((PASS + 1))
echo "PASS: $prompt"
else
echo "FAIL: $prompt"
echo " expected: $expected"
echo " got: ${response:0:120}"
fi
}
check "Return only the number 42." "42"
check "Explain what a UUID is in one sentence." "identifier"
check "Write a Python function that sums a list." "def "
SCORE=$((PASS * 100 / TOTAL))
echo "Score: $SCORE%"
echo "Date: $(date -u +%Y-%m-%d)"
echo "Model: $MODEL"
if [ "$SCORE" -lt "$THRESHOLD" ]; then
echo "Gate: FAILED"
exit 1
fi
echo "Gate: PASSED"
The script checks three things. Prompt one verifies exact output. Prompt two verifies domain vocabulary. Prompt three verifies code generation.
The script needs curl and jq. Both are standard on any Linux server. The free server already has them.
The score is a percentage. The threshold is a team decision. Eighty percent is a reasonable starting point.
The exit code is the handoff signal. Zero means the Gatekeeper can proceed. One means the Evaluator must investigate.
The results file is the evidence. It records the date, the model, and the score. It lives in the wiki next to the runbook.
Attach the raw output, not just the score. A score without output is a claim. A claim without evidence is a rumor.
The date stamp matters. Model releases change the landscape weekly. Re-run the script before every model change, not after an incident.
Set a cadence. Run the script weekly or after every model release. Pick the shorter interval.
Teams can extend the checks easily. Add a prompt that demands valid JSON. Add a latency measurement with curl -w. Add a tool-call test when the agent uses functions.
The wiki page keeps the process honest. Here is a template the Maintainer can paste.
# Model Evaluation Runbook
Owner: <Maintainer>
Last run: <date>
Current model: <alias>
Gate status: <PASSED|FAILED>
## Roles
Evaluator: <name>
Gatekeeper: <name>
Maintainer: <name>
## Evidence
- Date: <date>
- Score: <score>%
- Threshold: <threshold>%
- Command: `./evaluate.sh <model>`
- Output: <link to results>
## Decision
<Gatekeeper> approved or rejected <model> on <date>.
The template forces clarity. A one-page constraint removes the noise. Teams can see the gate status without reading a report.
This playbook has limits. It tests prompts, not real workloads. It checks substrings, not semantics.
It cannot catch a model that sounds right but computes wrong. Complex agentic flows need stronger evaluation. Strict compliance needs human review of every output.
This runbook is a gate, not a guarantee. Teams with a mature eval pipeline should skip it. Teams with per-request tracing needs should build something deeper.
The playbook works best for small teams. It works best for teams adopting their first AI coding assistant. It works best when a bad model costs a few wasted hours.
Teams that skip the playbook learn the same lesson twice. Teams that run it learn once. The wiki page is the memory.
Run the script against a free endpoint this week. The evidence will outlast the hype.
Top comments (0)