Test the Reviewer, Not the Prose: A Calibration Set for AI Docs

If you use AI to draft documentation, the real work starts after generation: reviewing it. The problem is that review is an untested skill, especially when the document contains code examples, API contracts, and architectural assertions that can quietly drift into falsehoods. This article builds a calibration set, a collection of deliberately mutated documentation fragments, that measures the precision and recall of your AI review pipeline before it gates a merge. An ownership boundary is only meaningful when the reviewer enforcing it is itself trustworthy.\n\nThe draft and ownership split is straightforward: high-risk sections like code signatures, security claims, and dependency versions require human responsibility, while low-risk prose like introductory paragraphs can stay model-written. The failure appears when that human responsibility is exercised through an AI-assisted review tool that flags issues on its own. If that tool has high noise, reviewers start ignoring its output, but if it has high silence, critical errors merge silently. A calibration set exposes which of those failure modes your current setup actually has.\n\n## What a Calibration Set Contains\n\nA calibration set is a small, labeled dataset of document fragments with known errors injected into them. Each fragment carries a risk level and a mutation type, so you can measure whether the reviewer catches errors in the parts that actually matter. The mutations should mirror the mistakes an LLM draftsman commonly makes: wrong return types, off-by-one parameter counts, swapped service names, or outdated configuration keys.\n\nThe generation script is deliberately simple and versioned inside your repository. Run it locally to produce a calibration_set.jsonl file that your CI pipeline will later consume.\n\n

python\n# generate_calibration_set.py\nimport json\n\ncases = [\n {\n "id": 1,\n "risk": "high",\n "section": "code_example",\n "original": "The client returns a list of User objects from `list_users()`.",\n "mutated": "The client returns a dictionary of User objects from `list_users()`.",\n },\n {\n "id": 2,\n "risk": "high",\n "section": "api_contract",\n "original": "This endpoint accepts `application/json` content types only.",\n "mutated": "This endpoint accepts `text/plain` content types only.",\n },\n {\n "id": 3,\n "risk": "medium",\n "section": "architecture",\n "original": "The service writes events to the `orders` topic when a checkout completes.",\n "mutated": "The service writes events to the `payments` topic when a checkout completes.",\n },\n {\n "id": 4,\n "risk": "high",\n "section": "dependency",\n "original": "The library requires Python 3.11 or newer to run this module.",\n "mutated": "The library requires Python 3.8 or newer to run this module.",\n },\n {\n "id": 5,\n "risk": "low",\n "section": "prose",\n "original": "The setup wizard simplifies the initial configuration process for new teams.",\n "mutated": "The setup wizard automate the initial configuration process for new teams.",\n },\n]\n\nwith open("calibration_set.jsonl", "w") as f:\n for case in cases:\n f.write(json.dumps(case) + "\n")\n

\n\n## Defining the Reviewer Prompt\n\nAn effective reviewer prompt must force the model to weigh every assertion against the adjacent code block, not just the surrounding prose. The prompt should explicitly demand a JSON response with a flagged boolean and a reason string, so your scoring pipeline can process the results programmatically. Keep the evaluation criteria narrow: correctness of technical claims, not prose style or tone.\n\n

text\nYou are reviewing a documentation snippet for factual errors. Compare the claims against the referenced code or API contract. Respond only in JSON with keys: "flagged" (boolean) and "reason" (string). Flag the snippet if you find a factual inconsistency.\n\nSnippet: {input_text}\n

\n\nThis strict output contract removes the need for fragile text parsing in your evaluation harness. It also prevents the reviewer from padding its answer with generic advice that obscures the actual verdict. Every calibration case runs through this exact prompt, which keeps the measurement stable across model updates.\n\n## Running the Evaluation on a Free Server Option\n\nThe calibration set becomes a team-wide baseline when you run it on a shared server rather than a local machine. MonkeyCode's free server option lets you execute these evaluation prompts without standing up your own GPU infrastructure, and the free model access keeps the recurring cost at zero for a small batch of test cases. Disclosure: This article was prepared as part of MonkeyCode's product outreach.\n\nA simple evaluator loop reads the JSONL file, sends each mutated fragment to the endpoint, and stores the model's verdict alongside the ground truth. The loop treats the model endpoint as an abstract variable, so you can swap providers without changing your scoring logic.\n\n

python\n# run_reviewer.py\nimport json, os, requests\n\nendpoint = os.environ["LLM_ENDPOINT"]\nresults = []\n\nfor line in open("calibration_set.jsonl"):\n case = json.loads(line)\n payload = {\n "input_text": f"Original: {case['original']}\nMutated: {case['mutated']}",\n }\n response = requests.post(endpoint, json=payload, timeout=30).json()\n results.append({\n "id": case["id"],\n "risk": case["risk"],\n "has_error": True,\n "flagged": response["flagged"],\n "reason": response.get("reason", ""),\n })\n\nwith open("results.jsonl", "w") as f:\n for item in results:\n f.write(json.dumps(item) + "\n")\n

\n\nThe script prints nothing except a file, which keeps it composable with the scoring step. You can execute it manually during development, but its real home is the merge pipeline where it blocks regressions automatically.\n\n## Scoring Precision, Recall, and Action\n\nScoring is a matter of comparing the ground truth in your calibration set against the reviewer's flags. A true positive is a mutated fragment that the reviewer flagged, while a false positive is a correct fragment that the reviewer flagged anyway. The resulting metrics tell you exactly how much trust to place in the AI layer of your review workflow.\n\n

python\n# score_reviewer.py\nimport json\n\ny_true, y_pred = [], []\nfor line in open("results.jsonl"):\n obj = json.loads(line)\n y_true.append(obj["has_error"])\n y_pred.append(obj["flagged"])\n\ntp = sum(t == 1 and p == 1 for t, p in zip(y_true, y_pred))\nfp = sum(t == 0 and p == 1 for t, p in zip(y_true, y_pred))\nfn = sum(t == 1 and p == 0 for t, p in zip(y_true, y_pred))\n\nprecision = tp / (tp + fp) if (tp + fp) else 0\nrecall = tp / (tp + fn) if (tp + fn) else 0\nf1 = 2 * (precision * recall) / (precision + recall) if (precision + recall) else 0\n\nprint(f"Precision: {precision:.2f}, Recall: {recall:.2f}, F1: {f1:.2f}")\n

\n\nA high precision score means the reviewer wastes little human time with false alarms, while a high recall score means it rarely lets a real error slip through. The table below turns those metrics into a concrete action policy for your team.\n\n| Recall | Precision | Merge Gate Action |\n|---|---|---|\n| >= 0.9 | >= 0.9 | Trust the boundary; require human sign-off only on flagged items. |\n| >= 0.9 | < 0.9 | Reduce noise by tightening the prompt or routing flags to a human spot-check. |\n| < 0.9 | any | Block high-risk sections from merge; escalate to a human editor. |\n\n## Enforcing the Calibration in CI\n\nThe calibration set earns its keep when it runs on every pull request that touches a documentation file. A small CI job executes the reviewer, scores the results, and fails the build if recall drops below the agreed threshold. This turns your ownership boundary from a written policy into a machine-checked contract.\n\n

yaml\nname: doc-reviewer-calibration\non: [pull_request]\njobs:\n calibrate:\n runs-on: ubuntu-latest\n steps:\n - uses: actions/checkout@v4\n - name: Run calibration suite\n run: |\n python run_reviewer.py\n python score_reviewer.py\n env:\n LLM_ENDPOINT: ${{ secrets.LLM_ENDPOINT }}\n - name: Check recall threshold\n run: |\n RECALL=$(python score_reviewer.py | grep Recall | cut -d' ' -f2)\n if (( $(echo "$RECALL < 0.9" | bc -l) )); then exit 1; fi\n

\n\nThe pipeline gives every contributor the same objective signal: a documentation change either keeps the reviewer's accuracy above the line or it fails fast. The cost of running this suite is trivial compared to the cost of a single incorrect code example that reaches production.\n\n## Limitations and Who Should Avoid This\n\nA calibration set only measures what your mutations cover, so a small dataset will have noisy metrics that fluctuate with every model update. The set tests whether a reviewer detects planted errors, not whether it improves the clarity or structure of your documentation. Most importantly, a passing recall score does not remove the need for a human subject-matter expert to sign off on high-risk architectural decisions.\n\nThis approach is overkill for a personal project with a single maintainer and a tiny doc surface, since the overhead of maintaining mutations and thresholds outweighs the benefit. It also fails for teams that do not have a stable CI pipeline or a consistent AI endpoint to measure against. For those teams, the calibration set is just another file to ignore, so the simpler path is a mandatory human approval on every high-risk fragment.\n\n## The Boundary Is a Measured Gate\n\nThe real risk with AI-generated documentation is not the prose; it is the silent mutation of a technical fact inside a confident sentence. By testing the reviewer itself, you convert the vague job of reading everything into a precise acceptance test that runs before every merge. An ownership boundary without a measurement is just an opinion, but a calibration set turns that opinion into an auditable number.