Hand an AI agent a QA checklist and here's what you get: every item inspected top to bottom with perfectly uniform attention. And with attention spread uniformly, the places that actually break get missed.
Running QA across 600+ multilingual web tools with AI agents, I ended up at a design I'd describe as giving QA a prior probability. The entire implementation is markdown files.
The problem: static checklists distribute attention evenly
A veteran human QA engineer doesn't read a checklist evenly. They carry a heuristic — "this type of tool usually breaks here" — and aim their attention at the suspicious spots first.
Give an AI a checklist (mine catalogs 70+ known violation patterns) and that gradient of attention doesn't exist. It dutifully verifies everything, burns context doing it, and still walks past the frequent offenders.
The fix: accumulate violation history, predict before inspecting
What I did is almost embarrassingly simple.
Every violation found in QA gets logged to a knowledge base (a markdown file): which category of tool, which violation pattern. That's the accumulation side.
Then at QA start, before touching the checklist, the agent predicts the top 3–5 likely violations from the tool's category × historical frequency. My category-level predictions look like:
- Calculators: overexposed "real-time calculation" marketing copy / missing unit toggles / broken output-card layout
- Visual tools: initialization flicker / preview overflow / export-element size breakage
- Text tools: multilingual label overflow / hardcoded UI strings
The predictions miss sometimes. Even so, just aiming the first pass of visual attention at "places that actually broke before" cut the misses noticeably. In Bayesian terms: stop starting QA from a uniform distribution; start from a prior.
Two designs that compounded the effect
Kill everything grep can catch before visual QA starts
Some violation patterns are mechanically detectable: banned words in visible copy, inline onclick, hardcoded UI strings. Run those as one batch grep sweep before any visual inspection. The expensive check (eyes on pixels) should only cover what eyes alone can catch. Cheap checks first, expensive checks second — obvious sequencing that AI agents skip unless you force it.
Separate check and fix into different sessions and branches
Let the QA agent fix things in the same session and you contaminate the verdict: the desire to have "fixed it" bleeds into the pass/fail judgment. I once got a QA record where "PASS" and a list of unresolved issues coexisted in the same file.
Since then: the check role detects and reports, and never touches code. Fixes happen in a separate intent, separate branch, separate session. If the same finding survives three rounds, escalate to the human instead of continuing to fix (shelving is a human-only decision).
Worth noting: the "pass + unresolved issues" record corruption may be less likely with today's tooling. But the design principle — QA objectivity is guaranteed by not holding the fix permissions — stays valid after any specific cause is patched.
Implementation cost
No special tools. The violation catalog is one markdown file. The KB (violation history) is one markdown file, layered into always-read and dig-when-needed sections. Prediction and routing live in the skill file (the procedure doc) handed to the agent.
So: it's all just markdown. No training, no external database. One extra step — "consult history, predict" — inserted into the procedure. That one step made the QA qualitatively different from a static checklist.
Limits and caveats
A prior needs accumulation. Early on the KB is empty and this does nothing; it starts working after a few dozen QA runs of logged violations.
Sloppy categories produce sloppy predictions. I started with roughly "calculator / visual / text" granularity, and that was enough.
There's a decay risk in trusting predictions too much — the agent stops looking beyond the predicted 3–5. The predictions only reorder attention; the full sweep never gets skipped.
Verified: April–June 2026. Environment: Claude Code + custom skills/knowledge base (markdown) / QA across a multilingual web-tool fleet.
Top comments (0)