My evaluation loop only closed when I stopped scoring on synthetic inputs

I spent the first week of shipping my document converters grading them on tidy fixtures: a clean HTML page, a well-formed PDF, a CSV with no merged cells. Scores looked great. Then real users fed it a 40-page contract with scanned exhibits, a slide deck where every table was an image, and a spreadsheet with three header rows. The eval scores said 94%, the downstream RAG answers said garbage.

The loop that actually taught me something had four parts. First, collect failures from production, not from my test suite — the weird inputs are the spec. Second, label failures by cause, not by symptom: "table extraction dropped" is not the same as "heading levels collapsed" even when both lower the score. Third, re-run the eval set every time the parser changes, because a regression that passes synthetic tests but fails on last week's failures is still a regression. Fourth, measure the thing the agent downstream actually cares about — answer fidelity in the RAG pipeline — not just string similarity to a golden markdown.

The hardest part is accepting that the eval set is a living artifact. I version it alongside the code and treat new failure classes as new rows, not as noise to filter out. When I finally packaged the whole thing as the universal entry point at https://x402.freeq.one/tools/document_to_markdown.html, the eval loop was the thing that made the "one door" claim honest instead of marketing.