Vendor benchmarks are rightly distrusted: pick a friendly dataset, tune on the test set, round up, publish. I sell a document-extraction API (Kynth Core), I wanted to publish accuracy numbers, and I couldn't find a form of "trust me" that I would accept from someone else. So the benchmark became a repo instead of a claim.
The rules I couldn't let myself break
- Public datasets only, pinned. FATURA for invoices (synthetic — disclosed loudly), SROIE + CORD-v2 for receipts, FinTabNet for tables, and the Bankstatemently open benchmark for statements — where the ground truth is held server-side by a third party, so I literally cannot tune against it. Every dataset pinned by revision with SHA-256s.
- Subsets pre-registered. The seeded selection of document IDs was committed before any API was called. No document was dropped after seeing results.
- No fuzzy matching in headline numbers. Amounts to the cent, dates exact (ISO), strings after whitespace/punctuation normalization. ANLS is a secondary column only.
-
Raw responses committed. Every vendor's verbatim API output is in the repo — anyone can re-score everything with zero API keys (
pnpm replay && pnpm report), or challenge my adapters' normalization line by line. - CI runs it, not me. Fork PRs run keyless replay-scoring only, so nobody's keys leak.
Results
| Suite | Metric | Kynth Core | AWS Textract | Google DocAI |
|---|---|---|---|---|
| Invoices (FATURA, n=100) | field accuracy | 99.4% | 92.2% | 93.1% |
| Receipts (SROIE, n=100) | field accuracy | 88.8% | 68.8% | 52.8% |
| Receipt line items (CORD, n=100) | F1 | 41.9% | 77.1% | 36.6% |
| Receipt totals (CORD, n=100) | accuracy | 67.1% | 95.5% | 82.2% |
| Tables (FinTabNet, n=100) | TEDS | 0.791 | 0.836 | 0.365 |
Where Kynth lost, plainly: Textract beats it decisively on CORD line-item extraction (77.1% vs 41.9% F1) and receipt totals from the same set, and edges it on table structure (0.836 vs 0.791 TEDS) — where Kynth also logged 14 outright failures on hard tables, counted against it rather than dropped. Kynth is also roughly 10× slower per document than Textract. Kynth wins where fields need semantic understanding (invoice fields, messy scanned receipts); the hyperscalers win on dense structural extraction. That trade is the honest takeaway.
Notes that keep the table honest: Textract has no bank-statement product (N/A, not zero). Bank statements are pending entirely — the third-party evaluator currently 400s on its own published files; the n=5 parses are committed and will score when it's fixed. Veryfi and LlamaParse adapters are implemented but unscored — no credentials this cycle; I wasn't going to pay Veryfi's $500/mo minimum to benchmark a competitor. FATURA is synthetic, which flatters everyone equally, but flatters.
What I'd want you to take from this
Not "Kynth wins" — on three of five rows it doesn't. The claim is narrower: flat per-document pricing, billed only on success, with accuracy you can reproduce is a different product shape than per-page-per-attempt with a pipeline you own. The repo is MIT: doc-extract-bench. PRs that improve competitors' adapters are especially welcome — that's the failure mode a vendor benchmark can't recover from on its own.
Live scores: api.kynth.studio/benchmarks. The engine behind it runs 73 production apps; that story is on my site.
Top comments (1)
This is the protocol I've been missing. I've promised a WER benchmark in three different posts and never shipped it, because I couldn't find an honest way to do it. The rule that actually kills cherry-picking here is pre-registering the subset before the first API call, and speech-to-text has no equivalent of held-out ground truth for Quebec French: public French corpora skew toward France French, so picking the sentences is already a form of tuning. Your line about synthetic datasets flattering everyone equally has a direct audio twin too. Studio-quality recordings flatter every model the same way, hiding exactly the accents and noise conditions that break in the real world. If no public dataset covered your real user distribution, how would you have pre-registered a subset at all?