I published a benchmark of document-extraction APIs where one of the entrants is my own. That is a credibility problem you cannot write your way out of. The only fix is to make the numbers recomputable by a stranger who trusts nothing I say — including on the runs where my product loses.
The design decision that makes that possible is small: the runner never scores anything it just fetched.
Split the run in half at the raw response
Every vendor call writes one verbatim JSON file and stops there:
// results/raw/<vendor>/<dataset>/<docId>.json
const rec = { vendor, dataset, docId, at: new Date().toISOString(),
latencyMs: r.latencyMs, raw: r.raw };
writeJson(rawPath(vendor, dataset, id), rec);
// planning a run: anything already on disk is not re-fetched
const ids = allIds.filter((id) => !existsSync(rawPath(adapter.name, ds.name, id)));
Scoring is a separate entry point (--replay) that reads those files and never touches the network. All 1,210 raw responses are committed. Anyone can clone the repo, run pnpm replay && pnpm report, and regenerate the README tables with zero credentials and zero spend.
Three things fall out of that split, and only one of them was the goal.
Reproduction is keyless. That was the point.
Fixing a scoring bug is free. Metrics are where a benchmark is most likely to be wrong, and re-running 500 paid API calls to correct a normalizer is the kind of cost that quietly discourages fixing it. Here a metric fix is a replay.
Crashed runs resume for free. That existsSync filter was written for idempotency and turned out to be the thing that saved the run, because of what happened next.
The stall was not where I was looking
A full run wedged. No error, no exit, just a process sitting on one document forever. My first instinct was rate limiting, and I went looking at retry logic.
The retry logic was fine. It just never ran. My retry wrapper does three attempts with backoff, and the AWS SDK client had maxAttempts: 3 on top of that — but retries only fire when something fails, and a hung socket does not fail. It sits. Every layer of resilience I had was downstream of an error that was never going to arrive.
So the fix was deadlines, not retries. Every fetch in every adapter got signal: AbortSignal.timeout(120_000), including the 3-second job-poll loop, which got its own shorter one. The AWS client got an explicit NodeHttpHandler({ connectionTimeout: 10_000, requestTimeout: 120_000, socketTimeout: 120_000 }). Two commits, one for the fetch calls and one for the SDK client, because the SDK does not respect the same mechanism and I found it as a second stall a half hour later.
If you take one thing from this: a hung request is not a slow request, and no amount of retry configuration converts one into the other. The timeout is what gives maxAttempts something to attempt.
Pre-register the subset before the first call
Selection is the easiest place to cheat, usually without meaning to. So the subset ID lists were committed before the adapters could bill anything, generated by seeded shuffle with the seed in the manifest (20260710). The methodology fixes one rule that costs me points: no document is excluded after seeing results. A vendor failure is a failed document, not a dropped one.
Headline metrics are exact match after type-aware normalization — amounts within 0.005, dates to ISO, strings NFKC-lowercased with punctuation stripped. ANLS (fuzzy Levenshtein similarity) is computed, reported, and never allowed into a headline number, because a similarity threshold is a dial and I would be the one turning it.
What that honesty actually cost
On invoices my API leads at 99.4% field accuracy against 93.1% and 92.2% — at a median 12.8s per document versus Textract's 1.8s. On CORD-v2 line items it gets 41.9% F1 against Textract's 77.1%. On FinTabNet tables it scores 0.791 TEDS against 0.836, with 14 of 100 documents failing outright on model returned unparseable output.
Those last two are in the README, in bold, in the same table style as the wins. Publishing them is the only reason the first number means anything.
The extraction API benchmarked here is Kynth Core, and this is how we built it: https://kynth.studio
Top comments (0)