Shrijal Acharya

Posted on Sep 21

I Tested 7 Document Parsing APIs on Real Scanned PDFs (2026): Code and Results

#ai #programming #agents #productivity

Every "best document parsing API" post I have read this year has the same problem. Nobody actually calls the APIs.

They copy the pricing page, add a feature table, rank everything, and call it a day. Zero PDFs were harmed in the making of that content. 🥴

So I did something else. I took 11 real documents from public datasets. Invoices, a French bank statement, a photographed receipt, a handwritten cheque, two contracts, and a medical EOB.

Then I wrote one JSON schema for each document type and sent the exact same files through Extend, Reducto, LlamaExtract, AWS Textract, Mistral OCR, Claude, and GPT. 119 API calls in total.

I saved every response too, so you can check the work yourself. This one got interesting pretty fast.

Let's get into it.

ℹ️ NOTE Everything below comes from stored API responses. The harness is public too, so you do not have to take my word for any of it.

TLDR

Category	Winner	Runner up	Notes
Raw field accuracy	Claude direct 0.991	GPT direct and Reducto 0.982	11 docs, fixed scoring, no LLM judge
Never hallucinated a value	Extend 0 out of 11	Claude, GPT, Reducto also 0	Textract 2, Mistral 3, LlamaExtract 3
Table rows never missed	Extend row F1 1.00	Everyone else 0.99	Only perfect row F1
Cheapest per correct field	GPT direct $0.00047	Mistral OCR $0.00049	Textract was 14x GPT
Fastest	Mistral OCR 4.2s median	Claude direct 6.4s	Mistral also hallucinated 3 times
Developer experience	Claude and GPT about 10 min	Extend about 25 min	Textract took about 45 min plus another LLM step
Agent support	Extend llms.txt, agents.md, MCP, plugin	Reducto 3 of 4	AWS has nothing specific for Textract
Hardest to justify	AWS Textract		Highest cost in this test and some ugly misses

The short version is pretty simple. Claude had the best raw accuracy. GPT was the cheapest per correct field. Extend was the only document API with zero hallucinations and perfect row recall.

That last part matters a lot once the output is going into a real system. If a human is checking the JSON, raw accuracy can be enough. If software is going to act on the result, I care a lot more about made up values.

Textract had the roughest run here. I will show you why.

What I tested

There are six products in the test. Claude and GPT run directly, so the result table has seven columns.

#	Provider	Endpoint	Native schema	Mode
1	Extend	`POST /extract`	yes	default, API `2026-02-09`, `extend-ai` 1.19.0
2	Reducto	`POST /extract`	yes	standard, `deep_extract` off
3	LlamaExtract	`extract_stateless()`	yes	`BALANCED`
4	AWS Textract	`AnalyzeDocument` FORMS + TABLES	no	plus LLM normalizer, us-east-1
5	Mistral OCR	`/v1/ocr` + `document_annotation_format`	yes	`mistral-ocr-latest`
6a	Claude direct	`/v1/messages`, forced tool	yes	`claude-opus-5`
6b	GPT direct	`/v1/responses`, `json_schema`	yes	`gpt-5.5`, `strict: false`

A couple of things before we start.

LlamaParse structured output is deprecated. If you want schema extraction on LlamaCloud now, you use LlamaExtract.

The old llama-cloud-services package is also deprecated, so I used llama-cloud>=1.0.

AWS Textract does not return your schema as JSON. It gives you blocks like LINE, KEY_VALUE_SET, TABLE, and CELL.

So I had to turn those blocks into text and pass that text to claude-haiku-4-5 to get the final JSON. That extra model call counts toward Textract cost and latency.

💡 In simple words, Textract could read the page, but it needed another model to shape that reading into the same output every other tool was asked to return.

Keep that in mind for the numbers later.

The documents

I used 11 documents and 35 pages in total. All of them came from public datasets.

The mix was roughly 3 easy, 4 medium, and 4 hard.

#	Document	Source	Pages	Why it is here
01	invoice_clean	Voxel51 synthetic invoices	1	clean invoice, 7 rows, European decimals
02	invoice_complex	FATURA2	1	different layout, discount row, weird total
03	receipt_photo	SROIE 2019	1	low resolution and handwriting on top
04	contract_simple	CUAD	7	agreement date is blank
05	contract_complex	CUAD pages 1 to 8	8	governing law is outside the page limit and `$6,200,000` is not the contract value
06	bank_statement	Bankstatemently	2	French Canadian, 25 transactions, watermark, 2 digit years
07	noisy_form	FUNSD	1	scanned form with 41 label and value pairs
08	handwritten_note	handwritten cheque dataset	1	written amount and number box disagree
09	EOB	CMS sample	1	medical EOB with `XXXXXX` placeholders
10	mixed_packet	built from 01, 04, and 09	5	checks page boundaries
11	contract_scanned	scanned copy of 04	7	same contract, but rasterised at 200dpi

Every file is locked with a sha256 in manifest.lock.json. If one byte changes, the runner stops.

That sounds a bit extreme, but it saves a lot of headaches later.

💡 It means every provider saw the exact same file. No accidental OCR. No changed PDF. No weird hidden advantage.

I also made every image source into a pixel only PDF with img2pdf. Then the build script checks it with pdftotext.

If any text layer exists, the build fails. Why bother

Because if I OCR the file before the benchmark, I have basically handed every provider the answer key. And that kinda kills the whole test. 💁

The traps

This is probably the most useful part of the corpus.

A few fields are genuinely missing from the documents. The right answer for those is null.

If a provider gives me a value anyway, it made something up.

Doc 04 has a blank agreement date. The effective date exists, so it is very easy to copy the wrong date.
Doc 05 stops at page 8. The governing law is on page 26. It also contains $6,200,000, but that number is a financing cap, not the contract value.
Doc 01 has no due date and no discount.
Doc 08 disagrees with itself. The words say one amount and the number box says another. The correct output keeps both.

💡 These are not OCR tricks. I wanted to see what happens when the safest answer is simply "I do not know".

How I kept the test fair

The numbers are useless if every provider gets a different setup. So each one got the same thing.

The exact same PDF
The exact same JSON schema
The exact same instruction
The same 2 second poll time and 300 second timeout
Retries only for network or server errors
The normal default mode for each product

No custom prompt for one provider. No premium mode for another.

A wrong answer did not get a second try. The prompt was tiny too.

# bench/providers/base.py
SHARED_INSTRUCTIONS = (
    "Extract the fields defined by the schema from this document. "
    "Use exactly the values printed in the document. "
    "Return dates as ISO 8601 (YYYY-MM-DD) and amounts as plain numbers "
    "without currency symbols or thousands separators. "
    "If a field is not present in the document, return null for it "
    "rather than guessing or inferring a value."
)

POLL_INTERVAL_S = 2
TIMEOUT_S = 300

And a schema looks like this.

{
  "type": "object",
  "properties": {
    "vendor_name": { "type": ["string", "null"] },
    "due_date": { "type": ["string", "null"], "description": "ISO 8601" },
    "line_items": {
      "type": "array",
      "items": {
        "type": "object",
        "properties": {
          "description": { "type": ["string", "null"] },
          "quantity": { "type": ["number", "null"] },
          "amount": {
            "type": ["number", "null"],
            "description": "line total EXCLUDING tax"
          }
        }
      }
    }
  }
}

The important bit is that fields can return null. That lets the provider say the value is missing instead of forcing it to guess.

💡 null is not a failure here. Sometimes it is exactly the right answer.

That "line total EXCLUDING tax" line exists because I got burned while building this. My first schema just said amount.

The invoice has Net price, Net worth, and Gross worth. Every provider picked something different and for about an hour I thought I had found a huge accuracy gap.

But nope.

My schema was just vague. 🫠

One adapter for every provider

Every company has a completely different way to say "send PDF, give JSON".

extend        POST /extract         upload -> file id   -> inline schema config
reducto       POST /extract         upload -> reducto:// -> instructions.schema
mistral       POST /v1/ocr          upload -> signed URL -> document_annotation_format
llamaextract  extract_stateless()   upload -> file id   -> ExtractConfig, maybe poll
claude        POST /v1/messages     base64 inline       -> forced tool, input_schema IS the schema
gpt           POST /v1/responses    base64 inline       -> text.format json_schema
textract      AnalyzeDocument       per-page blocks     -> LLM normalizer stage

Some upload first. Some want a signed URL. Some poll. Some return the JSON inside another object.

Mistral even returns document_annotation as a JSON string. Thanks Mistral. 😭

I hide all of that behind one function.

class Provider(Protocol):
    name: str
    def extract(self, pdf: Path, schema: dict, schema_id: str) -> ExtractResult: ...

That is what keeps the runner fair. The runner does not know which provider it is calling.

It just sends a PDF and expects the same result shape back.

The Claude adapter is tiny.

# bench/providers/direct_llm.py (trimmed)
def extract(pdf: Path, schema: dict, schema_id: str) -> ExtractResult:
    b64 = base64.b64encode(pdf.read_bytes()).decode()
    resp = client.messages.create(
        model="claude-opus-5",
        max_tokens=8192,
        tools=[{"name": "emit", "description": "Return the extracted fields.",
                "input_schema": schema}],
        tool_choice={"type": "tool", "name": "emit"},
        messages=[{"role": "user", "content": [
            {"type": "document",
             "source": {"type": "base64", "media_type": "application/pdf", "data": b64}},
            {"type": "text", "text": SHARED_INSTRUCTIONS},
        ]}],
    )
    block = next(b for b in resp.content if b.type == "tool_use")
    return ExtractResult(
        data=block.input,
        usage=Usage.from_anthropic(resp.usage),
        raw=resp.model_dump()
    )

No upload. No job ID. No polling.

The schema itself becomes the tool input.

Textract is the opposite. It needs about 60 lines just to rebuild rows from CELL blocks, then another model call to turn that into the target JSON.

# bench/providers/textract.py (trimmed)
def _table_rows(blocks_by_id, table):
    cells = [
        blocks_by_id[i]
        for r in table.get("Relationships", [])
        if r["Type"] == "CHILD"
        for i in r["Ids"]
    ]

    grid = defaultdict(dict)

    for c in cells:
        grid[c["RowIndex"]][c["ColumnIndex"]] = _cell_text(blocks_by_id, c)

    return [
        [grid[r].get(col, "") for col in sorted(grid[r])]
        for r in sorted(grid)
    ]

normalized = normalizer.run(text=rendered, schema=schema)

💡 Same task on paper. Very different amount of work in code.

Scoring without an LLM judge

I did not use another LLM to decide who won. That just creates a second argument.

The scorer is fixed.

Normalise first Money becomes Decimal, dates become ISO, and strings get cleaned so tiny formatting changes do not count as errors.
Scalars Exact after normalisation. Longer text uses rapidfuzz token_set_ratio. A score of 90 or more counts as correct.
Verdicts A field can be correct, near, wrong, missing, or hallucinated.
Arrays Rows are matched using text fields only.
Missed rows count If a provider returns 1 perfect row out of 7, it does not magically get 100 percent. 🥴

Before scoring the providers, the scorer checks itself. First, all 11 ground truth files are scored against themselves.

Every one must return exactly 1.000.

Then I deliberately break the data. Wrong total. Fake date. Missing vendor. Dropped rows.

Every one of those changes has to lower the score. Dropping 4 of 7 rows gives 0.579.

Not some fake 1.0.

$ uv run bench score
self-test  identity     11/11  =1.000  OK
self-test  corruption   6/6    reduce  OK
scoring 77 runs ...

run and score are separate commands. run saves the raw API response to disk. score only reads those saved files.

That meant I could rewrite the scorer four times without paying for the APIs again. Pretty useful when you find a bug at 1 AM. 👍

1. Accuracy

There were 77 main runs. Zero API errors. Zero invalid JSON responses.

Every provider returned something usable on every document.

Provider	Field accuracy	Row F1	Hallucinated	Missing	Median latency
Claude direct	0.991	0.99	0	2	6.4s
GPT direct	0.982	0.99	0	2	10.1s
Reducto	0.982	0.99	0	4	10.1s
Extend	0.962	1.00	0	2	22.1s
AWS Textract	0.936	0.99	2	1	14.4s
LlamaExtract	0.903	0.99	3	3	22.5s
Mistral OCR	0.884	0.99	3	3	4.2s

I kept field accuracy and row F1 separate. I did not turn everything into some made up score out of 10.

Because then half the benchmark becomes me deciding how much each number should matter.

Claude came first on raw field accuracy. GPT and Reducto tied next.

Extend landed at 0.962, but it had the only perfect row F1 in the whole test. It did not miss a single table row across 25 bank transactions, 7 invoice lines, 41 form pairs, and all the EOB rows.

That is a pretty important detail. 0.991 and 0.962 look different on a leaderboard.

Across these 11 documents, the gap was under 3 points.

Per document

Document	Claude	GPT	Reducto	Extend	Textract	LlamaExtract	Mistral
01_invoice_clean	1.00	1.00	1.00	1.00	1.00	1.00	1.00
02_invoice_complex	1.00	1.00	1.00	1.00	0.97	1.00	1.00
03_receipt_photo	1.00	0.92	1.00	1.00	1.00	1.00	1.00
04_contract_simple	1.00	1.00	1.00	0.93	0.86	1.00	0.86
05_contract_complex	0.93	0.93	0.86	0.93	0.79	0.86	0.64
06_bank_statement	1.00	1.00	0.99	1.00	0.98	1.00	0.99
07_noisy_form	0.98	0.95	0.95	0.95	0.86	0.74	0.93
08_handwritten_note	1.00	1.00	1.00	1.00	1.00	0.67	0.83
09_EOB	1.00	1.00	1.00	1.00	1.00	0.75	0.95
10_mixed_packet	1.00	1.00	1.00	0.92	0.92	0.92	0.67
11_contract_scanned	1.00	1.00	1.00	0.86	0.93	1.00	0.86

A few things stand out.

The clean invoice is a 7 way tie at 1.00. If your documents all look like that, honestly, pick the cheapest one and move on. 🤷‍♂️

The complex contract is where Mistral drops hard to 0.64. LlamaExtract had the roughest time on the noisy form, handwriting, and EOB.

Now the fun part.

The failures.

2. Where things broke

This is why I saved every raw response. A final accuracy number hides way too much.

The hallucination traps

The first trap was the blank agreement date in Doc 04. The correct answer is null.

Mistral returned 2012-02-17. Textract returned 2012-02-17.

Claude, GPT, Reducto, Extend, and LlamaExtract returned null.

The second trap was the $6,200,000 financing cap in Doc 05. Again, the correct contract value is null.

Mistral, LlamaExtract, and Textract returned 6200000. Claude, GPT, Reducto, and Extend returned null.

Three products turned a financing limit into a contract value. That is not the kind of error I want quietly flowing into an AP system.

💡 The scary errors are not always random garbage. Sometimes the wrong answer looks completely reasonable.

Textract dropped digits from money

On the bank statement, I found these two.

1 200,45 $ became 200.45

6 675,00 $ became 675.00

These are not formatting misses. The first digit is just gone.

And this is money. 🤧

I checked the summary on the same page too. The credits, debits, and closing balance all match the ground truth.

Then I checked Textract raw blocks. The digit was already missing before Haiku saw anything.

So the normalizer did not cause this one.

LlamaExtract shifted a whole column

The EOB result was sneaky. LlamaExtract found the right rows, but some values moved one column to the left.

Field	Expected	LlamaExtract
`services/0/plan_paid`	2.15	0.0
`services/0/patient_responsibility`	0.0	2.15
`services/1/billed_amount`	375.00	118.12
`services/1/allowed_amount`	118.12	35.00

Every number exists in the row. It is just under the wrong field.

That is exactly the kind of JSON you can skim and think looks fine.

Mistral found pages that do not exist

The mixed packet has five pages. Mistral said the contract was on pages 2 to 10 and the EOB was on 11 to 13.

Every other provider got 1-1, 2-4, and 5-5.

I still do not know where page 13 came from. 🫩

Handwriting fooled GPT

The receipt has tan chay yee written by hand at the top. GPT used that as the merchant_name.

The printed merchant is OJC MARKETING SDN BHD. Every other provider ignored the handwriting.

So yeah, best overall does not mean best on every file.

Mistral answered a slightly different question

Doc 05 calls the two parties Sekisui and Qualigen. Mistral returned Distributor and Manufacturer.

Those words make sense in the contract, but they are not the defined names I asked for. Part of this is schema sensitivity.

Still, it cost two fields.

Apparently the pipe symbol was hard

One invoice row says this.

Dell Core 2 Duo Desktop Computer | Windows XP Pro | 4GB | 500GB

Textract read | as I. Mistral removed the pipes.

Everyone else got it exactly right.

Tiny character. Same invoice. Two different OCR mistakes.

Lol.

Extend never made up a value

Extend had a very different failure pattern.

Zero hallucinations across all 11 documents
Passed both traps
Only perfect row F1 at 1.00
8 of 11 documents scored 1.000
0.962 field accuracy compared with 0.991 for the leader

Most of the lost points came from the contracts. A couple of fields were present, but Extend returned null.

That is still wrong for the benchmark. But it is a much easier failure to deal with in a production flow.

A missing value can go to review. A confident wrong number can move straight into the next system.

💡 Extend was more likely to leave a field empty than guess. Depending on your workflow, that can be a useful trade.

3. Pricing

Here is what the 35 page test cost.

Provider	Total	Cost per document	Cost per 1000 pages	Cost per correct field
GPT direct	$0.1736	$0.0158	$4.96	$0.00047
Mistral OCR	$0.1750	$0.0159	$5.00	$0.00049
Claude direct	$0.3734	$0.0339	$10.67	$0.00100
LlamaExtract*	$0.4375	$0.0398	$12.50	$0.00126
Reducto*	$0.5250	$0.0477	$15.00	$0.00142
Extend	$2.1875	$0.1989	$62.50	$0.00593
AWS Textract	$2.3553	$0.2141	$67.30	$0.00663

Pricing for Reducto and LlamaExtract could not be fully verified from usage data.

GPT was easily the cheapest per correct field. Textract was the most expensive in this test.

And because it also needed the extra Haiku step, its real setup is not just the Textract call.

What you get with Extend pricing

Extend reports both extraction credits and total credits.

"usage": { "credits": 3.0, "totalCredits": 5.0 }

Across my runs, that came to 5 credits per page. On pay as you go pricing, that is about $62.50 per 1,000 pages.

But you are also getting more than a JSON response. Extend includes parsing, structured extraction, field citations, confidence scores, and a review flow in the dashboard.

Scale pricing drops to roughly $50 per 1,000 pages. Enterprise pricing is custom.

💁 So I would not compare this only as price per page. The extra cost is paying for the workflow around extraction too.

Textract pricing gets messy fast

Textract prices features separately. FORMS is $0.05 per page. TABLES is $0.015 per page.

Using both comes to $0.065 per page. For 35 pages, that was $2.275.

Then I had about $0.08 of Haiku cost for the JSON normalizer. That brings the matrix to around $2.36.

The billing screenshot can show more than that because it also includes my reruns and integration tests.

There is also DetectDocumentText. That costs $0.0015 per page and has a bigger free tier.

I did not test that setup here. But plain OCR plus a cheap model would be a very interesting follow up.

Reducto and LlamaExtract pricing

I could not fully verify Reducto extraction cost from the usage dashboard. It was still showing the usage page as under construction when I checked.

So the $15 per 1,000 pages number uses the published parse rate. Extraction can add more credits.

LlamaExtract uses its published rate in the table with the same caveat.

4. Developer experience

I also tracked how annoying each integration was to build. Very scientific metric, I know. 😆

Rank	Provider	Time	What happened
1	Claude	about 10 min	worked
1	GPT	about 10 min	worked
3	Extend	about 25 min	documented path worked first try
4	Reducto	about 20 min	runtime typing issue and serializer crash
5	Mistral	about 15 min	documented import failed
6	LlamaCloud	about 30 min	old package docs and mixed pydantic versions
7	AWS Textract	about 45 min	no schema output, table rebuild code, extra LLM step

Claude and GPT were simple because the PDF can go straight into the model call. Extend was the easiest document API.

Its documented path worked on the first try. That should be normal.

It was not. ✌️

Mistral import issue

The docs showed this.

from mistralai import Mistral

On v2.9.4 I got this.

ImportError: cannot import name 'Mistral' from 'mistralai' (unknown location)

The working import was this.

from mistralai.client import Mistral

15 minutes of my life. 😴

LlamaCloud setup

LlamaCloud sent me through two old paths before I even started. LlamaParse structured output is deprecated.

llama-cloud-services is also deprecated.

Then the client lives at llama_cloud.client.LlamaCloud, and ExtractConfig still uses pydantic v1 style while the rest of the package uses v2.

It works. It just took more digging than I expected.

Extend setup

Extend was boring in the best way. The documented SDK flow worked.

No weird import hunt. No second package. No guess and check loop.

That is also where its agent files helped.

5. Agent experience

This part is becoming more important now. Can Claude Code, Codex, or another coding agent actually understand the SDK without me spoon feeding it everything

I checked what each vendor ships for agents.

What vendors publish

Checked on September 8 2026.

Provider	llms.txt	agents.md	MCP server	Official agent plugin
Extend	yes	yes	yes	yes `extend-hq/extend-agent-plugin`
Reducto	yes	yes	yes	no
Mistral	yes	no	no	no
LlamaCloud	yes	no	no	no
Anthropic	yes
OpenAI	yes
AWS Textract	org wide only	no	no	no

llms.txt alone is not that special anymore. Six of the seven have one.

Extend agents.md had all four. Reducto had three.

AWS has a general AWS llms.txt, but nothing specific for Textract.

What actually helped

Extend agents.md had one small line that saved real time. It says that REST paths and SDK methods can have different names.

For example

POST /extract_runs

maps to

client.extract_runs.create()

That sounds tiny. But it stopped the exact guess and check loop I hit with another SDK.

💡 Agent docs are useful when they answer the weird little questions a coding agent would otherwise guess.

Things I got wrong

This benchmark also caught one annoying problem with the datasets themselves. Published ground truth can be wrong too.

The SROIE answer key says the receipt postcode is B1750. The receipt clearly says 81750.

FUNSD has Macket where the form says Market.

So if I blindly used the dataset labels, I would punish a provider for reading the document correctly.

Nice.

This is another reason I kept the raw files and responses. Benchmarks can be wrong in boring ways.

Which one should you pick

There is no one winner for every workflow.

Pick Claude directly when

You want the highest raw field accuracy from this test
A human will review the result
You do not need field citations or confidence scores
You already use the Anthropic API

Pick GPT directly when

Cost matters a lot
You still want very strong raw accuracy
A human can check the weird cases

Pick Extend when

A made up value is worse than a missing one
You need field citations
You need confidence scores
You want a review UI
The extracted data goes into a real production workflow
Your team uses coding agents and good agent docs matter

Pick Reducto when

You want document specific tooling with strong raw accuracy
You want zero hallucinations in this test
You have checked the extraction pricing for your own workload

Pick Mistral when

Speed is the main thing you care about
A human checks the result before anything acts on it

Textract

For this exact setup, Textract was hard to justify. It cost the most, needed an extra model step, hallucinated twice, and dropped leading digits from money.

Your workload may be different. But based on these 11 documents, I would test the other options first.

Reproduce it

Everything is in the repo: shricodev/document-parsing-api-benchmark

uv sync
uv run bench doctor
uv run bench run --all
uv run bench score
uv run bench report

bench doctor checks credentials, corpus hashes, and spend limits. bench run will not call an API again if a successful result is already saved.

So if the run stops halfway through, you can resume without paying for the same documents twice.

$ uv run bench run --provider extend --doc 06_bank_statement
extend / 06_bank_statement  cached (runs/extend/06_bank_statement/default.json)  skipping

Every raw response lives in runs/. That means every result in this post can be scored again without an API key.

The source PDFs are not committed because of licensing. corpus/build/fetch.py downloads and freezes them instead.

Conclusion

The result was closer than I expected.

Claude had the best raw field accuracy. GPT was extremely cheap for how well it scored. Reducto was right behind them.

And Extend had the cleanest safety result in the group. Zero hallucinations. Zero missed table rows. Its SDK path worked on the first try too.

That is why I would not pick one of these from the accuracy column alone.

If a human is reading the output, calling Claude or GPT directly can make a lot of sense. If the JSON is going straight into AP, claims, finance, or another production system, I care much more about citations, confidence, review, and what happens when the model is not sure.

That is where Extend made the strongest case in this test.

And yeah, run your own benchmark. Freeze the files, save the responses, and add a few nasty cases.

This one cost me $9.67. It changed what I would actually use in production. ✅