TL;DR
A document parsing API turns PDFs, scans and images into structured JSON that software can use directly. Demos run on clean samples, so a good demo says little about your real inbox. This tutorial builds a small Python harness that runs your hardest documents through a candidate API and reports field accuracy, silent errors, latency and failures. It then shows the same harness against Sensible's API and lists what else to check before you commit: cost per document, who maintains the extraction logic, and how results get verified.
Business software depends on structured data, yet much of what companies receive arrives inside documents. Invoices, statements and forms show up as PDFs, scans and photos, each laid out differently by whoever produced it. For years the options were manual entry or brittle scripts.
A parsing API packs that reading work into one request, and many products now offer it. They all look good on a clean sample invoice, which makes them hard to compare. By the end of this tutorial you will know how to build a test set from your own documents, which numbers to measure, and how to read the results.
What One Supplier Invoice Looks Like After Parsing
Suppose we have an operations team at a mid-size distributor that receives supplier invoices from hundreds of vendors. Each vendor uses its own layout, some send digital PDFs, and others send scans. A person currently types the invoice number, date, line items and total into the accounting system.
A parsing API takes over that typing. The team's software sends a file and gets back JSON, a text format that programs read directly, with a named field for each value. The field names below are illustrative, because they depend on the configuration a team writes or clones.
{
"status": "COMPLETE",
"parsed_document": {
"invoice_number": { "type": "string", "value": "INV-1042" },
"total": { "type": "string", "value": "1,284.50" }
}
}
OCR, short for optical character recognition, only converts the pixels of a scanned page into text. Parsing goes further by deciding which text is the invoice number and which is the total, so OCR is one step inside most parsing APIs rather than a substitute for them.
Four Approaches Sit Behind Most Parsing APIs
Knowing which approach a vendor leans on predicts where it will perform well and where it will need attention. Many products combine more than one.
OCR Services That Return Text From Each Page
OCR and text detection services return text, tables and form fields from a page. They suit teams that want raw material and will write the logic that decides which value is which.
Rule-Based Extraction Reads Values From Fixed Positions
Rule-based extraction defines where a value sits, using positions, labels or patterns. Results are predictable for stable layouts, though each new layout needs its own rules.
LLM-Based Extraction Answers Questions About the Page
LLM-based extraction has a language model answer questions such as "what is the invoice total?" It handles layouts it has never seen, but it can return a plausible wrong answer and can be slower and costlier per page.
Hybrid Extraction Chooses a Method for Each Field
Hybrid extraction applies rules to fields with predictable structure and an LLM to fields that vary, then validates the results. Each field uses the method that suits it.
How to Pick Between the Four Approaches
No single approach wins everywhere. A fixed government form favors rules, a free-text contract favors an LLM, and a mixed inbox favors a hybrid with a way to detect when either one is unsure.
How to Build a Test Set From Your Hardest Documents
A vendor demo on a clean sample invoice tells you little about production. A better test uses 20 to 30 of your own documents, chosen to include the cases that cause trouble today. Hand-label the correct values once, then reuse the set for every vendor you try.
Sort Documents Into Folders by Difficulty
Sort the files into folders by the kind of difficulty they carry:
-
clean/holds a few easy digital PDFs as a control. -
scans/holds photographed or scanned pages with skew, noise or stamps. -
variants/holds the same document type from many vendors, so one field appears in several layouts. -
long/holds multi-page documents with repeating rows, such as a 40-line invoice. -
high-stakes/holds documents where a wrong value costs money, such as payment amounts.
docs/
clean/invoice-001.pdf
scans/invoice-014.pdf
variants/invoice-021.pdf
long/invoice-033.pdf
expected/
invoice-001.json
invoice-014.json
Label the Correct Values Once by Hand
Each file in expected/ shares its name with a document and lists the correct values. Set aside a handful of documents and don't use them while you tune anything, so you have an unbiased check at the end.
{
"invoice_number": "INV-1042",
"subtotal": 1180.00,
"total": "1,284.50",
"line_items": [{ "amount": 590.00 }, { "amount": 590.00 }]
}
A Vendor-Neutral Python Harness for Comparing Parsing APIs
How the Vendor Adapter Feeds the Test Runner
The harness below treats every vendor the same way. An adapter takes a file path and returns two things: a dictionary of extracted fields and a flag for whether the vendor marked the result as doubtful. The runner compares each field with your expected value and prints results per folder.
The runner also recomputes one rule of its own, that line items add up to the subtotal. That keeps the result from depending on whether a vendor chooses to flag a problem. Install requests, save the file as bench.py, and adjust the field names to match your documents.
import json
import pathlib
import statistics
import sys
import time
from collections import defaultdict
from adapter_sensible import extract # swap in any vendor's adapter with the same shape
def norm(v):
if isinstance(v, list):
return [norm(x) for x in v]
if isinstance(v, dict):
return {k: norm(x) for k, x in v.items()}
if v is None:
return None
s = str(v).strip().lower().replace(",", "").replace("$", "")
try:
return round(float(s), 2)
except ValueError:
return s
def reconciles(got):
"""Your own check, so the result never depends on the vendor flagging a problem."""
items, subtotal = got.get("line_items"), got.get("subtotal")
if not items or subtotal is None:
return True
try:
total = sum(float(str(i["amount"]).replace(",", "").replace("$", "")) for i in items)
return abs(total - float(str(subtotal).replace(",", "").replace("$", ""))) < 0.01
except (KeyError, ValueError):
return False
def run(adapter, docs_dir, expected_dir):
rows, failures = [], []
for pdf in sorted(pathlib.Path(docs_dir).rglob("*.pdf")):
want = json.loads((pathlib.Path(expected_dir) / f"{pdf.stem}.json").read_text())
started = time.perf_counter()
try:
got, vendor_flagged = adapter(pdf)
except Exception as exc:
failures.append((pdf.name, str(exc)))
continue
secs = time.perf_counter() - started
flagged = vendor_flagged or not reconciles(got)
for field, expected in want.items():
rows.append({
"group": pdf.parent.name, "doc": pdf.name, "field": field,
"ok": norm(got.get(field)) == norm(expected),
"flagged": flagged, "secs": secs,
})
return rows, failures
def report(rows, failures):
by_group = defaultdict(list)
for r in rows:
by_group[r["group"]].append(r)
for group, rs in sorted(by_group.items()):
acc = sum(r["ok"] for r in rs) / len(rs)
silent = sum((not r["ok"]) and (not r["flagged"]) for r in rs)
print(f"{group:<12} fields={len(rs):<4} accuracy={acc:.1%} silent_errors={silent}")
secs = sorted({(r["doc"], r["secs"]) for r in rows}, key=lambda t: t[1])
times = [s for _, s in secs]
if times:
p95 = times[min(len(times) - 1, int(len(times) * 0.95))]
print(f"latency p50={statistics.median(times):.1f}s p95={p95:.1f}s failed_docs={len(failures)}")
for name, err in failures:
print("FAILED", name, err)
if __name__ == "__main__":
report(*run(extract, sys.argv[1], sys.argv[2]))
Why Silent Errors Are the Number to Watch
The number to watch is silent_errors. A silent error is a wrong value that arrives without any flag, so nothing stops it from reaching your database. A wrong value that gets flagged costs a review; a wrong value that does not get flagged costs whatever the value controls.
What the Harness Output Looks Like
A dry run with three fake documents prints lines in this shape, and your own run will show the same columns with real numbers:
clean fields=4 accuracy=100.0% silent_errors=0
scans fields=8 accuracy=75.0% silent_errors=1
latency p50=0.0s p95=0.0s failed_docs=0
Adapting the Harness to Sensible's REST API
Calling the Synchronous Endpoint for Testing
Sensible exposes a REST endpoint per document type. The synchronous call is intended for testing, so it suits a harness: post the file bytes with a matching Content-Type and read parsed_document from the response. Authentication uses an Authorization: Bearer header with a key from the account page.
The adapter below also strips Sensible's {"type": ..., "value": ...} wrappers so results compare cleanly with your expected JSON. It reads validation_summary to see whether any validation rule raised an error or a warning. Set SENSIBLE_API_KEY in your environment and pass the name of your document type.
import os
import pathlib
import time
import requests
BASE = "https://api.sensible.so/v0"
HEADERS = {"Authorization": f"Bearer {os.environ.get('SENSIBLE_API_KEY', '')}"}
def plain(node):
"""Strip Sensible's {"type": ..., "value": ...} wrappers down to plain values."""
if isinstance(node, dict) and "value" in node:
return plain(node["value"])
if isinstance(node, dict):
return {k: plain(v) for k, v in node.items()}
if isinstance(node, list):
return [plain(v) for v in node]
return node
def from_response(body):
summary = body.get("validation_summary") or {}
vendor_flagged = (summary.get("errors", 0) + summary.get("warnings", 0)) > 0
return plain(body.get("parsed_document", {})), vendor_flagged
def extract(path, doc_type="invoices"):
"""Synchronous endpoint: intended for testing, not for production volume."""
resp = requests.post(
f"{BASE}/extract/{doc_type}",
headers={**HEADERS, "Content-Type": "application/pdf"},
data=pathlib.Path(path).read_bytes(),
timeout=120,
)
resp.raise_for_status()
return from_response(resp.json())
def extract_async(document_url, doc_type="invoices", poll_secs=2, timeout_secs=300):
"""Production path: submit a URL, then poll until the extraction finishes."""
start = requests.post(
f"{BASE}/extract_from_url/{doc_type}",
headers={**HEADERS, "Content-Type": "application/json"},
json={"document_url": document_url},
timeout=60,
)
start.raise_for_status()
extraction_id = start.json()["id"]
deadline = time.monotonic() + timeout_secs
while time.monotonic() < deadline:
body = requests.get(f"{BASE}/documents/{extraction_id}", headers=HEADERS, timeout=60).json()
if body.get("status") == "COMPLETE":
return from_response(body)
if body.get("status") == "FAILED":
raise RuntimeError(f"extraction {extraction_id} failed")
time.sleep(poll_secs)
raise TimeoutError(f"extraction {extraction_id} still running after {timeout_secs}s")
Calling the Asynchronous Endpoint for Production
The extract_async function shows the production path. It submits a URL the API can fetch, then polls GET /documents/{id} until the status is COMPLETE or FAILED. Sensible's asynchronous tutorial walks through the same calls, and a webhook can replace the polling loop.
Run the harness with python bench.py docs expected. To compare another vendor, write an adapter with the same two return values and change one import line.
Six Checks to Read Out of the Harness Results
The numbers only help when you know what each one is testing. These six checks turn a run into a decision.
Check 1: Accuracy Per Field Across Vendor Layouts
Read the variants/ line first, because layout variation is where most parsing projects stall. If accuracy there is far below clean/, the product depends on layouts it has already seen. Also look at which fields fail, since one weak field, such as a tax amount, can do more damage than a low average suggests.
Check 2: Silent Errors Versus Flagged Errors
Count the silent errors per folder and compare vendors on that number as well as accuracy. Ask whether you can write your own rules and what happens to results that fail them. In Sensible, you write validation rules against the extracted values, for instance confirming that line items sum to a stated subtotal, and any field that fails gets flagged for review instead of passing through unverified.
Each rule is a JsonLogic condition with a severity of error, warning or skipped, and results appear in the validations array and the validation_summary counts. Sensible's validation page shows a rule that checks a field is present, written as {"exists":[{"var":"invoice_number.value"}]} when adapted to our invoice field.
Confidence signals are a separate tool. They are available only for the Query Group method, with values such as multiple_possible_answers and answer_not_found, and an LLM can report them inaccurately. Rules are the safer check for any field where a wrong value is costly.
Check 3: Repeating Rows on Long Multi-Page Documents
Long tables expose limits that short samples hide. Look at the long/ line and compare the number of extracted rows with the expected count, since a missing row can still leave other fields looking correct. Ask each vendor how it handles output that exceeds a model's response limit.
Sensible's List method, for example, splits a document into chunks, scores each chunk against your field descriptions, combines the top-scoring ones, and truncates the list if the output exceeds an LLM engine's limit. The Sections method suits repeating groups such as accounts on a statement, and it returns them as an array of objects.
Check 4: Latency From Submit to Result
Time each call and look at the median and the 95th percentile, because the slowest documents decide whether a workflow feels reliable. Test the synchronous endpoint for correctness, then time the asynchronous path from submit to COMPLETE, since production volumes use asynchronous calls. Latency also depends on how you call the API, so measure it from the environment where it will run.
Check 5: Who Fixes It When a Vendor Changes Format
Layouts change without notice, and somebody has to update the extraction logic when they do. Some teams have engineers write and maintain it, while others prefer prebuilt configurations or a vendor services team. Sensible keeps its configurations in Sensible, versioned through the API, and offers a library of more than 150 prebuilt ones that you clone to your account and test in the Extract tab.
Teams that do not want to build configurations themselves can have Sensible's managed services team build them from their own sample documents and adjust them when a vendor changes its format. Whichever route you choose, put a changed layout in your test set and see how long the fix takes.
Check 6: Cost Per Document at Your Volume
A per-page headline price rarely matches your bill, so divide the plan fee plus any overage by your monthly volume. Sensible's pricing page lists Growth at \$449 per month billed annually, or \$499 billed monthly, with 750 documents included and \$0.57 per extra document. For example, a team that processes 1,000 documents a month on annual billing pays \$449 plus 250 overage documents at \$0.57, which is about \$0.59 per document.
Scale lists \$1,349 per month billed annually, with 3,200 documents included and \$0.53 per extra document. Enterprise pricing is custom from 10,000 documents a month. SOC 2 Type II and GDPR appear on every plan, and HIPAA with a signed BAA is listed on Enterprise.
How Sensible Maps to the Six Checks
Sensible is a hybrid platform that combines AI extraction with deterministic, rule-based methods behind one REST API. Its unit of work is a document type, a named set of similar documents such as invoices, and each type holds one or more configurations written in SenseML, a JSON-based configuration language. Each field can use either kind of method: layout-based rules where a layout is predictable, and an LLM method where it varies. Both live in one configuration and return the same schema.
The notes below show how that hybrid design answers the earlier checks.
Layout Variation Follows an 80/20 Pattern
Sensible follows an 80/20 pattern, with deterministic methods handling most fields first and a generalized LLM-based template covering the rest.
Fallbacks Cover Fields That Return Nothing
Consecutive fields with the same ID let an LLM field back up a layout field, or the reverse, when the first returns an empty array or null.
Classification Routes Each Document Before Extraction
Classification matches a document to a document type before extraction, so a team can decide what to extract or where to route it in a mixed inbox.
Validation Rules Plus a Review Queue
Validation rules you configure check the output, and flagged documents can be corrected in a review queue.
The quickest way to judge all of this is to run your variants/ and long/ folders through a trial and read the harness output. The same harness works for every other API you shortlist.
Where Should You Start
A parsing API is easiest to judge on the documents that cause the most trouble: unusual layouts, repeating groups and values that cannot be wrong. Accuracy per folder, silent errors, tail latency, maintenance effort and cost per document together give a fuller picture than any demo. Keep the test set, because you will want it again when a vendor ships an update or your document mix changes.
Sensible offers a 14-day free trial with no credit card to start, so you can run the harness above against your own files. Start with the invoice extraction page to see a document type in the app, then send your hardest ten documents through the API.
Frequently Asked Questions What Does a Document Parsing API Return?
It accepts a file such as a PDF, image or spreadsheet and returns the data inside it as named fields, usually JSON. The values come back with a type, so software can act on them without retyping.
Is OCR Enough for Extracting Invoice Fields?
OCR converts the pixels of a scan into text and nothing more. A parsing API also decides which text is the invoice number, the date or the total. Most parsing APIs run OCR as one internal step when a file has no embedded text.
How Many Documents Should a Parsing Test Set Contain?
There is no fixed number. Twenty to thirty documents spread across clean, scanned, variant, long and high-stakes folders gives a useful first read. Add more documents wherever two vendors score close together.
Should I Test With Synchronous or Asynchronous Calls?
Use synchronous calls to check correctness, because they return the result in the same response. Use asynchronous calls to measure production behavior, where you submit a document and collect the result later by polling or through a webhook.
Does Sensible Use LLMs or Rules to Extract Data?
Both. Each field in a Sensible configuration can use a layout-based method for predictable structure or an LLM-based method for content that varies, and both return the same schema.
Does Sensible Validate Extracted Data Automatically?
Validation is configured by the team. You write rules in JsonLogic against the extracted values, and results that fail appear in the validations array so they can be flagged for review.
Top comments (0)