Building a Document Intelligence Pipeline (Or: 80% of Your Data Is in PDFs)

Here's an uncomfortable estimate for anyone building data pipelines in a document-heavy domain: somewhere between 60% and 80% of what the business knows isn't in a database at all. It's in PDFs, scans, faxes-turned-images, and email attachments.

Every ML project in these domains follows the same arc: starts confident, hits the document wall, quietly scopes down to whatever's already structured — which is the least interesting subset of what the organisation knows.

This post is about building the pipeline that fixes that, and the design decisions that determine whether it works.

Why the 2015 approach failed (and why that no longer applies)

Most orgs tried this before with template-based OCR and were disappointed. Worth understanding why, so you don't rebuild the same thing.

Template OCR works by knowing where on the page a value sits — extract the text at coordinates (420, 310). It's brittle to the point of uselessness the moment a vendor changes their invoice layout, a form is photographed at an angle, or a new document type appears. Teams built templates for the top five formats, covered ~40% of volume, and abandoned the long tail.

Modern document AI doesn't locate values by coordinates. It reads the document and understands what things are — "total repair cost" is found because it means that, not because of its position. That single difference makes the long tail tractable, and it's genuinely new since the last time most teams evaluated this.

The pipeline, stage by stage

1. Ingest anything

PDFs, scans, photos, emails, faxes. If a human has to convert a file before your pipeline accepts it, you've reintroduced the bottleneck you're trying to remove. Normalise to images/text at the boundary, not upstream in someone's inbox.

2. Classify before you extract

Before extraction, determine what the document is: medical report, repair estimate, police report, contract. This matters more than it sounds — extraction schemas are per-type, and a misclassification makes everything downstream garbage. Classification errors are silent; extraction errors on a misclassified doc look like valid data.

Budget real effort here. It's the cheapest place to prevent expensive nonsense.

3. Extract to a schema — with confidence scores

Not "here's the text." That's just a different unstructured blob. Extract to defined fields, each with a confidence value:

{
  "doc_type": "repair_estimate",
  "fields": {
    "total_cost":   { "value": 48250.00, "confidence": 0.97 },
    "vehicle_vin":  { "value": "MA3...", "confidence": 0.62 },
    "labour_hours": { "value": 14.5,     "confidence": 0.91 }
  },
  "source": { "doc_id": "d-8842", "page": 2 }
}

The confidence score is what makes the next stage possible.

4. Route by confidence — the decision that makes it work

This is the design choice that separates working pipelines from demos. Don't aim for perfect automation. Aim for a tunable threshold.

• Above threshold → straight through, no human.
• Below threshold → human review queue, where a person corrects it.

Now automation rate is a dial your business can turn based on its risk appetite, not a binary that has to be 100% before anyone trusts it. Start conservative, raise the threshold as you gather evidence.

5. Feed corrections back

The human corrections from stage 4 are labelled training data, generated for free by work you were already doing. A pipeline that doesn't capture them is throwing away its single best improvement signal. Log every correction with the original extraction and the source doc.

6. Keep provenance

Every extracted field must point back to its source document and page. When someone asks where roof_age: 18 came from — and in a regulated domain they will — "the model said so" is not an answer. Store the doc id, page, and ideally the bounding region.

The trap: don't extract everything

The instinct is to digitise the archive — all forty years, all types, all fields. That programme takes three years and gets cancelled at eighteen months.

Invert it. Ask: which specific fields are blocking a decision right now? Usually it's a handful — a cost, a date, a classification, a couple of attributes. Extract those, from the document types that carry them, for the documents you're actively processing.

Ship in six weeks instead of never. The archive can wait; nobody's asking for a searchable 1998 contract. They're asking why a decision still takes three days.

Why this is enabling infrastructure, not a feature

Document intelligence is unglamorous and it isn't a use case anyone demos. But it's the thing that has to work before half the roadmap becomes possible — because the data those models want is sitting in a scanned fax.

Teams that treat it as infrastructure rather than a project tend to find several stalled initiatives quietly start moving at once.

We build document pipelines and the structured data foundations that AI depends on. More at IntelliBooks.