Invoice Extraction with Human Review: Route on Checks, Not on Confidence

A design case study: why a confidence score can’t decide which invoices get paid, and what to route on instead.

About this case study. This is a composite scenario built on synthetic invoices. It shows a design and how to measure it. The numbers are planning figures, not benchmark results.

The problem

A mid-sized distributor gets about 6,000 supplier invoices a month. They arrive as PDFs, scans and phone photos in more than one language. Three accounts-payable clerks type each one into the ERP: vendor, invoice number, dates, line items, tax, total, and the purchase order it belongs to.

The ask from finance was simple: “Let the AI read the invoices, and have people check only the ones it is unsure about.”

That sentence hides the hard part. How does the system know when it is unsure?

Today every field is typed by hand. The goal splits invoices into clean and unsure, but who decides which is which?

The obvious answer is the model’s confidence score: auto-approve above 90%, send to a person below it. That is the design this post argues against.

Why confidence alone is the wrong gate

Extraction services return a confidence number for each field. Amazon Textract’s AnalyzeExpense, for example, gives separate scores for the label it found (“Total:”) and the value next to it (“$55.64”). Those scores are useful, but they answer a narrow question: how sure am I that I read these characters correctly?

A total read with 99.9% confidence on an invoice that should never be paid.

They do not answer the questions finance cares about:

• Do the line items add up to the total?
• Is this a vendor we actually do business with?
• Does it match an open purchase order?
• Have we already paid this invoice?
• Are the bank details the same ones we have on file?

A model can read a total of $12,480.00 with 99.9% confidence on an invoice that is a duplicate, that belongs to no purchase order, or that asks us to pay a new bank account. High confidence, wrong outcome. The worst errors in accounts payable are not misread digits. They are perfectly read invoices that should never be paid.

So the rule for this system is the same one Post 1 used for permissions:

Model output is a claim. Checks against our own data are the evidence. The model extracts; deterministic checks decide.

The design in one picture

Extract, validate, route: three outcomes, a 2% audit, and every correction stored.

Five decisions shape this design.

Decision 1: The extractor is replaceable; the routing is the product

There are two reasonable extraction options on AWS today:

Textract vs. Bedrock Data Automation.

The extractor sits behind an adapter. The checks, routing and review loop stay the same.

We start with AnalyzeExpense and put it behind a small interface that returns one normalized Invoice object: fields, values, confidence and page geometry. If BDA does better on our golden set later, we swap the extractor and change nothing downstream.

That is the point of this decision. Extraction quality will keep improving and vendors will keep changing. The checks, the routing and the review loop are where our business rules live, so they should not be tied to one extractor.

Decision 2: Validate with deterministic checks

After extraction, every invoice runs the same checks. None of them uses a model.

Two checks stop an invoice outright; the rest send it to review with a reason.

Two rules are worth pointing out.

Match vendors by tax ID, not name. “ACME Ltd”, “Acme Limited” and “ACME LTD.” are the same company to a person and three companies to a string comparison. Names are for display; IDs are for joins.

Never update bank details from an invoice. Changed bank details on an invoice are the most common sign of payment fraud. If the invoice disagrees with the vendor master, the invoice goes to a separate queue that only the vendor-management team can clear, and they confirm by calling a phone number already on file. No confidence score can turn this off.

Decision 3: Thresholds per field, set by the cost of being wrong

One global threshold treats a misread invoice date the same as a misread total. Those mistakes do not cost the same.

Higher cost of error, higher bar to skip review.

These starting values are not magic. They are tuned on the golden set (Decision 5), and the target is set per field: what is the most auto-approval we can get while keeping this field’s escaped error rate under X?

Arithmetic checks also give a second signal that confidence cannot. If the total was read with 99% confidence but the line items do not add up, one of the numbers is wrong, and confidence did not catch it.

Decision 4: Build the review queue; show reviewers only what is flagged

Why not Amazon A2I

Amazon Augmented AI (A2I) used to be the default answer for “human review on AWS.” In June 2026 AWS announced that A2I, along with SageMaker Ground Truth, is moving to maintenance and stopped accepting new customers on July 30, 2026. A new system should not be built on it.

The review loop we need is small enough to own:

• Step Functions with a task token. The workflow pauses at the review step with .waitForTaskToken. The review UI calls SendTaskSuccess with the corrected invoice, and the workflow continues to the ERP. No polling and no extra state machine.
• DynamoDB holds the queue: invoice ID, reasons for review, assigned reviewer, SLA timestamp.
• A small web UI behind Cognito shows the PDF with bounding boxes only around the flagged fields , plus the reason (“line items sum to 11,980.00; total says 12,480.00”).

Left: the workflow pauses on a task token until a reviewer finishes. Right: the reviewer sees only the flagged fields, with the reason.

Design the screen for speed, not completeness

The first review screen showed all 25 fields. Reviewers re-checked every one, so reviewing took almost as long as typing. Showing only the flagged fields, with the reason and the highlighted region, turns a review into a 20-second decision instead of a three-minute re-key.

Two rules keep this honest:

1. Reviewers must touch every flagged field. Each flagged field needs an explicit “confirm” or a correction, so a reviewer can’t approve the whole invoice in one click.
2. Hard stops never go to the normal queue. Duplicate and bank-detail cases have their own queue, their own role, and a four-eyes rule.

Decision 5: Measure what escapes, not just what gets reviewed

This is the decision most teams skip, and it is the one that matters most.

If you only look at invoices that went to review, you learn how often reviewers correct the model. You learn nothing about the invoices that were auto-approved, and those are the ones that turn into wrong payments.

The random audit is the only window into invoices nobody looked at.

So we measure four things:

Four numbers to watch. The escaped error rate is the one that matters most.

The 2% random audit is the key. A small random sample of auto-approved invoices goes to review anyway, without telling the reviewer it was auto-approved. That is the only unbiased estimate of the escaped error rate. Without it, a rising straight-through rate looks like progress even if it comes from errors slipping through.

Every correction is stored as (invoice, field, extracted value, corrected value, reason). That store becomes:

• the golden set for comparing extractors (Textract vs BDA) and tuning thresholds;
• a vendor watch list : vendors whose invoices are corrected often get a stricter threshold or a vendor-specific blueprint;
• the release gate for any change to the extractor or thresholds (more on that in Post 4).

A sketch of the router

The routing logic is short, and it should be. It is the most important code in the system, so it needs to be easy to read and test.

from dataclasses import dataclass
from enum import Enum

class Route(Enum):
    AUTO_APPROVE = "auto_approve"
    REVIEW = "review"
    HARD_STOP = "hard_stop"

HARD_STOP_CHECKS = {"duplicate", "bank_details"}

@dataclass
class CheckResult:
    name: str
    passed: bool
    reason: str = ""
    field: str | None = None

def route(checks, amount, auto_limit, audit_sample):
    failed = [c for c in checks if not c.passed]

    # 1. Fraud and duplicates: nothing overrides these.
    if any(c.name in HARD_STOP_CHECKS for c in failed):
        return Route.HARD_STOP, failed

    # 2. Any failed check: review, with the reasons.
    if failed:
        return Route.REVIEW, failed

    # 3. Large invoices always get a person.
    if amount > auto_limit:
        why = CheckResult("amount_limit", False,
                          f"over {auto_limit}")
        return Route.REVIEW, [why]

    # 4. Random audit of clean invoices.
    if audit_sample:
        why = CheckResult("audit_sample", False,
                          "random audit")
        return Route.REVIEW, [why]

    return Route.AUTO_APPROVE, []

The router, top to bottom. Confidence never short-cuts the checks.

Notice what is missing: there is no line that says “if confidence is high, skip the checks.” Confidence is only one of the checks (per field), never a way around the others.

What we would expect, and how we would know

Each stage opens only when the data from the previous one says it is safe.

For a rollout like this, a reasonable plan is:

• Weeks 1–4, shadow mode. Everything goes to review. The system’s route is recorded but not used. This builds the golden set and shows what the straight-through rate would have been.
• Then, turn on auto-approve for one vendor group with clean history and low amounts. Watch the escaped error rate from the audit sample.
• Widen only when the audit says so. Each new vendor group needs its escaped error rate under the agreed limit for a full month.

A realistic early target is a straight-through rate between 40% and 60%, with most of the rest needing a fast review of one or two fields instead of full re-keying. The exact number depends on the vendor mix: clean, machine-generated PDFs from large suppliers will pass far more often than phone photos from small ones.

What we would do differently

• Start the corrections store on day one. Shadow-mode data is the cheapest golden set you will ever get.
• Agree the escaped error limit with finance before launch, not after the first wrong payment. It is a business decision, not an engineering one.
• Treat the vendor master as part of the system. Half of the “extraction errors” in a project like this turn out to be stale vendor data. The checks are only as good as the data they check against.

Key takeaways

1. Confidence measures reading, not correctness. Route on deterministic checks against your own data.
2. Thresholds belong to fields, and are set by the cost of being wrong.
3. Some cases never auto-approve: duplicates, bank-detail changes and large amounts.
4. Show reviewers only what is flagged, and make them touch each flagged field.
5. Audit a random sample of auto-approved invoices. It is the only honest measure of risk.
6. Keep the extractor replaceable. The routing and the review loop are the product.

Next in the series: Case 03, an LLM gateway that cut the bill (GCP). The design and decision records for this case, starting with “route on verified checks, not model confidence,” are in the series repository.

AWS service details reflect public documentation as of October 2026. Check current AWS docs before you build.