Short answer: use semantic search plus an LLM classifier to narrow invoice-field candidates, then produce a typed result only when the evidence clears an explicit gate. Keep the fast path small. A supplier invoice that takes 900 ms to tag is inconvenient; one that silently turns an invoice number into a purchase-order number is an operational incident waiting to happen. The numbers are policy choices, not universal benchmarks. Start with a 300 ms retrieval budget and an 800 ms end-to-end target only if your queue and document mix support them; measure before tightening either limit.
The useful design is a two-speed runbook: retrieve a bounded set of field definitions and examples, classify against that evidence, and abstain when the score, agreement, or required field shape is wrong.
What should happen when an invoice is ambiguous?
Treat ambiguity as a valid output. An invoice may contain Invoice No., Order No., Account, and a supplier-specific reference on the same page. A keyword match can find all four. Semantic similarity can find the right neighborhood, but similarity alone does not prove that a value belongs to the requested field.
For a concrete contract, represent each candidate as a field name, a description, an extraction rule, and a small set of approved examples. Embed the searchable text, retrieve the top 8 candidates, and rerank those candidates with signals that the retrieval index does not know: page location, nearby labels, data type, and supplier-specific aliases. The classifier then receives the winning evidence and must return either a field/value pair or an abstention reason.
That last branch is the part teams skip. They tune a confidence threshold until the dashboard looks calm, then discover that a plausible value is worse than a missing value because downstream payment automation treats both as ordinary strings. A missing field can enter a review queue. A wrong field can become a bad payment.
How should semantic search plus an LLM classifier handle invoice docs?
The pipeline should have four observable boundaries. First, normalize the OCR text while retaining page and bounding-box metadata. Second, retrieve candidate definitions from a vector-capable store. Third, rerank with deterministic document signals. Fourth, classify into a closed schema and validate the result before persistence.
A simple record is enough to make the boundary explicit:
type FieldCandidate struct {
Name string
Description string
Value string
Page int
Similarity float64
Evidence string
}
func accept(c FieldCandidate, minSimilarity float64) bool {
if c.Name == "" || c.Value == "" {
return false
}
if c.Page < 1 || c.Similarity < minSimilarity {
return false
}
return true
}
This is intentionally boring. The retrieval layer supplies candidates; the gate owns acceptance. In production, the classifier response should be decoded into a schema with a fixed enum for field names, a normalized value, source page, and evidence span. Reject unknown fields and values that fail type checks. For dates, decide whether the contract stores an ISO date or preserves the source string. For amounts, require currency context rather than guessing from punctuation.
The embedding query should describe the field you need, not dump an entire invoice into one vector. supplier invoice number and purchase order number are close in vocabulary, so the reranker needs the local label and the expected format. A Node.js service can apply the same contract at its queue boundary, while the classifier remains replaceable behind a typed interface. This is where quality costs latency: another pass has value only when it can see a signal the previous pass could not.
Keep retries outside the classifier decision. RFC 9110 distinguishes request semantics and makes idempotency relevant to safe retries; your job queue should assign an operation ID and make persistence idempotent before a timeout can cause a duplicate write. A retry that repeats an extraction is acceptable. A retry that appends a second accounting record is not.
Which signals are worth the extra latency?
The right question is not “Should we add reranking?” It is “Which errors are expensive enough to justify another decision boundary?”
For supplier invoices, page-local evidence is usually a stronger operational signal than a tiny change in embedding similarity. A candidate labelled Invoice No. on page 1 should beat a semantically similar Order No. found in a line-item description when the target field is the invoice identifier. A regular expression can validate a shape, but it cannot establish meaning. Use it as a gate, not as the classifier.
A practical policy is to split fields by consequence. Invoice number, total amount, currency, and supplier identity get strict gates and a review path. Low-impact tags such as department or document topic can tolerate a lower threshold if they never authorize payment. This is a decision rule, not a claim that any fixed threshold works for every supplier.
Postgres with a vector extension can keep candidate definitions beside ordinary relational metadata, which is useful when a field revision must be audited with its effective date. The extension's documentation describes exact and approximate nearest-neighbor options; choose based on measured recall and query latency for your corpus, not on a generic promise. Keep the embedding model version and preprocessing revision in the row so a reindex is reproducible. The limitation is clear: this approach is a poor fit for tiny document sets where a relational lookup is already clear, or for documents whose OCR quality is too low to provide trustworthy evidence; the extra semantic stage then adds latency without adding a useful decision signal. The trade-off is quality versus latency, and the right choice can be the simpler path.
How do you verify it before changing the gate?
Build a fixture set from representative invoices and label the target fields, source spans, abstentions, and reasons. Do not use only clean samples. Include rotated scans, duplicate labels, multi-page totals, missing currency, and supplier aliases. The test should answer two different questions: did retrieval include the correct candidate, and did the final gate accept only a correct value?
Record these dimensions separately:
- retrieval recall at the configured top-k;
- accepted-field precision by field type;
- abstention rate by supplier and document shape;
- p50 and p95 latency for each pipeline boundary;
- retry count, timeout reason, and idempotency outcome.
The p95 number matters at 3 a.m. because a queue of slow documents can look like a model-quality problem while it is really a concurrency or OCR bottleneck. Dashboards should show the page that fired: supplier, field, evidence span, model revision, and gate result. An aggregate “classification confidence” is not enough to reconstruct why a payment field was accepted.
A safe rollout changes one policy at a time. First shadow the new retrieval or reranking rule and compare decisions without writing them. Then enable it for a bounded supplier cohort. If accepted precision falls, restore the previous gate and preserve the shadow trace for diagnosis. If latency rises while quality is flat, remove the extra stage rather than hiding it behind a larger worker pool. Capacity can conceal a bad decision boundary until the next traffic spike.
The small contract that keeps the system honest
An extraction service should return a status, not just a string. accepted means the value passed schema and evidence checks. review means a human or a later workflow owns the decision. rejected means the input could not support the requested field. Include the operation ID, source page, evidence span, and revisions in every result.
This contract makes failure visible without turning every uncertainty into an outage. It also makes rollback concrete: stop consuming new policy revisions, route new documents to the last accepted revision, and leave already persisted records untouched until a separately approved correction job runs.
The useful optimization is selective effort. Spend retrieval and reranking latency where a wrong answer has a financial consequence; keep low-risk tags on the shorter path. Quality is the gate. Latency is the budget. Neither should be inferred from a single score.
References
- RFC 9110, “HTTP Semantics”: https://www.rfc-editor.org/rfc/rfc9110
- pgvector, “Open-source vector similarity search for Postgres”: https://github.com/pgvector/pgvector
Top comments (0)