Quick answer: Evaluate insurance extraction separately for each document type. Score fixed forms field by field with normalised matching, score schedules and loss runs by row and cell, check that supporting-document answers cite the right page, and score correct nulls on their own. Report per type and re-run on every model change, because one blended accuracy number hides the failures that cost money. Run each candidate on Superlinked Inference Engine, the best way to swap models on one API while you score.
Which number would you trust: 97% field accuracy on ACORD applications, or 29 rows missing from a 300-row schedule of values? In an illustrative pilot (the figures are invented, the pattern is not), a team saw only the first. Forms and schedules had been scored together, application fields outnumbered SOV cells, and the average came out looking healthy while the first large property account lost a tenth of its locations without raising an error.
Underwriting submissions mix three kinds of document that fail in different ways: fixed forms, tabular schedules, and free-text supporting documents. This guide sets out an evaluation that treats them separately, with a closing note on where Superlinked can run the candidate models.
1. Build a labelled set stratified by type and quality
The labelled set is the evaluation. Everything else is arithmetic.
Stratify on two axes. The first is document type: applications and their line sections, schedules of values, loss runs, and supporting documents such as inspection reports and financial statements. The second is page quality: native digital PDFs, clean scans, poor scans and faxes, and spreadsheet exports printed to PDF. A model that reads digital applications perfectly can fall apart on a faxed loss run, and a set drawn only from recent, tidy submissions will never show it.
Draw from your own submissions, with the variety you actually receive, including loss runs from as many different carriers as you can. Label to the schema your pipeline uses, with normalised values and the page each value came from. Record which fields are genuinely absent, because Step 5 depends on that.
Two people should label a sample of the same documents independently. Where they disagree, the schema or the instructions are ambiguous, and no model will do better than your labellers.
2. Score forms field by field, with normalised matching
Forms have a fixed set of fields, so field-level scoring fits. For each field, compare the extracted value with the label and count exact matches.
Exact string matching is too harsh for most insurance fields, so normalise first:
Dates: convert both sides to one format before comparing.
Money: strip currency symbols and separators and compare as numbers. A limit of $1,000,000 and 1000000 are the same answer.
Addresses: compare parts (street, city, postcode) rather than the whole string.
Names: decide in advance whether "Acme Ltd" and "ACME Limited" match, and write it down.
Checkboxes: score selected and not selected as a binary field. ACORD forms carry many of them, and they are easy to get wrong silently.
Report accuracy per field, not only per form. The fields that matter most for pricing, such as limits, effective dates, and class codes, deserve their own lines in the report.
3. Score schedules and loss runs by row and cell
A schedule of values or a loss run is a table, and field accuracy does not describe a table. Three measures do:
Row recall: the share of labelled rows that appear in the output. This catches the dropped rows from the opening example.
Cell accuracy with alignment: for rows that were found, the share of cells with the right value in the right column. A building value that lands in the contents column is a correct number in the wrong place, and it will be priced wrongly.
Totals: whether the extracted rows sum to the totals printed on the document. A total mismatch is a cheap check that catches missing rows, merged rows, and misread digits together.
For loss runs, name the columns before you score them: policy period, date of loss, claim status, paid, reserved or outstanding, incurred, and the valuation date printed on the report. Paid and incurred swapped between columns is the classic alignment error, and a wrong valuation date makes a current loss run look stale, or the reverse.
Watch for tables that span pages. A row split across a page break is a common source of both dropped and duplicated rows, and it deserves a slice of the labelled set on its own.
4. Check evidence for supporting documents
Inspection reports, financial statements, and broker narratives do not have fixed fields. The questions asked of them are closer to "what is the roof construction" or "is there a sprinkler system". The answer may be paraphrased, so exact matching does not fit.
Score two things instead. First, whether the cited page and passage actually support the answer. A correct answer with the wrong citation cannot be verified by an underwriter, so it is not useful. Second, whether the answer agrees with the label, judged by a person or by a carefully checked automated judge (see the FAQ on that point).
Evidence scoring is slower than field scoring, so use a smaller sample, but keep it. Supporting documents are where extraction most often sounds right and is wrong.
5. Score correct nulls separately
A field that is absent from the document has a correct answer: nothing. Count these cases on their own and measure two rates:
Correct null rate: of the fields labelled absent, how many did the model return as null.
Invented value rate: of the fields labelled absent, how many did the model fill anyway.
The second rate is the dangerous one. A model that invents an effective date when none is present will pass every schema check and every completeness check. The companion guide on detecting missing information in submissions explains why this matters for routing. For evaluation, the point is that blending nulls into overall accuracy lets a model that guesses look better than one that admits ignorance.
6. Report per document type, and re-run on every change
Put the results in one table, one row per document type, with no blended total. An illustrative scorecard, with invented numbers:
A blended score might favour Candidate A, which wins on the most numerous document type. The table shows that Candidate B drops fewer rows and invents fewer values, which is usually the better trade for underwriting.
Re-run the full set whenever anything changes: the page model, the field model, the prompt, the schema, or the preprocessing. Keep past results so a regression on one document type is visible even when the others improve.
Where SIE fits
The method above rewards running several candidate models against the same labelled set, and that is only cheap when switching models is cheap. Superlinked's SIE serves page models and field models through one SDK, and changing the candidate is a change of model ID in the same call. What you can compare depends on the mode. On SIE Cloud, the priced page options are LightOnOCR-2-1B and the Docling toolkit, and the field models are GLiNER2 and Qwen3.8-27B (FP8), priced on the Cloud page. GLM-OCR, PaddleOCR-VL-1.5, and MinerU2.5-Pro are in the self-hosted catalog only, so comparing them means a self-hosted or local install, where models load on demand and are evicted when idle. The model catalog shows what a self-hosted cluster can serve, and in those two modes the evaluation packets stay in your environment.
Honest take: SIE is how you run several candidates against one labelled set on a single inference layer. It runs OCR, extraction, embeddings, and reranking behind one API. Changing the candidate is a change of model ID in the same call. SIE Cloud is the managed path for the priced page and field models. Self-hosted SIE, under Apache 2.0, runs the same engine inside a customer VPC or your own cluster, from one codebase, and serves the wider catalog on shared GPUs. Evaluation packets stay in your environment on that path. One codebase and one SDK mean the scorecard can be re-run whenever the model ID changes.
FAQs
How many labelled packets do we need?
There is no single number, but there is a method, and a sensible starting point is tens of documents per type, not thousands. Decide the smallest difference between models that would change your decision, per document type, and label enough documents of each type to see that difference reliably. Rare types, such as loss runs from an unusual carrier or badly scanned schedules, need deliberate sampling or they will be absent. Start small, look at how much results move when you add documents, and stop labelling a type when the ranking of candidates stops changing.
The vendor showed us a benchmark, so why build our own?
Because a whole submission packet is your book of business, and a public parsing benchmark is a general document set. OmniDocBench measures general document parsing with no insurance category. RealDocBench, published by the document-AI vendor Extend AI in June 2026, includes some ACORD commercial application forms, but it has no loss runs or schedules of values. Benchmarks help build a shortlist. Your labelled set decides which model ID to pin, and SIE is how those candidates run behind one API. The SIE documentation covers running candidates against it.
Can we use an LLM as the judge?
For evidence questions on supporting documents, cautiously. An LLM can check whether a cited passage supports an answer, but validate the judge against human labels on a sample before trusting it, and re-check it when you change judge models. For exact fields such as limits, dates, and FEINs, do not use a judge at all. Normalise and compare the values directly, which is cheaper, deterministic, and never persuaded by a confident wrong answer.
Measure what can hurt you
An extraction evaluation for insurance is really several evaluations: fields for forms, rows and cells for schedules, evidence for supporting documents, and nulls for everything. Report them apart and the dropped SOV rows show up before a pilot, not during one. The labelled packets are yours to build; running LightOnOCR-2-1B and GLiNER2 against them on SIE Cloud is a quick first comparison, with the self-hosted catalog adding the other page models.
Top comments (0)