We Benchmarked Our Extraction Pipeline Four Ways. The Obvious Knobs Lied.

Paragon666 — Sat, 14 Nov 2026 01:01:03 +0000

We are a small independent lab in Los Angeles, and this is the lab notebook
of one real production job — not a synthetic benchmark. The job:
verbatim-quote extraction over a fixed private text corpus. For each chunk,
the model returns up to a fixed quota of exact quotes with anchors. We ran
that job several ways on public models and public software — a local 9B
model on a consumer GPU, and two hosted flagships — and measured the things
that decide whether the output is usable: ground-truth completeness (share
of the reference quote set recovered), parse failure rate, the non-verbatim
rejection share (rejected quotes over all returned quotes), and time.
Within each test the arms share inputs and scoring; where a test uses a
different metric, it says so on the spot. The headline claims travelling
with this stack did not survive contact with a real task. Four tests
changed how we configure everything — and the scoreboard near the end sums
up what is better where, so you do not have to assemble it from the tables.

1. A server upgrade that wins and loses on the same day

llama.cpp, Qwen3.5-9B-Q4_K_M, RTX 3060 12 GB. Production build b11669 vs
release v1.0.0 (b12359), side by side:

Build	Completeness, quota 20	Completeness, quota 40
b11669	0.4288	0.4676
v1.0.0	0.5469	0.3916

The newer build is +11.8 points at quota 20, −7.6 points at quota 40, and
slower at both: +1.5% wall-clock at quota 20, +16.9% at quota 40. The
ranking reverses with a prompt parameter. We kept the old build. "Newer is
better" is not a measurement; it is a hope.

2. Bigger chunks, worse results

Same server and model, context fixed at 65,536 tokens, quote quota fixed
at 20 per chunk; only chunk size changes:

Chunk	Completeness	Wall-clock, same 9 units
48K chars	0.4288	724.1 s
192K chars	0.2460	335.9 s (assembled — see below)
384K chars	0.1553	— (different context and offload; not comparable)

One large unit (603,683 chars) tells the same story: 0.664 at 48K chunks,
0.4063 at 96K, and an outright failure at 192K — at ~2.8 chars/token the
prompt simply exceeds a 65,536-token window. Char-based chunk limits are
not token-safe. One sample unit returned zero quotes as a single big
chunk: valid JSON, empty list, a silent whole-unit miss. At 48K it yields
16 quotes.

The bigger chunk bought a 2.16× assembled-time ratio and destroyed
coverage: with a fixed per-chunk quota, a bigger chunk is a smaller sample
of the text. Note what this test does not show: it does not isolate
context-window size — the context was fixed. The obvious fix, scaling the
quota with the chunk, was tested next and failed too: 96K chunks at quota
40 scored 0.2670, worse than 48K at either quota. The calibration's answer
was boring: 48K chunks.

3. The output budget is a quality parameter

Hosted models, same task. One metric warning, because it decides what can
be compared: the hosted figures below are union overlap — how much of
our local baseline's accepted quote set each arm also returned, on a
common 8-unit / 14-chunk subset. Our local baseline (Qwen3.5-9B
self-hosted) is scored differently: 42.9% ground-truth completeness on
nine units (28.9% on the same eight), 190 accepted quotes, 12.0% of its
returned quotes rejected as non-verbatim. Different metrics, no common
scale — we do not rank the hosted arms against the local bar.

ERNIE 5.5 Flash at a 4,800-token output budget (our pipeline's setting,
not a verified vendor default): 71.4% parse failures on the subset,
overlap 0.142. The cause was measurable: hidden reasoning consumed ≈86%
of the output budget before any answer existed. On the full 23-chunk run
at that budget: 52.2% parse failures, overlap 0.151. The same model at a
16,000-token budget: parse failures 21.4%, overlap 0.479, with a 1.5%
non-verbatim rejection share.

Qwen4-Max-Preview beat ERNIE's 16k arm on all four extraction metrics on
their common subset: 257 accepted quotes, overlap 0.537, zero measured
non-verbatim rejects, zero parse failures in 14 chunks (and 0 of 23 on
its full extension). One method note, because it matters: its
subscription terms allow interactive use only, so that arm ran
tool-mediated inside agent sessions rather than as a raw API — and 257 is
257 of a nominal 280 prompt slots, not a hard ceiling, so what a higher
quota would have bought is unmeasured. The inconvenient axes: it was far
slower per call (111.2 s vs 26.6 s on average) and its accepted quotes had
a lower Cyrillic share (0.307 vs 0.42).

And the honest half: on a separate free-form synthesis task — one prompt
per model — ERNIE outscored Qwen4-Max (5/8 vs 3/8 on a fact rubric),
while both models invented the same two numbers that were not in the
source, and ERNIE invented two more besides. Extraction quality and
synthesis trustworthiness are different axes, and a single leaderboard
hides that.

4. Two production blocks, two different models

With the geometry fixed by tests 1–3 (48K chunks, quota 20 — no scaled
quota anywhere), the pipeline ran at production scale, and the two blocks
ran on different models:

Block 1, local Qwen3.5-9B: 727 of 727 unit jobs completed, 1,187 chunks, 15,841 accepted quotes. 96 chunk responses (8.1%) failed strict parsing and were skipped by the code — not recovered inside the run; a separate rescue shard re-runs them through the hosted model.
Block 2, Qwen4-Max-Preview: 345 of 345 prompts completed across two lines, at 30–32 chunks per active session-hour per line. No reply was zero-byte; two were valid JSON with empty quote lists. We have no scored completeness figure for block 2 yet, and we claim none.

The advertising claims, checked

Four marketing-style claims travel with this stack — stated here in our
words, as we encountered them; we sourced no vendor copy. We measured
what we could.

"New major version: faster, better." v1.0.0 was slower at both quotas we run (+1.5% and +16.9%), and its quality ranking flipped with a prompt parameter. Not confirmed on this task.
"Bigger context windows let the model use more of your data." What we actually varied was chunk size at a fixed context, and coverage fell from 0.4288 to 0.2460; one unit silently returned nothing; one chunk did not fit the window at real token density. Long prompts were accepted mechanically — capacity is real. The quality implication is not confirmed; window size alone was never isolated.
"Flagship reasoning, reliable out of the box." At our pipeline's 4,800-token output budget: 71.4% parse failures on the subset — and in prose, invented numbers. At ~3.3× the budget the same model's extraction improved sharply. Blanket reliability: not confirmed.
"Flagship hosted model, dependable." For extraction on the tested calls it was the cleanest arm we measured: zero measured rejects, zero parse failures. For prose it invented the same two numbers as its rival and scored below it on the synthesis rubric. Dependable is per task shape, not per model — partially not confirmed.

The scoreboard: what is better where, and how to apply it

Server version (b11669 vs v1.0.0): no outright winner — v1.0.0 is better at quota 20 (+11.8 points of completeness), b11669 is better at quota 40 (+7.6 points) and faster at both. Apply: benchmark an upgrade on your own quota profile before adopting it. We stayed on b11669.
Chunk geometry (48K vs 192K chars): 48K wins quality (0.4288 vs 0.2460 completeness at fixed context and quota); 192K wins only raw speed — and loses the data. Scaling the quota with the chunk does not fix it (tested: 0.2670). Apply: 48K chunks, quota 20; count tokens, not characters.
Reasoning-model output budget (4.8k vs 16k tokens): 16k, decisively — parse failures fell from 71.4% to 21.4%, because hidden reasoning ate ≈86–87% of the budget either way. Apply: give reasoning models ≥16k output tokens for structured extraction, or do not use them for it.
Hosted extraction (Qwen4-Max vs ERNIE at 16k): Qwen4-Max for quote fidelity — 257 accepted quotes, overlap 0.537, zero measured rejects, zero parse failures, against ERNIE's 193 / 0.479 / 1.5% / 3-of-14. ERNIE wins speed (26.6 s vs 111.2 s per call on average) and Cyrillic share (0.42 vs 0.307). Apply: fidelity → Qwen4; throughput or language mix → ERNIE.
Local 9B vs hosted flagships: the local model is the only arm scored on ground truth at production scale — 15,841 accepted quotes in block 1, data never leaving the box, zero API cost — at a 12.0% rejection share in the bake-off sample, where Qwen4 measured zero. Apply: bulk on local, hard residue to the hosted model. That rescue pattern is what our pipeline actually runs.
Free-form synthesis (either hosted model): neither, unverified. ERNIE scored higher (5/8 vs 3/8, one prompt each) — and both invented the same two numbers that were not in the source. Apply: any number in model prose gets checked against the source before use. No exceptions.

What we take from this

The largest single lever we measured was chunk size (−18.3 points for 48K → 192K at fixed context and quota). A version swap moved +11.8 points; a quota change moved +3.9. Measure the configuration, not the label.
With a fixed extraction quota, larger chunks measurably destroy coverage. Context-window size by itself was not isolated in these tests, and we make no claim about it either way.
For reasoning models, the output budget is a quality parameter: raising it ~3.3× moved one model from 71.4% parse failures to 21.4% (its own metric — no cross-metric ranking is made).
Report the inconvenient half. Every table above includes the arm that makes our own production choice look worse on some axis — because it was.

Method note: we report wall-clock on the real task rather than synthetic
tokens/second; the one assembled figure is labelled where it appears. One
cautionary datapoint for co-tenancy fans: a co-resident sandbox instance
on partial offload on the same box generated at 2.31 tok/s — a
measurement rig, not a serving mode.

Verification

These numbers were cross-checked before release, by named reviewers on different models. An independent audit session (Daybreak Blue program, GPT-6 Sol, ultra reasoning) returned a not fit to publish verdict with 18 findings — including a model-attribution error of ours that our own first audit had missed. A parallel Daybreak review flagged three further defects. All three were confirmed against the records; the 18 findings were reconciled the same way, and every figure was re-derived from the raw run records. The text above is the corrected version. Where a checker and the records disagreed, the records won.

Follow **Paragon 666* on DEV — we publish reproducible bench notes: method and raw numbers with every result, including the inconvenient halves.*

DEV Community: Paragon666