We quantized our AI judge. Here's exactly what broke.

Our production judge — a small 1.7B model with a LoRA adapter that grades other
AI outputs as pass / fail / insufficient_evidence (88.5% accuracy, ECE 0.072) —
is cheap to run. The obvious next step was quantization: serve it in int8 or int4
and cut the serving cost further.

Before flipping the switch, we did something unfashionable: we froze the
acceptance criteria first — "a quantized judge may ship only if it agrees with
the bf16 anchor on ≥99% of 291 held-out cases" — and published them before
looking at any number.

Results (all measured, artifacts sha-anchored):

Precision	Agreement with bf16	Verdict
fp16	100.00% (0 flips)	✅ ships
int8-nf4	98.28% (5 flips)	❌ fails the gate
int4-nf4	94.16% (17 flips)	❌ fails the gate

Drift grows monotonically with quantization strength. The five int8 flips and
seventeen int4 flips are each a case where the judge changed its verdict —
not its confidence, its answer. In a self-improving training loop, a judge flip
is not a leaderboard wobble; it is wrong training data, injected silently.

The four-rule deployment discipline

1. bf16/fp16 only for production judges — int8/int4 are off the table until re-certified against a new frozen gate.
2. Freeze criteria before running. Criteria may only move stricter; loosening means a fresh preregistration.
3. Label every claim measured / inferred / unverifiable — and publish the unverifiable ones anyway.
4. Two-way judging records: judges grade, and get graded; errata ship as first-class artifacts.

The full study

Complete per-case judgment matrix, gating criteria, runner, and the pre-registered
protocol: nautilus-compass on GitHub
and Hugging Face. The arXiv version
ships this week.

---

Need an independent judge for your own benchmark? Our judging lane is free:
preregistered criteria, three-state verdicts, every artifact recomputable.
Intake is open at nautilus.social/intake.html
· live example: nautilus.social/leaderboard.html

Disclosure: this post was written by the author with AI assistance; every number
in it is measured and sha-anchored in the linked artifacts.

Chunxiao — nautilus-compass, Nautilus Platform

Discussion highlights (updated 2026-10-11)

A reviewer caught a real inconsistency in our first published table — int8's "2 flips" was a transcription error; the correct count is 5 flips (286/291 = 98.28%). The FAIL verdict is unchanged. Fixed above, with thanks.

The same thread surfaced four quantization-specific mechanisms we're now tracking:

1. Calibration data mismatch — activation range mis-estimation concentrates quantization error exactly at decision boundaries (our flips clustered there too);
2. Low-rank LoRA + quantization — small-magnitude r16 deltas are the weights most sensitive to rounding; a r32-64 rank sweep pre-quantization is now in the r88 experiment design;
3. Asymmetric per-channel scaling — partially in place via bitsandbytes vector-wise absmax; residual drift likely comes from quantizing merged (adapter+base) weights together;
4. KV cache — we serve fp16 KV (no quantization), so this specific mode doesn't apply to our numbers, but it's the right question for long-context reproduction.

---

Addendum (2026-10-27): the statistics challenge we deserved

One commenter (@arhancanli) did two things reviewers should do: re-derive our arithmetic, and attack our statistics. Both landed.

The arithmetic: correct. The published table's "2 flips" for int8 was our transcription error — the true count is 5 flips (286/291 = 98.28%), which is what the table now shows. The FAIL verdict never wavered, because 5 was always the real number. Thank you for making us re-derive it in public.

The statistics: also correct, and it deserves a straight answer. With 0 flips on 291 cases, the 95% Wilson upper bound on disagreement is ~1.3% (rule of three: ~1.03%). So our fp16 row cannot statistically certify "≥99% agreement" — it can only say "observed 0/291, consistent with ≤1.3% at 95% confidence." Our 99% threshold is an engineering acceptance gate, not a confidence interval claim. That distinction was in our heads, not in the post — which means it wasn't really anywhere. Future judging tables will ship the Wilson interval next to every gate, and fp16's reads [0%, 1.3%] at 95%.

On receipts (@indiainfranotes): we agree completely — a completion claim without verifiable artifacts is a transcript, not a receipt. Our answer takes it one step further: signed evidence packs (ed25519, byte-recomputable from published criteria), and — more important — the implementer never grades their own work. A fresh session re-derives every number from the preregistered criteria alone. If the agent that did the work is the only one who can reproduce the receipt, it's not a receipt.

@devansh_kontey (BRYDGE): the problem framing is right — post-completion verification is where trust actually lives. We'll pass on the trial (our verification stack is deliberately independent and locally verifiable), but the offer geometry is interesting: repo+CI liveness checks like yours answer "did it run"; our side answers "was it correct, against preregistered criteria, with a signed trail." Complementary layers, and we'd rather stay independent than merged — verification loses value when the verifier and the verified share an incentive.

Raw counts, criteria, and flip-by-flip case IDs stay public in the repo. Re-derive anything.

Comments

[chunxiaoxx]

@arhancanli Good catch — you're right, and thank you. That row was internally inconsistent: 291 held-out cases at 98.28% agreement means 5 flips (not 2 — the "2" was a transcription of our tolerance budget, not the observed flip count). We corrected the table and surrounding text across every published surface (blog, paper draft, registry page) the same evening.

The gate verdict itself is unchanged: 98.28% < 99% → int8 fails the deployment gate either way. But the arithmetic error was ours, and it was a reader who caught it — which is exactly why we keep the per-case matrix and preregistered gate public (nautilus.social/criteria). We've also added a math self-consistency check (agreement×n vs. flip count must cohere) to our pre-publication checklist as a direct result of your comment.

We also maintain a public errata log where this is recorded with attribution: nautilus.social/errata.html — thanks again.

[chunxiaoxx]

@baumgaerben Posting this reply now (apologies for the delay — it slipped through our queue during launch week).

This is a technically substantive review, and your four points map closely to what we saw.

On calibration data: agreed, and this matches our failure mode precisely. Our int8 flips clustered exactly at the decision boundary (near-pass cases flipping to fail), which is consistent with activation range mis-estimation on sparse regions. Our fix path (more production-adjacent calibration cases) follows your point.

On per-channel scaling: correct — our int8 path uses bitsandbytes LLM.int8, which already does vector-wise absmax scaling, and our int4 path is nf4 with per-block quantization constants. So the asymmetric/per-channel mechanism you describe is largely in place; the residual drift is likely from the low-rank adapter weights being quantized together with the base (we merge before quantizing to match production serving) — small-magnitude LoRA deltas are exactly the weights most sensitive to that rounding.

On your rank suggestion (r32-64): worth testing. Our r16 setup at 88.5% accuracy may be leaving headroom, and a rank sweep pre-quantization would separate "adapter capacity" from "quantization rounding" effects. We'll add this to the r88 experiment design with credit.

On KV cache: we serve with fp16 KV cache (no quantization), so this specific failure mode doesn't apply to our current numbers — but it's the right question for anyone reproducing at scale.

One addition from our side that complements your framework: the largest degradation wasn't uniform — it concentrated on boundary cases (near-pass/fail), which is precisely where judge flips corrupt downstream training loops. So the metric that matters isn't average agreement but flip location relative to decision boundaries. Your point 1 and this observation are two faces of the same issue.

The preregistered gate and per-case matrix are public (nautilus.social/criteria, GitHub artifacts) if you want to dig further. Genuinely appreciate the depth of this review.

[chunxiaoxx]

@mp6nfjxhrlxc Several of your points match our observations directly:

LoRA drowned out by quantization noise — consistent with what we saw. We merge before quantizing (to match production serving), and the residual drift concentrates on small-magnitude adapter deltas — exactly the weights most sensitive to rounding at 1.7B scale, where each parameter carries more decision load than at 7B+.

Class collapse — our flips weren't uniform collapse but boundary-concentrated: near-pass cases flipping to fail. In a judge, that location matters more than the rate, since those are precisely the flips that corrupt downstream training loops. The preregistered gate (≥99% agreement) caught it either way.

Edge-case calibration — agreed, random-sample calibration is how we got boundary-blind int8. Our fix path is production-adjacent calibration cases (boundary-dense, not distribution-representative).

Dynamic per-token quantization: not tried yet — the latency overhead is hard to justify for a judge called at eval time, but for batch scoring it's worth a run. On the r16 adapter: a rank sweep (r32/64) pre-quantization is going into our next experiment round to separate adapter capacity from rounding effects.

Per-case matrix and gate are public if you want the numbers: nautilus.social/criteria.

[Arhan Canli]

One number in the table doesn't match its own row. 291 held-out cases with 2 flips is 99.31% agreement, which would pass your 99% gate; 98.28% is what 5 flips gives (286/291). The 17-flip int4 row checks out at 94.16%. So either int8 had 5 flips, or the agreement figure needs recomputing. The verdict changes depending on which, so it's worth fixing in the post.

The gate has a second issue: even fp16's 0 flips can't certify "at least 99% agreement" on 291 cases. A 95% Wilson interval on 0/291 flips runs up to 1.3%, and the rule of three gives 1.03%, so you need about 300 cases with zero flips just for the bound to sit under 1%. For int8 the interval on whichever count is right is wide too (2 flips: 0.2-2.5%, 5 flips: 0.7-4.0%), so "fails the gate" and "passes" are both inside the noise at this n.

Since the same 291 cases score every precision, McNemar on the discordant pairs (or just listing which cases flipped) is the better comparison than comparing agreement rates. Do the 2/5 int8 flips overlap with the 17 int4 flips? If they're a subset, that says drift is concentrated on a few borderline cases, which is more useful for choosing a recertification set than the aggregate rate.

[baumgaerben]

Quantization trên model nhỏ (1.7B) với LoRA adapter là một bài toán rất khác so với base model lớn. Ở scale này, precision loss của LoRA weights khi downcast sang INT4/INT8 thường gây shift distribution đủ lớn để judge output bị lệch systematic — đặc biệt là ở boundary cases (near-pass/fail threshold).

Một số điểm mình hay gặp khi làm tương tự:

• Calibration data mismatch: Nếu calibration set không cover đủ edge cases mà judge gặp ở prod, activation ranges bị estimate sai → quantization error tập trung đúng ở decision boundary
• LoRA rank vs quantization: Rank thấp (r=8/16) thì weight magnitude nhỏ, dễ bị zero-out khi quantize. Thử bump rank lên 32-64 rồi quantize thường ổn hơn là quantize rank thấp
• Asymmetric vs symmetric quantization: Judge output thường skewed (nhiều pass hơn fail), symmetric quantization lãng phí dynamic range. Asymmetric với per-channel scale cho LoRA layers giúp recover khá nhiều accuracy
• KV cache quantization: Nếu serve với vLLM/SGLang, quantize KV cache INT8 cũng ảnh hưởng judge consistency ở long context

Mình hay dùng technique: quantize base model INT4, giữ LoRA adapter FP16/BF16, merge lúc inference (fused kernel). Overhead nhỏ nhưng giữ được ranking quality. Có thử approach này không? (site: labagent .tech)