Here's a fact that still bothers me: a four-bit quantized model file tells you how many elements it has and what dtype tag they carry — and almost nothing else that matters. Not which rule derived the shared scales. Not the zero-point convention. Not the sub-byte packing order. Not the layout its scale plane was written in. The file works only because the producer and the consumer happen to agree on all of it, silently, in code.
In 2026 alone, six documented incidents across vLLM and SGLang turned exactly those silent agreements into silently wrong model output. Not crashes — wrong numbers. Scale tensors dropped on load. Signed scales read as unsigned — roughly four orders of magnitude of dequantization error. A swizzled producer wired to a linear consumer. Every one of them loaded cleanly.
2026, vLLM + SGLang: six documented incidents, five failure classes — every one loaded cleanly
what silently went wrong
the field that makes it loud
Signed scales read as unsigned
≈ 4 orders of magnitude of dequant error
exact format identity (s8 ≠ u8)
Swizzled producer → linear consumer
layout disagreement, both sides "correct"
placement id mismatch
Scale tensors dropped on load
plane simply missing, nothing complained
§7.3 plane-size equations
Silent E8M0 truncation
scale format narrowed without a word
closed format ladder
GPTQ "zero point minus one" (as late as Jan 2026)
tens of thousands of zero points → NaN perplexity
declared zpc flag
Not crashes — wrong numbers. Each is a question something at the boundary could have answered.
The documented 2026 failure classes across vLLM and SGLang, and the GRIT field that turns each one from silently-wrong-numbers into a loud boundary failure. None of these crashed; all of them loaded cleanly and produced wrong output.
So I spent a stretch of this year building GRIT — the Grouped Reduced-precision Interchange Type. It's my answer to a simple question: what is the smallest thing a checkpoint could carry that would have made those failures loud? The paper is on Zenodo, the spec and all five implementations are on GitHub, and everything below reproduces from a clean checkout.
What GRIT actually is
A GRIT array is a quadruple: (Grade, Placement, Planes, Shape). The Grade is the complete numeric contract — element format, scale-derivation rule, zero-point convention, rounding, sparsity, the works — with a canonical string form and a 64-bit id. The Placement is the physical layout, carried as a value rather than baked into a type parameter, so "which layout" and "which numeric contract" stop being one fused enum name like marlin_24. The Planes are up to four byte buffers — data, two scale levels, aux. The Shape comes from the caller.
A GRIT array is a quadruple — and the whole quadruple travels with the bytes
Grade
the numeric contract:
formats · scale rule · zp
rounding · sparsity
Placement
physical layout,
carried as a value —
a new layout is a constant
Planes
up to 4 byte buffers:
data · scale0
scale1 · aux
Shape
rank + extents,
carried in the view
- 64-byte POD descriptor grade + hashed frame + shape ⇒ 128-bit gid
grit_check at every boundary — O(1), total, no undefined behaviour
any 64 bytes → exactly one status · equal gid + equal planes ⇒ bit-identical decode()
The quadruple. Grade carries the numeric contract, Placement carries the layout as a hashed value, Planes carry the bytes, Shape comes from the view — and all of it serializes to one 64-byte descriptor whose agreement with the bytes is checked in O(1) at every producer-to-consumer boundary.
All of it fits in a 64-byte plain-old-data descriptor — every field at a fixed offset, little-endian, no variable-length anything. The descriptor plus the shape determines every plane's exact byte length, so "does this descriptor match these bytes?" is decidable in O(1) at a boundary crossing. And the check is total: any 64 bytes you throw at it, including adversarial garbage, terminate with exactly one status and no reads outside the buffer.
The whole contract: 64 bytes, every field at a fixed offset, little-endian
magic·verlevels·flags
elem·scale0scale1·zp
axisk0·k1
sparse·container
note
placement × 3data · scale · metadata
grade_idFNV-1a-64
08162228325664
Flag bits carry what folklore used to: zero-point presence and convention, sparsity, bit order, interleave.
note (bytes 28–32) is assumption provenance — the one field excluded from every fingerprint,
so annotating a file never changes its identity.
Descriptor + shape ⇒ every plane's exact byte length ⇒ "does this descriptor match these bytes?" is O(1).
The 64-byte descriptor, byte-accurate. Identity and flags, then formats from a closed ladder, then group geometry, sparsity and container packing, the fingerprint-excluded note field, three 64-bit placement ids, and the FNV-1a-64 grade id. The grade, hashed frame and shape fingerprint together into a 128-bit gid: two tools holding the same gid hold the same contract.
The part I'd defend hardest isn't the descriptor, though. It's the law set attached to decode(): NaN poison, sparse-wins-over-poison, slice honesty, group-axis transpose as requantization rather than a view, fixed outermost-first evaluation order, no FMA substitution, placement invariance. Formats tell you what bytes mean; almost nobody writes down what the decoder is allowed to do. That's where the silent divergence lives — and it's exactly the edge where existing semantics bow out: StableHLO's 0 < scales constraint affirmatively excludes the hardest of these cases, so the two semantics are disjoint precisely where implementations diverge in practice.
The two guarantees the whole project stands on, both testable:
Equal gid and equal plane bytes ⇒ bit-identical decode() on every conformant implementation;
-
grit_checknever has undefined behaviour, even on adversarial 64-byte descriptors.
My favorite finding: two files, identical bytes, opposite meanings
Nobody rewrites their checkpoint format because a blog post asked nicely. So the wedge is grit scan: a read-only auditor that checks a declared contract against bytes where a descriptor exists, and infers one from container evidence where it doesn't — which today means GGUF and safetensors files you already have on disk.
I pointed it at four real Hugging Face checkpoints. My favorite finding: a GPTQ file and an AWQ file whose zero-point planes are identical in byte count, shape, dtype and tensor name — and carry opposite conventions. One stores zero points as-is; the other stores them minus one. Load one as the other and every weight shifts by a full quantization step. Nothing anywhere in either file records which convention is inside.
The experiment you can run today: two files, identical zero-point bytes, different numbers
GPTQ-Int4 · qzeros
0x77 0x77 0x77 0x77 …
same shape · same dtype · same name
AWQ-Int4 · qzeros
0x77 0x77 0x77 0x77 …
same shape · same dtype · same name
⇩ decode ⇩
zpc = minus1 : (w − (z+1)) · s
the GPTQ folklore convention
zpc = asis : (w − z) · s
the AWQ convention
Load one as the other ⇒ every weight shifts by one full quantization step — and the file loads cleanly.
GRIT's fix is one declared flag bit: zpc ∈ { asis, minus1 } — folklore promoted to a checkable field.
The ambiguity you can download today: GPTQ and AWQ zero-point planes that are byte-identical — same shapes, dtypes, tensor names — while decoding to different numbers, because the minus-one convention lives in tool source code instead of in the file. GRIT's zpc flag is one declared bit that ends the guessing.
That one genuinely surprised me. I expected the scanner to find sloppy metadata; I did not expect two of the most widely deployed quantization families to be formally indistinguishable at the byte level while meaning different numbers. It's the purest possible specimen of the whole problem: the meaning isn't in the file. It's in a comment thread somewhere, and in the source of whichever loader you happen to use.
The scanner, and an experiment with a control group
grit scan — the wedge that needs zero adoption
GGUF ·safetensors
- descriptor found → verify it
no descriptor (today) → infer
size equations · gid diffs ·
convention ambiguity · grade drift
Inferred contracts are labeled inferred, never declared;
what the grammar can't express is reported inferred_inexpressible, never approximated.
Graded, CI-gateable exits:
0 clean
1 violation
2 disagreement
3 warn
4 parse
pip install grit-datatype && python3 -m grit.scan --deep --json PATH...
grit scan verifies declared descriptors where they exist and infers contracts from ggml block structs, llama.cpp file types, and the GPTQ / AWQ / compressed-tensors families where they don't — labeling every inferred contract as inferred, never declared, and reporting what the v1 grammar can't express as inferred_inexpressible instead of approximating. Graded exit codes make it a one-line CI gate.
Claims about scanners are cheap, so the field study is reported with both runs, warts first. We downloaded four real, popular checkpoints — GGUF Q4_K_M, GPTQ-Int4, AWQ-Int4, compressed-tensors W4A16 — and read them byte by byte. Run 1, hand analysis plus the scanner as first shipped: real mismatches confirmed in three of the four files, but the tool auto-caught only one of them, indirectly. Honest score: not good enough. We closed exactly three inference gaps and re-ran on bit-identical bytes. Run 2: 12 findings became 349, and every mismatch class now fires automatically. The part that makes the number mean something: a false-positive control on two known-clean files stayed at zero findings, exit 0. The complete experiment log, both runs, is audit/scan-experiment.md in the repo.
Four real checkpoints, two runs, the same bytes
GGUF Q4_K_M · GPTQ-Int4 · AWQ-Int4 · compressed-tensors W4A16 — read byte by byte
Run 1 — scanner as first shipped
12 findings (hand analysis confirmed mismatches in 3 of 4 files; the tool auto-caught 1, indirectly)
Run 2 — three inference gaps closed, re-run on bit-identical bytes
349 findings — every class fires
False-positive control — two known-clean files
0 findings, exit 0
bar length ∝ findings
The two-run structure, reported as run: 12 findings from the scanner as first shipped, 349 after closing three inference gaps and re-scanning the same bytes — with a known-clean control at zero findings. n=4 checkpoints: this establishes the mismatch classes exist in the wild, not how prevalent they are.
Proof over promises
The project is built to be checked rather than trusted. One normative spec (2,164 lines), an executable Python reference, and five zero-dependency implementations — C11, C++20, Rust, pure-stdlib Python, strict TypeScript — that reproduce a 68-vector SHA-256-pinned conformance suite and agree bit-for-bit on 96/96 cross-language descriptor fingerprints, verified by a committed harness and a CI job that re-proves it on every push, not a one-off script. The Rust implementation is differentially fuzzed; the C++ one runs under AddressSanitizer and a strict build.
Five implementations, zero dependencies each — one bit-for-bit contract, CI-proven
C11 910 checks · 0 failures
C++20 801 checks · 0 failures · ASan+strict
Rust 37 tests + full conformance suite
Python 124 tests · stdlib only, numpy never imported
TypeScript 91 tests · strict mode
bar length ∝ suite check count (not coverage) — every suite at 0 failures
Cross-language gids: 96/96 identical across all five + the reference — 96 distinct gids
The shared suite: 68 SHA-256-pinned vectors
34 positive15 negative16 fingerprint3 supersedes
The verification surface: five implementations with zero dependencies each, every suite at zero failures, all five plus the reference agreeing on 96/96 cross-language gids — and the shared 68-vector conformance suite broken out by kind. One command reproduces the cross-language proof: bash spec/crosslang/run.sh.
And the check is cheap enough to leave on. A level-1 structural check costs 296 ns in C (887 cycles) and 687 ns in Rust; on a synthetic 8-shard, 1.07 GB MXFP4 checkpoint, checking every tensor at load time adds 2.7 ms, 33.4 KB of headers, and 108 KiB of resident memory for 192 checks. Against a multi-second checkpoint load, the safety margin is effectively free.
What the boundary check costs — one call, level-1 (structure)
log scale (100 ns → 10 µs) · measured on one i9-13900HK, indicative not guaranteed · harnesses in bench/
C 296 ns (887 cycles)
Rust 687 ns
Python9.8 µs
100 ns1 µs10 µs
A whole checkpoint: 1.07 GB · 8 shards · 192 checks at load time
+2.7 ms wall clock · +33.4 KB of headers · +108 KiB RSS
Per-call cost of the level-1 structural check on a log scale, and the whole-checkpoint picture: 192 checks on a 1.07 GB shard set cost 2.7 ms of wall clock. One machine, indicative numbers — the harnesses that produced them ship in bench/ and the methodology is §8.4 of the paper.
What GRIT does not claim
This section exists because the project's motto has to apply to its own marketing. The spec carries a component-by-component claims table with the closest prior art for each piece, and the honest scope statement next to it. Parameterised quantized types, nested two-level scales, and scale-plus-sparsity in one format are not GRIT's inventions — see MLIR's sub-channel quantized types, compressed-tensors, and Qualcomm's LPBQ. The arithmetic-contract-as-a-value is deployed art in StableHLO and JAX's DotAlgorithm. Canonicalise-then-fingerprint is Apache Avro's discipline, step for step — GRIT changes the object being hashed. The portable POD descriptor pattern is the Khronos Data Format Specification's and DLPack's, down to the same LSB-first sub-byte packing rule. And the FNV-1a fingerprints defend against drift and mislabeling, not against an adversary — there is no collision-resistance claim.
The v1 grammar also has real holes, all named in the spec rather than papered over: AMD's FNUZ FP8 variants are inexpressible; there is no codebook/LUT element class, so NF4 and the llama.cpp IQ* families are out of scope; GPTQ act-order g_idx grouping is unsupported, because act-order can't be hidden in Placement without making the placement-invariance law false; and dense-nibble zero-point planes — GPTQ/AWQ qzeros, two values per byte — are not byte-representable under the v1 padding rule. The scanner reports all of these as inferred_inexpressible instead of pretending. And the field study is n=4: it establishes that the mismatch classes exist in the wild, not how prevalent they are — prevalence needs a stratified sweep of hundreds of checkpoints, which is future work.
How it was built
GRIT started as a dare, not a product idea: find something genuinely missing, and only build it if the gap survives an adversarial attempt to prove it already exists. Before any code, the idea was attacked with a sweep of the closest prior work — MLIR sub-channel types, StableHLO DotAlgorithm, compressed-tensors, Avro, Khronos DFS, DLPack, torchao, TOSA block-scaled types, OCP MX, IEEE P3109 — and every novelty claim that did not survive was retracted before publication. What survived was not a format but a missing contract, and that decided everything else: a checkable type, not another container.
I'll be straightforward about the method, because it's part of the story: I built GRIT in an intensive collaboration with Claude (Anthropic) — spec drafting, orchestrated implementation across the five languages, and above all adversarial verification: exact-arithmetic oracle sweeps against every encoder, a hostile three-reviewer panel run against the paper before release, a novelty sweep against everything from StableHLO to OCP MX to P3109, and a citation audit that fetched every reference against the published record. The working rules were simple: every number must regenerate from a clean checkout, and every reviewer finding is either fixed or documented as a limitation — never softened. Every number in the paper traces to a repo artifact. The motto of the whole project applies to its own construction: nothing here asks to be trusted; everything here asks to be checked.
Try it on a checkpoint you already have
# the scanner — zero adoption required
pip install grit-datatype
python3 -m grit.scan --deep --json path/to/checkpoints/
# the type, in your language of choice
cargo add grit-datatype
npm install grit-datatype
# reproduce every claim in this post from a clean checkout
bash spec/crosslang/run.sh # 96/96 on every implementation
GRIT on GitHub — spec (normative), paper, all five implementations, benchmarks, the field-study log
Project site · interactive workbench — build and corrupt descriptor bytes in your browser and watch the check catch them; it runs the byte-identical npm build
The paper — DOI 10.5281/zenodo.21817716
PyPI · crates.io · npm — the repo tag and all three registries move in lockstep
Try the scanner on a checkpoint you already have. If it finds something I didn't predict, that's the most useful thing you could possibly tell me.
GRIT is open source under Apache-2.0, built in the open by a human–AI team — direction, constraints and the standard of evidence by the author, with Claude (Anthropic) as repository co-author. Cost figures are one-machine measurements presented as indicative; the harnesses that produced them are in the repo. The field study is n=4 and claims existence of the mismatch classes, not prevalence.
Top comments (0)