Every other 4-bit format gives you 4 bits. EXL2 files are labelled things like 4.65bpw or 3.0bpw, and the fractional part is not marketing rounding — it is a real average over layers that were individually quantized to different integer widths. That design is what lets you fill a specific card exactly rather than choosing between too small and too big.
Why 4.65 bits is a coherent thing to ask for
The constraint that actually matters when running a model locally is not a bit width. It is the amount of memory you have, minus what the KV cache will need for the context length you want, minus the runtime’s own overhead. That leaves a number of gigabytes for weights, and it is essentially never equal to the size of a whole-model 4-bit bake or a whole-model 5-bit bake.
Integer-width formats leave you rounding down and wasting the gap. A 24 GB card holding a 4-bit 34B model has several gigabytes unused, and nothing to spend them on. EXL2’s answer is to make the bitrate a continuous dial: ExLlamaV2 supports 2, 3, 4, 5, 6 and 8-bit quantization and mixes them within a model to reach any average between 2 and 8 bits per weight. The quantization method underneath is the same GPTQ-style error-compensating reconstruction described in GPTQ explained; the novelty is the allocation.
The arithmetic of an average bitrate
An average bitrate is a weighted mean over parameters, not over layers, and the weights are the layers’ parameter counts. Take a model with 7.0e9 parameters in quantizable linear layers and suppose the allocator has put 20% of them at 6 bits, 60% at 4 bits and 20% at 3 bits:
bpw = 0.20*6 + 0.60*4 + 0.20*3
= 1.2 + 2.4 + 0.6
= 4.2 bits per weight
weights on disk = 7.0e9 * 4.2 / 8 bytes
= 3.675e9 bytes
= 3.675 GB (3.42 GiB)
Two caveats on that number and both matter. First, it counts only the quantizable linear weights: embeddings, the output head and the norm parameters are stored separately, and in ExLlamaV2 the head is typically kept at a higher precision than the body, so the file is larger than the formula says. Second, the stated bpw for an EXL2 checkpoint already includes the per-group scale overhead, which is why a “4.0bpw” EXL2 file and a 4-bit GPTQ file with group size 128 are near each other in size rather than the EXL2 one being smaller — see group size arithmetic for where those extra fractions of a bit come from.
Run the same arithmetic backwards to size a bake. If you have 22 GiB free after reserving cache and want a 34B model in it:
target_bytes = 22 * 1024**3 = 2.362e10
bpw = target_bytes * 8 / 34e9
= 5.56 bits per weight
That is the number you pass to the converter — minus a margin for the head, the embeddings and fragmentation.
What the measurement pass measures
Handing the allocator a target average is only useful if it knows where the bits should go. ExLlamaV2’s conversion is therefore two passes, and the first one exists purely to answer that.
The measurement pass quantizes each linear layer several times, once per candidate setting — different bit widths, different group sizes — and records the reconstruction error each setting produces against the calibration data. The result is a per-layer error curve: for this matrix, going from 4 bits to 3 costs this much, and for that matrix it costs much more. Those curves are written to a measurement.json in the working directory, and the second pass solves the allocation problem against them — spend the bit budget where the curve is steepest.
The practical consequence is that the measurement pass is the expensive half and it is reusable. It depends on the model and the calibration data, not on the target bitrate, so quantizing one model to five different bitrates means one measurement and five cheap conversions. ExLlamaV2 exposes this directly: -om writes the measurement out and exits, and -m reads one back in.
Flag names, defaults and the set of supported bit widths belong to one project’s CLI and have changed across ExLlamaV2 releases. Check the conversion documentation in the ExLlamaV2 repository for the version you have installed rather than trusting a flag copied from a blog post.
Running the conversion
- Obtain the full-precision weights. Many strong open models are gated on Hugging Face and require accepting a licence on the model page before the download works; that gate is the licence being enforced, and it is the correct route.
- Install ExLlamaV2 and prepare a working directory with plenty of free space — the converter holds intermediate state on disk, and the temporary footprint exceeds the output.
- Run the measurement pass once, writing it out:
python convert.py -i /models/base -o /work -om /work/measure.json. This is the long step. - Run each bitrate against the saved measurement:
python convert.py -i /models/base -o /work -m /work/measure.json -cf /out/4.65bpw -b 4.65. Repeat with a different-band-cfper target. - Load the result and check the context length you actually need fits alongside it. A bake that leaves no room for the KV cache is a model that runs until the third turn of a conversation.
Choosing a bitrate for a card you own
The honest position on where quality falls off is that nobody publishes a per-model curve you can trust for your workload, and reconstruction error on calibration text is not the same as your task getting worse. What can be said from the format’s own structure is narrower and more useful:
- The marginal bit is not worth the same everywhere. Going 4.0 → 4.65 buys the allocator budget to lift the layers with the steepest error curves, which is a different and better purchase than lifting everything by 0.65 bits uniformly. This is the same principle GGUF’s K-quant mixes apply with a fixed recipe rather than a solved one.
- Below about 3 bits per weight the format is doing triage. At that budget most layers are at 2 bits and the allocator is protecting a minority. Whether that is acceptable is a question about your task, and the only way to answer it is to run your own evaluation on both bakes.
- Bitrate trades against context, not just against quality. Memory freed by a lower bitrate goes to the KV cache. A 4.0bpw model with 16k of context may serve you far better than a 5.5bpw model that only reaches 4k, and the format makes that trade explicit in a way integer formats do not.
Top comments (0)