Per-tensor quantisation on the card you already have

A number is a weight format

A model file stores its weights as numbers. The width of each number - eight bits, four, three - is a choice, and it trades memory and bandwidth for accuracy. Quantisation is that choice made on purpose, per tensor, instead of one global format forced on the whole model.

The effect is physical. Halve the bits and the file roughly halves; the weights you stream per token halve too, and read time follows memory, so the model gets faster on the same machine. That is why a quantised model answers on a card the full-precision file could not touch. It is not a smaller brain. It is the same brain written down more compactly, with the cost of the loss visible and measured.

Per tensor, because layers are not equal

Uniform quantisation is the blunt version. Per-tensor quantisation sets a scale for each tensor, so a layer that is sensitive to rounding keeps a wider format and one that tolerates it goes narrower. Modern formats keep attention and a few critical projections wider while the bulk of the expert weights go tight.

That is what makes an oversized model usable rather than merely loadable. The gap between "it starts" and "it still answers well" usually lives in the tensors the quantiser chose to protect. A GGUF type name tells you which widths are in play, which is why the engine version and the exact format belong beside any speed figure we publish.

The honest part

Quantisation is lossy, so the right posture is to measure it, not assert it. We run the same prompts and the same read/write timing across formats on the same box and report both throughput and a quality check, so a four-bit figure and an eight-bit figure compare as a pair rather than as a win. The benchmark index holds them side by side, versions included.

None of this is a subscription decision. It runs on the hardware already in front of you - Metal, CUDA, Vulkan, SYCL, or CPU alone - and inside your building, so the files doing the answering never leave it. Ask about an on-site assessment at [e-mail].

Throughput and quality per format are recorded in the benchmark index: https://localai.hex7.com/benchmark.html.