Quantizing a Core ML Model for iOS

#ai #machinelearning #tutorial #llm

Core ML has two different compression families and picking the wrong one wastes the effort. For a model you are shipping inside an app, where the constraint is bytes on disk and the tolerance for accuracy loss is small, the one you want is palettization — and the step most guides omit is checking that the compressed model still produces the same answers.

Palettization, not linear quantization

Linear quantization maps a float range onto an integer range with a scale and a zero point. Palettization does something different: it clusters the weights in a tensor into 2^nbits distinct values, stores that lookup table once, and replaces every weight with an index into it. A 4-bit palette stores 16 float values plus a 4-bit index per weight.

The reason this suits Apple silicon is the shape of the error. Weight distributions in a trained network are not uniform — they are concentrated near zero with a long tail — and a k-means palette spends its levels where the weights actually are, while a linear grid spends most of its levels on empty range. For the same bit width you generally lose less. The general background is in inference quantization and the Core ML guide.

There is a second difference that matters for what you are optimising. Palettization is a weight-only compression: it changes how the parameters are stored, not what the arithmetic is done in. That is exactly right when your constraint is the size of the download and the bytes that have to move from memory per inference, and it is the wrong tool if your constraint is arithmetic throughput, because the values are still expanded before they are multiplied. If what you want is faster compute rather than a smaller file, linear activation quantization is the other family, it needs calibration data, and it costs more accuracy for the same nominal bit width.

The practical decision rule is therefore about which limit you are against. A model that fits comfortably but makes the app download too large wants palettization at as few bits as accuracy allows. A model that is fast enough but will not fit in memory on an older device wants the same thing. A model that is already small and simply too slow is not helped much by either, and the answer there is a smaller architecture rather than a smaller representation of the same one.

Deployment target gates the features

This is the trap that produces a confusing error before you have written any real code. Apple’s coremltools documentation states that palettized weight representation for mlprogram models is available from iOS 16 / macOS 13 onwards, and that 8-bit lookup tables and the per_grouped_channel granularity arrived with iOS 18 / macOS 15 (coremltools palettization overview). If your conversion sets a lower minimum_deployment_target, the newer options are simply not available to it.

coremltools gates compression features on deployment target and the set moves with each release. Check the version of coremltools you have installed against Apple’s documentation for that version rather than a general guide.

Compressing the model

Install the toolchain on macOS: pip install coremltools. The conversion and compression steps run on the Mac, not the device.
Convert your source model to an mlprogram with an explicit deployment target, and keep this fp16 model — it is the reference you will compare against.
Palettize the weights with coremltools.optimize.coreml.palettize_weights.
Save both models and compare their sizes on disk.
Run the same inputs through both and compare the outputs numerically. Do not skip this.

import numpy as np
import coremltools as ct
import coremltools.optimize as cto

# 2. Convert, keeping the fp16 model as the reference.
mlmodel_fp16 = ct.convert(
    traced_model,
    inputs=[ct.TensorType(name="input", shape=(1, 3, 224, 224))],
    convert_to="mlprogram",
    minimum_deployment_target=ct.target.iOS18,
)
mlmodel_fp16.save("model_fp16.mlpackage")

# 3. Palettize to a 4-bit palette, clustered per group of 16 channels.
op_config = cto.coreml.OpPalettizerConfig(
    nbits=4,
    mode="kmeans",
    granularity="per_grouped_channel",
    group_size=16,
    weight_threshold=512,     # skip tensors smaller than this many elements
)
config = cto.coreml.OptimizationConfig(global_config=op_config)
mlmodel_4bit = cto.coreml.palettize_weights(mlmodel_fp16, config)
mlmodel_4bit.save("model_4bit.mlpackage")

weight_threshold defaults to 2048 and exists because palettizing a tiny tensor costs you a lookup table and saves you almost nothing. mode takes kmeans, unique or uniform; kmeans is the default choice for post-training compression and the other two are for weights that are already discrete. The full parameter list is in Apple’s palettization API reference.

Verifying against the original

The size reduction is arithmetic you can predict: a weight that took 16 bits takes 4, so weight storage falls to roughly a quarter, plus the lookup tables and minus any tensor below the threshold. What you cannot predict is the accuracy, and a compression that silently degrades one class of input is worse than no compression.

import os

def size_mb(path):
    total = 0
    for root, _, files in os.walk(path):
        for f in files:
            total += os.path.getsize(os.path.join(root, f))
    return total / 1e6

print(f"fp16: {size_mb('model_fp16.mlpackage'):.1f} MB")
print(f"4bit: {size_mb('model_4bit.mlpackage'):.1f} MB")

# Compare outputs on real inputs, not random noise.
ref = ct.models.MLModel("model_fp16.mlpackage")
cmp = ct.models.MLModel("model_4bit.mlpackage")

worst = 0.0
for x in calibration_inputs:          # a few dozen real samples
    a = ref.predict({"input": x})["output"]
    b = cmp.predict({"input": x})["output"]
    cos = float(np.dot(a.ravel(), b.ravel()) /
                (np.linalg.norm(a) * np.linalg.norm(b) + 1e-12))
    worst = max(worst, 1.0 - cos)
print(f"worst cosine distance: {worst:.5f}")

Use real inputs. Random noise passes through a compressed network differently from real data, and a comparison on noise will tell you the model is fine when it is not. If the cosine distance on real samples is unacceptable, the ordered set of things to try is: raise nbits from 4 to 6 or 8; move from per_tensor to per_grouped_channel with a smaller group_size; exclude specific layers with op_name_configs; and only then consider palettization-aware training.

What goes wrong

Comparing on the Mac and shipping to a phone. predict on macOS runs on whatever compute unit Core ML picks there. The device may partition the model differently, so a final check on real hardware is not optional.
The .mlpackage is a directory. os.path.getsize on it returns the size of the directory entry, not the model. Walk it, as above.
Compression is weight-only. Palettization shrinks the stored parameters. It does not shrink activations or the intermediate memory a large input needs, so a model that runs out of memory on a long input will still do so.
Not every layer benefits. Small tensors, embedding tables you index rather than multiply, and anything below weight_threshold are left alone, so the realised saving is always somewhat less than the bit-width ratio suggests. Measure the file rather than predicting it.
Accuracy loss is not uniform across inputs. An aggregate metric on a held-out set can stay flat while one class or one input distribution degrades badly, because compression error concentrates wherever the weight distribution had a long tail. Compare per-class or per-slice, not just in aggregate, and pay particular attention to the rare inputs your product cares about.
Nobody publishes the size or accuracy you will get. The compression ratio depends on how much of your model’s weight mass sits in tensors above the threshold, which is a property of your architecture. The accuracy cost depends on your data. Both are cheap to measure with the code above and impossible to look up, which is why the verification step is the tutorial rather than an appendix to it.

One last sequencing note, because it saves a rebuild. Do the conversion and the compression as two separate, saved artefacts, as above, rather than as one pipeline that only writes the compressed output. You will want the fp16 model again — to try a different nbits, to compare a second granularity, to answer “was it always like this or did we do it” when a regression appears weeks later — and re-running the conversion to get it back is the slow half of this process.