Converting a Hugging Face Model to MLX Format

#ai #machinelearning #tutorial #llm

Conversion to MLX is much less dramatic than conversion to GGUF. There is no new container format: the output is still a directory of safetensors files plus a tokenizer and a config.json. What changes is the parameter naming, the dtype, and — if you ask for it — the numeric representation of the weights.

What conversion actually changes

mlx_lm.convert loads the checkpoint through MLX’s own model implementation for that architecture, which is where the renaming happens: the Python module for, say, Llama declares its own parameter tree, and saving it back out writes keys in that tree’s shape. It then writes the tokenizer files through unchanged and writes a config.json that records what it did.

Three things are decided at conversion time and baked into the output directory:

dtype. --dtype accepts float16, bfloat16 or float32. Left unset, the converter defaults to the torch_dtype in the source config, or the dtype the weights are already in.
Quantization. -q turns it on and writes a quantization block into config.json recording bits, group size and mode. That has its own page — quantizing a model for MLX — and this one converts at full precision first so you can see the size difference for yourself.
Nothing else. There is no calibration set, no importance matrix, no per-model tuning step. Conversion is close to mechanical, which is why the whole mlx-community organisation can be produced by running this command repeatedly.

Install and check the Python

Confirm you are on a native Python. Apple’s MLX installation page tells you to run python -c "import platform; print(platform.processor())" and expect arm; i386 means an x86 interpreter under Rosetta and MLX will not import. Fix that first — nothing below works otherwise.
Install the package: pip install mlx-lm. It pulls in mlx, transformers and the Hugging Face hub client.
If the model you want is gated, authenticate now with hf auth login and a token from your Hugging Face account settings. Accepting the licence is done on the model page in a browser, on the account the token belongs to.

Convert

The minimal invocation names a source and a destination. The source can be a Hub repository id or a local directory:

mlx_lm.convert \
  --hf-path meta-llama/Llama-3.1-8B-Instruct \
  --mlx-path ./llama-3.1-8b-mlx \
  --dtype float16

--model is accepted as an alias for --hf-path in current releases, and the project’s own README now uses it; older material and older versions use --hf-path. Both parse today. If you are writing this into a script, pick one and pin the mlx-lm version.

Two flags are worth knowing before you need them. --revision pins a specific commit on the source repository, which is the only way to make a conversion reproducible given that upstream repositories are mutable. --trust-remote-code is required for architectures whose tokenizer ships custom Python; it executes code from the repository, so use it on repositories you would run code from.

--mlx-path must not already exist. The converter raises Cannot save to the path ... as it already exists. Please delete the file/directory or specify a new path to save to. rather than overwriting, which is deliberate and saves you from clobbering a good conversion with a failed one. Delete or rename before re-running.

Verify it loads

A conversion that wrote files is not a conversion that works. Load it and generate something:

from mlx_lm import load, generate

model, tokenizer = load("./llama-3.1-8b-mlx")

messages = [{"role": "user", "content": "Name three prime numbers."}]
prompt = tokenizer.apply_chat_template(messages, add_generation_prompt=True)

text = generate(model, tokenizer, prompt=prompt, max_tokens=64, verbose=True)

With verbose=True the generation loop prints three lines after the text — the prompt token count and prompt tokens per second, the generated token count and generation tokens per second, and the peak memory in gigabytes. Those are the numbers to note down; they are the only honest tokens-per-second figures for your hardware, and they are what the bandwidth arithmetic predicts a ceiling for.

The equivalent one-liner, if you only want to know whether it runs, is mlx_lm.generate --model ./llama-3.1-8b-mlx --prompt "hello".

Worth a moment of scepticism: a conversion can load, generate fluent text, and still be wrong, because a mismatched chat template produces plausible prose rather than an error. The check that catches it is to render the template and look at it — tokenizer.apply_chat_template with tokenize=False returns the string the model will actually see, and comparing that against the upstream repository’s template takes ten seconds. Role markers are the thing most likely to have been copied through from a stale source.

Gated weights, and the errors you will hit

Llama, Gemma and several other families are gated: the weights are public but the repository requires you to accept the licence with your account before the Hub will serve the files. There is no flag that bypasses that, and the mirrors that appear to are redistributing weights under terms their uploader may not have. Accept the licence on the model page and log in; if your organisation cannot accept the terms, choose a model whose licence it can.

Three failures account for most of the rest:

An unsupported architecture. mlx-lm needs a Python module for the model type in config.json. If there isn’t one, conversion fails at load, and the fix is to wait for the implementation or use a family that has one — not to force it.
Disk, not memory. Conversion downloads the full precision checkpoint into the Hugging Face cache under ~/.cache/huggingface and then writes a second copy at --mlx-path. A 70B in bf16 is roughly 141 GB before you have an MLX copy of it. The cache is not cleaned up for you.
Memory during load. The converter loads lazily, so it does not need the whole model resident to convert it — but the machine still has to hold what it is writing. If you see MLX’s allocator complain, that is the memory-pressure page, not a conversion bug.

One asymmetry to know about before you delete the source. The converter has a -d / --dequantize flag that reverses the direction, turning a quantized checkpoint back into full-precision tensors — but it restores the dtype, not the information. Dequantizing a 4-bit model gives you an fp16 file with 4-bit values in it, four times the size and none of the precision. Quantization is lossy in one direction only, so keep the source or keep the ability to re-download it.

Once it loads, the natural next step is to make it smaller. Convert again with -q and compare the two directories side by side.