Convert MLX Model to GGUF (KL Divergence Proof)

Converting a fine-tuned MLX model to GGUF for llama.cpp is one merge script and one convert_hf_to_gguf.py call away. Plenty of guides stop there. Nothing in that path tells you the GGUF you end up with actually matches the model you fine-tuned — a broken merge, a lossy container conversion, or an over-aggressive quantization all produce a file that loads and answers questions, just wrong ones, some of the time, in ways a quick chat test won't catch. The model in this tutorial is the same one we fine-tuned with LoRA on one Mac Studio; this picks up exactly where that tutorial leaves off, with a trained adapter in hand and llama.cpp as the target runtime instead of MLX.

This is the method we used to prove it, on a real 27B fine-tune, with the exact commands and the real numbers. Three separate links, each measured inside the one tool that can actually measure it, because there is no single tool that can compare an MLX model and a GGUF model directly: llama.cpp's own KL-divergence tool only compares GGUF against GGUF inside its own runtime, and MLX has no view into llama.cpp's logits.

[!NOTE]
Prerequisites

• A LoRA-fine-tuned MLX model: the original Hugging Face bf16 checkpoint plus the trained adapter, in mlx-lm's adapter format.
• llama.cpp built or installed (we used Homebrew's llama.cpp at tag b10330), plus a source checkout with a Python environment for convert_hf_to_gguf.py — the pip package and the conversion script are usually separate from the compiled binaries.
• Disk: roughly 2x your model's bf16 size free at the peak (merged bf16 plus bf16 GGUF co-resident, before you delete the intermediates). For a 27B model that was about 106 GB.
• A held-out validation text file for the KL-divergence measurement — we used our training mix's own validation split, not a separate calibration corpus.

[[steps]]

1. Verify the merged bf16 matches the original: merge the adapter into the source Hugging Face checkpoint, not the MLX base, and check it.
2. Convert to GGUF bf16, then verify the GGUF bf16 conversion matches across runtimes.
3. convert_hf_to_gguf.py and llama-quantize: the conversion commands for the actual quantized files.
4. Measuring quantization loss with llama-perplexity --kl-divergence: GGUF quantization KL divergence, the metric that actually proves parity.
5. Task-probe the served model, not just the logits.

Verify the merged bf16 matches the original

The adapter has to be merged into the original Hugging Face bf16 checkpoint — not the MLX-converted, quantized base you actually trained on top of. The low-rank update is the same arithmetic either way (W' = W + scale * (lora_a @ lora_b).T, fp32, stored back in bf16), but convert_hf_to_gguf.py needs the original HF tensor layout and naming to work at all, and the adapter's own keys need a rename to match it (language_model.model.layers.N.<module>.lora_a becomes model.language_model.layers.N.<module>.weight in the merged output).

MLX_DISABLE_COMPILE=1 python fuse/merge_lora_hf.py \
  --base models/Qwen3.8-27B-bf16 \
  --adapter results/t9-27b-anchor/adapter \
  --out models/Qwen3.8-27B-Atlassian-hf-bf16

[merge] 176 adapted modules, scale 2.0, base models/Qwen3.8-27B-bf16 -> models/Qwen3.8-27B-Atlassian-hf-bf16
[merge] DONE 176/176 modules merged into models/Qwen3.8-27B-Atlassian-hf-bf16 in 28s

The script asserts every adapter pair is consumed exactly once and every module's shapes match before it writes anything, so a mismatched key or a shape error stops the merge rather than silently skipping a module.

That is link 1: does the merged bf16 actually behave like base-plus-adapter? Check it the same way you'd check any merge, with a greedy generation comparison on a fixed prompt set, before you spend the next twenty minutes converting a file that was already wrong:

python probe/greedy_gen.py render
python probe/greedy_gen.py gen   # base+adapter, then the merged bf16, same 20 prompts
python probe/greedy_gen.py compare

AGREE 16/20 identical; mean common-prefix fraction 0.888

How you know it worked: most of the 20 prompts should produce byte-identical output between base-plus-adapter and the merged model, and the ones that don't should diverge late, at a genuine near-tie between two tokens, not from the first token. A much lower agreement than the 16 of 20 we measured here is a sign the merge itself is wrong, and no amount of downstream GGUF work will fix that.

Convert to GGUF bf16, then verify it matches across runtimes

convert_hf_to_gguf.py on a bf16 source is meant to be a lossless container change, not a lossy step — same weights, different file format. That is worth checking rather than assuming, because it is exactly the kind of step that fails silently when a model architecture is new enough that the converter's tensor-name mapping hasn't caught up.

.venv-convert/bin/python convert_hf_to_gguf.py \
  models/Qwen3.8-27B-Atlassian-hf-bf16 \
  --outtype bf16 \
  --outfile models/gguf/Qwen3.8-27B-Atlassian-bf16.gguf

If your model ships a multi-token-prediction or speculative-decoding head, check those tensors specifically survived the conversion — ours did, converted from torch.bfloat16 to the GGUF F32 metadata format, under names like blk.64.nextn.shared_head_norm.weight.

Then the second link: does the GGUF bf16 file, read by llama.cpp, produce the same output as the merged bf16 file read by mlx-lm? Two different runtimes, two different tensor libraries, same weights — this is the check that catches a converter bug a pure file-diff would miss.

python probe/llama_gen.py --gguf models/gguf/Qwen3.8-27B-Atlassian-bf16.gguf \
  --prompts prompts.json --out llama-gen.json
# compared against the same 20 prompts' mlx-lm output from link 1

AGREE 17/20 identical; mean common-prefix fraction 0.945

How you know it worked: agreement in the same range as link 1 — you're comparing the same underlying weights through two different inference engines, so you'd expect similar or slightly better agreement than the merge check, not worse. If cross-runtime agreement is meaningfully lower than your merge-verification agreement, the conversion step introduced its own divergence and it's worth checking the converter's tensor-mapping table for your specific architecture before trusting anything built from this file.

convert_hf_to_gguf.py and llama-quantize: the conversion commands

With a verified bf16 GGUF in hand, quantizing it is the fast, mechanical part. We shipped two quant levels, Q8_0 and Q6_K, both using an importance matrix built for this model family:

llama-quantize --imatrix models/unsloth-imatrix/imatrix_unsloth.gguf \
  models/gguf/Qwen3.8-27B-Atlassian-bf16.gguf \
  models/gguf/Qwen3.8-27B-Atlassian-Q8_0.gguf Q8_0

llama-quantize --imatrix models/unsloth-imatrix/imatrix_unsloth.gguf \
  models/gguf/Qwen3.8-27B-Atlassian-bf16.gguf \
  models/gguf/Qwen3.8-27B-Atlassian-Q6_K.gguf Q6_K

model size 52115.19 MiB (16.00 BPW) -> quant size 27690.97 MiB (8.50 BPW)   # Q8_0, 29,047,084,672 bytes on disk
model size 52115.19 MiB (16.00 BPW) -> quant size 21381.38 MiB (6.56 BPW)   # Q6_K, 22,431,000,192 bytes on disk

A third, more aggressive quant level (Q4_K_M) is possible with the same command and a different quant-type argument — we deliberately did not ship one for this release. A KL divergence and task-probe number has to be published alongside a quant before it can carry a "matches the original" claim, and we didn't run that ladder for Q4_K_M this time. Don't ship a quant level you haven't measured, even if the command to produce it is one word different from the one you did measure.

How you know it worked: the reported bits-per-weight for each quant should roughly match the quant name's own bit width (Q8_0 near 8.5, Q6_K near 6.5 — the extra fraction over the nominal bit count is metadata and scale factors), and the file size should land close to (model parameters × bits-per-weight) / 8.

GGUF quantization KL divergence: the metric that actually proves parity

File size and bits-per-weight tell you how much you compressed. They tell you nothing about how much the model's actual output distribution moved. That is what KL divergence measures directly: for each position, how far the quantized model's probability distribution over the next token has drifted from the unquantized reference's distribution at the same position. A small KL divergence means the quantized model is, on average, predicting almost the same thing the full-precision model would have.

Measuring quantization loss with llama-perplexity's --kl-divergence flag is a two-pass process, and this is the step where we broke our own measurement the first time.

# pass 1: build the reference distribution from the bf16 GGUF
llama-perplexity -m models/gguf/Qwen3.8-27B-Atlassian-bf16.gguf \
  -f valid-text.txt --kl-divergence-base bf16.kld -c 2048 --chunks 40 -ngl 99

# pass 2: compare each quant against that reference
llama-perplexity -m models/gguf/Qwen3.8-27B-Atlassian-Q8_0.gguf \
  -f valid-text.txt --kl-divergence --kl-divergence-base bf16.kld -c 2048 --chunks 40 -ngl 99

The bug: --kl-divergence-base on its own is the same code path as --save-all-logits — it builds and saves a reference file. Without the separate --kl-divergence flag on the second pass, llama-perplexity happily runs, writes a clean exit code, and just saves another logits file instead of comparing anything. We ran the per-quant passes this way the first time, got no error, and had no measurement at all — the fix was adding the one flag we'd assumed --kl-divergence-base alone implied.

With both flags present, the real numbers, 40 chunks of 2,048 tokens each, held-out validation text neither the quant nor the reference had been tuned on:

-- Q8_0
Mean ln(PPL(Q)/PPL(base)): 0.000925 ± 0.000460
Mean KLD: 0.000842 ± 0.000276
Maximum KLD: 10.878710

-- Q6_K
Mean ln(PPL(Q)/PPL(base)): 0.004168 ± 0.000556
Mean KLD: 0.003060 ± 0.000273
Maximum KLD: 6.626617

Both mean KLD figures are small fractions of a nat, and both are well inside the range the quantization community treats as a safe 8-bit and 6-bit quant respectively. The maximum KLD numbers (10.9 and 6.6) look alarming next to the mean, and that's expected, not a red flag by itself: a maximum is the single worst token position across 81,920 measured positions (40 chunks × 2,048), and one rare, high-entropy position moving further than average doesn't say anything about the other 81,919. Look at the mean and the standard deviation for the parity claim; keep the maximum as a thing to spot-check, not the headline number.

How you know it worked: you get an actual Mean KLD line for each quant, not just a Final estimate: PPL line with no divergence figure — if all you have is perplexity, the divergence pass didn't run and you're missing the step this whole section exists for.

Task-probe the served model

KL divergence tells you the quantized model's output distribution is close to the reference's. It doesn't tell you the model still does the specific thing you fine-tuned it to do. The last link is running your actual evaluation — whatever probes you use to judge the model in the first place — against the GGUF served through llama-server's OpenAI-compatible endpoint, and comparing those results to the same probes run against the MLX release.

llama-server -m models/gguf/Qwen3.8-27B-Atlassian-Q8_0.gguf \
  --port 8097 -ngl 99 -c 40960 --jinja --reasoning-format deepseek

Two flags are worth testing deliberately rather than assuming: whether your chat template's thinking-mode toggle is actually honored through the server (we found enable_thinking=false via chat_template_kwargs was honored on a dry run against the smaller sibling model in this same family, which is not guaranteed by every chat template on every llama.cpp version — test it on yours before trusting a thinking-off probe run), and, if your model ships a speculative-decoding head, whether --spec-type draft-mtp actually engages it and at what speedup. Every published MTP number for llama.cpp we could find is measured on CUDA; our own earlier llama.cpp-vs-MLX MTP measurement on this same model family is the closest Metal precedent we have, and it did not show the kind of speedup CUDA numbers report — so if you're on Apple Silicon, measure your own number rather than assume a CUDA figure transfers.

We hit two more server-side surprises running this against the real 27B build, both in exactly the category the paragraph above warns about — a param that looked handled but wasn't. First, llama-server's chat template silently defaults reasoning_effort to its highest setting when the field is absent from the request, and our serving client was sending it as a top-level body field the way our MLX server honors it, which llama-server does not; the identifier probe ran at max reasoning depth and 11 of the first 22 answers hit the token cap before we caught it and started sending reasoning_effort inside chat_template_kwargs instead, which both servers honor. Second, --reasoning-budget 0 under --reasoning-format none did not actually suppress thinking — the first 13 shape briefs came back with a <think> block sitting in the content and zero of them passed — where the per-request enable_thinking=false kwarg from the paragraph above did work, so that's the flag to reach for, not the budget one. Neither bug was in the weights; both were in how two different servers route the same-sounding parameter.

How you know it worked: your probe suite's pass rate on the served GGUF should land close to the MLX release's own pass rate, within the noise your probe suite normally shows between two runs. On our own run, once both flag bugs above were fixed, the Q8_0 file matched the MLX release on identifiers (85% pre-April accuracy, 15% post, same split) and landed within the gate's own noise on shape (28.7 of 35 apps, mean of three passes, against the MLX release's own 30.0). A GGUF that scores meaningfully worse than its three measured links would suggest has a problem the logit-level comparisons didn't catch, usually in how the server applies the chat template or handles the reasoning/thinking toggle rather than in the weights themselves.

What this doesn't prove, and what we didn't ship

This method proves the GGUF's weights and immediate output distribution track the source model closely. It does not, on its own, prove every capability the source model has survives identically — that's what the task-probe step is for, and it's worth running your real evaluation there, not just the logit-level checks. We use this same pattern, a rule with named conditions and printed inputs rather than a single blended score, to decide whether any new training round replaces the one before it; the model this tutorial converts is graded by the same kind of rule before it ever reaches this pipeline.

Two things we deliberately left out of this release, worth naming so you don't have to rediscover why: we did not ship a runtime LoRA GGUF (convert_lora_to_gguf.py plus --lora at serve time) — the mlx-lm-format adapter needs a PEFT-layout conversion first, that conversion path is reported unmaintained for mlx-lm's specific layer naming, and it would move the merge to the user's own runtime, which is exactly the step this whole method exists to measure once, here, rather than leave to chance on someone else's machine. And we did not ship the vision projector output — our source models are text-only fine-tunes, so there was nothing there to convert.

The two files this whole method produced are public: Qwen3.8-27B-Atlassian-Q8_0-GGUF (29.0 GB) and Qwen3.8-27B-Atlassian-Q6_K-GGUF (22.4 GB), both on Hugging Face, both carrying the same parity numbers this tutorial walked through.