Your parity gate must enforce the number you publish: a testing methodology for porting ML models across runtimes

You ship an ML model port from Python to another runtime, say Swift, C++ or ONNX. The build succeeds. The model loads. You feed it the canonical test image and the output looks right: valid JSON, sensible structure, plausible coordinates.

Then you measure pixel-for-pixel against the Python reference. One bbox edge is off by 9 px. Or 200 px. Deterministically. On every test image. It has been that way for weeks, because your test gate ran at a 30 px tolerance and never flagged it.

Silent drift like this hits any cross-runtime ML port that doesn't make a strict parity gate a first-class part of the build. Here is the four-component setup that catches it, generalized for your own port.

Your parity gate has to enforce the number you publish. A gate looser than the claim is not a gate. It's a decoration that gives you false confidence.

The 30 px gate that hid a 9 px bug

The published claim was "≤ 2 px parity on all 8 edges of both panels" against the Python mlx-vlm reference. The gate enforcing that claim ran at TOLERANCE=30 px, and had run at 30 px since early prototyping, when anything sub-50-px was a win.

Two of three canonical test images were drifting by ~9 / ~8 / ~5 px on outer edges. Nobody noticed, because the gate was 15× looser than the published number it was supposed to enforce.

Preparing to upstream the patch forced the forensic re-measurement, which put the actual measurements next to the published prose and exposed the gate as a decorative artifact rather than a check.

The fix wasn't a single bug, though there was a bug, in chat-template content ordering. The deeper fix was recognizing that the test infrastructure no longer matched the claim it was supposed to enforce. Once a strict gate was wired to the actual published number, the bug surfaced on the first run.

Identical weights do not mean identical output

A port keeps the weights and the architecture, and that does not guarantee the same output. The places where sub-pixel error accumulates:

• Image preprocessing. Every framework's "Lanczos" is a different Lanczos. PIL, Core Image, OpenCV, and TensorFlow's tf.image.resize all produce subtly different output for the same input. A 1- or 2-level difference at 8-bit precision in a few hundred edge pixels propagates through attention.
• Numerical precision boundaries. When the model mixes FP16/BF16 in some places and FP32 in others, the exact ordering of cast → multiply → sum can move a least-significant bit. Across a few thousand multiplies, the LSBs add up.
• Tokenizer / chat-template subtleties. If your runtime emits image-token positions one off from what the reference does, attention is one position off everywhere. The output looks plausible because the model degrades gracefully. It just degrades wrong.
• Initialization timing. Anything that persists between calls: caches, position-ID buffers, RoPE state. A buffer that resets in Python but persists in your port produces different output on the second image than the first.

Each alone produces small drift; stacked, they produce wildly wrong output. None of them throws an error: they all pass shape checks and they all return something.

You only catch them if you compare your runtime's output against a known-correct reference, at the strictness you care about, on every edge of every output, every time the gate runs.

Four things have to exist together

1. A reference that pins three things

The reference is the source of truth. Every other piece compares against it.

canonical_baselines.json (the name in this project; call it whatever) holds the Python reference output for every canonical test image, generated at temperature=0 with a pinned model snapshot:

{
  "model": "mlx-community/Qwen2.5-VL-7B-Instruct-4bit",
  "snapshot_hash": "fdcc572e8b05ba9daeaf71be8c9e4267c826ff9b",
  "mlx_vlm_version": "0.4.3",
  "max_edge_delta_allowed": 2,
  "prompt": "Detect these two UI panels...",
  "images": [
    {
      "name": "leetcode_test",
      "path": "_zero_px_test/leetcode_test.png",
      "size_px": [3078, 2114],
      "model_resize": [1260, 868],
      "panels": {
        "question": [1, 146, 421, 626],
        "editor":   [421, 146, 881, 626]
      }
    }
  ]
}

This file pins three things: the model snapshot hash, the reference framework's version, and the input dimensions / preprocessing parameters that produced the output. All three are required. Change one and the saved output stops being a valid reference, so the gate must be re-baselined deliberately rather than passing trivially.

The published claim, "≤ 2 px on every edge", sits in that file as max_edge_delta_allowed: 2. The number now lives in code, not in prose: publishing a different number means editing this file, and the gate adjusts with it.

2. Compare every edge, not every box

strict_2px_gate.py runs the freshly-built non-reference binary on every canonical image, parses the model's output, compares each edge of each output element against the saved reference, and exits non-zero if any edge exceeds the allowed delta.

for img in spec["images"]:
    out = run_runtime(args.binary, ipath, rw, rh)
    panels = parse_bboxes(out)
    for label, ref_bbox in img["panels"].items():
        actual = panels.get(label)
        if not actual:
            failed.append((img["name"], label, "missing"))
            continue
        for i, edge_name in enumerate(["x1", "y1", "x2", "y2"]):
            delta = abs(actual[i] - ref_bbox[i])
            if delta > max_allowed:
                failed.append((img["name"], label, edge_name, delta))

Two non-obvious things this gate gets right:

• Per-edge, not per-bbox. A bbox can average an IoU of 0.95 against the reference and still be wrong by 12 px on one specific edge, which makes IoU too forgiving as the headline metric. Per-edge maxes catch the asymmetric drift that the most subtle preprocessing bugs produce.
• Missing outputs are failures. A bbox for "question" but none for "editor" is not "no measurement." It is a failure, and the gate flags it loudly.

3. Prove the patch lines landed

setup_and_verify.sh is the only sanctioned way to set up a parity-test clone. It fresh-clones the upstream library at the pinned commit, applies your patch, builds the reproducer with the production-equivalent toolchain, and runs the gate, aborting loudly on any failure.

# After git apply, sanity-check the patch landed the canonical fix line:
if ! grep -q 'message.images.map { _ in ["type": "image"] }' \
        Libraries/MLXVLM/Models/Qwen2VL.swift; then
    echo "Patch applied but Qwen2VL.swift is missing the image-first chat-template fix."
    echo "The patch file is incomplete — re-derive from working tree."
    exit 1
fi

The grep-the-patched-checkout trick catches the most insidious version of this problem: a patch that applies cleanly but is missing a fix line, because the patch file itself was generated incompletely. The patched build looks fine and runs to completion, and it produces output that's wrong by 9 px. Your gate would have to be tight enough to catch that.

The grep makes the gate stricter by proving that the patch landed the specific lines that fix the bug you claim to have fixed.

4. Cross-check adapters at zero tolerance

If your project has more than one consumer (an SDK, an MCP server, a dynamic-library entry point, an alternative runtime path), wire them all to one cross-adapter gate:

# scripts/parity/cross_adapter_gate.sh — runs the same image through every adapter
A_BBOX=$(./run_adapter_a "$IMAGE")
B_BBOX=$(./run_adapter_b "$IMAGE")
C_BBOX=$(./run_adapter_c "$IMAGE")
diff <(echo "$A_BBOX") <(echo "$B_BBOX") || fail "A vs B"
diff <(echo "$A_BBOX") <(echo "$C_BBOX") || fail "A vs C"

This catches integration drift: one adapter holding stale state, taking a different image preprocessing path, or drifting from the canonical implementation. The expected result here is 0 px tolerance. All your adapters call the same model with the same preprocessing, so any difference is a bug.

In this project it runs at zero tolerance across every adapter that ships, catching anyone who breaks adapter parity before the field does.

There is no fuzzy pass

The gate runs in this order on every parity check:

1. Fresh-clone the reference library at the pinned commit. (No reusing dirty trees.)
2. Apply your patch. Fail if it doesn't apply cleanly.
3. Grep the patched tree for the literal lines your patch was supposed to introduce. Fail if any are missing.
4. Build with the production-equivalent toolchain (in this project's case xcodebuild, not swift build, which takes different code paths).
5. Run the binary on every canonical image.
6. Compare every edge of every output element to the saved reference.
7. Run the cross-adapter gate.
8. Pass = no edge of any output exceeds max_edge_delta_allowed AND every adapter agrees at zero tolerance.

Any step that fails aborts the whole gate with a specific error. There is no "warning"; there is no "fuzzy pass." There is "the published number holds" or "the published number does not hold." Binary.

Build the gate before the bug-hunt

Build the gate first, before the bug-hunt, because the bugs you find in your port are the ones the gate flags. A gate at the right tolerance turns "this is mysteriously wrong" into "this specific edge of this specific output exceeds the threshold by N px on this specific image." That's debuggable. A gate at the wrong tolerance produces no information at all.

The specific scripts above are tuned for a Swift port of Qwen2.5-VL on Apple Silicon with mlx-swift-lm as the runtime, and they replaced the 30 px gate. The methodology is platform-agnostic.

If you're porting an ML model across runtimes, the minimum-viable parity setup looks like this:

• One reference output file, pinned to a specific reference-framework version and model snapshot, with the number you'd publish embedded as a hard limit.
• One gate script that runs the non-reference binary on every test case and compares per-output-element, per-edge (or per-token, per-pixel, whatever the units of your model output are) against the reference, with a strict tolerance.
• One bootstrap script that builds the binary under test from a clean checkout with the canonical patch fully applied, and aborts if any patch line is missing.
• One cross-adapter sanity check if you have multiple consumers.

Four components: a few short scripts plus a JSON file. It runs in a few minutes after the first build, which dominates wall time anyway.

The parity gate must enforce the number you publish. A gate that's looser than the claim is not a gate. Build it first.

Provenance

The bug-hunt war story (failed model approaches, the bug list, what eventually worked) is on dev.to as On-Device Document Grounding on macOS: Getting Qwen2.5-VL to Actually Work in Swift. This post is the testing-methodology companion.

The scripts I described (canonical_baselines.json, strict_2px_gate.py, setup_and_verify.sh, cross_adapter_gate.sh) live in the project's scripts/parity/ directory if you want to adapt them.