The dual-preprocessing gap is the one that quietly ruins the most models, and it's rarely the quantization itself — it's that normalization constants live in two files in two languages that nobody diffs, and they only have to agree once to pass review. Making the Dart side unable to restate the manifest is the right structural fix: you can't drift from a value you're not allowed to redeclare.
Where I'd extend it: the manifest + content-hash catches "the artifact changed," but not "the artifact is numerically worse within tolerance." Those are different failures. The hash gate is binary; quality regression is a distribution. A parity gate that a build can fail on needs a golden set — a frozen batch of inputs with expected outputs — run post-export and post-quantization, failing the build if top-1 agreement or max abs error against the pre-quant model crosses a threshold. That turns "compare offline vs online, once, the week before launch" into something CI enforces every commit, which is exactly the never-again-after-launch trap you named. Is Fluttorch's parity gate doing pass/fail on a tolerance, or just reporting the delta?
Pass/fail, and it fails the build. The gate replays a frozen set of goldens
captured from the source model before lowering, so what it measures against is
the pre-quant reference, and the check is an absolute plus relative bound per
output tensor with an optional cosine floor. The default bound depends on the
quantization recipe and the precision, and it was measured rather than picked. If
you hand it a recipe nobody has measured it refuses to run instead of inventing a
number, and an empty bundle fails too, because a gate that passes with nothing to
check is the exact thing it exists to prevent.
You're right about what's missing, though. Everything it measures is elementwise,
so a classifier can sit comfortably inside the tolerance and still flip its argmax
on a borderline input, and the gate would call that green. Top-1 agreement isn't a
stricter version of what's there, it's a different measurement, and I hadn't
written it. It's open now as github.com/NaCode-Studios/Fluttorc..., with your comment linked:
the ordering-versus-distance framing is the part that made it obvious.
Two things I'd rather say than have you find. When the recipe is static, the
exporter currently calibrates on the same goldens it later grades itself against,
which makes the drift look better than it is. And the on-device half doesn't run
in CI yet: the Dart side does, against a committed export, but the workflow in the
docs is the shape it takes, not something a device proves on every commit.
For further actions, you may consider blocking this person and/or reporting abuse
We're a place where coders share, stay up-to-date and grow their careers.
The dual-preprocessing gap is the one that quietly ruins the most models, and it's rarely the quantization itself — it's that normalization constants live in two files in two languages that nobody diffs, and they only have to agree once to pass review. Making the Dart side unable to restate the manifest is the right structural fix: you can't drift from a value you're not allowed to redeclare.
Where I'd extend it: the manifest + content-hash catches "the artifact changed," but not "the artifact is numerically worse within tolerance." Those are different failures. The hash gate is binary; quality regression is a distribution. A parity gate that a build can fail on needs a golden set — a frozen batch of inputs with expected outputs — run post-export and post-quantization, failing the build if top-1 agreement or max abs error against the pre-quant model crosses a threshold. That turns "compare offline vs online, once, the week before launch" into something CI enforces every commit, which is exactly the never-again-after-launch trap you named. Is Fluttorch's parity gate doing pass/fail on a tolerance, or just reporting the delta?
Pass/fail, and it fails the build. The gate replays a frozen set of goldens
captured from the source model before lowering, so what it measures against is
the pre-quant reference, and the check is an absolute plus relative bound per
output tensor with an optional cosine floor. The default bound depends on the
quantization recipe and the precision, and it was measured rather than picked. If
you hand it a recipe nobody has measured it refuses to run instead of inventing a
number, and an empty bundle fails too, because a gate that passes with nothing to
check is the exact thing it exists to prevent.
You're right about what's missing, though. Everything it measures is elementwise,
so a classifier can sit comfortably inside the tolerance and still flip its argmax
on a borderline input, and the gate would call that green. Top-1 agreement isn't a
stricter version of what's there, it's a different measurement, and I hadn't
written it. It's open now as
github.com/NaCode-Studios/Fluttorc..., with your comment linked:
the ordering-versus-distance framing is the part that made it obvious.
Two things I'd rather say than have you find. When the recipe is static, the
exporter currently calibrates on the same goldens it later grades itself against,
which makes the drift look better than it is. And the on-device half doesn't run
in CI yet: the Dart side does, against a committed export, but the workflow in the
docs is the shape it takes, not something a device proves on every commit.