Hello, everyone.
Point a camera at something and get a sentence back describing what is happening there. For a device like that, how many seconds it takes matters more than how clever the model is. Three seconds is useful for watching a doorway. Thirty seconds is a log file.
Today I port Microsoft's Mage-VL to Apple Silicon as an independent MLX implementation that does not depend on mlx-vlm, and confirm float32 agreement on all four paths: static images, frame-sampled video, proactive streaming, and codec-native sparse video. What each path agrees with differs, though. Images, video, and codec are compared against fixtures generated from the official PyTorch implementation; the streaming gate alone is compared against a plain-PyTorch implementation of the published reference semantics, because mamba-ssm has no macOS build. On top of that, I built an implementation that processes each segment as it arrives, and measured how far it can keep up with live input.
On a fixed clip, the best configuration is the codec path with 2-second segments on a Mac Studio M4 Max: at worst 3.48 seconds from the event to the complete sentence. A MacBook Pro M1 Max needs 4-second segments and reaches 6.59 seconds. These come from playing a 768x512 video file, not from a live camera.
What Mage-VL Is
Mage-VL is a vision-language model Microsoft released in July 2026 along with a paper. It combines a Qwen3-4B-Instruct-2507 decoder with its own Mage-ViT for about 4.74 billion parameters, and handles static images, video, temporal grounding, and event-driven proactive streaming from a single checkpoint. The official checkpoint is about 10.8 GB in BF16.
Its distinguishing feature is the codec-native path. Instead of tokenizing every frame uniformly, it uses patches from I frames plus the patches that the motion vectors and residual energy of P frames mark as having changed the most. Microsoft reports that this path cuts visual tokens by more than 75% and runs up to 3.5x faster than uniform frame sampling. That is a result from Microsoft's paper, not a number I reproduced here.
There is one trained model and four paths. The count is easy to misread, so here are the roles and the flow of data.
| Path | Input | What preprocessing does | Output |
|---|---|---|---|
| Static image | One image | smart_resize and patching | Generated text |
| Frame-sampled video | Video | Patches every uniformly sampled frame | Generated text |
| Codec-native sparse video | Video | Picks changed patches from I frames and motion vectors, packs them onto canvases | Generated text |
| Proactive streaming | Video (codec assumed) | The above plus a Mamba mixer gate | speak / silent decision, then text |
The ported code is published as kiarina/mage-vl-mlx. It is torch-free at run time: src/ and the two inference scripts do not need PyTorch (weight conversion and parity checking do).
Licenses
| Subject | License |
|---|---|
| microsoft/Mage | MIT |
| microsoft/Mage-VL checkpoint | Apache-2.0 |
| state-spaces/mamba | Apache-2.0 |
| This port, kiarina/mage-vl-mlx | MIT |
The port contains no upstream source and redistributes no weights. scripts/convert_weights.py downloads the checkpoint from Hugging Face and converts it locally.
One caveat is worth knowing. codec-video-prep, which the codec path requires, declares no license in its PyPI metadata and publishes no source. It is Microsoft's own dependency for that path, but check it before shipping anything if licensing matters to you.
What This Article Verifies
Four things.
- Whether all four Mage-VL paths can be reproduced as an independent MLX implementation, with float32 parity shown for each
- Whether the token efficiency of the codec-native path holds under every way of comparing
- What the streaming gate is good for, and what it is not
- How far Apple Silicon can keep up with live input
Relation to Earlier Implementations
This is not "the first MLX implementation." The positioning first.
- An MLX implementation for static images exists in
Blaizzy/mlx-vlmpull request 1745, merged on 2026-07-29 - An MLX implementation covering video and streaming is published as
rsravanreddy/Mage-VL-MLX. End-to-end logit parity has not been confirmed for it, so I treat it as a design reference, not as evidence of agreement
This port implements all four paths independently of both, and verifies float32 agreement path by path — against the official fixtures, except for the gate's reference as described above. That fact is the whole claim.
Reproducing the Environment
You need an Apple Silicon Mac, mise, uv, FFmpeg, and Git. The first run downloads the Mage-VL checkpoint (about 10 GB) and converts it to MLX format.
git clone --depth 1 --filter=blob:none --sparse \
https://github.com/kiarina/labs.git
cd labs
git sparse-checkout set .mise/tasks 2026/08/27/mage-vl-realtime-benchmark
mise trust . && mise trust 2026/08/27/mage-vl-realtime-benchmark
mise -C 2026/08/27/mage-vl-realtime-benchmark run
Pass only directories to sparse-checkout. Cone mode always checks out the files at the repository root, so there is no need to list Makefile or mise.toml; listing them fails on git 2.42 and later with fatal: '.gitignore' is not a directory. A fresh clone's config is untrusted, so run mise trust before the task.
The default run executes only the frames-backend matrix. The rest are separate tasks.
| Task | What it does | Extra requirement |
|---|---|---|
run (default) |
The frames matrix | — |
run codec |
The codec matrix (two passes: 24 fps and 8 fps) | Docker |
run tokens |
Sweeps max_new_tokens over 16 / 32 / 64 |
Docker |
run saturation |
Saturates the camera path and measures delay and loss | — |
run memory |
Samples the memory of a running Web UI | Start the UI separately |
The headline numbers in this article come from the codec path, so reproducing them means running this one. It needs Docker.
mise -C 2026/08/27/mage-vl-realtime-benchmark run codec
The parity work lives in a separate lab per path, but those are records rather than runnable labs. Each README carries how the fixtures were made, the commands compared, the attempts that failed, and the bar that was set; reproducing them is done from the ported repository at the commit that lab pins.
| Lab | Path |
|---|---|
2026/08/25/mage-vl-mlx-stage1-image-parity |
Static images |
2026/08/25/mage-vl-mlx-stage2-video-parity |
Frame-sampled video |
2026/08/25/mage-vl-mlx-stage3-streaming-gate |
Streaming gate |
2026/08/26/mage-vl-mlx-stage4-codec-native |
Codec-native sparse video |
To read them, list the directories in the checkout. mise trust is only needed for labs that have tasks.
git sparse-checkout set .mise/tasks \
2026/08/25/mage-vl-mlx-stage1-image-parity \
2026/08/27/mage-vl-realtime-benchmark
The verification plan and the results for every stage are collected in mage-vl-mlx-port.md.
[1] All Four Paths Match in float32
Compared against fixtures generated from the official PyTorch implementation (the streaming gate's reference differs; see below).
| Path | Result |
|---|---|
| Weight key mapping | 696 for the model, 64 for the gate; zero missing, unused, or shape mismatches |
| Vision tower, images | Relative error 8.9e-06 to 1.6e-05, cosine 1.00000000 |
| Greedy 64 tokens, images | Identical on all 3 images |
| Video preprocessing | Frame indices, grid, patch_positions, and pixel values bit-identical |
| Vision tower, video | Relative error 8.892e-06 to 3.606e-04, cosine 1.000000 |
| Greedy 64 tokens, video | Identical on all 3 clips |
| Streaming gate Mamba mixer | Max absolute error 2.682e-07 to 4.396e-07 (the bar set in advance was 1.0e-5) |
| Codec preprocessing | patch_positions and pixel values bit-identical (max_abs 0.0) |
| Greedy 64 tokens, codec | Identical on both clips, logits cosine 1.000000 |
Preprocessing came out bit-identical because Qwen2-VL's smart_resize, the fused rescale/normalize order, and the patch layout were all matched against Qwen2VLImageProcessor. A single bit of drift there makes downstream agreement unmeasurable.
This agreement is specific to float32. bfloat16 accumulates rounding differences between backends, so decisions flip near the threshold. Concretely, at time 0 of cut_event float32 gives 0.5022 and bfloat16 gives 0.4977, turning speak into silent. Running in bfloat16 and verifying in float32 is the practical split.
One more caveat. The reference for the streaming gate is not the mamba-ssm CUDA kernel. mamba-ssm has no macOS build, so the reference is a plain-PyTorch rewrite following the published reference semantics. Agreement with the CUDA kernel is unverified.
[2] Codec-Native Efficiency Depends on What You Compare It To
This was the most interesting part. Four ways of comparing the same video (soccer_goal, 193 frames). Uniform sampling is a flat 384 tokens per frame.
| How you compare | Result |
|---|---|
| Match temporal coverage | 384 → 18.4 tokens per frame (95% fewer) |
| Against a fixed 32-frame budget (12,288 tokens) | 71% fewer, while seeing 6x as many source frames |
| Against the default 8 frames (3,072 tokens) | Codec uses more (3,528 tokens) |
| Speed under those same conditions (vs 8 frames) | 12.6-14.3 → 9.6-11.0 tokens/s, so codec is slower |
In other words, "codec is fast" and "codec is slow" are both available if you leave the comparison unstated. Microsoft's up-to-3.5x figure reads as a comparison between the frame counts needed for equivalent understanding, which is not the same condition as the 8-frame row above.
For segment-by-segment processing, though, the reduction pays off directly. On the M4 Max with 4-second segments, handing the segment to codec preprocessing at its original 24 fps takes 2.230 seconds to generate; resampling it to 8 fps first takes 0.828 seconds. Same four seconds of video, 2.7x difference in generation time, purely from how it is handed over.
That comes back later in this article.
[3] The Streaming Gate Does Not Point at Event Times
The proactive streaming gate emits a speak / silent decision per segment. I measured it hoping it would fire on the moment a goal is scored. It does not.
| Observation | Result |
|---|---|
| Separating content types | Soccer broadcast 0.69-0.79 vs a quiet corridor 0.04-0.11 |
| Event times | Not indicated. Segments containing the shot sometimes score below the control |
| Firing on frames input | 0.0009-0.0062. Lowering the threshold does not turn it into a filter |
| Survives re-encoding of stills | Yes. Sports 0.79-0.82 vs a static scene 0.12-0.14 |
What the gate sees is "is this the kind of footage worth talking about," not "did something just happen." So in practice the right use is a low-threshold pre-filter, with the actual event decision made from the generated text. The Web UI's soccer goal preset is calibrated that way, on the codec backend with the gate at 0.3.
One more important fact: the gate assumes codec input. It barely fires on frames input, so pairing the frames backend with the streaming gate is not a working configuration.
Note that the preset was calibrated with one positive clip and one control clip. That is not a sample size that supports a precision / recall claim.
[4] How Far Apple Silicon Keeps Up
Now the main question. First, a definition of "real time."
- RTF (real-time factor): processing time divided by media duration. Above 1, a continuous stream accumulates delay forever
- First text: from the moment a segment closes to the first character appearing
-
Worst case, event to full text: the delay for an event that happens at the start of a segment, which is
segment length + full response
Fixed clip (glass_fall), bfloat16 model with a float32 gate, 16-token cap, median of 3 runs.
| Machine | Configuration | RTF | First text | Worst event → full text |
|---|---|---|---|---|
| M4 Max | codec (8 fps), 2 s | 0.734 | 1.263 s | 3.48 s |
| M4 Max | codec (8 fps), 4 s | 0.400 | 1.366 s | 5.62 s |
| M4 Max | codec (24 fps), 4 s | 0.862 | 3.180 s | 7.51 s |
| M1 Max | codec (8 fps), 4 s | 0.644 | 2.205 s | 6.59 s |
| M1 Max | codec (24 fps), 4 s | 1.542 | 7.166 s | — |
| M4 Max | frames, 2 s (best for frames) | 1.047 | 2.130 s | — |
| M1 Max | frames, 2 s (best for frames) | 1.991 | 6.766 s | — |
The frames backend never keeps up, on either machine, at any segment length. Its best result is 1.047 on the M4 Max.
Where the Time Goes
The bottleneck is not decoding; it is the generation prefill over visual tokens. Here are the per-stage medians for 4-second segments on the M4 Max.
Codec adds to preprocessing because of the container startup (0.084 s → 0.498 s), but everything after the visual tokens are made sparse gets shorter. Generation drops from 3.357 s to 0.828 s, and the vision tower from 0.774 s to 0.128 s. The gate costs under 0.05 s either way.
Resampling Decides What Is Achievable
The 2.7x difference from [2] shows up directly in RTF.
Without resampling, the M1 Max cannot keep up under any condition I measured. With it, 4-second segments push RTF below 1 for the first time. The M4 Max's best was 4-second segments before resampling; afterwards it keeps up at 2 seconds, and its worst-case delay drops from 7.51 s to 3.48 s — less than half.
The reason is the number of canvases in codec preprocessing. Frames are grouped with --group_size 32, and each group produces --images_per_group 4 canvases, so up to 32 frames it stays at 4 canvases and 576 visual tokens.
| Frames in the window | Canvases | Visual tokens |
|---|---|---|
| 8 / 16 / 32 | 4 | 576 |
| 64 | 12 | 1,728 |
| 120 | 20 | 2,880 |
So going from 2 fps to 8 fps on a 4-second window quadruples the codec's temporal resolution without adding any load to the model. Conversely, handing over a 24 fps clip untouched makes a 4-second window 96 frames, which lands on the growing side of that table. 32 frames is the ceiling for raising temporal resolution without adding visual tokens.
What I measured here is canvas count, visual tokens, and preprocessing time — nothing about how accuracy changes as temporal resolution goes up. So keeping window seconds × capture rate near 32 is a guide to not adding cost, not an optimum that accounts for quality.
Each machine needs its own settings. The M4 Max's 2-second configuration breaks on the M1 Max, at RTF 1.157.
Generation Length Is Also a Setting
There is one more axis that is easy to miss: max_new_tokens. Sweeping only the cap, on the M4 Max with codec (8 fps) and 2-second segments:
| Max tokens | Tokens actually generated | Generation | RTF |
|---|---|---|---|
| 16 | 16, 16, 16, 16 (all capped) | 0.851 s | 0.779 |
| 32 | 32, 31, 32, 32 (capped) | 1.163 s | 0.936 |
| 64 | 61, 31, 64, 50 | 1.626 s | 1.126 |
Four things follow.
-
A cap is free unless it binds. Under the same conditions,
glass_fallproduces character-for-character identical text at all three caps (9-15 tokens, then EOS), and RTF stays at 0.723 / 0.726 / 0.727. "Raising the cap makes it slower" is simply wrong -
When it binds, sentences get cut off. The 16-token output ends like
'...dribbling the ball, while a player'. Speed and complete sentences trade against each other -
Losing real time drags first text down with it. First text at cap 64 is
1.669s, worse than1.273/1.277s at 16 and 32, because the backlog delays the start of the next segment -
Generation length is not set by the question alone. With an identical question,
glass_fallfinishes in 9-15 tokens whilesoccer_goalpins the cap. How much there is to describe in the footage is what decides it
Note that the two tables above use different videos and cannot be compared row to row (bitrates of 490 kB and 1469 kB, and codec-native preprocessing responds to that). Compare only within a table.
Memory: MLX Peak Is Not the Number to Size For
MLX peak memory per segment was 11.96-14.27 GB for frames and 11.96-13.08 GB for codec. Read alone, that suggests a 16 GB machine might be enough.
It is not the number to size for. A light configuration (1 s stride, 4 s window, 2 fps, 2 tokens) held a steady 22 GB macOS footprint over a long session, but moving to a heavy one (8 s stride, 16 s window, 4 fps, 64 frames, 32 tokens) took it up to 50 GB. At the moment the heavy run was stopped, the footprint was 49 GB with 17.2 GB of swap in use, on a 64 GB machine. MLX peak at that same moment read 22.04 GB.
The cause is MLX's buffer cache. Under every condition, the IOAccelerator (graphics) region of the footprint tracked MLX active + cache almost exactly, so it was not Metal reserving memory. The cache holds the high-water mark of what was ever allocated and is not returned when the run stops or sits idle.
mx.clear_cache() works, releasing about 34 GB immediately; called while idle after a stop, the footprint fell to 12 GB. Calling it mid-run only helps briefly, though — keep the same settings and 150 seconds later the cache is back at 28.68 GB and the footprint at 46 GB. It matters when moving from a heavy configuration to a light one, and when stopping a run. The port now calls mx.clear_cache() when a run stops.
Measure the footprint under the heaviest configuration you actually use, then size the unified memory from that.
What Happens When It Cannot Keep Up
What does a configuration with RTF above 1 do if you just keep streaming into it? The intuitive answer is that the delay grows forever. The measurement says otherwise.
The lag does not diverge. The frame queue has a bound, and once it is full the oldest frames are discarded, so the steady state settles at "queue wait (queue length / capture rate) plus the processing time of one segment." That was 13.400 s on the M1 Max and 10.415 s on the M4 Max.
What is bounded is the delay, not the loss. Holding that steady state costs 58-75% of the input (75.1% on the M1 Max, 58.3% on the M4 Max). The discarded frames stretch the window in wall-clock terms, so a nominally 4-second window covers 10 to 19 seconds.
This path returns a slow response about the latest footage, not a slow summary of all the input. For watching a doorway that is the desirable behaviour. For anything that must not miss a moment, it is fatal.
Whether the display is honest can be judged without sharing a clock with the server: if both sides are honest, end_s + lag_s must equal the client's own elapsed time. Across 21 and 42 segments the offset stayed constant, varying by less than 0.01 s.
Seeing It Run
Per-segment processing is exposed as a reusable API, RealtimeSession. A reference local Web UI ships with it and shows the generated text, gate score, latency, and backlog beside the live image, for either a played video file or the Mac's camera.
This was shot on a phone's rear camera in fullscreen mode, with the model running on a Mac Studio M4 Max reached over tailscale. Only the head and tail of the 43 seconds were trimmed — nothing in the middle is cut.
Two things need explaining.
First, Display delay. In this clip, the feature that holds the picture back by a few seconds to line it up with the response is set to auto. The model is not answering instantly. Nothing removes the delay; the viewer's timeline is shifted so the response and the footage can be watched side by side. The undelayed live picture stays visible in the corner.
Second, the numbers on screen. They come from a single live camera session. The phone sends video over the tailnet, so network conditions mix into them. Every number in this article was measured on a 768x512 video file; no live-camera numbers are quoted. The sweet spot found on fixed clips does not carry over to a live camera unchanged — the segment length has to grow.
The model's mistakes are left in, too. Around 00:16 it reports "a person using a circular saw to cut a large, white, plastic toy" and "the video is in black and white," when in reality a finger is being pointed at a white stuffed animal. Terse, per-segment descriptions with a 64-token cap include mistakes of this kind. Editing them out would misrepresent what the model actually does.
What It Can and Cannot Be Used For
Translating the numbers into what they allow.
It cannot react instantly to what is in front of it. From an event to a complete sentence takes 3.48 seconds even on the fastest machine here. A person reacts to a play in one or two seconds, so this does not reach that. Anything that has to stay in step with reality itself — holding a conversation with the person pointing the camera, say — does not work.
Delay the picture by a few seconds, though, and you can have live commentary. With RTF below 1 the lag does not accumulate, so holding the displayed video back by the response time makes the footage and the generated text advance side by side. That is what Display delay does in the demo clip, and it is the same idea as broadcast delay: to the viewer, the commentary arrives with the picture. Just do not mistake it for removing the delay — it fixes the delay as a constant offset.
Watching a place, and event summaries, are within reach as far as latency goes. Notifying you a few seconds later that "someone is at the door" or "a package was left" is entirely fine at 3 to 7 seconds. What matters more is the other property: the delay settles instead of growing.
What this article measured, though, is latency and sustained throughput — nothing else. Detection accuracy on real footage, missed events, and reliability over long sessions are all unevaluated. The demo clip contains a mistake from the model, and the gate preset was calibrated on one positive and one control clip. Deciding whether it actually works for your case needs an accuracy evaluation on your own footage.
It is a poor fit for "summarize everything afterwards." Run it in a configuration that cannot keep up and it discards 60-70% of the input to stay near the live edge. If coverage matters, give up real time and do it as post-processing.
The hardware requirement is concrete now. Even an M1 Max generation machine keeps up on the codec path with 4-second segments at 8 fps. But the settings differ per machine, and carrying the M4 Max settings over breaks it. For memory, measure the footprint of the configuration you actually run: 22 GB for a light one, over 50 GB for a heavy one.
Limitations and Open Questions
- The parity claim is specific to float32; bfloat16 flips decisions near the threshold
- The streaming gate reference is a plain-PyTorch reimplementation, not the
mamba-ssmCUDA kernel - Codec preprocessing needs an ARM64 Linux container and cannot be done in native macOS alone
-
codec-video-prepdeclares no license and publishes no source - Every video used here is synthetic or LTX-2 generated; real footage and long streams are unmeasured
- The fixed-clip matrix uses
glass_fallwith a 16-token cap, which is a region where the cap does not bind. Footage with more to describe will be slower than the matrix suggests - Each saturation measurement is a single session, not a median of three runs
- Codec numbers are affected by how many cv-preinfer containers run at once. Everything quoted was run alone and re-measured with no other GPU load
- Shortening generation through the question itself has not been swept. Unlike a cap it does not truncate sentences, but I have not measured it systematically
- The gate preset was calibrated on one positive and one control clip, which cannot support a precision / recall claim
- Quantization is not supported
- The gate replays the accumulated visual history for every segment. That matches the official whole-stream result, but it is not a stateful incremental Mamba, so the long-stream cost is reported as a measured limitation
Closing Thoughts
Deciding what to compare against took longer than the port itself. Codec-native can look like a 95% reduction, no reduction at all, or an increase. All three describe the same implementation on the same video. Producing numbers is easy; writing down which comparison each number answers is the hard part, and that was the strongest impression from this round.
The surprise came from fixing a bug where 24 fps clips were handed to codec preprocessing untouched. Applying the capture rate when cutting the segment dropped the M1 Max's RTF from 1.542 to 0.644. No change to the model, no quantization. What kept it from keeping up was how the input was handed over, not how heavy the model is. If a Mac feels slow here, counting how many of what you are passing in is a good first move.
The streaming gate not pointing at event times was a disappointment, but learning that it separates content types was worth having. Put it in front as a cheap pre-filter and let the generated text make the call; split that way, each part does what it is good at.
A device that answers in a few seconds when you point a camera at it is plainly fun. A doorway, a workbench, an aquarium — there is no shortage of places you would want to set one down.
Top comments (0)