DEV Community: Marco Rinaldi

One image schema for four VLM providers: we stopped reformatting payloads

Marco Rinaldi — Mon, 01 Jun 2026 16:51:55 +0000

TL;DR: We reconstruct grayscale frames from event-camera data and send them to vision-language models for weak scene labels. Four providers, four slightly different ways to attach an image, and our payload-building code had grown three branches. Putting Bifrost in front of the VLMs gave us one OpenAI-compatible image schema. Here's the honest version, including where LiteLLM and Portkey do it better.

So, the thing is, on my team at Prophesee we don't have classic RGB frames. We have event streams, and for some of our offline tooling we reconstruct short grayscale frames from those events so a vision-language model can give us a rough scene description. "Person crossing, low light, motion blur on the left." Weak labels. Useful for triaging which clips a human should look at first.

We send those reconstructed frames to four VLMs depending on the job: OpenAI's gpt-4o, Anthropic's Claude, Gemini through Google Vertex, and occasionally Mistral's vision model. Same picture, four annoyingly different request bodies.

The actual pain

Let me give you the full picture here. The OpenAI shape wants an image_url content block, and the URL can be a base64 data URI. Fine. Then you go to Vertex and the structure shifts, the field names shift, and the detail hint you were passing to control token cost stops meaning anything. Anthropic wants its own source object with a media type. None of this is hard. It's just three if provider == branches in a function that should do one thing.

We had maybe 80 lines of payload-shaping code in our Python annotation service. Every time a provider tweaked an API version, something silently broke and a batch of frames came back unlabelled at 2am during an overnight run. Not dramatic. Just paper cuts that add up over a 6-person team.

What we changed

We put Bifrost (an open-source AI gateway written in Go) in front of all four providers. It exposes a single OpenAI-compatible API, including multimodal, so our service now builds exactly one image message and never branches on provider again.

Running it locally was a one-liner:

npx -y @maximhq/bifrost
# or
docker run -p 8080:8080 maximhq/bifrost

Then our call looks identical no matter who serves it. We just change the model string:

curl -X POST http://localhost:8080/v1/chat/completions \\
  -H "Content-Type: application/json" \\
  -d '{
    "model": "vertex/gemini-1.5-pro",
    "messages": [{
      "role": "user",
      "content": [
        {"type": "text", "text": "Describe the scene. One line."},
        {"type": "image_url",
         "image_url": {"url": "data:image/png;base64,iVBORw0K... "}}
      ]
    }]
  }'

Swap vertex/gemini-1.5-pro for openai/gpt-4o or anthropic/claude-... and the body doesn't move. That's the whole point. The gateway translates to each provider's native multimodal format on the way out. (Streaming and multimodal docs.)

Two things came along for free that I didn't plan for. First, automatic fallback. If Vertex throws a 503 mid-batch, the request retries on a configured backup model instead of dropping the frame. (Fallbacks docs.) Second, semantic caching. Our reconstructed frames are repetitive, lots of near-identical low-motion clips, so caching on semantic similarity cut a chunk of redundant VLM calls. (Semantic caching docs.)

How it compares

I won't pretend Bifrost is the only thing that does this. It isn't.

Concern	Bifrost	LiteLLM	Portkey
Unified multimodal API	Yes, OpenAI-compatible	Yes, very mature	Yes
Implementation	Go, self-hosted	Python, self-host or lib	Hosted-first, self-host option
Provider breadth	23+	Largest I've seen	Broad
Observability UI	Prometheus metrics	Functional	Strongest dashboards

LiteLLM has been doing image normalisation longer and its provider list is wider than anyone's. If you're already deep in a Python stack and want a library call rather than a separate service, LiteLLM is genuinely the pragmatic pick. Portkey's hosted dashboards and guardrail tooling are more polished than what we run; for a team that wants observability out of the box without wiring Prometheus, that matters.

We picked Bifrost mostly because it's a single Go binary we self-host next to our annotation service, and the OpenAI-compatible surface meant near-zero changes to client code. Different teams will weigh that differently.

Trade-offs and Limitations

It's a network hop. You're adding a gateway between your service and the provider, so there's a small latency cost and one more thing that can fall over. For an offline labelling pipeline I don't care. For a tight real-time loop you'd measure it first.

You inherit the gateway's abstraction. If a provider ships a brand-new multimodal parameter, you wait for the gateway to expose it rather than calling the raw API. So far that's been fine for our image-description use, but it's a real constraint if you live on the bleeding edge of one provider's features.

And to be precise: this solved a payload-normalisation and reliability problem. It did nothing for our actual model quality. The VLM still mislabels heavy motion blur, and reconstructing good frames from sparse events is still the hard part of my week. The gateway just stopped me rewriting JSON shapes. If you can't make the upstream model smaller or the labels cleaner, no amount of plumbing fixes that.

One espresso's worth of setup, a quieter on-call. That trade I'll take.

Our event-camera detector lost 6 mAP to a badly chosen accumulation window

Marco Rinaldi — Mon, 01 Jun 2026 07:21:48 +0000

TL;DR: We spent three weeks chasing a 6 mAP regression in an event-camera object detector. The model was fine. The bug was the accumulation window we used to turn raw events into tensors, and we had picked it once, eighteen months earlier, on a different dataset. Here is how we tune it now.

So, the thing is, with event cameras you do not get frames. You get a stream of events, each one a tuple of (x, y, t, polarity), fired asynchronously whenever a pixel sees a brightness change. Microsecond timestamps. No global shutter, no exposure. Beautiful for high-speed motion. Annoying when you want to feed a convolutional detector that expects a dense tensor.

So everyone accumulates. You take all the events inside a time window, say 10 ms, and you build a representation out of them. A 2D histogram, a voxel grid, a time surface. That window length is a hyperparameter. And in my experience at Prophesee, it is the one people set once and never look at again.

The regression that was not a model regression

Last spring we retrained a small detector for a logistics conveyor setup. Boxes moving at roughly 1.8 m/s past a Gen4 sensor. New training run, new augmentations, and the val mAP came back at 41.2 against a previous baseline of 47.5.

Six points. Gone. We blamed the LoRA-style fine-tune first, then the augmentation pipeline, then a teammate's data split. Two of us, the better part of three weeks.

The actual cause: the old baseline accumulated events over 33 ms, the new pipeline defaulted to 10 ms. At 10 ms the boxes barely produced enough events to fill the histogram. The detector was looking at near-empty tensors. Sparse input, low recall, lost mAP. Nothing wrong with the weights at all.

What the window actually trades

A short window gives you crisp spatial structure but few events, so thin or slow-moving objects vanish. A long window collects plenty of events but smears fast motion across pixels, and the network sees a blurred ghost. The right value depends on object speed and event rate, which means it depends on your scene.

Here is the core of how we build the representation now, with the window made explicit instead of buried in a default:

import torch

def events_to_voxel(events, window_us, num_bins, height, width):
    # events: (N, 4) tensor of [x, y, t_us, polarity]
    t0 = events[:, 2].min()
    rel_t = events[:, 2] - t0
    keep = rel_t < window_us
    ev = events[keep]

    bin_idx = (ev[:, 2] - t0) / window_us * num_bins
    bin_idx = bin_idx.clamp(0, num_bins - 1).long()

    voxel = torch.zeros(num_bins, height, width)
    pol = ev[:, 3] * 2 - 1  # {0,1} -> {-1, +1}
    voxel.index_put_(
        (bin_idx, ev[:, 1].long(), ev[:, 0].long()),
        pol, accumulate=True,
    )
    return voxel

We now sweep window_us as a first-class part of validation, the same way we sweep learning rate. Cheap to run, since it is a preprocessing change and the weights stay fixed for the inference-time sweep.

The numbers from our conveyor set

Same model, same checkpoint, same 4,100-frame validation set. Only the accumulation window changes. Latency measured on a Jetson Orin NX at INT8.

Window	Events/frame (median)	mAP@0.5	Preproc + inference
5 ms	1,900	38.0	7.4 ms
10 ms	4,300	41.2	8.1 ms
20 ms	9,800	46.9	9.3 ms
33 ms	17,400	47.6	11.0 ms
50 ms	28,500	45.1	13.8 ms

The curve is not monotonic. It climbs, plateaus around 20 to 33 ms, then falls as motion blur sets in. For this scene the sweet spot was 20 ms, which gave us almost all the accuracy of 33 ms with 1.7 ms less latency per frame. We had been leaving both accuracy and speed on the table.

How we audit windows now

We added a small step to dataset curation. For a random 300-frame subset we render the accumulated voxel back to a grayscale-ish preview and run it past a vision-language model to flag frames where the target is unreadable, blurred, or empty. It catches degenerate windows faster than a human scrubbing through previews. We route that call through Bifrost so the same code can hit one provider in CI and a cheaper one for bulk runs without rewriting anything, and that is the whole extent of the LLM involvement here. The detector itself never touches a model bigger than 6 MB.

It is not a substitute for the mAP sweep. It is a sanity filter before we trust the sweep.

Trade-offs and Limitations

The window that wins on a conveyor at 1.8 m/s is wrong for drones or automotive. Scene speed changes everything, so these exact numbers do not transfer. Treat the method, not the 20 ms.

Sweeping the window inflates validation time. Five windows means five full preprocessing passes over the val set. For us that is a few minutes; for a million-frame set it is real compute you have to budget.

A fixed window also assumes roughly constant scene dynamics. The honest answer for variable-speed scenes is an adaptive or event-count-based window, which we are testing but do not yet trust in production. And the VLM audit costs money per frame, so we cap it to a subset rather than the full set.

Structured channel pruning got our detector under 12ms on a Jetson

Marco Rinaldi — Fri, 29 May 2026 07:22:38 +0000

TL;DR: Our frame-based defect detector ran at 31ms on a Jetson Orin Nano, and the production line needed 12. Structured channel pruning at 45% plus a three-epoch fine-tune got us to 11.4ms for a 0.6 mAP drop. Unstructured pruning looked beautiful on paper and gave exactly zero real-world speedup, so we deleted it.

So, the thing is, the model was never the problem. The detector hit 0.91 mAP in the lab and everyone was happy. Then we put it on the actual Orin Nano sitting next to the conveyor, and it ran at 31ms per frame. The line moves a stamped part every 12ms. You can imagine how that meeting went.

Let me give you the full picture here. This was a side project away from my usual event-camera work at Prophesee, a plain RGB detector for surface defects on stamped metal panels. Small team, three of us, a hard latency budget and no room for a bigger GPU on the line. Cutting model size was the only lever we had.

The trap I fell into first

I reached for unstructured pruning because the literature loves it. torch.nn.utils.prune, magnitude based, zero out the smallest 60% of weights, retrain. The sparsity numbers were gorgeous. Sixty percent of the parameters gone, mAP barely moved.

Latency on the Jetson: 30.8ms. Same as before, down to the decimal.

Here is why it does nothing. Unstructured pruning keeps every tensor the exact shape it was and writes zeros into it. cuDNN still runs a dense convolution over those zeros. Unless you have a sparse kernel plus hardware that exploits 2:4 structured sparsity (Ampere does, partially, and the Orin Nano's GPU is not the right SKU for it), you bought a smaller checkpoint file and nothing else. The thing was multiplying by zero at full price.

Where the latency actually lives

Structured pruning removes whole filters and channels, so the tensors get physically smaller. A conv layer with 256 output channels becomes 140. FLOPs drop, memory traffic drops, the kernel finishes sooner. Real speedup.

The painful part is dependency tracking. Remove an output channel in one layer and you must remove the matching input channel everywhere it feeds, across residual adds, concats, the lot. Do it by hand and you will spend a Sunday debugging shape mismatches instead of eating lunch in Bologna. torch-pruning traces the whole graph with its DepGraph and prunes coupled layers together.

import torch_pruning as tp

model = load_detector().eval()
example = torch.randn(1, 3, 640, 640)

# group importance by L2 norm of each filter group
imp = tp.importance.GroupNormImportance(p=2)

pruner = tp.pruner.MetaPruner(
    model,
    example,
    importance=imp,
    pruning_ratio=0.45,
    ignored_layers=[model.head.cls],  # leave the classification head alone
)

pruner.step()
macs, params = tp.utils.count_ops_and_params(model, example)
print(f"{macs/1e9:.1f} GFLOPs, {params/1e6:.2f}M params")

After pruning, the model is broken in the accuracy sense. You fine-tune to recover. Three epochs on the same training set pulled mAP back from 0.86 to 0.904. Not free, but cheap compared to retraining from scratch.

The numbers

Approach	Params removed	FLOPs	Jetson latency	mAP
Baseline (FP16)	0%	17.2 GFLOPs	31.0 ms	0.910
Unstructured 60%	60%	17.2 GFLOPs (dense)	30.8 ms	0.905
Structured 45%	41%	9.4 GFLOPs	11.4 ms	0.904

Structured pruning removed fewer parameters on paper and delivered all of the speed. That gap between "params removed" and "FLOPs removed" is the whole lesson. FLOPs only fall when the shapes shrink.

Trade-offs and limitations

Pruning ratio has a cliff. At 0.45 we lost 0.006 mAP. At 0.60 the small defect class collapsed and no amount of fine-tuning brought it back, because those filters were genuinely doing work. You have to sweep the ratio per model, there is no universal number.

The head is sensitive. We pinned the classification head with ignored_layers after an early run where pruning it tanked recall on the rare defect type. Pruning is not uniform across a network and treating it as such will cost you.

Speedup is hardware-shaped. Our 9.4 GFLOPs model runs at 11.4ms on the Orin Nano and at 4ms on a desktop 4070. Pruning helps compute-bound layers; if your bottleneck is memory bandwidth or a fat softmax, channel pruning moves the needle less than you hope. Profile first with trtexec or Nsight before you assume FLOPs equal time.

And retraining needs data. We had labels, so recovery was painless. For the handful of ambiguous border crops where annotators disagreed, we sent them to a larger vision model for a second opinion and routed those calls through an AI gateway (we run Bifrost, some teams use LiteLLM) so nobody had to wire up provider SDKs by hand. That kept the labelling loop moving without a separate integration project.

One more honest note. Structured pruning fights quantisation a little. Our INT8 TensorRT export of the pruned model needed re-calibration, and the calibration cache from the dense model was useless. Budget time for that step.

What I would tell past me

Stop optimising the metric that does not move the clock. Sparsity is a vanity number unless your runtime can spend it. Measure latency on the real device, with the real batch size, before and after every change. The espresso machine in our office has more consistent timing than my early benchmark scripts did, and I trusted it more.

If you cannot make the model smaller and faster on the actual target, you do not yet understand where its time goes. Pruning forced me to learn that, layer by layer.

VLM-scored calibration sets for INT8 quantisation, routed through Bifrost

Marco Rinaldi — Thu, 28 May 2026 16:58:23 +0000

TL;DR: We pick the 512 hardest images for INT8 PTQ calibration by scoring a candidate pool with a small VLM. Bifrost sits between our calibration pipeline and four providers, gives us semantic caching, per-engineer virtual keys, and hard budget caps so a runaway loop can't burn EUR800 of API spend over the weekend.

So, the thing is, calibration set selection is one of those topics nobody writes about until the day your INT8 model is 4.2 points of mAP worse than the fp16 reference and you have a release branch already cut. Pick the wrong 512 images and your activation histograms get biased toward easy frames. Pick the right 512 and the gap closes to 0.6 points without any QAT work at all.

For about a year we picked them by hand. Then we tried random sampling, then stratified by class. The thing that finally worked, on an industrial defect detector we shipped to a customer last quarter, was scoring our candidate pool with a vision-language model and biasing toward the high-difficulty tail. That sounds neat in a paper. In production it meant roughly 80k API calls per release cycle, four providers in rotation, three engineers all running their own ablations, and one very pointed email from finance about the November bill.

That's where Bifrost came in.

What we actually wanted from a gateway

A short list, in order of how much pain each item was causing us:

One API surface so the calibration script doesn't care if we're hitting GPT-4o, Claude Sonnet, Gemini, or our self-hosted Qwen2-VL.
Per-experiment budget caps. If someone's loop goes infinite over a Friday night, I want it to die at EUR50, not EUR5,000.
Semantic caching, because we score the same images across many ablations and paying twice for the same prompt+image hash is wasteful.
Per-engineer virtual keys so I can see who spent what without parsing four billing consoles.
Real observability. Prometheus metrics over a custom dashboard built by the intern who's now in Berlin.

Bifrost gave us all five. Setup took less than an afternoon, mostly spent arguing about key naming conventions.

The scoring loop

The actual code is boring, which is good. Our calibration scorer is a single async function that streams candidates from S3, calls Bifrost, parses a difficulty score from the JSON response, and writes it back to a Parquet shard.

providers:
openai:
keys:
- value: env:OPENAI_KEY_A
weight: 0.5
- value: env:OPENAI_KEY_B
weight: 0.5
anthropic:
keys:
- value: env:ANTHROPIC_KEY
governance:
virtual_keys:
- id: vk_calib_marco
budget_eur: 200
- id: vk_calib_giulia
budget_eur: 200
semantic_cache:
enabled: true
similarity_threshold: 0.92

Each engineer gets their own virtual key. The budget is hard. Hit EUR200 and Bifrost returns 429 until the cap is lifted. This sounds harsh until you remember the November bill.

Semantic caching pays for itself faster than I expected. About 38% of our candidate frames recur across ablations because we keep iterating on the difficulty prompt rather than the image set itself. Cache hits cost us nothing and return in under 40ms.

How it compares to LiteLLM and Portkey

We did genuinely evaluate both before committing.

Capability	Bifrost	LiteLLM	Portkey
OpenAI-compatible API	Yes	Yes	Yes
Multimodal request routing	Yes	Yes	Yes
Semantic caching	Built-in	Plugin	Built-in
Hierarchical virtual keys / budgets	Built-in	Limited	Built-in
Self-hosted single Go binary	Yes	Python	Hosted-first
Native Prometheus metrics	Yes	Yes	Yes

Portkey's hosted dashboards are more polished. If you don't want to run anything yourself, that's the right call. LiteLLM's Python ecosystem is broader and their community is bigger. We picked Bifrost because we wanted a self-hosted Go binary that fit alongside our existing inference services, with a governance model that matched how the team already splits experiments by engineer.

Trade-offs and Limitations

A few things to be honest about.

The semantic cache is great for repeated calibration runs and useless if your prompt template changes per experiment. We pin the prompt template and version it like model weights. Without that discipline the cache hit rate dropped to about 6%.

VLM scoring is not deterministic. Two scoring passes on the same 80k images gave us a Spearman correlation of 0.94, which is fine for picking calibration sets but would not be fine for, say, regulatory model documentation.

Bifrost's MCP support is good but we don't use it for this workflow. The scoring loop is a plain HTTP client, no tool use. I mention it because the README puts MCP front and centre and you might assume you need it. You don't.

The built-in UI is React-based and feels a generation behind Grafana for ad-hoc exploration. We export Prometheus metrics to our existing Grafana stack and ignore the bundled dashboard. Personal preference, not a bug.

QAT vs PTQ on our edge vision model: 6 months of A/B data

Marco Rinaldi — Thu, 28 May 2026 07:21:54 +0000

TL;DR: We ran post-training quantisation (PTQ) and quantisation-aware training (QAT) side by side on the same defect-classification model deployed on a Jetson Orin Nano. After six months in production, QAT recovered 3.1 mAP points over PTQ on rare defect classes, but cost us roughly two engineer-weeks of pipeline work and a 4x slower training cycle.

So, the thing is, every time someone shows me a quantisation benchmark on ImageNet, I want to ask them what their actual deployment looks like. Because ImageNet validation accuracy at INT8 tells you almost nothing about whether your model will still detect the 0.4% of defect samples that pay for the whole project. We learned this the hard way at the end of last year, when the first quarter of production data came back from one of our partner sites and our PTQ model was missing scratches that the FP16 baseline caught fine.

This post is the writeup. Six months, one model architecture (ResNet-18 trunk with a custom anchor-free head), two quantisation paths, two hardware targets. No synthetic benchmarks, no toy datasets.

The model and the constraint

The model is a defect classifier on a steel rolling line. Inference runs on a Jetson Orin Nano, 8GB version, sharing the SoC with a stereo depth pipeline. Latency budget for the classification path is 14ms. Memory budget after the depth pipeline takes its share is around 180MB. Five classes including background, with class imbalance roughly 92/3/2/2/1 percent.

FP16 baseline: 17.3ms, 92.4 mAP, 71MB of activations.
INT8 PTQ: 8.9ms, 88.1 mAP, 38MB.
INT8 QAT: 9.2ms, 91.2 mAP, 38MB.

Numbers look fine in aggregate. But aggregate hides the problem.

Where PTQ falls apart

Post-training quantisation with TensorRT's entropy calibrator does a reasonable job when your dataset is balanced. Ours is not. The calibration set we initially used was drawn proportionally from the training distribution, which meant 92% of the calibration data was clean background. The activation histograms ended up dominated by the background distribution, and the quantisation scales were tuned for that.

The result was that the rare defect classes (cracks at 1% prevalence, embedded particles at 2%) lost between 5 and 9 mAP points each. Hot pixels in the defect feature maps got clipped into the background range. We caught this in a customer review meeting in early March. Not my favourite Tuesday.

What we tried

Approach	Calibration / training	mAP (rare classes)	Engineer-weeks	Notes
PTQ default	5000 random samples	79.2	0.5	Baseline pain
PTQ rebalanced	2000 samples, defects oversampled 10x	84.6	1.0	Better, still gaps
PTQ percentile	99.99 percentile + rebalanced	86.1	1.5	Marginal gain
QAT (10 epochs)	Real training loop, fake-quant ops	89.4	2.5	The keeper

The QAT path used torch.ao.quantization with custom fake-quant observers per layer, exported through ONNX to TensorRT. We had to write a small shim because the default ONNX → TRT path stripped some of our QDQ nodes silently in TensorRT 10.4. The fix was forcing explicit precision on the affected layers.

# Snippet from our QAT training step
from torch.ao.quantization import QConfig, FakeQuantize, MovingAverageMinMaxObserver

per_channel_qconfig = QConfig(
    activation=FakeQuantize.with_args(
        observer=MovingAverageMinMaxObserver,
        quant_min=-128, quant_max=127,
        dtype=torch.qint8,
        qscheme=torch.per_tensor_symmetric,
    ),
    weight=FakeQuantize.with_args(
        observer=MovingAverageMinMaxObserver,
        quant_min=-128, quant_max=127,
        dtype=torch.qint8,
        qscheme=torch.per_channel_symmetric,
    ),
)
model.qconfig = per_channel_qconfig
torch.ao.quantization.prepare_qat(model, inplace=True)

The interesting bit was per-channel weight quantisation. Per-tensor lost us another 1.5 mAP on rare classes. Per-channel costs roughly nothing at inference on the Orin's NVDLA. You almost always want per-channel for vision models with significant filter diversity.

What QAT does not fix

QAT is not free and it's not magic. A few things bit us.

The training cycle goes from 2 hours to 8 hours when you turn on fake-quant. We optimised some of this by only quantising from epoch 4 onwards (training in FP16 first), which the literature calls warm-start QAT. Recovered most of the wall-clock cost.

Operator coverage in TensorRT for QDQ nodes is decent but not complete. Our custom group-norm replacement broke quantisation entirely and we had to fall back to batch-norm for the deployment branch. Annoying. Worth checking before you commit to an architecture.

The eval pipeline itself needed work too. We use an LLM-driven workflow to triage failure modes from our weekly inspection batches (sorting false positives by visual similarity, basically), routed through Bifrost as our internal gateway so the rest of the org can share quota. Found one whole class of failures (specular highlights confused as cracks) we were not tracking before.

Trade-offs and limitations

PTQ is still the right call for prototyping. If you're not sure your architecture is final, do not pay the QAT tax. Iterate with PTQ until the design freezes.
QAT amplifies dataset noise. Mislabeled samples hurt more under QAT than under FP training, because the fake-quant adds noise on top. We re-curated about 800 ambiguous labels before our final run.
The mAP gap shrinks with mixed precision. We tried mixed INT8/FP16 per layer and recovered 1 mAP for a 1.4ms latency hit. For our 14ms budget, full INT8 won. Yours may differ.
Different hardware, different story. The Orin's INT8 throughput is excellent. On a Coral TPU or a Cortex-M7, the analysis changes completely.

Six weeks of Bifrost in a factory QA pilot: real cost numbers

Marco Rinaldi — Wed, 27 May 2026 16:52:34 +0000

TL;DR: Six weeks running an AI gateway between our edge cameras and three cloud VLM providers cut our pilot VLM spend by 58% and gave us actual failover during a 90-minute Anthropic blip last month. Bifrost handled it. Here's what worked, what didn't, and how it compared to LiteLLM and Portkey on the same workload.

So, the thing is, when our team at a partner factory near Bologna wired up a defect inspection pilot with cloud VLMs in the loop, the cost story turned ugly within the first ten days. We had 28 stations, each catching anomalies from local event-camera and frame fusion, then escalating ambiguous frames to GPT-4o-mini or Claude Sonnet for a second opinion. The VLM bill landed at €4,800 in week one. Production was running 11 hours a day. Nobody had budgeted for that.

The pilot also stalled twice. Once because OpenAI returned 429s for 22 minutes during what I assume was a regional capacity issue, and once because a key rotated wrong and half the fleet froze. Neither outage was the model's fault. Both were avoidable.

We picked Bifrost as a gateway and ran it for six weeks. This is a writeup of what we measured. Independent perspective. I have no commercial relationship with them.

The setup

Our stack: Jetson Orin Nano per station, edge model (a distilled student of CLIP for class triage), and an escalation rule. If the edge confidence falls below a threshold, the frame plus context gets sent to a cloud VLM through Bifrost. Bifrost runs on a small VM in our partner's DMZ, two replicas behind a TCP load balancer. We use the OpenAI-compatible endpoint so our existing inference client didn't change.

Three things mattered for us, in this order. First, fallback semantics. Second, per-station budgeting. Third, observability metrics that hit Prometheus without extra scaffolding. We tried LiteLLM and Portkey before settling on Bifrost. More on that below.

Configuration was a single YAML and two environment variables. This is roughly the relevant part:

providers:
  openai:
    keys:
      - env: OPENAI_KEY_PRIMARY
        weight: 0.7
      - env: OPENAI_KEY_FAILOVER
        weight: 0.3
  anthropic:
    keys:
      - env: ANTHROPIC_KEY
  bedrock:
    keys:
      - env: BEDROCK_KEY
        region: eu-central-1

routes:
  - models: ["openai/gpt-4o-mini"]
    fallbacks:
      - "anthropic/claude-sonnet-4-6"
      - "bedrock/anthropic.claude-3-5-haiku"

That's the whole fallback chain. When OpenAI started rate limiting on April 18, traffic shifted to Anthropic within the configured timeout and we lost zero frames. Our oncall got a Prometheus alert on the fallback rate metric, which is exposed natively, and we caught the issue 90 seconds before any operator noticed.

What we measured

Six weeks of data from our pilot, three weeks before Bifrost and three weeks after. Same shift schedule, same product mix. Numbers below are real, scrubbed of station-specific identifiers.

Metric	Before gateway	After Bifrost	Notes
Weekly VLM spend	€4,640 avg	€1,920 avg	Semantic cache hit at 41%
P95 escalation latency	1.9s	1.4s	Some hits served from cache
Outage minutes	112	0	One Anthropic blip auto-mitigated
Operator interventions	7	1	Most were cost-driven
Per-station cost visibility	none	per virtual key	New capability

The 41% semantic cache hit rate surprised me. I expected maybe 15%. Factory floor frames have a lot of repeated context (same product variant, same lighting, similar defect prompts), and the cache exploits that pattern. Documented behaviour, see the semantic caching docs linked at the bottom.

Bifrost vs LiteLLM vs Portkey

Honest comparison, because we tested all three on the same workload.

Feature	Bifrost	LiteLLM	Portkey
OpenAI-compatible API	yes	yes	yes
Self-host as binary	yes (Go binary or Docker)	yes (Python proxy)	hosted primary
Semantic caching	yes	via Redis plugin	yes
Virtual keys with budgets	yes	yes	yes
Prometheus metrics	native	requires setup	hosted dashboard
Throughput in our test	~9k RPS sustained	~2k RPS before tuning	hosted (couldn't test fairly)

LiteLLM is more mature in some places. Its Python plugin ecosystem is bigger, and if your team already lives in Python middleware, that's a real advantage. Portkey has the slickest hosted dashboard out of the box, no question. We couldn't put a hosted service in the path for this pilot because the partner factory had restrictions on outbound traffic from the OT network.

Bifrost won for us on two specific points. The single Go binary deployed without a fight on the small VM we had. And the per-virtual-key budgeting let us bill each station's VLM cost back to the production line owner, which mattered for getting the pilot extended past phase one.

Trade-offs and limitations

The semantic cache occasionally returns a cached response when product variants change mid-shift. We had two false-negative defect reports in week three traced to a cache hit on a near-identical SKU. We tuned the similarity threshold and added a variant ID to the cache key. Lesson learned, but the takeaway is that semantic caching needs context-aware keys for industrial workloads. Not a Bifrost bug, but a real failure mode you have to design around.

Bifrost's MCP integration is interesting but we didn't use it for this pilot. Our cameras don't need tool use. If you are building agentic flows on top, that calculus changes.

Documentation gaps exist. The clustering setup for HA was less detailed than I would have liked when we first read it. Their team answered on Discord within an hour, which helped a lot.

You don't need a gateway if you are single-provider and single-region. The added complexity is worth it once you have real failover requirements or cost attribution problems. We had both. Your factory may not.

Distilling SAM 2 into a 6MB student for industrial inspection

Marco Rinaldi — Wed, 27 May 2026 07:22:52 +0000

TL;DR: We took Meta's SAM 2 small (around 224M params) and distilled it into a 6.3MB student that runs at 31 FPS on a Jetson Orin Nano for an automotive surface-defect pipeline. Mask IoU drops from 0.91 to 0.84, which is acceptable for the defect shapes we care about. The single biggest lever was a feature-alignment loss on the image embedding, not the mask logits.

So, the thing is, most of my year goes into event-camera work at Prophesee, but a side contract this spring with an automotive supplier outside Brescia ate two months of my evenings. They make aluminium body panels and they wanted real-time masks for surface defects: scratches, dents, paint pinholes. Cameras are boring CMOS at 25 FPS and 4MP. Target hardware is a Jetson Orin Nano because the PLCs on the line already talk to one over Ethernet.

First thing we tried was to fine-tune SAM 2 small directly and ship it with TensorRT FP16. About 1.2 seconds per image on the Orin. That's roughly 30x too slow for a moving line. We needed something a lot smaller.

The student architecture

We did a MobileSAM-style backbone transplant but kept going further. TinyViT-5M as the image encoder, prompt encoder stripped down to dense point prompts only (we feed it candidate locations from a cheap saliency head, no box prompts), and we cut the mask decoder's upsample to 1/2 instead of 1/4. That last bit lost us a tiny amount of edge precision but it was the difference between 19 FPS and 31 FPS on the Orin.

The student is 1.6M params. Storage is 6.3MB after INT8 weight-only quantisation. The teacher was held in FP16 on a separate A6000 during training.

The loss that actually worked

Here is the part where I want to be very honest. We spent two weeks on cleverness: multi-scale logit matching, attention transfer, contrastive losses on the prompt embedding. None of it beat the obvious thing once we put it in.

The obvious thing: align the student's image embedding to the teacher's in cosine space, alongside the usual soft mask BCE and a supervised dice term on the actual ground truth.

def distill_loss(s_emb, t_emb, s_logits, t_logits, gt_mask):
    # feature alignment in cosine space
    s_n = F.normalize(s_emb.flatten(2), dim=-1)
    t_n = F.normalize(t_emb.flatten(2), dim=-1)
    feat = 1.0 - (s_n * t_n).sum(-1).mean()

    # soft mask distillation with temperature
    soft = F.binary_cross_entropy_with_logits(
        s_logits, torch.sigmoid(t_logits / 2.0)
    )

    # supervised dice on hand-labelled defects
    p = torch.sigmoid(s_logits)
    inter = (p * gt_mask).sum((1, 2, 3))
    union = p.sum((1, 2, 3)) + gt_mask.sum((1, 2, 3))
    dice = 1.0 - (2 * inter / (union + 1e-6)).mean()


    return 0.4 * feat + 0.3 * soft + 0.3 * dice

Without the feat term, we plateaued at IoU 0.71 on the held-out set of 4,200 defect crops. With it, 0.84. Same student, same data, same schedule. That is the kind of result that makes you doubt your prior assumptions about what knowledge transfer actually means.

What the numbers look like

Variant	Params	Size	IoU	FPS (Orin)
SAM 2 small (teacher)	224M	884 MB	0.91	0.8
MobileSAM-style transplant	9.8M	39 MB	0.78	18
Ours, no feature loss	1.6M	6.3 MB	0.71	31
Ours, with feature loss	1.6M	6.3 MB	0.84	31

Where a VLM judge crept in

One thing that surprised me during training: about 6% of held-out crops had teacher masks that were visibly wrong. Pinholes the teacher missed entirely, or scratches it bled into adjacent reflections. If you blindly distill on those, the student learns the teacher's bad habits.

For the crops where student and teacher disagreed by more than 0.15 IoU, we ran a VLM-as-judge step. Claude 4.5 Sonnet on the primary path, Gemini 2.5 Pro as a backup, asked to pick which mask better followed the actual defect contour given the original crop and both overlays. We routed those calls through Bifrost (https://github.com/maximhq/bifrost) because we needed automatic failover between the two providers during peak hours when one would rate-limit us, and writing that retry logic ourselves felt like wasted time. About 38% of disagreements were judged in the student's favour, so we re-weighted those samples in the next epoch.

I will say: the VLM-as-judge is not magic and we hand-checked a stratified sample of its calls. The agreement with our QA lead was around 84%, which was good enough to drive sample re-weighting but not good enough to use as ground truth.

Trade-offs and limitations

Hairline scratches below about 80 microns at our working distance are still missed. The teacher catches some of those. We are debating whether to keep a slow second pass for ambiguous frames.
The feature-alignment loss requires teacher and student image embeddings at the same spatial resolution. That ruled out a couple of backbone choices we wanted to try.
Training was nine days on 4x A6000s. Distillation is not cheap, even if the result is.
We have not validated on glass or carbon-fibre panels. Aluminium only.
INT8 quantisation cost us 0.6 IoU points. Worth it for the size but not free.

Capping VLM spend per CV researcher: hierarchical budgets in practice

Marco Rinaldi — Tue, 26 May 2026 16:52:17 +0000

TL;DR: Our 11-person CV team at Prophesee was burning through €3-4k weeks of VLM spend on dataset annotation with no idea which researcher caused which spike. We put Bifrost between the labelling scripts and the providers, mapped one virtual key per person with monthly caps, and the receipt-chasing stopped. Took an afternoon to wire up. Real savings came from enforcement, not from clever routing.

So, the thing is, when you let eleven people each script their own VLM annotation passes against three different providers, you get one giant invoice at the end of the month and no idea who is responsible for the €700 Tuesday spike. We hit that exact wall in late February. Two consecutive weeks at around €4k each, and our team lead asked, very politely, over an espresso, whether maybe we should think about it.

The honest answer was: yes, six months ago.

What we were actually doing

Our annotation pipeline labels event-camera frames reconstructed from recordings (indoor robotics, low-light driving, drone footage). For each scene we run a VLM pass to produce captions, bounding box suggestions, and a sanity-check description. The VLM is the teacher; a smaller distilled model is what we deploy on the edge.

The traffic profile looked like this:

Source	Calls/week	Provider	Notes
Bulk caption jobs	~80k	gpt-4o-mini	overnight cron, predictable
Bounding box suggestions	~12k	claude-3.5-sonnet	interactive, daytime
Sanity-check descriptions	~6k	gemini-2.0-flash	per-researcher scripts
Ad-hoc exploration	who knows	mixed	the actual problem

That last row was biting us. Researchers were testing prompt variations, sweeping over thousands of frames out of curiosity, forgetting they kicked off the script before going home for the weekend.

What we wired up

We deployed Bifrost as a Docker container on the same internal box that already hosts our annotation queue. Ten minutes. Then we mapped one virtual key per researcher (docs) with a monthly cap, and two teams (vision research, robotics demos), each with their own pooled budget on top.
The config looked something like this:

teams:
  - id: vision-research
    monthly_budget_eur: 1800
  - id: robotics-demos
    monthly_budget_eur: 1200

virtual_keys:
  - id: vk_marco
    team: vision-research
    monthly_budget_eur: 220
    allowed_models: ["openai/gpt-4o-mini", "anthropic/claude-3.5-sonnet"]
  - id: vk_sofia
    team: vision-research
    monthly_budget_eur: 220
    allowed_models: ["openai/gpt-4o-mini"]

When a researcher's key hits 90% of their monthly cap, the gateway rejects further requests until the next cycle. No silent overspend. The team-level budget acts as a second ceiling: even if every individual cap stays under, the team budget kicks in.

We pointed every script at http://bifrost.internal:8080/v1/chat/completions and replaced the OpenAI/Anthropic SDK calls with the unified endpoint. The drop-in replacement meant we didn't touch the actual annotation code beyond the base URL.

What we got back

Three things, in honest order of value:

Per-key spend visibility. Prometheus metrics out of the box. We pipe them into our existing Grafana, and now I can see at a glance that Sofia spent €43 on Monday running a prompt sweep. Not to shame anyone. To understand the pattern.
Automatic failover. When OpenAI had its 47-minute hiccup on March 14, our overnight caption job didn't die. Bifrost rerouted to Anthropic. We found out from the metrics, not from a panicked Slack message at 7am.
Semantic caching on the bulk caption jobs. About 18% cache hit rate on repeated frames from similar scenes. Modest. Not the savings argument I would lead with, but real.

Trade-offs and limitations

I should be honest about where this is and isn't worth it.

If you're a solo engineer with one API key and one provider, none of this applies. Put your OPENAI_API_KEY in .env and move on. The hierarchical budget story matters when you have multiple humans with their own scripts.

LiteLLM is the obvious comparison. We tried it first. It works, the routing logic is solid, and the Python-native feel suits a research team. What pushed us to Bifrost was the governance side: virtual keys with per-team and per-user budgets as a first-class concept, not bolted on. Portkey has similar governance features but felt heavier than we needed for an 11-person internal deployment.

The Bifrost web UI is functional but not pretty. The semantic cache config has some sharp edges (the embedding model choice matters more than the docs imply). And if your provider mix is exotic (we briefly wanted Mistral's hosted vision endpoint, which wasn't supported at the time we tested), you'll need to wait or contribute.

The real limitation though: a gateway cannot save you from a researcher who genuinely needs to spend €500 on a legitimate experiment. It can only make that spend visible and intentional. Which, honestly, is what we wanted.

Falling back from edge detection to a cloud VLM when confidence drops

Marco Rinaldi — Tue, 26 May 2026 07:22:10 +0000

TL;DR: We deploy a 4MB SSD detector on an ARM edge box and cascade low-confidence frames to a cloud VLM. About 3% of frames make the trip. The interesting part is not the model but the routing layer that decides when to ask for help and how to fail gracefully when the network is hostile.

Last winter we shipped a vision system for an industrial inspection client. The constraint was familiar: a Cortex-A72 box bolted to a conveyor, no GPU, intermittent 4G uplink. The detector had to run at 25 FPS and flag defects. Easy enough. The hard part came later.

So, the thing is, on a clean validation set the small detector hit 91% mAP at IoU 0.5. In production it hovered around 78%. The drop came from frames the model had never imagined: unusual lighting on Mondays after the floor was washed, parts placed at angles outside our training distribution, partial occlusions from operator gloves. The honest answer was that we didn't have a large enough labeled set to cover the tail.

Rather than ship a bigger model the box couldn't run, we built a cascade. The edge model handles every frame. When its top-1 detection confidence drops below 0.55 or its second-best is within 0.15 of the first, we send the cropped patch to a cloud VLM for a second opinion. About 3% of frames trigger this path.

The cascade in practice

Here is the gist of the gating logic, simplified:

def should_escalate(detections):
    if not detections:
        return False
    top = detections[0].score
    if top < 0.55:
        return True
    if len(detections) > 1:
        runner_up = detections[1].score
        if (top - runner_up) < 0.15:
            return True
    return False

A patch is 224x224 around the predicted bounding box, encoded as JPEG quality 80. Average payload is 11 KB. Round-trip latency to the cloud VLM is 800-1400ms over 4G, which is acceptable because the conveyor has a 2-second buffer.

Why a routing layer mattered

The first version called one provider directly. Then we had a 17-minute outage and the line operator had to override frames manually. Not great. The second version routes through a gateway that fans out across two VLM providers with automatic failover. We use Bifrost (https://github.com/maximhq/bifrost) on a small VPS for this, though LiteLLM or a custom proxy would also work. With one provider as primary and another as secondary, when the primary returns 5xx or times out past 2 seconds, the next one takes over. Semantic caching helps for near-identical frames during steady-state operation.

The choice of gateway is not the headline. What matters is having one. Calling provider SDKs directly from edge devices is a debugging nightmare once you have failover logic and retries layered on top.

What we measured

After 6 weeks of production data on three client lines:

Metric	Edge only	Edge + VLM cascade
mAP (production)	78.2%	86.9%
Avg latency / frame	38 ms	41 ms
p99 latency / frame	52 ms	1290 ms
Cloud cost / day	$0	$4.20
Frames escalated	0%	3.1%

The p99 jump is real and unavoidable. We absorb it in the conveyor buffer. Without that slack the cascade would not be an option.

A subtle bug we hit

Early on, the cascade rate drifted upward over a week, climbing from 3% to 7%. Cost was rising too. After some digging we found the detector was responding to slow camera lens contamination. Dust accumulating on the optics was lowering confidence everywhere. The cascade masked the real problem because the cloud VLM was strong enough to handle the dirty frames.

Now we alert when the cascade rate moves beyond a 5% rolling band. The cloud should be a backup, not a crutch.

Trade-offs and limitations

Latency variance. The p99 number above is honest. For applications without buffer slack (closed-loop robotics, automotive vision), this approach falls apart. It works for our throughput-tolerant inspection setup. It would not work for collision avoidance.

Network dependence. When the 4G link drops the system degrades to edge-only. We log every frame that would have escalated and run a batch job once connectivity returns. About 8% of escalations end up batched on a bad day. The client accepted this. Another client might not.

Cost predictability. Per-frame cost is small but it is real. A noisier production environment can double the cascade rate overnight. We have a hard daily ceiling that disables escalation if exceeded, and the line falls back to edge-only with a flag for operator review.

Privacy. The cropped patch is sent to a third-party provider. For industrial inspection of inert parts this was fine. For anything involving people we would need on-prem VLMs, which changes the economics considerably.

Calibration drift. The confidence threshold of 0.55 was tuned on validation data from one factory. The other two factories needed different thresholds (0.48 and 0.61). We now run a small calibration script after first deployment.

Closing thought

Cascade architectures are old. Hierarchical classifiers and coarse-to-fine pipelines have been around since before I started this job. What changed is that the "ask for help" tier is now a generally capable model rather than a slightly larger specialist. That shifts the economics for the long tail of unusual inputs that no training set will ever cover.

Auto-labelling 1.2M robotics frames with VLMs: a failover story

Marco Rinaldi — Mon, 25 May 2026 16:53:43 +0000

TL;DR: We needed to caption 1.2M reconstructed event-camera frames using vision-language models for auxiliary supervision. The first run died at 340K from Anthropic rate limits. Putting Bifrost in front of three VLM providers cut the rerun cost by 22% and finished in 9 hours.

So, the thing is, when you work at a neuromorphic vision startup, your training data looks strange. At Prophesee we accumulate event streams into time-binned windows that we render into pseudo-frames. For a self-supervised pretraining run on a new asynchronous backbone, we wanted natural-language captions on every window. Not because we're going language-first. The captions act as auxiliary targets for a contrastive head that sits alongside the actual event tensor.

1.2M frames. Three candidate VLMs: GPT-4o, Claude 3.7 Sonnet, Gemini 2.5 Pro. All three caption our weird greyscale reconstructions differently enough that we wanted a mix per frame.

I tried Anthropic first because the captions were qualitatively the best on our pilot set. Job died at 340,317 captions on a sustained TPM cap. That was a Friday evening before a long weekend in Bologna. I lost the weekend.

Choosing a gateway over more retry code

My first instinct was to write a smarter retry loop. Every CV engineer has this instinct when they discover REST APIs aren't deterministic. After about three hours of writing what was clearly going to become a half-baked rate-limit handler with provider-specific quirks, I stopped.

The actual problem was that I had multiple providers, all with their own SDKs and their own error formats. I needed something in the middle that knew about quotas, retries, and fallback chains, and that wasn't going to require me to learn yet another vendor lock-in.

I looked at LiteLLM, Portkey, and Bifrost. Ended up running Bifrost in Docker on the same node as the batch dispatcher.

The setup

Bifrost runs as a single Go binary or container. The config that mattered for us was the fallback chain. Here's the trimmed version we shipped:

providers:
  openai:
    keys: [${OPENAI_KEY_1}, ${OPENAI_KEY_2}]
    weight: 0.5
  anthropic:
    keys: [${ANTHROPIC_KEY_1}]
    weight: 0.3
  vertex:
    keys: [${VERTEX_KEY_1}]
    weight: 0.2

fallbacks:
  - model: openai/gpt-4o
    next: [anthropic/claude-3-7-sonnet, vertex/gemini-2.5-pro]
  - model: anthropic/claude-3-7-sonnet
    next: [openai/gpt-4o, vertex/gemini-2.5-pro]

Our batch dispatcher called http://bifrost:8080/v1/chat/completions with whatever model we picked for that frame. If a provider was over quota, Bifrost handled the failover and the dispatcher never saw the error. That part is documented under retries and fallbacks.

We also turned on semantic caching for the prompt template because we caption a lot of near-identical static scenes. Robotics demos have long boring stretches. Cache hit rate landed around 14% on the full run, which isn't huge but covered the cost of running the gateway itself.

How it compared

Concern	LiteLLM	Portkey	Bifrost
Multi-provider failover	Yes	Yes	Yes
Self-hosted in our VPC	Yes	Paid tier	Yes (Docker)
Semantic caching built-in	Plugin	Yes	Yes
Prometheus metrics native	Partial	Yes	Yes
Single binary deploy	No (Python)	N/A (SaaS)	Yes (Go)
800 req/s sustained	GIL issues	N/A	Held

LiteLLM was the most familiar option for our team because we already use it for eval scripts. Honestly for offline single-process work it's fine. The problem hit us when we tried to push sustained throughput through one Python process. Bifrost being Go meant we didn't fight the GIL. Portkey's hosted product is genuinely nice and the analytics UI is better than what Bifrost shipped, but we needed everything inside our VPC for frames covered by client confidentiality.

Results

The full 1.2M caption run finished in 9 hours and 14 minutes. Total cost was $4,180, down from a projected $5,360 if we'd run everything on GPT-4o. The 22% saving came from routing roughly a third of traffic to Gemini, which is cheaper per token for our prompt length.

Two providers had transient 429 spikes during the run. I didn't have to do anything about either. The gateway absorbed them. I noticed only because the per-provider request graph in the Bifrost dashboard had a visible dip on Anthropic around hour four.

Trade-offs and limitations

Not everything was clean.

Latency overhead. Bifrost adds a hop. For batch labelling it didn't matter. For an interactive vision app streaming a webcam, I'd benchmark carefully before putting any gateway in the path.

Caption drift across providers. Captions from Gemini and Claude are stylistically different even with the same prompt. We had to normalise downstream with a small T5 rewriter. The gateway doesn't solve this for you.

Config sprawl. Once you have weights, fallbacks, virtual keys, and cache rules in one YAML, it gets hard to reason about which path a given request actually took. Bifrost's logging helped but I had to dig.

MCP and tool use. We didn't need them. If you're building an agent product instead of a labelling pipeline, the MCP support might matter more than failover.

What I'd do differently

Run a 5K-frame pilot before launching the full job. We did 50K, which was enough to catch the rate-limit issue conceptually but not enough to see what 800 req/s sustained does to a Python process. Also: drink the espresso before debugging gateway configs at 1am, not after.

Quantising event-camera networks to run under 1MB on a Cortex-M7

Marco Rinaldi — Mon, 25 May 2026 07:22:24 +0000

TL;DR: I shrunk a gesture-recognition model for a Prophesee EVK4 event camera from 4.2MB down to 780KB so it could run on an STM32H7 at 15ms per inference. The trick was not the quantisation itself, it was rethinking what an "image" even means when your sensor produces events instead of frames.

So, the thing is, most computer vision tutorials assume you start with a tensor of shape '[B, 3, H, W]' and end with a classification head. Event cameras break that assumption on day one. A Prophesee sensor doesn't give you frames at 30fps. It gives you a sparse stream of events, each a tuple of '(x, y, t, polarity)', fired only when a pixel changes brightness. You can get millions of these per second during motion and almost nothing when the scene is static.

That changes the entire optimisation game. Let me give you the full picture here.

The starting point

We had a gesture model trained on the DVS128 Gesture dataset (11 classes, hand movements recorded with a DAVIS sensor). The baseline was a small ResNet-ish backbone running on event frames, accumulated over 50ms windows. It hit 94.1% test accuracy at 4.2MB in fp32. Inference on a Cortex-M7 at 480MHz took 68ms per window, which is too slow when your events are arriving in real time.

Target: sub-1MB, sub-20ms, accuracy drop under 2 percentage points.

Step one: stop pretending events are frames

The first 30% of the size came from not training on frames at all. Event-frame accumulation throws away the temporal resolution you paid for when you bought a €3,000 sensor. We switched the input representation to a voxel grid of shape '[2, 5, 128, 128]' (2 polarities, 5 temporal bins per window). That alone let us drop the first conv block from 64 channels to 24, because the input was already temporally structured.

def events_to_voxel(events, num_bins=5, height=128, width=128):
    voxel = torch.zeros(2, num_bins, height, width)
    t_min, t_max = events[:, 2].min(), events[:, 2].max()
    t_norm = (events[:, 2] - t_min) / (t_max - t_min + 1e-9)
    bin_idx = (t_norm * (num_bins - 1)).long()
    for i, (x, y, _, p) in enumerate(events):
        voxel[int(p), bin_idx[i], int(y), int(x)] += 1.0
    return voxel

Accuracy actually went up to 94.6%. Smaller and better. This happens more often than the literature admits.

Step two: QAT, not PTQ

Post-training quantisation is fast but it lies to you on event data. The activation distributions are wildly bimodal because most pixels are zero. Standard min/max calibration collapses the useful range.

We did quantisation-aware training with PyTorch's torch.ao.quantization pipeline, qint8 weights and activations, per-channel for convs, per-tensor for the linear head. 15 epochs of QAT on top of the fp32 checkpoint. The observer matters: MovingAverageMinMaxObserver with averaging_constant=0.01 worked, the default 0.1 did not.

Stage	Size	Accuracy	M7 latency
fp32 baseline (frames)	4.2 MB	94.1%	68 ms
fp32 + voxel input	3.1 MB	94.6%	51 ms
PTQ int8	820 KB	89.2%	19 ms
QAT int8	780 KB	93.4%	15 ms

The 4-point gap between PTQ and QAT is the whole story. If anyone tells you PTQ "just works" on sparse inputs, ask them to show you the per-class confusion matrix.

Step three: deployment plumbing

Getting a quantised PyTorch model onto a microcontroller is the part nobody writes about. We export to ONNX, then run it through X-CUBE-AI from ST to get C code we can flash to the H7. The flow is finicky around quantisation parameters, so we wrote a small validator that runs the same 200 inputs through PyTorch, ONNX Runtime, and the on-device binary and compares logits. Three places to get a mismatch.

For the cloud side, when we're benchmarking different model variants we need an LLM to summarise long evaluation logs and propose hyperparameter tweaks. Our team of four uses Bifrost in front of OpenAI and a local Ollama instance so we can switch between them when the OpenAI rate limit bites during a sweep. It's one less thing to babysit.

Trade-offs and limitations

This pipeline only really works because gesture recognition is forgiving. A 1% accuracy drop on hand waves doesn't kill anyone. For automotive event-camera workloads (pedestrian detection at 30m), the same int8 quantisation pushed our internal benchmark from 87.3% AP to 81.0% AP, which is unacceptable. You'd need mixed-precision (int8 backbone, fp16 detection head) or a larger budget.

Also: the voxel-grid representation assumes you can buffer 50ms of events before inferring. If you need sub-10ms reaction time on a 1kHz event stream, you're back to recurrent spiking architectures or sparse convs, and that's a different blog post.

The X-CUBE-AI toolchain is closed-source and the error messages will make you reconsider your career on a Tuesday afternoon. TFLite Micro is more portable but has worse op coverage for the things event-vision pipelines actually need (per-channel quantised group convs especially).

What I'd do differently

Train with QAT from epoch zero, not as a fine-tune. The fp32-then-quantise habit comes from natural-image work where it's basically free. On event data, the activation statistics shift enough during QAT that you might as well let the model converge to them properly. We're trying this on the next project.

And benchmark on the actual hardware early. The first three weeks I tuned everything against PyTorch latency on a workstation. Useless. The M7 has a different cache hierarchy and SIMD profile, and the model that "should" be fastest on paper was 2x slower in silicon.

Quantising event-camera networks to run under 1MB on a Cortex-M7

Marco Rinaldi — Fri, 22 May 2026 07:22:38 +0000

So, the thing is, most computer vision tutorials assume you start with a tensor of shape [B, 3, H, W] and end with a classification head. Event cameras break that assumption on day one. A Prophesee sensor doesn't give you frames at 30fps. It gives you a sparse stream of events, each a tuple of (x, y, t, polarity), fired only when a pixel changes brightness. You can get millions of these per second during motion and almost nothing when the scene is static.

That changes the entire optimisation game. Let me give you the full picture here.

The starting point

Target: sub-1MB, sub-20ms, accuracy drop under 2 percentage points.

Step one: stop pretending events are frames

The first 30% of the size came from not training on frames at all. Event-frame accumulation throws away the temporal resolution you paid for when you bought a €3,000 sensor. We switched the input representation to a voxel grid of shape [2, 5, 128, 128] (2 polarities, 5 temporal bins per window). That alone let us drop the first conv block from 64 channels to 24, because the input was already temporally structured.

def events_to_voxel(events, num_bins=5, height=128, width=128):
    voxel = torch.zeros(2, num_bins, height, width)
    t_min, t_max = events[:, 2].min(), events[:, 2].max()
    t_norm = (events[:, 2] - t_min) / (t_max - t_min + 1e-9)
    bin_idx = (t_norm * (num_bins - 1)).long()
    for i, (x, y, _, p) in enumerate(events):
        voxel[int(p), bin_idx[i], int(y), int(x)] += 1.0
    return voxel

Accuracy actually went up to 94.6%. Smaller and better. This happens more often than the literature admits.

Step two: QAT, not PTQ

Stage	Size	Accuracy	M7 latency
fp32 baseline (frames)	4.2 MB	94.1%	68 ms
fp32 + voxel input	3.1 MB	94.6%	51 ms
PTQ int8	820 KB	89.2%	19 ms
QAT int8	780 KB	93.4%	15 ms

The 4-point gap between PTQ and QAT is the whole story. If anyone tells you PTQ "just works" on sparse inputs, ask them to show you the per-class confusion matrix.

DEV Community: Marco Rinaldi

One image schema for four VLM providers: we stopped reformatting payloads

The actual pain

What we changed

How it compares

Trade-offs and Limitations

Further Reading

Our event-camera detector lost 6 mAP to a badly chosen accumulation window

The regression that was not a model regression

What the window actually trades

The numbers from our conveyor set

How we audit windows now

Trade-offs and Limitations

Further Reading

Structured channel pruning got our detector under 12ms on a Jetson

The trap I fell into first

Where the latency actually lives

The numbers

Trade-offs and limitations

What I would tell past me

Further Reading

VLM-scored calibration sets for INT8 quantisation, routed through Bifrost

What we actually wanted from a gateway

The scoring loop

How it compares to LiteLLM and Portkey

Trade-offs and Limitations

Further Reading

QAT vs PTQ on our edge vision model: 6 months of A/B data

The model and the constraint

Where PTQ falls apart

What we tried

What QAT does not fix

Trade-offs and limitations

Further Reading

Six weeks of Bifrost in a factory QA pilot: real cost numbers

The setup

What we measured

Bifrost vs LiteLLM vs Portkey

Trade-offs and limitations

Further reading

Distilling SAM 2 into a 6MB student for industrial inspection

The student architecture

The loss that actually worked

What the numbers look like

Where a VLM judge crept in

Trade-offs and limitations

Further reading

Capping VLM spend per CV researcher: hierarchical budgets in practice

What we were actually doing

What we wired up

What we got back

Trade-offs and limitations

Further reading

Falling back from edge detection to a cloud VLM when confidence drops

The cascade in practice

Why a routing layer mattered

What we measured

A subtle bug we hit

Trade-offs and limitations

Closing thought

Further reading

Auto-labelling 1.2M robotics frames with VLMs: a failover story

Choosing a gateway over more retry code

The setup

How it compared

Results

Trade-offs and limitations

What I'd do differently

Further reading

Quantising event-camera networks to run under 1MB on a Cortex-M7

The starting point

Step one: stop pretending events are frames

Step two: QAT, not PTQ

Step three: deployment plumbing

Trade-offs and limitations

What I'd do differently

Further reading

Quantising event-camera networks to run under 1MB on a Cortex-M7

The starting point

Step one: stop pretending events are frames