Autonomous vehicles need to process frames in 50-100ms to react to obstacles. Augmented reality needs 16ms frame times to hit 60fps without feeling laggy. Industrial inspection has to keep pace with a moving line. These are hard latency budgets, and hitting them consistently takes engineering across the whole pipeline, not just a fast model.
Latency Is Not Just Inference Time
End-to-end latency in a vision system breaks down into several stages, each with its own typical range and its own optimization lever:
| Stage | Typical Latency | Optimization Lever |
|---|---|---|
| Image capture | 1-33ms (depends on fps) | Camera selection, exposure settings |
| Data transfer | 1-10ms | DMA, zero-copy buffers, GPU-direct capture |
| Preprocessing | 2-10ms | GPU-accelerated resize, on-device normalization |
| Model inference | 5-100ms | Model optimization, quantization, hardware selection |
| Post-processing | 1-10ms | NMS optimization, result filtering |
| Action/response | 1-5ms | Direct hardware control, efficient IPC |
A team that spends all its optimization budget on the model while capture and transfer stay untouched is optimizing the wrong variable. A holistic view of the pipeline is what actually gets you under budget.
Batching: Throughput vs Latency
GPUs are massively parallel and sit underutilized when fed one image at a time. Batching fixes utilization but costs latency, so the strategy matters:
- Static batching: wait to collect a fixed number of requests before running inference. Simple, but adds wait time equal to however long it takes to fill the batch.
- Dynamic batching: process whatever requests arrived within a short time window (e.g., 5ms). This is what Triton Inference Server and TorchServe support natively, and it balances throughput against latency better than static batching.
- Micro-batching: for streaming video from multiple cameras, batch frames across streams rather than across time from a single stream.
The right batch size is a function of your latency budget. If you have 100ms to work with and inference takes 30ms per image, a batch of 2-3 might be the sweet spot — bigger batches raise throughput but blow past the target.
Asynchronous Pipelines
Running every stage sequentially means your total latency is the sum of every stage's latency. Decoupling the stages so they run concurrently on different hardware changes that:
# Conceptual async pipeline
while running:
# These run concurrently on different hardware
future_frame = camera.capture_async() # Camera sensor
future_preprocess = preprocess_async(frame) # CPU
future_inference = model.infer_async(batch) # GPU
future_postprocess = postprocess(results) # CPU
# Pipeline: while GPU processes frame N,
# CPU preprocesses frame N+1,
# and camera captures frame N+2
With this pipeline parallelism, no piece of hardware sits idle waiting on another. The camera captures frame N+2 while the CPU preprocesses N+1 and the GPU infers on N. Throughput ends up bound by the slowest stage in the pipeline, not the sum of all of them.
Streaming Video Has Its Own Rules
Continuous streams introduce failure modes that don't show up when you're just benchmarking single images:
- Frame skipping: if inference takes longer than the frame interval, don't let a queue build up. Drop stale frames and process the most recent one instead of working through a growing backlog.
- Temporal redundancy exploitation: run full inference on keyframes and cheaper updates on the frames in between. If the scene hasn't changed much, reuse the previous result instead of recomputing from scratch.
- Region-of-interest processing: run a lightweight detector across the full frame first, then reserve the expensive model for the regions that actually matter.
- Multi-resolution strategies: detect at low resolution, then crop and process only the interesting regions at full resolution.
Hardware Options for Edge Deployment
The right hardware depends on your power, cost, and flexibility constraints:
- NVIDIA Jetson (Orin, Xavier): 10-275 TOPS, the standard for edge AI in robotics and autonomous vehicles.
- Google Coral (Edge TPU): 4 TOPS for TFLite models at 2W, well suited to always-on vision devices.
- Intel Movidius (VPU): neural compute sticks common in smart cameras and drones.
- Apple Neural Engine: 15.8 TOPS on M1, up to 38 TOPS on M3 Max, accessed through CoreML.
- Custom ASICs: maximum efficiency for a fixed model architecture, at the cost of zero flexibility if requirements change.
Benchmarking Pitfalls
Always profile on the actual target hardware, not a dev workstation. Three things reliably distort results:
- Cold start overhead: the first inference is slower because of memory allocation and kernel compilation. Warm up the model before you start timing.
- Memory contention: other processes competing for GPU or CPU memory can introduce latency spikes that don't show up in isolated tests.
- Thermal throttling: embedded devices can show degraded performance after several minutes of sustained load, so short benchmarks can be misleadingly optimistic.
Closing
Meeting a real-time latency budget is a systems problem, not a model problem. The camera, the transfer path, the batching strategy, the pipeline concurrency, and the hardware all contribute, and any one of them can become the bottleneck if ignored.
This is a condensed version of the full lesson, which goes into more depth on each of these topics: Real-Time Vision Systems, free on NeutralBlock.
Top comments (0)