"Real-time" is one of the most misused words in engineering. People say it to mean "fast." It doesn't. Real-time means on time — a result that arrives after its deadline is wrong, no matter how correct it is. A tracking system that identifies a person perfectly but half a second too late has already failed. Building Netra as a live camera system made me think in deadlines, not speeds.
Fast is an average; on-time is a guarantee
Optimizing for speed gets you a good average. Real-time work cares about the worst case, because the frame you miss is the one that mattered. A pipeline that runs in 20ms most of the time but occasionally spikes to 300ms isn't a fast system with a hiccup — for anything with a deadline, it's a system that fails intermittently and unpredictably, which is worse than one that's slow but steady.
So the question stops being "how fast can this go?" and becomes "can this reliably finish before the next frame needs it?" That reframing changes what you optimize. You stop chasing peak throughput and start hunting the tail — the rare slow path that blows the budget.
Everything gets a time budget
Once you're designing to a deadline, the frame interval becomes a fixed budget that every stage has to share: capture, detect, decide, act. Each stage spends a slice, and the slices have to sum to less than the deadline — every time, not on average.
- Budget the pipeline, not the peak. If detection can occasionally eat the whole frame, the parts after it starve. You size each stage against its worst case so the total still fits.
- Choose the model that fits the budget. A heavier model that's more accurate but blows the deadline is the wrong model here — a slightly simpler one that always finishes on time is strictly better, because a late answer isn't a slightly-worse answer, it's no answer.
- Steady beats bursty. A predictable, consistent frame time is more valuable than a lower average with an ugly tail — jitter is what makes a live system feel broken even when it's "fast."
The takeaway
Building anything live — a camera, a game loop, an audio engine, a control system — flips your intuition. Correctness alone isn't enough; correctness by the deadline is the whole spec. The discipline is to treat time as a hard resource you budget as carefully as memory, and to design for the worst frame, not the average one.
Netra taught me to think in deadlines. The full real-time pipeline — capture, detection, and the control loop — is on the project page.
👉 See it: www.divyakush.com/projects/netra
Divyakush Punjabi — Full-Stack & AI Systems Engineer
🌐 https://www.divyakush.com · 💼 LinkedIn · 💻 GitHub
Top comments (0)