On-Device Computer Vision Under 16 Milliseconds

In the development of clinical AI and real-time health monitoring, the difference between a functional interface and a broken user experience is measured in milliseconds. When I led the architecture for Synapsis Medical Technologies as the first engineering hire, I was tasked with building systems that bridged the gap between high-stakes medical data and immediate user feedback. Across the 18 production applications I have shipped, one constraint remains absolute: if you want a user to perceive a computer vision task as "real-time," you have exactly 16.6 milliseconds to complete your work.

This is the 60 FPS (frames per second) budget. In a browser or mobile environment, this budget must cover camera capture, frame preprocessing, model inference, post-processing, and the final UI render. If your inference takes 25ms, you have already failed the latency test, resulting in "jank" that erodes user trust—a critical failure in HealthTech.

To hit these targets, we are seeing a fundamental shift away from server-side inference toward on-device execution using MediaPipe and WebGPU. By moving the compute to the edge, we eliminate the 100ms–500ms round-trip latency of a network request, but we inherit the brutal constraints of mobile hardware.

The Problem: The Latency Floor

When building the architecture at Synapsis, where I owned the React Native, Next.js, and NestJS stack from 0 to 1, I encountered the "latency floor." Even with a highly optimized NestJS backend, a request originating from a mobile device on a 4G connection faces physical limitations.

For a computer vision task—such as detecting a patient's range of motion or analyzing a skin lesion—sending raw video frames to a cloud-based GPU is architecturally expensive. You face:

1. Ingress Costs: High-resolution frames consume significant bandwidth.
2. Privacy Concerns: Handling PHI (Protected Health Information) requires HIPAA-aligned pipelines. While I have maintained 99.9% uptime for HIPAA-aligned RAG pipelines, the most secure data is the data that never leaves the device.
3. Jitter: Network fluctuations make a consistent 60 FPS frame rate impossible.

The solution is to treat the client’s GPU as the primary compute resource. However, the traditional WebGL approach for browser-based AI is reaching its limits. WebGL was designed for drawing triangles, not for general-purpose parallel computation. This is where WebGPU changes the design requirements.

Context: The Shift to WebGPU and MediaPipe

The ecosystem has recently reached a tipping point. Chrome 113 introduced WebGPU by default, and MediaPipe has evolved from a C++ internal Google tool to a cross-platform framework that supports WebAssembly (Wasm) and WebGPU acceleration.

Unlike WebGL, which requires "faking" compute by storing data in textures and using fragment shaders, WebGPU provides direct access to GPU compute shaders. This allows for more efficient memory layouts and reduced overhead when passing data between the CPU and GPU. For engineers building React Native or web-based medical interfaces, this means we can now run models like BlazeFace or MediaPipe Landmarker at speeds that were previously reserved for native C++ implementations.

Technical Explanation: The 16ms Pipeline

To achieve sub-16ms latency, the pipeline must be non-blocking. In my experience scaling engineering teams and systems, the most common bottleneck isn't the model itself, but the data transfer between the CPU and GPU.

The pipeline generally follows this flow:

1. Texture Acquisition: The camera feed is uploaded to the GPU as a texture.
2. Preprocessing: Resizing, normalization, and color space conversion occur via compute shaders.
3. Inference: MediaPipe executes the model using the WebGPU backend.
4. Post-processing: Converting tensors back into human-readable coordinates or masks.
5. Rendering: Drawing the results onto a <canvas> or overlaying them on a React Native view.

If you use the CPU for step 2 or 4, you will likely exceed your 16ms budget. The "Zero-Copy" principle is essential here. You want the data to stay in GPU memory from the moment the camera captures the frame until the moment the result is rendered to the screen.

Architecture and Trade-offs

During my 8+ years of professional engineering, I have found that every performance gain comes with a trade-off in complexity or accuracy.

Quantization vs. Precision

To fit a model into the 16ms window, you often have to move from Float32 to Int8 or Float16 quantization. In a clinical AI context, this is a sensitive trade-off. A lower-precision model might run in 8ms instead of 20ms, but if it increases the error rate for a diagnostic tool, the speed is irrelevant. The architect's role is to define the "minimum viable precision" required for the use case.

Wasm vs. WebGPU

While WebGPU is faster, Wasm (WebAssembly) with SIMD (Single Instruction, Multiple Data) is more compatible across older devices. When I overhauled CI/CD cycles from 2 days to 4 hours, one of the primary goals was enabling rapid testing across a fragmented device landscape. If your user base is using five-year-old Android devices, a WebGPU-only architecture will fail. A hybrid approach—detecting WebGPU support and falling back to Wasm—is the standard for production-grade applications.

A Worked Example: Hand Tracking

Consider a scenario where we need to track 21 3D hand landmarks. Using MediaPipe’s WebGPU delegate, the initialization looks like this:

const vision = await FilesetResolver.forVisionTasks(
  "https://cdn.jsdelivr.net/npm/@mediapipe/tasks-vision/wasm"
);

const handLandmarker = await HandLandmarker.createFromOptions(vision, {
  baseOptions: {
    modelAssetPath: `hand_landmarker.task`,
    delegate: "GPU" // Forces WebGPU/WebGL usage
  },
  runningMode: "VIDEO",
  numHands: 2
});

The critical optimization happens in the animation loop. Using requestAnimationFrame, we must ensure the detectForVideo call returns within our budget. If the inference takes 12ms, we have 4.6ms left for UI updates. In a React environment, this means avoiding state updates that trigger heavy re-renders. Instead, we should use useRef to hold the landmark data and update a Canvas API directly, bypassing the React reconciliation engine for the high-frequency frame updates.

What it Cost to Learn

Building these systems at scale taught me that the "happy path" in documentation rarely survives production. When I scaled the Synapsis team from 0 to 21 engineers, we learned that the biggest performance killer wasn't the GPU—it was thermal throttling.

On mobile devices, running a GPU-intensive model at 60 FPS will heat the device rapidly. Within minutes, the OS will throttle the clock speed, and your 16ms inference time will balloon to 40ms. We had to implement "Adaptive Frame Rates." If the device temperature rose or the battery was low, we would drop the target to 30 FPS. This preserved the utility of the application at the cost of some smoothness, which is a necessary compromise in a professional environment.

Furthermore, integrating these vision tasks into a HIPAA-aligned environment meant ensuring that the frames processed in GPU memory were never cached or logged. Even on-device, data persistence must be strictly controlled.

Practical Recommendations

For architects looking to implement on-device vision, I recommend the following:

1. Profile the "Bus," not just the Model: Use Chrome DevTools' Performance tab to see how long texImage2D or device.queue.writeBuffer takes. Often, moving the data to the GPU takes longer than the actual inference.
2. Use Web Workers: Run the MediaPipe inference in a Dedicated Worker. This keeps the main thread free for user interactions, ensuring the UI remains responsive even if a frame takes longer than 16ms to process.
3. Optimize the CI/CD for Models: As I did with our production systems, automate the model quantization and conversion process. When a data scientist updates a model, the CI pipeline should automatically generate the Wasm and WebGPU-optimized versions, running automated latency benchmarks on real devices before the code is even reviewed.
4. Prioritize FHIR/HL7 Integration early: If the vision data needs to eventually reach a clinical record, ensure your post-processing step outputs data in a format compatible with healthcare standards. At Synapsis, integrating wearables and FHIR data was as important as the AI itself.

Conclusion

Achieving sub-16ms computer vision on the web is no longer an experimental feat; it is a requirement for modern, high-performance applications. By leveraging MediaPipe and WebGPU, we can build tools that were previously impossible without native development.

The transition from cloud-centric AI to edge-compute architecture requires a disciplined approach to the frame budget. As an architect, your job is to manage the tension between model accuracy, device thermal limits, and the unrelenting 60 FPS clock. When these elements are balanced, the result is a seamless, secure, and highly responsive user experience that respects both the user’s time and their data privacy.

---

Amit Chakraborty is a founding engineer and senior architect — React Native, AI/RAG systems and production architecture. Portfolio: www.amitchakraborty.dev · LinkedIn · GitHub. Open to senior and founding engineering roles, remote worldwide.