The GPU Delegate Isn't Always Faster
TFLite's GPU delegate crashes on Jetson devices more often than it accelerates inference. The promise is simple: offload compute to the GPU, get faster inference. The reality? Segfaults, cryptic CUDA errors, and latency that's worse than CPU.
I've seen this pattern across Jetson Nano, Xavier NX, and Orin modules. You enable the GPU delegate, run inference, and get either a crash or performance that makes you wonder why you bothered. The XNNPACK delegate, meanwhile, just works — and often beats GPU latency by 20-40% on common vision models.
This isn't a hardware limitation. It's a mismatch between what TFLite's GPU delegate expects and what Jetson actually provides.
Why the GPU Delegate Fails
Jetson devices run NVIDIA's Tegra architecture, which combines ARM CPU cores with CUDA-capable Maxwell/Pascal/Ampere GPUs. TFLite's GPU delegate was primarily designed for mobile GPUs (Mali, Adreno) using OpenGL ES compute shaders. Jetson support exists, but it's treated as a secondary target.
Continue reading the full article on TildAlice
Top comments (0)