Raghu Bharadwaj

Posted on Aug 18 Originally published at techveda.live

Operator Fallback: Why Your Accelerated Model Still Runs on the CPU

#machinelearning #embedded #linux #performance

When a model runs on an NPU or GPU, the runtime only offloads the operators that backend supports. Everything else stays on the CPU, which splits one graph into many small pieces that run one after another with copies and synchronisation between them. On published measurements from three Android phones, this fragmentation made one image classifier more than ten times slower through the accelerator path than on the CPU alone. Before you assume an accelerator is helping, count the partitions and profile per operator.

Most embedded teams enable an accelerator delegate, see a number that is not much better than the CPU number, and conclude the hardware was oversold. The usual cause is operator fallback: the runtime hands the accelerator only the operators its backend implements, and every remaining operator runs on the CPU. The model is still correct, but it is no longer one contiguous piece of work. This article covers what that does to execution, what it measures out to on real devices, and how to check your own model.

What operator fallback actually is

An inference runtime such as TensorFlow Lite, ONNX Runtime or ExecuTorch asks the backend which operators it can execute. Supported operators are grouped into partitions and handed over; the rest are left with the CPU implementation. Two things commonly trigger this:

Unsupported kernels. A backend implements a fixed operator set. A newer attention variant, an unusual normalisation, or an operator at a data type the backend does not accelerate will not be taken.
Dynamic shapes and control flow. Detectors emit a variable number of boxes, speech models use beam search, and text encoders take variable sequence lengths. Backends built around static shapes either reject those regions or push the whole graph back to the CPU.

The result is a mixed graph. In TensorFlow Lite the runtime tells you this directly in its log, in a line of the form Replacing N out of M node(s) with delegate (…) node, yielding K partitions for the whole graph. K is the number you care about. K equal to 1 means the accelerator got one contiguous region. K in the double digits means the graph has been split into many small pieces.

Why fragmentation costs more than the operators it saves

Each partition boundary is a handover. Tensors are copied between the CPU and the accelerator's memory, and each side waits for the other to finish its section. Under the scheduling that stock runtimes use, the two processors take turns rather than working at the same time, so while one runs, the other is idle.

The scale of the fragmentation is easy to underestimate. The counts below are execution stages — groups of work that must run one after another — and not neural-network layers; delegation does not change a model's architecture, only how its graph is scheduled. Measurements on a Google Pixel 6 show Whisper-Tiny holding 627 nodes across 75 stages before delegation. Afterwards the node count fell to 202, which sounds like progress, but the stage count rose to 184. SwinV2-Tiny on the same device went from 151 stages to 270. The operator count went down and the number of synchronisation points went up.

Dynamic models carry a second cost. Runtimes plan memory ahead of time from fixed tensor shapes, so when a shape is only known at run time the planner has to invalidate and reallocate large regions. That adds work and prevents independent parts of the graph from running at the same time.

What the numbers look like on real devices

Published measurements across three Android phones — a Google Pixel 6 (Google Tensor), a Huawei P30 Pro (Kirin 980) and a Redmi K50 (Dimensity 8100) — show how far this can go. For a SwinV2-Tiny image classifier on the Pixel 6, TensorFlow Lite ran 96–108 ms on the CPU alone and 1107–1994 ms through the accelerator path. ONNX Runtime showed the same pattern: 82–87 ms on the CPU, 1323–1726 ms through the accelerator. The accelerator path was more than ten times slower, and it was slower because of the boundaries, not because the accelerator is weak.

The same measurements record a detail worth remembering when you read a datasheet. The Kirin 980 contains a Mali-G76 GPU and a dual-core NPU, and neither was reachable through Android's Neural Networks API, so the experiment used TensorFlow Lite's OpenCL backend instead. Silicon that no runtime can address is not available compute.

What a fallback-aware runtime does differently

None of the stock runtimes does this today; it is an active research area, and the results below come from a published experimental runtime rather than something you can enable with a flag. A runtime that treats fallback as normal rather than exceptional recovers much of the loss with three mechanisms:

Refuse small offloads. A region is worth sending only if the compute it saves exceeds the dispatch and transfer cost. One published cost model accepts a candidate only at 3 or more operators, at least 1×10⁹ MACs of compute, and a boundary transfer of at most 0.1 bytes per MAC. Smaller regions stay on the CPU, which removes boundaries instead of adding them.
Run independent branches at the same time. The graph is analysed as a DAG and CPU branches run in parallel with the accelerator's branch instead of waiting for it.
Give each branch its own memory arena. Global buffer reuse is what makes parallel branches unsafe, because two branches then share a buffer. Per-branch arenas with liveness-based reuse inside each arena keep parallelism safe, and a scheduler that queries free memory at run time with a 30–50% safety margin decides how many branches run together.

The reported gains were 15–31% lower latency for CPU-only execution of large-input models, and 9–46% lower latency in the mixed accelerator path against ONNX Runtime (20–45% against TensorFlow Lite). At the level of a single stage the effect is visible directly: one SwinV2-Tiny stage made of a single Pixel 6 TPU branch plus three CPU branches fell from 13.46 ms to 7.61 ms, 43.5% faster than TensorFlow Lite ran the same stage. Averaged across stages the improvement was 35.5%.

How to measure operator fallback on your own device

You do not need a research framework to find out whether this affects you. The stock benchmark tool reports both the partition count and per-operator timings:

raghu@techveda.org:~$ ./benchmark_model --graph=detector.tflite --use_gpu=true --enable_op_profiling=true
INFO: Created TensorFlow Lite delegate for GPU.
INFO: Replacing 30 out of 96 node(s) with delegate (TfLiteGpuDelegateV2) node, yielding 9 partitions for the whole graph.

Those numbers illustrate the log format; read your own line. Nine partitions across a 96-node graph means the accelerator is entered and left nine times per inference, and the per-operator profile shows which operator types were left behind. Vendor delegates print the same line with their own name, for example NeutronDelegate on NXP i.MX 95.

On ONNX Runtime the equivalent controls are provider options on the TensorRT execution provider: trt_min_subgraph_size sets the smallest node count a subgraph must have before it is sent to TensorRT, and trt_dump_subgraphs writes the partitions out as ONNX files for inspection. The older environment variables ORT_TENSORRT_MIN_SUBGRAPH_SIZE and ORT_TENSORRT_DUMP_SUBGRAPHS still work but are marked deprecated in the current documentation, so prefer the provider options in new code. Raising the minimum subgraph size is a simpler form of the pruning rule described above, and on a fragmented model it is usually worth testing.

Where this breaks down

The gains are not free, and the published limitations are as useful as the results.

Memory grows. Per-branch arenas give up some buffer reuse. Peak runtime memory rose 26.5% on average, and in the worst case 45.3 MB became 72.9 MB, an increase of 60.9%. On a memory-constrained device that alone can decide the design.
Energy can get worse. In CPU-only runs on one device, energy fell 18.3% against ONNX Runtime and 30.0% against ExecuTorch for a text encoder, but rose 47.0% for a detector and 92.2% for a text classifier against ExecuTorch. Parallel CPU branches draw more power, and the reported work did not optimise for energy.
Dynamic shapes are still not solved. For models with variable sequence lengths the experimental runtime had no accelerator path at all, and neither did TensorFlow Lite or ExecuTorch. ONNX Runtime did reach the accelerator on those models and remains the stronger option there. Fine-grained partitioning reduces the cost of fallback; it does not make a backend support an operator it never implemented.
The evidence is from Android smartphones. All three test devices were phones, and two of them used Android's Neural Networks API; the P30 Pro reached its GPU through TensorFlow Lite's OpenCL backend instead. Nothing here was measured on an embedded Linux board, and NPU driver stacks on i.MX, Jetson or Rockchip parts differ enough that you must re-measure on your own hardware. Note also that the NNAPI and Hexagon delegates are now deprecated and no longer supported upstream, so this particular offload path is not one to build on.

What this means for embedded and kernel engineers

Treat the accelerator as a scheduling problem rather than a configuration flag. Four questions decide your frame rate: how many partitions the graph produces, how much data crosses each boundary, how much memory the runtime needs at peak, and whether the CPU sits idle while the accelerator runs. All four are visible with tools you already have.

It also changes how you evaluate silicon. A part with a large advertised TOPS figure but a narrow operator set can lose to a part with lower nominal throughput and better coverage of your model. Ask the vendor for the supported operator list and the delegate's partitioning behaviour before you ask for the TOPS number.

Key takeaways

Operator fallback splits a model between the accelerator and the CPU; the split, not the operator count, usually decides latency.
Delegation can raise the number of sequential execution stages even as it lowers the node count — in one measured case from 75 stages to 184.
An accelerator path can be much slower than the CPU path: 1107–1994 ms against 96–108 ms for one classifier on a Pixel 6.
Pruning small offloads, running independent branches in parallel and isolating memory per branch recovered 9–46% latency in published research, at the cost of 26.5% more peak memory on average. No stock runtime does this yet.
Count your partitions with the benchmark tool before concluding the hardware is at fault.

Frequently asked questions

How do I know if my model is affected by operator fallback?
Run the TensorFlow Lite benchmark tool with the delegate enabled and read the log line that reports how many nodes were replaced and how many partitions resulted. More than a few partitions means the runtime is entering and leaving the accelerator repeatedly during every inference.

Why can the accelerator path be slower than the CPU path?
Each partition boundary adds a data copy and a synchronisation point, and under the scheduling stock runtimes use, the CPU and the accelerator take turns instead of working at the same time. In the measurements quoted here, one image classifier ran 96–108 ms on the CPU and 1107–1994 ms through the accelerator on the same device.

Does running CPU branches in parallel with the accelerator cost anything?
Yes. Isolating memory per branch reduces buffer reuse, so peak runtime memory rose 26.5% on average and by 60.9% in the worst reported case, and energy consumption increased for some models even though latency fell.

Do these results apply to embedded Linux boards?
Not directly. The measurements were taken on three Android smartphones, two of them through the Neural Networks API and one through an OpenCL backend. The mechanism is the same on i.MX, Jetson or Rockchip parts, but the numbers must be re-measured on your own board and driver stack.

DEV Community