SoftwareDevs mvpfactory.io

Posted on Jul 27 • Originally published at mvpfactory.io

Wiring Android Neural Networks API to llama.cpp: Bypassing JNI Overhead with Shared Memory and Zero-Copy Tensor Buffers

#webdev #programming

---
title: "Zero-Copy LLM Inference on Android: Wiring llama.cpp to NNAPI Without the Hidden CPU Fallback"
published: true
description: "Wire llama.cpp's Android backend to NNAPI via TFLite shim, eliminate JNI marshaling on token tensors, and ensure your Snapdragon NPU actually runs your quantized model."
tags: android, mobile, architecture, performance
canonical_url: https://blog.mvpfactory.co/zero-copy-llm-inference-nnapi-llama-cpp-android
---

What We Are Building

By the end of this tutorial you will know how to wire llama.cpp to Android's Neural Networks API correctly — verifying that your Snapdragon NPU is actually doing the work, eliminating per-token JNI copy overhead using AHardwareBuffer, and splitting prefill from decode across the right accelerators. This is the full stack: delegate selection, shared memory allocation, and zero-copy tensor handoff.

Let me show you a pattern I use in every on-device LLM project.

Prerequisites

Android NDK r25+
A Snapdragon device (Gen 2 or later recommended)
llama.cpp checked out with Android backend enabled
A quantized model in GGUF format (Q8_0 for broadest NPU coverage, Q4_K_M if you are on Gen 3+)
Familiarity with JNI and TFLite basics

Step 1: Check Whether Your NPU Is Actually Running

Most teams enable the NNAPI delegate flag, ship it, and assume the NPU is running. It almost certainly is not.

The NNAPI delegate performs capability checks at runtime. If any operator in your model graph is unsupported — common with non-standard quantization schemes like Q4_K_M — it silently partitions the graph. Unsupported ops fall back to CPU. In the worst case, the entire model runs on CPU with added delegation overhead on top.

Here is the minimal setup to get delegation logging working:

val options = Interpreter.Options().apply {
    addDelegate(
        NnApiDelegate(NnApiDelegate.Options().apply {
            setExecutionPreference(NnApiDelegate.Options.EXECUTION_PREFERENCE_SUSTAINED_SPEED)
            setAllowFp16(true)
            setModelToken("llama_q4")
        })
    )
}

Then run:

adb logcat | grep -i nnapi

If you see OperationNotSupported, you are paying delegate overhead for zero NPU benefit.

Step 2: Understand the Accelerator Stack

The NNAPI stack on Snapdragon routes ops through the Hexagon DSP driver when available, falling back to GPU (via OpenCL or Vulkan compute), then CPU. You do not control this — the HAL implementation decides.

Accelerator	Typical latency (prefill)	Power draw	Quantization support
Hexagon NPU	Lowest	Lowest	INT8, INT4 (varies by SoC)
Adreno GPU	Moderate	Moderate	FP16, INT8
CPU (NEON)	Baseline	Highest	All formats

One hard constraint: INT4 (Q4_K) NPU support is SoC-generation dependent. On Snapdragon 8 Gen 1, Q4_K ops partition to CPU. On Gen 3 and later, Hexagon v75 includes native INT4 support. For broad NPU coverage today, Q8_0 is the safer target.

Step 3: Eliminate JNI Overhead with AHardwareBuffer

Here is the gotcha that will save you hours: every ByteBuffer.array() call on a tensor crosses the JNI boundary and forces a heap allocation. Per-token JNI crossings compound with sequence length.

The fix is to allocate input tensor buffers using AHardwareBuffer in native code — a single memory region accessible by CPU, GPU, and DSP simultaneously.

AHardwareBuffer_Desc desc = {
    .width  = kEmbeddingDim,
    .height = kMaxSeqLen,
    .layers = 1,
    .format = AHARDWAREBUFFER_FORMAT_BLOB,
    .usage  = AHARDWAREBUFFER_USAGE_CPU_WRITE_OFTEN |
              AHARDWAREBUFFER_USAGE_GPU_DATA_BUFFER |
              AHARDWAREBUFFER_USAGE_DSP_COMPUTE
};
AHardwareBuffer* buffer = nullptr;
AHardwareBuffer_allocate(&desc, &buffer);

void* data = nullptr;
AHardwareBuffer_lock(buffer, AHARDWAREBUFFER_USAGE_CPU_WRITE_OFTEN,
                     -1, nullptr, &data);
memcpy(data, token_ids, token_count * sizeof(int32_t));
AHardwareBuffer_unlock(buffer, nullptr);

On the Kotlin side, wrap this as a HardwareBuffer and hand it directly to the NNAPI delegate input. Zero copy, zero JNI per-token cost.

Step 4: Split Prefill and Decode Across Accelerators

llama.cpp's Android backend does not natively expose an NNAPI path yet. The practical approach is a TFLite shim: export your GGUF model's attention layers as TFLite flatbuffers, run them through the NNAPI delegate for prefill, and use llama.cpp's native NEON path for autoregressive decode.

The docs do not mention this, but the reasoning is straightforward: prefill is a large matrix multiply — ideal for NPU batch throughput. Decode is memory-bound and sequential; NPU scheduling overhead dominates there and does not pay off.

Gotchas

Silent CPU fallback is the most common hidden regression. Wire adb logcat | grep nnapi into your CI pipeline against your target SoC. This catches regressions before they ship.

Q4_K_M on older Snapdragons will silently fall back. Verify quantization support per SoC generation before picking your model format. Q8_0 is your safe default for broad NPU coverage.

Forgetting DSP_COMPUTE in the AHardwareBuffer usage flags means your buffer cannot be accessed by the Hexagon driver. Allocation succeeds but the delegate falls back to a copied buffer.

Conclusion

Three things to lock in before shipping any on-device LLM on Android: verify NNAPI delegation is actually routing to the NPU, allocate embedding and KV-cache buffers natively with AHardwareBuffer and the right usage flags, and split prefill through NNAPI while routing decode through llama.cpp's NEON path. Each phase belongs on the accelerator it actually benefits from.

The stack is fiddly but the latency and power gains are real once you get each layer right.

Resources:

DEV Community