Is Your Phone's LLM 37x Too Slow? Check This One Compiler Flag

Part 3 of a series.

Part 1: found the missing -march (2.5x faster).

Part 2: ruled out scheduling and frequency scaling (negative results).

Part 3: found the real culprit — -O0.

---

1. The result first

If you build llama.cpp for Android with a Gradle debug variant, you get an -O0 binary. It is 37x slower than Release.

Metric	Debug (-O0)	Release (-O3)	Improvement
Generation	0.5 tok/s	18.5 tok/s	37x
First token	1863 ms	52 ms	36x
Prefill, 43 tokens	42 s	0.556 s	75x
Model load	24 s	3.9 s	6x

Device: Pixel 4 (Snapdragon 855, 2019). Model: Qwen2.5-1.5B-Instruct Q4_K_M.
Nothing changed except two build config lines.

---

2. How I missed something this basic

What I had already checked (parts 1 and 2):

Check	Verdict
Missing -march	✅ Real, fixed — 2.5x on prefill
Threads on little cores	❌ Ruled out
OpenMP not threading	❌ Ruled out
mmap pages evicted	❌ Ruled out
CPU frequency throttling	⚠️ Real, but only ~20%
Optimisation level	🔴 Never checked

I spent days on how the CPU was running, and never checked whether the compiled code was optimised at all.

Worse: I had disassembled the binary and confirmed sdot (the ARM dotprod instruction) was present, and concluded "the build is fine".

sdot only tells you the instruction set. It says nothing about the optimisation level. At -O0 the compiler still emits sdot — it just does nothing else.

---

3. What turned it around

A reader on DEV left this comment:

Build type. A Gradle debug variant hands CMake CMAKE_BUILD_TYPE=Debug unless you override it, and ggml at -O0 with asserts on is 20-50x slower than release.

sdot showing up in the disassembly does not tell you the optimisation level.

Add -DCMAKE_BUILD_TYPE=Release to that arguments list, or benchmark the release variant.

I would bet the build type.

He called it exactly.

---

4. Verifying takes one command

Look at what the build system actually passes to the compiler:

find app/.cxx -name build.ninja

grep -o '\-O[0-3s]' build.ninja | sort -u

My result:

(no output)

Not a single -O flag.

And:

CMAKE_BUILD_TYPE:STRING=Debug

So ggml, llama.cpp and my own JNI layer were all compiled with zero optimisation.

---

5. The fix

Change 1: force optimisation in CMakeLists.txt

This must come before add_subdirectory(llama.cpp), otherwise it won't affect ggml.

set(CMAKE_C_FLAGS_DEBUG   "-O3 -g" CACHE STRING "" FORCE)
set(CMAKE_CXX_FLAGS_DEBUG "-O3 -g" CACHE STRING "" FORCE)

set(CMAKE_C_FLAGS_RELWITHDEBINFO   "-O3 -g -DNDEBUG" CACHE STRING "" FORCE)
set(CMAKE_CXX_FLAGS_RELWITHDEBINFO "-O3 -g -DNDEBUG" CACHE STRING "" FORCE)

Change 2: tell Gradle to use Release

externalNativeBuild {
    cmake {
        arguments += listOf(
            "-DUSE_LLAMA_CPP=ON",
            "-DGGML_CPU_ARM_ARCH=armv8.2-a+dotprod+fp16",
            "-DCMAKE_BUILD_TYPE=Release"    // key
        )
    }
}

Both, because AGP can override what you pass via Gradle, while FORCE in CMakeLists wins.

---

6. Verify the fix actually landed

Don't trust the flag you passed. Check what happened.

Check 1: build.ninja

grep -c '\-O3' build.ninja
grep '\-DNDEBUG' build.ninja

Mine:

-O3 appears 82 times
-DNDEBUG present

Check 2: binary size (the fastest smell test)

Library	Debug (-O0)	Release (-O3)
libggml-cpu.so	3.92 MB	0.95 MB
libllama.so	7427 KB	3404 KB

A large drop in size means optimisation actually happened — inlining, dead code elimination and constant folding all shrink the binary.

This is my favourite quick check: you don't need to disassemble anything, just look at the file size.

---

7. Why it's this dramatic

What -O0 turns off:

Optimisation	-O0	-O3
Function inlining	❌	✅ on hot paths
Loop unrolling	❌	✅
Auto-vectorisation	❌	✅
Register allocation	conservative	aggressive
Dead code elimination	❌	✅
Asserts	kept	removed (-DNDEBUG)

For LLM inference the killers are auto-vectorisation and inlining.

Quantised matrix multiplication — 99% of the compute — is made of tight loops. At -O0 those loops are not vectorised, so every element is computed and fetched individually. Tens of times slower is expected.

That explains every anomaly we saw:

CPU at full clock ✅
threads on big cores ✅
dotprod instructions present ✅
utilization: 1% ❌

The CPU was running full speed on unoptimised element-by-element code.

---

8. How to avoid this

If you're building llama.cpp for Android

# after building, check the optimisation flags
find app/.cxx -name build.ninja -exec grep -c '\-O3' {} \;

# or just look at the binary size
ls -lh app/build/intermediates/cxx/*/obj/arm64-v8a/*.so

libggml-cpu.so around 1 MB → optimised. Around 3-4 MB → probably -O0.

A general debugging order

1. Optimisation level (-O0/-O1/-O2/-O3)     <- start here
2. Architecture flags (-march / -mcpu)
3. Thread count and affinity
4. Memory and IO
5. The algorithm itself

I worked through step 5 before coming back to step 1.

---

9. Lessons

1. The more basic something is, the easier it is to skip.

We checked CPU architecture, thread scheduling, frequency, bandwidth — and never checked whether the code was optimised. Because "basic things don't look like the problem".

2. "The instruction exists" ≠ "the build is right".

Instruction set and optimisation level are different axes.

3. Binary size is an underrated signal.

A big size change is the simplest evidence that optimisation happened, and it's much faster than reading disassembly.

4. One good comment can beat days of solo debugging.

That reader spotted it in a sentence. And he could only do that because I published the negative results — if I'd only written about the successes, he wouldn't have known where I was stuck.

---

10. Where it stands now

18.5 tok/s, fully offline, on a 2019 phone.

• A 20-character sentence finishes in about a second
• First token in 52 ms — essentially no perceptible delay
• 1.3 GB RSS

For a five-year-old phone, that's genuinely usable.

---

Part 1: Why Your Phone Runs LLMs 80x Slower Than It Should

Part 2: Threads on Big Cores, CPU at Full Clock — Still Slow

Part 3: this post
Code: https://github.com/Pingredsai/local-ai-android