I Got 9.9 Lower TTFT on a Real Android Phone by Reusing llama.cpp KV State

bossandboss — Sat, 11 Jul 2026 21:17:57 +0000

Local LLM inference has an expensive habit:
It recomputes prefixes it has already seen.
A system prompt.
A reused RAG document.
A few-shot block.
A long static context.
If the prefix is identical, why pay the prefill cost again?
That's the problem I explored with EdgeSync-LLM.
The idea
The mechanism is simple:
Prompt = shared prefix + new suffix
On the first request, EdgeSync prefills the prefix and captures its KV cache state.
On the next request sharing that exact prefix, it restores the state and decodes only the new suffix.
No llama.cpp fork.
No patch.
The current validated path uses the public:
llama_state_seq_get_data
and
llama_state_seq_set_data
APIs.
Measured on a real Android ARM64 phone
Model:
Qwen2.5-0.5B-Instruct Q4_K_M
Shared prefix:
123 tokens
40 requests. 4 threads. Release build.
Path
Mean TTFT
p50
p95
Cold
4828 ms
4752 ms
5297 ms
KV state reuse
486 ms
476 ms
569 ms
9.9× lower TTFT on cache hits.
The warm path was approximately:
363 ms to decode the 10-token suffix
123 ms to restore the state blob
Fragment size: 1.64 MB
I also measured the same mechanism on x86-64.
Cold mean TTFT: 1395 ms
Warm mean TTFT: 185 ms
That's 7.5× on cache hits.
But I almost published a fake 8.8× speedup
This was the most important part of the project.
My first implementation directly copied raw K/V tensors.
It was fast.
Very fast.
The benchmark reported an 8.8× speedup.
There was one problem.
It was wrong.
llama.cpp tracks more than the K/V tensor values. Cache cells also have position and sequence metadata used to construct the attention mask.
Copying tensor values without restoring that bookkeeping produced an inert fragment.
The model skipped prefix computation...
...but attention could not actually see the restored prefix.
14 of 24 cache hits reproduced, token for token, the output of a generation with no prefix at all.
The “speedup” was dropped context.
So I discarded it.
Timing is not enough
A broken cache can be fast.
That's why EdgeSync now runs two correctness checks.
First:
Warm output must match cold output for the same prompt.
Results:
ARM64: 28/28
x86-64: 24/24
Second:
The warm output is compared with a generation from the suffix alone, without the prefix.
If both outputs match, the injected fragment may be doing nothing.
Results:
ARM64: 0/28 matches
x86-64: 0/24 matches
This second control is what caught the broken raw-tensor implementation.
A latency benchmark alone would not have caught it.
What I learned
There is a broader lesson here for LLM infrastructure benchmarking:
Skipping computation is not proof that state reuse works.
If a cache implementation silently drops context, it can look extremely fast.
Correctness must be part of the benchmark.
Not a separate test.
Not an assumption.
Part of the measurement itself.
What EdgeSync does not claim
I deliberately marked several parts of the project as unvalidated or unsound.
Semantic approximate prefix reuse is not proven.
Reusing KV state between merely “similar” prefixes is incorrect with the current mechanism.
Cross-engine reuse does not work with opaque llama.cpp sequence-state blobs.
Per-layer striding cannot preserve output.
The Android JNI bridge and several cache management components still need end-to-end validation.
The on-device ARM64 sampling is also thin and needs to be rerun with more timed repeats.
I would rather publish a smaller result that can be falsified than a larger claim that cannot survive reproduction.
What's next
The next steps are:
Rerun ARM64 with multiple timed repeats.
Compare complete generated strings instead of one token.
Benchmark HNSW against an exact radix-tree lookup.
Measure state restoration cost as prefix length increases.
Remove or redefine the current PARTIAL-hit mechanism.
The code, benchmark methodology and measured manifests are public.
EdgeSync-LLM:
https://github.com/bossandboss/EdgeSync-LLM
If you work on llama.cpp, local LLM inference, KV cache management or edge AI, I would especially value attempts to reproduce or break the benchmark.
Finding a flaw is more useful to me than a GitHub star.

DEV Community: bossandboss

I Got 9.9 Lower TTFT on a Real Android Phone by Reusing llama.cpp KV State

llm #cpp #android #ai