Building a 10.2x Faster Search-Only Retrieval Path on Arm64

NeonRecall's best accepted native Arm64 workload showed a 10.214x three-run median throughput speedup over its FP32 retrieval baseline.

That number needs a clear boundary. This is a single-threaded, search-only benchmark over already-produced embeddings. It does not include model inference, embedding generation, index construction, or corpus and query quantization.

Within that boundary, the result was consistent across three vector widths:

Workload	Per-run speedups	Three-run median	INT8 / FP32 payload
384 dimensions	9.349x, 10.465x, 10.214x	10.214x	25.26%
768 dimensions	9.311x, 9.921x, 9.354x	9.354x	25.13%
1536 dimensions	4.988x, 9.243x, 7.517x	7.517x	25.07%

Every optimized run was faster and had lower p95 search latency. The benchmark ran on native Arm64 Linux and selected the aarch64-neon-dotprod kernel.

The problem I wanted to isolate

A semantic-retrieval service usually has at least two distinct stages:

1. A model turns text into embeddings.
2. A search layer compares a query embedding with stored document embeddings.

It is easy to mix the two and report a speedup that is hard to interpret. NeonRecall deliberately isolates the second stage.

Both benchmark paths receive the same deterministic vectors and use the same top-k ranking implementation. The difference is the representation and dot-product arithmetic:

• The baseline stores and searches FP32 vectors.
• The optimized path stores symmetric per-vector INT8 vectors and uses an Arm-specific integer dot product.

This makes the comparison narrow, but it also makes the result easier to audit.

Per-vector INT8 instead of FP32

Each FP32 vector is quantized independently. NeonRecall finds the vector's largest absolute value and derives one scale:

scale = max(abs(vector)) / 127
q[i]  = clamp(round(vector[i] / scale), -127, 127)

The index stores one signed byte per dimension plus one FP32 scale per vector. During search, the integer dot product is rescaled before ranking:

const std::int64_t integer_score =
    dot_int8_selected_impl(selected.kind, query.values, vectors[i].values);

scores[i] = static_cast<float>(integer_score)
          * query.scale
          * vectors[i].scale;

For these workloads, that representation used 25.1% to 25.3% of the FP32 encoded-vector payload. This is a payload comparison, not a claim about total process memory or container overhead.

Selecting the right Arm kernel at runtime

NeonRecall provides three INT8 paths:

• A portable scalar implementation.
• An AArch64 NEON widening implementation.
• An AArch64 NEON dot-product implementation.

The binary checks the CPU at runtime and selects the strongest supported path:

if (!capabilities.is_aarch64 || !capabilities.neon) {
  return {KernelKind::ScalarInt8, "scalar-int8"};
}

if (capabilities.dotprod) {
  return {KernelKind::NeonDotProduct, "aarch64-neon-dotprod"};
}

return {KernelKind::NeonWidening, "aarch64-neon-widening"};

On a CPU with the dot-product extension, the hot loop processes 16 signed bytes at a time:

const int8x16_t left = vld1q_s8(lhs.data() + offset);
const int8x16_t right = vld1q_s8(rhs.data() + offset);
accumulator = vdotq_s32(accumulator, left, right);

The scalar path still matters. It gives the optimized representation a portable correctness reference, while runtime dispatch keeps the binary usable across different machines.

Measuring quality without overstating it

Quantization is only useful if retrieval remains acceptable, so the evidence pipeline includes a separate quality sanity check.

A revision-pinned qint8 MiniLM ONNX encoder produces one set of real embeddings. NeonRecall then compares FP32 retrieval arithmetic with INT8 retrieval arithmetic over those same embeddings.

Across three repetitions, both paths reached macro recall@10 of 1.0 on the included 18-document, six-query demo, with identical top-ten lists.

That is encouraging, but it is not a general retrieval-quality claim. It is a bounded regression check on a small authored dataset. It is also not a full-FP32 model versus INT8 model comparison; both retrieval paths use embeddings from the same pinned encoder.

The evidence pipeline is designed to reject bad runs

Performance work becomes much less convincing when the benchmark screenshot cannot be traced back to a binary, source commit, CPU, and exact input.

The accepted NeonRecall aggregate comes from three repetitions on a native ubuntu-24.04-arm GitHub Actions runner. Before accepting the result, the workflow:

• verifies the source commit and uploaded binary hashes;
• re-executes every uploaded binary on the aggregate runner;
• confirms ELF64 little-endian AArch64 identity;
• regenerates disassembly and requires the selected kernel's Arm instruction;
• locks deterministic input fingerprints and cross-run checksums;
• recomputes recall from raw ranked document IDs and repository qrels; and
• rejects scalar dispatch, performance regressions, excessive payload, or recall below the documented threshold.

The public aggregate records commit 904c8bd7120ef419a68019ed5ab92f43bf49d55c, workflow run 29539633599, the selected kernel, the compiler, the model revision, every individual speedup, and the quality results.

Reproducing the short path

On native Arm64 Linux, the basic correctness and dispatch path is:

git clone https://github.com/LubuSeb/neonrecall-arm.git
cd neonrecall-arm
cmake -S . -B build -DCMAKE_BUILD_TYPE=Release -DBUILD_TESTING=ON
cmake --build build --parallel
ctest --test-dir build --output-on-failure
python3 -B -m unittest discover -s scripts/tests -v
./build/neonrecall selftest --json
./build/neonrecall search-demo

The repository documents the longer native evidence route, including the pinned model assets, three benchmark repetitions, raw retrieval results, ELF checks, disassembly, and aggregation gates.

Where this technique fits

This design is relevant when retrieval arithmetic is a meaningful part of a CPU-bound RAG, recommendation, or vector-search workload and the embeddings already exist.

It does not answer whether end-to-end application latency will improve by the same factor. That depends on model inference, data movement, index structure, concurrency, filtering, and the surrounding service. A useful next step would be testing larger corpora and a production index while measuring end-to-end latency and resident memory separately.

For this experiment, the narrower result is the useful one: replacing FP32 search arithmetic with a purpose-built INT8 representation and an Arm NEON dot-product path produced a substantial, reproducible speedup on native Arm64.

Project links

• Source code and reproduction guide
• Accepted native Arm64 workflow
• Machine-readable aggregate
• 1:24 silent, captioned demo
• Arm Create: AI Optimization Challenge submission

NeonRecall is Apache-2.0 licensed and was built for Track 2: Cloud AI in the Arm Create: AI Optimization Challenge.