The ARMv9 Speed Paradox: Auditing Real-World Code with Gemma 4 on Bare Metal

Huge — Sun, 25 Oct 2026 09:45:03 +0000

This article is submitted for the DEV Hacktoberfest 2026 Open-Source AI Challenge: Week 3 ("Slow and Steady"), competing in the **Best Use of Gemma* category.*

Running automated security audits on real-world codebases with frontier LLMs usually comes with two painful tradeoffs: cloud API bills and headaches or stripped-down context windows that miss cross-file vulnerabilities.

For Week 3 of Hacktoberfest 2026, we took the "Slow and Steady" ethos literally: what happens when you run Google's Gemma 4 family on bare-metal ARMv9 silicon with a massive 65,536-token context window, evaluating real open-source targets against known ground-truth vulnerabilities?

We deployed a headless test runner across Google Cloud ARM64 instances (c4a-standard-8, 8 vCPUs on Neoverse-V2 silicon, 32 GB RAM) and pitted multiple architectures and quantization schemes against each other:

unsloth/gemma-4-12b-it-GGUF (Dense 12B across 4 quantization levels: UD-Q2_K_XL, IQ4_XS, UD-Q4_K_XL, and UD-Q6_K_XL)
bottlecapai/ThinkingCap-gemma-4-26B-A4B-it-GGUF (Mixture-of-Experts with 26B total parameters and 4B active: Q4_K_M and Q6_K)
prism-ml/Ternary-Bonsai-2-27B-gguf (1.58-bit ternary quantization: PTQ1_0 executed via custom PrismML-Eng/llama.cpp kernels)

Here is what we discovered about ARMv9 SIMD execution, context scaling, and vulnerability detection precision.

The Test Targets: Verified Ground Truth

Rather than synthetic benchmarks or toy examples, we scanned two real-world Go projects where our team previously discovered and remediated zero-day security flaws:

jhaals/yopass checked out at commit 16bc350a (pre-fix commit before the merged patch in PR #3973).
- Scanned files: pkg/server/oidc.go, pkg/server/apitoken.go, and pkg/server/middleware.go (646 lines, 22,181 characters).
- Ground truth flaw: Missing validation of the email_verified claim in the OIDC userinfo handler, allowing unverified identities to bypass access controls.
rs/curlie checked out at commit 5dfcbb17 (pre-fix commit before patches PR #104 and PR #105).
- Scanned files: main.go, args/httpie.go, args/parse.go, formatter/json.go, formatter/color.go, and formatter/binaryfilter.go (723 lines, 15,312 characters).
- Ground truth flaws: Arbitrary local file read in args/httpie.go via HTTPie-compatible @filename multipart form syntax, and terminal ANSI escape sequence injection in formatter/json.go.

The Full Benchmark Dataset

Across dozens of hours of headless execution on Neoverse-V2 cores, here are the complete measurements:

Model & Quantization	Size (GB)	Prompt Eval	Token Gen	yopass (Time / Triage)	curlie (Time / Triage)
Gemma-4-12B `UD-Q2_K_XL`	4.34 GB	8.30 t/s	8.38 t/s	4,040 s (0 valid / 0 rejected)	8,720 s (2 valid / 0 rejected)
Gemma-4-12B `IQ4_XS`	5.94 GB	10.83 t/s	11.16 t/s	9,260 s (1 valid / 3 rejected)	18,260 s (6 valid / 7 rejected)
Gemma-4-12B `UD-Q4_K_XL`	6.86 GB	15.12 t/s	11.08 t/s	7,375 s (1 valid / 3 rejected)	12,715 s (4 valid / 6 rejected)
Gemma-4-12B `UD-Q6_K_XL`	9.95 GB	14.92 t/s	8.06 t/s	8,730 s (1 valid / 3 rejected)	19,739 s (2 valid / 11 rejected)
ThinkingCap-26B-A4B `Q4_K_M`	15.84 GB	55.24 t/s	21.78 t/s	3,170 s (0 valid / 5 rejected)	6,404 s (2 valid / 12 rejected)
ThinkingCap-26B-A4B `Q6_K`	21.70 GB	54.37 t/s	21.04 t/s	3,213 s (1 valid / 4 rejected)	OOM crash at 2,500 s (32 GB RAM ceiling)
Ternary-Bonsai-27B `PTQ1_0`	5.54 GB	6.85 t/s	3.68 t/s	22,970 s (1 valid / 4 rejected)	27,090 s (1 valid / 3 rejected)
Gemma-4-26B-A4B (Vertex MaaS, Cloud)	Cloud API	~650 t/s	55.10 t/s	62 s (0 valid / 4 rejected)	90 s (4 valid / 9 rejected)

Bold values indicate best performance among local bare-metal CPU runs. Managed cloud API is included as an external reference baseline.

Four Engineering Insights from Bare-Metal Execution

1. The ARMv9 Hardware Paradox: 4-bit Beats 2-bit

Conventional intuition assumes smaller quants run faster because of reduced memory traffic. On 8 DDR5-backed Neoverse-V2 cores, the opposite occurred:

UD-Q4_K_XL (6.86 GB) achieved 15.12 tokens/second on prompt evaluation—nearly double the throughput of UD-Q2_K_XL (8.30 tokens/second), despite reading 58% more weight data per token!
Neoverse-V2 features dual 128-bit vector pipelines with native ARMv9 i8mm and dotprod instructions. Byte-aligned 4-bit and 6-bit blocks unpack with minimal instruction overhead. By contrast, non-linear 2-bit packing (Q2_K) and variable-rate lookup grids (IQ4_XS) force the CPU into heavy bit-shifting, masking, and table indirection on scalar/integer ALUs. On modern CPU architectures, SIMD unpacking overhead easily dwarfs memory bus contention.

2. The Mixture-of-Experts Supremacy on CPU

The sparse ThinkingCap-Gemma-4-26B-A4B architecture routed every token to only 4 billion active parameters out of 26 billion total. On bare CPU, it sustained over 55 tokens/second during prompt ingestion and 21.8 tokens/second during generation. It completed the full repository audits in half the wall-clock time of 12B dense models, while retaining the reasoning breadth of a 26B parameter model.

3. The 1.58-Bit Quantization Floor and Triage Immunity

At 1.58 bits per parameter, Ternary-Bonsai-2-27B fits 27 billion parameters into 5.54 GB of RAM. However, during deep 65k context code analysis on CPU:

It generated at 3.68 tokens/second via custom PrismML bit-emulation routines.
More importantly, extreme quantization degraded cross-domain semantic coherence: when analyzing Go source code (oidc.go), it hallucinated C functions (void parse_packet with memcpy) from its pre-training data.
The system immune response: Our multi-round skeptical triage layer immediately caught the hallucination. The verifier grepped the target codebase, confirmed no such C construct existed, and rejected the finding with a confidence of 0% ([III→I]). A disciplined multi-round pipeline successfully neutralized model hallucinations even under severe quantization breakdown.

4. 65k Context Sizing on 32 GB RAM

When deploying a 65,536-token context window with 8-bit KV caching (-ctk q8_0 -ctv q8_0), the KV cache expands by 8 to 10 GB under deep reasoning chains:

Models up to 16 GB (12B dense and 26B-A4B-Q4_K_M) operated with comfortable safety margins on a 32 GB RAM instance.
The 21.70 GB 26B-A4B-Q6_K model operated smoothly during single-turn evaluations, but hit the Linux kernel OOM limit during recursive multi-turn triage on larger files. For 65k contexts on 26B+ models, 48 GB to 64 GB of RAM is the recommended hardware baseline.

Taming the 65,536 Context Window in `llama-server`

Feeding entire repository components into local LLMs requires specific inference server tuning:

nice -n 10 llama-server \
  -m ./models/ThinkingCap-gemma-4-26B-A4B-it-Q4_K_M.gguf \
  -c 65536 \
  -n 4096 \
  -np 1 \
  -fa on \
  -ctk q8_0 -ctv q8_0 \
  -t 8 \
  --reasoning-format deepseek \
  --reasoning-budget 1536 \
  --host 127.0.0.1 \
  --port 8080

Three configuration flags proved essential:

-np 1: By default, llama-server divides -c 65536 across multiple concurrent slots (e.g. 4 slots of 16k tokens). Passing -np 1 guarantees the full 64k window belongs to the security pipeline.
-fa on -ctk q8_0 -ctv q8_0: Flash Attention combined with 8-bit quantized KV cache reduced context memory overhead by 48%, keeping RAM usage predictable even past 45,000 prompt tokens.
--reasoning-format deepseek --reasoning-budget 1536: Gemma 4 outputs structured thought tokens (<|thought|>). Separating thought tokens from the final JSON payload prevented grammar parsing errors when enforcing strict JSON schemas.

Two Pipeline Fixes for Local 65k Models

Transitioning our scanner pipeline (AISLE nano-analyzer) to local 65k inference revealed two practical bugs:

1. Greedy Markdown Code Block Parsing

The original parser used a greedy regex to locate JSON payloads:

# Buggy: matches the first code fence, which may be target code inside thought reasoning
match = re.search(r'```

(?:json)?\s*\n?(.*?)

```', text, re.DOTALL)

When Gemma 4 analyzed code, its internal reasoning frequently quoted snippets of Go source wrapped in

```go

fences. The parser matched the first snippet, failed JSON deserialization, and dropped the audit finding.

We resolved this by iterating backward through all fenced blocks to find the true payload:

# Fixed: inspect blocks from bottom to top to identify the valid JSON payload
for block in reversed(re.findall(r'```

(?:json)?\s*\n?(.*?)

```', text, re.DOTALL)):
    try:
        return json.loads(block.strip())
    except json.JSONDecodeError:
        continue

2. Runaway Generation Loops

Without an explicit max_tokens cap in client payloads and server flags (-n 4096), repetition loops after exhausting the reasoning budget can cause the model to generate tokens until filling the 65,536-token ceiling. Enforcing explicit caps prevented multi-hour hangs on edge-case files.

Compute Economics: Bare-Metal CPU vs. Cloud APIs

Benchmarking deep context windows often raises concerns about cloud infrastructure costs. For this study on Google Cloud Platform, our hardware setup consisted of c4a-standard-8 instances (8 vCPUs on ARM Neoverse-V2 silicon, 32 GB RAM) billed at $0.282 / hour with standard hyperdisk storage.

The total compute cost across all experimental worker instances:

gemma-arm-26b (ThinkingCap-26B-A4B MoE on yopass and curlie across Q4_K_M and Q6_K): ~5.5 hours runtime = $1.55
gemma-arm-c4a (Gemma-4-12B dense across four quantization levels): ~13.5 hours runtime = $3.80
gemma-arm-bonsai (Ternary-Bonsai-27B 1.58-bit execution): ~15.0 hours runtime = $4.23

Running the full local benchmark across all seven model configurations cost under $10.00 in total GCP compute expenses.

For comparison, running the same audit against Google Cloud's managed Vertex AI Model Garden (google/gemma-4-26b-a4b-it-maas at $0.15/M input and $0.60/M output) completed both repository audits in 2.5 minutes with a combined API cost of $0.06. By contrast, running the same 65k multi-round skeptical triage through commercial frontier closed-source APIs regularly reaches $25 to $60 per repository.

For teams working within restricted networks or private VPCs where source code cannot leave the boundary, running sparse MoE models on self-hosted ARMv9 CPU hardware offers high-depth context inspection for pennies per run.

Key Takeaways for Local Security Auditing

Do not default to 2-bit quants on modern CPUs: On ARMv9 Neoverse cores with DDR5 memory, integer SIMD execution makes standard 4-bit quants faster and substantially more capable than 2-bit variants.
MoE architectures excel on CPU: Sparse routing provided the best throughput-to-parameter ratio, sustaining over 21 tokens/second generation and 55 tokens/second prompt ingestion.
Multi-round skeptical triage is mandatory: Candidate scans produce false alarms; pairing discovery with an automated skeptical review round filters out benign patterns and neutralizes model hallucinations.

The test runner, inference configurations, and reproduction instructions are available in the open-source repository at Huge/nano-analyzer. For the interactive web report and case studies, visit BudgetScan Stories.

DEV Community: Huge