DEV Community

Huge
Huge

Posted on Originally published at huge.github.io

The ARMv9 Speed Paradox: Auditing Real-World Code with Gemma 4 on Bare Metal

#devchallenge #hf26challenge #gemma #opensource

Hacktoberfest Open-Source AI Challenge Week 3: Slow and Steady Submission 🦥

This article is submitted for the DEV Hacktoberfest 2026 Open-Source AI Challenge: Week 3 ("Slow and Steady"), competing in the **Best Use of Gemma* category.*

Running automated security audits on real-world codebases with frontier LLMs usually comes with two painful tradeoffs: cloud API bills and headaches or stripped-down context windows that miss cross-file vulnerabilities.

For Week 3 of Hacktoberfest 2026, we took the "Slow and Steady" ethos literally: what happens when you run Google's Gemma 4 family on bare-metal ARMv9 silicon with a massive 65,536-token context window, evaluating real open-source targets against known ground-truth vulnerabilities?

We deployed a headless test runner across Google Cloud ARM64 instances (c4a-standard-8, 8 vCPUs on Neoverse-V2 silicon, 32 GB RAM) and pitted multiple architectures and quantization schemes against each other:

  1. unsloth/gemma-4-12b-it-GGUF (Dense 12B across 4 quantization levels: UD-Q2_K_XL, IQ4_XS, UD-Q4_K_XL, and UD-Q6_K_XL)
  2. bottlecapai/ThinkingCap-gemma-4-26B-A4B-it-GGUF (Mixture-of-Experts with 26B total parameters and 4B active: Q4_K_M and Q6_K)
  3. prism-ml/Ternary-Bonsai-2-27B-gguf (1.58-bit ternary quantization: PTQ1_0 executed via custom PrismML-Eng/llama.cpp kernels)

Here is what we discovered about ARMv9 SIMD execution, context scaling, and vulnerability detection precision.

The Test Targets: Verified Ground Truth

Rather than synthetic benchmarks or toy examples, we scanned two real-world Go projects where our team previously discovered and remediated zero-day security flaws:

  1. jhaals/yopass checked out at commit 16bc350a (pre-fix commit before the merged patch in PR #3973).

  2. rs/curlie checked out at commit 5dfcbb17 (pre-fix commit before patches PR #104 and PR #105).

The Full Benchmark Dataset

Across dozens of hours of headless execution on Neoverse-V2 cores, here are the complete measurements:

Model & Quantization Size (GB) Prompt Eval Token Gen yopass (Time / Triage) curlie (Time / Triage)
Gemma-4-12B UD-Q2_K_XL 4.34 GB 8.30 t/s 8.38 t/s 4,040 s (0 valid / 0 rejected) 8,720 s (2 valid / 0 rejected)
Gemma-4-12B IQ4_XS 5.94 GB 10.83 t/s 11.16 t/s 9,260 s (1 valid / 3 rejected) 18,260 s (6 valid / 7 rejected)
Gemma-4-12B UD-Q4_K_XL 6.86 GB 15.12 t/s 11.08 t/s 7,375 s (1 valid / 3 rejected) 12,715 s (4 valid / 6 rejected)
Gemma-4-12B UD-Q6_K_XL 9.95 GB 14.92 t/s 8.06 t/s 8,730 s (1 valid / 3 rejected) 19,739 s (2 valid / 11 rejected)
ThinkingCap-26B-A4B Q4_K_M 15.84 GB 55.24 t/s 21.78 t/s 3,170 s (0 valid / 5 rejected) 6,404 s (2 valid / 12 rejected)
ThinkingCap-26B-A4B Q6_K 21.70 GB 54.37 t/s 21.04 t/s 3,213 s (1 valid / 4 rejected) OOM crash at 2,500 s (32 GB RAM ceiling)
Ternary-Bonsai-27B PTQ1_0 5.54 GB 6.85 t/s 3.68 t/s 22,970 s (1 valid / 4 rejected) 27,090 s (1 valid / 3 rejected)
Gemma-4-26B-A4B (Vertex MaaS, Cloud) Cloud API ~650 t/s 55.10 t/s 62 s (0 valid / 4 rejected) 90 s (4 valid / 9 rejected)

Bold values indicate best performance among local bare-metal CPU runs. Managed cloud API is included as an external reference baseline.

Four Engineering Insights from Bare-Metal Execution

1. The ARMv9 Hardware Paradox: 4-bit Beats 2-bit

Conventional intuition assumes smaller quants run faster because of reduced memory traffic. On 8 DDR5-backed Neoverse-V2 cores, the opposite occurred:

  • UD-Q4_K_XL (6.86 GB) achieved 15.12 tokens/second on prompt evaluation—nearly double the throughput of UD-Q2_K_XL (8.30 tokens/second), despite reading 58% more weight data per token!
  • Neoverse-V2 features dual 128-bit vector pipelines with native ARMv9 i8mm and dotprod instructions. Byte-aligned 4-bit and 6-bit blocks unpack with minimal instruction overhead. By contrast, non-linear 2-bit packing (Q2_K) and variable-rate lookup grids (IQ4_XS) force the CPU into heavy bit-shifting, masking, and table indirection on scalar/integer ALUs. On modern CPU architectures, SIMD unpacking overhead easily dwarfs memory bus contention.

2. The Mixture-of-Experts Supremacy on CPU

The sparse ThinkingCap-Gemma-4-26B-A4B architecture routed every token to only 4 billion active parameters out of 26 billion total. On bare CPU, it sustained over 55 tokens/second during prompt ingestion and 21.8 tokens/second during generation. It completed the full repository audits in half the wall-clock time of 12B dense models, while retaining the reasoning breadth of a 26B parameter model.

3. The 1.58-Bit Quantization Floor and Triage Immunity

At 1.58 bits per parameter, Ternary-Bonsai-2-27B fits 27 billion parameters into 5.54 GB of RAM. However, during deep 65k context code analysis on CPU:

  • It generated at 3.68 tokens/second via custom PrismML bit-emulation routines.
  • More importantly, extreme quantization degraded cross-domain semantic coherence: when analyzing Go source code (oidc.go), it hallucinated C functions (void parse_packet with memcpy) from its pre-training data.
  • The system immune response: Our multi-round skeptical triage layer immediately caught the hallucination. The verifier grepped the target codebase, confirmed no such C construct existed, and rejected the finding with a confidence of 0% ([III→I]). A disciplined multi-round pipeline successfully neutralized model hallucinations even under severe quantization breakdown.

4. 65k Context Sizing on 32 GB RAM

When deploying a 65,536-token context window with 8-bit KV caching (-ctk q8_0 -ctv q8_0), the KV cache expands by 8 to 10 GB under deep reasoning chains:

  • Models up to 16 GB (12B dense and 26B-A4B-Q4_K_M) operated with comfortable safety margins on a 32 GB RAM instance.
  • The 21.70 GB 26B-A4B-Q6_K model operated smoothly during single-turn evaluations, but hit the Linux kernel OOM limit during recursive multi-turn triage on larger files. For 65k contexts on 26B+ models, 48 GB to 64 GB of RAM is the recommended hardware baseline.

Taming the 65,536 Context Window in llama-server

Feeding entire repository components into local LLMs requires specific inference server tuning:

nice -n 10 llama-server \
  -m ./models/ThinkingCap-gemma-4-26B-A4B-it-Q4_K_M.gguf \
  -c 65536 \
  -n 4096 \
  -np 1 \
  -fa on \
  -ctk q8_0 -ctv q8_0 \
  -t 8 \
  --reasoning-format deepseek \
  --reasoning-budget 1536 \
  --host 127.0.0.1 \
  --port 8080
Enter fullscreen mode Exit fullscreen mode

Three configuration flags proved essential:

  • -np 1: By default, llama-server divides -c 65536 across multiple concurrent slots (e.g. 4 slots of 16k tokens). Passing -np 1 guarantees the full 64k window belongs to the security pipeline.
  • -fa on -ctk q8_0 -ctv q8_0: Flash Attention combined with 8-bit quantized KV cache reduced context memory overhead by 48%, keeping RAM usage predictable even past 45,000 prompt tokens.
  • --reasoning-format deepseek --reasoning-budget 1536: Gemma 4 outputs structured thought tokens (<|thought|>). Separating thought tokens from the final JSON payload prevented grammar parsing errors when enforcing strict JSON schemas.

Two Pipeline Fixes for Local 65k Models

Transitioning our scanner pipeline (AISLE nano-analyzer) to local 65k inference revealed two practical bugs:

1. Greedy Markdown Code Block Parsing

The original parser used a greedy regex to locate JSON payloads:

# Buggy: matches the first code fence, which may be target code inside thought reasoning
match = re.search(r'```

(?:json)?\s*\n?(.*?)

```', text, re.DOTALL)
Enter fullscreen mode Exit fullscreen mode

When Gemma 4 analyzed code, its internal reasoning frequently quoted snippets of Go source wrapped in

```go

fences. The parser matched the first snippet, failed JSON deserialization, and dropped the audit finding.

We resolved this by iterating backward through all fenced blocks to find the true payload:

# Fixed: inspect blocks from bottom to top to identify the valid JSON payload
for block in reversed(re.findall(r'```

(?:json)?\s*\n?(.*?)

```', text, re.DOTALL)):
    try:
        return json.loads(block.strip())
    except json.JSONDecodeError:
        continue
Enter fullscreen mode Exit fullscreen mode

2. Runaway Generation Loops

Without an explicit max_tokens cap in client payloads and server flags (-n 4096), repetition loops after exhausting the reasoning budget can cause the model to generate tokens until filling the 65,536-token ceiling. Enforcing explicit caps prevented multi-hour hangs on edge-case files.

Compute Economics: Bare-Metal CPU vs. Cloud APIs

Benchmarking deep context windows often raises concerns about cloud infrastructure costs. For this study on Google Cloud Platform, our hardware setup consisted of c4a-standard-8 instances (8 vCPUs on ARM Neoverse-V2 silicon, 32 GB RAM) billed at $0.282 / hour with standard hyperdisk storage.

The total compute cost across all experimental worker instances:

  • gemma-arm-26b (ThinkingCap-26B-A4B MoE on yopass and curlie across Q4_K_M and Q6_K): ~5.5 hours runtime = $1.55
  • gemma-arm-c4a (Gemma-4-12B dense across four quantization levels): ~13.5 hours runtime = $3.80
  • gemma-arm-bonsai (Ternary-Bonsai-27B 1.58-bit execution): ~15.0 hours runtime = $4.23

Running the full local benchmark across all seven model configurations cost under $10.00 in total GCP compute expenses.

For comparison, running the same audit against Google Cloud's managed Vertex AI Model Garden (google/gemma-4-26b-a4b-it-maas at $0.15/M input and $0.60/M output) completed both repository audits in 2.5 minutes with a combined API cost of $0.06. By contrast, running the same 65k multi-round skeptical triage through commercial frontier closed-source APIs regularly reaches $25 to $60 per repository.

For teams working within restricted networks or private VPCs where source code cannot leave the boundary, running sparse MoE models on self-hosted ARMv9 CPU hardware offers high-depth context inspection for pennies per run.

Key Takeaways for Local Security Auditing

  1. Do not default to 2-bit quants on modern CPUs: On ARMv9 Neoverse cores with DDR5 memory, integer SIMD execution makes standard 4-bit quants faster and substantially more capable than 2-bit variants.
  2. MoE architectures excel on CPU: Sparse routing provided the best throughput-to-parameter ratio, sustaining over 21 tokens/second generation and 55 tokens/second prompt ingestion.
  3. Multi-round skeptical triage is mandatory: Candidate scans produce false alarms; pairing discovery with an automated skeptical review round filters out benign patterns and neutralizes model hallucinations.

The test runner, inference configurations, and reproduction instructions are available in the open-source repository at Huge/nano-analyzer. For the interactive web report and case studies, visit BudgetScan Stories.

Top comments (1)

Collapse
 
naitik0009 profile image
naitik0009 •

Amazing