TL;DR: tool-calling is now a workload a 45M-parameter model can carry. Needle 2 (Apache-2.0) ships as a single 14MB binary that runs a full session in about 28MB of RAM, targeting phones, Raspberry Pi, and MCUs. I have not tested or run it; this is a review of the official README, model card, product page, and architecture paper.
The numbers check out arithmetically. 45M params at CQ2 (~2 bits/weight, quantized during pretraining rather than post-hoc) is 45e6 × 2 / 8 = 11.25 MB — the bulk of the 14MB binary, consistent with "weights never decompress into RAM." A 256-token sliding KV window with tools pinned as KV sinks keeps session memory near 28MB regardless of conversation length. Official speed claims: ~500 tok/s decode on Raspberry Pi 5, 400–1,500 tok/s on VR headsets, 300–700 tok/s on sub-$200 phones, ~70 MFLOPs/token vs ~460 for LFM2.5 and ~6,000 for Apple FM.
The design is a bet on context, not weights. The FFN is replaced by a fixed orthonormal Walsh–Hadamard transform (n log n, no weights to read); memory comes from hashed n-gram "engram" tables; routing is Sinkhorn-normalized. The README cites arXiv:2607.18363, a controlled study where attention-only transformers match standard transformers at matched parameter count (0.006 nats gap) but are better at context-grounded answers and worse where knowledge must live in weights — which fits a tool-calling model and misfits a chatbot (17.0% on DroidCall).
Behavior contract. Text in, JSON out: a byte-level grammar compiled from your declared schemas constrains every token, so calls cannot be malformed. Requests no declared tool can serve return the empty call []; there is no free-text fallback. Arguments contain only values evidenced by the input — optional fields are omitted, not guessed. Confidence is the min of a calibrated head and the call's decoding probability; below threshold it escalates instead of executing. Above five tools, a retrieval head embeds schemas once and only the top-5 enter context — unselected tools are unreachable, not merely unlikely.
How to read the benchmarks. Mobile Actions: 63.7% (LFM2.5 69.1%, FunctionGemma 64.0%, Apple FM 57.6%); BFCL v4 single-turn 42.6% overall with 93.4% well-formed; Seal-Tools 32.6% in-domain / 28.7% out-of-domain. Note the asymmetry: Needle runs at 2 bits with a 256-token window while baselines run f16 with full context, and all figures are vendor-reported with no third-party reproduction yet.
Who should look. Edge and embedded engineers building tool callers. Schemas compile into the decode grammar; LoRA fine-tuning on the frozen base merges at export into one .cact file that runs on the same engine — no recompilation. Skip it if you need a general assistant: training data is proprietary (115B + 38B tokens), there is no free-text fallback, and the 256-token window is a hard budget. For context, Pebble already runs it locally in the Index 01 watch app.
Project: https://github.com/cactus-compute/needle · README: https://github.com/cactus-compute/needle/blob/main/README.md · Model card: https://huggingface.co/Cactus-Compute/needle2 · Paper: https://arxiv.org/abs/2607.18363 · Product page: https://cactuscompute.com/needle
Top comments (0)