A model that ships as a 14MB binary and runs a full agentic session in 28MB of RAM sounds like a marketing number engineered for a headline. Needle 2, the second release from Cactus Compute, is currently sitting near 6,800 GitHub stars and climbing, which means the headline worked. What's more interesting than the size claim is what the team actually did to hit it, and what that architecture quietly costs you once you read past the README.
Needle 2 is a 45-million-parameter model whose entire job is tool calling, device control, and structured data extraction — nothing else. No chat, no creative writing, no general Q&A. It's meant to sit inside a phone app, a wearable, a smart-home hub, or a robot, translate a spoken or typed request into a function call, and do it without a network round trip. That's a narrow mandate, and narrowing the mandate is exactly how they got the model this small. The rest of this piece is about the specific engineering choices behind that trade, what they buy you, and — more usefully — what the docs don't emphasize about what they cost.
What it actually does
Install it with pip install cactus-needle, decorate a Python function, and hand it to a Needle agent:
import needle
@needle.tool
def get_weather(city: str):
"Get the current weather for a city."
return {"city": city, "temp_c": 27, "sky": "clear"}
agent = needle.Needle(tools=[get_weather])
print(agent.run("what's it like in Lagos right now?")["results"])
# [{'city': 'Lagos', 'temp_c': 27, 'sky': 'clear'}]
run() closes the whole loop: the model reads the query, decides which function to call and with what arguments, Needle executes your Python function, feeds the result back, and returns the final answer. There's also extract(), which reframes structured extraction as tool calling with exactly one declared "tool" — the schema you want filled:
from pydantic import BaseModel
class Invoice(BaseModel):
vendor: str
total: float
due_date: str
invoice = needle.extract("Invoice from Acme Corp, $1,200.00, due 2026-09-01", Invoice)
The type hints and docstrings on your function are the whole interface — Needle reads the Args: block for per-parameter descriptions, treats a default value as "optional," and turns a Literal["heat", "cool", "auto"] into a closed set the model literally cannot emit outside of. Constraints attached with needle.Field (ranges, regex patterns, string lengths, item counts) get compiled straight into the decoding grammar, so a malformed or out-of-range argument isn't something you catch after the fact — the model can't produce it in the first place.
That "reads the schema, obeys the schema" contract is the whole product. It's a much smaller ambition than a general-purpose small LLM like Google's Gemma line or Meta's Llama family, and that's deliberate — Needle isn't competing with them on breadth, it's competing with them on what fits in an on-device tool-calling slot.
How it works: the architecture behind the number
The 14MB figure isn't just "a small transformer, quantized hard." The README calls the design a Simple Attention Network, and it makes several non-default choices that are worth unpacking because they explain both the size and the sharp edges.
Hadamard MLP instead of a feed-forward block. Standard transformer blocks spend a huge fraction of their weights on the FFN. Needle replaces it with a fixed, orthonormal Walsh-Hadamard transform — a matrix that requires zero learned weights and runs in O(n log n) time. You're trading a block that normally has to be trained and stored for one that's structurally fixed. That's one of the more unusual choices in a shipped model rather than a research paper, and it's a meaningful chunk of where the parameter count goes.
Grouped Query Attention (GQA). Nothing exotic here — the same attention-head-sharing trick used in Llama 2 and most modern small models to cut KV-cache memory.
Engram key-value memory. Instead of only attending over the live context, the model also pulls key/value rows from hashed n-gram tables — a lookup-style memory mechanism layered on top of attention. The README describes this as prioritizing "contextual grounding over knowledge stored in weights," which is a fair description of the trade: a 45M-parameter model has almost nowhere to store world knowledge, so it leans on retrieval-style lookups and the immediate context instead of memorized facts.
Multi-lane hyper-connections with sandwich normalization. Both the attention and MLP residual paths run through gated, sandwich-normed hyper-connections rather than a single residual stream, with routing logits passed through Sinkhorn iteration to keep them doubly stochastic. This is standard modern "how do we stabilize a very compressed network" plumbing — it doesn't add capability so much as it keeps a heavily quantized model trainable at all.
CQ2-bit quantization, baked in during pretraining. This is the detail that matters most for evaluating the benchmark claims later: Needle isn't a full-precision model quantized after the fact. "Cactus Quants" compress to roughly 2 bits per weight during training, which generally produces better results than post-hoc quantization of the same architecture — but it also means every number that follows is being generated by a fundamentally lower-precision model than its competitors.
Byte-level grammar-constrained decoding. Every tool schema is compiled into a grammar at the byte level, and decoding is constrained to that grammar the entire time. This is the mechanism behind "the model can't emit malformed JSON" — it's not a claim about training quality, it's a hard decode-time constraint, which is a much stronger guarantee.
A 256-token sliding window with tools pinned as KV sinks. This is the mechanism that keeps memory flat at ~28MB "no matter how long the conversation runs" — and it's also the sharpest limitation in the whole design, which I'll come back to.
The full derivation is published as a paper — arXiv:2607.18363 — with authorship credited to an eight-person team at Cactus Compute, a Y Combinator-backed company. This isn't a weekend side project dressed up with a README; there's an actual architecture behind the size claim, which is more than you can say for a lot of "tiny model" releases that are just aggressive pruning plus a good marketing paragraph.
Why not just quantize a bigger model?
The obvious question for anyone who's already run a small model on-device: why not skip the custom architecture and just run a 1B–3B model through llama.cpp or Ollama at 4-bit? That path works today, and plenty of teams already ship it. The difference is what you're optimizing for. A 4-bit quant of a 1–3B general model still lands in the 500MB–1.5GB range and pulls hundreds of megabytes of RAM at inference time — fine for a modern phone, a non-starter for a wearable, a smart speaker, or a microcontroller-adjacent embedded board. Needle's bet is that if your task is genuinely narrow (pick a function, fill its arguments), you don't need general-purpose language capability sitting idle in the weights; you need a model built from the ground up to be small, and you accept a hard mandate on what it can do in exchange. That's a real trade-off, not a strictly better one — a quantized general model, badly as it may run on a smartwatch, still has a free-text fallback and broader context. Needle has neither, by design.
How fine-tuning actually works
The adaptation story is more fleshed out than most tiny-model releases. Data synthesis, training, and export are three separate, documented steps rather than a research script you're left to interpret. needle generate-data calls out to OpenRouter (or any OpenAI-compatible gateway, via OPENROUTER_URL) to synthesize labeled examples from a tool schema file, or to expand an existing dataset — useful because hand-labeling "query → tool call" pairs at the volume small models need is tedious. needle finetune then runs LoRA on the frozen base checkpoint, with the usual knobs exposed (rank, alpha, learning rate, epochs, validation split), and prints held-out validation loss per epoch. The step that matters for deployment is needle build: it merges the LoRA adapter into the base weights and re-quantizes the result into a single .cact file that runs on the same inference engine as the base model — no separate runtime, no recompilation, no adapter-loading logic in your app. Training itself is plain JAX, so it runs on CPU, NVIDIA CUDA (cactus-needle[gpu]), or Apple Silicon Metal (cactus-needle[metal]) without any custom kernel work on your end.
Offline deployment gets the same level of documentation. The inference engine — the actual 14MB binary — is fetched once from Hugging Face and cached at ~/.cache/cactus-needle/<engine version>/; every call after that touches zero network. For hardware that will never see a network at all, the path is: needle fetch --platform-tag <target> on a connected machine to pull the right binary for the target architecture, copy it into the cache path (or bundle it inside the installed package directory) on the air-gapped device, and set HF_HUB_OFFLINE=1 so a cache miss fails loudly instead of hanging on a download attempt. It's a small detail, but it's the difference between "technically works offline" and "was actually designed for a device that will never once touch the internet" — regulated environments and embedded shipping pipelines care about exactly this kind of operational detail.
The confidence gate — and where it breaks
The feature Cactus leans on hardest in the pitch is calibrated confidence. Every response carries a confidence score computed as the minimum of two independent signals: a calibrated post-hoc head scoring the full prompt plus the generated call, and the raw decoding probability of the call tokens. Both have to agree before a call is trusted. The intended usage pattern is straightforward and, frankly, good practice: pick a threshold for your product, act automatically above it, and escalate to a human or a bigger model below it.
{
"type": "call",
"success": true,
"function_calls": [{"name": "set_lights", "arguments": {"room": "living room", "on": true, "brightness": 30}}],
"reasoning": "'living room' -> room; 'dim' -> on true, brightness 30",
"confidence": 0.94,
"decode_tps": 850.0,
"peak_ram_mb": 28.5
}
For a model meant to autonomously trigger real-world actions — unlocking a door, sending a payment, adjusting a thermostat — a calibrated confidence gate is arguably more important than raw accuracy. It's the difference between "the model is sometimes wrong" and "the model tells you when it's likely wrong."
Here's the part that isn't in any of the launch coverage: fine-tuning silently disables it. The docs state this plainly if you read the API reference — "Calibration holds for the base model only. Fine-tuning does not update the head, so an agent running tuned weights reports confidence as None and warns once at construction." LoRA fine-tuning, which is the officially supported and encouraged path for adapting Needle to your own tools, updates the base weights via an adapter but leaves the confidence head frozen and now-uncalibrated for the new distribution — so Cactus just turns it off and returns None instead of a number that would be actively misleading.
That's the technically correct decision. It's also a real gap: the safety feature that makes an autonomous, on-device, function-calling model safe to trust with real actions is exactly the feature most developers will lose the moment they adapt the model to their actual product, which is the whole point of shipping a fine-tunable base model in the first place. If you're building for a domain with real-world consequences and you plan to fine-tune — which the whole toolchain (needle generate-data, needle finetune, LoRA merge, .cact export) is built to encourage — you need your own confidence signal, because Cactus's stops at the base model's door. A one-line warning at construction time is easy to miss in a background service that only logs errors.
What changed vs. the rest of the on-device landscape
Needle isn't the first attempt at small, local, function-calling models — it's arguably the most narrowly optimized one. The README's own comparison claim is that Needle "trades wins with other small models like FunctionGemma 270M, LFM2.5 230M and Apple's on-device Foundation Models, at 5x to 70x smaller, and 2 bits against their f16." Independent coverage citing the model's own benchmark tables reports 63.7% on Mobile Actions, 42.6% overall on BFCL v4 (with 93.4% well-formed JSON output), and 32.6% in-domain on Seal-Tools, alongside a compute estimate of roughly 70 MFLOPs per decoded token versus roughly 460 for a comparably scoped Liquid AI LFM2.5 model and roughly 6,000 for Apple's on-device model.
Take that comparison with real skepticism. Needle runs at 2-bit precision with a 256-token context; the models it's compared against run at f16 with substantially larger context windows. Comparing accuracy-per-parameter or accuracy-per-FLOP across models operating at different precision and different context budgets isn't invalid, but it's not a controlled experiment either — it's closer to "here's how far compression got us," which is a legitimate thing to report, just not the same claim as "beats models 70x its size on equal footing."
| Needle 2 | FunctionGemma 270M | LFM2.5 230M | Apple Foundation Models (on-device) | |
|---|---|---|---|---|
| Parameters | 45M | 270M | 230M | Undisclosed (larger) |
| Precision | 2-bit (CQ2) | f16 | f16 | f16 |
| Package size | 14MB | ~500MB+ (unquantized) | ~450MB+ (unquantized) | OS-bundled |
| Context window | 256 tokens | Larger, model-dependent | Larger, model-dependent | Larger, OS-managed |
| Deployment | Standalone binary, any platform | Requires runtime/framework | Requires runtime/framework | iOS/macOS only, via Apple's framework |
| Fine-tuning | LoRA, official pipeline | Varies by provider | Varies by provider | Not user fine-tunable |
| License | MIT (weights + code) | Varies | Varies | Proprietary, platform-gated |
The columns that matter most for a developer decision aren't really the accuracy numbers — they're deployment surface and lock-in. Needle runs anywhere from a Raspberry Pi 5 to WebAssembly to watchOS; Apple's Foundation Models only run inside Apple's own stack; FunctionGemma and LFM2.5 need whatever inference runtime their providers ship. If your product has to run identically on a $150 Android phone, a Quest headset, and a Linux gateway box, Needle's platform matrix (macOS, Linux across x86-64/ARM64/ARMv7/RISC-V/MIPS32el, Windows, Android, iOS/watchOS/tvOS, and WASM) is doing real work that the accuracy table doesn't capture.
Why this actually matters for developers
Cost. Zero inference cost after the download — no API metering, no per-token billing, no rate limits. For a product with millions of trivial "turn on the lights" requests, that's not a marginal saving, it's the difference between a viable unit economics story and a subsidized one.
Latency. Reported decode throughput is around 500 tokens/sec on a Raspberry Pi 5 and 300–700 tokens/sec on sub-$200 Android phones — with a 256-token cap on output, that's a full response in well under a second, with no network round trip to a cloud endpoint at all. For voice assistants and device-control UIs, that latency gap is the actual product experience, not a footnote.
DX. The tool-declaration API is genuinely pleasant — decorate a function, and the docstring plus type hints become the interface. needle playground spins up a local browser UI to iterate on prompts and tool schemas before you write any deployment code, and the fine-tuning pipeline (needle generate-data → needle finetune → needle build) is a real, documented path from base model to a shipped .cact file, not a research checkpoint you're on your own to productionize.
Lock-in. MIT-licensed, weights on Hugging Face, runs on an open engine you can bundle yourself. The one asterisk: training data is proprietary, so you can fine-tune the released model but you can't reproduce it from scratch or audit what it was originally trained on.
Security and privacy. Inference does zero networking by design — the engine downloads once, caches locally, and every subsequent call stays on-device. For anything handling location, health, or home-security data, "the model literally cannot phone home" is a stronger privacy story than "we promise not to log your requests."
Practical use cases
- Smart-home hubs and IoT gateways that need to parse "dim the kitchen to 30%" into a device command without waiting on a cloud round trip or losing function on a dropped connection.
- Wearables and AR/VR devices (the README specifically benchmarks Meta Quest 3S and Apple Vision Pro) where battery and thermal budget rule out anything heavier.
- Offline-first mobile apps — receipt/invoice extraction, form-filling from free text, voice command routing — in regions or contexts where connectivity isn't guaranteed.
- Robotics command interpretation, where a fixed, small action space (move, grip, rotate, stop) maps naturally onto Needle's closed-vocabulary, grammar-constrained tool calling.
- Air-gapped or regulated environments where "the model can't reach the network" is a compliance requirement, not just a nice-to-have.
Limitations the docs undersell
Beyond the confidence-gate gap already covered, three things in the docs are stated plainly but easy to skate past on a skim:
-
There is no free-text fallback, period. A request no declared tool can serve doesn't get a hedge, a clarifying question, or an "I'm not sure" — it gets the empty call
[]. That's a deliberate design decision that makes the model predictable, but it also means Needle cannot gracefully degrade; every query outside your declared tool surface is a hard miss, and your product has to handle that UX case explicitly. - The 256-token window isn't a soft default, it's the whole memory model. Because tools are pinned as KV sinks inside that window, a long multi-turn conversation with a large tool catalogue is competing for a genuinely small budget. This is what keeps RAM flat at 28MB, but it means Needle is not a fit for anything resembling extended dialogue — it's a fit for short, transactional turns.
- Tool retrieval means unreachable, not just unlikely. Above five declared tools, a contrastive retrieval head selects the top five per turn and the decoding grammar is rebuilt over only that subset — an unselected tool literally cannot be called that turn, regardless of how well it matches the user's intent. If your catalogue is large and your retrieval head picks wrong, there's no way for the model to reach past it; you're depending on retrieval accuracy as much as generation accuracy, and the docs don't publish retrieval accuracy numbers at all.
None of these are hidden — they're documented in doc/apis.md — but they don't show up in the "14MB! 45M params! Beats models 70x bigger!" framing that's been driving the coverage this week, and they're the constraints that will actually determine whether Needle fits your product.
Independent read
The engineering here is real. A fixed Hadamard transform replacing a trained FFN, quantization baked in during pretraining rather than bolted on after, grammar-constrained decoding, and a two-signal confidence gate are all substantive design choices, not just "shrink the model and call it a day." The team publishing a paper alongside the release, rather than only a blog post, is a good sign for a company two models into this space.
Where I'd push back is on the framing, not the engineering. "Trades wins with models 5x to 70x larger" is true only if you accept that comparing a 2-bit, 256-token-context model against f16, larger-context baselines is a fair fight — and it isn't, quite. It's a genuinely impressive compression result, but it's being marketed with language that implies parity, and the actual numbers (low-40s% on BFCL v4, low-30s% in-domain on Seal-Tools) describe a model that's useful for narrow, well-scoped tool sets, not one that's quietly as capable as its much larger comparison points. The confidence-gate-breaks-on-fine-tune issue is the sharper concern: it's exactly the kind of gap that shows up in production three months after a team has already shipped, not during the demo.
Who should try it, who should wait
Try it now if you're building a narrowly scoped, offline-capable device-control or extraction feature with a small, well-defined tool catalogue (well under five tools, or willing to validate retrieval accuracy on your own tool set) — smart home, voice-triggered actions on mobile, structured extraction from receipts/forms/invoices where zero network cost and hard offline capability matter more than open-ended reasoning.
Wait if your product needs conversational memory beyond a handful of turns, a tool catalogue too large to trust to unverified retrieval, or you're planning to fine-tune for a safety-relevant domain and were counting on the confidence score surviving that process — build your own escalation signal first.
Ignore it if you need general-purpose language capability, multi-turn reasoning, or anything resembling a chat assistant; that's explicitly not what this model is for, and no amount of prompt engineering will get a 45M-parameter, 256-token-window model there.
What's your read on the "compare compressed-model results against full-precision baselines" pattern more broadly — is MFLOPs-per-token even the right axis to compare tiny on-device models on, or does it just reward whoever quantizes hardest?
Sources:
Top comments (0)