Published on TekMag | Category: AI
Needle 2 is a 14MB open-source foundation model from Cactus Compute that runs tool calling, device control, and structured data extraction entirely offline on tiny hardware. At just 45 million parameters compressed to 2-bit precision, it fits in about 28MB of RAM and delivers 500+ tokens per second on a Raspberry Pi 5. The model reached #1 on GitHub Trending on August 14, 2026, after the repo gained nearly 5,000 stars in its first days.
Key Takeaways
- Needle 2 is a 45M-parameter, 14MB model optimized for tool calling and structured extraction
- It runs entirely offline using CQ2-bit quantization from Cactus Quants
- Auto-retrieval limits context to the top 5 tools when catalogs exceed five
- Confidence scoring prevents execution below a user-defined threshold
- Benchmark performance competes with models 5–70x its size on tool-calling tasks
- LoRA fine-tuning produces portable .cact files that run on the same engine
What Needle 2 Is
Needle 2 is a tiny language model built specifically for one job: calling tools and extracting structured data from text. It does not generate free-form prose. When a user request falls outside what the declared tools can handle, Needle returns an empty call and refuses to answer.
The model was designed by Cactus Compute and released under the MIT license on GitHub. The entire weights live in a single 14MB binary. Inference uses approximately 28MB of RAM. There are no separate model files to manage, and once the engine downloads from Hugging Face, it caches locally and runs completely offline.
How It Works
Needle uses what the authors call a Simple Attention Network. Instead of a standard feed-forward network, it applies a Hadamard transform (a fixed orthonormal matrix computed in O(n log n) time with no learned weights). Attention uses grouped-query attention with engram key-value memory. The architecture also includes multi-lane hyper-connections and gating mechanisms.
Every response is a function call. The model does not generate text between calls. Arguments come back as structured JSON constrained by a byte-level grammar compiled directly from your schema definitions. If a parameter is declared as a Literal, the grammar only admits those exact values. If a field has a numeric range, the model cannot emit a value outside it.
Tool retrieval kicks in automatically when you declare more than five tools. A built-in contrastive embedding head scores every tool against the current query and only the top five enter the context window. The grammar rebuilds over just that subset. Tool embeddings persist across sessions via a tool_index_path file keyed to a fingerprint of the schemas, so re-embedding only happens when the tool definitions change.
Memory and Confidence
Needle maintains a 256-token sliding KV window for conversation history. The tools themselves are pinned as permanent KV sinks, meaning they stay in memory regardless of how long the session runs. This is what keeps total memory near 28MB even during extended multi-turn interactions.
Every call carries a confidence score between 0 and 1. The score combines two signals: a calibrated post-hoc head that evaluates the full prompt plus the predicted call, and the decoding probability of the call tokens. Both signals must agree for the call to pass. Below your chosen threshold, the model escalates rather than executes a questionable call. The off-screen failure mode is refusal, not wrong execution.
Benchmarks
Needle 2 competes with models 5 to 70 times larger. Here is how it landed on the published benchmarks:
- Mobile Actions: Needle 2 scored 63.7%, compared to FunctionGemma 270M at 64.0% and LFM2.5 230M at 69.1%
- DroidCall: Needle 2 achieved 17.0%, versus FunctionGemma 270M at 17.5% and LFM2.5 230M at 11.0%
- Seal-Tools in-domain: Needle 2 reached 32.6% against LFM2.5 230M at 26.9% and FunctionGemma 270M at 16.3%
- Seal-Tools out-of-domain: Needle 2 posted 28.7%, while LFM2.5 230M hit 17.0% and FunctionGemma 270M hit 15.6%
- BFCL v4 single-turn overall: Needle 2 scored 42.6%, with FunctionGemma 270M at 46.1% and LFM2.5 230M at 60.8%
- Well-formed rate: Needle 2 maintained a 93.4% rate of structurally valid JSON outputs
The BFCL v4 gap reflects the benchmark's emphasis on complex multi-step reasoning. On narrower tool-calling tasks, Needle holds its ground against models an order of magnitude larger.
Installation and Usage
Install with pip:
pip install cactus-needle
The inference engine downloads once from Hugging Face and caches locally. No build step is required. You can review the official research background in the Simple Attention Network paper.
Here is a minimal example using the decorator API:
import needle
@needle.tool
def get_weather(city: str) -> dict:
"""Get the current weather for a city."""
return {"city": city, "temp_c": 27, "sky": "clear"}
agent = needle.Needle(tools=[get_weather])
result = agent.run("What's it like in Lagos right now?")
print(result["results"])
[{'city': 'Lagos', 'temp_c': 27, 'sky': 'clear'}]
For structured extraction, declare a Pydantic model and call extract():
from pydantic import BaseModel
class Invoice(BaseModel):
vendor: str
total: float
due_date: str
invoice = needle.extract("Invoice from Acme Corp, $1,200.00, due 2026-09-01", Invoice)
print(invoice.vendor, invoice.total)
-> Acme Corp 1200.0
Schema constraints use needle.Field with Annotated types. Supported validators include enum, const, ge/le/gt/lt, pattern, format, and length bounds. These compile directly into the decode grammar.
Fine-Tuning
Needle supports LoRA fine-tuning on the frozen base weights. The adapter merges cleanly at export, producing a single .cact file that runs on the same engine with no recompilation.
needle finetune data.jsonl --epochs 3 --generate 300 --lora-rank 16 --lora-alpha 32
needle build checkpoints/needle2.pkl --lora checkpoints/needle_lora.pkl --out my_needle.cact
Data format is JSONL. Each line contains a query, the tool schema, and the expected answer. The optional reasoning field lets you include the model's short derivation (e.g., 'kitchen' -> room; 'dim to 10' -> brightness 10), which improves fine-tuning quality without being grammar-constrained.
You can also synthesize training data from your tool schemas using an OpenRouter API key, which expands your examples before fine-tuning.
Where It Runs
Needle targets devices where larger models simply cannot fit:
- Raspberry Pi 5: 500+ tokens/sec decode
- VR headsets: 400–1,500 tokens/sec
- Phones under $200: 300–700 tokens/sec
- ESP32-class microcontrollers and other embedded platforms
Cactus Compute has shipped Needle in production inside the Pebble Index Ring, running locally within the Index 01 app. The model handles tool selection and structured responses without any network dependency.
Should You Use It?
Needle is not a general-purpose chat model. It will not write essays, summarize articles, or answer trivia. If your application needs to interpret natural language into tool calls, extract structured data from text, or control devices offline, it is purpose-built for that job.
The trade-off is size versus capability. At 45M parameters, Needle covers a narrow band of functionality exceptionally well. For applications that need broader reasoning alongside tool use, a larger model may be more appropriate. But for edge deployments, privacy-sensitive environments, or offline-first products, Needle removes the cloud dependency entirely.
Frequently Asked Questions
Q: Does Needle 2 require an internet connection?
A: Only for the initial engine download from Hugging Face. After caching, inference runs completely offline with no network calls.
Q: Can I use Needle for general text generation?
A: No. Needle is designed exclusively for tool calling and structured extraction. Off-topic prompts return an empty call rather than generating free text.
Q: What is the difference between Needle 1 and Needle 2?
A: Needle 2 uses the Simple Attention Network architecture with CQ2-bit quantization, significantly reducing model size while maintaining competitive benchmark performance compared to the original release.
Q: How does tool retrieval work with large tool catalogs?
A: When you declare more than five tools, a built-in contrastive embedding head scores each tool against the current query. Only the top five enter the context window, and the decode grammar constrains responses to that subset.
Q: Is Needle suitable for production use on microcontrollers?
A: Yes. Cactus Compute reports successful deployment on ESP32-class devices, and the model is already running in the Pebble Index Ring production app.
Conclusion
This article has examined the key developments, regulatory dynamics, and market implications of this topic. As the situation continues to evolve, stakeholders should monitor upcoming milestones and assess how these changes align with their strategic priorities.
References
- [1] cactus-compute/needle — GitHub repository (MIT license) — https://github.com/cactus-compute/needle
- [2] Needle 2 model weights — Hugging Face — https://huggingface.co/Cactus-Compute/needle2
- [3] Simple Attention Network paper (arXiv:2607.18363) — https://arxiv.org/abs/2607.18363
- [4] Cactus Compute product page — https://cactuscompute.com/needle
- [5] Coverage: MarkTechPost — https://www.marktechpost.com/2026/08/13/cactus-compute-needle-2-45m-parameter-tool-calling-model
{
"@context": "https://schema.org",
"@type": "FAQPage",
"mainEntity": [
{
"@type": "Question",
"name": "Does Needle 2 require an internet connection?",
"acceptedAnswer": {
"@type": "Answer",
"text": "Only for the initial engine download from Hugging Face. After caching, inference runs completely offline with no network calls."
}
},
{
"@type": "Question",
"name": "Can I use Needle for general text generation?",
"acceptedAnswer": {
"@type": "Answer",
"text": "No. Needle is designed exclusively for tool calling and structured extraction. Off-topic prompts return an empty call rather than generating free text."
}
},
{
"@type": "Question",
"name": "What is the difference between Needle 1 and Needle 2?",
"acceptedAnswer": {
"@type": "Answer",
"text": "Needle 2 uses the Simple Attention Network architecture with CQ2-bit quantization, significantly reducing model size while maintaining competitive benchmark performance compared to the original release."
}
},
{
"@type": "Question",
"name": "How does tool retrieval work with large tool catalogs?",
"acceptedAnswer": {
"@type": "Answer",
"text": "When you declare more than five tools, a built-in contrastive embedding head scores each tool against the current query. Only the top five enter the context window, and the decode grammar constrains responses to that subset."
}
},
{
"@type": "Question",
"name": "Is Needle suitable for production use on microcontrollers?",
"acceptedAnswer": {
"@type": "Answer",
"text": "Yes. Cactus Compute reports successful deployment on ESP32-class devices, and the model is already running in the Pebble Index Ring production app."
}
}
]
}
Read the full article: https://tekmag.thsite.top/needle-the-14mb-open-source-foundation-model-for-tiny-devices/
Top comments (0)