I Tried Needle2 for Local Tool Calling. I Ended Up With llama.cpp + Granite

I’ve been working on a small experiment around a semantic shell.

The basic idea is that the user types something like:

copy report.pdf to backup

and a small local model maps that to a known tool:

filesystem.copy

The shell then takes over. It validates arguments, asks for missing ones, shows confirmation UI when needed, and finally executes the operation.

The model is not supposed to generate shell commands. It only needs to understand intent.

That sounded like a very good fit for Needle2.

Needle2 is tiny, focused on tool calling and structured extraction, and designed to run locally. On paper it looked almost purpose-built for what I needed.

So I tried it.

The first version worked. Then I started adding more realistic cases.

And that’s where things became more interesting.

Five tools is very different from fifty

Most examples of tool-calling models are some variation of:

turn on the kitchen light

with two or three tools available.

That’s fine as an API example, but it doesn’t tell you much about how the model behaves in a real application.

Even my first filesystem package already had things like:

filesystem.copy
filesystem.move
filesystem.delete
filesystem.create_file
filesystem.create_directory
filesystem.list_directory
filesystem.navigate
filesystem.current_directory
filesystem.find

Needle2 uses retrieval when the number of tools gets larger. It selects a small candidate set and then resolves the tool from there.

That makes sense.

But it also means there are really two decisions now:

Which tools should be considered?

and then:

Which of those tools is the right one?

If the correct tool does not make it into the shortlist, the second stage never gets a chance.

For many applications that may be perfectly acceptable. In my case I wasn’t comfortable relying on it without knowing more about how often it happens.

I started thinking about bypassing retrieval entirely by dividing tools into groups of five and running all groups.

Needle is small enough that this is not completely ridiculous.

Something like:

group 1 -> result
group 2 -> result
group 3 -> result
...

then take the strongest candidate.

But immediately another question appears.

Are confidence scores from different candidate groups actually comparable?

Maybe they are. Maybe they are not.

I couldn’t find enough information to feel confident about treating them as globally calibrated scores.

That does not make Needle bad. It just means that once you move away from the demo path, you start needing answers that are not obvious from the examples.

Tool descriptions matter a lot

Another thing I learned quickly was how sensitive a small model is to tool descriptions.

A description like:

Moves files from one place to another.

is not enough if the neighboring tools are:

filesystem.copy
filesystem.navigate
filesystem.rename

You need to be explicit.

For example, “move” should probably say that the original item no longer remains at the source, and that this is not the same thing as changing the shell’s current directory.

The descriptions start looking less like documentation and more like a tiny classification dataset written in English.

That was workable, but it changed how I thought about the model.

It is less “small agent” and more “probabilistic semantic parser with a constrained vocabulary.”

Which is actually fine for this use case.

Then I tried the C++ runtime

For the shell itself I wanted native integration.

No Python process on the side, no extra service, and no awkward lifetime management. The C++ library looked like the right way to do it.

This is where I hit the biggest practical problem.

I was developing on an Intel Mac.

The Cactus Engine code contains ARM NEON includes and intrinsics in many places. Not just one isolated backend file, but spread through engine and kernel code.

That surprised me.

If ARM is the main target, this is understandable. iOS, Android, and Apple Silicon are all important platforms.

But from my side it meant x86 support was not just a matter of enabling another optimized backend.

I ended up spending time patching things simply to answer a much more basic question:

Is Needle good enough for my use case to justify carrying this dependency?

At that point I decided to try a different route before spending more time on the port.

llama.cpp and Granite

So I integrated llama.cpp.

Then I tried Granite 4 350M for the same tool-calling task.

I expected this to be more of a baseline than a solution.

It turned out to work better than I expected.

For the normal cases it selected the tools I wanted.

More importantly, I tried a few deliberately awkward inputs where no good tool really existed, and instead of forcing a call it simply failed to find one.

For my application, that is good behavior.

I would much rather see:

No suitable tool found.

than a confident guess.

The shell can recover from that.

It can ask the user to rephrase, narrow the scope, or choose between two possible interpretations.

A wrong tool call is much harder to recover from once you start dealing with destructive operations.

This changed the architecture less than I expected

The funny part is that switching models did not really affect the shell design.

The important part was already outside the model.

The flow is roughly:

user input
    ↓
intent resolution
    ↓
known semantic capability
    ↓
argument validation
    ↓
ask for missing arguments
    ↓
policy / confirmation
    ↓
execution

If the user says:

copy file

the model only needs to resolve:

filesystem.copy

The shell can notice that source and destination are missing and show proper selectors.

If two tools are plausible, for example copy versus move, the shell can show two buttons.

There is no need to force the model to solve every ambiguity.

That ended up being one of the more useful conclusions from the experiment.

A small local model does not need to be brilliant if the surrounding system is designed to handle uncertainty.

The part I wish documentation focused on more

I still think Needle2 is interesting.

What I missed was more information about the edges.

Not another:

turn on the light

example.

I wanted answers to things like:

• how accuracy changes as the tool catalogue grows;
• how often retrieval drops the correct tool;
• whether confidence remains comparable across different candidate sets;
• how similar tools should be described;
• what happens on no-match input;
• how confidence behaves on incorrect calls;
• how much fine-tuning helps with ambiguous tools;
• what the realistic native-platform support is.

Those details are not as flashy, but they are what decides whether something works in a real system.

The same applies to structured extraction.

Showing that a model can extract an invoice total is useful.

Showing how it behaves when there are subtotal, tax, total, previous balance, OCR errors, and two dates on the same page is much more useful.

Where I am now

For the moment I’m continuing with:

llama.cpp
Granite 4 350M
typed semantic tools
deterministic argument resolution
small contextual UI
policy-controlled execution

That does not mean Needle2 is a bad model.

It may be an excellent fit for the platforms and use cases Cactus is targeting.

It just stopped being the obvious choice for mine.

And that was probably the main thing I got from the experiment.

A component can look almost perfect from the feature list and still be the wrong foundation once you start testing the boring parts: portability, failure modes, ambiguity, scaling, and integration cost.

For this kind of software, I’m becoming much more interested in how a model fails than in how impressive its best demo looks.