DEV Community

Vigneshwaran M
Vigneshwaran M

Posted on

On-device tool calling in React Native: 228 ms, no server, no API key

#reactnative #android #ai #opensource

Say "Turn the kitchen lights down to 30 percent" and get back { room: "kitchen", level: 30 } in 228 ms — on the phone, offline, with no server, no API key and no per-token cost.

That's react-native-needle, a React Native binding I built for Cactus Compute's Needle 2 — a 45M-parameter model that runs entirely on-device. As far as I can tell it's the first RN binding for this model.

npm install react-native-needle
npx react-native-needle fetch-model
Enter fullscreen mode Exit fullscreen mode

Below: what it does well, the numbers I measured rather than the ones in the docs, and exactly where it breaks — including one failure you want to know about before you build on it.

Why a 45M model at all

Needle 2 isn't a chatbot. Give it a tool schema and it emits a schema-conforming call or declines. That's a narrower job than "generate text," and a much better fit for phones: no server, no API key, no per-token cost, no network, and nothing leaves the device.

It's also small enough that binding it is not a project. The C API is four functions — that's the whole thing:

int  needle_load(const unsigned char* cact, unsigned long long n);
int  needle_init(const char* system_prompt, const char* tools_json, const char* tool_index_path);
int  needle_complete(const char* input, int max_new_tokens, char* out, int out_capacity);
void needle_reset(void);
Enter fullscreen mode Exit fullscreen mode

I'd budgeted days for the native work. Four functions behind a JNI bridge and a CMake target is an afternoon.

Using it 

import { needleSupported, loadBundledModel, configure, extract } from 'react-native-needle';

if (needleSupported) {
  await loadBundledModel();

  await configure('You control smart home devices. Call a tool for the user request.', [
    {
      name: 'set_brightness',
      description: 'Set the brightness of the lights in a room',
      parameters: {
        type: 'object',
        properties: {
          room: { type: 'string', description: 'Which room' },
          level: { type: 'number', description: 'Brightness percentage, 0-100' },
        },
        required: ['room', 'level'],
      },
    },
  ]);

  const args = await extract<{ room: string; level: number }>(
    'Turn the kitchen lights down to 30 percent'
  );
  // → { room: 'kitchen', level: 30 }
}
Enter fullscreen mode Exit fullscreen mode

The tool schema is the extraction schema. extract() returns null rather than throwing when the model declines — a caller pre-filling a form wants "no answer", not an exception.

What it's genuinely good at

I ran eight measured cases on an arm64 emulator. Clean, single-intent commands are excellent:

Input Output Time
Turn the kitchen lights down to 30 percent {room: "kitchen", level: 30} 228 ms
Set a timer for 12 minutes {minutes: 12} 136 ms
Your parcel weighing 2.4 kg has left the depot, tracking AB4471. {weight_kg: 2.4, tracking: "AB4471"} 619 ms
What is the capital of France? declined — no matching tool 174 ms

That last row matters as much as the others. Asked something outside its tools, it returns an empty call list and a reason rather than inventing an answer.

For device control and voice-command routing, this is genuinely usable: sub-second round trips, offline, free.

The one build gotcha

arm64-v8a linked cleanly. armeabi-v7a did not:

undefined symbol: std::__ndk1::__hash_memory
Enter fullscreen mode Exit fullscreen mode

Cactus's 32-bit archive is built against a newer libc++ than NDK 27 exports. No flag fixes that — it's an ABI mismatch in a prebuilt binary. So the package is arm64-only, with a needleSupported flag so 32-bit devices degrade instead of crashing with an UnsatisfiedLinkError.

Worth knowing before you plan around "it's only 14 MB": in the APK it's 28 MB — 14.5 MB of engine plus 13.7 MB of weights.

Known limits

This is the part I'd want to read before adopting something, so here it is in full.

A leading unrelated clause kills the whole request. Given:

Preheat to 200 C and bake 25 minutes, serves 4. Add 250 grams of flour to the list.

with an add_item tool available, it declined outright:

{ "function_calls": [], "reasoning": "No tool available for preheating or baking." }
Enter fullscreen mode Exit fullscreen mode

The second sentence on its own works fine. It latches onto the first instruction and gives up instead of scanning for the part it can serve. Segment multi-intent input yourself.

String fields over-capture. In a denser message, tracking came back as "AB4471, ref 522119876543" — the code plus the next field glued on.

Numbers in crowded strings get fabricated. Given a message with several numbers in it, the model returned 1230.0 for an input value of 560.00 — at confidence 1.0000, with a written rationale that quoted the correct number and then produced a different one. Reproducible. Tightening the schema didn't help; adding "copy verbatim, never calculate" to the system prompt produced byte-identical output.

Memory. Documented at ~28 MB per session. Measured peak RSS: 500–530 MB. That's the real deployment constraint, not the model file size.

The signal to gate on

Here's the part worth stealing even if you never touch this package. Needle tells you when it's making things up:

"validation": { "ungrounded": ["record_item.amount"], "negation": false }
Enter fullscreen mode Exit fullscreen mode

It flagged the exact field it had fabricated — in the same response where it reported confidence 1.0.

So the confidence score was useless and the self-report was correct. The package exposes this directly:

import { complete, ungroundedFields } from 'react-native-needle';

const raw = await complete(userText);
if (ungroundedFields(raw).length) {
  // The model is telling you it made these up. Discard them.
}
Enter fullscreen mode Exit fullscreen mode

If you build on any model that exposes something like this, gate on the grounding flag and ignore the confidence number. I'd have shipped a wrong value if I'd trusted the metric that looked like the trustworthy one.

Two numbers I got wrong myself

My initial notes said "9–15 s per completion." Re-measuring across eight runs gave 140 ms – 1.6 s. The original figure came from one cold run that included initialisation, and I'd written it down as steady-state. Measure more than once before you publish a benchmark — including in your own README.

I also shipped a real bug by guessing. I wrote the response parser from what I assumed the envelope looked like:

return parsed?.arguments ?? parsed?.parameters ?? parsed;
Enter fullscreen mode Exit fullscreen mode

The real output nests one level deeper:

{"type":"call","function_calls":[{"name":"set_brightness","arguments":{...}}]}
Enter fullscreen mode Exit fullscreen mode

So extract() handed callers the whole envelope instead of their fields, and a decline came back as a truthy object — indistinguishable from a real answer. Fixed in 0.1.1. The tests now run against real captured device output, not my idea of the shape; every one of my original hand-written tests had passed against a format the model never emits.

One npm packaging trap

files in package.json is an allowlist, and it beats .npmignore. My .npmignore excluded the 20 MB engine and 14 MB weights. npm ignored it and cut a 97.6 MB tarball. The exclusions have to be ! negations inside files:

"files": ["build", "android", "!android/libs", "!android/src/main/assets/*.cact"]
Enter fullscreen mode Exit fullscreen mode

With that, 18.6 kB. There's now a prepack check that fails the pack if those negations ever go missing, because the failure is invisible until you actually inspect the tarball.

Should you use it?

Yes, if you want offline tool calling or voice-command routing: declare your tools, get a conforming call back in 140–600 ms, and ship with no backend at all. The engine is fetched and checksum-pinned on install; the weights are one explicit command.

Be careful if a wrong number is expensive in your app. Gate on ungroundedFields(), segment multi-intent input yourself, and treat confidence as decoration.

npm install react-native-needle
npx react-native-needle fetch-model
Enter fullscreen mode Exit fullscreen mode

Android/arm64, Expo SDK 51+. MIT; the engine is Apache-2.0 from Cactus Compute.

The natural next step is a LoRA fine-tune on domain data, which cactus-needle supports — a 14 MB model that's actually good at one narrow job is a far more useful thing than a general one. If you try that, I'd like to hear how it goes.

Top comments (0)