DEV Community: pielouNW

Triage support tickets locally with a 449 MB decision model

pielouNW — Thu, 29 Oct 2026 16:25:54 +0000

Every support inbox asks the same three questions of every ticket: who should handle it, does the customer want their money back, and how much is on fire. The usual fix is to ask an LLM, write a parser for whatever it decides to say, and then write a second parser for the day it answers "Billing (probably)" instead of "billing".

NobodyWho's new decision model support for Flutter, React Native, Swift, Kotlin and Godot bindings, remove the parser from that story. A decision model scores answers you have already written. You give it the options up front, it returns a probability for each one, and it never generates a single token, so it has no way to invent a fourth department.

TypeSafe calls this kind of model a System One model, after the fast, intuitive half of Kahneman's Thinking, Fast and Slow, and describes its own model, Jev, as "unstructured state in, typed probabilistic decisions out". NobodyWho runs the same kind of decision on your own hardware, using Laya, an Apache-2.0 decision model whose Q8_0 GGUF file is 449 MB.

Load the model

Install NobodyWho with pip install nobodywho, then point a Model at the Laya file on Hugging Face, or at a local GGUF path if you already have it.

from nobodywho import Decision, Model

model = Model(model_path="hf://ggml-org/Laya-GGUF/Laya-Q8_0.gguf")
decisions = Decision(model=model, n_ctx=1024)

Ask the ticket three questions

Each question has an ID, a type and plain-English instructions. A choice question picks one option, a noul question returns the probability that the answer is yes, and a score question places the ticket on a scale you define, from two to ten levels.

message = "I was charged twice. Please refund the duplicate payment."

questions = {
    "department": {
        "type": "choice",
        "instructions": "Which team should handle this?",
        "criteria": {
            "billing": "Payments and refunds",
            "technical": "Bugs and login problems",
        },
    },
    "refund": {
        "type": "noul",
        "instructions": "Is a refund requested?",
    },
    "urgency": {
        "type": "score",
        "instructions": "How urgent is this?",
        "criteria": ["can wait", "soon", "urgent"],
    },
}

result = decisions.decide(state={"message": message}, questions=questions)

team = result["answers"]["department"]["choice"]
probabilities = result["answers"]["department"]["probabilities"]
refund_probability = result["answers"]["refund"]["noul"]
urgency = result["answers"]["urgency"]["score"]

assert result["usage"]["output_tokens"] == 0

That last line is the whole pitch in one assert. The model read the ticket, answered three questions and wrote nothing.

What actually comes back

NobodyWho's test suite runs a very similar ticket through Laya: "I was charged twice for my order last week and nobody has replied." It asks which of billing, shipping or technical should take it, whether the customer is angry, and how urgent it is on a four-step scale from "can wait" to "right now". These are the reference answers the suite checks against on CPU, rounded:

{
  "route": {
    "choice": "billing",
    "probabilities": {"billing": 0.975, "shipping": 0.014, "technical": 0.011},
    "confidence": 0.962
  },
  "angry": {"noul": 0.784},
  "urgency": {
    "score": 1.914,
    "legend": {"0": "can wait", "1": "this week", "2": "today", "3": "right now"},
    "probabilities": {"0": 0.040, "1": 0.458, "2": 0.050, "3": 0.452},
    "confidence": 0.006
  },
  "usage": {"input_tokens": 166, "output_tokens": 0}
}

The routing answer is as clean as it gets: billing at 97.5%, with a confidence of 0.96. The model also puts the chance that this customer is angry at 78%, which anyone who has been charged twice will find conservative.

The urgency answer is where it gets interesting. A score is a weighted average of the levels, and 1.914 rounds neatly to "today", but the model only gives "today" 5%. What it actually believes is that this ticket is either a "this week" problem (45.8%) or a "right now" problem (45.2%), and the confidence of 0.006 is its way of telling you it can't pick. If your code only read the score, it would file the ticket under an answer the model gave a 5% chance.

Send the unsure ones to a person

Choice and score answers both carry that confidence value, from 0 to 1, which measures how concentrated the probabilities are. That gives you a simple rule for triage: route confident answers automatically and put the rest in front of a human.

answer = result["answers"]["department"]
queue = answer["choice"] if answer["confidence"] >= 0.6 else "human-review"

Applied to the reference output above, the route goes straight to billing at 0.96 and the urgency goes to a person at 0.006, which is exactly the split you want. Treat 0.6 as a starting point to tune against your own tickets, and keep an eye on the queues for the first week, because the docs are clear that high confidence does not guarantee a correct answer.

Running it in your app

DecisionAsync gives you the same API with await, which suits a web handler that triages tickets as they arrive. Decision models are also in the Flutter, React Native/Expo, Swift, Kotlin and Godot bindings, so the same questions can run inside a mobile or desktop app, and the ticket text never leaves the device.

The full Python guide can be found here.

If this saved you a parser, a star on the repo is the easiest way to tell the team to keep going ❤️

(How) Can LLMs run in the browser?

pielouNW — Wed, 28 Oct 2026 09:45:51 +0000

A big push here at NobodyWho is to get language models running in the browser using WebAssembly.

After working on it for a few weeks we've gotten a good idea of the landscape. One thing that's very clear, and this will come as a surprise, is that browsers are actually kinda complicated.

First, we use Rust at NobodyWho, and there are (at least) three different compilation targets for Rust for WASM: Emscripten, WASI, and wasm32-unknown-unknown with wasm-bindgen. All with different levels of support in various dependencies and within Rust itself.

Second, there are (at least) three different hardware backends: CPU, WebGPU, and WebNN, with the last two being the only reasonably fast ones and both being very new or experimental.

Third, there's the question of whether the LLM will be running in a website or a browser extension.

Fourth, and importantly, there are many different browsers and many different browser versions. Many of the features needed for running LLMs in a browser are relatively new, and browser support can be spotty, especially if your users haven't updated for a while.

Spoiled for choice

Fortunately, many of these dimensions can get flattened by technical constraints or a specific use case.

At NobodyWho we use both llama.cpp and the ONNX Runtime as inference engines, and they both assume a system library. That's an issue for wasm-bindgen, since the second unknown in wasm32-unknown-unknown means "unknown OS", i.e. no system library. This leaves WASI and Emscripten, with Emscripten being the obvious choice given that WASI in the browser is… experimental.¹

The story for hardware backends is just as clear: run with WebGPU when available and fall back to CPU when WebGPU isn't available (or when on Firefox, since apparently GPU is slower there). Why not WebNN? Again because it's… experimental.²

At NobodyWho we're focusing on running LLMs in a website rather than a browser extension, since that's what we have a use case for, but there are some interesting tradeoffs between the two. A big issue with running an LLM in the browser is that useful models are big. Even the small models are typically a few hundred MB, which is much more than a typical website. Given how important load times are, that can be a huge blocker. For an extension, you download the model once, deliberately, when you install the extension, so it taking some time is less of an issue,³ while website use is generally more ephemeral. This is less true for websites that users typically return to often, since the site can then cache the model so the download cost is only paid once.

Lastly we have browsers, and it's a mess. The most relevant features are WebGPU, Memory64 (see my previous post), and SharedArrayBuffer (for multi-threading). All of these have patchy support across browsers and platforms. Often, support just isn't implemented yet. Other times it is, but requires a feature flag to be turned on in the browser (good luck getting your non-technical users to do that). Sometimes, it is officially supported, but just works really poorly. The best example of this is Firefox's WebGPU support mentioned above.

Multi-threading and SharedArrayBuffer is its own can of worms because it requires strict cross-origin headers (COOP/COEP) which can be a real problem for some sites. And of course the easiest way to meet these requirements are different for each browser. On top of that, browsers vary wildly in how much storage they allow for websites, what their eviction policies are, how aggressive they are in killing workers, etc. The only good news is that of the large number of different browsers, in practice most people use Chromium-based browsers like Chrome and Edge,⁴ and that is where support is by far the best.

LLMs do run in the browser

But not cleanly. There are a lot of options and target platforms/browsers/use cases all requiring you to handle stuff differently. Until browser support for the necessary features becomes much more mature, a library for browser LLM inference that just works in all browsers, on all platforms, for most use cases, will require a lot of work on top of the basic browser APIs. A bit ironic given that part of the dream of WASM is to not depend on specific platforms so heavily.

I think incorporating such a general web LLM library into NobodyWho as part of our mission to support all platforms would be super cool, but for now we're exploring a few specific use cases where Emscripten and WebGPU with CPU fallback are the right choice. We haven't figured out how to handle the spotty browser support yet, but we'll get to that in time. Hopefully you'll see a release announcement right here within the next few weeks!

Notes

¹ Everything is shimmed ATM, which makes it pretty damn slow.

² It's supported nowhere. But at least there's a bug report on that for Chromium!

³ Extensions have other issues though. Browsers generally aggressively kill background processes like those an extension would hold an LLM in, so without some hacky solutions (e.g. offscreen documents) to keep them alive you'd need to constantly reload the multi-GB model from disk.

⁴ I'm not saying this is a good thing, just that it is the case, unless you're targeting some specific demographics. E.g. iOS browsers are all WebKit based (like Safari) while the proportion of Firefox users is higher among Google ~~hating~~ skeptical tech-nerds.

How to build an offline transcription app with Whisper

pielouNW — Mon, 26 Oct 2026 14:25:05 +0000

At NobodyWho, we've spent the last months shipping new features, like Speech-to-Text or Voice Activity Detection, to complete our local AI stack, and now felt like a good time to run a few experiments with our software.

I decided to build a simple transcription app with React Native that you can use fully offline once you have downloaded the whisper model (small, int8 quantization). With this app, you can easily record 10+ minutes of English speech and save either the entire transcript or just the parts you are interested in. Let's take a closer look at the implementation and see how it compares to a cloud transcription service.

Recording and transcription

The interface is straightforward: a button to start the recording and a timer, with the first sentences appearing as soon as they are processed. Let's have a look at what happens when the user presses the recording button:

First the OS asks permission for microphone access.
Then the audio streaming starts and the stream of audio buffers arrives.
Each buffer is processed by transcribeChunk (via onChunk), which calls NobodyWho's .transcribePcm method and returns the corresponding text. Since Whisper by default, returns non-verbal human sounds like coughs, laughs, yawning and so on, we remove them from the transcription (see cleanText).
When the user presses the stop button, we stop the recording and wait for any queued chunks to be processed thanks to pendingRef.

Here is what the simplified streaming process with buffers looks like in the app:

const { stream } = useAudioStream({
  sampleRate: SAMPLE_RATE,
  channels: 1,
  encoding: 'int16',
  onBuffer: buffer => {
    if (!chunkerRef.current) {
      chunkerRef.current = new Chunker(buffer.sampleRate, onChunk);
    }

    chunkerRef.current.push(toMonoInt16(buffer)); // .push() also calls onChunk
  },
});

const start = async () => {
  await requestRecordingPermissionsAsync();
  await stream.start();
};

const onChunk = (chunk: AudioChunk) => {
  const index = chunksRef.current.length;
  const live: LiveChunk = { start: chunk.start, end: chunk.end, text: null };
  chunksRef.current = [...chunksRef.current, live];

  const job = transcribeChunk(chunk)
    .then(text => {
      chunksRef.current = chunksRef.current.map((c, i) =>
        i === index ? { ...c, text } : c,
      );

      setChunks(chunksRef.current);
    });

  pendingRef.current.push(job);
};

const transcribeChunk = async (chunk: AudioChunk): Promise<string> => {
  const text = await whisperModel.transcribePcm(chunk.samples, chunk.sampleRate).completed();
  return cleanText(text);
}

Once the job queue is empty, the user is redirected to the Transcript screen where they can see the full transcript, export or listen to each chunk individually or compact them into one audio and text file:

Results and comparison

Alright, but how does it compare to other transcription tools? Since I'm French and my English is well, not the best as you can imagine, I often use subtitles on YouTube videos. Therefore I did a little comparison, using latest MKBHD tech review of the iPhone Duo:

YouTube transcript

First-generation devices are always the most fun to review. And for the past, like, month, everyone I talk to has been asking me, "What do you think of the iPhone Duo? Like, should I get a foldable iPhone?? Is that for me? Who is it even really for?? You've been testing it for the past week, right?? Yeah... So most new pieces of tech are fundamentally like a solid idea at their core. And then they can be a little bit rough around the edges, but that'll get better over time. The iPhone Duo specifically, as the first folding iPhone, is very polished around the edges, but kind of rough in the middle, if that makes sense.

App transcript

First generation devices are always the most fun to review. And for the past like month everyone I've talked to has been asking me, "What do you think of the iPhone Duo? Should I get a foldable iPhone? Is that for me? Who is it even really for?" You've been testing it for the past week, right? Yeah. So most new pieces of tech are fundamentally like a solid idea at their core and then they can be a little bit rough around the edges but that'll get better over time. The iPhone Duo specifically as the first folding iPhone is very polished around the edges but kind of rough in the middle if that makes sense.

The differences are minor. A "like" has been omitted by the Transcriber app, "I've talked" isn't correct in the Youtube transcript and punctuation is sometimes better in one version and sometimes in the other one, that's it!

This app has been vibe-coded and manually adjusted in half a day. It obviously needs more polishing, but it clearly shows potential for being a solid alternative to paid-per-token, cloud based solutions.

Adding Voice Activity Detection (VAD) would be a nice way to visually separate chunks of text, and using another more powerful whisper model (medium, large... with less quantization) would also improve results.

For more details, you can find the app’s repo on GitHub where you can download the Android apk to test it on your phone.

Want to build your app with on-device AI features? Star NobodyWho on GitHub and dive into the docs.

Letting Rust support models bigger than 4GB in the browser

pielouNW — Wed, 21 Oct 2026 09:40:44 +0000

At NobodyWho we aim to make language models run everywhere. We already have support for mobile and desktop apps, the Godot game engine, and even the Apple Watch. Nowadays though, most of the software people use isn't desktop apps running directly on their architecture of choice, but instead running in a browser on some website. It's therefore a huge hole in our promise that we don't support running language models in the browser. We know that, which is part of why we recently started work on adding WASM support to our library.

One thing that most people know about large language models is that they are, well, large, with even the "small" language models that NobodyWho specializes in providing easily taking many GB. This is a problem when wasm32 has a hard cap of 4 GB memory . Luckily we have wasm64 to save us. Great news is that Rust does support a wasm64 target. Bad news is, it doesn't support a wasm64+ Emscripten target.

	`unknown` (no C interop)	`emscripten` (C/C++ interop)
wasm32 (4 GB cap)	✅ `wasm32-unknown-unknown`	✅ `wasm32-unknown-emscripten`
wasm64 (Memory64)	✅ `wasm64-unknown-unknown`	❌ missing

The most straightforward way to fix this would be to create a custom target and use that, but of course Rust is open source, so we can fix it at the source and others can benefit as well. One issue was that a lot of the Rust standard library assumes that the only non-unknown wasm target was 32-bit. The Rust compiler has a fair bit of conditional compilation conditional on whether the target is wasm or not, but most of it just checked for target_arch="wasm32" instead of wasm targets in general so first step was fixing that by replacing target_arch="wasm32" with target_family="wasm" (PR).

Next step is actually creating the target. We have a PR for this active in the Rust repo, but it'll probably take some time for this to get merged, and even longer before it lands in stable.

Of course all of this still requires us to get NobodyWho to run in WASM, which both requires large changes in our own library and more upstream work, which we are making progress on. Even after that there's the exciting problem of figuring out how to make this useful. Figuring out problems like getting the models to the user's machine (even a small half GB model can be a lot to download while on a website), and how to make them run reasonably fast.

Want to run LLMs in the browser? Star and watch NobodyWho on GitHub to get notified when WebAssembly support drops!

You don’t need [insert expensive code review agent here]. All you need is Pi.

pielouNW — Tue, 20 Oct 2026 09:32:57 +0000

Our code review action powered by Pi and GLM 5.3

As we’re shipping out more and more code, it’s becoming increasingly hard to keep up. There’s a whole economy of code-review-as-a-service companies that are tackling that space. CodeRabbit, Greptile, Macroscope, are just some of them. These are great, but they are also expensive. Additionally - in most of them, you don’t actually pay per review or lines of code, or even tokens, you pay per seat!

At NobodyWho, we are all about keeping control. And if we can avoid a fat bill while we are at it, we will. Instead of adopting an expensive and fancy code review service, we just use Pi.

Need a review in your PR? Just comment /ai-review and an agent will review your code. We control the model, we control the prompt, and we control when and how it runs.

Here’s how it works.

The core: Code review skill

We stole the original review skill from the Earendil folks and their pi-review extension (we are big fans!). In our repo it sits right under .agents/skills/ai-review/SKILL.md. Anyone can run that skill locally before they open a Pull Request.

It’s a single file with code review instructions that our team can iterate on. One of the things we like the most about it is that it categorizes issues according to priority: P0 (fix this now), P1 (you should fix it), etc.

We are completely in control of what the agent reviews, and how it reviews it. Great - now how do we run it?

The engine: GitHub Actions

Our GitHub Action is also straightforward:

It runs whenever someone posts /ai-review as a comment in a PR
Adds an ai-review label and acknowledges that it started reviewing the code
It gets the details and diff of the PR
It installs Pi
The code review launches using the Pi CLI, our selected model from OpenRouter (we like GLM 5.3!) and using the code review skill from the main branch (to avoid prompt injection shenanigans)
When it’s done it posts the review to the PR as a comment
Finally, it saves the review as a GitHub Action artifact we can inspect and debug - which uses Pi’s export functionality

Inspecting the transcript from an AI code review run

Check our action here!

Why do we prefer this?

Control: We know what model ran, we know what prompt ran, and we can change those whenever we want according to what we need.
Price: We pay per review or per token. We don’t pay per seat. No subscriptions, no surprises, no rug pulls.
Customizability: Want to test that new model? Go for it. Too verbose? Adjust the skill. Swarm of agents? Go nuts. Want to make it more fancy and have it comment the diff and suggest? Possible!
Debuggability: Thanks to GitHub Actions and Pi, we save a full transcript every time we run it. This means we can see exactly what the agent inspected and how it reasoned about it. We could even train our own code review agent in the future given the team’s feedback.
Simple: One YAML file, that’s all you need.

We hope some day that these can even run on local models in the future. Some argue we are almost there. And we are all about local models at NobodyWho.

Why isn't Pi written in Lisp? Pi Durable in Erlang?

pielouNW — Mon, 19 Oct 2026 13:33:55 +0000

I'm a big fan of pi-coding-agent. I'm also a big fan of thinking about programming languages and their properties. When I combine the two, something feels dissonant.

Pi feels lisp-y

Since I first started using pi, I haven't been able to shake the hunch that it feels like something best written in lisp.

Why? Its goals align with the goals of lisp programs: minimal, radically extensible, self-modifiable and dynamic.

If you squint a little, emacs stans start to sound like pi stans.

Yeah you can do that in emacs. Emacs even includes the tools to change to emacs in that way! Just whip up the elisp repl and there you go!

Yeah you can do that in pi. Pi even includes the tools to change pi in that way! Just yeet a prompt like this and there you to!

I did hit a concrete snag when messing around with pi at one point: I was toying with the idea of storing the JSONL conversation history in the git tree (not as files, but as notes attached to the history). Eventually I convinced myself that this was not possible to do well without forking pi itself. Maybe this would have gone different in a homoiconic language?

Pi Durable feels BEAM-y

I had this deja-vu feeling again when reading the Pi Durable blogpost, but for a different technology this time.

It highlights a number of properties that come straight from the mouth of a Erlang/Elixir/Gleam/BEAM/OTP die-hard.

Earendil writes:

We want to change the code of a running agent without stopping it.

A tool call that is already running finishes on the code it started with

That allows the harness to run on one machine while its tools run on another. Your env function builds the environment for every tool call, from the conversation's working directory, so each conversation can run in a different place.

Installing an extension under a name that is already installed replaces it in one step.

If the process dies, a new process opens the same storage, finds the unfinished tasks, and continues each one from its last checkpoint.

Aborting a task aborts what it owns, bottom-up, so every task cleans up its own effects first, and a task only finishes once the work it owns has finished.

All of these sound like exactly features that are already implemented and battle-tested in BEAM/OTP. Is this an instance of Greenspun's Tenth Rule for Erlang?

Should LLMs write in mainstream languages?

A common argument is that "LLMs are just better at writing typescript". Does the agentic era imply that we should double-down on the most mainstream tech stacks?

I often think about Richard Feldman coping with having put years of his life into making Roc, a new programming language, just as LLMs were getting good.

At first he was distraught: Roc code is not well-represented in the LLMs' training dataset, and since LLMs are better at writing stuff they'd seen before, Roc would never grow popular enough to be represented in the dataset. Thus all new languages are doomed.

But then he did a 180: LLMs are better at writing in new languages than humans are. Just throw the docs and a few examples at the model, and it outputs good and valid code- accomplishing in minutes what would take an unassisted human days or weeks. LLMs are a great boon to niche technologies.

The Roc project then released a text file to throw at your coding agent and kept trucking.

It does matter which programming language your program is written in. A programming language carries with it some properties that transfer onto the program artifact. A program written in Rust simply is more likely to be memory safe than a program written in C, even if you tell your agent to make no mistakes.

While it is possible to write your program in an arbitrary language, in such a way that it implements the properties you want, this is exactly how you end up with Greenspun's Tenth Rule.

The point being: yes, while LLMs are better at writing typescript than other languages, they're also very good at writing in a language they've never seen before. Because your technological foundation does have a significant impact on the result, I believe the advantage of the softened learning curve far outweighs the LLM-familiarty advantage.

Back to Pi

Earendil does address this in their single-entry FAQ for Pi Durable:

Why TypeScript again?

Because it is the easiest way to bootstrap this. But as everybody knows by now, it's very easy to port everything to Rust or assembler. We're not ruling this out in the future, but at the moment we are focusing on TypeScript

I read this as "It's not an informed choice. We just went with the tool that we had at hand", and that's fine by me.

...and what about our code?

We write NobodyWho in Rust because it gives us the properties we want. It gives us confidence in correctness and reliability. It's highly portable with good cross-compilation for many targets. It has excellent FFI support (made legible via a strong macro system) which we use in two directions: binding downwards into llama.cpp for the juicy GGML acceleration, and binding upwards to the seven different programming languages you can use NobodyWho from.

People sometimes ask me if I had trouble finding people with sufficient Rust experience to recruit for this project. My answer is usually: "Why would they need Rust experience?"

NobodyWho vs RunAnywhere: On-Device Inference Engine Comparison

pielouNW — Fri, 09 Oct 2026 08:21:27 +0000

On paper, NobodyWho and RunAnywhere look almost identical. They run LLMs locally on consumer devices such as laptops and phones, are built on llama.cpp, and both list the same features across Kotlin, Swift, Python, Flutter and React Native. The differences only show up once you put them in the same app and start a real conversation. That's exactly what I did, side by side on iPhone and Android.

Here is a brief summary of the technical findings:

Speed: NobodyWho starts answering sooner on a single prompt, and RunAnywhere gets slower with every turn of a conversation, up to 33× slower after 20 turns on Android.
Multimodal: RunAnywhere accepts one image per request and no audio. NobodyWho lets you mix several images and audio files in one prompt.
Tool calling: RunAnywhere forgets previous tool results, so it can't answer follow-up questions about them.

Every result can be reproduced with my test app on GitHub.

Before getting to the benchmarks, let's start with an overview of both libraries: engine, model format and features, platform support and licensing.

Engine, model format and features

NobodyWho runs any GGUF model through llama.cpp, loaded straight from Hugging Face, a URL or a local path, with no conversion step.

RunAnywhere also uses llama.cpp, and can additionally register MLX and QHexRT (Qualcomm Hexagon NPU) backends. Registering them did not change any of my results below.

Both libraries offer hardware acceleration and a similar feature set: text generation, multimodal input, embeddings, RAG, speech-to-text, text-to-speech, structured output, voice activity detection and tool calling.

Platform support

NobodyWho and RunAnywhere both support Kotlin, Swift, Python, Flutter, React Native and Expo.

RunAnywhere also supports Electron and WebAssembly. NobodyWho has started work on WebAssembly, which will be available soon. NobodyWho also ships for Godot, and runs on Apple Vision Pro and Apple Watch.

Licensing

NobodyWho uses EUPL-1.2, an OSI-approved open-source licence, allowing proprietary and commercial projects to use the engine free of charge. However, if you distribute a modified version of NobodyWho itself, those engine changes must be open sourced.

RunAnywhere describes its licence as "RunAnywhere License (Apache 2.0 based, with additional commercial-use terms)." For companies, the free grant only applies to organizations with both "Less than $1,000,000 USD in total funding" and "Less than $1,000,000 USD in gross annual revenue." Organizations outside the listed criteria "must obtain a separate commercial license." If either threshold is exceeded later on, "a commercial license must be obtained within thirty (30) days."

In practice, NobodyWho stays free no matter how much funding your company raises or how much revenue it earns. RunAnywhere is free for a company only while it stays under $1M in total funding and $1M in annual revenue. Once it passes either figure, a paid commercial licence is required.

Technical comparison

To compare both engines under the same conditions, I added NobodyWho to the RunAnywhere React Native Starter App and ran both side by side on an iPhone Air and a Samsung S25. The comparison covers speed, multimodal input and tool calling.

I used the React Native SDKs, but none of these issues are specific to React Native. They come from RunAnywhere's inference engine and APIs, so they affect every platform and language it supports. You can check out the test app and run it on your own device to reproduce every result. Note that the RunAnywhere starter app needed a few fixes before it would build with Xcode 27, which you can find listed in the appendix at the end of this article.

Speed

The speed tests were done on both phones with the Qwen3 0.6B model and the same configuration.

On a single prompt, NobodyWho starts answering sooner on both phones, but RunAnywhere generates faster on Android:

iPhone Air: RunAnywhere generates 70.5 tokens per second (tok/s), with a time to first token (TTFT) of 201 ms. NobodyWho generates 70.8 tok/s, with a TTFT of 46 ms.
Samsung S25: RunAnywhere generates 52.4 tok/s, with a TTFT of 1268 ms (fluctuate between 300 and 1300 ms in my tests). NobodyWho generates 32.2 tok/s, with a TTFT of 230 ms.

The real problem appears in a conversation, where RunAnywhere's TTFT grows with every turn. Over 20 turns, it climbs from 147 ms to 471 ms on the iPhone Air (3× slower), and from 333 ms to over 11 seconds on the S25 (33× slower). NobodyWho stays between 33 ms and 60 ms on the iPhone Air, and between 218 ms and 896 ms on the S25. It means that after a few prompts, NobodyWho is answering faster, even on Android.

This happens because on every turn, RunAnywhere starts from an empty cache and re-processes the system prompt, the entire chat history and the new message. NobodyWho keeps the conversation in its KV cache and only processes the new message. With RunAnywhere, the longer the conversation, the slower the response.

NobodyWho currently shows the same growing TTFT issue with hybrid models like Qwen3.5, since the cache can't yet be reused between turns in the same way. A fix is in progress.

Multimodal input

Multimodal LLMs like Gemma 4 can take images and/or audio files along with a prompt. Let's see how each library handles them.

With RunAnywhere, you cannot send an audio file to the model, and you can only send one image at a time:

// node_modules/@runanywhere/core/src/Public/Api/Vlm.ts

export const vlm = {
  async generate(
    image: ImageInput,
    prompt: string,
    options?: LlmOptions
  ): Promise<GenerationResult> { ... },

  generateStream(
    image: ImageInput,
    prompt: string,
    options?: LlmOptions
  ): AsyncIterable<GenerationEvent> { ... },
};

In both generate and generateStream, the image is a required argument. Every follow-up question about the same image means sending it again and having the model process it again, instead of continuing the conversation naturally.

It is also not possible to interleave several images and audio files in a single prompt, which NobodyWho allows:

const response = await chat
  .ask(
    new Prompt([
      Prompt.Text("Tell me what you see in the images and what you hear in the audio."),
      Prompt.Image("/path/to/dog.png"),
      Prompt.Image("/path/to/cat.png"),
      Prompt.Audio("/path/to/sound.mp3"),
    ]),
  )
  .completed();

Tool calling

Tool calling lets your LLM call predefined functions when needed. For example, if you give the LLM a get_weather function, it will call it whenever the user asks about the weather.

Both libraries handle tool calling properly, but RunAnywhere does not keep previous tool calls and their results in the conversation history, so follow-up questions get answered incorrectly.

The screenshot above compares the official NobodyWho and RunAnywhere apps, both running Qwen3 4B. After a get_weather call, NobodyWho answers "What is the humidity?" from the earlier result, while RunAnywhere has forgotten it and asks for the location again.

Choosing between NobodyWho and RunAnywhere

The two libraries share the same feature list, but the differences become obvious once you use them. With RunAnywhere, the chat slows down with every message, the assistant forgets what its tools just returned, and you can only send one image at a time.

Requirement	Engine
OSI open-source licence with no revenue ceiling	NobodyWho
Wide platform and language support	Both
Fast multi-turn conversations	NobodyWho
Multimodal support	NobodyWho
Tool calling	NobodyWho

RunAnywhere is a good fit if you need Electron or WebAssembly support right now. For everything else, NobodyWho is the better choice for on-device AI: answers stay fast as conversations grow, tool results carry over to follow-up questions, and the licence stays free however big your company gets.

Appendix: building the RunAnywhere starter app with Xcode 27

The RunAnywhere starter app doesn't build out of the box with Xcode 27 on macOS 27. Both the starter app and the latest RunAnywhere SDK use older versions of React Native (0.83 and 0.85) instead of the current 0.87, and they conflict with Xcode 27's stricter compiler. To get it running, I fixed four issues in the Podfile (an outdated fmt C++ library, Swift 6 concurrency errors, an Objective-C class hidden from Swift and an undeclared static library), and a method called by the wrong name with a Yarn patch.

Info

This article was written on October 9, 2026. Results may have changed since then, on both the NobodyWho and RunAnywhere sides.
The @runanywhere dependencies in package.json are set to v0.20.19. Updating them to the latest published version, v0.20.27, did not change the results.

When Chat Templates Go Wrong

pielouNW — Thu, 01 Oct 2026 08:59:33 +0000

A model running locally loads cleanly and answers correctly, then keeps generating past its answer, stops following the system prompt, or drops an instruction it followed a turn earlier. The same model served through a hosted API behaves normally.

This behavior usually traces back to the chat template or the token metadata stored in the GGUF file.

An instruction-tuned model is trained on conversations written in one exact format. The chat template reproduces that format at inference, so the model receives the same layout it saw during training. When the rendered prompt deviates from it, the model is working from input it was never trained on, and its behavior degrades.

Most models ship a correct template, so the fault is usually in a specific file or setup, and it can be fixed without touching the weights.

The Chat Template

In that training format, control tokens mark where each turn begins and ends and which role is speaking. The wording inside a turn can vary, because that varies in training, but the control tokens and the structure around them are what the model keys on, and behavior degrades as that structure drifts from what the model saw in training.

The chat template defines that structure. It is a Jinja program that renders a list of messages into the single string the model receives, special tokens included, and it runs every time a message is added. These programs are not trivial, and the template Gemma ships with runs to roughly 250 lines. In a GGUF file the template is stored in the metadata under the tokenizer.chat_template key. (The wider tour of what else lives in GGUF metadata is in What's in a GGUF, besides the weights.)

A common layout, ChatML, structures a turn like this:

<|im_start|>system
You are a helpful assistant.<|im_end|>
<|im_start|>user
What is the capital of Denmark?<|im_end|>
<|im_start|>assistant

<|im_end|> is a single token with its own vocabulary id, which the model was trained to emit at the end of a turn and which the engine watches for to stop generation. The trailing <|im_start|>assistant line, with no content after it, is the generation prompt, the signal that the assistant's turn has begun, and without it the model has no indication that it should respond instead of continuing the preceding text.

Control tokens differ across model families, so Gemma marks turns one way and a ChatML-style model such as LFM2 or Qwen another, and a template written for one family does not align with the tokens another was trained on. Some models also ship more than one template, typically a plain one and a separate one for tool calling, and rendering the wrong variant changes the string with nothing to signal the substitution.

The template is a program that has to be run on every message, and the engine that runs it is not the same across tools. Hugging Face transformers uses jinja2, llama.cpp's server and CLI use their own Jinja implementation, and NobodyWho uses minijinja, a pure-Rust reimplementation.

Model authors write and test their templates against transformers, so Python's jinja2 is the reference. Templates often call Python string methods such as .strip(), or helpers that transformers adds such as raise_exception, and the other engines have to reproduce them. When they do, every engine renders the same string and the difference between them is speed. When an engine is missing something a template uses, the template fails to render or produces a different string, and the model is back to input it was never trained on. NobodyWho defines raise_exception itself and turns on minijinja's Python compatibility layer, which supplies string and dictionary methods such as .split() and .get().

A separate case is an older libllama call, llama_chat_apply_template, which ignores the model's embedded template and applies one of a small set of formats hardcoded in C++. This means that the template format in use is not the one the model shipped with, and a model outside that hardcoded set is rendered with a format that does not match it. The model can then ignore instructions or run past the end of a turn, even though the template stored in the file is correct.

This is also why the same model behaves correctly through a hosted API, where the serving stack applies the model's registered template and tokenizer and that layer is never exposed. Run locally, that layer is the caller's responsibility.

Invalid Token IDs

Special tokens are referenced by id, as indices into the token list at tokenizer.ggml.tokens, and the metadata records which entry is beginning-of-sequence, which is end-of-sequence, which ends a turn, and so on. When one of those ids is negative or points past the end of that list, the file still opens and passes surface inspection. Current llama.cpp catches it at load, logs a bad special token warning, and falls back to a default id, so the model runs with a special token the metadata never intended.

Silent Failures

When the token the model ends a turn on is not the token the engine treats as the stop signal, the end of the turn goes unrecognized and generation continues into the next one, with the model producing both sides of the conversation. Current llama.cpp recognizes common end-of-turn tokens such as <|im_end|> by name, even when the metadata names a different stop token. The run-on shows up with older engines, wrappers that stop only on the end-of-sequence id in the metadata, end-of-turn tokens outside the names llama.cpp checks, or files where the special tokens did not survive conversion.

A chat model's end-of-turn token often differs from the base end-of-sequence token, so a ChatML model stops on <|im_end|> rather than the model's plain <|endoftext|> token, and that end-of-turn token has to be recorded in the GGUF metadata as the token to stop on. When it is absent or wrong, an engine that stops only on the recorded token watches for a token the model does not emit at the end of a turn, and generation runs past the assistant's response. This was reported in llama.cpp issue #5040. A related conversion fault produces the same result: when the special tokens are not preserved as single tokens, the template emits the text <|im_end|> while the tokenizer splits that string into ordinary sub-word tokens, and the trained stop token never reaches the engine.

A missing generation prompt causes the model to continue the user's text instead of answering it. Incorrect whitespace around a role header tokenizes differently from the training format and lowers quality with no visible failure. Some fields do not survive conversion: the think_token separating a model's reasoning from its answer is frequently absent from GGUF conversions even when the upstream model defines it, and tool-call formats vary between families, so a template rendering them incorrectly produces calls the parser cannot read.

Checking a Template

To check a template, run three checks:

Render the final string the template produces immediately before tokenization, with the special tokens in place, and compare it against the format on the model card character for character.
Tokenize that string and confirm each special token resolves to a single dedicated id. A special token that splits into several ordinary tokens is why the model fails to stop.
Check what the run stopped on. A generation that ends at the token limit without emitting a stop token means the stop token is wrong or unregistered.

Diagnosing a Stop Token Mismatch

Take a ChatML model whose end-of-turn token, <|im_end|>, is not the model's base end-of-sequence token and is not recorded in the GGUF metadata as the token to stop on, running on an engine that stops only on the token recorded in the metadata. A single question returns a correct answer, then generation continues: the model writes a follow-up as the user, answers it, and halts only at the token limit. No error is raised and the transcript reads fluently.

The rendered prompt matches the model card, so the template text is correct. The turn markers tokenize to single ids, so the vocabulary is intact. Generation stopped on the length limit without the engine recognizing a stop token, and the stop token recorded in the metadata is the base end-of-sequence token rather than <|im_end|>. The weights are correct, so the fix is a metadata edit: set the stop token to <|im_end|>, and the same prompt answers once and stops.

Correcting the Metadata

Each of these faults is a small edit, often a single field, yet correcting one has until recently required either re-quantizing the model or running ad-hoc Python scripts against the file, a change of a few bytes in an otherwise correct header. gguf-surgeon opens the metadata block in a GGUF, reads and rewrites fields in place including the chat template and the token ids, and writes the file back with the tensor data byte-identical, so the model loads exactly as before.

An out-of-range id can be caught before publication with a single check:

gguf check model.gguf

It reports the offending key and that its id is out of range, in the wording of its validator: token id N is out of range (tokens array has M elements). Correcting a token id is one command, because set parses the value as the existing key's type. Replace 106 with the id of the correct token in the model's own vocabulary:

gguf set model.gguf tokenizer.ggml.eos_token_id 106 -y

Editing the template is a matter of reading it out, changing it, and writing it back:

gguf get model.gguf tokenizer.chat_template > template.j2
$EDITOR template.j2
gguf set model.gguf tokenizer.chat_template @template.j2 -y

In each case the tensor data is untouched and the file continues to load in llama.cpp.

NobodyWho vs Ollama: When to Use Each

pielouNW — Mon, 28 Sep 2026 09:31:42 +0000

NobodyWho and Ollama both run language models locally, without a cloud API, so they are often compared as alternatives. They are built for different jobs. Ollama runs a model on a machine you control. NobodyWho embeds a model inside an application that runs on your users' devices. Both build on the same foundation, so the decision comes down to where the model runs and who runs it.

Built on llama.cpp

Both trace back to llama.cpp, the open-source engine that made it practical to run language models on ordinary hardware, and both read the GGUF model format. Ollama began as a Go server wrapping llama.cpp and now runs its own ggml-based engine for multimodal and other newer models. NobodyWho builds on llama.cpp directly and contributes its fixes upstream. Because the inference layer under both is the same, the choice between them is about deployment: where the model needs to run, and who runs it.

Ollama

Ollama runs a model on a machine you administer, your own workstation or a server.

Installs as a background service exposing an HTTP API on 127.0.0.1:11434, with a CLI for pulling and running models.
Manages the model lifecycle: downloading, loading, and keeping a model resident between requests.
Offers an OpenAI-compatible API route, so existing OpenAI client code works against a local model with almost no change.
Pulls from its own model registry, and imports a GGUF file directly.
Runs on macOS, Linux, and Windows, with GPU acceleration on Nvidia, AMD, Apple, and Intel hardware.

It fits work where the model and its caller sit on infrastructure you control: evaluating several models locally, backing a script or notebook with a local endpoint instead of a paid API, or self-hosting inference for tools on your own network.

NobodyWho

NobodyWho embeds a model inside an application you ship to other people, so the model runs on the user's own hardware.

Compiles into the app as a library, with no separate service or daemon to install.
Runs in the app's own process, on the end user's device, fully offline.
Ships as a package for Godot, Flutter, React Native, Swift, Kotlin, and Python.
Loads plain GGUF files from Hugging Face or any URL.
Runs on Android, iOS, and desktop, down to hardware as small as an Apple Watch.
Keeps user data on the device, since inference never leaves it.

You cannot bundle a daemon like Ollama inside a mobile app or an app-store game and require every user to install and run it, which is the gap NobodyWho fills. An on-device assistant in a mobile app, an NPC in a Godot game, or a desktop tool that works with no network all fit here.

Using Them Together

The two commonly appear in one project at different stages. Ollama is a fast way to evaluate models during development, comparing candidates from your own machine. Once a model is chosen and the work turns to shipping it inside an application that runs on your users' devices, NobodyWho is what embeds it. Running a model for yourself and shipping one to your users are different questions, and a project often answers both.

NobodyWho vs Cactus compared on engine design, model format, hardware, platforms, cloud, and licensing.

pielouNW — Wed, 16 Sep 2026 18:35:15 +0000

Choosing an on-device inference engine sets where the model runs and under what terms. NobodyWho and Cactus both run models on the user's device, with no API key, no per-request cost, and no data leaving the hardware once the model is downloaded. Underneath, they are different engines. NobodyWho Edge is built on llama.cpp and runs GGUF models. Cactus is a from-scratch engine with its own quantization format, and positions itself as a llama.cpp alternative. This is a technical comparison of NobodyWho vs Cactus across engine and model format, hardware, installation, platform coverage, cloud behaviour, and licensing. The better fit depends on the target device and the commercial model of the product.

Engine and model format

NobodyWho runs GGUF models through llama.cpp. It loads any GGUF file from Hugging Face or a URL directly, with no conversion step, so any model already published in GGUF, at any of the quantization levels the ecosystem provides, runs as-is.

Cactus runs its own format. Cactus Quants (CQ) is a rotation-and-codebook quantization applied to every weight tensor, from 4-bit down to 1-bit, and the engine runs from CQ bundles. You get a bundle by downloading one Cactus has pre-built for its own model catalog (cactus download) or by converting a source model yourself (cactus convert), which Cactus documents as experimental for models it has not pre-built.

NobodyWho runs the existing GGUF catalog with no conversion step. Cactus's CQ is tuned in-house for on-device size and quality, with published accuracy tables per bit-width, at the cost of depending on its own bundles or an experimental conversion for anything outside its catalog.

Both also cover the parts of an app beyond chat: embeddings, speech-to-text, text-to-speech, and tool calling. NobodyWho generates the tool-calling grammar from your function signatures and constrains generation to it, so you pass plain functions and the output conforms to the expected structure without writing schemas by hand. Cactus takes OpenAI and MCP-style tool definitions, ships a vector index for retrieval, and provides Needle, a 26M-parameter model dedicated to tool calling.

Hardware acceleration

Cactus is built around the mobile processor. Its kernels are hand-written in ARM NEON SIMD for the CPU, with Metal on Apple GPUs and the Apple Neural Engine for some vision models. Qualcomm, MediaTek, and Exynos NPU support is on its roadmap, not shipped. Cactus publishes per-device benchmarks in its README, listing tokens per second and peak RAM for named iPhone and Mac hardware.

NobodyWho takes a hardware-aware approach to acceleration, using Vulkan and Metal for GPU execution rather than targeting the NPU. This is the sharpest hardware split between them. On older or low-end phones that lean on the CPU, Cactus's hand-written kernels are the advantage. On devices with a capable GPU, NobodyWho runs on the accelerator most platforms already expose.

Installation and first call

NobodyWho installs through each ecosystem's own package manager: pip install nobodywho for Python, flutter pub add nobodywho for Flutter, npm install react-native-nobodywho for React Native, ai.nobodywho:nobodywho from Maven Central for Kotlin, Swift Package Manager for Swift, and the in-editor asset library for Godot. Each binding is a thin wrapper over one shared Rust core.

Cactus installs its engine by cloning the repository and running a setup script, then building the binding and downloading a model bundle:

git clone https://github.com/cactus-compute/cactus && cd cactus && source ./setup 
cactus build --python 
cactus download LiquidAI/LFM2-VL-450M

Cactus also publishes per-platform packages on pip, Maven, pub, and npm.

The difference carries into the first call. NobodyWho's Python API handles model loading and lifecycle for you, and returns the reply as a string:

from nobodywho import Chat

chat = Chat("hf:NobodyWho/Qwen_Qwen3-0.6B-GGUF/Qwen_Qwen3-0.6B-Q4_K_M.gguf") 
response = chat.ask("What is the capital of Denmark?").completed() 
print(response)

Cactus's Python binding is a ctypes FFI over its C engine, with explicit model lifecycle and JSON message payloads:

from cactus import ensure_model, cactus_init, cactus_complete, cactus_destroy
import json

bundle = ensure_model("LiquidAI/LFM2-VL-450M")
model = cactus_init(str(bundle), None, False)
messages = json.dumps([{"role": "user", "content": "What is the capital of Denmark?"}])
result = cactus_complete(model, messages, None, None, None)
print(result["response"])
cactus_destroy(model)

Both snippets are each project's own documented quick-start.

Platform support

Cactus targets mobile and embedded. Its C and C++ core runs on wearables, smart-home devices, robots, and Raspberry Pi, with bindings for Flutter, React Native, Kotlin Multiplatform, Swift, Python, and Rust.

NobodyWho targets mobile, desktop (Linux, macOS, Windows), Python, and the JVM, and ships a binding for the Godot game engine.

Both engines cover phones and wearables, and Cactus reaches further into embedded hardware like robots and Raspberry Pi. Cactus also supports desktop deployment on macOS and ARM Linux. NobodyWho ships a Godot binding and targets desktop app runtimes across Linux, macOS, and Windows through its JVM and Python bindings. Cactus has no game-engine binding. Neither ships a browser or WebAssembly target. NobodyWho has an open GitHub issue tracking WASM export, and Cactus does not target the browser.

Cloud behaviour

Both run inference locally by default. Cactus adds an optional cloud handoff that routes low-confidence queries to a hosted model. NobodyWho has no cloud router in the engine. In deployments that prohibit outbound network calls, NobodyWho has nothing to disable, and Cactus's handoff must be turned off and verified.

Licensing

NobodyWho uses EUPL-1.2, an OSI-approved open-source licence. It permits proprietary and commercial use with no revenue limit. The only obligation is that redistributing a modified version of the engine requires publishing those engine changes.

Cactus is source-available under its own licence. That licence grants free use to individuals, students, non-profits, and organizations with "Less than $2,000,000 USD in total funding" and "Less than $2,000,000 USD in gross annual revenue." An organization that does not meet those criteria "must obtain a separate commercial license," and a qualifying organization that later crosses either threshold has "thirty (30) days" to obtain one. Both licences were verified on 9 September 2026. Cactus has revised its terms before, so check the current file before shipping a commercial product.

Both engines are free under those thresholds. Once a company crosses either one, Cactus requires a paid commercial licence and NobodyWho stays free.

Choosing between NobodyWho and Cactus

The decision follows target hardware and commercial model.

Requirement	Engine
Run any existing GGUF model with no conversion step	NobodyWho
In-house quantization tuned for on-device size and quality	Cactus
Hand-optimized CPU performance on ARM without a capable GPU	Cactus
Robotics, smart-home, or Raspberry Pi	Cactus
Desktop applications alongside mobile	NobodyWho
A model running locally inside a Godot game	NobodyWho
OSI open-source licence with no revenue ceiling	NobodyWho

The two are different engines with different model formats, so the choice is set by where the model has to run and how the licence treats the product, not by a shared core. The NobodyWho source and per-binding docs list current platform and format support.

On-Device AI in Flutter

pielouNW — Wed, 02 Sep 2026 13:50:47 +0000

In this Flutter tutorial, you'll learn how to run a large language model (LLM) directly on a user's device: no server, no API key needed. We'll start from scratch with a simple chat exchange, and progressively introduce more advanced features: multimodal input, speech-to-text, text-to-speech, voice activity detection, tool calling and RAG.

Each concept is explained before the code, so you can follow along whether you're new to on-device AI.

Why run AI On-Device?

Most AI features rely on a cloud API: you send a request to a remote server, it runs the model, and sends a response back. That works well, but it comes with tradeoffs.

Running the model directly on the device avoids all of them:

Works offline — no internet connection required
Privacy by design — user data never leaves the device
Low latency — no network round-trip
No cloud costs — inference is free

The tradeoff is raw capability: on-device models are smaller and less powerful than frontier cloud models. But for many use cases like summarization, chatbots, or local search, they're more than good enough.

About NobodyWho

We'll use the NobodyWho library throughout this tutorial. It wraps llama.cpp in Rust and ships bindings for several languages and frameworks: Kotlin, Python, Expo/React Native, Swift, Flutter, and Godot. It exposes a clean Dart API for running any model locally in .gguf format, on iOS and Android.

Add it with:

flutter pub add nobodywho

Import it under the nobodywho namespace, since the package uses common names like Model and Chat that could otherwise collide with your own code:

import 'package:nobodywho/nobodywho.dart' as nobodywho;

Before calling anything else, initialize the native bindings exactly once in main before runApp():

await nobodywho.NobodyWho.init();

Loading a Model

NobodyWho can download a GGUF model for you directly from Hugging Face, cache it, and reuse it on every subsequent launch. That means you don't need to bundle anything into your app or manage downloads yourself:

final chat = await nobodywho.Chat.fromPath(
  modelPath: 'huggingface:NobodyWho/Qwen_Qwen3-0.6B-GGUF/Qwen_Qwen3-0.6B-Q4_K_M.gguf',
);

The first time this runs, the model is downloaded to the platform cache directory. Every call after that loads the model directly.

modelPath accepts a few different forms:

Form	Example	Notes
HuggingFace reference	`hf:owner/repo/file.gguf`	Downloaded and cached on first use
HTTPS URL	`https://example.com/model.gguf`	Downloaded and cached on first use
Local path	`./model.gguf`	Used as-is, no download

The HuggingFace prefix is case-insensitive and the // is optional, so hf:, hf://, huggingface:, and huggingface:// are all equivalent. You can also pass "auto" to let NobodyWho pick a chat model based on the device's available memory, which is a handy default if you don't want to think about model selection at all.

You can track a remote download by passing onDownloadProgress to Chat.fromPath:

final chat = await nobodywho.Chat.fromPath(
  modelPath: 'huggingface:NobodyWho/Qwen_Qwen3-0.6B-GGUF/Qwen_Qwen3-0.6B-Q4_K_M.gguf',
  onDownloadProgress: (downloaded, total) {
    print('$downloaded / $total bytes');
  },
);

You can find thousands of LLMs in .gguf format on Hugging Face here.

Basic Chat

With a model loaded, you're ready to start a conversation:

final chat = await nobodywho.Chat.fromPath(
  modelPath: 'huggingface:NobodyWho/Qwen_Qwen3-0.6B-GGUF/Qwen_Qwen3-0.6B-Q4_K_M.gguf',
);
final response = await chat.ask('Is water wet?').completed();
print(response); // Yes, indeed, water is wet!

chat.ask() sends your message and returns a TokenStream. Calling .completed() waits for the whole response and gives you back the final string, which is fine for a one-off question. But a real chat interface needs to stream tokens as they arrive, otherwise users stare at a blank screen until generation finishes.

Streaming Tokens

final response = chat.ask('What is the capital of Denmark?');

await for (final token in response) {
  print(token);
}

A token is the smallest unit a model generates, typically a word, or a fragment of a word.

Multimodal Models

Some models can natively ingest images and audio. To use them, you need two things: a multimodal LLM, and its projection model that converts images and/or audio into tokens the LLM can consume (usually named with mmproj in it). A solid default that handles both image and audio is Gemma 4 with its BF16 projection model.

final model = await nobodywho.Model.load(
  modelPath: "./vision-model.gguf",
  projectionModelPath: "./mmproj.gguf",
);
final chat = nobodywho.Chat(model: model);

To actually send image or audio content, build a Prompt mixing text, images, and audio, and pass it to chat.askWithPrompt() instead of a plain string:

final response = await chat.askWithPrompt(nobodywho.Prompt([
  nobodywho.TextPart("Tell me what you see in the image and what you hear in the audio."),
  nobodywho.ImagePart("./dog.png"),
  nobodywho.AudioPart("./sound.mp3"),
])).completed();

Keep in mind that images and audio consume context fast, so you'll likely want a bigger contextSize than you'd use for text-only chat. Also note that the language model and its projection model have to be trained together — you can't mix an LLM and a projection model you happen to like and expect them to work.

Speech to Text

If you'd rather transcribe spoken audio into text than have the model listen to it directly, NobodyWho integrates Whisper models in ONNX format through SpeechToText.

final stt = await nobodywho.SpeechToText.load(source: 'hf://onnx-community/whisper-base');

final text = await stt.transcribeFile('recording.mp3').completed();
print(text);

source is a Hugging Face repo (hf://owner/repo) or a local directory laid out the same way. Browse the Whisper ONNX models on Hugging Face to find one that fits your accuracy and speed needs.

If your audio comes from a buffer rather than a file, use transcribePcm:

final text = await stt.transcribePcm(samples, 16000).completed();

The buffer needs to be mono i16 PCM samples. The sample rate can be anything, NobodyWho resamples internally to what Whisper expects. And just like chat, transcription can be streamed piece by piece instead of waiting for the full result:

import 'dart:io';

await for (final piece in stt.transcribeFile('recording.mp3')) {
  stdout.write(piece);
}

Text to Speech

Going the other direction, TextToSpeech turns text into WAV audio you can play back or save.

import 'dart:io';

final tts = await nobodywho.TextToSpeech.load(
  source: 'hf://NobodyWho/Kokoro-82M',
  voice: 'bf_emma',
  language: 'en-gb',
);

final wav = await tts.synthesize(text: 'Hello from NobodyWho!');
await File('out.wav').writeAsBytes(wav);

Three architectures are supported, all ONNX-based: Kokoro, Pocket TTS, and Supertonic. NobodyWho infers which one you're using from the source string, so you only need to set architecture explicitly when loading from a custom local folder.

Each architecture has its own voice and language options that need to agree with what the model supports.

Voice Activity Detection

Before transcribing audio, it helps to know when someone is actually speaking rather than relying on a fixed silence timeout. VoiceActivityDetection uses a small model to reliably tell speech and silence apart, and pairs naturally with SpeechToText.

For streaming microphone input, push chunks in as they arrive:

final vad = await nobodywho.VoiceActivityDetection.load(
  sampleRate: 16000,
  source: 'hf://onnx-community/silero-vad',
);
final stt = await nobodywho.SpeechToText.load(source: 'hf://onnx-community/whisper-base');

while (true) {
  final chunk = readMic();
  if (vad.push(chunk) == nobodywho.VoiceActivityDetectionEvent.speechEnded) break;
}

final speech = vad.finish();
final transcription = await stt.transcribePcm(speech, 16000).completed();
print(transcription);

Each push() call reports the current state (speechStarted, speechEnded, speech, or silence), and finish() hands you back the buffered speech segment while resetting internal state for the next turn.

If you already have a full recording and just want to pull out the speech segments from it, segment() does that in one pass:

final audio = readWavPcm('recording.wav');

for (final speech in vad.segment(audio)) {
  final transcription = await stt.transcribePcm(speech, 16000).completed();
  print(transcription);
}

Sensitivity is tunable via threshold, minSpeechDurationMs, minSilenceDurationMs, and prerollDurationMs (how much audio to keep before the detected start, so you don't clip the beginning of a sentence). The defaults are a reasonable starting point, but VAD is one of those things that usually benefits from tuning to your actual environment.

Tool Calling

Tools let the model call out to real functions in your app rather than just generating text. Any Dart function that returns a String or Future<String> becomes a tool by wrapping it in Tool() with a name and description:

import 'dart:math' as math;

final circleAreaTool = nobodywho.Tool(
  name: "circle_area",
  description: "Calculates the area of a circle given its radius",
  function: ({ required double radius }) {
    final area = math.pi * radius * radius;
    return "Circle with radius $radius has area ${area.toStringAsFixed(2)}";
  },
);

final chat = await nobodywho.Chat.fromPath(
  modelPath: 'huggingface:NobodyWho/Qwen_Qwen3-0.6B-GGUF/Qwen_Qwen3-0.6B-Q4_K_M.gguf',
  tools: [circleAreaTool],
);

NobodyWho also ships two general-purpose tools out of the box, a Python interpreter and a Bash interpreter, for models that need to reason precisely or compute something:

final chat = await nobodywho.Chat.fromPath(
  modelPath: 'huggingface:NobodyWho/Qwen_Qwen3-0.6B-GGUF/Qwen_Qwen3-0.6B-Q4_K_M.gguf',
  tools: [nobodywho.Tool.python(), nobodywho.Tool.bash()],
);

Not every model supports tool calling well, the Qwen family is a solid choice if you need it to be reliable. See the Tool Calling documentation for more.

RAG

Retrieval-Augmented Generation combines document search with LLM generation, so the model grounds its answers in your own knowledge base instead of what it happened to learn during training. NobodyWho provides an Encoder for embeddings and a CrossEncoder for reranking:

final encoder = await nobodywho.Encoder.fromPath(modelPath: './embedding-model.gguf');

final queryEmbedding = await encoder.encode(text: "What is the return policy?");
final docEmbeddings = await encoder.encodeBatch(texts: knowledge);
final similarities = docEmbeddings.map((doc) => nobodywho.cosineSimilarity(
  a: queryEmbedding.toList(),
  b: doc.toList(),
)).toList();

For better precision, rerank the top candidates with a CrossEncoder before handing them to the model:

final crossencoder = await nobodywho.CrossEncoder.fromPath(modelPath: './reranker-model.gguf');
final ranked = await crossencoder.rankAndSort(query: "What is the return policy?", documents: topDocs);

See the Embeddings & RAG documentation for the full walkthrough, including how to wire this up as a tool the model calls automatically.

What's Next?

You now have a complete foundation for building on-device AI features in Flutter:

Download and run a GGUF model
Send messages and get streamed tokens back
Feed images and audio directly into a multimodal model
Transcribe speech, synthesize it back, and detect when someone's actually talking
Extend the model with tool calling and perform search with RAG

You can also have a look at the Flutter starter example to see a full implementation.

Links

Github - pub.dev - Docs

On-Device AI in Python

pielouNW — Wed, 02 Sep 2026 11:26:56 +0000

In this Python tutorial, you'll learn how to run a large language model (LLM) directly on a user's device: no server, no API key needed. We'll start from scratch with a simple chat exchange, and progressively introduce more advanced features: multimodal input, speech-to-text, text-to-speech, voice activity detection, tool calling and RAG.

Each concept is explained before the code, so you can follow along whether you're new to on-device AI.

Why run AI On-Device?

Most AI features rely on a cloud API: you send a request to a remote server, it runs the model, and sends a response back. That works well, but it comes with tradeoffs.

Running the model directly on the device avoids all of them:

Works offline — no internet connection required
Privacy by design — user data never leaves the device
Low latency — no network round-trip
No cloud costs — inference is free

About NobodyWho

We'll use the NobodyWho library throughout this tutorial. It wraps llama.cpp in Rust and ships bindings for several languages and frameworks: Kotlin, Python, Expo/React Native, Swift, Flutter & Godot. It exposes a clean API for running any model locally in .gguf format.

Install it with:

pip install nobodywho

Or, preferably:

uv add nobodywho

Loading a Model

NobodyWho can download a GGUF model for you directly from Hugging Face, cache it, and reuse it on every subsequent launch. That means you don't need to manage downloads yourself:

from nobodywho import Chat

chat = Chat('huggingface:NobodyWho/Qwen_Qwen3-0.6B-GGUF/Qwen_Qwen3-0.6B-Q4_K_M.gguf')

The first time this runs, the model is downloaded to the platform cache directory. Every call after that loads the model directly.

model_path accepts a few different forms:

Form	Example	Notes
HuggingFace reference	`hf:owner/repo/file.gguf`	Downloaded and cached on first use
HTTPS URL	`https://example.com/model.gguf`	Downloaded and cached on first use
Local path	`./model.gguf`	Used as-is, no download

The HuggingFace prefix is case-insensitive and the // is optional, so hf:, hf://, huggingface:, and huggingface:// are all equivalent.

You can track a remote download by calling download_model directly and passing an on_download_progress callback. It receives (downloaded_bytes, total_bytes) and is skipped for cached or local files — if you don't pass anything, NobodyWho prints a default terminal progress bar:

from nobodywho import download_model, Chat

model_path = download_model(
    'huggingface:NobodyWho/Qwen_Qwen3-0.6B-GGUF/Qwen_Qwen3-0.6B-Q4_K_M.gguf',
    on_download_progress=lambda downloaded, total: print(f"{downloaded}/{total} bytes"),
)
chat = Chat(model_path)

You can find thousands of LLMs in .gguf format on Hugging Face here.

Basic Chat

With a model loaded, you're ready to start a conversation:

from nobodywho import Chat

chat = Chat('huggingface:NobodyWho/Qwen_Qwen3-0.6B-GGUF/Qwen_Qwen3-0.6B-Q4_K_M.gguf')
response = chat.ask('Is water wet?').completed()
print(response)  # Yes, indeed, water is wet!

chat.ask() sends your message and returns a TokenStream. Calling .completed() blocks until the model is done generating and gives you back the final string, which is fine for a one-off question. But a real chat interface needs to stream tokens as they arrive, otherwise users stare at a blank screen until generation finishes.

Streaming Tokens

response = chat.ask('What is the capital of Denmark?')

for token in response:
    print(token, end="", flush=True)

A token is the smallest unit a model generates, typically a word, or a fragment of a word.

If you'd rather not block synchronously while a model loads or generates, swap Chat for ChatAsync — the API stays the same, so await chat.ask(...).completed() or async for token in chat.ask(...) both work.

Multimodal Models

from nobodywho import Model, Chat

model = Model("./vision-model.gguf", projection_model_path="./projection_model.gguf")
chat = Chat(model, system_prompt="You are a helpful assistant, that can hear and see stuff!")

To actually send image or audio content, build a Prompt mixing text, images, and audio, and pass it to chat.ask() instead of a plain string:

from nobodywho import Audio, Image, Prompt, Text

prompt = Prompt([
    Text("Tell me what you see in the image and what you hear in the audio."),
    Image("./dog.png"),
    Audio("./sound.mp3"),
])
response = chat.ask(prompt).completed()

Keep in mind that images and audio consume context fast, so you'll likely want a bigger n_ctx than you'd use for text-only chat. Also note that the language model and its projection model have to be trained together — you can't mix an LLM and a projection model you happen to like and expect them to work.

Speech to Text

If you'd rather transcribe spoken audio into text than have the model listen to it directly, NobodyWho integrates Whisper models in ONNX format through SpeechToText.

from nobodywho import SpeechToText

stt = SpeechToText(source="hf://onnx-community/whisper-base")

text = stt.transcribe_file("recording.mp3").completed()
print(text)

source is a Hugging Face repo (hf://owner/repo) or a local directory laid out the same way. Browse the Whisper ONNX models on Hugging Face to find one that fits your accuracy and speed needs.

If your audio comes from a buffer rather than a file, use transcribe_pcm:

text = stt.transcribe_pcm(samples, sample_rate=16000).completed()

for piece in stt.transcribe_file("recording.mp3"):
    print(piece, end="", flush=True)

Text to Speech

Going the other direction, TextToSpeech turns text into WAV audio you can play back or save.

from pathlib import Path
from nobodywho import TextToSpeech

tts = TextToSpeech(
    source="hf://NobodyWho/Kokoro-82M",
    voice="bf_emma",
    language="en-gb",
)

wav = tts.synthesize("Hello from NobodyWho!")
Path("out.wav").write_bytes(wav)

Each architecture has its own voice and language options that need to agree with what the model supports.

Voice Activity Detection

For streaming microphone input, push chunks in as they arrive:

from nobodywho import VoiceActivityDetection, VoiceActivityDetectionEvent, SpeechToText

vad = VoiceActivityDetection(source="hf://onnx-community/silero-vad", sample_rate=16000)
stt = SpeechToText(source="hf://onnx-community/whisper-base")

while chunk := read_mic():
    if vad.push(chunk) == VoiceActivityDetectionEvent.SpeechEnded:
        break

speech = vad.finish()
transcription = stt.transcribe_pcm(speech, sample_rate=16000).completed()
print(transcription)

Each .push() call reports the current state (SpeechStarted, SpeechEnded, Speech, or Silence), and .finish() hands you back the buffered speech segment while resetting internal state for the next turn.

If you already have a full recording and just want to pull out the speech segments from it, .segment() does that in one pass:

audio = read_wav_pcm("recording.wav")

for speech in vad.segment(audio):
    transcription = stt.transcribe_pcm(speech, sample_rate=16000).completed()
    print(transcription)

Sensitivity is tunable via threshold, min_speech_duration_ms, min_silence_duration_ms, and preroll_duration_ms (how much audio to keep before the detected start, so you don't clip the beginning of a sentence). The defaults are a reasonable starting point, but VAD is one of those things that usually benefits from tuning to your actual environment.

Tool Calling

Tools let the model call out to real functions in your app rather than just generating text. Any synchronous function that returns a string becomes a tool with the @tool decorator:

import math
from nobodywho import tool, Chat

@tool(description="Calculates the area of a circle given its radius")
def circle_area(radius: float) -> str:
    area = math.pi * radius ** 2
    return f"Circle with radius {radius} has area {area:.2f}"

chat = Chat('./model.gguf', tools=[circle_area])

NobodyWho inspects the function's name, parameter names, and types to describe the tool to the model — add a params dict to @tool if a parameter needs more explanation than its name alone gives.