How can an AI answer with no network? On-device AI explained

On-device AI answers with no network because the model executes on your own machine: its weights, tokenizer and config are files on the disk, and a runtime such as llama.cpp does the math on the local chip. What it cannot do offline is reach anything live, so current information must come from a supplied document or tool.

Definitions {#definitions}

Definition

Local inference: executing the model on the user's device. Keeping prompts on the device also depends on the application's networking and storage behavior.

Definition

Weights: the learned numbers of the model, almost all of its size.

Definition

Quantization: storing each weight in fewer bits (about 4 to 5 instead of 16), so the model takes much less space.

How does on-device AI answer without a network? {#files}

For a typical text model, GGUF can package the weights, tokenizer data and architecture metadata in one file. The weights are almost all of the bytes; the tokenizer turns text into token IDs and back; the metadata gives layers, attention heads and context length. The GGUF specification used by llama.cpp says it aims to contain "all the information needed to load a model", and it also allows a large model to be split into shards.

The runtime loads that file, tokenizes your prompt and runs the layers on the CPU or GPU. None of those steps needs a remote server. A local application can also use RAG retrieval to select document passages before passing them to the model.

A first estimate of the size counts only the weights:

weight GB = parameters × average bits per weight / 8 / 1e9
small: 1.5e9 × 4.85 / 8 / 1e9 = 0.909375 GB ≈ 0.91 GB
8B:    8e9 × 4.85 / 8 / 1e9 = 4.85 GB

The real GGUF file is a little larger because of the tokenizer, metadata and alignment.

How fast can a laptop write an answer? {#speed}

There are two phases. Prefill processes the prompt tokens in parallel, often in chunks. Decode then writes one token at a time, which is why replies stream in word by word. For a dense model generating one sequence, decode is often limited by memory bandwidth when the weights are larger than the processor cache, because each new token has to stream the weights through the chip. A rough estimate is effective bandwidth divided by weight bytes. It leaves out attention-cache traffic and runtime overhead.

The worked numbers below assume that regime: dense model, batch size one, weights in RAM.

tokens/s ≈ bandwidth x efficiency / weight bytes
small: 120 GB/s x 0.65 / 0.91 GB ≈ 86 tokens/s ≈ 64 words/s  (0.75 words per token)
8B:    120 GB/s x 0.65 / 4.85 GB ≈ 16 tokens/s ≈ 12 words/s

The simulation prints 65.4 and 12.2 words/s after applying seeded timing variation, rounding assumed word counts and averaging per-prompt rates. These speeds are hypothetical; we have not measured a real model yet. Outside this regime the ratio changes. Batching shares one read of the weights across many sequences, mixture-of-experts models read only some weights per token, and long contexts add cache reads. A model five times smaller is not guaranteed to be five times faster. The llama.cpp performance tips also show large effects from GPU offloading and thread settings.

How much memory does it need? {#memory}

Memory also has to hold the attention (KV) cache, which stores keys and values for every token in the context window. Here kv_heads is the number of key/value heads and head_dim is the size of each head; the formula assumes 2-byte elements and one sequence.

KV cache = layers x 2 (K and V) x kv_heads x head_dim x 2 bytes x context
small: 28 x 2 x 2 x 128 x 2 x 4096 = 0.12 GB  -> subtotal 0.91 + 0.12 + 0.3  = 1.33 GB
8B:    32 x 2 x 8 x 128 x 2 x 4096 = 0.54 GB  -> subtotal 4.85 + 0.54 + 0.55 = 5.94 GB

The last term is an assumed working buffer, so these are estimated model, cache and buffer subtotals. Both are below 16 GB; actual memory use needs measurement. An idealized disk-transfer time is weight bytes divided by disk speed: 0.3 s and 1.62 s at an assumed 3 GB/s. That is not the full time until the model is ready.

What our lab shows, and what it does not {#lab}

The ai-16 lab patches seven Node network functions so any call throws before a socket opens, checks that all 7 throw, then sends 20 prompts through a test cloud client. The cloud client got 0 of 20 replies. That shows what a cut network does to an app that depends on a remote model, inside one process.

The same script also prints results for two "local models", but those are preset strings, not inference, so we leave them out here. The lab tests a JavaScript interceptor, which says little about offline AI itself. The real test is a GGUF model in llama.cpp with outbound traffic blocked by the operating system, recording the model, quantization, runtime version, command, output and a connection monitor showing zero attempts. We have not run that test yet.

The core of the interceptor:

// block.mjs: make network calls fail before a socket exists (Node 22)
import net from 'node:net';
const log = [];
function block(kind, target) {
  log.push({ kind, target });
  const err = new Error(`blocked ${kind} -> ${target}`);
  err.code = 'ENETUNREACH';
  throw err;
}
globalThis.fetch = async (url) => block('fetch', String(url));
net.connect = net.createConnection = (opts) => block('net.connect', `${opts.host}:${opts.port}`);

try { await fetch('https://probe.invalid/'); } catch (e) { console.log(e.code); } // ENETUNREACH
console.log(log.length, 'call(s) blocked');

What can an offline model not know? {#limits}

An offline model cannot independently observe new external events. It can reason from its learned patterns and supplied information, but checking this morning's headline requires a current source, such as a document you give it or a tool the app connects.

The date works the same way. An application can read the device clock, which needs no network, and put today's date into the prompt. You can also type the date yourself, or open a local file that contains it.

Trade-offs {#trade-offs}

Speed and RAM figures below are the estimates above, for the assumed configurations only.

Option	Good for	Costs you	Use when
Small local model (about 0.9 GB of weights)	Low RAM (about 1.3 GB estimated), can run without a remote inference service	Fewer parameters	Short tasks on modest hardware
Mid local model (about 4.9 GB of weights)	More parameters, can run without a remote inference service	Slower estimated decode, about 5.9 GB estimated RAM	A machine with RAM to spare
Cloud model	Can run models beyond the device's memory capacity	Needs network; your prompt goes to a remote service	Local memory or compute is insufficient

Our take {#our-take}

The size and memory formulas are a sound first check before you download a model: parameters times bits per weight gives the weight bytes, and the KV cache and buffers come on top. The speed formula is rougher. It holds only for dense decode of one sequence with weights in RAM, and it ignores cache traffic and runtime overhead. We have not measured speed or answer quality on a real model yet, so treat 86 and 16 tokens/s as estimates for one assumed laptop until our OS-level offline run is in.