TL;DR — Local, open-weight LLMs are already shipping on flagship phones and laptops in 2026 — not as a future promise, but as the engine behind offline transcription, summarization, and translation. The real constraint isn't the chip's TOPS rating, it's memory bandwidth, and that ceiling is quietly deciding which model sizes ever reach consumer hardware.
Somewhere between "AI PC" marketing and "on-device AI" press releases, an actual fact got buried: open-weight language models are running locally on shipping consumer hardware right now, doing real work, with no network call. Not a demo. Not a roadmap slide. A Llama-class model summarizing a document on a phone you can buy today. That's the story worth unpacking, because the model choice, the RAM budget, and the feature list are all downstream of the same hardware constraint — and that constraint is more interesting than any benchmark chart.
What's actually running today
Asus's Zenfone 12 Ultra ships with Meta's Llama 3-8B running locally for article and document summarization, alongside AI Call Translator and AI Transcript features that work with no network dependency, per a rundown from mtechresearch.tech. That's an 8-billion-parameter open-weight model, quantized down to fit a phone's memory, doing a job that a year or two ago meant a round-trip to a cloud API.
Apple's approach is architecturally different but philosophically adjacent: Apple Intelligence runs its writing tools, summarization, and Image Playground features on-device where possible, and only escalates to Private Cloud Compute — Apple's attested, audited server tier — when the task outgrows what's on the silicon. Per elephas.app, the whole on-device layer costs about 7GB of storage on an M-series Mac or an iPhone 15 Pro or later, and it's free — no subscription unlocks more of it. Google's parallel play is Gemini Nano powering Pixel 8 Pro's Recorder summaries and smart replies offline, and Samsung's Galaxy AI runs Circle to Search and Live Translate with on-device modes gated behind its Knox attestation layer, both cited in the same mtechresearch.tech analysis.
None of these vendors are shouting "open weights" from the App Store listing — but Llama's presence on Zenfone is the clearest public admission that a named, open-weight checkpoint is the thing actually doing the inference, not a black-box proprietary model nobody can name.
The RAM ceiling nobody puts on the spec sheet
The number that actually gates what runs on your phone isn't the NPU's TOPS rating — it's memory bandwidth. A breakdown from weeklyreviewer.com lays out the math plainly: a 7B-parameter model at 4-bit quantization needs roughly 3.5GB of working memory, and at the 40GB/s LPDDR5X bandwidth standard on current Copilot+ PCs, that generates 25–35 tokens per second — fast enough to feel conversational. Push to 13B parameters (about 7GB quantized) and throughput drops to 12–18 tokens per second, described as "the outer edge of interactive usability." Apple's M4 Neural Engine sits at 38 TOPS — lower than several competing x86 NPUs on paper — but the same source credits Apple's hardware-software co-design with delivering more consistent latency on Apple Intelligence workloads than higher-TOPS rivals running equivalent quantized weights.
Put plainly: the practical ceiling for mainstream device memory in 2026 is roughly Llama 3.1 8B at 4-bit as the top end of what fits comfortably. Anything needing GPT-4-class reasoning, hundred-thousand-token context, or frontier multimodal generation stays cloud-dependent — not because of a policy choice, but because the memory isn't there.
Where this is landing in real apps
The clearest evidence that this is a usable stack, not a lab curiosity, is a small ecosystem of curated model-picker apps built specifically for phone constraints. Marmot, an open-source iOS/Android app, runs inference on-device via llama.cpp — Metal-accelerated on iOS, ARM-optimized on Android — and instead of exposing thousands of models, ships one open-weight "champion" per weight class with a RAM-fit badge computed from the device's actual memory. Its July 2026 lineup: SmolVLM 256M for on-device receipt and screenshot understanding at 365MB, Qwen3.5 0.8B as the featherweight pick for any phone, Qwen3.5 2B for 4–6GB devices, SmolLM3 3B for 6GB devices, Qwen3.5 4B for 8GB devices, and Gemma 4 E4B — described by the project as "8B weights at a 4B footprint" — as the heavyweight ceiling for 12GB+ RAM phones. Marmot's meeting mode uses this stack for continuous transcription with a tap-to-speak contribution card, saving searchable transcripts entirely offline.
Who this is actually for
This is where the use case gets concrete instead of aspirational. Field technicians, clinicians doing bedside note-taking, journalists in low-connectivity regions, and travelers all share the same requirement: the assistant has to work with no signal, and the data often shouldn't leave the device at all. A local 3–8B open-weight model handles document summarization, meeting transcription, and short-form translation well enough for these jobs — not because it matches a frontier cloud model's reasoning, but because the job doesn't need frontier reasoning. It needs to run, offline, in under a second per response, on hardware the user already owns.
A first-person account from remio.ai is the plainest version of this trade-off: a local LLM on Android replaced a paid cloud AI subscription for several everyday tasks, with the model staying available with no network and no recurring bill. That's the honest pitch for local open-weight models in consumer tech — not "as good as the cloud," but "good enough, always available, and free after the download."
The trade-off cuts the other way too. If your task genuinely needs 100K-token context, multi-step agentic planning, or frontier-grade multimodal generation, none of these on-device models will get you there — that's what attested cloud offload exists for, and every vendor above builds a path to it rather than pretending the phone can do everything.
Where this breaks down
The RAM-class picker model — Marmot's weight-class table is a good proxy for the whole industry's current state — means your phone's memory tier silently decides your model quality ceiling, and most users never see that trade being made. A 4GB-RAM phone gets Qwen3.5 2B; a 12GB flagship gets something close to Gemma 4 E4B's quality. Nobody labels this on the box, and the gap between those two experiences is real. Battery drain under sustained local inference, thermal throttling during long transcription sessions, and the lack of a standard way to verify which exact checkpoint and quantization a "summarize" button is invoking are all unresolved rough edges — this is infrastructure that works, not infrastructure that's finished.
Credits & sources
Thanks to the maintainer of Marmot for the open-source weight-class model picker and its documented RAM-fit lineup. Device and feature data on Apple Intelligence came from elephas.app. The cross-vendor breakdown of Apple, Google, Samsung, and Asus on-device strategies came from mtechresearch.tech. Memory bandwidth and TOPS analysis came from weeklyreviewer.com. The first-person account of a local LLM replacing paid cloud AI tasks came from remio.ai.
Tomorrow's episode looks at DeepSeek-V4-Flash — a model built for a very different constraint than the phone in your pocket, and worth understanding on its own terms.
Appendix — the field in one chart
Top comments (0)