An individual recently executed a 400-billion parameter AI model on an iPhone. No server connection. No cloud streaming. It was in airplane mode running on a phone with 12GB of RAM.
Technically, the model requires over 200GB of memory. Yet the phone only has 12. How does this defy the laws of computer science?
The Trick: Stream, Don't Load
A developer known as @anemll posted a demo over the weekend using an open-source project named Flash-MoE. It implements a Mixture of Experts architecture that only wakens 4 to 10 experts per token out of 512 in total. Instead of preloading the complete model into memory, it streams weights directly from the phone's NVMe SSD to the GPU upon demand.
Imagine a library where you only take books off the shelf when you get a question. The remainder of the books stay on the shelf. Except the "shelf" is a 2GB/s SSD and the "books" are neural network weights.
The Speed? 0.6 Tokens Per Second
That's crazy slow for a chatbot. You'd be looking at 30 seconds for a sentence.
Here's why it's blowing developers' minds anyway.
This Isn't About Today
This isn't about going into production today. This is about setting the next upper bound.
Two years ago, a 7B model was the big dog for local runtimes. Last year, people were hyped over 70B on a Mac Studio with 192GB of unified memory. Now you've got a phone brute-forcing 400B by treating storage as virtual memory for neural networks.
Apple knows this is possible because they published their "LLM in a Flash" research paper. The core idea is that SSD bandwidth, not RAM capacity, becomes your upper bound. Flash-MoE turns that all the way up to 11.
Privacy Is the Real Story
Here's the real interest for developers: a model you can run offline, on device, with no network means your prompts never leave the phone.
No API logs. No data center slicing your medical questions or legal drafts.
Apple has been building to this. The Foundation Models framework they shipped with iOS 26 lets developers integrate on-device intelligence into apps with a 4096 context window. We are not there yet, but the waypoints are clear.
The Cloud Pricing Question ☁️
The question nobody is asking is, what is cloud AI pricing going to look like when you can run it on your phone?
OpenAI charges per token. Anthropic charges per token. Google charges per token. Their entire business model is predicated on needing their servers.
Let's say on-device inference gets within the ballpark of usable for a reasonably large model or two, and Apple ships smaller MoE models optimized for their hardware. Why pay the toll when the phone can do it all for free?
We are probably a couple hardware generations away from that being practical. Apple expects 7B-14B MoE models to hit 10-20 tokens per second on flagship iPhones by 2027.
But the line on the graph is clear. And some squad of product managers from Azure, AWS, and GCP are no doubt right now sweaty in a conference room over a back-of-the-envelope R&D budget.
The demo is phone-on-a-desk offline. No Internet. No fallback. Just silicon and storage doing what used to require a data center.
Is on-device AI a real threat to cloud inference pricing, or are we still too far from shore? 🤔
Top comments (0)