A friend of mine runs a small logistics company. Last year he paid an API bill of about four hundred dollars a month to summarise delivery notes. This year his new handheld scanners can do it on-device, offline, in under a second, for the price of the electricity. He did not change his software. He changed his hardware generation.
That is the quietest and most consequential shift in AI right now, and it does not get keynote slots because it is not a bigger model. It is a cheaper token.
The number that decides where inference runs
Forget parameter counts. The metric that actually governs deployment decisions is tokens per second per watt, measured on the device that has to do the work.
Three things fall out of that chart:
- The improvement is not uniform. Phone and laptop NPUs improved by roughly 3.5x in two years, because that is where the silicon roadmaps are pointed — every handset vendor wants on-device summarisation and photo editing without a round trip.
- Rack GPUs improved far less in efficiency terms (about 2.4x). They are optimised for absolute throughput and memory bandwidth, not joules. This matters: the cloud is getting faster, the edge is getting cheaper, and those are different curves.
- The gap between a phone and a rack GPU widened in the phone's favour. The phone went from 2.1 to 7.8 tok/s/W; the rack GPU from 1.2 to 2.9. In 2024 the phone was 1.75x better per watt; in 2026 it is 2.7x better. The edge is pulling away on efficiency even as it loses on total capacity.
That divergence is the whole story. Cloud inference wins on capability per dollar of capital. Edge inference wins on capability per joule — and joules are the thing you cannot scale by raising a round.
The TCO arithmetic, done honestly
Every "edge vs cloud" argument I read compares a GPU's sticker price to an API's per-token price. That is not a comparison; that is a category error. You have to compare five-year total cost of ownership for a fixed workload.
The model behind that chart assumes a small model (1-3B parameters, 4-bit weights), 1M requests per day, and a 3-year hardware refresh.
- Cloud API scales linearly and starts cheap. At 1M requests/day you are already at a five-figure monthly bill, and it compounds because you keep paying for the same tokens forever.
- Edge fleet has a brutal upfront: you buy the hardware, you pay for provisioning, you own the RMA process, and you eat the failure rate. But after year two the marginal cost per request is electricity and amortisation.
The crossover in my model lands around 2028 at roughly 90% fleet utilisation. Below that utilisation, cloud wins on flexibility. Above it, edge wins on cost — and the gap widens every year the workload stays the same.
The trap is that utilisation is the hidden variable nobody forecasts. If your fleet sits at 40% because demand is spiky, the edge case never pays back and you have bought a warehouse full of depreciating silicon. This is why the honest answer to "should we run this on-device?" is almost always "only for the steady-state portion of the workload".
What this changes in ordinary life
Your phone stops being a thin client for intelligence. Once a 3B model runs at 8 tok/s/W, the interesting features stop being "summarise this page" and start being "watch everything I do and be useful about it" — continuous, private, offline. Dictation that works in a basement. Translation that does not upload your conversation. Photo search that does not require a network round trip to a datacentre that logs the query.
The social consequence is under-discussed: privacy becomes a hardware property rather than a policy promise. "We don't train on your data" is a claim. "The data never left the device" is an architecture. Those are different levels of assurance, and regulators are starting to notice the difference.
Small businesses get capabilities that used to require an ML team. The logistics example is not exotic; it is the template. Invoice coding, delivery-note summarisation, defect spotting on a production line, first-line customer replies — all of these are now within reach of a device that costs less than a month of cloud spend, provided the workload is steady.
The API business model gets squeezed from below. If 60% of inference volume is short, repetitive, low-stakes work, and that 60% migrates to devices, the remaining cloud traffic is the hard, high-margin, long-context stuff. That is a much smaller market with much higher price sensitivity. Expect per-token prices to keep falling while frontier per-token prices fall more slowly — a barbell, not a uniform decline.
The engineering consequences nobody mentions
Running a model on a fleet of devices is a distributed systems problem disguised as an ML problem. Four things bite immediately:
- Quantisation drift. Your 4-bit model does not fail loudly; it fails on 2% of inputs, in ways that correlate with exactly the users you care most about.
- Version skew. A fleet of 40,000 devices will run nine different model versions for six months. Your evaluation harness has to be per-version or you will ship regressions you cannot see.
- Thermal throttling is your new p99. The benchmark you ran on a cold device is not the latency your user gets after twenty minutes of continuous use.
- Update bandwidth. A 2GB model update pushed to a fleet on metered connections is a real cost line, and delta-update infrastructure is unglamorous work that nobody budgets for.
What to watch
- tok/s/W on shipping hardware, not datasheets. Vendors quote peak NPU throughput at a fictional batch size. Look for measurements at batch 1 with the model you actually intend to run.
- The share of inference that is "ambient" — continuous, low-stakes, always-on. That is the segment that migrates first, and it is much larger than the chat-assistant segment everyone benchmarks.
- Model distillation cadence. Every time a 70B model is distilled into a 3B model with acceptable quality on a narrow task, another slice of workload becomes edge-eligible. That cadence, not chip announcements, sets the migration speed.
The uncomfortable conclusion for anyone building an AI product: the model is becoming a commodity input, the placement of the model is becoming the differentiator, and placement is a hardware and operations decision. The teams that treat inference as a deployment problem rather than an API call will be the ones whose unit economics survive the next two years.
If this was useful
Working out these curves takes real effort — reading the benchmark noise, normalising the units, and throwing out the numbers that do not survive a sanity check. If it saved you some of that, a tip is genuinely appreciated.
SOL: HCqPmPJM3NVtS6YrmVaTZYnuP8XZ2PFgQZKT94VK1LkS
Creating this takes real effort — thank you for the support.
Top comments (0)