Weights on NVMe, INT4 in production, and the economics of self-hosting

Two of today's stories are about squeezing frontier models onto hardware you already own. A third is about what the people serving them at scale actually do to make that work. Taken together they say the ceiling moved, and the reason to care about that has changed.

An 80B model in 4.3 GB of RAM

Swiftlet showed up on Hacker News today with 217 points and 99 comments. It streams weights off NVMe storage instead of holding them in memory, which lets it run an 80B Qwen on a Mac using about 4.3 GB of RAM, and a 35B model on an iPhone.

Throughput is the number to look at here: the author reports roughly 4.5 to 5 tokens per second on an M5. One commenter measured about half an hour to work through 10k tokens. That is not a chat experience. It is a batch job.

What this proves is that the memory ceiling is softer than people assumed. Whether local inference can replace an API call for interactive work is a separate question, and this does not answer it.

Cloudflare shows its quantization homework

Cloudflare published how it serves open models at scale: INT4 for weights, FP8 for the KV cache, plus cache tagging so entries do not get reused across contexts they should not. Evaluation centered on Kimi K2.6. The post landed at 230 points and 58 comments.

The comment thread found the gaps rather than the wins. Sensitivity to KV quantization is not uniform across model families, so a result on one family does not transfer. Long-context coding tasks can accumulate errors that standard benchmarks never surface. And the quantization details were not prominently disclosed on the model listing pages.

That last one is the practical takeaway. If a hosted model is quantized and you do not know how, your evaluation is measuring a different artifact than the one the benchmark measured.

Single-GPU frontier models, and why the math still fails

I covered V4 Flash's price cut yesterday. This is the other side of that coin: the same cheap API is what makes self-hosting it stop making sense.

Someone got DeepSeek V4 Flash running on one AMD MI300X. The thread put cloud rental for that card at about $1.99 per hour on AMD Developer Cloud. Reported throughput was around 830 tokens/sec burst and roughly 500 sustained across 8 concurrent users.

A commenter then ran those numbers against DeepSeek's own API pricing and found the obvious problem. Unless you are keeping that card saturated with concurrent users, the rental costs more than the tokens are worth.

For most of the last two years the argument for self-hosting was cost. Cheap inference APIs have quietly removed that argument for small operators, leaving only the reasons that were always the real ones: data residency, availability guarantees, and not building on top of a pricing page someone else controls.

Three more, briefly

Apple shipped the rebuilt Siri in the iOS 27 public beta in July. It runs on Apple Foundation Models, which Apple refined with help from Google's Gemini, on Apple silicon and private cloud. General release is expected in September.

Intelligence, the company behind Design Arena, raised $7.9M led by Index Ventures. The product is A/B voting on AI-generated output, sold to labs as human preference data. The company reports 5.3M users and $60M ARR, both self-reported. A company at $60M ARR raising a $7.9M seed is an odd pair of numbers, so treat both as company claims.

Lilian Weng wrote on harness engineering for self-improving agents. The core argument: keep the evaluation harness independent, or the feedback loop will corrupt the thing it is supposed to measure. If you have ever watched a test suite get quietly adjusted until it passed, you already know the failure mode.

FFmpeg 9.0 is out.