Sam Altman says the RAM in your computer is a bad use of the world's memory. He's half right, and the half he skips is the reason a lot of our clients ask us about local models in the first place.
Here's what he said. On the Cheeky Pint podcast with Stripe co-founder John Collison, about 29 minutes in, Altman was asked about supply chain shortages. As quoted by The Verge on October 30, he said:
"A lot of people want local inference. And somehow, they assume that the problem is solved now because we are going to have open-source models, we're going to be able to run them on our computers, and so we're not going to need all the stuff in the data center. But from a global allocation perspective, actually, it's quite terrible to have everybody have a bunch of the world's limited supply of DRAM sitting in their own computer and being used five percent of the time. You want maximum utilization in the cloud if you're trying to deal with this very supply constrained environment."
Developers did not take it well. There's a busy Hacker News thread, r/LocalLLaMA lit up, and most of the reaction boils down to "the man who sells cloud inference says cloud inference is better." Fair enough. But I build AI features for clients for a living, and I think both camps are arguing past the decision teams actually face.
So here's my honest take: his arithmetic is mostly right, his framing is wrong, and neither one tells you where to run your model.
What's actually true in what he said
Start with the part people don't want to admit. Utilization matters a lot for cost.
A GPU or a block of memory in a data center serving thousands of requests is busy most of the time. The same hardware on a developer's desk is idle overnight, idle in meetings and idle at lunch. If you divide the hardware cost by the tokens it actually produces, a busy shared machine wins easily. That's not an OpenAI talking point; it's the same reason most companies stopped running their own mail servers.
And the memory market really is tight right now. TweakTown, reporting on the same comments, pointed out that consumer DDR5 kits have jumped sharply in price over the past year and cited TrendForce forecasting another 10% to 15% rise in conventional DRAM contract prices this quarter as suppliers prioritise AI server memory. The Verge noted that Samsung and SK Hynix have long-term deals with cloud operators and projects like OpenAI's Stargate that reserve much of their output for the next few years.
So if your only goal is "cheapest tokens for a general chatbot", the cloud usually wins. I've told clients exactly that.
What the framing leaves out
The problem is the premise that people run models locally to save memory, or to save money. In our work, they almost never do.
When a client asks me about local or self-hosted models, the reason is nearly always one of these:
- The data can't leave. Contracts, health data, internal financials, source code under a strict NDA. For some of these, "the vendor promises not to train on it" isn't enough for their compliance people.
- They want nobody else's policy in the loop. A hosted model can be deprecated, rate-limited, repriced, or start refusing a task your product depends on. A model file on your own hardware can't change under you.
- It has to work offline or at the edge. Field apps, factories, a laptop on a plane, a device with bad connectivity.
- The workload is steady and boring. Classifying documents all day, every day, at a predictable volume. Here owned hardware runs hot, and the idle-time argument mostly goes away.
None of those is an allocation question. Altman's quote treats local inference as a worse way to do the same thing. For these teams it's a different product: one where control and privacy are features, not inefficiencies.
There's also an irony nobody in the thread missed. The scarcity he describes is driven by AI data center demand, including his own company's. Asking consumers to give up local memory so the cloud can have more is a hard sell from the biggest buyer in the market. I don't think that makes him wrong about utilization. It does mean "global allocation" isn't a neutral frame coming from him.
How we actually decide: a routing table, not a religion
Here's the rule of thumb we use on client projects. It's not clever, but it's saved us from both extremes.
| Workload | Where it runs | Why |
|---|---|---|
| Frontier reasoning, long context, agents | Hosted API | Capability you can't match locally, bursty usage |
| Sensitive data, strict compliance | Self-hosted open-weight model (client's cloud or on-prem) | Data stays inside their boundary |
| High-volume, simple, predictable (classify, extract, tag) | Small self-hosted model or hosted small model, whichever is cheaper at real volume | Utilization decides; measure it |
| Offline or edge | On-device small model | No network, no choice |
| Prototype or MVP | Hosted API | Speed of iteration beats everything early on |
Notice the middle option that both sides of the debate skip: open-weight models running in a data center you control. You get most of the utilization benefit Altman describes, and most of the control the local crowd wants, because you can move the weights tomorrow. For a lot of B2B products that's the sweet spot.
Make the choice reversible in code
The bigger mistake I see isn't choosing the wrong side. It's hard-wiring the choice. If your Django views call one vendor SDK directly, switching later is a rewrite. A thin interface keeps it a config change:
# ai/router.py
from dataclasses import dataclass
from django.conf import settings
@dataclass
class Task:
name: str # "classify_invoice", "draft_reply", ...
sensitive: bool # contains data that must stay in our boundary
text: str
def pick_backend(task: Task) -> str:
if task.sensitive:
return "self_hosted" # e.g. an open-weight model behind an OpenAI-compatible server
return settings.AI_ROUTES.get(task.name, "hosted")
def complete(task: Task) -> str:
backend = pick_backend(task)
client = settings.AI_CLIENTS[backend] # both expose the same .complete()
return client.complete(task.text)
Many self-hosted inference servers expose an OpenAI-compatible endpoint, so the "client" for each backend is often the same SDK pointed at a different base URL with a different model name. Routing becomes a settings dictionary, and moving a task from hosted to self-hosted (or back) is something you can test with an eval set instead of debating in a meeting.
Two more habits that matter more than which side you pick:
- Log cost and latency per task, per backend. Then the "is local cheaper?" argument is a query, not an opinion.
- Keep a small eval set per task. Thirty to fifty real examples is enough to tell whether a smaller or self-hosted model is good enough for that job.
Where I've landed personally
I run small models on my own machine for experiments and for poking at client data I'd rather not send anywhere during early discovery. The RAM is already in the laptop. I'm not buying 128GB at today's prices to save on API bills, because at my personal volume the cloud is cheaper and I know it. When I need serious reasoning, I use a hosted frontier model and don't feel guilty about it.
For client products, I've stopped treating "local vs cloud" as a values question. It's a routing question, answered per task, and it should be cheap to change your mind. The teams that get burned are the ones who picked a side in a Twitter argument and then built their whole stack around it.
So yes, Sam, idle memory is wasteful. But for a lot of the teams I work with, the RAM sitting idle on their own hardware is what control costs, and many of them are happy to pay it.
What do you think?
Where do you draw the line: which AI tasks in your product would you never send to a hosted API, and which would you never bother running yourself?
Top comments (0)