My program crashed with java.lang.OutOfMemoryError. My machine had 12 GB of RAM free.
Both sentences are true, and the reason why is the same reason you cannot run a big AI model on your laptop. Real numbers from my machine below.
The ceiling you never chose
Your program does not use the machine's memory directly. Java creates a separate space for it, the heap, and by default that heap is about a quarter of your RAM. On my 16 GB machine: 4,096 MB. I never chose that.
Why smaller on purpose? Two reasons. The garbage collector has to walk everything still alive, and a bigger heap can mean a longer walk (last episode: 6 million living objects cost 29 ms per collection). And your program is not the only thing on the machine.
The ceiling is a policy
Here is the part that surprised me. I ran the same allocation loop three times on the same machine, changing only the rule:
| run | ceiling | died holding | RAM free at death |
|---|---|---|---|
| -Xmx256m | 256 MB | 224 MB | 1,924 MB genuinely free |
| default (25% of RAM) | 4,096 MB | 3,840 MB | plenty (see note) |
| -XX:MaxRAMPercentage=50 | 8,192 MB | 7,264 MB | still headroom |
Same machine. Same program. Three completely different crash points.
The limit is not a fact about your computer. It is a policy. And a percentage does not care whether the machine is large or small: give Java half of a tiny container and there is nothing left for Java itself (thread stacks, metaspace, the JIT). That is why MaxRAMPercentage matters in the container era, and it is not a Java quirk: Node sizes its heap from available memory too.
An AI model is the same wall, one floor up
An AI model is not magic. It is a very long list of tuned numbers, and writing one word of output uses every one of them. Then the next word uses them all again.
7,000,000,000 numbers x 4 bytes = 28 GB
That is the floor, before loading overhead. A normal graphics card holds 12 GB. The model does not fit, and "does not fit" has a price measured in words per second.
The two wires
The maths runs on the graphics card, and the card has its own memory soldered right next to the chip: VRAM. The card's internal wire moves roughly 1,000 GB/s. The plug connecting the card to the rest of your machine moves roughly 60 GB/s. That is a 16x difference in wire width.
- Numbers fit on the card: they cross the narrow plug once, then every word runs on the wide wire. ~70 words/second.
- Numbers spill: the leftovers cross the narrow plug for EVERY word. ~4 words/second.
The worker was never the problem. The wire feeding it was. (Both speeds are spec-sheet arithmetic applied to a 14 GB model, not a benchmark: real inference varies with batching and caching. The ratio is the lesson.)
Shrink the bytes, not the count
You cannot delete the numbers, but you can store each one with less detail: 4 bytes to 2 to 1 takes 28 GB to 14 to 7. Like rounding $99.99 to $100: close enough, much smaller. 7 GB fits on a 12 GB card. The price is precision: the model gets slightly worse. That trade is called quantization, and it is why "run a 7B locally" is possible at all.
Why AI is an API
Bigger models are 70 billion numbers. At 2 bytes each that is 140 GB. No card you can buy holds that, and the numbers cannot live in normal RAM without crossing the narrow plug on every word.
So companies load those 140 GB onto racks of specialised cards once, for everyone. You send words. Words come back. The numbers never move. That is not a business decision first; it is physics and arithmetic. AI arrives as an API because the model literally has no home on your machine.
One law under everything
From a 256 MB heap ceiling to a 140 GB model: your program never gets the whole machine.
Honesty notes: the three crash points are measured (my machine, Java 21, 16 GB; your exact numbers will differ, the policy will not). The default run showed less "free" RAM at death because macOS counts file cache as used; the -Xmx256m run is the clean proof with 1,924 MB genuinely free. Model sizes are arithmetic, not measurements. "16 GB" prints as 16,384 MB because marketing gigabytes and binary gigabytes are different units.
The full video draws the whole thing frame by frame, including the three deaths and the camera drop onto the graphics card:
Which one has bitten you: a heap limit you never chose, or a model that would not fit on your card?
Top comments (0)