32B LLM on a 2008 Xeon: When RAM Matters More Than VRAM

Read the original in Russian: 32B LLM on a 2008 Xeon: When RAM Matters More Than VRAM

Experiment screencast: YouTube · continuation of the Forth + AI story

Recently I wanted to run 19–20 GB quantized models locally — on my work laptop. RTX 4070, Windows, LM Studio. Maybe I misconfigured something. I expected the model would spread across VRAM + RAM + NVMe, but all I got was a hard system freeze.

Then I remembered my old 2008 server — 2× Intel Xeon E5440, 64 GB RAM, no GPU. I get the current llama.cpp, built it without any trouble, and ran it in pure CPU mode.

Model: deepseek-r1-distill-qwen-32b-q4_k_m.gguf

Prompt: write factorial in Forth

Goal: not speed, but an answer to the question: can old hardware with enough memory run a model that is quality-wise out of reach on my laptop?

Two machines, one model class

	Laptop	Server
CPU	modern notebook	2× Xeon E5440
RAM	16 GB	64 GB
GPU	RTX 4070 8 GB	—
Stack	LM Studio (Windows)	llama.cpp (CPU)
Result	freeze	loads, runs

The server takes quite a while to start with a model like this. Generation speed is about ~0.01 tokens per second.

The experiment started around 23:56. Then I turned on the screencast recorder and went to sleep. Woke up a couple of times to check the results. By morning I basically had two answer variants. In the recording I used fast forward, except for the moments where I test the code.

Seven hours, two factorial attempts, none correct

At 05:48 the model still had not finished its reasoning chain, but there was already a Forth-looking snippet. I copied version 1, ran it in gforth — did not work: stack or syntax error.

But the stream was still going. So I waited.

At 06:51 there was a second factorial attempt. I ran that one too. Failed again.

Full output

llama-ui
New chat
MCP Servers
Settings
Recent conversations

Write a program to calculate the factorial. On GForth language

Okay, so I need to write a factorial program in GForth. I'm not super familiar with GForth, but I know it's a version of Forth, which is a stack-based programming language. Let me think about how to approach this.

First, I remember that factorial of a number n is the product of all positive integers up to n. So, for example, 5! is 54321 = 120. That makes sense.

Now, in Forth, programs are written in a way that uses a stack. So, I need to think about how to structure this using a stack-based approach. I'll probably need a loop that multiplies the numbers from 1 up to n.

I should start by defining a word, let's say 'factorial', which will take a number from the stack and return its factorial. So, the syntax would be something like : factorial ... ;

Next, I'll need to handle the case where n is zero or one because the factorial of both is 1. So, maybe I can have a conditional check at the beginning. If n is zero or one, just return 1.

For the loop, I'll need to initialize a counter and a product variable. Since we're using a stack, maybe I can push the initial value onto the stack and then iterate, multiplying each time.

Wait, but in GForth, how do I handle loops? I think there's a loop structure like begin ... while ... repeat, or maybe using a counted loop. Let me think. Since I need to multiply from 1 to n, perhaps using a loop that starts at 1 and goes up to n, multiplying each time.

But how do I accumulate the product? Let's see. I can initialize the product as 1, then for each number i from 2 to n, multiply the product by i.

So, the steps would be:

Check if n is 0 or 1. If yes, return 1.
Otherwise, initialize product to 1.
Loop from 2 to n:
a. Multiply product by i.
After the loop, return the product.

Now, translating this into GForth code. Let's outline it:

: factorial
dup 0 = if
drop 1
else
1
1
dup 1 = if
drop 1
else
begin
2dup * >r
r> 1+
dup 1 > while
r> 2dup * >r
r> 1+
repeat
r> drop
end
then
then
;

Wait, that seems a bit complicated. Maybe there's a simpler way. Alternatively, I could use a local variable to keep track of the product.

Wait, GForth supports local variables with the 'var' keyword. So, maybe I can define a variable to hold the product.

Let me try another approach:

: factorial
dup 0 = if
drop 1
else
1 >r ( r: product )
1
begin
2dup *
r> * >r ( multiply and store back )
1+
dup n > while
repeat
r> drop
then
;

Wait, no, that doesn't seem right. Maybe I should use a loop that runs from 2 to n, multiplying each time.

Alternatively, I can use a recursive approach. Factorial can be defined as n * factorial(n-1), with the base case of factorial(0) = 1.

But recursion
Context: 787/131072 (1%)Output: 772/∞0.0 t/s
Chat 16.06.2026, 23:56:42 - llama-ui

: factorial
dup 0 = if
drop 1
else
1 >r
1
begin
2dup *
r> * >r
1+
dup n > while
repeat
r> drop
then
;

New takeaway: a model not trained specifically on a programming language (Forth, for example) cannot really "program" in it — even when you use a large model.

The Xeon swan song

It is practically hard to find practical value in experiments with 2008 hardware. What can such an old box do in the LLM era? Almost nothing — unless your goal is to heat the room.

For launching large Q4 models, 64 GB of system RAM on my server beats 16 GB + 8 GB GPU. But that is only the fact that it runs. You would not call 0.01 t/s comfortable to work with. I think I just need to tweak a few LM Studio settings and everything will work on the laptop.

For my goal of getting working Forth code, the large model did not help. I need the "factory": algorithm in a model-friendly IR, deterministic transpile, gforth as the judge. That is what I am building around Cursor — rules, skills, and MCP.

The E5440 platform is weak by modern standards — I would not pick it for PCIe GPU offload either. Here RAM volume was the only win condition.

Takeaways

1. VRAM is not the only barrier. System RAM matters when the whole model lives in host memory on CPU. But in that case it is very slow.
2. Slow inference is still inference. 0.01 tok/s is more of a joke; but for the question "will it run at all?" — we got an answer.
3. Do not buy a GPU rack to find something out. I already wrote myself a rule: buy hardware after the experiment, not to finally start the experiment — otherwise you too will have monsters like this in the closet: "old but not useless" (or are they useless after all?).
4. Fail format. Well, in the end I got two broken factorials in seven hours. Bad. But that lesson was cheaper than a month of Opus thinking loops (until you count how much went into building the Xeon).

Update

I made some measurements of llama-bench with parameters -r 1 -p 16 -n 16

Short test

Model	Params	File size	pp16 (t/s)	tg16 (t/s)
TinyLlama 1.1B Q6_K	1.1B	861 MiB	13.20	6.57
Gemma 4 E2B Q4_K_M	4.7B	3.2 GiB	8.08	3.37
Qwen2.5-Coder 7B Q4_K_M	7.6B	4.4 GiB	2.51	1.25
Gemma 4 E4B Q4_K_M	7.5B	5.0 GiB	3.89	1.72
Qwen2.5-Coder 14B Q4_K_M	14.8B	8.4 GiB	1.22	0.65
Qwen2.5-VL 32B Q4_K_M	32.8B	18.5 GiB	0.52	0.29
DeepSeek-R1 Distill Qwen 32B Q4_K_M	32.8B	18.5 GiB	0.52	0.29

llama-bench shows the ceiling on a short context (~0.29 tok/s for 32B). A real chat with reasoning and thousands of tokens on a Xeon CPU averages ~0.03 tok/s — not because the model is different, but because each token gets more expensive as the context grows.