I run a local inference server that handles thousands of agent requests a day. It cost about $2,000 in used parts, and the newest silicon in it taped out around 2016. This series is the story of standing it up, and more honestly, the story of how much of what I "knew" about it turned out to be wrong.
I didn't pick this hardware to prove a point. I picked it because it's what I could afford. It turned out to be the best teacher I could have bought.
The bill of materials
Server guts used from the enterprise-surplus market (eBay receipts, January 2026); case, PSU, and cooler new from Newegg:
| Component | Cost |
|---|---|
| AMD EPYC 7302P + Supermicro H11SSL-i (16 cores / 32 threads) | $555 |
| 128 GB DDR4-2666 ECC (8x 16 GB) | $551 |
| 2x NVIDIA Tesla P40 (24 GB each) | $403 |
| 1 TB Intel DC P4510 U.2 NVMe | $100 |
| U.2 adapter and power cabling | $63 |
| Case, PSU, CPU cooler (Newegg, approximate) | ~$330 |
| Total | ~$2,000 |
For context, that's roughly two months of what an always-on agent workload would cost me in frontier API bills. The machine paid for itself before I finished tuning it.
The P40 is the heart of the build and the source of most of the pain. It's a Pascal datacenter card from 2016: 24 GB of VRAM, compute capability 6.1, no Tensor Cores, no NVLink. Two of them give you 48 GB of VRAM on paper. One of the first hard lessons in my notes is titled "a 47GB model does not fit in 48GB." The driver reserves about 6%, so usable is 45 GiB. Budgeting at the spec-sheet number OOMs.
What this hardware cannot do
Knowing the "no" list up front would have saved me weeks. Here it is:
No vLLM. Compute capability 6.1 is too old. I didn't take the documentation's word for it; I have an experiment directory proving it. Ruled out for real.
No Tensor Cores means FP16 is a trap. On Pascal, FP16 math runs at 1/64th the rate of FP32. The card's strength is INT8 through the dp4a instruction, about 47 TOPS. Everything about a working Pascal config flows from that one fact: quantized models, integer matmul kernels, and skepticism toward any advice written for newer cards.
No concurrent GPU models. One large model resident at a time. Swaps take about 30 seconds through the Portainer API. You design around it or you fight it forever.
No fast cold starts. 20 to 40 seconds to load a model, depending on size.
Most advice doesn't apply. This one cost me the most. The internet's LLM performance guidance is written on Ampere and newer. Some of it transfers to Pascal, some of it is irrelevant, and some of it is actively destructive. A "40% faster" split mode I found recommended in a vendor blog crashes Pascal outright with an illegal memory access. Telling those categories apart is most of the work, and it's the subject of this whole series.
What it can do
Here's what those constraints actually bought, with measured numbers from my own logs, not estimates:
- A 26B mixture-of-experts model (Gemma 4 26B-A4B, Q8) decodes at 41 tokens/sec. That's the fast path.
- The daily workhorse, a 27B dense model at Q6_K, runs 13-17 tokens/sec single-stream with speculative decoding, around 15 tokens/sec aggregate across four parallel slots.
- That same stack serves a 262k-token context window in about 23 GB of VRAM, using quantized KV cache.
- On the CPU side, small 4B models handle classification and routing at 15-25 tokens/sec on a handful of cores, always on, never competing with the GPUs.
- Over one measured 40-hour production window the stack processed 8,129 requests with a 0.17% failure rate and zero manual interventions.
Twelve different GPU model stacks are compiled, deployed, and benchmarked on this machine today, swapped on demand behind a single OpenAI-compatible endpoint. Agents talk to it all day. Most requests never touch a paid API.
None of that requires modern hardware. It requires knowing the machine you actually have.
Why old hardware teaches you more
On an H100, plenty of mistakes just cost you a little throughput you never notice. On a P40, mistakes fail loudly. The wrong split mode crashes. The wrong precision runs 64 times slower. The wrong context budget OOMs. The hardware gives you honest feedback because there's no headroom to hide in.
That feedback forced habits I now think of as the real payoff of the project:
Measure on your own hardware. Community numbers are hypotheses, not facts. Every claim in my notes carries a date and the file the measurement lives in.
Change one variable at a time. I learned this by breaking it. A "modernized" rebuild changed four things at once and prompt processing collapsed from 153 tokens/sec to 29. Nothing was attributable until a clean A/B isolated each variable. That incident became a standing rule.
Verdicts expire; mechanisms survive. The most useful things I know about this machine aren't rules like "flag X is always right." They're conditions: "flag X wins on these architectures because of this mechanism, and loses on those." Over six months, roughly half of my March conclusions were overturned or narrowed by September. Each reversal has a measurement behind it. That's not embarrassing; that's the discovery process working.
That last point is the spine of this series. Performance guidance for a moving target like llama.cpp is perishable. Flags I tuned around got deleted upstream. An environment variable I exported religiously turned out to be dead code that was never read. A "critical" recompile fixed a problem the running binary didn't have. Every one of those stories is a post.
Where this started
The homelab behind this goes back to 2016: Docker, networking, self-hosted services, a decade of running my own infrastructure because I wanted to know how things work. The AI chapter is recent. I started messing with LLMs in the fall of 2025, and the substrate was already there, which is the only reason the timeline in this series is months instead of years.
I work in SQL and data pipelines by day, on donor management systems for nonprofits. Nights and weekends I pointed the homelab at inference. This series is the honest record of what happened next: what I believed, what the machine proved, and what I had to unlearn.
The series
- This post. The hardware, the constraints, the thesis.
- The flag we tuned around got deleted. Row-split was the P40 answer, until upstream removed it. What replaced the lost throughput came from somewhere I wasn't looking.
- Cargo-cult flags. The "critical" env var that was never read at runtime, and how reading kernel source settled it.
- When the reason changes, the flag flips. Flash Attention was measurably wrong on Pascal, until quantized KV cache made it mandatory. Same silicon, new condition.
- Measure the binary you run. Two documents argued opposite positions about a SIMD flag that was already enabled. The running binary knew; the build directory lied.
- Template beats quant. A full quantization step moved my benchmark score not at all. The wrong chat template zeroed it.
- Buying speed with architecture. What MoE sparsity, speculative decoding, and parallel slots actually bought, with the A/B numbers.
- The gate caught me cheating. The promotion ledger that invalidated my own candidate because I skipped the test I designed.
- Eggs, cholesterol, and GPU flags. The wrap-up: a six-rung ladder of evidence quality, and why the mature answer is a condition, not a verdict.
Everything in it comes from dated, recorded measurements. Where something was never tested, it's labeled as never tested. Where I was wrong, the original wrong belief is quoted, because the reversal is the content.
$2,000, ten-year-old silicon, and more real systems education than any cloud bill has ever bought me.
Top comments (2)
The driver reservation trap on P40s is where most people get burned when stacking dual 24GB cards. Between CUDA context initialization, page tables, and driver memory overhead, you lose around 1.2 to 1.5 GiB per card before loading a single tensor. If a weight budget counts on raw 48 GiB instead of 45 GiB, the process OOMs on the first long context allocation.
The chat template point in item 6 is also the hardest failure to catch without automated evals. A subtle whitespace or BOS token drift in a custom jinja template quietly breaks instruct adherence while throughput benchmarks look completely green.
The line that hit hardest was "the internet's LLM performance guidance is written on Ampere and newer." I don't run Pascal, but I do run an always-on agent fleet against a mix of local and API backends, and the same law holds one level up: most serving advice assumes batch economics a small deployment simply doesn't have. The "paid for itself before I finished tuning it" math matches what I see on the API side — two months of frontier bills for an always-on agent workload is about right, and after that the burden of proof is on the cloud.
The constraint I'd most like to hear more about is the single-resident-model rule. Agent traffic is bursty in a bad way: one request wants the small router model, the next wants the 27B, the third clearly needs the long context window. With 30-second swaps through the Portainer API, how do you keep an agent that bounces between models from triggering swap storms? Pin a default and route only confirmed-heavy requests to the big model, or queue and eat the latency?
Two smaller questions from your numbers. The 262k context in ~23 GB via quantized KV cache — which KV quant, and did you check retrieval quality at the far end of the window, or is that context mostly there so long agent transcripts survive? And on the 8,129 requests / 0.17% failure window: were the failures timeouts on cold loads, or something else? That failure budget is the part I'd want to replicate before the throughput numbers.