How Far Can Local AI Go with Just a CPU — Building Auto-Tracking and Benchmarking Loops, and My Conclusions

#ai #cpullm

Local AI on CPU Without a GPU: How Far Can It Go? — Building an Auto-Tracking, Benchmarking Loop, and the Conclusions I Reached

Riel's chat runs on a cloud API (Google Gemini). For a while, though, I've been stuck on the hypothetical: "What if I couldn't use this API?" Whether due to model export regulations or policy changes, I wanted to build a minimal Korean chatbot fallback that runs solely on my home desktop in case external AI access is cut off. This post is a record of how far I pushed this on CPU-only, without a GPU, and why it's no longer my primary solution.

The Condition: Effectively No GPU

My desktop has a Ryzen 5700X (8 cores, 16 threads), 32GB RAM, and an RX580 4GB for graphics. The problem is, the RX580 (Polaris) doesn't support the latest ROCm, meaning I can't properly use the GPU for local LLM acceleration. Plus, 4GB isn't enough to load a 7B model entirely. In the end, the actual engine was the CPU. "How far can I go without a GPU, on CPU?" was the real question from the start.

What I Built — A Loop of Measuring, Tracking, and Self-Replacement

I didn't want to just run a model and call it a day. My goal was a system that could automatically swap out models for better CPU-compatible ones when they become available, even if I miss it.

Bench Harness — I created Korean auto-grading questions (math, multi-step reasoning, logical negation, unit conversion, facts, instruction following) to measure accuracy and speed (tok/s). It started with 8 questions, but they lacked discriminative power (more on this later). I increased it to 25, then to 50 questions.
Automatic Tracking of New Hugging Face Models — I detect new releases from manufacturers (EXAONE, Qwen, Google, etc.) on a weekly basis and add them to a candidate queue. Initially, I scraped "trending" models, which brought in a lot of 2023 community fine-tunes. I refined this by using a watchlist of authors + filters for size and family to cut down on noise (from 78 candidates down to 24).
Autonomous Benchmarking Loop (auto_bench) — My desktop automatically pulls new models from the queue, runs the 50-question benchmark, and uploads the results to a server. The server then automatically replaces the currently serving model if the new one is definitively better (either in accuracy margin or speed advantage). This loop actually discovered and swapped in models I hadn't explicitly scouted.
Public Demo (/local-ai) — Using an outbound polling worker structure that doesn't open any inbound ports on my home network, anyone could experience this CPU model while my desktop was on. ## Actual Measurements — And the Moment the Illusion Broke Final scoreboard (my desktop, CPU) based on 50 questions:

Model	Accuracy	Speed
`exaone3.5:7.8b`	90%	14 tok/s
`exaone3.5:2.4b`	84%	53 tok/s
`qwen2.5:3b`	84%	43 tok/s
`gemma2:2b`	76%	31 tok/s

The biggest takeaway here was how much the measurement design influences the conclusion. gemma2:2b initially scored 100% on the first 8 questions, making it seem "the best." But as I increased the question count, its score dropped continuously from 84% (25 questions) to 76% (50 questions). It was an overestimation caused by easy questions. Conversely, the value of exaone 7.8b only became apparent with harder questions (it tied with 2.4b on the initial 8 questions). Seeing rankings remain stable even with an expanded sample size finally became a mark of true reliability.
Ultimately, the optimal balance for CPU serving was exaone3.5:2.4b (84% accuracy + fastest).

Why It Stepped Down From Primary Use

To be honest, the engineering was successful, and that's precisely what clearly showed its limitations. The auto-tracking -> auto-benchmarking -> auto-replacement loop actually works. However:

The capabilities of models runnable on CPU (roughly 2~8B parameters) are about GPT-3.5 level. The gap is too large to replace the latest cloud models Riel's chat uses.
The realistic answer to "local AI gets better on its own" isn't that models inherently become smarter (weights are fixed), but rather that they detect new models and replace them with better ones. But the ceiling for that replacement is inherently low.
To reach true API-level performance, you need a GPU (which means money). I decided not to make that investment at this stage. So, local CPU AI has been demoted from primary use to a "fallback and experimental demo" status. The /local-ai endpoint is still alive, allowing direct experience if my desktop is on, but it no longer handles Riel's actual responses. ## What Remains (Lessons Learned)
Measurement comes first. That 100% on 8 questions was an illusion. If your yardstick is flimsy, you'll pick the wrong model.
Autonomous loops can be built, but they can't overcome hardware ceilings. Even if you automate detection, benchmarking, and replacement with software, if the best you can afford is GPT-3.5 level, that's where you'll stay.
Knowing when to fold is also a result. Confirming with numbers, not just gut feeling, that something "works/doesn't work" and stepping back — and documenting that record — is valuable. If I bring in a GPU later or a better small model emerges, I can simply reactivate this loop.