This month a developer ran a working language model on a Sony PSP-2000. That is a 333 MHz MIPS handheld from 2007 with 64 MB of RAM, and it is streaming English text onto its LCD at one to two tokens per second. The model is Karpathy's stories15M, int8-quantized down to about 17 MB, running on roughly 1,100 lines of pure C. A 64-token completion takes about a minute.
I have no plans to buy a PSP, and you probably do not either. But I keep seeing the same failure pattern in developer threads: people pick a model first, usually whatever is trending, then discover it does not fit their hardware, then OOM at inference time after a 45-minute download. The PSP experiment is the extreme version of the right way around. Start from the hardware you actually have, then find the biggest model that fits it with headroom. This article is that process, with real numbers.
One disclosure: I have not run a model on a PSP myself. I do run local GGUF models on my own servers for summarization and embedding work, and everything below about quantization sizing comes from published docs and benchmarks I link as I go. Where a claim comes from a project's own marketing, I say so.
The small-model wave is not one gimmick, it is a trend
The PSP build got attention because it is a fun extreme, but it sits in the middle of a real shift. Three data points from the last few weeks:
- An LLM on a 2007 handheld. The PSP project writeup details a 15M-parameter TinyStories model, int8-quantized (group size 64, symmetric) to 17 MB on disk, with a 32,000-token vocabulary and 256-token context. The author cross-compiled pure C with pspdev in Docker. One detail worth stealing for any inference project: they also built the reference C runtime with matching floating-point flags and diffed the output byte for byte against the PSP build. If the diff is not clean, the engine is wrong somewhere. That is a test surface, not a vibes check.
- Micro-models that do one job well. The Falcon-H1-Tiny 90M family discussion on r/LocalLLaMA describes a 90M-parameter tool-caller that hits 94.44% relevance detection, meaning it knows when a function should be called, while matching a 270M-parameter Function Gemma on that task. At roughly 90 MB quantized to Q8_0, a model like this runs on any modern phone or Raspberry Pi.
- Quantization got much better at the small end. Unsloth Dynamic 3.0 GGUFs, which hit the Hacker News front page on August 19, 2026, report up to 10% better top-1 accuracy at the same file size versus other providers, using post-training quantization with a curated calibration dataset. Their UD-IQ1_S file for Qwen3.8-27B is 6.2 GB, 89% smaller than the original model, while retaining about 72% of top-1 accuracy.
The common thread: the floor for "hardware that can run a useful model" keeps dropping. GPT-3 was 175 billion parameters in 2020. Useful work now happens at 15M on a game console and at 90M on a phone.
What tiny models are actually good for
Before the how-to, a reality check, because "runs on a PSP" does not mean "writes your code." Based on the documented results above and my own use, here is the honest capability map:
- Routing and classification. The 90M tool-caller's 94.44% relevance detection is the killer app. Deciding which big model or function should handle a request is a small task, and paying a frontier API for it is waste. This is exactly how I use a small local Qwen GGUF: cheap triage before anything expensive runs.
- Autocomplete and constrained generation. The Falcon-H1-Tiny line ships a code-completion variant that runs inside VS Code through the Continue plugin. Small models excel when the output space is narrow.
- Structured extraction. Pulling fields from text into JSON is a classification problem wearing a generation costume. Sub-1B models do this well when prompted strictly.
- Not general chat, and definitely not agents. Unsloth's own docs are blunt about this: accuracy drops sharply below their UD-Q2_K_XL format, 1-bit models can produce empty responses in non-thinking mode, and tool calls with heavily quantized models are explicitly not recommended because they fail, repeat, or never fire. Tiny models retain general knowledge, not procedure.
A useful mental model from the community: if you scaled a well-designed 90M architecture to about 1B parameters, you would likely cover most everyday local use cases like chat, tool calling, and light coding, all under 500 MB quantized. We are not there yet across the board, but the direction is clear.
The hands-on part: size a model to your hardware in 15 minutes
Here is the process I use, in order. It assumes llama.cpp, which is the standard GGUF runtime and what both the PSP project's quantization approach and Unsloth's files target.
Step 1: Measure real free memory, not spec memory. Run free -h on Linux or check Activity Monitor on macOS. The number you care about is available memory with your normal workload running, because your model shares the machine with everything else. A "16 GB" laptop with a browser open has maybe 8-10 GB to give.
Step 2: Budget for more than the file size. The loaded model needs weight memory plus context memory plus compute buffers. A practical rule from the llama.cpp community: expect total usage around 1.3 to 1.5x the GGUF file size at moderate context. A 5 GB file wants 7 GB of headroom.
Step 3: Pick your quant tier by budget, using the current data. For the Qwen3.8-27B example from Unsloth's published tables:
- UD-Q4_K_XL: near-original quality, roughly 24 GB class. This is the "big laptop or desktop GPU" tier. Start here if it fits.
- UD-Q2_K_XL: 9.83 GB, with about 8% better accuracy than the next-best file at that size per Unsloth's tests. This is the 16 GB machine tier, and per the HN discussion, UD-IQ3_XXS at 10.9 GB is another good fit for 16 GB machines.
- UD-IQ1_S: 6.2 GB, retaining about 72% of top-1 accuracy. The fallback for 8 GB machines. Fine for extraction and classification, weak for reasoning.
Note these are vendor benchmarks, not independent ones. The HN thread on the release includes pushback that 2-bit is "a lobotomy" for a model this capable, which matches my experience: quality cliffs are real, and they are task-dependent. Test on your own workload before trusting any table, including this one.
Step 4: Download and smoke-test with llama.cpp. The minimal loop is:
# install llama.cpp, then either point -hf at a repo/quant...
llama-cli -hf unsloth/Qwen3.8-27B-GGUF:UD-Q2_K_XL \
--jinja --temp 0.6 --top-p 0.95 --min-p 0.01 \
--ctx-size 8192 --seed 3407
# ...or run a local file directly
llama-cli -m ./model-UD-Q2_K_XL.gguf -p "Summarize: <your text>"
If you see repeated loops on a small quant, Unsloth's docs recommend setting presence_penalty to 1.5 or higher. That one flag fixed most loop degeneration in their testing.
Step 5: Steal the PSP project's test discipline. Whatever your model does in production, write down the exact prompt, the seed, and the temperature, and keep the expected output. When you swap quantizations or upgrade llama.cpp, re-run the fixed prompt and diff. The PSP author proved their whole stack correct by getting a byte-identical completion to the x86 reference. You do not need byte equality for a temperature-0 summary, but you do need a regression test that tells you when a model swap changed behavior.
The decision checklist (save this)
- 8 GB RAM or less: sub-1B specialist models for classification and routing, or a 6-7 GB 1-bit/2-bit quant of a bigger model for extraction only. Expect narrow competence.
- 16 GB RAM: a Q2/Q3-class dynamic quant of a 27B model, or a Q4_K_M of a 7-8B model. My pick for the best single-machine experience today.
- 24 GB+ VRAM: Q4-class 27B and up. This is where local coding assistance becomes genuinely useful.
- Anything doing tool calls or agent work: stay at Q3 or better, per Unsloth's own guidance, and test the tool-call path explicitly.
- Every tier: pin the model revision you tested, keep the fixed-prompt regression output, and prefer UD-prefixed Dynamic 3.0 files when your size class matches, since they are the accuracy leaders at equal size per the vendor's published comparisons.
And if you want the true edge case for fun: the PSP runtime targets 5-15 tokens per second once the author's planned VFPU kernel, using the console's vector coprocessor, replaces the scalar matmul. Even the floor is rising.
What I would do differently if I started today
I spent real money on API tokens in 2025 doing classification and routing with frontier models, because I did not know micro-models existed. If I were starting today: a 90M-class tool-caller running locally for all routing decisions, a 7-8B Q4 model on the 16 GB box for extraction and summarization, and frontier APIs only for the tasks that genuinely need reasoning. The PSP experiment is a reminder that the constraint was never the hardware. It was knowing how small "useful" actually is.
I write about AI infrastructure, developer tools, and the unglamorous engineering underneath the headlines every week. Subscribe, it is free.
Now you: do you run any models locally, and what hardware? I am curious whether people are still maxing out 16 GB machines or whether the micro-model wave has some of you downgrading your model ambition on purpose to keep latency near zero. Tell me what you run and what it refuses to do well.
Sources: PSP LLM project writeup, analysis of the PSP experiment's benchmarks, Falcon-H1-Tiny 90M r/LocalLLaMA thread, Unsloth Dynamic 3.0 documentation, Unsloth Dynamic 3.0 coverage of the accuracy and size claims, Hacker News discussion of the release
Top comments (0)