How to Run AI Locally: Models, Hardware, and Real-World Speed
Running AI locally has gone from a niche experiment to something ordinary people actually do. If you have been wondering whether you can run AI locally on your own hardware — without a cloud subscription, without your data leaving the building — the answer in 2026 is yes. And you do not need a server room to do it.
This guide covers how to run AI locally: which models work, what hardware you actually need, and what real-world performance looks like for everyday tasks.
Why Run AI Locally at All?
The case for local AI comes down to three things: privacy, cost, and always-on availability.
Cloud AI tools are fast and convenient, but every prompt you send travels to a data center somewhere. For personal notes, client data, internal documents, or anything you would not paste into a public form, that matters. Running AI locally means your data stays on your device — full stop.
Cost is the other factor. A stack of monthly AI subscriptions adds up. A one-time hardware purchase that runs AI indefinitely can be cheaper over two or three years, depending on how much you use it.
Finally, local AI is always on. No outages, no rate limits, no quota resets at midnight.
What Models Can You Run Locally?
The local AI model ecosystem has grown significantly. The most common options:
Small to medium open-weight models (3B–14B parameters)
These run well on consumer-grade hardware with 8–16GB of RAM or GPU memory. Good for summarisation, Q&A, writing assistance, code help, and light automation. Models in this range include various Llama, Mistral, Gemma, and Phi families.
Quantised versions of larger models
Quantisation reduces a model's memory footprint by lowering numerical precision. A 30B or 70B model quantised to 4-bit can fit in 8–16GB. Quality is slightly lower than the full-precision version, but the gap has narrowed considerably.
Specialised models
Some models are fine-tuned for coding, function calling, or tool use. If your workflow is specific — writing code, answering questions from a document, or running structured automations — a smaller specialised model can outperform a larger general-purpose one.
Optional cloud providers alongside local
Running locally does not mean you can never use a cloud model. A well-designed local AI setup lets you route requests to a cloud provider (like Claude or GPT) when you want a stronger model, and handle everything else locally. This hybrid approach gives you the best of both worlds.
What Hardware Do You Actually Need to Run AI Locally?
This is where most people get confused. The honest answer: it depends on what you want to run and how fast you need responses.
A standard laptop or desktop (CPU only)
You can run small quantised models on any modern laptop. Expect 1–5 tokens per second on a CPU — slow for a conversation, but usable for background tasks.
A machine with a dedicated GPU
A mid-range GPU with 8–12GB of VRAM dramatically improves speed. You can run 7B–13B models at 20–50 tokens per second, which feels close to real-time.
Dedicated AI hardware (edge inference boards)
Boards built for AI inference — like the NVIDIA Jetson Orin Nano Super — are designed to deliver consistent AI performance at low power. The Jetson Orin Nano Super 8GB offers around 67 TOPS (tera-operations per second) and runs on roughly 20W. It is not a gaming GPU, but for always-on AI assistants that need to run 24/7 without heating a room or running up an electricity bill, the efficiency trade-off makes sense.
Minimum practical specs for a usable local AI setup:
- 8GB RAM (16GB recommended for larger models)
- 20GB+ free storage for models
- Any modern CPU (ARM or x86)
- Optional: a GPU or dedicated AI accelerator for faster inference
Real-World Speed: What to Expect
Speed in local AI is measured in tokens per second (tok/s). One token is roughly ¾ of a word.
| Hardware | Model size | Typical speed |
|---|---|---|
| CPU only (laptop) | 7B Q4 | 2–6 tok/s |
| Mid-range GPU (RTX 3060) | 7B Q4 | 40–80 tok/s |
| Jetson Orin Nano Super | 7B Q4 | 10–20 tok/s |
| High-end GPU (RTX 4090) | 13B Q4 | 60–100+ tok/s |
For reference, reading speed is around 4–5 words per second, or roughly 5–7 tokens per second. Anything above that feels near-instant in a chat interface.
The Jetson-class hardware sits in the 10–20 tok/s range — fast enough for real conversation, email drafts, document summaries, and automation tasks, at a fraction of the power draw of a full GPU workstation.
What Can You Actually Do With Local AI?
Once you run AI locally, the practical use cases multiply fast:
- Summarise documents and emails without uploading them anywhere
- Draft and edit text with a private writing assistant
- Answer questions from your own files (RAG-style retrieval)
- Automate repetitive tasks — checking inboxes, responding to templates, organising data
- Write and review code without it leaving your machine
- Control smart home or local services through an agent that acts on your behalf
The constraint is always speed and model quality. Local models at the 7B–13B range are genuinely capable for most everyday tasks. For the heaviest reasoning or the most complex writing, you might still route to a cloud model — but for 80–90% of daily use, local handles it cleanly.
The Easiest Way to Run AI Locally
If you want to run AI locally without spending a weekend on configuration, the options are:
DIY on a spare PC — install Ollama or LM Studio, download a model, and start experimenting. Free, but requires setup time and ongoing maintenance.
A dedicated AI appliance — hardware that comes with the software pre-installed and configured. You plug it in, scan a QR code, and it is running. ClawBox is one example: a Jetson Orin Nano Super 8GB with 512GB NVMe and OpenClaw pre-installed for €549. The appeal is not the specs — it is that you do not configure anything.
The DIY route gives you more control. The appliance route gives you your time back.
FAQ
Can I run AI locally on a Raspberry Pi?
Yes, with limitations. Small quantised models (1B–3B) run on a Pi 5, but speeds are slow — 1–3 tok/s. Usable for scripts and automation, not great for real-time conversation.
Does local AI work without an internet connection?
Yes. Once the model is downloaded, it runs entirely offline. You only need an internet connection if you route to a cloud provider.
How much storage do models take?
A 7B model at 4-bit quantisation is roughly 4–5GB. A 13B model is around 8GB. Plan for 20–50GB if you want to keep several models available.
Is local AI as good as ChatGPT?
For many tasks, yes — and for private tasks, it is the only option that keeps your data fully offline. For the most complex reasoning, frontier cloud models still have an edge, which is why a hybrid local-plus-cloud setup is often the best of both.
Running AI locally is no longer a project for researchers and hobbyists. The hardware is accessible, the models are capable, and the tooling has caught up. Whether you build your own setup or start with something pre-configured, the question is not whether you can run AI locally — it is which approach fits your workflow.
Put AI to work on hardware you own — [)
Top comments (0)