What It Really Costs to Run a Local LLM 24/7 (Power, Amortization, Cost per Query)

Overnight Desk — Tue, 22 Sep 2026 10:45:16 +0000

Everyone asks what hardware a local LLM needs. Almost nobody asks what it costs to leave one running. I did the math for a Raspberry Pi 5 serving a small model around the clock.

What the board actually draws

A Pi 5 idles around 2.7-3.5 W with a minimal OS. Sustained LLM inference on all four cores pushes it to roughly 7-9 W, more with an NVMe hat or USB SSD. Add an active cooler (0.5-1 W) and any accelerator HAT, and budget headroom on top.

The official 27 W (5V/5A) USB-C PSU exists for a reason: undervoltage during a long generation run shows up as silent errors and throttled clocks before it ever shows up as a reboot.

The monthly cost reality

At a sustained 8 W average, 24/7 operation is about 5.8 kWh per month. At typical residential rates that's roughly $1-2/month - cheap, but not zero, and it triples with a second board or an accelerator.

The number that actually matters: cost per query

Take monthly power + hardware amortization (a complete Pi 5 build over 3 years is ~$4-5/month) and divide by your monthly queries.

Below a few thousand queries a month, a cloud API at fractions of a cent per call usually wins. Above it, local wins and keeps winning forever. Your crossover point depends on your rates and query sizes - which is why this needs a calculator, not a blog table.

Do your own math

I packaged both sides of this: a Power Budget Planner that specs PSU, cooling draw and runtime budgets in one sheet, and a Total Cost Calculator that runs hardware + power + time against cloud API prices and gives you cost per month and per query.

More guides: Local AI on Raspberry Pi - hub

What's your setup costing you? Curious whether people actually meter their boards.

How to Benchmark LLMs on a Raspberry Pi 5 (llama.cpp, Step by Step)

Overnight Desk — Tue, 22 Sep 2026 10:43:52 +0000

Most "Raspberry Pi AI benchmark" numbers on the internet are one-off runs with unknown settings. If you want numbers you can trust - and compare - you need a method. This is the one I use.

1. Install llama.cpp

sudo apt update && sudo apt install build-essential cmake git -y
git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp && cmake -B build && cmake --build build --config Release -j4

Build from source. Prebuilt binaries rarely match your kernel and flags, and a mismatched build can cost you 20% throughput.

2. Pick models that actually fit

A Pi 5 has 8 GB of RAM shared with the OS. Stay under ~5 GB for model + context or you'll swap and your numbers are garbage. Reliable picks in Q4: TinyLlama 1.1B, Qwen 1.5B/3B, Phi-3-mini (tight but works), Gemma 2B. Skip 7B+ unless you enjoy watching swap thrash.

3. Measure honestly

Use llama-bench with a fixed prompt and generation length. Log three numbers per run: prompt processing tok/s, generation tok/s, and SoC temperature at start and end. A run without a temperature is not a benchmark - thermal throttling on an uncooled Pi 5 shows up around 85°C and can cut throughput by a third mid-run.

./build/bin/llama-bench -m models/qwen-3b-q4.gguf -p 128 -n 256 -r 3

4. Keep runs repeatable

Fix the CPU governor (performance) for the run and note it down.
Same prompt set every time. Different prompts = different numbers.
Cool the board between runs, or say so in the log.
Record ambient temperature when comparing across days.

Going further

I run this protocol often enough that I packaged it: a Benchmark Workbook with the log sheets and test matrix pre-built, and a Report Generator that turns raw logs into a shareable report.

Full guide with more detail: Local AI on Raspberry Pi - guides hub

What are you running on your Pi? Curious what tok/s people are seeing on 3B-class models.

DEV Community: Overnight Desk