Will that 70B model fit on your machine? My CLI answers before you download 40 GB

Hamza — Sat, 18 Jul 2026 14:12:29 +0000

Every local LLM user knows this cycle: find a promising model, download 40 GB of GGUF, watch it OOM - or worse, watch it almost fit and crawl at 2 tokens/s while your context gets silently evicted. The information to predict all of this exists. It's just annoying math nobody wants to do by hand.

So I built a CLI that does it. But first, a confession.

The version nobody could install

Four months ago I posted a weekend hack called llm-neofetch-plus - a neofetch-style hardware tool with local-LLM extras. People actually tried it, which is how I learned two embarrassing things:

1. The PyPI package was fundamentally broken. My pyproject.toml used packages.find with directories that had no __init__.py, so the wheel shipped without the main module. pip install llm-neofetch-plus → ModuleNotFoundError. It only ever worked from a repo clone. For months. 🙃

2. YAML has no \033 escape. My config stored ANSI colors as "\033[1;34m". YAML parses that as \0 (a NUL byte!) followed by the literal text 33[1;34m - so on any machine where the config actually loaded, the "colors" were printed as visible garbage: 33[1;36m╔════.... This bug shipped on day one and I never saw it, because a separate encoding bug prevented the config from loading on my Windows machine at all. Two bugs in a trench coat, hiding each other.

Enough people cared despite all this that I did a full rewrite. Here's what it became.

"Can I run it?" - answered properly

$ llm-neofetch can-run llama3.1:8b --context 8192

┌──────────────────────────────────────────────────────────┐
│  ✓ llama3.1:8b (8B) can run on this system               │
└──────────────────────────────────────────────────────────┘
  Context: 8,192 tokens • VRAM: 2.0 GB (2.0 free) • RAM: 15.7 GB (3.8 free)

  Quant   Memory  Capacity  Right now  ~Speed    Max ctx
  Q2_K    4.0 GB  ✓ CPU     ✗ not now  25 tok/s  75K
  Q4_K_M  6.0 GB  ✓ CPU     ✗ not now  14 tok/s  60K
  Q8_0    9.4 GB  ✓ CPU     ✗ not now  8 tok/s   32K

Two verdicts per quantization, because they answer different questions:

Capacity - can this hardware run it at all? (totals)
Right now - does it fit in the memory that's actually free at this moment, counting VRAM used by other apps and RAM held by your 47 browser tabs?

That second column exists because a commenter on my original post pointed out that a 24 GB GPU with 4 GB already in use should not get a "70B fits" verdict. They were right. It took me four months, but it's fixed.

Exit code is 2 when nothing fits, so you can script it.

The math (the fun part)

Total memory is estimated as weights + KV cache + overhead:

Weights: params × bits_per_weight / 8. Q4_K_M is ~4.5 effective bits, so an 8B is ~4.2 GiB.
KV cache: 2 × n_layers × n_kv_heads × head_dim × 2 bytes per token, from a lookup table of common architectures. This is GQA-aware - it matters enormously. Llama-2 13B (full MHA) burns ~25× more KV memory per token than Qwen2.5 14B (GQA with 8 KV heads), despite similar parameter counts.
Context scales linearly: an 8B at 4K context needs ~0.5 GiB of KV cache; at 64K it needs ~8 GiB. Same model, wildly different memory story - which is why every verdict is computed at a specific context length.

Speed estimates use the memory-bandwidth bound: generating one token reads every weight once, so tok/s ≈ 0.7 × bandwidth / weight_bytes. It's labeled as an upper bound, because that's what it is. If you want truth instead of estimates, --bench-llm runs a real generation through the Ollama API and reports actual tok/s from eval_count / eval_duration.

It also knows what you already have

The tool detects installed backends (Ollama with version + running state + model count, LM Studio, llama.cpp, vLLM), scans your downloaded models, and renders a verdict for each one:

  Model                     Size     Source   Fits
  qwen2.5:14b-q4_K_M        8.4 GB   Ollama   ✓ CPU (RAM)
  llama3.1:70b-q4_K_M       39.6 GB  Ollama   ✗ too big

Plus: running LLM processes with per-process VRAM, AI runtime detection (CUDA/ROCm/Vulkan/DirectML/Metal), a --watch live monitor, --json for scripts, HTML export, and themes.

Try it

pip install llm-neofetch-plus   # and it genuinely works now
llm-neofetch                     # full hardware report
llm-neofetch can-run llama3.1:70b --context 16384

HFerrahoglu / llm-neofetch-plus

LLM-Neofetch++ is an advanced system information tool designed specifically for local LLM (Large Language Model) usage. It provides detailed hardware detection with personalized recommendations for running AI models on your system.

🚀 LLM-Neofetch++

Advanced System Information Tool for Local LLM Usage

Show detailed hardware specs optimized for running local AI models

✨ Features

🔍 Comprehensive Hardware Detection

✅ CPU: Model, cores, threads, frequency, temperature, usage
✅ GPU: NVIDIA (nvidia-smi/pynvml), AMD (amd-smi/rocm-smi), Intel Arc
✅ VRAM: Total, used, and available video memory
✅ NPU: Intel AI Boost, AMD XDNA, Apple Neural Engine
✅ RAM: Capacity, module speed, and bandwidth estimate
✅ Storage: Disk type (NVMe/SSD/HDD), capacity, speed benchmarks
✅ Battery: Charge level, power status, time remaining (laptops)
✅ Apple Silicon: M1-M4 variants with GPU cores and memory bandwidth
✅ AI Runtimes: CUDA, ROCm, Vulkan, DirectML, Metal versions
✅ Environment: WSL, Docker, and cloud VM (AWS/GCP/Azure) detection

🎯 Smart AI/LLM Features

🤖 Context-Aware Recommendations: max context length and token/s estimates per model tier, computed from weights + KV cache on your hardware
…

View on GitHub

Honest caveats: the estimates are heuristics with clearly-marked assumptions, and the AMD / Apple Silicon detection paths are written defensively but untested on real hardware - if you run either, a bug report (or even a "works fine") would genuinely help. Tell me where the math is wrong. That's how the last version got better.

DEV Community: Hamza