Every local-LLM tool tells you what a model is. Almost none of them tell you, before you spend twenty minutes and twelve gigabytes of bandwidth, whether it will actually run well on the machine in front of you. llmfit, a Rust CLI that shot to 32,000+ GitHub stars this month, exists to answer exactly that question — and on August 17 it shipped a change that's more interesting than the tool itself: it stopped trusting its own math and started asking users to prove it with real hardware.
That's not a minor changelog line. It's a small, well-executed example of a pattern more infra tools should copy: ship an estimate, be honest that it's an estimate, and build the machinery to slowly replace it with verified reality.
What llmfit actually does
llmfit's pitch is blunt: "Hundreds of models & providers. One command to find what runs on your hardware." Run llmfit with no arguments and you get an interactive TUI that detects your RAM, CPU, and GPU, then scores every model in its catalog — sourced from the Hugging Face API and embedded into the binary at compile time — across four axes: quality, speed, fit, and context window. The table tells you not just "this model exists" but "this model will run well, this one will be slow, and this one won't fit at all."
For scripting and agent use there's a classic CLI mode:
llmfit fit # table of all models ranked by fit
llmfit recommend --json # top picks as JSON
llmfit info "<model>" # one model: fit analysis, estimate basis, verify commands
llmfit bench # measure real tok/s against your running provider
It's installable via Homebrew, Scoop, MacPorts, uv/pip, a Docker image, or straight from crates.io, and it's MIT-licensed with signed Windows binaries. It doesn't run any model itself — it hands off to whatever's already installed: Ollama, llama.cpp, MLX, Docker Model Runner, LM Studio, or RamaLama, whose auto-discovery landed in the August 17 release (v1.1.10) alongside Qwen3.5 family support.
The mechanism: memory bandwidth, not vibes
The interesting part isn't the TUI, it's the estimation model underneath it, and llmfit's docs are unusually candid about how it works.
Hardware detection is multi-vendor: nvidia-smi for NVIDIA (aggregating VRAM across multiple cards, falling back to name-based estimation if reporting fails), rocm-smi for AMD, sysfs/lspci for Intel Arc, system_profiler for Apple Silicon (where unified memory doubles as VRAM), and npu-smi for Ascend chips. From there it identifies the acceleration backend — CUDA, Metal, ROCm, SYCL, or CPU — because that determines both what fits and how fast it runs.
Memory sizing isn't naive parameter-count math. llmfit walks a quantization hierarchy from Q8_0 down to Q2_K, picking the highest quality that actually fits in available memory, and it treats Mixture-of-Experts models correctly: Mixtral 8x7B has 46.7B total parameters but activates only ~12.9B per token, so llmfit estimates ~6.6GB of VRAM with expert offloading instead of naively assuming the full 23.9GB. That single detail matters — the maintainers note that a rival tool, llm-checker, treats all models as dense and will overestimate memory needs for anything MoE, including DeepSeek-V3.
Speed estimation is grounded in a genuinely simple physical model: token generation in LLM inference is memory-bandwidth-bound, since every token requires reading the full model weights from VRAM at least once. So the formula is:
speed_estimate = (bandwidth_GB_s / model_size_GB) × efficiency_factor
llmfit ships a lookup table of measured bandwidth for roughly 80 GPUs across NVIDIA, AMD, and Apple Silicon, with an efficiency factor (0.55 by default, tunable in the TUI's Advanced Configuration panel) that accounts for kernel overhead and KV-cache reads. For GPUs it doesn't recognize, it falls back to flat per-backend speed constants (CUDA at 220, Metal at 160, CPU x86 at 70, and so on). The whole approach is validated against public llama.cpp community benchmarks on Apple Silicon and NVIDIA T4 hardware — and, crucially, llmfit info <model> shows you the exact inputs behind any given number so you can sanity-check it yourself rather than taking it on faith.
That transparency is the setup for the actual news.
What shipped on August 17: turning guesses into a dataset
Until now, every number llmfit produced was a prediction from a formula, never a measurement. The new feature — flagged at the top of the README as "benchmark & share" — closes that loop. The workflow, walked through step by step in the project's benchmarking guide, looks like this:
- Find a model in the TUI, press
dto download it through whichever provider has it (Ollama, llama.cpp, etc.). - Serve it locally —
llama-server -m ~/.cache/llmfit/models/... --port 8080 -ngl 99, or start it in Ollama/vLLM/MLX. llmfit auto-detects the running provider. - Press
bto open the leaderboard. Because the model is installed and live, llmfit offers to benchmark it first — three real inference passes against the running server, measuring actual tokens/sec and time-to-first-token. - Optionally press
Spacebefore running to opt into sharing. When the run finishes, llmfit forks the repo, commits your result, and opens a pull request automatically — authenticating via GitHub's device flow, with noghCLI and no separate account setup required.
Every run is saved locally first (~/.config/llmfit), so nothing is lost if you decline to share, and you can batch-upload a backlog later with llmfit bench --share. Once a submission is merged, it ships in the next release: your measured number replaces the formula's estimate in the fit table, marked with a checkmark, and anyone else on identical hardware gets a calibrated number before they've run a single benchmark themselves. There's also a live community leaderboard (b, with H to browse any GPU model) and an "Inference Bench" mode (I) that batch-benchmarks every installed model across every running provider at once.
Mechanically, this is a crowdsourced calibration loop for a physics-based estimator — a pattern closer to how folding@home-style distributed projects or MLPerf submissions work than to how most CLI tools handle "accuracy." It's a genuinely clever piece of open-source infrastructure design: instead of asking users to trust a formula, or forcing them to run a heavyweight benchmark suite themselves, it makes contributing a byproduct of something they were going to do anyway (download a model, check if it's fast enough).
Why this matters more than it looks
The proliferation of open-weight models is the whole reason this problem exists. There are now dozens of viable families — Llama, Qwen, Mistral, Gemma, Phi, DeepSeek, GLM, Kimi, Granite — each shipping in half a dozen quantizations and often as MoE variants with wildly different effective footprints. "Will this run on my machine" used to mean downloading a multi-gigabyte file, trying to serve it, and finding out the hard way that it swaps to disk or produces two tokens a second. That's real cost: bandwidth, time, and — for anyone provisioning a fleet of dev laptops or CI runners for local inference — actual money spent guessing at GPU specs before a purchase.
An estimator that's honest about being an estimator, and that gets more accurate the more people use it, is a meaningfully better trust model than either extreme: blind trust in vendor-reported specs, or the alternative of a tool that only tells you the truth after you've already paid the cost of running the model. It also sidesteps a lock-in concern that dogs a lot of AI tooling right now — llmfit doesn't care which runtime or provider you use, it's a decision layer that sits above Ollama, llama.cpp, MLX, and friends rather than replacing any of them.
Practical use cases
A few concrete ways this actually gets used, based on the CLI surface: a team choosing a coding model to run inside an agent harness on developer laptops can run llmfit recommend --use-case coding --json and get a ranked, hardware-scored shortlist instead of guessing from a leaderboard that assumes datacenter GPUs. Someone deciding whether a $1,500 GPU upgrade is worth it can check llmfit info "<model>" against their current card and a hypothetical one side by side before spending anything. A platform team sizing CI runners or edge devices for local inference can script against the JSON output to gate which models are even offered as options, instead of discovering mid-pipeline that a runner OOMs on a 30B model. And for anyone building an agent that needs to pick its own local model at runtime — the stated audience for the --json flag and the OpenClaw integration doc in the project — llmfit becomes a dependency-free hardware oracle rather than a human decision.
On the security and maintainability side, the project is reasonably careful: the README states plainly that llmfit "will not transfer any information to other networked systems unless specifically requested by the user," meaning hardware fingerprinting stays local unless you explicitly download a model, query a runtime provider, or opt into the benchmark-sharing flow. The optional GITHUB_TOKEN used for the PR-sharing feature is standard device-flow OAuth rather than a long-lived credential baked into config, and Windows binaries are signed through SignPath's free open-source program — a small but real signal of maintenance seriousness for a tool people are about to run with GPU and filesystem access.
Where the story is thinner than the pitch
A few things the announcement doesn't dwell on:
Coverage is still the bottleneck. The bandwidth table covers ~80 recognized GPUs; anything outside it falls back to flat per-backend constants that ignore the actual card. The benchmark-and-share loop fixes this over time, but only for whatever hardware the community happens to own and bothers to submit — a random ROCm card or an unusual multi-GPU rig could stay on the generic fallback indefinitely.
"Ships in the next release" is a real lag. Merged benchmark PRs don't update anyone's numbers live; they land in the embedded model database only when a new binary is cut. Between releases, the community leaderboard shows real numbers, but the default fit table a first-time user sees is still whatever was baked in at the last build.
Trust flows through one maintainer's review queue. Every shared benchmark becomes a GitHub PR that has to be reviewed and merged by the project (currently maintained essentially by @AlexsJones) before it counts. That's a reasonable quality gate against bad or gamed submissions, but it also means the crowdsourcing loop is only as fast as one person's PR review bandwidth — worth watching as the star count (and presumably submission volume) keeps climbing.
It's an estimate of inference fit, not a training or fine-tuning tool. llmfit tells you whether a model will load and generate tokens at a usable speed; it says nothing about fine-tuning memory requirements, batch serving under concurrent load, or accuracy/quality regressions from aggressive quantization beyond a generic "quality" score.
Competitive context
The closest direct alternative the project names itself is llm-checker, a Node.js CLI that takes the opposite philosophical approach: rather than estimating from specs, it actually pulls and runs models via Ollama to measure real performance. That's more accurate per-run but slower and more expensive to use for browsing dozens of candidates, and — per llmfit's own README — it doesn't understand MoE architectures, so its memory estimates for models like Mixtral or DeepSeek-V3 are based on total parameter count rather than active parameters. Ollama and LM Studio themselves don't attempt this kind of pre-download compatibility scoring at all; they'll happily let you start pulling an 80GB file that was never going to fit. Hugging Face's own model pages give you parameter counts and file sizes, but nothing hardware-aware.
llmfit's real innovation is refusing to pick a side in the estimate-vs-measure tradeoff — it estimates by default for speed of browsing, then gives you a frictionless path to convert any specific candidate into a real measurement, and now a frictionless path to donate that measurement back to everyone else's estimate.
An honest read
This is a well-scoped, technically serious tool solving a real and growing pain point, built by someone clearly steeped in how GPU inference actually behaves (the MoE-aware sizing and memory-bandwidth formula are not things you get right by accident). The benchmark-and-share feature is the smartest part of the whole project: it turns an inherent weakness — "our numbers are just estimates" — into a growth loop, and it does it without asking users to leave the tool or create new accounts. Device-flow GitHub auth for a one-off PR is a nice piece of UX craft that a lot of larger, better-funded projects skip.
The risk isn't in the mechanism, it's in scale. A single-maintainer PR review queue and a release-gated data pipeline both work fine at today's traffic; whether they hold up if this keeps trending is an open question the project hasn't had to answer yet. And the whole premise — memory-bandwidth-bound token generation — is a good first-order model but not the whole story once you're doing batched serving, speculative decoding, or anything beyond single-user local inference, none of which llmfit claims to estimate.
Who should actually use this
Try it now if you regularly evaluate open-weight models for local or self-hosted use — picking a coding model for an agent harness, sizing a GPU purchase, or deciding what a CI runner or edge device can realistically serve. The --json output makes it trivial to wire into a script or an agent's tool-calling layer for automated model selection.
Wait if you're on unusual or very new hardware where the bandwidth table and community leaderboard are thin — you'll get the generic fallback estimate, which is honest but not especially precise, until more people on your exact GPU contribute numbers.
Ignore it if you're exclusively calling hosted model APIs (OpenAI, Anthropic, Google) with no local or self-hosted inference in the picture — there's no hardware-fit question to answer.
Given how quickly the open-weight model catalog is expanding and how differently every quantization and MoE variant behaves on real silicon, the estimate-then-verify loop llmfit just shipped is a small idea worth more attention than it's gotten: what other developer tools are still shipping confident numbers that nobody has actually measured, and would a similar crowd-verification loop fix them — or just shift the trust problem from "the vendor's estimate" to "whoever bothered to submit a PR"?
Sources:
Top comments (0)