DEV Community

Cover image for Finding the Right Local LLM Doesn't Have to Take Hours
Rijul Rajesh
Rijul Rajesh

Posted on

Finding the Right Local LLM Doesn't Have to Take Hours

#tools #productivity #llms

Hello, I'm Rijul. I'm building git-lrc, a micro AI code reviewer that runs on every commit. It's free and source-available on GitHub. Star git-lrc to help more developers discover the project. Do give it a try and share your feedback

Choosing a local LLM often comes down to trial and error. If you're not sure which models your hardware can comfortably run, here's a tool worth checking out.

Meet LLMFit

Repository:
https://github.com/AlexsJones/llmfit

llmfit analyzes your hardware and recommends the models that best fit your machine.

Instead of manually comparing VRAM requirements, quantizations, and model sizes, it tells you what is likely to run well on your system.

The installation instructions are available in the project's README.

For my machine, I installed it using:

curl -fsSL https://llmfit.axjns.dev/install.sh | sh
Enter fullscreen mode Exit fullscreen mode

Once installed, simply run:

llmfit
Enter fullscreen mode Exit fullscreen mode

This is the output I got:

Understanding the Output

At first glance, the table can look overwhelming.

Let's go through each column one by one.

Model

The name of the language model.

Examples include Llama 3.1 8B, Qwen2.5 14B, and many others.

Provider

The organization or repository that publishes the model.

Examples include Meta, bartowski, Unsloth, TheBloke, and other Hugging Face publishers.

Params

The number of parameters in the model.

Generally, larger models are more capable, but they also require more memory and compute.

Score

An overall recommendation score between 0 and 100.

It combines multiple factors such as:

  • Model quality
  • Generation speed
  • Hardware compatibility
  • Context length

Higher scores are generally better.

Tok/s

The estimated number of tokens the model can generate every second.

Higher values mean faster responses.

Quant

The quantization format used by the model.

Examples include:

  • Q8_0
  • Q6_K
  • Q4_K_M
  • Q3_K

Lower quantization levels usually reduce memory usage and improve speed, while slightly reducing quality.

Disk

The amount of disk space required by the selected model.

GPU

Shows how the model will actually run on your machine.

Examples include:

  • GPU
  • CPU + GPU
  • CPU

Mem %

The percentage of your available RAM or VRAM that the model is expected to use.

A value between 50% and 80% is usually a comfortable range.

Ctx

The maximum context window supported by the model.

A larger context window allows the model to process longer conversations and larger documents.

Date

The release date or last update date of the model.

This helps you judge how recent the model is.

Fit

Indicates how well the model matches your hardware.

  • Perfect — Excellent fit.
  • Good — Runs comfortably.
  • Marginal — Usable, but close to the hardware limit.
  • Too Tight — Not recommended for your machine.

Use Case

The primary scenario the model is optimized for.

Examples include:

  • General
  • Chat
  • Multimodal

That's all there is to it.

If you're planning to experiment with local LLMs, llmfit is a handy tool for figuring out which models are most suitable for your hardware before you spend time downloading them.

AI agents write code fast. They also silently remove logic, change behavior, and introduce bugs -- without telling you. You often find out in production.

git-lrc fixes this. It hooks into git commit and reviews every diff before it lands. 60-second setup. Completely free.

Any feedback or contributors are welcome! It's online, source-available, and ready for anyone to use.

Give it a ⭐ star on Github

Top comments (0)