DEV Community

TECHDEV
TECHDEV

Posted on Originally published at github.com

Before you rent the GPUs, check the map: I built a CLI that picks your LLM serving engine

#llm #machinelearning #python #devops

DeepInfer collected ~19k stars in a single week (GitHub Trending scan 2026-10-28). The headlines say "production inference engine". My weekend said: it serves exactly one model (DeepSeek-V4.1-Flash), needs SM100+ or Ascend950 silicon, and runs offline-only with no KV cache system, no PD disaggregation, no load balancing. Checked against the upstream repo 2026-11-01.

That is not a knock. That is the gap a tool should close: before you rent GPUs or rewire your stack, you need the map — what you actually have, what the model actually costs in memory, and which engines are actually viable on that combination.

So I built servefit: a zero-dependency Python CLI that inventories your hardware, does the memory math, checks a dated engine-support matrix, and recommends an engine with reasons. Two minutes, no account, and the offline path needs no network at all.

What it does

  • servefit inventory — parses nvidia-smi (GPUs, VRAM, compute capability with name fallback), AMD via rocm-smi, CPU/RAM. Apple Silicon is reported as detected-with-limits, never guessed.
  • servefit fit --model Qwen/Qwen3-8B — weights + KV cache + overhead math with every assumption printed, then viable engines from a curated 6-engine matrix. Every row carries a check date (all 2026-11-01); CI fails if any row goes 90 days stale, so staleness is loud, not silent. One recommendation, one starter config. Missing production components (load balancing, disaggregation) are listed per engine, never hidden.
  • servefit probe --endpoint <url> — OpenAI-compatible SSE probe measuring TTFT and approximate tokens/sec on your hardware.
  • Every command also writes --json manifests for automation.

A real run

servefit demo

fit on a Qwen3-8B lookup from the Hugging Face Hub: pulls the real arch (36 layers, 8 kv-heads, 128 head-dim) and the 8.19B param count from safetensors metadata, computes 14.9 GiB weights + 4.5 GiB KV cache at 32k context = 20.4 GiB total, and recommends SGLang; DeepInfer is ruled out with the chip and model reasons printed. The demo transcript labels what is synthetic (the inventory runs against a synthetic RTX 4090 profile) and what is scripted (the probe runs against a local fake OpenAI-compatible server); the timing it measures is this machine's, honestly labeled.

Method, stated honestly

Memory math: weights = params × bytes-per-param; KV cache = 2 × layers × kv_heads × head_dim × context × bytes-per-param; plus a stated overhead line. Every assumption is printed, so you can argue with it. Hand-computed fixtures in the test suite (8B bf16: 14.9 GiB weights, 4.5 GiB KV @ 32k, 20.4 GiB total) all match; 37 tests, all green, CI across Python 3.10–3.13.

The engine matrix is the opinionated part: a preference order (SGLang > vLLM > TensorRT-LLM > DeepInfer > llama.cpp > Ollama) gated by chip, model-family, and memory rules. Verdicts are never silent — every "viable" or "non-viable" comes with its reasons.

What it is NOT

  • It does not replace profiling. Memory math gets you to the right engine; real throughput comes from measuring your workload.
  • Apple Silicon: GPU VRAM is reported unknown by design; recommendations lean on llama.cpp/Ollama with a manual RAM check note.
  • Probe token counts are a whitespace-split approximation, labeled in the output — not the model's tokenizer.
  • The matrix covers 6 engines checked 2026-11-01. New engines and hardware appear as rows with fresh check dates, or they fail CI.

Where rivals beat us

Honest comparison, checked 2026-11-01:

  • Hugging Face's model pages and hub calculators win on data freshness and model coverage: every new model's arch is there the day it drops. servefit does its own lookup but only maintains a curated engine matrix.
  • Per-engine docs (vLLM, SGLang, TensorRT-LLM) win on depth: the full config surface, performance tuning, and multi-node guides. servefit gives one starter config, not a manual.
  • DeepInfer wins on the exact thing it was built for: serving DeepSeek-V4.1-Flash fast on SM100/Ascend hardware. servefit's matrix correctly rules it out for everything else — that is the point of the map.

Install

The servefit name on PyPI is still free; until a release is published there, install from source:

pip install git+https://github.com/techdevfun/servefit
servefit inventory
servefit fit --model Qwen/Qwen3-8B
Enter fullscreen mode Exit fullscreen mode

Roadmap: more engines, multi-GPU/NUMA topologies, and a servefit watch mode that re-checks the matrix on a schedule. Repo: github.com/techdevfun/servefit. Apache-2.0 licensed.

Top comments (0)