DEV Community

oVu X
oVu X

Posted on Originally published at github.com

Your Local LLM Is Slower Than It Should Be. This Free Skill Measures Why

Your Local LLM Is Slower Than It Should Be. This Free Skill Measures Why

Your local model answers slowly. You guess at settings. You change three flags at once. You run one test.

The number looks better. You keep the change. That number may be noise.

A new free skill stops the guesswork. It is called inference-optimisation. It turns speed tuning into a measured loop. It works on any hardware and any quantization. It works with Claude Code and any agent that reads SKILL.md files.

What it is

inference-optimisation is an agent skill. An agent skill is a file of steps that a coding agent follows. This skill tells the agent how to hill-climb the speed of a local LLM server. Hill-climb means change one setting, measure the result, and keep only real wins.

It covers decode, prefill, speculative decoding, KV cache size, startup time, and time to first token. Decode is the speed at which the model writes tokens. Tokens are the small text units a model reads and writes. This article writes tokens per second as tok/s. Prefill is the speed at which the model reads a prompt.

Time to first token, or TTFT, is the wait before the first word appears. KV cache is the memory that holds prompt context. Speculative decoding is a trick where a small draft model guesses the next tokens and the large model checks them.

The skill is free and open source under the MIT license. MIT is a permissive license that lets anyone use the code. The repo appeared on 16 November 2026. It had 1,428 stars and 129 forks when I checked it on 21 November 2026.

How it works

The skill starts with a ceiling. A ceiling is the fastest speed your hardware can reach in theory. The skill computes it from memory bandwidth and compute power. It does this before it changes anything. If the ceiling says 100 tokens per second, a target of 200 is above the physics. You learn this early, not after a week of tuning.

Then the loop runs. The agent measures a baseline. It picks one lever. A lever is one setting to change. It tests the change against interleaved controls. Interleaved controls are baseline runs placed before and after each test run.

The change must beat both controls. It must not slow anything else by more than 3 percent. A result must also pass a long-context needle test. A needle test checks that the model still finds a fact buried in a long prompt.

Every result lands in a log, including the failures. The skill says most of the value sits in the list of things that did not work and why. When the gains stop, the skill pins a clean build. A clean build is a fixed set of versions and settings that others can repeat.

Three parts ship in the repo. SKILL.md holds the procedure. The references folder holds the detail: how to compute ceilings, which levers exist, which traps to avoid, and how to publish results.

The scripts folder holds a small harness in plain Python and bash. bench.py measures any server that speaks the OpenAI API. ceiling.py computes the ceilings. queue.sh runs control and test runs in turn. mem.sh reports real memory use.

What it measured

The numbers below come from the skill's own hill-climbs. Each number names its hardware. Your numbers will differ. The method is the part that travels.

Hardware Model and setup Before → after
Intel Arc B390 laptop GPU Ornith-1.5-35B-A3B, SGLang Decode, 1 user: 43.6 → 53 tok/s. Decode, 4 users: 84.7 → 103 tok/s. Prefill at 256K: 297 → 628 tok/s. Startup: 113 → 43 s.
Intel Arc Pro B70 Qwen3.8-27B, vLLM Decode, 1 user: 91.2 tok/s. Decode, 16 users: 365 tok/s. Same card with llama.cpp: 25.0 tok/s. Agent session replay: 193 → 147 s.
RTX 3090 Qwen3.8-27B, EXL3 3 bpw 96.2 tok/s prose, 141.1 tok/s code, 1 user.
4× H200 NVL DeepSeek-V4.1-Flash, vLLM Crashed under load → 229.1 tok/s decode, 17,546 tok/s prefill. All six gates pass.

The B390 run is the one the skill grew from. Its ceiling math said 100 tok/s for one user needed two useful speculative drafts. The draft head accepts about 14 percent at depth 2. The practical target was about 88 to 97 tok/s. The team reached 53. That is about 55 percent of the ceiling.

The rest is kernel efficiency. The skill makes the agent compute this first. It tells you early when a target sits above the physics.

How to use it

First step: install it for Claude Code. Run these three commands in a terminal:

git clone https://github.com/sybil-solutions/inference-optimisation-skill
mkdir -p ~/.claude/skills
cp -r inference-optimisation-skill/skills/inference-optimisation ~/.claude/skills/
Enter fullscreen mode Exit fullscreen mode

Then ask the agent for one job. One prompt to try: hill-climb decode and prefill for the model on port 8000 under a 40 GiB cap. A second prompt to try: what is the theoretical max tok/s for this model on this machine?

No agent? The skill also runs by hand. The repo gives a quick start. Run ceiling.py on the example file to see your ceilings. Copy the example config and edit the server address and model name. Run bench.py for one measured run.

Use queue.sh to run baseline, test, baseline in turn. Run bench.py show to read the ledger table.

Try the ceiling step first, before you touch any setting. It takes minutes. It tells you whether your target is real or above the physics. That one step saves the most wasted time.

Honest limits

The numbers are hardware-specific. A B390 result says nothing about your RTX card. The skill gives you the method, not the settings. You still run the loop on your own machine.

You need a local server already running. The skill tunes a server. It does not install one. If you have no local model yet, start with Ollama or llama.cpp first. Then come back.

Single runs lie. The skill states noise at 5 to 10 percent. A single run that looks 4 percent faster proves nothing. The interleaved controls exist for this reason. Skip them and you will keep fake wins.

The skill is new. The repo holds one commit and dates from 16 November 2026. The results link to outside sources for methods and raw data. Check those sources before you trust a number. The method is sound, but the track record is short.

Speculative decoding is not free speed. The skill says it is about cost per step, not acceptance rate. A draft head that accepts often can still lose if each step costs too much. The skill prunes the draft vocabulary and sweeps the draft length for each case.

Verdict

Most local LLM tuning is guesswork with a stopwatch. This skill replaces the guesswork with a loop: ceiling, baseline, one change, control, log. The loop is the product. The numbers are proof that the loop works on real hardware, from a laptop GPU to four H200 cards.

Install it, run the ceiling step, and measure your own server before you change a flag. After reading this, you can do that in one evening. That is the point.

Follow along

I test AI tools and build in public:

Top comments (0)