Two questions before you run a local LLM: does it fit, and how fast is it really?

Finn Liu — Sat, 17 Oct 2026 01:02:00 +0000

I keep seeing the same two failures when people try to run a model locally. The first is buying or allocating hardware that cannot hold the model plus its context. The second is trusting a single tokens/s number that was measured under completely different conditions.

This is not a product pitch. It is the checklist I use now, plus a small tool I built because I kept redoing the arithmetic.

1. Separate memory fit from speed

These are two different questions and they deserve two different kinds of evidence.

Memory fit is arithmetic you can do yourself. Start with the stored size of the quantized weights, add the KV cache for the context you actually want, then leave headroom for the runtime, the driver and the desktop. A 70B model at Q4 is roughly 40 GB of weights before you add anything else, so "it runs on a 24 GB card" is usually a configuration that offloads to system RAM and then runs slowly.

Speed is a measurement, and it only means something with its conditions attached: exact hardware, exact checkpoint, quantization, runtime and version, context and prompt length, concurrency, and how the number was produced (prefill / TTFT / decode). A decode number from a single stream is not a throughput number from eight concurrent requests. A vendor claim is not a measurement. If the source does not name all of that, the honest answer is "unknown", not a green check.

Note especially: prefill time and tokens/s are not interchangeable. If you only have TTFT, you are looking at a derived prefill estimate, and it should be labelled that way.

2. Pick the runtime before you shop

The runtime decides what is even possible. CUDA, ROCm, Metal, Vulkan, CPU and NPU are separate lanes with separate floors, and a feature that exists on one lane is not proof for another. On Apple silicon, for example, MLX is the native path; on an NVIDIA server, vLLM or SGLang are the throughput paths; on a mixed CPU+GPU box, an offloading runtime for mixture-of-experts models is a different trade again.

3. Serving Apple silicon as an OpenAI-compatible endpoint

A detail worth knowing if you run MLX: mlx-lm ships mlx_lm.server, an OpenAI-compatible HTTP server. It exposes /v1/chat/completions, /v1/completions and /v1/models, defaults to 127.0.0.1:8080, and takes --model plus --host / --port. It is convenient for pointing an existing OpenAI client at your Mac.

The important caveat is upstream's own: mlx_lm.server implements only basic security checks and is not recommended for production. Keep it bound to localhost unless you add authentication and network controls in front of it. For a single-user local setup that is fine; for anything exposed, it is not.

4. Write the conditions down

Whatever you measure, record the conditions next to the number. Six weeks later, "42 tok/s" without "M5 Max, Q4, 8k context, single stream, mlx-lm x.y" is useless to you and misleading to anyone else.

If you want a starting point, I keep a sourced reference that separates memory fit, practical context, measured speed, vendor claims and unknowns, and links every number to its source: LocalForge. It is a decision guide, not a benchmark lab - it publishes no numbers of its own.

Build Notes: A Browser-Local Image Compressor for Everyday Upload Problems

Finn Liu — Thu, 08 Oct 2026 23:41:28 +0000

I built Quick Image Kit as a small browser-local image toolkit: quickimagekit.com

The product is intentionally practical. Many people do not need a full photo editor. They need to make a phone photo small enough for an upload form, profile page, email attachment, document portal, school system, job application, or favicon package.

The design constraint is simple: the image should be processed in the browser where possible, so the original file does not need to be uploaded to a processing server.

Why local processing matters

Image compression sounds low-risk until the files are personal. The same workflow often handles:

ID photos
passport-style photos
document scans
school records
legal or medical paperwork
internal screenshots
application form attachments

For those files, the safest default is not "upload first, compress later." A browser-local flow lets the user load the image, create a smaller copy, download the result, and then upload only that final copy to the actual destination.

That is the main positioning behind Quick Image Kit's browser-local image compressor.

The main UX problem is not the codec

For an image compressor, it is tempting to start with codecs, quality settings, or performance. Those matter, but they are not the first user problem.

The first problem is that users usually do not know what changed:

Did the image get smaller?
Is it now under the upload limit?
Did the dimensions change?
Is the face still clear?
Is document text still readable?
Is the output format accepted by the form?

So the interface has to explain the result in normal upload-language, not just expose a quality slider.

Resize before heavy compression

Large phone photos often start at 3000 to 4000 pixels wide. If the target is a small profile photo or form attachment, resizing usually improves the result more than pushing JPEG quality lower and lower.

The practical order is:

Crop away unnecessary background.
Resize when the source image is far larger than the upload needs.
Compress toward the target size.
Check readability before downloading.

That is why a useful image toolkit needs resizing, cropping, conversion and compression together, even if each tool stays simple.

Exact KB pages are workflow pages, not just SEO pages

Searches like "compress image to 20KB" or "compress photo under 200KB" look repetitive, but the user intent is real. Someone is blocked by a form that rejects their file.

The page should not only say "upload your image." It should explain what to try when the target is too small:

crop first if there is empty background
resize if the source is a large phone photo
use JPG for photo-like images when the form accepts it
avoid shrinking documents so far that text becomes unreadable
aim slightly below the limit, not dramatically below it

For that reason, Quick Image Kit has a separate exact-size workflow.

The growth lesson

The technical build was the easy part. The harder part is that a new utility site does not get traffic just because the pages exist.

The current growth work is deliberately simple:

ship focused pages for real upload scenarios
create useful external tutorials
submit to relevant tool lists
track whether links become live instead of treating submissions as wins
look for search impressions and referrals, not just sitemap counts

The useful distinction is this: a submitted link is not a live backlink. A live backlink is only counted after the external page actually links back.

What I would keep if rebuilding it

If I rebuilt the tool from scratch, I would keep these decisions:

keep the core image work local in the browser
make the output file size and dimensions visible
create separate workflows for exact KB, forms, passport-style uploads, and documents
keep the original file unchanged
avoid login requirements for the core task
make each page answer one concrete upload problem

The tool is here: quickimagekit.com

And the main compression workflow is here: compress an image in the browser

DEV Community: Finn Liu