This requires Apple Silicon (M-series) and macOS 13 (Ventura) or later. If you're on Intel, Linux, or Windows, the numbers in this article will not apply to your hardware.
TL;DR: Squish is a local LLM inference server for Apple Silicon, 1.15 to 14.7× faster than Ollama depending on how much your prompts repeat. Skip the story and install it →
Since February, many of my commits have been written by a local AI in under two seconds. No API keys, no rate limits, no internet connection, and no data leaving my machine. Getting it working took considerable effort, but once it did, it's been very reliable across dozens of repositories. The local AI software I'm describing didn't exist, not without heavy modifications to source code you don't control, or building your own from scratch. So I built it, and it solved my problem. It may not solve yours, but you can clone it, fork it, take it apart, and have fun with it.
It started with Gemini. I wired the API to a script I wrote that automates git commits and pushes. It was fast and intelligent, until I hit the hard free tier rate limits. In my case, I hit them every day. Sometimes on the second commit, sometimes on the first. Gemini returned a response message saying I was rate limited. I used it once or used it twice and was cut off. This was for simple stuff, maybe 500 to 1,000 tokens for a commit, a little more for a large diff. I put up with it for a while, but the annoying limits drove me to drop it. I searched around but couldn't find anything that could solve my problem.
So I went local, off the cloud entirely, with no rate-limiting. I pointed the script at Ollama running Mistral and configured and tested it for weeks: built a custom Modelfile, iterated prompts, fine-tuned output until the commit messages described the diff accurately. The descriptions came out great. They also took too long. End to end, a response landed anywhere from seven to ten seconds on a normal commit, and north of a minute for a large diff. So I pulled every lever and turned every knob, but the adjustments failed to reduce the response time, and my problem was still unsolved. The slow and unpredictable responses were the hard wall. When the software is written by someone else, their speed limit is your speed limit. I wanted a coherent commit message in under five seconds, ideally under three. Ollama and the models it serves could not deliver. I thought about it for over a month and came to the conclusion that I had to build something myself. Something lean and elegant that doesn't just work, it beats the baseline outright. That something is called Squish, a local AI inference server.
What Squish Is
To be clear: I did not build a model. I built the server that runs models, a framework that quantizes and compresses them, and a local format that reduces how much memory they need to run. Squish is an MLX-based local inference server, and five architectural components set it apart from existing tools:
1. A persistent daemon that keeps the model awake. Ollama loads its model lazily, on the first request, and that first request pays for it: twenty to thirty seconds of cold start while the weights load, before a single token comes back. Worse, the model gets unloaded once it sits idle, so an intermittent workflow pays that toll over and over. Squish loads the model once, when the daemon starts, and keeps it resident for as long as it's running. The cost is paid a single time; every request after is warm. Load once, never load again.
2. A two-tier KV cache that remembers. Before a model can answer a prompt, it runs prefill. It reads the entire prompt and builds the internal attention state required in memory before producing a single token. That memory state is called the KV cache. Normally the KV cache is discarded once a response is completed, and on the next request it gets rebuilt from scratch. Squish retains the KV cache, in two layers.
- The prompt cache handles exact repeats, an identical prompt sent again. If that prompt is resent, there's nothing to rebuild. Time-to-first-token (TTFT) drops to roughly 4 to 11 ms, versus about 800 ms to prefill from scratch.
- The block cache handles prompts that partially overlap previous prompts. The cache stores KV state in fixed-size blocks on disk. A block is computed only once. Any future prompt that contains overlapping blocks reuses the stored copy. This ensures the model only computes tokens it hasn't seen before. Examples include agent loops that resend a long system prompt each turn, and multi-turn conversations.
Exact or partial prompts are processed once, never repeated.
3. INT3 quantization that genuinely works, on some models. A model's knowledge lives in its parameters, also called weights, the values it learned during training. Quantization stores each parameter at lower precision, like rounding a long decimal to a couple of places. This saves memory, and on Apple Silicon it also makes the model faster. The reason is simple: the slow part of producing each token is not the math, it is moving the model's weights out of memory. Smaller weights mean less to move, and less to move means more tokens per second. The tradeoff is accuracy. Three-bit quantization (INT3) is aggressive enough that it usually breaks a model outright. However, for some model families, INT3 is stable, more on that below.
4. The optimization engine: where the speed and memory savings come from. To generate each new token, the GPU works through every weight in the model. Most tools expand the rounded-off weights back to full precision in memory before the GPU can use them. Squish never expands the full model to full-size in memory, saving a significant amount of memory. To achieve this, the quantized model weights are streamed from memory in small blocks by the GPU. During this process, each quantized block is expanded to full-size, then processed, and the next block overwrites the previous block sequentially. Only a few blocks are expanded to full-size inside the GPU at a time. Squish runs the same computation as full precision, without expanding the model weights to full-size in memory. With this memory savings, fewer bytes move through memory, improving speed at the same time.
5. A memory governor that reacts to real pressure. On a 16 GB Mac, everything shares the same memory, including the model, long conversation contexts, macOS, and every other running application. If all of these exceed 16 GB, the memory spills into disk space, which is slow and can cause thrashing, or macOS may kill the process outright. Squish watches memory-pressure signals from macOS. As memory pressure rises, the governor reduces the KV cache's size, first gently, then more aggressively if pressure keeps climbing. Under memory pressure, new requests are also given a smaller context window. This prevents allocating space the machine does not have. If memory pressure is critical, Squish rejects new requests with a clear response instead of crashing. Any requests already generating remain to finish processing. Once memory pressure reduces to a normal level, the original KV cache's size and prompt context window limits are restored.
The Benchmarking Methodology
Before I present any numbers, I'll share my benchmarking protocol. Plenty of "Ollama vs X" articles contain at least one apples-to-oranges measurement that favors the tool the author is selling. The most overlooked one is thermal. Run the favored tool first on a cool machine, then run the competing tool second once the machine is hot. This manufactures a win out of nothing but execution order. So I controlled for it. Each inference server was measured from the same 50°C baseline. A consistency check confirmed the first and last runs were taken at the same chip temperature and held to within 1.7%, so performance didn't degrade as things heated up. The temperature of the chip's silicon (its die) was logged live throughout the benchmark tests. These numbers reflect each inference server, not the benchmark order.
Hardware: Apple M3 MacBook Pro, 16 GB unified memory, running macOS 26 Tahoe, the current OS version. Normal hardware, controlled procedure: no external memory, SSD, or compute, and no fresh reboot to game the result.
Models: Qwen2.5-7B-Instruct for both, Q4_K_M on Ollama and INT4/INT3 on Squish. The two models are comparable in parameter count and quantization level. Ollama shipped a major version jump partway through this project, so I ran the full suite against both 0.18.2 and 0.30.7. They came out identical, matching to a tenth of a token per second at short and medium context, so the comparison below isn't pinned to a single convenient version. The numbers that follow use 0.30.7, the current release.
Protocol: I ran five runs per metric, reported the median result value, and used identical prompts and lengths for both Ollama and Squish. The benchmarks included prompt sizes of 75, 2000, and 4000 tokens. 75-token prompts are what most benchmarks publish. Coding agents and document Q&A workloads are typically around 4000 tokens. The raw per-run JSON results are committed in the repo, and every number can be reproduced with an M3.
The Honest Benchmark Comparison
With that protocol settled, here is how the two inference servers compare (apples-to-apples). Every number is reproducible from the repo via benchmarks/ollama_vs_squish/bench_thermal_h2h.py.
| Metric | Ollama 0.30.7 | Squish |
|---|---|---|
| Cold start: load + first token (1.5B) | 20–30 s | ≈0.5 s (54×) |
| Full response @ 4000-token prompt (repeated exactly)* | 37.5 s | 3.8 s (9.8×) |
| Decode throughput @ 75 tokens | 20.3 tok/s | 24.0 tok/s (INT3) |
| Decode throughput @ 2000 tokens | 19.7 tok/s | 22.6 tok/s (INT3) |
| Inter-token tail p95 @ 75 tokens | 52.4 ms | 42.7 ms (INT3) |
| Inter-token tail p95 @ 2000 tokens | 52.9 ms | 45.4 ms (INT3) |
| Repeat-prompt TTFT (KV cache hit) | ~160 ms | 4–11 ms |
| Peak RAM during inference | 5.14 GB | 3.50 GB |
| Disk: 7B INT4 / INT3 | 4.36 GB / n/a | 4.00 / 3.56 GB |
See "How Prompt Reuse Scales" below, this is the ceiling, not the typical case.
Ollama 0.18.2 was tested identically. Its decode throughput and latency matched 0.30.7 to within noise.
Squish leads on every metric in this table: decode throughput, tail latency, peak memory, and full end-to-end response time. An earlier version of this comparison also reported an Ollama win on short-prompt TTFT, 167 ms vs 192 ms, using the same fixed prompt sent five times. That number didn't hold up under scrutiny: it was measuring Ollama's own cache hit on the repeat sends, not a genuinely cold comparison. Under real cold, unique-prompt conditions, covered in the next section, Squish leads TTFT too, at every length tested, down to the shortest prompt measured.
Let's say you have an agent resending the system prompt every turn. For a 4,000-token prompt, Ollama takes around 37.5 seconds to return a full response. Squish's full response returns in 3.8 seconds flat. It's 9.8× faster, and it's the difference between a tool that responds in a timely manner and one where users stare at the screen wondering why it hasn't responded yet.
How Prompt Reuse Scales
The 9.8× number above assumes the prompt repeats exactly. What if the prompt is completely unique with nothing to reuse at all? I benchmarked Qwen2.5-7B-Instruct using Squish INT4 vs Ollama Q4_K_M. Tested five context lengths and verified 0% reuse for every single request.
| Context | Ollama TTFT | Squish TTFT | Ollama decode | Squish decode | Ollama E2E | Squish E2E | Speedup |
|---|---|---|---|---|---|---|---|
| 75 | 812 ms | 800 ms | 16.4 tok/s | 17.5 tok/s | 14.12 s | 12.30 s | 1.15× |
| 512 | 3.90 s | 3.33 s | 16.7 tok/s | 19.1 tok/s | 16.27 s | 13.74 s | 1.18× |
| 1024 | 10.01 s | 8.59 s | 10.8 tok/s | 14.8 tok/s | 28.93 s | 21.98 s | 1.32× |
| 2048 | 19.79 s | 17.31 s | 10.5 tok/s | 14.2 tok/s | 38.84 s | 32.24 s | 1.20× |
| 4096 | 41.50 s | 36.14 s | 9.4 tok/s | 11.9 tok/s | 62.80 s | 52.93 s | 1.19× |
Squish's 4096-token TTFT ranged from 25.4 to 43.6 seconds across its five measured runs, wider than the other rows, though each run independently confirmed 0% cache hit. This row should be treated as the least precise of the five. The 75-token result is the closest race of the set, Squish's TTFT edge there is just 12 ms, about 1.5%, still a real win but a thin one.
Even with 0% prompt reuse, Squish wins TTFT, decode, and end-to-end response time at every context size tested, down to the shortest prompt measured. The floor is 1.15 to 1.32× faster than Ollama.
The 9.8× ceiling resulted from the same 200-token, thermally-controlled benchmark as the floor above, so the numbers are directly comparable. The floor tests completely unique prompts; the ceiling resends an identical prompt every time: 1.15 to 1.32× on unique prompts, up to 9.8× on exact repeats. The benchmark below measures the prompt reuse effect in isolation and reaches 14.7× with 95% overlapping prompts.
I also wanted to see how speedup scales as prompt reuse increases from 5% to 95%. The second benchmark isolates and tests the reuse effect at four context lengths and five overlap percentage levels. Each result was identically validated byte-for-byte against a cold run to confirm reuse output is the exact same.
| Overlap | 512 tok | 1024 tok | 2048 tok | 4096 tok |
|---|---|---|---|---|
| 5% | 1.15× | 1.07× | 1.06× | 1.07× |
| 25% | 1.32× | 1.30× | 1.34× | 1.30× |
| 50% | 1.55× | 1.84× | 1.97× | 1.91× |
| 75% | 2.75× | 3.23× | 3.51× | 3.64× |
| 95% | 5.86× | 6.95× | 11.0× | 14.7× |
The results above reflect three runs, lighter than the 5-run protocol above. These five overlap levels are quartiles pulled from a finer sweep measured at every 5% interval between 5% and 95%. The full 19-point sweep isn't committed to the repo, reproduce it with python -m benchmarks.prefix_reuse_curve --contexts 512,1024,2048,4096.
This table's results are higher than the floor-to-ceiling range above. An 8-token generation is almost entirely prefill and the ratio isolates the prompt reuse effect at full strength. The 9.8× ceiling includes a full 200-token response including decode time. Reuse can't speed up decode, reducing the ratio. Both measure the same mechanism, one in isolation, and the other inside a real response. Speedup is modest at low overlap percentages and improves significantly at 50% and above. At 95% overlap, a 512-token prompt sees 5.9× while a 4096-token prompt sees 14.7×. At 75% and 95% overlap, the longer the context the faster the response, though the pattern is less clean right at 50%.
Agent loops and multi-turn conversations benefit heavily from increasing context with overlapping prompts. However, reuse only helps when there's previously processed content to reuse. The other lever is making the weights themselves smaller, and that's where things get interesting.
Ollama version 0.31.1 was released five days after the benchmark above was measured. A spot-check using 0.31.1 on the cold/unique floor reproduced the exact same 1.32× result at 1024 tokens. The reuse-ceiling spot-check initially failed its own drift check due to unrelated machine load; after clearing it, a clean re-run landed at 8.8×, within noise of the published 9.8×.
The Most Interesting Thing I Found
I wanted local models to run faster and use less memory. I started with INT4 quantization, which is well studied and a stable standard. But I wanted to know exactly how small the models would compress, so I tried quantizing using INT3. Most models broke. Qwen2.5-7B didn't. The INT3 quantized model remained stable, and tied INT4 on the arc_easy reasoning benchmark. The scores were 0.551 to 0.541, INT3 slightly ahead but within the margin of error. The INT3 model decodes roughly 18% faster and compresses the model from 4.00 GB to 3.56 GB on disk. The other Qwen models I tested remained stable using INT3, within roughly 1% of the original model's performance. I also quantized Gemma-3 using INT3, and it collapsed, a fifteen-point drop in accuracy.
The Qwen models I tested could handle INT3, in some cases marginally beating INT4 within the margin of error. With that finding in hand, I wanted to find the actual floor, so I moved on to INT2. Qwen broke, and for good reason. The model responded to my prompt with gibberish: IFYINGIFYIN, completely incoherent, basically random. When a 16-bit model is crushed down to 2 bits, its intelligence is lost. Aggressive quantization has a hard floor, and eventually every model breaks. That breaking point is different for every model and every model family. That's why Squish has a quantization safety mechanism that blocks INT3 for the families it fails on and INT2 altogether. The result is a tool that compresses every model as far as it can safely go, and no further.
How You Use It
There are many ways to use Squish, but at its most basic level it serves an OpenAI API on a local port (11435). I built it as a drop-in replacement for Ollama and for any application that speaks the OpenAI API endpoint spec.
Installing Squish via Homebrew is recommended. Nothing compiles, and every dependency comes bundled:
brew tap konjoai/squish
brew install squish
Or through Python:
pip install squish-ai # Python 3.11–3.14 required - OR
pipx install squish-ai --python python3.13 # Isolated in its own environment
With Squish installed, start a model with squish run. Specifying a model is optional. With no argument, Squish pulls a default sized to your machine's RAM, so you get the largest model that comfortably fits.
squish run # or specify a model, e.g. squish run qwen2.5:7b
Behind the scenes on that first run, Squish pulls a pre-squished model from the squishai Hugging Face community, the conversion and accuracy-gating already done, then loads and serves it. Once the model is loaded and ready, the web UI opens automatically.
A caveat before you go looking for models: Squish doesn't run every model, only those in MLX format (fp16 or bf16 weights). More on that in the next section.
With the server running, point any OpenAI client at it. Set your base URL to the local endpoint and use squish as the API key:
export OPENAI_BASE_URL=http://localhost:11435/v1
export OPENAI_API_KEY=squish
Ollama clients work too: point OLLAMA_HOST at the same port and they won't know the difference.
From there, there are several ways to work with the running model:
- The web UI, a chat client backed by a live instrument panel: the KV cache filling and reusing in real time, a quantization comparator, a latency waterfall splitting prefill from decode, and Apple Silicon power telemetry.
- The VS Code extension, the same chat plus agentic features and repo access, right in your editor.
- The macOS app, in two parts: a menu bar showing the Squish logo and live tokens per second, and a chat window for talking to the loaded model.
- The command line, POSTing directly to the OpenAI API endpoint.
Here are the basic Squish CLI commands:
squish run # start the server (auto-picks a model for your RAM)
squish pull # download and compress a model for your machine
squish doctor # check your environment is set up correctly
squish catalog # browse available models
To keep the server always available, squish daemon install registers it to start at login. For the full command list, see the repo.
What Squish Doesn't Do
Squish only runs on Apple Silicon (M-series). There is no support for CUDA, Intel, Windows, or Linux. These platforms, chips, and kernels are not on the roadmap. Squish's speed comes from the Mac's Metal and unified memory. If you're not on an M-series Mac, Squish will not work for you.
Squish doesn't run every model on Hugging Face. It works with models already in MLX format, the fp16 or bf16 weights the mlx-community organization publishes, which is the layout Apple's MLX framework expects. If a model is already in that format, you can point Squish's convert command at it, and it quantizes and packs the weights into Squish's own format. What you can't do is hand it an arbitrary checkpoint in some other layout and expect it to work.
What I Use It For Now
When I first started building Squish, my goal was to get my commit messages written quickly and without rate limits. I accomplished that goal and then some. My commit and pull request scripts still run against Squish. However, my agents now write many of those commits and PRs themselves. Now my local Squish workflow has grown to include:
- Private and local chat, every prompt and response stays on the machine.
- Local code review and codebase Q&A, the Squish agent reads my repo without a third party involved.
- A drop-in OpenAI endpoint for testing other applications against a real model, no API bill.
- Model benchmarking and inference metrics: TTFT, end-to-end latency, RAM/GPU utilization live in the web dashboard and VS Code extension.
Squish solves my problems, and it may or may not solve yours. But if you have more than 16 GB RAM, Squish can run larger models than I've been able to while using significantly less RAM and hard disk space. If you want to modify it, the source is available and every benchmark number is reproducible from the repo.
I built Squish because I was tired of waiting. It keeps the model loaded, the cache warm, and answers before I've moved my hands. What I haven't seen is what it can do beyond my one machine.
Join the Squish Community
I've run and tested Squish on one machine, an M3 MacBook Pro with 16 GB of RAM. Every number in this article reflects that same laptop. I have no idea what Squish looks like on a Mac Studio with 128 GB, or what happens when you point it at a 70B model instead of a 7B one, and I'd like to find out.
If you have access to a Mac with 32, 64, or 128+ GB of unified memory, running Squish against larger models and publishing what you find would answer questions I can't answer myself:
- How does the memory governor behave when there's real headroom to work with?
- Does the reuse curve hold the same shape for larger models?
- Where is the actual ceiling beyond the 16 GB RAM constraint?
A few other ways to help, beyond high RAM benchmarking:
- Try to break the numbers in this article. Every benchmark here is reproducible from the repo. If you can find a configuration where the claims don't hold, that's a more valuable contribution than a benchmark that confirms what's already published.
- Test model families I haven't. The INT3 accuracy gate currently covers a handful of Qwen models and Gemma-3. There are a lot of other model families out there I simply haven't had time to run through it.
- File real bugs from actual use. Edge cases from actual daily workflows surface things a benchmark suite never will.
- Contribute to the model catalog. If there's an MLX-format model on Hugging Face you think belongs in Squish's catalog, open an issue or a PR.
- Improve the docs. If something in this article or the repo's docs confused you, it'll confuse someone else too.
Open an issue or a PR on GitHub, whichever of these you want to take on. I built Squish alone, but I'd rather not be the only one who knows what it can do. Break something and have fun.
Requests for models to add to the squishai Hugging Face community, or trouble installing or running Squish? Open an issue on GitHub.
GitHub -
konjoai
/
squish
⚡️ The fastest way to run local LLMs on Apple Silicon — sub-second model loads, beats Ollama on throughput, tail latency, and full-response time. OpenAI/Ollama-compatible. No cloud, no API keys.
Squish
The fastest way to run local LLMs on Apple Silicon.
Sub-second model loads. Beats Ollama on throughput, tail latency, and full-response time. One OpenAI/Ollama-compatible daemon — no cloud, no API keys, fully offline.
███████╗██╗ ██╗██╗ ██╗ █████╗ █████╗ ██╗ ██╗ ██████╗ ██████╗ ██╗ ██╗
██╔════╝██║ ██║╚██╗██╔╝ ██╔══██╗ ██╔══██╗╚██╗██╔╝ ╚════██╗╚════██╗╚═╝██╔╝
███████╗███████║ ╚███╔╝ ╚██████║ ╚█████╔╝ ╚███╔╝ █████╔╝ █████╔╝ ██╔╝
╚════██║╚════██║ ██╔██╗ ╚═══██║ ██╔══██╗ ██╔██╗ ╚═══██╗██╔═══╝ ██╔╝
███████║ ██║██╔╝ ██╗ █████╔╝██╗╚█████╔╝██╔╝ ██╗ ██████╔╝███████╗██╔╝██╗
╚══════╝ ╚═╝╚═╝ ╚═╝ ╚════╝ ╚═╝ ╚════╝ ╚═╝ ╚═╝ ╚═════╝ ╚══════╝╚═╝ ╚═╝
faster cold start faster long-prompts less RAM
██████╗ ███████╗███████╗ ██████╗ ██╗ ██╗ ██╗███╗ ██╗████████╗██████╗
██╔═████╗ ██╔════╝██╔════╝ ╚════██╗██║ ██║ ██║████╗ ██║╚══██╔══╝╚════██╗
██║██╔██║ ███████╗███████╗ █████╔╝███████║ ██║██╔██╗ ██║ ██║ █████╔╝
████╔╝██║ ╚════██║╚════██║ ██╔═══╝ ╚════██║ ██║██║╚██╗██║ ██║ ╚═══██╗
╚██████╔╝██╗███████║███████║ ███████╗ ██║ ██║██║ ╚████║ ██║ ██████╔╝
╚═════╝ ╚═╝╚══════╝╚══════╝ ╚══════╝ ╚═╝ ╚═╝╚═╝ ╚═══╝ ╚═╝ ╚═════╝
cold load · 0.33–0.53s tok/s · beats Ollama quant default
██╗ ██╗███╗ ███╗███████╗ ██████╗ ███████╗ ██████╗ ██╗ ██████╗ ██████╗Hugging Face -
PyPI -
squish-ai · PyPI
Local LLM inference server for Apple Silicon. Block-level paged KV cache for long-context workloads. 5.4× faster end-to-end on 4K-token prompts vs Ollama, less RAM, INT3 support for Qwen3. OpenAI-compatible API.
pypi.org
Squish Website -
Squish: fast local LLMs on Apple Silicon - Squish
Fast local LLMs on Apple Silicon: sub-second model loads, faster than Ollama on long prompts. OpenAI and Ollama-compatible.
squish.run
Original Article -
I Couldn't Find a Local LLM Tool Fast Enough, So I Built My Own Called Squish - Squish
A local LLM inference server for Apple Silicon, 1.15 to 14.7× faster than Ollama depending on how much your prompts repeat, with the honest benchmarks.
squish.run
Top comments (3)
I found it really interesting that you were able to achieve a 1.15 to 14.7× speed increase with Squish compared to Ollama, especially since you were motivated by the need for faster commit message generation. I've been exploring local LLM tools myself and I'm curious, have you considered adding support for other models or architectures in Squish, or is the current MLX-based approach a deliberate design choice? Additionally, I'd love to know more about the quantization and compression techniques you used to reduce memory requirements for the models.
I'm open to adding more model architectures, but I'm limited by my Mac only hardware and 16GB RAM. That's why I'm asking for community help. MLX was deliberate, because of my hardware. I'm considering Linux and Windows, but Apple silicon is easiest for me to test currently. The article explains both the quantization and compression methods and techniques in detail. The most important being the block native fused multiple engine.
I just added the remaining animated diagrams to better visualize the Squish architecture.