If you have a MacBook, you already have a GPU with more memory than most graphics cards. This is because Apple silicon has unified RAM. This article will explain how to find, choose, and run models locally on Apple devices with MLX.

saba-ch — Wed, 15 Jul 2026 18:48:18 +0000

Jul 15

A guide on running models locally

#ai #agents #machinelearning #mlx

6 min read

A guide on running models locally

saba-ch — Wed, 15 Jul 2026 18:45:11 +0000

If you have a Macbook, you already have a GPU with more memory than most graphics cards. This is because Apple silicon has unified RAM that is shared across the GPU and CPU. The latest MacBook Pro comes with 24 GB of unified memory, that is effectively 24 GB of GPU memory. This is modest in the AI world, but it can definitely run a few quantized models locally. This article will explain how to find, choose, and run models locally on Apple devices with MLX.

What is MLX?

MLX is a machine learning framework designed for Apple silicon. It takes full advantage of Apple's unified memory and runs operations across GPU and CPU. It's similar to PyTorch. You can use MLX to train and run neural networks on MacOS. We are specifically interested in mlx-lm, a package built on top of MLX to run and fine-tune large language models. You can just run uv tool install mlx-lm to install it.

How do we choose the models?

When it comes to running models locally, we don't have many options because we are heavily constrained by our working memory, to be able to generate tokens, you should be able to hold the entire model in RAM. The latest MacBook Pro comes with 24GB memory, and macOS constrains us to only use 75% of that in the GPU that means we have an 18GB budget. In this article, we will be looking for models that can fit in our budget rather than do quick generation or solve the hardest problems.

If you head to Hugging Face, you will see a bunch of similar cryptic names of models. In those names, there is useful information about model size, which you will have to learn how to decipher. Let's try to decipher this one model: mlx-community/Qwen3.6-27B-bf16.

First, Qwen3.6, which is the name of the model, is one of the favorites of the local community. Then we get 27B, this means we have 27 billion parameters in this model. You can think of parameters as a bunch of numbers, such as vectors and matrices. Next up, have bf16 this tells us how many bits each parameter is. Models usually come out with 16 bits per parameter, which is then quantized into 8 and 4 bits.

Using deciphered params, we can use the formula to calculate how big the model actually is in bytes: parameters * bits_per_parameter / 8 = size in bytes.

27B * 16 / 8 = 54GB :(

Sadly there is no way we can fit this our budget (18GB). This is where quantization comes into play.

Quantization

You can think of quantization as scaling something down. Imagine a number between 1 and 1000. Let's say you chose 427, quantizing this number means storing it in a smaller space like 1 to 100. In this space, it translates to 43. When you need the original number back, you multiply it back by the scale, and you get 430.

During quantization, you lose accuracy, but you end up with smaller numbers, which means a smaller memory footprint.

Here is the quantized model mlx-community/Qwen3.6-27B-4bit.

In our equation, this translates to: 27B * 4 / 8 = 13.5 GB but we have to add 20-25% overhead to quantized models because there is overhead to scaling and smart quantization sometimes spares few layers for accuracy, we won't get into these details. After overhead, it would be around 16GB. This is borderline within our limit, but still within the budget. So we can run this on our machine, right?

Context window and KV Cache

I can see why you would think that, yes you can fit the model on the machine and greet it, but you won't be able to use it much because we haven't accounted for the context window yet.

LLMs predict the next token from all the tokens before it. To do this effectively, they store the computed vectors of each token in memory, called the KV cache. Without this, generating the next token would mean recomputing the whole sequence from scratch, which would be very compute-intensive. Luckily, each token's cache has a fixed size that scales linearly. To calculate how many bytes each token needs, you can use:

kv_per_token (bytes) = 2 × n_attn_layers × n_kv_heads × head_dim × cache_bytes

You can check out the config.json of the model to get these values. n_attn_layers=16, n_kv_heads=4, head_dim=256, cache_bytes=2. This means:

kv_per_token = 2 × 16 × 4 × 256 × 2 = 65,536 bytes = 64 KB per token

At a 32k context length, it would allocate 2GB of memory, which is already too much, and this model has a 262k context window; at full length, it would need 16GB of working memory! So we can throw this option out of the window.

Running the model

Let's check out the smaller model mlx-community/Qwen3.5-9B-4bit.

Let's plug this into our memory equation to calculate memory: 9B * 4 / 8 = 4.5GB if you add 20% overhead, you get 5.5 GB. So far so good. Now let's plug in context window numbers:

2 × 8 layers × 4 kv_heads × 256 head_dim × 2 bytes = 32,768 bytes = 32KB per token

This means at full 262k we would only use 8.4GB. Only is ironic here, full context seems to be larger than the model itself, but the good news is in total it needs 14GB memory, which is well below our limit, so we can safely use it.

Let's try it! To test it, you can run

mlx_lm.chat --model mlx-community/Qwen3.5-9B-4bit --max-tokens 8192

This will download and give you an ugly interactive chat interface to test it at first.

You can see you can message it and get the response back. Now you understand what I meant by an ugly interface. LLMs' outputs are not really formatted the way they're supposed to. You can play around with it and see how smart it is. Next up, we are gonna use it for coding!

Coding with the model

We were able to chat with the model from the built-in interactive interface. How can we use it in other harnesses? Most of the models today are exposed behind standard OpenAI-compatible API servers; it's the closest thing to standard we have today, and almost all harnesses support plugging in custom models that run behind one.

To run your model behind a server locally, run:

mlx_lm.server --model mlx-community/Qwen3.5-9B-4bit --max-tokens 8192

I will be plugging in my local model in PI Coding Agent. It's minimal and customizable by design, and that's just what we need now. if use pi as well and you want to add your own local provider, you can create the ~/.pi/agent/extensions/local-mlx.ts file and paste the following content:

import type { ExtensionAPI } from "@earendil-works/pi-coding-agent";

export default function (pi: ExtensionAPI) {
  pi.registerProvider("local-mlx", {
    name: "Local MLX",
    baseUrl: "http://127.0.0.1:8080/v1",
    api: "openai-completions",
    apiKey: "mlx-local",
    compat: {
      supportsDeveloperRole: false,
      supportsReasoningEffort: false,
      supportsUsageInStreaming: false,
      maxTokensField: "max_tokens"
    },
    models: [
      {
        id: "mlx-community/Qwen3.5-9B-4bit",
        name: "Qwen3.5 9B 4-bit (MLX local)",
        reasoning: false,
        input: ["text"],
        contextWindow: 131072,
        maxTokens: 8192,
        cost: { input: 0, output: 0, cacheRead: 0, cacheWrite: 0 }
      }
    ]
  });
}

After this run:

pi --model local-mlx/mlx-community/Qwen3.5-9B-4bit

Happy prompting! Here is how mine did:

Creating your own Agent Workspaces

If you liked self-hosting models, you are gonna love this one. I am working on a project called Wolli. It lets you create local Agent Workspaces that plug into your tools and help automate routine tasks. For example, you could integrate GitHub and make your own Code Reviewer.

You can give it a shot and let me know what you think!

DEV Community: saba-ch

If you have a MacBook, you already have a GPU with more memory than most graphics cards. This is because Apple silicon has unified RAM. This article will explain how to find, choose, and run models locally on Apple devices with MLX.

A guide on running models locally

A guide on running models locally

What is MLX?

How do we choose the models?

Quantization

Context window and KV Cache

Running the model

Coding with the model

Creating your own Agent Workspaces