DEV Community: Piotr Stachowski

Running Local LLMs on Apple Silicon: MLX Framework, mlx-community, and the Ultimate Mac Model Selection Guide

Piotr Stachowski — Wed, 22 Jul 2026 20:47:22 +0000

Introduction: Why MLX is a Game Changer for Mac AI

Running Large Language Models (LLMs) locally on consumer hardware used to require massive multi-GPU server rigs or complex CUDA setups. However, Apple Silicon (M1 - M5) changed the game with Unified Memory Architecture (UMA).

Unlike traditional PC architectures where the CPU and discrete GPU maintain separate memory pools and bottleneck over PCIe buses, Apple Silicon grants the CPU, GPU, and Neural Engine direct zero copy access to the exact same high-bandwidth RAM (up to 800 GB/s on M-series Ultra chips).

To capitalize on UMA, Apple released MLX open-source array framework designed specifically for machine learning on Apple Silicon. Modeled with PyTorch-like APIs and leveraging Metal performance shaders, MLX enables blazing-fast local LLM inference and fine-tuning directly on your Mac with zero compilation friction.

1. What is mlx-community on Hugging Face?

While the core mlx package provides the mathematical runtime engine, downloading raw weights and manually quantizing models can be cumbersome.

Enter mlx-community on Hugging Face:
👉 Official Hugging Face Hub: https://huggingface.co/mlx-community

The mlx-community repository is a central hub maintained by open-source contributors and Apple engineers. It hosts thousands of open-source models pre-converted into the native MLX format with optimized 4-bit, 8-bit, and FP16 quantization scheme choices.

Popular models ready to run instantly include:

Meta Llama 3.1 & 3.3 (8B, 70B)
Qwen 2.5 & Qwen 2.5 Coder (1.5B, 7B, 14B, 32B, 72B)
DeepSeek-R1 Distillations (Qwen-1.5B, Qwen-14B, Llama-70B)
Google Gemma 2 (2B, 9B, 27B)
Mistral & Mixtral MoE (7B, 8x7B, 8x22B)
Microsoft Phi-3.5 & Phi-4

2. How to Use MLX on Mac: Step-by-Step Quickstart

Getting started with MLX takes less than two minutes in your Mac terminal.

Step 1: Install the MLX LM Package

Ensure you are running newest macOS on an Apple Silicon Mac with Python 3.10 or newer:

# Install the official MLX LLM helper tool
pip install mlx-lm

# Verify installation and test a prompt directly from CLI
mlx_lm.generate \
    --model mlx-community/Qwen2.5-7B-Instruct-4bit \
    --prompt "Explain quantum superposition in 3 simple sentences."

3. Using MLX in Python Scripts

For building local apps, CLI tools, or background agent workflows, mlx_lm provides an intuitive Python API with lazy weight loading and streaming support:

from mlx_lm import load, generate

# Load model and tokenizer directly from the mlx-community hub
model, tokenizer = load("mlx-community/Meta-Llama-3.1-8B-Instruct-4bit")

# Generate response with streaming output
prompt = "Write a clean Python function for calculating Fibonacci using dynamic programming."
response = generate(
    model, 
    tokenizer, 
    prompt=prompt, 
    max_tokens=500,
    temp=0.2,
    verbose=True
)

print(response)

4. How to Choose the Right Model for Your Mac (RAM Sizing Guide)

Choosing the right model for your Mac requires understanding your Mac's Unified Memory (RAM) budget and quantization levels.

Memory Overhead & The 75% Rule

macOS reserves ~4 GB to 6 GB of RAM for the display server, window management, and background applications. Furthermore, macOS enforces a default process limit restricting any single application to ~75% of total RAM (though this can be raised via sysctl iogpu.wired_mem_limit).

Memory Calculation Rule of Thumb

4-bit Quantization (INT4): Requires ~0.7 GB RAM per Billion parameters. (Best balance of speed, low memory footprint, and ~95% intelligence retention).
8-bit Quantization (INT8): Requires ~1.2 GB RAM per Billion parameters.
Full Precision (FP16): Requires ~2.1 GB RAM per Billion parameters.

Mac RAM Sizing Matrix Table

16 GB Unified Memory (Base M1/M2/M3/M4, MacBook Air, Mac mini)
- Recommended Models: 7B to 9B parameters in 4-bit quantization.
- Top Picks: mlx-community/Meta-Llama-3.1-8B-Instruct-4bit, mlx-community/Qwen2.5-7B-Instruct-4bit, mlx-community/gemma-2-9b-it-4bit.
- Expected Speed: 35–60 tokens/second.
- RAM Footprint: ~5.5 GB (leaves 10 GB free for Chrome and IDEs).
24 GB / 36 GB Unified Memory (M2/M3/M4 Pro & Max)
- Recommended Models: 14B 4-bit/8-bit or 32B 4-bit models.
- Top Picks: mlx-community/Qwen2.5-14B-Instruct-4bit, mlx-community/DeepSeek-R1-Distill-Qwen-14B-4bit, mlx-community/Qwen2.5-32B-Instruct-4bit.
- Expected Speed: 25–45 tokens/second.
- RAM Footprint: ~10 GB to 20 GB.
64 GB / 96 GB Unified Memory (M1/M2/M3/M4 Max & Ultra)
- Recommended Models: 32B 8-bit or 70B 4-bit models.
- Top Picks: mlx-community/Meta-Llama-3.1-70B-Instruct-4bit, mlx-community/DeepSeek-R1-Distill-Llama-70B-4bit, mlx-community/Qwen2.5-Coder-32B-Instruct-8bit.
- Expected Speed: 18–30 tokens/second.
- RAM Footprint: ~38 GB to 48 GB.
128 GB to 192 GB Unified Memory (Mac Studio & Mac Pro Ultra)
- Recommended Models: 70B 8-bit, 120B+ models, or multi-model agent ensembles running concurrently.
- Top Picks: mlx-community/Meta-Llama-3.3-70B-Instruct-8bit, mlx-community/Mixtral-8x22B-Instruct-v0.1-4bit.
- Expected Speed: 15–25 tokens/second.

+--------------------+-------------------------+----------------------------------------------+--------------------+
| Mac RAM Capacity   | Max Model Parameter Size| Recommended mlx-community Model Path         | Generation Speed   |
+--------------------+-------------------------+----------------------------------------------+--------------------+
| 16 GB UMA          | 7B - 9B (4-bit)         | mlx-community/Meta-Llama-3.1-8B-Instruct-4bit | 35 - 60 tok/sec    |
| 24 GB / 36 GB UMA  | 14B - 32B (4-bit)       | mlx-community/Qwen2.5-32B-Instruct-4bit      | 25 - 45 tok/sec    |
| 64 GB / 96 GB UMA  | 70B (4-bit) / 32B (8-bit)| mlx-community/DeepSeek-R1-Distill-Llama-70B | 18 - 30 tok/sec    |
| 128 GB+ UMA        | 70B (8-bit) / 120B+     | mlx-community/Meta-Llama-3.3-70B-8bit        | 15 - 25 tok/sec    |
+--------------------+-------------------------+----------------------------------------------+--------------------+

5. Quantization & Bandwidth Bottlenecks

Why is 4-bit quantization so effective on Apple Silicon?

In Large Language Model inference, generation speed is almost entirely memory-bandwidth bound, rather than compute bound. Every token generated requires streaming the entire model weight tensor from Unified RAM into the GPU cores.

By compressing model weights from 16-bit float to 4-bit integer:

The model memory size drops by 75%.
The amount of data that must travel over the memory bus per token is reduced 4x.
On a MacBook Pro with 300 GB/s bandwidth, an 8B 4-bit model (~4.5 GB) achieves up to 60+ tokens/second!

# Fine-tune an MLX model on custom JSONL dataset using LoRA locally
mlx_lm.lora \
    --model mlx-community/Meta-Llama-3.1-8B-Instruct-4bit \
    --data ./my_dataset \
    --train \
    --iters 600 \
    --batch-size 4

Conclusion & Resources

Apple Silicon combined with MLX and mlx-community provides the most frictionless, high-performance local AI ecosystem available today. By matching model parameter counts to your Mac's unified memory tier and choosing 4-bit quantization, you can run private, latency-free LLM intelligence right on your workstation.

Explore the models and join the local AI revolution:

📦 Hugging Face Hub: https://huggingface.co/mlx-community
🐙 MLX GitHub Repository: https://github.com/ml-explore/mlx
⚡ MLX LM Documentation: https://github.com/ml-explore/mlx-examples/tree/main/llms/mlx_lm

AI Commit: Crafting an Automated Git Commit Message Generator for Neovim

Piotr Stachowski — Wed, 15 Jul 2026 18:06:15 +0000

A deep dive into how I built the ai_commit plugin for Neovim (nvim) using asynchronous system subprocesses, specialized prompt engineering, and external LLM APIs with zero heavy language wrappers.

Introduction & Repository

Writing meaningful, informative, and standardized git commit messages is one of those daily disciplines that keeps codebases healthy—but it's also a point of constant friction. Developers often write rushed commits like fix issue, wip, or refactor some stuff to move on quickly, which turns the git history into an uninformative timeline.

While there are many CLI utilities and heavy node-based wrappers that solve this problem externally, doing so requires switching context away from your editor. As an avid Neovim user, I wanted a solution integrated directly into my editing workspace.

The plugin is fully open-source and easy to integrate:
👉 Official GitHub Repository: https://github.com/klusekP/ai_commit

The goal was clear: select or stage files, type a simple Neovim command (like :AICommit), and let Neovim automatically request an intelligent, conventional-commit-formatted message based on the actual code diff and inject it right into Neovim's active commit buffer.

1. Core Plugin Architecture

To build a native Neovim plugin, we need to balance performance with user experience. A naive synchronous HTTP request would freeze Neovim completely during the network round-trip, resulting in a laggy editor.

Therefore, ai_commit uses a fully asynchronous subprocess model. Here is the operational pipeline:

Change Detection: The plugin verifies if there are staged changes by spawning git diff --staged --name-only as an asynchronous background job.
Diff Ingestion: If changes are staged, it executes git diff --staged to harvest the precise code additions and deletions.
Payload Construction: It bundles the raw diff with a highly tailored system prompt, ensuring the LLM formats the response strictly following the Conventional Commits specification.
Asynchronous Dispatch: Rather than relying on a heavy Python, Node, or Rust binary, Neovim spawns a non-blocking curl process in the background.
Buffer Injection: Once curl exits with a successful response, Neovim processes the JSON, extracts the commit message, and populates either the active commit buffer or opens a floating menu for user review.

2. Technical Prerequisites & Dependencies

One of the core design goals of ai_commit was minimal bloat. To install and run the plugin, only a handful of ubiquitous system dependencies are required:

Neovim (v0.8.0 or newer): Required for modern Lua APIs, job control spawning, and floating buffer manipulation.
Git CLI: The system must have git installed in the system PATH, as the plugin runs subprocesses to retrieve diff metadata.
curl (System Utility): The plugin issues direct network requests using the standard curl command-line tool. This avoids needing external Lua HTTP socket libraries.
Plenary.nvim (Optional but Standard): Used for elegant asynchronous job coordination and cleaner network requests. Most Neovim users already have Plenary as a dependency for Telescope.
API Access Token: A valid API key for your chosen provider (such as OpenAI, Anthropic, Gemini, or a locally hosted Ollama instance running llama3).

3. Complete Installation & Configuration Guide

Integrating ai_commit into your Neovim configuration is simple and supports standard modern package managers.

Installation via lazy.nvim

Add this plugin declaration block directly to your plugins directory:

-- Neovim Plugin Integration using lazy.nvim
return {
  "klusekP/ai_commit",
  dependencies = { "nvim-lua/plenary.nvim" }, -- optional helper library
  config = function()
    require("ai_commit").setup({
      provider = "openai",          -- Options: "openai" | "gemini" | "ollama"
      model = "gpt-4o-mini",        -- Model alias to invoke
      prompt_language = "en",       -- Output language: "en" (English) or "pl" (Polish)
      temperature = 0.2,            -- Generative creativity scale
      key_maps = {
        generate = "<leader>ca",    -- Custom keymap to trigger generation
      }
    })
  end
}

4. Provider Setup & Custom Prompts

The plugin is highly versatile and lets you leverage cloud APIs or locally run LLMs with ease.

Authentication: The plugin reads your API tokens directly from your environment variables, keeping your secrets out of your public dotfile repository (e.g., loads $OPENAI_API_KEY or $GEMINI_API_KEY).
Ollama (Local Offline Mode): Set provider = "ollama" and configure your local address (usually http://localhost:11434) to run completely offline without an internet connection.
Polish Language Output: Setting prompt_language = "pl" forces the system prompt to instruct the model to write clean, polish conventional commits, which is rare in other standard tools.

5. Command API & Keybindings

Once configured, the plugin exposes clean, native command-line bindings:

:AICommit - Triggers the asynchronous diff capture, calls the LLM, and populates the current buffer with the conventional commit message.
Keymaps configured under key_maps.generate (such as <leader>ca) let you execute this workflow instantly with simple keystrokes.

This ensures a rapid cycle where you stage changes with your preferred git UI (like fugitive or neogit), open your commit message editor, hit your keymap, and immediately commit your changes.

Conclusion

By bypassing heavy external runtimes and leveraging Neovim's built-in asynchronous job control coupled with direct API endpoints, ai_commit provides a lightning-fast, zero-bloat workflow that keeps your git timeline pristine and readable.

Check out the repository, install it today, and start generating professional, structured commit messages in your preferred language!

🔗 Star the project on GitHub: https://github.com/klusekP/ai_commit

Expiring Trip Links: Designing a Secure, High-Performance Location Sharing System with Python and Redis

Piotr Stachowski — Wed, 15 Jul 2026 17:59:05 +0000

How to build an elegant web-tracking architecture for unauthenticated guests that self-destructs after arrival, ensuring strict privacy and database efficiency.

Introduction

Real-time location tracking has become a standard expectation in modern logistics, ride-hailing, and delivery services. When a user is on a trip, they often want to share their progress with a friend or family member.

However, sharing live telemetry raises critical architectural and privacy challenges:

Unauthenticated Access: The recipient (guest) should be able to view the trip instantly on a web browser without downloading an app, creating an account, or authenticating.
Data Privacy: The tracking link must only be valid during the active trip. Once the vehicle arrives or the ride is cancelled, the link must immediately expire, preventing any further location leakage.
Database Performance: GPS coordinates are high-frequency data streams (often pinging every 1-3 seconds). Querying or writing this directly to a primary relational database creates unnecessary write amplification and can easily degrade system performance.

In this article, we'll design an elegant, secure, and production-ready system architecture using Python and Redis to generate self-expiring cryptographic hash links that allow guests to monitor trips with zero friction.

1. Core System Architecture Flow

To build this system efficiently, we separate the internal authenticated state from the public guest state.

We introduce a fast, in-memory caching layer (Redis) that sits between our web servers and the main database.

Here is how the lifecycle of a shared trip link works:

Trip Initialization: When a trip is activated, the core mobile app initiates location sharing. The server takes the trip_id, timestamp, and estimated duration, and generates a secure, unique tracking Hash.
Caching in Redis: The server maps this Hash to the active Trip ID in Redis, setting a Time-To-Live (TTL) equal to the trip's estimated duration plus a small buffer (e.g., 15 minutes).
Public URL Dispatch: The server returns a lightweight web link: https://track.services.com/share/{hash}.
Real-Time Coordinates: As the vehicle moves, its telematics system pushes GPS updates. These coordinates are written directly to Redis (as a fast key-value or Geo-spatial data structures), keeping our main database clean of transient telemetry slop.
Guest Web Portal: The guest opens the link in their browser. The web frontend queries the backend with the hash. The backend does a single O(1) lookup in Redis to verify the hash is active, retrieves the latest coordinate, and sends it to the guest.
Automatic Eviction: Once the trip concludes or the Redis TTL expires, Redis automatically evicts the hash. Any subsequent request to the guest link instantly returns a 404 (Not Found) or 410 (Expired), securing the driver's privacy.

2. Generating the Expiring Secure Hash

We generate the public hash in a way that prevents URL guessing (enumeration attacks). Instead of using auto-incrementing integers or predictable IDs, we combine a cryptographically secure random value, the trip's metadata, and a timestamp, then hash them using SHA-256.

By including the timestamp and a random salt, we ensure that even if the same user takes the same trip multiple times, the generated shareable links are completely unique and non-linkable.

import hashlib
import time
import secrets

def generate_trip_share_hash(trip_id: str, user_id: str) -> str:
    """
    Generates a cryptographically secure, non-guessable hash for guest sharing.
    Incorporates current epoch timestamp and secure high-entropy random bytes.
    """
    timestamp = int(time.time() * 1000)
    # Generate 16 bytes of cryptographically secure random salt
    salt = secrets.token_hex(16)

    # Create unique string representation
    payload = f"{trip_id}-{user_id}-{timestamp}-{salt}"

    # Generate SHA-256 hash
    return hashlib.sha256(payload.encode('utf-8')).hexdigest()

3. Fast Storage & TTL Management in Redis

Once the hash is generated, we store it in Redis. We map the hash to the underlying trip_id and set a time-to-live (TTL). This ensures that we don't have to clean up links manually; Redis handles memory reclamation natively.

We use two Redis keys for this design:

share:hash:{hash} -> A string containing the trip_id (with a TTL of duration + buffer).
trip:coords:{trip_id} -> A Redis hash or a simple string holding the latest GPS coordinates (lat, lng, heading, speed).

When a guest requests tracking updates via the hash, the server first resolves the hash to a trip_id from Redis, and then fetches the coordinates.

import redis

# Establish connection to Redis
r = redis.Redis(host='localhost', port=6379, db=0, decode_responses=True)

def register_share_link(share_hash: str, trip_id: str, duration_seconds: int) -> None:
    """
    Registers a shareable hash in Redis mapped to its underlying Trip ID.
    Enforces a secure TTL (time-to-live) so that links self-destruct automatically.
    """
    safety_buffer_seconds = 15 * 60  # 15-minute buffer
    total_ttl = duration_seconds + safety_buffer_seconds

    # Store the mapping hash -> trip_id with TTL in a single atomic transaction
    r.set(
        name=f"share:hash:{share_hash}",
        value=trip_id,
        ex=total_ttl
    )
    print(f"[Redis] Registered share:hash:{share_hash[:8]}... mapped to {trip_id} (TTL: {total_ttl}s)")

4. Designing the API Routes (FastAPI/Python)

On our API gateway, we expose two primary endpoints:

POST /api/trips/share: An authenticated endpoint used by the mobile client to initiate sharing, generate a hash, and register it in Redis.
GET /api/share/{hash}: A public, unauthenticated endpoint accessed by the guest's browser to fetch the active trip's real-time location.

By decoupling the guest endpoint, we avoid hitting the main relational database completely on telemetry reads.

import json
from fastapi import FastAPI, HTTPException, status
from pydantic import BaseModel

app = FastAPI()

class ShareRequest(BaseModel):
    trip_id: str
    user_id: str
    duration_minutes: int

@app.post("/api/trips/share")
def initiate_share(request: ShareRequest):
    """
    Mobile client triggers location sharing. Generates the hash and caches in Redis.
    """
    try:
        share_hash = generate_trip_share_hash(request.trip_id, request.user_id)
        duration_seconds = request.duration_minutes * 60

        # Save mapping to Redis
        register_share_link(share_hash, request.trip_id, duration_seconds)

        return {
            "success": True,
            "hash": share_hash,
            "tracking_url": f"https://track.services.com/share/{share_hash}",
            "expires_in_seconds": duration_seconds + (15 * 60)
        }
    except Exception as e:
        raise HTTPException(status_code=500, detail="Failed to initialize sharing link")

@app.get("/api/share/{share_hash}")
def get_guest_tracking(share_hash: str):
    """
    Public web portal fetches active trip location (Unauthenticated).
    Returns 410 Gone if the Redis TTL expired or link doesn't exist.
    """
    # 1. Resolve hash to trip_id in Redis (O(1) search)
    trip_id = r.get(f"share:hash:{share_hash}")

    if not trip_id:
        raise HTTPException(
            status_code=status.HTTP_410_GONE,
            detail="This tracking link has expired or is invalid."
        )

    # 2. Fetch coordinate stream or last coordinates
    coords_data = r.get(f"trip:coords:{trip_id}")
    coordinates = json.loads(coords_data) if coords_data else None

    return {
        "active": True,
        "trip_id": trip_id,
        "coordinates": coordinates  # e.g. {"lat": 37.77, "lng": -122.41}
    }

5. Production Security & Scaling Considerations

To run this solution robustly at a large scale, keep these security and optimization practices in mind:

Rate Limiting on Guest Endpoint: Since the guest endpoint GET /api/share/{hash} is unauthenticated, it is open to public scrapers. Implement strong IP-based rate limiting (e.g., using Redis-based token bucket algorithms) to prevent denial of service.
Obfuscate Internal IDs: Even if the hash is secure, avoid returning internal postgres UUIDs or raw driver names to the client unless necessary. Keep the payload returned to the guest strictly stripped of private user and driver fields.
Geo-fencing Expire Trigger: Rather than relying purely on time-based TTL, the backend server can actively listen to a 'Trip Arrived' event from the matching service. Upon receiving this event, the server can fire a r.delete(f"share:hash:{share_hash}") command to immediately disable the link the exact second the customer arrives at their destination.
Redis Replica Scaling: At high scales (millions of concurrent guest sessions), read traffic will dominate. You can scale your system horizontally by routing guest API read requests to read-only Redis replica instances, reserving the Redis master strictly for coordinates writes.

Conclusion

By leveraging Redis TTL and SHA-256 cryptographic hashes, we've built a high-performance location-sharing architecture that is secure by design.

This approach minimizes read operations on your primary relational databases, maintains absolute privacy for the user after their ride completes, and scales horizontally with ease.

This architecture forms the backbone of modern on-demand tracking features, balancing user experience, security, and developer sanity perfectly.