Local LLMs for Agentic Coding: A Practical Guide for Modest Hardware

In just 20 minutes, you can have your local AI engine and agent ready to serve.

This guide focuses on two highly efficient engines: llama.cpp and MTPLX. I'm not affiliated with either project. I discovered these through extensive experimentation with Ollama and oMLX. While Ollama and oMLX are excellent tools, I found llama.cpp and MTPLX to be more stable for my specific workflow. Feel free to pick whichever engine you prefer. This guide remains applicable regardless of your choice.

Llama.cpp offers broad hardware flexibility across macOS, Linux, and Windows, while MTPLX is optimized solely for Apple Silicon.

Note on testing methodology: I benchmarked all models on a MacBook M4 Pro with 48 GB of RAM. GPU figures are based on community research and documentation, not personal benchmarks. Speed figures are from my M4 Pro. NVIDIA/AMD owners should treat the speed table as indicative and re-tune quant/context for their VRAM; for smaller GPUs (e.g., 8 GB), you will need more aggressive quantization.

The Engine: Inference

Choosing the right inference engine is critical for agentic workflows. Unlike standard chat interfaces where a single slow response is merely an inconvenience, an agent relies on a continuous loop of reasoning, planning, and execution. In this context, even slight fluctuations in stability or speed can disrupt the entire logic chain. This is why I settled on llama.cpp and MTPLX. They provide the consistent performance required to maintain "agentic momentum" during complex tasks.

The difference between llama.cpp and MTPLX.
Llama.cpp is a bit slower than MTPLX, but offers superior memory efficiency and cross-platform versatility. For example, it allows me to run a 5-bit quantized Qwen3.8–27B with a 70k context window.

In contrast, MTPLX is optimized specifically for Apple Silicon to deliver faster performance. But on MTPLX, the 4-bit quant of Qwen3.8–27B can consume the same amount of RAM as the 5-bit quant on llama.cpp.

Installation can be handled via the respective official sites for llama.cpp and MTPLX.

Models
When selecting a model, my primary strategy is to estimate its memory footprint. As a rule of thumb, the model size should be roughly half your available RAM. With 48 GB of RAM, I typically target ~20 GB models. While it is possible to push up to 30 GB models, you must limit the context window to under 32K tokens to maintain stability. This is critical for agentic workflows. The Pi coding agent can easily consume ~10K tokens on the first prompt due to skills and AGENTS.md processing. Context can increase by nearly twofold when using Claude Code.

If you are working with 16 GB or 24 GB of RAM, focus on 9B to 14B parameter models like Qwen3.5-9B and Qwen3-14B. During the development of DocScan (detailed in my recent article), I found that ~70K tokens per task was more than enough for this mid-range application.

For heavy-duty coding and complex logic, the Qwen and Gemma families are a top-tier choice.

Recommended Models

Use Case	Model class	Recommended Models	Best For...
Autocompletion	Light	Qwen2.5-Coder-7B-Instruct, Qwen3.5-4B, Gemma-4-E4B	Real-time text, low latency
General Assistance and Coding	Mid-Range	Qwen3.5-9B, Qwen3-14B, Gemma-4-12B	Chat, documentation, simple scripts
Coding (Complex)	Dense	Qwen3.8-27B, Gemma-4-31B	Architecture planning, deep debugging
Coding (Execution)	MoE	Qwen3.6-35B-A3B, Gemma-4-26B-A4B	Rapid implementation, boilerplate, refactoring

Speed

Model Type	Prefill (tps)	Decode (tps)	MTPLX vs llama.cpp Performance
Dense	~140	10–20	MTPLX is ~1.5x faster
MoE	~900	60–80	Nearly identical speed

Model Architectures

1. Dense Models: In these models, every parameter is activated for every token generated. This full computational load makes them highly precise and excellent for complex reasoning, brainstorming, or coding, though they are typically slower than MoE models of a similar size.
2. Mixture of Experts (MoE): They have a small subset of the total parameters active per token. This allows the model to possess a massive total parameter count while maintaining high inference speeds, making it an efficient executor.

Models come in different quants. The best analogy for a quant is a movie resolution: a 16-bit model is a 4K movie, 3-bit is 720p, 1-bit is 320p. You can still get the same emotional palette if you like cinema and have enough imagination.

llama.cpp Configuration

To run an optimized server, use the following command as a baseline:

  llama serve \
    --models-preset ~/{your path}/models.ini \
    --flash-attn on \
    --parallel 2 \
    -ngl all \
    --threads 10 --threads-batch 12 \
    --cache-type-k q5_1 --cache-type-v q5_1 \
    --load-mode mmap+mlock \
    --jinja \
    --props --metrics \
    --chat-template-kwargs '{"preserve_thinking": true}' \
    --sleep-idle-seconds 300 \
    --models-max 1

Key Parameters Explained

🚀 Performance & Speed

• flash-attn: Enables Flash Attention, which significantly speeds up processing and reduces memory usage during long context sessions.
• threads / threads-batch: Sets the number of CPU threads used for generation and prompt processing. (e.g., leave a few cores free to avoid system lag).
• parallel: The number of independent conversation slots you want to maintain simultaneously.

🧠 Memory Management

• -ngl (Number of GPU Layers): Controls how much of the model is offloaded to your GPU/Metal. Using all allows llama.cpp to maximize your hardware usage automatically.
• cache-type-k/v: Sets the precision for the Key-Value cache. Using quantized formats instead of the default f16 significantly reduces RAM consumption, allowing for much larger context windows.
• load-mode mmap+mlock: Uses memory-mapping and "locks" the model in physical RAM to prevent the OS from swapping it to disk, ensuring stable latency.

⚙️ Model Behavior

• models-preset: Loads a predefined configuration (temperature, top-p, etc.) for specific models via an .ini file.
• chat-template-kwargs: Allows you to pass special instructions to the model's parser. For example, {"preserve_thinking": true} is essential for seeing the "Chain of Thought" reasoning.

Models Preset (models.ini)

Download model GGUFs from Hugging Face, directly or via the HF CLI.

Once you have downloaded your models, you need to register them in a models.ini file. This lets you switch between them quickly. You'll find temp, top-p, top-k, and other parameters on an appropriate model page.
Here is an example of how to structure your models.ini:

version = 1

[*]
c = 76800
spec-type = ngram-mod

[Qwen3.6-35B-A3B-MTP]  
model = {your model path}/Qwen3.6-35B-A3B-UD-Q5_K_XL.gguf  
mmproj = {your mmproj path}/mmproj-F16.gguf  
spec-type = draft-mtp  
spec-draft-n-max = 2  
rea = on  
reasoning-format = deepseek  
temp = 1.0  
top-p = 0.95  
top-k = 20  
min-p = 0.0  
presence-penalty = 1.5  
repeat-penalty = 1.0 

[Qwen3.8-27B]  
model = {your model path}/Qwen3.8-27B-UD-Q5_K_XL.gguf  
mmproj = {your mmproj path}/mmproj-F16.gguf  
rea = on  
reasoning-format = deepseek  
spec-type = ngram-mod  
spec-draft-n-max = 5  
spec-draft-p-min = 0.6  
temp = 1.0  
top-p = 0.95  
top-k = 20  
min-p = 0.0  
presence-penalty = 0.0  
repeat-penalty = 1.0

One note: don't be confused with reasoning-format = deepseek for qwen model. In llama.cpp, the reasoning-format controls how the server parses and returns the model’s chain-of-thought output. The name “deepseek” comes from the fact that the message.reasoning_content API field was first introduced by DeepSeek’s hosted API.

Parameter Guide

🚀 Speed & Optimization (Speculative Decoding)

• spec-type: The strategy used to "guess" upcoming tokens to speed up generation. draft-mtp uses Multi-Token Prediction for speed gains. ngram-mod is a universal method that works on any model.
• spec-draft-n-max: How many tokens the engine attempts to predict ahead of time.

🧠 Reasoning & Logic (Chain of Thought)

• rea: Enables "Reasoning" mode, allowing the model to perform a visible Chain of Thought before answering.
• reasoning-format: Tells the server how to parse the reasoning output.

🎨 Creativity & Randomness (Sampling)

• temp (Temperature): Controls randomness. Lower values (e.g., 0.6) make the model focused and deterministic; higher values make it more creative but unpredictable.
• top-p / top-k: Limits the pool of words the model chooses from, preventing "nonsense" tokens from being selected.
• min-p: A modern alternative to Top-P that ignores highly unlikely tokens based on their probability threshold.
• presence-penalty: Encourages variety by penalizing words that have already appeared in the conversation. Use a higher value (e.g., 1.5) for creative writing and lower/zero for strict coding tasks.

For NVIDIA and AMD GPU Owners

Backend: Use CUDA for NVIDIA (fastest) or ROCm for AMD. Vulkan is the best cross-vendor option.
VRAM & Sizing: To avoid slow PCIe transfers, ensure the model + KV cache fits entirely in VRAM.

• 24 GB: Ideal for ~20 GB quants (e.g., 27B Q4/Q5) or moderate 35B MoE.
• 12–16 GB: Suitable for 9B–14B dense models.
• 8 GB: Limited to small models (~9B Q4) with short context.

Optimization: Use -ngl all (or 99) to offload all layers to the GPU. If the model spills into system RAM, speed will drop significantly. When in doubt, use a smaller quantization.

Speed vs. capacity: A high-end NVIDIA GPU such as the RTX 4090 is often faster for token generation when the model and its working memory fit in VRAM. Macs can accommodate much larger models in high-memory configurations because CPU and GPU share unified memory, but model capacity and generation speed are separate trade-offs.

MTPLX Configuration

MTPLX has a friendly GUI and step-by-step setup.

Two things I changed in the settings:

• Fan Mode: default (I didn't want it pinning fans at max during decode)
• Context window: 76,800 (check it after a new version update; it may be dropped to defaults)

Decreasing the context window in the server settings doesn’t decrease the model capacity. So it doesn’t mean the model will be more unreliable when it reaches more than a half context boundary.

MTPLX models

MTPLX includes an embedded model picker and supports importing models via a direct link. You can find models on Hugging Face that fit your configuration. Always check the file size by clicking “View on HF” to ensure it fits your hardware.
Rule: The model size should be roughly half your RAM.

The Orchestrator: Agent Harness

I use Pi, but you can choose any orchestrator: OpenCode, Vibe from Mistral, Claude Code, Codex, etc. The key step is configuring the backend endpoint.
For Pi configuration: Set the baseUrl in ~/.pi/agent/models.json.

~/.pi/agent/models.json example:

{  
   "providers": {  
      "llamacpp": {  
         "api": "openai-completions",  
         "apiKey": "llama",  
         "baseUrl": "http://127.0.0.1:8080/v1",  
         "compat": {  
            "supportsDeveloperRole": true,  
            "supportsReasoningEffort": true  
         },  
         "models": [  
            {  
               "contextWindow": 76800,  
               "id": "Qwen3.6-35B-A3B-MTP",  
               "input": [  
                  "text",  
                  "image"  
               ],  
               "name": "Qwen3.6 35B-A3B MTP Thinking",  
               "reasoning": true  
            },  
            {  
               "contextWindow": 76800,  
               "id": "Qwen3.8-27B",  
               "input": [  
                  "text",  
                  "image"  
               ],  
               "name": "Qwen3.8 27B",  
               "reasoning": true  
            }
         ]  
      },  
      "mtplx": {  
         "api": "openai-completions",  
         "apiKey": "mtplx",  
         "authHeader": true,  
         "baseUrl": "http://127.0.0.1:8000/v1",  
         "models": [  
            {  
               "contextWindow": 76800,  
               "id": "Youssofal--Qwen3.8-27B-MTPLX-Optimized-Speed",  
               "input": [  
                  "text",  
                  "image"  
               ],  
               "name": "Qwen3.8-27B-MTPLX",  
               "reasoning": true  
            },  
            {  
               "contextWindow": 76800,  
               "id": "Youssofal--Qwen3.6-35B-A3B-MTPLX-Optimized-Speed",  
               "input": [  
                  "text",  
                  "image"  
               ],  
               "name": "Qwen3.6-35B-A3B-MTPLX",  
               "reasoning": true  
            }  
         ]  
      }  
   }  
}

Verify it's alive. Open a terminal and hit the server on port 8080 (llama.cpp) or 8000 (MTPLX):

curl http://127.0.0.1:8080/v1/models

If you see your model IDs in the JSON response, the backend is serving. You can also open http://127.0.0.1:8080 in your browser. The built-in WebUI gives you a quick chat window to sanity-check a response.
The setup is complete, and your system is ready.

Case Study: Real-world Implementation

My workflow rules

1. Understand Latency Stages (Prefill vs. Decode): Be patient during the initial request. Every response has two phases: prefill and decode.

   • Prefill processes the entire prompt with AGENTS.md and other instructions in parallel; it is compute-bound (FLOPs) and typically saturates GPU utilization.
   • Decode generates tokens sequentially, making it memory-bandwidth bound. This is why llama.cpp and MTPLX show similar prefill speeds on Apple Silicon, but MTPLX pulls ahead during decoding due to optimized memory scheduling.

2. Respect the Memory Ceiling: Your context window must fit within your available RAM. For a 48 GB system, aim for roughly 75K tokens. For extremely long tasks, use an extension like plannotator or break the work into a structured plan.md file to be implemented across multiple sessions.

3. Monitor Resource Pressure: Keep a system monitor (Activity Monitor on macOS, htop on Linux, or Task Manager on Windows) open at all times. Watch for rising memory pressure, which is a leading indicator of performance degradation.

4. Minimize Prompt Complexity: Keep your "skills" targeted. Instead of one massive skill set, use specific ones (e.g., Code Search, Software Engineering). For domain-specific logic (Jetpack Compose, SwiftUI, Flutter, RN), move those instructions to the project level to keep the agent's context clean and modular.

5. Keep AGENTS.md Granular: Avoid bloated instruction files. Keep individual AGENTS.md files under ~100 lines. In Feature-First Modular Architectures, utilize separate AGENTS.md files within specific feature or layer directories to maintain precision.

6. Use RAG to Mitigate Hallucinations: Retrieval-Augmented Generation (RAG) is essential for providing the model with external data, but it is not a silver bullet. LLMs can still "hallucinate" plausible-looking methods, mismatched versions, or fused class names.

7. Adopt a Multi-Stage Workflow: For complex tasks, use a specialized three-stage approach: Planning (using a high-precision dense model), Coding (using a high-speed MoE model), and Review (returning to a dense model for verification).

Implementation

I used local models to build the document detection project from my recent article. Now, I’ll add a new feature: exporting scan results to a file.

Goal: Add a file export feature.
Scope: A basic prompt to test what the current setup can handle.

make a plan
Feature: save scan result to a PDF file
ScanResult screen should show a "save to file" button when it has a scan result
Review whether it's possible to save a file without runtime permissions or any other unnecessary user interaction. Check if there's a convenient API.
Use web search for this.
put a plan to save-scan-plan.md

The Outcome: The feature works, but the architecture is slightly off. The PdfExporter.kt ended up in the presenter layer.

Next prompt:

1. New: presenter/filescan/PdfExporter.kt (or data/utils per layering)
it should be data/utils
make an updated plan in save-scan-plan-mod-1.md

The plan file has been regenerated to match the new scope, but this is mostly for educational purposes—you don’t need to commit a new plan version every time.

Why? I didn’t emphasize the clean-arch layers in the previous prompt. If I had specified something like "use the DocumentDetectorRepository data flow," the placement would have been correct from the start.

Last prompt:

make a note that there should be no direct dependency between data and presenter layers
it should work through domain
make an updated plan in save-scan-plan-mod-2.md

This task took about an hour: 30 minutes to implement and another 30 minutes to polish, double-check the agent, and test. While MTPLX would be significantly faster, the result at 10-15 tps (planning) and ~60 tps (implementation) is excellent. You’ve just gotten a local-storage feature with additions across all architecture layers and fallback compatibility for an hour of work.

Implementation review

What went well:

• End-to-end implementation: The feature was built from scratch with no missing pieces, warnings, or critical errors.
• Robust state handling: Includes a proper fallback logic branch for edge cases.
• Testing: Covered with simple JUnit tests for PDF filename generation.
• Documentation: Updated AGENTS.md files across all affected layers.

Areas for improvement:

• Memory management: The Bitmap.toPdf extension has a potential memory leak.

...
val pdf = PdfDocument()
val page = pdf.startPage(PdfDocument.PageInfo.Builder(width, height, 1).create())
...
val bytes = ByteArrayOutputStream()
pdf.writeTo(bytes)
pdf.close()

The next prompt found all issues:

review Bitmap.toPdf unhandled errors

The fix took only 1 minute and 3 seconds.

Just one button at first glance. But under the hood, it is a complete feature: PDF export with its full architectural layers (IScanPdfRepository, ScanPdfRepositoryImpl, and PdfExporter), robust file saving with API fallback logic, comprehensive state handling, runtime permissions, Koin bindings, documentation updates, and unit tests. A task that would traditionally take 3-4 hours was completed in a fraction of the time.

Conclusion

At its core, running local LLMs means shifting the intelligence from a distant cloud server directly onto your own hardware. While hosted services are powerful, local solutions offer three critical advantages that cloud providers cannot match: Control, Privacy, and Flexibility.

• Total Configuration Control: You are no longer limited by a provider's presets. You choose the model, fine-tune the settings, and connect it to specific local tools.
• Data Sovereignty: Your prompts, sensitive files, and project code never leave your device.
• Economic & Operational Independence: By eliminating per-prompt cloud fees, you can run heavy workloads indefinitely without cost concerns. You are no longer at the mercy of service outages, changing usage limits, or providers discontinuing your favorite model.

The Potential is Limitless
Once you have established this local intelligence layer, the applications are vast. It can power seamless IDE integration, automate document summarization, or serve as the core engine for your Personal Knowledge Management system. From assisting with emails and browser tasks - such as tracking shopping items, analyzing real-estate listings, or curating daily news digests. The only limit is how you choose to implement it.

Comments

[Sara]

The half-of-RAM rule of thumb matches what I've seen too. One thing that bit me was KV cache growth: a model that loads fine can still start swapping once the agent's context fills up. Have you tried quantizing the KV cache to q8 to stretch the context window?