How I Get Frontier-Quality Output from Local Models That Are 10x Smaller

I run Qwen 3.5 9B on my Mac Mini and Qwen 3 Coder 30B on an RTX 3060. That's the hardware reality — I'm not fitting a 200B+ parameter model locally. Not even close.

But here's the thing: most of my daily output from these models is indistinguishable from what I used to get from Claude or GPT-4o. Not because the models are that good. Because I learned how to talk to them.

This isn't another "here are 10 generic prompt tips" post. This is the specific, weird stuff I do to squeeze frontier-quality output from models that are 10-20x smaller than the cloud alternatives. Stuff that actually changed my day-to-day results.

Why Prompt Engineering Matters More for Small Models

Big models are forgiving. You can throw a vague prompt at GPT-4o and still get something usable. The model has enough parameters to infer what you probably meant.

Small models don't have that luxury. A 9B model takes your prompt literally. If you're vague, the output is vague. If you're specific, the output is specific. The quality gap between a badly-prompted small model and a well-prompted small model is massive — far bigger than the gap between a well-prompted small model and a well-prompted big model.

In other words: the same 9B model can produce garbage or gold depending entirely on how you ask.

Trick 1: The "Role + Format + Constraint" System Prompt

Most system prompts I see are one-liners: "You are a helpful assistant." That's wasted space on a small model.

My system prompts always have three sections:

You are a senior Python developer who writes clean, minimal code.

FORMAT: Respond with code only. No explanations unless asked. Use type hints.

CONSTRAINTS:
- Never use global variables
- Always include error handling for network calls
- Prefer standard library over third-party packages
- Maximum 50 lines per function

The role tells the model who to be. The format tells it what shape the answer should take. The constraints tell it what not to do.

Small models are terrible at deciding what to leave out. The constraints section forces the model to make fewer choices, which means fewer wrong choices.

Before this pattern: My 30B coder model would return 200-line functions with inline comments explaining every line. Useful, but I spent more time deleting than coding.

After: 50-line functions, type hints, no fluff. I went from 40% of the output being noise to maybe 5%.

Trick 2: One-Shot Examples Over Zero-Shot

Small models are pattern matchers. Give them a pattern and they follow it. Don't give them a pattern and they guess — badly.

Instead of:

Summarize this article in 3 bullet points.

I do:

Summarize articles as 3 bullet points.

ARTICLE: "Apple announced..."
SUMMARY:
- Apple released new MacBook Pro with M4 chip
- Starting at $1,599, shipping next week
- Performance claims 2x faster than previous generation

ARTICLE: "The Federal Reserve..."
SUMMARY:

That example does more work than any instruction. The model sees the exact format, tone, and level of detail I want. It mirrors that pattern for the next article.

This is the single highest-impact change I made. One-shot examples turned my 9B model from "sometimes okay" to "reliably good" for structured tasks like summarization, extraction, and formatting.

Trick 3: Decompose, Don't Compose

I used to ask small models for big things:

Build me a REST API with authentication, CRUD operations, rate limiting, and tests.

And I'd get 500 lines of interconnected code that looked reasonable but had subtle bugs in the auth middleware, the rate limiter didn't actually work, and the tests were testing the wrong things.

Now I break it into steps, running each one separately:

Step 1: "Create a FastAPI app skeleton with health check endpoint"
→ I review, test, fix

Step 2: "Add JWT authentication to this app. Here's the current code: [paste]"
→ I review, test, fix

Step 3: "Add CRUD endpoints for users with SQLAlchemy. Current app: [paste]"
→ I review, test, fix

Each step produces 20-40 lines. Small models are excellent at 20-40 lines. They're mediocre at 200 lines. They're terrible at 500 lines.

This is more work for me — I'm running 4 prompts instead of 1. But the output quality is dramatically better, and I spend less time debugging because each piece is correct before I build on it.

Trick 4: Temperature Isn't What You Think

I used to set temperature to 0.0 for "factual" tasks and 0.7 for "creative" tasks. That's what every tutorial says.

For small models, I've found the opposite works better:

• Code and structured output: 0.0 — no surprise, this is standard
• Summarization and extraction: 0.0 — small models hallucinate more at higher temperatures. Keep it deterministic.
• Creative writing: 0.3-0.4 — not 0.7. At 0.7, a 9B model produces wild tangents. At 0.3, it's slightly varied but stays on track.
• Brainstorming: 0.5-0.6 — this is the only time I go above 0.5 on a small model.

The key insight: small models have less "knowledge density" per parameter. Higher temperatures let them drift further from their training distribution, which means more hallucination. Keep the temperature low and guide the creativity through your prompt instead.

Trick 5: The "Bad Example" Anti-Pattern

Small models learn from examples, and they learn from what not to do just as much as what to do.

I include explicit bad examples in my prompts:

Write a technical explanation. Do NOT write like this:

"Machine learning is a fascinating field that has revolutionized countless industries.
In today's rapidly evolving technological landscape, leveraging artificial intelligence
has become paramount for organizations seeking to maintain a competitive edge."

Instead, write like this:

"ML models map inputs to outputs. You train them on data, then they predict on new data.
The three types: supervised (labeled data), unsupervised (no labels), reinforcement (reward signals)."

Your turn. Explain distributed systems:

The bad example shows exactly the kind of verbose, corporate, filler-heavy output I don't want. The good example shows the concise, specific, jargon-free style I do want.

This is especially effective for small models because it gives them two reference points instead of one. They can triangulate the style more accurately.

Trick 6: Post-Processing Pipeline (The Unsexy Secret)

Here's the thing nobody wants to hear: my local model output goes through a second pass.

Not another LLM call. A simple Python script that:

1. Removes filler phrases — "It's important to note that", "In today's world", "As we can see"
2. Enforces sentence length limits — any sentence over 30 words gets flagged for manual review
3. Checks for repetition — if two paragraphs say essentially the same thing, it removes one
4. Validates code blocks — runs python -c "compile(code, ...)" to check for syntax errors

This script catches maybe 15% of the model's output. But that 15% is the difference between "obviously AI-generated" and "reads like I wrote it."

FILLER_PHRASES = [
    "it's important to note",
    "in today's world",
    "as we can see",
    "it's worth mentioning",
    "needless to say",
    "at the end of the day",
    "in this rapidly evolving",
    "leverage",  # as a verb
    "delve into",
    "tapestry",
]

def clean_output(text: str) -> str:
    for phrase in FILLER_PHRASES:
        text = text.replace(phrase.capitalize(), "")
        text = text.replace(phrase.lower(), "")
    return text.strip()

Simple. Dumb. Effective. The model doesn't know I'm doing this, and it doesn't need to.

Trick 7: Context Recycling for Multi-Turn Work

Small models have small context windows. My 9B model handles about 8K tokens of context before it starts losing the thread.

When I'm working on a multi-turn coding session, I don't keep the full conversation history. After every 3-4 exchanges, I summarize the key decisions into a "state" block and start fresh:

[STATE]
Project: Task queue manager in Python
Stack: FastAPI + Redis + SQLite
Decisions so far:
- Using Redis for job queue, SQLite for persistence
- Async workers with asyncio.gather
- Auth via JWT with 5-minute expiry
Current file: worker.py (in progress)
Known issues: Need to handle worker crashes mid-task

Continue implementing the worker crash recovery:

This keeps the context lean and focused. The model gets exactly what it needs — no conversation history padding, no stale context from turns 1-3 that's no longer relevant.

Without context recycling: After 5-6 turns, the model starts contradicting decisions from turn 2. It forgets what it decided and re-decides differently.

With context recycling: I can work for 20+ turns and the model stays consistent because it only sees the current state, not the full history.

Where These Tricks DON'T Help

Let me be honest about the gaps:

Complex reasoning. A 9B model can't trace a 6-step logical argument no matter how you prompt it. I still route complex reasoning to the 30B model or to cloud APIs. Prompt engineering closes the gap for structured, well-defined tasks. It does not create intelligence that isn't there.

Long-context understanding. If I need to analyze a 20-page document, no amount of prompting will make a 9B model with 8K context do it well. I chunk the document, summarize each chunk, then feed the summaries to the model. It works, but it's a workaround, not a solution.

Factual accuracy about recent events. Local models have training cutoffs. My Qwen 3.5 doesn't know what happened in 2026. No prompt trick fixes stale training data. For current information, I use cloud APIs or RAG with real-time data.

Nuanced code architecture. The 30B coder model can write a single module well. It cannot design a multi-service system where auth, data, and queue layers interact correctly. I still design the architecture myself and have the model implement each piece.

The Numbers (Before and After Prompt Engineering)

I tracked output quality for a month before and after I started using these tricks consistently:

Task	Before (raw prompts)	After (engineered prompts)	Cloud (GPT-4o)
Code generation (< 50 lines)	6/10	9/10	9.5/10
Code generation (> 200 lines)	3/10	5/10	8/10
Summarization	5/10	9/10	9/10
Data extraction	6/10	8.5/10	9/10
Creative writing	4/10	7/10	8.5/10
Debugging	5/10	7/10	9/10

For tasks under ~50 lines of code, the gap between well-prompted local and cloud is almost gone. For longer tasks, local models still lag, but prompt engineering narrows it from "embarrassing" to "acceptable."

The Quick Reference

If you take one thing from this article, make it this checklist:

1. System prompt = Role + Format + Constraints — never just "you are helpful"
2. One-shot examples — always show the model exactly what you want
3. Decompose big tasks — 4 small prompts beat 1 big prompt
4. Low temperature — 0.0 for code, 0.3-0.4 for creative, never above 0.6 on small models
5. Bad examples — show what NOT to write
6. Post-processing — a 20-line Python script catches 15% of AI-isms
7. Context recycling — summarize state, don't keep full history

These aren't theoretical. I use every single one daily on my Mac Mini and RTX 3060 setup. They took my local models from "good enough for rough drafts" to "good enough for production output."

The model didn't change. My prompts did.

Comments

[Baptiste Le Bouquin]

The role + format + constraints split is underrated — most people write one paragraph that does all three jobs badly. On a small model I've found the FORMAT section matters most, because a 9B will happily wrap code in an essay if you don't forbid it.

One thing I'd add: for tool-use or agentic loops, keep the system prompt constant and vary only the user message. Small models drift when the instructions change mid-conversation, and re-reading a long system prompt eats context you can't spare at 9B scale.

Do you temperature-tune differently for small models too, or is the prompting alone enough in your case?