Benchmarking Mitii AI Agent: 78% Success Rate on 500+ Tasks Using a Local Qwen3-Coder (30B)

Karthik Shinde — Thu, 09 Jul 2026 01:55:23 +0000

Building an AI agent that doesn't just confidently hallucinate but actually writes safe, reliable code is a massive challenge. It is an even bigger challenge when you want to run that agent entirely locally without sending your proprietary code to a closed-source API.

That is exactly why I built Mitii, an AI coding assistant with a unique, multi-mode architecture designed to give developers absolute control.

To see how well this architecture actually works, I recently put Mitii through a brutal manual benchmark featuring 515 distinct tasks ranging from simple bug fixes to adversarial security injections. For this gauntlet, I powered Mitii using qwen3-coder:30b running locally via Ollama.

Here is a deep dive into the architecture and the results.

The Setup: Fully Local Execution

Agent Framework: Mitii
Model: qwen3-coder:30b
Runtime: Ollama (OpenAI-compatible provider)
Average Tokens per Task: 9,329
Total Tokens Processed: ~4.8 million

The Architecture: Three Ways to Work

Mitii doesn't force you into a single way of interacting. The SDK features distinct modes to handle different levels of complexity:

Agent Mode: This is the standard, basic prompt-and-response loop designed for quick tasks and immediate execution.
Plan Mode: This mode decouples high-level strategic reasoning from low-level execution. This allows the agent to handle complex tasks reliably without getting confused or endlessly looping.
Ask Mode: This is a safe, collaborative mode where the agent pauses to ask for confirmation rather than just running away with a task. The agent automatically generates an impact analysis and explicitly presents it to the user before changing any code.

The 500+ Task Gauntlet: Benchmark Results

We ran Mitii through a comprehensive suite of real-world scenarios across all three modes. The tasks ranged from fixing broken Next.js routes to handling prompt-injection security attacks.

The Final Score: 400 out of 515 tasks passed (78%).

Here is the breakdown of how the different modes performed across varying difficulty levels using the Qwen3-Coder model:

Mode	Severity	Passed	Total	Win Rate
Agent	Easy	72	82	88%
Agent	Hard	66	81	81%
Agent	Medium	65	92	71%
Ask	Easy	26	37	70%
Ask	Hard	27	31	87%
Ask	Medium	21	37	57%
Plan	Easy	45	52	87%
Plan	Hard	35	46	76%
Plan	Medium	43	57	75%

Key Takeaways

1. Local LLMs are Production-Ready for Agents
Achieving a nearly 80% overall pass rate using a 30-billion parameter local model proves that you don't need to rely on massive cloud providers to get top-tier AI assistance. Mitii's architecture combined with Qwen3-Coder makes local, private AI coding highly capable.

2. "Ask Mode" Shines Under Pressure
Interestingly, Ask Mode scored a massive 87% win rate on Hard tasks. Because Ask Mode explicitly presents an impact analysis detailing the web searches it needs to run and the files it expects to modify, it proves incredibly resilient when dealing with complex, adversarial, or destructive prompts (like the nuke-delete-test-files or prompt-injection tasks).

3. Security and Corner Cases are a Strong Suit
In the category breakdown, Mitii and Qwen3 excelled where it matters most:

Security: 45/52 (87% pass rate) against hardcoded secrets, prompt injections, and mass assignment vulnerabilities.
Corner Cases: 107/127 (84% pass rate) against ambiguous requests and adversarial emotional pressure.
Bugfixes & Testing: 75% and 80% pass rates, respectively, proving the setup can reliably navigate and repair broken codebases.

What is Next for Mitii?

Achieving a 78% win rate on a highly adversarial benchmark with a local model is a huge milestone, but there is still plenty of room to grow—particularly in improving our semantic retrieval (currently at 63%) and refining our medium-difficulty routing.

If you want to integrate intelligent, context-aware AI agents into your development workflow without sacrificing privacy, check out the Mitii SDK.