DEV Community: Ayush

Fine-Tuning Qwen3-8B for Smarter LLM Routing: Why More Data Didn't Mean Better Accuracy

Ayush — Sun, 18 Oct 2026 16:48:00 +0000

ai #machinelearning #llm #opensource #devchallenge

The Problem

Not every AI request needs an expensive language model.

Think about it:

Extracting a value from JSON.
Formatting a block of text.
Summarizing a document.
Debugging a complex algorithm.
Analyzing a system design problem.

Would you use the same model for all these tasks?

Probably not.

Simple tasks can often be handled by smaller, cheaper models, while complex reasoning tasks may require more capable models. The challenge is figuring out which model to use before processing the actual request.

That's where LLM routing comes in.

What Is Complexity Scorer?

Complexity Scorer is a project designed to estimate how much reasoning a prompt requires.

It uses a fine-tuned Qwen3-8B model to assign a complexity score between 0.00 and 1.00.

Score	Level	Example
0.00	Trivial	Extracting a value or formatting text
0.25	Easy	Following a few straightforward steps
0.50	Moderate	Combining information and constraints
0.75	Hard	Analyzing trade-offs and complex requirements
1.00	Very hard	Deep reasoning across complex systems

The model doesn't answer the user's prompt. It estimates its complexity so that another component can decide which model should handle it.

For example:

User Prompt
     |
     v
Complexity Scorer
     |
     v
Complexity Score
     |
     v
Model Router
     |
     +----> Cheaper Model
     |
     +----> Advanced Reasoning Model

The goal is to reduce unnecessary API spending without sacrificing answer quality.

The Experiment

The project used Tinker to fine-tune Qwen3-8B and compare its performance against the original model.

The results were interesting.

Experiment	Training Examples	Accuracy
Original Qwen3-8B	0	50%
Fine-tuned model	50	86%
Fine-tuned model	500	74%
Fine-tuned model	2,500	76%

The evaluation dataset contained 50 examples, and the same evaluation set was used for these comparisons.

The final model trained on 2,500 examples achieved 76% accuracy, a 26-percentage-point improvement over the original model.

But here's the surprising part: the model trained on only 50 examples achieved the highest accuracy on this particular evaluation set.

Why Didn't More Data Improve Accuracy?

More training examples don't automatically translate into better results.

There are several possible explanations.

1. Synthetic data quality

AI-generated training examples can contain inconsistent labels. Two similar prompts might receive different complexity scores, making it harder for a model to learn consistent boundaries between classes.

2. Evaluation dataset size

The evaluation set contained only 50 examples. That's a small sample, so the 86% result could reflect how well the model performed on those particular examples rather than generalize to unseen prompts.

3. More data isn't always better data

Increasing the dataset from 500 to 2,500 examples improved accuracy from 74% to 76%.

That's only one additional correct prediction out of 50.

A larger, more diverse evaluation set would be needed to determine whether that improvement is meaningful.

The takeaway: data quality, label consistency, and evaluation methodology matter just as much as dataset size.

Building the API

After fine-tuning the model, the next step was to make it accessible to developers.

The project included a .NET Minimal API with features such as:

API key authentication.
Rate limiting.
Exception handling.
Integration with the Tinker inference API.
A browser-based playground for testing prompts.

The model adapter was also published to Hugging Face, making it available for other developers to explore.

This separates the complexity-estimation step from the application that ultimately answers the user's question.

Why Open-Weight Models Matter

Using a closed AI model for every classification request would be one way to build this system, but it would introduce additional provider dependencies and inference costs.

An open-weight model provides more flexibility:

Fine-tune it for a specific task.
Experiment with different training datasets.
Share the resulting model with other developers.
Explore local deployment with a compatible runtime.

Local deployment still requires suitable hardware, and open weights alone don't guarantee lower total costs. However, they provide more control over how the system is deployed and improved.

Lessons Learned

This experiment reinforced a few important ideas for building AI-powered applications:

1. Optimize for the real objective.

A router needs to balance inference cost, latency, and the risk of sending a difficult request to an insufficient model.

2. Treat evaluation results carefully.

A small test set can make performance differences look more meaningful than they really are.

3. Build guardrails around AI tools.

Automated agents that can access paid APIs or run training jobs should have clear spending limits and permission boundaries.

4. Don't assume that scaling data will solve everything.

Better data generation, consistent labels, and stronger evaluation methods can be more valuable than simply increasing the number of examples.

What's Next?

There are several directions worth exploring:

Evaluating the model on a much larger, held-out dataset.
Testing whether complexity scores correlate with actual task difficulty.
Measuring routing decisions against real inference costs and answer quality.
Comparing the fine-tuned model with simpler classification approaches.
Experimenting with confidence thresholds and fallback mechanisms.

Ultimately, a useful LLM router shouldn't just predict complexity accurately. It should help applications make better decisions about cost, speed, and reliability.

Discussion

How would you approach building an intelligent LLM router?

Would you start with a fine-tuned model, a lightweight classifier, or a rule-based system?

I'd be interested to hear how other developers are approaching this problem.

Note: The results described above are from the experiment summarized in this article. The small evaluation set limits how confidently we can generalize its accuracy results.

🌱 TrailQuest - An Open-AI Adventure Agent That Wants You to Use Your Phone Less

Ayush — Thu, 08 Oct 2026 07:33:03 +0000

🌱 TrailQuest

"AI that wants you to use your phone less."

Most AI applications today demand your undivided attention. They want you staring at a glowing rectangle, typing prompts, scrolling chats, or watching streaming tokens.

We built TrailQuest with the exact opposite mission for Hacktoberfest 2026 Week 1 ("Touch Grass"):

What if open-source AI's primary metric of success was how fast it convinced you to put your phone in your pocket and step outside?

TrailQuest is an open-AI adventure agent that turns open-weight models (Gemma, Llama 3.2, vLLM, Ollama) into real-world outdoor micro-adventures tailored to your time budget, mood, and surroundings where the screen is intentionally the shortest part of the experience.

🌲 Why Open AI Matters for "Touch Grass"

When you take AI outdoors into state parks, local trailheads, urban green spaces, or backyard garden patches traditional closed-cloud APIs fall short:

Zero Signal on the Trail (Local & Offline First): Cellular signals drop the moment you step off the paved path. TrailQuest features a built-in browser-edge fallback engine powered by local open weights (Ollama/vLLM/Gemma). Your adventure agent generates, adapts, and speaks guidance with zero internet connection required.
Total Location & Sensory Privacy: Planning an outdoor quest requires location context, local environment sensory data, and optional reflection photos. With open-weight models running on your device or self-hosted server, your personal location and photos never hit a centralized proprietary cloud API.
Unrestricted Customization & Model Swapping: Closed models enforce rigid guardrails that make dynamic real-world outdoor quest design formulaic. Open weights allow fine-tuning (via Thinking Machines' Tinker) and local prompt engineering for hyper-local botanical, birding, and foliage quests.

🚀 How TrailQuest Works: The 3-Minute Screen Experience

TrailQuest follows a strict "Screen-Minimizing" design workflow:

1. The Touch Grass Wizard (`/quest`)

Select your Time Budget (15 min micro-walk, 30 min park exploration, or 60+ min nature trek), your Vibe/Mood (Curious, Mindful, Energetic, Playful), and your Difficulty Level. Optional weather and geolocation sensors automatically contextualize the quest.

[⏱️ 15 Min] + [🧘 Mindful] + [🌿 Micro-Nature] ➔ 🎯 "The Pine Needle & Cloud-Tracking Quest"

2. AI Agent Reasoning & Safety Blueprint (`/quest/[id]`)

In seconds, the open model agent generates:

🎯 Primary & Secondary Objectives (e.g., "Find three distinct tree leaf textures without picking them", "Observe bird calls for 2 minutes motionless").
📜 Quest Rules & Constraints (e.g., "No phone usage during travel between markers").
🛡️ Safety Cautions & Environmental Tips (e.g., "Watch for poison ivy along low fences; sunset in 45 minutes").
🔍 Sentry Agent Trace: Full transparent breakdown of LLM tool execution, prompt construction, and latency.

3. Outdoor Active Focus Mode (`/quest/[id]/active`)

The centerpiece of TrailQuest. The moment you press "Begin Outdoor Quest":

📵 Phone Away Screen: A 5-second countdown prompts you to place your phone in your pocket or backpack.
🔊 Hands-Free Audio Guidance: ElevenLabs / Web Speech API voice synthesis narrates quest checkpoints into your earbuds so you never need to look at the screen.
🌿 Ultra-Minimal Display: Shows only an ambient pulse, elapsed outdoor time, and a voice toggle button.

4. Post-Quest Reflection & Leveling (`/quest/[id]/complete`)

Upon return, complete a 30-second reflection, review your Score Ring, gain XP, level up your Adventurer Rank, and unlock outdoor badges ("Oak Whisperer", "Streak 7: Grass Master").

🛠️ Technical Architecture & Stack

TrailQuest is built with Next.js 16 (App Router), React 19, TypeScript, Tailwind CSS v4, and Framer Motion.

                 ┌────────────────────────────────┐
                 │       🌱 TrailQuest Web        │
                 └──────────────┬─────────────────┘
                                │
                 ┌──────────────▼─────────────────┐
                 │    ⚡ Unified Provider Layer   │
                 └──────┬─────────────────┬───────┘
                        │                 │
    ┌───────────────────▼───┐         ┌───▼───────────────────┐
    │  🤖 Open LLM Engine   │         │  🌐 Edge Offline Engine│
    │ (Gemma/Ollama/vLLM)   │         │ (Browser WebSpeech/JS)│
    └───────────────────────┘         └───────────────────────┘

Pluggable Provider System (`src/services/ai/`)

Every core subsystem in TrailQuest is built behind strict TypeScript interface abstractions (src/types/providers.ts):

LLMProvider: Switch seamlessly between GemmaProvider (Ollama/vLLM/OpenRouter) and MockLLMProvider (demo mode).
SpeechProvider: High-definition ElevenLabsProvider with Web Speech API fallback.
MemoryProvider: Long-term quest memory via BackboardProvider and local storage cache.
VectorStoreProvider: Hybrid outdoor RAG using TigerDataProvider (pgvector).
SearchProvider: Real-time weather and trail grounding via SerpApiProvider.

🏆 Partner Category Integrations

Partner Category	Integration in TrailQuest	Implementation
🤖 Gemma (Best Use of Gemma)	Primary open-weight LLM powering quest generation, constraint checking, and safety analysis.	`GemmaProvider` (`gemma-3-27b-it`)
📊 TabPFN (Prior Labs)	Predicts optimal quest difficulty & completion probability based on historical user attempt logs.	TabPFN Tabular Predictor Integration
☁️ Render (Best Use of Render)	Production web runtime hosting the Next.js frontend and Node API routes.	Render Web Service deployment configuration
🔊 ElevenLabs	Crystal-clear voice narration for hands-free audio checkpoints during active outdoor mode.	`ElevenLabsProvider`
🍃 MongoDB Atlas	Vector Search & persistence layer storing user profiles, quest attempts, and unlocked achievements.	MongoDB Atlas schemas & native API routes
📡 Sentry Agent Tracing	Transparent agent execution tracing detailing prompt tokens, execution latency, and step traces.	`/developer` & `/quest/[id]` Sentry agent trace
🔍 SerpApi	Live weather data & regional trail search grounding for safe outdoor adventure planning.	`SerpApiProvider`
🐯 Tiger Data	Spatial vector database for regional flora/fauna RAG retrieval.	pgvector Tiger Data provider
💾 Backboard	Persistent user quest memory across sessions.	`BackboardProvider`

📸 Screenshots & Product Walkthrough

1. Landing Page — "Touch Grass"

2. Quest Generator Wizard

3. Active Outdoor Mode — Phone Away UX

4. Developer Diagnostic Dashboard

💻 Local Quickstart

Want to run TrailQuest locally? It works out of the box with zero setup:

# Clone the repository
git clone https://github.com/Gitayush1/trailquest.git
cd trailquest/trailquest-app

# Install dependencies
npm install

# Start local server (Demo Mode enabled by default)
npm run dev

Open http://localhost:3000 in your browser!

🧠 Open Innovation Reflection

Building TrailQuest reinforced why open-source AI is essential for the future of ambient computing:

When software is designed to enrich offline, physical human lives rather than extract maximum online screen time, open-weight models are not just an alternative they are the foundational enabler. They allow intelligence to exist on the edge, respect human privacy, and run freely anywhere under the open sky.

Submitted for **Hacktoberfest 2026 Open-Source AI Challenge: Week 1 ("Touch Grass").
Tags: #devchallenge #hf26challenge

💬 Let's Connect & Discuss!

I'd love to hear your thoughts, feedback, and ideas! Please leave a comment below with any questions, suggestions, or your favorite outdoor spots.

Why I Wrote TrailQuest

Like many developers, I often find myself spending 10+ hours a day glued to screens debugging code, reading docs, and tweaking prompts. I realized that even when I wanted to step away and recharge outside, friction (deciding where to go, what to do, or how long to stay) would pull me right back to my phone.

I built TrailQuest to prove that AI doesn't have to compete for our screen time it can act as a catalyst to get us outside, explore our surroundings mindfully, and enjoy the physical world.

Whether you're looking for micro-walk ideas between work sprints, want to try screen-free outdoor challenges, or are interested in local open-weight AI architectures, let me know in the comments below! What outdoor micro-adventures would you like TrailQuest to generate for you? 🌿