ai #machinelearning #llm #opensource #devchallenge
The Problem
Not every AI request needs an expensive language model.
Think about it:
- Extracting a value from JSON.
- Formatting a block of text.
- Summarizing a document.
- Debugging a complex algorithm.
- Analyzing a system design problem.
Would you use the same model for all these tasks?
Probably not.
Simple tasks can often be handled by smaller, cheaper models, while complex reasoning tasks may require more capable models. The challenge is figuring out which model to use before processing the actual request.
That's where LLM routing comes in.
What Is Complexity Scorer?
Complexity Scorer is a project designed to estimate how much reasoning a prompt requires.
It uses a fine-tuned Qwen3-8B model to assign a complexity score between 0.00 and 1.00.
| Score | Level | Example |
|---|---|---|
| 0.00 | Trivial | Extracting a value or formatting text |
| 0.25 | Easy | Following a few straightforward steps |
| 0.50 | Moderate | Combining information and constraints |
| 0.75 | Hard | Analyzing trade-offs and complex requirements |
| 1.00 | Very hard | Deep reasoning across complex systems |
The model doesn't answer the user's prompt. It estimates its complexity so that another component can decide which model should handle it.
For example:
User Prompt
|
v
Complexity Scorer
|
v
Complexity Score
|
v
Model Router
|
+----> Cheaper Model
|
+----> Advanced Reasoning Model
The goal is to reduce unnecessary API spending without sacrificing answer quality.
The Experiment
The project used Tinker to fine-tune Qwen3-8B and compare its performance against the original model.
The results were interesting.
| Experiment | Training Examples | Accuracy |
|---|---|---|
| Original Qwen3-8B | 0 | 50% |
| Fine-tuned model | 50 | 86% |
| Fine-tuned model | 500 | 74% |
| Fine-tuned model | 2,500 | 76% |
The evaluation dataset contained 50 examples, and the same evaluation set was used for these comparisons.
The final model trained on 2,500 examples achieved 76% accuracy, a 26-percentage-point improvement over the original model.
But here's the surprising part: the model trained on only 50 examples achieved the highest accuracy on this particular evaluation set.
Why Didn't More Data Improve Accuracy?
More training examples don't automatically translate into better results.
There are several possible explanations.
1. Synthetic data quality
AI-generated training examples can contain inconsistent labels. Two similar prompts might receive different complexity scores, making it harder for a model to learn consistent boundaries between classes.
2. Evaluation dataset size
The evaluation set contained only 50 examples. That's a small sample, so the 86% result could reflect how well the model performed on those particular examples rather than generalize to unseen prompts.
3. More data isn't always better data
Increasing the dataset from 500 to 2,500 examples improved accuracy from 74% to 76%.
That's only one additional correct prediction out of 50.
A larger, more diverse evaluation set would be needed to determine whether that improvement is meaningful.
The takeaway: data quality, label consistency, and evaluation methodology matter just as much as dataset size.
Building the API
After fine-tuning the model, the next step was to make it accessible to developers.
The project included a .NET Minimal API with features such as:
- API key authentication.
- Rate limiting.
- Exception handling.
- Integration with the Tinker inference API.
- A browser-based playground for testing prompts.
The model adapter was also published to Hugging Face, making it available for other developers to explore.
This separates the complexity-estimation step from the application that ultimately answers the user's question.
Why Open-Weight Models Matter
Using a closed AI model for every classification request would be one way to build this system, but it would introduce additional provider dependencies and inference costs.
An open-weight model provides more flexibility:
- Fine-tune it for a specific task.
- Experiment with different training datasets.
- Share the resulting model with other developers.
- Explore local deployment with a compatible runtime.
Local deployment still requires suitable hardware, and open weights alone don't guarantee lower total costs. However, they provide more control over how the system is deployed and improved.
Lessons Learned
This experiment reinforced a few important ideas for building AI-powered applications:
1. Optimize for the real objective.
A router needs to balance inference cost, latency, and the risk of sending a difficult request to an insufficient model.
2. Treat evaluation results carefully.
A small test set can make performance differences look more meaningful than they really are.
3. Build guardrails around AI tools.
Automated agents that can access paid APIs or run training jobs should have clear spending limits and permission boundaries.
4. Don't assume that scaling data will solve everything.
Better data generation, consistent labels, and stronger evaluation methods can be more valuable than simply increasing the number of examples.
What's Next?
There are several directions worth exploring:
- Evaluating the model on a much larger, held-out dataset.
- Testing whether complexity scores correlate with actual task difficulty.
- Measuring routing decisions against real inference costs and answer quality.
- Comparing the fine-tuned model with simpler classification approaches.
- Experimenting with confidence thresholds and fallback mechanisms.
Ultimately, a useful LLM router shouldn't just predict complexity accurately. It should help applications make better decisions about cost, speed, and reliability.
Discussion
How would you approach building an intelligent LLM router?
Would you start with a fine-tuned model, a lightweight classifier, or a rule-based system?
I'd be interested to hear how other developers are approaching this problem.
Note: The results described above are from the experiment summarized in this article. The small evaluation set limits how confidently we can generalize its accuracy results.
Top comments (0)