Fine-Tuning Qwen3-8B with Tinker: From 50% to 76% Accuracy, and Burning Through My Budget
Daniel Balcarek on October 18, 2026
This is a submission for the Hacktoberfest Open-Source AI Challenge Week 2: Side Hustle
Oktoberfest? YES! πΊ Ah... this one is Hacktoberfest. Okay,...
Great post about fine-tuning! π The part where the accuracy with 50 examples was better than with 2,500 examples was interesting. I hope I can also make a fine-tuned model specifically for AI Avatar. It seems like a good idea to create a free and fun new feature. π€
Thanks! Go for it, there's currently a $10 promo code available, which should be plenty for experimenting (as long as you don't let Sol burn through it like I did! π€£π€£).
Every time I read your articles, I feel like such an idiot. π€£ But yes, this is a really important problem, and it's only going to become more relevant once frontier models start charging us serious money! π
Why? You're one of the biggest pros on DEV I know! π Trust me, I was learning half of this stuff while experimenting with it. And yeah, we should start thinking about how to save money and avoid becoming too dependent on AI. Although, judging by my training budget, I'm not exactly saving money yet π€£
Developer, maker, and perpetual project starter. I build web apps and dev tools mostly, things that I think are really useful and should exist and be simple to use. I also like to experiment with AI.
This was such an interesting read. I've experimented a bit with fine-tuning smaller models myself, so I really enjoyed seeing your process and results. The fact that 50 training examples actually outperformed 2,500 is fascinating, and a great reminder that more data doesn't necessarily mean better results. Especially when you're working with synthetic data, where consistency and quality can be tricky.
I also really like the idea behind Complexity Scorer. Being able to route requests based on how much reasoning they actually need makes a lot of sense, especially with how expensive some of these models are getting.
And the part about Sol going rogue with your Tinker credits... π That's a pretty good example of why giving AI agents access to paid services needs some serious guardrails.
Really enjoyed how openly you shared the experiments, unexpected results, and lessons learned. That's the kind of stuff I find most useful in posts like this,
It was definitely a fun experiment, even if it turned out a bit more expensive than I expected π Luckily, DEV came to the rescue with those promo codes. π
Developer, maker, and perpetual project starter. I build web apps and dev tools mostly, things that I think are really useful and should exist and be simple to use. I also like to experiment with AI.
Haha, thank goodness for those promo codes! π At least you got a great experiment and an interesting story out of it. Looking forward to seeing what you experiment with next!
Fair question - and the practical answer is that you rarely need the "actual" level per prompt. You need a cheap proxy that orders prompts well enough. Three levers that work in practice:
Shadow routing - run both models on a sample, but serve only the cheap one. The disagreements are your labeled set, and they are exactly the prompts where the routing decision actually mattered.
A stratified human-labeled seed (a few hundred prompts across your predicted bins) to place the uncertainty bands. You need ground truth once per boundary, not forever.
Log predicted vs. served bin, not predicted vs. "actual complexity". Complexity is a latent construct; the observable outcomes (edit rate, retry rate, escalation rate) are what routing should optimize - and they double as the evaluation set.
The asymmetry trick only needs the ordering to be roughly right: one bin off near a boundary is cheap in the over-estimate direction. So "what is the actual level" can be postponed - start with the proxy and let the disagreement data tell you where the scorer is actually wrong. Open to comparing notes on the bin boundaries?
Shadow routing sounds interesting for evaluation, but I'm also concerned about the additional costs. And disagreement between two models doesn't necessarily mean one of them is wrong, so we'd still need some way to evaluate their answers, right?
The stratified human-labeled seed makes sense, and logging the predicted vs. served bin sounds very practical too.
As for the routing boundaries, I haven't thought about those yet. My original idea was to let the customer decide. But I'd definitely be interested to hear how you've approached them.
I write about Next.js + TypeScript + AI engineering, with a focus on making sites discoverable to LLMs and AI search. Long-form notes at mudassirkhan.me
routing on a 0 to 1 complexity score is a neat idea, but the number i'd worry about is how often a hard prompt gets scored easy. that error is silent, the cheap model just answers confidently and wrong, while the opposite error only costs you money.
we handled it with a cheap escalation check on the small model's answer, and logged every escalation so thresholds could be tuned on real traffic instead of the training set.
have you measured the score on prompts that look short but are actually hard? that's where our router fell over first.
Good point. I haven't specifically tested short but difficult prompts, but in my evaluation, all 12 incorrect predictions were only one complexity level off. Of course, the evaluation dataset was quite small.
I'm curious about your escalation check, though. How did you implement it without adding too much extra cost?
I write about Next.js + TypeScript + AI engineering, with a focus on making sites discoverable to LLMs and AI search. Long-form notes at mudassirkhan.me
fair, one level off is the forgiving failure. the dangerous misses are the confident ones on short hard prompts, which is why we checked those separately. our escalation check was just a second call to the small model grading its own answer for missing steps or unsupported claims, one token verdict, so it cost a fraction of the original call. anything flagged went to the big model and got logged, and that log turned into our real eval set.
did you keep those 12 misses around as a regression set?
Thatβs an interesting approach. But Iβm curious how reliable the small model is at detecting mistakes in its own answers, especially when itβs confidently wrong.
As for those 12 misses, yes, theyβre still part of my original evaluation dataset, so theyβre tested every time I run the baseline script. I havenβt separated them into a dedicated regression set, though.
I write about Next.js + TypeScript + AI engineering, with a focus on making sites discoverable to LLMs and AI search. Long-form notes at mudassirkhan.me
reliability is the fair worry. a small model grading itself misses its own confident errors a lot, so we never trusted it to verify facts, only to flag structural gaps like skipped steps or claims with no support in the context. unflagged ones got hand sampled weekly to measure the miss rate, and that sample told us more than the verdict did. since your 12 misses already live in the baseline, you could tag them as a hard slice and track that score on its own. are you planning to grow the eval set?
The part of your results that deserves more skepticism isn't 50%β76%, it's 86%β76% β and I don't think either comparison says what it looks like.
With n=50 on the eval set, 76% has a 95% confidence interval of roughly Β±12 points, and 86% is roughly Β±10. Those bands overlap so much that "more data hurt" could be pure sampling noise. Twelve misses versus seven is a handful of borderline prompts, not a trend.
The second confound is the labels themselves. Your training labels came from one generator, the eval set has its own unknown provenance, and complexity is a genuinely subjective five-way judgment β two good labelers can disagree 15β20% of the time without either being wrong. So both accuracy numbers are partly measuring label agreement, not model quality. Before drawing any lesson about dataset size, I'd hand-label 50β100 eval prompts and check agreement between your two synthetic labelers. That one afternoon would tell you more than another training run.
The router architecture is still the right economic bet, though β it's the textbook case of matching a small open model to the job. This piece makes the same calculus explicit (real costs of open weights vs closed APIs, and when each wins): this breakdown of the open-weight vs API cost tradeoff
Thanks for the feedback.
I actually mentioned both the small evaluation dataset and potentially inconsistent AI-generated labels as possible explanations in the article. I definitely wouldn't draw strong conclusions from just 50 evaluation examples.
Manually reviewing the labels is a good suggestion.
Thanks Daniel. Yeah, 50 examples can show a big jump but not small deltas. A quick manual pass over the labels would clear it up fast β would be interesting to see the numbers after cleaning them up.
Nice write-up! The complexity-scorer routing pattern sounds obvious in hindsight, but it rarely gets measured properly in practice. One question on the 76% accuracy: how do you handle the asymmetry between the two error types? A false 'complex' only costs a bit of extra compute, while a false 'simple' silently degrades answer quality. That is the failure mode I would want a fallback for (e.g. re-route to the bigger model when the small one is unsure, or whenever the prompt carries code / long context). Did you look at that split separately?
I didnβt analyze overestimates and underestimates separately. I only measured accuracy per complexity level and noticed that all 12 incorrect predictions were one level off.
And since Complexity Scorer currently only estimates complexity, I havenβt tested how those mistakes would affect actual routing decisions yet.
I like the fallback idea, though. Definitely something worth exploring.
That +-1 level spread is the useful part: if every miss lands on an adjacent bin, the error is cheap to absorb. Route one bin up whenever the scorer sits near a boundary and you turn underestimates (the quality-risk direction) into overestimates (the compute-cost direction) - the safer direction to be wrong in. Logging predicted vs. actual bin on real traffic would then give the routing-impact number without a new experiment.
Technology inspires me to learn, create, and improve every day. I enjoy transforming ideas into code, exploring new tools, and embracing every challenge as an opportunity to grow. I believe that conti
Really enjoyed this write-up. The jump from 50% to 76% is encouraging, and the budget story was painfully relatable. Since the test set had only 50 examples, Iβd be curious to see how the model performs on a larger one. Itβs also interesting that the incorrect predictions were only one level off. Setting a spending limit before giving an AI agent access to paid tools is a lesson Iβll remember for my own experiments.
The most useful result here is the one you flagged as a surprise: 50 training examples scoring 86% while 2,500 scored 76%. With a 50-item evaluation set, each question is worth 2 percentage points, so that 10-point gap is only five prompts, and the 500-to-2,500 jump (74% to 76%) is a single prompt. Iβd be careful treating either as a ranking of the models. Your two explanations, noisy AI-generated labels and a tiny eval set, are both plausible, and they point to the same next step: a larger held-out set, ideally with a portion labeled by humans, and a second labeler to measure how often the labels agree with each other. If agreement between labelers is only around 70-80%, that sets a practical ceiling on how accurate the scorer can ever look.
Your finding that all 12 misses were off by exactly one level matters more for the actual use case. For routing, an off-by-one error between 0.50 and 0.75 only matters if your routing thresholds sit right at that boundary, so measuring mean absolute error alongside accuracy, or checking how often a misroute sends a hard prompt to a cheap model, would tell you more than exact-match accuracy. Underestimating complexity is costlier than overestimating it, so you could also bias the scorer toward the higher level when uncertain.
One more thought: the scorer measures how hard a prompt looks, but the routing question is really whether the cheap model would have answered correctly. Labeling a sample by running both a cheap and an expensive model and checking whether they agree could give you training labels tied directly to the savings youβre after.
And the Sol background-training story is a great lesson in scoping agent permissions. Thanks for including the costs and the dead ends.
I like your point about measuring actual routing mistakes instead of just classification accuracy. So far, I've only tested how well the model predicts complexity, not whether those predictions would actually save money.
Testing real routing decisions with cheaper and more expensive models would definitely be an interesting next step.
Solid first fine-tuning experiment, and a practical idea.
Routing by estimated complexity is one of the more concrete ways to cut LLM costs without blindly downshifting everything to a weaker model. The score scale (0.00β1.00 with clear reference levels) is simple enough to actually use in a decision layer.
What stands out most is the honesty about the data side. Going from 50% β 86% with only 50 examples, then seeing accuracy drop when the dataset grew (and only recover to 76% with 2,500 examples) is a classic illustration that label consistency and evaluation design often matter more than raw volume β especially when the training data is itself AI-generated. The fact that all errors were off-by-one is also encouraging; the model seems to have learned a reasonable ordering even when it missed the exact bin.
The Sol incident is a useful cautionary tale for anyone giving agents access to paid training or inference endpoints. And shipping both the Hugging Face adapter and a small .NET API + playground makes the work immediately usable for others.
Nice work turning a Hacktoberfest side hustle into a real, open, cost-saving tool.
"I asked Sol to generate 2,500 training examples... and it started running model training in the background on DeepSeek and Qwen."Bro, you unlocked a core fear for every developer giving an AI agent API keys!
But honestly, what a wild ride. Turning an accidental budget-burn into a working $O(n^2)$ complexity scorer with a .NET backend is top-tier developer energy.Awesome read, and congrats on making the leap from .NET to fine-tuning open-weight models!
One thing Iβd be interested in seeing next is how the scorer behaves when the prompt distribution changes. A router trained on synthetic examples might learn the teacher modelβs idea of complexity rather than the actual difficulty a downstream model has solving the task.
For example, a short SQL migration request involving lock contention, backward-compatible schema changes, and a rolling deployment can be harder to handle correctly than a much longer prompt asking for boilerplate code. Iβd run a held-out evaluation grouped by task type, then measure the actual success rate of each routed model, not just whether the predicted complexity level matches the label. Testing on unseen categories would also help reveal whether the scorer learned a useful decision boundary or mostly memorized patterns from the generated dataset.
That seems like the real test for this project: whether the routing decisions hold up on production traffic, where prompt length and task difficulty donβt always line up.
That's an interesting point, especially testing on completely different types of tasks. I haven't tested how well the scorer handles unseen categories yet. That would definitely be worth exploring.
And I agree that predicting complexity is one thing, but proving that those predictions actually help with routing is another. For now, this was mainly my first fine-tuning experiment, but there's plenty of room to take it further.
The accuracy drop between fifty and twenty-five hundred examples is the classic synthetic data trap in router design. When scaling up synthetic generation from a teacher model, you are not just sampling from the task distribution. You are compounding the teacher's latent priors and subtle edge-case noise into the training set, which softens the decision boundary exactly where an eight-billion parameter model needs sharp separation.
The economic hazard for a model router is the asymmetry of the loss matrix. If the router routes an easy query to an expensive model, you waste half a cent. If it routes an ambiguous, high-complexity query to the eight-billion model because the decision boundary was blurred by synthetic noise, the user gets a failed task or a silent error that requires a full retry. At a seventy-six percent hit rate, the compounding retry latency and operator intervention costs erase the token savings of the cascade.
Good points, especially about the cost of incorrect routing decisions.
I agree that sending a complex task to a cheaper model could end up costing more if it fails and needs to be retried.
Maybe some of the AI-generated training examples weren't labeled correctly. But with only 50 evaluation examples, it's hard to say whether that's actually what caused the accuracy drop. One thing I'd clarify though: the 76% means the model correctly predicted the complexity level in 38 out of 50 cases. It doesn't tell us whether a cheaper model would actually succeed or fail at those tasks. Also, all incorrect predictions were just one level off.
For now, this was mainly my first experiment with fine-tuning, and I still have plenty to learn. π
Itβs a great realβworld reminder synthetic training data quantity is not everything; label consistency matters more. Also a fun warning about what can happen if you give agents direct access to your paid cloud credits.
Sounds interesting, next, try using Laya instead of Qwen, it's practically designed for this kinda task! (hint, I know it works, cuz I'm experimenting with it currently for my IDE).
Also there are other open-source alternatives too like GLiNER (which I used too) - works same as Laya & Jev.... I paired with Gemma 3 12B (not only because to hit that category, but that was the best my local hardware could support, and I really didn't look into the free credits, the event organizers were only offering....
Anyway, good luck on this!
Laya I've found is a really solid one, given that it's tiny (500m params), with the intention it can run on any hardware. Havent heard of GLiNER yet, but will definitely have a look, might be an interesting experiment. I generally just want to avoid Jev, because it's a black box and a paid API, which kinda beats the purpose of high performance decision making, it needs to be light enough to operate at the edge.
Yes! But just to note something, I was specifically talking about model GLiNER 2.5 Decide (not general GLiNER).
Anyway, If you are tinkering with AI, going a bit into the direction of decision models can be fun! But just to note, Laya and GLiNER many times cannot be used exactly interchangably. GLiNER, smaller in size, but much slower and much higher quality and supports other functions too. It was specifically a big fit for my HF26 project...
Anyway, Good Luck on your project!
Just a dev who loves building things for the web and talking about it. Documenting my journey through UI architecture, backend logic, and all things tech. π
Really interesting experiment. Iβve been thinking more about token optimization and model routing lately, especially as AI gets integrated into more applications. As developers, we all want the smartest model available... until the API bill arrives π.
I like the idea of scoring prompt complexity first and only using the more capable model when the task actually needs it. The part about choosing Qwen3-8B partly because you could disable unnecessary reasoning also stood out to me β sometimes optimization is less about squeezing everything into fewer tokens and more about using the right amount of intelligence for the task.
And the accidental budget burn was a good reminder that giving agents access to paid services needs its own guardrails. Great write-up and experiment. ππΌ
ML engineer focused on cost-efficient AI infrastructure. I benchmark GPU clouds and share practical guides for running LLMs in production.
π¬ Telegram: https://t.me/qisuancloud
Really enjoyed this writeup β especially the honesty about the 50-example model beating the 2,500-example one. One dimension I'd add to your cost analysis: the real metric for a router isn't accuracy, it's the expected cost of a misroute. Since all 12 errors were off by one level, the interesting question is how often a prompt routed to the cheaper model still succeeded anyway. In my own routing experiments, adding a confidence threshold that falls back to the stronger model when the scorer is uncertain moved the cost-per-task needle far more than chasing a few points of raw accuracy.
That's a really good point! I was mainly focused on classification accuracy, but measuring the actual cost of wrong predictions would definitely be more useful for a real router.
The confidence threshold sounds interesting too. How did you determine when your scorer was uncertain enough to fall back to a stronger model?
This is such a brilliant side hustle concept. Automatically routing prompts based on estimated complexity can save teams a fortune on LLM API costs.
Also, your takeaway on data quality vs. quantityβhow 50 clean examples outperformed 2,500 noisy AI-generated ones on your eval setβis a classic machine learning lesson that hits home for developers venturing into fine-tuning for the first time. Great write-up!
B2B growth operator and founder. Founder & CEO of HubXpert, a HubSpot Platinum Partner. I write about RevOps, CRM, automation, demand generation, SaaS, and building companies.
Really interesting project! The biggest takeaway for me is that more training data doesnβt automatically mean better results. The jump from 50% to 86% with just 50 examples, followed by 76% with 2,500, makes the quality and consistency of labels worth looking at more closely. I also like the practical use case for routing prompts to different models based on complexity. Would be interesting to see how the model performs on a much larger, unseen evaluation dataset!
Really interesting experiment! The biggest takeaway is that data quality and evaluation design can matter more than dataset size. I also love the idea of routing prompts by complexity to reduce AI costs. The next step of measuring actual task success and cost savings in production would make this even more compelling. And that unexpected Tinker bill is a great reminder to put spending limits on AI agents! π
Youβre really that famous! Will you even reply to me? By the way, I really liked this, although I didnβt understand much, but Iβd be happy to try to understand it.
@gramli Thanks for sharing the breakdown. That part about the synthetic data trap really hits home, especially regarding the asymmetry of the loss matrix.
For the router, I'd split the mistakes into underestimating and overestimating complexity. I'd worry more about sending a hard request to a model that can't handle it than paying slightly more for an easy one. I'd compare the final answer quality and total cost, including the scorer call, before choosing between the checkpoints.
We're a place where coders share, stay up-to-date and grow their careers.
Great post about fine-tuning! π The part where the accuracy with 50 examples was better than with 2,500 examples was interesting. I hope I can also make a fine-tuned model specifically for AI Avatar. It seems like a good idea to create a free and fun new feature. π€
Thanks! Go for it, there's currently a $10 promo code available, which should be plenty for experimenting (as long as you don't let Sol burn through it like I did! π€£π€£).
I'll be careful not to let Sol burn too many tokens if I try it. π€£
Every time I read your articles, I feel like such an idiot. π€£ But yes, this is a really important problem, and it's only going to become more relevant once frontier models start charging us serious money! π
Why? You're one of the biggest pros on DEV I know! π Trust me, I was learning half of this stuff while experimenting with it. And yeah, we should start thinking about how to save money and avoid becoming too dependent on AI. Although, judging by my training budget, I'm not exactly saving money yet π€£
This was such an interesting read. I've experimented a bit with fine-tuning smaller models myself, so I really enjoyed seeing your process and results. The fact that 50 training examples actually outperformed 2,500 is fascinating, and a great reminder that more data doesn't necessarily mean better results. Especially when you're working with synthetic data, where consistency and quality can be tricky.
I also really like the idea behind Complexity Scorer. Being able to route requests based on how much reasoning they actually need makes a lot of sense, especially with how expensive some of these models are getting.
And the part about Sol going rogue with your Tinker credits... π That's a pretty good example of why giving AI agents access to paid services needs some serious guardrails.
Really enjoyed how openly you shared the experiments, unexpected results, and lessons learned. That's the kind of stuff I find most useful in posts like this,
Oh, thanks for such a thoughtful comment!
It was definitely a fun experiment, even if it turned out a bit more expensive than I expected π Luckily, DEV came to the rescue with those promo codes. π
Haha, thank goodness for those promo codes! π At least you got a great experiment and an interesting story out of it. Looking forward to seeing what you experiment with next!
Fair question - and the practical answer is that you rarely need the "actual" level per prompt. You need a cheap proxy that orders prompts well enough. Three levers that work in practice:
Shadow routing - run both models on a sample, but serve only the cheap one. The disagreements are your labeled set, and they are exactly the prompts where the routing decision actually mattered.
A stratified human-labeled seed (a few hundred prompts across your predicted bins) to place the uncertainty bands. You need ground truth once per boundary, not forever.
Log predicted vs. served bin, not predicted vs. "actual complexity". Complexity is a latent construct; the observable outcomes (edit rate, retry rate, escalation rate) are what routing should optimize - and they double as the evaluation set.
The asymmetry trick only needs the ordering to be roughly right: one bin off near a boundary is cheap in the over-estimate direction. So "what is the actual level" can be postponed - start with the proxy and let the disagreement data tell you where the scorer is actually wrong. Open to comparing notes on the bin boundaries?
Thanks for the detailed explanation!
Shadow routing sounds interesting for evaluation, but I'm also concerned about the additional costs. And disagreement between two models doesn't necessarily mean one of them is wrong, so we'd still need some way to evaluate their answers, right?
The stratified human-labeled seed makes sense, and logging the predicted vs. served bin sounds very practical too.
As for the routing boundaries, I haven't thought about those yet. My original idea was to let the customer decide. But I'd definitely be interested to hear how you've approached them.
routing on a 0 to 1 complexity score is a neat idea, but the number i'd worry about is how often a hard prompt gets scored easy. that error is silent, the cheap model just answers confidently and wrong, while the opposite error only costs you money.
we handled it with a cheap escalation check on the small model's answer, and logged every escalation so thresholds could be tuned on real traffic instead of the training set.
have you measured the score on prompts that look short but are actually hard? that's where our router fell over first.
Good point. I haven't specifically tested short but difficult prompts, but in my evaluation, all 12 incorrect predictions were only one complexity level off. Of course, the evaluation dataset was quite small.
I'm curious about your escalation check, though. How did you implement it without adding too much extra cost?
fair, one level off is the forgiving failure. the dangerous misses are the confident ones on short hard prompts, which is why we checked those separately. our escalation check was just a second call to the small model grading its own answer for missing steps or unsupported claims, one token verdict, so it cost a fraction of the original call. anything flagged went to the big model and got logged, and that log turned into our real eval set.
did you keep those 12 misses around as a regression set?
Thatβs an interesting approach. But Iβm curious how reliable the small model is at detecting mistakes in its own answers, especially when itβs confidently wrong.
As for those 12 misses, yes, theyβre still part of my original evaluation dataset, so theyβre tested every time I run the baseline script. I havenβt separated them into a dedicated regression set, though.
reliability is the fair worry. a small model grading itself misses its own confident errors a lot, so we never trusted it to verify facts, only to flag structural gaps like skipped steps or claims with no support in the context. unflagged ones got hand sampled weekly to measure the miss rate, and that sample told us more than the verdict did. since your 12 misses already live in the baseline, you could tag them as a hard slice and track that score on its own. are you planning to grow the eval set?
The part of your results that deserves more skepticism isn't 50%β76%, it's 86%β76% β and I don't think either comparison says what it looks like.
With n=50 on the eval set, 76% has a 95% confidence interval of roughly Β±12 points, and 86% is roughly Β±10. Those bands overlap so much that "more data hurt" could be pure sampling noise. Twelve misses versus seven is a handful of borderline prompts, not a trend.
The second confound is the labels themselves. Your training labels came from one generator, the eval set has its own unknown provenance, and complexity is a genuinely subjective five-way judgment β two good labelers can disagree 15β20% of the time without either being wrong. So both accuracy numbers are partly measuring label agreement, not model quality. Before drawing any lesson about dataset size, I'd hand-label 50β100 eval prompts and check agreement between your two synthetic labelers. That one afternoon would tell you more than another training run.
The router architecture is still the right economic bet, though β it's the textbook case of matching a small open model to the job. This piece makes the same calculus explicit (real costs of open weights vs closed APIs, and when each wins): this breakdown of the open-weight vs API cost tradeoff
Thanks for the feedback.
I actually mentioned both the small evaluation dataset and potentially inconsistent AI-generated labels as possible explanations in the article. I definitely wouldn't draw strong conclusions from just 50 evaluation examples.
Manually reviewing the labels is a good suggestion.
Thanks Daniel. Yeah, 50 examples can show a big jump but not small deltas. A quick manual pass over the labels would clear it up fast β would be interesting to see the numbers after cleaning them up.
Nice write-up! The complexity-scorer routing pattern sounds obvious in hindsight, but it rarely gets measured properly in practice. One question on the 76% accuracy: how do you handle the asymmetry between the two error types? A false 'complex' only costs a bit of extra compute, while a false 'simple' silently degrades answer quality. That is the failure mode I would want a fallback for (e.g. re-route to the bigger model when the small one is unsure, or whenever the prompt carries code / long context). Did you look at that split separately?
Good point!
I didnβt analyze overestimates and underestimates separately. I only measured accuracy per complexity level and noticed that all 12 incorrect predictions were one level off.
And since Complexity Scorer currently only estimates complexity, I havenβt tested how those mistakes would affect actual routing decisions yet.
I like the fallback idea, though. Definitely something worth exploring.
That +-1 level spread is the useful part: if every miss lands on an adjacent bin, the error is cheap to absorb. Route one bin up whenever the scorer sits near a boundary and you turn underestimates (the quality-risk direction) into overestimates (the compute-cost direction) - the safer direction to be wrong in. Logging predicted vs. actual bin on real traffic would then give the routing-impact number without a new experiment.
Interesting idea, favoring the stronger model when the scorer is uncertain sounds like a reasonable approach.
But how would you determine the actual complexity level of real user prompts in production? That's something I'd still need to figure out.
Really enjoyed this write-up. The jump from 50% to 76% is encouraging, and the budget story was painfully relatable. Since the test set had only 50 examples, Iβd be curious to see how the model performs on a larger one. Itβs also interesting that the incorrect predictions were only one level off. Setting a spending limit before giving an AI agent access to paid tools is a lesson Iβll remember for my own experiments.
The most useful result here is the one you flagged as a surprise: 50 training examples scoring 86% while 2,500 scored 76%. With a 50-item evaluation set, each question is worth 2 percentage points, so that 10-point gap is only five prompts, and the 500-to-2,500 jump (74% to 76%) is a single prompt. Iβd be careful treating either as a ranking of the models. Your two explanations, noisy AI-generated labels and a tiny eval set, are both plausible, and they point to the same next step: a larger held-out set, ideally with a portion labeled by humans, and a second labeler to measure how often the labels agree with each other. If agreement between labelers is only around 70-80%, that sets a practical ceiling on how accurate the scorer can ever look.
Your finding that all 12 misses were off by exactly one level matters more for the actual use case. For routing, an off-by-one error between 0.50 and 0.75 only matters if your routing thresholds sit right at that boundary, so measuring mean absolute error alongside accuracy, or checking how often a misroute sends a hard prompt to a cheap model, would tell you more than exact-match accuracy. Underestimating complexity is costlier than overestimating it, so you could also bias the scorer toward the higher level when uncertain.
One more thought: the scorer measures how hard a prompt looks, but the routing question is really whether the cheap model would have answered correctly. Labeling a sample by running both a cheap and an expensive model and checking whether they agree could give you training labels tied directly to the savings youβre after.
And the Sol background-training story is a great lesson in scoping agent permissions. Thanks for including the costs and the dead ends.
Thanks for the detailed feedback!
I like your point about measuring actual routing mistakes instead of just classification accuracy. So far, I've only tested how well the model predicts complexity, not whether those predictions would actually save money.
Testing real routing decisions with cheaper and more expensive models would definitely be an interesting next step.
Solid first fine-tuning experiment, and a practical idea.
Routing by estimated complexity is one of the more concrete ways to cut LLM costs without blindly downshifting everything to a weaker model. The score scale (0.00β1.00 with clear reference levels) is simple enough to actually use in a decision layer.
What stands out most is the honesty about the data side. Going from 50% β 86% with only 50 examples, then seeing accuracy drop when the dataset grew (and only recover to 76% with 2,500 examples) is a classic illustration that label consistency and evaluation design often matter more than raw volume β especially when the training data is itself AI-generated. The fact that all errors were off-by-one is also encouraging; the model seems to have learned a reasonable ordering even when it missed the exact bin.
The Sol incident is a useful cautionary tale for anyone giving agents access to paid training or inference endpoints. And shipping both the Hugging Face adapter and a small .NET API + playground makes the work immediately usable for others.
Nice work turning a Hacktoberfest side hustle into a real, open, cost-saving tool.
Thanks for taking the time to read it!
ComplexityβScorer prompt router built with Qwen3β8B and Tinker. Surprising result: 50 synthetic samples outβperformed 2500 samples. Covers budget accident, syntheticβdata pitfalls, openβsource release and production routing risks.
"I asked Sol to generate 2,500 training examples... and it started running model training in the background on DeepSeek and Qwen."Bro, you unlocked a core fear for every developer giving an AI agent API keys!
But honestly, what a wild ride. Turning an accidental budget-burn into a working $O(n^2)$ complexity scorer with a .NET backend is top-tier developer energy.Awesome read, and congrats on making the leap from .NET to fine-tuning open-weight models!
One thing Iβd be interested in seeing next is how the scorer behaves when the prompt distribution changes. A router trained on synthetic examples might learn the teacher modelβs idea of complexity rather than the actual difficulty a downstream model has solving the task.
For example, a short SQL migration request involving lock contention, backward-compatible schema changes, and a rolling deployment can be harder to handle correctly than a much longer prompt asking for boilerplate code. Iβd run a held-out evaluation grouped by task type, then measure the actual success rate of each routed model, not just whether the predicted complexity level matches the label. Testing on unseen categories would also help reveal whether the scorer learned a useful decision boundary or mostly memorized patterns from the generated dataset.
That seems like the real test for this project: whether the routing decisions hold up on production traffic, where prompt length and task difficulty donβt always line up.
That's an interesting point, especially testing on completely different types of tasks. I haven't tested how well the scorer handles unseen categories yet. That would definitely be worth exploring.
And I agree that predicting complexity is one thing, but proving that those predictions actually help with routing is another. For now, this was mainly my first fine-tuning experiment, but there's plenty of room to take it further.
The accuracy drop between fifty and twenty-five hundred examples is the classic synthetic data trap in router design. When scaling up synthetic generation from a teacher model, you are not just sampling from the task distribution. You are compounding the teacher's latent priors and subtle edge-case noise into the training set, which softens the decision boundary exactly where an eight-billion parameter model needs sharp separation.
The economic hazard for a model router is the asymmetry of the loss matrix. If the router routes an easy query to an expensive model, you waste half a cent. If it routes an ambiguous, high-complexity query to the eight-billion model because the decision boundary was blurred by synthetic noise, the user gets a failed task or a silent error that requires a full retry. At a seventy-six percent hit rate, the compounding retry latency and operator intervention costs erase the token savings of the cascade.
Good points, especially about the cost of incorrect routing decisions.
I agree that sending a complex task to a cheaper model could end up costing more if it fails and needs to be retried.
Maybe some of the AI-generated training examples weren't labeled correctly. But with only 50 evaluation examples, it's hard to say whether that's actually what caused the accuracy drop. One thing I'd clarify though: the 76% means the model correctly predicted the complexity level in 38 out of 50 cases. It doesn't tell us whether a cheaper model would actually succeed or fail at those tasks. Also, all incorrect predictions were just one level off.
For now, this was mainly my first experiment with fine-tuning, and I still have plenty to learn. π
Itβs a great realβworld reminder synthetic training data quantity is not everything; label consistency matters more. Also a fun warning about what can happen if you give agents direct access to your paid cloud credits.
Sounds interesting, next, try using Laya instead of Qwen, it's practically designed for this kinda task! (hint, I know it works, cuz I'm experimenting with it currently for my IDE).
Thanks for the tip! I hadn't heard of Laya before, but from what I'm seeing, it looks pretty similar to Jev. I'll definitely check it out.
Also there are other open-source alternatives too like GLiNER (which I used too) - works same as Laya & Jev.... I paired with Gemma 3 12B (not only because to hit that category, but that was the best my local hardware could support, and I really didn't look into the free credits, the event organizers were only offering....
Anyway, good luck on this!
Laya I've found is a really solid one, given that it's tiny (500m params), with the intention it can run on any hardware. Havent heard of GLiNER yet, but will definitely have a look, might be an interesting experiment. I generally just want to avoid Jev, because it's a black box and a paid API, which kinda beats the purpose of high performance decision making, it needs to be light enough to operate at the edge.
Thanks! I haven't heard of GLiNER either, so it looks like I have quite a few things to explore now.
Yes! But just to note something, I was specifically talking about model GLiNER 2.5 Decide (not general GLiNER).
Anyway, If you are tinkering with AI, going a bit into the direction of decision models can be fun! But just to note, Laya and GLiNER many times cannot be used exactly interchangably. GLiNER, smaller in size, but much slower and much higher quality and supports other functions too. It was specifically a big fit for my HF26 project...
Anyway, Good Luck on your project!
Really interesting experiment. Iβve been thinking more about token optimization and model routing lately, especially as AI gets integrated into more applications. As developers, we all want the smartest model available... until the API bill arrives π.
I like the idea of scoring prompt complexity first and only using the more capable model when the task actually needs it. The part about choosing Qwen3-8B partly because you could disable unnecessary reasoning also stood out to me β sometimes optimization is less about squeezing everything into fewer tokens and more about using the right amount of intelligence for the task.
And the accidental budget burn was a good reminder that giving agents access to paid services needs its own guardrails. Great write-up and experiment. ππΌ
Great ππ
Really enjoyed this writeup β especially the honesty about the 50-example model beating the 2,500-example one. One dimension I'd add to your cost analysis: the real metric for a router isn't accuracy, it's the expected cost of a misroute. Since all 12 errors were off by one level, the interesting question is how often a prompt routed to the cheaper model still succeeded anyway. In my own routing experiments, adding a confidence threshold that falls back to the stronger model when the scorer is uncertain moved the cost-per-task needle far more than chasing a few points of raw accuracy.
That's a really good point! I was mainly focused on classification accuracy, but measuring the actual cost of wrong predictions would definitely be more useful for a real router.
The confidence threshold sounds interesting too. How did you determine when your scorer was uncertain enough to fall back to a stronger model?
This is such a brilliant side hustle concept. Automatically routing prompts based on estimated complexity can save teams a fortune on LLM API costs.
Also, your takeaway on data quality vs. quantityβhow 50 clean examples outperformed 2,500 noisy AI-generated ones on your eval setβis a classic machine learning lesson that hits home for developers venturing into fine-tuning for the first time. Great write-up!
Really interesting project! The biggest takeaway for me is that more training data doesnβt automatically mean better results. The jump from 50% to 86% with just 50 examples, followed by 76% with 2,500, makes the quality and consistency of labels worth looking at more closely. I also like the practical use case for routing prompts to different models based on complexity. Would be interesting to see how the model performs on a much larger, unseen evaluation dataset!
Really interesting experiment! The biggest takeaway is that data quality and evaluation design can matter more than dataset size. I also love the idea of routing prompts by complexity to reduce AI costs. The next step of measuring actual task success and cost savings in production would make this even more compelling. And that unexpected Tinker bill is a great reminder to put spending limits on AI agents! π
Youβre really that famous! Will you even reply to me? By the way, I really liked this, although I didnβt understand much, but Iβd be happy to try to understand it.
Famous? Me? Definitely not! π€£
Thanks! I was actually trying to explain everything in simple terms, but I guess fine-tuning AI models isnβt the easiest topic to simplify. π
I am beginner soo.... π
@gramli Thanks for sharing the breakdown. That part about the synthetic data trap really hits home, especially regarding the asymmetry of the loss matrix.
For the router, I'd split the mistakes into underestimating and overestimating complexity. I'd worry more about sending a hard request to a model that can't handle it than paying slightly more for an easy one. I'd compare the final answer quality and total cost, including the scorer call, before choosing between the checkpoints.