A scene
A customer of Fern & Thread, a small clothing shop in Pune, writes:
"how long will shipping take"
One policy, one fact: standard shipping takes 5-7 business days and costs ₹79. A 4B model can answer that for a fraction of a paisa.
A second customer writes:
"can I return my sale item and also change my delivery address?"
Now two policies interact, and sale items are final-sale. That needs a bigger model, or a careful one.
A third writes:
"my parcel arrived damaged and I want my money back"
That is a damage claim and a refund dispute. A human must handle it, and a model guessing here is worse than no answer.
Sloth Router sorts these three before anyone spends money.
The idea
A small open model (Qwen3.5-4B) sits in front and does one job: pick ANSWER (one policy, clearly covered), DEFER (several policies, or unsure: hand to a bigger model), or ABSTAIN (missing information, or a human must handle it). It is lazy on purpose, for the "Slow and Steady" theme.
Python, not the model, owns the safety default: a malformed output, a tie, or a low-confidence ANSWER becomes DEFER, never ANSWER.
The three rules
The router follows these rules, in order, written into its prompt:
- ABSTAIN if the request is off-topic, essential information is missing, or a human must handle it (refund disputes, damage claims, chargebacks, personal data). It never collects passwords, cards, Aadhaar or PAN.
- DEFER if the message asks about two or more policies, needs arithmetic across policies, or the model is unsure. This applies even when each policy is in the knowledge base.
- ANSWER only if the message asks about exactly one policy, a single knowledge-base fact supports the answer, and the request is clear.
If rules overlap, ABSTAIN beats DEFER, and DEFER beats ANSWER. The costly mistake is a confident ANSWER where a DEFER or ABSTAIN was right, so that is the number I treat as the headline: the dangerous error.
What I built
- A synthetic help desk for a fictional Pune shop, with a fixed knowledge base.
- An answer-first data generator and a frozen split: 3,000 train, 300 validation, 600 test.
- 12 unit tests for the labels and the threshold logic.
- A Tinker LoRA fine-tuning pipeline for Qwen3.5-4B.
Code: https://github.com/sssimransingh26-bit/Sloth-Router
How the data is built
Data is answer-first. Every ANSWER example starts from a knowledge-base sentence, and the question is written and styled afterwards, so the right answer is known before the question exists. DEFER examples combine two policies. ABSTAIN examples drop a needed detail, go off-topic, or demand a human.
Training uses polite, terse, typo-ridden and Hinglish phrasings. The test set uses email, gen-z, ALL CAPS and staccato styles, plus knowledge-base entries held out from training, so the model can't pass by memorising a style or an entry. Hashes of the frozen files are stored in the repo.
What I measured
The untuned 4B router on the 600 test items:
| Measure | Result |
|---|---|
| Route accuracy | 64.2% |
| Dangerous errors (answered when it should have deferred or abstained) | 4.2% (95% CI 2.8% to 6.1%) |
| Answer correct | 26.3% |
| Cost per 1,000 requests | $0.41 |
| Mean latency | 3.2 s |
These come from an earlier version of my prompt, before I removed a fake confidence number (see below). A 64% untuned baseline leaves real room for fine-tuning to help.
The fine-tuning run
A LoRA fine-tune of Qwen3.5-4B on Tinker. A first smoke test used 64 examples for three epochs: loss fell from 1.70 to 0.25 in the first epoch and reached about 0.02 by the third. That is mostly memorisation of 64 examples, so I treat it as a pipeline check only.
The real run used 400 examples for one epoch (50 steps), where every example is seen once:
| Step | Loss |
|---|---|
| 0 | 1.41 |
| 10 | 0.048 |
| 20 | 0.041 |
| 30 | 0.004 |
| 40 | 0.003 |
This shows the model learned the output format quickly. It does not show routing accuracy, because the loss averages over mostly easy format tokens. I have not scored the tuned model on the test set, so I make no claim that it beats the baseline.
What went wrong
- My first training target gave every ANSWER the same confidence of 0.91 and every DEFER 0.74. That number carried no information, so a threshold curve built on it would have been meaningless. I removed it. Real confidence has to come from the model's own probabilities.
- My training examples were first cut off at 728 tokens, which removed exactly the answer I was training on. Training had zero signal until I raised the limit.
- For hours the first Tinker training step would not return. It started working late on deadline day, so I ran out of time to train on the full dataset and score the tuned model.
Limits
All data is synthetic, from one fictional shop. There are no real users. Any saving would hold for this help desk only. The project calls Tinker, so it is not fully offline.
Why open
A small open-weight model can be fine-tuned, inspected and swapped without changing the rules around it. The safety logic lives in plain Python, not in the model.
Next
Score the tuned router against the baseline on the test set, add the always-big and keyword baselines, sweep the threshold on probability-based confidence, and build a small web page.
Top comments (0)