By Renchao Wu
My small routing model became much better at returning the expected fields after fine-tuning. It also became slightly worse at returning valid JSON.
On 576 held-out synthetic requests, strict all-field accuracy rose from 7.6% to 47.2%. JSON validity fell from 98.6% to 95.5%. The same adapter that improved the headline score introduced more responses a strict parser could not read.
I published the MLX adapter and reproduction package, the synthetic dataset, and a Kaggle notebook for checking the frozen outputs. The useful result is the tradeoff, together with the evidence needed to inspect it.
The task: turn a request into a small routing decision
The experiment uses Qwen2.5-0.5B-Instruct to handle English, Chinese, and mixed-language requests. The output contract covers notes, scheduling, calculations, requests needing clarification, and unsupported requests.
A response must satisfy several checks. It must parse as strict JSON, match the permitted schema, choose the correct route, and contain exactly the expected field values. Passing one check does not imply passing the next. A parser can accept an invented route; a schema validator can accept a correctly typed but wrong value.
The schema is present in the prompt for both the base model and the adapter. Neither arm gets credit for guessing a hidden interface.
This is a personal synthetic experiment. Its templates share vocabulary across splits, and their language variants were written by one author. It does not measure open-domain assistant quality or production routing reliability.
A family split was not enough
My first version split template families before expanding language variants. That avoided putting a direct translation of the same example in both training and test, but it left another problem: related families could still describe almost the same situation.
A complete scheduling request and its missing-time sibling belong together. Different family IDs do not make them independent scenarios.
For v2, I bound 77 families into 23 explicit scenario clusters, including paraphrases and complete/missing-information siblings. I split those clusters before expanding English, Chinese, and mixed variants. The resulting splits contain 1,026 training rows, 291 validation rows, and 576 test rows. The test covers 23 families in seven held-out clusters.
Those clusters are a documented judgment, not proof of semantic independence. I froze the manifest before the v2 run and kept v1 as a separate history. The v1 and v2 test scores are not comparable. The split changes and audit are recorded in the public review.
One consequence mattered a lot: the entire percent_of scenario landed in test. The training set never taught the model that operation. Calculation results therefore include an out-of-training probe, rather than an ordinary sample of familiar calculations.
The result depends on what gets counted
The frozen v2 test produced these rounded percentages:
| Check | Base model | LoRA adapter |
|---|---|---|
| Strict JSON | 98.6% | 95.5% |
| Schema-valid | 40.6% | 95.0% |
| Correct route | 35.6% | 72.2% |
| Strict all-field match | 7.6% | 47.2% |
| Cross-language equal and correct | 2.6% | 43.1% |
The base model usually produced parseable JSON, but often used fields or routes outside the contract. The adapter learned much more of the interface and selected the right route more often. That gain coexisted with a regression in parseability.
Strict all-field match requires schema-valid, type-checked output and exact expected values. The scorer rejects duplicate keys, nonfinite values, Markdown fences, and extra text. It also rejects Python's tempting shortcuts: true does not count as integer 1, and 3.0 does not satisfy an integer-only field.
The frozen scorer keeps operand order as part of the gold contract. A separate commutative_ok metric exists for diagnosis; it does not silently replace the headline score. In this v2 run, it did not rescue any swap-only failures.
Validation strict accuracy was almost flat: 25.4% for the base and 25.8% for the adapter. The large test gain was concentrated in notes and scheduling. Treating the 47.2% aggregate as a general improvement would hide that unevenness.
The failures changed the next experiment
The adapter's strict calculation score was zero. For held-out percentage requests, it produced familiar operations such as div or add. It had learned useful routing patterns without acquiring the missing operation through this training set.
Clarification and rejection were also weak. Their route accuracies were about 24% and 47%. An underspecified request was still often treated as an action to execute. In a real system, that behavior would matter even if an aggregate score looked encouraging.
I initially described all 26 invalid adapter outputs as coming from the unseen operation. A later question prompted a recount of the original frozen predictions. That explanation was too broad. The corrected breakdown is:
| Gold task | Invalid JSON outputs |
|---|---|
Calculation using percent_of
|
13 |
| Percentage request missing a number; clarification required | 10 |
| Underspecified note request; clarification required | 3 |
The final group included repeated keys or truncation. No other test families produced invalid JSON. The language counts were eight English, nine Chinese, and nine mixed.
That correction matters to diagnosis. Adding training examples for an unseen operation would not, by itself, explain the underspecified-note failures. I would test missing-information handling separately and examine the repeated-key outputs before deciding what to change.
The corrected repository-root review is the source for this breakdown. The existing experiment.zip and Kaggle frozen package still contain the earlier review text. Their outputs, weights, and aggregate scores have not changed. This was a recount, not another training run.
What actually ran
Training ran locally on Apple silicon with MLX. The published adapter is in MLX-LM format; it is neither a standalone model nor a PEFT adapter.
The base is Qwen/Qwen2.5-0.5B-Instruct, fixed at revision 7ae557604adf67be50417f59c2c2f167def9a775. The run used LoRA rank 8, scale 20, 16 layers, learning rate 1e-4, batch size 1, and 1,500 iterations. The seed was 42. I used the final checkpoint as declared, rather than selecting it against the test results.
Evaluation loaded the saved adapter in fresh processes. Both arms used greedy decoding with the same prompt and a 96-token output limit. A second training run with the same seed produced a bit-identical adapter; evaluation was not repeated. That checks repeatability under this setup, not robustness across seeds or environments.
There are only seven test clusters. The published family-bootstrap intervals for strict accuracy are wide: roughly 1–16% for the base and 28–67% for the adapter. The 576 rows include language triples and related examples; they are not 576 independent real-world situations.
Two different ways to reproduce it
If you want to inspect scoring without running a model, use the Kaggle frozen-output notebook. It loads saved predictions, checks the frozen artifacts, and recalculates scores on CPU. Its successful cloud run is evidence for that scoring path. It does not establish cloud training or inference performance.
To rerun training on compatible Apple silicon, follow the model repository instructions. After downloading the repository and extracting its experiment.zip at the repository root:
unzip experiment.zip
cd reproduction
bash scripts/run_all.sh
The script installs pinned dependencies, downloads the fixed base revision, regenerates data, checks the scorer, trains, reloads the adapter, and evaluates both arms. New run outputs remain separate from the published frozen outputs. Runtime and memory will depend on the machine.
Code, data, and adapter are published under Apache-2.0 with license and notice files. The base weights are not redistributed. Read the corrected root review alongside the historical package before interpreting failure attribution.
What I would change next
I would keep this test frozen and build a separate experiment around the weaknesses it exposed: missing-information requests, unsupported requests, and operations absent from training. I would also compare ordinary decoding with a constrained-output path on the same requests, reporting format compliance and field correctness separately.
Those experiments have not run. They would answer different questions from this release.
For this run, the adapter was better at the declared routing contract overall, still failed every strict held-out calculation, and made valid JSON less dependable. I would want all three facts in front of me before deciding whether to put it behind an API.
Top comments (0)