In our previous blog, we described how we successfully fine-tuned Qwen2.5-1.5B-Instruct using QLoRA to extract structured JSON from unstructured text for a domain-specific use case. The results were promising, with the model achieving 62% exact match accuracy and 97% field match accuracy.
However, a natural question arose: How does our fine-tuned small model compare against state-of-the-art frontier models like Claude Opus 5?
Would a massive general-purpose model outperform our domain-specific fine-tuned model?
We decided to run the same evaluation pipeline on Claude Opus-5 to find out.
Evaluation Setup
We used the identical test set and the same evaluation metrics to ensure a fair comparison:
- Exact Match: Binary per sample (1 if all fields match exactly, else 0). Aggregated across the test dataset.
- Field Match: Partial credit per sample (correct fields / total fields). Aggregated across the test dataset.
Results
Despite being ~1,000x smaller than Claude Opus-5, our fine-tuned Qwen2.5-1.5B achieved:
~15 percentage points higher field match accuracy (96.72% vs 81.81%)
62% exact match accuracy vs Claude's 0%
Much less expensive
Much more control over your data
Conclusion
This comparison demonstrates a powerful lesson in modern AI development,
a well-tuned small model can outperform even the largest frontier models on domain-specific use-cases.
Top comments (0)