The first experiment gave clear results: Spell things out and the model performs well. Then we made payroll agreements more complicated – not to be annoying, but to reflect the nature of payroll agreements for shift work.
The clean result did not survive.
We ran Qwen2.5-1.5B-Instruct in GGUF format across a wide range of quantization levels, from highly compressed Q2 to near-full-precision FP16, on 2500 payroll clauses that varied in levels of complexity. Complexity here refers to the number of different class types worked at different rates. The agreements varied structurally from:
- abbreviated text (S1) to
- semi-structured language (S2) to
- full, grammatically correct sentences (S3).
F1 scores are displayed in the heatmap and show:
Pay agreement complexity is the primary driver of extraction accuracy — the more class types and rate structures a worker has, the harder the model works.
For simple agreements — a salaried worker, a single flat rate — semi-structured text is sufficient. Full prose adds no value and marginally reduces accuracy.
For complex agreements — the shift-based arrangements that define hospitality, fitness, and care work — full grammatical sentences outperform structured shorthand by nearly 21 percentage points. This is where off-the-shelf models struggle most, and where a fine-tuned model delivers the clearest commercial advantage.
Does a bigger model solve the problem?
We are building toward a fine-tuned model for payroll rule extraction.
We need to know 2 things before we get there:
- Performance ceiling of an off-the-shelf model
- the best accuracy-to-compute tradeoff for production environment
So we ran the same experiment across four quantization levels of the same base model: 2-bit, 4-bit, 8-bit, and 16-bit.
The results were clear:
- The 2-bit model is not viable.
- The 4-bit model underperforms on complex structures, which makes it inadequate for shift work.
- The 8-bit and 16-bit models are statistically indistinguishable at every complexity level.
8-bit currently offers the best tradeoff we have measured. But this experiment tested one model family. The next step is running the same experiment across a selection of model architectures to understand if different base models yield different results.
They likely do - and that comparison will inform which model we fine-tune.
Top comments (0)