DEV Community: Jebra

Fine-Tuned Qwen2.5-1.5B vs Claude-Opus-5 for JSON Extraction

Jebra — Mon, 24 Aug 2026 16:09:54 +0000

In our previous blog, we described how we successfully fine-tuned Qwen2.5-1.5B-Instruct using QLoRA to extract structured JSON from unstructured text for a domain-specific use case. The results were promising, with the model achieving 62% exact match accuracy and 97% field match accuracy.

However, a natural question arose: How does our fine-tuned small model compare against state-of-the-art frontier models like Claude Opus 5?
Would a massive general-purpose model outperform our domain-specific fine-tuned model?

We decided to run the same evaluation pipeline on Claude Opus-5 to find out.

Evaluation Setup

We used the identical test set and the same evaluation metrics to ensure a fair comparison:

Exact Match: Binary per sample (1 if all fields match exactly, else 0). Aggregated across the test dataset.
Field Match: Partial credit per sample (correct fields / total fields). Aggregated across the test dataset.

Results

Despite being ~1,000x smaller than Claude Opus-5, our fine-tuned Qwen2.5-1.5B achieved:

~15 percentage points higher field match accuracy (96.72% vs 81.81%)
62% exact match accuracy vs Claude's 0%
Much less expensive
Much more control over your data

Conclusion

This comparison demonstrates a powerful lesson in modern AI development,
a well-tuned small model can outperform even the largest frontier models on domain-specific use-cases.

Fine-Tuning with QLoRA for JSON Extraction

Jebra — Mon, 10 Aug 2026 19:51:09 +0000

We needed to extract structured JSON from unstructured text for a domain-specific use case. We decided to train the Qwen2.5-1.5B-Instruct model with quantization levels of 4-bit, 8-bit and 16-bit separately.

Dataset
Total samples: ~280

Training
Fine-tuned using QLoRA

Evaluation Metrics

We measured two scores per split:

Exact Match: Binary per sample (1 if all fields match exactly, else 0). Aggregated across the split.
Field Match: Partial credit per sample (correct fields / total fields). Aggregated across the split.

Example:

Gold JSON

{
  "first_name": "Jordan",
  "last_name": "Kaur",
  "employment_type": "contractor",
  "employer_name": "Grounded Movement",
  "employer_email": "payroll@goat.com",
  "employer_address_line_1": "41 King Street",
  "employer_address_line_2": "Madison, UK",
  "employer_registration_number": null,
  "employer_tax_number": null,
  "currency": "USD",
  "pay_rate": 45.0,
  "pay_rate_basis": "per_shift",
  "incentive_pay": 5.0,
  "incentive_type": "per_reservation_over_n",
  "n": 15,
  "reservation_types": [
    "check_ins"
  ],
  "exclude_staff_reservations": true,
  "exclude_cancelled_shifts": true,
  "max_pay": null,
  "payment_frequency": "weekly"
}

Predicted JSON (model output)

{
  "first_name": "Jordan",
  "last_name": "Kaur",
  "employment_type": "contractor",
>>"employer_name": "Grounded",<<
  "employer_email": "payroll@goat.com",
  "employer_address_line_1": "41 King Street",
  "employer_address_line_2": "Madison, UK",
  "employer_registration_number": null,
  "employer_tax_number": null,
  "currency": "USD",
  "pay_rate": 45.0,
>>"pay_rate_basis": null,<<
  "incentive_pay": 5.0,
  "incentive_type": "per_reservation_over_n",
  "n": 15,
  "reservation_types": [
    "check_ins"
  ],
  "exclude_staff_reservations": true,
  "exclude_cancelled_shifts": true,
  "max_pay": null,
  "payment_frequency": "weekly"
}

Mismatch 1: "employer_name": "Grounded" (Predicted) vs "Grounded Movement" (Gold)
Mismatch 2: "pay_rate_basis": null (Predicted) vs "per_shift" (Gold)

Total Fields = 20
Correct Fields = 18
Wrong Fields = 2

Exact Match
Score: 0/1

Field Level Match
Score: 18/20 = 0.9  (90%)

Results (for 16-bit)

While we experimented with 4-bit, 8-bit and 16-bit QLoRA, the lower-bit experiments are omitted here for brevity.

-------------------------------------------------------------------
Model                    Exact Accuracy    Field Match
-------------------------------------------------------------------
Baseline                 0.00              0.54
Fine‑tuned               0.62              0.97

Observations

Field match improved from ~54% to ~97% after fine-tuning.
Exact match, while lower (62%), improved from 0%.

The field-level result indicates that the adapter successfully learned the correct key-to-value mapping for the majority of fields.

Dicts and Docs: The Value of Grammar and Documentation for LLM-Based Automation

Jebra — Thu, 16 Jul 2026 14:06:03 +0000

What structure (grammar) and documentation (definitions) buys you – before any training

Our instinct is to pay up for bigger and newer models when an LLM's output is underwhelming but there are two quick fixes:

Documentation which operates on how the model understands the task by injecting (at inference time) semantics that pre-training excluded because they belong to a niche domain. Documentation, in this experiment, is API-style documentation that unifies docs from leading HR & Payroll companies.
Grammar which operates on the model’s output is a schema that the model cannot disobey. We compile our 20-field schema into a formal grammar to constrain decoding. At each step the sampler can only pick tokens the grammar allows, so the model doesn't emit malformed JSON or a value outside what the schema permits.

We ran the experiment on 264 payroll agreements spanning formal contracts through to informal email exchanges and meeting transcripts — offer letters, job postings, and staff handbooks in between. Each had an associated hand-verified gold answer.

Setup:

Models: Qwen2.5-Instruct, run on llama.cpp
Model sizes: 3B and 14B
Quantization: Q8_0, quantized locally using llama.cpp's convert_hf_to_gguf from the published fp16 weights.

We ran each model four times — once for each combination of the two things we were testing: field names alone versus full definitions, and free decoding versus grammar-constrained decoding. That comes to 2,112 extractions in total.

Here’s what we found:

Documentation substitutes for grammar. With definitions in place, the 14B model is accurate across 92% of fields and 31% of records.
Grammar does not substitute for documentation. Grammar improves but can’t teach a convention.
Grammar without documentation can hurt – it forces answers on questions the document never addresses. When a contract says nothing about a policy, the model must still pick a value — and it picks false 50 times where the right answer was null. Definitions removed the error entirely.

Field accuracy scores (at 92%) may make it seem like the models are near-perfect. But a record is only right when all twenty fields are — and less than a third of the records survive that test. The culprits have a name: bottlenecks. We’ll discuss these in the next post.

When payroll gets complicated, the machine gets harder to please

Jebra — Thu, 02 Jul 2026 15:30:55 +0000

The first experiment gave clear results: Spell things out and the model performs well. Then we made payroll agreements more complicated – not to be annoying, but to reflect the nature of payroll agreements for shift work.

The clean result did not survive.

We ran Qwen2.5-1.5B-Instruct in GGUF format across a wide range of quantization levels, from highly compressed Q2 to near-full-precision FP16, on 2500 payroll clauses that varied in levels of complexity. Complexity here refers to the number of different class types worked at different rates. The agreements varied structurally from:

abbreviated text (S1) to
semi-structured language (S2) to
full, grammatically correct sentences (S3).

F1 scores are displayed in the heatmap and show:

Pay agreement complexity is the primary driver of extraction accuracy — the more class types and rate structures a worker has, the harder the model works.
For simple agreements — a salaried worker, a single flat rate — semi-structured text is sufficient. Full prose adds no value and marginally reduces accuracy.
For complex agreements — the shift-based arrangements that define hospitality, fitness, and care work — full grammatical sentences outperform structured shorthand by nearly 21 percentage points. This is where off-the-shelf models struggle most, and where a fine-tuned model delivers the clearest commercial advantage.

Does a bigger model solve the problem?

We are building toward a fine-tuned model for payroll rule extraction.
We need to know 2 things before we get there:

Performance ceiling of an off-the-shelf model
the best accuracy-to-compute tradeoff for production environment

So we ran the same experiment across four quantization levels of the same base model: 2-bit, 4-bit, 8-bit, and 16-bit.

The results were clear:

The 2-bit model is not viable.
The 4-bit model underperforms on complex structures, which makes it inadequate for shift work.
The 8-bit and 16-bit models are statistically indistinguishable at every complexity level.

8-bit currently offers the best tradeoff we have measured. But this experiment tested one model family. The next step is running the same experiment across a selection of model architectures to understand if different base models yield different results.

They likely do - and that comparison will inform which model we fine-tune.

Write for the machine

Jebra — Fri, 19 Jun 2026 14:49:30 +0000

Write for the machine, not the expert

The effect of abbreviation, syntax, redundancy, and filler words on structured information extraction

Does how we write matter when feeding unstructured text to an LLM? Experiment 1 isolates this question using a deliberately simple test case: a single flat payroll agreement containing 11 fields.

Experimental Setup

• Dataset: 1,000 rows of text total (the same payroll agreement written in 5 distinct styles across 200 unique workers).
• Model: Qwen2.5-1.5B-Instruct in GGUF format (tested across a wide range of quantizations from Q2 to FP16 in a CPU-only environment).
• Inference Pipeline: To prevent malformed JSON, we utilized grammar-constrained decoding via LlamaGrammar.from_json_schema(), restricting token generation to the target structure.
• Metric: Accuracy was measured using field-level extraction quality:

Accuracy = True Positives / Total Unique Fields

Here is the exact JSON used as the target baseline for a test case:

{
            "first_name": "Layla",
            "last_name": "Silva",
            "worker_type": "contractor",
            "class_name": "Boxing",
            "pay_type": "per_class",
            "class_rate": 40.0,
            "bonus_amount": 4.0,
            "bonus_threshold": 8,
            "head_count_basis": "check_ins_late_cancels",
            "holiday_rate_multiplier": 1.5,
            "cap_per_session": 100.0
 }

S1
Label
Shorthand
Description
Abbreviations, symbols, no verbs
Example
Layla Silva [C]: Boxing $40/class. $4/head >8 att (CI+LC). Holiday 1.5x. Cap $100/session.

S2
Label
Spelled-out
Description
Abbreviations replaced with full words, colons as separators
Example
Layla Silva [contractor]. Rate: $40/class — Boxing. $4/head when attendance exceeds 8 (check-ins and late cancels). Holiday rate: 1.5x standard. Session cap: $100.

S3
Label
Explicit prose
Description
Full sentences, one clean statement per field
Example
Layla Silva operates as an independent contractor. She is paid $40 for each Boxing class she teaches. For every attendee above 8, she receives $4 extra, counting check-ins and late cancels. Public holidays attract a 1.5x multiplier on her $40 base rate. Session earnings are capped at $100.

S4
Label
Explicit redundant prose
Description
Same as S3 but key values repeated across sentences
Example
Layla Silva is hired as a contractor. Her Boxing rate is $40 per class. Each Boxing class she teaches pays $40. For Boxing classes, a per-head bonus of $4 applies for each attendee above 8, counting check-ins and late cancels. The $4 per-head threshold is 8. The Boxing rate of $40 increases to $60.00 on holidays (1.5x the $40 base). Per-session Boxing earnings are limited to $100 in total.

S5
Label
Explicit prose with filler
Description
Same as S3 but legal boilerplate text introduced.
Example
Layla Silva is contracted as a self-employed instructor. Her contractual rate for Boxing is $40 per class. For the purposes of this arrangement, she is entitled to a per-head bonus of $4 for each attendee above 8, counting check-ins and late cancels. As per the agreed terms, work on designated public holidays is paid at 1.5 times the standard rate. Total per-session earnings are capped at $100.

Key Findings

The results highlight a distinct hierarchy in how language style impacts extraction accuracy:

• Spelling out words is paramount: Replacing cryptic abbreviations with full words (S1 to S2) yielded the largest performance jump, nearly 11 percentage points.
• Syntax matters less than semantics: Moving from spelled-out shorthand to full grammatical sentences (S2 to S3) made no meaningful difference. If the vocabulary is explicit, the model extracts efficiently without prose.
• Redundancy degrades accuracy: Repeating key values across sentences (S4) underperforms. Instead of providing helpful "anchors," duplication introduces token ambiguity that misleads the model.
• Filler text introduces noise: Adding filler text like “subject to the terms of the agreement” also had a negative impact.