Fine-Tuning with QLoRA for JSON Extraction

#ai #llm #machinelearning

We needed to extract structured JSON from unstructured text for a domain-specific use case. We decided to train the Qwen2.5-1.5B-Instruct model with quantization levels of 4-bit, 8-bit and 16-bit separately.

Dataset
Total samples: ~280

Training
Fine-tuned using QLoRA

Evaluation Metrics

We measured two scores per split:

Exact Match: Binary per sample (1 if all fields match exactly, else 0). Aggregated across the split.
Field Match: Partial credit per sample (correct fields / total fields). Aggregated across the split.

Example:

Gold JSON

{
  "first_name": "Jordan",
  "last_name": "Kaur",
  "employment_type": "contractor",
  "employer_name": "Grounded Movement",
  "employer_email": "payroll@goat.com",
  "employer_address_line_1": "41 King Street",
  "employer_address_line_2": "Madison, UK",
  "employer_registration_number": null,
  "employer_tax_number": null,
  "currency": "USD",
  "pay_rate": 45.0,
  "pay_rate_basis": "per_shift",
  "incentive_pay": 5.0,
  "incentive_type": "per_reservation_over_n",
  "n": 15,
  "reservation_types": [
    "check_ins"
  ],
  "exclude_staff_reservations": true,
  "exclude_cancelled_shifts": true,
  "max_pay": null,
  "payment_frequency": "weekly"
}

Predicted JSON (model output)

{
  "first_name": "Jordan",
  "last_name": "Kaur",
  "employment_type": "contractor",
>>"employer_name": "Grounded",<<
  "employer_email": "payroll@goat.com",
  "employer_address_line_1": "41 King Street",
  "employer_address_line_2": "Madison, UK",
  "employer_registration_number": null,
  "employer_tax_number": null,
  "currency": "USD",
  "pay_rate": 45.0,
>>"pay_rate_basis": null,<<
  "incentive_pay": 5.0,
  "incentive_type": "per_reservation_over_n",
  "n": 15,
  "reservation_types": [
    "check_ins"
  ],
  "exclude_staff_reservations": true,
  "exclude_cancelled_shifts": true,
  "max_pay": null,
  "payment_frequency": "weekly"
}

Mismatch 1: "employer_name": "Grounded" (Predicted) vs "Grounded Movement" (Gold)
Mismatch 2: "pay_rate_basis": null (Predicted) vs "per_shift" (Gold)

Total Fields = 20
Correct Fields = 18
Wrong Fields = 2

Exact Match
Score: 0/1

Field Level Match
Score: 18/20 = 0.9  (90%)

Results (for 16-bit)

While we experimented with 4-bit, 8-bit and 16-bit QLoRA, the lower-bit experiments are omitted here for brevity.

-------------------------------------------------------------------
Model                    Exact Accuracy    Field Match
-------------------------------------------------------------------
Baseline                 0.00              0.54
Fine‑tuned               0.62              0.97