What structure (grammar) and documentation (definitions) buys you – before any training
Our instinct is to pay up for bigger and newer models when an LLM's output is underwhelming but there are two quick fixes:
- Documentation which operates on how the model understands the task by injecting (at inference time) semantics that pre-training excluded because they belong to a niche domain. Documentation, in this experiment, is API-style documentation that unifies docs from leading HR & Payroll companies.
- Grammar which operates on the model’s output is a schema that the model cannot disobey. We compile our 20-field schema into a formal grammar to constrain decoding. At each step the sampler can only pick tokens the grammar allows, so the model doesn't emit malformed JSON or a value outside what the schema permits.
We ran the experiment on 264 payroll agreements spanning formal contracts through to informal email exchanges and meeting transcripts — offer letters, job postings, and staff handbooks in between. Each had an associated hand-verified gold answer.
Setup:
- Models: Qwen2.5-Instruct, run on llama.cpp
- Model sizes: 3B and 14B
- Quantization: Q8_0, quantized locally using llama.cpp's convert_hf_to_gguf from the published fp16 weights.
We ran each model four times — once for each combination of the two things we were testing: field names alone versus full definitions, and free decoding versus grammar-constrained decoding. That comes to 2,112 extractions in total.
Here’s what we found:
- Documentation substitutes for grammar. With definitions in place, the 14B model is accurate across 92% of fields and 31% of records.
- Grammar does not substitute for documentation. Grammar improves but can’t teach a convention.
- Grammar without documentation can hurt – it forces answers on questions the document never addresses. When a contract says nothing about a policy, the model must still pick a value — and it picks false 50 times where the right answer was null. Definitions removed the error entirely.
Field accuracy scores (at 92%) may make it seem like the models are near-perfect. But a record is only right when all twenty fields are — and less than a third of the records survive that test. The culprits have a name: bottlenecks. We’ll discuss these in the next post.
Top comments (0)