General-purpose models know a lot about construction in the abstract. Ask one what a vapor barrier does or why snow load matters and you'll get a good answer. But a construction business runs on specifics: how long a manufacturer takes, what a permit requires, which installation step comes next on this building. That knowledge is local, it changes often, and almost none of it is on the public internet.
This post walks through a benchmark that measured that gap, and what it suggests for anyone building retrieval-augmented systems on top of company data.
The setup
- 387 questions, each with a single verifiable answer drawn from the company's own systems.
- 13 categories, covering specifications (wind, snow and foundation requirements), timelines (delivery and installation), rules (permits, engineering, financing and company policy), products and installation process.
- 6 models: Claude Opus 5.5, DeepSeek V4 Flash, Llama 4 Maverick, GPT-6.1 Sol, Mistral Large 3 and Gemini 2.5 Flash.
- Two conditions per question: the model alone, and the model with the relevant source pages retrieved into the prompt. Same pages for every model. Simple retrieval, no fine-tuning.
Scoring:
- Numeric answers are scored against a tolerance.
- Written answers are graded by a separate model that sees the reference answer and is blind to which model wrote the response.
Two models ran all 387 questions. Four ran a fixed 199-question subset with the same category mix, because of rate limits.
Results
Share correct on the 199 questions every model answered:
| Model | Alone | With retrieval |
|---|---|---|
| Claude Opus 5.5 | 25% | 96% |
| DeepSeek V4 Flash | 18% | 94% |
| Llama 4 Maverick | 14% | 93% |
| GPT-6.1 Sol | 25% | 92% |
| Mistral Large 3 | 11% | 85% |
| Gemini 2.5 Flash | 21% | 83% |
Averages: about 19% alone, about 90% with retrieval.
A few category-level numbers (averaged across models) show how uneven the "alone" baseline is:
| Category | Alone | With retrieval |
|---|---|---|
| Delivery timelines | 3% | 94% |
| Permits and engineering | 8% | 93% |
| Company policies | 18% | 91% |
| Structural specs (wind and snow) | 21% | 97% |
| Installation steps | 25% | 88% |
| Foundation specs | 38% | 96% |
| Financing rules | 12% | 78% |
Observations
1. Model choice mattered less than data access. Without retrieval, the gap between the best and worst model was about 14 points, and every model failed most questions. With retrieval, every model passed most questions, and a small open model outscored the strongest model without retrieval by almost four to one.
2. Models fill gaps with "typical company" behavior, and don't flag it. The clearest failures were questions where all six models agreed and were wrong. Asked whether change orders can be added after a customer signs a concrete quote, all six said yes, because that's normal in the industry. This company's policy is different. Nothing in the answers signaled that the models were guessing.
3. Retrieval can't be more consistent than the corpus. The misses that survived retrieval, financing rules in particular, mostly traced back to internal documents that disagreed with each other: two sets of payment terms, two different fees for the same thing. The models faithfully reproduced whichever page they were shown.
Takeaways if you're building RAG on company data
- Evaluate with and without retrieval. The "alone" score tells you how much your system depends on the corpus, and how badly it fails when retrieval misses.
- Grade blind. Use a separate grader model that sees the reference answer but not which model answered.
- Score numbers with a tolerance rather than exact match.
- Look for consensus-wrong answers. When every model agrees and is wrong, you've found a place where your business differs from the industry default.
- Audit the corpus for contradictions before blaming the model. In this benchmark, fixing the documents is the obvious next step before a rerun.
Limitations
Four of six models ran a 199-question subset. Some newer models weren't available and weren't tested. Written answers were graded by a model, which can be wrong.
The full write-up, including the category breakdown and methodology, was published by Metal America.
Top comments (0)