The hardest part of my AI quote checker was teaching it not to guess

Michael Choi — Thu, 17 Sep 2026 03:39:46 +0000

Two roofers quote the same roof. One proposal is shorter and cheaper. Does that make it better value—or just less specific?

I'm building ReadMyBid, a free beta for reviewing U.S. residential roofing quotes. The interesting engineering problem hasn't been making the model sound knowledgeable. It's making the output less confident when the document doesn't support a conclusion.

Missing is not excluded

If a quote doesn't mention flashing, that doesn't prove the contractor won't replace it. It also doesn't prove the existing flashing needs replacement. All we can honestly say is that the document doesn't state the scope clearly.

We separate included, unclear, not stated, and explicitly excluded scope. The useful next step is often a question for the contractor, not a red flag delivered with dramatic certainty. The roof already has enough potential leaks; the reasoning shouldn't add another.

A strict schema isn't a truth filter

A model can produce perfectly valid JSON containing an unsupported finding. Schema validation helps with predictable rendering, but it doesn't establish that a claim came from the quote.

For revised quotes, we moved price differences and scope-status transitions into server-side calculations. The model extracts structured information; ordinary code calculates the differences. That still depends on accurate extraction, but it removes an unnecessary layer of model arithmetic and interpretation.

First establish that the documents belong together

A comparison can sound convincing even when the files describe different contractors or jobs. We treat uncertain document matches conservatively. A contractor-name mismatch overrides a model's likely-match assessment, and unmatched pairs don't receive revision-improvement claims.

Name normalization has limits: removing a legal suffix isn't proof of identity. This is a guardrail, not an identity-verification system.

We removed a feature instead of adding another prompt

Initially, free-text lists of new exclusions and new risks could make restated terms sound newly introduced. Adding phrase filters helped individual examples but made the behavior increasingly brittle.

We stopped asking the model to generate those lists. The server now derives them from a fixed set of structured scope transitions. An included item becoming explicitly excluded can support a new-exclusion finding. A generic sentence about a written approval process possibly causing delays cannot.

The tradeoff is real: a contract-wide exclusion that doesn't map to a structured field may not appear as a new finding. For this beta, we'd rather miss a possible finding than confidently invent one. The UI shouldn't imply that the review is exhaustive.

Tests are not homeowner validation

We've used sample and synthetic documents to test validation, scope transitions, document mismatches, and report behavior. Those checks aren't evidence that real homeowners find the product useful. That's still what we're trying to learn.

The tool reviews written documents. It doesn't inspect a roof, establish local fair pricing, approve a contractor, or give legal advice.

If you've built document-review systems, I'd be interested in how you test the boundary between not stated and genuinely adverse. What failure cases would you put into the test set?

Disclosure: I built ReadMyBid. It's currently a free beta. Uploaded quotes are sent to OpenAI for analysis and aren't saved to our app's database; users should remove personal details first.