Large language models are great at sounding right. In a diagnostic context, sounding right and being right are very different things, and the gap costs money.
I build software, and I also own equipment, so I kept running into the same wall: type a heavy-equipment fault code into a chatbot and it gives you a confident, fluent, often wrong answer. It invents code meanings. It makes up torque specs. It never tells you it's guessing. I wanted to see if I could build a diagnostic assistant that refuses to do that. Here's how it came together.
The core problem
An LLM doesn't retrieve a fault code's meaning, it predicts plausible text. For common OBD-II codes there's enough training data that it's usually fine. For heavy equipment, where codes are manufacturer-specific and the real information lives in service manuals, the model fills gaps with fiction and has no mechanism to flag that it's doing so.
The fix isn't a bigger model. It's retrieval grounding: force the model to look the answer up in a real dataset before it speaks, and make it cite the source. No source, no claim.
Step 1: the data
The foundation is a fault-code dataset — roughly 645,000 codes across 156 manufacturers, on-highway and off-highway. Cleaning and normalizing this was most of the work. Codes come in different formats (SAE J1939 SPN/FMI, OEM-specific strings), so the lookup has to handle several representations of "the same" code.
I put it in SQLite with FTS5 for full-text search, opened read-only in production. The search runs as a cascade: exact SPN/FMI match first, then normalized-code exact match, then code-prefix match, then a parsed-SPN attempt, and finally a full-text bm25 ranking as the fallback. Cheap exact matches first, fuzzy last.
Step 2: grounding the model
The key design decision: the model can't answer from memory. It's given a tool to query the fault-code database, and the system prompt forces retrieval before any diagnostic claim. The retrieved record is passed back as context, and the model's job is to explain it, not to invent it.
Two rules do most of the heavy lifting:
- Every substantive claim has to trace to a retrieved source.
- If retrieval returns nothing, the correct answer is "I don't have a verified source for this, check your manual" — not a plausible guess.
That second rule is the one that matters. A wrong answer with no source is worse than no answer, because it sends someone chasing the wrong repair.
Step 3: verification
Grounding reduces fabrication but doesn't eliminate it, so there's a second pass: a cheaper model acts as an LLM-as-judge, checking whether the generated answer is actually supported by the retrieved sources. It runs in log mode first so you can measure false-flag rates before you let it gate anything. Trust the loop only after the numbers say you can.
What I ended up with
A free fault-code lookup and a grounded AI troubleshooter for heavy equipment, live at heavyequipmentfix.com. No login, nothing for sale. The assistant cites its sources and is built to say "not sure" instead of bluffing.
The takeaway
Most "AI is unreliable" problems in narrow domains aren't model problems, they're grounding problems. If you constrain the model to real data, force citation, and verify the output, you can get something genuinely useful in a field where a confident lie is dangerous.
If you work on this stuff, I'd honestly rather you try to break it than believe me. Punch in a code you know cold and tell me if it gives you garbage — the tool's here.
Top comments (1)
good to read