When a language model reads a transcript and returns a list of tools, two kinds of mistakes appear. The first is invention: a tool that was never mentioned. The second is subtler: a real tool paired with the wrong link or the wrong meaning. Both erode trust. This post lists the techniques that helped us reduce them, and the numbers we saw, with the caveat that our samples are small.
Mistake 1: tools that were never said
The cure is to make the model show its evidence and to check it with code.
- Require a verbatim quote for every extracted item
- Check that the tool's name appears in the transcript, and drop the item if it does not
- Check the quote and the timestamp against the transcript too, and remove whichever cannot be found, so nothing unverified is shown as evidence
A caution: a naive check of the form "the tool name must appear in the transcript" can reject real tools when captions split or respell them (a product written as FastAPI may appear as "fast API"). Match names with some tolerance, and log rejections so you can see whether you are throwing away good results.
Mistake 2: real tools, wrong meaning
This one surprised us. In one test run on twenty technical videos, a large share of the official links our resolver returned were wrong. A database called Neon led to a cinema studio. An assistant called Serena led to a tennis player's website. Codex led to a food-safety standard. The model had named the tools correctly; the failure was in resolving ambiguous names with a web search.
What fixed most of it:
- Add context to the query: include the category and a short description of the tool, not just its name
- Accept canonical repositories: many CLI and open-source tools have no website, only a GitHub repo
- Block unreliable sources as official: aggregators, course platforms, forums, social networks
- Score and threshold: compute a confidence for each match, and reject anything below a threshold
- Return nothing when unsure: an empty field is honest, a wrong link is not
Improve the prompt, then measure
Prompts matter. We rewrote ours with an explicit definition of what counts as a tool, a list of exclusions and an instruction to quote verbatim. On four videos of very different lengths, from 47 minutes to over 7 hours, and labelling the results by hand, precision moved from about 89 to about 96 percent and recall from about 76 to about 93 percent. Those are indicative figures: four videos, one run each, a model with some randomness. Treat them as direction, not proof.
The point is the method: label a small set by hand once, then re-run it after every prompt change, so you notice regressions.
Don't hide partial results
A different kind of hallucination is silent partial coverage. If the transcript is too long and you truncate it, the output looks complete but only reflects the beginning. We hit this: a seven-hour video was analysed on its first twelve minutes. The fix was to process the whole transcript in chunks and to report coverage explicitly, with a warning when an analysis is partial.
A short checklist
- Name found in the transcript, quote required, quote verified
- Context in link resolution
- Confidence threshold, with
Noneas a valid answer - A small labelled set to catch regressions
- Honest coverage reporting
Where this led
We applied these rules in VidScope, which lists the tools mentioned in a YouTube video with timestamps and official links, and prefers no link to an uncertain one.
FAQ
Can hallucinations be eliminated?
No, but requiring and verifying evidence reduces them a lot.
Why return nothing instead of a best guess?
A wrong link looks authoritative; an empty field tells the user to check.
How do I measure accuracy on a small project?
Label a few videos by hand and re-run them after every change, while remembering the sample is small.
Top comments (0)