DEV Community

DeadY2J
DeadY2J

Posted on

How I stopped an LLM from inventing facts in my daily YouTube Shorts

#ai #python #ffmpeg #opensource

I run a channel that publishes one Short every day, unattended. The first version let the LLM write the script from a topic. It worked until it didn't: it slipped in numbers that weren't in my notes and presented rumours as facts. On a channel about facts, that is the one thing you can't afford.

So I flipped the design. I write the verified facts once (with sources). The AI only turns them into a script, and the script has to pass guards written in plain code.

The guards

  1. Number whitelist. Every number in the script must appear in the facts (plus a few allowed ones, like a countdown). Anything else is rejected and the model retries.
  2. Repetition check. Repeated n-grams and repeated ideas across recent videos are rejected.
  3. Rumour labelling. Facts prefixed with RUMOR: must be presented as rumours, or the script is rejected.
  4. A second pass at temperature 0 that lists claims not supported by the facts.
  5. A facts-only fallback. If the model keeps failing, the video is built straight from the facts. It is less punchy, but it is always correct.

What a reader taught me

After I shared it, a reader pointed out the weak spot: guard #4 is another LLM call, so it is the only check that code doesn't verify, and the number whitelist can't see sentences without numbers (which are exactly the ones models invent most).

His suggestion, which I'm tracking as an issue: make the model cite, for every sentence, the index of the fact it comes from, and check in code that the sentence's numbers appear in that fact. The index says where to look; the numeric check proves the sentence agrees with it. Each guard reinforces the other.

Fallbacks everywhere

Same philosophy for the rest of the pipeline: voice (Edge TTS, then ElevenLabs, Cloudflare, offline system voice) and images (Cloudflare FLUX, NVIDIA, your own generator, archive, procedural gradient) each have a chain, so one service being down never stops the channel.

Try it / help

It is open source (MIT, Python + FFmpeg), runs with zero API keys, and I built it with a lot of AI assistance, so design feedback is very welcome.

Top comments (0)