The latest articles on DEV Community by DeadY2J (@deady2j). https://dev.to/deady2j https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4215577%2Fdb567b9d-4eff-4cd0-8dec-609a9f518a45.png https://dev.to/deady2j en DeadY2J Thu, 29 Oct 2026 15:09:50 +0000 https://dev.to/deady2j/how-i-stopped-an-llm-from-inventing-facts-in-my-daily-youtube-shorts-28cg https://dev.to/deady2j/how-i-stopped-an-llm-from-inventing-facts-in-my-daily-youtube-shorts-28cg <p>I run a channel that publishes one Short every day, unattended. The first version let the LLM write the script from a topic. It worked until it didn't: it slipped in numbers that weren't in my notes and presented rumours as facts. On a channel about facts, that is the one thing you can't afford.</p> <p>So I flipped the design. <strong>I write the verified facts once (with sources). The AI only turns them into a script, and the script has to pass guards written in plain code.</strong></p> <h2> The guards </h2> <ol> <li> <strong>Number whitelist.</strong> Every number in the script must appear in the facts (plus a few allowed ones, like a countdown). Anything else is rejected and the model retries.</li> <li> <strong>Repetition check.</strong> Repeated n-grams and repeated ideas across recent videos are rejected.</li> <li> <strong>Rumour labelling.</strong> Facts prefixed with <code>RUMOR:</code> must be presented as rumours, or the script is rejected.</li> <li> <strong>A second pass at temperature 0</strong> that lists claims not supported by the facts.</li> <li> <strong>A facts-only fallback.</strong> If the model keeps failing, the video is built straight from the facts. It is less punchy, but it is always correct.</li> </ol> <h2> What a reader taught me </h2> <p>After I shared it, a reader pointed out the weak spot: guard #4 is another LLM call, so it is the only check that code doesn't verify, and the number whitelist can't see sentences without numbers (which are exactly the ones models invent most).</p> <p>His suggestion, which I'm tracking as <a href="https://github.com/DeadY2J/factreel/issues/11" rel="noopener noreferrer">an issue</a>: make the model cite, for every sentence, the index of the fact it comes from, and check in code that the sentence's numbers appear in <em>that</em> fact. The index says where to look; the numeric check proves the sentence agrees with it. Each guard reinforces the other.</p> <h2> Fallbacks everywhere </h2> <p>Same philosophy for the rest of the pipeline: voice (Edge TTS, then ElevenLabs, Cloudflare, offline system voice) and images (Cloudflare FLUX, NVIDIA, your own generator, archive, procedural gradient) each have a chain, so one service being down never stops the channel.</p> <h2> Try it / help </h2> <p>It is open source (MIT, Python + FFmpeg), runs with zero API keys, and I built it with a lot of AI assistance, so design feedback is very welcome.</p> <ul> <li>Repo: <a href="https://github.com/DeadY2J/factreel" rel="noopener noreferrer">https://github.com/DeadY2J/factreel</a> </li> <li><code>pip install factreel</code></li> <li>Good first issues (new language packs, a JSON validator, local TTS): <a href="https://github.com/DeadY2J/factreel/issues" rel="noopener noreferrer">https://github.com/DeadY2J/factreel/issues</a> </li> </ul> python ai opensource ffmpeg