AI blog drafts fact-checked: my drafter invented a bug, 8 of 9 needed fixes

On 18 September I published a post called AI blog writing pipeline without made-up facts. Its central claim was that the model only phrases facts a script has gathered, so "it never had the chance to fabricate one", and that the whole system "fails toward silence, never toward fabrication".

Six days later I fact-checked every draft sitting in the review folder against the code, the logs and my own notes. The claim did not hold. This is what I found, and what I have changed.

The draft that invented a bug

The drafter had written a post about my podcast pipeline, which turns an article into a two-voice episode using Piper text-to-speech over the Wyoming protocol. My log recorded it as the first draft to pass a new set of checks. Those checks were about format: title length, description length, tags. It passed them all.

The content was mostly fiction. The draft said:

• the bug was endianness, fixed by flipping a > to a <;
• the text-to-speech model had a "reasoning field" that caused trouble;
• the fix took two weeks of debugging;
• synthesis ran at 90% real-time.

It also contained, in the body text, a sentence that began "wait, no. Let me be precise." The model had corrected itself mid-draft and the correction had been published into the post.

None of those four things happened. The real bugs were two, and both are in the code:

1. The Wyoming frame has three parts, not two. A header line can declare a data_length, and the event's data then follows as its own block of bytes after the newline, not inline in the header. My first reader went header, then payload, landed in the middle of a JSON object, and failed with Extra data: line 1 column 62. That looks like a broken server. It was a field I hadn't read.
2. The model writing the script returned nothing usable. Not the speech model: the language model writing the dialogue. Its answer came back with content empty and the whole token budget inside a thinking key. The pipeline logged a cheerful "0 turns" rather than an error. The code now raises when content is empty and thinking is populated, and it switches thinking off in the request body.

The speed figure was invented too. What I actually measured was 8.4 seconds of audio in 2.9 seconds of wall time, a real-time factor of about 0.35.

I rewrote the post from the code and published it the same day as Piper TTS over Wyoming: the frame format and empty model that broke my podcast. The fabricated original is kept, marked as such, so I can compare the two.

The irony is worth stating plainly. The invented bug was more conventional than the real one. My guess is that endianness is what a model reaches for when it knows a post is about binary framing and doesn't know the details. The true story, a missed field in a three-part frame, is the one other people will hit.

It wasn't the first time

Looking back, the warnings were already in my notes:

• 16 September: the topic radar picked up a public repo that turned out to have no code in it, only a README. The drafter wrote a whole build story from the one-line description. It was convincing enough that it was read as a real project. I renamed the draft so it couldn't be queued and added a guard that skips any repo with no commits.
• 20 September: a fix that let malformed drafts survive (the model was wrapping its front matter in a code fence) also let through a draft that invented a brownout threshold and a soldering step, neither of which was in the facts it was given.
• 21 September: on several runs the model emitted its own tool-call syntax instead of a post, because it was trying to call a tool inside a text-only request.

Each time I fixed the symptom in front of me. None of those fixes made a draft true.

Nine drafts, checked line by line

On 24 September the queue was empty and there were nine drafts waiting. I checked each claim against the thing it described. The result:

• 1 had a false core. A post about an Ollama deadlock credited the fix to a configuration setting that, on its own, did not hold. What actually fixed it was a small guard script. The draft also said I had pinned the model myself, which I hadn't, and it wrapped the whole thing in a debugging story that never happened.
• 8 needed fixes, from heavy to small.
• All 9 were missing a description in the front matter.

The errors were not random. They fell into a handful of types.

Invented process story. Drafts described alternatives I "tried" and rejected. One post about using Discord as the interface for my agents listed two other chat platforms I had supposedly evaluated. It also described a rate-limit error and a message-chunking fix that didn't exist, and said voice notes worked, which they don't. Models write the post they expect, and posts usually have a "what I tried first" section.

A confident wrong fix. The deadlock post above. This is the dangerous type, because a reader will copy the fix.

Stale state. Counts that were true once: the number of smart bulbs responding, the number of containers in the estate, the number of agent profiles. One draft still included a service I had decommissioned. Another described logging tables that had been dropped ten days earlier.

Mixed-run numbers. A post about a price-watching agent's sources combined figures from different runs into one table, called a source that had already been fixed broken, and described sources as shops.

Half-true configuration. A Docker post said log size limits were set. The config file did exist, but containers created before it weren't covered.

Claims that were never true. A post about my secrets manager said the automated identity had read-only access. It never did. That also turned out to be in a post already live since 6 September.

Security-posture detail. Some drafts described access arrangements more specifically than I want public. Those lines were cut rather than corrected.

Checking the drafts found two live bugs

The most useful outcome wasn't the corrected posts. To check whether "every agent is traced" was true, I had to look, and it wasn't: when the tracing service moved address, the main configuration was updated but twelve per-profile environment files still pointed at the old one. The specialist agents had been silently untraced.

Checking a claim about which model the agents use turned up another. A canary that tests the paid cloud model had reverted four profiles to a local model on 7 September, because its probe prompt came back redacted by a privacy filter. It only restores automatically after a credit problem, not after this kind of revert, so they sat on the local model for more than 17 days while its log said there was nothing to watch. Both were fixed the same day: the twelve files repointed, and the canary changed so it re-probes after any revert and moves back when a paid model passes.

A draft making a confident claim about my own systems is a cheap audit prompt. It is only useful if someone checks the claim.

What the 18 September post got wrong

Reading it again:

• "The model never went looking for a fact, so it never had the chance to fabricate one." It doesn't need to go looking. Given a gap, it fills it.
• The leak gate is real and it works, but it is structural. It catches an IP address or a credential shape. It cannot know that "two weeks of debugging" is false.
• "A ten-second read by the person whose name is on it can" catch a subtly overstated claim. The endianness bug reads fine. You only catch it by opening the code.
• Topics were "derived from things that demonstrably happened". The empty repo shows a topic can come from a description alone.

That live post about the secrets manager has been corrected in place, with a dated correction note saying what was wrong, rather than edited silently.

What I do now

• "Drafted OK" is not "true". Passing the format and leak checks means the file is well formed. It says nothing about the content.
• Every draft is checked against the code, logs or notes before it is queued, claim by claim. For numbers, a live read beats a note, and a note beats a code comment.
• Check the predictable categories first: any "I tried X first", any fix, any count, any statement about access or permissions.
• Correct published posts visibly, with a date.

The nine drafts have been fixed and queued one a day. The pipeline still saves me the typing. It does not save me the checking.

I wrote the full pattern up, with the validators, the canary that lied and a checklist, as a short field report: Keeping AI Agents Honest.

---

🤖 Drafted with AI assistance from my own homelab notes, logs and repos, then reviewed and edited before publishing.