Most automated verification tools assume the repository is the source of truth. If the code says X, X is true. If the commit history documents Y, Y happened. It's a reasonable default — until it isn't.
Two failures I hit this year had the same structure: an automated pass reasoned confidently from the repo, produced a well-formed answer, and was wrong because the actual state lived somewhere the tool couldn't see.
Case 1: the newsletter that didn't exist
While correcting a published article, an automated pass flagged a line claiming there was an active manual newsletter workflow — the article described sending posts by hand to a specific email platform.
The checker read the article, searched the repo for workflow evidence, found nothing committed, and marked the claim unverifiable. That was the right call for the wrong reason. The repo had nothing because the newsletter platform had nothing.
One API call settled it: get the publication, list its posts. The response came back with zero posts and zero drafts. Not "posts I couldn't find in the repo" — zero posts on the platform itself. An active manual workflow is impossible when there is nothing to send. The article was wrong about the mechanism; the API told me the mechanism had never been used at all.
The repo couldn't have told me that. The repo can document what was committed; it cannot document the absence of activity on an external platform. Those are different things, and treating them the same is how the automated pass stayed confidently wrong for a while.
Case 2: the rejection count that kept changing
A different article mentioned four AdSense rejections. The number came from the repo's documented history: four entries, each timestamped.
An automated fact-check pass later changed the article to seven rejections, sourced from the author's own records kept outside the repo. A second automated pass changed it back to four, re-reading the repository.
The actual count is seven. Rejections five, six, and seven arrived as verbal notifications — emails read but never committed. The repo's four was accurate for what was committed; it understated the real count by three.
Two automated passes disagreed with each other not because of a bug but because they each read different sources, both internally consistent, with no signal indicating which was authoritative. The repo said four and was internally consistent. The external records said seven and were internally consistent. The only way to resolve the conflict was to check which source had the more complete view — and the answer was "neither alone."
The fix was mechanical: commit the three missing entries. The harder question is the design one underneath it: when is a repository entry complete, and when is it a documented subset of a larger set?
For this project, the honest answer is that commit history is the subset. Things happen that don't generate commits — API calls that fail silently and get noticed weeks later, verbal information that comes in during a meeting, external systems that change state without notifying the repo. A tool that only reads the repo will confidently "correct" true statements into false ones whenever the truth lives somewhere the repo cannot see.
The same pattern on a Raspberry Pi
The shelf scanner I run on a Raspberry Pi runs a scan every hour and writes the result to a JSON log on the Pi's local filesystem. That log then syncs to cloud storage.
Whether the scan actually ran is not in the GitHub repository. CI has a record of each firmware deploy — that's the repo's version of events. But a green deploy commit is evidence the software was updated, not that the scan executed. The freshness of the scan history lives in the Pi's own output, not in git.
When I built a check for stale scan history — "has the scanner produced a result in the last two hours?" — I had to read the Pi's actual output file rather than querying git log. The repo couldn't answer that question. The scan log could.
This is the same structure as the newsletter and rejection cases: the authoritative source for a fact depends on what the fact is, not on where the code that produces it lives.
What changed in practice
For the newsletter: the article was corrected to remove the manual-workflow claim. One API call, one correction, done.
For the rejections: three entries were committed to the repo. The article now reflects seven. The automated checker no longer contradicts itself across passes.
The broader fix was a distinction in the pipeline between two different failure modes:
- "Claim not found in repo" — this means: look externally before marking it false. The repo doesn't hold everything.
- "Claim contradicted by repo" — this is a real flag. If the repo's own evidence contradicts a claim, that needs investigation.
Treating these identically was the actual bug. Output inspection — reading what the pipeline actually produced — is different from process inspection, and this is a case where the output lives outside the repo entirely. The repo is a very good source of truth for code and deliberate commits. It's a poor source of truth for anything that was never committed.
Part of an ongoing 6-month experiment running three AI-curated directory sites. The technical claims here are real; this article was AI-assisted.
Top comments (1)
This matches a rule I learned the expensive way with my own monitoring scripts: any automated check that claims something about an external system has to read the external system at claim time, not a cached projection of it. I track a GPU fleet and an API account with cron-driven monitors, and a balance-delta approach (record the balance, subtract usage, report) was wrong so often I stopped trusting it — the provider's accounting had line items my local projection couldn't see. Now every report re-reads the live endpoint and "the script said so" counts as a hypothesis, never as evidence.
The AdSense case is the nastier variant precisely because both sources were internally consistent. When two consistent sources disagree, "which one has the more complete view" is the only tiebreaker that works, and your fix (commit the missing entries so the repo converges on reality) is the right direction — it makes the repo become the complete record instead of pretending it already was.
Did you end up encoding the authoritative-source mapping anywhere, or is it still implicit in each checker? I keep toying with declaring it per-claim (this field: git; that field: live API) so a future checker can't silently pick the wrong source — but it starts looking like a small trust database with its own maintenance cost, and I haven't convinced myself the overhead pays off.