After publishing Two times an external API overruled my own repository, a reader left a question I couldn't stop thinking about. @raknaos asked whether I had encoded the authoritative-source mapping anywhere, or if it was still implicit in each checker — and noted that declaring it per claim starts to look like a small trust database with its own maintenance cost.
Short answer: it's still implicit. Longer answer: the failure modes that motivated the original article aren't isolated. I found two more cases in the same pipeline with the same shape. The per-claim approach is what I should have built; this article is about why, and what it would look like.
The thesis in one sentence: when multiple data sources can each describe the same claim, and an automated checker must pick one, the choosing must be declared in configuration — not inherited as a convention from which source happens to be easiest to query.
The class of failure: confident wrong corrections
Both failures from the previous article had this structure: an automated checker read a claim, found evidence from its primary source, and updated the claim to match. The update looked correct from the checker's perspective. From the article's perspective, it introduced an error.
This isn't a quality-gate problem in the usual sense. The quality contract v2 frontmatter I added to every article includes verified_at — a date for when claims were checked. That field tells me a check ran; it doesn't tell me which source was authoritative. Two auto-correction passes using different sources can each attach a valid verified_at to contradicting facts, and both timestamps look legitimate.
The problem is not "did the checker run?" It's "which source did the checker consult, and was that the right one for this particular claim?" A green verified_at doesn't answer that.
Two cases from my pipeline
Case 1: Rejection count. A published article described the project's rejection history. The repo had four events committed. The author's own notes showed seven — three rejections arrived as verbal notifications and were never committed. A checker reading git saw four as the ground truth and updated any article showing seven. A different pass reading another source saw seven and corrected it back. Each pass was confident. Each correction was wrong half the time.
The claim "how many rejections did this project receive?" has one correct answer. The problem: two sources (git commits; author notes) gave different answers, and each looked authoritative to whichever checker happened to query it.
Case 2: Newsletter status. An article described an active manual newsletter workflow. Git commit history showed publishing setup work. The Beehiiv API, when queried, showed zero posts, zero drafts — the publication had been configured but never used. The claim was settled by the source the automatic checker hadn't been reading.
The claim "is the newsletter active?" has one correct answer. The problem: git commits documented intent (the setup work), while the provider API documented outcome (zero sends). These are subtly different propositions, but the article stated the outcome, so only the outcome source was authoritative.
In both cases the failure signal was invisible. The job produced a well-formed file. The updated number looked plausible. The output inspection principle says to verify the output rather than the job status — but even inspecting the output file doesn't catch "this number looks right but came from the wrong source."
What per-claim source declaration looks like
This is the design I haven't built yet.
The minimum viable form: a configuration that lists facts with multiple plausible sources and names the authoritative one for each:
rejection_count:
authoritative: author_notes
fallback: none
reason: git records committed events only; verbal notifications stay in notes
newsletter_active:
authoritative: provider_api
fallback: none
reason: git records intent; the API records outcome
The checker reads this config before querying any source. It queries only the declared authoritative source. If that source is unavailable, it returns unknown — not a silent fallback to whatever is readable. The failure-closed behavior is load-bearing: a silent fallback to git is exactly what created the four-vs-seven pattern in Case 1.
The config lives in the repo. Changes to it are in git history, reviewable, attributable — the same auditability that systematic review of implicit assumptions provides for GitHub Actions configuration.
A parallel from a different project makes the value clear. The shelf scanner I run on a Raspberry Pi 3 uses NCNN for inference — a lightweight framework designed for embedded hardware. The YOLO model detects empty space — single class, no product identification. Which SKU belongs in that gap is resolved by a planogram (a shelf-layout master file), not the model. Two distinct sources, two explicit roles: the model is authoritative for "is there empty space here?"; the planogram is authoritative for "what should be stocked there?" Conflating them would require the model to do SKU recognition — a much harder problem that needs per-store retraining. The explicit boundary makes the system maintainable in a way a conflated design wouldn't. I want the same clarity in the article pipeline.
The maintenance overhead question
@raknaos framed this accurately: a per-claim trust config is a small database with its own maintenance cost. Add a new data source and you need to update the config. Change which source is authoritative for a claim and you need to reason through every place the old source was cited. An unmaintained config that points to a deprecated field is worse than no config — it routes confidently to the wrong place.
Against that: what's the cost of the current approach?
The two-corrections case produced no visible error signal. Each pass produced a well-formed file with a plausible-looking number. I found the discrepancy manually, while fact-checking something else. The Dev.to API sync behaviors I've documented are similar: some fields have exactly one plausible source and are safe to treat as authoritative implicitly; others — like reaction counts on a published article — have a correct value that only the live API knows, not a cached projection. The difference between those two categories is currently in my head, not in the code.
My rough estimate: the maintenance overhead of keeping a trust config current is lower than the debugging overhead of a confident-wrong correction in a pipeline that runs unattended. But the answer depends on how often authority shifts. In a stable pipeline reading from a fixed set of sources, the config is set once and rarely touched. In a pipeline with active ETL from multiple third-party APIs — where a provider can deprecate a field, add a more-granular endpoint, or change what a field means — the maintenance cost rises, and the config earns its overhead most visibly precisely when authority shifts.
Where implicit mapping stays acceptable
Not every claim needs a declaration. Some have exactly one plausible source:
- Article word count: the file is the source, and no external authority can override it.
- Publication timestamp: the platform API records it; the repo stores it after the fact. One source.
- Inference time measured on your own device: your measurements, no alternative.
These don't need declarations because "which source?" has a single obvious answer. Implicit is fine when there's nothing to declare.
The pattern that warrants explicit declaration: whenever a developer could construct two plausible stories about the same fact from two different sources, and both stories would sound defensible, and an automated system must pick one without asking. That is the decision point the config exists to make explicit and auditable.
Looking back at what the quality contract v2 implementation actually introduced: verified_at records when verification ran, and original_evidence records what was checked. Neither records which source was authoritative when multiple sources exist. Adding a per-claim source declaration field to the contract is the next logical step.
What I would do differently
The verified_at field was the right idea applied at the wrong granularity. It stamps the article with "verified on this date" but doesn't record what source was queried for each claim. Two articles verified the same day from different sources can contradict each other; both have valid timestamps.
What I'm going to build: a source_declarations section in the frontmatter contract for claims that have multiple plausible sources. Each entry names the claim, the authoritative source, and a short reason. The checker script enforces it: if a claim type has an entry in source_declarations, the checker queries only that source and fails closed if it's unavailable.
This won't prevent every wrong correction. It will prevent the specific class of error where an automated checker picks the accessible source over the authoritative one, and does so silently, and the output looks correct.
FAQ
What if the declared authoritative source is unavailable when the checker runs?
Return unknown and skip the update. A checker that silently falls back to a secondary source reintroduces the failure mode you're designing around. An article with one field missing is recoverable; an article confidently carrying a number from the wrong source can sit there for weeks before anyone notices — or it can trigger a correction pass that confidently restores the wrong number again.
How do you identify which claims need declarations before you've had a failure?
You mostly don't, which is why the retroactive approach is more practical than prospective enumeration. Every confident-wrong correction reveals a claim that needed a declaration. Keep a running list. Each failure is free documentation of a gap in the config.
Is this the same as data lineage tracking?
Related but narrower. Lineage tracks how a value was transformed: "this field was computed from X using function Y at time Z." Per-claim source declaration answers a simpler, earlier question: when I need to verify this specific claim, which source do I query? The two can coexist and reference each other; neither replaces the other.
What did you actually implement as of this writing?
Nothing beyond verified_at and original_evidence in the frontmatter. The per-claim config is the next step. Writing this post is how I'm committing to the shape of it before building it. The two-corrections failure is documented in the prior article; this is the proposed design for making that failure structurally impossible next time.
Part of an ongoing 6-month experiment running three AI-curated directory sites. The technical claims here are real; this article was AI-assisted.
Related:
Top comments (0)