Three PDCA patterns that closed a 92-day detection lag in my content pipeline

The daily analytics report for my content pipeline is detailed: fleet views, A/B arm comparisons, queue depth, publish status. But for three months, it kept missing problems that had been present for months. A hardcoded multiplier that skewed all my analytics. A traffic channel registering zero for 71 consecutive days. A pipeline bug that ran wrong on every upload for over three months.

The common thread: the report described what happened yesterday but never compared a prior prediction against its eventual outcome. There was no mechanism to say "on August 1st, you expected X by September 1st — did X happen?"

That's the gap these three patterns close. The implementation lives in scripts/yt-analytics/pdca.py and pdca_meta.py, but the patterns are generic enough that I'd apply them to any automated pipeline that produces daily reports.

Pattern 1: Machine-checkable predictions with deadlines

A typical daily report entry looks like: "Pipeline fixed. Video generation back to normal." That's a description of an action, not a prediction. A week later, "normal" has no operational definition.

A ledger entry instead looks like:

{
  "id": "voc-launch-volume",
  "kind": "prediction",
  "claim": "VoC review corpus reaches 1000 total entries",
  "metric": "voc_reviews_total",
  "op": ">=",
  "value": 1000,
  "deadline": "2026-09-07",
  "on_fail": "Check appid selection — MAX_APPIDS cap or Steam rate limiting"
}

Every field is machine-readable. metric names a value the checking script knows how to read. op and value define the test. deadline is the date the test becomes binding. on_fail is the pre-committed response — written before the outcome is known, so a failure triggers a specific diagnostic path instead of a fresh debate about what might have gone wrong.

The checking engine (pdca.py --resolve) reads data/pdca-ledger.json, looks up each active prediction's metric from the daily history file, and writes MET or FAILED back into the ledger along with the actual observed value. Idempotent: running it twice on the same day produces the same result.

Pattern 2: Pre-committed fail actions

The detail in pattern 1 that matters most: on_fail is written when you create the prediction, not when it fails.

This is the part I kept skipping before I formalized it. A prediction without a fail action is an intention, not a commitment. When the metric fails, you end up re-analyzing from scratch: why did this fail? what should we check? The 92.5-day detection lag in our baseline happened partly because failures would be noticed, trigger a discussion, produce no decision, and then be reported again the following week as if newly discovered.

Pre-committing the fail action changes the failure path from "schedule a discussion" to "run this specific diagnostic." For the VoC corpus prediction above, if voc_reviews_total hasn't hit 1000 by September 7th, the pre-committed action is "check appid selection — MAX_APPIDS cap or Steam rate limiting." That's checkable in under five minutes. The prediction becomes a tripwire with a pre-loaded response.

Writing on_fail before the outcome is known also forces a useful discipline: if you can't write a specific fail action, you probably don't understand the prediction well enough to make it. "Investigate" is not a fail action. "Check collector's error log for HTTP 429 responses" is.

Pattern 3: A meta-audit layer that watches the checkers

Patterns 1 and 2 work if the ledger is kept current. But the ledger can rot in ways that L1 checking (pdca.py) can't see from the inside:

• A deadline passes without L1 resolving it (the checker ran but got the metric wrong)
• Code changes that should have a ledger entry don't have one
• A resolved prediction is missing its outcome reflection (the "what did we learn" field)

This is what pdca_meta.py (the L3 layer) watches. It doesn't check whether predictions came true — that's L1's job. It checks whether L1 is doing its job correctly.

The most useful check: coverage. pdca_meta.py lists every feat/fix/ops commit from the last 14 days that touched generation or strategy code but doesn't match any ledger entry. Before I added this, 18 significant code changes had no corresponding predictions. That's 18 changes where I was flying blind — no hypothesis about what they were supposed to accomplish, no mechanism to detect if they regressed.

The coverage list runs every day. A commit with no ledger entry is listed with its message and date until someone either creates a prediction for it or explicitly acknowledges it with a reason (e.g., "this was a pure refactor with no behavioral effect"). The acknowledgment must be explicit — silence doesn't clear the flag.

The three-layer structure is worth naming clearly:

• L1 (pdca.py): deterministic prediction-vs-outcome checks from history data
• L2: daily LLM tuner acts on the L1 report (chooses which hypothesis to explore next)
• L3 (pdca_meta.py): audits L1 and the ledger for staleness, missing coverage, and consistency errors

L3 runs in the same daily pipeline as L1 and surfaces its findings in the same report. When L3 flags something, it's a signal that the checking machinery itself needs maintenance — not the content pipeline.

The daily cron that runs L1 and L3 uses a standard GitHub Actions schedule trigger — the same pattern that runs the content ETLs for aiappdex.com — so the pipeline runs regardless of whether any manual action is taken that day.

What this doesn't fix

Daily reports still describe what happened, and the ledger doesn't replace that. The ledger adds a parallel structure that tracks whether the future we predicted arrived. The two serve different functions.

The hardest part of maintaining the ledger is discipline: writing predictions before you want to, not after you've seen the outcome. In practice, I've added on_fail retroactively when I realized a prediction was missing one, which is better than nothing but loses the main benefit — the fail action should be written while you still don't know the answer.

The patterns work best for hypotheses with measurable, time-bounded outcomes. "This change will increase view count" is not ledger-worthy unless you can specify which metric, by how much, by when. The discipline of specifying those three things before committing a change is most of the value.

---

Part of an ongoing 6-month experiment running three AI-curated directory sites. The technical claims here are real; this article was AI-assisted.