DEV Community: MORINAGA

Three ETL failure patterns I now write into the output file, not just the logs

MORINAGA — Fri, 14 Aug 2026 07:05:44 +0000

My Reddit scraper returned empty arrays for 71 days. Nothing broke visibly. The market-listening output file kept being written, kept being committed to git, kept looking like a healthy daily snapshot. Nobody caught it until the interpretation layer noticed that Reddit-sourced signals hadn't changed in two months.

The three patterns I added afterward are small. Combined, they mean that failure mode can't happen silently again — the artifact itself reports what went wrong, not just the runner logs that nobody reads unless something is already on fire. All three are implemented in scripts/market-listening/collect.mjs, with the 71-day blind spot documented in the inline comments.

Pattern 1: sources_ok lives in the artifact, not the logs

The daily output JSON now contains a sources_ok object with a boolean per source:

{
  "sources_ok": { "youtube": true, "autocomplete": true, "bluesky": true },
  "errors": [],
  ...
}

If any YouTube request fails, sources_ok.youtube flips to false. The file that gets committed to git carries its own health signal. That's the core change: the failure state lives in the artifact, not in a log stream.

Why this matters over logging: the committed artifact shows up in git diffs, gets reviewed in the daily health check, and is read by the interpretation layer before it uses any of the data. Logs from a GitHub Actions cron are reviewed only if someone suspects a problem. The artifact is reviewed on every downstream read.

The pipeline health monitor checks sources_ok on every run and opens a GitHub issue if any source is false. The interpretation layer that reads these files is fail-closed on sources_ok: if sources_ok.youtube === false, the YouTube signals for that day are ignored rather than mixed into the weekly aggregate. Separating collection from interpretation makes this guard practical — the interpretation step reads the artifact and checks health before consuming any of the results.

Pattern 2: errors[] alongside the results

The boolean sources_ok tells you something failed. The errors array tells you what and when:

function recordError(source, detail) {
  const entry = { source, detail, at: new Date().toISOString() };
  errors.push(entry);
  console.error(`ERROR [${source}] ${detail}`);
  return entry;
}

Every failed fetch calls recordError with the source name, the error detail (including HTTP status codes), and an ISO timestamp. These accumulate in the errors array and are written into the same artifact as the successful results.

The practical difference: if a run produces errors: [{ "source": "bluesky", "detail": "HTTP 403", "at": "2026-08-13T07:..." }], I can see the exact failure mode without opening any runner log. After eight days of that, the error messages have been consistent enough that I know exactly what the Bluesky API is rejecting — not just "bluesky failed."

The at timestamp matters too. If a script partially completes and then errors, the timestamps in errors[] tell you at what point in the run things started going wrong, without reconstructing it from log correlation.

This is different from the approach I described in catching silent failures in a GitHub Actions pipeline, which focuses on job-level failures in CI. These failures are internal to a single job that mostly succeeds — partial failures within a healthy-looking run, which CI-level checks won't catch.

Pattern 3: the vacuous-true guard on .every()

The sources_ok flag for YouTube is computed by checking whether every query produced valid voted results:

const ok =
  QUERIES.length > 0 &&
  QUERIES.every((q) => queries[q].reps_ok >= MIN_APPEARANCES && queries[q].accepted > 0);
if (!ok) recordError("youtube", "one or more queries produced no voted results");

Without QUERIES.length > 0, an accidentally-empty QUERIES array would satisfy .every() vacuously — [].every(fn) returns true for any fn. The script would commit an artifact with sources_ok.youtube = true and zero actual data inside.

This is more plausible than it sounds. Any config change that blanks the query list — a bad merge, an over-aggressive deduplication, an env substitution that resolves to empty string — would silently produce an artifact that reports itself as healthy. The length guard is three tokens.

The same pattern applies to any validation using .every(), .all(), or equivalent over a collection that can be empty. If an empty input is a failure condition — and it usually is — guard on length first. The validation chain length > 0 && every(condition) handles it cleanly.

What the three patterns give you together

A single fetch failure: sources_ok flips to false. The git diff shows it. The health check catches it. The interpretation layer ignores that source for that day. The error detail explains what happened.

A config accident that empties the query list: the length guard trips, the source reports false, the artifact contains no data but reports that explicitly rather than vacuously reporting success.

An extended outage (like the current Bluesky block): eight consecutive artifacts, each with sources_ok.bluesky = false and matching error entries. The pattern in the committed history is unmistakable and requires no log archaeology.

The 71-day Reddit blind spot was eventually caught from the outside — someone noticed the data wasn't changing. These patterns are designed to make that kind of failure visible from the inside, in the artifact itself, immediately on the day it happens.

Part of an ongoing 6-month experiment running three AI-curated directory sites. The technical claims here are real; this article was AI-assisted.

What I learned about repeat-and-vote sampling for non-deterministic search results

MORINAGA — Fri, 14 Aug 2026 07:05:39 +0000

Repeating a YouTube search query three times and keeping only results that appear in two or more of those fetches is more reliable than treating any single fetch as ground truth. The Jaccard similarity between two fetches of the same query — within a single minute — sits around 0.43–0.88. Today's run measured 0.447 for the "more steam reviews than" query. That means roughly half the video IDs flipped between fetches. A single fetch isn't a signal; it's a snapshot of one shuffled result.

I built this pattern into a daily market-signal collector for three directory sites (AI tools, indie games, open-source alternatives). The collection layer fetches five YouTube queries every morning, and the full implementation is in scripts/market-listening/collect.mjs. This article is about how I made that collection reliable despite YouTube search being inherently non-deterministic.

Why a single YouTube search fetch isn't a signal

YouTube's search ranking is legitimately non-deterministic. Freshness boosts, A/B test bucketing, CDN-level caching, and real-time personalization all interact to produce different rankings on the same query across consecutive requests. This isn't an API bug or detection avoidance — it's how the system works.

I confirmed this by measuring. The script stores the mean pairwise Jaccard across all rep-pairs per query in the output file. Today's run:

Query	Mean pairwise Jaccard	Items kept	Items dropped as noise
"more steam reviews than"	0.447	measured across 3 reps	28

The code stores the mean only — not individual pairwise values — so 0.447 is what I can actually cite. At that value, under half the video IDs appear in both halves of any given fetch-pair. Treating one fetch as "the ranking" means you're looking at roughly a coin flip on which videos actually show up. You can't tell signal from noise from a single observation.

The problem is worse for market-signal use cases, where you're not trying to understand any one video's rank — you're trying to identify which videos appear consistently over time. A video that appears in one fetch but not the next isn't evidence of anything. Consistent appearance across independent fetches is.

The repeat-and-vote approach

The fix is three fetches per query, with a jitter delay (5–15 seconds, randomized) between each. A video that appears in fewer than two of the three fetches is filtered out and counted in dropped_noise. Only items that appear in at least two of three fetches are considered signal.

Today's run dropped 28 items as noise for the first query. Those videos were surfaced by YouTube's shuffler at least once, but didn't clear the consistency threshold. Some of them might be genuinely relevant; most are one-time shuffler artifacts. I'm trading recall for precision: I'd rather miss a few real signals than include noise that drives incorrect market conclusions.

A few implementation notes that aren't obvious:

The MIN_APPEARANCES=2 threshold is a parameter. At REPS=3, requiring 2/3 appearances means a video must appear consistently across the majority of fetches. Raising it to 3/3 would be stricter — better for high-confidence signals, worse for coverage. I've run with 2/3 since the collector was built and haven't found a reason to change it.

The jitter delay is calibrated for request politeness, not cache-busting. The sleep intervals I use across different APIs are different for different services — YouTube gets 5–15s between requests. This introduces some time variance between reps, which helps with cache freshness but isn't guaranteed to force a cache miss.

I don't use the official YouTube Data API for this. The search HTML surface is sufficient for market-listening purposes, and it avoids a key rotation dependency. I prefer polling public endpoints without API key registration where the data surface is adequate.

Median rank instead of position from a single fetch

Once I determine which videos cleared the consistency threshold, I face a second problem: what rank to assign them. A video might rank 2nd in one fetch, 3rd in another, and 15th in a third. None of those individual positions is trustworthy.

The solution is median rank. For each kept video, I collect its rank from every rep where it appeared, sort those ranks, and take the median:

// voteReps() in scripts/market-listening/collect.mjs
const sorted = [...v.ranks].sort((a, b) => a - b);
kept.push({
  ...v,
  median_rank: sorted[Math.floor(sorted.length / 2)],
});

A video ranked 2, 3, 40 across three reps gets median rank 3. The outlier position (40th) doesn't drag it down. A video ranked 1, 2, 1 gets median 1. For a two-element list — a video that only appeared in 2/3 reps — median is the second element: ranks [2, 8] returns 8, since floor(2/2) = 1.

I use median specifically because search rank distributions have long tails. A video can appear in position 30+ on a bad shuffle while legitimately being a top-10 result. Mean would be dragged by those outlier positions; min would reward lucky placements. Median is stable against both. I've applied the same outlier-resistance property when using Jaccard for duplicate detection — the logic generalizes across signal-extraction problems.

Jaccard as a built-in diagnostic

After computing the vote for each query, the script computes the mean Jaccard similarity across all pairs of reps for that query and stores it in the output:

const pairs = [];
for (let i = 0; i < reps.length; i++) {
  for (let j = i + 1; j < reps.length; j++) {
    pairs.push(jaccard(
      reps[i].map((v) => v.videoId),
      reps[j].map((v) => v.videoId)
    ));
  }
}
queries[query].jaccard_mean = pairs.length
  ? Number((pairs.reduce((a, b) => a + b, 0) / pairs.length).toFixed(3))
  : null;

This means every daily file in data/market-listening/ carries a per-query Jaccard mean. It's not just an implementation detail — it's the primary diagnostic for whether the repeat-and-vote pattern is still justified.

If jaccard_mean climbs toward 1.0 for a query, the shuffler has become deterministic for that query. Either caching locked in, or the query narrowed to so few results there's no variation left. That's the cue to lower REPS (three fetches of the same cached result isn't three independent samples) or rethink the query entirely.

The pipeline health monitor reads these files daily. I could add Jaccard-based alerts — flag if any query exceeds 0.9 (suspiciously deterministic) or drops below 0.25 (unusually chaotic). I haven't added those yet because 0.447 has been stable enough that I don't have a calibrated threshold. This is an honest gap in the current implementation.

The Jaccard function itself:

export function jaccard(a, b) {
  const A = new Set(a);
  const B = new Set(b);
  if (A.size === 0 && B.size === 0) return 1;
  let inter = 0;
  for (const x of A) if (B.has(x)) inter++;
  return inter / (A.size + B.size - inter);
}

Two empty sets return 1 (vacuously equal). That edge case matters: if two reps both returned empty results, this would report perfect similarity — which is wrong from a health standpoint. I handle this upstream via the sources_ok logic, which fails a source if any query produced zero accepted results. But the Jaccard function itself doesn't surface that failure; it needs external context.

Where this approach breaks down

The repeat-and-vote pattern assumes independent fetches. Several failure modes break that assumption.

CDN-level caching. If all three requests hit the same YouTube edge cache, they receive the same response — effectively one sample, not three. The jitter delay and randomized inter-rep timing help, but can't guarantee cache-misses. Watching jaccard_mean over time is the only way to catch this from the outside; a sustained climb toward 1.0 across all queries simultaneously is the clearest signal.

Egress IP stability. GitHub Actions runners can change egress IPs between runs but typically don't change between steps within a single run. If all three reps within a run use the same egress IP, they might receive the same regional ranking. This is less likely than CDN determinism, but it's a real risk for queries with strong geo-sensitivity.

Temporal autocorrelation. The 5–15s jitter between reps is short enough that trending content could genuinely rank differently between rep 1 and rep 3. For market-signal purposes, where I want stable-over-time signals, this is actually fine — a video that's trending at second-level resolution isn't what I'm tracking. But if you were trying to detect short-lived viral content, the jitter window would need to collapse.

What I'd do differently now: log per-query Jaccard variance alongside the mean. The mean tells me how similar the fetches were on average, but variance would tell me whether they were clustered (three fetches all near 0.45) or spread out (one at 0.2, one at 0.8). That's diagnostic information I don't currently have. I'd also add reps_attempted vs. reps_succeeded at the per-rep level, so I can identify which specific fetch failed without reading runner logs.

The broader pattern — collect, vote, report the vote metadata — applies to any non-deterministic API. The implementation details change (different HTML parsing, different jitter), but the logic is the same. And having Jaccard built into the output gives you a continuous health signal rather than having to re-instrument later when something starts behaving differently. This is related to what I cover in catching silent ETL failures and applies equally to survivorship-bias concerns in any analytics pipeline — the data you collect determines what you can conclude, and knowing its collection conditions is part of using it responsibly.

FAQ

Why not use the YouTube Data API's search endpoint instead?

The search.list endpoint has its own caching and returns different results than the search HTML surface in ways that are harder to inspect. More importantly, I prefer polling public endpoints without registering for a key where sufficient — every API key is a credential to rotate and a quota to manage. For market-listening at this scale, the HTML surface is adequate.

What happens if one of the three reps fails to return any results?

A failed rep reduces the pool for voting. If only one rep succeeds, nothing can reach the MIN_APPEARANCES=2 threshold and the query produces zero accepted items. This flips sources_ok.youtube to false in the output artifact, and the daily health check surfaces it. It's not silent.

Would REPS=5 produce better results?

Probably yes, at linear cost. Each additional rep adds several minutes to the run (5–15s jitter per request × 5 queries). For a daily market-signal that only needs directional accuracy, REPS=3 filters most single-fetch noise without being prohibitively slow. I'd increase it if I needed sub-day granularity or was tracking a more volatile query set.

Does this approach work for other search APIs?

Yes. The repeat-and-vote pattern and the Jaccard diagnostic are search-API-agnostic. You'd change the HTML extraction and the appropriate jitter timing, but the voting logic and the sources_ok/Jaccard output structure are reusable. The only input the vote function needs is a list of per-rep result sets where each item has a stable ID and a rank.

Part of an ongoing 6-month experiment running three AI-curated directory sites. The technical claims here are real; this article was AI-assisted.

What 14 days of YouTube hook A/B parity data revealed—and what it's hiding

MORINAGA — Thu, 13 Aug 2026 07:09:30 +0000

I've been running a hook A/B test on my YouTube Shorts for 14 days. The variant arm leads on views-per-day by a meaningful-looking margin. The data is also telling me that the margin is almost entirely explained by one outlier video on each side — not by the hook text.

Here's what the numbers look like and why I'm not changing anything yet.

The test setup

The YouTube short-generation pipeline assigns hook arm by the publish day's UTC parity. Even UTC day = CONTROL: number-first hook, leading with a concrete data point. Odd UTC day = VARIANT: story-first or underdog-emotion hook, burying the data inside the script rather than leading with it.

Every video spec now records "hook_arm": "control" or "hook_arm": "variant" in its JSON. The analytics history appends this to the per-video JSONL on every daily fetch. The A/B logic is described in the survivorship bias fix write-up — without full-fleet JSONL, there would be no arm-level data to analyze.

The 14-day window (2026-07-29 to 2026-08-12) covers 11 videos: 3 in the control arm, 8 in the variant arm. The imbalance is calendar arithmetic — videos only upload when specs are ready, not on a fixed cadence, so the even/odd split doesn't guarantee equal arm sizes.

The 14-day numbers

Parity-inferred medians (these are computed from current views-per-day, not age-controlled day-7 data — the JSONL doesn't have enough history for day-7 on all 11 videos yet):

Arm	n	Videos	Median vpd
Control (even day)	3	Brotato/AC Mirage, Undertale/Spider-Man, Project Zomboid	5.3 vpd
Variant (odd day)	8	Geometry Dash, Binding of Isaac, RimWorld/FF7R, Undertale/Jul29, Don't Starve, Celeste, Valheim, Among Us	7.4 vpd

The two-day threshold for flagging a directional signal — variant leading for 2 consecutive days — was crossed today.

The confound

The control arm contains one video with 0.7 vpd: Brotato/AC Mirage. This is a "flop anchor" pair. AC Mirage had poor anchor quality relative to the live-table rankings at the time of publication, and the video's performance reflects that.

The variant arm contains Geometry Dash (18.7 vpd, age 3 days). This video uses a current live-table anchor with EA FC 26 — a different quality tier from anything in the control arm. Geometry Dash's high early vpd is consistent with a strong anchor pairing, not with the hook text being particularly effective.

When I exclude one outlier from each arm — Geometry Dash from variant, Brotato from control — the medians converge:

Variant (n=7, ex-Geometry Dash): 6.0 vpd
Control (n=2, ex-Brotato): 5.7 vpd

A 0.3 vpd gap on n=2 vs n=7 is not interpretable. The original 2.1 vpd gap was driven by which anchor each arm happened to get, not by the hook text.

Why the outliers ended up in these arms

The arm assignment is mechanical: even UTC publish day = control, odd = variant. Geometry Dash was published on an odd day because the generation routine fired on that day. Brotato was published on an even day for the same reason. Neither assignment was intentional from a design standpoint — the calendar parity doesn't track anchor quality.

This is the core design problem: the arm assignment correlates with publish date, and anchor quality partially correlates with publish date (because the live-table ranking changes over time, and the best anchors get used when specs happen to be ready). Getting a clean measurement requires either randomizing anchor quality across arms or stratifying — making sure each arm has a comparable distribution of anchor tiers.

Currently the live-table anchor gate ensures minimum quality (R6b compliance) but doesn't control for distribution within arms.

What I'm waiting for before drawing a conclusion

Three conditions need to be met before the hook text comparison is valid:

1. Tagged data instead of inferred data. The hook_arm field was added partway through the 14-day window. Several of the videos in the parity table are "parity-inferred" — I know their publish day and computed their arm from that, but they don't have hook_arm in their spec JSON. Inferred arms are correct, but they make the history brittle to any change in how the pipeline runs.

2. At least 5 tagged observations per arm. The current control arm has n=3, with one outlier consuming most of the signal. At n=5 (minimum), excluding one outlier still leaves 4 data points — enough to compute a median that isn't dominated by the sample distribution.

3. Anchor parity between arms. Before publishing the next batch of videos under the hold, I'm tracking which anchor tier lands in which arm. If the next three control-arm videos all get weak anchors and the next three variant-arm videos get strong anchors again, the arm comparison stays confounded regardless of sample size.

The anchor parity check is the one I can't automate easily with the current pipeline. It requires comparing the live-table ranking of the anchor at the time of publish across the two arms — which means storing the table state at generation time, not just the anchor name. That's a data collection change, not a filtering change.

What the 2-day threshold is actually telling me

The "variant leads for 2 consecutive days" signal is real and worth noting. It means there's a genuine directional trend in the current data, not one-day noise. But the threshold was designed to trigger review, not to trigger a directive change. The next step is the anchor-parity audit, not switching all future videos to the variant hook.

If after 5 tagged observations per arm with comparable anchors the variant still leads, that's a different situation. At that point the hook text itself is the more plausible explanation for the gap.

Until then: record hook_arm in every new spec JSON, and wait.

Part of an ongoing 6-month experiment running three AI-curated directory sites. The technical claims here are real; this article was AI-assisted.

What I learned separating daily collection from weekly interpretation in a cron pipeline

MORINAGA — Thu, 13 Aug 2026 07:09:19 +0000

The question came from a reasonable place: if market-listening sweeps only run once a week, are meaningful day-over-day changes getting missed? I spent a few hours measuring before answering, and the measurement led to an architectural split I hadn't planned on.

The answer: collect daily, interpret weekly. Not because weekly is the right cadence for collecting, but because the instrument I'm querying doesn't have daily resolution.

The instrument's resolution problem

YouTube search ranking sounds like a high-resolution signal. A video that gained 10,000 views overnight should move in the results the next day. In practice, the ranking surface is more like a weather map: high-resolution in principle, but what you see at any moment is partly the weather and partly display non-determinism.

I measured this by fetching the same five search queries, then fetching them again roughly 60 seconds later, and comparing the two result lists using Jaccard similarity. A Jaccard of 1.0 means the lists are identical; 0.0 means no shared entries.

The same-minute refetch returned Jaccard similarity between 0.43 and 0.88. A genuine two-day difference — comparing two days' fetches of the same query — returned 0.03 to 0.58.

The same-minute measurement noise (0.43–0.88) substantially overlaps with the actual two-day signal range (0.03–0.58). When the noise and signal ranges aren't separated, you can't reliably attribute a change to "something changed in the real world" versus "the ranking shuffler surfaced different items." View counts have the same problem: across two consecutive daily fetches, 59% of view counts were byte-identical. The changing ones had mostly crossed a display-rounding threshold ("1.2K" vs "1,247"), not a real change in views.

Conclusion: this instrument has weekly resolution for persistent trends, not daily resolution. Running the sweep daily doesn't change that.

Why daily interpretation means publishing noise as strategy

Four downstream systems read the shared market-trends.md file: the YouTube Short generation routine, the Bluesky post queue, the article routine, and the market monitoring sweep. I described how that file works as a shared context hub in detail last week. The Bluesky routine reads its FLAT/CHANGED columns directly into subject selection.

When market-trends.md was updated daily, single-fetch noise propagated immediately into Bluesky subject selection. A video that appeared in the top-15 because the shuffler surfaced it would become a "trending subject" for the next day's posts. Those posts would go out on a subject with no genuine search demand — just a one-day shuffler artifact.

The failure was invisible. The posts looked normal, the subjects had the right category, nothing in the logs flagged a problem. The degradation was slow drift toward subjects with no measured demand. This is the category of failure that's hardest to catch: the pipeline runs, the outputs look plausible, and the error accumulates silently. Recognizing the shape of that failure is what the pipeline health monitor pattern is designed to surface — but only if you've defined what "healthy output" means. I hadn't.

Three design rules in the new collection layer

The fix was a dedicated daily collection layer with no interpretation step. scripts/market-listening/collect.mjs runs at 17:00 UTC every day and writes to data/market-listening/YYYY-MM-DD.json. Three rules in the code are treated as non-optional.

Rule 1: Repeat-and-vote. Each query is fetched three times. A video only makes the output if it appears in at least two of the three fetches. The constants are REPS = 3 and MIN_APPEARANCES = 2.

// Repeat-and-vote parameters. 3 reps / 2 votes is the cheapest setting that
// removes single-fetch flukes: at Jaccard ~0.6 a genuine top result appears in
// all 3 reps, a shuffler artifact typically in 1.
const REPS = 3;
const MIN_APPEARANCES = 2;

At a Jaccard of ~0.6, a video that genuinely ranks in the top 15 appears in all three fetches. A shuffler artifact that randomly surfaced once might appear in one fetch, rarely in two, almost never in all three. In live dry-runs before launch, this rule dropped between 16% and 36% of unique videos per query. Over the first full collection run on 2026-08-12, 86 of 287 unique candidate videos were filtered as noise. That's more than a quarter of the raw output — gone before it reached the interpretation layer.

Rule 2: Never swallow a failure. The previous trends-fetch workflow used a fetch helper that caught HTTP 403 errors and returned an empty array. Two Reddit sources went dead and nobody noticed for 71 days because the collection file kept being written and kept looking plausible. Empty arrays aren't obviously wrong; a file with fewer signals just looks like a quiet week.

In the new script, every HTTP failure is written to an errors[] array in the output JSON and the matching entry in sources_ok is flipped to false. The committed file itself is the alert:

{
  "sources_ok": { "youtube": true, "reddit": false, "autocomplete": true },
  "errors": [
    { "source": "reddit/game-recs", "detail": "HTTP 403", "at": "2026-08-12T17:14:22Z" }
  ]
}

A consumer reading the file sees immediately that reddit-sourced signals are absent. The same principle drives the GitHub Issues monitor — make failures visible in the committed artifact, not just in runner logs that nobody reads daily.

Rule 3: Parsed JSON only, never raw HTML. The initial collection prototype wrote full YouTube search HTML to disk for later parsing. A single full sweep was 15MB. Annualised, that's ~5.5GB of git history for one workflow. The parsed-rows-only format runs ~98KB per day — about 150× smaller. The lessons from managing GitHub Actions artifact storage apply here: storage costs are real, but reviewability is the bigger reason. A 98KB JSON diff is readable when something goes wrong; a 15MB HTML diff is not.

The interpretation layer stays weekly

The collection script has one explicit comment that appears twice:

// This script never interprets, it only measures.

The weekly interpretation sweep reads multiple consecutive days of collection output and looks for two types of signal above the noise floor:

signals.persistent_new: a video that appeared in the top-15 for three consecutive daily collection runs. Persistence across three independent days of REPS=3 collection means this video appeared in 2-of-3 fetches for three days straight — 6 of 9 possible observation slots. That's a much stronger signal than any single-day reading, and it has a measured false-positive rate (the 86/287 noise drop gives an upper bound).
signals.autocomplete_shift: YouTube's autocomplete suggestions change slowly — on a days-to-weeks timescale rather than minute-to-minute. Day-over-day autocomplete changes are above the noise floor where ranking isn't.

The interpretation sweep still uses an LLM — the same session-coordination pattern from this write-up — but only once a week, and only when the persistent signals give it something real to reason about. A daily LLM interpretation pass over single-fetch noise would produce outputs that look thoughtful and describe the shuffler.

The timing was also designed around conflict avoidance. The auto-tuner lands at 20:42–20:47 UTC in practice. Collection at 17:00 UTC leaves 3.5 hours of clearance. The cron concurrency discipline for this monorepo uses workflow-level groups, so the time separation matters.

What downstream consumers gained

Before the split, the market-trends.md file's updated: date meant "when someone last ran a sweep" — which conflated collection age with interpretation age. A consumer had no reliable way to distinguish stale interpretation from no data.

After the split, three distinct states are visible:

State	Signal	Consumer action
`sources_ok: all true` + recent daily commit	Collection healthy	Use data normally
`sources_ok: youtube=false`	Source broken	Use remaining signals, flag gap
`updated:` older than 14 days	Interpretation stale	Fail-closed to defaults

The freshness gate described in the shared context file write-up now has sharper semantics because collection freshness (daily commit history) is separate from interpretation freshness (updated: date). The pipeline health monitor checks sources_ok flags on every run and would open a GitHub Issue if both YouTube sources failed simultaneously — impossible to detect before the split.

What I'd do differently

Measure the instrument's day-over-day resolution before designing the collection frequency. The Jaccard measurement took about two hours. I spent a month assuming daily sweeps produced daily-resolution signals because the sweep ran daily. Those are not the same thing.

The middle ground I considered — keeping daily interpretation but applying a 7-day rolling average to raw scores before writing the file — wouldn't have worked. Averaging noise doesn't produce signal; it produces a smoother representation of noise. The Jaccard non-determinism problem is that individual observations aren't reliably measuring the underlying ranking. Averaging them gives you a smoother non-signal, not a real one.

The correct insight is that you need consensus across independent samples taken at the same moment (the REPS=3 vote), not smoothing across repeated moments. Voting filters out non-signal at observation time; no amount of post-hoc averaging replaces that.

FAQ

Why 3 fetches specifically?

At Jaccard ~0.6, a genuine top-15 result appears in all three fetches. A shuffler artifact typically appears in one, rarely two, almost never all three. The 2-of-3 threshold is the cheapest configuration that removes single-fetch flukes while keeping collection under 3 minutes (5 queries × 3 reps with 5–15s jitter between hits). REPS=5 would have pushed runtime past 5 minutes and tripled the request count to YouTube's servers.

What's autocomplete_shift and why is it above the noise floor?

YouTube autocomplete returns suggested completions for seed prefixes ("stardew valley vs...", "best indie game..."). These suggestions update on a days-to-weeks timescale rather than minute-to-minute. When the autocomplete response for a seed changes between two consecutive daily fetches, that indicates genuine search-demand movement — not ranking non-determinism.

Doesn't the 17:00 UTC cron conflict with other writes?

The auto-tuner lands at 20:42–20:47 UTC in practice, giving 3.5 hours of clearance. Earlier attempts at 20:00 UTC caused occasional push conflicts. The collection script's push step includes a 3-attempt retry with git pull --rebase on conflict — the same pattern used by other write workflows in this monorepo — but time-based separation is simpler than relying on retry logic.

Could you just interpret daily with a longer rolling window?

The problem isn't window length; it's that each individual observation is unreliable at daily resolution. Averaging seven noisy daily fetches gives a smoother curve but each data point still carries Jaccard noise of 0.43–0.88. The vote removes non-signal at observation time. No amount of post-hoc averaging replaces that.

Three things I learned designing a shared context file for automated routines
What I learned building a pipeline health monitor that opens GitHub Issues automatically

Part of an ongoing 6-month experiment running three AI-curated directory sites. The technical claims here are real; this article was AI-assisted.

Pausing a GitHub Actions cron: the yaml trap that breaks all workflow triggers

MORINAGA — Wed, 12 Aug 2026 07:05:36 +0000

After 11 uploaded videos disappeared from YouTube overnight, I wanted to stop the automated upload schedules while I investigated. The fix seemed trivial: comment out the cron lines in two workflow files. I've done this before without thinking about it. Today it broke two unrelated things.

Here's what went wrong and how to pause a cron correctly.

The mistake: commenting out the cron line but leaving the key

The original yt-publish.yml looked like this:

on:
  push:
    branches: [main]
  schedule:
    - cron: '0 21 * * 1,3,5'

My first attempt at pausing was:

on:
  push:
    branches: [main]
  schedule:
    # - cron: '0 21 * * 1,3,5'

That leaves schedule: as a key with no value. In YAML terms, schedule: with no value is a null scalar, which is valid YAML — but GitHub Actions doesn't accept it. Its workflow schema requires schedule to be a sequence. An empty schedule: key fails validation.

I did the same thing in yt-publish-longform.yml. Two workflows now in a broken state.

How the failure shows up

The cryptic part: the error doesn't say "invalid schedule". GitHub's runner rejects the whole workflow file and the failure appears on every trigger, including push.

The Actions tab shows:

Run status: failed
Run duration: 0s
Message: "This run likely failed because of a workflow file issue."

At 0s, no job has started. It's a pre-parse failure. Because both affected workflows had a push trigger as well as schedule, every commit to main for the next hour showed red checks. The failing workflow was the upload pauser, not any of the actual build or publish workflows — so at first the red commits looked related to something I'd pushed, not to the yaml edit.

The tell: 0s duration. Any real job failure takes at least a few seconds to allocate a runner. A 0s failure almost always means the workflow file didn't parse.

The fix: comment out the entire schedule key

The correct way to pause a cron without disabling other triggers is to comment out the key itself. GitHub's workflow syntax docs require schedule to be a non-empty sequence — an empty mapping key fails schema validation.

on:
  push:
    branches: [main]
  # schedule:   # ⚠ PAUSED 2026-08-11 — uncomment this line AND the cron line below to resume
  #   - cron: '0 21 * * 1,3,5'

With schedule: itself commented out, the on: block only contains push. GitHub parses this without complaint. The push trigger still fires; manual workflow_dispatch calls still work.

I also added a note in the comment explaining why it's paused and exactly which line to uncomment. This is worth the extra characters — "uncomment to resume" comments have saved me from re-investigating why a schedule was disabled when I come back to it two weeks later.

The useful side effect: workflow_dispatch still works

When only schedule is disabled, workflow_dispatch remains active. This turned out to be exactly what I needed: I could test the disappearance detection code by triggering the analytics workflow manually from the Actions tab, without the risk of automatically uploading more videos to a channel with unexplained deletions.

There's a common pattern where pausing a cron is paired with needing manual testing:

Something went wrong with the automated run
You want to stop the automation while you debug
But you want to run it once manually to verify a fix

workflow_dispatch covers this exactly. Commenting out schedule while leaving workflow_dispatch gives you manual control without re-enabling the automation.

Relevant to four cron timing bugs I've hit before: this is the opposite problem — not a cron that fires at the wrong time, but a cron that needs to stop while preserving manual access. The yaml trap is easy to avoid once you've seen it, but the failure mode is confusing enough the first time that it's worth writing down.

Summary

What you do	What GitHub sees	What breaks
Comment out just the cron line	`schedule:` with null value	Whole workflow fails (0s) on all triggers
Comment out the `schedule:` key	Valid `on:` with only remaining triggers	Nothing — push and workflow_dispatch still work
Remove `schedule:` key entirely	Same as above	Nothing

The second and third rows are equivalent. I prefer commenting over deleting because it preserves the cron expression for when I want to re-enable, and the "PAUSED" comment documents the decision.

Part of an ongoing 6-month experiment running three AI-curated directory sites. The technical claims here are real; this article was AI-assisted.

How I detected deleted YouTube videos using JSONL history diffing

MORINAGA — Wed, 12 Aug 2026 07:05:31 +0000

The YouTube analytics cron ran without incident on August 10. Health checks were green. Then I looked at the channel manually and counted 79 videos where I expected 89. Ten were gone. A quick oEmbed check on every tracked ID confirmed eleven total deletions — the YouTube Data API v3 omitted them entirely, which means deleted rather than unlisted or private.

One of the eleven was published by this pipeline at 21:47 UTC on August 10. By 01:14 UTC on August 11 it was gone. The pipeline that uploaded it never noticed.

This article is about the blind spot that caused that, what the implementation looks like, and a subtle bug in the first version that I would have hit the next day.

The blind spot: health checks only watch the input side

The pipeline health monitor I built in July checks whether videos have been queued, generated, and uploaded according to schedule. It opens GitHub Issues when a cron doesn't fire, when the queue goes stale, or when an upload timestamp is missing.

What it doesn't do — and what I never thought to add — is ask YouTube whether the videos it previously published still exist. The monitor is entirely input-side: local files, local timestamps, queue state. If something on YouTube's end deletes a video, every local artifact still looks valid. The upload timestamp is real. The video ID is in the queue's history. All green.

This is the same category of problem I wrote about in fixing survivorship bias in my analytics — a measurement that looks at what you produced, not at the current state of what exists. Silent failure detection in this pipeline has always been a coverage problem: the tests check output, not outcome.

The fix needs to work the other direction: take today's live fetch from the YouTube Data API and compare it against what was known to exist yesterday. Anything absent is a candidate deletion.

Building detect_disappeared() using JSONL history

The analytics script already maintained a JSONL history file — one row per (date, video_id) — because fixing survivorship bias required having all-time fleet data rather than just the current top/bottom snapshot. That existing history is the comparison baseline.

def previous_snapshot(history_path: Path = HISTORY_PATH) -> tuple[str, dict[str, dict]]:
    """(date, {video_id: row}) of the most recent day already in history."""
    by_date: dict[str, dict[str, dict]] = defaultdict(dict)
    if history_path.exists():
        for line in history_path.read_text().splitlines():
            try:
                d = json.loads(line)
            except ValueError:
                continue
            if d.get("date") and d.get("video_id") and d.get("status") != "disappeared":
                by_date[d["date"]][d["video_id"]] = d
    today = datetime.now(timezone.utc).strftime("%Y-%m-%d")
    prior = sorted(d for d in by_date if d != today)
    return (prior[-1], by_date[prior[-1]]) if prior else ("", {})

def detect_disappeared(stats_videos: list[dict], history_path: Path = HISTORY_PATH) -> list[dict]:
    prev_date, prev_rows = previous_snapshot(history_path)
    if not prev_rows:
        return []
    live = {v.get("id") for v in stats_videos if v.get("id")}
    return [
        {"video_id": vid, "last_seen": prev_date, "title": row.get("title", ""),
         "published_at": row.get("published_at", ""), "views": row.get("views", 0)}
        for vid, row in sorted(prev_rows.items())
        if vid not in live
    ]

stats_videos is the live fetch from videos.list — everything the Data API returns for the channel today. previous_snapshot() reads the most recent date already written to history (not today). The diff is a set subtraction: IDs in the snapshot that are absent from the live set.

The result goes two places. First, the run prints a ::warning:: annotation that surfaces in the GitHub Actions step summary:

::warning::11 video(s) disappeared from YouTube since 2026-08-10: abc123(24v), def456(0v), ...

Second, the daily report gains a "Disappeared from YouTube" table with each ID, its last-known view count, and its publish date. Both outputs are loud by design — this is the one failure mode the input-side health checks structurally cannot catch.

The subtle bug: disappearance markers that look like presence

The first implementation had a defect I would have hit on the very next run.

When detect_disappeared() finds deletions, they get appended to the history file as rows with status: "disappeared". That keeps a permanent record of the deletion and lets day7_by_archetype() exclude those videos from median calculations.

The bug: previous_snapshot() read every row in the history file to build the "what existed yesterday" map. A status: "disappeared" row has a valid date and video_id — so it was being counted as "this video was present on that date." The next day's run would read the disappeared marker for a video, conclude that video was present yesterday, not find it in the live fetch (because it was deleted), and report it as disappeared again. The same 11 videos would fire every single day with a drifting last_seen.

The fix is one filter condition:

if d.get("date") and d.get("video_id") and d.get("status") != "disappeared":
    by_date[d["date"]][d["video_id"]] = d

Disappeared markers record an absence, not a presence. Including them when building the comparison snapshot inverts their meaning. The test that covers this:

def test_disappearance_is_reported_once_not_every_day(self):
    # Day 1: three alive. Day 2: one vanishes and is marked.
    # Day 3: the marker must not read as "present on day 2" and re-fire.
    ...
    self.assertEqual(run.detect_disappeared(live, self.path), [])

I built this test after the fix rather than before, but it locks in the correct semantics.

Excluding deleted videos from archetype medians

A secondary issue: if a deleted video's status: "disappeared" row were included in the day-7 median calculation, it would drag medians down. The 11 videos that were deleted all sat at 0-25 views. A product_findindiegame Short at 12 views included in the archetype's day-7 median would look like a weak performer, when actually it never had a chance to accumulate views before disappearing.

This is survivorship bias in the reverse direction: not over-counting winners by excluding losers, but over-penalizing the archetype by including phantom underperformers.

day7_by_archetype() already has a guard for this:

if d.get("status") == "disappeared":
    continue  # deleted from YouTube — must not count toward any median

The comparison pipeline now looks like this:

Data path	What it tracks	Disappeared handling
`fetch_uploads()`	Local upload records (timestamps, IDs)	Unaware of YouTube deletions
`fetch_stats()`	Live YouTube Data API response	Only shows surviving videos
`append_history()`	JSONL per (date, video_id)	Records `status: "disappeared"` rows
`detect_disappeared()`	Diffs live vs snapshot	Fires :⚠️: and report table
`day7_by_archetype()`	Day-7 view medians by archetype	Skips `status: "disappeared"` rows

What I'd change about the original design

Adding output-side monitoring from day one would have caught this. The simplest form is a daily count check: fetch channel.statistics.videoCount from the Data API and compare it against your own count of known-uploaded IDs. A drop of more than 1-2 between runs is an anomaly worth alerting on.

The auto-tuner's archetype directive system runs daily and produces a directive that the video generation routine consumes. That directive is built on the day-7 median — so a corrupted median directly produces a wrong archetype bias. I'd caught one survivorship-bias version of this problem two days earlier. This was the inverse arriving from an unexpected direction.

Input-side health checks give you confidence that your automation ran correctly. Output-side checks give you confidence that what you produced still exists. For a publishing pipeline, you need both. The pipeline health checks I had were comprehensive on the input side and completely blind on the output side. That's the structural gap this detection closes.

FAQ

Does this distinguish between deleted and unlisted/private?
In this case, yes — the YouTube Data API omits videos entirely when they're deleted. Private videos owned by the authenticated account appear in response if you're using OAuth; public videos that go private return no results on a public key request, same as deleted. For this pipeline the distinction didn't matter: all 11 were confirmed deleted via oEmbed returning 404 and the Data API omitting them with a public key.

What if the API itself has an outage and returns fewer videos?
detect_disappeared() doesn't distinguish between a partial API response and a real deletion. If the API returns 60 of 79 videos, 19 will be flagged as disappeared. For now I treat this as an acceptable false positive rate — the :⚠️: annotation requires manual confirmation, and a full-channel API outage is obvious from the report. Adding a sanity check on the returned video count (is it within 10% of yesterday's) would reduce noise.

Should the disappeared rows ever be pruned from history?
Not for now. They're useful as an audit trail and the file is small. If the history grows past a few MB, pruning rows older than 90 days while keeping disappeared markers would be the right call.

What happened with the 11 deleted videos?
Both publish schedules are paused pending a YouTube Studio check for a policy notice. The pipeline never received one. The channel itself is healthy (public, 79 videos). I don't have a confirmed explanation yet — I'll post an update when I do.

Part of an ongoing 6-month experiment running three AI-curated directory sites. The technical claims here are real; this article was AI-assisted.

Three things I learned designing a shared context file for automated routines

MORINAGA — Tue, 11 Aug 2026 06:45:14 +0000

I run four GitHub Actions cron routines that all need to know the same thing: which game subjects have live search demand right now. The YouTube Short generation routine picks matchup titles from it. The Bluesky post queue uses it to weight the gaming slot. The article generation routine uses it to anchor VoC topics. A market monitoring sweep updates it.

For the first month of this setup, each routine had its own baked-in assumptions. The YT script used a hardcoded list of "safe" matchup pairs. The Bluesky routine had a separate file with a different classification of "hot" subjects. The article routine ignored market signal entirely. Every system was drifting independently, and updating any one of them required touching multiple files.

The fix was one shared markdown file — docs/market-trends.md — with three design rules that I got progressively more right over time.

Rule 1: counted numbers only, every claim traceable

The first version of this file was just notes. "Underdog narratives seem to do well." "Players talk about price gaps." Useful to write down, useless to automate against. A routine reading that file has no way to prefer one subject over another when the file's evidence is impressionistic.

The TERUS method — every number is a count from a specific source, every source is saved raw — forced the file into something a routine can act on. "3.1M views, 'Indie Dev Beat AAA Studios With 0 Marketing'" is something a routine can compare against "5K views, most recent build-in-public short." An impression is not.

This also matters for the author. When you have to write a specific count, you have to actually count. You can't hand-wave "players care about X" when the claim has to be backed by "I read 43 comments and 11 mentioned X." The discipline of traceable numbers improved the file's accuracy more than any review process would have.

Rule 2: fail-closed freshness gate

The second version of the file had a format problem: it could be months old and none of the consumers would know. The YT script would anchor on "hot subjects" that had been stale for two months, and the routine would behave as if the data were current.

The fix was a updated: date field at the top of the file and a freshness rule that consumers must check before using the file's content:

updated: 2026-08-10

Freshness rule (fail-closed): if updated: is more than 14 days
old, consumers MUST ignore this file and fall back to their own defaults —
stale trends are worse than no trends.

Fail-closed here means: if the file is stale, the routine ignores the market-trends data and uses conservative defaults (established evergreen matchups, not trend-chasing). It doesn't silently use the stale data, and it doesn't crash. The 14-day window matches the social-listening sweep cadence — I update the file weekly when things are moving, less often when they're stable.

The key design choice is making freshness the consumer's responsibility rather than the publisher's. The consumer checks the updated: date on every run. This means a routine added six months from now will inherit the freshness check automatically if it follows the documented pattern — I don't need to update a registry or notify consumers when the file goes stale.

Rule 3: separate what's measured from what's prescribed

The third version made the biggest difference: splitting the file into an observations section (what I measured) and a prescriptions section (what consumers should do with it).

The original file mixed these together: "Underdog narratives get high views, so use them." That forces consumers to interpret the claim, and different routines interpret it differently. The Bluesky routine might take "use underdog narratives" to mean write underdog captions. The YT routine might take it to mean use underdog-narrative titles. Neither is wrong, but they diverge over time.

The current structure separates the measurement table from the action directive:

Observations section: the raw data table — video format, measured view counts, evidence notes. This is what I actually found. Consumers can read this to understand the evidence.
Hot subjects list: the prescriptions — specific matchup subjects, anchor candidates, and what-to-avoid notes. This is what the routine should act on. The YT script reads this section and picks from the "anchor OK" rows; the Bluesky routine reads it to weight game subjects for the 70% gaming slot.

The separation means I can update the observations without changing any consuming routine's behavior. If I notice something new in the measurement data, I can add it to the observations section without touching the prescriptions — the prescriptions only change when I've decided what action to take based on the evidence.

What I'd do differently

Build the freshness check first. I spent a month using stale market data because it felt like a future concern. It wasn't — the first time I updated the file and forgot to check whether the routines were actually reading the new version, I lost a week of YT anchor selection to outdated data.

The other thing: write the consuming routine's expected behavior in the file itself. docs/market-trends.md now has a "Who reads this (mandatory)" section that names each consumer and what it should do with the file. When I added the article routine as a fourth consumer, I had a clear template for how to wire it in — and a record of what the existing consumers were doing, so I could check for overlap or contradiction.

A shared markdown file with a freshness gate and a traced-evidence discipline is a long way from a config service, but for a four-system automation project running on a $25/month budget, it's the right level of infrastructure.

Part of an ongoing 6-month experiment running three AI-curated directory sites. The technical claims here are real; this article was AI-assisted.

How I fixed survivorship bias in my YouTube analytics by logging all videos

MORINAGA — Tue, 11 Aug 2026 06:45:09 +0000

My YouTube analytics script ran for three months and reported two tables every day: Top 5 videos by views-per-day and Bottom 5. I used those tables to decide which video archetypes to produce the next day. It seemed reasonable — high performers tell you what's working, low performers tell you what isn't.

The problem, which I finally wrote down explicitly during a directive audit this week, is that Top5/Bottom5 is exactly the wrong sample for comparing archetypes. You're always selecting on the outcome. That means you can't compute a real median for any archetype, can't tell whether a typical video of type A outperforms a typical video of type B, and can't run any form of hook A/B experiment. All you can say is which videos happen to be at the distribution's edges on a given day.

The original report and why it felt informative

The daily cron fetched all video stats from the YouTube Data API v3, computed views-per-day for each video older than 24 hours, sorted by that rate, and wrote the top 5 and bottom 5 into a markdown report in docs/yt-analytics/. The script then read those tables to build a "prefer X, avoid Y" bias hint for the next day's video generation directive.

This caught obvious failures. The build_in_public archetype really did crater — I could see it collapsing from a median around 34 views-per-day to around 8 over about three weeks. But obvious failures are easy. The harder question is: among archetypes that aren't catastrophically bad, which one has a better median? That's exactly what Top5/Bottom5 cannot answer.

Here's why. If product_findindiegame is the dominant archetype — which it is, the directive has favored it for weeks — then it will appear more often in both tails just because there are more of them. A Top5 table with three findindiegame entries and two unknowns tells me nothing about whether findindiegame is better than, say, ossfind. I need the middle of the distribution, not the edges.

The tables are also silent about new archetypes with only 2-3 videos. Those videos rarely hit Top5 or Bottom5 because neither cluster is large enough to dominate. I could run a new archetype for a month and have no data on it at all from this report.

What a valid comparison requires

To compare archetypes fairly, I need three things:

Approach	What you see	What you miss	Good for
Top5/Bottom5 daily	Extreme performers	Median distribution	Catching catastrophic failures
views/day, all videos	Current rate, full fleet	Age confound — a 2-day Short looks worse than a 20-day one	Quick daily snapshot
Full-fleet JSONL + day-7	All videos at age ~7 days	Nothing — full population	Archetype A/B, hook arm tests

The age problem matters: YouTube Shorts plateau fast. A fresh video has a high views-per-day because its 24-hour velocity is being divided by 1 or 2 days. A 30-day-old video has a low rate even if it accumulated more total views. Comparing views-per-day across videos of different ages is structurally biased toward new content.

Age-controlled: observe every video at the same point in its lifecycle. Day 7 is a good choice because Shorts on this channel reach roughly 99% of their lifetime views by that point — the day-7 number is a close proxy for total views, without the age confound.

To get day-7 observations, I need a snapshot of every video's view count taken seven days after it was published — which means logging all videos daily so I can find the snapshot closest to day 7 in post.

The implementation

The new append_history() function in scripts/yt-analytics/run.py does this logging:

def append_history(stats_videos: list[dict], history_path: Path = HISTORY_PATH) -> int:
    today = datetime.now(timezone.utc).strftime("%Y-%m-%d")
    seen_today: set[str] = set()
    if history_path.exists():
        for line in history_path.read_text().splitlines():
            try:
                d = json.loads(line)
            except ValueError:
                continue
            if d.get("date") == today:
                seen_today.add(d.get("video_id"))
    rows = []
    for v in stats_videos:
        vid = v.get("id")
        if not vid or vid in seen_today:
            continue
        s = v.get("statistics") or {}
        snippet = v.get("snippet") or {}
        rows.append(json.dumps({
            "date": today,
            "video_id": vid,
            "title": snippet.get("title", "")[:120],
            "published_at": snippet.get("publishedAt", ""),
            "archetype": v.get("_archetype", "unknown"),
            "hook_arm": v.get("_hook_arm", ""),
            "views": _stat_int(s, "viewCount"),
            "likes": _stat_int(s, "likeCount"),
            "comments": _stat_int(s, "commentCount"),
        }, ensure_ascii=False))
    if rows:
        history_path.parent.mkdir(parents=True, exist_ok=True)
        with history_path.open("a") as f:
            f.write("\n".join(rows) + "\n")
    return len(rows)

Idempotent per (date, video_id) — if the cron runs twice the same day, the second run writes nothing. One pass over the file to build seen_today, then append. No extra API quota: the full channel fetch was already happening; I just stopped discarding the non-tail rows.

The day7_by_archetype() function reads the history and finds each video's closest snapshot in the 6-to-8-day window:

def day7_by_archetype(history_path: Path = HISTORY_PATH):
    best: dict[str, tuple[float, int, str]] = {}  # vid -> (dist_from_7, views, archetype)
    for line in history_path.read_text().splitlines():
        d = json.loads(line)
        pub = (d.get("published_at") or "")[:10]
        date = d.get("date") or ""
        age = (datetime.fromisoformat(date) - datetime.fromisoformat(pub)).days
        if not 6 <= age <= 8:
            continue
        dist = abs(age - 7)
        cur = best.get(d["video_id"])
        if cur is None or dist < cur[0]:
            best[d["video_id"]] = (dist, d.get("views", 0), d.get("archetype", "unknown"))
    by_arch: dict[str, list[int]] = defaultdict(list)
    for _dist, views, arch in best.values():
        by_arch[arch].append(views)
    return sorted(
        ((a, statistics.median(vs), len(vs)) for a, vs in by_arch.items()),
        key=lambda x: x[1], reverse=True,
    ), len(best)

The 6-8 day window matters because the cron doesn't run at a precise offset from video publish time. A video might get its first post-launch observation at age 6.2 days or 7.8 days depending on publish time vs cron schedule. Taking the closest observation in that window gets the most age-accurate reading without requiring exact timing.

When the new metric takes over

Day-7 data is only trustworthy once there are enough samples per archetype. Two videos is not enough to compute a meaningful median:

DAY7_MIN_PER_ARCH = 3   # minimum observations per archetype
DAY7_MIN_ARCHES = 2     # minimum archetypes meeting that threshold

Until both conditions are met, strategy_ranking() falls back to the old views/day bias — which still has the age confound, but is less wrong than trusting a 1-2 sample day-7 median. The daily report shows which metric drove the decision, so I can see when the day-7 path has enough data to activate.

This is the same pattern as the frozenset guard that fixed the directive self-contradiction: use a known-good default until the data is genuinely ready, fail closed rather than making a decision with noise. A too-eager switch to the new metric would have replaced one bias (age) with another (sparse sample noise).

What this unlocks: the hook A/B experiment

The survivorship bias fix was a prerequisite for the hook A/B experiment, not an end in itself. The market sweep I ran (see docs/market-trends.md, updated 2026-08-10) found that the top-40 videos in this niche are almost all story-first or emotion-first titles — none of the niche's 1M+ performers lead with a data point. But that's a survivor-biased observation: maybe data-first hooks just fail and get filtered out before they accumulate enough views to appear in the sample. I can't tell from looking at the winners.

So I'm running a proper A/B split:

Even UTC day = CONTROL: number-first hook, as the original R7 rule specified
Odd UTC day = MARKET VARIANT: story/underdog-emotion hook (review count data cited inside the script as proof, not as the title's lead)

Each video in the generation queue now records "hook_arm": "control" or "hook_arm": "variant" at generation time. append_history() writes that field to the JSONL. After ≥20 videos per arm accumulate a day-7 observation, the ranking comparison between arms will be possible.

Without full-fleet JSONL, this A/B test produces no result. Top5/Bottom5 gives you too few samples per arm per period, and they're censored samples at that. The three approaches I use for silent failure detection also apply here: the cron now emits a history: appended N rows line that monitoring can check against zero to catch days where the full-fleet fetch silently returned nothing.

What I'd do differently

Start the JSONL on day one of the channel. I have approximately three months of daily analytics runs where only the top and bottom five videos made it to disk. That data is unrecoverable — the YouTube Data API does not expose historical view counts at past timestamps. I know that product_findindiegame outperformed build_in_public by roughly 4x in that period based on the Top5/Bottom5 tables, but I can't get the full distribution shape or the hook-arm breakdown because I didn't log it.

The other thing I'd do differently is index the JSONL into a proper database from the start. The current implementation reads the entire history file on every run to build seen_today. It's fast at the current scale — a few hundred rows — but a growing channel would slow this down linearly. Turso's free-tier limits shaped my data model for the directory sites in a similar way; I chose JSONL here because it's zero-config and the channel is small, but a proper table indexed on (video_id, date) would make the day-7 window query O(log n) instead of O(n).

The deeper lesson is about what "informative" means in a report. Top5/Bottom5 is informative if your goal is to catch extreme failures quickly. It's not informative if your goal is to compare population medians. Those are different goals and require different data collection strategies. I conflated them for three months because the report looked like it was answering the question I cared about.

FAQ

Why day 7 specifically, and not day 14 or day 30?

Shorts on this channel plateau fast. The view accumulation curve is close to flat by day 5-6 for most videos; day 7 is safely past the plateau for nearly all of them. Using day 14 or day 30 would require waiting longer for the data to mature, and for the A/B experiment to produce a usable result with 20+ samples per arm, that could add 2-3 extra weeks of delay. Day 7 captures most of the signal with the shortest wait.

Does the 6-8 day window cause measurement noise?

Some. A day-6 snapshot will be slightly lower than a day-7 snapshot and a day-8 snapshot slightly higher. The window exists because exact-day-7 matching would miss videos where the cron ran before the 7-day mark. In practice the difference between a day-6 and day-8 reading is small relative to the difference between archetype medians — the noise is real but not dominant.

What happens to videos I published before the JSONL started?

They don't get a day-7 observation. The day7_by_archetype() function only returns videos with a snapshot in the 6-8 day window. Pre-JSONL videos won't appear in the archetype rankings. That's correct behavior — I don't want to impute or fabricate historical data. The system starts clean from the first JSONL entry.

When does the day-7 metric replace views/day in strategy decisions?

When DAY7_MIN_ARCHES (currently 2) archetypes each have at least DAY7_MIN_PER_ARCH (currently 3) day-7 observations. Both thresholds must be met simultaneously. The daily report shows the metric label that drove the decision, so it's transparent when the switchover happened.

Part of an ongoing 6-month experiment running three AI-curated directory sites. The technical claims here are real; this article was AI-assisted.

Three public HTTP APIs I read daily without registering for a key

MORINAGA — Mon, 10 Aug 2026 07:06:01 +0000

My daily trends fetch is a Node.js script that runs in GitHub Actions, hits three APIs, and writes a JSON file that feeds my X-drafts pipeline later in the day. None of the three sources require an API key. No OAuth flow, no dashboard signup, no rate-limit token to rotate.

That sounds trivial, but it matters for CI pipelines. Every API key stored in GitHub Secrets is a secret that can expire, a secret that has to be rotated when a team member leaves, a secret that creates a failure surface. Keyless reads eliminate all of that for the sources where it's possible.

Here are the three I use and the practical limits of each.

Hacker News Firebase API

The HN API is publicly documented on GitHub and hosted on Firebase. No key, no auth header, no rate limit published in the official docs.

const ids = await fetchJSON(
  "https://hacker-news.firebaseio.com/v0/topstories.json"
);
const items = await Promise.all(
  ids.slice(0, 20).map((id) =>
    fetchJSON(`https://hacker-news.firebaseio.com/v0/item/${id}.json`)
  )
);

This returns the top story IDs, then fetches each item. The item objects include title, url, score, and descendants (comment count). For my use case — grabbing the top 20 stories to identify what's trending in dev — that's everything I need.

The practical limit is latency. Fetching 20 items individually over Firebase takes 2–4 seconds depending on cold-start behavior. Parallelizing with Promise.all handles this fine in a script context. In a browser context, sequential fetches would be painful.

Firebase occasionally throttles aggressively for a few minutes if you hit it from many IPs in a short window. In four months of daily runs from GitHub Actions I've hit this once. The fix was a 500ms delay between item fetches, which is enough to avoid triggering it in practice.

dev.to public API

dev.to exposes a read-only articles endpoint that works without authentication for public content. The top=1 parameter returns articles sorted by recent reaction count.

const items = await fetchJSON(
  "https://dev.to/api/articles?per_page=12&top=1"
).catch(() => []);

The response includes title, url, public_reactions_count, comments_count, and tag_list on each article. The catch(() => []) is load-bearing: dev.to's API returns 503 occasionally, and swallowing that to an empty array lets the rest of the trends fetch continue.

One thing that surprised me: top=1 doesn't mean "top from the last 1 day". The documentation is ambiguous about the time window. In practice, articles from the previous two to three days appear in the results. For trending detection that's fine; for building a "published today" feed it would be wrong.

The API does not require auth for reading public articles. It has a rate limit — 10 requests per second per IP, per the docs — which I've never come close to hitting with a once-daily run.

Reddit .json endpoints

Reddit's most underused feature is the .json suffix that works on almost any listing URL. Append .json to a subreddit URL and you get the full listing data without OAuth.

const j = await fetchJSON(
  "https://www.reddit.com/r/programming/top.json?t=day&limit=10"
).catch(() => null);

The response structure is {data: {children: [{data: {title, url, score, ...}}]}}. Each child has title, url, score, num_comments, author, and selftext (body for self-posts).

The User-Agent header matters here. Reddit blocks requests without a user agent string, and returns 429 for user agents that look like bots making too many rapid requests. I use my-app/1.0 as the UA string; that's been stable across months of daily runs.

const r = await fetch(url, { headers: { "user-agent": "my-app/1.0" } });

Reddit's .json endpoint does not require an API key for public subreddit content. Accessing private subreddits or performing writes requires OAuth. For the reading use case — grabbing the top posts from r/programming and r/SideProject — no registration is needed.

The limit I hit in practice: the t=day (top of the day) filter doesn't always return 10 results early in UTC because the day is new and few posts have accumulated significant score yet. Running at 06:00 UTC I sometimes get 3–4 results instead of 10. Running at 22:00 UTC I consistently get the full batch.

What I do with the combined output

The script writes a single JSON file to content/trends/YYYY-MM-DD.json with all three sources merged:

const all = [
  ...hn.map((x) => ({ ...x, source: "HN" })),
  ...devto.map((x) => ({ ...x, source: "dev.to" })),
  ...reddit.map((x) => ({ ...x, source: `r/${sub}` })),
];
await writeFile(OUT, JSON.stringify({ date: today, items: all }, null, 2));

A later step in the same day's workflow reads this file and uses it to draft X posts via Claude. The trends file is the handoff artifact between the data-collection step (no API key, runs first) and the generation step (requires Anthropic API key, runs second).

That separation also means I can test the collection step in isolation without spending any API budget. The trends fetch fails silently on individual source errors — Reddit 503, HN throttle, dev.to rate limit — and still writes a file with whatever it managed to collect. Downstream steps get degraded input rather than a broken run.

Where the keyless approach breaks down

Not every useful data source has a public read endpoint. GitHub's search API requires auth past 60 requests per hour. HuggingFace's models API requires a token for private repos but is keyless for public models — I use the keyless endpoint for that too, but it's limited to what HF considers "public". Steam's storefront API has undocumented endpoints that work without auth, though the behaviors are inconsistent enough that I treat them carefully.

The three sources above — HN, dev.to, Reddit .json — have been stable for the four months I've used them. That's not a guarantee of permanence. Reddit has changed its API policies before; dev.to's read rate limits could tighten. The fallback behavior (catch to empty array) means the pipeline degrades gracefully rather than breaking when that happens.

Part of an ongoing 6-month experiment running three AI-curated directory sites. The technical claims here are real; this article was AI-assisted.

Notable this week: WeatherNext, Oracle OpenJDK AI ban, DeepMind reshuffle, Shieldstral

MORINAGA — Mon, 10 Aug 2026 07:05:57 +0000

Five things I bookmarked this week. They span a range — a climate AI model, an open-source policy move, a lab restructuring, an operational scraper account, and a new small moderation model. None of these appeared in last Saturday's notable releases post, which focused on agent frameworks and frontier models.

1. Google DeepMind restructuring: Demis Hassabis to Chair, Jeff Dean departs

HN score: 366, 494 comments on August 5 — one of the higher-comment threads of the week. Google announced that Demis Hassabis moves from CEO to Chair of Google DeepMind, and Jeff Dean is departing. The announcement came through Google's official blog. These transitions at frontier research labs tend to shift what gets prioritized over the following 12-18 months. Moving a founder to Chair often signals a pivot toward product or commercialization under new leadership — which says something about where DeepMind sits in Google's roadmap. I'm watching whether this changes DeepMind's stance on open weights; that's been a point of differentiation between DeepMind and the pure commercial labs, and it tends to get re-litigated when leadership changes. Source: Google blog.

2. DeepMind WeatherNext: breakthrough in cyclone forecasting

Score: 341, 106 comments on August 8. WeatherNext is DeepMind's new weather forecasting model with announced improvements on cyclone trajectory prediction. The specific framing — breakthrough on severe weather events rather than general forecast accuracy — is meaningful because cyclone prediction errors have historically had direct emergency-response consequences. I don't have a clear view yet on how it compares to ECMWF operationally; that comparison matters and I'll revisit once there's independent benchmarking. What seems defensible now: AI weather models crossing from "research interesting" to "potentially useful for emergency response" is a distinct capability threshold from the frontier LLM race, and worth tracking separately. Source: DeepMind blog.

3. Oracle bans AI-generated code from OpenJDK

Score: 317, 225 comments on August 7. Oracle prohibited AI-generated code contributions from being merged into OpenJDK, despite Larry Ellison's public claim that Oracle isn't writing its own code. The stated rationale is copyright uncertainty — the same underlying concern that made the Linux kernel's maintainers cautious. What's notable is the stance: Oracle is drawing a harder line than most major open-source projects, where disclosure requirements rather than outright bans have been the working consensus. Whether that reflects legitimate copyright concerns or something competitive is genuinely unclear to me. The operational consequence is interesting regardless: this forces contributors to prove code origin in ways that nobody has a reliable method for yet, and it will come up in every downstream OSS project that imports from OpenJDK. Source: Dealroom.

4. A year of fighting scrapers on a 1.5-million-page website

Score: 345, 405 comments on August 7 — the highest-comment thread I saw in this week's snapshot. The patronview.com post is a detailed operational account of what bot traffic looks like at scale: 99% non-human, the economics of blocking tools, what countermeasures work, and what doesn't. Running three programmatic sites with far fewer pages, I still need to plan for this. The finding I'm holding: even well-configured bot blocking doesn't reliably get you below roughly 30% non-human traffic on a large public site. The post covers Cloudflare Bot Fight Mode, rate limiting, user-agent heuristics, and honeypot traps. The honeypot trap section is the most practically useful — it's a method that scales without per-request compute cost. Source: patronview.com.

5. Mistral Shieldstral: 3B open-weights multimodal moderation model

Score: 249, 62 comments on August 4. Mistral released Shieldstral, a 3-billion-parameter open-weight model for multimodal content moderation. The architectural pitch is that it runs alongside a generation model — classifying or filtering inputs at inference time rather than as a separate API call to a third party. Small moderation models running locally matter for pipelines where routing content to a classification API creates cost or privacy issues. I haven't tested it in my ETL — my content pipeline ingests public model metadata rather than user-generated content — but it would be directly relevant for anyone building a UGC product where third-party classification isn't viable. Source: Mistral AI blog.

Part of an ongoing 6-month experiment running three AI-curated directory sites. The technical claims here are real; this article was AI-assisted.

What I learned building a pipeline health monitor that opens GitHub Issues automatically

MORINAGA — Sun, 09 Aug 2026 06:36:30 +0000

The trigger was a four-day gap I noticed in retrospect. The Bluesky queue had silently stalled because an API token expired mid-run, every daily workflow returned exit code 0, and nothing in GitHub Actions went red. The Bluesky QC gate was working correctly — it blocked a bad post, then the token expired before the next run, and after that the gate started rejecting everything because authentication was broken. Four days later I noticed the account had gone quiet.

The conclusion I took from that: job status is not the same as output progress. I'd built the three-site content pipeline assuming those were equivalent. The assumption was wrong.

The fix is scripts/pipeline-health.py, a Python script that runs every night in GitHub Actions at 23:30 UTC — after the day's content routines complete — and checks three separate signals. It uses the GitHub Actions REST API and the GitHub Issues API via urllib.request, so there are no dependencies beyond the Python standard library and the GITHUB_TOKEN that Actions injects automatically. When any signal fires, it opens a GitHub Issue with a structured diagnostic body. When everything clears, it closes the Issue automatically. The rest of the time, it's silent.

The three failure modes and why job status misses two of them

The script watches four content workflows by display name: yt-publish, Publish articles, yt-publish-longform, and Bluesky queue post. For each, it queries the GitHub Actions API for runs in the last 26 hours and reports any failure or cancelled conclusion. That catches the obvious failure: a job that crashed or was abandoned.

The obvious failure is not the dangerous one.

Signal 2 is no-progress detection. The YouTube publish workflow reads from a queue directory and exits 0 whether it published a video or found an empty queue — both cases look identical from CI. The script reads content/yt-queue/uploaded/ and checks the timestamp of the most recent file. If nothing in that directory is newer than 36 hours, video production has stalled even though no workflow failed. The same threshold applies to Bluesky: 36 hours without a logged post triggers the alert.

Signal 3 is queue staleness. The Bluesky queue is a directory of pending posts, and the script checks whether the oldest file in that queue is more than 14 days old. A post sitting in the queue for 14 days almost certainly missed its window — either a QC gate blocked it permanently, or the queue processor is stuck on it. Neither is visible from job status.

The three signals catch distinct failure classes:

Signal	What it catches	What it misses
Workflow failures	Crashed jobs, timeouts, config errors	Successful no-ops
No-progress (36h)	Stalled pipelines with clean CI	Bad-but-published content
Stale queue (14d)	Content backed up past its useful lifetime	Short-term stalls under the threshold

The gaps in coverage matter as much as the coverage itself. The health monitor doesn't read article content, doesn't verify publish quality, and doesn't know whether a video is good. The content quality gate handles article-level problems; the ETL health queries catch data-level stalls. The pipeline monitor only asks whether something shipped at all.

Why a single deduplicated Issue beats individual notifications

The first version opened a new Issue every time the watchdog detected a problem. Within two weeks there were 47 open Issues about variations of the same stalled pipeline. I stopped reading them.

The second version writes to one Issue. The script embeds  as a marker in the Issue body, searches for that marker on each run, and updates the existing Issue in place if found. No new Issue is opened while an alert is active; the body is replaced with the current diagnostic state and the timestamp of detection.

When all three signals clear — no workflow failures, progress markers are recent, queue isn't stale — the script calls the GitHub Issues API to close the open alert Issue. The next failure opens a new one. The effect: the Issue tracker contains at most one pipeline-health issue at any moment, and its open/closed state maps directly to pipeline health.

The closed-issue history becomes a free audit trail. Filtering closed Issues by the pipeline-health label shows exactly when the pipeline stalled and for how long, without any additional storage or logging infrastructure.

I considered Slack or email notifications instead. The problem with both: they create a parallel notification stream I have to check separately from my normal work. A GitHub Issue surfaces in the same place I track everything else. It auto-assigns priority by being open. When it closes, I don't have to manually mark anything resolved.

Implementation: stdlib only, no npm install, no external packages

The script uses only Python's standard library: urllib.request, json, os, datetime, sys. No requests, no PyGithub, no install step.

def _req(method, path, body=None):
    url = path if path.startswith("http") else f"{API}{path}"
    data = json.dumps(body).encode() if body is not None else None
    req = urllib.request.Request(url, data=data, method=method)
    req.add_header("Authorization", f"Bearer {TOKEN}")
    req.add_header("Accept", "application/vnd.github+json")
    req.add_header("X-GitHub-Api-Version", "2022-11-28")
    if data:
        req.add_header("Content-Type", "application/json")
    with urllib.request.urlopen(req, timeout=30) as resp:
        raw = resp.read().decode()
        return json.loads(raw) if raw else {}

The GitHub Actions step needs two permissions: actions: read to query workflow run history, and issues: write to open and close the alert Issue. GITHUB_TOKEN and GITHUB_REPOSITORY are provided automatically in the Actions context — no additional secrets setup.

The workflow runs once daily at 23:30 UTC, not on every push or every workflow completion. Running on a fixed schedule avoids double-alerting when a single stall causes multiple overlapping failures. The 30-minute offset after the top of the hour puts it after the nightly content jobs finish, which avoids flagging in-progress work.

on:
  schedule:
    - cron: "30 23 * * *"
  workflow_dispatch: {}

The workflow_dispatch trigger lets me run it manually to check state without waiting for the nightly run. This was useful while debugging the no-progress thresholds.

How I picked the thresholds

The 36-hour no-progress threshold came from failure data. The shortest real stall that caused content gaps was about 30 hours. The GitHub Actions cron timing bugs I'd hit previously mean the watchdog itself might fire off schedule. 36 hours gives one missed daily run plus a six-hour margin before it trips; 24 hours produced false positives in testing.

The 14-day queue staleness threshold mirrors QUEUE_MAX_AGE_DAYS in yt-publish.yml. The YouTube publisher drops files older than 14 days rather than publish them out of context. A file that old in the queue is one the publisher won't touch — it signals a stall that's gone past the point of self-repair.

The 26-hour window for checking workflow runs is slightly longer than a calendar day. This prevents a gap when the nightly run schedule drifts by a few minutes and a failure from the previous night's run falls just outside a 24-hour window.

I changed these thresholds once after the initial deployment: the Bluesky no-progress threshold started at 24 hours and produced a false alert after I intentionally paused posting for a day without updating the configuration. 36 hours now gives me room to pause for a day without triggering an alert.

What changed after I had this running

The monitoring changed how I think about the GitHub Actions free quota. When I was cutting workflow runs to stay under the free tier ceiling, I evaluated each workflow by its compute cost. The pipeline health monitor runs for about 20 seconds and queries the API a handful of times — nearly free in compute terms. But its value is asymmetric: 20 seconds of Actions minutes catches failures that would otherwise take me days to notice. I kept it without hesitation.

It also surfaced a pattern I hadn't noticed: the Bluesky JSONL queue stalls more often than the YouTube queue, and almost always from auth issues rather than content issues. The no-progress signal has fired three times total. Two of those three were Bluesky auth failures; one was an unrelated disk quota issue in the upload step. That distribution told me where to add better error handling, which the monitoring made visible.

The monitor doesn't eliminate production failures. It shrinks the detection window from "whenever I notice the feed went quiet" to "the morning after the failure occurred."

What this doesn't catch

Bad-but-published content. If a YouTube video uploads with corrupted audio, or a Bluesky post goes out with a broken image, the pipeline-health monitor doesn't know. It only asks whether something was published, not whether what was published was correct. That's a separate concern that belongs in the QC gates before publish, not in a post-hoc health check.

Cascade failures where the pipeline restarts fast enough that no 36-hour window opens. A job that fails and restarts within an hour — GitHub's built-in retry behavior — looks healthy from the no-progress signal even if it failed and retried five times.

Intentional pauses. If I decide to take a break from publishing, the monitor trips after 36 hours. I handle this by closing the alert Issue manually and ignoring it for the pause duration, then re-opening if things haven't resumed. There's a cleaner solution — a PAUSED_UNTIL environment variable the script could check — but I've hit this scenario infrequently enough that I haven't implemented it.

FAQ

Does this replace checking the GitHub Actions UI directly?

No. The UI shows job-level status and step logs, which the health monitor doesn't replicate. This adds a layer on top: watching outputs rather than jobs, and creating a persistent artifact (the Issue) that survives past the run.

Why Python instead of a Node.js script like the other monitoring tools in this project?

No strong reason. I wrote it during a session already in Python and urllib.request handles everything without an npm install step. A Node.js version would work equally well.

What's the QUEUE_MAX_AGE_DAYS threshold based on?

It mirrors the hard cap in the publisher: the queue processor drops files older than 14 days rather than publish stale content. A file that's been in the queue past that threshold is one the publisher won't handle automatically — the stall is permanent until a human intervenes.

Could GitHub's built-in failure notifications replace this?

GitHub can notify on workflow job failures. It can't detect no-progress signals or queue staleness. Those two signals are specific to my pipeline's output semantics and require code that understands what "progress" means in this context.

Why auto-close the Issue instead of requiring manual review?

Open issues accumulate and lose signal value. A closed issue with the self-resolution timestamp is more useful for retrospective analysis than an issue that requires someone to manually close it after confirming everything's working. If the pipeline is healthy, there's no issue to read; if it's not, there's exactly one issue that says what's wrong.

Part of an ongoing 6-month experiment running three AI-curated directory sites. The technical claims here are real; this article was AI-assisted.

Notable this week: NOOA, Prime Agent, Qwen3.8-Max, DeepSeek V4-Flash official

MORINAGA — Sun, 09 Aug 2026 06:36:25 +0000

Five things I bookmarked this week. Each one covers a different angle — agent frameworks, frontier models, and tooling — so there's some breadth here.

1. NVIDIA NOOA — agents as plain Python classes

NVIDIA Labs released NOOA (NVIDIA Object-Oriented Agents) on August 7 under Apache 2.0. The core design decision: an agent is a single Python class, not a YAML config or a JSON scaffold. Six harness capabilities — typed I/O, pass-by-reference, code as action, programmable loops, explicit state, and model-callable APIs — drive the design. LiteLLM handles model routing, so the same agent code runs against Claude, GPT-5, Ollama, or a local vLLM endpoint without changes. Install is pip install nooa (v0.0.8, alpha, Python 3.12–3.13).

The number they lead with — 82.2% on SWE-bench Verified using GPT-5.5, with half the token budget of prior state-of-the-art — is worth paying attention to. I'm more interested in whether the object-oriented model holds up outside coding tasks, where SWE-bench stops being useful signal. That's the question I'll be watching through the rest of the year. Source: MarkTechPost.

2. PrimeIntellect/prime-agent — self-improving RLM harness

Prime Agent dropped August 5, MIT license. The architecture is unusual enough to read past the headline. Rather than a tool-call loop, the agent runs inside a persistent IPython kernel. Context is a variable the agent actively manages; memory, skill definitions, and subagent specs are durable state it can update (CRUD) from within its own trajectory. They call this the Recursive Language Model (RLM) abstraction.

With Claude Opus 5, they report 95.5% on ARC-AGI-3, fractionally above the 95.4% human expert baseline. I'm generally cautious about single-benchmark numbers, but the architectural difference from ReAct-style agents is real enough that the technical report is worth reading rather than skimming. The repo is early-stage; the GitHub star count has been climbing fast since the August 5 announcement.

3. Qwen3.8-Max — 2.4T parameter MoE from Alibaba

Alibaba published Qwen3.8-Max on August 3 (MarkTechPost). Architecture is MoE: 2.4 trillion parameters total, 95 billion active per token. Context window is 1 million tokens. Native multimodal support is included.

Alibaba is pricing international API access at roughly 40% of Claude Opus 5 input and 24% output — aggressive positioning. Benchmark positions on launch: fifth Text Arena, second Vision Arena. Open weights are scheduled for release this week; I'll form a more concrete view once I can run it locally rather than through the API. The pricing point means it will probably land in my comparison ETL on aiappdex.com — I've been adding new frontier models to the pairwise compare pages as weights become available.

4. DeepSeek V4-Flash 0731 — official GA with re-post-trained agent checkpoint

DeepSeek moved V4-Flash from preview to official general availability on July 31 (MarkTechPost). Same 284B CSA+HCA backbone as the preview checkpoint; what changed is a re-post-training pass focused on agentic tasks. DeepSeek reports official V4-Flash beating V4-Pro-Preview across nine agent benchmarks. The model is MIT-licensed on Hugging Face and the API is at $0.14/$0.28 per 1M input/output tokens.

I switched my comparison ETL to the preview three weeks ago and the official checkpoint runs noticeably cleaner on instruction-following edge cases — specifically the cases where the preview would produce partial JSON before hitting a stop sequence. That single improvement is enough to keep it in my pipeline for the foreseeable future. The price point also makes it viable for high-volume extraction tasks where using a flagship model would be economically absurd.

5. addyosmani/agent-skills — 76k GitHub stars, production skills for coding agents

agent-skills by Addy Osmani (Engineering Lead at Google) crossed 76k GitHub stars this week and has been one of the fastest-growing developer tool repos of 2026. Twenty-four skills in plain Markdown — security review, migration strategy, testing discipline — compatible with Claude Code, Cursor, GitHub Copilot, and Codex. Install: npx skills add addyosmani/agent-skills.

What distinguishes it from other prompt template repos: the skills are specific about what to check, not vague about intent. Rather than "be thorough," the security-review skill names concrete vulnerability classes. I've been incorporating a few of the patterns into how I prompt Claude Code in my own pipeline. Worth reading even if you don't use the installer — the framing is useful standalone.

Part of an ongoing 6-month experiment running three AI-curated directory sites. The technical claims here are real; this article was AI-assisted.

DEV Community: MORINAGA

Three ETL failure patterns I now write into the output file, not just the logs

Pattern 1: sources_ok lives in the artifact, not the logs

Pattern 2: errors[] alongside the results

Pattern 3: the vacuous-true guard on .every()

What the three patterns give you together

What I learned about repeat-and-vote sampling for non-deterministic search results

Why a single YouTube search fetch isn't a signal

The repeat-and-vote approach

Median rank instead of position from a single fetch

Jaccard as a built-in diagnostic

Where this approach breaks down

FAQ

What 14 days of YouTube hook A/B parity data revealed—and what it's hiding

The test setup

The 14-day numbers

The confound

Why the outliers ended up in these arms

What I'm waiting for before drawing a conclusion

What the 2-day threshold is actually telling me

What I learned separating daily collection from weekly interpretation in a cron pipeline

The instrument's resolution problem

Why daily interpretation means publishing noise as strategy

Three design rules in the new collection layer

The interpretation layer stays weekly

What downstream consumers gained

What I'd do differently

FAQ

Related

Pausing a GitHub Actions cron: the yaml trap that breaks all workflow triggers

The mistake: commenting out the cron line but leaving the key

How the failure shows up

The fix: comment out the entire schedule key

The useful side effect: workflow_dispatch still works

Summary

How I detected deleted YouTube videos using JSONL history diffing

The blind spot: health checks only watch the input side

Building detect_disappeared() using JSONL history

The subtle bug: disappearance markers that look like presence

Excluding deleted videos from archetype medians

What I'd change about the original design

FAQ

Three things I learned designing a shared context file for automated routines

Rule 1: counted numbers only, every claim traceable

Rule 2: fail-closed freshness gate

Rule 3: separate what's measured from what's prescribed

What I'd do differently

How I fixed survivorship bias in my YouTube analytics by logging all videos

The original report and why it felt informative

What a valid comparison requires

The implementation

When the new metric takes over

What this unlocks: the hook A/B experiment

What I'd do differently

FAQ

Related

Three public HTTP APIs I read daily without registering for a key

Hacker News Firebase API

dev.to public API

Reddit .json endpoints

What I do with the combined output

Where the keyless approach breaks down

Notable this week: WeatherNext, Oracle OpenJDK AI ban, DeepMind reshuffle, Shieldstral

1. Google DeepMind restructuring: Demis Hassabis to Chair, Jeff Dean departs

2. DeepMind WeatherNext: breakthrough in cyclone forecasting

3. Oracle bans AI-generated code from OpenJDK

4. A year of fighting scrapers on a 1.5-million-page website

5. Mistral Shieldstral: 3B open-weights multimodal moderation model

What I learned building a pipeline health monitor that opens GitHub Issues automatically

The three failure modes and why job status misses two of them

Why a single deduplicated Issue beats individual notifications

Implementation: stdlib only, no npm install, no external packages

How I picked the thresholds

What changed after I had this running

What this doesn't catch

FAQ

Notable this week: NOOA, Prime Agent, Qwen3.8-Max, DeepSeek V4-Flash official

1. NVIDIA NOOA — agents as plain Python classes

2. PrimeIntellect/prime-agent — self-improving RLM harness

3. Qwen3.8-Max — 2.4T parameter MoE from Alibaba