DEV Community: takahiro hashito

Two checks that stop a bot from announcing a 404

takahiro hashito — Sun, 30 Aug 2026 14:41:11 +0000

Background

The most embarrassing thing an update announcement can do is link to a 404. A typo in a slug, or a URL that has not been deployed yet, and the link is dead.

When a bot posts those announcements, it does not get embarrassed. It just keeps producing broken links on schedule.

So the pipeline runs two independent checks before anything is posted. The important part is not either check on its own — it is that they answer two different questions, and are deliberately not merged.

How it works

The pipeline runs the two checks at two different moments, as sketched below.

[1] at generation time  -- does this URL exist?
      the site's sitemap.xml is the list of published URLs.
      a URL that is not in it never becomes an announcement.

[2] just before posting -- can I reach it right now?
      hit the URL over HTTP and check for 2xx/3xx,
      while distinguishing "broken" from "temporarily unreachable".

Checking only over HTTP is fragile right after a deploy: the page is correct, the propagation just has not finished. Checking only the sitemap cannot tell you that something has since gone missing. Each check covers the other's blind spot, so both run, for different reasons.

Implementation

The sitemap is the source of truth for existence

Every site emits sitemap.xml at build time — that is, by construction, the list of URLs that will be published. Matching against it needs no dependency and no XML parser:

const fs = require("fs");

function loadSitemapUrls(path) {
  const xml = fs.readFileSync(path, "utf8");
  const set = new Set();
  for (const m of xml.matchAll(/<loc>([^<]+)<\/loc>/g)) {
    set.add(m[1].trim().replace(/\/$/, "")); // normalize the trailing slash
  }
  return set;
}

function assertInSitemap(url, urls) {
  const key = url.trim().replace(/\/$/, "");
  if (!urls.has(key)) throw new Error("URL not in sitemap (may be unpublished): " + url);
  return url;
}

const urls = loadSitemapUrls("public/sitemap.xml");
assertInSitemap("https://example.net/articles/foo", urls); // throws if absent

Normalizing the trailing slash on both sides is what makes the comparison usable; a sitemap that writes /foo/ and a generator that writes /foo will otherwise disagree about every URL.

Put assertInSitemap at the entrance of the announcement generator and a post pointing at a non-existent URL becomes structurally impossible to build.

Reachability distinguishes "broken" from "unknown"

The pre-post check tries HEAD first, falls back to GET, and sorts the outcome into three buckets. Collapsing that into "anything but 200 is bad" throws away correct URLs during deploy propagation.

async function checkUrl(url, { timeoutMs = 8000, client = axios } = {}) {
  if (!/^https?:\/\//i.test(url))
    return { ok: false, definitive: true, reason: "invalid-url" };

  const opts = { timeout: timeoutMs, maxRedirects: 5, validateStatus: () => true };
  const evaluate = (status) => ({
    ok: status >= 200 && status < 400,
    status,
    definitive: status >= 400 && status < 600, // <- unambiguously broken
    reason: status < 400 ? "ok" : `http-${status}`,
  });

  try {
    const head = await client.head(url, opts);
    // codes that commonly mean "HEAD not supported" -- re-check with GET
    if ([403, 405, 501].includes(head.status)) return evaluate((await client.get(url, opts)).status);
    return evaluate(head.status);
  } catch (headErr) {
    try {
      return evaluate((await client.get(url, opts)).status); // HEAD failed -> one GET
    } catch (getErr) {
      // status unknown = possibly transient. never exclude permanently
      return { ok: false, definitive: false, reason: getErr.code || "network-error" };
    }
  }
}

How the result is used is the whole point:

result	meaning	action
`ok: true`	reachable, 2xx/3xx	safe to announce
`definitive: true`	4xx/5xx, clearly broken	exclude permanently
`definitive: false`	timeout or unknown network state	leave in the queue, retry next run

Keeping definitive separate is what lets genuinely dead URLs be dropped forever while propagation lag and brief outages get picked up on the next scheduled run.

The two paths are visible in the logs, which is exactly what you want when reviewing a run:

Skip posting (broken link http-404): https://example.net/articles/typo
Defer posting (url unreachable now: ETIMEDOUT): https://example.net/articles/new-post
Done. Posted 3/5 due tweets (skipped 2 for unreachable/broken links).

The first line is a permanent exclusion. The second is not — that item is still queued and will be tried again.

Gotchas

Trailing slashes. If the sitemap and the generator disagree about /, every comparison fails. Normalize both before comparing.
Do not let deploy lag leak into the sitemap check. Existence is answered by the build output; "is it up right now" is answered by HTTP. Keeping the two questions apart is what makes both of them stable.
HEAD is not universally supported. A CDN (the cache layer sitting in front of an origin) or a WAF (a web application firewall that filters requests) may answer 403, 405, or 501 to a perfectly good URL. Always keep a GET fallback.
Never exclude permanently on a timeout. Dropping a correct URL because of a transient failure means it is never announced again. That is precisely the case definitive: false exists to prevent.

There is one more trap in the same family, one directory over. A separate job compares the number of URLs in the live sitemap against what Search Console reports. Sitemaps can be index files pointing at child sitemaps, and if a child fetch fails, returning the partial total would silently show up as "Search Console is behind." So the counter returns nothing at all when any child cannot be read:

const children = locsOf(xml).slice(0, MAX_CHILDREN);
let total = 0;
for (const c of children) {
  const childXml = await getText(c, fetchImpl); // on failure -> catch (never return a partial count)
  total += locsOf(childXml).length;
}

Same principle as definitive: false: an incomplete answer must not be presented as a complete one.

The result

One of the sites whose update announcements run through both checks: https://manga.autoarticles.net

Wrap-up

Broken links in automated announcements mostly disappear once you do three things:

Guarantee existence with the sitemap, at generation time.
Confirm reachability over HTTP, immediately before posting.
Never merge the two.

On the reachability side, the split between "broken" and "unknown" is what makes the whole thing tolerant of deploy lag. It comes to well under a hundred lines in total, and after it, an embarrassing 404 announcement stops being something that can happen.

Serving a static site from a Cloud Functions seed bundle

takahiro hashito — Sun, 30 Aug 2026 14:40:54 +0000

Background

I run a handful of small static sites as a side project. All but one work the same way: a build script emits HTML, and firebase deploy --only hosting ships it.

One site is different. Its articles live in Firestore, and the HTML is assembled by a Cloud Function that pushes pages through the Firebase Hosting REST API. This post explains why I made that exception and what it cost me.

Some terms up front, since I will use them throughout:

Firebase Hosting — a CDN (Content Delivery Network: copies of your files are kept on servers around the world so visitors are served from a nearby one). Normally you upload a local directory to it.
Cloud Functions — serverless functions on Google Cloud. Two of them here are plain HTTP endpoints.
Firestore — a document database. One article equals one document.
seed bundle — when you deploy a Cloud Function, everything under functions/ is packed into the runtime. The article Markdown files ride along inside that pack.

How it works

The publish path has three stages, shown below. The Markdown files on the left are the ones I actually edit; everything to the right of them is derived.

functions/seed/articles/<slug>.md   (Markdown, in the repo)
        |
        |  (1) firebase deploy --only functions
        v
seed/articles/ inside the deployed function bundle
        |
        |  (2) HTTP: setupInitialData
        v
     Firestore "articles" collection
        |
        |  (3) HTTP: redeployHosting
        v
All pages rebuilt and shipped via the Hosting REST API

Nothing in that diagram is firebase deploy --only hosting. On this site that command is not merely useless — it would overwrite custom-delivered production with the contents of the local dist directory.

Step (1) is the one that surprises people. The import function reads Markdown like this:

const articlesDir = path.join(__dirname, "seed", "articles");

__dirname points inside the deployed runtime, not at your laptop. So the set of articles the function can see is frozen at the moment of the last firebase deploy --only functions. Dropping a new .md into your local checkout changes nothing on the function's side.

That is the central consequence of this design: because the article source lives inside a deployed artifact, every new article requires a function redeploy first.

Implementation

Why put Firestore in the middle at all? Because I want to rebuild every page at once.

Footer cross-links, structured data, breadcrumbs, OGP (Open Graph Protocol: the meta tags that decide the title, description, and image shown when a link is shared on social media) — none of these mean anything unless they are consistent site-wide. If each article owns a frozen HTML file, changing a shared fragment means regenerating everything, and you still have to decide where that regeneration happens.

With the articles in Firestore, redeployHosting reads "all articles as of now" and rebuilds every page from that. No local build output is involved.

There is a second trap. Cloud Functions only bundles what is under functions/. A shared config sitting at the repo root will require fine locally and throw MODULE_NOT_FOUND in production. I sync it in a predeploy hook:

"functions": {
  "source": "functions",
  "runtime": "nodejs22",
  "predeploy": [
    "node tools/copy-fleet-sites.js"
  ]
}

Static assets follow the same logic and are listed explicitly:

const STATIC_ASSETS = {
  "og-image.png": "image/png",
  "og-image.svg": "image/svg+xml",
  "favicon.svg": "image/svg+xml",
  "apple-touch-icon.png.svg": "image/svg+xml",
};

Production consists of the Firestore articles plus exactly this list. A file that is not listed here does not exist in production. Before I wrote that map, og:image returned 404 on every page — the file was sitting in the local directory that custom delivery never reads.

Gotchas

The worst one was duplicate slugs.

The slug is the Firestore document ID. Two articles with the same slug means the second import overwrites the first, and the overwritten article loses its public URL and starts returning 404.

What makes it nasty is that the build cannot catch it. The static site generator reads src/content/blog; the delivery path reads functions/seed/articles. The build stays green while production quietly loses an article.

So I put the check at the entrance to delivery, not in the build:

node build-and-deploy.js --seed
  → npm run test:slugs        (duplicates stop everything here)
  → firebase deploy --only functions
  → setupInitialData

Guarding the build would have been pointless: you can skip the build and call the import directly, and any check that can be bypassed will be.

The reverse problem exists too. Deleting a Markdown file does not delete the Firestore document, so renamed or merged articles leave orphans serving stale URLs. The import reports slugs that exist in Firestore but not in the seed; deletion only happens when explicitly requested. One line guards it:

if (!seedSlugs.size) return [];  // never treat "could not read the seed" as "every article is an orphan"

If reading the seed fails and you trust the resulting empty set, every article in Firestore looks orphaned. Combine that with the delete flag and the site loses all of its content. A failed read is not a result of zero.

Result

The site running on this setup: https://gadget.autoarticles.net

Takeaway

One thing generalizes from all of this.

When the single source of truth lives inside a deployed artifact rather than on your dev machine, a multi-step publish flow is not bad engineering — it is the direct consequence of that choice.

The steps are not the problem. The problem is papering over them with a runbook that a human has to remember. Trying to shorten the flow by reaching for firebase deploy --only hosting is worse than useless here: it overwrites custom-delivered production with whatever is in your local dist.

The right move is to collapse the steps into a single command and put the checks at its entrance. The bar I hold this to is simple: it should not break when nobody remembers the procedure.

Why my announcement bot posts through an IFTTT webhook instead of the X API

takahiro hashito — Mon, 24 Aug 2026 16:00:11 +0000

Background

I wanted the sites I run to announce their own updates on X. Small bot, simple job.

I started out planning to use the X API. After looking at the free-tier limits, the app review, and the ongoing token rotation, I ended up posting through an IFTTT webhook instead. This post is about that trade-off, and about the link-check gate that runs immediately before every post.

How it works

Generation and posting are two separate programs, and the operating system's scheduler sits between them.

Generation: a Node script turns the day's updates into announcement text plus a link, attaches a scheduled time, and appends it to a local queue.json.
Posting: launchd (the macOS service manager that starts jobs on a schedule) wakes periodically, picks the queued items whose time has arrived, and POSTs them to an IFTTT webhook. An IFTTT applet receives that and posts to X.

queue.json stays local and is not committed. Splitting generation from posting is what makes the rest of this manageable — the queue is the only interface between them, so the transport at the far end became replaceable.

Implementation

Posting is one JSON POST

Instead of calling the X API, the bot POSTs to an IFTTT Maker webhook. Authentication is a key embedded in the URL, so there is no OAuth token to refresh. IFTTT maps value1 to the body and value2 to an image URL.

async sendTweet(tweetText, imageUrl = null, webhookUrl = null) {
  const url = webhookUrl || this.baseUrl;      // per-account webhook can override
  const payload = { value1: tweetText };
  if (imageUrl) payload.value2 = imageUrl;      // attach the og:image
  const res = await this.client.post(url, payload, {
    headers: { "Content-Type": "application/json" },
  });
  if (res.status !== 200) throw new Error(`IFTTT failed: ${res.status}`);
  return { success: true };
}

Allowing the webhook URL to be swapped per account means one posting path serves both a general account and several genre-specific ones. Queue entries carry an optional webhook and fall back to the default.

A link check immediately before posting

The worst failure mode in an automated announcement is linking to an article whose deploy has not propagated yet — a 404, posted confidently. So reachability is verified right before the post. HEAD first, falling back to GET when a server refuses or does not support it.

async function checkUrl(url, { timeoutMs = 8000, client = axios } = {}) {
  const opts = { timeout: timeoutMs, maxRedirects: 5, validateStatus: () => true };
  const evaluate = (status) => ({
    ok: status >= 200 && status < 400,
    status,
    definitive: status >= 400 && status < 600,   // permanently broken
    reason: status < 400 ? "ok" : `http-${status}`,
  });
  try {
    const head = await client.head(url, opts);
    if ([403, 405, 501].includes(head.status)) {  // common "HEAD unsupported" codes -> retry with GET
      return evaluate((await client.get(url, opts)).status);
    }
    return evaluate(head.status);
  } catch {
    // network error / timeout -> status unknown. never exclude permanently
    return { ok: false, status: null, definitive: false, reason: "network-error" };
  }
}

The point is separating "definitely broken" (4xx/5xx) from "temporarily unknown" (network failure). The first is dropped from the queue forever; the second is skipped this round and retried next time.

const res = await checkUrl(q.url);
if (!res.ok) {
  if (res.definitive) { q.brokenUrl = true; q.brokenReason = res.reason; } // permanent
  else { /* possibly transient -> retry on the next process run */ }
  continue;   // either way, not posted this round
}

Collapse those two and one blip permanently discards a perfectly good URL, which nothing will ever announce again.

Only what is due, and only once

The posting run selects items whose time has come and that have not gone out, recording posted URLs in posted.json:

const due = queue.filter(q =>
  !q.posted && !q.brokenUrl &&
  new Date(q.scheduledAt).getTime() <= now && !postedUrls.has(q.url));

This is what makes the run repeatable — running it twice does not post twice, because the second pass sees the entries already marked. It also handles a laptop that was asleep when a scheduled time passed: the next run simply finds several items due and sends them together.

Gotchas — which double as the reasons I left the X API

Credential operations disappear. The X API brings rate-limit monitoring, token expiry and re-issuance, and app review with it. For a small personal integration, a single webhook is less to keep alive.
Guard against double posting. A posted flag plus the set of already-sent URLs in posted.json makes a repeated run a no-op rather than a duplicate.
RunAtLoad=false. The posting job runs on StartInterval (30 minutes) only. With RunAtLoad set to true, an unscheduled post fires the instant the job is loaded — which happens on login, on reload, and every time you edit the job.
Accept what the webhook cannot do. Threads, quote posts, and fine-grained control are weaker than with the API. For announcements that was fine, so the trade was a thinner integration in exchange for fewer features.

The result

One of the sites whose updates are announced this way: https://gadget.autoarticles.net

Wrap-up

Even for something as simple as "site updated, tell people", the choice between calling an API directly and passing through a webhook changes the ongoing cost far more than the code does.

If durability matters more than capability, this shape held up well: split generation from posting behind a queue, verify link reachability immediately before posting (distinguishing permanent from transient failure), and let the OS scheduler own execution. The transport at the end becomes a detail you can swap without touching anything upstream — which is worth more than any individual feature the API would have given me.

Reordering an overflowing job queue does not shrink it. Splitting it does.

takahiro hashito — Mon, 24 Aug 2026 16:00:01 +0000

Background

I run about 20 small sites and update them automatically. A batch runs four times a day, decides which sites have work due, and then does only that work.

The decision itself lives in one script, whats-due.js. Its only inputs are tasks.json (task definitions), state.json (run state), and today's date. No network calls, no randomness, so the same inputs always produce the same answer. That determinism is what makes the whole thing safe to re-run after a crash.

But a correct decision and a workable decision are two different things. This post is about the second one: what to do when the number of due tasks keeps exceeding what a single run can finish.

How it works

The shape of the decision is this.

tasks.json + state.json + today(JST) + slot
        ──> whats-due.js ──>
{ anyDue, priorityOrder, staleWarnings, configWarnings, tasks: [ { id, site, due, reason, ... } ] }

Every task carries a quota.kind, and cadenceOf() folds those kinds into four cadences.

per-run (research-append / ranked-append / new-critical-cves) — runs every batch, because freshness is the point
per-day (min-articles-per-day and friends) — a daily quota; once met, the task reports done-today
periodic (periodic-days) — every N days
on-demand — never due in the daily run; the definition exists only so other tooling can look up the repo path and deploy command

state.json holds very little per site: lastRun, lastRunArticlesAdded, knownSlugs. Quota checks are just a comparison between lastRun and today.

That part was fine. The trouble started downstream.

Implementation

What was actually happening

On the morning of 2026-08-15, the run had 19 due tasks and finished 12 of them. Seven spilled over. The next morning, the same thing.

My first fix was a priorityOrder that sorted by starvation, longest-neglected first. It did nothing. Sorting changes which tasks get dropped; it does not change how many. With a per-run capacity of roughly five or six items and 19 items due, no ordering saves you. The problem was the count, not the order.

Fix 1: split the count into slots

I divided the day into four slots (morning 10:00, noon 14:00, evening 18:00, night 22:00) and made each task declare its slots in tasks.json. whats-due.js runs the normal quota logic first, then drops anything not assigned to the current slot:

for (const r of result) {
  const t = tasksDoc.tasks.find((x) => x.id === r.id);
  r.slots = (t && Array.isArray(t.slots) && t.slots.length ? t.slots : null) || SLOT_NAMES;
  r.slot = slot;
  if (r.due && !taskRunsInSlot(t, slot)) {
    r.due = false;
    r.slotDeferredReason = r.reason;
    r.reason = "other-slot";
    r.nextSlots = r.slots;
  }
}

The detail that matters is slotDeferredReason. The original reason for being due is preserved rather than overwritten, so the output alone explains why a task is absent from this run. Drop that field and, a day later, you cannot tell from the logs whether a task is broken or simply scheduled elsewhere.

Fix 2: count misses per run, not per day

The detection side had a matching hole. The old counter, consecutiveMisses, is derived from lastRun, which is a date. A per-run task touched in the morning sets lastRun to today, so dropping it that same night leaves the counter at zero. A task that is handled every morning and skipped every night is structurally invisible at day granularity.

So state.run.ledger now keeps a per-run record, and this counts how many recent runs in a row a task was skipped:

function missedRuns(state, taskId, opts = {}) {
  const runs = recentRuns(state, { excludeCurrent: true, ...opts });
  if (!runs.length) return null;
  let n = 0;
  for (const r of runs) {
    const t = r && r.tasks && r.tasks[taskId];
    if (t && ATTEMPTED.has(t.status)) break;
    n++;
  }
  return n;
}

Returning null rather than 0 for an empty ledger is deliberate. 0 reads as "never dropped", which quietly conflates "no data" with "healthy".

Ledger entries use four statuses: done, noop (looked, found nothing new), failed, and deferred. Collapsing noop into done makes every legitimate empty check show up as a warning until nobody reads the warnings any more. Pushing noop toward "not attempted" collides with the rule that a run must never end with zero output. Four values, then.

With that in place, the first sort key of priorityOrder moved from days to missedRuns:

const priorityOrder = result
  .filter((r) => r.due)
  .map((r, i) => ({ r, i }))
  .sort((a, b) => {
    const ma = a.r.neverRun ? Infinity : a.r.missedRuns || 0;
    const mb = b.r.neverRun ? Infinity : b.r.missedRuns || 0;
    if (mb !== ma) return mb - ma;
    const da = a.r.neverRun ? Infinity : a.r.daysSinceLastRun || 0;
    const db = b.r.neverRun ? Infinity : b.r.daysSinceLastRun || 0;
    if (db !== da) return db - da;
    return a.i - b.i;
  })
  .map((x) => x.r.id);

Anything dropped at night surfaces the next morning with missedRuns=1 and lands at the front, so the same set cannot be dropped twice in a row. Environments without a ledger get missedRuns=null, are treated as 0, and fall back to the old day-based ordering.

What the output looks like

The decision is reproducible with no side effects. --no-mark skips the start marker, so you can run it as often as you like without touching state.json.

node automation/scripts/whats-due.js --no-mark --date 2026-08-20 --batch noon

Here is one task from that output:

{
  "id": "menrui-daily",
  "site": "menrui",
  "due": false,
  "reason": "other-slot",
  "cadence": "per-run",
  "slots": ["evening"],
  "slot": "noon",
  "slotDeferredReason": "research-and-append",
  "nextSlots": ["evening"],
  "lastRun": "2026-08-20",
  "neverRun": false,
  "daysSinceLastRun": 0,
  "consecutiveMisses": 0,
  "missedRuns": 3,
  "attemptRate": { "attempted": 1, "of": 6 }
}

lastRun is today, so consecutiveMisses is 0 and the day-based view calls this healthy. The run-based view disagrees: three consecutive runs skipped, and only one attempt in the last six. That is exactly the shape day granularity cannot see.

Running the same command per slot and tallying due against other-slot shows what the split bought:

morning  due=6  other-slot=14
noon     due=5  other-slot=15
evening  due=5  other-slot=15
night    due=6  other-slot=14

Before slots, all 20 of those (the sum of both columns) were due in every run. Now each run gets five or six. Nothing was made less frequent: the daily total is preserved by the slots assignments, and sites that are supposed to update twice a day are assigned to two slots.

Gotchas

1. Configuration mistakes need their own exit, separate from the due check

Change a quota.kind from research-append to research-apend and the task falls into the unknown-kind branch, which safely sets due=false. So far so good. The problem was that staleWarnings filtered on r.due, so I measured a task sitting unrun for 207 consecutive days while both the warning list and the starvation list stayed empty and the process exited 0. One typo, and a task disappears without a sound.

Unknown kinds now always land in configWarnings:

const configWarnings = result
  .filter((r) => r.reason === "unknown-quota-kind")
  .map((r) => ({
    id: r.id,
    site: r.site,
    problem: "unknown-quota-kind",
    quotaKind: r.quotaKind,
    lastRun: r.lastRun,
    daysSinceLastRun: r.daysSinceLastRun,
    hint: "...",
  }));

Missing slots (silently means "every slot", so the count never drops) and unknown slot names (silently means "never due") go to the same place, for the same reason: neither is fixable until somebody sees it.

2. "Not part of the daily run" deserves a declared kind, not just enabled:false

Some sites are measured but carry no daily production quota. Excluding them with enabled:false alone means that re-enabling the entry silently adds daily work. A dedicated quota.kind: "on-demand" states the exclusion explicitly, doubles up with enabled, and keeps an intentional opt-out distinguishable from a typo in the output.

3. Make the start marker a side effect, not a step

"Record the start of the run" was step zero of the runbook. It was skipped on essentially every run, which meant the silent-failure detector never had anything to compare against. It now happens inside whats-due.js via markStart(). The decision always runs first, so making it a side effect of the decision makes it unskippable.

4. Read order matters around that marker

The state used for the ledger has to be re-read after markStart(). markStart() rewrites run.lastStart, so reusing the state object read at the top of the script makes "the current run" point at the previous one, which excludes the most recent run and undercounts misses by one.

5. Two machines need two sets of markers

Once runs alternate between two machines, a single pair of start/complete markers breaks: the next machine's start overwrites the previous machine's record. Either an interruption vanishes entirely, or a late completion marker gets matched against a different run and reports a false positive. Markers now live under state.run.hosts[<host>] and are compared within a host. The shared lastStart / lastComplete fields are still written for existing readers.

The result

One of the sites this keeps updated daily: https://manga.autoarticles.net

Wrap-up

An overflowing queue does not shrink when you reorder it. Split the count to match per-run capacity.
Keep the reason you deferred something (slotDeferredReason) in the output, so a smaller queue stays observable.
Count misses per run, not per day. Day granularity cannot see a task that is handled in one run and dropped in another.
Absent data is not healthy data. An empty ledger returns null, not 0.
Give configuration errors their own exit (configWarnings). Failing safe is not enough if failing safe is also failing silently.

Making the decision deterministic is only the starting point; the automation settles down only once the decision's output is sized to what one run can actually finish.

Enforcing a style rule with a linter that actually fails the build

takahiro hashito — Sun, 23 Aug 2026 21:24:11 +0000

Background

I run a fleet of static sites that publish new content every day, mostly unattended. One of the house style rules is simple: no emoji anywhere in our own copy.

That rule is impossible to hold by hand. A single site builds a few hundred HTML files, and emoji can slip into nav icons, button labels, <title>, the RSS feed, or JSON-LD (the JSON-formatted metadata embedded in a page to describe its structure to search engines). Nobody is going to review all of that before every deploy.

So I wrote emoji-lint, a check that exits 1 the moment it finds a single emoji. It sits in the pre-deploy gate, which means a failure stops that day's publish.

This post is not about the regex. It's about what happens when you put a failing check into real operation: you immediately discover the places where the rule must not apply.

How it works

The core is unremarkable. A regex holds the emoji code point ranges, the scanner walks each file line by line, and matching lines are reported as JSON.

const EMOJI_RE =
  /[\u{1F000}-\u{1FAFF}\u{2600}-\u{27BF}\u{2B00}-\u{2BFF}\u{1F1E6}-\u{1F1FF}\u{FE0F}\u{200D}\u{2049}\u{203C}\u{2122}\u{2139}]/u;

\u{FE0F} (variation selector) and \u{200D} (ZWJ) are in there because emoji are not always a single code point. Arrows and similar symbols used in ordinary technical writing are deliberately left out. Catch everything and the check drowns in false positives, at which point people stop reading it.

The interesting part came later. Three categories of content look exactly like a violation but must not be treated as one:

Verbatim quotes from other people
Real proper nouns whose official spelling contains a symbol
Passages where the emoji itself is the subject being explained

Delete the emoji in any of those and you break something more important than the style rule.

One term up front: "masking" here means replacing a range with spaces so the scanner cannot see it. Nothing is deleted from the file.

Implementation

Scope exclusions to an explicit marker, never to a CSS class

Quotes are excluded only when the element carries data-quote="verbatim".

function maskVerbatimQuotes(text) {
  let masked = 0;
  const out = text.replace(VERBATIM_RE, (m, open, tag, inner, close) => {
    masked++;
    // keep newlines, blank everything else -> line numbers stay intact
    return open + inner.replace(/[^\n]/g, " ") + close;
  });
  return { text: out, masked };
}

The obvious alternative is to exclude anything with a quote-ish class such as .quote. I rejected it. If a cosmetic class buys you an exemption, anyone who wants an emoji just adds that class, and the rule is gone. Excluding only on a marker that declares intent keeps the escape hatch narrow.

inner.replace(/[^\n]/g, " ") preserves newlines on purpose. Line numbers found in the masked text map straight back to the original file. Shrink the text and every reported line number is off, which makes the report useless for actually fixing anything.

Proper nouns can't be covered by a marker

The second case needed a different unit. Take the manga title ラブ★コン: the ★ (U+2605) is part of the official title. Rewrite it as ラブコン and the title is simply wrong.

The catch is that this string also expands into <title>, meta description, JSON-LD, and even href="/tags/ラブ★コン.html". An element-level attribute cannot cover all of that.

So for proper nouns, the unit of exclusion is the name itself:

{
  "names": ["ラブ★コン", "聖☆おにいさん", "うたの☆プリンスさまっ♪", "ニドラン♀"]
}

Those exact strings are blanked before the scan. There's a guard against the obvious abuse: an entry consisting only of symbols (a bare ★, for instance) is rejected with exit 2. Allowing "★ is always fine" would defeat the point of scoping to names.

Numeric character references were invisible to the scanner

The third finding wasn't an exception at all, it was a hole. emoji-lint scans raw HTML, so it could not see a single emoji written as 📚 (📚). The browser decodes it and renders the emoji, which produced the worst possible state: emoji live in production while the check reports green.

Measured: one site had 26 such instances in production (13 of them the 📚 in a favicon), another had 10 across 8 places in its Japanese and English pages (✓ ✓ and ✗ ✗). Both were reporting zero findings.

The fix is to decode, before scanning, only those numeric references whose code point falls in an emoji range. Structural references like <, >, and & are left untouched.

Order matters: decode first, mask second. Masking inspects tag and attribute structure, so restoring emoji entities beforehand does not disturb it. Reverse the order and an emoji written as an entity inside a quote escapes the exclusion and gets flagged.

Gotchas

Never exclude silently. This turned out to be the single most valuable decision.

Adding exclusions naturally makes the check easier to pass. If you can't see that it got easier, you eventually arrive at a check that inspects nothing while still reporting success. So every exclusion is counted in the output:

{
  "ok": true,
  "count": 0,
  "quotedSkipped": 0,
  "sampleSkipped": 0,
  "nameSkipped": 0,
  "namesUsed": {},
  "entityDecoded": 0,
  "namesUnused": ["ラブ★コン", "聖☆おにいさん", "ニドラン♀", "ニドラン♂"]
}

That is a real run against one site's public/ directory. count: 0 means no violations; the zeroes elsewhere mean this particular site has nothing to exclude. Had it printed nameSkipped: 117, you would know the check passed after skipping 117 spots. "Clean" and "looked at nothing" become distinguishable.

namesUnused runs the other direction: allow-list entries that never matched. Titles that are no longer published pile up, and a growing allow list drifts toward being an escape hatch, so this is the cleanup signal.

The payoff is measurable. Before the proper-noun handling, one manga-deals site reported 97 false positives. When 97 lines are permanently red, a genuine violation hiding among them is invisible; the check ran but did not function. Afterward: zero false positives, with exclusions limited to 117 occurrences across two registered titles, and genuine emoji detection intact.

One more near-miss. For the "emoji as subject matter" case, my first instinct was to blanket-exclude <code> and <pre>. I dropped it: decorative emoji that happen to land in a code block would sail straight through. The measurement backed this up. On one site, of 417 detections, 54 (across 8 files) were genuinely the subject matter and 160 were plainly decorative. Blanket-deleting and blanket-excluding were both wrong. Exclusion applies only where the emoji is the topic; icons, nav, and button labels stay violations.

The result

One of the sites this runs against: https://cve.autoarticles.net

Wrap-up

Enforce a style rule mechanically and it will collide with things that matter more than the rule: quotation accuracy, correct proper nouns, an article that still makes sense. You get three options. Refuse exceptions and ship wrong content. Let them leak through cosmetic markers and the rule quietly dies. Or scope exclusions to explicit declarations and always report the counts.

Pick the third and the check stops being a pass/fail light. It becomes a tool that tells you what it looked at and what it deliberately did not. If you intend to keep a failing check in the pipeline for years, that is the information you need. A check that excludes silently will eventually check nothing at all.

Don't validate the output. Validate that you were allowed to generate it.

takahiro hashito — Sun, 23 Aug 2026 21:24:03 +0000

Introduction

I run a site that collects overseas viewer comments about individual anime episodes, translates them, and publishes them. It updates automatically every day.

There is one failure mode that matters more than all the others: creating a page for an episode that has not aired yet.

If the page exists before the broadcast, there are no comments to put on it. But the heading "Episode 8 — overseas reactions" is already live. An empty page is recoverable. What is not recoverable is a pipeline that decides an empty page looks bad and fills it with something plausible. At that point invented sentences are wearing the face of real people.

This post is about the check that prevents that, and about the day it actually fired.

The overall shape

The obvious implementation is arithmetic on dates:

Take the air date of episode 1
Assume weekly broadcast
Count the weeks elapsed until today
Treat every episode up to that number as aired

That works for producing candidates, but it is not evidence that anything aired. A skipped week makes the real count lower. So does a recap episode. Calendar arithmetic never observes the broadcast; it only restates an assumption.

So I split the pipeline in two:

Candidate generation — weekly arithmetic, guessing which episodes are missing. Guessing is fine here.
Existence check — does an observation from outside my system exist for this episode? No guessing allowed here.

For the existence check I use the per-episode discussion threads on MyAnimeList (a large anime database; MAL from here on). One thread is created per episode after it airs, and the timestamp of the first post in that thread is readable.

Viewers post after watching. So that timestamp is external evidence that the broadcast happened. Better still, it lives in exactly the same place I fetch the comments from, so verification costs no additional data source.

The core of the implementation

The check is a subtraction between what I claim and what the outside world recorded.

/**
 * @param {string} expectedAired  the air date my data claims (YYYY-MM-DD)
 * @param {string} firstPostedAt  first post in MAL's thread for that episode (ISO 8601, UTC)
 */
function checkAirDate(expectedAired, firstPostedAt, limitDays = 5) {
  const expected = Date.parse(expectedAired + 'T00:00:00Z');
  const observed = Date.parse(firstPostedAt);
  const gapDays = Math.round((observed - expected) / 86400000);

  return {
    expectedAired,
    firstPostedAt,
    gapDays,
    limitDays,
    status: Math.abs(gapDays) <= limitDays ? 'ok' : 'mismatch',
  };
}

Running it on real values:

checkAirDate('2026-07-28', '2026-07-28T14:25:51.000Z');
// => { gapDays: 1, status: 'ok', ... }

checkAirDate('2026-08-23', '2026-07-28T14:25:51.000Z');
// => { gapDays: -25, status: 'mismatch', ... }

The arithmetic is trivial. What does the work is the asymmetry: the left operand is my claim, the right operand is an outside observation. Comparing two values from the same side of that line verifies nothing.

The five-day tolerance absorbs time zones and streaming delays. Japanese late-night slots shift the local date, and overseas releases can lag by days. Demanding an exact match would reject correct episodes.

Finally the per-episode results are aggregated:

const conclusive = episodes.every((ep) => ep.airDateCheck.status === 'ok');
if (!conclusive) return; // don't write any fetched comments

Every episode must pass, or nothing is written. Partial acceptance is wrong here: if some episodes disagree, the air-date data was produced incorrectly, which means the ones that appear to agree may only be agreeing by accident.

Where it bit me

The gate fired today, on my own code.

Episodes 4 through 7 were missing, so I created four page stubs and went to fetch comments:

"conclusive": false,
"mismatched": [
  "ep 4: first post 2026-07-28T14:25:51Z is -25 days from claimed air date 2026-08-23",
  "ep 5: first post 2026-08-04T14:26:23Z is -18 days from claimed air date 2026-08-23",
  "ep 6: first post 2026-08-11T14:24:02Z is -11 days from claimed air date 2026-08-23"
]

All four stubs claimed the same air date: today. The stub generator was defaulting the air date to the current date instead of leaving it unset.

This is the part worth keeping. That bug is invisible by inspection. 2026-08-23 is a well-formed date and a perfectly possible value; nothing about the record looks wrong. It only becomes wrong next to an outside observation, and then it becomes wrong with a number attached: minus twenty-five days.

After filling in the real air dates and re-running:

"conclusive": true,
"mismatched": [],
"totals": { "titles": 1, "episodes": 4, "reactions": 32 }

Thirty-two comments from the same threads, against eight while the run was failing — a failed run stops fetching early.

One more thing. The same check also prevents under-collection. The weekly arithmetic proposed episode 8 as a candidate, but MAL has no thread for episode 8 yet; the last observable one is episode 7. Arithmetic said 8, observation said 7. Observation wins.

The result

Here is the site this runs on: https://anime.autoarticles.net

Each episode page carries the poster's handle, a per-post URL, and the original English text alongside the translation. A translation on its own gives the reader no way to confirm that the quote is real. Making it confirmable is part of the same design decision.

Conclusion

Automated pipelines don't emit convincing falsehoods because the generator is too clever. They emit them because nothing in the system is able to say that a precondition stopped holding.

The stub generator that defaulted the air date to today raised no error at all. The value was present and well-formed. What stopped it was a single component comparing that value against a record from outside the system.

So put the gate before generation, not after it. Auditing generated prose for plausibility is a losing game, because plausible prose is cheap to produce. Verifying that you were allowed to generate — against a fact you did not author — is cheap and decisive.

And when you can't confirm the precondition, fail loudly rather than quietly shipping zero. "Nothing today" and "I could not check today" are different messages to the person reading the logs tomorrow, and that person is usually you.

Reordering an overflowing job queue does not shrink it. Splitting it does.

takahiro hashito — Sat, 22 Aug 2026 16:17:23 +0000

Background

I run about 20 small sites and update them automatically. A batch runs four times a day, decides which sites have work due, and then does only that work.

But a correct decision and a workable decision are two different things. This post is about the second one: what to do when the number of due tasks keeps exceeding what a single run can finish.

How it works

The shape of the decision is this.

tasks.json + state.json + today(JST) + slot
        ──> whats-due.js ──>
{ anyDue, priorityOrder, staleWarnings, configWarnings, tasks: [ { id, site, due, reason, ... } ] }

Every task carries a quota.kind, and cadenceOf() folds those kinds into four cadences.

per-run (research-append / ranked-append / new-critical-cves) — runs every batch, because freshness is the point
per-day (min-articles-per-day and friends) — a daily quota; once met, the task reports done-today
periodic (periodic-days) — every N days
on-demand — never due in the daily run; the definition exists only so other tooling can look up the repo path and deploy command

state.json holds very little per site: lastRun, lastRunArticlesAdded, knownSlugs. Quota checks are just a comparison between lastRun and today.

That part was fine. The trouble started downstream.

Implementation

What was actually happening

On the morning of 2026-08-15, the run had 19 due tasks and finished 12 of them. Seven spilled over. The next morning, the same thing.

Fix 1: split the count into slots

for (const r of result) {
  const t = tasksDoc.tasks.find((x) => x.id === r.id);
  r.slots = (t && Array.isArray(t.slots) && t.slots.length ? t.slots : null) || SLOT_NAMES;
  r.slot = slot;
  if (r.due && !taskRunsInSlot(t, slot)) {
    r.due = false;
    r.slotDeferredReason = r.reason;
    r.reason = "other-slot";
    r.nextSlots = r.slots;
  }
}

Fix 2: count misses per run, not per day

So state.run.ledger now keeps a per-run record, and this counts how many recent runs in a row a task was skipped:

function missedRuns(state, taskId, opts = {}) {
  const runs = recentRuns(state, { excludeCurrent: true, ...opts });
  if (!runs.length) return null;
  let n = 0;
  for (const r of runs) {
    const t = r && r.tasks && r.tasks[taskId];
    if (t && ATTEMPTED.has(t.status)) break;
    n++;
  }
  return n;
}

Returning null rather than 0 for an empty ledger is deliberate. 0 reads as "never dropped", which quietly conflates "no data" with "healthy".

With that in place, the first sort key of priorityOrder moved from days to missedRuns:

const priorityOrder = result
  .filter((r) => r.due)
  .map((r, i) => ({ r, i }))
  .sort((a, b) => {
    const ma = a.r.neverRun ? Infinity : a.r.missedRuns || 0;
    const mb = b.r.neverRun ? Infinity : b.r.missedRuns || 0;
    if (mb !== ma) return mb - ma;
    const da = a.r.neverRun ? Infinity : a.r.daysSinceLastRun || 0;
    const db = b.r.neverRun ? Infinity : b.r.daysSinceLastRun || 0;
    if (db !== da) return db - da;
    return a.i - b.i;
  })
  .map((x) => x.r.id);

What the output looks like

The decision is reproducible with no side effects. --no-mark skips the start marker, so you can run it as often as you like without touching state.json.

node automation/scripts/whats-due.js --no-mark --date 2026-08-20 --batch noon

Here is one task from that output:

{
  "id": "menrui-daily",
  "site": "menrui",
  "due": false,
  "reason": "other-slot",
  "cadence": "per-run",
  "slots": ["evening"],
  "slot": "noon",
  "slotDeferredReason": "research-and-append",
  "nextSlots": ["evening"],
  "lastRun": "2026-08-20",
  "neverRun": false,
  "daysSinceLastRun": 0,
  "consecutiveMisses": 0,
  "missedRuns": 3,
  "attemptRate": { "attempted": 1, "of": 6 }
}

Running the same command per slot and tallying due against other-slot shows what the split bought:

morning  due=6  other-slot=14
noon     due=5  other-slot=15
evening  due=5  other-slot=15
night    due=6  other-slot=14

Gotchas

1. Configuration mistakes need their own exit, separate from the due check

Unknown kinds now always land in configWarnings:

const configWarnings = result
  .filter((r) => r.reason === "unknown-quota-kind")
  .map((r) => ({
    id: r.id,
    site: r.site,
    problem: "unknown-quota-kind",
    quotaKind: r.quotaKind,
    lastRun: r.lastRun,
    daysSinceLastRun: r.daysSinceLastRun,
    hint: "...",
  }));

2. "Not part of the daily run" deserves a declared kind, not just enabled:false

3. Make the start marker a side effect, not a step

4. Read order matters around that marker

5. Two machines need two sets of markers

The result

One of the sites this keeps updated daily: https://manga.autoarticles.net

Wrap-up

An overflowing queue does not shrink when you reorder it. Split the count to match per-run capacity.
Keep the reason you deferred something (slotDeferredReason) in the output, so a smaller queue stays observable.
Count misses per run, not per day. Day granularity cannot see a task that is handled in one run and dropped in another.
Absent data is not healthy data. An empty ledger returns null, not 0.
Give configuration errors their own exit (configWarnings). Failing safe is not enough if failing safe is also failing silently.

Making the decision deterministic is only the starting point; the automation settles down only once the decision's output is sized to what one run can actually finish.

Enforcing a style rule with a linter that actually fails the build

takahiro hashito — Sat, 22 Aug 2026 16:17:06 +0000

Background

I run a fleet of static sites that publish new content every day, mostly unattended. One of the house style rules is simple: no emoji anywhere in our own copy.

So I wrote emoji-lint, a check that exits 1 the moment it finds a single emoji. It sits in the pre-deploy gate, which means a failure stops that day's publish.

This post is not about the regex. It's about what happens when you put a failing check into real operation: you immediately discover the places where the rule must not apply.

How it works

The core is unremarkable. A regex holds the emoji code point ranges, the scanner walks each file line by line, and matching lines are reported as JSON.

const EMOJI_RE =
  /[\u{1F000}-\u{1FAFF}\u{2600}-\u{27BF}\u{2B00}-\u{2BFF}\u{1F1E6}-\u{1F1FF}\u{FE0F}\u{200D}\u{2049}\u{203C}\u{2122}\u{2139}]/u;

The interesting part came later. Three categories of content look exactly like a violation but must not be treated as one:

Verbatim quotes from other people
Real proper nouns whose official spelling contains a symbol
Passages where the emoji itself is the subject being explained

Delete the emoji in any of those and you break something more important than the style rule.

One term up front: "masking" here means replacing a range with spaces so the scanner cannot see it. Nothing is deleted from the file.

Implementation

Scope exclusions to an explicit marker, never to a CSS class

Quotes are excluded only when the element carries data-quote="verbatim".

function maskVerbatimQuotes(text) {
  let masked = 0;
  const out = text.replace(VERBATIM_RE, (m, open, tag, inner, close) => {
    masked++;
    // keep newlines, blank everything else -> line numbers stay intact
    return open + inner.replace(/[^\n]/g, " ") + close;
  });
  return { text: out, masked };
}

Proper nouns can't be covered by a marker

The second case needed a different unit. Take the manga title ラブ★コン: the ★ (U+2605) is part of the official title. Rewrite it as ラブコン and the title is simply wrong.

The catch is that this string also expands into <title>, meta description, JSON-LD, and even href="/tags/ラブ★コン.html". An element-level attribute cannot cover all of that.

So for proper nouns, the unit of exclusion is the name itself:

{
  "names": ["ラブ★コン", "聖☆おにいさん", "うたの☆プリンスさまっ♪", "ニドラン♀"]
}

Numeric character references were invisible to the scanner

The fix is to decode, before scanning, only those numeric references whose code point falls in an emoji range. Structural references like <, >, and & are left untouched.

Gotchas

Never exclude silently. This turned out to be the single most valuable decision.

{
  "ok": true,
  "count": 0,
  "quotedSkipped": 0,
  "sampleSkipped": 0,
  "nameSkipped": 0,
  "namesUsed": {},
  "entityDecoded": 0,
  "namesUnused": ["ラブ★コン", "聖☆おにいさん", "ニドラン♀", "ニドラン♂"]
}

The result

One of the sites this runs against: https://cve.autoarticles.net

Wrap-up

Guaranteeing You Never Show an Expired Offer, by Design

takahiro hashito — Sat, 22 Aug 2026 13:41:12 +0000

Introduction

I run a handful of sites that publish time-limited deals. The hard part is not writing the articles. It is that published content turns false on its own, without anyone touching it.

"500 yen off, through August 10" is correct the day it is written. On August 11 the exact same string is misinformation. Not one byte of the file changed.

Cleaning up by hand does not work. On 2026-08-02 I walked my own sites and found an article about a Prime Day sale that ended on July 13 still displaying "listed as running through July 13". Strictly speaking that sentence is not a lie. But nothing on the page tells the reader the sale is over.

So I moved to a different shape: store the deadline as data, and compare it against today every time the page is rendered. This post is about the design reasoning. The implementation details live in a separate post.

The shape

Articles are a JSON array. One object per article; the deadline lives in metrics.deadline as an ISO 8601 date (YYYY-MM-DD).

data/articles.json          article data (metrics.deadline / ended)
        │
        ├─ build.js         compares against "today (JST)" at render time
        │                     0 = current / 1 = grey / 2 = ended
        │                     → dropped from listings, RSS, sitemap; article URL kept
        │
        └─ preflight        checks the GENERATED HTML for expired entries in listings
                              not generated yet → fail, never pass

Three points matter:

The deadline lives in the data, not in Japanese prose inside the body
The "is it over" decision happens at render time, not at write time
Whether the rule actually works is verified against the generated HTML

The core of the implementation

The end-of-sale test is this small:

// Ended if there is a deadline and it is in the past, relative to the survey date.
// An article with no deadline is never treated as ended (do not fabricate an ended state).
function isEnded(a) {
  if (a && a.ended === true) return true;          // a human marked it confirmed-ended
  const dl = a && a.metrics && a.metrics.deadline;
  return !!dl && daysUntil(dl) < 0;
}

Short, but three design decisions are packed in.

One. An article with no deadline is not treated as ended. Some deals never publish an end date. If you resolve "no deadline" to "probably over by now", you display an ended state for something that is still running. The general rule is: do not resolve an unknown toward a confident assertion.

Two. The human-set ended: true is checked first. This part was genuinely broken for a while. The old code decided purely on the deadline date, so a hand-written ended: true — meaning "I confirmed this ended early" — was read by nothing and silently ignored. There was effectively no way to pull a sale that ended ahead of schedule. The lesson: when you add an automatic decision, separately confirm you have not blocked the manual override path.

Three. The reference date is "today", not "the day I surveyed it". This was the highest-leverage call.

// Order by "today (JST)". Anchoring to the survey date produces an error whose direction is
// "the staler the data, the more in-period an expired deal looks", which keeps ended deals
// pinned near the top of listings.
const ORDER_TODAY_JST = new Date(Date.now() + 9 * 3600e3)
  .toISOString()
  .slice(0, 10);

Concretely. An article surveyed on July 10 with a deadline of July 13. If the reference date is the survey date, it still reads "3 days left" in August. The staler the article, the more current it looks. That is the worst possible direction for the error: neglected articles look the healthiest.

Anchor to today and the direction flips. Neglected articles sink faster. One line, and the decay runs the right way.

I ended up with three tiers:

// 0 = current / 1 = grey (possibly ended) / 2 = ended
function freshRank(a) {
  if (isEnded(a)) return 2;
  if (isStale(a)) return 1;
  return 0;
}
// Shared comparator: freshness tier → newest first → slug (keeps the build deterministic)
function byFreshness(a, b) {
  return (
    freshRank(a) - freshRank(b) ||
    String(b.date).localeCompare(String(a.date)) ||
    String(a.slug).localeCompare(String(b.slug))
  );
}

"Grey" means not confirmed over, but suspicious: either (1) the published deadline is past relative to today, or (2) no deadline was published and the last confirmation is more than 14 days old. Note the constraint on (2): an article whose deadline is published and still in the future never goes grey, no matter how old the confirmation is. The deadline is the source's published value, and my own staleness should not override it.

The third sort key is slug to keep the build deterministic. The same data always produces the same HTML, so any diff is evidence that data changed.

What tripped me up

Do not delete it. Keep it and say it ended.

At first I only dropped ended articles from the listings. But as noted above, anyone arriving at the URL directly still sees a body written on the assumption that the price is still good. Being gone from listings does not stop search traffic.

Returning 410 would hit indexing abruptly. So instead of removing, I keep the page and state the end explicitly.

function endedNoticeHtml(a) {
  if (!isEnded(a)) return "";
  const dl = a && a.metrics && a.metrics.deadline;
  const when = dl ? `（出典表記の期限: ${jday(dl)}）` : "";
  return `<p class="ended-note"><strong>このセールは終了しています</strong>${when}。` +
    `以下は終了時点までの記録で、掲載中の価格・割引率は現在のものではありません。</p>\n`;
}

The banner says: this sale has ended (source-stated deadline: ...); what follows is a record as of that point, and the listed prices and discount rates are not current. I also prefix the <title> with a marker for "ended" and downgrade the call-to-action from "see the sale price" to "see the current price". The body stays as written; only the frame is redrawn against today.

The same idea was needed for release announcements. "Goes on sale starting August 18, 2026" stays in the future tense on August 20. I found a few such cases per site. Rewriting the prose just goes stale again next week, so this is rendered from today as well. One caveat: a metrics.start of 2026-08 (no day) is not evaluated, because "on sale in August 2026" is not past during August.

The biggest trap was verifying the rule actually works.

Writing the exclusion tells you nothing about whether it fires. So a pre-publish check inspects the generated public/index.html for links to expired articles. Checking the artifact, not the data, is the point.

That check produced a false positive once. A naive substring match on the slug hit the JSON index used for on-site search, which intentionally retains ended entries. Measured on 2026-07-28: a site that was excluding correctly failed the check. Readers can only click links, so the check now looks for the slug appearing as an href.

One more. If public/index.html has not been generated, the check fails rather than passes. "Could not verify" is not "nothing wrong". Resolving a missing measurement to a pass means the check goes quietly dead the day someone reorders the build.

The result

One of the sites running this in production: https://beer.autoarticles.net

Compare the listing order against what you see when you open an expired article directly; the tiers described here show up as-is.

Summary

"Never show expired information" is decided by data structure and choice of reference date, not by operational diligence.

Store the deadline in one machine-readable field, not in prose
Decide freshness at render time, and anchor to "today" rather than "the survey date"
Do not resolve unknowns toward assertion (no deadline is not the same as ended)
When you add automation, check you have not blocked the manual override
Verify the exclusion against the generated artifact, and never let an unverifiable state pass

The third point is worth repeating: one line decides which way your content decays. Anchored to the survey date, neglected articles look the freshest. Anchored to today, they sink. Same mechanism, opposite outcome.

A snapshot-style ops dashboard with nothing but the Python standard library

takahiro hashito — Sat, 22 Aug 2026 13:41:01 +0000

Background

Once you run more than a handful of small static sites, you want one screen that answers "what is the state of everything right now" — analytics sessions, Search Console impressions, what got updated today, what the cloud bill is at.

So I built one. Each run of the collector produces a single snapshot, the server stores it, and history accumulates.

The constraint I set was that the server side gets no dependencies at all. No Flask, nothing installed. Just Python's http.server and sqlite3 handling ingest, storage and HTML delivery. I wanted deployment to be "copy the file, run python3 server.py", whether that is a small Linux box or my laptop.

How it works

Three pieces:

A collector (Node) runs twice a day at 10:00 and 22:00, gathers analytics, Search Console, per-site updates and cloud cost into one JSON object, and POSTs it to /api/ingest.
The receiver (Python http.server) stores that payload as one snapshot in SQLite.
The page reads the latest snapshot and renders bars in plain CSS, with a detail modal per site.

All the gathering logic lives in the collector. The receiver only authenticates, stores and retrieves — which is why it has needed almost no changes as sites were added.

Implementation

Dependency-free HTTP with token auth

Subclass BaseHTTPRequestHandler and implement POST /api/ingest. Writes always require a token in an X-Token header, read from an environment variable or a .ingest_token file sitting next to the script, so nobody can push snapshots from outside.

import json, os, sqlite3
from http.server import ThreadingHTTPServer, BaseHTTPRequestHandler

class Handler(BaseHTTPRequestHandler):
    def do_POST(self):
        if self.path != "/api/ingest":
            return self.send_error(404)
        if self.headers.get("X-Token") != TOKEN:      # writes need a token
            return self.send_error(403)
        n = int(self.headers.get("Content-Length", 0))
        payload = json.loads(self.rfile.read(n) or b"{}")
        sid = ingest(payload)
        self._json({"ok": True, "snapshot_id": sid})

ThreadingHTTPServer rather than the single-threaded one, so an ingest arriving while a page is being rendered does not block.

A successful push looks like this from the collector's side — the returned snapshot_id is the row that was just created, and it increments twice a day:

push ok -> snapshot_id=101  (sites=23 / GA=true / SC=true / GCP=130 JPY / tweets_today=19)

A normalized schema keyed by snapshot id

Dropping the whole POST body into one JSON column is tempting and costs you later: every chart has to parse it again. Instead, one row in snapshots, and every other concern joined to it by snapshot_id — per-site metrics, per-service cloud cost, the day's updates. Time-series aggregation then stays pure SQL.

def init_db():
    c = db()
    c.executescript("""
        CREATE TABLE IF NOT EXISTS snapshots(
            id INTEGER PRIMARY KEY AUTOINCREMENT,
            ts TEXT, ts_jst TEXT, gcp_cost TEXT, meta TEXT);
        CREATE TABLE IF NOT EXISTS site_metrics(
            snapshot_id INTEGER, site_key TEXT, label TEXT,
            ga_sessions_y REAL, ga_pv_y REAL,
            sc_clicks REAL, sc_impressions REAL, sc_ctr REAL, sc_position REAL);
        CREATE TABLE IF NOT EXISTS gcp_services(
            snapshot_id INTEGER, service TEXT, cost REAL);
        CREATE INDEX IF NOT EXISTS idx_sm_snap ON site_metrics(snapshot_id);
    """)
    c.commit(); c.close()

def ingest(p):
    c = db(); cur = c.cursor()
    cur.execute("INSERT INTO snapshots(ts,ts_jst,gcp_cost,meta) VALUES(?,?,?,?)",
                (p.get("ts"), p.get("ts_jst"), p.get("gcp_cost"),
                 json.dumps(p.get("meta") or {})))
    sid = cur.lastrowid
    for s in p.get("sites", []):
        cur.execute("INSERT INTO site_metrics VALUES(?,?,?,?,?,?,?,?,?)",
            (sid, s.get("key"), s.get("label"), s.get("ga_sessions_y"),
             s.get("ga_pv_y"), s.get("sc_clicks"), s.get("sc_impressions"),
             s.get("sc_ctr"), s.get("sc_position")))
    c.commit(); c.close(); return sid

Metrics with a settled shape go in columns; loose extras go in the meta JSON column. That split keeps the schema firm without making every new field a migration.

With that layout, the trend chart is one query — total sessions per snapshot, most recent 60:

SELECT s.id sid, s.ts_jst tsj, COALESCE(SUM(m.ga_sessions_y),0) tot
FROM snapshots s
LEFT JOIN site_metrics m ON m.snapshot_id = s.id
GROUP BY s.id ORDER BY s.id DESC LIMIT 60

Gotchas

Positional inserts break the day you add a column

The INSERT INTO site_metrics VALUES(?,?,?,...) above is the version that eventually bit me. Adding one column to the collector made every push fail with table has N columns but M values supplied. The comment now sitting above that statement:

# Always INSERT into site_metrics with explicit column names.
# With positional `VALUES(?,?,...)`, adding a single column makes every push
# fail with "table has N columns but M values supplied" -- an operational
# incident, and worse than breaking quietly would have been.
SITE_METRIC_COLS = ["snapshot_id", "site_key", "label", "url",
                    "ga_sessions_y", "ga_users_y", "ga_pv_y", "ga_sessions_7d", "ga_pv_7d",
                    "sc_clicks", "sc_impressions", "sc_ctr", "sc_position", "added",
                    "ga_organic_y", "ga_organic_7d", "ga_organic_engaged_7d"]

Tables whose shape is still moving get named columns. Small, stable ones keep the positional form. The rule is not "always name columns" but "name them where the shape is still changing."

Migrations only ever add

When the collector starts sending a new field, an old database lacks the column and everything falls over. At startup, check PRAGMA table_info and ALTER TABLE ... ADD COLUMN anything missing:

def ensure_columns(c, table, cols):
    """Adds only the columns that do not exist yet. Idempotent, and never
    rewrites existing rows (new columns become NULL, which is correct:
    that push simply did not collect them)."""
    have = {r["name"] for r in c.execute(f"PRAGMA table_info({table})").fetchall()}
    for name, decl in cols:
        if name not in have:
            c.execute(f"ALTER TABLE {table} ADD COLUMN {name} {decl}")

Adding a column in SQLite is a metadata-only operation, so history survives untouched. Add columns, never drop them, and any build works against any database.

Do not deduplicate the same day

Two pushes a day means two rows a day. Rather than upserting the second over the first, every push appends. Trend and "change since last run" only exist because the history does. There is no unique constraint on the timestamp and that is deliberate.

Fix the timezone at write time

Store ts as ISO UTC and also write ts_jst ("YYYY/MM/DD HH:MM JST") at insert time, keeping both. If the display layer converts on every render, one of those conversions eventually drifts.

Missing is not zero

Every numeric column distinguishes them, and the renderer does too:

def mval(v, title="欠測（取得できず）。実測ゼロ(0)ではありません"):
    """Missing (None) renders as an em dash; a measured zero renders as 0.
    Never silently collapse one into the other."""

NULL means the collector failed to fetch. 0 means it fetched successfully and the number was zero. Merging them turns a broken credential into "traffic disappeared", which is a conclusion you will act on.

The result

The dashboard sits alongside the other tools I have built: https://hashitosystem.com

Wrap-up

http.server plus sqlite3 is genuinely enough for an internal ops dashboard. The division of labour is what makes it hold up:

Collection lives in the collector. The receiver has no idea what an "impression" is.
Settled metrics become columns; loose extras go in a meta JSON column.
Timezone is resolved at write time, not at render time.
Migrations only add columns, so old databases keep working.
Append every push, because history is the entire point of snapshots.

The receiver has barely changed as the number of sites grew, and that is the whole return on keeping it dependency-free and logic-free.

Enforcing a style rule with a linter that actually fails the build

takahiro hashito — Thu, 20 Aug 2026 15:50:22 +0000

Background

I run a fleet of static sites that publish new content every day, mostly unattended. One of the house style rules is simple: no emoji anywhere in our own copy.

So I wrote emoji-lint, a check that exits 1 the moment it finds a single emoji. It sits in the pre-deploy gate, which means a failure stops that day's publish.

This post is not about the regex. It's about what happens when you put a failing check into real operation: you immediately discover the places where the rule must not apply.

How it works

The core is unremarkable. A regex holds the emoji code point ranges, the scanner walks each file line by line, and matching lines are reported as JSON.

const EMOJI_RE =
  /[\u{1F000}-\u{1FAFF}\u{2600}-\u{27BF}\u{2B00}-\u{2BFF}\u{1F1E6}-\u{1F1FF}\u{FE0F}\u{200D}\u{2049}\u{203C}\u{2122}\u{2139}]/u;

The interesting part came later. Three categories of content look exactly like a violation but must not be treated as one:

Verbatim quotes from other people
Real proper nouns whose official spelling contains a symbol
Passages where the emoji itself is the subject being explained

Delete the emoji in any of those and you break something more important than the style rule.

One term up front: "masking" here means replacing a range with spaces so the scanner cannot see it. Nothing is deleted from the file.

Implementation

Scope exclusions to an explicit marker, never to a CSS class

Quotes are excluded only when the element carries data-quote="verbatim".

function maskVerbatimQuotes(text) {
  let masked = 0;
  const out = text.replace(VERBATIM_RE, (m, open, tag, inner, close) => {
    masked++;
    // keep newlines, blank everything else -> line numbers stay intact
    return open + inner.replace(/[^\n]/g, " ") + close;
  });
  return { text: out, masked };
}

Proper nouns can't be covered by a marker

The second case needed a different unit. Take the manga title ラブ★コン: the ★ (U+2605) is part of the official title. Rewrite it as ラブコン and the title is simply wrong.

The catch is that this string also expands into <title>, meta description, JSON-LD, and even href="/tags/ラブ★コン.html". An element-level attribute cannot cover all of that.

So for proper nouns, the unit of exclusion is the name itself:

{
  "names": ["ラブ★コン", "聖☆おにいさん", "うたの☆プリンスさまっ♪", "ニドラン♀"]
}

Numeric character references were invisible to the scanner

The fix is to decode, before scanning, only those numeric references whose code point falls in an emoji range. Structural references like <, >, and & are left untouched.

Gotchas

Never exclude silently. This turned out to be the single most valuable decision.

{
  "ok": true,
  "count": 0,
  "quotedSkipped": 0,
  "sampleSkipped": 0,
  "nameSkipped": 0,
  "namesUsed": {},
  "entityDecoded": 0,
  "namesUnused": ["ラブ★コン", "聖☆おにいさん", "ニドラン♀", "ニドラン♂"]
}

The result

One of the sites this runs against: https://cve.autoarticles.net

Wrap-up

A snapshot-style ops dashboard with nothing but the Python standard library

takahiro hashito — Thu, 20 Aug 2026 15:50:12 +0000

Background

So I built one. Each run of the collector produces a single snapshot, the server stores it, and history accumulates.

How it works

Three pieces:

A collector (Node) runs twice a day at 10:00 and 22:00, gathers analytics, Search Console, per-site updates and cloud cost into one JSON object, and POSTs it to /api/ingest.
The receiver (Python http.server) stores that payload as one snapshot in SQLite.
The page reads the latest snapshot and renders bars in plain CSS, with a detail modal per site.

All the gathering logic lives in the collector. The receiver only authenticates, stores and retrieves — which is why it has needed almost no changes as sites were added.

Implementation

Dependency-free HTTP with token auth

import json, os, sqlite3
from http.server import ThreadingHTTPServer, BaseHTTPRequestHandler

class Handler(BaseHTTPRequestHandler):
    def do_POST(self):
        if self.path != "/api/ingest":
            return self.send_error(404)
        if self.headers.get("X-Token") != TOKEN:      # writes need a token
            return self.send_error(403)
        n = int(self.headers.get("Content-Length", 0))
        payload = json.loads(self.rfile.read(n) or b"{}")
        sid = ingest(payload)
        self._json({"ok": True, "snapshot_id": sid})

ThreadingHTTPServer rather than the single-threaded one, so an ingest arriving while a page is being rendered does not block.

A successful push looks like this from the collector's side — the returned snapshot_id is the row that was just created, and it increments twice a day:

push ok -> snapshot_id=101  (sites=23 / GA=true / SC=true / GCP=130 JPY / tweets_today=19)

A normalized schema keyed by snapshot id

def init_db():
    c = db()
    c.executescript("""
        CREATE TABLE IF NOT EXISTS snapshots(
            id INTEGER PRIMARY KEY AUTOINCREMENT,
            ts TEXT, ts_jst TEXT, gcp_cost TEXT, meta TEXT);
        CREATE TABLE IF NOT EXISTS site_metrics(
            snapshot_id INTEGER, site_key TEXT, label TEXT,
            ga_sessions_y REAL, ga_pv_y REAL,
            sc_clicks REAL, sc_impressions REAL, sc_ctr REAL, sc_position REAL);
        CREATE TABLE IF NOT EXISTS gcp_services(
            snapshot_id INTEGER, service TEXT, cost REAL);
        CREATE INDEX IF NOT EXISTS idx_sm_snap ON site_metrics(snapshot_id);
    """)
    c.commit(); c.close()

def ingest(p):
    c = db(); cur = c.cursor()
    cur.execute("INSERT INTO snapshots(ts,ts_jst,gcp_cost,meta) VALUES(?,?,?,?)",
                (p.get("ts"), p.get("ts_jst"), p.get("gcp_cost"),
                 json.dumps(p.get("meta") or {})))
    sid = cur.lastrowid
    for s in p.get("sites", []):
        cur.execute("INSERT INTO site_metrics VALUES(?,?,?,?,?,?,?,?,?)",
            (sid, s.get("key"), s.get("label"), s.get("ga_sessions_y"),
             s.get("ga_pv_y"), s.get("sc_clicks"), s.get("sc_impressions"),
             s.get("sc_ctr"), s.get("sc_position")))
    c.commit(); c.close(); return sid

Metrics with a settled shape go in columns; loose extras go in the meta JSON column. That split keeps the schema firm without making every new field a migration.

With that layout, the trend chart is one query — total sessions per snapshot, most recent 60:

SELECT s.id sid, s.ts_jst tsj, COALESCE(SUM(m.ga_sessions_y),0) tot
FROM snapshots s
LEFT JOIN site_metrics m ON m.snapshot_id = s.id
GROUP BY s.id ORDER BY s.id DESC LIMIT 60

Gotchas

Positional inserts break the day you add a column

# Always INSERT into site_metrics with explicit column names.
# With positional `VALUES(?,?,...)`, adding a single column makes every push
# fail with "table has N columns but M values supplied" -- an operational
# incident, and worse than breaking quietly would have been.
SITE_METRIC_COLS = ["snapshot_id", "site_key", "label", "url",
                    "ga_sessions_y", "ga_users_y", "ga_pv_y", "ga_sessions_7d", "ga_pv_7d",
                    "sc_clicks", "sc_impressions", "sc_ctr", "sc_position", "added",
                    "ga_organic_y", "ga_organic_7d", "ga_organic_engaged_7d"]

Tables whose shape is still moving get named columns. Small, stable ones keep the positional form. The rule is not "always name columns" but "name them where the shape is still changing."

Migrations only ever add

When the collector starts sending a new field, an old database lacks the column and everything falls over. At startup, check PRAGMA table_info and ALTER TABLE ... ADD COLUMN anything missing:

def ensure_columns(c, table, cols):
    """Adds only the columns that do not exist yet. Idempotent, and never
    rewrites existing rows (new columns become NULL, which is correct:
    that push simply did not collect them)."""
    have = {r["name"] for r in c.execute(f"PRAGMA table_info({table})").fetchall()}
    for name, decl in cols:
        if name not in have:
            c.execute(f"ALTER TABLE {table} ADD COLUMN {name} {decl}")

Adding a column in SQLite is a metadata-only operation, so history survives untouched. Add columns, never drop them, and any build works against any database.

Do not deduplicate the same day

Fix the timezone at write time

Store ts as ISO UTC and also write ts_jst ("YYYY/MM/DD HH:MM JST") at insert time, keeping both. If the display layer converts on every render, one of those conversions eventually drifts.

Missing is not zero

Every numeric column distinguishes them, and the renderer does too:

def mval(v, title="欠測（取得できず）。実測ゼロ(0)ではありません"):
    """Missing (None) renders as an em dash; a measured zero renders as 0.
    Never silently collapse one into the other."""

The result

The dashboard sits alongside the other tools I have built: https://hashitosystem.com

Wrap-up

http.server plus sqlite3 is genuinely enough for an internal ops dashboard. The division of labour is what makes it hold up:

Collection lives in the collector. The receiver has no idea what an "impression" is.
Settled metrics become columns; loose extras go in a meta JSON column.
Timezone is resolved at write time, not at render time.
Migrations only add columns, so old databases keep working.
Append every push, because history is the entire point of snapshots.

The receiver has barely changed as the number of sites grew, and that is the whole return on keeping it dependency-free and logic-free.