DEV Community: takahiro hashito

Embedding JSON-LD in a static site with template literals, and the one escape you cannot skip

takahiro hashito — Thu, 17 Sep 2026 13:17:25 +0000

Background

I run a small static site, a Japanese dictionary of words and phrases that are trending on X. Every page is generated daily by a dependency-free build.js: a Node.js script that only uses the built-in fs module to write HTML files.

Each page carries JSON-LD (JSON for Linked Data, the structured-data format where you describe what a page is about as JSON and put it in a <script type="application/ld+json"> tag for search engines). There is no template engine involved. The JSON-LD goes straight into the HTML through a JavaScript template literal.

This post walks through that setup, and through the single line of escaping that I consider mandatory whenever you embed JSON this way.

How it works

The structure is simple. Each page builds an array of plain objects it wants to publish as JSON-LD and hands that array to a shared head() function.

data/*.json（語のデータ）
   │
   ▼
renderWordPage(w)  ── DefinedTerm / Article / BreadcrumbList / FAQPage のオブジェクトを作る
   │  jsonld: [...]
   ▼
head(opts)         ── 配列を <script type="application/ld+json"> に1個ずつ変換して <head> に入れる
   │
   ▼
public/word/<slug>.html

(The labels in the diagram are the original Japanese notes: word data goes in, renderWordPage builds the DefinedTerm / Article / BreadcrumbList / FAQPage objects, and head turns each one into its own script tag inside <head>.)

Because the objects are ordinary JavaScript objects, adding or dropping properties conditionally is easy to read. The only place that builds an HTML string by hand is the very last step.

Implementation

1. head() turns the array into script tags

Here is the JSON-LD part of the real head() in build.js:

function head(opts) {
  const {
    title,
    description,
    canonicalPath,
    ogType = "website",
    ogImage = "/og-image.png",
    jsonld = [],
    noindex = false,
  } = opts;
  const url = CANON + canonicalPath;
  const jsonldStr = jsonld
    .map(
      (o) => `<script type="application/ld+json">${JSON.stringify(o).replace(/</g, "\\u003c")}</script>`,
    )
    .join("\n");

jsonld is an array of objects, and each object becomes one script tag. The important part is that every < in the JSON.stringify output is replaced with \u003c. The Gotchas section explains why.

2. Pages just build objects and pass them in

On a word page, the word itself is a DefinedTerm (the schema.org type for a defined term) and the explanation is an Article. This is an excerpt from the actual code:

const altNames = [w.reading, ...w.aliases].filter(Boolean);
const definedTerm = {
  "@context": "https://schema.org",
  "@type": "DefinedTerm",
  name: w.word,
  alternateName: altNames.length > 1 ? altNames : w.reading,
  description: w.meaning,
  inDefinedTermSet: {
    "@type": "DefinedTermSet",
    name: site.siteName,
    url: CANON + "/words",
  },
  url: CANON + path_,
};

If a word only has a reading, alternateName stays a plain string. It becomes an array only when the word also has alternative spellings. The Article object follows the same pattern: it skips keywords when there are no tags (...(w.tags.length ? { keywords: ... } : {})) and only sets datePublished when the word has a date. All of those decisions happen while building the object, not while building HTML.

Breadcrumbs appear on every page, so the JSON-LD version is generated by a function from the same array that renders the visible breadcrumb navigation:

function breadcrumbJsonLd(items) {
  return {
    "@context": "https://schema.org",
    "@type": "BreadcrumbList",
    itemListElement: items.map((it, i) => ({
      "@type": "ListItem",
      position: i + 1,
      name: it.label,
      item: CANON + it.href,
    })),
  };
}

Since both the on-page breadcrumb and the structured-data breadcrumb come from the same crumbs array, they cannot drift apart when one of them is edited.

This is the actual output on a generated word page (/word/50-50). The full page has more blocks, so only the BreadcrumbList is shown:

<script type="application/ld+json">{"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"トップ","item":"https://kotoba.autoarticles.net/"},{"@type":"ListItem","position":2,"name":"辞書","item":"https://kotoba.autoarticles.net/words"},{"@type":"ListItem","position":3,"name":"50-50","item":"https://kotoba.autoarticles.net/word/50-50"}]}</script>

The three input crumbs come out as ListItem entries with position 1 to 3, on a single line. (The name values are the Japanese labels for "Top" and "Dictionary".)

Gotchas

JSON.stringify alone lets the script tag close early

JSON.stringify gives you valid JSON. It does not give you something that is safe to drop into HTML. The HTML parser does not read the script body as JSON; it closes the tag as soon as it sees </script. If a word's description or a product name ever contains the string </script>, the JSON-LD gets cut off right there and the rest is parsed as HTML.

Here is a minimal script you can run yourself:

const o = { "@type": "DefinedTerm", name: "</script><script>alert(1)</script>" };
const raw = `<script type="application/ld+json">${JSON.stringify(o)}</script>`;
const safe = `<script type="application/ld+json">${JSON.stringify(o).replace(/</g, "\\u003c")}</script>`;
console.log(raw);
console.log(safe);
const body = safe.match(/<script type="application\/ld\+json">([\s\S]*?)<\/script>/)[1];
console.log(JSON.parse(body).name);

Running it with node prints:

<script type="application/ld+json">{"@type":"DefinedTerm","name":"</script><script>alert(1)</script>"}</script>
<script type="application/ld+json">{"@type":"DefinedTerm","name":"\u003c/script>\u003cscript>alert(1)\u003c/script>"}</script>
</script><script>alert(1)</script>

Line 1 (before the replace) has a literal </script> in the middle of the value, which is where the HTML parser closes the tag. In line 2 (after the replace) every < is \u003c, so </script never appears in the HTML. Line 3 shows that \u003c is a JSON string escape, so JSON.parse gives you the original value back. Search engines see exactly the same data; only the HTML structure is protected.

> and & are left alone. What ends a script body is the appearance of </script, so removing < is enough.

Add a test that fails if the replace disappears

To anyone who did not write it, this line looks like a pointless replace, which makes it an easy casualty of a refactor. So test/jsonld-escape.test.js checks two things (run with npm test, which is node --test test/):

It extracts every JSON-LD block from all generated HTML under public/ and asserts that no raw < remains and that each block passes JSON.parse.
It reads the build.js source and asserts that the expression embedding JSON-LD has .replace(/</g, "\\u003c") attached.

Check 1 on its own passes silently whenever the current data happens to contain no <. Check 2 catches the fix being removed regardless of the data. Both also fail when they find no JSON-LD at all, so the test cannot go green by checking nothing.

The result

This is the site that ships JSON-LD this way. Open the source of any word page and you will see the script tags described above: https://kotoba.autoarticles.net

The same one-line replace is also in the build.js of my other sites that are built the same way.

Wrap-up

For JSON-LD on a static site, build plain objects and let head() turn them into script tags. Conditional logic stays on the object side, and the code stays readable.
When embedding with a template literal, always run JSON.stringify(o).replace(/</g, "\\u003c"). Valid JSON and safe-to-embed-in-HTML are two different problems.
Defensive code whose purpose is not obvious should come with a test that fails the moment it is removed.

One service account for 21 GA4 and Search Console properties, with zero dependencies

takahiro hashito — Wed, 16 Sep 2026 13:14:54 +0000

Background

I run 21 small sites and I wanted one command that pulls GA4 (Google Analytics 4) numbers and Search Console numbers for all of them, every night.

Both are reachable through Google APIs, but each site is a separate property. Wire it up naively and you need one authorization per site. Browser-based OAuth is a bad fit for an unattended nightly job: nobody is there to click the consent screen.

So I made one service account (SA) and added it as a viewer on every property. An SA is a Google account for a program rather than a person; if you hold its private key you can mint tokens without a consent screen. Authentication collapsed into a single key file.

This post covers how that collapse was built, and — more importantly — how I misread the numbers it collected.

How it works

Here is the whole path. One key file becomes a JWT (JSON Web Token: a short signed string that asserts "I am this service account"), the JWT is exchanged for an access token, and the same token drives both APIs.

                 ┌───────────────────────┐
  1 SA key ────▶ │ sign an RS256 JWT      │
                 └──────────┬────────────┘
                            │ grant_type=jwt-bearer
                            ▼
                 oauth2.googleapis.com/token
                            │  access_token
             ┌──────────────┴──────────────┐
             ▼                             ▼
   GA4 Data API (runReport)      Search Console API (searchanalytics)
             │                             │
             └──────────────┬──────────────┘
                            ▼
                    per-site JSON (stdout)

No third-party library appears anywhere in that path. I kept dependencies at zero on purpose: googleapis would be easier, but every new machine that runs the nightly job then needs an npm install. Signing a JWT takes nothing but Node's built-in crypto.

Implementation

Getting an access token from the key is the entire trick. The SA JSON holds client_email (the account's address) and private_key (an RSA private key). Below is the part that assembles the JWT and posts it to Google's token endpoint. sa is the parsed key file, now is the current UNIX time in seconds, and b64url is a small Base64URL helper.

const claim = {
  iss: sa.client_email,
  scope: "https://www.googleapis.com/auth/analytics.readonly",
  aud: "https://oauth2.googleapis.com/token",
  iat: now,
  exp: now + 3600,
};
const unsigned = `${b64url(JSON.stringify(header))}.${b64url(JSON.stringify(claim))}`;
const sig = crypto.createSign("RSA-SHA256").update(unsigned).sign(sa.private_key);
const jwt = `${unsigned}.${b64url(sig)}`;

const res = await fetch("https://oauth2.googleapis.com/token", {
  method: "POST",
  headers: { "Content-Type": "application/x-www-form-urlencoded" },
  body: new URLSearchParams({
    grant_type: "urn:ietf:params:oauth:grant-type:jwt-bearer",
    assertion: jwt,
  }).toString(),
});

What comes back is an access_token valid for one hour. From there both APIs take the same Authorization: Bearer header, and swapping the scope string is the only change needed to reuse the function for Search Console.

The second change was less glamorous and paid off just as well: one ledger instead of two. The map from site name to numeric GA4 property ID was hardcoded in the fetch script. The same map also lived in the dashboard collector. I diffed all 21 entries and every value matched — but one key was spelled differently (doraemon-tools vs doraemon), and that one site silently failed to resolve.

const PROPERTY_IDS = (() => {
  const { SITES } = require("../dashboard/collect.js");
  const out = {};
  for (const s of SITES) if (s.ga) out[s.key] = String(s.ga);
  return out;
})();

Deriving the map from the collector removed the copy. A side effect: two sites that only existed in the collector's list became fetchable without passing --property by hand.

Gotchas

A missing environment variable wiped out a whole day of measurement. An unattended job does not inherit your interactive shell's environment. GOOGLE_APPLICATION_CREDENTIALS was unset and one night's log was a single line saying GA/SC could not be fetched. I added a fallback to a known key path (~/.config/ga/...). "No env var" and "no key" are different failures.

And the big one: I was reading total sessions as a result.

Some sites had zero rows in their 28-day Search Console query report while GA reported three-digit session counts. Measured on 2026-09-12: blog 146, cve-watch 83, anime-trend 50. If a site has never appeared in search results, then whoever showed up did not arrive through search. They came direct, from a referral, or they were bots. None of that is an SEO result.

The fix was to always pull the channel breakdown (sessionDefaultChannelGroup) and decide reach on the Organic Search figure alone.

function reachVerdict(channels, scImpressions) {
  const organic = Number(channels.organicSearch || 0);
  const total = Number(channels.total || 0);
  const out = {
    organicSearchSessions: organic,
    totalSessions: total,
    nonSearchSessions: Math.max(0, total - organic),
    scImpressions: scImpressions ?? null,
    reached: organic > 0,
  };
  if (organic === 0 && total > 0) {
    out.verdict = "not-reached";
  } else if (organic > 0) {
    out.verdict = "reached";
  } else {
    out.verdict = "no-sessions";
  }
  return out;
}

Three states, not two: not-reached (people arrive, but not from search), reached (search delivers), and no-sessions (nobody came at all). The old code folded the first and the last into a single "has sessions / has none" test, which is exactly how not-reached got read as success.

One detail I care about: when the Search Console figure cannot be fetched, it is not filled in with 0.

const sc = scImpressions === undefined || scImpressions === null
  ? null
  : Number(scImpressions);

A missing measurement and a measured zero are different facts. Writing 0 turns a day when the API was down into a day with zero impressions, and a week later that ledger shows a decline that never happened. It stays null, and the verdict text says so explicitly.

The result

A site that runs on this: https://hashitosystem.com

Wrap-up

One SA added to every property reduces unattended auth to a single key file. Signing the JWT needs only Node's crypto, so the dependency count stays at zero
Two copies of the same lookup table will drift in spelling even when every value matches. Derive it from one source
A session count is not evidence that search brought anyone. Pull the channel breakdown and judge on Organic Search
Never store a failed fetch as 0. Once a gap and a real zero share a type, you cannot tell them apart later

Generating per-article og:image with headless Chrome and zero npm dependencies

takahiro hashito — Mon, 14 Sep 2026 14:44:21 +0000

The problem

I run about twenty small sites that publish new articles every day through an automated pipeline. For a long time every single article on every single site shared one identical og:image.

og:image is the <meta property="og:image" content="..."> tag that decides which picture shows up when someone pastes your URL into X or Slack. When three of your articles land in the same timeline with the same picture, they read as one article. Whatever is different about them never gets across.

There are three obvious ways to fix this:

Call a hosted image-generation API (pay per image)
Add an image library such as sharp or canvas
Emit SVG and point og:image at it

Option 3 is out: the major social platforms do not render SVG for og:image. X and Slack both ignore it and fall back. Option 1 costs money that scales with article count. Option 2 pulls in a native build step, and I run this pipeline on both macOS and Windows, so I did not want one more thing that breaks differently per platform.

What I ended up with: lay the card out in HTML and CSS, then let the Chrome that is already installed take a screenshot of it. Zero npm dependencies, zero recurring cost. Here is how it works, plus the four traps I walked into.

The pipeline

The flow from an article's metadata to the meta tag on the published page looks like this:

article metadata (title / badge / accent / siteLabel / domain / expired)
  ↓  templateHtml()           … build a fixed 1200×630 HTML+CSS page
temp file og.html
  ↓  Chrome --headless=new --screenshot --window-size=1200,820
PNG (820px tall, padding at the bottom)
  ↓  cropPngHeight()          … cut to the top 630 rows using only Node's zlib
public/articles/<slug>.og.png
  ↓  each site's build.js      … uses this PNG if present, otherwise a shared image
the published page's <meta property="og:image">

Every step is deterministic: the same metadata always produces byte-identical output. The only inputs are the site's data/site.json and data/articles.json. One article's metadata renders one card. The caller looks like this:

const { generateOgImages } = require('./automation/scripts/ogp/generate-og.js');

await generateOgImages([
  {
    out: 'public/articles/campaign-famima-asa-combiwari.og.png',
    title: 'ファミマ朝5時〜11時「コンビ割」9月15日開始',
    badge: '最大145円引き',
    accent: '#6f4e37',
    siteLabel: 'コーヒーのセール情報',
    mark: '珈',
    domain: 'coffee.autoarticles.net',
    note: '価格・在庫は変動。最新はAmazonでご確認ください。',
    expired: false,
  },
]);

The interesting parts

Let CSS do the drawing

There is not a single line of image-manipulation code in here. The gradient, the rounded badge, the diagonal light streaks, the "expired" ribbon — all of it is CSS.

const grad = expired
  ? 'linear-gradient(135deg,#8a8f94,#6b7075)'          // expired deals go grey
  : `linear-gradient(135deg,${escHtml(accent)},${escHtml(endColor)})`;

The title font size steps down as the title gets longer, and -webkit-line-clamp:3 caps it at three lines. Those two rules alone eliminated overflow in practice:

h1{
  font-size:${fontPx}px; line-height:1.28;
  display:-webkit-box; -webkit-line-clamp:3; -webkit-box-orient:vertical; overflow:hidden;
}

The real payoff of staying in CSS is that you can iterate on the layout in a browser. Open the same HTML in your local Chrome and you see essentially the output. You do not get that if you are computing coordinates against an imaging library.

Take the shot

The assembled HTML goes into a temp file, and Chrome is launched exactly once to capture it. chromeBin() lazily resolves the browser binary, out is the destination PNG path, and htmlPath is the og.html written into a temp directory:

execFileSync(chromeBin(), [
  '--headless=new',
  '--disable-gpu',
  '--hide-scrollbars',
  '--no-sandbox',
  '--force-device-scale-factor=1',
  '--window-size=1200,820',
  `--screenshot=${out}`,
  fileUrl(htmlPath),
], { stdio: 'ignore' });
cropTopLeft(out, 1200, 630);

After that single call, out holds a 1200×820 PNG, and cropTopLeft trims it to the top 1200×630. The publishable file is finished at that point.

--force-device-scale-factor=1 is not optional. Without it a HiDPI machine hands you 2400×1260 and every downstream crop is wrong.

Do not redraw what has not changed

A hash of the input parameters is stored next to the PNG as <name>.og.hash, and a match means skip:

const hashPath = out.replace(/\.png$/i, '') + '.og.hash';
const h = hashOf(it);                       // `out` is excluded from the hash on purpose
if (!opts.force && fs.existsSync(out) && fs.existsSync(hashPath)) {
  if (fs.readFileSync(hashPath, 'utf8').trim() === h) { skipped++; continue; }
}

This matters more than it sounds. A real daily build logs:

[og] coffee: 生成 4 / スキップ 152 / 計 156

Four of 156 articles were redrawn — the ones actually touched that day. Chrome startup costs 0.5–1s per image, so rendering all 156 every day would add over two minutes to the nightly run.

Four traps

1. `--window-size` height is the viewport height

Passing --window-size=1200,630 does not give you 630px. Chrome's --headless --screenshot treats window height as viewport height, but the captured image can extend to the full page height, leaving padding at the bottom.

The fix is to shoot big and crop: capture at --window-size=1200,820, then cut to the top 630 rows.

2. Cropping without an imaging library

sharp would make this one line, but that breaks the zero-dependency premise. Reading the PNG spec, it turns out Node's built-in zlib is enough for this specific case.

PNG pixel data is stored as scanlines, each prefixed by a one-byte filter type, all concatenated and deflated into the IDAT chunks. Crucially, a filter only ever references rows above it. So if you inflate the stream and keep the first targetH rows, what you have is a valid image of the top targetH rows — no recompute needed.

const channels    = colorType === 6 ? 4 : 3;   // 6=RGBA, 2=RGB
const bytesPerRow = 1 + width * channels;      // leading byte is the filter type
const raw     = zlib.inflateSync(Buffer.concat(idat));
const cropped = raw.subarray(0, targetH * bytesPerRow);   // top targetH rows

const newIhdr = Buffer.from(ihdr);
newIhdr.writeUInt32BE(targetH, 4);             // patch only the height in IHDR
const out = Buffer.concat([
  PNG_SIG,
  pngChunk('IHDR', newIhdr),
  ...passthrough,
  pngChunk('IDAT', zlib.deflateSync(cropped, { level: 9 })),
  pngChunk('IEND', Buffer.alloc(0)),
]);

You do have to compute the chunk CRC32 yourself, but that is another thirty lines. Anything the code cannot handle — interlaced, non-8-bit depth, palette — throws instead of guessing. Chrome always emits 8-bit non-interlaced RGB/RGBA, so this holds in production.

This trick only covers vertical cropping. Cropping horizontally means cutting inside each row, which changes the bytes-per-row and makes the problem meaningfully harder. Here the width already matches the window width, so height is all that is needed.

3. Windows produced zero images, silently

This one hurt. The list of candidate Chrome paths contained no Windows entries at all. On a Windows box with Chrome plainly installed, the resolver threw "Chrome/Chromium not found."

Worse, the caller swallowed that exception and returned exit 0. Each site's build.js has a deliberately safe fallback: use the per-article PNG if present, otherwise the shared site image. So with generation totally dead, the build still succeeded, pages still shipped, and og:image quietly stayed on the shared image. Nothing in the logs said otherwise.

The fix came in two parts:

Split out a pure function. chromeCandidates(env, platform, homedir) only assembles candidate paths and never touches the filesystem; findChrome() takes that list and checks existence. Now a test running on macOS can assert that Windows candidates are produced.
Drop the fallback. If the binary cannot be resolved, fail.

4. The same shape of bug, somewhere else

There was a second instance. Passing --site <key> resolves the site through a task definition's repoPath (e.g. /Users/hashito/git/web/coffee). On Windows that macOS-absolute path is interpreted as drive-relative, so it resolves against whatever drive the process happens to be on and does not exist. The old code then silently fell back to a different clone of the site.

That clone also has a data/ directory, so generation succeeded. The log said success. But that tree is never deployed, so nothing reached production.

The fix: path.resolve to pin the drive before dispatching, and fail when the site cannot be resolved. The fallback to a different tree is gone; --dir exists for the case where you genuinely want to point elsewhere.

The result

One of the sites running on these generated og:images: https://blog.hashito.biz

Takeaway

If a process has a fallback, count how often the fallback fires and print it. Of the four traps above, the hard one was #2 (cropping PNG with nothing but zlib), but the expensive ones were #3 and #4. In both cases a well-intentioned "safe fallback" concealed the fact that generation had died completely.

A fallback limits the damage when something goes wrong; it is not evidence that nothing went wrong. Print generated / skipped / total on every run and "generated 0, skipped 0" jumps out of the log immediately. That single line is the difference between an implementation that tells you it is broken and one that ships the same placeholder image for weeks.

Your CVSS >= 7.0 filter is silently dropping the CVEs you most want

takahiro hashito — Sun, 13 Sep 2026 13:43:21 +0000

The problem: too much input

A CVE (Common Vulnerabilities and Exposures) is the globally unique ID assigned to a single vulnerability, in the form CVE-2026-85706. The NVD (National Vulnerability Database, run by the US government) collects them, and it registers anywhere from a few dozen to well over a hundred entries per day. Reading all of that by hand is not an option for a side project.

So I wrote a collector that keeps only the serious ones. CVSS (Common Vulnerability Scoring System) rates severity from 0.0 to 10.0, where 7.0 and above is High and 9.0 and above is Critical. It is a number, so a machine can cut on it.

The first version of that cut threw away exactly the entries I cared about most. This post is about where the threshold belonged.

The shape of the collector

Two sources feed one merge step:

                     ┌──────────────────────────┐
  CISA KEV ─────────▶│ diff against known IDs   │
  (exploited)        │ (data/vulns.json)        │──▶ new candidates
  NVD ──────────────▶│                          │
  (last N days)      └──────────────────────────┘

CISA KEV (short for Known Exploited Vulnerabilities Catalog) is a list of entries that CISA has confirmed are being exploited in the wild. It ships as a single JSON file. NVD is the full database, queryable by date range and severity.

The rule I settled on:

The OR is the whole point.

What went wrong

The obvious implementation merges first and filters once:

// first version (wrong)
const candidates = [...fromKEV(), ...fromNVD()];
const filtered = candidates.filter((c) => c.cvssScore >= 7.0);

Run that and zero KEV entries survive. The KEV feed has no CVSS field at all. Here is a real record:

{
  "cveID": "CVE-2026-85706",
  "vendorProject": "GitLab",
  "product": "Community Edition and Enterprise Edition",
  "vulnerabilityName": "GitLab Community Edition and Enterprise Edition Path Traversal Vulnerability",
  "dateAdded": "2026-09-11",
  "dueDate": "2026-09-14",
  "knownRansomwareCampaignUse": "Unknown"
}

No cvssScore. In JavaScript, undefined >= 7.0 is false, so every KEV candidate is dropped. No exception. Nothing in the logs. The count just goes to zero.

And what was being dropped was the highest priority half of the input. KEV membership means the vulnerability is being exploited right now, not that a scoring rubric says it could be bad. A 6.5 that attackers are using outranks a 9.8 that nobody has touched. A filter meant to keep the serious items was deleting the ones under active attack.

The fix: push the filter into the source

The filter moves out of the merge step and into each fetcher. The KEV fetcher below reads the catalog JSON, sorts by dateAdded descending, and skips only IDs already present in known (the set of CVE IDs the site has published). There is no score comparison anywhere in it.

// KEV source: no CVSS cut. Confirmed exploitation is the evidence.
async function fromKEV(known, max) {
  const data = await getJSON(KEV_URL);
  const vulns = (data && data.vulnerabilities) || [];
  vulns.sort((a, b) =>
    String(b.dateAdded || "").localeCompare(String(a.dateAdded || "")),
  );
  const out = [];
  for (const v of vulns) {
    const id = String(v.cveID || "").toUpperCase();
    if (!id || known.has(id)) continue;
    out.push({ cveId: id, source: "CISA KEV", /* ...trimmed... */ });
    if (out.length >= max) break;
  }
  return out;
}

// NVD source: not in KEV, so the score is the only evidence. Enforce 7.0 here.
for (const it of items) {
  const m =
    (metrics.cvssMetricV31 && metrics.cvssMetricV31[0]) ||
    (metrics.cvssMetricV30 && metrics.cvssMetricV30[0]);
  const cvss = m && m.cvssData ? m.cvssData.baseScore : null;
  if (cvss != null && cvss < 7.0) continue;
  out.push({ cveId: id, source: "NVD", cvssScore: cvss, /* ...trimmed... */ });
}

The NVD loop is the other half of the same file. It walks the API results, pulls the base score out of whichever CVSS metric block is present (v3.1 first, v3.0 as fallback), and applies the 7.0 cut there and only there.

The cvss != null guard matters. NVD occasionally serves an entry with no score yet (still being analysed). Those are kept rather than dropped: "cannot be judged by score" must not collapse into "fails the score check", or you have built the same silent-deletion path a second time.

Output after the fix:

$ node automation/scripts/cve/fetch-critical.js --max 3 --days 7
{
  "generatedFrom": ["CISA KEV", "NVD(last 7 days)"],
  "knownCount": 317,
  "candidates": [
    { "cveId": "CVE-2026-85706", "source": "CISA KEV", "vendorProject": "GitLab",  ... },
    { "cveId": "CVE-2026-86060", "source": "CISA KEV", "vendorProject": "MikroTik", ... },
    { "cveId": "CVE-2026-67277", "source": "CISA KEV", "vendorProject": "MikroTik", ... }
  ],
  "errors": []
}

Every candidate carries source, so the evidence behind each pick is auditable later. Without it there is no way to check whether the rule was right.

Two things that bit me

1. The NVD API returns ancient CVEs unless you pass dates.

Query with cvssV3Severity=CRITICAL alone and you get the head of the registration order, which is CVEs from around 2002. I was collecting "new critical vulnerabilities" and receiving twenty-year-old ones.

pubStartDate and pubEndDate must be sent as a pair, and the span is capped at 120 days. Ordering is not guaranteed either, so sort by publication date yourself after fetching.

function nvdUrl(days) {
  const span = Math.min(Math.max(Number(days) || 30, 1), 120);
  const end = new Date();
  const start = new Date(end.getTime() - span * 24 * 60 * 60 * 1000);
  const iso = (d) => d.toISOString().replace(/\.\d{3}Z$/, ".000");
  const q = new URLSearchParams({
    cvssV3Severity: "CRITICAL",
    resultsPerPage: "40",
    pubStartDate: iso(start),
    pubEndDate: iso(end),
  });
  return `https://services.nvd.nist.gov/rest/json/cves/2.0?${q}`;
}

2. "Could not fetch" and "found nothing" must not look alike.

Same failure shape as the threshold bug. A dropped connection or a rate limit both produce "0 candidates" if you write it naively, and the caller cannot tell that apart from a genuinely quiet day.

So errors are collected instead of swallowed, and a non-empty error list sets the exit code:

result.newCount = result.candidates.length;
if (result.errors.length > 0) process.exitCode = 1;
process.stdout.write(JSON.stringify(result, null, 2) + "\n");

A successful fetch that returned nothing exits 0 with errors: []. The caller can now treat the two cases differently. Never let a missing measurement pass as a green result.

Demo

One of the sites running this collector: https://cve.autoarticles.net

Takeaway

Cutting noise with a threshold was the right idea. The mistake was where the cut lived.

Generalised: write filters per data source, not over the merged set. Merge first and apply one shared condition, and every source lacking that field disappears completely and silently. Here that was the one source I could least afford to lose.

Before you make a field part of a condition, confirm it always exists on that path. If it might not, spell out the != null case so "unknown" does not collapse into "rejected". And always report a fetch failure through a different exit than a measured zero.

Decoding a UUID by hand: version, variant, and the embedded timestamp

takahiro hashito — Sat, 12 Sep 2026 13:42:36 +0000

Background

A UUID (Universally Unique Identifier) is a 128-bit value written as 32 hex digits, used as an identifier that will not collide across systems. I run a small UUID generator on a personal tools site. Recently I added the other half: paste a UUID in, and it tells you which version it is, which variant it belongs to, and — when the version carries one — the timestamp baked into it.

The reason was practical. IDs arriving from other systems come with no documentation. When you generate them yourself you know the scheme; when you receive them you are guessing. The spec makes this mechanically decidable, so I wrote the reader.

What this post is actually about is not the feature. It is how I convinced myself the decoder was correct, which turned out to be the interesting part.

How it works

A UUID is 32 hex digits. The canonical form groups them 8-4-4-4-12 with hyphens, but for reading you want the hyphens gone and a flat 32-character string.

Only two positions matter for classification (0-based indices into that flat string):

What	Index	Meaning
version	12	the UUID scheme (1/3/4/5/6/7/8)
variant	16	which family of specs this UUID belongs to

"Variant" is the less familiar one. UUIDs have several historical lineages, and the variant bits say which. Everything in normal use today is the RFC 4122 variant (carried forward by RFC 9562), meaning the top two bits of nibble 16 are 10.

Three versions embed a timestamp: v1, v6 and v7. This is where it gets awkward, because the epoch and the unit differ by version:

v7: the first 48 bits are milliseconds since the Unix epoch. Read them as a number, done.
v1 / v6: 100-nanosecond intervals since 1582-10-15, and in v1 the 60-bit timestamp is split across three fields stored out of order.

v4 is random and v3/v5 are name hashes, so there is no time to recover.

Implementation

Dependency-free JavaScript. Normalize the input, read the variant, then branch per version to recover the time. Hyphenated, unhyphenated, uppercase and brace-wrapped GUID strings all show up in practice, so everything that is not a hex digit is stripped before the length check.

const GREG_OFFSET_MS = 12219292800000; // ms between 1582-10-15 and 1970-01-01

function normalize(raw) {
  return String(raw).replace(/[^0-9a-fA-F]/g, "").toLowerCase();
}

function variantOf(nibble) {
  if ((nibble & 0x8) === 0) return "NCS (legacy)";
  if ((nibble & 0xc) === 0x8) return "RFC 4122 / RFC 9562";
  if ((nibble & 0xe) === 0xc) return "Microsoft GUID (legacy)";
  return "reserved for the future";
}

function inspect(raw) {
  const hex = normalize(raw);
  if (hex.length !== 32) return { input: raw, error: "not 32 hex digits" };
  const ver = parseInt(hex.charAt(12), 16);
  const variant = variantOf(parseInt(hex.charAt(16), 16));
  let unixMs = null;
  if (ver === 7) {
    unixMs = parseInt(hex.slice(0, 12), 16);
  } else if (ver === 1 || ver === 6) {
    const hi = hex.slice(0, 8), mid = hex.slice(8, 12), high = hex.slice(13, 16);
    const tick = ver === 1 ? BigInt("0x" + high + mid + hi)
                           : BigInt("0x" + hi + mid + high);
    unixMs = Number(tick / 10000n) - GREG_OFFSET_MS;
  }
  return { input: raw, version: ver, variant, unixMs,
           utc: unixMs == null ? null : new Date(unixMs).toISOString() };
}

Three things carry the weight here.

Test the variant masks from narrow to wide. & 0x8, then & 0xc, then & 0xe — the masks widen from one to three bits, so this order is the only correct one. Check Microsoft first and you will misclassify ordinary RFC UUIDs.
v7 needs no BigInt. 48 bits is about 2.8e14, comfortably inside Number.MAX_SAFE_INTEGER (2^53), so parseInt is enough.
v1 needs BigInt and a re-ordering. v1 splits its 60-bit timestamp into time_low (first 8 digits), time_mid (next 4) and time_hi (3 digits right after the version nibble). To rebuild it you concatenate high-to-low: time_hi + time_mid + time_low. v6 is v1 with those fields already in ascending order, so the concatenation reverses.

The final Number(tick / 10000n) - GREG_OFFSET_MS converts 100-ns Gregorian ticks to Unix milliseconds: divide by 10000 for ticks-to-ms, then subtract the 12219292800000 ms between 1582 and 1970.

Gotchas

How do you know code written from a spec is right? That is the real problem. Slide a field boundary by one digit and you do not get an obviously broken value — you get a plausible different date. And writing your own unit tests does not help, because you also author the expected values. If your reading of the spec is wrong, your test is wrong in exactly the same way.

What worked was agreement between two independent paths. RFC 9562 publishes one v1 example and one v7 example, and they are constructed to denote the same instant. Two completely different encodings, decoded by two completely different branches of my code — if they land on the same moment, both branches are validated at once.

Here is the run:

{"input":"017f22e2-79b0-7cc3-98c4-dc0c0c07398f","version":7,"variant":"RFC 4122 / RFC 9562","unixMs":1645557742000,"utc":"2022-02-22T19:22:22.000Z"}
{"input":"C232AB00-9414-11EC-B3C8-9F6BDECED846","version":1,"variant":"RFC 4122 / RFC 9562","unixMs":1645557742000,"utc":"2022-02-22T19:22:22.000Z"}
{"input":"9f1a7c54-8f3e-4b2a-b9d1-6e2c5a0f77aa","version":4,"variant":"RFC 4122 / RFC 9562","unixMs":null,"utc":null}
{"input":"{0199A9C0000070008000000000000000}","version":7,"variant":"RFC 4122 / RFC 9562","unixMs":1759489556480,"utc":"2025-10-03T11:05:56.480Z"}
{"input":"00000000-0000-0000-0000-000000000000","version":0,"variant":"NCS (legacy)","unixMs":null,"utc":null}

Line 1 is the v7 example, line 2 the v1 example. Both came out as 2022-02-22T19:22:22.000Z. The parseInt path and the BigInt re-ordering path arrived independently at the same answer. Get either field layout wrong and that agreement disappears.

Two more findings from the same run:

Line 4: {0199A9C0000070008000000000000000} — brace-wrapped, unhyphenated, uppercase — decodes identically, because normalization runs first. Without it, every GUID pasted out of a Windows tool would be rejected.
Line 5: the nil UUID (all zeros) reports version: 0 and variant: "NCS (legacy)". That is not a bug; it is what those bits literally say. But surfacing it as "a legacy NCS UUID" is misleading, because nil means "no value". Bit extraction and human-facing labels belong in separate layers — I had collapsed them. The shipped tool now special-cases nil and max (all f) before classification.

The result

A site that runs on this: https://hashitosystem.com/tools/uuid/

Generation (v1 / v4 / v7, up to 1000 at a time) and inspection live on the same page. Classification and time math happen entirely in the browser; pasted UUIDs are never sent to a server.

Wrap-up

Classifying a UUID comes down to reading two of its 32 digits. Order the variant masks narrow-to-wide, re-order v1's split timestamp before handing it to BigInt, and the implementation stays short.

The verification is the part worth keeping. Code written straight from a spec cannot be validated by tests you wrote from the same reading. Look for a case where the spec encodes one fact two different ways, and check that your two code paths agree. When two unrelated routes reach the same answer, your own misreading cannot explain it.

Why I dropped the X API for an IFTTT webhook - and what it cost me

takahiro hashito — Fri, 11 Sep 2026 13:36:09 +0000

Introduction

I run a small bot that announces updates to my own sites on X.
My first plan was to use the X API. I ended up dropping it and posting through an IFTTT webhook instead.

IFTTT is a SaaS that connects "when A happens, do B". One of its inputs is Webhooks: you get a URL,
you send one HTTP request to it, and the applet you configured runs. In other words, you get
"one HTTP call posts to X" without app registration or token refresh code.

This post is about what that trade bought me, what it cost me, and how I covered the cost.
Short version: what I gave up was confirmation that the post actually happened, and the fix was
to measure it through a separate path.

The overall shape

Three pieces:

Generate - build the announcement text from the day's updated articles and push it into a local queue.json, each item carrying a scheduledAt timestamp.
Post - a launchd job (launchd is the macOS service manager: you drop an XML plist file and it handles daemons and scheduled jobs, filling the role systemd and cron do on Linux) wakes every 30 minutes and sends only the items whose time has come.
Verify - read the public timeline separately and count what actually landed.

Generation and posting are split so the posts can be spread over a time window. If you generate and
send in the same pass, everything goes out at the same minute.

The core of the implementation

Sending is short. The IFTTT webhook takes fixed field names, value1 through value3.

const url = `https://maker.ifttt.com/trigger/${event}/with/key/${key}`;
const payload = { value1: tweetText };
if (imageUrl) payload.value2 = imageUrl;
const response = await axios.post(url, payload, {
  headers: { 'Content-Type': 'application/json' }
});

With the X API, this is where OAuth 1.0a signing or OAuth 2.0 refresh-token handling would live,
plus app registration. For a personal announcement bot that was the heaviest part. The webhook removes it.

The problem starts after this line. The code above looks successful whenever it gets a 200.
But the IFTTT webhook returns 200 when:

the applet is turned off
the X connection has been revoked
you have hit the free-tier applet limit

The webhook is answering "I accepted the event", not "the applet ran" and certainly not "it is on X".
I missed this, and my logs showed a 100% success rate the entire time delivery was completely dead.

I fixed it in two steps. First, stop looking only at the status code.

static extractError(data) {
  if (data == null) return null;
  if (typeof data === 'object') {
    const errs = Array.isArray(data.errors) ? data.errors : null;
    if (errs && errs.length) {
      return errs.map(e => (e && e.message) || String(e)).join(' / ');
    }
    return null;
  }
  const text = String(data);
  if (/"errors"\s*:/.test(text)) {
    try { return IFTTTService.extractError(JSON.parse(text)); }
    catch (e) { return text.slice(0, 200); }
  }
  if (/invalid key|not found|rate limit|too many requests/i.test(text)) {
    return text.slice(0, 200);
  }
  return null;
}

IFTTT can return an error in the body while the status stays 200. Send a key with one character
changed and you get HTTP 200 plus {"errors":[{"message":"You sent an invalid key."}]}; the function
above pulls that string out.

Second, split the names. Instead of one success flag, the call returns accepted and published.
accepted means the request was taken. published means it is live - and published is always
null (unknown), because the webhook cannot tell you.

Not writing true for something you cannot observe was the single most useful decision here.

Things that bit me

The posting job runs under macOS launchd with StartInterval set to 1800 (every 30 minutes).

<key>ProgramArguments</key>
<array>
    <string>/Users/hashito/.nodebrew/current/bin/node</string>
    <string>/path/to/src/index.js</string>
    <string>--process</string>
</array>
<key>EnvironmentVariables</key>
<dict>
    <key>PATH</key>
    <string>/Users/hashito/.nodebrew/current/bin:/usr/local/bin:/usr/bin:/bin</string>
</dict>
<key>StartInterval</key>
<integer>1800</integer>

node is an absolute path because launchd does not go through a login shell, so the nodebrew PATH
from .zshrc never arrives. The only evidence was node: command not found in the log. Setting
PATH explicitly in EnvironmentVariables fixed it.

The other one: a Mac sleeps. An interval job does not fire while asleep. launchd runs the missed
invocation after wake, so if the queue is ordered by time, the backlog drains on wake in order.
Give up on holding the interval, and let the queue hold the ordering - then sleep turns from a
failure into a delay.

The result

One of the sites this bot announces: https://gadget.autoarticles.net

Takeaway

Dropping the X API for a webhook removed all the auth code. It also removed the answer to
"did it post?".

Generalised: always check how far an external service's success response actually reaches.
If a response only guarantees "accepted" and you record it as "completed", you lose the ability to
detect breakage. Measure the unguaranteed part through another path, or at minimum keep it as
null / unknown. The moment you fill it with true, your monitoring starts lying to you.

Let the sitemap decide which URLs your bot is allowed to announce

takahiro hashito — Thu, 10 Sep 2026 13:40:50 +0000

The problem

I run about 20 small content sites that update themselves daily, and a bot announces each day's new articles on X. Every so often the announcement pointed at a URL that returned 404.

The cause was easy to find. The bot built announcement URLs from my own bookkeeping. The daily job writes "today I added this slug" (a slug being the short string that identifies an article) into state.json, and the bot glued that slug onto the site's base URL.

That is a lie in disguise. The ledger proves that the job intended to add a page. It does not prove the page is live. If the build failed halfway, or the deploy did not finish, the ledger row is still there. An announcement must not trust the ledger.

So I changed the rule: the bot may only announce URLs that appear as a <loc> in the site's own sitemap.xml.

The design

sitemap.xml is the site declaring, publicly, which pages exist. It is meant for search engines, but as a list of live pages it is far more trustworthy than my ledger. It is a build artifact, so a failed build leaves it stale, and an undeployed build never reaches production at all.

The pipeline became:

Take today's slugs from the ledger (state.json).
Fetch https://<site>/sitemap.xml and parse it into a list of <loc> values.
Find a <loc> containing the slug. Only those become announcement candidates.
If a site resolves zero candidates, report the count and drop the site.

Step 3 failing means "in the ledger, not published." Do not announce it.

The core of it

Resolving a slug to a real URL is a plain substring match. locs below is the array of URLs pulled from the sitemap:

_resolveSlugUrl(slug, locs) {
  if (!slug) return null;
  const matches = locs.filter(u => u.includes(slug));
  if (matches.length === 0) return null;
  // prefer the article itself over /tags/ and other index pages
  const preferred = matches.find(u => !/\/tags?\//i.test(u));
  return preferred || matches[0];
}

Returning null on no match is the whole point. The tempting "helpful" version — fall back to building the URL from the base path — puts you right back where you started.

The /tags/ branch came out of real runs. Most of these sites publish /tags/<same word as the slug>, so the tag index matches before the article does. Tag pages return 200, so a pure liveness check cannot reject them. They exist; they are just not what I meant to announce.

Where it broke

One site resolved zero candidates every single day while publishing normally. Its article URLs share no characters with the ledger index. The ledger holds CVE (Common Vulnerabilities and Exposures — the globally unique identifier assigned to a single known security flaw) numbers, while the public URLs look like /article/post_<epoch_ms>_<random>. Measured on 2026-08-20: of 1338 <loc> entries, 0 contained a CVE ID.

The mechanism was behaving exactly as specified. It was also silently discarding a site that published every day.

The fix had two parts. First, count what you drop: sites that ended with zero candidates now show up in the report as {key:"ai-news", added:1, tried:3} — "one article added, three strategies tried, nothing resolved." That line is what made the problem visible at all. Second, add a fallback that reads <lastmod> (each page's last-modified timestamp) and treats same-day entries as today's publications — with a guard:

const dated = entries.filter((e) => e.lastmod && lastmodJstDate(e.lastmod) === logDate);
if (dated.length === 0) return [];
/* If *every* page carries today's lastmod, the site regenerated itself.
 * That is not the same as "these pages were written today." */
if (dated.length / entries.length > 0.5) return [];

One site regenerates all 26 of its pages daily, so every lastmod is today's date. Calling those "new articles" would be a lie to readers, hence the ratio guard. The CVE site, by contrast, had exactly one same-day lastmod, matching the ledger's "1 added."

There was also a timezone bug worth naming. <lastmod> is UTC (2026-09-04T21:23:46.517Z), my calendar day is JST, and I was comparing slice(0, 10) of both. Articles published between 00:00 and 09:00 JST never matched. Announcements silently vanished on nights the job ran late — a wonderfully hard bug to reproduce on purpose.

Liveness, in three states

The final check does an actual HTTP request, and its verdict is deliberately not a boolean:

const evaluate = (status) => ({
  ok: status >= 200 && status < 400,
  status,
  definitive: status >= 400 && status < 600,   // genuinely broken
  reason: status >= 200 && status < 400 ? 'ok' : `http-${status}`
});

definitive is set only when a numeric 4xx/5xx came back. A timeout or DNS failure leaves it false. "Broken" and "could not determine" must not collapse into one value, or a transient network blip permanently evicts a perfectly good page — quietly, which is the worst part.

One of the live sites: https://manga.autoarticles.net

Takeaway

Derive outward-facing correctness from the published artifact, never from your own records. Your records prove intent; the sitemap proves publication. And whenever you add a filter, count what it removes and print that count — a rejection nobody counts is indistinguishable from a rejection that never happened.

A dependency-free snapshot dashboard: http.server + sqlite3, and the ALTER TABLE trap

takahiro hashito — Wed, 09 Sep 2026 13:37:59 +0000

Background

I run more than twenty small sites as a solo developer. Publishing is automated; looking at the results was not. Google Analytics and Search Console are both fine for one site, but opening two consoles per site, aligning the date ranges, and remembering yesterday's number does not scale to twenty. I stopped doing it after three days.

So I put a small server on a Mac mini at home and started pushing every site's numbers to it a few times a day. It uses only the Python standard library, http.server and sqlite3. There is no pip install step, no framework, and nothing to keep updated.

This post is about the storage design, and specifically about one decision that paid off and one bug it caused.

How it works

Three pieces, nothing else.

A collector (Node) gathers Analytics and Search Console numbers for every site into one JSON object.
It sends that object to POST /api/ingest, authenticated with a single shared token in an X-Token header.
The Python server appends it to SQLite and serves the dashboard HTML from GET /.

The whole server boot is this:

from http.server import ThreadingHTTPServer, BaseHTTPRequestHandler
import sqlite3, json

if __name__ == "__main__":
    init_db()
    ThreadingHTTPServer(("0.0.0.0", PORT), H).serve_forever()

ThreadingHTTPServer rather than plain HTTPServer because the plain one is sequential: the browser's favicon request blocks the HTML response. Even with a single viewer, opening two tabs makes it feel broken.

The ingest handler returns the id of the row it just created:

def do_POST(self):
    if self.path != "/api/ingest":
        return self._send(404, "not found", "text/plain")
    tk = token()
    if not tk or self.headers.get("X-Token") != tk:
        return self._send(401, json.dumps({"error": "unauthorized"}), "application/json")
    n = int(self.headers.get("Content-Length", "0"))
    payload = json.loads(self.rfile.read(n) or b"{}")
    return self._send(200, json.dumps({"ok": True, "snapshot_id": ingest(payload)}), "application/json")

Returning snapshot_id is the design in one line. Each push creates a new row rather than updating an existing one.

Implementation

One snapshot = one parent row plus N child rows

The schema below is what init_db() runs on every boot. snapshots holds one row per push, and site_metrics holds one row per site within that push, linked by snapshot_id:

CREATE TABLE IF NOT EXISTS snapshots(
    id INTEGER PRIMARY KEY AUTOINCREMENT, ts TEXT, ts_jst TEXT, gcp_cost TEXT, meta TEXT);

CREATE TABLE IF NOT EXISTS site_metrics(
    snapshot_id INTEGER, site_key TEXT, label TEXT, url TEXT,
    ga_sessions_y REAL, ga_users_y REAL, ga_pv_y REAL,
    sc_clicks REAL, sc_impressions REAL, sc_ctr REAL, sc_position REAL, added INTEGER);

CREATE INDEX IF NOT EXISTS idx_sm_snap ON site_metrics(snapshot_id);

So a push of 23 sites writes 1 row into snapshots and 23 into site_metrics, all carrying the same snapshot_id. site_metrics deliberately has no primary key. A row is not "the current state of a site", it is "how that site looked at that push". Dozens of rows per site_key is the correct state. Nothing is ever updated or deleted; reads always filter by the latest snapshot_id, which is why the index is on that column.

Adding a metric is one ALTER TABLE

Metrics always grow. This table started with a handful of columns and now has 29, after splitting organic search out of total sessions, then adding a per-channel breakdown, then a traffic-quality figure.

Append-only storage makes that cheap, because nothing has to be backfilled:

def ensure_columns(c, table, cols):
    """Idempotent migration: ALTER TABLE ADD COLUMN only for columns that are missing.
    Existing rows are never rewritten; the new column is NULL for them, which is correct."""
    have = {r["name"] for r in c.execute(f"PRAGMA table_info({table})").fetchall()}
    for name, decl in cols:
        if name not in have:
            c.execute(f"ALTER TABLE {table} ADD COLUMN {name} {decl}")

PRAGMA table_info(<table>) is SQLite's introspection query: it returns one row per column. Building the set of existing names and adding only what is missing makes the function idempotent, meaning it can run on every boot and produce the same result no matter how many times it has run before.

The important part is that NULL on old rows is semantically right. A July snapshot has no organic-search column because that metric did not exist yet, not because the value was zero.

That distinction is enforced end to end: a failed fetch stores NULL, a measured zero stores 0. Collapse both into 0 and a flat line in the chart becomes unreadable forever after, because nothing tells you whether traffic died or collection did.

And that is exactly what breaks positional INSERT

Cheap column additions mean frequent column additions. And INSERT INTO t VALUES(?,?,?) names no columns, so it breaks the moment the column count changes.

Here is a reproduction I actually ran:

import sqlite3
c = sqlite3.connect(":memory:")
c.execute("CREATE TABLE m(snapshot_id INTEGER, site_key TEXT, clicks REAL)")
c.execute("INSERT INTO m VALUES(?,?,?)", (1, "coffee", 3))
print("before ALTER:", c.execute("SELECT * FROM m").fetchall())

c.execute("ALTER TABLE m ADD COLUMN impressions REAL")
print("after ALTER :", c.execute("SELECT * FROM m").fetchall())

try:
    c.execute("INSERT INTO m VALUES(?,?,?)", (2, "beer", 5))
except Exception as e:
    print("positional INSERT ->", type(e).__name__, e)

c.execute("INSERT INTO m(snapshot_id,site_key,clicks) VALUES(?,?,?)", (2, "beer", 5))
print("named INSERT ok  :", c.execute("SELECT * FROM m").fetchall())

Output:

before ALTER: [(1, 'coffee', 3.0)]
after ALTER : [(1, 'coffee', 3.0, None)]
positional INSERT -> OperationalError table m has 4 columns but 3 values were supplied
named INSERT ok  : [(1, 'coffee', 3.0, None), (2, 'beer', 5.0, None)]

Three things to take from that. ALTER TABLE ADD COLUMN does not damage existing rows. The new column is NULL for them. And the unnamed INSERT fails from the very next write onward.

It does not fail for one site. Every site goes through the same statement, so the entire push fails. The change that caused it was a display change, which puts the cause and the symptom far apart.

The fix is to define the column list once and build the SQL from it:

SITE_METRIC_COLS = ["snapshot_id", "site_key", "label", "url",
                    "ga_sessions_y", "ga_users_y", "ga_pv_y", "ga_sessions_7d", "ga_pv_7d",
                    "sc_clicks", "sc_impressions", "sc_ctr", "sc_position", "added",
                    "ga_organic_y", "ga_organic_7d", "ga_organic_engaged_7d"]

sm_sql = ("INSERT INTO site_metrics(" + ",".join(SITE_METRIC_COLS) + ") VALUES("
          + ",".join("?" * len(SITE_METRIC_COLS)) + ")")

Adding a column now touches three places: the CREATE TABLE, the ensure_columns list, and SITE_METRIC_COLS. Three sounds like a lot, but forgetting one fails loudly and immediately instead of silently reaching production.

Gotchas

"It works today" is not evidence that positional INSERT is safe. As the repro shows, there is no warning until the first write after a column is added. Name columns from the start in any table you expect to grow.
Separate NULL from 0 on day one. You cannot reconstruct the difference later; a stored 0 gives you nothing to decide with. Watch the collector too, where a ?? 0 default quietly destroys it.
http.server is sequential. Switching to ThreadingHTTPServer is a one-line fix, but until you do, one favicon request stalls the page.
ALTER TABLE ADD COLUMN is a metadata-only operation in SQLite, so it is fast regardless of row count. The flip side is that no default is written into existing rows. For this design, not writing one was the correct behaviour.

The result

A live example of the sites this dashboard tracks: https://hashitosystem.com

Wrap-up

Store operational metrics as append-only snapshots. Adding a metric costs one ALTER TABLE ADD COLUMN, no observation is ever rewritten, and NULL keeps its natural meaning of "not measured then".

In exchange, never use INSERT INTO t VALUES(?,?,?) in a table whose columns will grow. Keep the column list as a constant and generate the statement from it. If you make adding columns cheap, pair it with writes that survive the addition.

My English page shipped with Japanese error messages, because the logic owned the strings

takahiro hashito — Wed, 09 Sep 2026 13:32:14 +0000

Background

I ship small single-file browser tools. Every tool has a Japanese page and an English page, and the rule I set for myself is that the two pages share the exact same logic. Same functions, same parsing, same edge cases. Only the wording differs.

Today I added a Cache-Control analyzer and shipped it. Then I opened the English page and the result panel said this:

Browser cache
  Freshness lifetime  365d
  Remaining           365d
  State               新鮮

新鮮 means "fresh". The labels were English, the values were Japanese.

The mistake was in what the shared function returned

The shared analyzer looked like this. analyze() takes a Cache-Control value and returns freshness plus a list of findings.

var DIRECTIVES = {
  "max-age": { arg: "delta", who: "both",
    desc: "応答が新鮮でいられる秒数。Age を差し引いた残りで判定されます。" },
  "no-store": { arg: "none", who: "both",
    desc: "どのキャッシュにも保存させない。最も強い指定です。" },
  // ...
};

function freshness(limit, age) {
  if (limit === null) return { lifetime: null, state: "指定なし（ヒューリスティックに委ねられる）" };
  var rem = limit - age;
  return { lifetime: limit, remaining: rem, state: rem > 0 ? "新鮮" : "古い（要再検証）" };
}

Both pages loaded this same file, and each page defined its own label table:

// on the English page
var L = { browser: "Browser cache", lifetime: "Freshness lifetime", state: "State" };

So the page owned the labels and the logic owned the values. That split looked fine while I was writing it, and it is exactly the bug. The English page had no way to say "fresh", because the only place that word existed was inside a Japanese string literal in the shared file.

Nothing failed. No console error, no missing key, no fallback. The page rendered perfectly in two languages at once.

What I changed

I pulled every human-readable string out of the logic and turned it into a message pack the page supplies. The logic keeps the branching; the page keeps the wording.

The message pack below is generated per language and concatenated in front of the shared logic, so MSG and DIRTEXT are already in scope when the parser runs.

// generated per language, concatenated before the shared logic
var DIRTEXT = {
  "max-age": "How many seconds the response stays fresh. Freshness is measured against Age.",
  "no-store": "No cache may store the response at all. This is the strongest directive."
};
var MSG = {
  stHeuristic: "Not set (left to heuristics)",
  stFresh: "Fresh",
  stStale: "Stale (needs revalidation)",
  noCacheMaxAge: function (n) {
    return "no-cache is present, so every reuse revalidates even though max-age=" + n + ".";
  }
};

DIRTEXT is keyed by directive name and feeds the explanation shown under each directive; MSG holds everything else the parser needs to say. The shared logic then reads like the code below, where limit is the max-age in seconds and age comes from the Age response header.

// shared logic, now with no literals a reader can see
function freshness(limit, age) {
  if (limit === null) return { lifetime: null, state: MSG.stHeuristic };
  var rem = limit - age;
  return { lifetime: limit, remaining: rem, state: rem > 0 ? MSG.stFresh : MSG.stStale };
}

The branching is untouched. The only difference is that the three outcomes now name a key instead of spelling out a word, so the English page renders "Fresh" and the Japanese page renders the same state in Japanese from the same call.

Messages that interpolate a number became functions rather than templates with placeholders. That is deliberate: word order differs between the two languages, and a function lets each language put the number wherever it belongs instead of forcing both into one sentence shape.

I checked the result with a grep over the shared file. Zero lines containing CJK characters, apart from one code comment. If a Japanese literal ever comes back, that check finds it.

The part I did not expect

The obvious win is the English page being English. The unexpected win was that writing the two message packs side by side made me reread every message.

Three of them were vague in Japanese and I only noticed while translating. One said the equivalent of "this directive does not work", which is not useful. Writing the English version forced me to say why: private stops shared caches from storing the response, so s-maxage has no audience. I then rewrote the Japanese one to match.

Translation turned out to be a review pass I had not budgeted for.

The rule I am keeping

A shared function that returns a display string has silently chosen a language for every caller. If some callers are in a different language, that choice is a bug, and it is a bug that renders fine.

So: return keys, or take the strings as a parameter. Either way, the page decides the words.

Both tools are here:

https://hashitosystem.com/en/tools/cachecontrol/

If you maintain a page in more than one language, grep your shared modules for literals in your own first language today. Anything that comes back is a string your other pages cannot translate, and it will ship looking fine.

Concatenating scene-split videos with ffmpeg: the offset that only breaks at scale

takahiro hashito — Tue, 08 Sep 2026 14:38:32 +0000

What I built

I run a handful of small content sites, and I wanted short product-comparison videos generated from their article data automatically. The pipeline takes a script as JSON, and for each scene it synthesizes speech, records an HTML/CSS stage with headless Chromium, muxes the two with ffmpeg, and finally joins every scene into one file.

The part that ate the most of my time was the final join, so that is all this post is about. If your clips have a fixed duration, a single concat demuxer call is the whole story. But the duration of synthesized speech is not known ahead of time. What happens when you cross-fade clips whose length is variable is the real subject here.

For context, the TTS layer tries VOICEVOX, then edge-tts, then macOS say, in that order. Which engine ends up being used changes the duration for the same script. Variable length is not an accident; it is baked into the design.

The shape of the pipeline

The order of operations below is the answer to the problem.

script JSON
  |- per scene (in parallel)
  |    1. synthesize a wav per line with TTS, measure it with ffprobe
  |    2. accumulate line start times to fix the scene duration `total`
  |    3. pass that `total` to the recorder and capture the HTML stage
  |    4. trim the video to `total` and mux the audio -> scene_NN.mp4
  |- join every scene with xfade / acrossfade -> final.mp4

The obvious approach is "record the video first, fit the audio afterwards". Do that and the speech gets cut off when it is longer than the clip, or the clip sits in silence when it is shorter. Deciding that audio owns the duration, and moving that decision to the top of the pipeline, was the fix.

The interesting parts

1. Fix the duration from audio, then record

Synthesize a wav per line, accumulate start times with a GAP of 0.25 seconds between lines. The sum is the scene duration.

lines = scene.get("lines", []); timed = []; wavs = []; t = 0.0; GAP = 0.25
for i, ln in enumerate(lines):
    w = os.path.join(sdir, f"l{i}.wav")
    r = sh(["python3", TTS, "--text", ln["text"],
            "--voice", ln.get("voice", "metan"), "--out", w, "--engine", engine])
    d = json.loads(r.stdout.strip().splitlines()[-1])["duration"]
    timed.append({"text": ln["text"], "start": round(t, 3), "dur": round(d, 3)})
    wavs.append((w, d)); t += d + GAP
total = max(t, 2.0)

Having the TTS script report its own duration is what makes this work: the length is settled the moment the audio exists. timed then goes straight to the recorder as subtitle timings. The max(t, 2.0) is a floor so that a scene with zero lines (a title card, say) does not come out at zero seconds.

For joining the lines, instead of inserting silence between them with aevalsrc, I concatenate first and stretch the whole thing to total with apad.

filt = "".join(f"[{i}:a]" for i in range(len(wavs))) \
     + f"concat=n={len(wavs)}:v=0:a=1[c];[c]apad=whole_dur={total}[a]"

You can build a filtergraph that pads between every line, but it grows with the line count and becomes unreadable. Fitting the duration once at the end turned out to survive script edits much better.

The mux is just trimming the video to total and attaching the audio.

ffmpeg -y -i scene.webm -i scene.wav -c:v libx264 -pix_fmt yuv420p \
       -c:a aac -b:a 128k -shortest scene_01.mp4

2. The xfade offset is a position on the output timeline

This is the heart of it. The offset of xfade is not where the second input starts. It is the time at which the cross-fade begins within the output you have assembled so far. Because each join overlaps by xf seconds, the output gets shorter than the sum of the inputs, so adding raw clip durations does not line up.

fc = []; vlab = "0:v"; alab = "0:a"; off = durs[0] - xf
for i in range(1, n):
    nv = f"v{i}"; na = f"a{i}"
    fc.append(f"[{vlab}][{i}:v]xfade=transition=fade:duration={xf}:offset={off:.3f}[{nv}]")
    fc.append(f"[{alab}][{i}:a]acrossfade=d={xf}[{na}]")
    vlab = nv; alab = na; off += durs[i] - xf
total = sum(durs) - (n - 1) * xf

Drop the - xf in off += durs[i] - xf and everything drifts further back with every scene you add. With three scenes it just feels "slightly off" and you move on; it only becomes unmistakable around ten. You have to raise the scene count to see it.

Concretely, with 10-second scenes and xf = 0.5:

joining clip	correct offset	offset without `- xf`
2nd	9.5	9.5
3rd	19.0	20.0
4th	28.5	30.0
10th	85.5	90.0

The first join matches exactly. A minimal two-clip test can never catch this.

Note that acrossfade takes no offset at all. It simply overlaps d seconds with whatever came before, so unlike the video side you do not carry an accumulator. The two filters look symmetric and are not; worth re-reading the docs every time you write this.

3. A short scene makes cross-fading impossible

Since durations are variable, a scene shorter than xf will eventually show up. Feeding that to xfade produces broken output, so detect it and fall back to a plain concatenation.

if n < 2 or xf <= 0 or any(d <= xf + 0.2 for d in durs):
    lst = os.path.join(wd, "concat.txt")
    open(lst, "w").write("".join(f"file '{s['mp4']}'\n" for s in scenes_out))
    sh(["ffmpeg", "-y", "-f", "concat", "-safe", "0", "-i", lst,
        "-af", LOUDNORM, "-c:v", "libx264", "-preset", "veryfast",
        "-pix_fmt", "yuv420p", "-c:a", "aac", final])
    return final

The + 0.2 margin exists because the duration ffprobe reports and the real container duration differ slightly, and I did not want clips sitting exactly on the boundary to blow up.

4. Background music and final loudness

The music is not length-matched at all. It is looped in with -stream_loop -1 and cut to the program by amix with duration=first. loudnorm runs exactly once, at the end.

LOUDNORM = "loudnorm=I=-16:TP=-1.5:LRA=11"
fc.append(f"[{bi}:a]volume=0.10,apad[bg]")
fc.append(f"[{alab}][bg]amix=inputs=2:duration=first[amx]")
fc.append(f"[amx]{LOUDNORM}[aout]")

The important part is not applying loudnorm per scene. Normalizing scene by scene lifts the quiet scenes and flattens the dynamics of the whole video. Normalize the final output once.

Things that bit me

Forgetting to accumulate the offset (above). It does not reproduce with two scenes, so write the test with at least four.
Choosing between the concat demuxer and filter_complex. When there is no music and a plain join will do, -c copy is dramatically faster, but it assumes every scene shares a codec, resolution and frame rate. If the recorder does not pin -r 30, -c copy desynchronizes the audio immediately.
Forgetting -shortest. Even with the video trimmed to total, some containers leave a few frames at the tail. With the flag it reliably stops at the shorter stream.
Parallelism. Scenes run TTS, recording and mux through a ThreadPoolExecutor, but the default is three workers. Raise it and headless Chromium eats memory until a recording dies mid-way. Even on a fast machine, not being greedy finishes sooner.

The result

One of the sites where these videos are published: https://manga.autoarticles.net

Wrapping up

When your source material has a variable duration, do not try to reconcile it at the join. Put the stage that decides the duration (here, speech synthesis) at the top of the pipeline and make the video, the subtitles and the lip sync all follow that number. Join exactly once, at the end.

Carrying the xfade offset as an accumulator is the same principle. Map every clip onto one shared output timeline instead of tracking each clip's local time, and the arithmetic holds no matter how many scenes you add. Skip that, and you get the worst kind of bug: one that only appears once the input grows.

Designing a zero-dependency build.js: static sites assembled from JSON

takahiro hashito — Mon, 07 Sep 2026 13:51:51 +0000

Introduction

I run more than twenty static sites as a solo developer. Article data lives in JSON, and a single build.js at the root of each repository turns it into static HTML. That output is served from Firebase Hosting. The sites update automatically four times a day.

The build.js has zero external dependencies. Both dependencies and devDependencies are absent from package.json. The only modules it loads are Node built-ins (fs, path, crypto) plus a local fleet-sites.json.

$ node -e "const p=require('./package.json');console.log(p.dependencies, p.devDependencies, p.scripts)"
undefined undefined { test: 'node --test test/' }

$ grep -oE 'require\("[^"]+"\)' build.js | sort -u
require("./fleet-sites.json")
require("crypto")
require("fs")
require("path")

The two undefined values mean the keys do not exist at all, not that they are empty objects. npm install is never needed.

I did not skip a static site generator for speed or for ideology. I did it because the one thing that matters here is that the build succeeds every single day, unattended. This post covers three design decisions that came out of that constraint.

The overall picture

The whole pipeline is shown below. Two JSON files go in, one script runs, and a directory of static files comes out.

data/articles.json   array of articles (one object per article)
data/site.json       site config and categories
        |
        v
build.js             validate -> sort -> render with template literals
        |
        v
public/              articles, categories, tags, sitemap.xml, feed.xml, 404

That is the entire build: read two JSON files, write a directory of HTML. There is no template engine. HTML is assembled with JavaScript template literals. A full build of 147 articles finishes in well under a second.

$ time node build.js
Done. 147 articles, 8 categories, 314 tags.
node build.js  0.76s user 0.21s system 115% cpu 0.832 total

The core of the implementation

1. Make the last sort key unique so the build is deterministic

List ordering uses three keys: freshness bucket, then newest first by date, then slug. The slug is not there for presentation. It is there so that the same input always produces the same output.

// Shared comparator: freshness bucket -> newest date first -> slug (keeps the build deterministic).
function byFreshness(a, b) {
  return (
    freshRank(a) - freshRank(b) ||
    String(b.date).localeCompare(String(a.date)) ||
    String(a.slug).localeCompare(String(b.slug))
  );
}

Articles added on the same day are a daily occurrence. If the comparison stops before the third key, the result of Array.prototype.sort depends on input order. Move a record inside the JSON file and the ordering changes, which means every page shows up as a diff even though nothing meaningful changed. The one article that actually changed gets buried in noise.

You can verify the fix by building twice and hashing the output.

$ for i in 1 2; do find public -name '*.html' -print0 | sort -z | xargs -0 cat | shasum; node build.js >/dev/null; done
c821dc4ca0dcf711564a3890d89de71e54a84a20  -
c821dc4ca0dcf711564a3890d89de71e54a84a20  -

Identical. Rebuilding does not move a single byte.

2. Anchor "today" to real time, not to the data

These sites cover time-limited sales, so entries that are likely over get pushed down the list. The choice of reference date decides how that behaves.

// The ordering baseline is "today" in JST. Using the survey date instead would make
// stale data look like it is still within its deadline.
const ORDER_TODAY_JST = new Date(Date.now() + 9 * 3600e3).toISOString().slice(0, 10);
const STALE_DAYS = 14;

function isStale(a) {
  if (isEnded(a)) return false;
  if (isExpiredToday(a)) return true;
  if (a && a.metrics && a.metrics.deadline) return false; // a published deadline beats our guess
  const conf = a && (a.confirmed || a.date);
  return !!conf && daysUntil(conf, ORDER_TODAY_JST) < -STALE_DAYS;
}

If the baseline were the date the data was collected, a site whose updates had stopped would look more current, not less. That is a failure mode that hides itself, which makes it the worst kind. Anchor time-based checks to real time.

The branch that skips the staleness rule when metrics.deadline exists follows the same principle: a deadline published by the source outranks our inference. Do not let a guess override a stated fact.

3. Put validation inside the build and fail on unbacked claims

The worst outcome for these sites is publishing a discount that does not exist. So the build scans for assertive claims such as "50% OFF", "half price", or "-> 1,980 yen", and only lets an article through when it has both a confirmed date and a source URL. Otherwise the build stops.

const assertive =
  /(\d{1,3}\s*[%％]\s*(OFF|オフ|還元(?!率)|割引)|半額|実質\s*無料|→\s*\d+\s*円|\d{2,4}\s*円均一)/;

The character class [%％] covers both the ASCII percent sign and its full-width counterpart, and there is a story behind that. For a long time this guard only matched the ASCII %. Meanwhile, the rendering code in the same build.js that draws the discount badge had handled [%％] from the beginning. Rendering understood full-width characters; validation did not.

As a result, an article written as 50％OFF never entered the guard at all and could ship without a source or a confirmation date. Measurement found three such articles. All three happened to carry both fields, so the published output was fine, but that was not the guard passing them; it was the guard never looking.

The lesson is not "remember full-width characters in your regex". It is that the moment you define the same rule in two places, one of them will eventually be fixed alone.

Things that bit me

Escaping HTML is the obvious cost of having no dependencies. You write it yourself, and then you have to route every output path through it: <title>, meta tags, JSON-LD (structured data for search engines, embedded as JSON inside a <script> tag), and href values. Raw <, > and & must never reach the output. Cutting corners here breaks pages.

The other one is handling artifacts that may not exist yet. Per-article Open Graph images are produced by a separate script, so sometimes they are missing.

function ogRel(a) {
  return fs.existsSync(path.join(PUB, "articles", `${a.slug}.og.png`))
    ? `/articles/${a.slug}.og.png`
    : "/og-image.png";
}

Check for the file and fall back to the shared image. Without these three lines, one failed OG generation run leaves every new article card pointing at a 404. Not letting another stage's failure corrupt this stage's output is worth doing regardless of your dependency count.

Tests run on node --test, which again needs nothing extra.

$ node --test test/
# tests 196
# pass 196
# fail 0

The result

Here is one of the sites this actually runs: https://ocha.autoarticles.net

Conclusion

Zero dependencies was not about being lightweight. It was about making sure a daily build never fails for a reason I did not write. Three decisions carried the most weight:

Give the sort a unique final key so the build is deterministic. Diffs stay readable
Anchor time-based logic to real time. Basing it on data timestamps hides breakage instead of exposing it
Validate inside the build and fail on unbacked claims, but never define the same rule in two places

The third one cost the most. Because the character class was written separately for rendering and for validation, the check silently missed cases for months. If a rule exists in two places, collapse it into one.

Routing 95 automation scripts through one Node.js entry point

takahiro hashito — Sun, 06 Sep 2026 14:38:00 +0000

Background

I run about 20 small sites as a solo side project. Behind them sits a pile of what I call deterministic scripts: plain Node programs that produce the same output from the same input, with no LLM involved. They fetch CVE data (CVE stands for Common Vulnerabilities and Exposures — the shared identifier assigned to each publicly disclosed software vulnerability), run SEO audits, lint for stray emoji, gate builds before deploy, and wrap the deploy itself.

Once that pile grows, two problems show up on the caller's side.

Nobody remembers the paths. Is it node automation/scripts/common/preflight.js or node automation/scripts/seo/audit.js? Every script sits somewhere slightly different.
Shared setup cannot be applied everywhere. Say every deploy path needs a longer timeout. You have to copy that line into every script that shells out to deploy. Miss one, and that single path breaks quietly.

So I put a single entry point, automation/run.js, in front of all of them. It is 344 lines and dispatches 95 ops. This post is about how it is built.

How it works

Every invocation has the same shape:

node automation/run.js <op> [--json '<JSON>'] [--json-file <path>] [other flags]

Internally:

node automation/run.js preflight --site coffee
        |
        v
  [ run.js ]
    1. resolve op -> script path via the OPS table
    2. expand --json / --json-file into argv
    3. inject run-time env defaults      <-- the real point
    4. spawnSync the child (stdio: inherit)
    5. exit with the child's exit code
        |
        v
  automation/scripts/common/preflight.js

OPS is just an object literal:

const OPS = {
  "whats-due": "whats-due.js",
  "update-state": "common/update-state.js",
  "emoji-lint": "common/emoji-lint.js",
  preflight: "common/preflight.js",
  "site:deploy": "common/site-build-deploy.js",
  // ... 95 entries
};

Names like site:deploy use a colon to namespace groups (seo:, cve:, anime:). The op name is decoupled from the directory layout, so moving a script does not change how it is called.

Implementation

1. The dispatcher

It is a table lookup plus spawnSync, and nothing else.

const rel = OPS[op];
if (!rel) {
  process.stderr.write(
    `unknown op: ${op}\navailable: ${Object.keys(OPS).join(", ")}\n`,
  );
  process.exit(2);
}

const target = path.join(SCRIPTS, rel);
const finalArgs = [target, ...opFlags, ...jsonArgs, ...passthrough];
const res = spawnSync(process.execPath, finalArgs, {
  stdio: "inherit",
  env: process.env,
});
process.exit(res.status == null ? 1 : res.status);

Two things matter here.

It spawns a child process instead of require()-ing the script. Each script must also work standalone (node automation/scripts/common/preflight.js --site coffee) and calls process.exit() itself. Requiring it in-process would let that exit tear down run.js too, destroying the meaning of the exit code.

It uses stdio: "inherit", so the child writes straight to the terminal with no buffering. One op — re-verifying every published CVE against primary sources — takes about 35 minutes for 299 records, and its progress lines stream out live.

The exit code is res.status == null ? 1 : res.status. status is null when the child died from a signal, so that branch exists to avoid reporting a killed process as success.

2. Run-time defaults, applied exactly once

This is the reason the single entry point earns its keep.

/* Deploy defaults go here, once.
 * Five different wrappers shell out to firebase deploy and pass env straight
 * through. Because the entry point is a single file, setting it here reaches
 * every path (no copying the same line into each wrapper). */
applyDeployEnvDefaults(process.env);
applyNetworkEnvDefaults(process.env);

applyNetworkEnvDefaults works around a real failure on my Mac: Node's fetch (undici) picks a broken IPv6 route and every request dies, while curl works fine. If that workaround lived in each fetching script, the one script I forgot would return "zero new items" — indistinguishable from a genuinely quiet day.

This is not the same as putting applyNetworkEnvDefaults() in a shared library and calling it from each script. A library call can be forgotten. An entry point cannot be skipped, because nothing runs without passing through it.

3. When `--json` meant two different things

Arguments can be passed as JSON, and keys expand into flags:

// {"site":"x"}       -> --site x
// {"days":28}        -> --days 28
// {"deploy":true}    -> --deploy       (false is dropped)
// {"episodes":[1,2]} -> --episodes 1,2
// {"_":["a","b"]}    -> a b            (positional)

That collided with existing usage. --json carried two meanings:

(1) pass arguments as JSON: --json '{"site":"x"}' — a run.js feature
(2) emit machine-readable output: --json — a flag several scripts already had

run.js always assumed (1), so the six ops documented with (2) (seo:goals, seo:triage, issues:sync, crosspost:verify, crosspost:draft-check, metrics:daily) failed with --json parse failed and exit 2 when invoked exactly as documented. One script had quietly invented a --json-out alias to dodge it, and that workaround is what exposed the bug.

The fix is to look at the next token:

const raw = argv[i + 1];
if (raw === undefined || raw.startsWith("-")) {
  passthrough.push(t);   // meaning (2): hand the flag to the child untouched
  continue;
}
i++;
jsonArgs = jsonToArgv(JSON.parse(raw));  // meaning (1): expand as arguments

JSON always starts with { or [, so it can never be confused with a token starting with -. When you cannot remove ambiguity with types, proving the two value sets do not overlap is the cheapest way out.

In practice:

$ node automation/run.js seo:goals --json
# --json is forwarded to the child, which prints JSON output

$ node automation/run.js update-state --json '{"site":"coffee","count":1}'
# expands to --site coffee --count 1 before reaching the child

Gotchas

Ops that double as subcommands. run:start, run:complete and run:check all call the same script in different modes. A separate OP_FLAGS table holds the flags to prepend per op. Argument order is [target, ...opFlags, ...jsonArgs, ...passthrough]; opFlags go first so a user-supplied duplicate flag wins by coming later.

Growth turns features invisible. At 95 ops, 23 of them appear in no runbook at all. A single entry point makes calling easy, but discoverability does not come for free. node automation/run.js list prints every op and its target path, which tells you the name — not how to use it. That is still an open item on the documentation side.

Assume some ops are slow. The CVE re-verification op takes roughly 35 minutes across 299 records. Because of stdio: "inherit", its progress lines ([cve:recheck] 144/299 done, 1006s elapsed) stream through untouched. A wrapper that swallows child output makes a long-running op indistinguishable from a hung one.

The result

One of the sites this pipeline updates daily: https://cve.autoarticles.net

It aggregates vulnerability information, and every step — fetching, re-verification, gating and deploy — runs through run.js.

Wrap-up

The usual pitch for a single entry point is "you stop memorizing commands." What actually paid off was being able to apply run-time behaviour to every path with certainty: env defaults, network workarounds, deploy timeouts. Put those in a shared library and someone forgets the call; the forgotten path then fails quietly. Put them at the entry point and there is no path that skips them.

The flip side is that consolidation does not solve discovery. Ninety-five ops behind one door still need a document describing what is in the room.

DEV Community: takahiro hashito

Embedding JSON-LD in a static site with template literals, and the one escape you cannot skip

Background

How it works

Implementation

1. head() turns the array into script tags

2. Pages just build objects and pass them in

Gotchas

JSON.stringify alone lets the script tag close early

Add a test that fails if the replace disappears

The result

Wrap-up

One service account for 21 GA4 and Search Console properties, with zero dependencies

Background

How it works

Implementation

Gotchas

The result

Wrap-up

Generating per-article og:image with headless Chrome and zero npm dependencies

The problem

The pipeline

The interesting parts

Let CSS do the drawing

Take the shot

Do not redraw what has not changed

Four traps

1. --window-size height is the viewport height

2. Cropping without an imaging library

3. Windows produced zero images, silently

4. The same shape of bug, somewhere else

The result

Takeaway

Your CVSS >= 7.0 filter is silently dropping the CVEs you most want

The problem: too much input

The shape of the collector

What went wrong

The fix: push the filter into the source

Two things that bit me

Demo

Takeaway

Decoding a UUID by hand: version, variant, and the embedded timestamp

Background

How it works

Implementation

Gotchas

The result

Wrap-up

Why I dropped the X API for an IFTTT webhook - and what it cost me

Introduction

The overall shape

The core of the implementation

Things that bit me

The result

Takeaway

Let the sitemap decide which URLs your bot is allowed to announce

The problem

The design

The core of it

Where it broke

Liveness, in three states

Takeaway

A dependency-free snapshot dashboard: http.server + sqlite3, and the ALTER TABLE trap

Background

How it works

Implementation

One snapshot = one parent row plus N child rows

Adding a metric is one ALTER TABLE

And that is exactly what breaks positional INSERT

Gotchas

The result

Wrap-up

My English page shipped with Japanese error messages, because the logic owned the strings

Background

The mistake was in what the shared function returned

What I changed

The part I did not expect

The rule I am keeping

Concatenating scene-split videos with ffmpeg: the offset that only breaks at scale

What I built

1. `--window-size` height is the viewport height

3. When `--json` meant two different things