Three things that cost me real time building a 25-source news aggregator

许立 — Sun, 27 Sep 2026 08:16:16 +0000

I built a small forestry news aggregator: 25 sources, rolling 10-day window, every headline machine-translated into Chinese/English/Spanish, every item linking back to the publisher. It runs on Astro (static) + Netlify Functions + Netlify Blobs as a key-value store.

The aggregation itself was the easy part. These three things were not.

1. Google News RSS links are JavaScript shells

If you've ever pulled https://news.google.com/rss/search?q=..., you know the <link> values look like this:

https://news.google.com/rss/articles/CBMiWkFVX3lxTE...

Fetching that URL and looking for the article body gets you nothing useful — it's a shell page that resolves the real publisher URL client-side. For an aggregator that wants to store a short summary, that's a dead end.

The redirect target is available, but not from the HTML. You have to:

Fetch the shell page.
Pull two attributes out of the markup: data-n-a-sg and data-n-a-ts.
POST them, plus the article ID, to Google's internal batchexecute endpoint.
Scrape the real URL out of the response.

The request body is a nested array serialized as f.req:

const html = await (await fetch(shellUrl)).text();
const sg = /data-n-a-sg="([^"]+)"/.exec(html)?.[1];
const ts = /data-n-a-ts="([^"]+)"/.exec(html)?.[1];
if (!sg || !ts) return '';   // shell changed shape — give up, don't guess

const payload = [[[
  "Fbv4je",
  JSON.stringify([
    "garturlreq",
    [["X","X",["X","X"],null,null,1,1,"US:en",null,1,null,null,null,null,null,0,1],
     "X","X",1,[1,1,1],1,1,null,0,0,null,0],
    articleId,
    ts,
    sg,
  ]),
  null,
  "generic",
]]];

const res = await fetch(
  "https://news.google.com/_/DotsSplashUi/data/batchexecute",
  {
    method: "POST",
    headers: { "content-type": "application/x-www-form-urlencoded;charset=UTF-8" },
    body: "f.req=" + encodeURIComponent(JSON.stringify(payload)),
  }
);
const real = (await res.text()).match(/https?:\/\/(?!news\.google\.com)[^\\"'\[\]\s]{10,300}/)?.[0] || '';

Two warnings from experience:

This is undocumented and can break without notice. Guard every step and return an empty string on failure — never guess a URL. My pipeline treats an unresolved link as "no summary available" and still publishes the headline with a link to the Google News page.
Expect partial success. Some publishers block the follow-up fetch even after you resolve the URL. In my last run, about a third of Google News items ended up without an in-site summary. That's a publisher-access problem, not a resolver problem, and no amount of retrying fixes it.

2. Static hosting rebuilds quietly eat your monthly budget

This one hurt because the design was obviously correct: the site is static, so when new items land in the store, rebuild to publish them.

Then I looked at the numbers. Netlify charges 15 credits per production deploy, and the plan I'm on includes 1000 credits per billing period. A daily rebuild is:

15 credits × 30 days = 450 credits

Nearly half the budget spent on publishing headlines. And the collector runs more than once a day in practice.

The fix was to stop conflating "fetch new data" with "rebuild the site":

The collector writes to the store and does not trigger a build.
Pages ship with a static snapshot (so crawlers and first paint are fine).
On load, the page calls a small read API, and if the store is newer than the snapshot, it re-renders the list client-side.

fetch(`/api/news/list?days=10&per=60`)
  .then(r => r.ok ? r.json() : null)
  .then(j => {
    // Static snapshot is already the newest thing we have — leave the DOM alone.
    if (!j?.ok || (j.updatedAt || 0) <= SNAPSHOT_UPDATED_AT) return;
    host.innerHTML = render(j.items);
  })
  .catch(() => {});   // can't reach the API? keep reading the snapshot

Now daily updates cost zero deploys, and credits are only spent when I actually change code.

The trade-off is real and worth stating: the static snapshot is authoritative for SEO and first paint, the client update is a progressive enhancement. If the API is down, readers still get the previous build.

3. Synchronous functions die at 10 seconds, and they die midway

My admin page has a "fetch now" button. It called an endpoint that ran the whole pipeline inline: fetch 25 sources → filter → dedupe → write to the store → generate summaries → translate.

The response was a consistent 504 Inactivity Timeout. But the failure mode was worse than a plain error: the collection and the store write had already completed, so new items existed — and then the summarisation phase got killed. The result was a batch of headlines with no summaries, plus a button that looked like it had failed. Users see a broken feature; the data is actually half-processed.

Netlify's synchronous functions have a 10-second limit. Background functions — identified by a -background suffix on the filename — get a much larger budget (15 minutes on my plan).

So the endpoint now does nothing but kick off the background function and return:

// POST /api/news/manage  { action: "runNow" }
const target = new URL('/.netlify/functions/news-collect-background', url.origin).href;
const r = await fetch(target, {
  method: 'POST',
  headers: { 'x-admin-token': process.env.FORUM_ADMIN_TOKEN || '' },
});
if (!(r.status === 202 || r.ok)) return json({ ok: false, error: 'could not start' }, 502);
return json({ ok: true, background: true });

The response time went from "504 after 31 seconds" to 0.7 seconds.

Two details that made this actually usable:

Write a progress marker first. A background job returns immediately, so the UI has nothing to show between the click and the first result. Write a started marker before doing any work, then update it per phase:

await saveState({ lastRun: { at: Date.now(), phase: 'started', startedAt } });
// …then: collected → digested → digest-filled → done

The frontend polls the read API every 5 seconds and renders the phase. Without the marker the button just sits there looking dead for a minute.

Protect the trigger. A background function URL is reachable directly. Without a token check, anyone can hit it in a loop and burn your model budget. It takes the admin token in a header and compares in constant time.

What I'd do differently

Treat "compute" and "publish" as separate concerns from day one. The rebuild-on-change instinct is correct for a blog and wrong for anything that updates on a schedule with per-deploy costs.
Measure the failure mode, not just the failure. The 504 wasn't the bug; the bug was that a killed function left the store in a half-processed state that looked like a different problem.
Assume any undocumented endpoint you depend on will break. The Google News resolver is the most fragile part of the system and it's wrapped so that failure degrades to "no summary" rather than "no item".

The site is at foreststellar.com if you want to see the result. The thinnest coverage is Latin America — if you read Spanish or Portuguese forestry press, I'd genuinely like to know what sources you actually open.

DEV Community: 许立

Three things that cost me real time building a 25-source news aggregator

1. Google News RSS links are JavaScript shells

2. Static hosting rebuilds quietly eat your monthly budget

3. Synchronous functions die at 10 seconds, and they die midway

What I'd do differently