DEV Community: Timothy Kelvin

Three event sites already hand you their data. Here's where each one hides it 🗂️

Timothy Kelvin — Mon, 09 Nov 2026 05:40:41 +0000

No CSS selectors, no headless browser. The data was in the HTML all along.

This week I built one that collects upcoming events in a city from Eventbrite, Meetup and Luma and puts them in one list. I expected three scrapers full of fragile selectors. I ended up writing almost none, because all three sites already ship their event data inside the page, in a form a program can read. Each one just keeps it somewhere different.

🚦 First, the rules

Before writing any code I read each site's robots.txt, keeping only the rules that apply to every bot (User-agent: *):

Eventbrite blocks its RSS and Atom feeds, checkout links and a few API paths. The city search pages (/d/...) and event pages are open.
Meetup blocks its API paths (/api, /meetup_api) and its feeds. The search page (/find/) and event pages are open.
Luma's robots.txt only has rules for Googlebot.

robots.txt isn't the whole story, so read each site's terms as well before you build on it. I also send an honest User-Agent that names the tool, and wait 1.5 seconds between requests to the same site.

🧩 Hiding place 1: schema.org JSON-LD

All three sites put a <script type="application/ld+json"> block in their pages, with each event described in schema.org's Event vocabulary. It's there for search engines, and it's the most standard of the three hiding places:

{
  "@type": "Event",
  "name": "Tech Networking Event In NYC",
  "startDate": "2026-11-18",
  "location": {
    "@type": "Place",
    "name": "Tap Haus 33",
    "address": { "streetAddress": "14 East 33rd Street", "addressLocality": "New York" },
    "geo": { "latitude": "40.7473297", "longitude": "-73.9842987" }
  }
}

The events can be nested inside @graph arrays or ItemLists, and the type can be MusicEvent or BusinessEvent rather than plain Event. So instead of guessing the shape, I walk the whole object and keep anything whose type ends in "Event":

export function jsonLdEvents(html) {
    const events = [];
    const walk = (node) => {
        if (!node || typeof node !== 'object') return;
        if (Array.isArray(node)) return node.forEach(walk);
        const type = [].concat(node['@type'] ?? []).join(' ');
        if (/Event\b/.test(type)) events.push(node);
        for (const value of Object.values(node)) walk(value);
    };
    for (const m of html.matchAll(/<script[^>]*application\/ld\+json[^>]*>([\s\S]*?)<\/script>/gi)) {
        try { walk(JSON.parse(m[1])); } catch { /* one broken block shouldn't hide the rest */ }
    }
    return events;
}

JSON-LD is great for the basics. But it's the search-engine version of each event, so it leaves things out. Meetup's had no coordinates and no end time, and Eventbrite's search page had no prices. For the rest, I went to the data each site's own front end uses.

🧩 Hiding place 2: a JavaScript object (Eventbrite)

Eventbrite's search page is about 800 KB, and most of that is one line:

window.__SERVER_DATA__ = {"env":{...},"search_data":{"events":{"results":[...],"pagination":{...}}}, ...};

It isn't a JSON script tag, so you can't just grab a script's contents and parse them. It's a JavaScript assignment, followed by more code. I find the marker, then walk forward counting braces until the object closes, skipping anything inside strings. Event descriptions are full of { and }, so strings have to be skipped, escaped quotes included:

export function objectAfter(html, marker) {
    const start = html.indexOf('{', html.indexOf(marker));
    let depth = 0, inString = false, escaped = false;
    for (let i = start; i < html.length; i++) {
        const c = html[i];
        if (inString) {
            if (escaped) escaped = false;
            else if (c === '\\') escaped = true;
            else if (c === '"') inString = false;
            continue;
        }
        if (c === '"') inString = true;
        else if (c === '{') depth++;
        else if (c === '}' && --depth === 0) return JSON.parse(html.slice(start, i + 1));
    }
    return null;
}

That object holds 20 events a page, each with the venue's full address, coordinates, time zone, categories and an is_online_event flag, plus page_count for paging with ?page=2.

🧩 Hiding place 3: Next.js page data (Luma and Meetup)

Luma and Meetup are both Next.js apps, so each page carries <script id="__NEXT_DATA__"> with the props it was rendered from.

Luma is the easy one. A city page like luma.com/london has its events at props.pageProps.initialData.data.events, with the ticket price in cents, whether it's free, the guest count and the host calendar. Luma's discover page lists the cities it covers (87 when I checked), each with coordinates, so I pick the nearest one to the place the user typed.

Meetup ships an Apollo cache instead, which is normalized: objects point at each other by reference rather than nesting.

const state = nextData.props.pageProps.__APOLLO_STATE__;
// ROOT_QUERY['eventSearch:{"filter":{"city":"New York",...}}'] = { edges: [{ node: { __ref: 'Event:316963808' } }, ...] }
// state['Event:316963808'] = { title, dateTime, venue, group: { __ref: 'Group:11913282' }, rsvps: { totalCount: 147 }, ... }
const resolve = (v) => (v && v.__ref ? state[v.__ref] : v);

The cache key even records the filter Meetup actually applied, city and country included. That turned out to matter.

🐛 The bug: zero Meetup events, but only sometimes

With a keyword ("tech"), Meetup returned about 20 to 30 events per city. Without one, it returned zero for every city. No error, just nothing.

The page wasn't empty. Without a keyword, Meetup doesn't run a search at all. It shows recommended events, and those are cached under a different key with a different shape:

// with a keyword:    eventSearch:{...}       -> edges: [{ node: { __ref: 'Event:1' } }]
// without a keyword: recommendedEvents:{...} -> edges: [{ __ref: 'RecommendedEventsEdge:{"node":{"id":"1"}}' }]

Each edge is a reference too, so the event is two hops away. The fix was to accept either key and resolve twice:

const key = keys.find((k) => k.startsWith('eventSearch')) ?? keys.find((k) => k.startsWith('recommendedEvents'));
const events = state.ROOT_QUERY[key].edges.map((edge) => resolve(resolve(edge)?.node));

Lesson: when a parser returns zero, test with and without every optional input. "Empty" was a different page shape, not an empty page.

📍 Every site wants the place spelled differently

The user types "London". Each site wants something else:

Site	What it wants
Eventbrite	`/d/united-kingdom--london/` (`/d/ny--new-york/` in the US)
Meetup	`location=gb--London` (`us--ny--New York` in the US and Canada)
Luma	the nearest of its city slugs: `london`

So the first step is a geocoder (OpenStreetMap's Nominatim) for the country, the state code and coordinates. It had two surprises:

OpenStreetMap calls London "Greater London", and New York "City of New York". My first London run searched "Greater London" on Eventbrite and Meetup. (Luma was fine, because it goes by coordinates.) Now a prefix like that is dropped when what's left is what the user typed.
Meetup quietly falls back to your own location. When I sent a location string it didn't understand (location=london, or bare coordinates), it didn't fail. It returned events near my IP address, which is in Nairobi. The venues gave it away: every one of them was in or around Nairobi. Now every source checks the place it actually searched, and a mismatch becomes an error row for that city instead of wrong events.

A real mismatch needs care too. Eventbrite searches "Brooklyn" as New York, which is fair, since Brooklyn is part of the city. So the check accepts the parent city the geocoder reports.

🕐 Three time formats, one output

Eventbrite: "start_date": "2026-11-18", "start_time": "18:30", "timezone": "America/New_York"
Luma: "start_at": "2026-11-08T23:00:00.000Z" plus a time zone
Meetup: "dateTime": "2026-11-10T19:30:00-05:00"

I output local time with the offset (2026-11-18T18:30:00-05:00), because that's when people turn up, and it still converts cleanly. Node's Intl gives the offset for any zone at a given moment:

function offsetAt(timeZone, utcMs) {
    const name = new Intl.DateTimeFormat('en-US', { timeZone, timeZoneName: 'longOffset' })
        .formatToParts(new Date(utcMs)).find((p) => p.type === 'timeZoneName').value; // "GMT-05:00"
    const m = name.match(/GMT([+-])(\d{1,2}):?(\d{2})?/);
    return m ? `${m[1]}${m[2].padStart(2, '0')}:${m[3] ?? '00'}` : '+00:00';
}

For Eventbrite's local times there's a catch: to know the offset you need the instant, and to know the instant you need the offset. I guess with the naive time, then ask again at the corrected instant, so a time near a daylight-saving change lands on the right side. The unit tests cover New York in November (-05:00) and July (-04:00), London (+00:00) and Kolkata (+05:30).

🔁 The same event on two sites

Organizers often list one event in two places. A test run in London found "Tech-Baithak London Launch Meet" on both Eventbrite and Meetup. The merge key is deliberately simple: the title's words, lowercased and stripped of punctuation, plus the local start date. The first listing is kept, the other goes into an alsoListedOn field, and any field the first listing lacks (coordinates, price) is filled from the second.

🎯 What I learned

Look for the data before you write selectors. JSON-LD, __NEXT_DATA__ and server data objects survive redesigns that break CSS selectors.
Every site hides it differently. One extractor for each kind of hiding place, plus one normalizer, beat three full scrapers.
Normalized caches need resolving. In Apollo's cache, even the list edges can be references.
Check what the site actually searched. A search that falls back to your IP location looks exactly like success.
Test every optional input both ways. My zero-results bug only appeared without a keyword.

X's API bills every post it returns, so I moved my filters into the query 💸

Timothy Kelvin — Thu, 29 Oct 2026 12:46:29 +0000

Six posts read, two kept, six paid for. One operator fixed that.

Hi! I'm Tim, a self-taught developer in Nairobi. I build small data tools that run as Apify Actors, serverless jobs that people pay for per result.

This week I built one that finds posts taking off on X for a topic, a list of accounts, or whatever is trending. The interesting part wasn't the code. It was the bill.

🚪 Why the official API

The first thing I checked was X's robots.txt. After the rules for named search engines and link previewers, it ends with:

User-agent: *
Disallow: /

Every other bot is told to stay out, and posts, search and like counts sit behind a login anyway. So the tool uses the official X API v2, with each user's own Bearer Token. Nobody hands over a password or cookies, and nothing needs a proxy.

💸 What a read costs

X's self-serve API is pay-per-use. You buy credits, and each call deducts what it returns. From X's pricing page in October 2026:

What X returns	Price
A post	$0.005
A user profile	$0.010
A trends request	$0.010

Two details matter. A post is billed once per UTC day, so reading it again the same day is free (X calls this a "soft guarantee"). And pay-per-use is capped at 3 million post reads a month.

At $0.005 a post, the usual pattern of "fetch a lot, filter in code" stops being free.

🎯 Put the threshold in the query

My first version was going to fetch recent posts for a topic and keep the ones with enough likes. Every post it threw away would still be on the bill.

Then I found three operators in X's recent-search docs: min_likes:, min_reposts: and min_replies:. (According to the same docs, min_faves: and min_retweets: from X's web search are rejected.)

My first live call:

"AI agents" min_likes:300 -is:retweet -is:reply -is:nullcast lang:en

X returned 10 posts. The least-liked one had 320 likes. Nothing below the line came back, so nothing below the line was billed.

Every query the tool builds now ends with the user's filters, threshold included:

export function filterOperators({ contentType = 'all', minLikes = 0, minReposts = 0, minReplies = 0, language = null, includeReplies = false, verifiedOnly = false }) {
    const ops = ['-is:retweet', '-is:nullcast'];   // no reposts, no ads
    if (!includeReplies) ops.push('-is:reply');
    if (verifiedOnly) ops.push('is:verified');
    if (language) ops.push(`lang:${language}`);
    ops.push(...(CONTENT_OPERATORS[contentType] ?? []));   // e.g. has:video_link for native videos
    if (minLikes > 0) ops.push(`min_likes:${minLikes}`);
    if (minReposts > 0) ops.push(`min_reposts:${minReposts}`);
    if (minReplies > 0) ops.push(`min_replies:${minReplies}`);
    return ops.join(' ');
}

Leaving reposts out matters for the same reason. A repost comes back as a post of its own, so without -is:retweet you can pay for copies sitting next to the original.

👤 The author can cost more than the post

To get a post's author (handle, followers, verified badge), you add expansions=author_id, and the users come back in includes.users. X's price list puts a user profile at twice the price of a post.

I haven't yet confirmed in the console whether every expanded author is billed as a user read. Either way, I made author details a setting that's on by default and easy to switch off. Without it, a link becomes https://x.com/i/status/<id>, which still works.

📏 512 characters per query

The docs give self-serve recent search a 512-character query limit. That's plenty for a topic, but not for 60 accounts written as from: clauses, and each extra query is an extra request.

So accounts are packed into as few queries as fit, with the filters counted in:

let group = [];
for (const h of handles) {
    const next = [...group, h];
    if (`(${next.map((x) => `from:${x}`).join(' OR ')}) ${ops}`.length > MAX_QUERY_LENGTH) flush();
    group.push(h);
}
flush();

60 handles became 3 queries, of 512, 503 and 256 characters, with every handle in exactly one of them. One query per handle would have been 60 requests.

🔗 Three kinds of link in one array

entities.urls mixes three things:

Attached media. Photos and videos show up as x.com/.../photo/1 or /video/1 links, and these carry a media_key.
Quoted posts. A quote's link points back to twitter.com/<user>/status/<id>.
Real outside links. Often with unwound_url, title and description already filled in by X.

For an "articles only" mode I query with has:links, then keep posts that link outside X (or to an X Article at x.com/i/article/...). In a test, X returned 10 posts for "AI agents" has:links min_likes:100, and 7 linked outside X. The other 3 were billed and dropped, which is the one place where filtering after reading is still unavoidable. The settings text says so.

📝 `text` is not the whole post

For posts over 280 characters, text is a shortened version. Ask for the note_tweet field and the full post is in note_tweet.text. On two long posts from my first call, text was 303 and 302 characters, and note_tweet.text was 464 and 311.

📈 Which number says "viral"?

Here are three posts from the same morning (2026-10-29), with the counts at the time I read them:

Post	Likes	Views	Engagement rate	Likes per hour	Views per follower
SpaceX's Starship static-fire video	17,631	1,854,564	1.17%	875	0.04
A 24-second meme video about AI agents, from an account with 80K followers	10,823	505,501	2.81%	565	6.30
A long post about AI agents, from an account with 152K followers	619	8,260	10.51%	37	0.05

Sort by likes and SpaceX wins, which is what you'd expect from 42 million followers. Sort by views per follower and the meme video wins: it got six views for every follower its account has. Sort by engagement rate and the long post wins, because the few people who saw it acted on it.

None of those is "the" viral score, so I don't compute one. Each row carries all of them, and the user picks the sort.

🌍 Trends come back as names only

GET /2/trends/by/woeid/{woeid} lists what's trending in a place. I checked all 16 places I offer, and every one answered. But for worldwide, every trend came back without tweet_count, so you can't rank trends by volume. And two of the top five worldwide trends were Japanese hashtags, which is why trend searches respect the language filter too.

🧾 No key, no crash

Apify runs every public Actor once a day with its example input, and flags it after repeated failures. A secret token can't be part of that example, so a run without one has to succeed anyway. It saves one row explaining how to get a token, charges nothing and exits cleanly. A bad token, an account with no credits, or an app without read access each get their own message (401, 402 and 403), and any posts already found are still saved.

🎯 What I learned

When reads cost money, the filter belongs in the request. With min_likes: in the query, every post I pay for is one I keep.
Read the price list before the endpoint docs. It changed what I fetched, and in what order.
Expansions aren't free extras. Make the expensive ones optional.
"Viral" isn't one number. Big accounts win on raw counts, and small ones win on reach and rate. Give people the columns and let them sort.
Check robots.txt first. It settled the whole design in two lines.

Testing all of this cost me under a dollar of X credits.

AI agents were told my inputs were optional, then rejected for leaving them out 🤖

Timothy Kelvin — Sat, 24 Oct 2026 21:11:17 +0000

One word in 29 config files: required. Every field it guarded already had a default.

Hi! I'm Tim, a self-taught developer in Nairobi. I build small data tools that run as Apify Actors, serverless jobs that people pay for per result. 100 of them are public.

This week Apify's DEV account published What AI agents see when they call your Apify Actors, by SIÁN Agency. One finding in it sent me straight to my own input schemas.

🤖 What an agent sees

A person runs an Actor from a form. The fields have labels, the dropdowns list the valid values, and an example is already filled in.

An AI agent connected through the Apify MCP server gets none of that. It gets the input schema as JSON Schema, and it sends what the schema says.

The article found that two parts of the MCP server read that schema differently:

tools/list tells the model what a tool accepts. It leaves a field out of required when that field has a default.
tools/call checks the model's arguments. It validates against the original schema and doesn't fill in defaults first.

So a field that is both required and has a default is listed as optional. When the model leaves it out, the call is rejected with a validation error (-32602) for the very field it was told it could skip.

An agent can recover by fetching the Actor's details and trying again. It still burns a round trip on the first call, every time. The author found this in 59 of the 253 schemas they checked.

🔍 How many of mine?

I wrote a short script that reads every .actor/input_schema.json in my portfolio. For each required field, it records whether the field has a default and whether that default equals the prefill, the example shown in the form.

	Count
Actors with at least one required field	29 of 100
Required fields that also had a default	32, in 26 Actors
Required fields with no default	3, in 3 Actors
Defaults identical to the prefill	32 of 32

Here's a typical one, from a vehicle safety ratings lookup (trimmed to the parts that matter):

{
    "required": ["year", "make", "model"],
    "properties": {
        "year":  { "type": "integer", "prefill": 2022,     "default": 2022 },
        "make":  { "type": "string",  "prefill": "Honda",  "default": "Honda" },
        "model": { "type": "string",  "prefill": "Accord", "default": "Accord" }
    }
}

All three fields are required, and all three have a default. In the form, nothing looks wrong, because the form is already filled in. To an agent, all three are listed as optional, and a call without them fails.

🙈 Why nothing ever caught it

Apify runs every public Actor once a day with its prefilled input, and flags it if that run fails three days in a row.

All 32 defaults were identical to the prefill. So the daily test always sent every required field, and it always passed. None of these Actors was ever flagged for it.

The test walks the path a person takes. An agent sending {} takes a different one, and nothing was walking it.

🚫 Why I didn't run the probe

The article comes with a probe that calls each Actor through the MCP server with {}. I didn't run it on mine. My own runs show up in my Actors' analytics as a user, and that analytics page is the only signal I have about real customers, so I debug locally and keep platform runs to a minimum.

That left the static half: what the schemas say. The article is clear that this half can't find everything. An Actor can report SUCCEEDED with nothing useful in it, and only running it shows that. So after the schemas, I read through the code paths of my best-earning Actor by hand, looking for that kind of silent result. More on that in fix 3.

🔧 Fix 1: a field with a default doesn't need `required`

Apify writes schema defaults into every run's input. A required field with a default is filled in whether the caller sends it or not, so the required flag adds nothing to the run. All it does is mislead the agent.

The rule I applied to the 26 Actors:

// Drop `required` only when every required field already has a default.
const noDefault = schema.required.filter((k) => schema.properties[k].default === undefined);
if (noDefault.length) throw new Error(`no default for ${noDefault.join(', ')}`);
schema.required = [];

Nothing else in any schema changed, and the script checked that too. A person in the form sees exactly the same thing as before.

🔧 Fix 2: three fields had no default at all

A court opinion search (query), a travel per diem lookup (locations) and a package health check (packages) each had a required field with no default.

These were honest: with no default, nothing gets dropped from the tool listing, so an agent is told to send the field. But their code also threw on empty input:

if (!query) {
    throw new Error('query is required.');
}

I wanted every one of my Actors to answer {} the same way: with the example from the form, not an error. Each of the three got a default equal to its prefill, and the code now falls back to that example:

// An empty call (an AI agent calling with {}, or an empty API call) searches the input form's example instead of failing.
const query = String(queryInput ?? '').trim() || 'antitrust';
if (!String(queryInput ?? '').trim()) log.info('No query given, so searching the example "antitrust".');

In the two that take a list, a value that isn't a list still fails with a clear message. Only "nothing at all" falls back to the example.

🔇 Fix 3: zero rows is not an answer

My best-earning Actor audits small business websites and scores each business as a lead for web designers. It has a filter: only keep businesses that score above a minimum.

Set that minimum high enough and every business gets filtered out. The run used to end SUCCEEDED, with zero rows. An agent reading that would tell its user "no leads found", as a fact about the world. The real answer was "your filter was stricter than every business it checked".

And an empty call ({}) used to fail outright with "Nothing to audit", even though the schema told an agent nothing was required. Now it audits the three example businesses from the form.

For the empty case, a run that saves nothing now saves one row explaining why, and doesn't charge for it:

// Never end with an empty dataset: say why nothing was saved, in one uncharged row.
if (stats.pushed === 0) {
    const note = emptyRunNote(stats, counters, options, search, categoryTitle);
    await Actor.pushData(note);
}

Here's the row it writes when the filter removes all 5 businesses it audited:

{
    "name": "No businesses saved in this run (not charged)",
    "websiteStatus": null,
    "opportunityScore": null,
    "note": "All 5 businesses audited scored below 100, so none were saved (filtered businesses aren't charged). Lower minScore or turn off onlyOpportunities to see them.",
    "processed": 5,
    "filteredOut": 5
}

The row says what happened and what to change. A person can read that. So can a model.

🚀 Shipping 29 Actors, one at a time

29 Actors meant 29 builds. I didn't want a loop that pushes everything and checks nothing, so each Actor went through the same steps, and the script stopped at the first problem (trimmed):

before=$(node live-build.js "$a")            # the build to roll back to
git commit -q -m "$msg" && git push -q origin HEAD
out=$(timeout 480 apify push 2>&1)
echo "$out" | grep -q "Build: SUCCEEDED" || { echo "STOP $a"; exit 1; }
verify=$(node live-build.js "$a" --verify)   # live schema: required is [] and matches local
echo "OK $a | before $before | after $verify"

The --verify step reads the input schema back from the new live build, not from my disk. All 29 passed, and the previous build of every Actor is saved in case one needs rolling back.

🎯 What I learned

My input schemas have a second reader now. I wrote them for a form. A model reads them as a contract, and a contradiction in the contract becomes a failed call.
required plus default is a contradiction. If the platform fills a field in anyway, don't tell callers they must send it.
A daily test only checks the path it takes. Passing it says nothing about any other input, including an empty one.
Zero rows needs a reason. If a filter removes everything, say so in a row, not in a log nobody opens.
There's more to check. 33 fields across 16 of my Actors still have a prefill but no default. A person sees the example. An agent sending {} gets nothing for that field. Some of those Actors already fall back to an example in code; I haven't checked the rest yet.

If you publish Actors, this one-liner lists the fields in a schema that are both required and defaulted:

node -e "const s=require('./.actor/input_schema.json');console.log((s.required||[]).filter(k=>s.properties[k]?.default!==undefined))"

An empty list means an agent gets an honest contract. Anything else is a field it was told it could skip, and then got rejected for skipping.

An RSS feed grew to 500 items and my crawler got flagged as broken 📰

Timothy Kelvin — Thu, 22 Oct 2026 14:04:14 +0000

I didn't change a line of code. The feed changed, and Google News had been saying no the whole time.

Hi! I'm Tim, a self-taught developer in Nairobi. I build small data tools that run as Apify Actors, serverless jobs that people pay for per result.

Apify runs every public Actor once a day with its default input. The run has to finish within 5 minutes and return at least one result. If it fails three days in a row, the Actor gets an "under maintenance" label on its Store page.

This week one of mine got that label: a news crawler that reads RSS feeds, fetches each new article, pulls out the clean text, and remembers what it has already returned so a scheduled run only gives you what's new.

Over the last 30 days, 24 of its runs passed and 3 timed out.

Let's find out why! 🔍

⏱️ Reproducing the daily test

The daily test uses the default input, which for this crawler is an empty form. With no sources given, it falls back to four built-in ones: a Google News search plus the TechCrunch, Electrek and Engadget feeds.

So I ran exactly that on my machine and timed it:

Source	Items found	Articles saved
Google News search	100	0
TechCrunch (keyword filter)	feed didn't load from my connection	0
Electrek	500	500
Engadget (keyword filter)	1	1

580 seconds. Nearly twice the limit, and the two big problems are right there in the table.

📜 Problem 1: the feed was 500 items long

Electrek runs on WordPress, where a feed shows the latest 10 items by default. This one lists 500, and my code never asked how long a feed was. It fetched every item:

for (const source of sources) {
    let items;
    try {
        items = await discover(source);
    } catch (err) {
        log.warning(`Discovery failed for ${source.name}`, { error: (err as Error).message });
        continue;
    }
    for (const item of items) {
        const article = await fetchAndParse(item.url);
        // ...dedupe, save, charge
    }
}

The crawler is polite: it waits at least a second between requests to the same site. That's the right thing to do, but it means 500 articles from one site take at least 500 seconds before anything else happens.

Dedupe didn't help either. It runs on the parsed article (canonical URL, title, publish time and a hash of the body), so even an article the crawler had already seen got fetched again before it could be skipped.

🚪 Problem 2: Google News says no

The Google News source produced 100 links and 0 articles. Here's the start of news.google.com/robots.txt:

User-agent: *
Disallow: /
Allow: /$
Allow: /?
Allow: /home$
Allow: /home?
Allow: /home/
Allow: /nwshp$
Allow: /topics/
Allow: /publications/
Allow: /stories/
...

Everything is disallowed except the homepage and a few HTML sections. The RSS search feed lives under /rss/search, and every article link in it is a news.google.com/rss/articles/... redirect. None of those paths are allowed.

My crawler checked robots.txt before fetching each article, and correctly refused all 100. But it never checked before fetching the feed itself. So it read a feed it wasn't allowed to read, then politely declined every link in it.

By those rules, the Google News source can't return a single article. It only ever cost time.

(Further down, the same file has a block that disallows everything for GPTBot, ClaudeBot, PerplexityBot and a few other AI crawlers by name.)

🐢 Problem 3, found while fixing: timeouts got three retries

While testing the fix, TechCrunch article pages kept timing out from my connection. The fetch helper retried transient failures three times with backoff, and it treated a timeout like any other transient error.

With a 15-second timeout, that's 4 attempts plus backoff: about a minute for one page. A site that's having a bad day could eat the whole 5 minutes on its own.

🔧 The fixes

1. Check only the newest N items per feed

A new maxArticlesPerSource input, default 20. Items are sorted newest first, then cut:

const maxPerSource = Number.isInteger(input.maxArticlesPerSource) && input.maxArticlesPerSource! > 0
    ? input.maxArticlesPerSource!
    : DEFAULT_MAX_ARTICLES_PER_SOURCE; // 20

const newestFirst = (a: { publishedAt: string | null }, b: { publishedAt: string | null }) =>
    (Date.parse(b.publishedAt ?? '') || 0) - (Date.parse(a.publishedAt ?? '') || 0);

if (items.length > maxPerSource) {
    items = [...items].sort(newestFirst).slice(0, maxPerSource);
}

For a monitor that runs on a schedule, the newest 20 is what you want anyway. Anything older was either returned by an earlier run or is old news.

2. Check robots.txt at the door too

Feeds now go through the same robots.txt check as articles, and Google News sources are skipped with a reason instead of failing quietly:

export async function discover(source: Source): Promise<DiscoveredItem[]> {
    if (source.type === 'google-news') {
        throw new Error("Skipped: Google News doesn't allow automated access to its RSS feeds (robots.txt). Add the publishers' own RSS feeds instead.");
    }
    const feedUrl = source.feedUrl;
    if (!(await isAllowedByRobots(feedUrl))) {
        throw new Error(`Skipped: the site's robots.txt doesn't allow automated access to ${feedUrl}`);
    }
    const feed = await fetchFeed(feedUrl);
    // ...
}

Google News also left the default sources. Most publishers have their own feed, often at /feed or /rss, and those are the ones to use.

3. One retry for a timeout, and a per-site limit

A page that timed out once gets one more try, not three:

} catch (err) {
    const timedOut = (err as Error).name === 'TimeoutError' || /timeout/i.test((err as Error).message);
    if (attempt === MAX_RETRIES || (timedOut && attempt >= 1)) throw err;
    await sleep(2 ** attempt * 500);
}

And after 2 failed articles from the same site, the rest of that site is skipped for this run:

for (const item of items) {
    const site = siteOf(item.url);
    if ((failuresBySite.get(site) ?? 0) >= MAX_FAILURES_PER_SITE) {
        skippedDisallowedOrFailed++;
        continue;
    }
    // fetch, and on failure: failuresBySite.set(site, failures + 1)
}

The articles aren't lost. They're still in the feed tomorrow, when the site may be answering again.

4. "Nothing new" is a result

There was one more trap waiting. The daily test needs at least one result, but this crawler's whole point is to return only new articles. A second run on the same day, or a quiet news day, could return nothing at all and look broken.

So a run with nothing new now saves one row saying so, and doesn't charge for it:

if (stored === 0) {
    await Actor.pushData({
        title: 'No new articles in this run (not charged)',
        note: skippedDuplicate > 0
            ? `All ${skippedDuplicate} articles found were already returned by an earlier run.`
            : 'The sources returned no articles that could be fetched and read.',
        discovered,
        skippedDuplicate,
        skippedDisallowedOrFailed,
        skippedSources,
        discoveredAt: new Date().toISOString(),
    });
}

If you're scheduling a monitor, an empty result and a broken run look the same from the outside. A row that says why tells them apart.

🧪 After

Run	Where	Time	Result
Empty input, first run	my machine	82 s	21 articles
Empty input, same storage again	my machine	122 s	1 row: "All 21 articles found were already returned by an earlier run."
A Google News + TechCrunch input	my machine	38 s	1 row, with the Google News skip reason in it
Empty input	Apify	29 s	25 articles (Electrek 20, TechCrunch 3, Engadget 2)

The repeat run was slower because TechCrunch was timing out from my connection. The per-site limit kicked in after two failures, and it still finished well inside the limit. On Apify's servers, TechCrunch answered fine.

🎯 What I learned

Inputs you don't control will grow. A feed, an API page, a sitemap. If the loop has no cap, the run time doesn't either.
Check robots.txt at the entry point, not just the pages. My crawler was careful about every article and careless about the feed that listed them.
A timeout is not a hiccup. Retrying a 15-second timeout three times turns one slow page into a lost minute.
"Nothing new" deserves a row. For anything that runs on a schedule, say why the result is empty.
A daily health check is a free alarm. The code hadn't changed since August. The world around it did, and the check is what told me.

The crawler is back to finishing in under a minute, and the under-maintenance label should clear after the next daily test.

One bad row failed a paying customer's whole run 🧯

Timothy Kelvin — Mon, 12 Oct 2026 02:28:53 +0000

The bug wasn't in the API call. It was in what I assumed people would type.

Hi! I'm Tim, a self-taught developer in Nairobi. I build small data tools that run as Apify Actors, which are serverless jobs that people pay for per result.

One of them takes a list of occupation codes and returns official US wage data from the Bureau of Labor Statistics (BLS) API. You give it 15-1252 and it gives you back what software developers earn, nationally or in one state.

This weekend it got its second paying user. They ran it three times. Two runs worked. One failed.

The failed run wasn't caused by BLS being down or by a timeout. My own code threw an error on one row, and that threw away every other lookup in the same run.

Let's look at how that happened! 👀

🤔 What I could see

Not much. On Apify, a developer can't open another user's runs unless that user chooses to share them, and this one hadn't.

So I had a failure rate (1 run of 3) and no input to reproduce it with.

Instead of guessing, I asked a different question:

What would a real person actually type into this form?

📋 The old code trusted the input

The tool has a simple input: one SOC code field for a quick lookup, plus an occupations JSON field for checking several at once. I added the single field a few weeks ago, after noticing that a raw JSON textarea scares people off. This weekend the JSON field bit back.

Here's what the old code did with each entry:

function buildSeriesId({ areaType, areaCode, socCode, dataTypeCode }) {
    const occupationCode = socCode.replace(/-/g, '');
    if (!/^\d{6}$/.test(occupationCode)) {
        throw new Error(`Invalid SOC code "${socCode}"`);
    }
    return `OEU${areaType}${areaCode}000000${occupationCode}${dataTypeCode}`;
}

And the run loop:

const results = await fetchOccupationWages({ occupations, apiKey });

for (const record of results) {
    await Actor.pushData(record);
    await Actor.charge({ eventName: 'occupation-checked' });
}

Two things are wrong here, and neither looks wrong on its own:

buildSeriesId throws on anything that isn't exactly a 6-digit code.
Nothing is saved until every lookup has finished.

Put together, one bad entry anywhere in the list means zero rows saved.

💥 Five inputs a real person might type

I wrote a quick script that called the lookup function directly with inputs I could imagine someone pasting in:

Input	Old result
`["15-1252", "29-1141"]` (a plain list of codes)	`Cannot read properties of undefined (reading 'replace')`
`{ "socCode": "15-1252.00" }`	`Invalid SOC code "15-1252.00"`
`{ "socCode": "15-1252", "stateCode": "California" }`	`Unknown state code "California"`
`{ "socCode": "software developer" }`	`Invalid SOC code "software developer"`
two good codes, then `15-1252.00`	Whole run fails, 0 rows saved

All five crashed.

The second one hurts the most. 15-1252.00 is how O*NET writes the same code, and O*NET is one of the first places you land when you search for an occupation code. To a person, it's obviously the same occupation. To my regex, it was garbage.

The last row is the real lesson. The two good lookups had already been sent to BLS, so the tool did the work and then threw the results away.

So I wrote down the rule I should have started with:

In a batch tool, the unit of failure is the row, not the run.

🔧 Fix 1: understand more, and explain the rest

Each entry now goes through normalizeEntry before anything else. It accepts more shapes, and when it can't make sense of an entry, it returns an explanation instead of throwing:

export function normalizeEntry(entry) {
    const raw = typeof entry === 'string' || typeof entry === 'number'
        ? { socCode: String(entry) }
        : (entry ?? {});
    const rawSoc = raw.socCode ?? raw.soc ?? raw.code ?? raw.occupationCode;
    // ...
    const digits = String(rawSoc).trim().replace(/\.\d{2}$/, '').replace(/[\s-]/g, '');
    if (!/^\d{6}$/.test(digits)) {
        return { socCode: rawSoc, error: `"${rawSoc}" isn't a SOC code. Use the 6-digit code, e.g. "15-1252" for software developers.` };
    }
    // ...states: "TX", "tx", "Texas" and "texas" all work; "US" or empty means national
}

A plain string is treated as a code, the .00 suffix is dropped, and full state names map to their two-letter codes.

Anything that still doesn't make sense becomes a row with found: false and an error saying what to fix. The run carries on with the rest of the list.

💸 Fix 2: don't charge for the error rows

This is a pay-per-result tool, so "return an error row" raised a second question: should that row cost money?

No. Every lookup now carries a billable flag, and only the lookups BLS actually answered get charged:

for (const { billable, ...record } of records) {
    await Actor.pushData({ ...record, checkedAt: new Date().toISOString() });
    if (billable) await Actor.charge({ eventName: 'occupation-checked' });
}

While I was in there, I found an older billing bug. When BLS refuses a request, for example once its daily limit is used up, the reply comes back with a status like REQUEST_NOT_PROCESSED instead of data. The old code turned that into a found: false row and charged for it. Now those rows say what happened and add "You weren't charged for this lookup."

📦 Fix 3: batch the requests

That BLS limit deserved a closer look. Without a registration key, BLS allows 25 queries a day with up to 25 series per query.

Each occupation lookup needs 5 series: employment, mean and median annual wage, and mean and median hourly wage. The old code sent one request per lookup, so 5 series per request and 25 lookups a day before BLS started refusing.

Now valid lookups are grouped, 5 per request (10 with a key, where the limit is 50 series per query):

const perRequest = Math.floor((apiKey ? 50 : 25) / 5);

for (let start = 0; start < valid.length; start += perRequest) {
    const batch = valid.slice(start, start + perRequest);
    // one POST with every series for every lookup in the batch,
    // then map each returned series back to its lookup by series ID
}

That's 5 times more lookups out of the same daily allowance.

🧪 Testing it with a deliberately messy list

I ran the tool locally with the messiest input I could think of, seven entries in mixed shapes:

{
  "occupations": [
    "15-1252",
    { "socCode": "29-1141.00", "stateCode": "California" },
    { "socCode": "software developer" },
    { "socCode": "15-1252", "stateCode": "Narnia" },
    { "soc": "11-1021", "state": "tx" },
    "13-2011",
    "15-1211"
  ]
}

Result:

Entry	Found	Annual median wage	Error	Charged
15-1252 (national)	✅	135,980		yes
29-1141, CA	✅	140,270		yes
software developer	❌		isn't a SOC code	no
15-1252, Narnia	❌		unknown state	no
11-1021, TX	✅	103,230		yes
13-2011 (national)	✅	83,680		yes
15-1211 (national)	✅	105,850		yes

7 rows out, 5 charged, and all 5 valid lookups went to BLS in a single request.

I also tested an empty form. Apify runs every public Actor once a day with its default input, and flags it as under maintenance if that keeps failing. An empty form now looks up an example code instead of throwing.

🙀 One false alarm

My first local test runs exited with code 127, which made me think I'd broken the shutdown.

The log said:

Assertion failed: !(handle->flags & UV_HANDLE_CLOSING), file src\win\async.c, line 94

Before touching anything, I stashed my changes and ran the old code. It printed the same assertion.

It's Node 24 on Windows asserting while the process shuts down, after all the work is done. The cloud runs these on Linux with Node 20, so it never shows up there. If you only check exit codes on a Windows machine, it's worth knowing about.

🎯 What I learned

Fail the row, not the run. In anything that takes a list, one bad item should produce one error row.
Save as you go, or at least don't throw away finished work. The old code had done the good lookups and still saved nothing.
Accept what people actually type. Codes copied from other sites, full state names and plain lists aren't edge cases. They're the input.
An error row needs a reason and a price. Say what to fix, and don't charge for it.
Read the upstream API's limits before they read you. Batching gave 5 times the headroom for free.

I still don't know exactly what that customer typed. But every input I could think of now gets either an answer or a clear reason why not.

When crt.sh's website is down, query its public database instead

Timothy Kelvin — Thu, 08 Oct 2026 18:20:42 +0000

I maintain a small tool that searches Certificate Transparency logs for new certificates on a domain. It reads crt.sh's JSON output, and it already retries the 404s and 502s that crt.sh returns when it's busy.

This week retrying stopped being enough. For more than an hour, every request to the website failed: a 502 in under a second, or no answer at all until my 40-second timeout. The platform I run the tool on tests it once a day with its default input, and it had failed 5 of its last 27 daily tests, which was enough to get it flagged as under maintenance.

crt.sh has a second door, though. Its database is open to the public, read-only:

psql -h crt.sh -p 5432 -U guest certwatch

While the website was returning 502s, the database answered in 9 to 35 seconds. That's slower than the website on a good day, but it's an answer. So the tool now tries the website three times and then falls back to the database. Getting that fallback right took four fixes.

1. The pooler rejects some connection options

My first connection set a server-side statement timeout:

new pg.Client({ host: 'crt.sh', port: 5432, user: 'guest', database: 'certwatch', statement_timeout: 60000 });

It failed straight away with unsupported startup parameter: statement_timeout. The database sits behind a connection pooler that only accepts a few startup parameters. node-postgres has a client-side query_timeout that does the same job without asking the server:

const client = new pg.Client({
    host: 'crt.sh', port: 5432, user: 'guest', database: 'certwatch',
    connectionTimeoutMillis: 20000,
    query_timeout: 60000,
});

2. The query that works, and the one that gets cancelled

crt.sh searches by identity. Its schema has a certificate_and_identities view with a full-text index, so this finds every unexpired certificate for a domain and its subdomains:

SELECT c.id,
       x509_commonName(c.certificate) AS common_name,
       ca.name AS issuer_name,
       encode(x509_serialNumber(c.certificate), 'hex') AS serial_number,
       to_char(x509_notBefore(c.certificate), 'YYYY-MM-DD"T"HH24:MI:SS') AS not_before,
       to_char(x509_notAfter(c.certificate), 'YYYY-MM-DD"T"HH24:MI:SS') AS not_after
FROM certificate c
JOIN ca ON ca.id = c.issuer_ca_id
WHERE c.id IN (
    SELECT cai.certificate_id
    FROM certificate_and_identities cai
    WHERE plainto_tsquery('certwatch', $1) @@ identities(cai.certificate)
      AND (cai.name_value ILIKE '%.' || $1 OR lower(cai.name_value) = lower($1))
    LIMIT 5000)
  AND x509_notAfter(c.certificate) > now()
  AND x509_notBefore(c.certificate) >= $2::timestamp
ORDER BY x509_notBefore(c.certificate) DESC
LIMIT $3

For a mid-sized domain this took about 35 seconds, and for a small one about 9.

I tried to make the 5,000-match cap keep the newest certificates by adding ORDER BY cai.certificate_id DESC inside the subquery. That version never finished. crt.sh cancelled it with canceling statement due to statement timeout after about 54 seconds: the server has its own limit, and sorting every match goes over it. So the subquery stays unordered, which means a very large domain can miss some recent certificates on this path. For a fallback, that's a fair trade.

3. A dropped connection kills the whole process

After a cancelled query, my test script crashed with an error I never caught:

error: client_idle_timeout
Emitted 'error' event on Client instance

The pooler closes idle and cancelled connections, and node-postgres reports that as an error event on the client. In Node, an error event with no listener is an uncaught exception, so the process dies, retry logic and all. One line fixes it:

client.on('error', (err) => console.debug('crt.sh connection closed:', err.message));

4. The timestamps are UTC, but they don't say so

The first fallback run on my laptop said a certificate became valid at 07:08:23. The same code in the cloud said 10:08:23. To find out which was right, I decoded the certificate itself:

import { X509Certificate } from 'node:crypto';

const { rows } = await client.query('SELECT certificate FROM certificate WHERE id = $1', [id]);
console.log(new X509Certificate(rows[0].certificate).validFrom); // Oct  3 10:08:23 2026 GMT

x509_notBefore() returns a timestamp without time zone that holds UTC. node-postgres turns that type into a JavaScript Date in the machine's local time zone. The cloud runs on UTC, so it was right by luck. My laptop is on UTC+3, so it was three hours off. Formatting the value in SQL with to_char, as in the query above, avoids the driver's guess entirely, and when I compare dates in JavaScript I add the Z myself:

const utc = (crtshTime) => new Date(`${crtshTime}Z`);

Fitting it into a time budget

The daily test expects a finished run with results within five minutes, and a monitor running on a schedule shouldn't hang either. So everything shares one 4-minute budget:

The website gets three attempts with a 20-second timeout each, capped at 75 seconds in total.
The database gets whatever is left, with one retry if at least 30 seconds remain.
If both fail, the run fails with a message that names both errors, instead of quietly reporting zero certificates.

One more cause: the example domain

The default input searched example.com for certificates from the last 30 days. Looking at 180 days of history in the database showed that example.com once went 59 days without a new certificate. So on some days the default run correctly found nothing, and an empty result counts as a failed test too. I switched the default to a domain with lots of subdomains on 90-day certificates, whose longest gap was 19.5 days.

The result

With the website still returning 502s, the fallback found four recent certificates in 32 seconds on my laptop and 40 seconds in the cloud.

What I'd pass on:

If a free service has a second way in (here, a public read-only database), wire it up as a fallback before you need it.
Behind a connection pooler, set timeouts on the client, not the server.
Always attach an error listener to database clients in Node.
Don't let a driver guess the time zone of a timestamp without time zone. Format it yourself.
Pick test inputs that can't legitimately come back empty.

Dead website or false alarm? Five cases from checking small-business sites

Timothy Kelvin — Thu, 01 Oct 2026 06:10:30 +0000

I've been writing a checker that looks at the website a local business lists (salons, barbers, plumbers) and decides whether it works. The first version was simple: fetch the URL, and if that fails, call the site broken.

Then I ran it on about 50 salon, spa and barber websites in Nairobi and read every verdict by hand. A few "broken" sites worked fine in a browser, one site came out dead in one run and healthy in the next, and a parked domain looked healthy. Below are the cases and what the checker does now. Everything here is Node 20 with the built-in fetch.

The helper used in the snippets:

async function fetchPage(url, init = {}) {
  try {
    const res = await fetch(url, { redirect: 'follow', signal: AbortSignal.timeout(15_000), ...init });
    return { ok: res.ok, status: res.status, finalUrl: res.url, body: await res.text() };
  } catch (err) {
    return { ok: false, status: 0, code: err.cause?.code ?? err.name };
  }
}

1. HTTP says 404, HTTPS works

I request http:// first, because whether plain HTTP redirects to HTTPS is worth knowing. One barbershop's server answered plain HTTP with a 404 and no redirect, while https:// served the real site. Chrome has defaulted to HTTPS for typed addresses since version 90, and search results link to HTTPS, so almost nobody sees that 404. Calling the site broken was wrong.

Now an error over HTTP only counts if HTTPS fails too. The missing redirect is still recorded, as its own smaller finding:

const http = await fetchPage(`http://${host}${path}`);
const redirected = Boolean(http.finalUrl?.startsWith('https:'));
let page = http;
if (!http.ok && httpsWorks && !redirected) {
  // Plain HTTP answers with an error page, but HTTPS visitors get the site.
  page = await fetchPage(`https://${host}${path}`);
}
const httpRedirectsToHttps = http.status > 0 ? redirected : null;

httpsWorks comes from an earlier request (I fetch https://host/robots.txt first, which doubles as the HTTPS check). The !redirected guard matters: if HTTP already redirected to HTTPS and that page is a 404, fetching it again tells you nothing new.

2. The listed link is dead, the site isn't

Business listings often point at a deep page rather than the homepage. One salon's listing linked to /beauty-salon, which returns 404, while the homepage works. "Your website is broken" is the wrong verdict. "Your listing links to a missing page" is the right one, and it's a smaller fix for the owner.

Falling back to the homepage is only safe when the domain belongs to the business, though. Another listing pointed at a page on a booking platform. That platform's homepage works, and it says nothing about the salon. So the fallback only runs when the domain carries a distinctive word from the business name:

import { getDomain } from 'tldts';

const GENERIC = new Set(['salon', 'hair', 'beauty', 'barber', 'barbers', 'studio', 'spa', 'nails', 'house', 'shop']);

function domainCarriesName(name, url) {
  const label = (getDomain(new URL(url).hostname) ?? '').split('.')[0];
  const words = name.toLowerCase().split(/[^a-z0-9]+/).filter((w) => w.length >= 4 && !GENERIC.has(w));
  return label.length >= 4 && words.some((w) => label.includes(w));
}

domainCarriesName('House of Treasures Beauty Salon', 'https://houseoftreasureskenya.com/beauty-salon'); // true
domainCarriesName('Posh Palace Hair Studio', 'https://booking-platform.example/p/12932'); // false

The generic-word list is what keeps "Glam Hair Salon" from matching a domain like salonbookings.com.

3. "Page Not Found" with a 200

That booking-platform link from case 2 returned status 200, with <title>Page Not Found</title> and almost no text. The status code says fine, the page says gone. On short pages I now check the title:

const NOT_FOUND_TITLE = /(^|[^a-z0-9])(404|page not found|not found|page (could not|cannot|can't) be found)([^a-z0-9]|$)/i;

function isSoft404($, visibleText) {
  return visibleText.length < 3000 && NOT_FOUND_TITLE.test($('title').first().text().trim());
}

The length guard keeps a real page from being flagged because its title happens to contain "not found". The check also runs after the parked-domain and default-server-page checks, so those keep their more specific labels.

4. A certificate chain missing its intermediate

One server sent only its own certificate, without the intermediate that links it to a trusted root. openssl s_client shows it plainly: one certificate in the chain and Verify return code: 21 (unable to verify the first certificate).

Browsers usually cope. Chrome and Edge fetch the missing intermediate from the URL inside the certificate, and Firefox ships a list of known intermediates. Node's fetch on a Linux server doesn't, and fails with UNABLE_TO_VERIFY_LEAF_SIGNATURE. (The same request succeeded on my Windows laptop with Node 24, so test where you deploy.)

For that one error code, the checker reads the page anyway and records the certificate problem as a finding instead of "down":

import { Agent, fetch as undiciFetch } from 'undici';

const lenient = new Agent({ connect: { rejectUnauthorized: false } });

let page = await fetchPage(url);
if (page.code === 'UNABLE_TO_VERIFY_LEAF_SIGNATURE') {
  const res = await undiciFetch(url, { dispatcher: lenient, signal: AbortSignal.timeout(15_000) });
  page = { ok: res.ok, status: res.status, finalUrl: res.url, body: await res.text(), certIssue: 'incomplete chain' };
}

This only reads public HTML and sends nothing. It's deliberately narrow: an expired, self-signed or wrong-host certificate makes browsers show a full-page warning, so those stay "broken".

5. Slow is not dead

One WordPress site took 34, 17 and 11 seconds to answer three requests in a row with curl. With a 15-second timeout and one retry, it "loaded in 8.6 s" in one run and came out as "not responding" in the next. Same site, opposite verdicts.

Now a page that times out gets one more try with 45 seconds, and the load time becomes a finding of its own:

let page = await fetchPage(url);
if (page.code === 'TimeoutError') {
  page = await fetchPage(url, { signal: AbortSignal.timeout(45_000) });
}

The first probe (robots.txt) keeps its short timeout and no long retry, so a host that doesn't answer at all still fails fast instead of holding up the batch for a minute.

And one the other way round

A parked domain returned 200 with a 114-byte page. There was no "this domain is for sale" text to match, only a script:

<script>window.location.href="/lander"</script>

Registrar parking pages often look like this. A tiny page whose only job is to send the visitor to /lander now counts as parked:

const PARKED_LANDER = /(window\.)?location(\.href)?\s*=\s*["']\/lander\b/i;
const isParked = html.length < 3000 && PARKED_LANDER.test(html);

Where it ended up

After these changes, 13 of the 57 businesses in that batch that list a website really did have a broken one: 10 domains no longer resolve, one is parked, one listing points at a booking page that's gone, and one server shows its default page. Four of the 13 have between 79 and 178 reviews each, so they're busy businesses still sending visitors to a dead link.

The general lesson for any uptime-style check: a failed request is a fact about your request, not yet a fact about the site. Check the other scheme, the homepage and the certificate, and give a slow site a second chance before calling it dead.

Untrusted certificate? Telling apart three different TLS problems in Node

Timothy Kelvin — Wed, 30 Sep 2026 19:31:34 +0000

When a TLS check fails, a lot of tools stop at "certificate not trusted". That's true, but it doesn't tell anyone what to fix, and the fix is different depending on the cause:

The certificate is self-signed. Nobody vouches for it. The fix is a certificate from a real CA.
The server isn't sending its intermediate certificate. The certificate itself is fine. The fix is in the server config: serve the full chain, not just the leaf.
The chain ends at a root the client doesn't trust. Either it's a private CA, or it's a public root that has since been retired. The fix is on the client side, or a certificate that chains to a current root.

I wanted a domain check I maintain to say which of these it is. Here's what Node actually gives you, tested against the deliberately broken hosts at badssl.com (Node v24).

Reading the certificate anyway

To inspect a bad certificate you have to connect without rejecting it:

const socket = tls.connect({ host, port: 443, servername: host, rejectUnauthorized: false }, () => {
  console.log(socket.authorized, socket.authorizationError);
});

This is for inspection only. You get the certificate and the trust verdict, and you don't send anything over the connection.

Gotcha: authorizationError is a string

My own code had socket.authorizationError?.message, which is always undefined, so the error field in my output was quietly null for every broken certificate. In practice authorizationError is a plain code string:

Host	`authorizationError`
self-signed.badssl.com	`DEPTH_ZERO_SELF_SIGNED_CERT`
incomplete-chain.badssl.com	`UNABLE_TO_VERIFY_LEAF_SIGNATURE`
untrusted-root.badssl.com	`SELF_SIGNED_CERT_IN_CHAIN`
expired.badssl.com	`CERT_HAS_EXPIRED`
wrong.host.badssl.com	`ERR_TLS_CERT_ALTNAME_INVALID`

Why the code alone isn't enough

Node reports one problem, the first it runs into. expired.badssl.com is a good example. Its certificate expired in 2015, and the chain behind it ends at AddTrust External CA Root, which expired in 2020 and isn't among the 120 roots bundled with Node. The code says CERT_HAS_EXPIRED. Renewing the certificate is the right first step, but it's not the whole story.

So besides the code, I look at the chain itself.

Walking the chain

getPeerCertificate(true) returns the leaf with issuerCertificate links to each issuer Node could find. A root, or any self-signed certificate, points to itself. If an intermediate is missing, the links simply stop.

import tls from 'node:tls';
import crypto from 'node:crypto';

const bundledRoots = new Set(
  tls.rootCertificates.map((pem) => new crypto.X509Certificate(pem).fingerprint256),
);

function diagnose(socket) {
  if (socket.authorized) return 'trusted';
  const leaf = socket.getPeerCertificate(true);
  if (leaf.issuerCertificate === leaf) return 'self-signed-leaf';

  const chain = [];
  let cert = leaf;
  while (cert && !chain.includes(cert)) {
    chain.push(cert);
    if (!cert.issuerCertificate || cert.issuerCertificate === cert) break;
    cert = cert.issuerCertificate;
  }
  const last = chain[chain.length - 1];
  if (last.issuerCertificate !== last) return 'chain-incomplete';
  return bundledRoots.has(last.fingerprint256) ? 'chain-ok' : 'unknown-root';
}

The chain.includes check stops the loop at the self-reference. Comparing the last certificate's fingerprint with tls.rootCertificates tells you whether the chain ends at a root Node ships with.

If the chain is fine but the socket still isn't authorized, check the other two usual suspects directly: valid_to for expiry, and tls.checkServerIdentity(host, cert), which returns an error when the name doesn't match.

Results

Host	Node's code	Chain	Other checks
badssl.com	none	trusted
self-signed.badssl.com	`DEPTH_ZERO_SELF_SIGNED_CERT`	self-signed-leaf
incomplete-chain.badssl.com	`UNABLE_TO_VERIFY_LEAF_SIGNATURE`	chain-incomplete
untrusted-root.badssl.com	`SELF_SIGNED_CERT_IN_CHAIN`	unknown-root
expired.badssl.com	`CERT_HAS_EXPIRED`	unknown-root	expired
wrong.host.badssl.com	`ERR_TLS_CERT_ALTNAME_INVALID`	chain-ok	name mismatch

Each row maps to a different fix:

self-signed-leaf: fine for internal testing, otherwise replace it with a CA-issued certificate.
chain-incomplete: the certificate is probably fine and the server config isn't. Browsers can often fill in a missing intermediate on their own, so a site can look fine in a browser and still fail for API clients and scripts.
unknown-root: a private CA, or a retired public root. With a private CA, the clients need its root. With a retired root, the server needs a certificate that chains to a current one.
chain-ok but not authorized: look at the expiry date and the hostname.

One caveat: the chain you get back is the one Node could build, from what the server sent plus its own root store. On badssl.com it ends at ISRG Root X1, which comes from Node's store, not from the server. That's fine for diagnosis, but it means the result describes what this particular client can verify.

Replacing GovTrack's retired bill API with official bulk data

Timothy Kelvin — Sun, 27 Sep 2026 09:00:55 +0000

I maintain a small tool that looks up new U.S. federal bills by keyword. For its whole life it read from GovTrack's API (https://www.govtrack.us/api/v2/bill). Last week a cloud run failed with a 15-second timeout, requests from my own machine got empty replies, and another HTTP client got a 403. A day later the same URL answered again.

Before adding another retry and moving on, I checked what I was actually depending on.

The API had been retired for years

In December 2016 GovTrack announced that its API and bulk data would shut down the following summer, and pointed developers to other sources.
The API documentation page, /developers/api, returns a 404. The Wayback Machine shows it as a 404 since at least February 2023.
robots.txt has Disallow: /api for every user agent, with a 30-second crawl delay.

The endpoint still answers, but nothing about it is supported. An intermittent failure there isn't something to retry around, so I looked for a replacement.

Two official options

The Congress.gov API. It's official and well documented, but it needs an api.data.gov key. The shared DEMO_KEY came back with X-Ratelimit-Limit: 10, which is nothing for a tool running from shared cloud IPs. The /bill list endpoint also has no keyword parameter, and list items don't include the sponsor, so every bill would need its own request.

GovInfo's BILLSTATUS bulk data. The Government Publishing Office publishes one XML file per bill at https://www.govinfo.gov/bulkdata/BILLSTATUS/{congress}/{type}/, built from Congress.gov data. Each file has the titles, sponsor, cosponsors, committees and the full action history. No key, no account, the files are marked public domain, and govinfo.gov's own robots.txt lists the bulk data sitemaps.

I went with the bulk data. It has no search either, but there's no request budget to protect.

Finding recent bills without a search endpoint

Each Congress and bill type has its own sitemap:

https://www.govinfo.gov/sitemap/bulkdata/BILLSTATUS/119hr/sitemap.xml

It lists every file (10,608 House bills in the 119th Congress when I checked). From where I am it downloaded in under 3 seconds, while the JSON directory listing for the same folder took 9 to 13.

async function listBillNumbers(congress, type) {
  const url = `https://www.govinfo.gov/sitemap/bulkdata/BILLSTATUS/${congress}${type}/sitemap.xml`;
  const xml = await (await fetch(url)).text();
  const re = new RegExp(`BILLSTATUS-${congress}${type}(\\d+)\\.xml`, 'g');
  const numbers = new Set([...xml.matchAll(re)].map((m) => Number(m[1])));
  return [...numbers].sort((a, b) => b - a);
}

Bill numbers are handed out in the order bills are introduced. So for "bills from the last 90 days" I walk each type (H.R., S., H.Res. and so on) from the highest number down and stop once three bills in a row are older than the window. Three rather than one, as a margin in case a number was introduced out of order.

Status isn't in the file

The XML has no "current status" field. It has a list of actions, newest first, with free-text descriptions:

<item>
  <actionDate>2026-09-24</actionDate>
  <text>Referred to the House Committee on Oversight and Government Reform.</text>
  <type>IntroReferral</type>
  <actionCode>H11100</actionCode>
</item>

GovTrack gets its data through the open source unitedstates/congress project, and statuses like "Passed House (Senate next)", "Failed Cloture" or "Pocket Vetoed" come from that project's parsing of these action texts. The rules live in parse_bill_action and new_status_after_vote in bill_info.py, about 400 lines of Python and regular expressions. I ported them to JavaScript rather than inventing my own, so existing status filters would keep meaning the same thing.

The core is a small state machine. After a vote, the new status depends on whether it passed, which chamber voted, and whether that's the chamber the bill started in. Simplified:

function statusAfterVote(voteType, passed, chamber, billType, suspension) {
  if (voteType === 'vote') { // vote in the originating chamber
    if (passed) {
      if (billType === 'hres' || billType === 'sres') return 'PASSED:SIMPLERES';
      return chamber === 'h' ? 'PASS_OVER:HOUSE' : 'PASS_OVER:SENATE';
    }
    if (suspension) return 'PROV_KILL:SUSPENSIONFAILED';
    return chamber === 'h' ? 'FAIL:ORIGINATING:HOUSE' : 'FAIL:ORIGINATING:SENATE';
  }
  // ...second-chamber, ping-pong, cloture, veto override and conference votes
}

Two details mattered:

The same vote often appears twice: once from the House or Senate floor system and once from the Library of Congress (sourceSystem code 9), with slightly different text. The original code drops the Library of Congress copy when its text ends with the text of the action just before it. Skip that and a single vote can count twice.
Actions are replayed oldest first, and each one sees the status the earlier ones produced.

Checking the port against the old source

While GovTrack still answered, I pulled bills from it across as many statuses as I could and compared field by field: status, which chamber acts next, status date, introduced date, whether the bill is still alive, and the sponsor's party and state. That was 203 bills over 18 different statuses, including failed veto overrides from earlier Congresses. All 203 matched.

That check is why I'm comfortable with the switch. A port of that many regular expressions is easy to get subtly wrong.

What got worse: keyword search

GovTrack's search looked at more than titles. For "artificial intelligence" over 90 days it returned 123 bills. Matching titles in the bulk data finds 57, all of which GovTrack also returned. I checked eight of the missing ones: the phrase appears nowhere in their BILLSTATUS files, and none had a CRS summary or legislative subject terms yet, including bills introduced in July.

Two changes won part of it back:

OR between alternatives. artificial intelligence OR AI finds 68, and 66 of those are in GovTrack's set. The other two mention AI in their titles but weren't in GovTrack's results.
Singular and plural both match, so "data centers" also finds the "Data Center Fair Share Act".

Real full-text search would mean downloading bill text as well, at least one more file per bill. I've left that out for now.

Keeping it cheap

A 90-day keyword search reads about 2,000 XML files, and most of them get thrown away. A full parse took about 2.5 ms per file on my machine, so each file goes through two stages:

const xml = await res.text();
// the bill's own introducedDate comes before any nested sections
const introduced = xml.match(/<introducedDate>([^<]+)<\/introducedDate>/)?.[1];
if (introduced < cutoff) return 'too-old';

// every word of the keyword has to be somewhere in the raw file
const lower = xml.toLowerCase();
if (!tokens.every((t) => lower.includes(t))) return 'no-match';

const bill = parser.parse(xml).billStatus.bill; // full parse only for candidates

The raw-text check can let extra files through but never drops a real match, because a title match means every word is in the file somewhere. The exact decision still runs on the parsed titles afterwards.

In a cloud container in us-east-1, 2,030 files took about seven seconds at 10 requests at a time. GovInfo sits behind Cloudflare, and plain requests at that rate went through without any challenge.

Takeaways

An endpoint that answers is not the same as a supported API. Check the docs page and robots.txt before building on one, and again when it starts failing.
When an API has no search anyway, bulk files plus a sitemap can be simpler than an API with a tiny rate limit.
If you replace a data source, run old and new side by side on real data before switching. 203 matching bills told me more than rereading the regular expressions ever would.

I Built an Economic Calendar From Government Sources Instead of Scraping ForexFactory

Timothy Kelvin — Fri, 25 Sep 2026 17:48:26 +0000

The most popular economic calendar actors on the Apify Store scrape ForexFactory or Investing.com. Their own docs tell the story: the calendar page blocks datacenter IPs, so you need residential proxies "at any meaningful volume," and a browser for anything past the current week.

But every number on those calendars comes from somewhere. The jobs report comes from the Bureau of Labor Statistics. GDP comes from the Bureau of Economic Analysis. The Fed publishes its own meeting dates. So I built a calendar that reads the agencies directly: no proxies, no browser, and a run finishes in about six seconds.

Getting there took more than five fetch calls. Here's what I ran into.

Five sources, four formats

Source	What it covers	Format
BLS	Jobs report, CPI, PPI, JOLTS	iCalendar feed
BEA	GDP, PCE inflation, trade	iCalendar feed
Census	Retail sales, housing starts, durable goods	HTML table
Federal Reserve	FOMC decisions	HTML page
Treasury	Bill, note and bond auctions	JSON API

Two agencies publishing .ics files was the nicest surprise. That's the format your phone's calendar app subscribes to, and it's meant to be read by software.

1. BLS returns 403 unless you say who you are

My first request to the BLS calendar came back 403 Forbidden. My instinct was that this was bot protection, and my rule for this portfolio is simple: if a site is actively blocking me, I don't try to get around it.

So I tried the honest version first: a User-Agent that names the tool and links to it.

// Agencies block anonymous clients; an honest, identifying User-Agent is what
// their access guidance asks for. This is identification, not disguise.
export const USER_AGENT = 'us-economic-calendar/0.1 (+https://apify.com/m_ctim/us-economic-calendar)';

200 OK. BLS doesn't block scripts, it blocks anonymous scripts. There's a real difference between telling a server who you are and pretending to be Chrome, and it's worth trying the first before assuming you'd need the second.

2. iCalendar lines fold

A long SUMMARY in an .ics file gets split across lines, with the continuation starting with a space:

SUMMARY:Gross Domestic Product\, 3rd Quarter 2025 (Updated Estimate)\, GDP
  by Industry\, and Corporate Profits (Revised)

Split on newlines naively and you get half a title. The fix is one regex before splitting, plus unescaping the commas and semicolons:

// RFC 5545 line folding: a line starting with a space or tab continues the previous one.
const lines = text.replace(/\r\n/g, '\n').replace(/\n[ \t]/g, '').split('\n');
// ...
const value = line.slice(colon + 1).replace(/\\n/gi, '\n').replace(/\\([,;\\])/g, '$1');

3. Two agencies, two ways of writing time

BLS writes local Eastern time with a timezone label:

DTSTART;TZID=US-Eastern:20260103T100000

BEA writes UTC:

DTSTART:20260122T133000Z

Traders want both Eastern (how releases are announced) and UTC (how their systems store time), so every row needs a correct conversion, and daylight saving makes that fiddly. 8:30 AM Eastern is 12:30 UTC in October and 13:30 UTC in November.

Rather than pull in a date library, I used Intl.DateTimeFormat to find the Eastern offset at a given instant. The one subtlety is that you don't know the offset until you know the instant, and you don't know the instant until you know the offset. Two passes settle it, including on the days the clocks change:

export function etToUtcIso(dateStr, timeStr) {
    const [y, m, d] = dateStr.split('-').map(Number);
    const [hh, mm] = timeStr.split(':').map(Number);
    const naive = Date.UTC(y, m - 1, d, hh, mm);
    // Two passes settle the offset correctly on DST transition days.
    let utc = naive - etOffsetMinutes(naive) * 60000;
    utc = naive - etOffsetMinutes(utc) * 60000;
    return new Date(utc).toISOString();
}

The test I trusted: the November 6 jobs report, after the clocks go back on November 1, comes out as 13:30:00Z. October releases come out as 12:30:00Z.

4. BEA names GDP two different ways

Each release gets an impact rating: high for the ones that move markets (jobs, CPI, GDP, the Fed), lower for regional and niche series. My rule for GDP was:

/^Gross Domestic Product/i

The first full-month test looked fine at a glance. Then I listed every row, low impact included, and found Q3 GDP sitting there rated low. BEA's feed calls some releases "Gross Domestic Product, ..." and others "GDP (Advance Estimate), ...". Same release family, two spellings.

// BEA titles it both ways: "Gross Domestic Product, ..." and "GDP (Advance Estimate), ...".
[/^(?:Gross Domestic Product|GDP)\b(?!.*\bby (?:State|County|Metro))/i, 'high', 'growth'],

The lesson I keep relearning: check the rows your filter excluded, not just the ones it kept. The high-impact list looked plausible precisely because nothing in it was wrong. Something was just missing.

5. Some releases belong to two agencies

The monthly trade report is a joint BEA and Census release, and both calendars list it. Without handling that, it shows up twice. Titles differ slightly ("U.S. International Trade in Goods and Services, August 2026" versus the same without the period), so the dedupe key uses the date, the time, and the title up to its first comma:

const normTitle = (t) => t.split(',')[0].toLowerCase().replace(/[^a-z]/g, '').slice(0, 40);
const key = `${raw.date}|${raw.timeEt}|${normTitle(raw.title)}`;

The row that survives records the other agency in an alsoPublishedBy field, so nothing is silently dropped.

6. The Fed's calendar has its own grammar

The FOMC page lists each meeting as a month and a day range. Most are simple, like March and 17-18*. Then there are:

Meetings that span two months: Apr/May with 30-1. The decision comes on the last day, so the date is May 1.
The asterisk: it marks meetings that come with the Summary of Economic Projections, the "dot plot." That goes in the row's title and notes.
Notation votes: entries like 22 (notation vote). There's no 2:00 PM announcement to schedule around, so they're skipped.

const days = dateText.match(/\d+/g);
const lastDay = Number(days[days.length - 1]);
const months = monthText.split('/');
const month = monthIndex(months[months.length - 1]);
const hasProjections = dateText.includes('*');

7. Don't invent data the source doesn't have

Treasury's Fiscal Data API lists announced auctions, with two quirks.

First, it republishes every auction daily under a new record_date, so the same auction appears several times. Keying on CUSIP plus auction date fixes that.

Second, it has no auction time. By convention bills close at 11:30 AM Eastern and notes and bonds at 1:00 PM, and it would have been easy to fill that in. But a convention isn't a published time, and a calendar people schedule trades around shouldn't guess. Auction rows return timeEt: null, and the README says why.

The same rule applies to the biggest thing this calendar lacks: consensus forecasts. Those come from private data vendors, not the agencies. The ForexFactory scrapers have them because ForexFactory compiles them. An official-source calendar can't, so this one doesn't pretend to.

What it returns

{
  "date": "2026-10-29",
  "timeEt": "08:30",
  "timestampUtc": "2026-10-29T12:30:00.000Z",
  "agency": "BEA",
  "title": "GDP (Advance Estimate), 3rd Quarter 2026",
  "period": "3rd Quarter 2026",
  "impact": "high",
  "category": "growth",
  "url": "https://www.bea.gov/news/schedule"
}

Filter by date range, agency, impact or keyword. It charges once per run, so a month of releases costs the same as a day.

Try it: US Economic Calendar on Apify
Source: github.com/timmKal01/us-economic-calendar

If you work with macro data, I'd like to know what you'd add. ECB and Bank of England calendars are the obvious next step.

What crt.sh's Error Pages Taught Me About Retry Logic

Timothy Kelvin — Sun, 20 Sep 2026 14:03:00 +0000

I built a small Apify actor that watches Certificate Transparency logs for a domain, basically "tell me every SSL cert that's been issued for this domain or its subdomains recently." Useful for catching phishing look-alikes and shadow IT before a customer complaint does. The data source is crt.sh, a free community-run search service over CT log data. Getting the actual query working took ten minutes. Making it reliable took a lot longer, and taught me more about "empty result" ambiguity than I expected from what's basically a single GET request.

First problem: 404 doesn't mean "no results."

crt.sh returns bare HTML error pages, not JSON, when it's under load. I first treated a 404 as "no certificates found," which is a completely reasonable read of a 404. Except I'd query the same domain twice in a row and get a 404 once and a real result the next time. Same query, same data, different outcome, because the 404 wasn't about the data at all, it was crt.sh being overloaded. Same story with 502/503/504. None of these are trustworthy "zero results" signals for this service, so they all need to be retried, not accepted.

Second problem: my retry budget wasn't big enough for a real outage.

I started with 4 attempts and exponential backoff, which felt generous. Then during a bad patch I watched it eat 6 consecutive 502s before a success came through, and the actor's rolling 30-day failure rate had climbed to something like 46%. Four attempts just wasn't enough runway to ride out a real bad stretch on a free community service. Bumped it to 8, and capped the backoff instead of letting it grow unbounded, since a successful response can itself take 10-20 seconds once crt.sh is under load, so the retry loop needs real time budget, not just more attempts fired quickly at each other.

Third problem: a hung connection defeats retries entirely.

Plain fetch() has no timeout. One test run just hung, no 502, no slow success, nothing, indefinitely. If a request never resolves, it never reaches the point where my retry logic would even kick in. Added an AbortController with a per-attempt timeout so a dead connection gets treated as a failure and retried like any other, instead of silently stalling the whole run.

Fourth problem, and the sneaky one: a field just disappeared from the response shape.

The actor filters results by date and sorts newest-first, originally using crt.sh's entry_timestamp field. At some point crt.sh's JSON output for this particular query stopped including that field. No error, no warning, just undefined. Which meant new Date(undefined) >= startDate silently evaluated false for every single entry, so the actor returned zero results, every time, regardless of what data actually existed. The scariest kind of bug for a monitoring tool: it fails by going quiet, not by throwing. Switched to not_before (when the cert becomes valid, which is essentially CT-log time anyway) as the log-time proxy instead, since that field is always present.

None of these were exotic failures. Bare error pages under load, a budget that was too small, a fetch with no timeout, a field that quietly vanished. But stacked together they're the difference between "works when I test it" and "actually trustworthy as a monitoring tool," which matters a lot more once the whole point of the thing is telling you about certificates you didn't expect.

If you want to poke at the actual retry code: github.com/timmKal01/certificate-transparency-monitor. The actor itself is live on Apify if you want to point it at a domain.

"Pull a company's open roles straight from their own job board API"

Timothy Kelvin — Fri, 18 Sep 2026 10:05:15 +0000

Most "company hiring signal" tools scrape LinkedIn. company-hiring-tracker does something simpler: most companies that use Greenhouse or Lever for hiring expose their open roles through a public, unauthenticated API on their own job board. No login, no proxy, no headless browser, just a fetch.

Give it a company's ATS (Greenhouse or Lever) and board slug, and it returns each open role: title, location, department, and posting date, as clean JSON.

{ "title": "Senior Backend Engineer", "location": "Remote", "department": "Engineering", "postedAt": "2026-09-10" }

Priced per board queried, not per job returned, so checking a company with 40 open roles costs the same as checking one with 2.

https://apify.com/m_ctim/company-hiring-tracker

DEV Community: Timothy Kelvin

Three event sites already hand you their data. Here's where each one hides it 🗂️

🚦 First, the rules

🧩 Hiding place 1: schema.org JSON-LD

🧩 Hiding place 2: a JavaScript object (Eventbrite)

🧩 Hiding place 3: Next.js page data (Luma and Meetup)

🐛 The bug: zero Meetup events, but only sometimes

📍 Every site wants the place spelled differently

🕐 Three time formats, one output

🔁 The same event on two sites

🎯 What I learned

X's API bills every post it returns, so I moved my filters into the query 💸

🚪 Why the official API

💸 What a read costs

🎯 Put the threshold in the query

👤 The author can cost more than the post

📏 512 characters per query

🔗 Three kinds of link in one array

📝 text is not the whole post

📈 Which number says "viral"?

🌍 Trends come back as names only

🧾 No key, no crash

🎯 What I learned

AI agents were told my inputs were optional, then rejected for leaving them out 🤖

🤖 What an agent sees

🔍 How many of mine?

🙈 Why nothing ever caught it

🚫 Why I didn't run the probe

🔧 Fix 1: a field with a default doesn't need required

🔧 Fix 2: three fields had no default at all

🔇 Fix 3: zero rows is not an answer

🚀 Shipping 29 Actors, one at a time

🎯 What I learned

An RSS feed grew to 500 items and my crawler got flagged as broken 📰

⏱️ Reproducing the daily test

📜 Problem 1: the feed was 500 items long

🚪 Problem 2: Google News says no

🐢 Problem 3, found while fixing: timeouts got three retries

🔧 The fixes

1. Check only the newest N items per feed

2. Check robots.txt at the door too

3. One retry for a timeout, and a per-site limit

4. "Nothing new" is a result

🧪 After

🎯 What I learned

One bad row failed a paying customer's whole run 🧯

🤔 What I could see

📋 The old code trusted the input

💥 Five inputs a real person might type

🔧 Fix 1: understand more, and explain the rest

💸 Fix 2: don't charge for the error rows

📦 Fix 3: batch the requests

🧪 Testing it with a deliberately messy list

🙀 One false alarm

🎯 What I learned

When crt.sh's website is down, query its public database instead

1. The pooler rejects some connection options

2. The query that works, and the one that gets cancelled

3. A dropped connection kills the whole process

4. The timestamps are UTC, but they don't say so

Fitting it into a time budget

One more cause: the example domain

The result

Dead website or false alarm? Five cases from checking small-business sites

1. HTTP says 404, HTTPS works

2. The listed link is dead, the site isn't

3. "Page Not Found" with a 200

4. A certificate chain missing its intermediate

5. Slow is not dead

And one the other way round

Where it ended up

Untrusted certificate? Telling apart three different TLS problems in Node

Reading the certificate anyway

Gotcha: authorizationError is a string

Why the code alone isn't enough

Walking the chain

Results

Replacing GovTrack's retired bill API with official bulk data

The API had been retired for years

Two official options

📝 `text` is not the whole post

🔧 Fix 1: a field with a default doesn't need `required`