DEV Community: Timothy Kelvin

One bad row failed a paying customer's whole run 🧯

Timothy Kelvin — Mon, 12 Oct 2026 02:28:53 +0000

The bug wasn't in the API call. It was in what I assumed people would type.

Hi! I'm Tim, a self-taught developer in Nairobi. I build small data tools that run as Apify Actors, which are serverless jobs that people pay for per result.

One of them takes a list of occupation codes and returns official US wage data from the Bureau of Labor Statistics (BLS) API. You give it 15-1252 and it gives you back what software developers earn, nationally or in one state.

This weekend it got its second paying user. They ran it three times. Two runs worked. One failed.

The failed run wasn't caused by BLS being down or by a timeout. My own code threw an error on one row, and that threw away every other lookup in the same run.

Let's look at how that happened! 👀

🤔 What I could see

Not much. On Apify, a developer can't open another user's runs unless that user chooses to share them, and this one hadn't.

So I had a failure rate (1 run of 3) and no input to reproduce it with.

Instead of guessing, I asked a different question:

What would a real person actually type into this form?

📋 The old code trusted the input

The tool has a simple input: one SOC code field for a quick lookup, plus an occupations JSON field for checking several at once. I added the single field a few weeks ago, after noticing that a raw JSON textarea scares people off. This weekend the JSON field bit back.

Here's what the old code did with each entry:

function buildSeriesId({ areaType, areaCode, socCode, dataTypeCode }) {
    const occupationCode = socCode.replace(/-/g, '');
    if (!/^\d{6}$/.test(occupationCode)) {
        throw new Error(`Invalid SOC code "${socCode}"`);
    }
    return `OEU${areaType}${areaCode}000000${occupationCode}${dataTypeCode}`;
}

And the run loop:

const results = await fetchOccupationWages({ occupations, apiKey });

for (const record of results) {
    await Actor.pushData(record);
    await Actor.charge({ eventName: 'occupation-checked' });
}

Two things are wrong here, and neither looks wrong on its own:

buildSeriesId throws on anything that isn't exactly a 6-digit code.
Nothing is saved until every lookup has finished.

Put together, one bad entry anywhere in the list means zero rows saved.

💥 Five inputs a real person might type

I wrote a quick script that called the lookup function directly with inputs I could imagine someone pasting in:

Input	Old result
`["15-1252", "29-1141"]` (a plain list of codes)	`Cannot read properties of undefined (reading 'replace')`
`{ "socCode": "15-1252.00" }`	`Invalid SOC code "15-1252.00"`
`{ "socCode": "15-1252", "stateCode": "California" }`	`Unknown state code "California"`
`{ "socCode": "software developer" }`	`Invalid SOC code "software developer"`
two good codes, then `15-1252.00`	Whole run fails, 0 rows saved

All five crashed.

The second one hurts the most. 15-1252.00 is how O*NET writes the same code, and O*NET is one of the first places you land when you search for an occupation code. To a person, it's obviously the same occupation. To my regex, it was garbage.

The last row is the real lesson. The two good lookups had already been sent to BLS, so the tool did the work and then threw the results away.

So I wrote down the rule I should have started with:

In a batch tool, the unit of failure is the row, not the run.

🔧 Fix 1: understand more, and explain the rest

Each entry now goes through normalizeEntry before anything else. It accepts more shapes, and when it can't make sense of an entry, it returns an explanation instead of throwing:

export function normalizeEntry(entry) {
    const raw = typeof entry === 'string' || typeof entry === 'number'
        ? { socCode: String(entry) }
        : (entry ?? {});
    const rawSoc = raw.socCode ?? raw.soc ?? raw.code ?? raw.occupationCode;
    // ...
    const digits = String(rawSoc).trim().replace(/\.\d{2}$/, '').replace(/[\s-]/g, '');
    if (!/^\d{6}$/.test(digits)) {
        return { socCode: rawSoc, error: `"${rawSoc}" isn't a SOC code. Use the 6-digit code, e.g. "15-1252" for software developers.` };
    }
    // ...states: "TX", "tx", "Texas" and "texas" all work; "US" or empty means national
}

A plain string is treated as a code, the .00 suffix is dropped, and full state names map to their two-letter codes.

Anything that still doesn't make sense becomes a row with found: false and an error saying what to fix. The run carries on with the rest of the list.

💸 Fix 2: don't charge for the error rows

This is a pay-per-result tool, so "return an error row" raised a second question: should that row cost money?

No. Every lookup now carries a billable flag, and only the lookups BLS actually answered get charged:

for (const { billable, ...record } of records) {
    await Actor.pushData({ ...record, checkedAt: new Date().toISOString() });
    if (billable) await Actor.charge({ eventName: 'occupation-checked' });
}

While I was in there, I found an older billing bug. When BLS refuses a request, for example once its daily limit is used up, the reply comes back with a status like REQUEST_NOT_PROCESSED instead of data. The old code turned that into a found: false row and charged for it. Now those rows say what happened and add "You weren't charged for this lookup."

📦 Fix 3: batch the requests

That BLS limit deserved a closer look. Without a registration key, BLS allows 25 queries a day with up to 25 series per query.

Each occupation lookup needs 5 series: employment, mean and median annual wage, and mean and median hourly wage. The old code sent one request per lookup, so 5 series per request and 25 lookups a day before BLS started refusing.

Now valid lookups are grouped, 5 per request (10 with a key, where the limit is 50 series per query):

const perRequest = Math.floor((apiKey ? 50 : 25) / 5);

for (let start = 0; start < valid.length; start += perRequest) {
    const batch = valid.slice(start, start + perRequest);
    // one POST with every series for every lookup in the batch,
    // then map each returned series back to its lookup by series ID
}

That's 5 times more lookups out of the same daily allowance.

🧪 Testing it with a deliberately messy list

I ran the tool locally with the messiest input I could think of, seven entries in mixed shapes:

{
  "occupations": [
    "15-1252",
    { "socCode": "29-1141.00", "stateCode": "California" },
    { "socCode": "software developer" },
    { "socCode": "15-1252", "stateCode": "Narnia" },
    { "soc": "11-1021", "state": "tx" },
    "13-2011",
    "15-1211"
  ]
}

Result:

Entry	Found	Annual median wage	Error	Charged
15-1252 (national)	✅	135,980		yes
29-1141, CA	✅	140,270		yes
software developer	❌		isn't a SOC code	no
15-1252, Narnia	❌		unknown state	no
11-1021, TX	✅	103,230		yes
13-2011 (national)	✅	83,680		yes
15-1211 (national)	✅	105,850		yes

7 rows out, 5 charged, and all 5 valid lookups went to BLS in a single request.

I also tested an empty form. Apify runs every public Actor once a day with its default input, and flags it as under maintenance if that keeps failing. An empty form now looks up an example code instead of throwing.

🙀 One false alarm

My first local test runs exited with code 127, which made me think I'd broken the shutdown.

The log said:

Assertion failed: !(handle->flags & UV_HANDLE_CLOSING), file src\win\async.c, line 94

Before touching anything, I stashed my changes and ran the old code. It printed the same assertion.

It's Node 24 on Windows asserting while the process shuts down, after all the work is done. The cloud runs these on Linux with Node 20, so it never shows up there. If you only check exit codes on a Windows machine, it's worth knowing about.

🎯 What I learned

Fail the row, not the run. In anything that takes a list, one bad item should produce one error row.
Save as you go, or at least don't throw away finished work. The old code had done the good lookups and still saved nothing.
Accept what people actually type. Codes copied from other sites, full state names and plain lists aren't edge cases. They're the input.
An error row needs a reason and a price. Say what to fix, and don't charge for it.
Read the upstream API's limits before they read you. Batching gave 5 times the headroom for free.

I still don't know exactly what that customer typed. But every input I could think of now gets either an answer or a clear reason why not.

When crt.sh's website is down, query its public database instead

Timothy Kelvin — Thu, 08 Oct 2026 18:20:42 +0000

I maintain a small tool that searches Certificate Transparency logs for new certificates on a domain. It reads crt.sh's JSON output, and it already retries the 404s and 502s that crt.sh returns when it's busy.

This week retrying stopped being enough. For more than an hour, every request to the website failed: a 502 in under a second, or no answer at all until my 40-second timeout. The platform I run the tool on tests it once a day with its default input, and it had failed 5 of its last 27 daily tests, which was enough to get it flagged as under maintenance.

crt.sh has a second door, though. Its database is open to the public, read-only:

psql -h crt.sh -p 5432 -U guest certwatch

While the website was returning 502s, the database answered in 9 to 35 seconds. That's slower than the website on a good day, but it's an answer. So the tool now tries the website three times and then falls back to the database. Getting that fallback right took four fixes.

1. The pooler rejects some connection options

My first connection set a server-side statement timeout:

new pg.Client({ host: 'crt.sh', port: 5432, user: 'guest', database: 'certwatch', statement_timeout: 60000 });

It failed straight away with unsupported startup parameter: statement_timeout. The database sits behind a connection pooler that only accepts a few startup parameters. node-postgres has a client-side query_timeout that does the same job without asking the server:

const client = new pg.Client({
    host: 'crt.sh', port: 5432, user: 'guest', database: 'certwatch',
    connectionTimeoutMillis: 20000,
    query_timeout: 60000,
});

2. The query that works, and the one that gets cancelled

crt.sh searches by identity. Its schema has a certificate_and_identities view with a full-text index, so this finds every unexpired certificate for a domain and its subdomains:

SELECT c.id,
       x509_commonName(c.certificate) AS common_name,
       ca.name AS issuer_name,
       encode(x509_serialNumber(c.certificate), 'hex') AS serial_number,
       to_char(x509_notBefore(c.certificate), 'YYYY-MM-DD"T"HH24:MI:SS') AS not_before,
       to_char(x509_notAfter(c.certificate), 'YYYY-MM-DD"T"HH24:MI:SS') AS not_after
FROM certificate c
JOIN ca ON ca.id = c.issuer_ca_id
WHERE c.id IN (
    SELECT cai.certificate_id
    FROM certificate_and_identities cai
    WHERE plainto_tsquery('certwatch', $1) @@ identities(cai.certificate)
      AND (cai.name_value ILIKE '%.' || $1 OR lower(cai.name_value) = lower($1))
    LIMIT 5000)
  AND x509_notAfter(c.certificate) > now()
  AND x509_notBefore(c.certificate) >= $2::timestamp
ORDER BY x509_notBefore(c.certificate) DESC
LIMIT $3

For a mid-sized domain this took about 35 seconds, and for a small one about 9.

I tried to make the 5,000-match cap keep the newest certificates by adding ORDER BY cai.certificate_id DESC inside the subquery. That version never finished. crt.sh cancelled it with canceling statement due to statement timeout after about 54 seconds: the server has its own limit, and sorting every match goes over it. So the subquery stays unordered, which means a very large domain can miss some recent certificates on this path. For a fallback, that's a fair trade.

3. A dropped connection kills the whole process

After a cancelled query, my test script crashed with an error I never caught:

error: client_idle_timeout
Emitted 'error' event on Client instance

The pooler closes idle and cancelled connections, and node-postgres reports that as an error event on the client. In Node, an error event with no listener is an uncaught exception, so the process dies, retry logic and all. One line fixes it:

client.on('error', (err) => console.debug('crt.sh connection closed:', err.message));

4. The timestamps are UTC, but they don't say so

The first fallback run on my laptop said a certificate became valid at 07:08:23. The same code in the cloud said 10:08:23. To find out which was right, I decoded the certificate itself:

import { X509Certificate } from 'node:crypto';

const { rows } = await client.query('SELECT certificate FROM certificate WHERE id = $1', [id]);
console.log(new X509Certificate(rows[0].certificate).validFrom); // Oct  3 10:08:23 2026 GMT

x509_notBefore() returns a timestamp without time zone that holds UTC. node-postgres turns that type into a JavaScript Date in the machine's local time zone. The cloud runs on UTC, so it was right by luck. My laptop is on UTC+3, so it was three hours off. Formatting the value in SQL with to_char, as in the query above, avoids the driver's guess entirely, and when I compare dates in JavaScript I add the Z myself:

const utc = (crtshTime) => new Date(`${crtshTime}Z`);

Fitting it into a time budget

The daily test expects a finished run with results within five minutes, and a monitor running on a schedule shouldn't hang either. So everything shares one 4-minute budget:

The website gets three attempts with a 20-second timeout each, capped at 75 seconds in total.
The database gets whatever is left, with one retry if at least 30 seconds remain.
If both fail, the run fails with a message that names both errors, instead of quietly reporting zero certificates.

One more cause: the example domain

The default input searched example.com for certificates from the last 30 days. Looking at 180 days of history in the database showed that example.com once went 59 days without a new certificate. So on some days the default run correctly found nothing, and an empty result counts as a failed test too. I switched the default to a domain with lots of subdomains on 90-day certificates, whose longest gap was 19.5 days.

The result

With the website still returning 502s, the fallback found four recent certificates in 32 seconds on my laptop and 40 seconds in the cloud.

What I'd pass on:

If a free service has a second way in (here, a public read-only database), wire it up as a fallback before you need it.
Behind a connection pooler, set timeouts on the client, not the server.
Always attach an error listener to database clients in Node.
Don't let a driver guess the time zone of a timestamp without time zone. Format it yourself.
Pick test inputs that can't legitimately come back empty.

Dead website or false alarm? Five cases from checking small-business sites

Timothy Kelvin — Thu, 01 Oct 2026 06:10:30 +0000

I've been writing a checker that looks at the website a local business lists (salons, barbers, plumbers) and decides whether it works. The first version was simple: fetch the URL, and if that fails, call the site broken.

Then I ran it on about 50 salon, spa and barber websites in Nairobi and read every verdict by hand. A few "broken" sites worked fine in a browser, one site came out dead in one run and healthy in the next, and a parked domain looked healthy. Below are the cases and what the checker does now. Everything here is Node 20 with the built-in fetch.

The helper used in the snippets:

async function fetchPage(url, init = {}) {
  try {
    const res = await fetch(url, { redirect: 'follow', signal: AbortSignal.timeout(15_000), ...init });
    return { ok: res.ok, status: res.status, finalUrl: res.url, body: await res.text() };
  } catch (err) {
    return { ok: false, status: 0, code: err.cause?.code ?? err.name };
  }
}

1. HTTP says 404, HTTPS works

I request http:// first, because whether plain HTTP redirects to HTTPS is worth knowing. One barbershop's server answered plain HTTP with a 404 and no redirect, while https:// served the real site. Chrome has defaulted to HTTPS for typed addresses since version 90, and search results link to HTTPS, so almost nobody sees that 404. Calling the site broken was wrong.

Now an error over HTTP only counts if HTTPS fails too. The missing redirect is still recorded, as its own smaller finding:

const http = await fetchPage(`http://${host}${path}`);
const redirected = Boolean(http.finalUrl?.startsWith('https:'));
let page = http;
if (!http.ok && httpsWorks && !redirected) {
  // Plain HTTP answers with an error page, but HTTPS visitors get the site.
  page = await fetchPage(`https://${host}${path}`);
}
const httpRedirectsToHttps = http.status > 0 ? redirected : null;

httpsWorks comes from an earlier request (I fetch https://host/robots.txt first, which doubles as the HTTPS check). The !redirected guard matters: if HTTP already redirected to HTTPS and that page is a 404, fetching it again tells you nothing new.

2. The listed link is dead, the site isn't

Business listings often point at a deep page rather than the homepage. One salon's listing linked to /beauty-salon, which returns 404, while the homepage works. "Your website is broken" is the wrong verdict. "Your listing links to a missing page" is the right one, and it's a smaller fix for the owner.

Falling back to the homepage is only safe when the domain belongs to the business, though. Another listing pointed at a page on a booking platform. That platform's homepage works, and it says nothing about the salon. So the fallback only runs when the domain carries a distinctive word from the business name:

import { getDomain } from 'tldts';

const GENERIC = new Set(['salon', 'hair', 'beauty', 'barber', 'barbers', 'studio', 'spa', 'nails', 'house', 'shop']);

function domainCarriesName(name, url) {
  const label = (getDomain(new URL(url).hostname) ?? '').split('.')[0];
  const words = name.toLowerCase().split(/[^a-z0-9]+/).filter((w) => w.length >= 4 && !GENERIC.has(w));
  return label.length >= 4 && words.some((w) => label.includes(w));
}

domainCarriesName('House of Treasures Beauty Salon', 'https://houseoftreasureskenya.com/beauty-salon'); // true
domainCarriesName('Posh Palace Hair Studio', 'https://booking-platform.example/p/12932'); // false

The generic-word list is what keeps "Glam Hair Salon" from matching a domain like salonbookings.com.

3. "Page Not Found" with a 200

That booking-platform link from case 2 returned status 200, with <title>Page Not Found</title> and almost no text. The status code says fine, the page says gone. On short pages I now check the title:

const NOT_FOUND_TITLE = /(^|[^a-z0-9])(404|page not found|not found|page (could not|cannot|can't) be found)([^a-z0-9]|$)/i;

function isSoft404($, visibleText) {
  return visibleText.length < 3000 && NOT_FOUND_TITLE.test($('title').first().text().trim());
}

The length guard keeps a real page from being flagged because its title happens to contain "not found". The check also runs after the parked-domain and default-server-page checks, so those keep their more specific labels.

4. A certificate chain missing its intermediate

One server sent only its own certificate, without the intermediate that links it to a trusted root. openssl s_client shows it plainly: one certificate in the chain and Verify return code: 21 (unable to verify the first certificate).

Browsers usually cope. Chrome and Edge fetch the missing intermediate from the URL inside the certificate, and Firefox ships a list of known intermediates. Node's fetch on a Linux server doesn't, and fails with UNABLE_TO_VERIFY_LEAF_SIGNATURE. (The same request succeeded on my Windows laptop with Node 24, so test where you deploy.)

For that one error code, the checker reads the page anyway and records the certificate problem as a finding instead of "down":

import { Agent, fetch as undiciFetch } from 'undici';

const lenient = new Agent({ connect: { rejectUnauthorized: false } });

let page = await fetchPage(url);
if (page.code === 'UNABLE_TO_VERIFY_LEAF_SIGNATURE') {
  const res = await undiciFetch(url, { dispatcher: lenient, signal: AbortSignal.timeout(15_000) });
  page = { ok: res.ok, status: res.status, finalUrl: res.url, body: await res.text(), certIssue: 'incomplete chain' };
}

This only reads public HTML and sends nothing. It's deliberately narrow: an expired, self-signed or wrong-host certificate makes browsers show a full-page warning, so those stay "broken".

5. Slow is not dead

One WordPress site took 34, 17 and 11 seconds to answer three requests in a row with curl. With a 15-second timeout and one retry, it "loaded in 8.6 s" in one run and came out as "not responding" in the next. Same site, opposite verdicts.

Now a page that times out gets one more try with 45 seconds, and the load time becomes a finding of its own:

let page = await fetchPage(url);
if (page.code === 'TimeoutError') {
  page = await fetchPage(url, { signal: AbortSignal.timeout(45_000) });
}

The first probe (robots.txt) keeps its short timeout and no long retry, so a host that doesn't answer at all still fails fast instead of holding up the batch for a minute.

And one the other way round

A parked domain returned 200 with a 114-byte page. There was no "this domain is for sale" text to match, only a script:

<script>window.location.href="/lander"</script>

Registrar parking pages often look like this. A tiny page whose only job is to send the visitor to /lander now counts as parked:

const PARKED_LANDER = /(window\.)?location(\.href)?\s*=\s*["']\/lander\b/i;
const isParked = html.length < 3000 && PARKED_LANDER.test(html);

Where it ended up

After these changes, 13 of the 57 businesses in that batch that list a website really did have a broken one: 10 domains no longer resolve, one is parked, one listing points at a booking page that's gone, and one server shows its default page. Four of the 13 have between 79 and 178 reviews each, so they're busy businesses still sending visitors to a dead link.

The general lesson for any uptime-style check: a failed request is a fact about your request, not yet a fact about the site. Check the other scheme, the homepage and the certificate, and give a slow site a second chance before calling it dead.

Untrusted certificate? Telling apart three different TLS problems in Node

Timothy Kelvin — Wed, 30 Sep 2026 19:31:34 +0000

When a TLS check fails, a lot of tools stop at "certificate not trusted". That's true, but it doesn't tell anyone what to fix, and the fix is different depending on the cause:

The certificate is self-signed. Nobody vouches for it. The fix is a certificate from a real CA.
The server isn't sending its intermediate certificate. The certificate itself is fine. The fix is in the server config: serve the full chain, not just the leaf.
The chain ends at a root the client doesn't trust. Either it's a private CA, or it's a public root that has since been retired. The fix is on the client side, or a certificate that chains to a current root.

I wanted a domain check I maintain to say which of these it is. Here's what Node actually gives you, tested against the deliberately broken hosts at badssl.com (Node v24).

Reading the certificate anyway

To inspect a bad certificate you have to connect without rejecting it:

const socket = tls.connect({ host, port: 443, servername: host, rejectUnauthorized: false }, () => {
  console.log(socket.authorized, socket.authorizationError);
});

This is for inspection only. You get the certificate and the trust verdict, and you don't send anything over the connection.

Gotcha: authorizationError is a string

My own code had socket.authorizationError?.message, which is always undefined, so the error field in my output was quietly null for every broken certificate. In practice authorizationError is a plain code string:

Host	`authorizationError`
self-signed.badssl.com	`DEPTH_ZERO_SELF_SIGNED_CERT`
incomplete-chain.badssl.com	`UNABLE_TO_VERIFY_LEAF_SIGNATURE`
untrusted-root.badssl.com	`SELF_SIGNED_CERT_IN_CHAIN`
expired.badssl.com	`CERT_HAS_EXPIRED`
wrong.host.badssl.com	`ERR_TLS_CERT_ALTNAME_INVALID`

Why the code alone isn't enough

Node reports one problem, the first it runs into. expired.badssl.com is a good example. Its certificate expired in 2015, and the chain behind it ends at AddTrust External CA Root, which expired in 2020 and isn't among the 120 roots bundled with Node. The code says CERT_HAS_EXPIRED. Renewing the certificate is the right first step, but it's not the whole story.

So besides the code, I look at the chain itself.

Walking the chain

getPeerCertificate(true) returns the leaf with issuerCertificate links to each issuer Node could find. A root, or any self-signed certificate, points to itself. If an intermediate is missing, the links simply stop.

import tls from 'node:tls';
import crypto from 'node:crypto';

const bundledRoots = new Set(
  tls.rootCertificates.map((pem) => new crypto.X509Certificate(pem).fingerprint256),
);

function diagnose(socket) {
  if (socket.authorized) return 'trusted';
  const leaf = socket.getPeerCertificate(true);
  if (leaf.issuerCertificate === leaf) return 'self-signed-leaf';

  const chain = [];
  let cert = leaf;
  while (cert && !chain.includes(cert)) {
    chain.push(cert);
    if (!cert.issuerCertificate || cert.issuerCertificate === cert) break;
    cert = cert.issuerCertificate;
  }
  const last = chain[chain.length - 1];
  if (last.issuerCertificate !== last) return 'chain-incomplete';
  return bundledRoots.has(last.fingerprint256) ? 'chain-ok' : 'unknown-root';
}

The chain.includes check stops the loop at the self-reference. Comparing the last certificate's fingerprint with tls.rootCertificates tells you whether the chain ends at a root Node ships with.

If the chain is fine but the socket still isn't authorized, check the other two usual suspects directly: valid_to for expiry, and tls.checkServerIdentity(host, cert), which returns an error when the name doesn't match.

Results

Host	Node's code	Chain	Other checks
badssl.com	none	trusted
self-signed.badssl.com	`DEPTH_ZERO_SELF_SIGNED_CERT`	self-signed-leaf
incomplete-chain.badssl.com	`UNABLE_TO_VERIFY_LEAF_SIGNATURE`	chain-incomplete
untrusted-root.badssl.com	`SELF_SIGNED_CERT_IN_CHAIN`	unknown-root
expired.badssl.com	`CERT_HAS_EXPIRED`	unknown-root	expired
wrong.host.badssl.com	`ERR_TLS_CERT_ALTNAME_INVALID`	chain-ok	name mismatch

Each row maps to a different fix:

self-signed-leaf: fine for internal testing, otherwise replace it with a CA-issued certificate.
chain-incomplete: the certificate is probably fine and the server config isn't. Browsers can often fill in a missing intermediate on their own, so a site can look fine in a browser and still fail for API clients and scripts.
unknown-root: a private CA, or a retired public root. With a private CA, the clients need its root. With a retired root, the server needs a certificate that chains to a current one.
chain-ok but not authorized: look at the expiry date and the hostname.

One caveat: the chain you get back is the one Node could build, from what the server sent plus its own root store. On badssl.com it ends at ISRG Root X1, which comes from Node's store, not from the server. That's fine for diagnosis, but it means the result describes what this particular client can verify.

Replacing GovTrack's retired bill API with official bulk data

Timothy Kelvin — Sun, 27 Sep 2026 09:00:55 +0000

I maintain a small tool that looks up new U.S. federal bills by keyword. For its whole life it read from GovTrack's API (https://www.govtrack.us/api/v2/bill). Last week a cloud run failed with a 15-second timeout, requests from my own machine got empty replies, and another HTTP client got a 403. A day later the same URL answered again.

Before adding another retry and moving on, I checked what I was actually depending on.

The API had been retired for years

In December 2016 GovTrack announced that its API and bulk data would shut down the following summer, and pointed developers to other sources.
The API documentation page, /developers/api, returns a 404. The Wayback Machine shows it as a 404 since at least February 2023.
robots.txt has Disallow: /api for every user agent, with a 30-second crawl delay.

The endpoint still answers, but nothing about it is supported. An intermittent failure there isn't something to retry around, so I looked for a replacement.

Two official options

The Congress.gov API. It's official and well documented, but it needs an api.data.gov key. The shared DEMO_KEY came back with X-Ratelimit-Limit: 10, which is nothing for a tool running from shared cloud IPs. The /bill list endpoint also has no keyword parameter, and list items don't include the sponsor, so every bill would need its own request.

GovInfo's BILLSTATUS bulk data. The Government Publishing Office publishes one XML file per bill at https://www.govinfo.gov/bulkdata/BILLSTATUS/{congress}/{type}/, built from Congress.gov data. Each file has the titles, sponsor, cosponsors, committees and the full action history. No key, no account, the files are marked public domain, and govinfo.gov's own robots.txt lists the bulk data sitemaps.

I went with the bulk data. It has no search either, but there's no request budget to protect.

Finding recent bills without a search endpoint

Each Congress and bill type has its own sitemap:

https://www.govinfo.gov/sitemap/bulkdata/BILLSTATUS/119hr/sitemap.xml

It lists every file (10,608 House bills in the 119th Congress when I checked). From where I am it downloaded in under 3 seconds, while the JSON directory listing for the same folder took 9 to 13.

async function listBillNumbers(congress, type) {
  const url = `https://www.govinfo.gov/sitemap/bulkdata/BILLSTATUS/${congress}${type}/sitemap.xml`;
  const xml = await (await fetch(url)).text();
  const re = new RegExp(`BILLSTATUS-${congress}${type}(\\d+)\\.xml`, 'g');
  const numbers = new Set([...xml.matchAll(re)].map((m) => Number(m[1])));
  return [...numbers].sort((a, b) => b - a);
}

Bill numbers are handed out in the order bills are introduced. So for "bills from the last 90 days" I walk each type (H.R., S., H.Res. and so on) from the highest number down and stop once three bills in a row are older than the window. Three rather than one, as a margin in case a number was introduced out of order.

Status isn't in the file

The XML has no "current status" field. It has a list of actions, newest first, with free-text descriptions:

<item>
  <actionDate>2026-09-24</actionDate>
  <text>Referred to the House Committee on Oversight and Government Reform.</text>
  <type>IntroReferral</type>
  <actionCode>H11100</actionCode>
</item>

GovTrack gets its data through the open source unitedstates/congress project, and statuses like "Passed House (Senate next)", "Failed Cloture" or "Pocket Vetoed" come from that project's parsing of these action texts. The rules live in parse_bill_action and new_status_after_vote in bill_info.py, about 400 lines of Python and regular expressions. I ported them to JavaScript rather than inventing my own, so existing status filters would keep meaning the same thing.

The core is a small state machine. After a vote, the new status depends on whether it passed, which chamber voted, and whether that's the chamber the bill started in. Simplified:

function statusAfterVote(voteType, passed, chamber, billType, suspension) {
  if (voteType === 'vote') { // vote in the originating chamber
    if (passed) {
      if (billType === 'hres' || billType === 'sres') return 'PASSED:SIMPLERES';
      return chamber === 'h' ? 'PASS_OVER:HOUSE' : 'PASS_OVER:SENATE';
    }
    if (suspension) return 'PROV_KILL:SUSPENSIONFAILED';
    return chamber === 'h' ? 'FAIL:ORIGINATING:HOUSE' : 'FAIL:ORIGINATING:SENATE';
  }
  // ...second-chamber, ping-pong, cloture, veto override and conference votes
}

Two details mattered:

The same vote often appears twice: once from the House or Senate floor system and once from the Library of Congress (sourceSystem code 9), with slightly different text. The original code drops the Library of Congress copy when its text ends with the text of the action just before it. Skip that and a single vote can count twice.
Actions are replayed oldest first, and each one sees the status the earlier ones produced.

Checking the port against the old source

While GovTrack still answered, I pulled bills from it across as many statuses as I could and compared field by field: status, which chamber acts next, status date, introduced date, whether the bill is still alive, and the sponsor's party and state. That was 203 bills over 18 different statuses, including failed veto overrides from earlier Congresses. All 203 matched.

That check is why I'm comfortable with the switch. A port of that many regular expressions is easy to get subtly wrong.

What got worse: keyword search

GovTrack's search looked at more than titles. For "artificial intelligence" over 90 days it returned 123 bills. Matching titles in the bulk data finds 57, all of which GovTrack also returned. I checked eight of the missing ones: the phrase appears nowhere in their BILLSTATUS files, and none had a CRS summary or legislative subject terms yet, including bills introduced in July.

Two changes won part of it back:

OR between alternatives. artificial intelligence OR AI finds 68, and 66 of those are in GovTrack's set. The other two mention AI in their titles but weren't in GovTrack's results.
Singular and plural both match, so "data centers" also finds the "Data Center Fair Share Act".

Real full-text search would mean downloading bill text as well, at least one more file per bill. I've left that out for now.

Keeping it cheap

A 90-day keyword search reads about 2,000 XML files, and most of them get thrown away. A full parse took about 2.5 ms per file on my machine, so each file goes through two stages:

const xml = await res.text();
// the bill's own introducedDate comes before any nested sections
const introduced = xml.match(/<introducedDate>([^<]+)<\/introducedDate>/)?.[1];
if (introduced < cutoff) return 'too-old';

// every word of the keyword has to be somewhere in the raw file
const lower = xml.toLowerCase();
if (!tokens.every((t) => lower.includes(t))) return 'no-match';

const bill = parser.parse(xml).billStatus.bill; // full parse only for candidates

The raw-text check can let extra files through but never drops a real match, because a title match means every word is in the file somewhere. The exact decision still runs on the parsed titles afterwards.

In a cloud container in us-east-1, 2,030 files took about seven seconds at 10 requests at a time. GovInfo sits behind Cloudflare, and plain requests at that rate went through without any challenge.

Takeaways

An endpoint that answers is not the same as a supported API. Check the docs page and robots.txt before building on one, and again when it starts failing.
When an API has no search anyway, bulk files plus a sitemap can be simpler than an API with a tiny rate limit.
If you replace a data source, run old and new side by side on real data before switching. 203 matching bills told me more than rereading the regular expressions ever would.

I Built an Economic Calendar From Government Sources Instead of Scraping ForexFactory

Timothy Kelvin — Fri, 25 Sep 2026 17:48:26 +0000

The most popular economic calendar actors on the Apify Store scrape ForexFactory or Investing.com. Their own docs tell the story: the calendar page blocks datacenter IPs, so you need residential proxies "at any meaningful volume," and a browser for anything past the current week.

But every number on those calendars comes from somewhere. The jobs report comes from the Bureau of Labor Statistics. GDP comes from the Bureau of Economic Analysis. The Fed publishes its own meeting dates. So I built a calendar that reads the agencies directly: no proxies, no browser, and a run finishes in about six seconds.

Getting there took more than five fetch calls. Here's what I ran into.

Five sources, four formats

Source	What it covers	Format
BLS	Jobs report, CPI, PPI, JOLTS	iCalendar feed
BEA	GDP, PCE inflation, trade	iCalendar feed
Census	Retail sales, housing starts, durable goods	HTML table
Federal Reserve	FOMC decisions	HTML page
Treasury	Bill, note and bond auctions	JSON API

Two agencies publishing .ics files was the nicest surprise. That's the format your phone's calendar app subscribes to, and it's meant to be read by software.

1. BLS returns 403 unless you say who you are

My first request to the BLS calendar came back 403 Forbidden. My instinct was that this was bot protection, and my rule for this portfolio is simple: if a site is actively blocking me, I don't try to get around it.

So I tried the honest version first: a User-Agent that names the tool and links to it.

// Agencies block anonymous clients; an honest, identifying User-Agent is what
// their access guidance asks for. This is identification, not disguise.
export const USER_AGENT = 'us-economic-calendar/0.1 (+https://apify.com/m_ctim/us-economic-calendar)';

200 OK. BLS doesn't block scripts, it blocks anonymous scripts. There's a real difference between telling a server who you are and pretending to be Chrome, and it's worth trying the first before assuming you'd need the second.

2. iCalendar lines fold

A long SUMMARY in an .ics file gets split across lines, with the continuation starting with a space:

SUMMARY:Gross Domestic Product\, 3rd Quarter 2025 (Updated Estimate)\, GDP
  by Industry\, and Corporate Profits (Revised)

Split on newlines naively and you get half a title. The fix is one regex before splitting, plus unescaping the commas and semicolons:

// RFC 5545 line folding: a line starting with a space or tab continues the previous one.
const lines = text.replace(/\r\n/g, '\n').replace(/\n[ \t]/g, '').split('\n');
// ...
const value = line.slice(colon + 1).replace(/\\n/gi, '\n').replace(/\\([,;\\])/g, '$1');

3. Two agencies, two ways of writing time

BLS writes local Eastern time with a timezone label:

DTSTART;TZID=US-Eastern:20260103T100000

BEA writes UTC:

DTSTART:20260122T133000Z

Traders want both Eastern (how releases are announced) and UTC (how their systems store time), so every row needs a correct conversion, and daylight saving makes that fiddly. 8:30 AM Eastern is 12:30 UTC in October and 13:30 UTC in November.

Rather than pull in a date library, I used Intl.DateTimeFormat to find the Eastern offset at a given instant. The one subtlety is that you don't know the offset until you know the instant, and you don't know the instant until you know the offset. Two passes settle it, including on the days the clocks change:

export function etToUtcIso(dateStr, timeStr) {
    const [y, m, d] = dateStr.split('-').map(Number);
    const [hh, mm] = timeStr.split(':').map(Number);
    const naive = Date.UTC(y, m - 1, d, hh, mm);
    // Two passes settle the offset correctly on DST transition days.
    let utc = naive - etOffsetMinutes(naive) * 60000;
    utc = naive - etOffsetMinutes(utc) * 60000;
    return new Date(utc).toISOString();
}

The test I trusted: the November 6 jobs report, after the clocks go back on November 1, comes out as 13:30:00Z. October releases come out as 12:30:00Z.

4. BEA names GDP two different ways

Each release gets an impact rating: high for the ones that move markets (jobs, CPI, GDP, the Fed), lower for regional and niche series. My rule for GDP was:

/^Gross Domestic Product/i

The first full-month test looked fine at a glance. Then I listed every row, low impact included, and found Q3 GDP sitting there rated low. BEA's feed calls some releases "Gross Domestic Product, ..." and others "GDP (Advance Estimate), ...". Same release family, two spellings.

// BEA titles it both ways: "Gross Domestic Product, ..." and "GDP (Advance Estimate), ...".
[/^(?:Gross Domestic Product|GDP)\b(?!.*\bby (?:State|County|Metro))/i, 'high', 'growth'],

The lesson I keep relearning: check the rows your filter excluded, not just the ones it kept. The high-impact list looked plausible precisely because nothing in it was wrong. Something was just missing.

5. Some releases belong to two agencies

The monthly trade report is a joint BEA and Census release, and both calendars list it. Without handling that, it shows up twice. Titles differ slightly ("U.S. International Trade in Goods and Services, August 2026" versus the same without the period), so the dedupe key uses the date, the time, and the title up to its first comma:

const normTitle = (t) => t.split(',')[0].toLowerCase().replace(/[^a-z]/g, '').slice(0, 40);
const key = `${raw.date}|${raw.timeEt}|${normTitle(raw.title)}`;

The row that survives records the other agency in an alsoPublishedBy field, so nothing is silently dropped.

6. The Fed's calendar has its own grammar

The FOMC page lists each meeting as a month and a day range. Most are simple, like March and 17-18*. Then there are:

Meetings that span two months: Apr/May with 30-1. The decision comes on the last day, so the date is May 1.
The asterisk: it marks meetings that come with the Summary of Economic Projections, the "dot plot." That goes in the row's title and notes.
Notation votes: entries like 22 (notation vote). There's no 2:00 PM announcement to schedule around, so they're skipped.

const days = dateText.match(/\d+/g);
const lastDay = Number(days[days.length - 1]);
const months = monthText.split('/');
const month = monthIndex(months[months.length - 1]);
const hasProjections = dateText.includes('*');

7. Don't invent data the source doesn't have

Treasury's Fiscal Data API lists announced auctions, with two quirks.

First, it republishes every auction daily under a new record_date, so the same auction appears several times. Keying on CUSIP plus auction date fixes that.

Second, it has no auction time. By convention bills close at 11:30 AM Eastern and notes and bonds at 1:00 PM, and it would have been easy to fill that in. But a convention isn't a published time, and a calendar people schedule trades around shouldn't guess. Auction rows return timeEt: null, and the README says why.

The same rule applies to the biggest thing this calendar lacks: consensus forecasts. Those come from private data vendors, not the agencies. The ForexFactory scrapers have them because ForexFactory compiles them. An official-source calendar can't, so this one doesn't pretend to.

What it returns

{
  "date": "2026-10-29",
  "timeEt": "08:30",
  "timestampUtc": "2026-10-29T12:30:00.000Z",
  "agency": "BEA",
  "title": "GDP (Advance Estimate), 3rd Quarter 2026",
  "period": "3rd Quarter 2026",
  "impact": "high",
  "category": "growth",
  "url": "https://www.bea.gov/news/schedule"
}

Filter by date range, agency, impact or keyword. It charges once per run, so a month of releases costs the same as a day.

Try it: US Economic Calendar on Apify
Source: github.com/timmKal01/us-economic-calendar

If you work with macro data, I'd like to know what you'd add. ECB and Bank of England calendars are the obvious next step.

What crt.sh's Error Pages Taught Me About Retry Logic

Timothy Kelvin — Sun, 20 Sep 2026 14:03:00 +0000

I built a small Apify actor that watches Certificate Transparency logs for a domain, basically "tell me every SSL cert that's been issued for this domain or its subdomains recently." Useful for catching phishing look-alikes and shadow IT before a customer complaint does. The data source is crt.sh, a free community-run search service over CT log data. Getting the actual query working took ten minutes. Making it reliable took a lot longer, and taught me more about "empty result" ambiguity than I expected from what's basically a single GET request.

First problem: 404 doesn't mean "no results."

crt.sh returns bare HTML error pages, not JSON, when it's under load. I first treated a 404 as "no certificates found," which is a completely reasonable read of a 404. Except I'd query the same domain twice in a row and get a 404 once and a real result the next time. Same query, same data, different outcome, because the 404 wasn't about the data at all, it was crt.sh being overloaded. Same story with 502/503/504. None of these are trustworthy "zero results" signals for this service, so they all need to be retried, not accepted.

Second problem: my retry budget wasn't big enough for a real outage.

I started with 4 attempts and exponential backoff, which felt generous. Then during a bad patch I watched it eat 6 consecutive 502s before a success came through, and the actor's rolling 30-day failure rate had climbed to something like 46%. Four attempts just wasn't enough runway to ride out a real bad stretch on a free community service. Bumped it to 8, and capped the backoff instead of letting it grow unbounded, since a successful response can itself take 10-20 seconds once crt.sh is under load, so the retry loop needs real time budget, not just more attempts fired quickly at each other.

Third problem: a hung connection defeats retries entirely.

Plain fetch() has no timeout. One test run just hung, no 502, no slow success, nothing, indefinitely. If a request never resolves, it never reaches the point where my retry logic would even kick in. Added an AbortController with a per-attempt timeout so a dead connection gets treated as a failure and retried like any other, instead of silently stalling the whole run.

Fourth problem, and the sneaky one: a field just disappeared from the response shape.

The actor filters results by date and sorts newest-first, originally using crt.sh's entry_timestamp field. At some point crt.sh's JSON output for this particular query stopped including that field. No error, no warning, just undefined. Which meant new Date(undefined) >= startDate silently evaluated false for every single entry, so the actor returned zero results, every time, regardless of what data actually existed. The scariest kind of bug for a monitoring tool: it fails by going quiet, not by throwing. Switched to not_before (when the cert becomes valid, which is essentially CT-log time anyway) as the log-time proxy instead, since that field is always present.

None of these were exotic failures. Bare error pages under load, a budget that was too small, a fetch with no timeout, a field that quietly vanished. But stacked together they're the difference between "works when I test it" and "actually trustworthy as a monitoring tool," which matters a lot more once the whole point of the thing is telling you about certificates you didn't expect.

If you want to poke at the actual retry code: github.com/timmKal01/certificate-transparency-monitor. The actor itself is live on Apify if you want to point it at a domain.

"Pull a company's open roles straight from their own job board API"

Timothy Kelvin — Fri, 18 Sep 2026 10:05:15 +0000

Most "company hiring signal" tools scrape LinkedIn. company-hiring-tracker does something simpler: most companies that use Greenhouse or Lever for hiring expose their open roles through a public, unauthenticated API on their own job board. No login, no proxy, no headless browser, just a fetch.

Give it a company's ATS (Greenhouse or Lever) and board slug, and it returns each open role: title, location, department, and posting date, as clean JSON.

{ "title": "Senior Backend Engineer", "location": "Remote", "department": "Engineering", "postedAt": "2026-09-10" }

Priced per board queried, not per job returned, so checking a company with 40 open roles costs the same as checking one with 2.

https://apify.com/m_ctim/company-hiring-tracker

"A Raw JSON Textarea Was Quietly Costing Me Users"

Timothy Kelvin — Fri, 18 Sep 2026 09:57:34 +0000

I almost missed this one because the actor itself worked fine. The bug wasn't in the code, it was in the input form.

Looking at usage funnels across a portfolio of small data-fetching actors (Apify calls them Actors — serverless scrapers/API tools), one number stood out: for company-buying-signal-report, 5 people opened the input page, and only 1 actually ran it. Everywhere else in the funnel, the drop-off was gradual. Here it was a cliff.

The only clue was the input schema itself. The actor's one real input field was companies, an array of objects ({ domain, greenhouseSlug, leverSlug }), rendered by Apify's console as a raw JSON editor. Technically correct, since most people running this actor only care about one company at a time, and a blank JSON textarea is not exactly a welcoming way to ask for that.

The fix was small: add flat, single-item fields (domain, greenhouseSlug, leverSlug) alongside the existing array field. In the actor's code, the single fields take priority when filled in, and fall back to the array for anyone who does want to check multiple companies in one run.

const companies = singleDomain
  ? [{ domain: singleDomain, greenhouseSlug, leverSlug }]
  : (companiesInput?.length ? companiesInput : defaultCompanies);

Once I saw it in one actor, I grepped the rest of the portfolio for the same pattern ("editor": "json" on an array-of-objects field with no matching single-item shortcut) and found four more actors with the exact same friction: a hiring tracker, a weather tracker, a risk-briefing tool, and a city council monitor. Same fix, same result, each verified live afterward.

None of this shows up in a code review. It only shows up when you look at where real people stop, not just whether the backend logic is correct.

There are three different things people mean by "check my SSL cert"

Timothy Kelvin — Tue, 15 Sep 2026 04:18:35 +0000

A cert expiring in production is one of those outages that feels avoidable in hindsight. Nobody meant to skip it. It just wasn't anyone's job to check.

The confusing part

"Check my SSL cert" can mean three different things, and they check three different systems.

Is the domain registration itself about to expire? That's a WHOIS/RDAP question, nothing to do with TLS.
Has a new certificate been issued for this domain recently, maybe one you didn't request? That's a certificate transparency log question, a public append-only record of every cert any browser-trusted CA has ever issued.
What certificate is the server actually presenting right now, and how many days until it expires? That's a live TLS handshake question, and it's the one that actually causes an outage when the answer is "zero."

I'd already built tools for the first two. This one covers the third, and it's the one people usually mean when a pager goes off.

How it works

Opens a direct TLS connection to each host, the same handshake a browser performs, and reads the certificate the server presents. Node's built-in tls module does the heavy lifting:

const socket = tls.connect(
  { host, port, servername: host, timeout: 8000, rejectUnauthorized: false },
  () => {
    const cert = socket.getPeerCertificate();
    const authorized = socket.authorized;
    // ...
  },
);

rejectUnauthorized: false looks backwards at first. The point isn't to skip trust checking, it's to keep the connection open long enough to actually read the certificate even when it's self-signed or the chain is broken. socket.authorized and socket.authorizationError still report the trust result separately, so a broken cert shows up as data ("isSelfSigned: true", a trust error message) instead of a connection failure with no detail. Silently failing to connect isn't the same as telling someone their cert is bad.

What it returns

One record per host: issuer, subject, valid dates, days until expiry, expired/self-signed flags, and the trust result. A host that can't be reached at all still returns a record, just with reachable: false and an error message, since "we tried to check and something's wrong" is itself a useful result for a host that should have been reachable.

Pricing shape

Billed per host checked, not per field returned. One charge whether the host is fine, expired, or completely unreachable.

Try it

Source: github.com/timmKal01/ssl-certificate-expiry-checker
Hosted version (Apify actor, no setup): apify.com/m_ctim/ssl-certificate-expiry-checker

If you're already checking domain expiry or watching CT logs, this is the third leg of that stool, the one that actually matches what's live on the wire right now.

Three small API-backed actors I've shipped recently (Docker Hub, lead extraction, SEC Form 4)

Timothy Kelvin — Fri, 11 Sep 2026 19:24:35 +0000

I build small, narrow Apify actors — each one wraps a single official API or data source and does one job well, rather than trying to be a general-purpose scraper. Here are three recent ones, each solving a different, specific problem.

Docker Hub Image Tracker

Docker Hub will repush a tag with a new digest, new architectures, or a different underlying image, and the version string stays the same. This actor hits hub.docker.com/v2/repositories directly and tells you what actually changed for a given repository within a time window — digest, size, supported architectures, last-pushed/last-pulled timestamps.

{ "repository": "postgres", "daysBack": 7, "maxResults": 50 }

Real gotcha from building it: the API's ordering param is backwards from the usual REST convention — ordering=last_updated sorts newest-first, and you need a - prefix to get oldest-first.

apify.com/m_ctim/docker-hub-image-tracker · source

Website Lead Extractor

Built on Crawlee's plain CheerioCrawler instead of a headless browser. Point it at a domain, it crawls within that domain up to a set depth, and pulls every email, phone number, and social profile link (LinkedIn, X, Facebook, Instagram, GitHub) it finds in the raw HTML — no Chromium, no render wait.

{ "startUrls": [{ "url": "https://example.com" }], "maxDepth": 1, "maxPagesPerDomain": 20 }

Tradeoff worth stating plainly: it only sees what's in the initial HTML response, so a client-side-only React SPA with no SSR needs a browser-based crawler instead. For the common case of a static or server-rendered marketing site, it's faster and has nothing to configure around headless-browser flakiness.

apify.com/m_ctim/website-lead-extractor · source

Insider Trading Alert

Every SEC Form 4 filing (executive/director/10%-owner stock buys and sells) is public, structured XML, filed within a couple of business days of the transaction. This actor reads EDGAR's filing feed and each filing's own XML directly — search one ticker or scan the market-wide feed of the newest filings.

{ "ticker": "AAPL", "transactionType": "disposed", "minTransactionValue": 100000 }

Real bug from building it: the boolean "is this person an officer/director" flags get encoded as 1/0 by some filers' software and true/absent by others, on the same form type. A naive === '1' check silently returned false for real officers depending on which software generated the filing.

apify.com/m_ctim/insider-trading-alert · source

All three are pay-per-event on Apify (no subscription), no proxy, and read directly from each source's own official API rather than scraping rendered pages.

Turning a list of company websites into a contact list with a plain HTTP crawl

Timothy Kelvin — Thu, 10 Sep 2026 09:20:16 +0000

Anyone who has done outbound sales or recruiting knows this task. You have a list of company websites and you need the contact info off each one. Open the Contact page, copy the email, check LinkedIn, move to the next site. A 50 domain list eats most of an afternoon done by hand.

The approach

Give it a start URL and it crawls within that same domain, up to a depth you choose, using Crawlee's CheerioCrawler. On each page it checks for:

Email addresses in the raw HTML
Phone numbers
Social profile links (LinkedIn, X, Facebook, Instagram, GitHub)

Everything found gets returned as one record per page, with the URL, domain, and whatever contact info was on it.

{
  "url": "https://example.com/contact",
  "domain": "example.com",
  "emails": ["hello@example.com"],
  "phones": ["+1 415-555-0132"],
  "socialProfiles": {
    "linkedin": "https://linkedin.com/company/example",
    "twitter": "https://x.com/example"
  }
}

What it deliberately doesn't do

No headless browser. It only reads what the site already renders in the raw HTML response to a plain request, the same thing any visitor's browser would see before JavaScript runs. That keeps it fast and cheap to run, and it means it only extracts information the site is already publishing publicly. Nothing behind a login, nothing a normal visitor couldn't already find on a Contact or About page.

Where it falls short, on purpose

Sites that render their content client side with JavaScript are a real weak spot here. A plain HTTP crawl never sees content that only shows up after JS runs, so a heavily client rendered site might come back with far less than it actually has. That's the honest tradeoff for staying fast, cheap, and proxy free. A browser based crawler would handle those sites but costs more to run and adds real complexity. For static or server rendered company sites, which is most small and mid size business sites, the plain HTTP approach works well.

Pricing shape

Billed per page where contact info was actually found. A domain with nothing public anywhere costs nothing to check, and a domain where three pages all list contact info costs about the same as pulling it from all three yourself, just a lot faster.

Try it

Source: github.com/timmKal01/website-lead-extractor
Hosted version (Apify actor, no setup): apify.com/m_ctim/website-lead-extractor

If you run it against a list of your own domains, curious to hear how the hit rate looks on real sites.