DEV Community: NeverEmpty

Your scraper's defaultCountry setting invents phone numbers

NeverEmpty — Fri, 28 Aug 2026 14:09:49 +0000

Most contact scrapers have a setting called something like defaultCountry or region. It looks harmless. It is the fastest way to fill a database with phone numbers that are structurally valid, pass every check you run, and belong to nobody.

Here is the measurement that convinced me to delete mine.

One company, 51 countries

I crawled the contact pages of ibm.com on 2026-08-28 — up to five pages per site, three were reachable, robots.txt honoured. What came back:


Email addresses	37
Phone numbers resolved to E.164	115
Distinct countries in those 115	51
Phone-shaped strings that could not be resolved to E.164	52

The 51: AT AU BE BG CH CN CY CZ DE DK EE EG FI FR GB GP GR HK HR HU IE IL KR LK LT LU LV MO MY NC NG NL NO PF PH PK PL PT RO RS SE SG SI SK TH TR TW UA US VN ZA.

That is one company's own contact pages. Not a directory, not an aggregator. The same crawl over suse.com gave 11 numbers across 8 countries, ionos.com 10 across 7.

So the premise behind a default country — "this is a US site, so bare numbers on it are US numbers" — is wrong before you write a line of parsing code. Corporate contact pages are international by nature. That is what they are for.

What a default country actually does

Take the strings that had no country code of their own and parse them as US:

import { parsePhoneNumberFromString } from 'libphonenumber-js';

let valid = 0;
for (const raw of unresolvedStrings) {          // 52 of them
  const n = parsePhoneNumberFromString(raw, 'US');
  if (n?.isValid()) valid++;
}
// valid === 18

18 of the 52 come back valid. Not "probably valid" — isValid() returns true, .number gives a clean E.164 string, and every downstream check passes.

Most of those 18 really are US numbers, so far so good. The problem is the ones that are not. From the suse.com crawl, a Budapest number printed on the page as a local string:

parsePhoneNumberFromString('361-489-4600', 'US').isValid()   // true
parsePhoneNumberFromString('361-489-4600', 'US').number      // '+13614894600'

+36 1 489 4600 is Budapest. +1 361 489 4600 is a real, dialable US area code. The default country did not fail loudly — it produced a different, perfectly valid number in a different country, and handed it to you as a fact.

A validator cannot save you from this. It validates the number you constructed, not the assumption you made. It is not lying: +13614894600 genuinely is a valid US number. It just is not the number that was printed on the page.

Sometimes the validator does catch it — 0180 0132 00049 from the same site parses as US and comes back invalid, so that one gets thrown away. But you cannot rely on being saved by luck. The failure mode you never see is the one that stays.

The shape that does not lie

Two fields, one rule:

phones        E.164, only for numbers that carried their own country code
phonesLocal   the raw string, essentially as printed, when one could not be resolved

The rule: never synthesise a country code. If the page did not say which country, you do not know which country.

That sounds like losing data. It is not — it is moving it. The 52 unresolved IBM strings are still in the output, still usable by code that has context you do not. A CRM that already knows the account is in Malaysia can resolve 1800-88-8558 correctly; your crawler cannot.

What you lose is the illusion of 52 extra E.164 numbers. That illusion is expensive, because nobody discovers it until someone dials.

While you are there: emails are obfuscated on purpose

Two patterns worth handling, both cheap:

name [at] example [dot] com and its bracketed variants. A plain /\S+@\S+/ extracts nothing usable from those — [at], (at) and {at} forms produce no match at all.
Cloudflare email protection. The address is replaced by <a href="/cdn-cgi/l/email-protection" data-cfemail="a1c4d9c0ccd1cdc4e1...">. It is an XOR encoding: the first byte is the key, each following byte is a character XORed with it.

function decodeCfEmail(hex) {
  const key = parseInt(hex.slice(0, 2), 16);
  let out = '';
  for (let i = 2; i < hex.length; i += 2) {
    out += String.fromCharCode(parseInt(hex.slice(i, i + 2), 16) ^ key);
  }
  return out;
}

This is not a security control — Cloudflare ships email-decode.min.js to every browser to undo it client-side, and their docs note it is enabled automatically on sign-up. It is there to stop naive harvesters, and it works: six lines is the entire difference between finding a site's contact address and reporting that it has none.

And when the fetch fails

The fourth site in my run was chiyodacorp.com. It returned:

{ "domain": "chiyodacorp.com", "ok": false, "reason": "fetch failed",
  "emails": [], "phones": [], "phonesLocal": [] }

Not an absent row. Not an empty row that reads like "this company publishes no contact details". A row that says which site failed and why, so the next run can retry that one and only that one.

A crawl that quietly drops what it could not reach reports a smaller, cleaner, wronger world every time.

The short version

Delete the default-country setting. It is not a convenience, it is a fabricator.
Keep E.164 and un-resolvable strings in separate fields, and never invent the difference.
Handle bracketed obfuscation and Cloudflare's XOR, or accept that you are silently missing contacts.
Emit a row with a reason for every site you failed on.

None of this is hard. All of it is skipped by default, which is why so many contact datasets are confidently wrong.

Written with AI assistance. Every count, country list and parse result above came from a live crawl executed on 2026-08-28 before publishing.

Greenhouse, Lever, Ashby, Workable: 4 job APIs, 1 dangerous bug

NeverEmpty — Fri, 28 Aug 2026 13:35:17 +0000

Most companies running Greenhouse, Lever, Ashby or Workable publish their open roles on a public JSON endpoint. No API key, no OAuth, no browser.

Here they are, with the counts I measured on 2026-08-28. Every number below came back from a live request while writing this — and job counts move daily, so yours will differ.

The four endpoints

# Greenhouse
curl -s "https://boards-api.greenhouse.io/v1/boards/stripe/jobs"
# → 200, jobs[] with 578 entries

curl -s "https://boards-api.greenhouse.io/v1/boards/monzo/jobs"
# → 200, jobs[] with 66 entries

# Lever
curl -s "https://api.lever.co/v0/postings/matchgroup?mode=json"
# → 200, a bare array with 71 entries

# Ashby
curl -s "https://api.ashbyhq.com/posting-api/job-board/linear"
# → 200, jobs[] with 29 entries

# Workable
curl -s "https://apply.workable.com/api/v1/widget/accounts/lyst?details=true"
# → 200, jobs[] with 8 entries

Three of the four are documented: Greenhouse, Lever, Ashby. The Workable one is different — that is the endpoint their embeddable careers widget calls, not the documented Workable API (which needs a Bearer token). It works today and it is public, but treat it as something that can change without a changelog.

Two container shapes, four vocabularies. Greenhouse, Ashby and Workable all wrap the list in jobs; Lever returns a bare array. Inside, nothing lines up — the job title is title in three of them and text in Lever. Greenhouse gives you absolute_url and internal_job_id, Ashby gives employmentType and secondaryLocations, Lever gives categories and hostedUrl.

The part that silently breaks

Here is what a naive integration does:

const res = await fetch(url);
const data = await res.json().catch(() => ({}));
return data.jobs ?? [];   // ← this is the bug

Run that against a company that does not use Greenhouse and you get []. Run it against a company that has zero open roles right now and you also get []. Run it while the endpoint is throwing a 502 and you get [] again. Point it at a Lever board and you get [] every single time, forever, because Lever's response has no jobs key at all.

Four completely different situations, one identical output. Downstream, a week later, someone asks why a client "stopped hiring" — and the answer is that nobody was ever hiring there on that ATS.

I measured this while checking companies for this post:

Request	Result
`boards-api.greenhouse.io/v1/boards/stripe/jobs`	200, 578 jobs
`boards-api.greenhouse.io/v1/boards/ramp/jobs`	404 `{"status":404,"error":"Job not found"}`
`api.lever.co/v0/postings/netlify?mode=json`	404 `{"ok":false,"error":"Document not found"}`
`api.lever.co/v0/postings/plaid?mode=json`	404

Those 404s do not mean those companies are not hiring. They mean the guess about which ATS they use, or the guess about their board slug, was wrong.

So the minimum honest shape is four outcomes, not one:

let res;
try {
  res = await fetch(url, { signal: AbortSignal.timeout(20000) });
} catch (e) {
  return { state: 'fetch-failed', reason: e.message };
}

if (res.status === 404) return { state: 'no-board-on-this-ats' };
if (!res.ok)            return { state: 'fetch-failed', status: res.status };

let list;
try {
  list = pickList(await res.json());        // vendor-specific: jobs[] or bare array
} catch (e) {
  return { state: 'not-json', reason: e.message };   // 200 + an HTML error page
}

return list.length
  ? { state: 'ok', jobs: list }
  : { state: 'board-exists-but-zero-open-roles' };

state costs you one string per row and removes an entire class of "the data looked fine" incidents.

The harder problem: which board belongs to whom

Guessing a slug from a domain name works often enough to be dangerous. Here is a real collision, both measured today:

curl -s "https://api.ashbyhq.com/posting-api/job-board/notion"
# → 200, 134 jobs, first one: "Software Engineer, Developer Platform"

curl -s "https://apply.workable.com/api/v1/widget/accounts/notion?details=true"
# → 200, "name": "Notion", 0 jobs
#   description: "We're a luxury lifestyle agency based in Shoreditch..."

Same slug, two entirely unrelated organisations, both returning HTTP 200. If your resolver tries Workable first and stops at the first 200, you have just labelled a London lifestyle agency as the productivity company.

It gets sillier. apply.workable.com/api/v1/widget/accounts/a — a single letter — returns 200 with "name": "a". A slug existing proves nothing about who owns it. (For contrast, zzqqxxnotarealcompany returns 404 on both Workable and Greenhouse, so the 200s above are real registrations, not a catch-all.)

The only thing that settles ownership is the company's own site. If notion.so links to that board, it is theirs. If it does not, what you have is a lead, not a fact — and it should be labelled that way rather than merged into the same column as verified results.

There is a second trap in the other direction. Investor portfolio pages, accelerator sites and job aggregators link to boards that belong to other companies. Crawl an investor's site, follow every board link, and you will end up attributing their portfolio companies' job boards to the investor. Ask "does this page list other companies' jobs?" before you attribute anything.

Normalising the shapes into one

A usable row needs at minimum:

source        greenhouse | lever | ashby | workable
company       the domain you started from
boardSlug     the slug that actually worked
jobId         vendor id, kept as a string
title         Lever calls this `text`
location      null when the vendor does not give one — not ""
remote        true | false | null   (null when unknown, not false)
department
url           the public apply URL
postedAt      ISO 8601, not the vendor's raw format

The null rule matters more than it looks. If "unknown" and "false" collapse into the same value, every downstream filter for on-site roles quietly includes the ones you simply had no data for.

The rule, if you build your own

Whatever you wrap these endpoints in, keep the three outcomes separate:

the board does not exist on this ATS (404)
the board exists and has zero open roles right now
the request failed (timeout, 5xx, HTML where JSON was expected)

Collapsing them into one empty array is the bug that costs you a week later, not today. The endpoints at the top of this post are the whole integration otherwise — no key, no proxy, no browser.

Written with AI assistance. Every endpoint, count, status code and error body in this post was executed live before publishing, on 2026-08-28.