DEV Community: SwiftKit

Company name website, logo and stock ticker for free: Wikidata is the enrichment API nobody mentions

SwiftKit — Wed, 07 Oct 2026 04:03:56 +0000

"I have 2,000 company names in a spreadsheet and need their websites" is a surprisingly common request. Paid enrichment APIs exist, but for well-known and mid-size companies there's a free, openly licensed source most people skip: Wikidata, the structured database behind Wikipedia. It's CC0 (public domain), has an official API, and asks only that bots identify themselves and don't hammer it.

Step 1: search by name

const UA = 'my-enrichment-script/1.0 (contact@example.com)';
const wd = async (params) =>
  (await fetch(`https://www.wikidata.org/w/api.php?${new URLSearchParams({ format: 'json', maxlag: '5', ...params })}`,
    { headers: { 'User-Agent': UA } })).json();

const { search } = await wd({ action: 'wbsearchentities', search: 'Stripe', language: 'en', type: 'item', limit: '7' });
// Q7624104 Stripe — Irish-American payment technology company
// Q3421342 stripe — long, narrow band of color…

Step 2: read the facts

Every result is an item with "claims" (property → values). The useful ones for companies:

Property	Meaning
P856	official website
P154	logo image (a file on Wikimedia Commons, often SVG)
P17 / P159	country / headquarters
P452	industry
P571	founded
P1128	employees
P414 + qualifier P249	stock exchange + ticker
P4264, P2002, P2037	LinkedIn company, X username, GitHub org

const { entities } = await wd({ action: 'wbgetentities', ids: search.map((s) => s.id).join('|'), props: 'claims|labels|descriptions' });
const best = (claims, p) => {
  const list = (claims[p] || []).filter((c) => c.rank !== 'deprecated' && c.mainsnak.datavalue);
  return list.find((c) => c.rank === 'preferred') || list.at(-1); // "preferred" = current value
};
const website = best(entities.Q7624104.claims, 'P856')?.mainsnak.datavalue.value; // https://stripe.com/

A logo filename turns into a direct image URL with https://commons.wikimedia.org/wiki/Special:FilePath/<file name>.

Values that point to other items (country, industry) come back as IDs like Q30. Fetch their labels in one batched wbgetentities call (up to 50 IDs). One gotcha: some newer items only have a mul (multilingual) label, not an English one, so ask for languages: 'en|mul' or you'll print raw Q140991293s.

Step 3: pick the right "Apple"

Searching a name returns fruit, record labels and albums too. What worked for me:

Drop anything whose type (P31) is a human, a disambiguation page or a creative work (album, film, song, book).
Require an official website (P856). You wanted the domain anyway, and it filters out most noise.
Prefer an exact label match, then the first remaining result. Wikidata's search ranking is already good.

That picked the right entity for Stripe, Toyota, Siemens, Canva, Mercari, Patagonia and Death Wish Coffee in my tests, and correctly returned nothing for "Taylor Swift".

Step 4: going the other way (domain → company)

Wikidata's search index can match the website statement directly:

await wd({ action: 'query', list: 'search', srsearch: 'haswbstatement:P856=https://www.allbirds.com/' });

It's an exact match, so try the https://, https://www. and trailing-slash variants.

Step 5: fill the gaps from the homepage

Smaller companies often aren't in Wikidata, and logos on Commons are sometimes missing. One polite visit to the homepage gets you the rest: <link rel="apple-touch-icon"> (a clean square icon), the favicon (prefer SVG), og:image, an <img> whose src/alt/class mentions "logo", JSON-LD Organization.logo and sameAs (official social links).

Limits

Coverage is great for listed companies and well-known brands, patchy for local small businesses.
Employee counts are community-maintained, usually from annual reports, and can be a year or two old.
Logos are trademarks: fine for identifying a company, not for pretending to be one.

The packaged version

I wrapped this as Company Domain & Logo Finder on Apify: paste names or domains, get website, logo, favicon, country, industry, founded year, employees, tickers and official social accounts per row. Companies only, never people. 10 companies take about 15 seconds.

Have you used Wikidata in production? I'm curious what else people pull from it.

Written with AI assistance; the code and numbers were tested before publishing.

A small SEO crawler in Node: the 20 checks worth running on every page (and a score that's easy to explain)

SwiftKit — Tue, 06 Oct 2026 02:26:32 +0000

Most technical SEO problems are boring and findable: a template that drops the meta description, a staging noindex that shipped to production, 40 pages with the same title, links to pages that were deleted last year. A crawler finds all of them in minutes.

Here's the one I built, what it checks, and what it found on a real docs site.

Crawl politely

Start at the home page, follow links on the same site (subdomains included), skip files (.pdf, .jpg, .zip…).
Read robots.txt first and skip disallowed paths (longest matching rule wins).
A few requests at a time, an honest User-Agent, and a hard page limit.

Parse with linkedom. It's a fraction of jsdom's weight and plenty for reading tags:

import { parseHTML } from 'linkedom';

const t0 = Date.now();
const res = await fetch(url, { headers: { 'User-Agent': 'my-seo-audit/1.0' }, redirect: 'follow' });
const responseTimeMs = Date.now() - t0;
const html = await res.text();
const { document } = parseHTML(html);

const meta = (k) => document.querySelector(`meta[name="${k}" i], meta[property="${k}" i]`)?.getAttribute('content')?.trim() || null;
const page = {
  title: document.querySelector('title')?.textContent.replace(/\s+/g, ' ').trim() || null,
  description: meta('description'),
  h1: [...document.querySelectorAll('h1')].map((h) => h.textContent.trim()).filter(Boolean),
  canonical: document.querySelector('link[rel="canonical" i]')?.getAttribute('href') || null,
  robots: (meta('robots') || '') + ' ' + (res.headers.get('x-robots-tag') || ''),
  imagesWithoutAlt: [...document.querySelectorAll('img')].filter((i) => !i.hasAttribute('alt')).length,
  viewport: !!meta('viewport'),
  lang: document.documentElement.getAttribute('lang'),
};

Note the X-Robots-Tag header: a noindex there doesn't show up in the HTML at all, and it's a classic leftover from staging.

The checks

Errors (the page is hurt):

missing <title>
noindex in meta robots or the X-Robots-Tag header
links to pages that return 4xx/5xx

Warnings (fix soon):

title over 60 characters or under 15
missing meta description
no <h1>
the same title on several pages
images without alt
under 200 words
server response over 1.5 s
no mobile viewport tag
plain HTTP

Notices (nice to fix):

meta description over 160 or under 50 characters
more than one H1
duplicate meta descriptions
canonical pointing elsewhere, or missing
no lang
no Open Graph tags
HTML over 1 MB
redirects

Broken links without hammering the site

Every page you crawl already gives you a status code. For links you didn't crawl (because of the page limit, or because they point elsewhere), check them once at the end: HEAD first, and GET if the server answers 405/403 to HEAD (many do). Cap the count, and keep external links optional.

A score people understand

I tried weighted formulas. Clients didn't trust them. What works is a checklist score: every page starts at 100, and each error costs 15, each warning 6 and each notice 2. Anyone can recompute it from the issue list, and fixing an issue visibly moves the number. It's a hygiene score, not a guess at Google's ranking, and I say so.

What it found on a real site

On 30 pages of a well-maintained open-source docs site:

27 titles over 60 characters. The template appends a long tagline to every page title.
24 pages sharing one meta description. It's the site-wide default.
10 pages with the same title, all versioned copies of one docs page.
1 page without an H1, 2 thin pages, no broken links.
Average score: 90.

Nothing dramatic, but each is a one-line template fix. That's the typical result: a handful of template issues multiplied across hundreds of pages.

The packaged version

I put this on Apify as SEO Audit Crawler. You get one row per page with the score and issues, plus a free site summary: issues ranked by pages affected, every broken link with the page it's on, duplicate groups, non-indexable pages and the slowest pages. 40 pages take about 15 seconds.

What's the most common SEO bug you keep finding on client sites?

Written with AI assistance; the code and numbers were tested before publishing.

Every Shopify store has a public products.json: how to export a full catalog (and the currency trap)

SwiftKit — Mon, 05 Oct 2026 01:54:22 +0000

If a store runs on Shopify, you can usually get its whole public catalog without scraping a single HTML page. Storefronts expose /products.json: the same product data the store shows to every visitor, as JSON.

curl "https://www.allbirds.com/products.json?limit=250&page=1"

You get up to 250 products per page, each with title, vendor, product type, tags, images, options and every variant (size, color) with price, compare-at price, SKU and an in-stock flag.

Paging through a whole catalog

const sleep = (ms) => new Promise((r) => setTimeout(r, ms));

async function allProducts(store) {
  const out = [];
  for (let page = 1; ; page++) {
    const res = await fetch(`${store}/products.json?limit=250&page=${page}`, {
      headers: { 'User-Agent': 'my-catalog-export/1.0 (contact@example.com)' },
    });
    if (res.status === 429) { await sleep(10000); page--; continue; } // too fast: wait, retry
    if (!res.ok) break;
    const { products } = await res.json();
    out.push(...products);
    if (products.length < 250) break; // last page
    await sleep(800); // one request at a time, with a pause
  }
  return out;
}

Two more endpoints are handy:

/collections/<handle>/products.json: only one collection, e.g. sale or new-arrivals.
/meta.json: the store's name, country, main currency and number of published products.

Gotcha 1: "on sale" lives in compare_at_price

There's no "discount" field. A variant is on sale when compare_at_price is set and higher than price. Both are strings like "87.50", so convert them before comparing ("125.00" > "87.50" is false as strings).

Gotcha 2: prices can change currency based on where you are

This one cost me an hour. I ran the same export from Japan and got a sneaker priced at 14300. From a US server it was 87.50. Same product, same endpoint.

Stores selling internationally (Shopify Markets) localize prices to the visitor's country, and the JSON doesn't say which currency you got. Ways to stay sane:

Run from the country whose prices you want (servers in the US usually get USD).
Read the main currency from /meta.json and sanity-check a known product.
Never compare prices between runs made from different locations.

Gotcha 3: stock is a yes/no flag

available: true/false per variant is all you get. There are no exact quantities in this data, so don't promise "units in stock" in your dashboard.

Is this OK to use?

It's public data the storefront publishes for everyone, and Shopify's default robots.txt allows crawling product and collection pages. Still: go one request at a time, back off on 429, identify yourself, and respect each store's terms. Some stores switch the endpoint off or password-protect the shop; move on when they do.

The code

The full script (CSV export, paging, 429 backoff, sale flags; MIT, no dependencies) is on GitHub: swiftkit-dev/shopify-catalog-export.

The packaged version

I put this on Apify as Shopify Store Products Scraper: paste store or collection links and get products or one row per variant, the store's currency, an "only new products" mode for daily alerts, and a free summary per store (price range and median, share on sale, top vendors, products added in the last 30 days). It handles the paging and 429s for you.

What do you use catalog data for: price tracking, competitor research, or something else?

Written with AI assistance; the code and numbers were tested before publishing.

Give your AI agent a 'read this URL' tool: clean Markdown in ~2 seconds, even for JavaScript apps

SwiftKit — Sun, 04 Oct 2026 00:17:15 +0000

Most AI agents need one boring tool: "read this URL". Raw HTML wastes tokens. A typical article page is 80–95% menus, scripts, cookie banners and share buttons. What the model actually needs is the main text as Markdown, with headings, lists, tables and links.

Here's the pipeline I ended up with, and the one trick that keeps it fast.

1. Download first, browser only if needed

A headless browser handles everything, but it's slow (seconds per page) and heavy. Most pages don't need one: blogs, docs, news and company sites ship their text in the HTML.

So: plain fetch first. Only if the result is nearly empty, open the page in Chrome:

const APP_SHELL = /<div[^>]+id=["'](root|app|__next|__nuxt)["'][^>]*>\s*<\/div>/i;

const html = await (await fetch(url)).text();
let c = convert(html, url);
const thin = c.text.length < 400 && (APP_SHELL.test(html) || c.text.length < 150);
if (thin) {
  const page = await browser.newPage();
  await page.goto(url, { waitUntil: 'domcontentloaded' });
  await page.waitForLoadState('networkidle', { timeout: 10000 }).catch(() => {});
  c = convert(await page.content(), page.url());
}

Two details that matter:

Block images, fonts and video in the browser (context.route). You only want text, and it halves load time.
Check the browser's HTTP status. A 403 error page renders beautifully and converts into confident-looking Markdown about "Request blocked". Treat status 400+ as a failure, not content.

2. Find the main content with Readability

Mozilla Readability is the engine behind Firefox Reader View. It works on any DOM, so on a server you can pair it with linkedom (much lighter than jsdom):

import { parseHTML } from 'linkedom';
import { Readability } from '@mozilla/readability';

const { document } = parseHTML(html);
const article = new Readability(document, { charThreshold: 200 }).parse();
// article.content = cleaned HTML, article.title, article.byline

If Readability returns too little (home pages, product listings), fall back to <main> or <body> minus nav, header, footer, aside and cookie banners.

3. HTML to Markdown with Turndown

import TurndownService from 'turndown';
import { gfm } from 'turndown-plugin-gfm';

const td = new TurndownService({ headingStyle: 'atx', codeBlockStyle: 'fenced' });
td.use(gfm); // tables and strikethrough
td.remove(['script', 'style', 'iframe', 'form', 'svg']);
// Icon-only links (share buttons) carry no text: drop them
td.addRule('emptyLinks', {
  filter: (n) => n.nodeName === 'A' && !n.textContent.trim() && !n.querySelector('img'),
  replacement: () => '',
});
const markdown = td.turndown(article.content);

Make links and image URLs absolute before converting (new URL(href, pageUrl)), or the Markdown breaks the moment it leaves the site.

4. Be a polite bot

Respect robots.txt (longest matching rule wins, like Google).
Send an honest User-Agent.
If the page title is "Just a moment…" or "Access denied", stop. That's a bot check, and getting around it isn't your call to make.

The packaged version

I run this as Website to Markdown for AI on Apify. You can crawl whole sites into RAG chunks, or use it as an instant API: one HTTP request with a URL, Markdown back in about 2 seconds, including JavaScript apps:

curl "https://swiftkit--web-to-markdown.apify.actor/?url=https://docs.apify.com&format=markdown" \
  -H "Authorization: Bearer YOUR_APIFY_TOKEN"

It's a fixed price per converted page, and failed or blocked pages are free.

What does your agent use for reading pages today: raw HTML, a reader API, or something home-made?

Written with AI assistance; the code and numbers were tested before publishing.

Run Lighthouse on hundreds of pages from Node (no PageSpeed API key), and why the scores move

SwiftKit — Sat, 03 Oct 2026 21:02:36 +0000

PageSpeed Insights is great for one URL. For 300 product pages, or "all our landing pages after every release", you want Lighthouse in a script. The good news: Lighthouse is an npm package, and you don't need an API key to run it yourself.

The minimal script

import lighthouse from 'lighthouse';
import * as chromeLauncher from 'chrome-launcher';

const chrome = await chromeLauncher.launch({
  chromeFlags: ['--headless=new', '--no-sandbox', '--disable-dev-shm-usage'],
});
try {
  const { lhr, report } = await lighthouse('https://www.wikipedia.org', {
    port: chrome.port,
    output: ['json', 'html'],
    onlyCategories: ['performance', 'accessibility', 'best-practices', 'seo'],
    logLevel: 'error',
  });
  console.log(Math.round(lhr.categories.performance.score * 100));
  // report[1] is the full HTML report: save it, it's what people actually read.
} finally {
  await chrome.kill();
}

For the desktop preset, pass Lighthouse's own config as the third argument:

const desktop = (await import('lighthouse/core/config/desktop-config.js')).default;
await lighthouse(url, flags, desktop);

Two small gotchas I hit:

In chrome-launcher 1.x, kill() isn't guaranteed to return a promise, so chrome.kill().catch(...) can throw. Wrap it in try/catch instead.
If you run in Docker with Playwright's image, point chrome-launcher at Playwright's browser: chromePath: chromium.executablePath().

What to keep from the result

The lhr object is huge. For a bulk table, this is what I keep per page:

const score = (c) => Math.round(lhr.categories[c].score * 100);
const ms = (id) => Math.round(lhr.audits[id].numericValue);
const row = {
  performance: score('performance'),
  seo: score('seo'),
  lcpMs: ms('largest-contentful-paint'),
  tbtMs: ms('total-blocking-time'),
  cls: lhr.audits['cumulative-layout-shift'].numericValue,
  // Top fixes, ranked by time saved:
  opportunities: Object.values(lhr.audits)
    .filter((a) => a.details?.type === 'opportunity' && a.details.overallSavingsMs >= 100)
    .sort((a, b) => b.details.overallSavingsMs - a.details.overallSavingsMs)
    .slice(0, 8)
    .map((a) => ({ title: a.title, savingsMs: Math.round(a.details.overallSavingsMs) })),
};

Also check lhr.runtimeError before reading anything. A page that failed to load still returns an lhr, just with garbage scores.

Run audits one at a time

It's tempting to audit 10 pages in parallel. Don't: Lighthouse simulates a slow device and measures CPU work, so parallel audits on one machine slow each other down and drag every performance score down. Run them one after another, each with a fresh Chrome. It's slower, but the numbers mean something.

Why the scores move

I ran the same pages on my laptop and on a 2 GB cloud container:

Page	Laptop (mobile)	Cloud (mobile)
wikipedia.org performance	98	92
wikipedia.org LCP	1,379 ms	1,497 ms

A heavier marketing site scored 25 for performance in the cloud, with an LCP over 9 seconds. Same code, same day.

Lab scores depend on the machine's CPU, and the mobile preset multiplies that with CPU throttling. So:

Compare pages within the same run, on the same machine, not across tools.
PageSpeed Insights also shows real-user data (CrUX) when Google has enough traffic. That's a different measurement, and often the one that matters for SEO.
Track trends (same pages, same setup, weekly) rather than single numbers.

The packaged version

If you'd rather not run Chrome fleets yourself, I put this on Apify as Lighthouse Audit: paste URLs, pick mobile, desktop or both, and get scores, Core Web Vitals, top fixes, failed SEO/accessibility checks and the full HTML report per page. It runs one audit at a time for the reasons above. Failed pages are free.

How do you track performance across many pages today? I'm curious whether people use CrUX data, lab runs, or both.

Written with AI assistance; the code and numbers were tested before publishing.

A live job search over 10,000 company career sites, built from public links (HN + the Internet Archive)

SwiftKit — Sat, 03 Oct 2026 03:27:20 +0000

Job aggregators are great, but they're copies: scraped, deduplicated, sometimes days old. If you care about a specific slice, like startups and tech companies, you can go to the source instead. Most of these companies post jobs through an applicant tracking system (Greenhouse, Ashby, Lever, Workday…), and those systems have public JSON endpoints that exist so jobs can be shown anywhere.

The hard part isn't fetching jobs. It's knowing which companies exist on which board.

Where do you get a list of company boards?

You can't list "all Greenhouse customers". But companies post their own board links in public places, and one of the best is Hacker News: every month, hundreds of companies reply to "Ask HN: Who is hiring?" with a link to their careers page.

HN has an official search API (by Algolia) that searches comment text:

const r = await (await fetch(
  'https://hn.algolia.com/api/v1/search_by_date?query=ashbyhq.com&tags=comment&hitsPerPage=100'
)).json();

Pull the board slugs out with a regex per system:

const PAT = {
  greenhouse: /(?:boards|job-boards)(?:\.eu)?\.greenhouse\.io\/([a-z0-9_-]+)/gi,
  ashby: /jobs\.ashbyhq\.com\/([a-z0-9_.%-]+)/gi,
  lever: /jobs\.(?:eu\.)?lever\.co\/([a-z0-9_.-]+)/gi,
  workday: /([\w-]+)\.(wd\d+)\.myworkdayjobs\.com\/(?:[a-z]{2}-[A-Z]{2}\/)?([\w-]+)/gi,
};

The API caps a query at 1,000 hits, so walk back in time with numericFilters=created_at_i<=…, using the timestamp of the last hit as the new upper bound.

This produced about 2,100 candidate boards across 10 systems.

Most of them are dead

Companies get acquired, switch ATS, or stop hiring. So check each one against its public API and keep only boards with at least one open job:

System	Candidates	Live with jobs
Greenhouse	911	319
Ashby	399	270
Workday	234	118
Workable	242	47
Lever	96	44
Personio	53	21
Recruitee	87	21
SmartRecruiters	52	16
Rippling	26	13
Teamtailor	32	11

880 live boards. A good start, but HN only sees companies that post there.

Scaling up with the Internet Archive

The Wayback Machine has a public CDX index of every URL it has archived, and job boards get archived a lot. You can list archived URLs under a host prefix:

https://web.archive.org/cdx/search/cdx?url=jobs.ashbyhq.com/&matchType=prefix&fl=original&collapse=urlkey&from=2025&limit=50000&showResumeKey=true

Take the first path segment as the slug, page through with resumeKey, and pause a few seconds between requests. It's a shared public service, so be gentle. That gave about 33,000 candidates. After checking each one against its board's API:

System	Live boards
Greenhouse	5,490
Ashby	3,642
Lever	1,453
Workday	118
Others	~170

About 10,800 live company boards, with 313,000 open jobs. One lesson: Workable started answering 429 halfway through the checks, so I stopped and left it alone. Rate limits are a request, not a challenge.

One snapshot a day, not 10,000 requests per search

With 880 boards you can check every board on every search. With 10,800 you shouldn't: it's slow, and it puts needless load on everyone's job boards. So a small job reads every board once a day and writes one gzipped, line-delimited snapshot (one JSON array per job, about 14 MB for 313k jobs, built in under 5 minutes). Searches stream that file line by line, so a 512 MB container handles it with under 100 MB of memory. My first version parsed the whole file at once, and the container quietly ran out of memory.

Workday is different: big companies have thousands of postings, paged 20 at a time. Downloading 43,000 jobs to find 140 data engineers would be wasteful. Workday's endpoint accepts a searchText, so let it search server-side:

await fetch(`https://${tenant}.${dc}.myworkdayjobs.com/wday/cxs/${tenant}/${site}/jobs`, {
  method: 'POST',
  headers: { 'Content-Type': 'application/json' },
  body: JSON.stringify({ appliedFacets: {}, limit: 20, offset: 0, searchText: 'data engineer' }),
});

Then still filter titles locally, because Workday's search also matches descriptions.

A search for "product designer" posted in the last 30 days returns 444 jobs across 10,800 companies in about a minute, most of which is spent on the live Workday searches.

Things that bit me

Dates differ by system. Greenhouse gives first_published, Workday says "Posted 3 Days Ago", and some boards only show the last update. Normalize them, and document which is which.
"Remote" is inconsistent. Lever and Ashby have structured workplace types; others only have free-text locations. Treat remote as "marked remote", not "definitely not remote".
Be polite. One request per board per run, modest concurrency, a User-Agent that says who you are.

Try it

The packaged version is on Apify as Job Search API: title keywords and exclusions, location, remote-only, posted-within-days, and an "only new jobs" mode for daily alerts. You can add your own company boards too. The snapshot refreshes daily, and Workday plus any boards you add are searched live.

What would you add to the list? If your company's board is missing, drop the link in the comments.

Written with AI assistance; the code and numbers were tested before publishing.

Validate email lists without sending a single email (DNS, typos, disposables, duplicates)

SwiftKit — Fri, 02 Oct 2026 05:26:13 +0000

Most "email verification" tutorials end with the same trick: open an SMTP connection to the recipient's mail server and ask whether the mailbox exists. It works less and less. Gmail, Outlook and most corporate servers accept everything (catch-all) or answer vaguely, and probing from a cloud IP is a quick way to get that IP blocklisted.

The good news: you can catch most of the junk in a list without contacting any mail server at all. Here's the checklist I ended up with.

1. Syntax, but realistic

The full RFC 5322 grammar allows things no real provider accepts (quoted local parts, comments, IP literals). For list cleaning, a practical rule is better: one @, a local part of normal characters, and a domain with a real TLD. Lowercase the domain, trim whitespace, and strip mailto: that people paste in.

2. Does the domain accept mail?

import { Resolver } from 'node:dns/promises';

const dns = new Resolver();
dns.setServers(['1.1.1.1', '8.8.8.8']);

async function acceptsMail(domain) {
  try {
    const mx = await dns.resolveMx(domain);
    // A "null MX" (RFC 7505) means: this domain never receives email.
    if (mx.length === 1 && mx[0].exchange === '') return false;
    return mx.length > 0;
  } catch (e) {
    if (e.code === 'ENODATA') {
      // No MX: mail falls back to the A record.
      try { return (await dns.resolve4(domain)).length > 0; } catch { return false; }
    }
    if (e.code === 'ENOTFOUND') return false; // domain doesn't exist
    throw e; // timeout etc.: report "unknown", don't guess
  }
}

Two lessons:

Treat timeouts as unknown, not invalid. Marking a real customer invalid because a DNS server hiccuped is worse than leaving one bad address in.
Cache per domain. A 50k list usually has only a few thousand distinct domains.

3. Typos of big providers

gmial.com, hotmal.com, gmail.con. These often do resolve (someone registered them), so DNS alone won't catch them. Compare the domain with a short list of popular providers using edit distance:

function editDistance(a, b) {
  const d = Array.from({ length: a.length + 1 }, (_, i) => [i]);
  for (let j = 1; j <= b.length; j++) d[0][j] = j;
  for (let i = 1; i <= a.length; i++)
    for (let j = 1; j <= b.length; j++)
      d[i][j] = Math.min(d[i - 1][j] + 1, d[i][j - 1] + 1,
        d[i - 1][j - 1] + (a[i - 1] === b[j - 1] ? 0 : 1));
  return d[a.length][b.length];
}
// distance 1–2 from gmail.com, outlook.com, yahoo.com… → suggest the fix

Only suggest when the domain is not itself a known provider, or you'll "correct" gmx.com to gmail.com.

4. Disposable and role addresses

Disposable: the community-maintained disposable-email-domains list (public domain, about 9,000 entries) catches most throwaway services. Refresh it regularly.
Role accounts: info@, sales@, support@, noreply@. They're real, but they're shared inboxes, and many email tools treat them as a risk. Flag them instead of deleting them.

5. Duplicates that don't look like duplicates

For Gmail, dots and +tags don't matter, so these are all the same inbox:

john.smith+news@gmail.com
johnsmith@googlemail.com
JohnSmith@gmail.com

Build a canonical form (lowercase, remove dots and the +tag for Gmail, map googlemail.com to gmail.com) and dedupe on that. Only remove +tags for providers where you know they're aliases.

6. Cheap extra signals

Domain age from the registry's RDAP service: a domain registered last week on a B2B sign-up form deserves a second look.
Parked domains: name servers like sedoparking.com mean "this domain is for sale", not a real company.
SPF / DMARC presence: not proof of anything alone, but real sending domains usually have them.

What this can't tell you

Honestly: none of this proves that a specific mailbox exists. "Valid" here means "well formed, the domain is real and accepts mail, and nothing looks off." For list hygiene before an import, that's usually the part that matters, and it's fast, cheap and doesn't touch your sender reputation.

If you'd rather not build it

I packaged this checklist as an Apify Actor: Bulk Email Validator. Paste emails or a CSV link, and you get valid / risky / invalid with the reason, a typo suggestion, disposable/role/free flags, mail provider, domain age, parked and SPF/DMARC flags, and duplicate detection. It never sends a test email. Pay-per-email, about $0.50 per 1,000.

What's the weirdest thing you've found in an email list?

Written with AI assistance; the snippets and numbers were tested before publishing.

How to detect any website's tech stack in bulk (and why a real browser finds 2x more)

SwiftKit — Fri, 02 Oct 2026 03:14:59 +0000

"What is this site built with?" sounds like a one-off question, until you need the answer for 5,000 domains. Sales teams want every Shopify store in a list. Agencies want to audit prospects. Analysts want to count which CMS wins in a niche.

Here's how technology detection actually works, and what I learned building an open-source detector.

Fingerprints: the core idea

Every tool leaves traces. The open-source webappanalyzer project (a community continuation of the old Wappalyzer data) catalogs 7,600+ technologies with patterns like:

"WordPress": {
  "meta": { "generator": "^WordPress ?([\\d.]+)?\\;version:\\1" },
  "html": "<link rel=[\"']stylesheet[\"'] [^>]+/wp-(?:content|includes)/",
  "implies": ["PHP", "MySQL"]
}

A detector loads the page and tests each pattern against:

Signal	Example
HTML	`/wp-content/` paths
Script URLs	`cdn.shopify.com/s/…`
Meta tags	`<meta name="generator" content="WordPress 6.8">`
Response headers	`server: cloudflare`, `x-powered-by: Next.js`
Cookies	`_shopify_y`, `hubspotutk`
DNS	MX records → Google Workspace or Microsoft 365, SPF includes
SSL issuer	Let's Encrypt, DigiCert…

Then it resolves implies (WordPress → PHP + MySQL), requires and excludes, and keeps only matches above a confidence threshold.

The catch: half the fingerprints need JavaScript

When I counted, 3,342 of 7,628 technologies have js patterns: things like window.ABTasty, window.Intercom or AFRAME.version. These only exist after the page's JavaScript runs. A plain HTTP request never sees them, and neither do tools that load later through tag managers.

So I ran the same sites both ways:

Site	HTML + headers + DNS	Headless Chrome
notion.so	18	38
allbirds.com	16	29
hubspot.com	33	56

Roughly 2× more detections with a real browser: analytics, chat widgets, A/B testing, consent tools and front-end frameworks.

Tips if you build this yourself:

Block images, fonts and media in the browser. You don't need them, and it makes each page much cheaper.
Evaluate all JS chains in one page.evaluate call, wrapped in try/catch (some getters throw).
Also feed the browser's network requests into the script-URL and XHR patterns. Many tools load from tag managers you won't see in the initial HTML.
Respect robots.txt, and expect some big sites to answer 403 to automation.

Open source + a hosted version

The detector is open source (GPL-3.0, like the fingerprint data):
github.com/swiftkit-dev/website-tech-stack-detector

If you'd rather not run browsers yourself, the same code runs on Apify as
Website Tech Stack Detector: paste a list of domains,
choose fast or deep scan, and get JSON/CSV with technologies grouped by category, versions and the
email provider. It's pay-per-site, and unreachable sites are free.

What would you use bulk tech detection for? I'm curious which categories people care about most.

Written with AI assistance; the endpoints, numbers and examples were tested before publishing.

How to get every open job from Greenhouse, Lever, Ashby and Workday with one API call

SwiftKit — Fri, 02 Oct 2026 03:14:42 +0000

If you've ever built a niche job board, a hiring-signal feed for sales, or just wanted to track who's hiring data engineers, you've probably hit the same wall: every company's careers page looks different.

The good news is that under the hood, most of them run on a small number of applicant tracking systems (ATS), and many of those publish open jobs through public, unauthenticated endpoints. They exist precisely so companies can show their openings anywhere. You don't need to scrape HTML.

Here's how the big ones work.

Greenhouse

curl "https://boards-api.greenhouse.io/v1/boards/airbnb/jobs?content=true"

The airbnb part is the company's board token, the same slug you see in boards.greenhouse.io/airbnb. You get title, location, departments, first-published date and (with content=true) the full description as HTML-escaped text.

Lever

curl "https://api.lever.co/v0/postings/palantir?mode=json"

Lever gives you categories.commitment (full-time, contract…), workplaceType (remote/hybrid/onsite), and a salaryRange object when the company publishes pay.

Ashby

curl "https://api.ashbyhq.com/posting-api/job-board/openai?includeCompensation=true"

Ashby is the friendliest for salary data: includeCompensation=true returns structured min/max, currency and interval. In a quick test, OpenAI's "Data Engineer" role came back as $235K – $385K.

Workday

Workday is the one everyone asks about, and the trickiest. Each company's career site exposes a JSON endpoint behind its public page:

curl -X POST "https://nvidia.wd5.myworkdayjobs.com/wday/cxs/nvidia/NVIDIAExternalCareerSite/jobs" \
  -H "Content-Type: application/json" \
  -d '{"appliedFacets":{},"limit":20,"offset":0,"searchText":""}'

Things to know:

You can't guess the URL from a company name. Tenant (nvidia), data center (wd5) and site name (NVIDIAExternalCareerSite) all vary, so start from the company's careers link.
Pages are capped at 20 jobs, and the total field is only filled in on the first page. If you read it on page 2, you'll stop early. (Ask me how I know.)
Dates come as text like "Posted 3 Days Ago", so you only get day precision.

The annoying part: normalizing everything

Each system has different field names, date formats, location shapes and salary structures. If you want one clean table, you end up writing a small adapter per ATS, plus:

detecting which ATS a company uses,
handling pagination and rate limits politely,
deduplicating, and
remembering which jobs you've already seen if you run it daily.

A shortcut

I packaged all of this as an Apify Actor: Company Jobs. You give it a list of companies (names, websites or careers URLs) and it returns one normalized row per open job from Workday, Greenhouse, Lever, Ashby, Workable, SmartRecruiters, Recruitee, Personio, Teamtailor and Rippling:

{
  "company": "openai",
  "title": "Data Engineer",
  "location": "San Francisco; Mountain View",
  "salaryMin": 235000,
  "salaryMax": 385000,
  "salaryCurrency": "USD",
  "platform": "ashby",
  "url": "https://jobs.ashbyhq.com/openai/…"
}

Two features I use most:

"Only new jobs since my last run": put it on a daily schedule and get just the new postings. Great for hiring alerts or "which of my target accounts just opened a data team?"
Title, location and remote filters, so you only pay for jobs you actually want.

It's pay-per-result (about $1 per 1,000 jobs), and it only reads the public job boards that companies publish themselves.

Be a good citizen

Whatever route you take:

Stick to the official public endpoints above, not HTML scraping.
Keep concurrency low and back off on 429s.
Don't collect recruiter names or emails that some boards include. You don't need them for job data.

If there's an ATS you'd like covered next, tell me in the comments.

Written with AI assistance; the endpoints, numbers and examples were tested before publishing.