DEV Community: Daniel Meshulam

Google's News API has been gone since 2016, and people still search for it

Daniel Meshulam — Fri, 25 Sep 2026 21:38:27 +0000

Search for "google news api" and a striking share of the results are people
looking for something that no longer exists. Google deprecated the News Search
API in May 2011 as part of a wider AJAX Search API cull, kept it limping until
the final shutdown in February 2016, and never shipped a replacement. A decade
later the searches are still there.

What almost nobody mentions is that Google News still exposes a perfectly good
public endpoint. It is just an RSS feed rather than a JSON API, it needs no key,
and it has quietly outlived the API that was supposed to be the real product.

The endpoint

https://news.google.com/rss/search?q=<query>&hl=en-US&gl=US&ceid=US:en

That is the whole thing. Swap hl, gl and ceid for a different language and
country and you get that edition instead. There is no key, no quota page, and no
sign-up.

The catches are real but small:

It is RSS, so you get XML. Title, link, publication date, source name and a short description. No article body, and no way to ask for one.
The links are Google redirects, not publisher URLs. You have to follow them if you want the destination.
The publisher name lives in a nested <source> element, not where a normal feed parser looks for it.
Roughly 100 items per query. Deeper history is not available through it.

None of that is a reason to go and pay for a news API. It is a reason to write
about forty lines once.

Two sources are better than one

Bing News publishes its own RSS in the same shape at a different URL. Running
both and merging costs one extra request and meaningfully changes what you see,
because the two indexes disagree about what is worth surfacing.

Here is a real run over both:

16 articles
sources: {'google:semiconductor export controls': 8,
          'bing:semiconductor export controls': 8}

[google] It Takes More Than Two to Tango: Creating Effective Export Con...
         Just Security                                     2026-07-09
[google] China considers tighter export controls on AI models and chips
         Reuters                                           2026-07-20
[google] China's Localization Drive in Semiconductors Gains Impetus...
         CSIS | Center for Strategic and International...   2026-03-24
[google] China's Domestic DUV Push Puts ASML's Export-Control Advantage...
         EU Today                                          2026-07-28

Every row carries title, url, publisher, publishedAt, summary,
source, image and guid.

One thing worth knowing if you parse Bing's feed yourself. It carries the
publisher in <News:Source>, and declares that prefix as

xmlns:News="https://www.bing.com/news/search?q=<your query>&format=RSS"

The namespace URI is the request URL, so it changes on every query and no fixed
namespace constant will ever match it. Match on the local element name instead.
I found this the way you would expect: publisher was null on exactly the Bing
half of a two-source run while the value was sitting in the feed the whole time.

The code

import json
import os
import urllib.request

ACTOR = "glitchbound~news-scraper"
API = "https://api.apify.com/v2"


def headlines(query, per_source=10):
    payload = {
        "searchQueries": [query],
        "maxArticlesPerSource": per_source,
        "language": "en",
        "includeBingNews": True,
        "dedupeHeadlines": True,
    }
    url = (f"{API}/acts/{ACTOR}/run-sync-get-dataset-items"
           f"?token={os.environ['APIFY_TOKEN']}&maxTotalChargeUsd=0.50")
    req = urllib.request.Request(
        url, data=json.dumps(payload).encode(), method="POST",
        headers={"Content-Type": "application/json"})
    with urllib.request.urlopen(req, timeout=300) as r:
        rows = json.load(r)
    # A feed that fails comes back as a row with an `error` field rather than
    # failing the run, and those are not charged.
    return [r for r in rows if not r.get("error")]


for a in headlines("semiconductor export controls", per_source=5):
    # Fields can be absent on any given row, so read them defensively rather
    # than subscripting. Feeds are not schemas.
    when = (a.get("publishedAt") or "")[:10] or "no date"
    who = (a.get("publisher") or "unknown")[:28]
    print(f'{when:10}  {who:28} {(a.get("title") or "")[:60]}')

That runs News Scraper on
Apify, which is the same forty lines plus the parts that are
tedious rather than hard: both feeds, the nested publisher element, the date
normalisation, and deduplication.

If you would rather own the code, the endpoint above is public and this post has
told you everything you need.

Deduplication is most of the value

One wire story appears on dozens of syndicating domains, and Google and Bing will
both hand you several copies. A hundred raw results is frequently thirty stories.

Dedupe on the URL first, then on a normalised headline, because the same Reuters
piece appears under slightly different titles across outlets. Doing it on
headline alone merges genuinely different stories that share a phrasing; doing it
on URL alone leaves most of the duplication in place.

Where this does and does not fit

Fits: brand and competitor monitoring, tracking a topic across outlets,
building a corpus of headlines, seeing which publishers cover a beat, feeding a
summarisation pipeline.

Does not fit: anything needing article text, anything needing more than a few
months of history, or academic archival work. For those you want a licensed
provider, and the honest answer is that no free endpoint replaces one.

What it costs

If you write it yourself, nothing beyond your own hosting.

Through the Actor it is $1.50 per 1,000 articles on the free tier, so a daily run
pulling 50 articles is about seven cents a month, plus a $0.02 Actor-start event
per run. Duplicates that get dropped are not charged, and neither are feeds that
fail.

The wider point

An enormous amount of "you need an API for this" is habit. Google News, Apple's
app charts, the SEC, the World Bank, the ECB and Open Food Facts all publish
usable public endpoints that need no key and get very little attention, largely
because there is no vendor with an incentive to tell you about them.

Worth checking before you pay for access to public data.

Code

Runnable examples are here:
github.com/danielhagever/apify-data-actors

Find startups the day they raise, before the press release, with 30 lines of Python

Daniel Meshulam — Fri, 25 Sep 2026 21:37:15 +0000

Funding announcements are old news by the time you read them. There's usually a
gap of weeks between a company closing a round and telling anyone about it. In
that gap, the round is already a matter of public record.

That's because of Form D. When a US company sells securities in a private
placement, it has to file one with the SEC, generally within 15 days of the first
sale. It's a legal requirement, it's public the moment it's accepted, and almost
nobody reads it.

So the question is not whether you can see funding rounds early. It's how to
filter thousands of Form Ds a month down to the ones you care about.

This post is the working version of that, end to end: ~30 lines of Python, real
output, and the two traps that cost me a couple of hours.

Why you can't just list Form Ds

The naive approach is "give me every Form D from the last 90 days." That's
thousands of filings: real estate syndicates, hedge funds, family offices, a
long tail of LLCs. Most of it is not what anyone means by "a startup raised."

You also can't filter by company name, because you don't know the company
names yet. That's the entire point of the exercise.

What you want is a full-text search over the body of the filings, scoped to
Form D and to a recent window. EDGAR supports exactly that, over every filing
since 2001.

The code

I'm using an Apify Actor I built for EDGAR,
SEC EDGAR Scraper, because
it handles the parts of EDGAR that are tedious: CIK resolution, the fair-access
rate limit, XBRL, and the date-range bug I'll get to below. One HTTP call runs it
and returns rows.

import datetime as dt
import json
import os
import urllib.request

ACTOR = "glitchbound~sec-filings-scraper"
API = "https://api.apify.com/v2"


def form_d_by_sector(sector: str, days: int = 90) -> list[dict]:
    since = (dt.date.today() - dt.timedelta(days=days)).isoformat()
    payload = {
        # Quote the phrase or EDGAR matches the words separately.
        "searchQuery": f'"{sector}"',
        "formTypes": ["D"],
        "filedSince": since,
        "maxSearchResults": 50,
    }
    url = (f"{API}/acts/{ACTOR}/run-sync-get-dataset-items"
           f"?token={os.environ['APIFY_TOKEN']}&maxTotalChargeUsd=0.50")
    req = urllib.request.Request(
        url, data=json.dumps(payload).encode(), method="POST",
        headers={"Content-Type": "application/json"})
    with urllib.request.urlopen(req, timeout=300) as r:
        rows = json.load(r)

    # A target the Actor couldn't resolve comes back as a row with an `error`
    # field instead of failing the whole run, and those aren't charged. Count
    # them separately or a run that fetched nothing still looks successful.
    return [r for r in rows if not r.get("error")]


for f in sorted(form_d_by_sector("artificial intelligence"),
                key=lambda r: r["filingDate"], reverse=True):
    print(f"{f['filingDate']}  {f['companyName']:55} {f.get('state') or '--'}")
    print(f"            {f['filingUrl']}")

Set APIFY_TOKEN from
console.apify.com/settings/integrations
and run it.

Real output

This is an actual run, not a mockup:

2026-07-06  Flo Artificial Intelligence, Inc.  (CIK 0001798621)      CA
            .../data/1798621/000179862126000003/primary_doc.xml
2026-06-29  Artificial Intelligence Financial Services Market Inc.   DE
            .../data/2142624/000214262426000001/primary_doc.xml
2026-06-29  Artificial Intelligence Financial Services Market Inc.   DE
            .../data/2142624/000214262426000002/primary_doc.xml
2026-06-18  Panorama Artificial Intelligence Corp.                   WA
            .../data/2001638/000200163826000001/primary_doc.xml
2026-05-26  Artificial Intelligence Progress Research, LP            --
            .../data/2133240/000213324026000002/primary_doc.xml
2026-05-15  Allied Innovation Artificial Intelligence Fund, LLC      NY
            .../data/2021379/000202137926000001/primary_doc.xml
2026-05-05  Artificial Intelligence Progress Research, LP            --
            .../data/2133240/000213324026000001/primary_doc.xml

(URLs truncated for width; the real ones are absolute sec.gov links.)

Seven filings, each linking straight to the primary document on sec.gov. Every one
is a real entity that filed a real Form D in that window.

Note that the same company can appear twice, with two separate accession numbers
on 2026-06-29. Those are genuinely two filings, not a duplicate row: an amended Form
D gets its own accession. Dedupe on companyName only if you actually want to.

You'll notice something honest about this list: it is heavy on entities with
"Artificial Intelligence" in the company name, plus a couple of funds. That's
what a full-text phrase search gives you. It matches the text of the filing, and
Form D is a short form without a business description. It's a strong signal,
not a magic sector classifier. More on that below.

Trap 1: EDGAR silently ignores a half-open date range

This one cost me real money before I caught it.

EDGAR's full-text endpoint takes startdt and enddt alongside
dateRange=custom. Send only startdt and it does not narrow anything, and it
does not tell you either. Measured against the live endpoint, same query, forms=D:

Parameters	Hits	Oldest result
`startdt` only	53	2015-07-28
no date params at all	53	2015-07-28
`startdt` + `enddt`	7	2026-05-05

startdt alone is byte-for-byte identical to sending no filter. If you're paying
per result, you're paying for eleven years of filings outside the window you
asked for, and the response looks perfectly normal.

If you're calling EDGAR directly: always send both bounds. Default the
missing one: 2001-01-01 is the floor for full-text search, and today for the
upper bound. (This is fixed in the Actor as of build 0.1.4; that's why the run
above returns 7 and not 53.)

Trap 2: "recent" isn't the sort you want for anything else

For funding discovery, newest-first is right. For almost every other EDGAR
question it isn't. If you're looking for the filing that matters on a topic,
recency is noise. Worth knowing before you build the wrong dashboard.

What this costs

The Actor charges per result: $3.00 per 1,000 on the free tier, so seven filings
is 2.1 cents, plus a one-time $0.02 Actor-start event, so about 4 cents for
the run. Paid Apify tiers pay less per result, down to half that on Gold.

The maxTotalChargeUsd=0.50 in the code above is a hard
ceiling the platform enforces. The run stops cleanly at the limit rather than
being killed mid-write, so you can't accidentally spend more than you meant to.

Rows that come back as errors, such as a ticker that doesn't resolve or a query
EDGAR rejects, aren't charged.

Going further

A few directions that work well from here:

Watch a sector on a schedule. Run it daily with days=2 and diff against
what you've already seen. New Form Ds show up the day they're accepted.

Pivot to the company. Once a name appears, feed its CIK back in with
includeFinancials: true to get XBRL figures: revenue, net income, total assets,
straight from what the company tagged, not scraped off a rendered page. A full
20-year history is one request, because it comes from companyfacts.

Search for other things entirely. formTypes: ["8-K"] with a phrase like
"material weakness" or "going concern" finds every company that disclosed it.
formTypes: ["4"] is insider trades. formTypes: ["13F-HR"] is what the big
funds hold.

Caveats worth stating plainly

Form D is a filing, not a validated funding announcement. It tells you securities were sold under a Reg D exemption. It does not confirm a venture-style round, and the amounts on the form are frequently the maximum offering rather than what closed.
Full-text phrase search matches text, not sectors. As the output shows, it finds companies whose filings contain your phrase, which correlates with the sector but isn't the same thing. For real classification you'd want to pull the filing and read it.
This is public disclosure, and that's the whole appeal. No login, no key, no scraping of rendered pages. The SEC asks callers to identify themselves and stay under 10 requests/second, which the Actor does.
Not investment advice, obviously.

Code

Full runnable scripts, including this one and a few others, are here:
github.com/danielhagever/apify-data-actors

Every example input in that repo is generated from the Actor's own input schema,
so they're valid by construction rather than by my memory of the field names.
That's a habit I picked up after the reviewedAt-vs-date mistake in a
different script cost me an afternoon of blank output.

If you find a case where EDGAR behaves differently than described here, I'd
genuinely like to know. That date-range trap only surfaced because I ran the
example instead of just writing it.

The App Store top charts are a public API, and nobody archives them

Daniel Meshulam — Fri, 25 Sep 2026 21:34:54 +0000

Apple's top charts are one of the few genuinely useful public rankings left.
They are free, they need no key, and they update daily.

They also have one property that makes them frustrating: there is no history.
The feed shows you today. Yesterday is gone. If you want to know whether an app
climbed or slid, or which category a publisher quietly took over last month, that
data does not exist anywhere unless somebody was writing it down.

Nobody is writing it down, which means the archive is worth more than the feed.

This post covers both halves: reading the charts today, and the small amount of
work it takes to have a year of history in a year's time.

The feed

Apple's RSS feeds sit at a predictable URL per country, chart type and category.
No key, no login, and no bot wall. That is unusual enough in 2026 to be worth
saying out loud.

What is awkward is the shape. Each entry buries the app id under
id.attributes["im:id"], the price under im:price.attributes, the developer
under im:artist.label, and the icon in an array where you want the last
element. Parsing it correctly is 30 lines of nested .get() calls that you write
once and never enjoy.

I put that in an Apify Actor,
App Store Top Charts Scraper,
so I could stop rewriting it. One HTTP call runs it and returns flat rows.

import json
import os
import urllib.request

ACTOR = "glitchbound~app-charts-scraper"
API = "https://api.apify.com/v2"


def top_charts(countries, charts=("top-free",), limit=50):
    payload = {
        "countries": list(countries),
        "charts": list(charts),
        "categories": ["all"],
        "limit": limit,
    }
    url = (f"{API}/acts/{ACTOR}/run-sync-get-dataset-items"
           f"?token={os.environ['APIFY_TOKEN']}&maxTotalChargeUsd=0.50")
    req = urllib.request.Request(
        url, data=json.dumps(payload).encode(), method="POST",
        headers={"Content-Type": "application/json"})
    with urllib.request.urlopen(req, timeout=300) as r:
        rows = json.load(r)
    # A country or chart that fails comes back as a row with an `error` field
    # rather than failing the run, and those are not charged. Split them out or a
    # run that fetched nothing still looks successful.
    return [r for r in rows if not r.get("error")]


rows = top_charts(["us", "gb", "de"], charts=["top-free"], limit=20)
for r in sorted(rows, key=lambda r: (r["country"], r["rank"])):
    print(f'{r["country"]}  #{r["rank"]:<3} {r["title"][:42]:42} {r["developer"][:24]}')

Real output, from an actual run, top of the German chart:

DE  #1   Claude by Anthropic                        Anthropic PBC
DE  #2   ChatGPT                                    OpenAI OpCo, LLC
DE  #3   Nect Wallet                                Nect
DE  #4   Google Gemini                              Google
DE  #5   EasyPark parken - Park App                 Easy Park AS
DE  #6   AusweisApp Bund                            Governikus GmbH & Co. KG
DE  #7   Mein dm                                    dm-drogerie markt GmbH +
DE  #8   Post & DHL                                 Deutsche Post AG

That German top ten is itself the argument for archiving. Four of those eight are
government or logistics apps that nobody would predict from the US chart, and the
top two are AI assistants that were not in this chart at all two years ago. You
cannot see either of those facts in a single pull.

Every row also carries appId, bundleId, price, currency, isFree,
genre, releaseDate, icon and url, so a chart pull doubles as an app
metadata pull.

The part that is actually valuable

One pull tells you the ranking today, which you could have read off your phone.
The interesting queries all need two pulls separated by time:

Which apps entered the top 20 this week
Which publisher gained the most chart slots this month
How long a given app has held its position
Whether a competitor's launch actually moved anything
Which categories are churning and which are frozen

None of that is answerable from the feed. All of it is answerable from a table
you append to daily.

So the whole trick is: run it on a schedule and never delete anything.

import datetime as dt

rows = top_charts(["us"], charts=["top-free", "top-paid"], limit=100)
stamp = dt.datetime.now(dt.timezone.utc).isoformat(timespec="seconds")
for r in rows:
    r["capturedAt"] = stamp          # the feed has no timestamp; add your own
# append to wherever you keep things: Postgres, SQLite, a CSV, a dataset

On Apify you get this without writing the storage layer: put the Actor on a
Schedule and every run appends to the same dataset, each with its own timestamp.
Three months later you have something nobody sells.

One detail that matters if you build this yourself: the feed carries no
timestamp of its own. If you do not stamp each capture at write time, a year
later you have a pile of rankings and no way to order them. Stamp it on the way
in, not on the way out.

What it costs

Charged per chart entry returned. A 100-app chart is 100 results, so at $3.00 per
1,000 on the free tier a daily 100-app capture for one country is about 30 cents
a month, plus a $0.02 Actor-start event per run. Countries and chart types
multiply that, so pick the markets you actually care about rather than sweeping
all of them.

Rows that come back as errors are not charged.

Caveats worth stating plainly

Apple only. Google Play publishes no equivalent public chart feed. Anything claiming a Play "top chart" API is scraping a rendered page, which breaks on redesign.
Charts are per country, per chart type, per category. An app can be absent from top-free while sitting high in top-grossing. "Not in the chart" is always relative to which chart you asked for.
Apple caps a feed at 200 entries. Beyond that the answer does not exist in the feed.
Rankings are Apple's, computed however Apple computes them. This reads them; it does not model them.

Code

Runnable examples, including this one, are here:
github.com/danielhagever/apify-data-actors

Every example input in that repo is generated from the Actor's own input schema,
so they are valid by construction rather than by my memory of the field names.

WHOIS is gone, RDAP replaced it, and a 404 does not mean what you think

Daniel Meshulam — Fri, 25 Sep 2026 21:32:53 +0000

If you last touched domain lookups a few years ago, the thing you remember is
WHOIS: a plaintext protocol on port 43 that returned an unparseable wall of text
in a different shape from every registry.

It is effectively gone. RDAP replaced it, it is RFC 9083, it returns JSON, and
every gTLD registry is required to run one. This is one of the rare cases where
the standards process produced something strictly better and nobody noticed.

Here is what a lookup looks like now, what you can actually get out of it after
GDPR, and the one trap that had my own tool confidently reporting registered
domains as free to buy.

What RDAP gives you, and what it does not

Post-GDPR, RDAP returns no registrant personal data for virtually every TLD. If
you are looking for who owns a domain, stop: that information is not there and
no legitimate source has it.

What is there is the operational picture, and that turns out to be the more
useful half:

registrar, registration date, expiry date, status codes, DNSSEC
whether the name is locked against transfer

Combine that with a DNS lookup and you get the thing people actually want, which
is what a company runs on:

domain        registrar                expires       age mail       dns         dmarc
stripe.com    SafeNames Ltd.           2027-09-11   30.9 google     aws-route53 reject
linear.app    CloudFlare, Inc.         2030-05-09    8.2 google     cloudflare  reject
vercel.com    Amazon Registrar, Inc.   2029-10-04   26.8 google     vercel      quarantine
github.io     -                        -               - -          aws-route53 -
angel.co      -                        -               - google     cloudflare  reject
notion.so     -                        -               - -          cloudflare  quarantine

That is a real run. Mail provider and DNS provider come from MX and NS records,
DMARC policy from a TXT lookup at _dmarc.<domain>. For sales qualification,
security auditing or competitive research, those three columns are worth more
than any registrant field ever was.

Note the bottom three rows have DNS but no registrar and no expiry. Hold that
thought.

The code

import json
import os
import urllib.request

ACTOR = "glitchbound~whois-domain-lookup"
API = "https://api.apify.com/v2"


def profile(domains):
    payload = {
        "domains": domains,
        "includeDns": True,
        "recordTypes": ["A", "MX", "NS", "TXT"],
        "includeEmailSecurity": True,
    }
    url = (f"{API}/acts/{ACTOR}/run-sync-get-dataset-items"
           f"?token={os.environ['APIFY_TOKEN']}&maxTotalChargeUsd=0.50")
    req = urllib.request.Request(
        url, data=json.dumps(payload).encode(), method="POST",
        headers={"Content-Type": "application/json"})
    with urllib.request.urlopen(req, timeout=300) as r:
        rows = json.load(r)
    # A domain that could not be looked up comes back as a row with an `error`
    # field rather than failing the run, and those are not charged.
    return [r for r in rows if not r.get("error")]


for r in profile(["stripe.com", "linear.app", "vercel.com"]):
    print(f'{r["domain"]:14} {r.get("mailProvider") or "-":10} '
          f'{r.get("dnsProvider") or "-":12} dmarc={r.get("dmarcPolicy") or "-"}')

That runs an Apify Actor I maintain,
WHOIS Domain Lookup, because the
tedious parts are the RDAP bootstrap, the vcard unpacking and the provider
fingerprinting rather than the HTTP call. If you would rather do it yourself,
https://rdap.org/domain/<name> is the whole API and it needs no key.

The trap: a 404 is ambiguous

rdap.org is a router. It reads IANA's bootstrap registry, finds the RDAP
server for your TLD, and redirects. A 404 back from it looks like it means one
thing: no registry holds this name, therefore the domain is available.

That is what my tool assumed. It was wrong, and the failure is ugly:

rdap.org/domain/github.io    404
rdap.org/domain/notion.so    404
rdap.org/domain/segment.io   404
rdap.org/domain/angel.co     404

All four are registered. All four returned 404.

The reason is that IANA's bootstrap does not cover every TLD. It lists RDAP
services for around 1,200 of them, and some of the most common startup TLDs are
not in there:

.io   .co   .de   .me   .so   .gg   .sh

For those, rdap.org has nowhere to send the query, so it 404s for every name
under them, registered or not. My Actor turned that into isAvailable: true and
would have told you github.io was free to register.

On a tool whose headline feature is availability checking, that is the worst
possible way to be wrong: confident, plausible, and acted upon. Nobody
double-checks a clear answer.

The fix, in two parts

Ask whether there is a registry at all. Fetch
https://data.iana.org/rdap/dns.json once, build the set of TLDs that have an
RDAP service, and only treat a 404 as "available" when the TLD is in it.

import json, urllib.request

def rdap_tlds():
    d = json.load(urllib.request.urlopen("https://data.iana.org/rdap/dns.json"))
    return {t.lower().lstrip(".") for entry in d["services"] for t in entry[0]}

Let DNS settle the rest. A name with NS records has been delegated by the
registry, and a registry does not delegate a name nobody registered. So for the
TLDs with no RDAP, NS records prove registration even though RDAP cannot.

The reverse does not hold, and this is the part worth being careful about: a
registered domain can have no NS at all. So "no NS and no RDAP" is not
"available", it is unknown, and it should be returned as null with an
explanation rather than as a guess. A null is a fact the caller can handle. A
plausible wrong answer is not.

Why this generalises

The bug was not really about RDAP. It was about reading an absence as a
definite answer.

404 means "I have nothing for you". Whether that is because the thing does not
exist, or because you asked somewhere that was never going to know, is not in
the response. Any time you convert a missing result into a positive claim, check
that you actually asked something capable of answering.

I have now hit the same shape four times in a month in different tools: an API
silently ignoring a filter it does not support, a page parser taking the first
number in a plausible range, this, and crt.sh, which answers 404 when it is
refusing traffic and reports "nothing found" as an empty list with a 200. All of
them produced output that looked completely normal.

What it costs

Charged per domain returned, at $2.00 per 1,000 on the free tier, so a hundred
domains is 20 cents plus a $0.02 Actor-start event. Domains removed by the
built-in filters, like onlyRegistered or expiringWithinDays, are not charged,
and neither are lookups that fail.

Notes

Some ccTLDs run thin RDAP servers that return fewer fields. You get whatever the registry publishes rather than an error.
DNS answers are returned sorted. Resolvers round-robin, and unsorted output makes a scheduled run look like it changed every time when nothing did.
Availability comes from the registry itself, not a reseller API, so there is no incentive for anyone to tell you a name is taken when it is not.

Code

Runnable examples are here:
github.com/danielhagever/apify-data-actors

Every example input in that repo is generated from the Actor's own input schema,
so it is valid by construction rather than by my memory of the field names.

10,789 American employers publish their whole job list in one HTML page

Daniel Meshulam — Fri, 25 Sep 2026 21:31:35 +0000

A Paylocity careers page looks like a React app:

https://recruiting.paylocity.com/recruiting/jobs/All/8f0bcd76-3ee7-42f2-b040-83afc6e6d564

Open the network tab and there is no job API call to find. The jobs are
already there, in the HTML, in one global:

import json, re, requests

UA = {"User-Agent": "Mozilla/5.0"}
guid = "8f0bcd76-3ee7-42f2-b040-83afc6e6d564"
html = requests.get(f"https://recruiting.paylocity.com/recruiting/jobs/All/{guid}",
                    headers=UA).text
data = json.loads(re.search(r"window\.pageData\s*=\s*(\{.*?\});\s*\n", html, re.S).group(1))

data["ModuleTitle"]        # 'City of Columbia'
len(data["Jobs"])          # 9

And a job is not a card scraped off a page, it is a record:

{'JobId': 4514780,
 'JobTitle': 'Building and Grounds Maintenance Part Time',
 'PublishedDate': '2026-09-17T13:43:33-05:00',
 'IsInternal': False, 'IsRemote': False,
 'JobLocation': {'Address': '1648 Trottwood Ave.', 'City': 'Columbia',
                 'State': 'TN', 'Zip': '38401', 'Country': 'USA'}}

A street address. Most career systems give you "Columbia, TN" if you are
lucky and "Ridley Park - Parks and Recreation" if you are not.

recruiting.paylocity.com answers 404 to /robots.txt, so there is nothing
there to disallow anything.

Four things that bite

The assignment ends at ;\n. Job descriptions contain semicolons. A
greedy match, or a match that stops at the first ;, both give you a broken
object. The newline is what makes it unambiguous.

A retired board answers 200. When an employer leaves Paylocity the guid
keeps serving a portal page, just without pageData. So "no pageData" is the
proof a board is gone; the status code never says so.

JobId is a global sequence, not a per-board one. Measured over ten
unrelated boards: the ranges interleave (a 40-person academy's ids sit inside
a hospital group's range) and no id appeared at two employers. That matters
because an employer can run several modules, and the same posting shows up on
more than one. Key on the id alone and they collapse correctly.

LocationName is not a location. It is whatever the employer labelled the
site: "Public Works", "Columbia Aquatics & Recreation Center", sometimes a
person's name. The address object is the only place a city comes from.

How many there are

A board is a guid, and nothing derives a guid from a company domain, so the
list has to come from somewhere. The Wayback CDX index holds every URL it has
archived on that host:

https://web.archive.org/cdx/search/cdx?url=recruiting.paylocity.com
  &matchType=domain&collapse=urlkey&fl=original&output=json
  &filter=original:.*[Rr]ecruiting/[Jj]obs/[Aa]ll/.*

Two things about that query cost me boards before they earned them. limit
stops at the first page and looks complete: the first sweep returned exactly
20,000 URLs and 14,836 guids, which is a suspiciously round stopping point.
And the filter is a case-sensitive regex, while the archive holds the path
capitalised as well; fixing the case alone found 1,821 more.

Read live on 2026-09-24: 16,936 candidate boards, 13,581 that still answer,
10,789 with open roles, 146,046 jobs. One GET each, six at a time, a
quarter of a second apart: 41 minutes for the lot, and not one request
refused.

The employers are the interesting part. This is not the Fortune 500, which
every jobs dataset already has three times over. It is DOCS Health, a Taco
Bell franchise group with 2,219 openings, Farmers Home Furniture, credit
unions, dialysis clinics, car dealerships, county governments, country clubs.
The half of the American labour market that never appears in an enterprise
ATS index.

If you want the rows rather than the code, that is what
ats-jobs-scraper does: a
company name in, every open posting out, from Paylocity and twenty-one other
career systems, as JSON.

BrassRing has a JSON job search, and Home Depot publishes 23,982 jobs through it

Daniel Meshulam — Fri, 25 Sep 2026 21:28:03 +0000

Best Buy's careers site is a ServiceNow portal. Its apply links go somewhere
else: sjobs.brassring.com, which is BrassRing (Infinite Talent, once IBM
Kenexa). One host serves every employer on it, and a board is a partner id
plus a site id:

https://sjobs.brassring.com/TGnewUI/Search/Home/Home?partnerid=25632&siteid=5649

That one is Best Buy. Home Depot's store jobs are 25526/5032, Walgreens
26336/5014. There is no robots.txt on the host (404), and the search the page
itself calls is plain JSON.

The two calls

The search page carries an anti-forgery token, and the Ajax calls want it as a
header named RFT. Without it every call is HTTP 500 with an HTML error page,
which is what makes this look closed when it is not.

import re, requests

s = requests.Session()
home = s.get("https://sjobs.brassring.com/TGnewUI/Search/Home/Home",
             params={"partnerid": "25632", "siteid": "5649"},
             headers={"User-Agent": "Mozilla/5.0"}).text
token = re.search(r'name="__RequestVerificationToken"[^>]*value="([^"]+)"', home).group(1)
H = {"RFT": token, "X-Requested-With": "XMLHttpRequest",
     "Content-Type": "application/json", "User-Agent": "Mozilla/5.0"}

first = s.post("https://sjobs.brassring.com/TgNewUI/Search/Ajax/MatchedJobs", headers=H, json={
    "PartnerId": "25632", "SiteId": "5649", "Keyword": "", "Location": "",
    "KeywordCustomSolrFields": "JobTitle,Location", "LocationCustomSolrFields": "Location",
    "FacetFilterFields": None, "TurnOffHttps": False, "Latitude": 0, "Longitude": 0,
    "PowerSearchOptions": {"PowerSearchOption": []}, "encryptedsessionvalue": ""}).json()

first["JobsCount"]                      # 2684
len(first["Jobs"]["Job"])               # 50

A job arrives as a list of question/value pairs rather than an object:

{q["QuestionName"]: q["Value"] for q in first["Jobs"]["Job"][0]["Questions"]}
# {'reqid': '3761989', 'jobtitle': 'Retail Sales Associate',
#  'lastupdated': '22-Sep-2026', 'formtext12': 'Greensboro',
#  'formtext10': 'North Carolina ', 'formtext1': 'Part time ', ...}

Pages come from a different endpoint, ProcessSortAndShowMoreJobs, whose keys
are lower-cased (partnerId, pageNumber). PageNumber on MatchedJobs
looks like it pages and does not.

The trap: the sort repeats pages

A site offers two sorts, date and title, and both are full of ties, so
consecutive pages overlap and skip. Sweeping Best Buy's 54 pages once by date
returned 2,299 distinct jobs of 2,680. Not a rate limit, not an error: just an
unstable sort under a paging API.

What works is sweeping more than once and joining on the requisition id:

date sweep                 2,299
title sweep                2,580
date + title + date        2,661   (99.3%)

Stop as soon as the union reaches the declared count, and most sites finish in
one sweep. Read that way, live on 2026-09-22: Best Buy 2,684 of 2,684 in 44
seconds, Performance Food Group 1,359 of 1,359, Home Depot 23,876 of 23,982 in
1,441 requests.

Which fields hold the place changes per site

The page config says what each site shows under a title:

"JobFieldsToDisplay": {"JobTitle": "jobtitle",
                       "Position3": ["formtext12", "formtext10", "formtext1"],
                       "Summary": "jobdescription"}

Best Buy shows city, state and schedule in custom fields. Home Depot has a
location field and uses it. Performance Food Group shows its division, then
"St Louis, Missouri (MO)", then a facility name, and its location field
holds the facility. Edward Jones shows the requisition number first, and
TeamHealth shows a medical specialty and no place at all.

A rule that survives all of those: prefer a shown value that reads as "City,
State"; else the site's own location; else join the shown values up to the
one naming a state, skipping ids and anything with a colon; and if none of
them names a state, publish no location rather than a wrong one.

How many employers are on it

The Wayback CDX index lists archived search pages: 727 partner/site pairs, 489
of which still answer, 120 with open roles. After dropping internal-only
sites, a template site and a network that re-posts other employers' jobs, 98
boards and 72,436 jobs, including Home Depot's stores (23,982), Walgreens
(22,446), Best Buy, Harbor Freight, Infosys, Performance Food Group, Hobby
Lobby, GardaWorld, ADM, Edward Jones, Publix, American Greetings, Bechtel,
GUESS, AAFES, U.S. Steel and Stantec.

One more thing worth knowing: an employer runs several sites over one partner
id (ADM in six languages, Edward Jones three, Walgreens two), and the same
requisition appears on more than one. The requisition is keyed per partner, so
partner + reqid is the job.

If you want the rows rather than the code, that is what
ats-jobs-scraper does: a
company name in, every open posting out, from BrassRing and twenty other
career systems, as JSON.

Workday says the board has 2,000 jobs. Target has 12,525. Reading the rest.

Daniel Meshulam — Fri, 25 Sep 2026 21:25:25 +0000

Ask a Workday career site for its jobs and the first answer states a total:

r = requests.post(
    "https://target.wd5.myworkdayjobs.com/wday/cxs/target/targetcareers/jobs",
    json={"appliedFacets": {}, "limit": 20, "offset": 0, "searchText": ""},
)
r.json()["total"]      # 2000

Target does not have 2,000 open jobs. It has 12,525. The number is the most
that search will return, and once you have paged to it the API just re-serves
page one: offsets past the ceiling come back as the first twenty postings
again, forever, which is how a paging loop that stops on "empty page" turns
into an infinite one.

Measured on 2026-09-22 across 3,916 Workday boards: 59 of them sit at exactly
that wall (a board shows 1,990-2,000 because postings of one requisition merge
into one job). Among them Target, Walmart, NVIDIA, Accenture, Citi, Airbus,
T-Mobile, PNC, Abbott, Kohl's, Sysco, Circle K and Trinity Health.

The counts are in the same answer you already have

The response carries a facets list, and those counts are of the whole board,
not of the 2,000:

Location_Country                 United States of America  12,452   India  73
Location_Region_State_Province   California 1,701   Texas 879   Florida 803 ...
jobFamilyGroup                   Stores 12,023   Internships 132 ...
workerSubType                    Regular 11,538   Seasonal 854   Intern 133
timeType                         Variable 11,427   Full time 1,098

Four of those sum to exactly 12,525. That agreement is the useful part: a
facet a posting carries once sums to the board, a facet it can carry several
times (skills, locations) sums higher, and one only some postings carry sums
lower. So the board's real size is the sum the most facets agree on.

Each facet value is its own search, with its own ceiling

appliedFacets narrows the search, and the narrowed search gets a fresh
2,000:

body = {"appliedFacets": {"Location_Region_State_Province": ["<value id>"]},
        "limit": 20, "offset": 0, "searchText": ""}

California is 1,701 postings, and 1,701 come back. So the board is read slice
by slice and the slices are joined on each posting's externalPath, which is
the posting's own address and therefore the one key that cannot collide:

Read the first page. If total is exactly 2,000, take the facets.
Work out the real size (the sum the most facets agree on).
Pick the facet that covers that size and whose every value is under the ceiling, fewest values first. Target: its 52 states.
Where one value is still over the ceiling, apply it and split again by a second facet. Petco needs this: 2,980 of its jobs are "Retail (Stores)", and inside that slice the schedule splits it 1,893 / 1,024.
Where no facet covers the slice at all, use the best partial one AND read the slice's own 2,000: Petco's schedule field is blank on 63 postings, and those come back in the plain read.

What it returns

Read whole, on 2026-09-22:

Target      12,499 of 12,536   655 requests
Michaels     5,071 of  5,071   279
Petco        3,774 of  3,798   298
Home Depot  (Workday corporate site, 1,031, no ceiling)

Walmart went from 1,990 rows to 21,202, Circle K from 2,000 to 11,397.

Two things worth keeping when you build this:

A failed slice falls back to the plain 2,000. Target is ~650 requests
against ~100, so a transient error is six times likelier; a caller who got
2,000 rows yesterday should not get an exception today for the sake of the
other 10,499.

Nothing splits under the ceiling. If the caller wants 1,000 rows, the
plain search already answers, and the split is pure cost.

The other thing in that answer

While measuring this I found that bulletFields[0], which most code (mine
included) uses as the job id, is not an id on 224 of those boards: Thales
prints "Regular Employee" there, Michaels the state, Cracker Barrel the
store's street address. Anything that de-duplicates on it collapses the board:
61,107 jobs became 16,314, and a run for Thales returned 7 rows for 200 jobs.
The requisition is in the posting's address instead
(/job/<place>/<Title-Slug>_<req>), and the bullet is only safe when it IS
that requisition.

If you want the rows rather than the code, this is what
workday-jobs-scraper
does: a company name in, every open posting out, past the ceiling, as JSON.

I bypassed my own Claude Code deny-list in eight ways. Only an allow-list held.

Daniel Meshulam — Fri, 11 Sep 2026 15:47:38 +0000

In April 2026 a coding agent at PocketOS hit a credential mismatch in staging, found an infrastructure token in an unrelated file, and deleted the production database and its volume backups in one API call. Thirty hours down. Every post-mortem named the same causes: a token within reach, no gate on destructive actions, backups in the same blast radius. Nobody blamed the model.

Most teams I talk to protect against this with a line in CLAUDE.md that says "never touch production". That is a sign, not a lock. So I built the lock, then spent an evening trying to break it. Here is what happened.

Round one: the deny-list

Claude Code has PreToolUse hooks: a small program that receives the tool call as JSON on stdin and answers with an exit code (0 allow, 2 deny). The obvious first guard for a protected folder is a deny-list. Block rm, mv, rmdir when the command mentions protected/. Ten lines. Done.

It took minutes to defeat. All of these wrote into the protected tree untouched:

Bypass	Command
Other verbs	`cp`, `tee`, `dd`, `sed -i`, `install`, `rsync` into `protected/`
Nested interpreter	`sh -c "rm protected/x"`, `perl -e 'unlink "protected/x"'`
Variable indirection	`P=protected; rm $P/x`
Subshell	`(cd protected && rm x)`
Acting flag	`find protected -delete`
Pipeline	`ls protected \
Clobber operator	{% raw %}`echo x >\
Symlink	{% raw %}`ln -s protected alias; rm alias/x`

Worse: two of the blocks the guard did produce came from a quoting parse error, not from detection. A guard that is wrong in both directions cannot be reasoned about at all.

Round two: invert the rule

The fix is not a longer deny-list. It is a different rule: referencing a protected path is denied unless every verb in the command is on a short read-only allow-list (cat, ls, grep, head, find without acting flags, and a few more). Anything unknown fails closed. A tool nobody predicted is covered by construction, because nothing has to be predicted.

Three details that only showed up under attack:

Resolve paths before comparing. ../protected/x, symlinks and cd in the middle of a command all change what a relative path means. The guard tracks the session cwd that Claude Code sends and resolves every token with realpath before deciding. Using the hook process's own cwd made the same command allowed or blocked depending on where the hook happened to be launched from.
Read every path key. The write guard originally read only tool_input.file_path. NotebookEdit sends notebook_path; MultiEdit nests paths under edits[]. Adding those tools to the matcher looked like coverage while the guard inspected nothing. A guard that is wired but cannot read its input is more dangerous than a missing one, because the settings file now lies to you.
Fail closed on confusion. Unparseable quoting, a missing config file, a malformed policy, a payload with no path: every one of these denies. A guard that waves something through when confused is worse than no guard.

Round three: destructive commands that touch no path

The path rule says nothing about git push --force, git reset --hard, curl | sh, terraform destroy or DROP TABLE. Those get a second layer: regex deny patterns checked on the raw command string, before any path logic, so quoting tricks around the path parser cannot route around them.

Prove it or it does not exist

Every bypass above is now a permanent regression test. The suite feeds each hook the JSON Claude Code would send and asserts the exit code, 125 assertions, and it snapshots and restores everything it mutates so a Ctrl-C mid-run cannot leave the repo in a broken state (the suite even kills a nested copy of itself at the dangerous moment to prove the trap works).

Every block also writes one JSON line to an audit log and, if you set a Slack or Discord webhook, posts the block to the team channel in the same call. A small script turns a month of that log into a report: how many agent actions were blocked, by reason, by tool, by day. A control nobody can see is a control nobody funds.

Postscript: the built-in rules had bypasses too

The Claude Code 2.1.268 changelog fixed deny and ask rules that did not apply on symlinked directories, and a case where a Read or Edit deny rule did not apply when env -C, eval or a similar command the permission checker cannot analyze was on the same line. That is not a criticism of the permission system; it is the reason to put a tested hook layer in front of it and to re-run the tests on every release.

Take it

The kit is public: agent-guardrails-kit. Copy hooks/, gates/ and config.json into a repo, edit protected_paths and deny_patterns, wire two hooks in .claude/settings.json, run the tests.

If you want to know where your current setup stands before touching anything, there is a grader that reads your settings.json and CLAUDE.md in the browser and scores it out of 10: agent-guardrails.meshulam791.workers.dev/grade. Nothing is uploaded. Here is what the full exposure report looks like on a fictional repo.

I install and test-prove this for teams on Claude Code and Cursor. Details on the same site.

Merging job postings from ten ATS platforms into one schema, and the four fields that fight back

Daniel Meshulam — Sat, 08 Aug 2026 11:56:22 +0000

Reading one ATS job board API is easy. Every one of them is public JSON with
no key. Reading ten and getting rows you can actually query together is a
different job, and it is almost entirely about four fields.

I run an index across Greenhouse, Ashby, Workable, Workday, BambooHR, Breezy,
Teamtailor, Personio, Recruitee and Rippling. Here is what actually broke,
with the measurements.

1. Remote is not a boolean, and one API will lie to you about it

Ashby postings carry isRemote. It looks like exactly the field you want.

Measured on Ramp's board, 1 August 2026: isRemote was true for 117 of 126
postings, while the same postings' workplaceType said Hybrid 101, Remote
16, OnSite 9.

Ashby's isRemote means "not fully on-site". Trust it and a user asking for
remote work gets a hundred hybrid roles at a New York office.

workplaceType is the honest field, and it has three values, because hybrid
is a real third state that a boolean cannot hold. Normalise to the three-way
enum and derive the boolean from it, never the reverse:

workplace_type = raw.get("workplaceType")            # Remote | Hybrid | OnSite
is_remote = (workplace_type == "Remote") if workplace_type else None

Note the None. A platform that publishes no arrangement at all gets unknown,
not False. "We do not know" and "we know it is not remote" are different
answers and collapsing them is how a filter silently drops good rows.

2. Dates arrive in three shapes, one of which is relative

ISO 8601 from some. Epoch milliseconds from others. And Workday hands you a
human sentence: "Posted 4 Days Ago", "Posted Today",
"Posted 30+ Days Ago".

Epoch handling has a cheap tell:

# Milliseconds since the epoch are 13 digits until the year 2286.
secs = n / 1000 if n > 10_000_000_000 else n

The relative ones need parsing, and one of them must be refused:

_AGO = re.compile(r"^posted\s+(\d+)\s+days?\s+ago$", re.I)

def workday_posted(text):
    t = (text or "").strip().lower()
    if t == "posted today":     days = 0
    elif t == "posted yesterday": days = 1
    else:
        m = _AGO.match(t)
        if not m:
            return None          # "posted 30+ days ago" lands here, on purpose
        days = int(m.group(1))
    return (datetime.now(timezone.utc).date() - timedelta(days=days)).isoformat()

"Posted 30+ Days Ago" returns None deliberately. It could be 31 days or

Turning it into "31 days" produces a date that sorts, filters and charts perfectly and is fiction. Keep the original string in a separate field and let the caller see what the source actually said.

3. Employment type can vary between two boards on the SAME platform

This one surprised me. It is not just that platforms disagree with each other.
Workday's timeType is tenant-configured, so two customers of the same ATS
publish different things.

Measured 2026-08-08 across ten Workday boards: two publish timeType on
every posting, and eight publish it on none. The only value that appeared at
all was "Full time".

So "does this platform have employment type" is not a question with an answer,
and any per-platform lookup table encodes a guess. Read the field where the
board provides it, map it through one shared vocabulary, and leave it null
everywhere else. I had this branch hard-coded to None for a while, which
meant 258,876 rows carried no employment type even where Workday had plainly
stated one.

The trap on the other side is a filter for "full-time" that silently excludes
the majority of your index, because most rows are null rather than false.
Filter on != 'part-time' if you mean "not part-time", and say in your docs
which one you did.

4. Company name is often not a company name

Several platforms return the board token where you expect the employer, so you
get acmecorp instead of Acme Corp, and on Workday you can get the hostname.
Pull the display name from the board envelope rather than the posting, and if
it is absent, say so instead of shipping a slug that looks like a name.

The rule underneath all four

Every one of these bugs has the same shape: a field existed, so it got
trusted. isRemote exists. A parsed "30+ days" exists. A board token
exists where a company name goes. Each one produces a value that is
well-formed, sorts correctly, and is wrong.

The fix is always the same and it is not technical. Carry unknown as unknown.
Three-state where reality has three states. Null where the source said nothing.
It makes your schema uglier and your answers true, and in this data the null
rate is itself information: it tells you which platform to stop promising
things about.

The index, if you want the rows rather than the code

53,000 company boards, 23 ATS platforms, 2,442,000 open roles in one schema,
rebuilt nightly, no login and no company list. On Apify as
ATS Jobs Search API, or
single-platform:
Greenhouse,
Workday,
Ashby,
Workable,
BambooHR.

If you want to see the merged fields on real rows before deciding whether the schema is worth it, every vertical page at glitchbound-jobs.pages.dev is built straight from it: game dev, nursing, devops.

Posted date is populated on 76.2% of rows and country on 78.0%. Those are the
real numbers, not the ones I would like, and the missing quarter is mostly
Workday's "30+ days" refusing to become a date.

How to know which job postings were removed, when nothing tells you

Daniel Meshulam — Sat, 08 Aug 2026 11:53:08 +0000

Every ATS job board API answers the same question: what is open right now.
Greenhouse, Ashby, Workable, Workday, all of them hand you the current list.

None of them tell you what disappeared. There is no closed_at, no
deletions feed, no tombstone. A role that was filled yesterday is simply not
in today's response, indistinguishable from a role that never existed.

And disappearance is usually the more interesting half. A company that opened
40 roles and closed 38 is not the same company as one that opened 40 and
closed none, and the open-roles endpoint reports both as 40.

The mechanic is a diff, and it is not hard

You cannot ask for closures. You have to have been watching.

CREATE TABLE seen (
    platform TEXT, boardToken TEXT, jobId TEXT,
    firstSeen TEXT,          -- first run this job appeared in
    lastSeen  TEXT,          -- most recent run it appeared in
    closedOn  TEXT,          -- set when it stops appearing
    PRIMARY KEY (platform, boardToken, jobId)
);

Each run: upsert everything you fetched with today's date in lastSeen, then
mark the rest closed.

UPDATE seen SET closedOn = :today
 WHERE closedOn IS NULL
   AND lastSeen < :today
   AND boardToken IN (SELECT boardToken FROM fetched_ok_today);

That last line is the one people leave out, and it is the difference between
a closure feed and a random number generator.

The three ways this goes wrong

A failed fetch is not a closure. If a board 500s, times out, or you get
rate limited, every job on it vanishes from your response. Diff naively and
you have just recorded 2,000 closures at one company on the day their CDN had
a bad afternoon. Only diff boards that answered successfully this run.
Everything else keeps yesterday's state untouched.

A paginated board that stopped early is the same bug wearing a hat. If you
read 3 of 7 pages and treat the result as the complete current state, pages
4 to 7 all "closed". Track per-board fetch completeness, not just HTTP 200.

You cannot know how long a job was open if you started watching after it
was posted. This is the subtle one. A job you first saw on the day you
started watching may have been open for a week or a year. Computing
daysOpen = closedOn - firstSeen on that row invents a number.

The fix is to record it as unknown rather than compute it anyway:

board_first_seen = min_first_seen_for(platform, board_token)
knowable = job.first_seen > board_first_seen   # strictly after, not equal
days_open = (closed_on - job.first_seen).days if knowable else None

"Open for 3 days" and "we do not know how long it was open" must never look
the same, because one of them is a signal about the company and the other is
an artefact of when you happened to start.

The same discipline applies upward: a company's closed30 should be absent
until you have watched it for 30 days, not 0. "Closed nothing" and "we have
not been looking long enough" send a reader to two different conclusions.

What it gets you once it runs

Closures turn a job board into a time series. Net change per company. Roles
that close in four days, which usually means an internal candidate. Roles that
sit open for six months, which usually means the requisition is a placeholder.
A company whose closures suddenly outrun its openings, which is a churn signal
worth a phone call if they are your customer.

None of that is visible in the endpoint everyone reads.

If you would rather not run it for a month first

That is the real cost here: this data does not exist until you have collected
it, and the first month produces nothing.

I have been running the diff across 53,000 company boards on 23 ATS platforms,
and the index currently holds 2,442,000 open roles and 874,000 recorded
closures, rebuilt nightly. It is on Apify as
ATS Jobs Search API, with
company-level opening and closing counts in
B2B Leads Finder.

The closure counts are readable without an account or a key at glitchbound-jobs.pages.dev, which has a page on closed roles rebuilt from the same nightly diff.

daysOpen is null wherever it is not knowable, for the reason above. If you
find a row where that rule looks wrong, tell me, because that is the exact
mistake this is trying not to make.

How to find any company's job board from just its domain, without a list

Daniel Meshulam — Sat, 08 Aug 2026 11:48:49 +0000

Most tech companies run their careers page on an ATS: Greenhouse, Ashby,
Workable, Workday, BambooHR and a handful of others. Nearly all of those
expose a public JSON endpoint for their postings, no key and no login.

The endpoint needs a board token, though, which is the company's slug
inside that ATS. And that is the actual problem. You have stripe.com. You
need to know that Stripe is on Greenhouse under the token stripe, and every
tutorial starts one step after that, with the token already in hand.

Here is what actually works, and the measurement that says so.

The obvious approach is the worse one

Fetch the company's careers page, look for a link to boards.greenhouse.io or
jobs.ashbyhq.com, pull the token out of the URL.

Measured across 30 companies: that finds the board 50% of the time.

It fails on exactly the companies you most want. Stripe's careers page is a
JavaScript application that names no board anywhere in the delivered HTML.
Render it in a headless browser and you have turned a 200ms JSON fetch into a
multi-second browser run, for a token that is five characters long.

Guess the token, then verify it

The better approach inverts it. Derive candidate tokens from the domain, then
ask each ATS whether that board exists. On these platforms a bogus token
returns 404 and a real one returns 200 with the postings, so a 200 is
proof. You never need the careers page at all.

import httpx

BOARDS = {
    "greenhouse": "https://boards-api.greenhouse.io/v1/boards/{t}/jobs",
    "ashby":      "https://api.ashbyhq.com/posting-api/job-board/{t}",
    "workable":   "https://apply.workable.com/api/v1/widget/accounts/{t}?details=true",
    "bamboohr":   "https://{t}.bamboohr.com/careers/list",
}

def resolve(domain: str):
    token = domain.split("//")[-1].split("/")[0].split(".")[-2]
    hits = []
    for ats, url in BOARDS.items():
        r = httpx.get(url.format(t=token), timeout=15)
        if r.status_code == 200:                 # a 200 here is proof
            hits.append((ats, token, r.json()))
    return hits

Measured on the same 30 companies: 90%, including Stripe, whose Greenhouse
board answers with more than 500 open roles the careers page never mentioned.
(Check it yourself: boards-api.greenhouse.io/v1/boards/stripe/jobs. The
count moves daily, which is rather the point.)

One attempt per guess, not a retry ladder. A guessed token either exists or it
does not, and retrying every wrong guess three times is what turns a sweep of
90 domains into 25 seconds each.

The part that will bite you

Two failure modes look identical to a naive script, and both produce a
confident wrong answer.

A company can run more than one ATS. Take the board with the most
postings, not the first one that answers 200. Otherwise a company with a stale
Lever board from 2023 and a live Greenhouse board resolves to the dead one.

An empty board is a real answer, and it is not the answer you want.
nvidia.com resolves to a genuine Workable account with zero open roles.
NVIDIA has about 2,000 postings. They are on Workday.

So "Workable, 0 roles" is true and completely misleading. If you return that
bare, your caller reads "NVIDIA is not hiring", which is the same lie a 404
tells when you read it as "nothing found". Return the emptiness as a fact of
its own:

best = max(hits, key=lambda h: len(h[2]))
result = {"ats": best[0], "token": best[1], "isEmpty": len(best[2]) == 0,
          "alternatives": [h[:2] for h in hits if h is not best]}

And Workday cannot be guessed at all. Its URLs look like
https://nvidia.wd5.myworkdayjobs.com/NVIDIAExternalCareerSite: a tenant, a
numbered data centre and a site name, none of which follow from the domain.
For Workday you need the careers URL itself. There is no trick here, and any
library that claims to find Workday from a domain is guessing.

If you would rather not run it yourself

I maintain an index built this way: 53,000 company boards across 23 ATS
platforms, 2,442,000 open roles, rebuilt nightly, no login and no company list
needed. It is on Apify as
ATS Jobs Search API, and the
single-platform ones are
Greenhouse,
Workday,
Ashby,
Workable and
BambooHR.

If you would rather look at what is in there before wiring anything up, the index is browsable at glitchbound-jobs.pages.dev, including a page for this exact problem.

The 30-company and 90-domain numbers above are from building it. If you hit a
case where guess-and-verify fails and the careers page would have worked, I
would genuinely like to know which company.

Find companies that are hiring, and how to reach them, without touching LinkedIn

Daniel Meshulam — Thu, 06 Aug 2026 11:34:20 +0000

Every B2B prospecting tool sells you the same thing: a list of companies, and maybe an email pattern. The problem was never the list. It is knowing which companies to call this week, and almost nothing in a static database tells you that.

Hiring does. A company that opened 40 roles this month has budget and is changing shape. A company opening its first sales, security or finance role has just changed strategy, and that is the single strongest timing signal you can get for free.

And it is genuinely free, because companies publish their openings through APIs designed to be read. Greenhouse, Ashby, Workable, Recruitee, Personio, Rippling and BambooHR each expose a public JSON or XML endpoint per company: no login, no cookies, no proxy.

curl -s "https://boards-api.greenhouse.io/v1/boards/stripe/jobs" | head -c 300

That is the whole trick, and it is why job aggregators exist at all.

The part everyone gets wrong

The per-company endpoint needs the company's board token. There is no directory of tokens. So "who is hiring for sales in Berlin" is unanswerable from these APIs directly: you would need the token of every company that might be, which is the list you were trying to build.

You have to invert it. Enumerate the boards first, then read all of them on a schedule. I run that index: 11,157 companies across those seven platforms, re-read every night.

Two things a live scraper structurally cannot give you

This is the part I actually want to write about, because it took me a while to see it.

1. You cannot see a job close if you were not watching when it opened.

A scraper that runs today sees today's postings. It cannot tell you that a company closed 30 roles this month, because "closed" is the absence of something it never recorded. Only an index with history has that, and it is the difference between a growth signal and a churn warning:

company              opened30   closed30   net
growing co               +48        -6     +42
shrinking co              +3       -37     -34

The second row is a customer about to churn, or a competitor in trouble, depending on who you are. No amount of scraping harder produces it.

2. "First time hiring for X" needs a baseline.

Right now 3,486 of the 11,157 companies are hiring into a department they were not hiring into before. You can only know that by having watched what they hired for previously.

company     opened30   just started hiring for
Bjak           +1230   A1 CEO Office, A1 Engineering, A1 HR
Renuity         +362   Corporate, Field
Mindrift        +274   Creator (Writer)
OpenAI          +270   Applied AI, Communications, Consumer Product

Getting the contact details, honestly

A hiring signal without a way to reach the company is half a product. The approach I settled on is deliberately narrow: fetch the company's own website and take only what it publishes as a way to be contacted. A role address like info@ or sales@, a phone number inside a tel: link, its own social profiles.

No guessed first.last@ patterns, no permutation, no verification service, no named individuals. Partly because guessing produces bounces, mostly because a person's work address was published so they could do their job, not so a stranger could buy it in a list.

That honesty has a price and it shows up as a number: 6,634 of 11,157 companies are reachable this way, with every one of them crawled. If a vendor tells you they have 95% coverage of decision-maker emails, they are permuting names against a mail server, and you are buying a bounce rate.

Two implementation notes that cost me time:

tel: links only. My first pass regexed phone-shaped strings out of the page and collected SVG path coordinates, longitudes and unix timestamps. A tel: href is the publisher asserting "this is a phone number".
Role allowlist, not a blocklist. Filtering out obvious personal addresses still let sarah@ through. Allowing only a known set of role mailboxes is the version that holds.

Absent is not zero

The rule that ended up mattering most: a company we have not crawled yet and a company whose site publishes nothing must never look the same. They send you to two different places. So every row carries contactsHarvested, and closed30 is absent rather than 0 until there is enough history to know the difference between "closed nothing" and "we do not know yet".

Same for salary: two of the seven platforms publish a band, so 92% of rows have no salary and say so, rather than carrying a plausible guess.

Try it

The index is on Apify: https://apify.com/glitchbound/b2b-leads-finder

{
  "enteringNewDepartments": true,
  "minOpenedLast30": 20,
  "withEmailOnly": true
}

One row per company, with the hiring signal and the contact details. The jobs themselves are a separate Actor if you want the postings instead: https://apify.com/glitchbound/ats-jobs-search

Happy to answer questions about any of the seven APIs in the comments. If you are pulling job data today, I am curious what you are using and what breaks.