DEV Community: Daniel Meshulam

Merging job postings from ten ATS platforms into one schema, and the four fields that fight back

Daniel Meshulam — Sat, 08 Aug 2026 11:56:22 +0000

Reading one ATS job board API is easy. Every one of them is public JSON with
no key. Reading ten and getting rows you can actually query together is a
different job, and it is almost entirely about four fields.

I run an index across Greenhouse, Ashby, Workable, Workday, BambooHR, Breezy,
Teamtailor, Personio, Recruitee and Rippling. Here is what actually broke,
with the measurements.

1. Remote is not a boolean, and one API will lie to you about it

Ashby postings carry isRemote. It looks like exactly the field you want.

Measured on Ramp's board, 1 August 2026: isRemote was true for 117 of 126
postings, while the same postings' workplaceType said Hybrid 101, Remote
16, OnSite 9.

Ashby's isRemote means "not fully on-site". Trust it and a user asking for
remote work gets a hundred hybrid roles at a New York office.

workplaceType is the honest field, and it has three values, because hybrid
is a real third state that a boolean cannot hold. Normalise to the three-way
enum and derive the boolean from it, never the reverse:

workplace_type = raw.get("workplaceType")            # Remote | Hybrid | OnSite
is_remote = (workplace_type == "Remote") if workplace_type else None

Note the None. A platform that publishes no arrangement at all gets unknown,
not False. "We do not know" and "we know it is not remote" are different
answers and collapsing them is how a filter silently drops good rows.

2. Dates arrive in three shapes, one of which is relative

ISO 8601 from some. Epoch milliseconds from others. And Workday hands you a
human sentence: "Posted 4 Days Ago", "Posted Today",
"Posted 30+ Days Ago".

Epoch handling has a cheap tell:

# Milliseconds since the epoch are 13 digits until the year 2286.
secs = n / 1000 if n > 10_000_000_000 else n

The relative ones need parsing, and one of them must be refused:

_AGO = re.compile(r"^posted\s+(\d+)\s+days?\s+ago$", re.I)

def workday_posted(text):
    t = (text or "").strip().lower()
    if t == "posted today":     days = 0
    elif t == "posted yesterday": days = 1
    else:
        m = _AGO.match(t)
        if not m:
            return None          # "posted 30+ days ago" lands here, on purpose
        days = int(m.group(1))
    return (datetime.now(timezone.utc).date() - timedelta(days=days)).isoformat()

"Posted 30+ Days Ago" returns None deliberately. It could be 31 days or

Turning it into "31 days" produces a date that sorts, filters and charts perfectly and is fiction. Keep the original string in a separate field and let the caller see what the source actually said.

3. Employment type can vary between two boards on the SAME platform

This one surprised me. It is not just that platforms disagree with each other.
Workday's timeType is tenant-configured, so two customers of the same ATS
publish different things.

Measured 2026-08-08 across ten Workday boards: two publish timeType on
every posting, and eight publish it on none. The only value that appeared at
all was "Full time".

So "does this platform have employment type" is not a question with an answer,
and any per-platform lookup table encodes a guess. Read the field where the
board provides it, map it through one shared vocabulary, and leave it null
everywhere else. I had this branch hard-coded to None for a while, which
meant 258,876 rows carried no employment type even where Workday had plainly
stated one.

The trap on the other side is a filter for "full-time" that silently excludes
the majority of your index, because most rows are null rather than false.
Filter on != 'part-time' if you mean "not part-time", and say in your docs
which one you did.

4. Company name is often not a company name

Several platforms return the board token where you expect the employer, so you
get acmecorp instead of Acme Corp, and on Workday you can get the hostname.
Pull the display name from the board envelope rather than the posting, and if
it is absent, say so instead of shipping a slug that looks like a name.

The rule underneath all four

Every one of these bugs has the same shape: a field existed, so it got
trusted. isRemote exists. A parsed "30+ days" exists. A board token
exists where a company name goes. Each one produces a value that is
well-formed, sorts correctly, and is wrong.

The fix is always the same and it is not technical. Carry unknown as unknown.
Three-state where reality has three states. Null where the source said nothing.
It makes your schema uglier and your answers true, and in this data the null
rate is itself information: it tells you which platform to stop promising
things about.

The index, if you want the rows rather than the code

24,280 company boards, 10 ATS platforms, 688,711 open roles in one schema,
rebuilt nightly, no login and no company list. On Apify as
ATS Jobs Search API, or
single-platform:
Greenhouse,
Workday,
Ashby,
Workable,
BambooHR.

If you want to see the merged fields on real rows before deciding whether the schema is worth it, every vertical page at glitchbound-jobs.pages.dev is built straight from it: game dev, nursing, devops.

Posted date is populated on 76.2% of rows and country on 78.0%. Those are the
real numbers, not the ones I would like, and the missing quarter is mostly
Workday's "30+ days" refusing to become a date.

How to know which job postings were removed, when nothing tells you

Daniel Meshulam — Sat, 08 Aug 2026 11:53:08 +0000

Every ATS job board API answers the same question: what is open right now.
Greenhouse, Ashby, Workable, Workday, all of them hand you the current list.

None of them tell you what disappeared. There is no closed_at, no
deletions feed, no tombstone. A role that was filled yesterday is simply not
in today's response, indistinguishable from a role that never existed.

And disappearance is usually the more interesting half. A company that opened
40 roles and closed 38 is not the same company as one that opened 40 and
closed none, and the open-roles endpoint reports both as 40.

The mechanic is a diff, and it is not hard

You cannot ask for closures. You have to have been watching.

CREATE TABLE seen (
    platform TEXT, boardToken TEXT, jobId TEXT,
    firstSeen TEXT,          -- first run this job appeared in
    lastSeen  TEXT,          -- most recent run it appeared in
    closedOn  TEXT,          -- set when it stops appearing
    PRIMARY KEY (platform, boardToken, jobId)
);

Each run: upsert everything you fetched with today's date in lastSeen, then
mark the rest closed.

UPDATE seen SET closedOn = :today
 WHERE closedOn IS NULL
   AND lastSeen < :today
   AND boardToken IN (SELECT boardToken FROM fetched_ok_today);

That last line is the one people leave out, and it is the difference between
a closure feed and a random number generator.

The three ways this goes wrong

A failed fetch is not a closure. If a board 500s, times out, or you get
rate limited, every job on it vanishes from your response. Diff naively and
you have just recorded 2,000 closures at one company on the day their CDN had
a bad afternoon. Only diff boards that answered successfully this run.
Everything else keeps yesterday's state untouched.

A paginated board that stopped early is the same bug wearing a hat. If you
read 3 of 7 pages and treat the result as the complete current state, pages
4 to 7 all "closed". Track per-board fetch completeness, not just HTTP 200.

You cannot know how long a job was open if you started watching after it
was posted. This is the subtle one. A job you first saw on the day you
started watching may have been open for a week or a year. Computing
daysOpen = closedOn - firstSeen on that row invents a number.

The fix is to record it as unknown rather than compute it anyway:

board_first_seen = min_first_seen_for(platform, board_token)
knowable = job.first_seen > board_first_seen   # strictly after, not equal
days_open = (closed_on - job.first_seen).days if knowable else None

"Open for 3 days" and "we do not know how long it was open" must never look
the same, because one of them is a signal about the company and the other is
an artefact of when you happened to start.

The same discipline applies upward: a company's closed30 should be absent
until you have watched it for 30 days, not 0. "Closed nothing" and "we have
not been looking long enough" send a reader to two different conclusions.

What it gets you once it runs

Closures turn a job board into a time series. Net change per company. Roles
that close in four days, which usually means an internal candidate. Roles that
sit open for six months, which usually means the requisition is a placeholder.
A company whose closures suddenly outrun its openings, which is a churn signal
worth a phone call if they are your customer.

None of that is visible in the endpoint everyone reads.

If you would rather not run it for a month first

That is the real cost here: this data does not exist until you have collected
it, and the first month produces nothing.

I have been running the diff across 24,280 company boards on 10 ATS platforms,
and the index currently holds 688,711 open roles and 29,027 recorded
closures, rebuilt nightly. It is on Apify as
ATS Jobs Search API, with
company-level opening and closing counts in
B2B Leads Finder.

The closure counts are readable without an account or a key at glitchbound-jobs.pages.dev, which has a page on closed roles rebuilt from the same nightly diff.

daysOpen is null wherever it is not knowable, for the reason above. If you
find a row where that rule looks wrong, tell me, because that is the exact
mistake this is trying not to make.

How to find any company's job board from just its domain, without a list

Daniel Meshulam — Sat, 08 Aug 2026 11:48:49 +0000

Most tech companies run their careers page on an ATS: Greenhouse, Ashby,
Workable, Workday, BambooHR and a handful of others. Nearly all of those
expose a public JSON endpoint for their postings, no key and no login.

The endpoint needs a board token, though, which is the company's slug
inside that ATS. And that is the actual problem. You have stripe.com. You
need to know that Stripe is on Greenhouse under the token stripe, and every
tutorial starts one step after that, with the token already in hand.

Here is what actually works, and the measurement that says so.

The obvious approach is the worse one

Fetch the company's careers page, look for a link to boards.greenhouse.io or
jobs.ashbyhq.com, pull the token out of the URL.

Measured across 30 companies: that finds the board 50% of the time.

It fails on exactly the companies you most want. Stripe's careers page is a
JavaScript application that names no board anywhere in the delivered HTML.
Render it in a headless browser and you have turned a 200ms JSON fetch into a
multi-second browser run, for a token that is five characters long.

Guess the token, then verify it

The better approach inverts it. Derive candidate tokens from the domain, then
ask each ATS whether that board exists. On these platforms a bogus token
returns 404 and a real one returns 200 with the postings, so a 200 is
proof. You never need the careers page at all.

import httpx

BOARDS = {
    "greenhouse": "https://boards-api.greenhouse.io/v1/boards/{t}/jobs",
    "ashby":      "https://api.ashbyhq.com/posting-api/job-board/{t}",
    "workable":   "https://apply.workable.com/api/v1/widget/accounts/{t}?details=true",
    "bamboohr":   "https://{t}.bamboohr.com/careers/list",
}

def resolve(domain: str):
    token = domain.split("//")[-1].split("/")[0].split(".")[-2]
    hits = []
    for ats, url in BOARDS.items():
        r = httpx.get(url.format(t=token), timeout=15)
        if r.status_code == 200:                 # a 200 here is proof
            hits.append((ats, token, r.json()))
    return hits

Measured on the same 30 companies: 90%, including Stripe, whose Greenhouse
board answers with more than 500 open roles the careers page never mentioned.
(Check it yourself: boards-api.greenhouse.io/v1/boards/stripe/jobs. The
count moves daily, which is rather the point.)

One attempt per guess, not a retry ladder. A guessed token either exists or it
does not, and retrying every wrong guess three times is what turns a sweep of
90 domains into 25 seconds each.

The part that will bite you

Two failure modes look identical to a naive script, and both produce a
confident wrong answer.

A company can run more than one ATS. Take the board with the most
postings, not the first one that answers 200. Otherwise a company with a stale
Lever board from 2023 and a live Greenhouse board resolves to the dead one.

An empty board is a real answer, and it is not the answer you want.
nvidia.com resolves to a genuine Workable account with zero open roles.
NVIDIA has about 2,000 postings. They are on Workday.

So "Workable, 0 roles" is true and completely misleading. If you return that
bare, your caller reads "NVIDIA is not hiring", which is the same lie a 404
tells when you read it as "nothing found". Return the emptiness as a fact of
its own:

best = max(hits, key=lambda h: len(h[2]))
result = {"ats": best[0], "token": best[1], "isEmpty": len(best[2]) == 0,
          "alternatives": [h[:2] for h in hits if h is not best]}

And Workday cannot be guessed at all. Its URLs look like
https://nvidia.wd5.myworkdayjobs.com/NVIDIAExternalCareerSite: a tenant, a
numbered data centre and a site name, none of which follow from the domain.
For Workday you need the careers URL itself. There is no trick here, and any
library that claims to find Workday from a domain is guessing.

If you would rather not run it yourself

I maintain an index built this way: 24,280 company boards across 10 ATS
platforms, 688,711 open roles, rebuilt nightly, no login and no company list
needed. It is on Apify as
ATS Jobs Search API, and the
single-platform ones are
Greenhouse,
Workday,
Ashby,
Workable and
BambooHR.

If you would rather look at what is in there before wiring anything up, the index is browsable at glitchbound-jobs.pages.dev, including a page for this exact problem.

The 30-company and 90-domain numbers above are from building it. If you hit a
case where guess-and-verify fails and the careers page would have worked, I
would genuinely like to know which company.

Find companies that are hiring, and how to reach them, without touching LinkedIn

Daniel Meshulam — Thu, 06 Aug 2026 11:34:20 +0000

Every B2B prospecting tool sells you the same thing: a list of companies, and maybe an email pattern. The problem was never the list. It is knowing which companies to call this week, and almost nothing in a static database tells you that.

Hiring does. A company that opened 40 roles this month has budget and is changing shape. A company opening its first sales, security or finance role has just changed strategy, and that is the single strongest timing signal you can get for free.

And it is genuinely free, because companies publish their openings through APIs designed to be read. Greenhouse, Ashby, Workable, Recruitee, Personio, Rippling and BambooHR each expose a public JSON or XML endpoint per company: no login, no cookies, no proxy.

curl -s "https://boards-api.greenhouse.io/v1/boards/stripe/jobs" | head -c 300

That is the whole trick, and it is why job aggregators exist at all.

The part everyone gets wrong

The per-company endpoint needs the company's board token. There is no directory of tokens. So "who is hiring for sales in Berlin" is unanswerable from these APIs directly: you would need the token of every company that might be, which is the list you were trying to build.

You have to invert it. Enumerate the boards first, then read all of them on a schedule. I run that index: 11,157 companies across those seven platforms, re-read every night.

Two things a live scraper structurally cannot give you

This is the part I actually want to write about, because it took me a while to see it.

1. You cannot see a job close if you were not watching when it opened.

A scraper that runs today sees today's postings. It cannot tell you that a company closed 30 roles this month, because "closed" is the absence of something it never recorded. Only an index with history has that, and it is the difference between a growth signal and a churn warning:

company              opened30   closed30   net
growing co               +48        -6     +42
shrinking co              +3       -37     -34

The second row is a customer about to churn, or a competitor in trouble, depending on who you are. No amount of scraping harder produces it.

2. "First time hiring for X" needs a baseline.

Right now 3,486 of the 11,157 companies are hiring into a department they were not hiring into before. You can only know that by having watched what they hired for previously.

company     opened30   just started hiring for
Bjak           +1230   A1 CEO Office, A1 Engineering, A1 HR
Renuity         +362   Corporate, Field
Mindrift        +274   Creator (Writer)
OpenAI          +270   Applied AI, Communications, Consumer Product

Getting the contact details, honestly

A hiring signal without a way to reach the company is half a product. The approach I settled on is deliberately narrow: fetch the company's own website and take only what it publishes as a way to be contacted. A role address like info@ or sales@, a phone number inside a tel: link, its own social profiles.

No guessed first.last@ patterns, no permutation, no verification service, no named individuals. Partly because guessing produces bounces, mostly because a person's work address was published so they could do their job, not so a stranger could buy it in a list.

That honesty has a price and it shows up as a number: 6,634 of 11,157 companies are reachable this way, with every one of them crawled. If a vendor tells you they have 95% coverage of decision-maker emails, they are permuting names against a mail server, and you are buying a bounce rate.

Two implementation notes that cost me time:

tel: links only. My first pass regexed phone-shaped strings out of the page and collected SVG path coordinates, longitudes and unix timestamps. A tel: href is the publisher asserting "this is a phone number".
Role allowlist, not a blocklist. Filtering out obvious personal addresses still let sarah@ through. Allowing only a known set of role mailboxes is the version that holds.

Absent is not zero

The rule that ended up mattering most: a company we have not crawled yet and a company whose site publishes nothing must never look the same. They send you to two different places. So every row carries contactsHarvested, and closed30 is absent rather than 0 until there is enough history to know the difference between "closed nothing" and "we do not know yet".

Same for salary: two of the seven platforms publish a band, so 92% of rows have no salary and say so, rather than carrying a plausible guess.

Try it

The index is on Apify: https://apify.com/glitchbound/b2b-leads-finder

{
  "enteringNewDepartments": true,
  "minOpenedLast30": 20,
  "withEmailOnly": true
}

One row per company, with the hiring signal and the contact details. The jobs themselves are a separate Actor if you want the postings instead: https://apify.com/glitchbound/ats-jobs-search

Happy to answer questions about any of the seven APIs in the comments. If you are pulling job data today, I am curious what you are using and what breaks.

Search every Greenhouse, Ashby and Workable job board at once, without a company list

Daniel Meshulam — Wed, 05 Aug 2026 20:12:35 +0000

Every guide to scraping job boards starts the same way: "first, get the company's board token." Greenhouse, Ashby and Workable each expose a public JSON API per company, so if you know the token, one HTTP call returns every open role. Clean, documented, no login.

The problem is the token. There is no directory of them. If you want "senior backend roles in Berlin" across the whole ecosystem, you would need the board token of every company that might be hiring, which is exactly the list you were hoping the data would give you. The per-company APIs answer "what is Stripe hiring?" but not "who is hiring?"

I run a nightly index that solves the inversion: it already knows 7,131 boards across Greenhouse, Ashby and Workable, and re-reads them every night. 230,508 open roles at the moment I write this. You query it like a search engine instead of crawling anything yourself.

One call, no company list

curl -s "https://api.apify.com/v2/acts/glitchbound~ats-jobs-search/run-sync-get-dataset-items?token=$APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "title": ["backend engineer*", "platform engineer*"],
    "countries": ["Germany"],
    "workArrangements": ["remote"],
    "postedAfter": "2026-07-01",
    "maxResults": 100
  }'

You get one JSON row per job: title, company, location, the real apply URL on the employer's own site, department, posting date, and salary where the employer published one. A trailing * does prefix matching, a multi-word line is a phrase, and countries is resolved rather than text-matched, so German roles arrive whether the board wrote Berlin, Germany, Munich, DE or DE - Berlin.

The same thing in Python, with the Apify client:

from apify_client import ApifyClient

client = ApifyClient(token="...")
run = client.actor("glitchbound/ats-jobs-search").call(run_input={
    "title": ["data engineer*"],
    "countries": ["Netherlands", "Germany"],
    "seniority": ["senior", "staff"],
    "maxResults": 200,
})
for job in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(job["company"], "|", job["title"], "|", job["url"])

Why this is not "just scrape LinkedIn"

LinkedIn scrapers need an account, and accounts get banned. This index reads the same employer-published APIs that job aggregators are invited to read: no login, no cookies, no proxy, and the apply link goes to the employer's own careers page rather than through an aggregator.

The trade-off is coverage. It holds companies that run their hiring on Greenhouse, Ashby or Workable, which is most of tech, and it does not hold the long tail of companies that only post to LinkedIn. Lever is missing too: their robots.txt blocks the crawler the index is seeded from, and a partial Lever list would be worse than none.

The part I have not seen elsewhere: companies as the row

Flip mode to companies and the same index answers a different question: who is hiring hardest right now?

{
  "mode": "companies",
  "hiringLabels": ["surging", "growing"],
  "enteringNewDepartments": true
}

Each row is a company with its open-role count, how many roles it opened in the last 30 and 7 days, and which departments are new this month. A company opening its first sales or security roles has just changed strategy. Because the index sees every board every night, it can also tell you when jobs close, which a live scraper cannot: you cannot see a posting disappear if you were not there when it existed.

Honest limits

42,894 of the roles are remote; the rest are not, and unlabelled jobs are left unlabelled rather than guessed into a bucket.
Salary appears on about 8% of jobs, from Ashby only, because Greenhouse and Workable publish no salary field at all.
The index refreshes nightly, not per-request. Follow the url field to the employer's page, which is always authoritative.

Pricing is per job returned, about a dollar per thousand, and a search that matches nothing costs nothing.

The Actor is here: https://apify.com/glitchbound/ats-jobs-search

Questions and edge cases welcome in the comments; if a filter you need is missing, say so and I will likely ship it.

Workday's job API tells you there are 2,000 jobs, then says 0 on page two

Daniel Meshulam — Sun, 02 Aug 2026 12:18:36 +0000

Workday is where large enterprises actually post. NVIDIA has 2,000 open roles
there, Salesforce 1,477, Adobe 832. It answers an anonymous POST with no key.

It also has two behaviours that are not in any documentation you can read
without an account, and both of them fail silently. One of them costs you 98%
of the board without raising anything.

The number that changes after page one

Ask for the first twenty postings and the response carries a total:

POST /wday/cxs/nvidia/NVIDIAExternalCareerSite/jobs
{"appliedFacets":{}, "limit":20, "offset":0, "searchText":""}

  20 jobPostings,  total: 2000

Ask for the next twenty and the count is gone:

offset 20   ->  20 jobPostings,  total: 0
offset 40   ->  20 jobPostings,  total: 0

Not null, not absent. Zero. The postings keep coming; only the count collapses.

Measured on four enterprise tenants:

tenant	total at offset 0	at offset 20	at offset 40
NVIDIA	2000	0	0
Salesforce	1477	0	0
Adobe	832	0	0
Sony	94	0	0

Same shape every time, so this is Workday and not one tenant's configuration.

Why that costs you 98% of the board

Here is the loop everyone writes, and it is not a bad loop:

offset, out = 0, []
while True:
    page = fetch(offset)
    posts = page["jobPostings"]
    if not posts:
        break
    out += posts
    offset += len(posts)
    if offset >= page["total"]:      # looks obviously right
        break

On page two page["total"] is 0, and 20 >= 0 is true. The loop exits,
reports no error, and hands back what it has.

I ran both versions against NVIDIA:

declared total on page one              2000
the obvious loop collected                40      2%
keeping the first total instead         2000    100%

Forty postings out of two thousand, and nothing anywhere says so. No exception,
no warning, no partial-result flag. Just a job board that looks very quiet.

The fix is one line moved:

offset, out, total = 0, [], None
while True:
    page = fetch(offset)
    posts = page["jobPostings"]
    if not posts:
        break
    out += posts
    offset += len(posts)
    if total is None:                # the first answer is the only honest one
        total = page.get("total") or 0
    if total and offset >= total:
        break

Two changes that matter. Keep the first total. And guard on total being
truthy, so a board that never reports a usable count still runs to the empty
page instead of stopping immediately.

The page size is capped at 20, and 21 is a 400

The other undocumented one. limit is not a suggestion:

limit=20  ->  HTTP 200
limit=21  ->  HTTP 400
limit=25  ->  HTTP 400
limit=50  ->  HTTP 400

Exactly twenty. And the failure arrives as a 400, which reads as "your request
body is malformed" rather than "that number is too big", so the natural
reaction is to go looking at your JSON.

Two thousand postings at twenty a page is a hundred round trips. That is the
price, and there is no parameter that lowers it.

You cannot build the URL from a company name

A Workday address carries three separate unknowns:

https://nvidia.wd5.myworkdayjobs.com/NVIDIAExternalCareerSite
        ^^^^^^ ^^^                    ^^^^^^^^^^^^^^^^^^^^^^^
        host   data centre            career site name

The host label is usually the company, but the data centre is wd1 through
wd12 with no pattern, and the site name is whatever whoever set it up typed.
External_Career_Site, external_experienced, SonyGlobalCareers,
NVIDIAExternalCareerSite. Guessing across that space is mostly 404s.

The API path adds a fourth part that the public URL does not show:

public   https://nvidia.wd5.myworkdayjobs.com/NVIDIAExternalCareerSite
api      https://nvidia.wd5.myworkdayjobs.com/wday/cxs/nvidia/NVIDIAExternalCareerSite/jobs
                                              ^^^^^^^^^^^^^^^

On all four tenants I checked, that middle segment is the host label again.
Build it with a literal None in there and you get a 422, not a 404, which
at least fails in a way that does not look like a wrong company name.

So take the careers URL from the customer. It is the one thing they always
have, and it is the only reliable source for all three parts.

What the postings look like

Sparse, and worth knowing before you design a schema around it:

{
  "title": "Distinguished Software Architect - Deep Learning",
  "externalPath": "/job/US-CA-Santa-Clara/...",
  "locationsText": "US, CA, Santa Clara",
  "postedOn": "Posted Today",
  "bulletFields": ["JR2001234"]
}

Five keys. No department, no employment type, no salary, no description, and
postedOn is a relative phrase rather than a date.

Counted across 80 NVIDIA postings:

Posted 30+ Days Ago    20
Posted 5 Days Ago      13
Posted 13 Days Ago     13
Posted 2 Days Ago      12
Posted Today            5
Posted Yesterday        3

Do not parse that into a timestamp. A quarter of them are "30+ Days Ago",
which has no date in it at all, and a timestamp you invented is worse than a
string you kept: the string is obviously approximate and the timestamp is not.
Give it its own field and let the consumer decide.

Measured on 2026-08-02. Four tenants carried a usable count: NVIDIA,
Salesforce, Adobe and Sony. Dell reported 0 at offset 0 as well, which is the
third trap below rather than a fifth data point. I
maintain a set of ATS scrapers that normalise
nine of these systems into one row shape, which is how I ran into all of it.

Ashby's isRemote flag is true for hybrid roles. I checked 1,668 postings.

Daniel Meshulam — Sun, 02 Aug 2026 11:59:21 +0000

If you build anything on Ashby's public job board API, there is one field that
will quietly ruin your filter.

isRemote is a boolean. It is not the boolean you think.

I pulled every posting from ten real Ashby boards on 2026-08-02, 1,668 in
total, and counted:

	postings	share
`isRemote: true`	1,072	64%
`workplaceType: "Remote"`	295	18%

Filter on the boolean and you get three and a half times as many roles as are
actually remote.

It is not noise, it is a definition

Cross-tabulating the two fields across nine of those boards, 1,637 postings,
leaves no ambiguity at all:

workplaceType    isRemote    postings
Hybrid           true             776
Remote           true             295
OnSite           false            271
(absent)         false            295

Zero exceptions in 1,637 rows. isRemote is exactly
workplaceType != "OnSite", with an absent type counting as on-site.

So the field is honest about what it computes and misleading about what it is
called. It means not fully on-site. Hybrid roles, 47% of everything I
pulled, all carry it.

The per-company spread is the part that bites

An average of 3.6x would be survivable. The variance is not:

board	postings	`isRemote` true	actually Remote	overstated
openai	754	485	32	15.2x
replit	88	88	6	14.7x
ramp	126	117	16	7.3x
notion	109	74	0	everything
cohere	142	131	100	1.3x
vanta	100	90	63	1.4x
cursor	120	47	41	1.1x
sierra	175	16	15	1.1x
linear	23	23	22	1.0x

Linear is honest by accident: nearly all its roles genuinely are remote, so the
two fields agree. Notion is the opposite extreme. It has zero roles marked
Remote and 74 postings flagged isRemote: true, because Notion is a hybrid
company and every hybrid role trips the flag.

Which means any sanity check you run on one company can pass and tell you
nothing. Test on Linear and the field looks perfect. Test on Notion and every
result is wrong.

Three states, not two

The real fix is not a better boolean, it is admitting that this is not a
boolean question. workplaceType has three values, and the third one is the
one people actually argue about:

Remote   295
Hybrid   776
OnSite   271

Collapsing that into yes-or-no has to lie in one direction or the other.
Ashby's boolean lies toward remote; a boolean built the other way would tell
776 people a hybrid role is on-site, which is equally wrong and would just
annoy a different group.

If you are normalising job data across sources, carry all three. A row that
says Hybrid is more useful than a row that says true, and it is the only
version you can aggregate honestly.

The 295 with nothing set

There is a fourth group worth knowing about: 295 postings, 18% of that
cross-tab, carry no workplaceType at all. They all come back isRemote: false.

That is a company that never filled the field, being reported as not remote. It
might be. Nobody said so. If you are ranking or filtering, treat an absent
workplace type as unknown rather than as a no, because the API has already made
that judgement for you and it had nothing to go on.

Getting the field at all

One more trap, unrelated but in the same request. Ashby omits compensation
entirely unless you ask:

GET /posting-api/job-board/ramp                            no compensation key
GET /posting-api/job-board/ramp?includeCompensation=true   "$211.4K - $290.6K"

I had that parameter wired to a description flag in my own client for a while,
which meant every run that did not ask for descriptions silently dropped
salary. Worth reading a response you are sure you already understand.

What I would actually do

def workplace(posting):
    """Three states, and unknown is not a no."""
    wt = posting.get("workplaceType")
    if wt in ("Remote", "Hybrid", "OnSite"):
        return wt
    return "Unknown"          # NOT False, and not isRemote

Four lines, and it is right on all 1,668.

None of this is Ashby's fault exactly. isRemote is doing what it says in the
code that produced it. It is just that the name promises an answer to a
question the data cannot answer, and a boolean will always find someone willing
to believe it.

Every number here is from the public job board API, no key, and reproducible:
api.ashbyhq.com/posting-api/job-board/<board>?includeCompensation=true. I
maintain a set of ATS scrapers that normalise
this across nine systems, which is how I ran into it.

Indeed 403s you with zero bytes. The companies it aggregates publish free APIs.

Daniel Meshulam — Sat, 01 Aug 2026 19:30:44 +0000

I spent an afternoon measuring which job sources actually answer a request, and
the result was cleaner than I expected: the aggregators are shut and the
sources they aggregate from are wide open.

Everything below was measured on 2026-08-01 from an ordinary residential
connection with normal browser headers. Not read from documentation.

The aggregators

indeed.com/jobs          403      0 bytes
glassdoor.com/Job/...    403      0 bytes
ziprecruiter.com         403      0 bytes
upwork.com/nx/search     403      0 bytes

Zero bytes. Cloudflare rejects these before the request reaches an application,
so there is nothing to parse and nothing to be clever about. That is from a
residential address, which is the friendliest IP you can bring. A datacenter
address does worse.

Reddit is worth a mention because it fails in a more interesting way:

reddit.com/r/programming/hot.json    403   189,908 bytes
old.reddit.com/.../hot.json          403   189,908 bytes
reddit.com/r/programming/.rss        200    40,349 bytes   <- once
reddit.com/r/programming/.rss        429         0 bytes   <- and every time after

Note the 189,908 bytes on a 403. If you check len(response.content) instead of
response.status_code, that block page reads as a large successful response. I
have written that bug. It is the most expensive one-line mistake in scraping,
because it does not fail, it lies.

The sources

Indeed does not originate job data. Neither does Glassdoor, ZipRecruiter or
LinkedIn Jobs. They aggregate from company career pages, and those career pages
run on applicant tracking systems. Every major one publishes a public JSON API
with no key, because companies want their openings indexed.

GET https://boards-api.greenhouse.io/v1/boards/stripe/jobs
200    548 jobs

GET https://api.ashbyhq.com/posting-api/job-board/openai
200    753 jobs

GET https://api.lever.co/v0/postings/palantir?mode=json
200    302 jobs

GET https://api.rippling.com/platform/api/ats/v1/board/rippling/jobs
200    768 jobs

Also open: Workable, Recruitee, SmartRecruiters, Personio. No key, no proxy, no
bot wall, and fresher than any aggregator, because an aggregator shows you its
last crawl and this shows you the board.

One distinction worth making, because these are not all the same thing.
Greenhouse, Lever, Ashby, Recruitee and SmartRecruiters document these
endpoints publicly as needing no authentication. Greenhouse says it plainly:
"Job Board data is publicly available, so authentication is not required for any
GET endpoints." Lever goes further and says published postings "may be scraped
by third parties."

Rippling and Workable are different. Rippling's developer documentation sits
behind a login, and Workable's documented API is a separate authenticated one.
The endpoints I used for those two are the ones their own public careers pages
call, they answer an anonymous request, and they serve postings a company chose
to publish. But "answers without a key" is not the same claim as "the vendor
documents it as public", and I would rather say which is which than let the
list imply all seven are equivalent.

Trap 1: you need the board token, and there is no directory

Every one of those URLs contains a token: stripe, openai, palantir. There
is no public mapping from a company to its token. This is the actual reason
people reach for Indeed instead.

The obvious fix is to fetch the company's careers page and look for an ATS link.
I tried that on 12 companies and it found 6. Stripe's careers page is a
JavaScript app that names no board anywhere in the HTML.

The fix that works is the other way round: derive candidates from the domain
and verify each against the live API.

def tokens(domain):
    stem = domain.split("//")[-1].split("/")[0].removeprefix("www.").split(".")[0]
    return [stem, stem.replace("-", "")]

PROBES = {
    "greenhouse": "https://boards-api.greenhouse.io/v1/boards/{}/jobs",
    "lever":      "https://api.lever.co/v0/postings/{}?mode=json",
    "ashby":      "https://api.ashbyhq.com/posting-api/job-board/{}",
    "recruitee":  "https://{}.recruitee.com/api/offers/",
    "workable":   "https://apply.workable.com/api/v1/widget/accounts/{}?details=true",
    "rippling":   "https://api.rippling.com/platform/api/ats/v1/board/{}/jobs",
}

stripe.com gives stripe, and boards-api.greenhouse.io/v1/boards/stripe/jobs
returns 548 jobs. Both methods, measured:

Method	Sample	Resolved
Read the careers page for a link	12 companies	50%
Derive the token and verify it	30 companies	90%

Trap 2: why guessing is sound here, and where it stops being sound

Guessing is only safe if a wrong guess is distinguishable from a right one. I
checked each API with a real token and a deliberately bogus one:

greenhouse   real 200      bogus 404
lever        real 200      bogus 404
ashby        real 200      bogus 404
recruitee    real 200      bogus 404
workable     real 200      bogus 404
rippling     real 200      bogus 404

A 404 for an unknown board is what makes a 200 proof rather than a maybe.
And a real board with nothing open is a third, distinct state:

ashby/deel     200   {"jobs":[],"apiVersion":"1"}
lever/kraken   200   []

So 404 means no such board, 200 [] means the board exists and is empty.
Clean semantics, and they let you tell "wrong token" apart from "not hiring".

SmartRecruiters is the exception, and it is worth knowing before you trust it:

smartrecruiters  real company Visa            200  {"totalFound":2,   "content":[...]}
smartrecruiters  company with nothing open    200  {"totalFound":0,   "content":[]}
smartrecruiters  company that does not exist  200  {"totalFound":0,   "content":[]}

The last two are byte-identical. The obvious second signal does not rescue it
either: GET /v1/companies/Visa returns 404, for a company whose postings
endpoint returns real jobs. So on SmartRecruiters there is no way to confirm a
guessed id, and the honest move is to require an exact one rather than emit a
confident zero. Company ids there are also case sensitive: Visa works, visa
does not.

This is the same shape as a family of bugs I keep meeting: an absence being
returned as an answer. RDAP does it too, where a 404 means "no server for this
TLD" and gets read as "domain available".

Trap 3: descriptions cost 18x, and change the schema

Greenhouse takes ?content=true. Same board, measured:

boards-api.greenhouse.io/v1/boards/figma/jobs                  111 KB
boards-api.greenhouse.io/v1/boards/figma/jobs?content=true    1.97 MB

Eighteen times the payload. What is less obvious is that departments and
offices only exist when content is on, so if you want the department of a
role you are paying for every description whether you wanted them or not.

Rippling goes the other way: its board API returns five fields and no
description at any setting. If you build a uniform schema across platforms, that
field is empty because the source has none, not because you missed it. Worth
saying in your output rather than shipping a null that reads as a bug.

What you get for the trouble

One request per company, no key, no proxy, and data that has not been through
somebody else's crawl schedule. Stripe's 548 roles, Databricks' 803, OpenAI's

Titles, locations, departments, offices, descriptions.

And on Ashby, salary. Of the seven JSON sources it is the only one that
publishes a compensation band, and it hides it behind a parameter:

GET /posting-api/job-board/ramp                             no compensation key
GET /posting-api/job-board/ramp?includeCompensation=true    "$211.4K - $290.6K"

Every Ramp and OpenAI posting carries a real band. Notion and Linear return the
field full of nulls, which is a company choosing not to publish rather than a
missing feature, and the two are worth telling apart.

Correction, added after publishing. I first wrote that Ashby was the only
one of the eight sources here with salary. That is wrong: Personio, the XML one,
publishes it too, and in a better shape. Ashby gives you a formatted string,
"$211.4K - $290.6K". Personio gives you the numbers:

<salaryInformation><min>45000.00</min><max>55000.00</max></salaryInformation>
<currencyCode>EUR</currencyCode><currencySymbol>&#8364;</currencySymbol>

Structured min, max and currency, which is what you want if you are sorting or
filtering rather than displaying. Personio is a European mid-market system, so
this is worth knowing if that is your segment. I had this parameter wired
to the description flag in my own client and was silently dropping salary on
every run that did not ask for descriptions, which is a good argument for
reading a response you think you already understand.

The aggregators are not gatekeepers of this data. They are readers of it, same
as you, and they are the only ones who have to be let in.

Every endpoint in this post is public and the measurements stand on their own,
so nothing here needs a product. I am packaging the eight clients and the domain
resolver as Actors on the Apify Store under
glitchbound, which is also where my other
scrapers live if this is the kind of thing you use.

Google's News API has been gone since 2016, and people still search for it

Daniel Meshulam — Thu, 30 Jul 2026 18:44:36 +0000

Search for "google news api" and a striking share of the results are people
looking for something that no longer exists. Google deprecated the News Search
API in May 2011 as part of a wider AJAX Search API cull, kept it limping until
the final shutdown in February 2016, and never shipped a replacement. A decade
later the searches are still there.

What almost nobody mentions is that Google News still exposes a perfectly good
public endpoint. It is just an RSS feed rather than a JSON API, it needs no key,
and it has quietly outlived the API that was supposed to be the real product.

The endpoint

https://news.google.com/rss/search?q=<query>&hl=en-US&gl=US&ceid=US:en

That is the whole thing. Swap hl, gl and ceid for a different language and
country and you get that edition instead. There is no key, no quota page, and no
sign-up.

The catches are real but small:

It is RSS, so you get XML. Title, link, publication date, source name and a short description. No article body, and no way to ask for one.
The links are Google redirects, not publisher URLs. You have to follow them if you want the destination.
The publisher name lives in a nested <source> element, not where a normal feed parser looks for it.
Roughly 100 items per query. Deeper history is not available through it.

None of that is a reason to go and pay for a news API. It is a reason to write
about forty lines once.

Two sources are better than one

Bing News publishes its own RSS in the same shape at a different URL. Running
both and merging costs one extra request and meaningfully changes what you see,
because the two indexes disagree about what is worth surfacing.

Here is a real run over both:

16 articles
sources: {'google:semiconductor export controls': 8,
          'bing:semiconductor export controls': 8}

2026-07-09  Just Security          It Takes More Than Two to Tango: Creating...
2026-07-20  Reuters                China considers tighter export controls on...
2026-03-24  CSIS                   China's Localization Drive in Semiconductors...
2026-07-28  EU Today               China's Domestic DUV Push Puts ASML's Export...
2026-06-24  Purdue University      Intellectual Property Protection and Export...
2025-11-20  Cfr.org                Protecting the Foundation: Strengthening...
2026-07-19  The Motley Fool        Semiconductor Trade Statistics: U.S. Imports...

Every row carries title, url, publisher, publishedAt, summary,
source, image and guid.

One thing worth knowing if you parse Bing's feed yourself. It carries the
publisher in <News:Source>, and declares that prefix as

xmlns:News="https://www.bing.com/news/search?q=<your query>&format=RSS"

The namespace URI is the request URL, so it changes on every query and no fixed
namespace constant will ever match it. Match on the local element name instead.
I found this the way you would expect: publisher was null on exactly the Bing
half of a two-source run while the value was sitting in the feed the whole time.

The code

import json
import os
import urllib.request

ACTOR = "glitchbound~news-scraper"
API = "https://api.apify.com/v2"


def headlines(query, per_source=10):
    payload = {
        "searchQueries": [query],
        "maxArticlesPerSource": per_source,
        "language": "en",
        "includeBingNews": True,
        "dedupeHeadlines": True,
    }
    url = (f"{API}/acts/{ACTOR}/run-sync-get-dataset-items"
           f"?token={os.environ['APIFY_TOKEN']}&maxTotalChargeUsd=0.50")
    req = urllib.request.Request(
        url, data=json.dumps(payload).encode(), method="POST",
        headers={"Content-Type": "application/json"})
    with urllib.request.urlopen(req, timeout=300) as r:
        rows = json.load(r)
    # A feed that fails comes back as a row with an `error` field rather than
    # failing the run, and those are not charged.
    return [r for r in rows if not r.get("error")]


for a in headlines("semiconductor export controls", per_source=5):
    # Fields can be absent on any given row, so read them defensively rather
    # than subscripting. Feeds are not schemas.
    when = (a.get("publishedAt") or "")[:10] or "no date"
    who = (a.get("publisher") or "unknown")[:28]
    print(f'{when:10}  {who:28} {(a.get("title") or "")[:60]}')

That runs News Scraper on
Apify, which is the same forty lines plus the parts that are
tedious rather than hard: both feeds, the nested publisher element, the date
normalisation, and deduplication.

If you would rather own the code, the endpoint above is public and this post has
told you everything you need.

Deduplication is most of the value

One wire story appears on dozens of syndicating domains, and Google and Bing will
both hand you several copies. A hundred raw results is frequently thirty stories.

Dedupe on the URL first, then on a normalised headline, because the same Reuters
piece appears under slightly different titles across outlets. Doing it on
headline alone merges genuinely different stories that share a phrasing; doing it
on URL alone leaves most of the duplication in place.

Where this does and does not fit

Fits: brand and competitor monitoring, tracking a topic across outlets,
building a corpus of headlines, seeing which publishers cover a beat, feeding a
summarisation pipeline.

Does not fit: anything needing article text, anything needing more than a few
months of history, or academic archival work. For those you want a licensed
provider, and the honest answer is that no free endpoint replaces one.

What it costs

If you write it yourself, nothing beyond your own hosting.

Through the Actor it is $1.50 per 1,000 articles on the free tier, so a daily run
pulling 50 articles is about seven cents a month, plus a $0.02 Actor-start event
per run. Duplicates that get dropped are not charged, and neither are feeds that
fail.

The wider point

An enormous amount of "you need an API for this" is habit. Google News, Apple's
app charts, the SEC, the World Bank, the ECB and Open Food Facts all publish
usable public endpoints that need no key and get very little attention, largely
because there is no vendor with an incentive to tell you about them.

Worth checking before you pay for access to public data.

Code

Runnable examples are here:
github.com/danielhagever/apify-data-actors

WHOIS is gone, RDAP replaced it, and a 404 does not mean what you think

Daniel Meshulam — Thu, 30 Jul 2026 18:02:58 +0000

If you last touched domain lookups a few years ago, the thing you remember is
WHOIS: a plaintext protocol on port 43 that returned an unparseable wall of text
in a different shape from every registry.

It is effectively gone. RDAP replaced it, it is RFC 9083, it returns JSON, and
every gTLD registry is required to run one. This is one of the rare cases where
the standards process produced something strictly better and nobody noticed.

Here is what a lookup looks like now, what you can actually get out of it after
GDPR, and the one trap that had my own tool confidently reporting registered
domains as free to buy.

What RDAP gives you, and what it does not

Post-GDPR, RDAP returns no registrant personal data for virtually every TLD. If
you are looking for who owns a domain, stop: that information is not there and
no legitimate source has it.

What is there is the operational picture, and that turns out to be the more
useful half:

registrar, registration date, expiry date, status codes, DNSSEC
whether the name is locked against transfer

Combine that with a DNS lookup and you get the thing people actually want, which
is what a company runs on:

domain        registrar                expires       age mail       dns         dmarc
stripe.com    SafeNames Ltd.           2027-09-11   30.9 google     aws-route53 reject
linear.app    CloudFlare, Inc.         2030-05-09    8.2 google     cloudflare  reject
vercel.com    Amazon Registrar, Inc.   2029-10-04   26.8 google     vercel      quarantine
github.io     -                        -               - -          aws-route53 -
angel.co      -                        -               - google     cloudflare  reject
notion.so     -                        -               - -          cloudflare  quarantine

That is a real run. Mail provider and DNS provider come from MX and NS records,
DMARC policy from a TXT lookup at _dmarc.<domain>. For sales qualification,
security auditing or competitive research, those three columns are worth more
than any registrant field ever was.

Note the bottom three rows have DNS but no registrar and no expiry. Hold that
thought.

The code

import json
import os
import urllib.request

ACTOR = "glitchbound~whois-domain-lookup"
API = "https://api.apify.com/v2"


def profile(domains):
    payload = {
        "domains": domains,
        "includeDns": True,
        "recordTypes": ["A", "MX", "NS", "TXT"],
        "includeEmailSecurity": True,
    }
    url = (f"{API}/acts/{ACTOR}/run-sync-get-dataset-items"
           f"?token={os.environ['APIFY_TOKEN']}&maxTotalChargeUsd=0.50")
    req = urllib.request.Request(
        url, data=json.dumps(payload).encode(), method="POST",
        headers={"Content-Type": "application/json"})
    with urllib.request.urlopen(req, timeout=300) as r:
        rows = json.load(r)
    # A domain that could not be looked up comes back as a row with an `error`
    # field rather than failing the run, and those are not charged.
    return [r for r in rows if not r.get("error")]


for r in profile(["stripe.com", "linear.app", "vercel.com"]):
    print(f'{r["domain"]:14} {r.get("mailProvider") or "-":10} '
          f'{r.get("dnsProvider") or "-":12} dmarc={r.get("dmarcPolicy") or "-"}')

That runs an Apify Actor I maintain,
WHOIS Domain Lookup, because the
tedious parts are the RDAP bootstrap, the vcard unpacking and the provider
fingerprinting rather than the HTTP call. If you would rather do it yourself,
https://rdap.org/domain/<name> is the whole API and it needs no key.

The trap: a 404 is ambiguous

rdap.org is a router. It reads IANA's bootstrap registry, finds the RDAP
server for your TLD, and redirects. A 404 back from it looks like it means one
thing: no registry holds this name, therefore the domain is available.

That is what my tool assumed. It was wrong, and the failure is ugly:

rdap.org/domain/github.io    404
rdap.org/domain/notion.so    404
rdap.org/domain/segment.io   404
rdap.org/domain/angel.co     404

All four are registered. All four returned 404.

The reason is that IANA's bootstrap does not cover every TLD. It lists RDAP
services for around 1,200 of them, and some of the most common startup TLDs are
not in there:

.io   .co   .de   .me   .so   .gg   .sh

For those, rdap.org has nowhere to send the query, so it 404s for every name
under them, registered or not. My Actor turned that into isAvailable: true and
would have told you github.io was free to register.

On a tool whose headline feature is availability checking, that is the worst
possible way to be wrong: confident, plausible, and acted upon. Nobody
double-checks a clear answer.

The fix, in two parts

Ask whether there is a registry at all. Fetch
https://data.iana.org/rdap/dns.json once, build the set of TLDs that have an
RDAP service, and only treat a 404 as "available" when the TLD is in it.

import json, urllib.request

def rdap_tlds():
    d = json.load(urllib.request.urlopen("https://data.iana.org/rdap/dns.json"))
    return {t.lower().lstrip(".") for entry in d["services"] for t in entry[0]}

Let DNS settle the rest. A name with NS records has been delegated by the
registry, and a registry does not delegate a name nobody registered. So for the
TLDs with no RDAP, NS records prove registration even though RDAP cannot.

The reverse does not hold, and this is the part worth being careful about: a
registered domain can have no NS at all. So "no NS and no RDAP" is not
"available", it is unknown, and it should be returned as null with an
explanation rather than as a guess. A null is a fact the caller can handle. A
plausible wrong answer is not.

Why this generalises

The bug was not really about RDAP. It was about reading an absence as a
definite answer.

404 means "I have nothing for you". Whether that is because the thing does not
exist, or because you asked somewhere that was never going to know, is not in
the response. Any time you convert a missing result into a positive claim, check
that you actually asked something capable of answering.

I have now hit the same shape four times in a month in different tools: an API
silently ignoring a filter it does not support, a page parser taking the first
number in a plausible range, this, and crt.sh, which answers 404 when it is
refusing traffic and reports "nothing found" as an empty list with a 200. All of
them produced output that looked completely normal.

What it costs

Charged per domain returned, at $2.00 per 1,000 on the free tier, so a hundred
domains is 20 cents plus a $0.02 Actor-start event. Domains removed by the
built-in filters, like onlyRegistered or expiringWithinDays, are not charged,
and neither are lookups that fail.

Notes

Some ccTLDs run thin RDAP servers that return fewer fields. You get whatever the registry publishes rather than an error.
DNS answers are returned sorted. Resolvers round-robin, and unsorted output makes a scheduled run look like it changed every time when nothing did.
Availability comes from the registry itself, not a reseller API, so there is no incentive for anyone to tell you a name is taken when it is not.

Code

Runnable examples are here:
github.com/danielhagever/apify-data-actors

Every example input in that repo is generated from the Actor's own input schema,
so it is valid by construction rather than by my memory of the field names.

The App Store top charts are a public API, and nobody archives them

Daniel Meshulam — Thu, 30 Jul 2026 16:19:14 +0000

Apple's top charts are one of the few genuinely useful public rankings left.
They are free, they need no key, and they update daily.

They also have one property that makes them frustrating: there is no history.
The feed shows you today. Yesterday is gone. If you want to know whether an app
climbed or slid, or which category a publisher quietly took over last month, that
data does not exist anywhere unless somebody was writing it down.

Nobody is writing it down, which means the archive is worth more than the feed.

This post covers both halves: reading the charts today, and the small amount of
work it takes to have a year of history in a year's time.

The feed

Apple's RSS feeds sit at a predictable URL per country, chart type and category.
No key, no login, and no bot wall. That is unusual enough in 2026 to be worth
saying out loud.

What is awkward is the shape. Each entry buries the app id under
id.attributes["im:id"], the price under im:price.attributes, the developer
under im:artist.label, and the icon in an array where you want the last
element. Parsing it correctly is 30 lines of nested .get() calls that you write
once and never enjoy.

I put that in an Apify Actor,
App Store Top Charts Scraper,
so I could stop rewriting it. One HTTP call runs it and returns flat rows.

import json
import os
import urllib.request

ACTOR = "glitchbound~app-charts-scraper"
API = "https://api.apify.com/v2"


def top_charts(countries, charts=("top-free",), limit=50):
    payload = {
        "countries": list(countries),
        "charts": list(charts),
        "categories": ["all"],
        "limit": limit,
    }
    url = (f"{API}/acts/{ACTOR}/run-sync-get-dataset-items"
           f"?token={os.environ['APIFY_TOKEN']}&maxTotalChargeUsd=0.50")
    req = urllib.request.Request(
        url, data=json.dumps(payload).encode(), method="POST",
        headers={"Content-Type": "application/json"})
    with urllib.request.urlopen(req, timeout=300) as r:
        rows = json.load(r)
    # A country or chart that fails comes back as a row with an `error` field
    # rather than failing the run, and those are not charged. Split them out or a
    # run that fetched nothing still looks successful.
    return [r for r in rows if not r.get("error")]


rows = top_charts(["us", "gb", "de"], charts=["top-free"], limit=20)
for r in sorted(rows, key=lambda r: (r["country"], r["rank"])):
    print(f'{r["country"]}  #{r["rank"]:<3} {r["title"][:42]:42} {r["developer"][:24]}')

Real output, from an actual run, top of the German chart:

DE  #1   Claude by Anthropic                        Anthropic PBC
DE  #2   ChatGPT                                    OpenAI OpCo, LLC
DE  #3   Nect Wallet                                Nect
DE  #4   Google Gemini                              Google
DE  #5   EasyPark parken - Park App                 Easy Park AS
DE  #6   AusweisApp Bund                            Governikus GmbH & Co. KG
DE  #7   Mein dm                                    dm-drogerie markt GmbH +
DE  #8   Post & DHL                                 Deutsche Post AG

That German top ten is itself the argument for archiving. Four of those eight are
government or logistics apps that nobody would predict from the US chart, and the
top two are AI assistants that were not in this chart at all two years ago. You
cannot see either of those facts in a single pull.

Every row also carries appId, bundleId, price, currency, isFree,
genre, releaseDate, icon and url, so a chart pull doubles as an app
metadata pull.

The part that is actually valuable

One pull tells you the ranking today, which you could have read off your phone.
The interesting queries all need two pulls separated by time:

Which apps entered the top 20 this week
Which publisher gained the most chart slots this month
How long a given app has held its position
Whether a competitor's launch actually moved anything
Which categories are churning and which are frozen

None of that is answerable from the feed. All of it is answerable from a table
you append to daily.

So the whole trick is: run it on a schedule and never delete anything.

import datetime as dt

rows = top_charts(["us"], charts=["top-free", "top-paid"], limit=100)
stamp = dt.datetime.now(dt.timezone.utc).isoformat(timespec="seconds")
for r in rows:
    r["capturedAt"] = stamp          # the feed has no timestamp; add your own
# append to wherever you keep things: Postgres, SQLite, a CSV, a dataset

On Apify you get this without writing the storage layer: put the Actor on a
Schedule and every run appends to the same dataset, each with its own timestamp.
Three months later you have something nobody sells.

One detail that matters if you build this yourself: the feed carries no
timestamp of its own. If you do not stamp each capture at write time, a year
later you have a pile of rankings and no way to order them. Stamp it on the way
in, not on the way out.

What it costs

Charged per chart entry returned. A 100-app chart is 100 results, so at $3.00 per
1,000 on the free tier a daily 100-app capture for one country is about 30 cents
a month, plus a $0.02 Actor-start event per run. Countries and chart types
multiply that, so pick the markets you actually care about rather than sweeping
all of them.

Rows that come back as errors are not charged.

Caveats worth stating plainly

Apple only. Google Play publishes no equivalent public chart feed. Anything claiming a Play "top chart" API is scraping a rendered page, which breaks on redesign.
Charts are per country, per chart type, per category. An app can be absent from top-free while sitting high in top-grossing. "Not in the chart" is always relative to which chart you asked for.
Apple caps a feed at 200 entries. Beyond that the answer does not exist in the feed.
Rankings are Apple's, computed however Apple computes them. This reads them; it does not model them.

Code

Runnable examples, including this one, are here:
github.com/danielhagever/apify-data-actors

Every example input in that repo is generated from the Actor's own input schema,
so they are valid by construction rather than by my memory of the field names.

Find startups the day they raise, before the press release, with 30 lines of Python

Daniel Meshulam — Thu, 30 Jul 2026 16:16:48 +0000

Funding announcements are old news by the time you read them. There's usually a
gap of weeks between a company closing a round and telling anyone about it. In
that gap, the round is already a matter of public record.

That's because of Form D. When a US company sells securities in a private
placement, it has to file one with the SEC, generally within 15 days of the first
sale. It's a legal requirement, it's public the moment it's accepted, and almost
nobody reads it.

So the question is not whether you can see funding rounds early. It's how to
filter thousands of Form Ds a month down to the ones you care about.

This post is the working version of that, end to end: ~30 lines of Python, real
output, and the two traps that cost me a couple of hours.

Why you can't just list Form Ds

The naive approach is "give me every Form D from the last 90 days." That's
thousands of filings: real estate syndicates, hedge funds, family offices, a
long tail of LLCs. Most of it is not what anyone means by "a startup raised."

You also can't filter by company name, because you don't know the company
names yet. That's the entire point of the exercise.

What you want is a full-text search over the body of the filings, scoped to
Form D and to a recent window. EDGAR supports exactly that, over every filing
since 2001.

The code

I'm using an Apify Actor I built for EDGAR,
SEC EDGAR Scraper, because
it handles the parts of EDGAR that are tedious: CIK resolution, the fair-access
rate limit, XBRL, and the date-range bug I'll get to below. One HTTP call runs it
and returns rows.

import datetime as dt
import json
import os
import urllib.request

ACTOR = "glitchbound~sec-filings-scraper"
API = "https://api.apify.com/v2"


def form_d_by_sector(sector: str, days: int = 90) -> list[dict]:
    since = (dt.date.today() - dt.timedelta(days=days)).isoformat()
    payload = {
        # Quote the phrase or EDGAR matches the words separately.
        "searchQuery": f'"{sector}"',
        "formTypes": ["D"],
        "filedSince": since,
        "maxSearchResults": 50,
    }
    url = (f"{API}/acts/{ACTOR}/run-sync-get-dataset-items"
           f"?token={os.environ['APIFY_TOKEN']}&maxTotalChargeUsd=0.50")
    req = urllib.request.Request(
        url, data=json.dumps(payload).encode(), method="POST",
        headers={"Content-Type": "application/json"})
    with urllib.request.urlopen(req, timeout=300) as r:
        rows = json.load(r)

    # A target the Actor couldn't resolve comes back as a row with an `error`
    # field instead of failing the whole run, and those aren't charged. Count
    # them separately or a run that fetched nothing still looks successful.
    return [r for r in rows if not r.get("error")]


for f in sorted(form_d_by_sector("artificial intelligence"),
                key=lambda r: r["filingDate"], reverse=True):
    print(f"{f['filingDate']}  {f['companyName']:55} {f.get('state') or '--'}")
    print(f"            {f['filingUrl']}")

Set APIFY_TOKEN from
console.apify.com/settings/integrations
and run it.

Real output

This is an actual run, not a mockup:

2026-07-06  Flo Artificial Intelligence, Inc.  (CIK 0001798621)      CA
            .../data/1798621/000179862126000003/primary_doc.xml
2026-06-29  Artificial Intelligence Financial Services Market Inc.   DE
            .../data/2142624/000214262426000001/primary_doc.xml
2026-06-29  Artificial Intelligence Financial Services Market Inc.   DE
            .../data/2142624/000214262426000002/primary_doc.xml
2026-06-18  Panorama Artificial Intelligence Corp.                   WA
            .../data/2001638/000200163826000001/primary_doc.xml
2026-05-26  Artificial Intelligence Progress Research, LP            --
            .../data/2133240/000213324026000002/primary_doc.xml
2026-05-15  Allied Innovation Artificial Intelligence Fund, LLC      NY
            .../data/2021379/000202137926000001/primary_doc.xml
2026-05-05  Artificial Intelligence Progress Research, LP            --
            .../data/2133240/000213324026000001/primary_doc.xml

(URLs truncated for width; the real ones are absolute sec.gov links.)

Seven filings, each linking straight to the primary document on sec.gov. Every one
is a real entity that filed a real Form D in that window.

Note that the same company can appear twice, with two separate accession numbers
on 2026-06-29. Those are genuinely two filings, not a duplicate row: an amended Form
D gets its own accession. Dedupe on companyName only if you actually want to.

You'll notice something honest about this list: it is heavy on entities with
"Artificial Intelligence" in the company name, plus a couple of funds. That's
what a full-text phrase search gives you. It matches the text of the filing, and
Form D is a short form without a business description. It's a strong signal,
not a magic sector classifier. More on that below.

Trap 1: EDGAR silently ignores a half-open date range

This one cost me real money before I caught it.

EDGAR's full-text endpoint takes startdt and enddt alongside
dateRange=custom. Send only startdt and it does not narrow anything, and it
does not tell you either. Measured against the live endpoint, same query, forms=D:

Parameters	Hits	Oldest result
`startdt` only	53	2015-07-28
no date params at all	53	2015-07-28
`startdt` + `enddt`	7	2026-05-05

startdt alone is byte-for-byte identical to sending no filter. If you're paying
per result, you're paying for eleven years of filings outside the window you
asked for, and the response looks perfectly normal.

If you're calling EDGAR directly: always send both bounds. Default the
missing one: 2001-01-01 is the floor for full-text search, and today for the
upper bound. (This is fixed in the Actor as of build 0.1.4; that's why the run
above returns 7 and not 53.)

Trap 2: "recent" isn't the sort you want for anything else

For funding discovery, newest-first is right. For almost every other EDGAR
question it isn't. If you're looking for the filing that matters on a topic,
recency is noise. Worth knowing before you build the wrong dashboard.

What this costs

The Actor charges per result: $3.00 per 1,000 on the free tier, so seven filings
is 2.1 cents, plus a one-time $0.02 Actor-start event, so about 4 cents for
the run. Paid Apify tiers pay less per result, down to half that on Gold.

The maxTotalChargeUsd=0.50 in the code above is a hard
ceiling the platform enforces. The run stops cleanly at the limit rather than
being killed mid-write, so you can't accidentally spend more than you meant to.

Rows that come back as errors, such as a ticker that doesn't resolve or a query
EDGAR rejects, aren't charged.

Going further

A few directions that work well from here:

Watch a sector on a schedule. Run it daily with days=2 and diff against
what you've already seen. New Form Ds show up the day they're accepted.

Pivot to the company. Once a name appears, feed its CIK back in with
includeFinancials: true to get XBRL figures: revenue, net income, total assets,
straight from what the company tagged, not scraped off a rendered page. A full
20-year history is one request, because it comes from companyfacts.

Search for other things entirely. formTypes: ["8-K"] with a phrase like
"material weakness" or "going concern" finds every company that disclosed it.
formTypes: ["4"] is insider trades. formTypes: ["13F-HR"] is what the big
funds hold.

Caveats worth stating plainly

Form D is a filing, not a validated funding announcement. It tells you securities were sold under a Reg D exemption. It does not confirm a venture-style round, and the amounts on the form are frequently the maximum offering rather than what closed.
Full-text phrase search matches text, not sectors. As the output shows, it finds companies whose filings contain your phrase, which correlates with the sector but isn't the same thing. For real classification you'd want to pull the filing and read it.
This is public disclosure, and that's the whole appeal. No login, no key, no scraping of rendered pages. The SEC asks callers to identify themselves and stay under 10 requests/second, which the Actor does.
Not investment advice, obviously.

Code

Full runnable scripts, including this one and a few others, are here:
github.com/danielhagever/apify-data-actors

Every example input in that repo is generated from the Actor's own input schema,
so they're valid by construction rather than by my memory of the field names.
That's a habit I picked up after the reviewedAt-vs-date mistake in a
different script cost me an afternoon of blank output.

If you find a case where EDGAR behaves differently than described here, I'd
genuinely like to know. That date-range trap only surfaced because I ran the
example instead of just writing it.