DEV Community: Daniel Meshulam

Google Play top charts API: Top Free, Top Grossing and Trending apps in 130 countries

Daniel Meshulam — Fri, 23 Oct 2026 07:01:07 +0000

Google Play has no public API for its top charts. SerpApi and a few data vendors resell them; the Play Store web app itself reads them from an internal RPC, and that is what everyone else ends up reading too. It is fast (a 100-app chart in well under a second) and it has one behaviour worth knowing before you trust a single row of it.

The charts there are

Five lists answer for any country and category:

Chart	What it ranks
Top Free	downloads, free apps
Top Paid	downloads, paid apps
Top Grossing	revenue
Trending	Google's "movers and shakers": what is rising fastest right now
Top New Free	recently released free apps

Each goes up to 200 apps (Top New Free runs shorter in small markets). Trending is the one most people miss, and it is not a reshuffle of Top Free: of the top 100 in the US on 2026-10-18 the two shared 7 apps. Top New Free shared 1. If you are looking for breakouts, those two lists are where they show up first.

An unknown country is not an error

Ask for a country code Google does not serve and you do not get a 400. You get a full, plausible chart, from somewhere else. Which somewhere was the surprise: the answer for an unknown code is the chart of the country the request comes from.

I found this the embarrassing way. My scraper only offers countries I have checked, and the check compared each country's chart with the answer for zz. I ran it in September from Israel; Israel's chart matched zz, so Israel went on the excluded list as "not a real market". Run again in October from the US, zz matched the US chart exactly, and Israel came back with a chart of its own: Rav-Kav Online, Moovit, Wolt and Hebrew-named apps in the top 40. The check had excluded Israel for being Israel.

The reference that does not move with you is the chart Google serves for a code with no Play market at all. Antarctica (aq), Bouvet Island, Heard Island and the other uninhabited territories all get the same list, and a country is real when its chart differs from that one and from every other country's. By that test 130 codes are real markets today. Five that look like they should be are not: Ethiopia, Montenegro, Afghanistan, Brunei and Barbados answer with the no-market chart. Russia answers Top Free only, since Google Play takes no payments there.

If you pull Play charts for many countries, run that test yourself: a wrong market fails silently, and you will be the one paying for its rows.

Reading them without writing the RPC

I maintain Google Play Top Charts Scraper on Apify, which reads all five charts for the 130 verified countries and 54 categories, refuses a country it has not verified rather than returning someone else's chart, and asks Google twice before calling a chart empty (Google sometimes answers a real chart with nothing, and a minute later in full).

{
  "countries": ["us", "il", "br", "jp"],
  "charts": ["top-grossing", "trending"],
  "categories": ["all-games"],
  "limit": 200
}

Every row carries rank, app id, title, developer, rating, install band and price. includeDetails adds exact installs, rating counts, release and update dates and the developer's contact email. For the iPhone side, the App Store sibling returns the same row shape, so the two join on title and developer.

Company career page scraper: every open job from any company, by name or domain

Daniel Meshulam — Fri, 23 Oct 2026 06:59:24 +0000

Nearly every company's careers page is a skin over one of a dozen applicant tracking systems: Workday, Greenhouse, Lever, Ashby, Oracle, SAP SuccessFactors, iCIMS, Eightfold, Phenom and a few more. Each of them publishes the open jobs as JSON, because job boards and Google need to index them. Scraping a career page is mostly two problems: knowing which system and which board a company uses, and reading that board completely. Both have traps that return plausible, wrong data.

Trap 1: the name on the board is not always the company

Guessing a board from a company name ("stripe" on Greenhouse) works for startups. For big brands it goes wrong in ways that look right:

A Greenhouse board that names itself "LinkedIn" belongs to Obsidian Security; it is the feed they post to LinkedIn. Ask a naive scraper for LinkedIn's jobs and you get Obsidian's account executives.
"Meta" on Recruitee is a one-job account of a much smaller company. "Honda" on Paylocity is a dealership in Akron with two openings. American Honda's real site is careers.honda.com, which runs on Phenom.
Google, Apple, IBM, Meta, LinkedIn and Tesla run careers software of their own. The honest answer for those is "not on a system I can read", not the nearest board with the right word in it.

What works is a directory built from boards that have actually answered, with one answer per name, plus a list of brands whose name must never be matched to somebody else's board.

Trap 2: the same job, twice

Citi and Netflix put Eightfold, a careers front end, over their Workday. A name finds both, and a scraper that reads both returns every job twice: Citi's "Young Apprentice - C00 - MUMBAI" came back from each. The front end serves the underlying system's requisition id, so (company, job id) identifies the job on either side. Keep one.

Trap 3: filtering after you read misses most matches

Big employers have thousands of open roles, and most scrapers read the newest N and filter by title afterwards. Measured against the newest 1,000 (300 for EY):

Company	Title	Found by reading, then filtering	Found by the site's own search
EY (SAP)	senior consultant	30	690
NVIDIA (Workday)	software engineer	196	442
Microsoft (Eightfold)	software engineer	177	415
Amazon	product manager	15	492

Every one of these systems has a keyword search in the same public feed: Workday's searchText, Amazon's base_query, Eightfold's query, SAP's q. Send the title there, keep the rows whose title actually holds it, and stop paging once a few pages in a row hold no match. The searches rank title matches first, so that rule kept 3,264 of 3,264 matches on Workday while reading 239 of the 502 pages.

Or skip all of it

I maintain Company Career Page Jobs Scraper on Apify, which does the above for 78,000 company names across 24 systems: type Stripe, stripe.com or Home Depot, add a title or a city, and get the employer's own postings. Brands that publish on their own software are named as such instead of guessed, a job seen through two systems is charged once, and filtered-out rows are not charged.

{
  "companies": ["EY", "NVIDIA", "Amazon"],
  "titleKeywords": ["software engineer", "senior consultant"],
  "locationKeywords": ["London"]
}

It is also callable from an AI agent through the Apify MCP server.

Workday Jobs Scraper: every posting from any Workday careers site, by company name

Daniel Meshulam — Fri, 23 Oct 2026 06:59:05 +0000

Large employers post their jobs on Workday: NVIDIA, Salesforce, Walmart, Target, Wells Fargo, Pfizer, Boeing. Every one of those *.myworkdayjobs.com career sites looks different in the browser and is the same API underneath:

import requests

r = requests.post(
    "https://nvidia.wd5.myworkdayjobs.com/wday/cxs/nvidia/NVIDIAExternalCareerSite/jobs",
    json={"appliedFacets": {}, "limit": 20, "offset": 0, "searchText": ""},
)
data = r.json()
data["total"], [j["title"] for j in data["jobPostings"][:3]]

No key, no login, no cookies: it is the endpoint the careers page itself calls. A scraper is a loop over offset. Three things turn that loop into a wrong answer, and none of them raises an error.

1. You need the URL, and a company name does not give it to you

The address carries three facts a name cannot derive: the host label (nvidia), the data centre (wd5; companies sit anywhere from wd1 to wd504 and move between them) and the career site's own name (NVIDIAExternalCareerSite). Guessing across all three mostly produces 404s and 422s.

What works is a map built from boards that have actually answered with real postings. The one this post's scraper ships holds 3,056 employers, so NVIDIA, Wells Fargo or PwC resolve without anyone hunting for a URL. A name it does not know is better refused than guessed.

2. `limit` is 20, and `total` is a ceiling

Ask for 50 a page and the answer is an error. Page past 2,000 and big boards just serve page one again, so a loop that stops on an empty page never stops. On those boards total says 2,000 whatever the real count is. I wrote up the ceiling and the way past it (the facet counts in the same answer) here.

3. Filtering after you read misses most of the matches

This is the one that costs the most, because the output looks fine.

Most scrapers read the newest N postings (often 1,000), then keep the ones whose title matches. On a small board that is everything. On a big one it is a sample. Measured on 2026-10-18 with "software engineer":

Board	Matches in the newest 1,000	Matches on the board
NVIDIA	196	442
Salesforce, "account executive"	250	448
Mastercard	191	225

The fix is already in the request: searchText. Workday searches title and description, so the answer is a superset, and you keep the rows whose title holds the words:

def search(url, text):
    out, offset = [], 0
    while True:
        page = requests.post(url, json={"appliedFacets": {}, "limit": 20,
                                        "offset": offset, "searchText": text}).json()
        posts = page.get("jobPostings") or []
        hits = [p for p in posts if text.lower() in p["title"].lower()]
        out += hits
        # Workday ranks title matches first: once a page past the second has
        # none, the rest are postings that only mention the words.
        if not posts or (offset >= 40 and not hits):
            return out
        offset += 20

On eight title and board pairs that kept every one of 3,264 title matches and read 239 of the 502 pages the searches declared. The same goes for a place: searchText="Israel" at NVIDIA finds 233 jobs, where the newest 1,000 held 61.

Or skip all of it

I maintain Workday Jobs Scraper on Apify, which does the three things above: a company name for 3,056 employers, the 2,000 ceiling, and titles and places sent to Workday's own search. A company that is not on Workday is looked up where it does post (Oracle, Eightfold, Phenom, SAP and more), and filtered-out rows are not charged.

{
  "companies": ["NVIDIA", "Wells Fargo"],
  "titleKeywords": ["software engineer"],
  "locationKeywords": ["Israel"]
}

It is also callable from an AI agent through the Apify MCP server, so "find software engineer openings at NVIDIA in Israel" is one tool call.

Google's News API has been gone since 2016, and people still search for it

Daniel Meshulam — Fri, 25 Sep 2026 21:38:27 +0000

Search for "google news api" and a striking share of the results are people
looking for something that no longer exists. Google deprecated the News Search
API in May 2011 as part of a wider AJAX Search API cull, kept it limping until
the final shutdown in February 2016, and never shipped a replacement. A decade
later the searches are still there.

What almost nobody mentions is that Google News still exposes a perfectly good
public endpoint. It is just an RSS feed rather than a JSON API, it needs no key,
and it has quietly outlived the API that was supposed to be the real product.

The endpoint

https://news.google.com/rss/search?q=<query>&hl=en-US&gl=US&ceid=US:en

That is the whole thing. Swap hl, gl and ceid for a different language and
country and you get that edition instead. There is no key, no quota page, and no
sign-up.

The catches are real but small:

It is RSS, so you get XML. Title, link, publication date, source name and a short description. No article body, and no way to ask for one.
The links are Google redirects, not publisher URLs. You have to follow them if you want the destination.
The publisher name lives in a nested <source> element, not where a normal feed parser looks for it.
Roughly 100 items per query. Deeper history is not available through it.

None of that is a reason to go and pay for a news API. It is a reason to write
about forty lines once.

Two sources are better than one

Bing News publishes its own RSS in the same shape at a different URL. Running
both and merging costs one extra request and meaningfully changes what you see,
because the two indexes disagree about what is worth surfacing.

Here is a real run over both:

16 articles
sources: {'google:semiconductor export controls': 8,
          'bing:semiconductor export controls': 8}

[google] It Takes More Than Two to Tango: Creating Effective Export Con...
         Just Security                                     2026-07-09
[google] China considers tighter export controls on AI models and chips
         Reuters                                           2026-07-20
[google] China's Localization Drive in Semiconductors Gains Impetus...
         CSIS | Center for Strategic and International...   2026-03-24
[google] China's Domestic DUV Push Puts ASML's Export-Control Advantage...
         EU Today                                          2026-07-28

Every row carries title, url, publisher, publishedAt, summary,
source, image and guid.

One thing worth knowing if you parse Bing's feed yourself. It carries the
publisher in <News:Source>, and declares that prefix as

xmlns:News="https://www.bing.com/news/search?q=<your query>&format=RSS"

The namespace URI is the request URL, so it changes on every query and no fixed
namespace constant will ever match it. Match on the local element name instead.
I found this the way you would expect: publisher was null on exactly the Bing
half of a two-source run while the value was sitting in the feed the whole time.

The code

import json
import os
import urllib.request

ACTOR = "glitchbound~news-scraper"
API = "https://api.apify.com/v2"


def headlines(query, per_source=10):
    payload = {
        "searchQueries": [query],
        "maxArticlesPerSource": per_source,
        "language": "en",
        "includeBingNews": True,
        "dedupeHeadlines": True,
    }
    url = (f"{API}/acts/{ACTOR}/run-sync-get-dataset-items"
           f"?token={os.environ['APIFY_TOKEN']}&maxTotalChargeUsd=0.50")
    req = urllib.request.Request(
        url, data=json.dumps(payload).encode(), method="POST",
        headers={"Content-Type": "application/json"})
    with urllib.request.urlopen(req, timeout=300) as r:
        rows = json.load(r)
    # A feed that fails comes back as a row with an `error` field rather than
    # failing the run, and those are not charged.
    return [r for r in rows if not r.get("error")]


for a in headlines("semiconductor export controls", per_source=5):
    # Fields can be absent on any given row, so read them defensively rather
    # than subscripting. Feeds are not schemas.
    when = (a.get("publishedAt") or "")[:10] or "no date"
    who = (a.get("publisher") or "unknown")[:28]
    print(f'{when:10}  {who:28} {(a.get("title") or "")[:60]}')

That runs News Scraper on
Apify, which is the same forty lines plus the parts that are
tedious rather than hard: both feeds, the nested publisher element, the date
normalisation, and deduplication.

If you would rather own the code, the endpoint above is public and this post has
told you everything you need.

Deduplication is most of the value

One wire story appears on dozens of syndicating domains, and Google and Bing will
both hand you several copies. A hundred raw results is frequently thirty stories.

Dedupe on the URL first, then on a normalised headline, because the same Reuters
piece appears under slightly different titles across outlets. Doing it on
headline alone merges genuinely different stories that share a phrasing; doing it
on URL alone leaves most of the duplication in place.

Where this does and does not fit

Fits: brand and competitor monitoring, tracking a topic across outlets,
building a corpus of headlines, seeing which publishers cover a beat, feeding a
summarisation pipeline.

Does not fit: anything needing article text, anything needing more than a few
months of history, or academic archival work. For those you want a licensed
provider, and the honest answer is that no free endpoint replaces one.

What it costs

If you write it yourself, nothing beyond your own hosting.

Through the Actor it is $1.50 per 1,000 articles on the free tier, so a daily run
pulling 50 articles is about seven cents a month, plus a $0.02 Actor-start event
per run. Duplicates that get dropped are not charged, and neither are feeds that
fail.

The wider point

An enormous amount of "you need an API for this" is habit. Google News, Apple's
app charts, the SEC, the World Bank, the ECB and Open Food Facts all publish
usable public endpoints that need no key and get very little attention, largely
because there is no vendor with an incentive to tell you about them.

Worth checking before you pay for access to public data.

Code

Runnable examples are here:
github.com/danielhagever/apify-data-actors

Find startups the day they raise, before the press release, with 30 lines of Python

Daniel Meshulam — Fri, 25 Sep 2026 21:37:15 +0000

Funding announcements are old news by the time you read them. There's usually a
gap of weeks between a company closing a round and telling anyone about it. In
that gap, the round is already a matter of public record.

That's because of Form D. When a US company sells securities in a private
placement, it has to file one with the SEC, generally within 15 days of the first
sale. It's a legal requirement, it's public the moment it's accepted, and almost
nobody reads it.

So the question is not whether you can see funding rounds early. It's how to
filter thousands of Form Ds a month down to the ones you care about.

This post is the working version of that, end to end: ~30 lines of Python, real
output, and the two traps that cost me a couple of hours.

Why you can't just list Form Ds

The naive approach is "give me every Form D from the last 90 days." That's
thousands of filings: real estate syndicates, hedge funds, family offices, a
long tail of LLCs. Most of it is not what anyone means by "a startup raised."

You also can't filter by company name, because you don't know the company
names yet. That's the entire point of the exercise.

What you want is a full-text search over the body of the filings, scoped to
Form D and to a recent window. EDGAR supports exactly that, over every filing
since 2001.

The code

I'm using an Apify Actor I built for EDGAR,
SEC EDGAR Scraper, because
it handles the parts of EDGAR that are tedious: CIK resolution, the fair-access
rate limit, XBRL, and the date-range bug I'll get to below. One HTTP call runs it
and returns rows.

import datetime as dt
import json
import os
import urllib.request

ACTOR = "glitchbound~sec-filings-scraper"
API = "https://api.apify.com/v2"


def form_d_by_sector(sector: str, days: int = 90) -> list[dict]:
    since = (dt.date.today() - dt.timedelta(days=days)).isoformat()
    payload = {
        # Quote the phrase or EDGAR matches the words separately.
        "searchQuery": f'"{sector}"',
        "formTypes": ["D"],
        "filedSince": since,
        "maxSearchResults": 50,
    }
    url = (f"{API}/acts/{ACTOR}/run-sync-get-dataset-items"
           f"?token={os.environ['APIFY_TOKEN']}&maxTotalChargeUsd=0.50")
    req = urllib.request.Request(
        url, data=json.dumps(payload).encode(), method="POST",
        headers={"Content-Type": "application/json"})
    with urllib.request.urlopen(req, timeout=300) as r:
        rows = json.load(r)

    # A target the Actor couldn't resolve comes back as a row with an `error`
    # field instead of failing the whole run, and those aren't charged. Count
    # them separately or a run that fetched nothing still looks successful.
    return [r for r in rows if not r.get("error")]


for f in sorted(form_d_by_sector("artificial intelligence"),
                key=lambda r: r["filingDate"], reverse=True):
    print(f"{f['filingDate']}  {f['companyName']:55} {f.get('state') or '--'}")
    print(f"            {f['filingUrl']}")

Set APIFY_TOKEN from
console.apify.com/settings/integrations
and run it.

Real output

This is an actual run, not a mockup:

2026-07-06  Flo Artificial Intelligence, Inc.  (CIK 0001798621)      CA
            .../data/1798621/000179862126000003/primary_doc.xml
2026-06-29  Artificial Intelligence Financial Services Market Inc.   DE
            .../data/2142624/000214262426000001/primary_doc.xml
2026-06-29  Artificial Intelligence Financial Services Market Inc.   DE
            .../data/2142624/000214262426000002/primary_doc.xml
2026-06-18  Panorama Artificial Intelligence Corp.                   WA
            .../data/2001638/000200163826000001/primary_doc.xml
2026-05-26  Artificial Intelligence Progress Research, LP            --
            .../data/2133240/000213324026000002/primary_doc.xml
2026-05-15  Allied Innovation Artificial Intelligence Fund, LLC      NY
            .../data/2021379/000202137926000001/primary_doc.xml
2026-05-05  Artificial Intelligence Progress Research, LP            --
            .../data/2133240/000213324026000001/primary_doc.xml

(URLs truncated for width; the real ones are absolute sec.gov links.)

Seven filings, each linking straight to the primary document on sec.gov. Every one
is a real entity that filed a real Form D in that window.

Note that the same company can appear twice, with two separate accession numbers
on 2026-06-29. Those are genuinely two filings, not a duplicate row: an amended Form
D gets its own accession. Dedupe on companyName only if you actually want to.

You'll notice something honest about this list: it is heavy on entities with
"Artificial Intelligence" in the company name, plus a couple of funds. That's
what a full-text phrase search gives you. It matches the text of the filing, and
Form D is a short form without a business description. It's a strong signal,
not a magic sector classifier. More on that below.

Trap 1: EDGAR silently ignores a half-open date range

This one cost me real money before I caught it.

EDGAR's full-text endpoint takes startdt and enddt alongside
dateRange=custom. Send only startdt and it does not narrow anything, and it
does not tell you either. Measured against the live endpoint, same query, forms=D:

Parameters	Hits	Oldest result
`startdt` only	53	2015-07-28
no date params at all	53	2015-07-28
`startdt` + `enddt`	7	2026-05-05

startdt alone is byte-for-byte identical to sending no filter. If you're paying
per result, you're paying for eleven years of filings outside the window you
asked for, and the response looks perfectly normal.

If you're calling EDGAR directly: always send both bounds. Default the
missing one: 2001-01-01 is the floor for full-text search, and today for the
upper bound. (This is fixed in the Actor as of build 0.1.4; that's why the run
above returns 7 and not 53.)

Trap 2: "recent" isn't the sort you want for anything else

For funding discovery, newest-first is right. For almost every other EDGAR
question it isn't. If you're looking for the filing that matters on a topic,
recency is noise. Worth knowing before you build the wrong dashboard.

What this costs

The Actor charges per result: $3.00 per 1,000 on the free tier, so seven filings
is 2.1 cents, plus a one-time $0.02 Actor-start event, so about 4 cents for
the run. Paid Apify tiers pay less per result, down to half that on Gold.

The maxTotalChargeUsd=0.50 in the code above is a hard
ceiling the platform enforces. The run stops cleanly at the limit rather than
being killed mid-write, so you can't accidentally spend more than you meant to.

Rows that come back as errors, such as a ticker that doesn't resolve or a query
EDGAR rejects, aren't charged.

Going further

A few directions that work well from here:

Watch a sector on a schedule. Run it daily with days=2 and diff against
what you've already seen. New Form Ds show up the day they're accepted.

Pivot to the company. Once a name appears, feed its CIK back in with
includeFinancials: true to get XBRL figures: revenue, net income, total assets,
straight from what the company tagged, not scraped off a rendered page. A full
20-year history is one request, because it comes from companyfacts.

Search for other things entirely. formTypes: ["8-K"] with a phrase like
"material weakness" or "going concern" finds every company that disclosed it.
formTypes: ["4"] is insider trades. formTypes: ["13F-HR"] is what the big
funds hold.

Caveats worth stating plainly

Form D is a filing, not a validated funding announcement. It tells you securities were sold under a Reg D exemption. It does not confirm a venture-style round, and the amounts on the form are frequently the maximum offering rather than what closed.
Full-text phrase search matches text, not sectors. As the output shows, it finds companies whose filings contain your phrase, which correlates with the sector but isn't the same thing. For real classification you'd want to pull the filing and read it.
This is public disclosure, and that's the whole appeal. No login, no key, no scraping of rendered pages. The SEC asks callers to identify themselves and stay under 10 requests/second, which the Actor does.
Not investment advice, obviously.

Code

Full runnable scripts, including this one and a few others, are here:
github.com/danielhagever/apify-data-actors

Every example input in that repo is generated from the Actor's own input schema,
so they're valid by construction rather than by my memory of the field names.
That's a habit I picked up after the reviewedAt-vs-date mistake in a
different script cost me an afternoon of blank output.

If you find a case where EDGAR behaves differently than described here, I'd
genuinely like to know. That date-range trap only surfaced because I ran the
example instead of just writing it.

The App Store top charts are a public API, and nobody archives them

Daniel Meshulam — Fri, 25 Sep 2026 21:34:54 +0000

Apple's top charts are one of the few genuinely useful public rankings left.
They are free, they need no key, and they update daily.

They also have one property that makes them frustrating: there is no history.
The feed shows you today. Yesterday is gone. If you want to know whether an app
climbed or slid, or which category a publisher quietly took over last month, that
data does not exist anywhere unless somebody was writing it down.

Nobody is writing it down, which means the archive is worth more than the feed.

This post covers both halves: reading the charts today, and the small amount of
work it takes to have a year of history in a year's time.

The feed

Apple's RSS feeds sit at a predictable URL per country, chart type and category.
No key, no login, and no bot wall. That is unusual enough in 2026 to be worth
saying out loud.

What is awkward is the shape. Each entry buries the app id under
id.attributes["im:id"], the price under im:price.attributes, the developer
under im:artist.label, and the icon in an array where you want the last
element. Parsing it correctly is 30 lines of nested .get() calls that you write
once and never enjoy.

I put that in an Apify Actor,
App Store Top Charts Scraper,
so I could stop rewriting it. One HTTP call runs it and returns flat rows.

import json
import os
import urllib.request

ACTOR = "glitchbound~app-charts-scraper"
API = "https://api.apify.com/v2"


def top_charts(countries, charts=("top-free",), limit=50):
    payload = {
        "countries": list(countries),
        "charts": list(charts),
        "categories": ["all"],
        "limit": limit,
    }
    url = (f"{API}/acts/{ACTOR}/run-sync-get-dataset-items"
           f"?token={os.environ['APIFY_TOKEN']}&maxTotalChargeUsd=0.50")
    req = urllib.request.Request(
        url, data=json.dumps(payload).encode(), method="POST",
        headers={"Content-Type": "application/json"})
    with urllib.request.urlopen(req, timeout=300) as r:
        rows = json.load(r)
    # A country or chart that fails comes back as a row with an `error` field
    # rather than failing the run, and those are not charged. Split them out or a
    # run that fetched nothing still looks successful.
    return [r for r in rows if not r.get("error")]


rows = top_charts(["us", "gb", "de"], charts=["top-free"], limit=20)
for r in sorted(rows, key=lambda r: (r["country"], r["rank"])):
    print(f'{r["country"]}  #{r["rank"]:<3} {r["title"][:42]:42} {r["developer"][:24]}')

Real output, from an actual run, top of the German chart:

DE  #1   Claude by Anthropic                        Anthropic PBC
DE  #2   ChatGPT                                    OpenAI OpCo, LLC
DE  #3   Nect Wallet                                Nect
DE  #4   Google Gemini                              Google
DE  #5   EasyPark parken - Park App                 Easy Park AS
DE  #6   AusweisApp Bund                            Governikus GmbH & Co. KG
DE  #7   Mein dm                                    dm-drogerie markt GmbH +
DE  #8   Post & DHL                                 Deutsche Post AG

That German top ten is itself the argument for archiving. Four of those eight are
government or logistics apps that nobody would predict from the US chart, and the
top two are AI assistants that were not in this chart at all two years ago. You
cannot see either of those facts in a single pull.

Every row also carries appId, bundleId, price, currency, isFree,
genre, releaseDate, icon and url, so a chart pull doubles as an app
metadata pull.

The part that is actually valuable

One pull tells you the ranking today, which you could have read off your phone.
The interesting queries all need two pulls separated by time:

Which apps entered the top 20 this week
Which publisher gained the most chart slots this month
How long a given app has held its position
Whether a competitor's launch actually moved anything
Which categories are churning and which are frozen

None of that is answerable from the feed. All of it is answerable from a table
you append to daily.

So the whole trick is: run it on a schedule and never delete anything.

import datetime as dt

rows = top_charts(["us"], charts=["top-free", "top-paid"], limit=100)
stamp = dt.datetime.now(dt.timezone.utc).isoformat(timespec="seconds")
for r in rows:
    r["capturedAt"] = stamp          # the feed has no timestamp; add your own
# append to wherever you keep things: Postgres, SQLite, a CSV, a dataset

On Apify you get this without writing the storage layer: put the Actor on a
Schedule and every run appends to the same dataset, each with its own timestamp.
Three months later you have something nobody sells.

One detail that matters if you build this yourself: the feed carries no
timestamp of its own. If you do not stamp each capture at write time, a year
later you have a pile of rankings and no way to order them. Stamp it on the way
in, not on the way out.

What it costs

Charged per chart entry returned. A 100-app chart is 100 results, so at $3.00 per
1,000 on the free tier a daily 100-app capture for one country is about 30 cents
a month, plus a $0.02 Actor-start event per run. Countries and chart types
multiply that, so pick the markets you actually care about rather than sweeping
all of them.

Rows that come back as errors are not charged.

Caveats worth stating plainly

Apple only. Google Play publishes no equivalent public chart feed. Anything claiming a Play "top chart" API is scraping a rendered page, which breaks on redesign.
Charts are per country, per chart type, per category. An app can be absent from top-free while sitting high in top-grossing. "Not in the chart" is always relative to which chart you asked for.
Apple caps a feed at 200 entries. Beyond that the answer does not exist in the feed.
Rankings are Apple's, computed however Apple computes them. This reads them; it does not model them.

Code

Runnable examples, including this one, are here:
github.com/danielhagever/apify-data-actors

Every example input in that repo is generated from the Actor's own input schema,
so they are valid by construction rather than by my memory of the field names.

WHOIS is gone, RDAP replaced it, and a 404 does not mean what you think

Daniel Meshulam — Fri, 25 Sep 2026 21:32:53 +0000

If you last touched domain lookups a few years ago, the thing you remember is
WHOIS: a plaintext protocol on port 43 that returned an unparseable wall of text
in a different shape from every registry.

It is effectively gone. RDAP replaced it, it is RFC 9083, it returns JSON, and
every gTLD registry is required to run one. This is one of the rare cases where
the standards process produced something strictly better and nobody noticed.

Here is what a lookup looks like now, what you can actually get out of it after
GDPR, and the one trap that had my own tool confidently reporting registered
domains as free to buy.

What RDAP gives you, and what it does not

Post-GDPR, RDAP returns no registrant personal data for virtually every TLD. If
you are looking for who owns a domain, stop: that information is not there and
no legitimate source has it.

What is there is the operational picture, and that turns out to be the more
useful half:

registrar, registration date, expiry date, status codes, DNSSEC
whether the name is locked against transfer

Combine that with a DNS lookup and you get the thing people actually want, which
is what a company runs on:

domain        registrar                expires       age mail       dns         dmarc
stripe.com    SafeNames Ltd.           2027-09-11   30.9 google     aws-route53 reject
linear.app    CloudFlare, Inc.         2030-05-09    8.2 google     cloudflare  reject
vercel.com    Amazon Registrar, Inc.   2029-10-04   26.8 google     vercel      quarantine
github.io     -                        -               - -          aws-route53 -
angel.co      -                        -               - google     cloudflare  reject
notion.so     -                        -               - -          cloudflare  quarantine

That is a real run. Mail provider and DNS provider come from MX and NS records,
DMARC policy from a TXT lookup at _dmarc.<domain>. For sales qualification,
security auditing or competitive research, those three columns are worth more
than any registrant field ever was.

Note the bottom three rows have DNS but no registrar and no expiry. Hold that
thought.

The code

import json
import os
import urllib.request

ACTOR = "glitchbound~whois-domain-lookup"
API = "https://api.apify.com/v2"


def profile(domains):
    payload = {
        "domains": domains,
        "includeDns": True,
        "recordTypes": ["A", "MX", "NS", "TXT"],
        "includeEmailSecurity": True,
    }
    url = (f"{API}/acts/{ACTOR}/run-sync-get-dataset-items"
           f"?token={os.environ['APIFY_TOKEN']}&maxTotalChargeUsd=0.50")
    req = urllib.request.Request(
        url, data=json.dumps(payload).encode(), method="POST",
        headers={"Content-Type": "application/json"})
    with urllib.request.urlopen(req, timeout=300) as r:
        rows = json.load(r)
    # A domain that could not be looked up comes back as a row with an `error`
    # field rather than failing the run, and those are not charged.
    return [r for r in rows if not r.get("error")]


for r in profile(["stripe.com", "linear.app", "vercel.com"]):
    print(f'{r["domain"]:14} {r.get("mailProvider") or "-":10} '
          f'{r.get("dnsProvider") or "-":12} dmarc={r.get("dmarcPolicy") or "-"}')

That runs an Apify Actor I maintain,
WHOIS Domain Lookup, because the
tedious parts are the RDAP bootstrap, the vcard unpacking and the provider
fingerprinting rather than the HTTP call. If you would rather do it yourself,
https://rdap.org/domain/<name> is the whole API and it needs no key.

The trap: a 404 is ambiguous

rdap.org is a router. It reads IANA's bootstrap registry, finds the RDAP
server for your TLD, and redirects. A 404 back from it looks like it means one
thing: no registry holds this name, therefore the domain is available.

That is what my tool assumed. It was wrong, and the failure is ugly:

rdap.org/domain/github.io    404
rdap.org/domain/notion.so    404
rdap.org/domain/segment.io   404
rdap.org/domain/angel.co     404

All four are registered. All four returned 404.

The reason is that IANA's bootstrap does not cover every TLD. It lists RDAP
services for around 1,200 of them, and some of the most common startup TLDs are
not in there:

.io   .co   .de   .me   .so   .gg   .sh

For those, rdap.org has nowhere to send the query, so it 404s for every name
under them, registered or not. My Actor turned that into isAvailable: true and
would have told you github.io was free to register.

On a tool whose headline feature is availability checking, that is the worst
possible way to be wrong: confident, plausible, and acted upon. Nobody
double-checks a clear answer.

The fix, in two parts

Ask whether there is a registry at all. Fetch
https://data.iana.org/rdap/dns.json once, build the set of TLDs that have an
RDAP service, and only treat a 404 as "available" when the TLD is in it.

import json, urllib.request

def rdap_tlds():
    d = json.load(urllib.request.urlopen("https://data.iana.org/rdap/dns.json"))
    return {t.lower().lstrip(".") for entry in d["services"] for t in entry[0]}

Let DNS settle the rest. A name with NS records has been delegated by the
registry, and a registry does not delegate a name nobody registered. So for the
TLDs with no RDAP, NS records prove registration even though RDAP cannot.

The reverse does not hold, and this is the part worth being careful about: a
registered domain can have no NS at all. So "no NS and no RDAP" is not
"available", it is unknown, and it should be returned as null with an
explanation rather than as a guess. A null is a fact the caller can handle. A
plausible wrong answer is not.

Why this generalises

The bug was not really about RDAP. It was about reading an absence as a
definite answer.

404 means "I have nothing for you". Whether that is because the thing does not
exist, or because you asked somewhere that was never going to know, is not in
the response. Any time you convert a missing result into a positive claim, check
that you actually asked something capable of answering.

I have now hit the same shape four times in a month in different tools: an API
silently ignoring a filter it does not support, a page parser taking the first
number in a plausible range, this, and crt.sh, which answers 404 when it is
refusing traffic and reports "nothing found" as an empty list with a 200. All of
them produced output that looked completely normal.

What it costs

Charged per domain returned, at $2.00 per 1,000 on the free tier, so a hundred
domains is 20 cents plus a $0.02 Actor-start event. Domains removed by the
built-in filters, like onlyRegistered or expiringWithinDays, are not charged,
and neither are lookups that fail.

Notes

Some ccTLDs run thin RDAP servers that return fewer fields. You get whatever the registry publishes rather than an error.
DNS answers are returned sorted. Resolvers round-robin, and unsorted output makes a scheduled run look like it changed every time when nothing did.
Availability comes from the registry itself, not a reseller API, so there is no incentive for anyone to tell you a name is taken when it is not.

Code

Runnable examples are here:
github.com/danielhagever/apify-data-actors

Every example input in that repo is generated from the Actor's own input schema,
so it is valid by construction rather than by my memory of the field names.

10,789 American employers publish their whole job list in one HTML page

Daniel Meshulam — Fri, 25 Sep 2026 21:31:35 +0000

A Paylocity careers page looks like a React app:

https://recruiting.paylocity.com/recruiting/jobs/All/8f0bcd76-3ee7-42f2-b040-83afc6e6d564

Open the network tab and there is no job API call to find. The jobs are
already there, in the HTML, in one global:

import json, re, requests

UA = {"User-Agent": "Mozilla/5.0"}
guid = "8f0bcd76-3ee7-42f2-b040-83afc6e6d564"
html = requests.get(f"https://recruiting.paylocity.com/recruiting/jobs/All/{guid}",
                    headers=UA).text
data = json.loads(re.search(r"window\.pageData\s*=\s*(\{.*?\});\s*\n", html, re.S).group(1))

data["ModuleTitle"]        # 'City of Columbia'
len(data["Jobs"])          # 9

And a job is not a card scraped off a page, it is a record:

{'JobId': 4514780,
 'JobTitle': 'Building and Grounds Maintenance Part Time',
 'PublishedDate': '2026-09-17T13:43:33-05:00',
 'IsInternal': False, 'IsRemote': False,
 'JobLocation': {'Address': '1648 Trottwood Ave.', 'City': 'Columbia',
                 'State': 'TN', 'Zip': '38401', 'Country': 'USA'}}

A street address. Most career systems give you "Columbia, TN" if you are
lucky and "Ridley Park - Parks and Recreation" if you are not.

recruiting.paylocity.com answers 404 to /robots.txt, so there is nothing
there to disallow anything.

Four things that bite

The assignment ends at ;\n. Job descriptions contain semicolons. A
greedy match, or a match that stops at the first ;, both give you a broken
object. The newline is what makes it unambiguous.

A retired board answers 200. When an employer leaves Paylocity the guid
keeps serving a portal page, just without pageData. So "no pageData" is the
proof a board is gone; the status code never says so.

JobId is a global sequence, not a per-board one. Measured over ten
unrelated boards: the ranges interleave (a 40-person academy's ids sit inside
a hospital group's range) and no id appeared at two employers. That matters
because an employer can run several modules, and the same posting shows up on
more than one. Key on the id alone and they collapse correctly.

LocationName is not a location. It is whatever the employer labelled the
site: "Public Works", "Columbia Aquatics & Recreation Center", sometimes a
person's name. The address object is the only place a city comes from.

How many there are

A board is a guid, and nothing derives a guid from a company domain, so the
list has to come from somewhere. The Wayback CDX index holds every URL it has
archived on that host:

https://web.archive.org/cdx/search/cdx?url=recruiting.paylocity.com
  &matchType=domain&collapse=urlkey&fl=original&output=json
  &filter=original:.*[Rr]ecruiting/[Jj]obs/[Aa]ll/.*

Two things about that query cost me boards before they earned them. limit
stops at the first page and looks complete: the first sweep returned exactly
20,000 URLs and 14,836 guids, which is a suspiciously round stopping point.
And the filter is a case-sensitive regex, while the archive holds the path
capitalised as well; fixing the case alone found 1,821 more.

Read live on 2026-09-24: 16,936 candidate boards, 13,581 that still answer,
10,789 with open roles, 146,046 jobs. One GET each, six at a time, a
quarter of a second apart: 41 minutes for the lot, and not one request
refused.

The employers are the interesting part. This is not the Fortune 500, which
every jobs dataset already has three times over. It is DOCS Health, a Taco
Bell franchise group with 2,219 openings, Farmers Home Furniture, credit
unions, dialysis clinics, car dealerships, county governments, country clubs.
The half of the American labour market that never appears in an enterprise
ATS index.

If you want the rows rather than the code, that is what
ats-jobs-scraper does: a
company name in, every open posting out, from Paylocity and twenty-one other
career systems, as JSON.

BrassRing has a JSON job search, and Home Depot publishes 23,982 jobs through it

Daniel Meshulam — Fri, 25 Sep 2026 21:28:03 +0000

Best Buy's careers site is a ServiceNow portal. Its apply links go somewhere
else: sjobs.brassring.com, which is BrassRing (Infinite Talent, once IBM
Kenexa). One host serves every employer on it, and a board is a partner id
plus a site id:

https://sjobs.brassring.com/TGnewUI/Search/Home/Home?partnerid=25632&siteid=5649

That one is Best Buy. Home Depot's store jobs are 25526/5032, Walgreens
26336/5014. There is no robots.txt on the host (404), and the search the page
itself calls is plain JSON.

The two calls

The search page carries an anti-forgery token, and the Ajax calls want it as a
header named RFT. Without it every call is HTTP 500 with an HTML error page,
which is what makes this look closed when it is not.

import re, requests

s = requests.Session()
home = s.get("https://sjobs.brassring.com/TGnewUI/Search/Home/Home",
             params={"partnerid": "25632", "siteid": "5649"},
             headers={"User-Agent": "Mozilla/5.0"}).text
token = re.search(r'name="__RequestVerificationToken"[^>]*value="([^"]+)"', home).group(1)
H = {"RFT": token, "X-Requested-With": "XMLHttpRequest",
     "Content-Type": "application/json", "User-Agent": "Mozilla/5.0"}

first = s.post("https://sjobs.brassring.com/TgNewUI/Search/Ajax/MatchedJobs", headers=H, json={
    "PartnerId": "25632", "SiteId": "5649", "Keyword": "", "Location": "",
    "KeywordCustomSolrFields": "JobTitle,Location", "LocationCustomSolrFields": "Location",
    "FacetFilterFields": None, "TurnOffHttps": False, "Latitude": 0, "Longitude": 0,
    "PowerSearchOptions": {"PowerSearchOption": []}, "encryptedsessionvalue": ""}).json()

first["JobsCount"]                      # 2684
len(first["Jobs"]["Job"])               # 50

A job arrives as a list of question/value pairs rather than an object:

{q["QuestionName"]: q["Value"] for q in first["Jobs"]["Job"][0]["Questions"]}
# {'reqid': '3761989', 'jobtitle': 'Retail Sales Associate',
#  'lastupdated': '22-Sep-2026', 'formtext12': 'Greensboro',
#  'formtext10': 'North Carolina ', 'formtext1': 'Part time ', ...}

Pages come from a different endpoint, ProcessSortAndShowMoreJobs, whose keys
are lower-cased (partnerId, pageNumber). PageNumber on MatchedJobs
looks like it pages and does not.

The trap: the sort repeats pages

A site offers two sorts, date and title, and both are full of ties, so
consecutive pages overlap and skip. Sweeping Best Buy's 54 pages once by date
returned 2,299 distinct jobs of 2,680. Not a rate limit, not an error: just an
unstable sort under a paging API.

What works is sweeping more than once and joining on the requisition id:

date sweep                 2,299
title sweep                2,580
date + title + date        2,661   (99.3%)

Stop as soon as the union reaches the declared count, and most sites finish in
one sweep. Read that way, live on 2026-09-22: Best Buy 2,684 of 2,684 in 44
seconds, Performance Food Group 1,359 of 1,359, Home Depot 23,876 of 23,982 in
1,441 requests.

Which fields hold the place changes per site

The page config says what each site shows under a title:

"JobFieldsToDisplay": {"JobTitle": "jobtitle",
                       "Position3": ["formtext12", "formtext10", "formtext1"],
                       "Summary": "jobdescription"}

Best Buy shows city, state and schedule in custom fields. Home Depot has a
location field and uses it. Performance Food Group shows its division, then
"St Louis, Missouri (MO)", then a facility name, and its location field
holds the facility. Edward Jones shows the requisition number first, and
TeamHealth shows a medical specialty and no place at all.

A rule that survives all of those: prefer a shown value that reads as "City,
State"; else the site's own location; else join the shown values up to the
one naming a state, skipping ids and anything with a colon; and if none of
them names a state, publish no location rather than a wrong one.

How many employers are on it

The Wayback CDX index lists archived search pages: 727 partner/site pairs, 489
of which still answer, 120 with open roles. After dropping internal-only
sites, a template site and a network that re-posts other employers' jobs, 98
boards and 72,436 jobs, including Home Depot's stores (23,982), Walgreens
(22,446), Best Buy, Harbor Freight, Infosys, Performance Food Group, Hobby
Lobby, GardaWorld, ADM, Edward Jones, Publix, American Greetings, Bechtel,
GUESS, AAFES, U.S. Steel and Stantec.

One more thing worth knowing: an employer runs several sites over one partner
id (ADM in six languages, Edward Jones three, Walgreens two), and the same
requisition appears on more than one. The requisition is keyed per partner, so
partner + reqid is the job.

If you want the rows rather than the code, that is what
ats-jobs-scraper does: a
company name in, every open posting out, from BrassRing and twenty other
career systems, as JSON.

Workday says the board has 2,000 jobs. Target has 12,525. Reading the rest.

Daniel Meshulam — Fri, 25 Sep 2026 21:25:25 +0000

Ask a Workday career site for its jobs and the first answer states a total:

r = requests.post(
    "https://target.wd5.myworkdayjobs.com/wday/cxs/target/targetcareers/jobs",
    json={"appliedFacets": {}, "limit": 20, "offset": 0, "searchText": ""},
)
r.json()["total"]      # 2000

Target does not have 2,000 open jobs. It has 12,525. The number is the most
that search will return, and once you have paged to it the API just re-serves
page one: offsets past the ceiling come back as the first twenty postings
again, forever, which is how a paging loop that stops on "empty page" turns
into an infinite one.

Measured on 2026-09-22 across 3,916 Workday boards: 59 of them sit at exactly
that wall (a board shows 1,990-2,000 because postings of one requisition merge
into one job). Among them Target, Walmart, NVIDIA, Accenture, Citi, Airbus,
T-Mobile, PNC, Abbott, Kohl's, Sysco, Circle K and Trinity Health.

The counts are in the same answer you already have

The response carries a facets list, and those counts are of the whole board,
not of the 2,000:

Location_Country                 United States of America  12,452   India  73
Location_Region_State_Province   California 1,701   Texas 879   Florida 803 ...
jobFamilyGroup                   Stores 12,023   Internships 132 ...
workerSubType                    Regular 11,538   Seasonal 854   Intern 133
timeType                         Variable 11,427   Full time 1,098

Four of those sum to exactly 12,525. That agreement is the useful part: a
facet a posting carries once sums to the board, a facet it can carry several
times (skills, locations) sums higher, and one only some postings carry sums
lower. So the board's real size is the sum the most facets agree on.

Each facet value is its own search, with its own ceiling

appliedFacets narrows the search, and the narrowed search gets a fresh
2,000:

body = {"appliedFacets": {"Location_Region_State_Province": ["<value id>"]},
        "limit": 20, "offset": 0, "searchText": ""}

California is 1,701 postings, and 1,701 come back. So the board is read slice
by slice and the slices are joined on each posting's externalPath, which is
the posting's own address and therefore the one key that cannot collide:

Read the first page. If total is exactly 2,000, take the facets.
Work out the real size (the sum the most facets agree on).
Pick the facet that covers that size and whose every value is under the ceiling, fewest values first. Target: its 52 states.
Where one value is still over the ceiling, apply it and split again by a second facet. Petco needs this: 2,980 of its jobs are "Retail (Stores)", and inside that slice the schedule splits it 1,893 / 1,024.
Where no facet covers the slice at all, use the best partial one AND read the slice's own 2,000: Petco's schedule field is blank on 63 postings, and those come back in the plain read.

What it returns

Read whole, on 2026-09-22:

Target      12,499 of 12,536   655 requests
Michaels     5,071 of  5,071   279
Petco        3,774 of  3,798   298
Home Depot  (Workday corporate site, 1,031, no ceiling)

Walmart went from 1,990 rows to 21,202, Circle K from 2,000 to 11,397.

Two things worth keeping when you build this:

A failed slice falls back to the plain 2,000. Target is ~650 requests
against ~100, so a transient error is six times likelier; a caller who got
2,000 rows yesterday should not get an exception today for the sake of the
other 10,499.

Nothing splits under the ceiling. If the caller wants 1,000 rows, the
plain search already answers, and the split is pure cost.

The other thing in that answer

While measuring this I found that bulletFields[0], which most code (mine
included) uses as the job id, is not an id on 224 of those boards: Thales
prints "Regular Employee" there, Michaels the state, Cracker Barrel the
store's street address. Anything that de-duplicates on it collapses the board:
61,107 jobs became 16,314, and a run for Thales returned 7 rows for 200 jobs.
The requisition is in the posting's address instead
(/job/<place>/<Title-Slug>_<req>), and the bullet is only safe when it IS
that requisition.

If you want the rows rather than the code, this is what
workday-jobs-scraper
does: a company name in, every open posting out, past the ceiling, as JSON.

I bypassed my own Claude Code deny-list in eight ways. Only an allow-list held.

Daniel Meshulam — Fri, 11 Sep 2026 15:47:38 +0000

In April 2026 a coding agent at PocketOS hit a credential mismatch in staging, found an infrastructure token in an unrelated file, and deleted the production database and its volume backups in one API call. Thirty hours down. Every post-mortem named the same causes: a token within reach, no gate on destructive actions, backups in the same blast radius. Nobody blamed the model.

Most teams I talk to protect against this with a line in CLAUDE.md that says "never touch production". That is a sign, not a lock. So I built the lock, then spent an evening trying to break it. Here is what happened.

Round one: the deny-list

Claude Code has PreToolUse hooks: a small program that receives the tool call as JSON on stdin and answers with an exit code (0 allow, 2 deny). The obvious first guard for a protected folder is a deny-list. Block rm, mv, rmdir when the command mentions protected/. Ten lines. Done.

It took minutes to defeat. All of these wrote into the protected tree untouched:

Bypass	Command
Other verbs	`cp`, `tee`, `dd`, `sed -i`, `install`, `rsync` into `protected/`
Nested interpreter	`sh -c "rm protected/x"`, `perl -e 'unlink "protected/x"'`
Variable indirection	`P=protected; rm $P/x`
Subshell	`(cd protected && rm x)`
Acting flag	`find protected -delete`
Pipeline	`ls protected \
Clobber operator	{% raw %}`echo x >\
Symlink	{% raw %}`ln -s protected alias; rm alias/x`

Worse: two of the blocks the guard did produce came from a quoting parse error, not from detection. A guard that is wrong in both directions cannot be reasoned about at all.

Round two: invert the rule

The fix is not a longer deny-list. It is a different rule: referencing a protected path is denied unless every verb in the command is on a short read-only allow-list (cat, ls, grep, head, find without acting flags, and a few more). Anything unknown fails closed. A tool nobody predicted is covered by construction, because nothing has to be predicted.

Three details that only showed up under attack:

Resolve paths before comparing. ../protected/x, symlinks and cd in the middle of a command all change what a relative path means. The guard tracks the session cwd that Claude Code sends and resolves every token with realpath before deciding. Using the hook process's own cwd made the same command allowed or blocked depending on where the hook happened to be launched from.
Read every path key. The write guard originally read only tool_input.file_path. NotebookEdit sends notebook_path; MultiEdit nests paths under edits[]. Adding those tools to the matcher looked like coverage while the guard inspected nothing. A guard that is wired but cannot read its input is more dangerous than a missing one, because the settings file now lies to you.
Fail closed on confusion. Unparseable quoting, a missing config file, a malformed policy, a payload with no path: every one of these denies. A guard that waves something through when confused is worse than no guard.

Round three: destructive commands that touch no path

The path rule says nothing about git push --force, git reset --hard, curl | sh, terraform destroy or DROP TABLE. Those get a second layer: regex deny patterns checked on the raw command string, before any path logic, so quoting tricks around the path parser cannot route around them.

Prove it or it does not exist

Every bypass above is now a permanent regression test. The suite feeds each hook the JSON Claude Code would send and asserts the exit code, 125 assertions, and it snapshots and restores everything it mutates so a Ctrl-C mid-run cannot leave the repo in a broken state (the suite even kills a nested copy of itself at the dangerous moment to prove the trap works).

Every block also writes one JSON line to an audit log and, if you set a Slack or Discord webhook, posts the block to the team channel in the same call. A small script turns a month of that log into a report: how many agent actions were blocked, by reason, by tool, by day. A control nobody can see is a control nobody funds.

Postscript: the built-in rules had bypasses too

The Claude Code 2.1.268 changelog fixed deny and ask rules that did not apply on symlinked directories, and a case where a Read or Edit deny rule did not apply when env -C, eval or a similar command the permission checker cannot analyze was on the same line. That is not a criticism of the permission system; it is the reason to put a tested hook layer in front of it and to re-run the tests on every release.

Take it

The kit is public: agent-guardrails-kit. Copy hooks/, gates/ and config.json into a repo, edit protected_paths and deny_patterns, wire two hooks in .claude/settings.json, run the tests.

If you want to know where your current setup stands before touching anything, there is a grader that reads your settings.json and CLAUDE.md in the browser and scores it out of 10: agent-guardrails.meshulam791.workers.dev/grade. Nothing is uploaded. Here is what the full exposure report looks like on a fictional repo.

I install and test-prove this for teams on Claude Code and Cursor. Details on the same site.

Merging job postings from ten ATS platforms into one schema, and the four fields that fight back

Daniel Meshulam — Sat, 08 Aug 2026 11:56:22 +0000

Reading one ATS job board API is easy. Every one of them is public JSON with
no key. Reading ten and getting rows you can actually query together is a
different job, and it is almost entirely about four fields.

I run an index across Greenhouse, Ashby, Workable, Workday, BambooHR, Breezy,
Teamtailor, Personio, Recruitee and Rippling. Here is what actually broke,
with the measurements.

1. Remote is not a boolean, and one API will lie to you about it

Ashby postings carry isRemote. It looks like exactly the field you want.

Measured on Ramp's board, 1 August 2026: isRemote was true for 117 of 126
postings, while the same postings' workplaceType said Hybrid 101, Remote
16, OnSite 9.

Ashby's isRemote means "not fully on-site". Trust it and a user asking for
remote work gets a hundred hybrid roles at a New York office.

workplaceType is the honest field, and it has three values, because hybrid
is a real third state that a boolean cannot hold. Normalise to the three-way
enum and derive the boolean from it, never the reverse:

workplace_type = raw.get("workplaceType")            # Remote | Hybrid | OnSite
is_remote = (workplace_type == "Remote") if workplace_type else None

Note the None. A platform that publishes no arrangement at all gets unknown,
not False. "We do not know" and "we know it is not remote" are different
answers and collapsing them is how a filter silently drops good rows.

2. Dates arrive in three shapes, one of which is relative

ISO 8601 from some. Epoch milliseconds from others. And Workday hands you a
human sentence: "Posted 4 Days Ago", "Posted Today",
"Posted 30+ Days Ago".

Epoch handling has a cheap tell:

# Milliseconds since the epoch are 13 digits until the year 2286.
secs = n / 1000 if n > 10_000_000_000 else n

The relative ones need parsing, and one of them must be refused:

_AGO = re.compile(r"^posted\s+(\d+)\s+days?\s+ago$", re.I)

def workday_posted(text):
    t = (text or "").strip().lower()
    if t == "posted today":     days = 0
    elif t == "posted yesterday": days = 1
    else:
        m = _AGO.match(t)
        if not m:
            return None          # "posted 30+ days ago" lands here, on purpose
        days = int(m.group(1))
    return (datetime.now(timezone.utc).date() - timedelta(days=days)).isoformat()

"Posted 30+ Days Ago" returns None deliberately. It could be 31 days or

Turning it into "31 days" produces a date that sorts, filters and charts perfectly and is fiction. Keep the original string in a separate field and let the caller see what the source actually said.

3. Employment type can vary between two boards on the SAME platform

This one surprised me. It is not just that platforms disagree with each other.
Workday's timeType is tenant-configured, so two customers of the same ATS
publish different things.

Measured 2026-08-08 across ten Workday boards: two publish timeType on
every posting, and eight publish it on none. The only value that appeared at
all was "Full time".

So "does this platform have employment type" is not a question with an answer,
and any per-platform lookup table encodes a guess. Read the field where the
board provides it, map it through one shared vocabulary, and leave it null
everywhere else. I had this branch hard-coded to None for a while, which
meant 258,876 rows carried no employment type even where Workday had plainly
stated one.

The trap on the other side is a filter for "full-time" that silently excludes
the majority of your index, because most rows are null rather than false.
Filter on != 'part-time' if you mean "not part-time", and say in your docs
which one you did.

4. Company name is often not a company name

Several platforms return the board token where you expect the employer, so you
get acmecorp instead of Acme Corp, and on Workday you can get the hostname.
Pull the display name from the board envelope rather than the posting, and if
it is absent, say so instead of shipping a slug that looks like a name.

The rule underneath all four

Every one of these bugs has the same shape: a field existed, so it got
trusted. isRemote exists. A parsed "30+ days" exists. A board token
exists where a company name goes. Each one produces a value that is
well-formed, sorts correctly, and is wrong.

The fix is always the same and it is not technical. Carry unknown as unknown.
Three-state where reality has three states. Null where the source said nothing.
It makes your schema uglier and your answers true, and in this data the null
rate is itself information: it tells you which platform to stop promising
things about.

The index, if you want the rows rather than the code

53,000 company boards, 23 ATS platforms, 2,442,000 open roles in one schema,
rebuilt nightly, no login and no company list. On Apify as
ATS Jobs Search API, or
single-platform:
Greenhouse,
Workday,
Ashby,
Workable,
BambooHR.

If you want to see the merged fields on real rows before deciding whether the schema is worth it, every vertical page at glitchbound-jobs.pages.dev is built straight from it: game dev, nursing, devops.

Posted date is populated on 76.2% of rows and country on 78.0%. Those are the
real numbers, not the ones I would like, and the missing quarter is mostly
Workday's "30+ days" refusing to become a date.

DEV Community: Daniel Meshulam

Google Play top charts API: Top Free, Top Grossing and Trending apps in 130 countries

The charts there are

An unknown country is not an error

Reading them without writing the RPC

Company career page scraper: every open job from any company, by name or domain

Trap 1: the name on the board is not always the company

Trap 2: the same job, twice

Trap 3: filtering after you read misses most matches

Or skip all of it

Workday Jobs Scraper: every posting from any Workday careers site, by company name

1. You need the URL, and a company name does not give it to you

2. limit is 20, and total is a ceiling

3. Filtering after you read misses most of the matches

Or skip all of it

Google's News API has been gone since 2016, and people still search for it

The endpoint

Two sources are better than one

The code

Deduplication is most of the value

Where this does and does not fit

What it costs

The wider point

Code

Find startups the day they raise, before the press release, with 30 lines of Python

Why you can't just list Form Ds

The code

Real output

Trap 1: EDGAR silently ignores a half-open date range

Trap 2: "recent" isn't the sort you want for anything else

What this costs

Going further

Caveats worth stating plainly

Code

The App Store top charts are a public API, and nobody archives them

The feed

The part that is actually valuable

What it costs

Caveats worth stating plainly

Code

WHOIS is gone, RDAP replaced it, and a 404 does not mean what you think

What RDAP gives you, and what it does not

The code

The trap: a 404 is ambiguous

The fix, in two parts

Why this generalises

What it costs

Notes

Code

10,789 American employers publish their whole job list in one HTML page

Four things that bite

How many there are

BrassRing has a JSON job search, and Home Depot publishes 23,982 jobs through it

The two calls

The trap: the sort repeats pages

Which fields hold the place changes per site

How many employers are on it

Workday says the board has 2,000 jobs. Target has 12,525. Reading the rest.

The counts are in the same answer you already have

Each facet value is its own search, with its own ceiling

What it returns

The other thing in that answer

I bypassed my own Claude Code deny-list in eight ways. Only an allow-list held.

Round one: the deny-list

Round two: invert the rule

Round three: destructive commands that touch no path

Prove it or it does not exist

Postscript: the built-in rules had bypasses too

Take it

Merging job postings from ten ATS platforms into one schema, and the four fields that fight back

1. Remote is not a boolean, and one API will lie to you about it

2. Dates arrive in three shapes, one of which is relative

3. Employment type can vary between two boards on the SAME platform

4. Company name is often not a company name

The rule underneath all four

The index, if you want the rows rather than the code

2. `limit` is 20, and `total` is a ceiling