DEV Community: Truffle Pig Data

AI Visibility: Find the Pages Where You Rank but Google's AI Cites a Competitor

Truffle Pig Data — Wed, 16 Sep 2026 19:09:37 +0000

Ranking and being cited are two different problems, and Google Search Console only shows you the first one. You can sit at position 4 on a query, watch an AI Overview answer it with somebody else's page, and see nothing in your reports except a CTR that quietly sags. This post is about closing that gap with evidence: the AI Overview Rewrite Queue on Apify joins your Search Console export with live citation data and hands back a scored list of which pages to rewrite first.

Disclosure: the Apify links in this post are affiliate links. If you run the Actor, I may earn a referral commission at no extra cost to you.

Why citation tracking alone is not enough

Most AI visibility tools answer one question: were you cited. That is useful and incomplete, because "not cited" on its own does not tell you whether to act. Not cited on a query you rank 40th for is noise. Not cited on a query you rank 6th for, where a competitor is cited instead, is a page Google already understands that somebody else is answering better. Telling those apart needs your ranking data and your citation data in the same row, which is the join this Actor does.

What the rewrite queue returns

The API returns one row per query, sorted so the rewrites worth the most come first, with your Search Console metrics and live citation data side by side.

Field	Example	Notes
`query`	`what is a crm`	The keyword
`tier`	`A`	A to X, biggest opportunity first
`tier_reason`	One plain sentence	Explains the call without a legend
`position` / `impressions`	`8.4` / `1900`	From your export
`ai_overview_present`	`true`	Whether an overview appeared at all
`citation_state`	`competitor_cited`	Cited, competitor cited, or no references
`reference_domains`	`["reddit.com", "ibm.com"]`	Every domain the overview cited, in order
`join_status`	`matched`	Or `gsc_only` / `check_only`, never dropped

A whole-run summary lands in the key-value store: counts per tier, the join rate, and the competitor domains cited most often across your keyword set.

The five tiers, and what to do with each

A is the money tier: you rank 5 to 20 and a competitor is cited instead of you. Rewrite these first. B is the one people miss, an answer-shape problem rather than an authority problem: you rank top 4 and the AI still passes you over, so restructure the answer instead of chasing links. If citation simply tracked ranking, tier B would always be empty. C means you are cited and still converting below your own baseline, so the overview is satisfying the search. D is ordinary SEO. X means the check did not complete, which is evidence rather than a conclusion.

Tier C compares each query against your own median CTR at a similar position, computed from the no-overview queries in your export. It never uses an industry benchmark table, because every account's normal is different.

Who this is for

SEO and content people who want a rewrite backlog built from evidence rather than opinion. Anyone tracking AI visibility over time who has noticed that a single check is noise, since the same query can cite you Monday and drop you Thursday. And builders wiring an agent that can pull the tier A list and draft the rewrites in the same session.

The manual way, and where it breaks

You can do this by hand: export Search Console, search each query, note who got cited, paste it into a sheet. It works for twenty queries and falls apart at two hundred. Worse, it quietly produces bad data, because an overview that failed to load looks identical to a query with no overview and both get written down as a zero. That is the failure mode that matters in measurement tools, where a plausible zero reads as an answer instead of a bug, and you end up confidently deprioritizing a page you should have rewritten.

The faster way: run the queue

Open the AI Overview Rewrite Queue, put your domain in target_domains, and give it your Search Console export as a published-Sheet CSV URL or as pasted rows. No Google sign-in, no OAuth, no service account: it reads an export you already have. From the command line:

curl -X POST "https://api.apify.com/v2/acts/johnvc~ai-overview-rewrite-queue/runs?token=YOUR_APIFY_TOKEN" -H "Content-Type: application/json" -d '{ "target_domains": ["example.com"], "search_console_csv_url": "https://docs.google.com/spreadsheets/d/e/EXAMPLE/pub?output=csv", "min_impressions": 50 }'

Run mechanics are in the Apify API docs.

To get the export: open the Performance report, pick a date range, open the Queries tab, then Export. For scheduled runs, send it to Google Sheets and use File, Share, Publish to web, comma-separated values, so every run picks up fresh data. Localized headers, semicolon delimiters, comma decimals, and percent signs on CTR all parse without editing.

Score your queries in Python

from apify_client import ApifyClient
client = ApifyClient("YOUR_APIFY_TOKEN")
run = client.actor("johnvc/ai-overview-rewrite-queue").call(run_input={"target_domains": ["example.com"], "search_console_csv_url": "https://docs.google.com/spreadsheets/d/e/EXAMPLE/pub?output=csv"})
for row in client.dataset(run["defaultDatasetId"]).iterate_items():
    if row.get("tier") == "A":
        print(row["query"], row["position"], row["impressions"], row["reference_domains"])

Filter to tier A and you have this month's rewrite list in a dozen lines.

Read the queue from Claude via MCP

Through the Model Context Protocol the Actor is a callable tool in Claude, Claude Code, and Cursor. Ask for the tier A list and an agent can pull it and start drafting in the same session, which is a better loop than exporting a CSV and pasting it into a chat window. You can read more about Claude at claude.ai.

FAQ about AI Overview citation scrapers

How does this scraper tell a real "no AI Overview" from a failed check?

They are separate fields and are never conflated. ai_overview_present: false means the query was checked and no overview appeared. A null with check_status of retrieval_failed or blocked means it could not be checked, and those land in tier X where they can never be scored as a rewrite priority. A failure that reads as a zero is worse than no data at all.

What does this scraper cost to run?

Pay per event: a one-time setup charge per run covering the export parsing, the citation checks, and your CTR baseline, plus a scored-row event per query returned. One thing to know is that the citation checks run on the Google AI Overview API under your own account and are billed separately under that Actor's pricing. That is the larger part of a run's cost. Use min_impressions to keep the checked set to queries with traffic worth having.

Can I schedule this scraper to track citations over time?

Yes, and you should. A single check is a snapshot, and overviews are volatile enough that one run is not a verdict. Put a published Sheet URL in search_console_csv_url, attach a Schedule to the rewrite queue, and trend the tiers instead of reading one run.

Can Claude drive this scraper over MCP?

Yes. Connect the Apify MCP server and it becomes a callable tool in any MCP client, so an agent can fetch the queue and act on it without leaving the conversation.

What can this scraper not tell me?

Whether you were recommended, as opposed to cited. It reports citation, meaning a link to your domain in the overview's sources. Whether the surrounding text recommends you, merely mentions you, or contradicts you is a text-analysis question this does not answer. It also runs the checks from datacenter addresses, so set gl, hl, and location deliberately or a lost citation is indistinguishable from a different exit node.

Can an AI agent pay for this scraper in USDC with x402?

Yes. The AI Overview Rewrite Queue supports agentic payments via the x402 protocol, so AI agents and MCP clients can pay for runs in USDC on Base with no Apify account or API token. Point your agent at the Apify MCP server and it can discover, pay for, and run the scraper autonomously; the Apify x402 announcement has the details.

More from Truffle Pig Data

The citation checks behind this one, usable on their own for raw overview text and sources: Google AI Overview API. The same visibility question on a different answer engine: Brave AI Mode API. For Korean search: Naver AI Overview API. And to find the question-shaped queries that trigger overviews in the first place: Google Autocomplete API.

Wrapping up

A citation trendline can sit flat and green while your clicks fall, because the overview is answering for you either way. Export Search Console once, point the AI Overview Rewrite Queue at it, and open the tier A rows first.

Companies Using Workday: How to Build a Customer List with Live Open-Role Counts in 2026

Truffle Pig Data — Tue, 15 Sep 2026 19:08:58 +0000

Ask a sales rep or a recruiter which companies run their hiring on Workday and you get a spreadsheet somebody built two years ago. The career sites are public, but each sits at {tenant}.{datacenter}.myworkdayjobs.com with a datacenter you cannot guess, and a big employer runs several. I'll show the manual route and where it breaks, then the shortcut: the Workday Career Sites API, which turns a company name into one JSON row per career site with a live open-role count.

Disclosure: the Apify links in this post are affiliate links. If you run the Actor, I may earn a referral commission at no extra cost to you.

Does Workday publish a list of companies using Workday?

No. Workday's own APIs are tenant-scoped: they serve a customer's HR integrations, need that customer's credentials, and never say who the customers are. What it does publish is the sites, thousands of public careers pages on myworkdayjobs.com and myworkdaysite.com. So a companies-using-Workday list is a discovery job: find the tenants, resolve each to its boards, verify they answer.

What the Workday Career Sites API returns

The Workday Career Sites API returns one row per Workday career site as structured JSON: tenant, datacenter, site slug, public careers URL, direct jobs API URL, and the open-role count at run time.

Field	Example	Notes
`tenant`	`kla`	Hostname prefix
`datacenter`	`wd1`	Not guessable from the name
`careersUrl`	`https://kla.wd1.myworkdayjobs.com/en-US/Penang_Semicon`	Ready to open
`apiUrl`	`https://kla.wd1.myworkdayjobs.com/wday/cxs/kla/Penang_Semicon/jobs`	Jobs endpoint, ready to POST
`totalJobs`	`37`	Empty when `verifyLive` is off

siteSlug, hostKind, status, slugSource, companyName, and discoveredAt ride along. Not included: job postings, applicant data, or companies on another hiring platform.

Who this is for

HR-tech sales teams who want companies using Workday as a target list, with open-role counts as a buying signal; job-board and recruiting-data builders who need a company-to-career-site map; and anyone giving an AI agent live answers about who runs Workday and how hard they are hiring.

The manual way, and where it breaks

You guess. Assume the tenant matches the company name (it often doesn't), then try wd1, wd5, wd108 and the rest until a hostname answers. Then you hunt for site slugs, because the main board is rarely the only one: KLA runs seven Workday career sites, Cadence eight. Repeat for two hundred companies, then again next month. Bulk is worse: no list to start from. I follow semiconductor hiring, and those seven KLA boards are what pushed me to build a discovery step instead of maintaining the spreadsheet.

The faster way: run the Workday Career Sites API

Apify Console

Open the Workday Career Sites API and click Try for free.
Put company names or domains in companies, or switch on discoverAll.
Run it and download the dataset as JSON, CSV, or Excel.

REST

curl -X POST "https://api.apify.com/v2/acts/johnvc~workday-career-sites-api/runs?token=YOUR_APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{ "companies": ["nvidia", "kla"], "verifyLive": true }'

Run endpoint reference: the Apify API docs.

Find companies using Workday in Python

Call the Actor with apify-client and keep the career_site rows:

from apify_client import ApifyClient

client = ApifyClient("YOUR_APIFY_TOKEN")

run = client.actor("johnvc/workday-career-sites-api").call(
    run_input={"companies": ["nvidia", "kla", "cadence.com"], "verifyLive": True}
)

for site in client.dataset(run["defaultDatasetId"]).iterate_items():
    if site.get("resultType") != "career_site":
        continue
    print(site["tenant"], site["datacenter"], site["siteSlug"], site["totalJobs"])
    print("   ", site["apiUrl"])

Build a list of companies using Workday

Set discoverAll to true and the Actor enumerates Workday career sites in bulk, with maxResults capping how many companies it processes. It is a wide net, not a census: some smaller employers are missing from the public index, while a named company is resolved exactly. Build a List of Companies Using Workday is that configuration, ready to start.

A Workday customer list for sales prospecting

Same bulk mode, different reader: to an HR-tech seller a Workday customer list is an account list, and totalJobs shows which accounts are hiring this week. Start from Build a Workday Customer List for Sales and join tenant to your CRM.

Count open roles across Workday career sites

One totalJobs number is a snapshot; the same run weekly is a time series per site, and that trendline (a subsidiary board doubling, a regional board going quiet) is what people pay for. Start from Count Open Roles Across Workday Career Sites and keep every dataset.

Find a company's Workday careers URL

Pass one name or domain and get back a careersUrl you can open; a company that isn't on Workday produces no rows and no charge. See Find a Company's Workday Careers URL.

Accenture, Barclays, and Boeing: worked examples

Three large tenants, each a one-click task: Accenture, Barclays, and Boeing. Each returns the company's career sites with a jobs API URL and open-role count per board.

Find every Workday job board a company runs

Tools that stop at the first board miss the regional and university sites, each with its own openings. Find Every Workday Career Site a Company Runs returns all of them, one row each.

Get the Workday jobs API URL for a company

apiUrl is what most developers are after: a direct jobs endpoint per site. Get the Workday Jobs API URL for a Company hands it over; the sibling Workday Careers API consumes it to pull the postings. Discovery here, extraction there.

Resolve the Workday tenant name for a company

For nvidia.wd5.myworkdayjobs.com the tenant is nvidia, the company's label inside Workday. Get the Workday Tenant Name for a Company returns tenant and datacenter together, the pair you need to build any Workday URL.

Workday MCP: use it from Claude, Claude Code, and Cursor

Apify exposes the Actor through the Model Context Protocol, so Claude, Claude Code, and Cursor get a tool that takes company names and returns career sites; an agent can then answer "which of these ten companies run Workday, and how many roles does each have open" from live data. The server URL is:

https://mcp.apify.com/?tools=actors,docs,johnvc/workday-career-sites-api

In Claude Code that is claude mcp add --transport http apify plus the URL; read more about Claude Code at claude.ai.

The example repo

johnisanerd / Apify-Workday-Career-Sites-API

companies using workday: Python + MCP quick-start for the Workday Career Sites API on Apify. Call it from Python (uv) or as an MCP tool in Claude and Cursor. Returns structured JSON for companies using Workday.

🏢 Workday Career Sites API: Find the Companies Using Workday

Find the companies using Workday and every career site each one runs, as clean structured JSON: tenant, datacenter, careers URL, direct jobs API URL, and a live open-role count.

Actor page: apify.com/johnvc/workday-career-sites-api Input schema: apify.com/johnvc/workday-career-sites-api/input-schema

Give this API a company name and it returns every Workday career site that company runs, each with the tenant, datacenter, public careers URL, the direct jobs API URL, and how many roles are open right now. Turn on discovery mode and it enumerates companies using Workday in bulk, so you can build a Workday customer list for sales or research. It returns every board a company runs, not just the main one: large employers keep separate sites for regions, subsidiaries, and university hiring.

Video Walkthrough

Text walkthrough

The primary use case is finding the companies using Workday and resolving each to its career sites…

View on GitHub

A uv-managed Python quick start with companies_list, every_site, and jobs_api_url recipes, plus MCP install steps.

FAQ about scraping Workday career sites

What does the Workday career sites scraper cost, and is there a free tier?

Pay per career site returned, plus a tiny actor-start and per-result event; no monthly minimum, and the current per-site price is on the Store page. Only verified live sites are billed unless you turn on includeInactive, and new Apify accounts include free platform credit.

What is a Workday tenant, and how does the scraper find it?

The company's own label inside Workday, and the first part of its hostname on myworkdayjobs.com, where Workday hosts customer career sites. The scraper resolves a name or domain to tenant, then datacenter, then every site slug, with slugSource as a confidence signal.

Can an AI agent call this scraper through MCP?

Yes. Point Claude, Claude Code, Cursor, or ChatGPT at the hosted Apify MCP server with this Actor attached (the URL above) and it shows up as a callable tool.

Can I schedule the scraper to track open-role counts over time?

Yes. Save the input as a task, attach an Apify schedule (weekly, 0 7 * * 1, suits a customer list), and each run appends fresh totalJobs and discoveredAt values to chart. Start from the Workday Career Sites API.

Does the scraper return the job postings themselves?

No. It returns the career sites and an apiUrl per site; the Workday Careers API pulls the postings. It never returns applicant data, and a company on another hiring platform comes back with no rows.

More from Truffle Pig Data

Related Actors: the Workday Careers API for the postings behind each apiUrl, the Oracle Fusion Recruiting and Taleo Jobs API for employers on Oracle instead of Workday, and the LinkedIn Company API for firmographics to enrich the list you just built.

Wrapping up

Workday will not hand you a customer list, but the career sites are public and this Actor reads them into JSON. Try the Workday Career Sites API, or clone the example repo and point it at your watch list.

Google Maps Photos API: Pull a Place's Whole Gallery in 2026 (Python, Street View, MCP)

Truffle Pig Data — Mon, 14 Sep 2026 19:09:55 +0000

Every business on Google Maps carries a gallery: the storefront, the interior, the dishes, the customer snapshots, the Street View panorama. Getting one photo out means right-clicking. Getting all of them, for a hundred places, means writing a scraper. I wanted the second thing and did not want to maintain the scraper, so I built the Google Maps Photos API on Apify, which turns a place name into one JSON row per photo.

Disclosure: the Apify links in this post are affiliate links. If you run the Actor, I may earn a referral commission at no extra cost to you.

Does Google Maps have a photos API?

Sort of, and the gap is the interesting part. Google's Places API has a Photo endpoint, but it returns a rendered image for a reference you already hold, and that reference comes from a separate Place Details call you also pay for. There is no "give me this restaurant's whole gallery, sorted, with stable IDs" call. Street View Static is the same shape: a rendered tile at coordinates you supply, not a listing of what is attached to a place. So the thing people mean by a Google Maps photos API is not what the official product does.

What the Google Maps Photos API returns

The Google Maps Photos API returns one row per photo as structured JSON: full-size image URL, thumbnail, gallery position, and a stable ID you can de-duplicate on across runs.

Field	Example	Notes
`image`	`https://lh3.googleusercontent.com/...`	Full-size photo URL
`thumbnail`	`https://lh3.googleusercontent.com/...=w200`	Smaller preview of the same shot
`position`	`3`	Where it sits in the place's gallery
`photo_id`	`AF1QipM...`	Stable across runs, so you can dedupe
`place_title`	`Mozart's Coffee Roasters`	Present when the place was found by search
`category_id`	`CgIgARICCAI`	The gallery section filter that was applied

Each place also gets one summary row carrying photos_returned, pages_fetched, and place_categories, the full list of gallery sections with the filter ID for each. That last field is how you discover a venue's own sections and then pull just one.

It does not return captions, EXIF, contributor names, or upload dates. If you need provenance, this is not the tool.

Who this is for

Listings sites that need real imagery instead of one hero thumbnail. Travel and hospitality apps pulling interiors and menus. Retail and real estate teams auditing what a location looks like online.

The manual way, and where it breaks

Load the place page in a headless browser, click into the photo grid, scroll until it stops loading, pull src off the tiles. It works the first afternoon. Then: the gallery lazy-loads in blocks, so you are scripting scroll-and-wait loops; image URLs are size-parameterized, so you must know which suffix gives the original; category tabs render client-side with opaque IDs; and the markup shifts often enough that selectors rot in weeks.

The faster way: run the Google Maps Photos API

Documented JSON in, documented JSON out, nothing to keep alive.

Apify Console

Open the Google Maps Photos API and click Try for free.
Type a place into searchTerm, or paste a Google Maps URL into placeUrls.
Run it and download the dataset as JSON, CSV, or Excel.

REST

curl -X POST "https://api.apify.com/v2/acts/johnvc~google-maps-photos-api/runs?token=YOUR_APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{ "searchTerm": "Mozart'"'"'s Coffee Roasters, Austin TX", "maxPlacesPerSearch": 1, "maxPhotosPerPlace": 20 }'

Run endpoint reference: the Apify API docs.

Get Google Maps photos in Python

Photo rows and the per-place summary arrive in the same dataset, separated by result_type:

from apify_client import ApifyClient

client = ApifyClient("YOUR_APIFY_TOKEN")

run = client.actor("johnvc/google-maps-photos-api").call(
    run_input={
        "searchTerm": "Mozart's Coffee Roasters, Austin TX",
        "maxPlacesPerSearch": 1,
        "maxPhotosPerPlace": 20,
    }
)

items = list(client.dataset(run.default_dataset_id).iterate_items())

for photo in [i for i in items if i.get("result_type") == "photo"]:
    print(photo["position"], photo["photo_id"], photo["image"])

for place in [i for i in items if i.get("result_type") == "place_summary"]:
    for category in place.get("place_categories", []):
        print(category["id"], category["title"])

Note run.default_dataset_id: apify-client 3.x returns a typed Run object, not a dict, which trips up older snippets.

Street View and 360 panoramas without coordinates

This is the capability I did not expect to be the most useful one. Set photoCategory to street_view and you get the panoramas attached to a place, addressed by name rather than by coordinates. Street View Static wants a latitude, longitude, and heading, and getting from "this restaurant" to "the right pano facing the right way" is its own problem. The worked example is Get Street View and 360 Photos for Any Place. The videos category works the same way.

Bulk photo download across a list of places

maxPlacesPerSearch turns this from a one-place lookup into a sweep. A searchTerm of coffee shops in Austin, TX resolves to many places and pulls each gallery in one run. Two tasks cover the shapes: Bulk Download Every Photo for a List of Places when you already have the list, and Photos for Every Business in a Neighborhood when you want the search to find them.

Menu photos, and telling owner uploads from customer snapshots

Food and drink venues expose extra gallery sections, so photoCategory: menu gets the menu boards and dish shots without the parking lot: Get Menu and Food Photos for Any Restaurant, or the Chinese-language 获取餐厅菜品照片和店内实拍图. And by_owner separates what the business posted from what customers uploaded, which is the whole question in an imagery audit: Get Owner-Uploaded Photos for a Business.

Use it from Claude and other MCP clients

Apify exposes the Actor over the Model Context Protocol, so Claude, Claude Code, and Cursor can pull a gallery mid-conversation and answer "show me the interior shots for this hotel" with real URLs. More on Claude Code at claude.ai.

The example repo

johnisanerd / Apify-Google-Maps-Photos-API

📸 Google Maps Photos API: every place photo as clean JSON

Pull every photo Google Maps holds for a business or landmark, as structured JSON. Type a place name and get back full-size image URLs, thumbnails, gallery positions, and a stable photo ID for each one.

Actor: Google Maps Photos API on Apify Store

Video Walkthrough

https://www.youtube.com/watch?v=jREWahDGhJM

Quick Start

git clone https://github.com/johnisanerd/Apify-Google-Maps-Photos-API.git
cd Apify-Google-Maps-Photos-API
cp .env.example .env          # paste your Apify token
uv sync
uv run google-maps-photos-api-example.py

Get a free Apify API key at apify.com.

Why Use This Google Maps Photos API?

No place ID needed. Search by name, or by category and city. Paste a Google Maps URL if you have one. Identifiers work too, and skip the search charge.
Filter to one gallery section. Menu, food and drink, vibe, by owner, videos, or Street View and 360.
Stable photo IDs. Every row carries one, so a re-run…

View on GitHub

Python quick start, the category-filter variants, and MCP setup for Claude, Cursor, and ChatGPT.

FAQ about scraping Google Maps photos

Is the Google Maps photos scraper free, and how much does it cost?

Billing is per photo returned, plus a small run start fee, so cost scales with what you receive rather than with pages fetched. Searching by name adds a per-search charge; passing a Google Maps URL or a place identifier skips it, which is the cheapest way to run at volume. New Apify accounts come with free platform credit, so a first run usually costs nothing.

How many photos will this scraper return for one place?

It varies, from a handful for a small office to several hundred for a busy restaurant. Photos arrive in blocks of twenty and maxPhotosPerPlace is your cap. The summary row reports photos_returned.

Can I run this scraper from Claude or another MCP client?

Yes. Connect the Apify MCP server and the Actor becomes a callable tool, so an agent pulls a gallery straight from your prompt.

Can I schedule this scraper to refresh galleries?

Yes, and the stable photo_id is what makes it worth doing: save a task, attach an Apify schedule, and dedupe on that ID to see only what is new. Start from the Google Maps Photos API.

Do the scraper's image URLs expire, and can it get historical photos?

The URLs are Google-hosted and can rotate, so re-host anything you intend to display long term rather than hotlinking. And no, it cannot give you a place's photos from five years ago: it returns the current gallery with no upload dates. Historical imagery belongs to Street View's own timeline, a different product.

Can an AI agent pay for this scraper in USDC with x402?

Yes. The Google Maps Photos API supports agentic payments via the x402 protocol, so AI agents and MCP clients can pay for runs in USDC on Base with no Apify account or API token. Point your agent at the Apify MCP server and it can discover, pay for, and run the scraper autonomously; the Apify x402 announcement has the details.

More from Truffle Pig Data

Same shape of JSON: Google Maps Places API to find the places whose photos you want, Google Maps Contributor Reviews API for what people said, and Apple Maps API for a second source.

Wrapping up

A place's gallery is public, structured, and annoying to collect by hand, a good description of most things worth automating. Try the Google Maps Photos API, or clone the example repo and point it at your own list of places.

iCIMS API: Pull Live Job Postings From Any Career Site in 2026

Truffle Pig Data — Sun, 13 Sep 2026 19:29:51 +0000

About a quarter of the Fortune 500 runs hiring through iCIMS, which means a huge share of real, current job postings sit on employer career sites that no public API will hand you. I wanted those postings as JSON, from the source rather than from an aggregator's week-old copy. This post covers why iCIMS is awkward to read, what breaks when you script it yourself, and the shortcut: the iCIMS API on Apify, which turns a career-site address into one JSON row per open role.

Disclosure: the Apify links in this post are affiliate links. If you run the Actor, I may earn a referral commission at no extra cost to you.

Does iCIMS have a public jobs API?

Not one you can sign up for. iCIMS sells API access to its own customers so they can read their own data, which is the right tool if you own the account. There is no public endpoint for reading an employer's live postings from outside. So the workable path is a scraper you call like an API: give it a career site, get that employer's open roles back as structured JSON. It is an unofficial tool, it is not affiliated with iCIMS, and it reads only pages that are already public.

What the iCIMS API returns

The iCIMS API returns every live job on a career site as a JSON row with title, employer, locations, requisition ID, salary, dates, the full description, and the apply link.

Field	Example	Notes
`url`	`https://careers-rambus.icims.com/jobs/23006/job`	Canonical, stable across title edits, the key to dedupe on
`id`	`2026-23006`	The employer's own requisition ID, not a number from the URL
`title`	`Design Verification Principal Engineer`	As published
`organization`	`Rambus`	Hiring employer
`locations_derived`	`["Hillsboro, OR, United States"]`	Readable City, Region, Country strings
`date_posted` / `date_updated`	`2026-08-03T04:00:00+00:00`	ISO-8601; `date_updated` is the field to monitor
`salary_raw`	Pay range as published	Never inferred
`apply_url`	Direct application link

Descriptions come back three ways: description_text, description_html, and description_markdown. The Markdown one is the friendliest if you are piping a job description into a model.

Who this is for

People tracking hiring at named accounts, who want a new requisition the day it appears rather than when a board syndicates it. Teams building or backfilling a job board, who need real posted dates and working apply links. And anyone doing recruitment market research across many employers on the same platform, where per-employer onboarding is the thing that kills the project.

The manual way, and where it breaks

Reading one career site by hand is fine; the trouble starts at scale. There are two kinds of iCIMS site, classic portals on icims.com and modern career sites on the employer's own domain, and they need different handling. iCIMS prints no posted date on the page, so a naive parser returns jobs with no usable date. Job URLs change when an employer edits a title, so dedupe quietly breaks and yesterday's role reappears as new. And some employers gate parts of the site to trusted networks, which looks like an empty result rather than an error. I lost an afternoon to that one before working out the site was fine and my code was being told to go away.

The faster way: run the iCIMS API

In the Apify Console, open the iCIMS API, click Try for free, paste a career-site root into startUrls or a bare company name into companies, then run it and export as JSON, CSV, or Excel. Every common address form works and the right approach is picked automatically: a site root, a sitemap.xml, a /jobs/search page, a single job URL, or a modern career site on the employer's domain.

From the command line it is one call:

curl -X POST "https://api.apify.com/v2/acts/johnvc~icims-careers-api/runs?token=YOUR_APIFY_TOKEN" -H "Content-Type: application/json" -d '{ "startUrls": [{"url": "https://careers-rambus.icims.com"}], "maxJobsPerSite": 50, "descriptionFormat": "both" }'

Run mechanics are in the Apify API docs.

Pull iCIMS jobs in Python

from apify_client import ApifyClient
client = ApifyClient("YOUR_APIFY_TOKEN")
run = client.actor("johnvc/icims-careers-api").call(run_input={"companies": ["rambus"], "maxJobsPerSite": 25})
for job in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(job["id"], job["title"], job.get("locations_derived"), job.get("date_posted"))

Swap companies for a list and you are reading several employers in the same run.

Track new postings without re-reading the whole site

Turn includeDetails off and you get list-only mode: job URL, requisition ID, and last-changed timestamp for every live job, in a single request per career site. Pair it with newerThan and you have a cheap daily change feed. The pattern that works is a list-only run every morning, a comparison of date_updated against what you stored yesterday, then a second run with details on only the jobs that moved.

Use it as an ATS API for your own tools

Most ATS API products are unification layers: you pay an integration vendor, connect an employer's account, and read their private data with permission. This is the other half of that problem. It reads what an employer has already published publicly, so there is no account to connect and no setup per employer, which is what makes "who is hiring in this market" a single run instead of a per-employer integration project. The companies field takes bare names, so ["rambus", "sas"] resolves without hunting for URLs, and duplicates across inputs are removed for you. For storage, url is a stable primary key and date_updated is the column to upsert on.

Read iCIMS jobs from Claude via MCP

Through the Model Context Protocol the Actor becomes a callable tool in Claude, Claude Code, and Cursor, so "what has Rambus posted in engineering this week" runs a live pull instead of guessing from training data. The input and output schema descriptions are written for a model to read, so an agent can work out which parameters it needs without much prompting. You can read more about Claude at claude.ai.

johnisanerd / Apify-iCIMS-Careers-API

iCIMS API example: pull live job postings from any iCIMS career site with Python or MCP. Covers both public iCIMS surfaces, with change detection so a daily poll only costs the jobs that moved.

iCIMS API: Pull Live Job Postings From Any iCIMS Career Site

iCIMS does not hand out a public read API for job postings, so most teams end up writing a one-off parser per employer. This repo shows you the shortcut: a working Python example plus MCP install steps for the iCIMS Careers API on Apify.

Point it at a career site, get back structured jobs: title, requisition ID, employer, locations, employment type, posted and updated dates, salary when the employer publishes it, and the apply link. It reads both public iCIMS surfaces, the classic careers-{tenant}.icims.com portals and the modern iCIMS career sites that live on the employer's own domain, so you do not have to know which one an employer runs.

Get a free Apify API key here: https://apify.com?fpr=9n7kx3

Text walkthrough

The icims api you are looking for is this Actor. You give it one thing, a career site, either…

View on GitHub

FAQ about scraping iCIMS job postings

How much does the iCIMS scraper cost to run?

Billing is per job returned, with a small run-start charge and no subscription. There are three per-job rates: full postings at the standard rate, list-only jobs much cheaper because the whole site is read in one request, and proxied jobs higher because iCIMS serves job pages uncompressed. Error rows are not billed at any of the three, so a career site that cannot be reached does not charge you for jobs it never returned.

Can Claude or another agent drive this scraper over MCP?

Yes. The Actor is MCP-server-compatible, so Claude, Claude Code, Cursor, and other MCP clients call it as a tool and pull live postings inside an agent workflow.

Can I schedule the scraper to catch new jobs daily?

That is the intended use. Save an input as a Task, attach a Schedule, and pair a nightly list-only run with newerThan to get a change feed for a set of employers. Start from the iCIMS API and route the output through webhooks, n8n, Make, or Zapier.

Which career sites can this scraper not reach?

Ones restricted to trusted networks. You get a labelled error row with error_code set to ip_gated rather than a silent gap. Residential proxies clear many of them, but employers who allow only their own corporate network cannot be reached from outside by anything.

Is this scraper affiliated with iCIMS, and is the data really public?

No affiliation, and yes. It reads career-site pages that are already open to any browser, with no login and no API key. You are responsible for your own use, including local law and platform terms.

Can an AI agent pay for this scraper in USDC with x402?

Yes. The iCIMS API supports agentic payments via the x402 protocol, so AI agents and MCP clients can pay for runs in USDC on Base with no Apify account or API token. Point your agent at the Apify MCP server and it can discover, pay for, and run the scraper autonomously; the Apify x402 announcement has the details.

More from Truffle Pig Data

The same job for the other big enterprise ATS: Workday Careers API. From the destination side of the market rather than the employer side: LinkedIn Jobs API. And for scoring the companies you turn up: Glassdoor Reviews API.

Wrapping up

Employer career sites are the original source, ahead of every board that syndicates from them. Point the iCIMS API at one, or clone the example repo and start from working Python.

Sold Property Prices by Suburb: Pull realestate.com.au Listings as JSON in 2026 (Python, MCP, No-Code)

Truffle Pig Data — Sat, 12 Sep 2026 19:09:19 +0000

You can look up what one house sold for on realestate.com.au, but nothing there lets you pull a whole suburb's sales into a table. There is no public API, and the price on a sold listing is usually an estimate, not a disclosed figure. I'll show the manual route and where it breaks, then the shortcut: the Realestate.com.au Property API on Apify, which takes a suburb and returns sold, rental, or for-sale listings as JSON.

Disclosure: the Apify links in this post are affiliate links. If you run the Actor, I may earn a referral commission at no extra cost to you.

Does realestate.com.au have an API?

Not a public one. There is no open developer API to sign up for and no key to request, so "realestate.com.au api" searches are mostly people asking for access that does not exist. In practice it means a scraper you consume like an API: suburb in, listing rows out as JSON.

What the Realestate.com.au Property API returns

The Realestate.com.au Property API returns one row per listing as structured JSON: address, property type, beds, baths, parking, land size, a price estimate, the confirmed sold date and selling agency on sold rows, the advertised rent on rental rows, agent details, and coordinates.

Field	Example	Notes
`soldDate`	`2021-08-03T00:00:00.000Z`	Confirmed, on sold rows
`estimatedPrice`	`$585,000`	The source's estimate; `estimatedPriceValue` is `585000`
`lastSoldAgency`	`Harcourts RG - Gold Coast`	Selling agency
`bedrooms`	`4`	`bathrooms` and `parking` too
`landSizeValue`	`454`	Parsed from `454m²`
`rentPrice`	`$650 per week`	Rental rows, with `rentCurrency`

Every row also carries a one-line summary an AI agent can read as is.

Who this is for

Buyer's agents and investors researching a suburb's sold market before they bid. Analysts benchmarking the Australian rental market by suburb. Agency principals who want to know which office holds a suburb's market share.

The manual way, and where it breaks

The DIY version is to build the sold-listings URL for a suburb, fetch each results page, and parse the cards. Pagination has to be walked by hand. Prices arrive as text, often "Contact Agent" or "AUCTION" on for-sale listings, so your parser needs a null path from day one. Several Australian suburbs share a name, so a bare suburb quietly gives you the wrong market. The markup shifts and your selectors rot with it. Add blocking and retries, and you own infrastructure to get a table of sales.

The faster way: run the Realestate.com.au Property API

Documented input in, documented rows out, no listing URL to find first.

Apify Console

Open the Realestate.com.au Property API and click Try for free.
Leave Mode on search, set Listing type to sold, and enter Coomera, QLD, 4209.
Run it. The Sold properties view shows dates and agencies; export as JSON, CSV, or Excel.

REST

curl -X POST "https://api.apify.com/v2/acts/johnvc~realestate-au-property-api/runs?token=YOUR_APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{ "mode": "search", "listingType": "sold", "locations": ["Coomera, QLD, 4209"], "maxResultsPerSearch": 50 }'

Run endpoint reference: the Apify API docs.

Get sold property prices in Python

Call the Actor with apify-client:

from apify_client import ApifyClient

client = ApifyClient("YOUR_APIFY_TOKEN")

run = client.actor("johnvc/realestate-au-property-api").call(
    run_input={
        "mode": "search",
        "listingType": "sold",
        "locations": ["Coomera, QLD, 4209"],
        "maxResultsPerSearch": 50,
    }
)

for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    if item.get("result_type") != "listing":
        print("no listings:", item.get("error_message"))
        continue
    print(item["streetAddress"], item.get("soldDate"), item.get("estimatedPrice"), item.get("lastSoldAgency"))

The result_type check is the piece I'd keep even in a throwaway script. An empty search comes back as one row with result_type: "error" and a plain-language error_message, with no listing charged, so a zero for a suburb is verifiable rather than a guess.

Realestate.com.au property data as JSON, any listing type

The general-purpose starting point is Realestate.com.au Property Data API: Listings as JSON: any suburb, buy, rent, or sold, one output shape for all three. Up to 20 locations fit in one run, and maxResultsPerSearch (1 to 2000) is the cost dial.

Check sold property prices in any Australian suburb

Check Sold Property Prices in Any Australian Suburb is the flagship: every recent sale the source holds for a suburb, up to your cap, with date, agency, and estimate. Run it weekly and keep each dataset, and after a couple of months the weekly sold count and median estimate become a trendline. The trendline is the product; the one-off check is the demo.

Track the Australian rental market by suburb

Flip listingType to rent and the same suburb returns current rentals with rentPrice and rentCurrency beside beds, baths, and parking, which is what a yield calculation needs. Track Weekly Rent Prices by Australian Suburb is built to be scheduled. The sold and rental searches also exist as Chinese-language tasks: sold listings and price estimates and rental listings and weekly rents.

Find which agency sold each home in a suburb

Sold rows carry lastSoldAgency, so a sold search grouped by that field is an agency market-share table, which is what Find Which Agency Sold Each Home in an Australian Suburb does. The agents block adds phone, rating, and review count; the individual agent's name is often missing, so group at agency level.

Export an Australian property dataset to CSV or Excel

Not everything needs code. Export an Australian Property Dataset to CSV or Excel pulls a suburb into a spreadsheet from the Console through the two built-in views; grab the full JSON export when you need land size, coordinates, or agents.

Use it from Claude and other MCP clients

Apify exposes the Actor through the Model Context Protocol, so Claude, Claude Code, and Cursor can run a suburb search mid-conversation and answer "which agency handled the most sales in Coomera QLD last year" with real rows. In Claude Code it is one command:

claude mcp add --transport http realestate-au "https://mcp.apify.com/?tools=actors,docs,johnvc/realestate-au-property-api"

Get Australian Sold Property Data in Claude via MCP is the shortest path to trying it, and you can read more about Claude Code at claude.ai.

The example repo

johnisanerd / Apify-Realestate-AU-Property-API

Realestate.com.au Property API on Apify: sold property prices, rentals, and for-sale listings by suburb as structured JSON. Python (uv) quick-start plus MCP install guides for Claude, Cursor, and ChatGPT.

🏡 Realestate.com.au Property API: sold property prices, rentals, and for-sale listings by suburb

Give it an Australian suburb. Get back structured JSON: sold property prices with confirmed sale dates, rental listings with advertised rent, and for-sale stock, all from one API with no listing URL to find first.

Actor page: apify.com/johnvc/realestate-au-property-api Input schema: apify.com/johnvc/realestate-au-property-api/input-schema

This repo is a working Python client for the Realestate.com.au Property API on Apify. The API turns Australian property listings into clean rows: street address, suburb, state, postcode, property type, beds, baths, parking, land size, floor area, agent details, photo URLs, and latitude and longitude. Ask it for sold and you get sold property prices with a confirmed soldDate and the selling agency. Ask it for rent and you get the advertised rent, which is how you track the australia rental market suburb by suburb. Ask it for buy and you get current australian property listings.

…

View on GitHub

A working Python client with four example runs (sold, rent, buy, and URL mode), each capped at 3 listings so a first run costs almost nothing, plus MCP install walkthroughs for Claude Cowork, Claude Code, Cursor, and ChatGPT.

FAQ about scraping realestate.com.au

What does the realestate.com.au scraper cost, and is there a free tier?

Billing is per event: one listing-scraped event per listing pushed to the dataset, plus a tiny actor-start event. A search that returns nothing is not charged, and maxResultsPerSearch caps a run's cost before it starts. Current rates are on the Store card, and new Apify accounts come with free platform credit.

How do I check what a property sold for with this scraper?

Set listingType to sold and search the suburb with its state and postcode. Each row carries the confirmed soldDate, the lastSoldAgency, and the price estimate. For one address, switch mode to url and pass the listing page in listingUrls.

Does the scraper return the actual sale price or an estimate?

An estimate. The sold date is confirmed; the price is the source's displayed estimate, because Australian sale prices are frequently not disclosed. Sold and rental rows almost always carry a number, for-sale rows often do not, so filter on a non-null estimatedPriceValue before averaging a buy search. There is no automated valuation in the output.

Will this scraper find commercial property sold prices?

No. It covers the residential listings on realestate.com.au, where buy, rent, and sold come from. Commercial stock in Australia lives on a separate site and is out of scope.

Does the scraper work as an MCP tool in Claude or Cursor?

Yes. Point any MCP client at the Apify MCP server with the URL above and the Actor appears as a callable tool in Claude, Claude Code, Claude Cowork, or Cursor; the example repo has the install steps for each.

How do I schedule the scraper to build a sold-price history?

Save one task per suburb or listing type, open Actions, choose Schedule, and attach a cron string such as 0 9 * * 1 for a Monday review; one schedule can trigger many tasks. Diff each run's dataset against the last on propertyId: a property that moves from buy to sold is a sale. Start from the Realestate.com.au Property API.

More from Truffle Pig Data

Related Actors with the same kind of JSON output: the Zoopla UK Property API for listings and sold house prices in Britain, the Google Maps Places Scraper for the schools, cafes, and transport around a shortlisted address, and the Google Maps Directions API for commute times from a shortlist.

Wrapping up

There is no public realestate.com.au API, but sold property prices, rentals, and for-sale stock for any suburb are one JSON call away. Try the Realestate.com.au Property API, or clone the example repo and point it at your own suburbs.

Greenhouse Job Board API: Track Who's Hiring Across 4,400+ Companies in 2026

Truffle Pig Data — Fri, 11 Sep 2026 19:09:33 +0000

Companies that hire through Greenhouse post their openings on a hosted board, like GitLab's Greenhouse job board, and those postings are the closest thing to a live headcount signal you can get. Getting them out means one board at a time, HTML descriptions, pay buried in prose, and no list of who uses Greenhouse. I'll show the manual route and where it breaks, then the shortcut: the Greenhouse Job Board API on Apify, which returns live jobs from any Greenhouse board as JSON with no Harvest key, and finds the 4,400+ companies hiring there.

Disclosure: the Apify links in this post are affiliate links. If you run the Actor, I may earn a referral commission at no extra cost to you.

Does Greenhouse have an API for job postings?

Yes, two. The Harvest API is Greenhouse's private one for employers; no key for a company you don't work at. The Job Board API is public and read-only, documented on Greenhouse's developer site for integrators building application flows, and for one company, once, just call it. It serves one board per call, though, you need the token first, and Greenhouse publishes no list of boards. The Actor fills those gaps: a directory of 4,400+ verified boards, parsed salaries, Markdown, timestamp filters, and rows filtered before you pay.

What the Greenhouse job board API returns

The Greenhouse Job Board API returns one row per open job as structured JSON: title, company, departments, offices, location, remote flag, employer timestamps, raw and parsed salary, and a Markdown description.

Field	Example	Notes
`title`	`Account Executive - Italy`	Plus `companyName`, `boardToken`, `id`, `url`
`location`	`Remote, Italy`	With `isRemote`, `departments`, `offices`
`dateUpdated`	`2026-08-03T16:43:10-04:00`	Employer's stamp; `datePublished` too
`salaryDerived`	`{ "min": 71400, "max": 126000, "currency": "EUR" }`	Parsed; `salaryRaw` is verbatim

Who this is for

Sales and GTM teams who read hiring as a buying signal; recruiters pulling every open role at target companies; talent analysts tracking posting velocity and pay transparency; and anyone handing live job data to an AI agent.

The manual way, and where it breaks

The DIY route is a script that calls the public endpoint per board token and diffs the JSON against last night's file. It works for three companies. At thirty you learn a token is not the company's name and start reading page source. Descriptions come back as HTML, the salary is a sentence inside it, and your regex misses the EUR postings. Your diff needs a database, and "every company on Greenhouse" has no endpoint at all. None of this is hard. All of it is maintenance, and you wanted a dataset.

The faster way: run the Greenhouse Job Board API

Apify Console

Open the Greenhouse Job Board API and click Try for free.
Put board tokens or board URLs in companies; add titleKeywords or updatedAfter for a subset.
Run it and download the dataset as JSON, CSV, or Excel.

REST

curl -X POST "https://api.apify.com/v2/acts/johnvc~greenhouse-job-board-api/runs?token=YOUR_APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{ "companies": ["gitlab", "stripe"], "titleKeywords": ["engineer"], "maxJobs": 10 }'

Endpoint reference: the Apify API docs.

Scrape Greenhouse job postings in Python

Call the Actor with apify-client and keep the rows whose resultType is job:

from apify_client import ApifyClient

client = ApifyClient("YOUR_APIFY_TOKEN")

run = client.actor("johnvc/greenhouse-job-board-api").call(
    run_input={
        "companies": ["gitlab", "stripe"],
        "titleKeywords": ["engineer"],
        "maxJobs": 10,
    }
)

for row in client.dataset(run["defaultDatasetId"]).iterate_items():
    if row.get("resultType") != "job":
        continue
    salary = row.get("salaryDerived") or {}
    print(row["title"], row["companyName"], row.get("location"), row["dateUpdated"])
    print("   ", salary.get("min"), salary.get("max"), salary.get("currency"), row["url"])

Use the Greenhouse job board API without a key

Nothing to sign up for. The default run reads Stripe's board with Markdown on and maxJobs at 100; the task Use the Greenhouse Job Board API Without a Key is that configuration.

A job board scraper for Greenhouse career sites

Paste whatever you have, board tokens, board URLs, embed URLs, or single job URLs, mixed freely across companies and startUrls. outputMode: urlsOnly gives a cheap index: id, title, company, link, timestamps. The task is Job Board Scraper for Greenhouse Career Sites.

Greenhouse job postings with salary data

Published pay ranges pass through verbatim as salaryRaw; a deterministic parser adds salaryDerived (min, max, currency, period) from the posting text. See Greenhouse Job Postings With Salary Data.

Find companies using Greenhouse

There is no page that lists all Greenhouse open jobs, so the Actor ships one. outputMode: companiesOnly returns one row per company from the bundled directory, live-verified, with boardToken, boardUrl, and a current jobCount; dead boards are never billed. discoveryQuery text-matches company names and maxCompanies caps a blind sweep. This is the run I do first when I want to know who is quietly staffing up in a niche. Task: Find Companies Using Greenhouse ATS.

Track new Greenhouse jobs daily

Every job carries the employer's own published and updated stamps, so updatedAfter: "25h" on a daily schedule returns only what changed since yesterday, no seen-list needed; publishedAfter narrows that to new roles. One run is a snapshot. A scheduled run is a time series, and that shows a company opening twelve engineering roles before any press release. Task: Track New Greenhouse Job Postings Daily.

Greenhouse jobs as Markdown for AI agents

includeDescriptionMarkdown is on by default, so every row carries a Markdown description an LLM can read without an HTML-stripping step, with stable field names. Task: Greenhouse Jobs as Markdown for AI Agents.

Greenhouse MCP server for job data: Claude, Claude Code, and Cursor

Apify exposes the Actor over MCP, so Claude, Claude Code, Cursor, and ChatGPT get a greenhouse-job-board-api tool that answers "which of these twenty companies posted sales roles this week" with live data. The server URL is:

https://mcp.apify.com/?tools=actors,docs,johnvc/greenhouse-job-board-api

In Claude Code that is claude mcp add --transport http apify plus the URL; Claude Code has a free trial at claude.ai.

The example repo

johnisanerd / Apify-Greenhouse-Job-Board-API

greenhouse job board api: Python + MCP quick-start for the Greenhouse API on Apify. Call it from Python (uv) or as an MCP tool in Claude and Cursor. Returns structured JSON for greenhouse job board api.

🌱 Greenhouse Job Board API: live jobs, companies, and new postings

A Python and MCP quick-start for the Greenhouse job board API on Apify. Pull live jobs from any Greenhouse-hosted career site, discover companies hiring through Greenhouse, and track new or changed postings, all without a Harvest key or a login.

Actor: Greenhouse Job Board API on Apify
Input schema: input parameters
Get a free API token: apify.com

Greenhouse is the applicant tracking system behind the career pages of thousands of companies, including Stripe, Airbnb, GitLab, and Anthropic. This Actor reads the public Greenhouse job board API live at run time, so every row reflects what the board says right now, not what an index remembered last week. Give it board tokens or URLs, or give it nothing and let it sweep a bundled directory of 4,400+ verified boards.

Text walkthrough

The Greenhouse job board API takes company board tokens…

View on GitHub

A uv Python quick start with five --example recipes and MCP install guides for five clients.

FAQ about scraping Greenhouse job boards

Is the Greenhouse scraper free, or what does it cost?

Per delivered row: one event per job record, plus add-ons (description formats, application questions, run report) billed only on rows that carry them. No start fee, no minimum, and filtered rows are never charged; with add-ons off a job row costs a fraction of a cent. Prices are on the store's pricing card, and new Apify accounts include free platform credit.

Is job-boards.greenhouse.io legit, and is a scraper allowed to read it?

Yes, job-boards.greenhouse.io and boards.greenhouse.io are both official Greenhouse domains. The scraper reads only the public job board layer behind them, never applicant or Harvest data; whether your use of public job data is fine where you live (California comes up a lot) is a question for your counsel, not me.

Can Claude run this scraper over MCP, and is it Greenhouse's MCP?

Claude, Claude Code, Cursor, and ChatGPT can, through the hosted Apify MCP server URL above. It is not Greenhouse's own MCP product, which is built for Greenhouse customers; this one is public job data for anyone building an agent.

How do I schedule this scraper for a daily new-jobs feed?

Save your input as a task with updatedAfter at 25h, attach an Apify schedule with a cron expression like 0 7 * * *, and each run returns only jobs that changed since the last one. Start from the Greenhouse Job Board API.

Where does the Greenhouse scraper stop?

At the EU data silo: boards on job-boards.eu.greenhouse.io are detected and reported but not yet scraped (EU support is planned). Nothing behind the Harvest API: no applicants, no internal requisitions. A bad token comes back as an in-band error row with errorCode: board_not_found.

More from Truffle Pig Data

Hiring data rarely lives on one ATS, so three siblings: the Ashby Job Board API for Ashby-hosted boards, the Workday Careers API for Workday career sites, and the Google Jobs Scraper for Google Jobs.

Wrapping up

Greenhouse gives you one board at a time; the Actor gives you all of them, salaries parsed, yesterday's changes filtered. Try the Greenhouse Job Board API, or clone the example repo and run --example companies on your niche.

Trustpilot Reputation Monitoring in 2026: Turn Public Reviews into a Dated JSON History

Truffle Pig Data — Thu, 10 Sep 2026 19:09:55 +0000

Getting Trustpilot company review pages into something you can query is a grind: reviews are paginated, replies are nested under each one, and the posting date is not the date the thing happened. I'll show the manual way and where it breaks, then a faster path in Python, REST, and MCP, plus a repo to clone. The faster path is the Trustpilot Reviews API on Apify: company URLs in, one JSON row per review out, built for reputation monitoring, a dated history of what changed.

Disclosure: the Apify links in this post are affiliate links. If you run the Actor, I may earn a referral commission at no extra cost to you.

Does Trustpilot have an API?

Yes, and it's worth being straight about. Trustpilot's own developer API sits behind a business account and its plan limits, scoped to that account's data. If you're a business reading your own reviews, use it; it won't let you point at a competitor with no account relationship. That's the gap a scraper-as-API fills: public review pages in, a fixed JSON shape out, billed per review rather than per plan. This Actor isn't affiliated with Trustpilot and doesn't claim feature parity with their API.

What the Trustpilot Reviews API returns

The Trustpilot Reviews API returns one JSON row per public review: star rating, title and text, posting date and a separate date of experience, verified flag, reviewer country, and any company reply, plus the company's overall rating and star breakdown.

Field	Example	Notes
`reviewRating`	`5`	1 to 5 stars
`dateOfExperience`	`2026-07-23T00:00:00.000Z`	When it happened; `reviewDate` is the posting date
`reviewReplies`	`[{ "text": "Thanks for the kind words.", "date": "2026-07-15" }]`	The business's reply, with date
`starBreakdown`	`{ "star5": { "count": 4083, "percent": 93 } }`	The 5-to-1 split, counts and percentages
`result_type`	`review`	Or `error` with an `error_message`, so nothing fails silently

Who this is for

Brand and CX teams who want the star breakdown as a time series. Competitive analysts who need a rival set in one export, replies included. Data scientists building a customer review dataset with labels attached.

The manual way, and where it breaks

The DIY version is a headless browser that loads the review page, pages through, and parses each card. It works in a quick test. Then you hit a company with four thousand reviews, find that reply text and date of experience live in different parts of the markup, and watch your selectors rot when the layout shifts. At twenty competitors you're running proxies and retries and maintaining infrastructure instead of a rating history.

The failure I care about most is the quiet one. When a script returns zero reviews, you can't tell whether the page moved, the company has no reviews, or you were blocked. An empty result is a claim, and claims need evidence.

The faster way: run the Trustpilot Reviews API

You send a documented JSON input and get a documented JSON output, with nothing to keep alive.

Apify Console

Open the Trustpilot Reviews API and click Try for free.
Paste company review-page URLs (or bare domains) into Company URLs and set Max reviews per company.
Run it and export as JSON, CSV, or Excel.

REST

curl -X POST "https://api.apify.com/v2/acts/johnvc~trustpilot-reviews-api/runs?token=YOUR_APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{ "companyUrls": ["https://www.trustpilot.com/review/www.dugood.org"], "maxReviewsPerCompany": 3 }'

Run endpoint reference: the Apify API docs.

Get customer review data in Python

Call the Actor with apify-client and handle the error rows:

from apify_client import ApifyClient

client = ApifyClient("YOUR_APIFY_TOKEN")

run = client.actor("johnvc/trustpilot-reviews-api").call(
    run_input={
        "companyUrls": ["https://www.trustpilot.com/review/www.dugood.org"],
        "maxReviewsPerCompany": 25,
    }
)

for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    if item["result_type"] == "error":
        print("error:", item["companyUrl"], item["error_message"])
        continue
    print(item["reviewRating"], item["dateOfExperience"], item["reviewTitle"])

That error branch answers the plausible zero: a URL that returns nothing produces an error row with a plain-language error_message, so a dead page, a company with no reviews, or a datePosted window that excluded everything comes back with a reason. (I'd rather see an error row than a suspiciously clean zero.) The runnable version is the task Get Trustpilot reviews by API from company URLs.

Reputation monitoring on a schedule, for any company

A single run is a check. The value is the trendline: the same input on a schedule, each run appending a dated snapshot with fetched_at, companyOverallRating, and starBreakdown. Because the breakdown carries counts per level, you see a one-star share climbing before the rounded average moves. Save the input as a task, schedule it with 0 7 * * *, and set datePosted to Last 30 days so recurring runs stay small. The task Monitor Trustpilot reputation for any company is already wired this way.

Audit Trustpilot reviews for fake review signals

Every row carries isVerifiedReview and reviewerTotalReviews, so you can weight an established reviewer against a one-time account. The task Audit Trustpilot reviews for fake review signals sets that up. It gives you the signals; the judgement is still yours.

Build a customer review dataset for sentiment analysis

Flat rows of reviewContent with reviewRating attached are already a labelled dataset, and reviewerLocation splits it by market. The task Build a customer review dataset for sentiment analysis exports exactly that shape.

Export Trustpilot reviews to CSV

For spreadsheet people, the task Export Trustpilot reviews to CSV for any company goes straight from company URLs to CSV, no code. Two recipes also exist as Chinese-language tasks: bulk collect Trustpilot reviews and export to CSV and monitor competitor Trustpilot ratings.

Use it from Claude and other MCP clients

Apify exposes the Actor through the Model Context Protocol, so Claude, Claude Code, and Cursor can pull reviews mid-conversation and answer "which competitor replies to complaints" with live data. In Claude Code:

claude mcp add --transport http trustpilot "https://mcp.apify.com/?tools=actors,docs,johnvc/trustpilot-reviews-api"

The task Read Trustpilot reviews in Claude via MCP has a working configuration; read more about Claude Code at claude.ai.

The example repo

johnisanerd / Apify-Trustpilot-Reviews-API

Reputation monitoring data from the Trustpilot Reviews API on Apify: a Python (uv) quick-start plus MCP install guides for Claude, Cursor, and ChatGPT. Returns structured JSON reviews, ratings, replies, and star breakdowns.

⭐ Trustpilot Reviews API: Reputation Monitoring Data as Structured JSON

The most efficient, reliable, and developer-friendly way to use the Trustpilot Reviews API.

Actor page: apify.com/johnvc/trustpilot-reviews-api Input schema: apify.com/johnvc/trustpilot-reviews-api/input-schema

Give it one or more company review-page URLs and it returns one clean JSON row per review: the star rating, the review title and full text, the posting date, the separate date of experience, the verified flag, the reviewer's country, and any reply the business posted. Every row also carries company context, so the same pull gives you the overall rating, the total review count, and the full 5-to-1 star breakdown with counts and percentages. It is built API-first and MCP-ready, so you can call it from Python or drive it as a tool from an AI agent.

Video Walkthrough

Text walkthrough

Most people arrive here because dashboards priced for enterprise reputation monitoring do not expose the underlying rows, and the…

View on GitHub

A Python quick start that asks for just three reviews, plus MCP walkthroughs for Claude, Cursor, and ChatGPT.

FAQ about scraping Trustpilot reviews

What does the Trustpilot scraper cost?

Per review returned, through one review-scraped event, plus a tiny actor-start event per run. A company that returns no reviews costs nothing, and maxReviewsPerCompany is enforced at the source, so your cap is the ceiling on a run's cost. Current rates are on the Store card.

How do I monitor a competitor's online reputation with this scraper?

Put every competitor URL in one companyUrls array, save it as a task, and attach a weekly schedule such as 0 9 * * 1. Each row carries companyName, companyOverallRating, and starBreakdown, and reviewReplies shows who answers complaints. Start from the Trustpilot Reviews API with datePosted set to Last 30 days.

Can Claude Code or Cursor call this scraper directly?

Yes. Add the hosted Apify MCP server with the Actor preloaded, as above, and it appears as a callable tool. Every row carries a one-line summary an agent can read as is.

How does media monitoring support reputation management alongside a review scraper?

Media monitoring tells you what's being said about you; review data tells you what customers experienced and whether you replied. Run both on the same cadence and a rating dip lines up against that week's coverage; this scraper is the review half.

Is this scraper enough for crisis prevention?

Early warning needs the full star distribution rather than the average, the date of experience separate from the posting date, and a history you keep. This scraper returns the first two and leaves storage and alerting to you. Its hard edges: no company search, so you build the URL list yourself; datePosted accepts four fixed windows, not calendar dates; and no reviewer email addresses.

More from Truffle Pig Data

Same Actor, another angle: the Trustpilot Reviews API write-up on LinkedIn. For B2B software, the G2 Reviews API returns the same kind of row.

Wrapping up

Trustpilot's API serves a business reading its own reviews; for any other company, this is the public data as dated JSON you keep. Try the Trustpilot Reviews API, or clone the example repo and point it at your own company list.

Google Flights Deals API: Telling a Real Bargain From a Cheap Short-Haul

Truffle Pig Data — Wed, 09 Sep 2026 19:09:12 +0000

Here is a thing I got wrong for a long time: a cheap flight and a good deal are not the same thing. Pull the cheapest destinations from any airport and the top of the list is mostly short-haul, cheap because it is close, not because the fare is unusual. The ones worth chasing are priced below their own norm. The Google Flights Deals API on Apify answers that by attaching a typical-price baseline to every result, sourced from Google Flights.

Disclosure: the Apify links in this post are affiliate links. If you run the Actor, I may earn a referral commission at no extra cost to you.

Is there an API for Google Flights deals?

No. Google retired QPX Express in 2018 and has never shipped a public flights API, deals tab included. What exists is a scraper you consume like an API: send an airport code, get structured JSON back. The framing matters because the deals feed is not a search: you give it a departure point and it tells you where is cheap right now, a different query from the point-to-point search most flight APIs answer.

What the Google Flights Deals API returns

The Google Flights Deals API returns the thirty cheapest destinations from an airport as structured JSON, each carrying its price, what that route typically costs, and a flag for whether it beats the baseline.

Field	Example	Notes
`price`	`312`	Fare in the requested currency
`average_price`	`498`	What this route typically costs
`savings_percent`	`37.3`	The gap, as a percentage
`is_below_typical`	`true`	The one field that separates bargains from short-hauls
`outbound_date`	`2026-09-12`	Exact dates for the specific itinerary
`flight_link`	`https://www.google.com/travel/flights/...`	Booking link

You also get name, country, a description and thumbnail, plus route, airline, stops, and flight_duration in minutes.

Stated plainly: this returns a snapshot with a baseline. It does not track prices over time, keep history, or send alerts. It is the data layer you build those on, not the alerting product.

Who this is for

Travel affiliates who need better than a list sorted by price. Fare newsletters picking five destinations a week and justifying each. Developers building a price-alert product on a source with a reference price attached. AI travel agents answering "where can I go cheaply from here".

The manual way, and where it breaks

Google Flights encodes its whole query into an opaque URL blob you reverse-engineer per filter, and results render through JavaScript, so a plain HTTP request comes back nearly empty. Worse, the typical-price figure is not printed next to each card as a number. Getting a baseline out of the DOM means parsing a comparative phrase rendered differently across locales, the kind of thing that silently returns None three weeks after it shipped.

The faster way: run the Google Flights Deals API

One airport code in, thirty scored destinations out.

Apify Console

Open the Google Flights Deals API and click Try for free.
Put a 3-letter IATA code in departureId. No arrival field, by design.
Run it and download the dataset as JSON, CSV, or Excel.

REST

curl -X POST "https://api.apify.com/v2/acts/johnvc~google-flights-deals-api/runs?token=YOUR_APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{ "departureId": "JFK", "maxDealsPerAirport": 30, "currency": "USD" }'

Run endpoint reference: the Apify API docs.

Find flight bargains in Python

Filter on the flag rather than sorting by price:

from apify_client import ApifyClient

client = ApifyClient("YOUR_APIFY_TOKEN")

run = client.actor("johnvc/google-flights-deals-api").call(
    run_input={"departureId": "JFK", "maxDealsPerAirport": 30, "currency": "USD"}
)

items = list(client.dataset(run.default_dataset_id).iterate_items())
deals = [i for i in items if i.get("result_type") == "deal"]
bargains = [d for d in deals if d.get("is_below_typical")]

for deal in sorted(bargains, key=lambda d: -(d.get("savings_percent") or 0)):
    print(deal["name"], deal["price"], "vs", deal["average_price"],
          f"({deal['savings_percent']}% below)", deal["outbound_date"])

Measured across three hubs during the build, between 6 and 30 percent of the feed clears its own baseline on a given day. That is the honest yield, and why the flag earns its place: sorting by price and taking the top five hands you the wrong five. The runnable version is Find Genuine Flight Bargains From Your Airport.

Build your own fare history

The Actor keeps no history, which sounds like a limitation and is really a design line: it hands you a fare and a reference price per row, and you decide what to keep. Save a task, attach a schedule, store rows keyed on departure, arrival, and date. After a few weeks you own a fare record nobody can revoke, every row carrying what that route normally cost that day. Start from Fare Data With a Typical-Price Baseline, which takes several airports at once.

Narrow it to the trip you would actually take

The filters map to how people search: maxStops: nonstop for direct routes, in Nonstop Flight Bargains From Any Airport; travelDuration: weekend for short trips, in Find Cheap Weekend Trips From Your Airport; travelClass: business for the premium cabin, in Cheap Business Class Deals From an Airport. Non-English locales work the same way: 查询从上海出发的低价机票目的地.

One filter behaves differently, and the Actor logs a warning when you set it: maxDurationMinutes makes the source stop returning the baseline, switching off savings and is_below_typical. I found that the hard way by changing several inputs at once and blaming the wrong one. Isolate variables.

Use it from Claude and other MCP clients

Apify exposes the Actor over the Model Context Protocol, so Claude, Claude Code, and Cursor can run a live lookup mid-conversation and answer "where can I fly cheaply from Chicago next month, and which are below normal" with current fares. More on Claude Code at claude.ai.

The example repo

johnisanerd / Apify-Google-Flights-Deals-API

✈️ Google Flights Deals API: cheap destinations as clean JSON

Give it an airport and get back the thirty cheapest destinations you can fly to from there, as structured JSON. Each one comes with the price, the typical price for that route, exact dates, airline, stops, duration, a photo, and a booking link.

The part nobody else gives you: which of them are actually bargains. A cheap fare and a good deal are not the same thing.

Actor: Google Flights Deals API on Apify Store

Video Walkthrough

https://www.youtube.com/watch?v=jREWahDGhJM

Quick Start

git clone https://github.com/johnisanerd/Apify-Google-Flights-Deals-API.git
cd Apify-Google-Flights-Deals-API
cp .env.example .env          # paste your Apify token
uv sync
uv run google-flights-deals-api-example.py

Get a free Apify API key at apify.com.

Why Use This Google Flights Deals API?

Bargain detection, not just cheap fares. Every row carries average_price, what that route typically costs, plus savings, savings_percent, and an is_below_typical flag.
One…

View on GitHub

Python quick start, filter variants, and MCP setup for Claude, Cursor, and ChatGPT.

FAQ about scraping Google Flights deals

Is this flight deals scraper free, and what does it cost?

Billing is per deal returned, plus a small run start fee, so a thirty-destination sweep is thirty billed rows with nothing hidden behind it. maxDealsPerAirport caps the count before a run starts. New Apify accounts come with free credit, so early runs usually cost nothing.

How do flight price alerts work, and can this scraper send them?

It cannot, and any tool claiming otherwise on top of this feed is doing the alerting itself. An alert is two pieces: a repeated observation and a rule about when to notify. This scraper is the observation half, and a good one, because each row arrives with a reference price, so your rule can be "below typical" rather than a threshold you guessed.

Can this scraper find deals without me picking dates?

Yes, and it is the default. Leave outboundDate blank and the feed picks cheap dates, returning the exact outbound_date and return_date it found. That is the opposite of a normal flight search, and why this answers "when should I go" as well as "where".

What does the scraper mean by "below typical price"?

It compares the fare to what that route usually costs, not to the other destinations in your results, so a $312 flight can be below typical while a $190 one is not. Negative savings mean a route is priced above its own norm, and those rows are useful too: they say what to skip.

Can this scraper run from Claude or another MCP client?

Yes. Connect the Apify MCP server and it becomes a callable tool in Claude, Claude Code, or Cursor, so an agent pulls live deals from a prompt.

Can an AI agent pay for this scraper in USDC with x402?

Yes. The Google Flights Deals API supports agentic payments via the x402 protocol, so AI agents and MCP clients can pay for runs in USDC on Base with no Apify account or API token. Point your agent at the Apify MCP server and it can discover, pay for, and run the scraper autonomously; the Apify x402 announcement has the details.

More from Truffle Pig Data

Same kind of JSON: Google Travel Explore API for ideas with hotel prices, Google Flights Data Scraper for point-to-point search, Google Hotels Search Scraper for the nights, and Google Maps Photos API for where you land.

Wrapping up

Cheapest and best-value are different sorts, and only one belongs in front of a reader. Try the Google Flights Deals API, or clone the example repo and point it at your own home airport.

Workday API Without a Tenant: Pull Every Job Posting as JSON in 2026

Truffle Pig Data — Tue, 08 Sep 2026 19:09:35 +0000

Every large employer on the Workday ATS posts its openings on a public careers site, like NVIDIA's Workday careers page, and none of them hand you that list as JSON. I'll show the manual route and where it breaks, then the shortcut: the Workday Careers API, which turns any careers URL into one row per job with no Workday tenant, recruiting license, or login.

Disclosure: the Apify links in this post are affiliate links. If you run the Actor, I may earn a referral commission at no extra cost to you.

Does Workday have an API for job postings?

Yes, and that is what makes the search confusing. Workday's official REST and SOAP APIs serve enterprise HR integrations: they need tenant credentials and never expose public job listings. In practice a Workday API for job postings means a scraper you call like an API: send a careers URL, get every posting back as JSON.

What the Workday API returns

The Workday Careers API returns every live posting on a Workday careers site as structured JSON: title, locations, ISO posted date, full description, pay range, employment type, remote status, requisition ID, and apply URL.

Field	Example	Notes
`title`	`Senior Software Engineer`	`company` and `jobReqId` (`JR1990000`) ride along
`postedDate`	`2026-07-30`	Exact ISO date; `postedOn` keeps the relative label
`locationsText`	`US, CA, Santa Clara`	`additionalLocations` lists the rest
`remoteType`	`Flex`	Remote, Flex, or Onsite when the site exposes it
`salaryMin` / `salaryMax`	`148000` / `230000`	Plus `salaryCurrency`; null when no range is published

Who this is for

Job boards refreshing Workday job listings with a stable jobReqId to dedupe on; recruiting and sales intelligence teams tracking a competitor's open roles and hiring velocity; labor-market researchers comparing pay-range disclosure and remote share; and anyone feeding live hiring signals to an AI agent.

The manual way, and where it breaks

The DIY route is a headless browser that pages through the list and opens every posting for its description. The list carries only relative dates, so a "new this week" filter means fetching every job's detail record. Pay ranges are prose you parse yourself. One employer runs several career sites across two URL families, and a big tenant lists thousands of postings, so you add pacing and retries or get rate limited. You end up maintaining a crawler for what is really a data feed.

The faster way: run the Workday Careers API

Apify Console

Open the Workday Careers API and click Try for free.
Paste careers URLs into startUrls; optionally add searchText, maxJobsPerSite, or postedAfter.
Run it and download the dataset as JSON, CSV, or Excel.

REST

curl -X POST "https://api.apify.com/v2/acts/johnvc~workday-careers-api/runs?token=YOUR_APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{ "startUrls": [{ "url": "https://nvidia.wd5.myworkdayjobs.com/NVIDIAExternalCareerSite" }], "searchText": "engineer", "maxJobsPerSite": 100, "includeDetails": true }'

Run endpoint reference: the Apify API docs.

Pull Workday jobs in Python

Call the Actor with apify-client; an error row means a bad URL, so keep the job rows:

from apify_client import ApifyClient

client = ApifyClient("YOUR_APIFY_TOKEN")

run = client.actor("johnvc/workday-careers-api").call(
    run_input={
        "startUrls": [{"url": "https://nvidia.wd5.myworkdayjobs.com/NVIDIAExternalCareerSite"}],
        "searchText": "engineer",
        "maxJobsPerSite": 50,
    }
)

for job in client.dataset(run["defaultDatasetId"]).iterate_items():
    if job.get("resultType") != "job":
        continue
    print(job["title"], job["locationsText"], job["postedDate"])
    print("   ", job.get("salaryMin"), job.get("salaryMax"), job.get("salaryCurrency"), job["applyUrl"])

Export Workday job listings to JSON, CSV, or Excel

Leave searchText empty and maxJobsPerSite at 0 to get the whole site, one row per posting; feed several sites into one run and join on tenant and siteId. The task Export Workday Job Listings to JSON, CSV or Excel is that configuration.

Monitor Workday job postings on a schedule

With postedAfter set to a recent ISO date, the run stops paging once a whole page is older than your cutoff, since Workday sorts newest first; a 2,000 job site with a 3 day cutoff reads about 280 listings, and filtered rows are never billed. Schedule it, diff on jobReqId, and new postings show up the day they appear. Start from Monitor New Workday Job Postings for Any Company.

Find remote Workday jobs

Every row carries remoteType (Remote, Flex, or Onsite, whenever the site exposes the label), and the SITE_SUMMARY key-value record counts each type for the whole site. Filter after the run, or clone Find Remote Jobs on Any Workday Careers Site.

Pull Workday jobs with salary ranges

When an employer publishes a pay range, the Actor extracts salaryMin, salaryMax, and salaryCurrency and keeps the snippet in salaryText; with no range stated, the fields are null rather than guessed. See Extract Jobs with Salary Ranges from Workday Sites.

One company, every opening: the NVIDIA example

Point the Actor at NVIDIA's careers URL with no filters and you get every live opening with locations and pay ranges, plus totalJobsOnSite to confirm nothing was dropped. Export All NVIDIA Jobs with Salaries and Locations is the one-click version; the example repo ships it as --example nvidia_jobs.

Workday MCP server: use it from Claude, Claude Code, and Cursor

Apify exposes the Actor through the Model Context Protocol, so Claude, Claude Code, and Cursor get a workday-careers-api tool that takes a careers URL and returns structured jobs, so an agent can answer "which of these companies posted engineering roles this week" or track headcount across a watch list. The server URL is:

https://mcp.apify.com/?tools=actors,docs,johnvc/workday-careers-api

In Claude Code that is claude mcp add --transport http apify plus the URL, and you can read more about Claude Code at claude.ai.

The example repo

johnisanerd / Apify-Workday-Careers-API

workday api: Python + MCP quick-start for the Workday Careers API on Apify. Call it from Python (uv) or as an MCP tool in Claude and Cursor. Returns structured JSON for workday api.

Workday Jobs API: Scrape Any Workday Careers Site from Python or MCP

This repo shows two ways to use the Workday Careers API on Apify: a Python quick start managed with uv, and MCP install guides for five AI clients (Claude Cowork Desktop, Claude Code, Claude on the web, Cursor, and ChatGPT).

Give the API any Workday careers URL, like https://nvidia.wd5.myworkdayjobs.com/NVIDIAExternalCareerSite, and it returns every live job posting as structured JSON: titles, locations, exact ISO posted dates, full descriptions, extracted pay ranges, employment type, remote status, requisition IDs, and direct apply URLs. Thousands of Fortune 500 employers run hiring on the Workday ATS, and this is the practical Workday API for their public job data: no login, no proxies, pay per result.

Video walkthrough

Text walkthrough

Searching for a Workday API for job postings usually leads to Workday's enterprise SOAP and REST APIs, which need tenant credentials and…

View on GitHub

A uv-managed Python quick start with nvidia_jobs, remote_jobs, and salary_ranges recipes, plus MCP install guides for five clients.

FAQ about scraping Workday job postings

How much does the Workday scraper cost, and is there a free tier?

About $0.10 per 1,000 job postings returned, plus a small actor-start event per run. Error rows and rows removed by postedAfter are never charged, and new Apify accounts include free platform credit. No subscription, no seat license: you pay for rows that reach your dataset.

Is Workday an ATS, and which myworkdayjobs.com sites can the scraper read?

Yes. Workday Recruiting is one of the most widely used enterprise applicant tracking systems, and every customer publishes jobs on a public careers site. myworkdayjobs.com is the official domain Workday hosts those sites on, so pages there are legitimate, and any site there or on myworkdaysite.com is readable with no login.

What is the best MCP server for Workday, and can Claude run this scraper?

For job data, the hosted Apify MCP server with this Actor attached, using the URL above. Claude, Claude Code, Cursor, and ChatGPT then call the scraper as a tool.

Can I schedule the scraper to catch new Workday job postings?

Yes. Save the run as a task and attach an Apify schedule with a cron expression like 0 7 * * *, using postedAfter for exact dates or list-only mode (includeDetails: false) plus a jobReqId diff when you only need what is new. Start from the Workday Careers API.

What will the scraper not give you?

Anything behind a Workday login: employee records, internal requisitions, applicant data, the HR APIs. On the public side, salary fields are null when no range is published, remoteType is filled only when the site shows the label, and postedAfter needs includeDetails on.

More from Truffle Pig Data

Same Actor, other angles: the Medium how-to, the LinkedIn write-up, and the Peerlist article.

Have company names instead of URLs? The companion Workday Career Sites API turns them into the tenant and site values this Actor consumes; the Oracle Fusion Recruiting and Taleo Jobs API does the same for employers on Oracle.

Wrapping up

Workday's own APIs will not give you a careers site, but you can still get every posting as clean JSON without a tenant. Try the Workday Careers API, or clone the example repo and point it at the employers you track.

LinkedIn Learning API: How to Get Course Data, Reviews, and Syllabus in 2026

Truffle Pig Data — Thu, 03 Sep 2026 14:11:58 +0000

I keep hitting the same wall on learning-data projects: the catalog I want sits on LinkedIn Learning, and no open endpoint returns it. The official LinkedIn Learning API needs the Partner Program or a purchased site license with admin-provisioned OAuth. So I built a hosted LinkedIn Learning API that reads the public, logged-out course pages and returns each course as JSON: ratings, reviews, instructors, and syllabus.

Disclosure: the Apify links in this post are affiliate links. If you run the Actor, I may earn a referral commission at no extra cost to you.

Does LinkedIn Learning have a public API?

Sort of, and the distinction matters. Both official APIs, the Learning API and the Learning Reporting API, sit behind the Partner Program or a paid site license and return learner activity for seats you own, not the public catalog most people want.

Here, a LinkedIn Learning API means a service you call with a keyword or a course URL and get that public catalog metadata back as JSON, no login. It reads only public pages, not private learner activity; for that, use LinkedIn's Reporting API.

	Official LinkedIn Learning API	This route (public guest pages)
Access	Partner Program or site license, admin OAuth	None; reads public logged-out pages
Fields	Learner activity, completions, admin reporting	Catalog metadata: `ratingValue`, `reviews`, `tableOfContents`, `instructors`, `skills`
Price	Enterprise contract	Pay per row: $0.10 per 1,000 search rows, $0.50 per 1,000 full records

What the LinkedIn Learning API returns

In one sentence: it reads LinkedIn Learning's public logged-out course pages as JSON, from ratings and written reviews to the full lesson-by-lesson syllabus, with no login. Search mode returns one row per course; details mode returns the full record for a course, lesson, or path URL. The fields worth pulling:

Field	Example	Notes
`ratingValue`	`4.7`	Average learner rating out of 5, with `ratingCount` alongside.
`reviews`	`[{rating, body, authorName, authorJobTitle, authorProfileUrl, datePublished}]`	Written review text plus reviewer identity, not just a star average.
`tableOfContents`	`[{section, items:[{title, description, durationSeconds, isFree}]}]`	Full syllabus: every lesson has a description and an `isFree` flag, plus `freeLessonCount`.
`instructors`	`[{name, jobTitle, profileUrl}]`	Structured people with profile links. Full records add `hasCertificate` and `skills`.

Who this is for

Learning and development teams benchmarking an internal library against ratings and learner counts.
Course-recommender and AI-agent builders who need structured records, syllabus included.
Analysts reading the written reviews across a topic, or anyone wanting an online courses dataset without an enterprise contract.

The manual way, and where it breaks

It works, up to a point. A guest page like linkedin.com/learning/search?keywords=python loads courses without an account, and each course page embeds a schema.org Course block as JSON-LD:

import json, re, requests

url = "https://www.linkedin.com/learning/python-essential-training-18764650"
html = requests.get(url, headers={"User-Agent": "Mozilla/5.0"}).text

for block in re.findall(r'<script type="application/ld\+json">(.*?)</script>', html, re.S):
    data = json.loads(block)
    if data.get("@type") == "Course":
        print(data["name"], data.get("aggregateRating", {}).get("ratingValue"))

That gets a name and a star average, not the per-lesson isFree flags, reviewer names, or the courses in a learning path. Those sit elsewhere and shift often, so selectors rot. Logged out, LinkedIn serves at most 50 results per query, so a real catalog means merging many queries.

The faster way: run the LinkedIn Learning API

Three ways to call it.

Apify Console

Open the LinkedIn Learning API and click Try for free.
Leave Mode on Search, type a keyword such as python, and narrow by level, length, or software.
Click Start and export the dataset as JSON, CSV, or Excel.

REST API

One call runs the Actor and returns the dataset in the same response:

curl -X POST "https://api.apify.com/v2/acts/johnvc~linkedin-learning-api/run-sync-get-dataset-items?token=YOUR_APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{ "mode": "search", "queries": ["project management"], "maxItems": 25 }'

Full run-endpoint documentation is in the Apify API docs.

MCP

Point any MCP client at the hosted Apify server and the Actor becomes a tool:

https://mcp.apify.com/?tools=actors,docs,johnvc/linkedin-learning-api

In Claude, Claude Code, or Cursor, ask it to search the catalog mid-conversation, a LinkedIn Learning MCP server with no glue code.

Get LinkedIn Learning course data in Python

The apify-client package mirrors the REST API. With enrichDetails on, the run mixes row types, so filter to course_detail before reading ratingValue:

from apify_client import ApifyClient

client = ApifyClient("YOUR_APIFY_TOKEN")

run = client.actor("johnvc/linkedin-learning-api").call(run_input={
    "mode": "search",
    "queries": ["python"],
    "maxItems": 5,
    "enrichDetails": True,
    "maxConcurrency": 5,
})

for row in client.dataset(run.default_dataset_id).iterate_items():
    if row.get("resultType") != "course_detail":
        continue
    print(row["title"], row.get("ratingValue"), row.get("ratingCount"), row["courseUrl"])

Get the LinkedIn Learning course catalog as JSON

Search mode with one query is the fastest way to a clean catalog. The LinkedIn Learning course data as JSON task returns one row per course with title, courseUrl, instructors, and difficultyLevel:

{ "mode": "search", "queries": ["project management"], "maxItems": 25 }

Pull a LinkedIn Learning course list with no login

Same pattern, a different topic, no account or cookie. The LinkedIn Learning course data without a login task reads the public catalog for a term like cybersecurity:

{ "mode": "search", "queries": ["cybersecurity"], "maxItems": 25 }

Run a LinkedIn Learning course catalog download

A single query caps at 50. expandWithFilters re-runs it across filter combinations and merges unique courses, the only way past that ceiling; enrichDetails adds rating and enrollment. The LinkedIn Learning course ratings dataset task shows the enriched shape:

{ "mode": "search", "queries": ["python"], "maxItems": 5, "enrichDetails": true, "maxConcurrency": 5 }

Enriched rows bill at the detail rate, never both.

Extract LinkedIn Learning reviews and ratings

The written review body with reviewer identity is the field no competing course scraper exposes. In details mode each row carries ratingValue, ratingCount, and a reviews array. The extract LinkedIn Learning course reviews task reads three courses at once:

{ "mode": "details", "courseUrls": ["https://www.linkedin.com/learning/python-essential-training-18764650", "https://www.linkedin.com/learning/agile-foundations", "https://www.linkedin.com/learning/sql-essential-training-3"] }

Find free preview lessons in a course

Every lesson in tableOfContents carries an isFree flag and freeLessonCount totals them, so you can rank courses by how much is watchable free and surface LinkedIn Learning free courses. The free preview lessons task returns the full syllabus with that flag on every lesson:

{ "mode": "details", "courseUrls": ["https://www.linkedin.com/learning/python-essential-training-18764650", "https://www.linkedin.com/learning/excel-essential-training-microsoft-365", "https://www.linkedin.com/learning/communication-foundations-2018"] }

Scrape LinkedIn Learning Excel course data

The softwareNames filter keeps only courses that teach a specific tool, using LinkedIn's own label. LinkedIn applies only the first label per run, so run each tool separately. The Excel course data task keeps courses that teach Microsoft Excel:

{ "mode": "search", "queries": ["excel"], "softwareNames": ["Microsoft Excel"], "maxItems": 25 }

Scrape LinkedIn Learning Figma course data

The same softwareNames pattern works for a design tool. The Figma course data task keeps only courses that teach Figma:

{ "mode": "search", "queries": ["figma"], "softwareNames": ["Figma"], "maxItems": 25 }

Map skills taught across the LinkedIn Learning catalog

An enriched search returns a skills array on every course, each linked to its topic page. The skills coverage task shows which skills a topic teaches:

{ "mode": "search", "queries": ["data analysis"], "maxItems": 5, "enrichDetails": true, "maxConcurrency": 5 }

Get LinkedIn Learning instructors and profile links

Enriched rows also return instructors with name, job title, and profile link. The instructor data task adds the course rating, level, and duration:

{ "mode": "search", "queries": ["leadership"], "maxItems": 5, "enrichDetails": true, "maxConcurrency": 5 }

Track new course releases and learning paths

Set sortBy to RECENCY and each run returns the newest courses first, so a scheduled run watches a subject for releases. The track new courses task uses:

{ "mode": "search", "queries": ["artificial intelligence"], "sortBy": "RECENCY", "maxItems": 25 }

For learning paths, set entityType to LEARNING_PATH or pass a /learning/paths/ URL; each path row carries pathUrl and its ordered member courses, with courseCount.

The example repo

johnisanerd / Apify-LinkedIn-Learning-API

LinkedIn Learning API: Python + MCP quick-start on Apify. Call it from Python (uv) or as an MCP tool in Claude and Cursor. Returns structured JSON for course data, learner reviews, syllabus and instructors, no login needed.

🎓 LinkedIn Learning API: Course Data, Reviews and Syllabus from Python and MCP

A Python and MCP quick-start for the LinkedIn Learning API on Apify. Search the public LinkedIn Learning course catalog by keyword, skill level, length or software, then pull the full record for any course: rating and rating count, written learner reviews, the complete syllabus with a description and a free-preview flag on every lesson, instructors with profile links, skills taught and certificate details. None of it needs a login, a cookie or a site license.

Actor: LinkedIn Learning API on Apify
Input schema: input parameters
Get a free API token: apify.com

LinkedIn's own Learning API is available only through its Partner Program or a purchased site license, with OAuth keys an admin has to provision. This Actor reads the pages LinkedIn Learning already publishes to logged-out visitors and returns the same catalog metadata as JSON: course title…

View on GitHub

The repo has a runnable Python quick start, a .env template, and the MCP server URL.

FAQ about scraping LinkedIn Learning

Does the LinkedIn Learning scraper show which courses give a certificate?

Yes, through hasCertificate and certificateName on every full record. It is a certificate of completion, not an accredited qualification.

Are LinkedIn Learning courses free, and what does the scraper cost?

Most need a subscription, but freeLessonCount and the per-lesson isFree flag show what is watchable free. Billing is pay per row: $0.10 per 1,000 courses found and $0.50 per 1,000 full records; the detail charge replaces the search charge.

How do you get a list of LinkedIn Learning courses with this scraper?

Run search mode with keywords or topic pages, and turn on expandWithFilters to pass the 50-per-query ceiling: it re-runs the query across filter combinations and merges unique courses.

Can the scraper tell me whether every course in a topic gives a certificate?

Run an enriched search and read hasCertificate across the rows. Not every course carries one, so the flag is per course, not per topic.

What does the scraper return for a LinkedIn Learning learning path?

Pass a /learning/paths/ URL in details mode, or set entityType to LEARNING_PATH in search. The row carries pathUrl and the ordered member courses in courses, with courseCount.

How does the scraper tell a learning path apart from a course?

By the entityType value: COURSE is one multi-lesson unit, VIDEO is a single lesson, and LEARNING PATH is a curated sequence of courses.

How do I find the free preview lessons with the scraper?

Every lesson in tableOfContents carries an isFree flag and freeLessonCount totals them, so a result set sorts by how much is watchable without paying.

Can the scraper download the LinkedIn Learning course catalogue?

There is no official export. Run queries or topic pages with expandWithFilters, then export as CSV, JSON, or Excel. LinkedIn caps a query at 50, so a broad catalogue comes from many queries; approximateTotalResults is a size signal, not an exact count.

Wrapping up

The official API is gated, but the public catalog is not: pull ratings, reviews, and the syllabus as JSON with no login. Start from the LinkedIn Learning API and run your first search free, or clone the example repo.

Disclosure: the Apify links in this post are affiliate links. If you run the Actor, I may earn a referral commission at no extra cost to you.

Last Updated: 2026.09.02

How to find investors for a startup in 2026 (with the cheque sizes attached)

Truffle Pig Data — Mon, 24 Aug 2026 15:25:47 +0000

Every founder I know hits the same wall on a raise. You can find investor names in an afternoon. What you cannot find, without a lot of tab-switching, is which of them writes a cheque the size of your round.

Signal by NFX publishes that. It maintains public investor lists organised by sector and stage, and each investor carries a cheque-size range. The lists are public, no login, but they paginate eight at a time through a private endpoint, so copying one into a spreadsheet by hand is a genuinely bad afternoon.

I did it by hand once. Then I stopped doing it by hand.

Disclosure: the Apify links in this post are affiliate links. If you run the Actor, I may earn a referral commission at no extra cost to you.

What the NFX Signal investor data actually contains

The NFX Signal Investor API returns venture investors, VC firms, and the investor-list catalog as structured JSON, including the minimum, target, and maximum cheque each investor writes.

What you get	Detail
Investor	name, position, profile link, headshot
Cheque size	`minInvestment`, `targetInvestment`, `maxInvestment` in USD
Firm	`firmName`, `firmUrl`
Context	investment locations, other lists they appear on, source list stage and sector
Catalog	349 lists across 103 sectors, four stages, and 14 geographies

The largest single list, fintech-seed, holds 8,775 investors. san-francisco-bay-area holds 8,246.

Who this is for

Founders raising, who need a target list ordered by the cheque they actually need.
Anyone selling to VCs, who needs a firm directory for a CRM rather than individual partners.
Analysts mapping a market, who want to see which firms show up across AI, FinTech, and climate lists.

The DIY path, and where it breaks

You can do this by hand. Open a list page, copy the eight visible investors, click load more, repeat. For a 300-investor list that is roughly 38 rounds of clicking.

Scripting it is not much better at first pass. The page is server rendered, so the first batch is in the HTML, but everything after that comes from a cursor-paginated endpoint that the page calls as you scroll. You end up reverse engineering the pagination, then handling the fact that cheque sizes come back as strings, that geography is modelled as a sector rather than a location field, and that a "firm" on a list is only a name and a link until you enrich it.

None of that is hard. It is just a day you did not plan to spend.

The shortcut: run it as an API

Console. Open the NFX Signal Investor API, pick a mode, hit Start.

REST. One call, no login to the source needed:

curl -X POST "https://api.apify.com/v2/acts/johnvc~nfx-signal-investor-api/run-sync-get-dataset-items" \
  -H "Authorization: Bearer $APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"mode":"investors","listSlugs":["saas-seed"],"maxItems":50}'

The run endpoint is documented in the Apify API reference.

MCP. Point any MCP client at the server below and an agent can call it directly:

https://mcp.apify.com/?tools=actors,docs,johnvc/nfx-signal-investor-api

That last one matters more than it looks. "Which seed investors back developer tools, and what do they write?" is a question an agent can now answer from live data instead of memory.

How to find investors for a startup, by stage and cheque size

The one I use most. Pick the lists that match your round, then sort by targetInvestment:

{ "mode": "investors", "listSlugs": ["saas-seed"], "maxItems": 100 }

{
  "result_type": "investor",
  "name": "Brendan Wallace",
  "position": "managing_partner",
  "minInvestment": 10000000,
  "targetInvestment": 12500000,
  "maxInvestment": 25000000,
  "firmName": "Fifth Wall Ventures",
  "personUrl": "https://signal.nfx.com/investors/brendan-wallace"
}

Ready to run: Find Investors for Your Startup by Stage and Check Size.

Getting a list of VC firms in San Francisco

Switch to firms mode and you get the firm directory instead of individual partners, which is the shape you want for a CRM import.

{ "mode": "firms", "listSlugs": ["san-francisco-bay-area"] }

Ready to run: List of VC Firms in San Francisco Bay Area.

Pre seed investors, with the cheque sizes attached

Pre-seed is the stage where cheque size matters most and is published least consistently. Three sector lists in one run:

{ "mode": "investors", "listSlugs": ["fintech-pre-seed", "saas-pre-seed", "ai-pre-seed"], "maxItems": 500 }

Ready to run: Pre Seed Investors List With Check Sizes.

Using it as a Crunchbase alternative

It complements those tools rather than replacing them, and the difference is worth being precise about. Crunchbase and PitchBook are company-centric: they tell you what a company raised. This is investor-centric: it tells you who invests, at what stage, and what they write. You can attach Crunchbase data per firm in the same run:

{ "mode": "investors", "listSlugs": ["ai-seed"], "maxItems": 100, "enrichWithCrunchbase": true }

Ready to run: Crunchbase Alternative for Investor and VC Firm Data.

Browsing the whole VC database first

Slugs are not guessable, so start here. lists mode returns the catalog with a slug, stage, sector, and investor count for each of the 349 lists.

{ "mode": "lists" }

Ready to run: VC Database Export by Sector and Stage.

Exporting a Signal NFX list to JSON or CSV

If you just want one list out of the browser and into a file, this is the plain version, no enrichment, no filtering: Export Signal NFX Investor Lists to JSON or CSV.

Tracking new investors entering your sector

An export is a snapshot. Save the input as a Task, attach a monthly Schedule, and diff the runs to see who entered your sector: Track New Investors Joining a VC List Each Month.

The example repo

johnisanerd / Apify-NFX-Signal-Investor-API

NFX Signal Investor API examples: find investors, check sizes, and a list of VC firms from Signal by NFX as JSON. Python + MCP quick start.

NFX Signal Investor API: find investors, check sizes, and VC firms

Python and MCP examples for the NFX Signal Investor API on Apify. It turns Signal by NFX investor lists into structured JSON: venture investors, the VC firms behind them, and the check size each investor actually writes.

Not affiliated with, endorsed by, or connected to NFX. It reads the same public investor lists any visitor can browse.

Text walkthrough

If you have ever searched for how to find investors for a startup, the hard part is not finding names, it is finding the right names with the right cheque size. This API starts from the public Signal NFX investor lists: 349 of them, spanning 103 sectors and four stages from pre-seed to Series B. Run mode: "lists" first and you get the whole catalog with a slug, a stage, a sector, and an investor count for each one. Pick…

View on GitHub

Python quick start with uv, five task-aligned helper functions, and MCP install walkthroughs for Claude Cowork, Claude Code, Claude on the web, Cursor, and ChatGPT.

FAQ about scraping NFX Signal investor data

Do I need an NFX account to use this scraper?

No. It reads the public investor lists on signal.nfx.com, so no account, cookie, or token is needed for the source. You only need an Apify token to run the Actor.

Should I build my own scraper for this instead?

If you need one list once, scrape it yourself, it is a fair afternoon of work. The pagination, the stringified cheque amounts, and the firm enrichment chain are the parts that make it a maintenance job rather than a script.

Can I use this scraper from Claude or another AI agent?

Yes, over MCP. Point the client at the server URL above and the agent can discover the input schema and run it, which turns "who invests in my sector" into a live query.

Can I schedule this scraper to run every month?

Yes. Save your input as a Task, attach a Schedule, and diff each run. The monthly tracking example is set up for exactly that.

What does this scraper not give me?

Two honest limits. Not every investor publishes a cheque range, and those fields come back empty rather than estimated. And firm enrichment resolves by company name, so a firm with an ambiguous name may come back without LinkedIn or Crunchbase fields attached; you are not charged for the ones that miss.

More from Truffle Pig Data

This Actor is catalogued on Alpha OSINT alongside the rest of the financial and company data sources.

Try it

Run the NFX Signal Investor API on a single list first, with maxItems set low, and see whether the cheque sizes change how you build your list. That is the part I did not expect to matter as much as it does.

How to Pull Remote Startup Jobs With Salary and Equity in 2026 (Python + a jobs API)

Truffle Pig Data — Thu, 20 Aug 2026 13:05:33 +0000

Pulling remote startup jobs with real salary and equity numbers attached is harder than it should be. Wellfound, the site formerly known as AngelList Talent, is one of the few places where companies publish both right on the posting, which is why I went there to answer what sounded like a simple question: what does a seed-stage startup actually pay a backend engineer?

Reading a few hundred postings by hand is not a research method. So here is the manual approach, where it falls over, and the jobs API I use instead.

Disclosure: the Apify links in this post are affiliate links. If you run the Actor, I may earn a referral commission at no extra cost to you.

What the API returns

The Wellfound Jobs API returns startup job listings as structured JSON, with the posted compensation parsed into separate salary and equity numbers.

Most job feeds hand you a string like $135k - $165k * 0.05% - 0.15% and wish you luck. This one gives you that string and the parsed numbers next to it:

Field	Example	Notes
`compensationRaw`	`$135k – $165k • 0.05% – 0.15%`	exactly as posted
`salaryMin` / `salaryMax`	`135000` / `165000`	parsed, `null` when unpublished
`equityMin` / `equityMax`	`0.05` / `0.15`	percent, `null` when unpublished
`company.ycFunded`	`true`	Y Combinator flag
`company.stage`	`early_stage`	funding stage
`remoteKind`	`REMOTE_ONLY`	remote arrangement
`description`	full markdown	the whole posting

You also get title, url, locationNames, acceptedRemoteLocationNames, jobType, yearsExperienceMin/Max, postedAt, and a company object with size, tagline, and the topInvestors and activelyHiring flags.

Who this is for

You are building a job board or an aggregator and want early-stage roles that the big feeds miss.
You are weighing an offer and want a defensible comp range instead of a vibe.
You are a founder setting a band and have nothing to compare against.

The DIY path, and where it breaks

Wellfound is a Next.js app, so the listing data is sitting in the __NEXT_DATA__ blob rather than the HTML. You can get surprisingly far with requests and a JSON walk:

import json, re, requests

html = requests.get("https://wellfound.com/role/r/software-engineer").text
blob = json.loads(re.search(r'__NEXT_DATA__[^>]*>(.*?)</script>', html, re.S).group(1))
apollo = blob["props"]["pageProps"]["apolloState"]["data"]
jobs = [v for v in apollo.values() if v.get("__typename") == "JobListingSearchResult"]
print(len(jobs), "jobs on page 1")

Three things break this, in the order you hit them:

The compensation is one string. $135k – $165k • 0.05% – 0.15% has to be split into four numbers, and the format varies. Salary only, equity only, hourly, "competitive".
Role slugs are not free text. /role/software is a 404. Only /role/software-engineer exists. You need the real slug list before you can query anything.
The page shapes differ. The role pages use JobListingSearchResult, but the site-wide feed uses a completely different typename with a different nesting. Parse one and the other silently returns nothing.

None of that is unsolvable. It is just a parser you now own forever.

The shortcut: run it as an API

The Wellfound Jobs API handles the parsing, the slug mapping, and both page shapes. Three ways in.

Console. Open the Actor page, click Try for free, set a role, and run it.

REST. Any HTTP client, using the Apify run-sync endpoint:

curl -X POST "https://api.apify.com/v2/acts/johnvc~wellfound-jobs-api/run-sync-get-dataset-items?token=$APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"roles":["software-engineer"],"remoteOnly":true,"maxItems":25}'

MCP. It runs as a tool for Claude and other MCP clients. Point your config at:

https://mcp.apify.com/?tools=actors,docs,johnvc/wellfound-jobs-api

Then ask in plain language: "find remote startup jobs for a senior backend engineer paying over 160k, and tell me which companies are YC backed." Setup docs are in the Apify MCP guide.

Find remote startup jobs

The one I reach for most. remoteOnly filters to listings that actually accept remote applicants, and acceptedRemoteLocationNames on each row tells you where they accept them from, which is the part that usually disqualifies you.

{"roles": ["software-engineer"], "remoteOnly": true, "minSalary": 150000, "maxItems": 50}

Ready to run: Find Remote Startup Jobs.

Find Y Combinator startup jobs

ycOnly narrows to YC-backed companies, which is the filter I have not found anywhere else. Pair it with companyStage so you are comparing seed against seed rather than seed against Series C.

{"roles": ["software-engineer"], "ycOnly": true, "includeDescription": false, "maxItems": 150}

Ready to run: Find Y Combinator Startup Jobs.

Startup jobs in San Francisco

locations takes city slugs and pairs with any role. Each role and city combination becomes its own search, so three roles across four cities is twelve searches, not seven. Watch maxItems accordingly.

{"roles": ["product-manager"], "locations": ["san-francisco"], "maxItems": 50}

Ready to run: Find Startup Jobs in San Francisco.

Startup jobs in New York

Same shape, different slug. Worth running separately rather than together, because the comp distributions are genuinely different and averaging across cities hides that.

{"roles": ["data-scientist"], "locations": ["new-york"], "minSalary": 140000, "maxItems": 50}

Ready to run: Find Startup Jobs in New York.

Startup jobs in London

Non-US cities work the same way. Check salaryCurrency on the rows before you compare anything, because a London posting quoting GBP next to a SF posting quoting USD is not a comparison.

{"roles": ["backend-engineer"], "locations": ["london"], "maxItems": 50}

Ready to run: Find Startup Jobs in London.

Get startup salary and equity data

The comp use case, with descriptions off because nobody averages prose. Drop the rows where the field is null before you compute anything, and report how many survived.

rows = [r for r in items if r.get("result_type") == "job"]
with_salary = [r for r in rows if r.get("salaryMin") is not None]
with_equity = [r for r in rows if r.get("equityMin") is not None]
print(f"{len(with_salary)}/{len(rows)} published salary, {len(with_equity)}/{len(rows)} published equity")

On a 30-row YC sample I ran while writing this, 28 published a salary and only 5 published equity. Those are different sample sizes and deserve separate sentences.

Ready to run: Get Startup Salary and Equity Data.

The example repo

johnisanerd / Apify-Wellfound-Jobs-API

Startup jobs API example: pull Wellfound (AngelList) jobs with parsed salary, equity, remote status and YC signals. Python + uv quick start and MCP setup.

💼 Wellfound Jobs API: A Startup Jobs API for Salary and Equity Data

A startup jobs API for Wellfound (formerly AngelList Talent). Search by role, location, and remote status, and get one clean JSON row per job: the full description, parsed salary and equity ranges, and the hiring startup's signals including Y Combinator backing, top investors, and funding stage.

Actor: https://apify.com/johnvc/wellfound-jobs-api?fpr=9n7kx3

Video Walkthrough

Text walkthrough

This jobs API turns Wellfound's public startup listings into structured data. You give it roles (Wellfound slugs like software-engineer, or plain words like "software" and "pm", which are mapped for you) and optionally locations such as san-francisco, new-york, or london. Each role and location pair becomes its own search, and the API walks the paginated results for you.

Every row comes back with title, url, the full markdown description, and compensation parsed into real numbers: salaryMin, salaryMax…

View on GitHub

Python quick start plus MCP setup walkthroughs for Claude Code, Cursor, and ChatGPT.

FAQ about scraping Wellfound jobs

Should I build my own Wellfound scraper or use an API?

Build it if the parser is the interesting part of your problem. Use the API if the data is. The three breakages above are the ones you will spend your time on, and they change without warning.

Can I use this jobs scraper from Claude or another AI agent?

Yes. It runs as an MCP tool at https://mcp.apify.com/?tools=actors,docs,johnvc/wellfound-jobs-api. There are also two open agent skills built on it, linked below.

Can I schedule this scraper to run automatically?

Yes, through Apify Schedules, on any cron you like. Dedupe on url, which is canonical and survives a company editing the job title.

Does the scraper return salary and equity for every job?

No, and this is the honest limitation that matters most. It returns what the company published. Plenty of postings carry neither. Always report your coverage count alongside any figure you quote.

Can the scraper search Wellfound by keyword?

Not site-wide, because Wellfound does not serve a site-wide free-text search. keyword filters the jobs on the pages a run already fetched. Drive the search with roles and locations and use keyword to narrow further.

Wrapping up

If you want the parser, the code above is a fine starting point. If you want the data, the Wellfound Jobs API bills per job returned with no start fee, so a run that matches nothing costs essentially nothing.