DEV Community: Truffle Pig Data

Local Lead Generation with Yelp Data: How to Scrape Yelp Search Results in 2026

Truffle Pig Data — Fri, 07 Aug 2026 19:09:36 +0000

If you sell anything to local businesses, Yelp is one of the best prospect databases that exists: ranked listings, review counts that signal how established a business is, price tiers, phone numbers, neighborhoods. Getting that out of the site programmatically is the annoying part. The Yelp Search API on Apify does it for you: a search term and a location go in, and the ranked business listings come out as clean JSON.

Disclosure: the Apify links in this post are affiliate links. If you run the Actor, I may earn a referral commission at no extra cost to you.

Doesn't Yelp already have an API?

It does. The Yelp Fusion API is official, documented, and worth knowing about. It also comes with the usual official-API friction: key registration, daily call quotas, terms that constrain how you store and reuse the data, and responses shaped by what Yelp chooses to expose rather than what the search page shows. The ranked results page, with its sponsored slots and refinement filters, is its own thing. A scraper you call like an API gives you that page as data, with no key and no quota negotiation, which is usually what local lead generation work actually needs.

What the Yelp Search API returns

The Yelp Search API returns ranked local business listings as structured JSON: name, rating, review count, price tier, categories, phone, neighborhood, and the place IDs you need to pull full details later.

Field	Example	Notes
Business name	`Brooklyn Smile Dental`	As ranked on the page
Rating	`4.7`	Star rating
Reviews	`284`	Review count, a proxy for how established they are
Price	`$$`	Yelp's price tier
Categories	`dentists, cosmetic dentists`	Yelp category aliases
Phone	`(718) 555-0139`	The outreach field

Each result also carries a filters object listing the valid category, price, feature, distance, and neighborhood refinements for that search, so you can discover legal filter values straight from the data instead of guessing.

Who this is for

Local lead generation shops building B2B lists by trade and metro. Agencies pitching reputation or marketing services who want review counts and ratings attached to every prospect. And data teams who need a ranked snapshot of a local market, whether that's dentists in Brooklyn or every $$ restaurant near a ZIP code.

The manual way, and where it breaks

Yelp's search results render through heavy client-side code, the markup is deliberately unfriendly to parsers, and unattended scripts meet the anti-bot wall quickly. The refinement system runs on URL parameters (find_desc, find_loc, cflt, attrs) that are documented nowhere, so you reverse-engineer them from your browser bar. I did exactly that while building this Actor, and I can report the parameters change just often enough to keep a DIY scraper permanently on your maintenance list.

The faster way: run the Yelp Search API

Apify Console

Open the Yelp Search API and click Try for free.
Set location (city, address, or ZIP) and optionally a search_term.
Run it and export the listings as JSON or CSV.

REST

curl -X POST "https://api.apify.com/v2/acts/johnvc~yelp-search-api/runs?token=YOUR_APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{ "search_term": "plumbers", "location": "Chicago, IL", "sort_by": "review_count", "max_pages": 2 }'

Standard Apify run endpoints; reference in the Apify API docs.

Search Yelp in Python

from apify_client import ApifyClient

client = ApifyClient("YOUR_APIFY_TOKEN")

run = client.actor("johnvc/yelp-search-api").call(
    run_input={
        "search_term": "dentists",
        "location": "Brooklyn, NY",
        "sort_by": "rating",
        "max_pages": 2,
    }
)

for biz in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(biz.get("rating"), biz.get("phone"), biz.get("categories"))

Twenty listings, sorted by rating, with phone numbers attached: that's a cold-call sheet in about ten lines.

Build B2B lists by ZIP code

The task Find dentists by ZIP code for B2B lead lists shows ZIP-level targeting, the granularity most sales territories are drawn in.

Target one borough at a time

Find dentists in Brooklyn for lead generation is the metro version of the same play, ready to clone for your own city and niche.

Cover the trades

Find plumbers in Chicago for lead generation applies the pattern to home services, where review count doubles as a longevity signal.

Rank the best restaurants in a city

Find the best restaurants in Newark uses rating sort for a market snapshot rather than a lead list.

Clone it for the next market

Find the best restaurants in Oakland is the same search pointed at another coast, which is the whole point: one saved task per market, run on demand.

Yelp MCP: let your agent search listings

Through Apify's MCP server, the Actor is callable from Claude, Claude Code, and Cursor as a tool. Ask "find the ten highest-rated plumbers in Chicago and give me their phone numbers" and the agent runs a real search instead of inventing plausible-sounding businesses. Setup is a one-time config, and you can read more about Claude at claude.ai.

FAQ about the Yelp scraper

Is the Yelp scraper free, and what does it cost after that?

Runs bill one cent per results page (about 10 businesses) plus a one-cent setup fee, so a 100-listing pull costs roughly eleven cents. New Apify accounts ship with free platform credit that covers early runs.

Why use a scraper instead of the Yelp Fusion API?

Use Fusion when its quotas and terms fit your project; it's a fine API. Use the scraper when you want the ranked search page itself, filters and all, without key management or daily caps, and in a shape built for exporting lead lists.

Can Claude run this Yelp scraper through MCP?

Yes. Registered as an MCP tool it works from Claude, Claude Code, and Cursor, returning the same structured listings a direct run produces.

Can I schedule the Yelp scraper for rank tracking?

Yes. Multi-location brands do this to watch where each location ranks for its core search. Save a task per location-query pair, attach an Apify schedule, and diff positions over time, starting from the Yelp Search API.

What data will the search scraper not give me?

Whatever the results page doesn't show: no email addresses, no full review text, no hours. It returns place IDs precisely so you can chain a second Actor for depth on the businesses that matter.

More from Truffle Pig Data

The chain I just mentioned: the Yelp Place API turns place IDs into full business detail, the Yelp Reviews API pulls the review text, and the Google Local API gives you the same market from Google's side for comparison.

Wrapping up

Local lead generation runs on fresh, ranked, filterable business data, and Yelp has exactly that. The Yelp Search API hands it to you as JSON; start with one niche in one city and scale from there.

Pricing Intelligence for iOS Apps: How to Pull App Store Product Data in 2026

Truffle Pig Data — Fri, 07 Aug 2026 11:11:27 +0000

A surprising amount of product strategy starts with somebody reading an App Store page: what a competitor charges, which in-app purchases they sell, what their last ten versions shipped, how their rating moves. I got tired of doing that reading by hand, so I built the Apple App Store Product API on Apify. Feed it App Store IDs or URLs and it returns the full product record for any iOS, iPadOS, or macOS app as structured JSON, one row per app per country.

Disclosure: the Apify links in this post are affiliate links. If you run the Actor, I may earn a referral commission at no extra cost to you.

Does the Apple App Store have an API?

Yes, partly, and the partly is why this Actor exists. Apple's iTunes Search and Lookup endpoints are real and free, and for basic metadata they're fine. But the payload reflects an older App Store: you won't find the privacy "nutrition" cards, the itemized in-app purchase price list, version-by-version release notes, or the sample reviews that sit on today's product page, and Apple publishes no firm rate guarantees for it. When the data you need is "everything on the modern product page, per storefront," a scraper consumed as an API is the practical route.

What the App Store product API returns

The Apple App Store Product API returns one complete app record per ID and country: title, developer, description, price, ratings, version history, screenshots, in-app purchases, supported languages, privacy cards, and a sample of reviews.

Field	Example	Notes
`title`	`Spotify: Music and Podcasts`	App name as listed
`developer`	`Spotify`	Publisher of record
`price_text`	`Free`	Localized price for the storefront queried
Ratings	`4.8`	Star average plus counts
In-app purchases	`subscription tiers with prices`	Itemized, the pricing intelligence core
`lookup_country`	`de`	Which storefront this row came from

Version history, privacy cards, and supported languages ride along in the same row, and include_reviews_sample controls whether the page's few visible reviews are attached.

Who this is for

Pricing intelligence is the headline use: analysts comparing what an app charges across 50 storefronts, or watching a competitor's in-app purchase ladder change quarter to quarter. Product and ASO teams doing competitive research are the second group. The third is AI agent builders who want app metadata as agent context, so "summarize this competitor's app and its monetization" runs on real data.

The manual way, and where it breaks

The product page is public, so scraping it yourself is tempting. The details fight back. Much of the page content lives in embedded data structures that shift with App Store redesigns, storefront selection depends on URL country prefixes and headers that are easy to get subtly wrong, and localized price formatting differs per region, so your parser needs currency logic on day one. Multiply all of that by a country matrix and a retry layer, and the quick script becomes a project. I wrote the throwaway version three times before admitting it wasn't throwaway.

The faster way: run the Apple App Store Product API

Apify Console

Open the Apple App Store Product API and click Try for free.
Paste App Store IDs or full URLs into product_ids and pick a country.
Run it and download the records as JSON or CSV.

REST

curl -X POST "https://api.apify.com/v2/acts/johnvc~apple-app-store-product-api/runs?token=YOUR_APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{ "product_ids": ["324684580"], "country": "us" }'

ID 324684580 is Spotify; a full URL like https://apps.apple.com/us/app/spotify/id324684580 works too, since the ID gets parsed out. Run mechanics are in the Apify API docs.

Fetch app records in Python

from apify_client import ApifyClient

client = ApifyClient("YOUR_APIFY_TOKEN")

run = client.actor("johnvc/apple-app-store-product-api").call(
    run_input={
        "product_ids": ["324684580", "310633997"],
        "countries": ["us", "gb", "de"],
    }
)

for app in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(app.get("title"), app.get("lookup_country"), app.get("price_text"))

Two IDs across three storefronts returns six rows, one per app-country pair, which is the shape you want for a pricing matrix.

Compare prices across country stores

The task App Store prices by country is the regional pricing sweep: one app, many storefronts, localized price_text per row.

Benchmark competitor apps

Compare competitor iOS apps on price and features runs a set of rival apps through one pull, which is the fastest way to build a feature-and-price comparison table that's current.

Extract in-app purchase price lists

Extract iOS app in-app purchase prices targets the monetization ladder specifically, the part Apple's official lookup won't give you.

Look up any app by ID

Get Apple App Store app details by app ID is the minimal single-app run, useful as a smoke test before you scale up.

Watch ratings and releases over time

Track iOS app ratings and version history shows the monitoring setup: run it on a schedule and you accumulate a longitudinal record of rating drift and release cadence.

App data as AI agent context

Hooked up over the Model Context Protocol, the Actor becomes a tool that Claude, Claude Code, and Cursor can call, so an agent can pull a live app record before answering questions about it. The task iOS app details in Claude MCP has the wiring, and there's more on Claude at claude.ai.

FAQ about scraping App Store product data

What does the App Store scraper cost to run?

Two cents per app record returned, plus a two-cent setup fee per run. A 100-app competitive sweep, the per-run maximum, lands around $2. Apify's free credit on new accounts covers plenty of experimentation first.

How is this scraper different from Apple's own lookup endpoints?

Apple's endpoints return the classic metadata slice. The scraper returns the product page as it exists now, including in-app purchase pricing, privacy cards, version history, and per-storefront localization, which is where pricing intelligence work actually happens.

Can an AI agent call this App Store scraper over MCP?

Yes. Registered as an MCP tool, it's callable from Claude, Claude Code, and Cursor, and each call returns full app records as tool output.

Can I schedule the scraper to monitor price changes?

Yes. Save your app list as a task, attach an Apify schedule, and compare price_text and in-app purchase entries between runs. Start from the Apple App Store Product API.

What won't this product scraper give you?

The full review corpus. It carries the small review sample Apple displays on the page, typically three, which is enough for flavor but not sentiment analysis. It also reports one storefront per row, so worldwide coverage means listing the countries you care about rather than assuming one row covers all.

More from Truffle Pig Data

The rest of the Apple toolkit: the Apple App Store Reviews API when you do need reviews at volume, Apple App Store Search for finding app IDs by keyword, and the Apple Maps API for Apple's places data.

Wrapping up

Competitive app research shouldn't mean reading product pages in fifty tabs. Point the Apple App Store Product API at your competitor set and get the whole picture as JSON.

Google Local AI: Get Local Pack Data as JSON for Agents, Lead Gen, and Rank Tracking

Truffle Pig Data — Thu, 06 Aug 2026 08:10:24 +0000

The local pack, that map-plus-three-businesses block at the top of Google Search, decides who gets the phone call for "plumber near me" in every city on earth. For local SEO work and lead generation you need that block as data, and Google doesn't hand it over. The Google Local API on Apify fixes that: send a query and a city, get back the local results as JSON with ratings, phones, hours, and coordinates.

Disclosure: the Apify links in this post are affiliate links. If you run the Actor, I may earn a referral commission at no extra cost to you.

Does Google have a local pack API?

Google has the official Places API, and it's genuinely good at what it does: look up businesses, get details, autocomplete addresses. What it will not tell you is what the local pack actually shows for a search in a specific city, in what order, on which device. Pack composition and ranking are SERP features, not Places data, and there's no official endpoint for them. Rank trackers and lead-gen tools need the page as served, which is why a scraper you call like an API fills the gap: it queries Google Local the way a user in that location would and returns the businesses it saw.

What the Google Local API returns

The Google Local API returns each local result as structured JSON: business name, rating, review count, address, phone, hours, GPS coordinates, place_id, and links.

Field	Example	Notes
Business name	`Smith & Sons Plumbing`	As shown in the pack
`rating`	`4.8`	Star rating
Reviews	`312`	Review count behind the rating
`phone`	`(813) 555-0182`	The lead-gen field
Hours	`Open until 6 PM`	Current status as rendered
`place_id`	`ChIJd8BlQ2Bx...`	Stable ID for joins with other Google data

GPS coordinates ride along for mapping, and you can target by google_domain, country (gl), language (hl), and device. Desktop pages carry about 20 businesses, mobile about 10, which matters because billing is per page.

Who this is for

Local SEO agencies tracking pack positions for clients. Lead-gen operators building contact lists of roofers, med spas, or HVAC contractors city by city. And builders wiring local business lookup into AI agents, where "find me the top-rated dentists in Chicago" should return data, not vibes.

The manual way, and where it breaks

Scraping the local pack yourself means faking a location. Google decides what you see from where it thinks you are, so you're forging uule strings, juggling consent pages, and rendering JavaScript before you parse a single card. Then the anti-bot layer notices your datacenter IP. Then the markup changes. I keep a mental list of scrapers I've written twice, and Google SERP parsers sit at the top of it. The location spoofing alone is why I'd rather call something hosted.

The faster way: run the Google Local API

Apify Console

Open the Google Local API and click Try for free.
Set q to a search like "plumbers" and location to a city.
Run it and export JSON or CSV.

REST

curl -X POST "https://api.apify.com/v2/acts/johnvc~google-local-api/runs?token=YOUR_APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{ "q": "plumbers", "location": "Tampa, Florida, United States", "max_pages": 1 }'

The run lifecycle is standard Apify; details in the Apify API docs.

Pull local businesses in Python

from apify_client import ApifyClient

client = ApifyClient("YOUR_APIFY_TOKEN")

run = client.actor("johnvc/google-local-api").call(
    run_input={
        "q": "dentists",
        "location": "Chicago, Illinois, United States",
        "device": "desktop",
        "max_pages": 1,
    }
)

for biz in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(biz.get("rating"), biz.get("phone"), biz.get("place_id"))

Loop the same call over a city list and you've got a scraper-backed prospecting pipeline in thirty lines.

Build lead lists for home services

The biggest family of published tasks does exactly this, one trade per task: roofer contacts in Dallas, HVAC contractors in Phoenix, and plumbers in Tampa. Each is a preconfigured run you can clone for your own metro.

Cover professional and medical niches

The same pattern extends to higher-ticket verticals: med spas in Scottsdale with ratings, dental implant leads in Phoenix, and immigration lawyers in New York.

Track local pack rankings for SEO

Rank tracking is the other half of the story. Google local pack rank tracker for SEO agencies is the agency template, track "plumber near me" local pack rankings covers the near-me query class, and monitor local SERP visibility for dentist SEO shows ongoing monitoring for one niche.

No code, just a spreadsheet

Export Google Local results to CSV is the Console-only path from search to sheet.

Google Local AI: give your agent the pack

Where this gets interesting in 2026 is agents. Over the Model Context Protocol, the Actor becomes a tool Claude, Claude Code, or Cursor can call, so a prompt like "who ranks in the local pack for roofers in Denver, and what are their ratings?" triggers a live run and comes back grounded in the actual SERP. Wire it up through Apify's MCP server, and read more about Claude at claude.ai.

FAQ about the Google Local scraper

What does the local pack scraper cost per run?

Two cents per page of results, plus a small setup fee per run. A desktop page holds around 20 businesses, so a city-level lead pull works out to about a tenth of a cent per business. max_pages caps spend, and new Apify accounts include free credit.

Can this scraper power a local rank tracker on a schedule?

Yes, that's the core SEO use. Save one task per query-city pair, attach an Apify schedule, and diff positions run over run. Start from the Google Local API and the agency rank-tracker task above.

Does the scraper support "near me" and precise locations?

It does. Use location for city-level targeting or a uule string for finer control, plus gl, hl, and google_domain for country and language. The device setting switches between desktop and mobile results, which often differ.

Can Claude call this scraper as an MCP tool?

Yes. Registered through Apify's MCP server, it shows up as a callable tool in Claude, Claude Code, and Cursor, and returns the same structured rows an API call would.

What won't a local pack scraper give you?

Email addresses, for one: Google Local shows phone, hours, and links, so email prospecting needs an enrichment step. And the pack is inherently local and personalized, so your run reflects the location and device you requested, not the exact page every user sees. Treat positions as a consistent measurement, not a universal truth.

More from Truffle Pig Data

Adjacent Actors for local data work: the Google Maps Places API for place detail lookups, the Google Local Services API for the sponsored pros block, and the Google Maps Directions API for travel times between the businesses you find.

Wrapping up

The local pack is where local revenue starts, and now it's queryable. Point the Google Local API at your niche and city, and you'll have the block as JSON before your coffee cools.

Business Development with SEC Data: How to Build Investment Advisor Lead Lists in 2026

Truffle Pig Data — Wed, 05 Aug 2026 19:09:42 +0000

Every SEC-registered investment advisor in the United States sits in a public registry, the Investment Adviser Public Disclosure database, and almost nobody in business development uses it directly because the raw data is a chore to work with. I kept rebuilding the same parsing pipeline for finance prospecting projects, so I turned it into the SEC Investment Advisors Search on Apify: 250,000+ investment professionals and 15,000+ RIA firms, queryable by name, location, or CRD number, returned as structured JSON.

Disclosure: the Apify links in this post are affiliate links. If you run the Actor, I may earn a referral commission at no extra cost to you.

Is there an API for SEC investment advisor data?

Sort of, and that "sort of" is the problem. The registry is public and the SEC does publish compiled advisor data, but as bulk snapshot files meant for downloading, not a query API you can hit with a city and a firm name. There's no endpoint for "give me the contacts at RIA firms in Texas updated since March." So the practical version of an SEC advisor API is a scraper-backed database you query like an API: filters in, contact and firm records out, with pagination and incremental updates handled for you.

What the advisor contacts API returns

The SEC Investment Advisors Search returns firms, contacts, or both as structured JSON, with each contact linked to their firm.

Field	Example	Notes
Contact name	`Jane Smith`	Individual advisor or representative
Email	`jsmith@examplecapital.com`	Includes a verification status
Phone	`+1 212 555 0140`	Contact-level number when present
Professional profile URL	`linkedin.com/in/...`	Link to the person's profile
Firm name	`Example Capital Advisors`	With firm-level identifiers like CRD
Office address	`New York, NY, US`	Plus firm website

Records carry timestamps and identifiers for deduplication, which matters once you start running this on a schedule and only want what changed.

Who this is for

The obvious crowd is business development and sales teams selling into financial services: fintech vendors, compliance software, custodians, anyone whose buyer is an RIA. The less obvious users are market researchers mapping how advisory firms distribute across geographies, and CRM operators who need to enrich a stale contact database with firm associations and verified emails.

The manual way, and where it breaks

The DIY path is to download the SEC's compiled advisor data, unpack a very large file, and write parsers to join individuals to firms. I've done it, and it works once. Then next month's snapshot lands and you rerun everything from scratch, because the bulk files don't tell you what changed. Filtering by geography means loading the whole country to keep one state. And the compilation gives you registrations, not an outreach-ready record; profile links and email verification are on you. None of it is hard, exactly. It's just a data-engineering project standing between you and a prospect list.

The faster way: run the SEC Investment Advisors Search

One JSON input, one filtered result set, pay per contact returned.

Apify Console

Open the SEC Investment Advisors Search and click Try for free.
Pick a query_type (firms, contacts, or both) and set filters like firm_city or firm_state.
Run it and download the dataset as JSON or CSV.

REST

curl -X POST "https://api.apify.com/v2/acts/johnvc~SECInvestmentAdvisorContacts/runs?token=YOUR_APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{ "query_type": "both", "firm_city": "New York", "firm_state": "NY", "contacts_limit": 25 }'

Endpoint mechanics are in the Apify API docs.

Query advisor contacts in Python

from apify_client import ApifyClient

client = ApifyClient("YOUR_APIFY_TOKEN")

run = client.actor("johnvc/SECInvestmentAdvisorContacts").call(
    run_input={
        "query_type": "contacts",
        "contact_firm_name": "Morgan Stanley",
        "contacts_limit": 10,
    }
)

for contact in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(contact.get("contact_id"), contact.get("name"), contact.get("email"))

Swap contact_firm_name for organization_crds when you already know the firms you're targeting; CRD numbers are exact where name matching is fuzzy.

Build an RIA list for one city

The task Build an RIA contact list for New York City shows the metro-area pattern: city plus state filters, firms and contacts together.

Turn registry data into lead lists

Build financial advisor lead lists from SEC data is the general business development recipe, going from filters to an outreach-ready sheet.

Skip the code entirely

Download SEC investment advisor data as CSV runs in the Console and ends in a spreadsheet, no Python required.

Cover a whole state

List SEC registered investment advisors in Texas demonstrates state-level coverage, useful for territory planning and market sizing.

Prospect from Claude via MCP

Because Apify exposes Actors over the Model Context Protocol, Claude, Claude Code, and Cursor can run advisor searches as a tool call. "Find me RIA firms in Austin and pull their contacts" becomes a prompt instead of a script, and the structured records land in the conversation. The task Prospect SEC investment advisors from Claude via MCP has the setup, and you can read about Claude itself at claude.ai.

FAQ about the SEC advisor scraper

What does the SEC advisor scraper cost per contact?

About a cent per contact returned, plus a couple of tenths of a cent in run setup fees. contacts_limit caps the spend before a run starts, and the free credit on a new Apify account covers a real first list.

Where does the scraper's contact data come from?

The underlying registry is the SEC's public Investment Adviser Public Disclosure data, covering registered firms and their professionals. The Actor layers structure on top: linked firm associations, profile URLs, and email verification status. It's public-record data, and your outreach still has to follow the usual email and telemarketing rules.

Can I use this scraper for CRM enrichment on a schedule?

Yes, and the date_updated filter is the key. Run it monthly with an Apify schedule, request only records updated since your last sync, and you pay for changes rather than the whole registry. Start from the SEC Investment Advisors Search and save the filter set as a task.

Does the scraper work from Claude or other MCP clients?

Yes. Connect it through Apify's MCP server and it appears as a callable tool, so an agent can pull advisor lists mid-conversation.

What won't this scraper give you?

It covers SEC-registered investment advisors, so brokers, insurance agents, and state-only registrants outside that registry won't appear. Emails carry a verification status but no guarantee of deliverability, and profile URLs exist only where a profile was found. Treat it as a strong starting list, not a finished campaign.

More from Truffle Pig Data

If you're building a finance prospecting stack, these sit next door: the LinkedIn Company API for firmographic detail, the Crunchbase Company API for funding context, and the PitchBook Company API for the private-markets view.

Wrapping up

The SEC already published the best advisor database in the country; it just didn't ship a query API for it. The SEC Investment Advisors Search fills that gap, one filtered run at a time.

Yandex Tracking at Scale: Build a Rank Tracker for 1,000 Keywords in 2026

Truffle Pig Data — Wed, 05 Aug 2026 13:15:10 +0000

Most rank trackers price Yandex like an afterthought, when they cover it at all. If your clients sell into Russia, Turkey, Kazakhstan, or anywhere else Yandex matters, the tooling gap is real: you either pay for an enterprise tier you don't need or you track rankings by hand. I built the Yandex Rank Tracker API to close that gap. It's a per-result Yandex SERP scraper: you pay for exactly the rows you pull, whether that's a 10-keyword daily check or a 1,000-keyword monthly sweep.

Disclosure: the Apify links in this post are affiliate links. If you run the Actor, I may earn a referral commission at no extra cost to you.

Does Yandex have a search API?

Technically yes, which makes this different from most sources I scrape. Yandex offers developer search APIs, but getting there means account registration in the Yandex ecosystem, API keys, quota tiers, and output built for site owners rather than rank trackers. What you don't get is the SERP as users see it: the ads, the knowledge graph card, the inline image and video blocks. A scraper-as-API fills that hole. You send a keyword and a region, and you get back the live results page as structured JSON, no key ceremony involved.

What the Yandex tracking data looks like

The Yandex Rank Tracker API returns each search result as its own JSON row: ranking position, title, target URL, snippet, favicon, publication date, and rich snippets when present.

Field	Example	Notes
Position	`3`	Rank on the page, the number you're tracking
Title	`CRM для малого бизнеса`	Result headline as served
URL	`https://example.ru/crm`	Target link for domain matching
Snippet	`Попробуйте бесплатно...`	Description text under the title
Publication date	`2026-06-12`	When Yandex shows one
Rich snippets	`sitelinks, ratings`	Present on enhanced results

Optional result types add paid ads with advertiser metadata, knowledge-graph entity cards, inline image strips, and video carousels. You can localize by domain (yandex.com, .ru, .by, .kz, .uz, .com.tr), pick from 19 interface languages, target a region ID, sort by relevance or date, and paginate as deep as you need.

Who this is for

SEO agencies with clients in CIS or Turkish markets, in-house teams that need daily Russian-language rank checks, ad-intelligence analysts watching who bids on which keywords, and builders giving an AI agent a second search engine beyond Google.

The manual way, and where it breaks

You can request a Yandex results page yourself and parse it. I tried, and the failure list got long fast. Yandex ships one of the more aggressive captcha walls in search, so unattended scripts stall within minutes. Regional targeting rides on an lr region ID system you have to map by hand. Cyrillic queries need careful encoding, the markup differs by domain and language, and ad blocks shift position between runs. Every one of those is fixable, and together they become a part-time job.

The faster way: run the Yandex Rank Tracker API

Documented input in, structured rows out, per-result billing.

Apify Console

Open the Yandex Rank Tracker API and click Try for free.
Set text to your keyword, pick a yandex_domain, and optionally a region.
Run it and export the rows as JSON or CSV.

REST

curl -X POST "https://api.apify.com/v2/acts/johnvc~yandex-scrape-yandex-search-results-at-scale---per-result/runs?token=YOUR_APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{ "text": "купить квартиру москва", "yandex_domain": "yandex.ru", "lr": "225", "max_pages": 1 }'

Region 225 is Russia; the run endpoint itself is covered in the Apify API docs.

Track Yandex rankings in Python

from apify_client import ApifyClient

client = ApifyClient("YOUR_APIFY_TOKEN")

run = client.actor("johnvc/yandex-scrape-yandex-search-results-at-scale---per-result").call(
    run_input={
        "text": "CRM для малого бизнеса",
        "yandex_domain": "yandex.ru",
        "lr": "225",
        "max_pages": 1,
    }
)

for row in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(row.get("position"), row.get("title"), row.get("url"))

Match url against your domain and you have a rank check; loop it over a keyword list and you have a tracker.

Track 1,000 keywords without a subscription

The flagship task, Track Yandex rankings for 1,000 keywords, shows the bulk pattern. Per-result pricing keeps the budget predictable: 1,000 keywords times a top-10 pull is 10,000 rows, about $150, whether that happens in one run or fifty.

Monitor SERPs and ads on a schedule

Bulk collect Yandex SERPs for SEO monitoring is the recurring-snapshot setup, and Monitor Yandex search ads by keyword points the same machinery at paid results, advertiser metadata included.

Pull knowledge graph, images, and video blocks

Rankings are not the whole SERP. Get Yandex knowledge graph entity data extracts the entity card, Extract Yandex inline images by query grabs the image strip, and Track Yandex video results by query covers the video carousel.

Go beyond Russian

There are ready-made tracks for 18 languages and several country markets, from Turkish SERPs to Kazakh SERPs to country-level runs like Yandex SERPs in Georgia. For the no-code crowd, Export Yandex search results to CSV skips the code entirely.

Let your AI agent read the SERPs

Over the Model Context Protocol, the Actor becomes a tool that Claude, Claude Code, or Cursor can call mid-conversation. Ask "where does mysite.ru rank for 'CRM для малого бизнеса', and who outranks it?" and the agent runs the check against live rows instead of guessing. The task Get Yandex SERP data in Claude via MCP walks through the config, and there's more on Claude at claude.ai.

FAQ about Yandex tracking with a scraper

How is this Yandex scraper different from the per-page Yandex Search API?

Same structured data, different meter. This Actor bills per result row, which wins for bulk rank tracking where you control exactly how many rows you pull. Its sibling, the Yandex Search API, bills per SERP page and wins for occasional full-page pulls.

What does the Yandex scraper cost per run?

About $0.015 per result row plus a small start fee, with no subscription underneath. A top-10 check on one keyword is roughly fifteen cents; new Apify accounts include free platform credit, so trial runs usually cost nothing.

Can Claude run this Yandex scraper through MCP?

Yes. Add it as an MCP tool and Claude or Cursor can run rank checks during a conversation, with the structured rows coming back as tool output.

How do I schedule the Yandex scraper for daily tracking?

Save your keyword list as a task, attach an Apify schedule with a cron expression, and each run appends fresh rows you can diff over time. Start from the Yandex Rank Tracker API and save your first task.

What won't this Yandex scraper tell you?

It reports the SERP for the domain, language, and region you request, which is not the personalized page any single logged-in user sees. It also can't see data Yandex doesn't render, search volume included; pair it with a keyword tool for that half of the picture.

More from Truffle Pig Data

I've written about the rank-tracking angle elsewhere: the Medium piece Build a Yandex Rank Tracker: Bulk SERP Monitoring at $0.015 per Result, the same story on LinkedIn, and a shorter build note on Peerlist.

Wrapping up

If your rank tracker goes blank east of Warsaw, this is the fix: per-result Yandex tracking that costs what you pull. Start with the Yandex Rank Tracker API and a ten-keyword test list.

Case Law API: How to Search US Court Opinions in 2026 (Python, MCP, AI Agents)

Truffle Pig Data — Tue, 04 Aug 2026 12:03:38 +0000

Commercial legal databases charge hundreds of dollars per seat per month, while Google Scholar's case law search is free and covers US federal and state opinions back well over a century. The catch is that it only exists as a web page. I wanted that corpus as JSON I could query from scripts and agents, which is what the Google Scholar Case Law API on Apify does: give it a query like "qualified immunity" and it returns opinions with courts, dates, and citations as structured data.

Disclosure: the Apify links in this post are affiliate links. If you run the Actor, I may earn a referral commission at no extra cost to you.

Does Google Scholar have a case law API?

No. Google Scholar has never shipped a public API for anything it indexes, and the case law collection is no exception. There is no key to request, no endpoint, no bulk export. The commercial alternatives license their own databases and price accordingly. So in practice a case law API means a scraper you consume like an API: send a query with court and year filters, get opinions back as JSON. That is what this Actor is, and it stays scoped to what Scholar actually publishes.

What the case law API returns

The case law API returns US court opinions as structured JSON: case title and party names, deciding court, decision date, reporter citation, a snippet, the cited-by count, and the list of cases each opinion cites.

Field	Example	Notes
Case title	`Harlow v. Fitzgerald`	Party names as Scholar renders them
Court	`Supreme Court of the United States`	The deciding court
Decision date	`June 24, 1982`	Argument and filed dates included when published
Citation	`457 US 800`	Reporter citations, page ranges, docket numbers
Cited by	`30,000+`	Citation count plus inline citation links
`result_id`	`9124178popularname...`	Identifier used to fetch the full case detail

A search hit is the light record. Pass its result_id back through caseIds, or set fetchCaseDetailsForResults to true, and you get the full case detail including the complete list of cases the opinion cites. That second layer is what makes citation-graph work possible.

Who this is for

Three groups keep showing up. Legal researchers and litigation support teams assembling jurisdiction-specific corpora or preloading case metadata before discovery. Developers doing citation analysis, where cited-by counts and cited-case lists become edges in a graph. And people wiring legal search into AI agent workflows, where an agent needs to check real precedent mid-conversation instead of guessing.

One caution worth stating plainly: this is a research tool that returns what Google Scholar indexes. It is not legal advice, and anything you plan to file should be verified against the official reporter.

The manual way, and where it breaks

The DIY version is a script that requests scholar.google.com/scholar?as_sdt=2006&q=... and parses the result cards. It works for about ten queries. Scholar is one of the most aggressively bot-protected properties Google runs, so you hit captchas fast, and from a datacenter IP you often get blocked on request one. The court filter is a numeric code system buried in the UI that you have to reverse-engineer, pagination tops out quietly, and the markup shifts under your selectors. I got a prototype working in an afternoon and then spent far longer keeping it alive than building it.

The faster way: run the Google Scholar Case Law API

The hosted route is a documented JSON input and a documented JSON output.

Apify Console

Open the Google Scholar Case Law API and click Try for free.
Enter a query, and optionally courts, yearFrom, and yearTo.
Run it and download the dataset as JSON or CSV.

REST

curl -X POST "https://api.apify.com/v2/acts/johnvc~google-scholar-case-law/runs?token=YOUR_APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{ "query": "qualified immunity", "courts": ["158"], "yearFrom": 2015, "maxResults": 20 }'

Court code 158 is the Supreme Court; leave courts empty to search every US state and federal court. Endpoint details live in the Apify API docs.

Search case law in Python

from apify_client import ApifyClient

client = ApifyClient("YOUR_APIFY_TOKEN")

run = client.actor("johnvc/google-scholar-case-law").call(
    run_input={
        "query": "Chevron deference",
        "courts": ["158"],
        "yearFrom": 2010,
        "maxResults": 20,
    }
)

for case in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(case.get("title"), case.get("court"), case.get("result_id"))

Feed the result_id values back in a second run through caseIds when you want full detail on specific opinions.

Scope a search to one circuit

The task Find Ninth Circuit opinions on qualified immunity shows court scoping in practice: one court code turns a national search into a jurisdiction-specific corpus.

Trace a doctrine through the Supreme Court

Find Supreme Court cases citing Chevron deference pairs a doctrine query with the SCOTUS court code, which is the starting point for tracking how a standard evolved.

Export results to a spreadsheet

Export case law search results to CSV is the no-code path: run the search, download the dataset as CSV, hand it to whoever works in Excel.

Start with the free credit

Search US case law with a free API shows the cheapest possible run. New Apify accounts include platform credit, and a search-only run bills half a cent per result, so first experiments usually cost nothing out of pocket.

Search US precedent in Chinese

A pair of tasks serve Chinese-speaking legal researchers who need US precedent. The first, US trade secret case search, covers trade secret disputes.

Pull US patent infringement precedents

The companion task US patent infringement case law does the same for patent litigation, using the language input to localize the interface.

Put case law inside AI agent workflows

Apify exposes the Actor over the Model Context Protocol, so Claude, Claude Code, and Cursor can call it as a tool. An agent can search a doctrine, pull the full detail for the top hit, and cite real opinions instead of hallucinating them, which is the failure mode that gets people sanctioned. The task Do legal research in Claude with a case law MCP has the setup, and you can read more about Claude at claude.ai.

FAQ about scraping Google Scholar case law

How much does the case law scraper cost to run?

Billing is per event: half a cent per search result and one cent per full case detail, plus a fraction of a cent when a run starts. A 20-result search-only run is about a dime. Full details are off by default so a cheap search stays cheap, and Apify's free credit covers early usage.

How do I run the case law scraper from Python?

Use apify-client as shown above: pass a query with optional courts and year bounds, then iterate the dataset. For citation-graph work, collect result_id values and run them back through caseIds.

Can an AI agent use this case law scraper through MCP?

Yes. Connected over MCP, the Actor shows up as a callable tool in Claude, Claude Code, or Cursor, so agent workflows can ground their legal answers in opinions that actually exist.

Can I schedule the scraper to watch for new opinions?

Yes. Save your query as a task, attach an Apify schedule, and enable sortByDate so the newest decisions surface first. Repeated runs against a watchlist query is how the monitoring use case works; start from the Google Scholar Case Law API.

What are the honest limits of a case law scraper?

It returns what Google Scholar indexes, nothing more. You get citations and cited-by counts, but not editorial treatment signals, so it will not tell you a case was overruled the way a paid citator does. Coverage of very recent or very obscure opinions depends on Scholar's own indexing. Treat the output as research input, not legal advice.

More from Truffle Pig Data

Related Actors from the same shop, all returning structured JSON: the Google Scholar API for academic papers and citations, the Google Scholar Lite API for lighter search runs, and the Google Patents API for the patent side of IP research.

Wrapping up

Google Scholar's case law collection is the best free corpus of US opinions, and now it behaves like an API. Run the Google Scholar Case Law API and start with a single query and a court code.

Google Scholar API for Bulk Paper Search: $1.50 per 1,000 Papers (2026)

Truffle Pig Data — Mon, 03 Aug 2026 17:09:01 +0000

Every literature review starts the same way: a search on Google Scholar, then hours of copying titles, authors, years, and citation counts into a spreadsheet. Scholar has no API, blocks scrapers with enthusiasm, and caps how fast a human can click. The Google Scholar Lite API on Apify is my answer for the bulk half of that problem: send a list of queries, get clean paper records back at $1.50 per 1,000 papers.

Disclosure: the Apify links in this post are affiliate links. If you run the Actor, I may earn a referral commission at no extra cost to you.

Does Google Scholar have an API?

No, and it never has. There is no official endpoint, no key to request, and the site is notoriously quick to throw captchas at anything that looks automated. For one-off searches that is a nuisance; for a systematic review with forty queries, it is a wall. So a Google Scholar API in practice means a scraper-style service you call like an API: queries in, structured paper metadata out.

What the Google Scholar Lite API returns

The Google Scholar Lite API returns one record per paper as structured JSON: title, authors, journal, publication year, citation count, result snippet, and links to the paper and its PDF or HTML full text.

Field	Example	Notes
title	"Attention Is All You Need"	Paper title
authors	Vaswani et al.	Author list
journal	NeurIPS	Venue or journal
year	`2017`	Publication year
citation count	`140000+`	The bibliometric headline number
links	paper page, PDF or HTML	Full-text link when one is available

Each record also carries a stable identifier and the originating query, which matters when you run dozens of searches in one batch.

Who this is for

Researchers and grad students building literature-review shortlists across many queries, bibliometrics folks assembling citation datasets, and developers giving a research agent a live paper-search tool instead of a stale index.

The manual way, and where it breaks

Scraping Scholar directly is a rite of passage that ends the same way for everyone: the first page works, the fifth page gets a captcha, and by page twenty your IP is on a timeout. Add parsing citation counts out of HTML and stitching pagination, and the collection script becomes the project. I wrote one in grad-school style years ago; the captchas won.

The faster way: run the Google Scholar Lite API

Apify Console

Open the Google Scholar Lite API and click Try for free.
Add your searchTerms, optionally a yearFrom and yearTo.
Run it and download the dataset as JSON, CSV, or Excel.

REST

curl -X POST "https://api.apify.com/v2/acts/johnvc~google-scholar-lite-api/runs?token=YOUR_APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{ "searchTerms": ["transformer attention mechanism"], "yearFrom": 2020, "maxResultsPerSearch": 50 }'

Run endpoint reference: the Apify API docs.

Bulk paper search in Python

import json
from apify_client import ApifyClient

client = ApifyClient("YOUR_APIFY_TOKEN")

run = client.actor("johnvc/google-scholar-lite-api").call(
    run_input={
        "searchTerms": ["transformer attention mechanism", "CRISPR gene editing"],
        "yearFrom": 2020,
        "maxResultsPerSearch": 50,
    }
)

for paper in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(json.dumps(paper, indent=2)[:300])
    print("---")

Each query is searched independently, so the records arrive pre-grouped for per-topic shortlists, with the citation count and full-text link in every row.

Export Scholar results to CSV in bulk

Most reviews live in spreadsheets, and the dataset export goes straight there. The task Export Google Scholar search results to CSV in bulk runs a batch and hands you the CSV.

Scrape Scholar for a PRISMA systematic review

Systematic reviews need reproducible, screening-ready records: fixed queries, year bounds, exported results. Scrape Google Scholar for a PRISMA systematic review is that protocol saved as a task.

It works in other languages too

The language input localizes results, and two Chinese-language tasks show it in practice: bulk export Google Scholar to CSV in Chinese and a literature review search by year range in Chinese.

Use it from Claude and other MCP clients

Research agents are the newest heavy users of Scholar data, and this Actor is MCP-ready: Claude, Claude Code, and Cursor can call it mid-conversation to ground an answer in actual papers. The task Bulk Google Scholar search for Claude agents via MCP has the setup, and you can read more about Claude and Claude Code at claude.ai.

FAQ about scraping Google Scholar

Does Google Scholar have an official API, or do I need a scraper?

There is no official API and no sign of one coming, so programmatic access means a scraper-style service. This one behaves like an API: JSON input, JSON output, no captchas on your side.

How much does the Google Scholar scraper cost?

Pay per paper, from $1.50 per 1,000 papers, with no setup or per-run fee. maxResultsPerSearch caps each query (default 100), so a forty-query review has a predictable ceiling, and new Apify accounts include free credit that covers a pilot run.

How is the Lite scraper different from the full Google Scholar API?

Lite does one thing at bulk prices: search results with core metadata (title, authors, venue, year, citation count, links). The full Google Scholar API adds the deeper Scholar surfaces: citation formats like BibTeX and APA, author profiles with h-index, per-article citation history, and co-author networks. Shortlists on Lite, deep dives on the full version.

Can the scraper support a PRISMA-style systematic review?

Yes. Fixed queries plus yearFrom and yearTo bounds give you a reproducible search protocol, and the exported records are ready for title-and-abstract screening. The PRISMA task above is the template.

Can Claude use this Scholar scraper over MCP?

Yes. Connected through the Apify MCP server, the Actor becomes a tool an agent can call, which is what people searching for a Google Scholar MCP are usually trying to build.

Can I schedule the scraper to track citations over time?

Yes. Save your query set as a task, attach an Apify schedule, and re-run monthly; comparing citation counts across runs gives you a citation-velocity view of a field. Start from the Google Scholar Lite API.

More from Truffle Pig Data

The academic shelf has three neighbors: the full Google Scholar API for citations, profiles, and networks, the Google Scholar Case Law API for court opinions, and the Google Patents API for the prior-art side of research.

Wrapping up

Literature reviews should start with data, not data entry. The Google Scholar Lite API turns a query list into a paper dataset for less than the coffee you would drink copying it by hand.

Reverse Image Search API: Trace Any Image Across the Web in 2026

Truffle Pig Data — Mon, 03 Aug 2026 12:10:08 +0000

Ask anyone who does image investigations and they will tell you the same quiet truth: Yandex Images is the strongest reverse image engine on the open web, especially for finding re-uses and earlier versions of a picture. The catch is that it only works one image at a time, in a browser. The Yandex Reverse Image Search API on Apify removes the catch: give it an image URL, get everything Yandex knows about that image as structured JSON.

Disclosure: the Apify links in this post are affiliate links. If you run the Actor, I may earn a referral commission at no extra cost to you.

Is there a reverse image search API?

Not from the engines themselves. The reverse image tools people actually rely on are browser interfaces, and Yandex has never shipped a public API for its image engine. Cloud vision products will label an image for you, but "what is in this picture" is a different question from "where does this picture appear online," and the second one is what usage tracking, fact-checking, and brand protection need. So a reverse image search API in practice means a scraper you call like an API: image URL in, typed result rows out.

What the Yandex Reverse Image Search API returns

The Yandex Reverse Image Search API returns typed JSON rows for one image: matching pages, visually similar images, other sizes, product matches, descriptive tags, and an entity card when the subject is recognizable.

result_type	What you get	Notes
`matching_page`	page title, link, snippet	Every page where the image appears
`similar_image`	thumbnail and source link	Visually similar content
`image_size`	other resolutions	Large, medium, small versions
`shopping_result`	product match with price	E-commerce listings that match
`image_tag`	descriptive search terms	What Yandex thinks is in the image
`knowledge_graph`	entity card	For recognizable people, places, brands

Every row carries the query image reference, the source domain, and a search timestamp, and section toggles control which types a run returns.

Who this is for

Photographers and brands tracking unauthorized use of their images, fact-checkers and OSINT researchers tracing a photo back to its first appearance, and e-commerce teams hunting marketplace listings that look suspiciously like their products.

The manual way, and where it breaks

The manual loop is uploading an image to Yandex, scrolling the sections, and copying links into a sheet, then doing it again for the next 200 product photos. Automating that interface yourself means driving a JavaScript-heavy page through regional domains and parsing several visually distinct result sections. It is exactly the kind of scraper that works on Tuesday and not on Friday. I would rather write the analysis code than babysit the collection code.

The faster way: run the Yandex Reverse Image Search API

Apify Console

Open the Yandex Reverse Image Search API and click Try for free.
Paste a public image_url and pick the result sections you want.
Run it and download the dataset as JSON, CSV, or Excel.

REST

curl -X POST "https://api.apify.com/v2/acts/johnvc~yandex-reverse-image-search/runs?token=YOUR_APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{ "image_url": "https://substack-post-media.s3.amazonaws.com/public/images/edbfb2cd-ebcb-4527-bec7-5315c182278f_445x445.png", "max_results": 20 }'

Run endpoint reference: the Apify API docs. The task Reverse image search with Yandex, results as JSON is this call saved and runnable.

Reverse image search in Python

from apify_client import ApifyClient

client = ApifyClient("YOUR_APIFY_TOKEN")

run = client.actor("johnvc/yandex-reverse-image-search").call(
    run_input={
        "image_url": "https://substack-post-media.s3.amazonaws.com/public/images/edbfb2cd-ebcb-4527-bec7-5315c182278f_445x445.png",
        "include_matching_pages": True,
        "include_similar_images": True,
        "max_results": 50,
    }
)

for row in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(row["result_type"], "|", row.get("source_domain") or row)

Filter on result_type and each section becomes its own dataset: pages for usage tracking, similar images for dataset building, shopping rows for brand protection.

Find where an image appears online

The core move for usage tracking: matching pages with title, link, and context per appearance. The task Find where an image appears online with Yandex runs it in one click.

Trace an image to its original source

For provenance work, other sizes and earlier appearances are the trail. Trace an image to its original source with Yandex is tuned for exactly that fact-checking workflow.

Find visually similar images

Similarity search feeds ML datasets and content curation. Find visually similar images from an image URL returns the lookalikes with thumbnails and source links.

Find products from an image

Turn on shopping matches and the Actor becomes a counterfeit detector: listings that match your product photo, with prices. Find products from an image with Yandex shows it working.

Use it from Claude and other MCP clients

Over MCP, Claude, Claude Code, and Cursor can run reverse image searches as a tool call: paste an image URL into a conversation and ask "where else does this appear online." The task Reverse image search in Claude via MCP has the config, and you can read more about Claude and Claude Code at claude.ai.

FAQ about scraping reverse image results

Is there a free reverse image search scraper?

This one bills per result row, and max_results puts a hard cap on a run before it starts, so a 20-row test run costs very little. New Apify accounts include free platform credit, which covers your first investigations.

Can an AI like Grok or Claude do reverse image search, or do I need a scraper?

Chat AIs can describe an image, but describing is not indexing; they cannot tell you which pages host that exact file. The scraper queries a real reverse-image index, and over MCP an AI can call it, which gets you both halves: retrieval from Yandex, reasoning from the model.

How does the scraper help with source hunting?

Source hunting is working backward from a compressed, cropped copy to the original. The other-sizes rows surface higher-resolution versions, and matching pages show every appearance with context, so you can follow the trail to the earliest, cleanest instance.

Does the scraper work in Claude via MCP?

Yes, and there is a saved task demonstrating it linked above. The Actor shows up as a callable tool once the Apify MCP server is connected.

Can I schedule the scraper to monitor image use?

Yes. Save a task per image set, attach an Apify schedule, and diff the matching-page domains between runs to catch new unauthorized uses early. Start from the Yandex Reverse Image Search API.

What are the limits of this reverse image scraper?

The image must be publicly reachable over http(s), since Yandex fetches it; localhost and private links will not work. Results reflect Yandex's index and can differ across the six regional domains. And its face-matching strength deserves the obvious caveat: use it responsibly and lawfully.

More from Truffle Pig Data

Two related Actors from the same shelf: the Yandex Search Scraper for Yandex's text SERPs, and the Google Images API for keyword-driven image search on the Google side.

Wrapping up

Reverse image search was the last search vertical without an API. The Yandex Reverse Image Search API fixes that with typed JSON rows you can filter, schedule, and build on.

Google News API at Scale: Monitor 100 Keywords for Headlines in 2026

Truffle Pig Data — Sun, 02 Aug 2026 17:09:43 +0000

News monitoring gets expensive fast when every article costs a full-fat scrape. Most monitoring jobs do not need full articles; they need headlines, sources, and links for a hundred keywords, every morning. Google News already aggregates that, so I built the Google News Lite API on Apify to read it cheaply: send search terms, get one JSON row per matching article.

Disclosure: the Apify links in this post are affiliate links. If you run the Actor, I may earn a referral commission at no extra cost to you.

Does Google News have an API?

No. Google retired its official News API years ago and never replaced it, so the question "how do I get a Google News API" keeps getting asked and keeps having the same answer. RSS feeds exist, but they were built for feed readers, not for monitoring: filtering is limited and running a hundred keyword feeds gets unwieldy. The working answer is a scraper you call like an API: search terms in, structured headline rows out.

What the Google News Lite API returns

The Google News Lite API returns one row per article as structured JSON: title, link, source, snippet, date, and image, plus the rank, the matching query term, and your targeting metadata.

Field	Example	Notes
`title`	"Apple Vision Pro gets enterprise push"	The headline
`link`	article URL	Direct to the publisher
`source`	"The Verge"	Publisher name
`snippet`	short excerpt	Enough for triage and sentiment
`date`	"3 hours ago"	Relative publication date
`image`	thumbnail URL	When available

Rows are de-duplicated by URL per query, and each run collects up to about 100 articles per search term.

Who this is for

PR and comms teams watching brand and executive coverage, analysts tracking markets or technologies for breaking developments, and developers feeding fresh headlines into agent workflows, alerting, or NLP pipelines.

The manual way, and where it breaks

Scraping Google News directly starts with consent pages and ends with parser archaeology. The markup rotates, dates arrive as relative strings in whatever language the region serves, and the same story appears five times under five URLs. By the time you have handled localization, de-duplication, and blocking, your "quick headline fetch" has a backlog. I keep a graveyard folder of half-finished news parsers; it is not small.

The faster way: run the Google News Lite API

Apify Console

Open the Google News Lite API and click Try for free.
Add your searchTerms and pick a timeRange like day.
Run it and download the dataset as JSON, CSV, or Excel.

REST

curl -X POST "https://api.apify.com/v2/acts/johnvc~google-news-lite-api/runs?token=YOUR_APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{ "searchTerms": ["Apple Vision Pro", "OpenAI"], "timeRange": "day", "maxResultsPerSearch": 25 }'

Run endpoint reference: the Apify API docs.

Monitor headlines in Python

from apify_client import ApifyClient

client = ApifyClient("YOUR_APIFY_TOKEN")

run = client.actor("johnvc/google-news-lite-api").call(
    run_input={
        "searchTerms": ["Apple Vision Pro", "OpenAI"],
        "timeRange": "day",
        "maxResultsPerSearch": 25,
    }
)

for article in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(article["title"], "|", article["source"], "|", article["link"])

Each term is searched separately, so the output is already grouped for per-keyword reporting.

Build a daily news digest feed

The digest is the classic setup: a fixed term list, timeRange set to day, one run every morning. The task Build a daily news digest feed by API is that configuration ready to clone.

Bulk monitor 100 keywords at once

Per-article billing is what makes wide keyword nets affordable, since a quiet keyword costs almost nothing. Bulk monitor news for 100 keywords at once shows the wide-net pattern.

It works in other languages too

Country and language are input fields, not an afterthought. Two Chinese-language tasks prove the point: a daily Chinese news digest and bulk brand monitoring in Chinese. Swap country and language and the same recipes run for any market.

Use it from Claude and other MCP clients

Because the Actor is MCP-ready, Claude, Claude Code, and Cursor can pull fresh headlines mid-conversation: "what happened around our brand in the last 24 hours" becomes a tool call with real citations. The task Feed Google News headlines to Claude via MCP has the setup, and you can read more about Claude and Claude Code at claude.ai.

FAQ about scraping Google News

Is this Google News scraper free?

It bills per article returned, with no setup fee, and maxResultsPerSearch caps each term at up to 100 articles so spend is bounded before the run starts. New Apify accounts include free platform credit, which covers a lot of headlines.

How is the Lite scraper different from the full Google News API?

Lite is the cheap, headline-level variant: title, link, source, snippet, date, and image, capped at 100 articles per term, with country and language targeting. The full Google News API adds location-based searches, safe search, and pagination control. If you need those, use the full version; if you are monitoring keywords at volume, Lite is the better price.

Does the scraper return full article text?

No, and that is a deliberate trade. You get the headline, snippet, source, and a direct link; if a story matters, fetch the article from the link. Keeping rows headline-level is what keeps 100-keyword monitoring cheap.

What does a media monitoring scraper actually collect?

Coverage over time: who wrote about a term, when, where, and with what framing in the headline and snippet. Stack daily runs and you have share-of-voice, source breakdowns, and a timeline of how a story developed.

Can Claude call the news scraper through MCP?

Yes. Connect the Apify MCP server and the Actor appears as a callable tool, so agents and chat sessions can ground themselves in the day's coverage instead of last year's training data.

Can I schedule the scraper for an automatic daily digest?

Yes, that is its natural habitat. Save your term list as a task, attach an Apify schedule (a cron line like 0 7 * * *), and read the fresh dataset each morning. Start from the Google News Lite API.

More from Truffle Pig Data

Related Actors for adjacent feeds: the full Google News API when you need deeper search control, the Google Finance API for the market-data side of a monitoring stack, and the Google Forums Search API for what discussion boards say before the press does.

Wrapping up

Headline monitoring should be boring and cheap, not a scraping project. The Google News Lite API keeps it that way: search terms in, one row per article out, billed per article.

Google Ads Transparency API: See Every Ad a Competitor Runs (2026 Guide)

Truffle Pig Data — Sun, 02 Aug 2026 12:10:28 +0000

Your competitor's entire Google ad history is public. The Google Ads Transparency Center lists every ad an advertiser runs: format, when it first appeared, whether it is still live. What it does not offer is any way to get that list out; you scroll and squint. The Google Ads Transparency API on Apify closes the loop: give it an advertiser ID, get the ad library back as structured JSON.

Disclosure: the Apify links in this post are affiliate links. If you run the Actor, I may earn a referral commission at no extra cost to you.

Does the Ads Transparency Center have an API?

No. Google built the Transparency Center as a browsing experience, and there is no official endpoint, no export button, and no bulk access. For a growth team that wants to diff a competitor's creatives month over month, that is a dead end. So a Google Ads Transparency API in practice means a scraper you call like an API: advertiser ID in, one JSON row per ad out.

What the Google Ads Transparency API returns

The Google Ads Transparency API returns every ad in an advertiser's public library as structured JSON: advertiser identity, creative identifier, ad format, first and last shown dates, total days running, and a preview link.

Field	Example	Notes
advertiser	name and ID	Whose library you pulled
creative ID	stable identifier	For diffing across runs
format	text, image, or video	Creative type
first shown	`2025-11-02`	When the ad first appeared
last shown	`2026-07-12`	Most recent sighting
days running	`252`	Longevity, the cheap proxy for performance

Long-running creatives are the interesting ones: nobody pays to run a loser for eight months.

Who this is for

Growth marketers who want a competitor ad library they can query, agencies auditing a prospect's creative mix before a pitch, researchers tracking political advertisers, and builders wiring competitive ad data into AI marketing agents.

The manual way, and where it breaks

The manual approach is opening the Transparency Center, scrolling an infinite list, and screenshotting ads into a slide deck. It produces an artifact you cannot query, sort, or diff, and it is stale the day you make it. Automating the browsing yourself runs into the usual wall: a JavaScript-rendered interface with pagination behavior and markup that shift under you. I did the screenshot-deck version once for a client review; never again.

The faster way: run the Google Ads Transparency API

Apify Console

Open the Google Ads Transparency API and click Try for free.
Paste an advertiserId, the AR... code from any Transparency Center advertiser URL.
Run it and download the dataset as JSON, CSV, or Excel.

REST

curl -X POST "https://api.apify.com/v2/acts/johnvc~google-ads-transparency-api/runs?token=YOUR_APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{ "advertiserId": "AR01614014350098432001", "maxResultsPerAdvertiser": 50 }'

Run endpoint reference: the Apify API docs.

Pull an ad library in Python

import json
from apify_client import ApifyClient

client = ApifyClient("YOUR_APIFY_TOKEN")

run = client.actor("johnvc/google-ads-transparency-api").call(
    run_input={
        "advertiserIds": ["AR01614014350098432001"],
        "maxResultsPerAdvertiser": 100,
    }
)

for ad in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(json.dumps(ad, indent=2)[:300])
    print("---")

Each row is one creative with its format, run dates, and preview link, which is everything a diff or a dashboard needs.

Spy on competitor Google ads

The starter move: pull a rival's full library and see what they are actually running, not what they say they are running. The task Spy on competitor Google ads is that pull, saved.

Audit an advertiser's creative mix

Formats tell a strategy story: an advertiser shifting from text to video is changing their funnel. Audit advertiser Google ad creatives breaks a library down by format.

Track how long competitor ads run

Days-running is the closest thing to public performance data. Track competitor ad duration surfaces the long-lived creatives worth studying.

Watch political advertisers

The same mechanics work for accountability research. Political advertiser Google ads pulls a political advertiser's public library into analyzable rows.

Export an ad library to CSV

For the spreadsheet crowd, Export advertiser Google ads to CSV runs the pull and hands you a CSV, no code involved.

Use it from Claude and other MCP clients

The Actor is MCP-ready, so Claude, Claude Code, and Cursor can call it as a tool: "pull the last 100 ads from these two competitors and tell me what changed since last month" becomes a prompt instead of a project. You can read more about Claude and Claude Code at claude.ai.

FAQ about scraping the Ads Transparency Center

Is there a free Google Ads Transparency scraper?

This one bills per ad returned, and maxResultsPerAdvertiser (default 50, up to 500) caps the count before a run starts. New Apify accounts include free platform credit, so a first competitor pull usually costs nothing out of pocket.

How does the scraper find an advertiser's ID?

Open the advertiser in the Transparency Center and copy the AR... code from the URL; that code is the input. You can pass one ID or a list, and the Actor de-duplicates them for you.

Can the scraper show ad spend or targeting data?

No, and be suspicious of anything claiming it can. The public record carries format, run dates, days running, and previews; spend and targeting are not published there. Days running is the honest proxy most teams use.

Does the scraper work in Claude via MCP?

Yes. Connect the Apify MCP server and the Actor appears as a callable tool in Claude, Claude Code, or Cursor, which is how the AI-marketing-agent use case runs.

Can I schedule the scraper to watch competitors' ad libraries?

Yes, snapshots are the whole game: run the same advertiser list weekly, diff the creative IDs, and you get alerts on new launches and quiet retirements. Start from the Google Ads Transparency API.

More from Truffle Pig Data

Adjacent lenses on the same competitive questions: the Google Local Services API for the Local Services Ads side of Google advertising, and the Google AI Overview API for how Google's AI answers describe your market.

Wrapping up

The Transparency Center already publishes the data; it just never offered a way to use it. The Google Ads Transparency API turns any advertiser's library into JSON you can diff, chart, and act on.

YouTube Shorts API: See What Ranks in Google Short Videos in 2026 (Shorts, Reels, TikTok)

Truffle Pig Data — Sat, 01 Aug 2026 17:08:53 +0000

Short-form video has its own SERP now. Search a topic on your phone and Google shows a dedicated short-videos block mixing YouTube Shorts, TikTok, and Instagram Reels, ranked. That block is the scoreboard for short-form content, and no official API returns it. The Google Short Videos API on Apify scrapes it for you: query in, flat rows of ranked short videos out.

Disclosure: the Apify links in this post are affiliate links. If you run the Actor, I may earn a referral commission at no extra cost to you.

Is there a YouTube Shorts API?

Sort of, and that is the problem. The official YouTube Data API is real and works for YouTube metadata, but it has no view of what ranks in Google's short-videos block, and it knows nothing about TikTok or Reels. Nobody ships an official cross-platform endpoint for "which short videos does Google rank for this topic." So a YouTube Shorts API in the ranking sense means a scraper you call like an API: send a query, get the ranked block back as JSON.

What the Google Short Videos API returns

The Google Short Videos API returns each short-form result as a structured row: rank, title, link, preview image, source platform, creator or channel, and clip duration, plus related-search suggestions on mobile and tablet.

Field	Example	Notes
rank	`3`	Position in the short-videos block
title	"10 minute no-equipment workout"	As Google shows it
link	video URL	Direct to the clip
source	YouTube, TikTok, Facebook	Platform tag per result
channel	creator name	Who posted it
duration	`0:42`	Clip length

On mobile or tablet device profiles you also get the "people also search for" suggestions, which is free keyword research.

Who this is for

Content strategists deciding what to film next, social teams tracking which platform wins a topic, and SEO folks mining the related-search suggestions for short-form keyword ideas.

The manual way, and where it breaks

Scraping this block yourself means emulating a mobile device, because the desktop page often does not even show it. So you are spoofing user agents, rendering JavaScript, and parsing a carousel whose markup was not designed to be read. Every result looks slightly different depending on the source platform, and the block's layout changes more often than regular results. I would rather maintain almost anything else.

The faster way: run the Google Short Videos API

Apify Console

Open the Google Short Videos API and click Try for free.
Enter a search query in q; keep device on mobile to get related searches.
Run it and download the dataset as JSON, CSV, or Excel.

REST

curl -X POST "https://api.apify.com/v2/acts/johnvc~google-short-videos-api/runs?token=YOUR_APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{ "q": "air fryer recipes", "device": "mobile", "max_pages": 1 }'

Run endpoint reference: the Apify API docs.

Get ranked short videos in Python

import json
from apify_client import ApifyClient

client = ApifyClient("YOUR_APIFY_TOKEN")

run = client.actor("johnvc/google-short-videos-api").call(
    run_input={"q": "workout tips", "device": "mobile", "max_pages": 1}
)

items = client.dataset(run["defaultDatasetId"]).list_items().items
print(f"{len(items)} results")
print(json.dumps(items[0], indent=2))

The first row shows the shape: rank, title, link, source platform, channel, and duration, ready for a dataframe.

Search across platforms in one call

One query returns YouTube Shorts, TikTok, and Reels results side by side, each tagged with its source, so platform comparison is a groupby instead of three scrapers. The task Cross-platform short video search API is that setup saved.

Scrape YouTube Shorts and Reels by keyword

For keyword-first research, two tasks cover the recipe: Scrape YouTube Shorts and Reels by keyword and Search YouTube Shorts by keyword via API. Point either at your topic list and export.

See which TikToks rank on Google

TikTok's own analytics will not tell you what Google ranks. See which TikToks rank on Google for air fryer recipes filters the block down to one platform for one niche, a pattern that transfers to any topic.

Use it from Claude and other MCP clients

Over MCP, Claude, Claude Code, and Cursor can run this research mid-conversation: "what short videos rank for 'meal prep' and which platform dominates" becomes one prompt. The task Do short video SERP research in Claude via MCP has the setup, and you can read more about Claude and Claude Code at claude.ai.

FAQ about scraping short video results

Is there a free YouTube Shorts scraper?

Billing here is per page fetched, with about 10 short videos per page and max_pages capping the spend before a run starts. New Apify accounts include free platform credit, so early research runs typically cost nothing out of pocket.

Does the scraper cover TikTok and Instagram Reels too?

Yes. The block itself is cross-platform, and every row carries a source tag, so you can compare how YouTube, TikTok, and Facebook or Instagram content splits the ranking for a topic.

Can the scraper get YouTube Shorts transcripts?

No, this one returns ranking metadata: titles, links, sources, channels, durations. For the words inside the videos, pair it with the YouTube Transcripts API, which takes the video links this scraper returns.

Can I use this scraper for AI-generated Shorts research?

It will not make videos for you, but it answers the question that should come first: what already ranks for your topic, in what format, at what duration. Plenty of AI-shorts pipelines use exactly this data to pick topics.

Does the scraper work in Claude via MCP?

Yes. Connect the Apify MCP server and it becomes a callable tool in Claude, Claude Code, or Cursor, which is what the SERP-research task above demonstrates.

Can I schedule the scraper to track short video trends?

Yes. Save your queries as a task, attach an Apify schedule, and the dataset accumulates a ranking history per topic, which is how you catch a format trend early. Start from the Google Short Videos API.

More from Truffle Pig Data

Two neighbors in the video and search family: the YouTube Transcripts API for what the videos actually say, and the Google Images API for the image side of the SERP.

Wrapping up

Short-form video rankings are public; they were just never queryable. The Google Short Videos API makes them a JSON feed you can build on.

Google Travel API: Get Destination Ideas with Flight Prices as JSON in 2026

Truffle Pig Data — Sat, 01 Aug 2026 12:10:20 +0000

Most flight tools answer "how much to fly from A to B?" The more interesting question is "where can I even go?", and Google Travel Explore answers it beautifully: a map of destinations from your airport with flight prices, hotel prices, and suggested dates. Beautifully, but only in a browser. The Google Travel Explore API on Apify turns that surface into structured JSON: one ranked destination list per departure airport.

Disclosure: the Apify links in this post are affiliate links. If you run the Actor, I may earn a referral commission at no extra cost to you.

Is there a Google Travel API?

No. Google retired its public flights API years ago and never shipped anything for the Explore surface, so there is no official endpoint that returns "destinations reachable from JFK with prices and dates." A Google Travel API today means a scraper you consume like an API: airport code in, ranked destination ideas out.

What the Google Travel Explore API returns

The Google Travel Explore API returns ranked destination ideas as structured JSON: destination name and country, destination airport with GPS coordinates, estimated round-trip flight price, estimated nightly hotel price, suggested trip dates, flight duration, and stops.

Field	Example	Notes
destination	"Lisbon, Portugal"	Name plus country and destination airport
flight price	`412`	Estimated round trip from your airport
hotel price	`98`	Estimated nightly rate
trip dates	suggested start and end	Date-bound, not vague
duration and stops	7h 45m, nonstop	Flight time and connection count
link	Google Travel URL	Plus a thumbnail and airline info

One run can batch several departure airports, and each destination row is self-contained enough to publish or feed to an agent as-is.

Who this is for

Travel content sites building "where can I go on a budget" features, agencies pitching clients ranked ideas instead of one-off quotes, newsletter writers hunting deals from specific hubs, and developers giving an AI travel agent real prices and dates to reason over.

The manual way, and where it breaks

Explore is an interactive map. You pan, hover, and read cards one at a time, and there is no stable URL that captures the result set you are looking at. Automating it means driving a JavaScript-heavy map interface in a headless browser and untangling how the destination data loads behind it, and every layout tweak resets your work. I tried to eyeball-copy twenty destinations into a sheet once and gave up at nine.

The faster way: run the Google Travel Explore API

Apify Console

Open the Google Travel Explore API and click Try for free.
Set departureId to an airport code like ATL, or batch several in departureIds.
Run it and download the dataset as JSON, CSV, or Excel.

REST

curl -X POST "https://api.apify.com/v2/acts/johnvc~google-travel-explore-api/runs?token=YOUR_APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{ "departureId": "ATL", "maxResultsPerDeparture": 25 }'

Run endpoint reference: the Apify API docs.

Get destination ideas in Python

import json
from apify_client import ApifyClient

client = ApifyClient("YOUR_APIFY_TOKEN")

run = client.actor("johnvc/google-travel-explore-api").call(
    run_input={
        "departureIds": ["JFK", "SFO"],
        "gl": "us",
        "maxResultsPerDeparture": 25,
    }
)

for destination in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(json.dumps(destination, indent=2)[:300])
    print("---")

Each row carries the destination, the estimated flight and hotel prices, the suggested dates, and the duration, which is everything a "where should I go" feature needs.

Cheapest places to fly from 20 major cities

The classic use case ships as a ready-made series: one saved task per departure city, twenty in all. A few to start from: Atlanta, New York, Los Angeles, Chicago, Seattle, and Toronto. Each one runs the Explore lookup for that hub and returns the ranked, priced list; the rest of the series lives on the Actor's examples tab.

Export travel destinations to CSV

For editorial workflows the spreadsheet is the product. Export travel destinations to CSV runs a lookup and hands you a CSV ready for sorting, filtering, and publishing.

Use it from Claude and other MCP clients

The Actor is MCP-ready, so Claude, Claude Code, and Cursor can call it live: ask "where can I fly from JFK for under $400 with a hotel under $120 a night" and the agent filters real rows instead of hallucinating fares. You can read more about Claude and Claude Code at claude.ai.

FAQ about scraping Google Travel Explore

Is there a free Google Travel scraper?

The Actor bills per destination returned, and maxResultsPerDeparture (default 50, max 200) caps the count before a run starts, so cost is predictable. New Apify accounts include free platform credit, which easily covers a first exploration.

How does the scraper choose which destinations to return?

It returns the ranked list Google Travel Explore shows for your departure airport, localized by your gl and hl settings. You are getting Google's ordering, not a re-ranking.

Are the scraper's prices exact bookable fares?

No, and it is worth being clear about this: flight and hotel figures are Google's estimates for the suggested dates, built for discovery rather than checkout. For live itineraries and bookable prices on a specific route, pair it with a flights-level search.

Can an AI travel agent use this scraper?

Yes, it was built with that in mind. Over MCP the Actor becomes a tool an agent can call, and each destination row arrives with price, dates, and duration attached, so the agent can filter and recommend without extra lookups.

Can I schedule the scraper to track deals from my airport?

Yes. Save your departure airport as a task, attach an Apify schedule, and each run appends a fresh ranked list, which over time becomes a record of how prices and reachable destinations shift. Start from the Google Travel Explore API.

More from Truffle Pig Data

Explore answers "where"; two sibling Actors answer the follow-ups. The Google Flights API prices a specific route with real itineraries, and the Google Hotels Search Scraper covers where you sleep when you get there.

Wrapping up

Destination discovery data used to be locked inside an interactive map. The Google Travel Explore API returns it as ranked, priced JSON from any departure airport you care about.