DEV Community: Apify

Investor prospecting lists from PitchBook and LinkedIn data

Crawler Bros — Thu, 17 Sep 2026 05:21:25 +0000

Investor prospecting lists from PitchBook and LinkedIn data

Fundraising research usually means either paying for a PitchBook or Crunchbase seat the founder only needs for a few months, or manually building a spreadsheet from investor websites and LinkedIn one profile at a time. Neither is a great use of the limited time founders have during an active raise.

Investor prospecting with public data means joining public investor profile data with public LinkedIn partner profiles into one targeted, sector- and stage-matched list, instead of paying for a full research-platform seat or building the list one profile at a time by hand.

Quick answer

Use PitchBook Investors Scraper to collect public investor profiles — firm name, focus areas, and stated investment criteria — without a PitchBook subscription. Use LinkedIn Profile Scraper to enrich individual partner or associate profiles with role, background, and current firm. Combine both into a prospecting list segmented by sector and stage fit.

The investor prospecting stack

The stack uses two CrawlerBros Actors:

PitchBook Investors Scraper for public investor profiles without a subscription.
LinkedIn Profile Scraper for public LinkedIn profile information without login.

The workflow:

\Sector and stage criteria -> PitchBook Investors Scraper (firm-level investor profiles) -> LinkedIn Profile Scraper (partner-level enrichment) -> segmented investor prospecting list \\

Key facts

Firm-level data answers "who invests in this"; partner-level data answers "who do I email." Both are needed for outreach that isn't generic.
Start from known profile URLs, not broad category browsing. Both Actors work best against specific, already-identified profile pages.
Focus fields go stale. A firm's stated sector/stage focus can change; re-verify before outreach, not just before building the list.
Fit beats volume. A well-matched list of 20-50 firms consistently outperforms an unfiltered list of hundreds.

Actor configuration that matters

For PitchBook Investors Scraper, start from firm or investor profile URLs discovered through sector research rather than trying to enumerate an entire investor category in one run — public profile pages are the reliable input; broad category browsing is not.

\json { "profileUrls": [ "https://pitchbook.com/profiles/investor/example-1", "https://pitchbook.com/profiles/investor/example-2" ], "includeInvestmentCriteria": true } \\

For LinkedIn Profile Scraper, feed it the specific partner or associate profile URLs found through the PitchBook data or firm websites, since public profile scraping works best against known URLs rather than broad search.

What the output looks like

The PitchBook row needs firm name, focus sectors, stage focus, and stated check size when available. The LinkedIn row needs name, title, current firm, and background summary. Joining the two gives a prospecting row that answers both "does this firm invest in what I'm building" and "who specifically should I reach out to."

Use case 1: targeted seed and Series A outreach

A founder raising a specific round size in a specific sector can filter PitchBook profiles for matching stage and sector focus, then use LinkedIn enrichment to identify the right partner to reach rather than sending a cold email to a generic firm inbox.

Use case 2: warm-intro path mapping

Once a target investor list exists, cross-referencing partner LinkedIn backgrounds against a founder's own network (previous companies, schools, mutual connections) helps prioritize which firms are reachable through a warm introduction versus cold outreach.

Use case 3: competitive fundraising intelligence

Tracking which investors a competitor has raised from, where that's publicly known, helps a founder understand which firms are already active in the category and how to differentiate outreach messaging for firms likely to be comparing the two companies.

Production notes

Start from known profile URLs, not broad scraping. Both Actors work best against specific, already-identified profile pages rather than attempting to enumerate an entire category of investors in one pass.

Respect what's public. Only collect information visible on public profile pages. Avoid combining this with any data source that would require login credentials or private access.

Re-verify stage and sector focus before outreach. Investor focus areas change; a profile scraped months ago may not reflect a firm's current thesis.

Prioritize fit over volume. A list of 20 well-matched investors outperforms a list of 200 firms scraped without sector or stage filtering.

Firm-level data alone produces low response rates. A firm-level list without partner-level context means outreach goes to generic inboxes. Adding LinkedIn partner enrichment makes outreach targeted enough to get meaningfully better response rates.

Cost comparison

Approach	Cost	Data freshness	Partner-level detail
PitchBook/Crunchbase subscription	Expensive, ongoing	High	Yes
Manual research	Free, very slow	Depends on effort	Inconsistent
Apify PitchBook + LinkedIn pipeline	Pay per run	On demand	Yes

Check the current Pricing tab on each Actor page before running at scale.

FAQ

Can Apify replace a PitchBook subscription?

It can replace the workflow of researching public investor profiles for a specific list; it doesn't replicate PitchBook's full paid database and analytics.

Is scraping public LinkedIn profiles allowed?

LinkedIn Profile Scraper collects publicly visible profile information without login; always review current platform terms before large-scale collection.

How big should a first investor list be?

Most founders get better results from a well-matched list of 20-50 firms than an unfiltered list of hundreds.

What's the fastest way to try this?

Pick 10 investor firms whose public focus matches your sector and stage, then run PitchBook Investors Scraper and LinkedIn Profile Scraper against them.

Try it yourself

Pick 10 investor firms whose public focus matches your sector and stage. Run PitchBook Investors Scraper against their profile pages, then use LinkedIn Profile Scraper to find the right partner at each. A short, well-matched list beats a long, generic one.

Facebook Marketplace intelligence: resale pricing and comment sentiment

Crawler Bros — Wed, 16 Sep 2026 05:15:01 +0000

Facebook Marketplace intelligence: resale pricing and comment sentiment

Facebook Marketplace is one of the largest resale and local-commerce platforms, and one of the least monitored. Most price-intelligence tools skip it entirely because it's local, listing-heavy, and constantly turning over.

Facebook Marketplace intelligence means combining public listing data with public comment data to price resale inventory, spot unauthorized brand resale, and read how a community actually reacts to a local post — instead of relying on manual browsing or reaction-emoji counts alone.

Quick answer

Use Facebook Marketplace Scraper to collect listing prices, condition, and seller data for a product category or location. Use Facebook Comments Scraper to pull public reactions on posts, whether that's a Marketplace listing, a local business page post, or a community group thread. Combine both to price resale inventory and read local sentiment in the same workflow.

The Marketplace intelligence stack

The stack uses two CrawlerBros Actors:

Facebook Marketplace Scraper for listing data, pricing, and seller information.
Facebook Comments Scraper for public comments on posts and videos with metadata.

The workflow:

\Product category or location -> Facebook Marketplace Scraper (listing prices, condition, seller) -> Facebook Comments Scraper (reactions on flagged listings or posts) -> resale price and sentiment dataset \\

Key facts

Location beats keyword. Marketplace is a local marketplace by design — a keyword search without a location filter returns inconsistent results.
Condition explains price, not irrationality. Two identical-looking listings at different prices are usually explained by condition, not by one seller being unreasonable.
Listing velocity beats a single price. A category with fast-turning listings at a given price point signals real demand better than a handful of stale unsold listings.
"Public" has a specific meaning here. Marketplace listings and public post comments qualify; private group content doesn't.

Actor configuration that matters

For Facebook Marketplace Scraper, scope by category and location rather than a broad keyword search — Marketplace results are heavily location-weighted, so a keyword search without a location filter returns inconsistent results.

\json { "category": "furniture", "location": "Austin, TX", "maxItems": 150, "sortBy": "date_listed" } \\

For Facebook Comments Scraper, feed it the specific post or listing URLs you want reactions on rather than trying to crawl comments broadly — Facebook's comment threading and pagination behave differently across post types, so targeted URL lists produce more reliable results than open-ended searches.

What the output looks like

The Marketplace row needs title, price, condition, location, seller name, listing date, and listing URL. Price and condition together are what make resale comps useful — a "good condition" $150 listing and a "like new" $150 listing aren't the same data point.

The Comments row needs comment text, author, timestamp, and post URL, so a spike in complaints or praise can be traced back to the specific listing or post that triggered it.

Use case 1: resale price benchmarking

A reseller flipping furniture, electronics, or collectibles can pull active listings for a category and location, then price new inventory against what's actually moving rather than guessing from a handful of manually checked listings.

Use case 2: gray-market and unauthorized resale monitoring

A brand that sells through authorized retailers can search Marketplace for its product names to spot bulk resale, counterfeit listings, or unauthorized liquidation stock showing up at prices well below MSRP — a pattern that's hard to catch by browsing manually.

Use case 3: local business post sentiment

A local business posting about a new menu item, event, or policy change can pull comments on that post to see the actual reaction, rather than relying on the reaction-emoji count alone. Comment text often reveals why a post landed well or badly, which the emoji summary doesn't.

Production notes

Scope by location first. Marketplace is a local marketplace by design — location filtering matters more here than on almost any other platform in this batch.

Track condition alongside price. Two identical-looking listings at different prices are often explained by condition, not by one seller being irrational.

Watch listing velocity, not just price. A category with fast-turning listings at a given price point tells you more about real demand than a handful of stale listings sitting unsold.

Respect what "public" means here. Only collect what's visible without logging in or joining a private group. Marketplace listings and public post comments qualify; private group content does not.

Price without condition confuses more than it clarifies. Price without condition and listing age produces comps that look contradictory. Adding condition and days-listed as required fields fixes most of the confusion.

Cost comparison

Approach	Coverage	Local granularity	Sentiment data
Manual browsing	One category at a time	High, but slow	None
General resale-price tools	Limited or no Marketplace coverage	Low	None
Apify Marketplace + Comments pipeline	Category and location scoped	High	Yes

Check the current Pricing tab on each Actor page before running at scale.

FAQ

Can Apify scrape Facebook Marketplace listings?

Yes. Facebook Marketplace Scraper extracts public listing data including price, condition, and seller information.

Can I pull comments from any Facebook post?

Facebook Comments Scraper works on public posts and videos; private group or profile content isn't accessible.

Is this useful outside of reselling?

Yes — brand protection, local sentiment tracking, and community research all use the same two-Actor pattern.

What's the fastest way to try this?

Pick one product category and city, run Facebook Marketplace Scraper for current listings, then run Facebook Comments Scraper on the most-commented listings.

Try it yourself

Pick one product category and one city. Run Facebook Marketplace Scraper for current listings, then run Facebook Comments Scraper on the two or three most-commented listings you find. The price spread plus the comment tone will tell you more about that local market than either dataset alone.

Finding flight price anomalies with Skiplagged data on Apify

Crawler Bros — Tue, 15 Sep 2026 04:41:46 +0000

Finding flight price anomalies with Skiplagged data on Apify

Flight pricing is famously inconsistent — the same route can price differently depending on the exact combination of origin, destination, and connection chosen, sometimes by a wide margin for what's effectively the same trip. Skiplagged built a business around surfacing exactly these gaps by comparing direct fares against fares for longer itineraries that happen to pass through the same destination.

Flight price arbitrage detection means systematically comparing fares across itinerary structures for the same route and destination, instead of checking prices manually route by route. For travel content sites, deal-alert businesses, and corporate travel teams watching cost, that pricing behavior is worth monitoring on a schedule.

Quick answer

Use Skiplagged Scraper to collect flight search results for specific routes and date ranges, including price, airline, and itinerary structure. Run it across a route set on a schedule and flag results where a longer or connecting itinerary prices meaningfully below the direct fare for the same destination.

The flight arbitrage setup

The setup uses one CrawlerBros Actor:

Skiplagged Flight Search Scraper for flight search results with prices, airlines, and booking data.

The monitoring loop:

\Route and date list -> Skiplagged Flight Search Scraper -> price-by-itinerary dataset -> anomaly flagging (direct vs. connecting fare gaps) \\

Key facts

The gap is the signal, not the price. The interesting data point is the difference between itinerary types for the same destination, not any single fare in isolation.
A single-date search misses most gaps. Fare anomalies often appear on specific days of the week; search across a date range, not one fixed date.
Fares move constantly. A one-time snapshot only shows one moment in a moving market — schedule the run instead.
Some routes show gaps consistently; most don't. A shortlist of routes worth monitoring closely beats tracking every possible route equally.

Actor configuration that matters

Search by origin, destination, and date range rather than a single fixed date — fare anomalies often appear on specific days of the week or specific date combinations, and a single-date search will miss most of them.

\json { "origin": "JFK", "destination": "LAX", "departureDateRange": ["2026-11-01", "2026-11-30"], "maxResults": 100 } \\

Run the same route across a rolling date window on a schedule rather than a one-time pull — fare gaps shift as airline pricing algorithms react to booking activity, so a single snapshot only shows one moment in a constantly moving market.

What the output looks like

Each result row should carry origin, destination, departure date, price, airline, and itinerary type (direct vs. connecting, and whether the booked destination differs from the traveler's actual destination). The itinerary-type field is what turns a plain price list into an arbitrage-detection dataset — without it, a cheap connecting flight and a cheap direct flight look identical.

Use case 1: travel deal content

A travel deals site or newsletter can run a set of popular routes on a schedule and automatically flag the biggest fare gaps each week, turning a manual research task into a structured feed worth publishing.

Use case 2: corporate travel cost monitoring

A company managing frequent travel on a fixed set of routes can track whether cheaper itinerary structures are consistently available on routes employees fly often, informing internal travel policy without requiring manual research for every trip.

Use case 3: route-level pricing trend research

Tracking the same route over months shows whether fare volatility on that route is increasing or decreasing, which is useful context for anyone deciding when to book rather than just what to book.

Production notes

Track itinerary structure, not just price. The interesting signal here is the gap between itinerary types for the same destination, not the raw price of any single result.

Run on a rolling schedule. Fares change constantly; a static dataset goes stale within days.

Be explicit about what "cheaper" means to your audience. Booking a longer or connecting itinerary to save money involves trade-offs (no checked bags on the unused leg, itinerary risk) that are worth stating clearly in any published content built from this data.

Watch for route-specific patterns. Some routes show fare gaps consistently; others rarely do. Building a shortlist of routes worth monitoring closely is more efficient than tracking every possible route equally.

Price alone doesn't flag anomalies. Without itinerary-type context, "found a cheap flight" alerts don't distinguish between a genuinely unusual gap and an ordinary low-season fare. Itinerary-type as a required field makes anomaly detection meaningful.

Cost comparison

Approach	Coverage	Update frequency	Anomaly detection
Manual fare checking	One route at a time	Whenever you check	Manual
Flight price alert apps	Broad, but black-box logic	Automated	Limited transparency
Apify Skiplagged pipeline	Custom route set	Scheduled	Transparent, itinerary-level

Check the current Pricing tab on the Actor page before running at scale — cost depends on route count and date-range breadth.

FAQ

Can Apify track flight price anomalies?

Yes. Skiplagged Flight Search Scraper collects fare data across itinerary types for a given route and date range.

Is this only useful for hidden-city ticketing?

No — the underlying data is useful for any fare-comparison or travel-cost-monitoring use case, independent of any specific booking strategy.

How often should routes be re-scraped?

Daily or every few days for routes you're actively monitoring; fare gaps can appear and disappear within a short window.

What's the fastest way to try this?

Pick one popular route and a 30-day date window, run Skiplagged Flight Search Scraper across it, then sort by the direct-vs-connecting price gap.

Try it yourself

Pick one popular route and a 30-day date window. Run Skiplagged Flight Search Scraper across that window, then sort by the gap between direct and connecting itinerary pricing. The largest gaps are your first candidates worth digging into further.

Giving AI agents live European fuel prices with Apify MCP

Viktor Kondas — Mon, 14 Sep 2026 15:45:05 +0000

Giving AI agents live European fuel prices with Apify MCP

How I turned a CEE + DACH fuel-price Actor into a tool Cursor can call — and what I had to change once a model, not a spreadsheet, was reading the output.

I built European Fuel Station Prices API (CEE + DACH) (https://apify.com/kondasviktor/cee-dach-fuel-intelligence) as a normal Apify Actor first: normalized station prices across Central Europe and DACH, scheduled monitoring, and a route cheapest-stop mode. Then I wired the same Actor into Cursor through the Apify MCP server (https://docs.apify.com/platform/integrations/mcp). This piece is about that second step — Theme 2 in Apify's Content Program: your Actor as a tool for AI agents.

What you need

A free Apify (https://apify.com) account
Cursor (https://cursor.com) (or another MCP-capable client)
Willingness to let the agent call cloud Actors (each run bills PPE; Austria smoke tests land on the $0.10 minimum)
Optional: a Fuelo (https://fuelo.net/about/api\_key\_request) API key if you want HU/PL/CZ/SK/CH/BG coverage (BYOK — the Actor never uses platform keys)

No local scrape setup. The Actor already runs on Apify; MCP is how the agent reaches it.

The guessing problem

Ask any LLM-based agent "where should I fill up between Budapest and Vienna?" and you get a fluent answer built from training data that is months or years stale. Pump prices move daily. Logistics teams want live, per-station, cross-border numbers — not a national average from memory.

CEE adds a second failure mode. Slovakia can apply dual pricing for foreign-plated vehicles (dualPricingForForeignPlates: true, verified 2026-07-24 in my Actor's regulatory config). Croatia can run price caps on regular stations while highway stations differ (priceCapActive: true, same date). Austria restricts when prices may increase to Mon/Wed/Fri (priceIncreaseRestrictedDays: ["MO","WE","FR"]). None of that belongs in model weights. It belongs in structured fields an agent reads at call time.

That gap — volatile, jurisdiction-specific data an LLM cannot reliably know — is what MCP tools are for. The interesting engineering is not scraping a price board. It is designing output so an agent knows how much to trust each number and receives few enough tokens to reason.

Why I built a regional Actor in Cursor

I live in CEE. The Apify Store had US/FR/AU fuel Actors; station-level CEE + DACH coverage did not. I wanted one schema across DE, AT, CH, HU, PL, CZ, SK, RO, HR, SI, and BG — official feeds where they exist, licensed aggregators elsewhere, EUR-normalized for cross-border comparison.

The Actor is live on Store (build 0.1.10 at time of writing), pay-per-event (https://docs.apify.com/platform/actors/publishing/monetize/pay-per-event), with public example tasks. No buyer key for AT, SI, HR, RO. BYOK: Tankerkönig for DE, Fuelo for CH/HU/PL/CZ/SK/BG. You can smoke-test Austria in under a minute for the $0.10 minimum run charge.

I built the Actor in Cursor — the same client that now calls it via MCP. That closed loop mattered: every schema tweak got exercised by an agent that actually had to answer a logistics question, not by me staring at a dataset table.

Data sources are official public APIs or licensed aggregators with BYOK. The Actor does not scrape pump websites behind a login or ignore site terms; for Poland and similar Fuelo countries, missing a key fails loudly instead of inventing prices.

How the Actor is built

Each country is a small adapter behind one interface. Austria reads E-Control's public JSON API, maps DIE/SUP/GAS into canonical diesel/petrol95/cng, and tags every row dataConfidence: OFFICIAL_REALTIME with an honest sourceMessage that E-Control returns cheapest-N, not the full market:

Code (typescript)

// src/adapters/at.ts (excerpt)

url.searchParams.set('fuelType', mapFuel(query.fuel));

// ...

return {

stationId: s.id != null ? String(s.id) : null,

country: 'AT',

diesel: typeof die === 'number' ? die : null,

petrol95: typeof sup === 'number' ? sup : null,

dataConfidence: 'OFFICIAL_REALTIME',

source: 'E-Control Spritpreisrechner',

sourceMessage:

'E-Control returns cheapest-N stations for the search — not every station nationwide',

regulatoryFlags: regulatoryFlagsFor('AT'),

};

Prices pass through ECB FX into *Eur and priceEur so a Budapest→Vienna comparison is not HUF guesswork against EUR memory.

Route mode keeps cheapest-stop math deterministic — haversine detour, not LLM arithmetic:

Code (typescript)

// src/route/geometry.ts (excerpt)

const detour = toStation + stationToDest - direct;

if (detour > opts.maxDetourKm) continue;

if (!best || price < best.price || (price === best.price && detour < best.detour)) {

best = { station: s, price, detour, fromOrigin: toStation };

}

Partial runs never invent rows. markCountryStatus classifies each country (ok, empty, missing_fuelo_key, …) and the run still succeeds for the rest:

Code (typescript)

// src/main.ts (excerpt)

if (errMsg) {

countryStatus[country] = classifyAdapterError(errMsg).status;

return;

}

Pricing is explicit PPE: station-price $0.0025, route-recommendation $0.05, optional ai-digest $0.15 (BYOK, off by default), $0.10 minimum per run.

Architecture: agent decides, Actor evidences

Code (text)

User question

→ Cursor agent

→ Apify MCP (call-actor / fetch-actor-details)

→ Fuel Intelligence Actor

→ country adapters → live feeds

→ normalized dataset

→ agent recommendation

The agent is the decision layer. The Actor is the evidence layer. MCP is the bridge that removes the manual Console handoff.

Wiring Cursor to Apify MCP

Apify hosts MCP at https://mcp.apify.com with OAuth on first connect — no API token in your config file. In .cursor/mcp.json:

Code (json)

{

"mcpServers": {

"apify": {

"url": "https://mcp.apify.com/?tools=fetch-actor-details,kondasviktor/cee-dach-fuel-intelligence"

}

}

}

That pin matches the snippet on the Actor's Store page: fetch-actor-details plus the Actor id. For day-to-day exploration I also use a broader pin (actors,docs,...) so the agent can search Store and read docs; for a fleet assistant that always calls this Actor, the narrow pin skips a search round trip.

Reload Cursor, sign in to Apify when prompted, then enable the server under Settings → Tools & MCP.

Cursor MCP configuration in mcp.json showing the Apify server URL with the cee-dach-fuel-intelligence Actor pinned as a tool

Cursor MCP tool list for the Apify server — all tools enabled including kondasviktor/cee-dach-fuel-intelligence

Typical discovery path: search-actors → fetch-actor-details (read the input schema (https://docs.apify.com/platform/actors/development/actor-definition/input-schema)) → call-actor. The input schema tells the agent required fields (mode, lat/lng for point, origin/destination for route). Apify can also infer output shapes from recent successful runs; my Actor's dataset schema documents row types (station-price, route-recommendation, run-status). The Actor output schema in Console is mainly deep links (dataset, runOverview) — field typing for agents comes from input + dataset schemas, not from those links alone.

Three real agent runs (August 2026)

I did not tell the agent "run my fuel Actor." I asked logistics questions and let it resolve the tool through MCP. All runs below are cloud runs via Apify MCP; run IDs are real. Origin in Console shows MCP.

A — Vienna petrol, key-free official data

Prompt: I need current petrol prices near central Vienna for a delivery van, and tell me how reliable that data is.

Input the agent constructed:

Code (json)

{

"mode": "point",

"countries": ["AT"],

"lat": 48.2082,

"lng": 16.3738,

"radiusKm": 8,

"fuel": "petrol95",

"maxStations": 8,

"enableAIDigest": false

}

Run: chJk2dCvN45DnWGKC · SUCCEEDED in 4.5s · 10 dataset items · countryStatus.AT: ok

Cheapest row returned:

Code (json)

{

"recordType": "station-price",

"stationId": "1494440",

"brand": "TMC",

"name": "TMC Werkstatt & Tankstelle",

"country": "AT",

"city": "Wien",

"petrol95": 1.669,

"petrol95Eur": 1.669,

"priceEur": 1.669,

"currency": "EUR",

"dataConfidence": "OFFICIAL_REALTIME",

"source": "E-Control Spritpreisrechner",

"sourceStatus": "OK"

}

Agent conclusion: TMC at €1.669 is the cheapest nearby petrol95; data is official E-Control realtime, but cheapest-N — not every Austrian station.

Scenario A run detail in Apify Console — run chJk2dCvN45DnWGKC, SUCCEEDED, 10 results, origin MCP

Scenario A dataset — Vienna petrol95 station-price rows sorted by price, TMC first at €1.669, all with OFFICIAL_REALTIME confidence

B — Budapest→Vienna diesel route

Prompt: I'm driving a diesel van from Budapest to Vienna today. Where should I stop to fill up without a big detour, and how sure are you about that price?

Input (Fuelo key redacted — buyers supply their own):

Code (json)

{

"mode": "route",

"countries": ["HU", "AT"],

"originLat": 47.4979,

"originLng": 19.0402,

"destinationLat": 48.2082,

"destinationLng": 16.3738,

"fuel": "diesel",

"maxDetourKm": 20,

"maxStations": 25,

"radiusKm": 15,

"enableAIDigest": false,

"fueloApiKey": "YOUR_FUELO_KEY"

}

Run: mT6vN8TOsCG7P5d6s · SUCCEEDED in 12.9s · 39 items · one route-recommendation row

Code (json)

{

"recordType": "route-recommendation",

"stationId": "fuelo:HU:36084",

"brand": "Slovnaft",

"country": "HU",

"city": "Bratislava",

"fuel": "diesel",

"price": 1.706,

"priceEur": 1.706,

"currency": "EUR",

"distanceFromOriginKm": 158.38,

"estimatedDetourKm": 1.03,

"dataConfidence": "COMMUNITY_AGGREGATOR_UNVERIFIED"

}

Agent conclusion: Cheapest stop within ~1 km detour is Slovnaft diesel at €1.706 — but the agent must hedge: COMMUNITY_AGGREGATOR_UNVERIFIED, not official regulator data. The same run also returned OFFICIAL_REALTIME E-Control rows for AT segments; mixing tiers in one answer without naming confidence would be wrong.

This is token economics in practice: route mode returns one recommendation record plus bounded station rows (maxStations), not an unbounded national dump. MCP still returns full field lists per row — context control is the Actor's modes and caps, not automatic truncation in MCP.

Scenario B dataset — Budapest-Vienna route, row 42 is the route-recommendation: Slovnaft Bratislava, diesel €1.706, ~1.03 km detour

C — Poland without a Fuelo key (honest failure)

Prompt: Current diesel near central Warsaw — same kind of answer as Vienna.

Input: point query for PL — fueloApiKey deliberately omitted even though I have a key locally.

Code (json)

{

"mode": "point",

"countries": ["PL"],

"lat": 52.2297,

"lng": 21.0122,

"radiusKm": 10,

"fuel": "diesel",

"maxStations": 10,

"enableAIDigest": false

}

Run: RJCT720eHg3INuOzg · SUCCEEDED in 3.6s · 0 stations · 1 run-status row

Code (json)

{

"recordType": "run-status",

"countryStatus": { "PL": "missing_fuelo_key" },

"stationCount": 0,

"okStationCount": 0,

"notes": [

"PL: PL uses Fuelo.net (BYOK). Set input fueloApiKey from https://fuelo.net/about/api\_key\_request — we never use platform keys."

]

}

Agent conclusion: Polish station data is unavailable until the caller supplies a Fuelo key — do not invent Warsaw diesel prices. Honest failure beats silent emptiness: the run succeeded, the status says why it is empty.

Scenario C run detail — run RJCT720eHg3INuOzg, SUCCEEDED with 1 result, origin MCP, completed in 3 seconds

Scenario C Actor log — ERROR line shows PL uses Fuelo.net BYOK, followed by INFO: stations=0, status=missing_fuelo_key

What MCP removed

Without MCP, a human (or agent pretending to be one) does this:

Open Apify Console
Find the Actor
Fill input JSON
Start run
Wait
Open dataset
Export or copy rows
Paste into chat
Ask the model to recommend

With MCP, steps 2–8 collapse into one agent turn: recognize live fuel data is needed → fetch-actor-details → call-actor → reason on structured rows. The manual handoff disappears.

Design choices that mattered for agents

A few patterns I would steal for any live-data Actor meant for MCP:

Put provenance on the row, not only in the README. dataConfidence, source, and sourceMessage ride on every station row so the agent can hedge inside one tool call. A spreadsheet buyer scrolls to methodology; an agent cannot.

Keep regulation in dated config, not model memory. regulatoryFlags come from JSON (lastVerified: 2026-07-24). The optional ai-digest PPE event stays off the hot path — narrative over aggregates is for scheduled batch runs, not every "where's cheapest" query.

Keep the core deterministic. Currency normalization, detour math, and cheapest-stop selection stay in TypeScript. Let the agent interpret; do not let it calculate pump prices.

Name fields so a model can infer meaning. estimatedDetourKm and missing_fuelo_key beat internal enums like tier2.

Design for agent call patterns. One route call beats eleven country calls. Cap maxStations. Batch countries in a single input when the schema allows it.

What broke while building this

AT always needs coordinates. Even country mode feeds a national centroid into E-Control's lat/lng search — there is no "whole country" dump.
asOfDate on E-Control throws explicitly when asOfDateStrict=true instead of silently serving live prices as history.
Tankerkönig free tier caps radius at 25 km — the input schema enforces the same cap.
Missing BYOK must not look like "no stations exist." Poland without fueloApiKey → missing_fuelo_key, not an empty success story.
Early MCP pins that only exposed call-actor without fetch-actor-details made the agent guess required fields. Pinning the Actor and details fixed that.

What I would do differently

I would ship the dataset schema and dataConfidence enums before the first public Store listing. I added provenance after watching an agent treat a Fuelo community price like E-Control realtime. I would also document MCP pin recipes on the Store page earlier — the narrow pin vs broad actors,docs pin is not obvious from Console alone.

Next on my list: richer route geometry than haversine (road network detours), and clearer city/country consistency checks when a community feed returns odd location labels.

Takeaways

One Actor now serves Console, API, schedules, automation tools — and agents via MCP. Fuel was the forcing function because bad answers are costly; the pattern generalizes to any live, messy, geographic dataset (EV charging, truck parking, tolls): normalize in the Actor, expose provenance on every row, keep math deterministic, let MCP handle the call, let the agent handle the question.

Try it: European Fuel Station Prices API (CEE + DACH) (https://apify.com/kondasviktor/cee-dach-fuel-intelligence) — AT/SI/HR/RO need no key. MCP setup: Apify MCP docs (https://docs.apify.com/platform/integrations/mcp).

Viktor Kondas · [Apify profile](https://apify.com/kondasviktor) · Built in Cursor with Apify MCP

Event and nightlife demand intelligence with Eventbrite and Resident Advisor

Crawler Bros — Mon, 14 Sep 2026 05:10:26 +0000

Event and nightlife demand intelligence with Eventbrite and Resident Advisor

Event promoters, venue bookers, and artist managers usually track demand through instinct: which venues seem busy, which artists seem to be blowing up, which ticket prices seem too high or too low. That's fine until you're deciding whether to book a venue in a new city or price tickets for an unfamiliar market.

Event and nightlife demand intelligence means turning scattered venue and promoter pages into one city-level dataset of event density, pricing, and lineup activity, instead of relying on instinct or manually checking dozens of individual pages. Eventbrite and Resident Advisor between them cover a large share of public event listings — general events and ticketing on one side, electronic music and nightlife specifically on the other.

Quick answer

Use Eventbrite Scraper to collect event listings, dates, venues, and ticket prices across categories and cities. Use Resident Advisor Scraper for electronic music and nightlife events specifically, including artist lineups and venue details. Combine both to build a city-level view of event density, pricing, and which artists or venues are drawing repeat bookings.

The event demand stack

The stack uses two CrawlerBros Actors:

Eventbrite Scraper for events including title, date, venue, ticket price, and tags.
Resident Advisor Scraper for upcoming events by city, artists, and venues.

The workflow:

\Target city or region -> Eventbrite Scraper (general events, ticket pricing) -> Resident Advisor Scraper (nightlife/music events, artists, venues) -> city-level event demand dataset \\

Key facts

Two platforms, two lenses: Eventbrite covers broad event categories; Resident Advisor is concentrated in electronic music and nightlife specifically.
Event count alone misleads: a city with fewer, higher-price events can be a healthier market than one with many low-price events competing for the same audience.
Venue-level frequency matters more than city totals: a venue hosting weekly events is a different business than one hosting quarterly.
RA's sparse coverage of a city doesn't mean the city is quiet: check Eventbrite's music category for that city before concluding demand is low.

Actor configuration that matters

For Eventbrite, filter by city and category (music, food and drink, business, arts) rather than pulling every event type in one run, since category mix varies a lot by city and a single run mixing categories makes pricing comparisons meaningless.

\json { "location": "Austin, TX", "category": "music", "maxItems": 150, "dateRange": "next_30_days" } \\

For Resident Advisor, scope by city and date range. RA's listings are concentrated in cities with an active electronic music scene, so a city with little RA coverage isn't necessarily quiet — it may just not be RA's core market, which is worth checking against Eventbrite's own music category for that city.

What the output looks like

The Eventbrite row needs event title, venue, date, category, ticket price (or price range), and organizer. The Resident Advisor row needs event title, venue, date, artist lineup, and event URL. Neither dataset alone answers "is this city's nightlife scene growing" — venue-level event frequency over several weeks does.

Use case 1: venue booking decisions for touring artists

An artist manager deciding which cities to book can check RA event density and typical lineup tier for target cities, cross-referenced with Eventbrite's broader event calendar to see what else is competing for the same audience on a given date.

Use case 2: ticket pricing benchmarks

A promoter pricing tickets for a new event can pull recent Eventbrite and RA listings for comparable events in the same city and genre to price against what's actually selling, rather than copying a price from a different market.

Use case 3: venue and city market research

A venue considering a second location can compare event density and category mix across candidate cities — a city with steady RA and Eventbrite activity across multiple venues suggests an established scene; a city with only occasional large events suggests a market that hasn't been built out yet.

Production notes

Don't compare cities on raw event count alone. A city with fewer, higher-price events can represent a healthier market than a city with many low-price events competing for the same audience.

Track venue-level frequency, not just city-level totals. A venue hosting events weekly is a different business proposition than one hosting quarterly.

Combine both sources before drawing genre conclusions. RA is strong for electronic music specifically; general demand for a city's live music scene needs Eventbrite's broader category coverage too.

Watch date ranges carefully. Both platforms list events well in advance, so a snapshot taken today under-represents a city's actual events happening next month unless the date range is wide enough.

One source alone under-represents demand. Using RA data alone works for electronic-music-focused research but badly under-represents general live music and community events. Adding Eventbrite's broader category coverage gives a much more complete city picture.

Cost comparison

Approach	Coverage	Pricing data	Artist/lineup data
Manual venue-by-venue checks	Narrow, slow	Inconsistent	Inconsistent
Ticketing platform dashboards	Owned events only	Yes	Limited
Apify Eventbrite + RA pipeline	Multi-venue, multi-city	Yes	Yes (RA)

Check the current Pricing tab on each Actor page before running at scale.

FAQ

Can Apify track event ticket pricing?

Yes. Eventbrite Scraper extracts public event listings including ticket price, venue, and date.

Is Resident Advisor only for electronic music?

Yes, RA's coverage is concentrated in electronic music and nightlife; use Eventbrite for broader event category coverage.

Can this help decide where to book a tour?

Yes — comparing event density, pricing, and lineup tier across candidate cities is a common use of this combination.

What's the fastest way to try this?

Pick one city and date range, run Eventbrite for the relevant category and Resident Advisor for the same city, then compare venue frequency and pricing.

Try it yourself

Pick one city and one date range. Run Eventbrite Scraper for the relevant category and Resident Advisor Scraper for the same city. Compare venue frequency and ticket pricing across both. The city with steady activity across multiple venues, not just one big event, is usually the stronger long-term market.

Mining Quora and Reddit for SEO content ideas

Crawler Bros — Sun, 13 Sep 2026 07:07:30 +0000

Mining Quora and Reddit for SEO content ideas

Most SEO content calendars start from a keyword tool, which shows what people search for but not how they actually phrase the question or what specifically confuses them. Quora and Reddit answer that second part directly — both platforms are full of people asking exact questions in their own words and getting corrected, argued with, or thanked in the replies.

Content idea mining from Quora and Reddit means turning a keyword into a specific, well-scoped article outline built from the actual questions people ask and the actual confusions that show up in the answers, instead of writing another generic "ultimate guide" from keyword volume alone.

Quick answer

Use Quora Search Scraper to find questions and existing answers related to a topic. Use Reddit Scraper to pull relevant subreddit discussions and Reddit Comment Scraper to capture the actual reply threads. Combine all three to build content outlines around the specific questions people ask and the specific confusions that come up in the answers.

The content research stack

The stack uses three CrawlerBros Actors:

Quora Search Scraper for questions, answers, and engagement metrics.
Reddit Scraper for entire subreddits with posts, titles, text, scores, and timestamps.
Reddit Comment Scraper for comments on specific posts with text, scores, and timestamps.

The research loop:

\Target topic or keyword -> Quora Search Scraper (questions + top answers) -> Reddit Scraper (relevant subreddit posts) -> Reddit Comment Scraper (reply-thread detail on the best posts) -> content outline backlog \\

Key facts

Question phrasing is a reusable asset. Quora question text is often close to how someone would actually search — closer than a paraphrased keyword-tool phrase.
Comment threads reveal the real content gap. Where commenters correct or argue with a post is a stronger signal of what's missing than keyword volume alone.
Run comment extraction selectively. Reddit Comment Scraper is worth running on the high-engagement posts Reddit Scraper already flagged, not on every post.
Search multiple phrasings of the same topic. Quora questions rarely match keyword-tool phrasing exactly; two or three variations catch more relevant questions.

Actor configuration that matters

For Quora Search Scraper, search by the same phrase variations you'd use for keyword research — Quora questions are often phrased more conversationally than search queries, so running a few phrasings of the same topic surfaces more relevant questions than a single exact-match search.

\json { "searchQuery": "how to price a freelance web design project", "maxItems": 50 } \\

For Reddit Scraper, target subreddits where your topic's actual audience discusses it, rather than a generic keyword search across all of Reddit — a freelance-pricing topic belongs in freelancing and web-design subreddits, not a broad site-wide search. For Reddit Comment Scraper, run it against the specific posts Reddit Scraper flagged as high-engagement, since comment-level detail is only worth extracting for posts that already show real discussion.

What the output looks like

The Quora row needs question text, top answer text, answer author, and vote count — question phrasing is often the most reusable asset, since it's frequently close to how someone would actually search. The Reddit post row needs title, subreddit, score, comment count, and URL. The Reddit comment row needs comment text, score, and parent post URL, so a repeated objection or correction can be traced back to its source thread.

Use case 1: FAQ and "People Also Ask"-style content

Quora question phrasing translates almost directly into FAQ section headers — using the actual question text (rather than a paraphrased version) both matches searcher intent more closely and often matches long-tail search phrasing search engines already index well.

Use case 2: identifying content gaps in existing guides

Reading Reddit comment threads on a topic often surfaces the specific point where people get confused or where an existing popular guide is wrong or outdated — that's a stronger content-gap signal than keyword volume alone, since it shows exactly what existing content is failing to answer.

Use case 3: building content in the audience's actual language

The phrasing patterns that repeat across Quora answers and Reddit comments — the specific words practitioners use, not marketing language — make useful headline and subheading material, since content written in the audience's own vocabulary tends to read as more credible to that audience.

Production notes

Search multiple phrasings, not just the primary keyword. Quora questions rarely match keyword-tool phrasing exactly; running two or three variations catches more relevant questions.

Only run Reddit Comment Scraper on posts worth the detail. Comment-level extraction is more useful selectively, on posts Reddit Scraper already flagged as high-engagement, than run broadly across every post.

Read the actual disagreement in comment threads. The most useful content ideas often come from where commenters correct or argue with the original post, not from the post itself.

Keep source links attached to every idea. An outline built from a Reddit thread or Quora answer should keep the source URL nearby, both for fact-checking and to avoid presenting someone else's specific anecdote as a generic example.

Question text alone misses the texture of confusion. Quora questions give headline ideas but miss where people actually get stuck. Adding Reddit comment-thread detail surfaces the specific misconceptions worth addressing directly in the content.

Cost comparison

Approach	Question phrasing	Discussion depth	Scale
Keyword tool only	No	None	High
Manual forum browsing	Yes	Yes	Low
Apify Quora + Reddit pipeline	Yes	Yes	High

Check the current Pricing tab on each Actor page before running at scale.

FAQ

Can Apify scrape Quora questions and answers?

Yes. Quora Search Scraper extracts questions, answers, and engagement metrics for a given search query.

Is this different from keyword research?

Yes — keyword research shows search volume and phrasing at a high level; this workflow shows the specific questions and confusions behind that volume.

Do I need to scrape every comment on every post?

No — target comment extraction at the specific high-engagement posts your subreddit-level scrape already flagged as relevant.

What's the fastest way to try this?

Pick one topic, run Quora Search Scraper for two or three phrasings and Reddit Scraper on two or three relevant subreddits, then pull comments on the best posts.

Try it yourself

Pick one topic you're already planning to write about. Run Quora Search Scraper for two or three phrasings, then run Reddit Scraper on the two or three subreddits where your audience actually hangs out. Pull comments with Reddit Comment Scraper on the best posts. Build your outline from the actual questions and disagreements, not from a keyword list alone.

Startup idea radar: finding validated ideas with Reddit, Search, and News data

Crawler Bros — Sat, 12 Sep 2026 06:55:11 +0000

Startup idea radar: finding validated ideas with Reddit, Search, and News data

Startup ideas rarely arrive as neat pitch decks.

They show up as complaints, repeated questions, "alternative to..." threads, half-solved workflows, and long comments explaining why an existing product almost works.

A startup idea radar is a repeatable research workflow that collects public pain signals — Reddit complaints, Google search intent, and news momentum — and scores them, instead of relying on a single brainstorm or gut feeling. This Apify idea radar collects those signals from Reddit, Google Search, and Google News so a founder can rank pains before building.

Quick answer

Use Reddit scraping to find raw user complaints, Google Search scraping to understand search intent and competing content, and Google News scraping to track market momentum. Combine the datasets to rank startup ideas by pain frequency, buyer intent, competition, and trend activity.

The startup research stack

The research stack uses:

Reddit Keywords and Reddit Scraper-style Actors for problem discovery.
Google Search Results Scraper for search intent, People Also Ask, and competitor pages.
Google News Scraper for market news and trend validation.

The idea discovery loop:

\Idea keywords -> Reddit complaint mining -> Google search intent -> news and trend context -> scored idea backlog \\

Key facts

3 sources, 3 jobs: Reddit finds pain, Google Search finds commercial intent, Google News finds market momentum.
A signal is strongest when it appears in more than one source: a Reddit complaint plus matching search queries plus category news is far more convincing than any one alone.
Exact phrases beat generic keywords: complaint phrases like "alternative to" or "too expensive" surface pain that broad keyword volume misses.
Boring, repeated pains tend to win: reporting, lead enrichment, price tracking, hiring signals, and data cleanup outperformed novelty ideas in practice.

Reddit finds pain

Reddit is useful because people complain in detail. They don't say "workflow optimization opportunity." They say the tool broke, the pricing is unfair, the integration is missing, or the support team ignored them.

Search for patterns like:

"alternative to"
"is there a tool for"
"I hate using"
"too expensive"
"doesn't integrate with"
"manual spreadsheet"
"looking for recommendations"

The output gives threads, titles, scores, subreddits, URLs, and sometimes comment data depending on the Actor stage.

Actor configuration that matters

For idea discovery, the exact query set matters more than the tool choice. Start Reddit Keywords with complaint and alternative phrases, then use Google Search Results Scraper for the same problem terms.

\json { "keywords": [ "alternative to airtable", "too expensive crm", "is there a tool for client reporting", "manual spreadsheet workflow" ], "resultLimit": 100, "sort": "relevance" } \\

Then pass the strongest terms into Google Search and Google News. Search shows whether the problem has commercial intent. News shows whether the category is moving.

What the output looks like

For each idea, keep both the evidence and the score:

\json { "idea": "client reporting automation for agencies", "source": "reddit", "query": "manual spreadsheet client reporting", "title": "How are you all sending weekly client reports?", "subreddit": "agency", "score": 87, "commentsCount": 42, "url": "https://old.reddit.com/r/..." } \\

The score is never enough on its own. Keep the original title, source, and URL because the wording is often the insight. A founder can learn more from one angry thread than from a sterile keyword-volume number.

In a small Reddit Keywords run for "alternative to airtable", the Actor returned five posts. The output included keyword, post_id, title, author, subreddit, content, score, num_comments, url, old_reddit_url, link_flair, created_at, is_stickied, and is_nsfw. This is the exact kind of public pain signal you want for idea validation.

Google Search finds intent

Reddit pain is not enough. Some complaints are loud but not commercial.

Google Search Results Scraper helps check whether people also search for the problem. Look at:

organic competitors
People Also Ask questions
related searches
result count
ranking page types
country-specific differences

If Reddit has pain and Google has commercial pages, the idea becomes more interesting.

News finds momentum

Google News adds market context. A boring problem in a growing market can be more valuable than an exciting problem in a shrinking one.

Monitor:

funding announcements
regulation changes
layoffs
category growth
enterprise adoption
platform policy shifts

This helps separate durable markets from temporary noise.

How to score ideas

A lightweight scoring model uses five dimensions:

pain frequency
pain intensity
buyer intent
competitive saturation
market momentum

A great idea doesn't need a perfect score. It needs enough evidence to justify deeper research.

What a good signal looks like

A useful idea signal usually appears in more than one source.

Reddit might show people complaining that existing tools are too expensive. Google Search might show "best X alternative" and "X pricing" queries. Google News might show funding or regulation in the category. Together, those signals suggest a market with pain, intent, and movement.

A weak signal looks different. One viral Reddit thread may create noise, but if Google has no related searches and news is quiet, the pain may be too narrow. That doesn't mean the idea is bad — it means you need interviews before treating it as a market.

Also look for language patterns. The exact words people use in complaints can become landing page copy later. "I just want a simple way to..." is more valuable than a generic keyword because it captures the job-to-be-done.

The research backlog

Every idea becomes a row in a backlog:

idea name
source keywords
strongest Reddit threads
strongest search queries
related competitors
recent news links
pain score
intent score
notes from manual review

This keeps the workflow practical. The goal is not to automate taste. The goal is to make your taste better informed.

Where Apify helps

The advantage of Apify is repeatability. A founder can run the same research workflow every week without rebuilding scrapers or copying search results by hand.

That matters because startup research changes. A pain point that looked small in January may explode after a platform policy change in March. A category that looked crowded may become interesting when users start complaining about pricing. Scheduled Actor runs create a trail of evidence instead of a pile of one-off notes.

Manual reading of the best threads is still worth it. Automation narrows the field. Judgment picks the idea. The best outcome is not more ideas — it's fewer, better-tested ones.

Production notes

Search exact phrases. Generic keywords produce generic results. Complaint phrases reveal pain.

Keep subreddit context. A complaint in a founder subreddit means something different from the same complaint in a hobby subreddit.

Read comments, not only titles. The best details are usually in comment threads.

Do not confuse trendiness with demand. News volume can mean hype. Pair it with search and Reddit evidence.

Store rejected ideas. Rejected ideas are useful when the market changes.

Novelty isn't the goal. The best ideas were rarely the weirdest ones — they were boring problems with repeated pain: reporting, lead enrichment, price tracking, hiring signals, and data cleanup. Scraping helps you see repetition, which is usually more valuable than surprise.

Cost comparison

Approach	Strength	Weakness
Founder intuition	Fast	Biased
Surveys	Direct answers	Slow and prompted
Keyword tools	Search volume	Weak on raw pain
Apify idea radar	Public pain plus intent	Needs human judgment

Check the current Actor Pricing tabs before running at scale. This workflow is usually cost-controlled by small, targeted runs instead of broad scraping.

FAQ

Can scraping help validate startup ideas?

Yes. Scraping public discussions, search results, and news can reveal recurring problems and market signals.

Why use Reddit for startup research?

Reddit has detailed public discussions where users describe problems in their own words.

Does this replace customer interviews?

No. It helps you choose better ideas before interviews.

What's the fastest way to try this?

Pick one market, run Reddit keyword searches for complaint phrases, scrape Google Search results for the same problem, and collect recent news.

Try it yourself

Pick one market. Run Reddit Keywords for complaint phrases, scrape Google Search Results Scraper for the same problem, and collect recent news with Google News Scraper. If all three sources point to the same pain, you have something worth investigating.

Turning a jobs scraper into an AI agent's tool with the Apify MCP server

Zahar Baranenko — Fri, 11 Sep 2026 12:07:54 +0000

There is a lot of good advice around on making your Actor agent-friendly. Almost none of it shows you the thing an agent actually sends over the wire — and that turned out to be the part that changed how I build.

I build Actors on Apify under the name primeflowio. One of them, ATS Jobs Scraper & Change Monitor, takes a list of companies and returns every open job they have, normalized into one schema — no matter which applicant tracking system the company uses. A few weeks after publishing it, I connected it to the Apify MCP server and an agent called it like a native tool: discovered it, filled the input, ran it, read the results back. No glue code on my side.

The Origin column says MCP: that run was started by an agent, not by me clicking Start.

So this article does two things. First it walks the protocol by hand — every request, every real response — so you can see exactly what your Actor is answering to. Then it goes through what I changed once I had read those responses, including a bug that would have told a customer that every job at a company had disappeared overnight.

The problem the Actor solves

Job postings are scattered across ATS platforms. If you track 200 target companies — as recruiters, B2B sales teams, and serious job seekers do — you'd need to visit 200 career pages, sitting on Workday, Greenhouse, Lever, Ashby, SmartRecruiters, Recruitee or Personio, each with its own markup.

The twist most people miss: you don't need to scrape any of those pages. The major ATS platforms expose documented, public endpoints for their job boards. Here is every platform the Actor supports, with what each one actually returned when I re-checked it by hand this week:

ATS	Endpoint	Live check (Sep 2026)
Greenhouse	`boards-api.greenhouse.io/v1/boards/{slug}/jobs`	stripe: 611, gitlab: 230
Lever	`api.lever.co/v0/postings/{slug}?mode=json`	spotify: 73
Ashby	`api.ashbyhq.com/posting-api/job-board/{slug}`	ramp: 143, linear: 28
Personio	`{company}.jobs.personio.de/xml`	personio: 1 open role
Recruitee	`{company}.recruitee.com/api/offers/`	yource: 1 open role
Workday	`{tenant}.{wd}.myworkdayjobs.com/wday/cxs/{tenant}/{site}/jobs`	nvidia: 2,000 — and slow
SmartRecruiters	`api.smartrecruiters.com/v1/companies/{slug}/postings`	see below

No API keys, no proxies, no anti-bot arms race. The hard part isn't access — it's detection (which ATS does this company use?), normalization (a different response shape per platform, one schema out), and change tracking.

Three of those rows deserve a footnote, and they are the same lesson in different clothes.

Workday is the expensive one. It answers, and it answers with everything — two thousand postings for one tenant — but a small run against it took 83 seconds where Ashby takes two. That is fine for a watchlist on a schedule and a bad idea for anything with a stopwatch on it, which is why my prefilled input does not point at Workday.

Recruitee gets the failure mode right, and it is worth copying. An unknown company returns 404 {"error":"Not Found"}. A real company with nothing open returns 200 {"offers":[]}. Those are different sentences, and a caller can act on the difference.

Which brings us to SmartRecruiters, which does not. It answers 200 {"totalFound": 0, "content": []} for every identifier I tried — including companies that visibly run their careers site on SmartRecruiters, and including identifiers that do not exist at all. An endpoint that cannot distinguish "no openings" from "wrong name" from "this company never switched the public API on" is not a data source; it is a coin flip with a JSON wrapper. The Actor now reports that board as unreadable instead of returning an empty list that looks like an answer — which turns out to be the same bug I had shipped myself, one layer up. More on that in lesson 6.

So the Actor does three things:

Auto-detect — given a company slug or careers URL, it probes the supported endpoints and figures out which ATS answers.
Normalize — every job becomes one record: title, department, team, location, remote, employment_type, salary_min/max, url, published_at, updated_at.
Monitor — in monitor mode it keeps the previous run's state in the Actor's key-value store and emits new / removed / changed records as a diff. Users point a webhook or an n8n flow at those and get "company X just opened 3 engineering roles" without re-downloading everything.

That last part is the reason the Actor exists at all: the incumbent job-data Actors make you re-pull the full dataset every run and charge per job every time. If what you actually want is changes, you're paying for the same rows daily.

Every Actor is already an MCP tool

Here's the part that cost me zero extra work. The Apify MCP server exposes any public Actor as a tool over the Model Context Protocol. You don't register anything, you don't write a manifest — the Actor's input schema is the tool definition. An agent connects to:

https://mcp.apify.com/?tools=primeflowio/ats-jobs-scraper

and gets a tool named primeflowio--ats-jobs-scraper, with parameters generated from the same input schema that renders the form in Apify Console. (?tools= is the parameter in Apify's docs; ?actors= is accepted too — I checked both today and they return the same five tools.)

I wanted to see exactly what an agent sees, so I did the whole handshake by hand with curl before touching any framework. MCP over streamable HTTP is three steps.

1. Initialize a session:

curl -s -D - -X POST "https://mcp.apify.com/?tools=primeflowio/ats-jobs-scraper" \
  -H "Authorization: Bearer $APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -H "Accept: application/json, text/event-stream" \
  -d '{"jsonrpc":"2.0","id":1,"method":"initialize",
       "params":{"protocolVersion":"2025-03-26","capabilities":{},
                 "clientInfo":{"name":"demo-agent","version":"1.0"}}}'

The response headers contain mcp-session-id — every later call carries it in an Mcp-Session-Id header, plus a notifications/initialized message to finish the handshake.

2. List tools. {"method":"tools/list"} returns, in my session:

primeflowio--ats-jobs-scraper
get-actor-run
get-dataset-items
get-key-value-store-record
abort-actor-run

Note what the server adds for free: the agent doesn't just get my Actor, it gets the helpers to check a run's status and page through the results dataset. That completes the loop — an agent can start a run, poll it, and fetch data without any Apify-specific code.

3. Call the tool:

curl -s -X POST "https://mcp.apify.com/?tools=primeflowio/ats-jobs-scraper" \
  -H "Authorization: Bearer $APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -H "Accept: application/json, text/event-stream" \
  -H "Mcp-Session-Id: $SESSION" \
  -d '{"jsonrpc":"2.0","id":3,"method":"tools/call",
       "params":{"name":"primeflowio--ats-jobs-scraper",
                 "arguments":{"companies":["https://jobs.lever.co/spotify"],
                              "mode":"pull","maxJobsPerCompany":3}}}'

The run finished in under two seconds and the MCP response came back with a summary the agent can act on directly:

"status":"SUCCEEDED",
"statusMessage":"1/1 companies, 3 jobs, 0 changes, 0 errors"
...
SUCCEEDED in 1.635s. 4 items; 24 fields available.
Use get-dataset-items with datasetId=pl8b6nClKueN3efCO and limit
(for example 20) to fetch items (4 total). Available fields (dot notation):
record_type, company, ats, job_id, title, department, team, location,
remote, employment_type, salary_min, salary_max, salary_currency, url, ...

The whole exchange from a plain client: five tools discovered, one called, a summary back in under three seconds.

Read that response again from an LLM's point of view. It's not a blob of JSON rows — it's an instruction: here's what happened, here's the tool call to make next, here are the fields you can ask for. The Apify MCP server writes that guidance itself. My only job was to make sure the run summary (statusMessage) and the dataset fields were worth reading.

The same flow in Python

For anything scripted I use ~60 lines of plain requests — no MCP SDK needed for the happy path. Full file in the repo; the core is:

MCP_URL = "https://mcp.apify.com/?tools=primeflowio/ats-jobs-scraper"

def rpc(session, method, params=None, msg_id=None):
    payload = {"jsonrpc": "2.0", "method": method}
    if params is not None: payload["params"] = params
    if msg_id is not None: payload["id"] = msg_id
    headers = dict(HEADERS)
    if session: headers["Mcp-Session-Id"] = session
    r = requests.post(MCP_URL, headers=headers, json=payload, timeout=300)
    r.raise_for_status()
    return r

init = rpc(None, "initialize", {...}, msg_id=1)
session = init.headers["mcp-session-id"]
rpc(session, "notifications/initialized")

tools = parse_sse(rpc(session, "tools/list", msg_id=2).text)

result = parse_sse(rpc(session, "tools/call", {
    "name": "primeflowio--ats-jobs-scraper",
    "arguments": {"companies": ["https://jobs.lever.co/spotify"],
                  "mode": "pull", "maxJobsPerCompany": 3},
}, msg_id=3).text)

One implementation detail that trips people up: responses arrive as server-sent events (Content-Type: text/event-stream), even for a single JSON-RPC reply. parse_sse just takes the last data: line and json.loads it.

Plugging the same URL into Claude, Cursor, or any MCP-capable client is a config entry instead of code — the point of the protocol is that the curl session above is exactly what those clients do under the hood.

What building for agents changed in my Actor design

I originally designed the input schema for humans filling a form in Apify Console. Agents read the same schema, but they punish different mistakes. Six lessons I'd apply to any Actor now:

1. The input schema is a tool contract — keep required fields to one. My schema has exactly one required field, companies. Everything else (mode, maxJobsPerCompany, includeSnapshot) has a sane default. An agent that has to guess six required parameters will guess some of them wrong; an agent that has to provide one list of company names almost can't fail.

2. Give agents a cost throttle. maxJobsPerCompany exists so a cautious caller can cap the bill of an exploratory run. My test cost under a cent: with pay-per-event pricing, 1 company check plus 3 job results is $0.005. An agent (or the human supervising it) can try the tool for pocket change before pointing it at a 500-company watchlist. If your Actor can generate unbounded output, give the caller a knob to bound it.

3. Make statusMessage machine-readable prose. "1/1 companies, 3 jobs, 0 changes, 0 errors" is short enough to fit in any context window and precise enough that an agent can decide what to do next without fetching the dataset at all. I treat that one string as part of the API surface now.

4. Put bulk data in the dataset, not the response. The MCP server already nudges this: it returns a summary plus "use get-dataset-items with this ID" rather than inlining rows. Fighting that — returning huge payloads inline — wastes the agent's context and your users' tokens. Design the output so the first page of dataset items answers the most common question.

One schema out, whichever ATS answered. This is the page the agent fetches after the summary tells it what's there.

5. Your defaults are load-bearing: the platform runs them. Apify's automated QA runs every public Actor with its prefilled input and expects success within about 5 minutes. Mine was ["linear", "https://jobs.lever.co/spotify"] — two boards, ~130 postings, chosen back when I was thinking about humans clicking "Start" and wanting to see something substantial. In August an upstream endpoint got slow, three consecutive QA runs failed, and the Actor was flagged "under maintenance" in the Store until the runs went green again. I found out from the notification emails, days later. Worth knowing what that clock is: under Apify's publishing terms, an Actor left broken for 30 days can be deprecated and removed.

The lesson isn't "make the Actor faster". It's that the prefill is a health check the platform runs on your behalf, so it should exercise the cheapest, most reliable path you have. I've since cut mine to a single small board: 28 postings, 2.2 seconds against a five-minute budget. The same defaults make agent test-calls cheap, which is a happy coincidence — QA bots and cautious agents want exactly the same thing from you.

Ivan Solovyev landed on the same wall from a different direction — his prefill hit four marketplaces at once and the test gave up waiting while paying users ran the Actor happily all day. Two of us tripped over the same default in the same month, which says something about how easy it is to write a prefill for the demo rather than for the robot.

6. An empty answer is not "everything disappeared". This one I found because of that article, and it is the most expensive bug I have shipped.

In monitor mode the Actor compares this run's jobs against the previous run's state and emits new / removed / changed. The failure path was already handled: if a job board errors out, the run records an error for that company, keeps the old state, and moves on. What I had not separated was a board that answers 200 with an empty list — which happens during migrations, when a board is briefly unpublished, or when the provider hiccups.

That path was ok, jobs = True, []. The diff then did exactly what it was told: marked every known job as removed, and overwrote the state with nothing. A customer watching Stripe would have received "611 jobs removed" one morning and "611 jobs new" the next, with every webhook firing twice on fiction.

The guard is small, and the shape of it generalises to any stateful Actor:

# A board that answers 200 with zero jobs is usually the source blinking, not
# every role closing at once. Keep the old state, say so, and only believe the
# zero once it repeats.
if prev_jobs and not jobs and streak + 1 < EMPTY_RUNS_BEFORE_TRUSTED:
    push_items([{"record_type": "warning", "company": slug, "ats": ats,
                 "detail": f"{ats} returned an empty board while "
                           f"{len(prev_jobs)} jobs were known; previous state "
                           f"kept, no changes emitted",
                 "empty_streak": streak + 1}])
    state_put(store_id, cache_key,
              {"ats": ats, "jobs": prev_jobs, "empty_streak": streak + 1})
    continue

I tested it by seeding the state with two known jobs for a board that currently answers empty. Before: two removed records and a wiped state. After: 1/1 companies, 0 jobs, 0 changes, 0 errors, 1 stale, one warning record, and both jobs still in state. Real mass closures are slow; sources blink instantly — so a zero has to earn its credibility over three consecutive runs before the Actor will act on it.

The general rule, which is the same one Solovyev arrives at from the billing side: an Actor with memory must distinguish what the world says from what it managed to hear. For a stateless scraper an empty result is a result. For anything that diffs, an empty result is an accusation.

Where this goes next: monitor mode plus agents

Pull mode answers "who is company X hiring right now?" — a natural one-shot agent query ("did Spotify open any Android roles this month?"). Monitor mode is more interesting for agent workflows because it inverts the direction: instead of the agent polling, the Actor runs on Apify's scheduler, emits only new / removed / changed records, and a webhook or n8n flow wakes the agent only when something changed. "Head of Data role appeared at a target account" is a buying signal; an agent that gets only diffs can react to it without burning tokens re-reading yesterday's 611 postings.

One platform detail makes that possible, and it is the part I expected to be hard. Monitor mode needs memory that survives between runs, and those runs belong to whoever calls the Actor, not to me. Actors run with limited permissions by default, which sounds like the end of the idea. The docs say a limited-permissions Actor may "create any additional storage, and write to that storage" and "read and write to storages created in previous runs", and my own runs — the log line reads Running under "LIMITED_PERMISSIONS" — create a named key-value store, then read that same state back on later runs and diff against it. So the state can live on the caller's account rather than in a database of mine, and I never have to ask anyone for a token. I have only exercised that on my own account, so treat the cross-account half as documented rather than as something I measured; the design decision it points to is what matters. If you are building anything stateful, look at named storage before you reach for infrastructure of your own.

The diff mode earns its keep on live data, by the way. While I was writing this, a watchlist run caught Databricks editing a marketing role 29 minutes after they touched it — one changed record out of thirteen companies and roughly three and a half thousand postings. That is the argument for a change feed in a single line: the agent reads one row instead of 3,500.

That's the pattern I'd summarize the whole experience with: the MCP server made my Actor callable by agents in an afternoon, but making it worth calling — one required input, bounded costs, a status line an LLM can parse, diffs instead of dumps — was ordinary API design, applied to a new consumer.

FAQ

Do I need to modify an Actor to expose it via MCP? No. Any public Actor works through mcp.apify.com immediately; the input schema becomes the tool signature. Whether it's a good tool is a design question — see the six lessons above.

What does a call cost? Whatever the Actor's pricing says — same as running it from Console or API. This Actor is pay-per-event: $2 per 1,000 companies checked, $1 per 1,000 job results, so the demo call above cost $0.005.

Is scraping ATS job boards allowed? The endpoints used here are the platforms' own public job-board APIs — Greenhouse Job Board API, Lever Postings API, Ashby Posting API, Recruitee's offers endpoint, Personio's job XML feed. They exist precisely so postings can be syndicated to aggregators and career-site widgets, which is why none of them need a key.

Can the agent also fetch the results? Yes — the MCP session includes get-actor-run and get-dataset-items helpers automatically, so the agent can page through results without any Apify SDK.

Try it yourself

Point an agent at your own watchlist. Add the MCP server to any MCP-capable client:

https://mcp.apify.com/?tools=primeflowio/ats-jobs-scraper

…then ask it something you'd actually want to know — "who is Stripe hiring in Berlin right now?" — and watch it pick the arguments out of the input schema on its own. A single company check plus a handful of postings costs half a cent, so the first call is cheap enough to be curious with. Run it a second time a week later with mode: "monitor" and you get the diff instead of the dump.

The Actor is ATS Jobs Scraper & Change Monitor on Apify Store. The demo client and the full curl transcript from this article are at github.com/strongboxr/ats-mcp-agent-demo.

I build data Actors on Apify as primeflowio. If a number in this article has a caveat, the caveat is in the sentence next to it.

Suggested meta description: How an Apify Actor that scrapes ATS job boards became an AI agent tool via the Apify MCP server — the full handshake, working code, and six design lessons.

Shopify competitor monitoring without a price intelligence SaaS

Crawler Bros — Fri, 11 Sep 2026 05:29:56 +0000

Shopify competitor monitoring without a price intelligence SaaS

Most e-commerce competitor research is too manual to stay current.

Someone checks competitor stores when a launch is coming. Someone copies prices into a sheet. Someone screenshots a sale page. Two weeks later, the data is already stale.

Shopify competitor monitoring is the practice of collecting a competitor's public product catalog — prices, variants, tags, availability, and discounts — on a schedule and diffing it week over week, instead of manually checking storefronts when a launch seems likely. For Shopify stores, this is especially avoidable: many storefronts expose public catalog data that can be collected with Shopify Scraper Pro on Apify.

Quick answer

You can monitor Shopify competitors by scraping public product catalogs, collections, prices, variants, vendors, tags, availability, and discount data with Shopify Scraper Pro on Apify. Run it on a schedule and compare datasets to detect price changes, new products, removed products, and sale campaigns.

The Shopify monitoring setup

The monitoring setup uses Shopify Scraper Pro.

The Actor supports multi-store batches, collection-scoped scraping, in-store search, sale tracking, endpoint fallbacks, image size controls, and filters for price, vendor, type, tags, title, sale status, availability, and dates.

The monitoring loop:

\Competitor store list -> Shopify Scraper Pro -> product catalog dataset -> weekly diff -> price and launch alerts \\

The most useful outputs are product title, handle, URL, vendor, type, tags, price, compare-at price, discount percent, availability, variants, images, and collection context.

Key facts

Handles, not titles, are the stable anchor. Product titles change for merchandising reasons; handles usually stay stable across weekly diffs.
Catalog-level scraping catches launches, not just price moves. Product-page-only monitoring fails the moment a competitor ships something new.
Sale status is derived, not typed. Many stores never write "sale" in the title — onSale, comparePrice, and discountPercent come from structured pricing fields.
Collections often matter more than price. A product moving into "best sellers" or "new arrivals" can be a stronger signal than a two-dollar price change.

Actor configuration that matters

Shopify Scraper Pro accepts storeUrls as full URLs, bare domains, or myshopify.com handles. For focused monitoring, collectionHandles can restrict the crawl to collections like sale, new-arrivals, or mens. The Actor also supports searchQuery, sortBy, maxItemsPerStore, maxItems, image quality controls, collection enrichment, availability filters, price filters, tag filters, onSaleOnly, minDiscountPercent, date filters, SKU filtering, and polite request delay.

\json { "storeUrls": ["allbirds.com", "kith.com"], "collectionHandles": ["new-arrivals", "sale"], "sortBy": "best-selling", "maxItemsPerStore": 250, "onSaleOnly": false, "minDiscountPercent": 20, "enrichWithCollections": true, "requestDelaySecs": 1 } \\

That level of filtering is why catalog scraping beats random page checks — you can build a tight monitoring job around exactly the competitor collections and product signals that matter.

What the output looks like

A product row needs enough detail to survive comparison over time. Keep store, title, handle, productUrl, vendor, productType, price, compareAtPrice, discountPercent, available, and tags.

The handle is the anchor. Titles and descriptions change often, but handles usually stay stable enough for weekly diffs.

In a small Shopify Scraper Pro run against allbirds.com, the Actor returned five product rows. The output included productId, handle, productUrl, title, vendor, productType, tags, price, comparePrice, onSale, discountPercent, variantCount, available, availableVariantCount, skus, variants, images, storeUrl, and scrapedAt.

Why Shopify is a good monitoring target

Shopify stores change constantly:

new products launch
variants sell out
prices move
collections get reorganized
seasonal sale pages appear
best-selling collections shift
product tags reveal merchandising logic

Manual monitoring misses those changes. A scheduled Actor run catches them.

For a DTC brand, this matters because competitors rarely announce every test. A new bundle page, a 25% discount, or a "back in stock" product can explain shifts in paid search, email campaigns, or conversion performance.

The advantage of catalog-level data

Most price monitoring tools focus on product detail pages. That works when you already know which products matter. It fails when competitors launch new items.

Catalog-level scraping lets you detect:

new product handles
removed products
changed variants
price changes
sale status changes
collection additions
tag changes

This is how you catch product strategy, not just pricing.

Sale tracking

Shopify Scraper Pro derives sale information from price and compare-at price fields, giving you onSale, comparePrice, and discountPercent style analysis.

This is useful because many stores don't write "sale" in the product title. The discount exists in structured pricing data. Once the output is in a dataset, sort by discount percent and identify aggressive promotions.

The weekly diff

The most useful part of the workflow is the diff between two runs. Compare the latest catalog against the previous snapshot and tag each product as new, removed, unchanged, price changed, variant changed, or sale changed.

That diff gives the team a practical report:

products launched this week
products removed from the catalog
products newly discounted
products with deeper discounts
products back in stock
products that sold out
collections that changed

This is much easier to read than a full catalog export. A 2,000-product competitor catalog is not an insight. A list of 37 changed products is.

Keep product handles as stable identifiers. Titles change for merchandising reasons, but handles are usually more stable. When handles disappear, that often means a product was removed or replaced.

Example workflow

For a skincare brand, monitor five competitor stores and focus on the "new arrivals," "best sellers," and "sale" collections. The weekly report would show which products launched, which bundles appeared, and which items received discounts.

For a fashion brand, variant availability matters more. If a competitor keeps selling out of medium sizes in a specific collection, that may indicate demand for a silhouette or color trend.

For a supplements brand, tags and product types are useful. Competitors often expose product positioning through tags like sleep, recovery, focus, energy, or vegan.

What to add next

The next layer is alerting. A Slack alert when a competitor launches a new product is useful, but a smarter alert is better: notify only when a new product appears in a tracked collection, has a discount above 20%, or matches a keyword such as bundle, subscription, starter kit, or limited edition.

It's also worth connecting catalog changes to ad monitoring. If a new Shopify product appears and Facebook ads for the same product launch two days later, that tells a much clearer story than either dataset alone.

Production notes

Run discovery and tracking separately. First, crawl the whole store to understand catalog structure. Then monitor specific stores or collections on a schedule.

Watch variants. A product can stay live while important sizes or colors sell out. Variant availability often matters more than product availability.

Track collections. Collection membership reveals merchandising. A product moving into "best sellers" or "new arrivals" can be a stronger signal than a price change.

Normalize currencies downstream. If you track international Shopify stores, keep original currency fields and convert later.

Do not scrape checkout or private customer data. Public catalog monitoring is enough for competitive intelligence.

SKU-level monitoring alone misses strategy. Watching individual products catches price changes but misses strategy. Collection-level scraping shows launches, merchandising changes, and sale timing, which usually matters more than one product moving by two dollars.

Cost comparison

Approach	What it catches	Weakness
Manual browsing	Obvious price changes	Misses launches and variants
Price intelligence SaaS	Known SKU monitoring	Can be expensive and rigid
Shopify Scraper Pro on Apify	Catalog, price, variant, collection changes	Requires your own reporting layer

Check Shopify Scraper Pro's Actor page for the current pricing model before running at scale — Apify Actor pricing can change.

FAQ

Can Apify scrape Shopify stores?

Yes. Shopify Scraper Pro extracts structured public catalog data from Shopify-powered stores.

Can I monitor product price changes over time?

Yes. Schedule runs and compare current prices against the previous dataset.

Do I need login credentials?

No. This workflow uses public storefront catalog data.

What's the fastest way to try this?

Pick three competitor Shopify stores, run Shopify Scraper Pro with a small limit first, then export two weekly snapshots and diff them.

Try it yourself

Pick three competitor Shopify stores. Run Shopify Scraper Pro with a small limit first, then expand to the full catalog. Export two weekly snapshots. The changed rows will show you new products, discounts, availability shifts, and catalog strategy without another SaaS subscription.

I had never read my own Actors the way an agent reads them

Ai-Q Labs — Fri, 11 Sep 2026 05:18:24 +0000

I have 23 Actors on the Apify Store. I have read their Console input forms hundreds of times - every
field label, every hint, every checkbox, because I wrote them and then fixed them and then fixed them
again.

I had never once read the tool definition an AI agent receives when it calls them through the
Apify MCP server.

This is what I kept looking at. It is not what my callers see.

Last week I did. Of the 246 field descriptions I have written across those 23 Actors, 53 arrive at
the agent exactly as I typed them. That is 21.5%. The rest are altered on the way out.

Most of those alterations are documented, sensible, and probably improvements. That is not the point.
The point is that I shipped 23 tools without ever looking at what my callers actually receive, and
when I finally looked, I found three things the docs do not mention and one mistake that was entirely
mine.

Here is how to look, and what I found.

Reading your own tool definitions

The Apify MCP server speaks Streamable HTTP, the transport defined by the
Model Context Protocol. You post JSON-RPC to it, you get
tools/list back, and that response is the ground truth for what an agent sees.

The first trap costs a minute: the endpoint is /, not /mcp. Post to /mcp and the server
tells you off in plain English:

There is nothing at route POST /mcp?actors=... This Model Context Protocol (MCP) server supports the Streamable HTTP transport.

The second trap costs a little more: notifications/initialized answers with an empty body, so a
naive JSON.parse on every response throws. I wrote that bug while writing this article.

// Read your own Actors the way an AI agent reads them.
// Usage: APIFY_TOKEN=... node read-my-tools.mjs aiqlabs/sitemap-checker aiqlabs/pdf-inspector

const token = process.env.APIFY_TOKEN;
const actors = process.argv.slice(2).join(',');
const url = `https://mcp.apify.com/?token=${token}&actors=${encodeURIComponent(actors)}`;

async function rpc(body, sessionId) {
    const headers = { 'content-type': 'application/json', accept: 'application/json, text/event-stream' };
    if (sessionId) headers['mcp-session-id'] = sessionId;
    const res = await fetch(url, { method: 'POST', headers, body: JSON.stringify(body) });
    const sid = res.headers.get('mcp-session-id') ?? sessionId;
    const text = await res.text();
    if (!text.trim()) return { sid, payload: null }; // notifications answer 202 with no body
    const line = text.split('\n').find((l) => l.startsWith('data:'));
    return { sid, payload: JSON.parse(line ? line.slice(5) : text) };
}

const init = await rpc({
    jsonrpc: '2.0', id: 1, method: 'initialize',
    params: { protocolVersion: '2024-11-05', capabilities: {}, clientInfo: { name: 'read-my-tools', version: '1.0' } },
});
await rpc({ jsonrpc: '2.0', method: 'notifications/initialized' }, init.sid);
const { payload } = await rpc({ jsonrpc: '2.0', id: 2, method: 'tools/list' }, init.sid);

Point it at three of mine - say sitemap-checker and two
others - and it prints:

aiqlabs--github-repository-audit  (490 chars, 16 params)
  injected by the server : waitSecs
  prefill left in schema : repos, packages
  truncated at 500 chars : repos

aiqlabs--sitemap-checker  (421 chars, 12 params)
  injected by the server : waitSecs
  prefill left in schema : domains

Three lines of output, three things worth knowing. Take them in order.

The budget nobody tells you about

The description an agent reads is not your description. It is this:

This tool calls the Actor "<user>/<slug>" and retrieves its output results.
Use this tool instead of the "call-actor" if user requests this specific Actor.
Actor description: <your description>

The README states that descriptions are truncated at 500 characters (MAX_DESCRIPTION_LENGTH). That
limit applies to the whole string above, preamble included - so what you actually get is 500 minus
the preamble.

And the preamble is not a fixed cost. Your Actor's full name sits inside it, which means a longer
slug buys you a shorter description:

budget = 338 − length("<user>/<slug>")

Across my 23 that works out to 303 to 317 characters. domain-availability-checker - my longest
slug at 35 characters - gets 303. pdf-inspector gets 317. Nothing anywhere told me that naming an
Actor spends part of its description.

None of mine are over. But the margins are thinner than I would have guessed: my tightest is
github-repository-audit at 10 characters of headroom, then hacker-news-link-rot at 11 and
chrome-extension-audit at 12. Median description across the 23 is 278. Three of them are one
average sentence away from being delivered with the last clause replaced by ....

I found the real numbers late, and only because I checked. My first draft of this section said the
preamble was "174 characters, fixed" and the budget was 326 - measured off a single Actor, which
happened to be one of the longest-named ones, and then stated as a constant.

What the server changes, in numbers

Across 23 Actors and 246 author-written fields:

Change	Fields	Documented
`Example values: ...` appended	188	yes
`items.description` generated where I wrote none	36	no
`prefill` kept in the exposed JSON Schema	33	no
`waitSecs` parameter added	23	no
`REQUIRED` prefix added	17	yes
`Possible values: ...` appended	8	yes
description truncated at 500 chars	1	yes

Zero of my 246 fields had an empty description, so none of this is blanks being filled in. It is text
I wrote being changed.

The documented half is in the server's README and it is
worth reading once:

Descriptions are truncated to 500 characters (as defined in MAX_DESCRIPTION_LENGTH).

Required fields are explicitly marked with a REQUIRED prefix in their descriptions for compatibility with frameworks that may not handle the JSON schema properly.

I checked the second claim rather than trusting it: 17 fields are declared required in my schemas,
and all 17 carry the prefix in the tool definition. It does what it says.

The truncation that undoes itself

Exactly one of my 246 fields is longer than 500 characters: the repos field on
github-repository-audit, at 624. Its top-level description arrives cut, with the last 124 characters
replaced by ....

Those 124 characters read:

...they could not be checked, not because they came back clean. Give the package name in Packages instead if you need those three.

That is not decoration. That sentence exists to stop a caller reading a null as a clean result - the
single misreading this Actor was built to prevent. Truncation ate exactly the warning.

Except the agent still gets it. The server had also generated an items.description for that array
field - something I never wrote - containing the full 624 characters, byte for byte. A documented
truncation was quietly cancelled by an undocumented generation.

I want to be precise about how much this shows. One field in 246 was long enough to test it. I am
reporting a mechanism, not a rate. But the mechanism is worth knowing in both directions: if your long
description sits on an array, the full text survives in items; if it sits on a string, nothing
catches it.

Where does items.description come from? Of my 39 array fields, 36 got one. Twenty-four are exact
copies of the parent description, eleven are the parent plus the **REQUIRED** prefix, and one is a
structural expansion of editor: requestListSources into its url / method / payload / headers
shape. The three that got nothing all have an enum on their items.

That last case caught me out while I was measuring. I first recorded it as "my 138-character
description was replaced by a 19-character string" and nearly published that. It was wrong: the parent
description arrives intact, and the 19 characters are the title of the expanded object type. Nothing
was lost. I mention it because it is the kind of error that reads perfectly well in a draft.

The parameter you did not write

Every one of my 23 tools carries a parameter I have never declared:

{
  "type": "integer", "minimum": 0, "maximum": 45, "default": 30,
  "description": "Max seconds (0–45, default 30) to cap the wait for the Actor run to reach terminal state. For long-running Actors the response returns at the cap with the current run status; follow `nextStep` to poll via get-actor-run. Set to 0 to fire-and-forget."
}

It is on the server's own tools too - get-actor-run carries it - so this is a server-wide
convention rather than something done to Actors specifically.

I am not reporting this as a defect. The description tells the agent precisely how to recover, which
is more than most timeouts do. I am reporting it because an author who has never read their tool
definition does not know their Actor now has a 45-second contract with its callers. If your Actor
takes four minutes on a realistic input, every agent call returns before it finishes and the caller's
experience depends on whether their client follows nextStep. That is a design constraint on your
Actor, and it arrived without you.

One key, two layers

prefill survives into the exposed JSON Schema on 33 of my fields. It is not a
JSON Schema keyword; nothing in
the spec says what a consumer should do with it. An agent reading the schema sees a value
sitting next to my field and can reasonably read it as a suggestion.

I have met this key before. In an earlier article I traced a run that returned four rows when I had
asked about one repository: the platform was treating my prefill as a default and injecting it
into every call that omitted the field. Same key, different layer, different behaviour - and neither
behaviour is one I asked for when I filled in a Console form hint.

Then I let an agent choose between all 23

Reading the definitions tells you what arrives. It does not tell you whether the catalog works. So I
ran a second trial: give an agent all 23 descriptions and one realistic task, and see which tool it
reaches for.

I wrote seven tasks, fixed the intended answer for each one in a file before launching anything,
and wrote down what I expected to happen. Two independent agents per task, fourteen trials, no Actor
executed.

I expected failures. My catalog has obvious traps in it - three PDF tools, three tools that all touch
broken links, two that both check domains.

Fourteen out of fourteen picked the tool I intended. Three of my seven predictions were wrong, and
every one of them was wrong in the same direction: the agents were more careful than I gave them
credit for. Task 7 is a fair example. I expected http-status-checker to pull votes away from
dead-link-checker; instead both agents explained, unprompted, that a status checker can only test
URLs you already have and therefore cannot recover the URLs of a blog that moved two years ago.

So the selection layer was not the story. This was:

Confidence fell to medium in exactly 4 of the 14 trials, and a tie was declared in exactly those
same 4. The other ten were unanimous, high, and tie-free. Those four trials are the two pairs
where I had written overlapping descriptions.

Here is one of them, in my own words, from the live catalog:

seo-ai-visibility-auditor - "classic SEO health and for AI/LLM discoverability - AI crawler access in robots.txt, llms.txt, structured data, metadata"
seo-audit-tool - "on-page and technical SEO, and adds four checks general auditors skip: AI crawler access in robots.txt, llms.txt, redirects that drop the path, and noindex sent via X-Robots-Tag"

The phrase "AI crawler access in robots.txt, llms.txt" appears verbatim in both. I wrote both
descriptions, months apart, and had never once read them next to each other.

Asked "is my robots.txt blocking AI crawlers like GPTBot", both agents picked
seo-ai-visibility-auditor, and both told me why the pick was thin:

both entries explicitly list "AI crawler access in robots.txt" as a check, so either would answer
the question; I picked seo-ai-visibility-auditor because AI/LLM discoverability is its stated
primary purpose rather than one of four add-on checks. Neither description names GPTBot
specifically, so the match rests on the "AI crawler access in robots.txt" wording alone.

That is a correct answer arrived at by elimination, with the reasoning shown and the weakness
declared. The agent did the honest thing. The duplication it had to route around is mine.

What I wanted to conclude here was: the signal is the confidence, not the choice - every medium
landed on a real duplication, so watch where the agent hesitates. I had the sentence written. Then I
measured it and had to take it out. That is the next section.

Two caveats regardless. n is 2 per task, all Claude, all in one harness - this is a check on my
catalog, not evidence about agents in general. And the agents saw name and description only; a real
client also reads the input schema, which might break a tie my prose leaves open.

Fixing it, and measuring the fix the same way

The two SEO Actors were never duplicates in the code. One takes a starting URL and crawls
(startUrls, crawlSite, maxPages); the other takes a list and audits it page by page (urls,
maxUrls). They are genuinely different tools. My descriptions simply never said so - they both led
with the checks, and the checks overlap.

So I rewrote both to lead with the input shape, and to name each other:

seo-ai-visibility-auditor - "...Give it a domain to crawl - for a fixed list of pages, use SEO Audit Tool instead."
seo-audit-tool - "Give it a list of URLs... One row per URL - to crawl a whole site instead, use SEO & AI Visibility Auditor."

Then I pulled tools/list again to confirm the agent-facing text had actually changed - no tool in
the catalog still carries the shared phrase - and ran the same task past three fresh agents. All
three: high, no tie. One explained it without being asked:

aiqlabs/seo-audit-tool also covers "AI crawler rules" but is scoped to a supplied list of URLs, and
both descriptions explicitly cross-reference each other to resolve that split, so this is not a tie.

Clean result. I nearly stopped there.

The control that ruined it

I also re-ran task 4 - the domain pair - as a control. I had not touched either of those two
descriptions.

pair	edited?	before	after
SEO	✅ yes	medium ×2, tie ×2	high ×3, tie none
domain	❌ no	medium ×2, tie ×2	high ×2, tie none

The pair I did not fix improved exactly as much as the pair I did. So the confidence change cannot be
attributed to my rewrite, and the sentence I wanted to publish - watch where the agent hesitates -
does not survive its own control.

The cause is a flaw in my design, not a mystery. My before and after prompts were not identical. The
before agents were told "answer in exactly this structure and nothing else"; they produced answers and
then sat on them, and I had to chase them with a second message asking them to actually send it. The
after agents had the delivery instruction from the start. One changed variable, sitting right next to
the one I was trying to measure.

What survives is narrower and still worth having. The duplication was real: the same clause, verbatim,
in two live descriptions I wrote months apart. Four of fourteen before-trials declared a tie and all
four landed on the two pairs where my descriptions genuinely overlapped - no false alarms among the
other ten. The fix shipped, and the shared phrase is gone from what agents receive. What I cannot tell
you is whether fixing it changed anything measurable, because I broke my own instrument while using it.

If I had measured only the pair I edited - which was my plan until I added the control as an
afterthought - I would have published a causal claim resting on a changed prompt.

One more honest note on that edit. While writing the new description I listed "soft 404s answering 200" as
one of the four checks. The Actor does detect those - there is a soft_404 code in its source - but
my own README defines the four as AI crawler rules, /llms.txt, path-dropping redirects, and
X-Robots-Tag noindex. I had quietly swapped one out. I caught it by reading the README before
publishing rather than after, which is the only reason it is a footnote instead of a correction.

What I changed

Four things, none of them large:

I compute the budget per Actor - 338 − length("<user>/<slug>") - instead of assuming one number. Three of mine sit within 12 characters of losing a sentence.
No field description over 500 characters unless the field is an array. On an array, items catches the overflow. On a string, it does not.
tools/list before publish. It takes ten seconds and it is the only view of your Actor that your non-human callers actually have.
Read your descriptions as a set, not one at a time. This is the one that actually cost me something. I wrote two SEO Actors months apart, gave them the same clause, and never once put the two sentences side by side. A catalog is a document; mine was 23 documents that had never met.

The wider point is smaller than a bug and more annoying than one. I built 23 tools for AI agents to
call and spent all of my review time in a form built for humans. The one interface my actual callers
use, I had never opened.

prefill and default look the same in Console. Only one of them lets an agent skip your required field.

Ai-Q Labs — Fri, 11 Sep 2026 05:18:13 +0000

I have 23 audit Actors on the Apify Store. They all do a version of the same thing: take a public record, check whether what it still claims is true, and write the verdict into a dataset.

Last week I set out to prove they were unsafe for AI agents to chain together. I had a specific accusation in mind, I built the experiment to demonstrate it, and the experiment refused. What I found instead was a one-line schema fix that I have now applied, and a distinction between two input-schema settings that I had treated as interchangeable for months.

The accusation I was going to make

Chaining means running Actor A, taking something out of its output, and passing it into Actor B. For an agent to do that through the Apify MCP server, B has to be callable with a value that only A can supply. MCP is the Model Context Protocol, the interface that exposes your Actor to clients like Claude and Cursor as a tool they can call on their own.

So I read my own catalogue with a script instead of from memory. Of my 23 published Actors, 9 declare a URL or a domain in their input schema's required array. Counting every ordered pair where A emits the kind of identifier B requires, I get 61 possible chains: 36 joined by a URL, 25 by a domain.

A tenth Actor requires an identifier too, and it is the interesting one. dataset-to-github-issues takes a dataset ID and a repository name. Nothing else in my catalogue emits either, so no agent can ever reach it from another Actor's output - it is a second stage with no possible first stage. I had not noticed that until the script told me.

Then I looked at what those 9 do when the value never arrives.

I had swept my catalogue for target-naming defaults once before and found them, so the count itself was not the surprise. Restricting it to the Actors that can actually be a second stage sharpens it: seven of the nine carry a default that names a concrete target.

Actor	field	default
`bulk-domain-checker`	`domains`	`["apify.com"]`
`dead-link-checker`	`domains`	`["docs.apify.com"]`
`domain-availability-checker`	`domains`	`["apify.com"]`
`http-status-checker`	`urls`	`["https://apify.com/store"]`
`seo-ai-visibility-auditor`	`startUrls`	`[{"url":"https://apify.com"}]`
`sitemap-checker`	`domains`	`["apify.com"]`
`tech-stack-detector`	`domains`	`["apify.com"]`

All seven are also listed in required. I had assumed that word did some work.

My accusation was going to be this: in a two-stage workflow, an agent that drops the value between stages gets no error. The second Actor runs on apify.com, returns a well-formed dataset, and the agent reports a confident answer about a website the user never asked about. A wrong answer with no failure attached to it.

First, the part that turned out to be true

Before involving any agent, I called two of my own Actors with an empty input from the command line.

The first, seo-audit-tool, has urls in required and no default:

apify call aiqlabs/seo-audit-tool --input '{}'

Run: Calling Actor aiqlabs/seo-audit-tool (ZjU3YtyaWgqpNriNj)

Error: Input is not valid: Field input.urls is required

No run was created. No dataset, no compute, no partial result to misread. The platform named the missing field and stopped.

The second, http-status-checker, has urls in required and a default:

apify call aiqlabs/http-status-checker --input '{}'

INFO  Checking 1 URL(s) with 8 in flight. HTML analysis: on.
INFO  Done. 1 URL(s): 0 error, 0 warning, 0 info, 1 clean.

Apify call result: SUCCEEDED
Run ID: sMLok9a5KMSSdtnD1

I sent nothing. It audited something. Reading the input the platform stored for that run shows what:

{"urls":["https://apify.com/store"],"analyzeHtml":true,"respectRobotsTxt":true,
 "robotsAgent":"Googlebot","onlyIssues":false,"maxConcurrency":8,
 "requestTimeoutSecs":20,"maxRedirects":8,"slowResponseMs":3000,"maxUrls":10000}

Same platform, same account, same minute, same field name, both fields marked required. One call was refused and one was silently completed. The only difference is the default.

There is a second thing in that pair worth pulling out. seo-audit-tool does carry a prefill on urls - two example URLs that appear in the Console form. It was still rejected. prefill populates the form for a human; default is substituted for whoever omits the field.

The input schema specification is not vague about this. On prefill:

this field is only used in the user interface but does not affect the Actor functionality and API

and, in the same paragraph:

the Prefill value won't be used by existing integrations such as Actor tasks or API calls, but the Default will be if specified

On default, it names the callers explicitly: the platform passes the value when the user omits it "via any means (API, CLI, scheduler, or user interface)". An agent calling through the MCP server is one of those means.

I had read that page. I read it as a description of two features and used both, on the same field, for months.

There is a detail in http-status-checker that makes the gap concrete. The two settings on urls do not even hold the same value:

"urls": {
  "prefill": ["https://apify.com/store",
              "https://apify.com/this-page-does-not-exist",
              "http://apify.com/"],
  "default": ["https://apify.com/store"]
}

Three URLs for the form, one URL for anyone who omits the field. I wrote both, months apart, and never put them side by side. A caller that skips urls does not get the demo I built - it gets a shorter thing I stopped thinking about, and no part of the interface ever shows the two together.

Then the accusation collapsed

To test the chain I needed a question that one Actor cannot answer on its own. That took two attempts.

My first idea was sitemap-checker into http-status-checker: enumerate a site's sitemap, then check the URLs. It is a bad example, because sitemap-checker has checkUrlStatus set to true by default and already requests every URL. My second idea was hacker-news-link-rot into domain-availability-checker, and that fails the same way - the first Actor already emits domainRegistrationStatus.

That is worth pausing on. I had built every one of these Actors to answer a question completely, which is right for a standalone Store listing and is exactly what makes them poor chain stages. Composable and self-contained are not the same design, and I had only ever aimed at one of them.

The question I settled on works because it needs HTML analysis, which the sitemap Actor does not do:

Among the URLs listed in crawlee.dev's sitemap, which ones are marked noindex, and which ones are soft 404s?

A noindex page is one that tells search engines to keep it out of the index; a soft 404 is a missing page that returns 200 anyway. Both need the HTML of the page. sitemap-checker emits 11 fields and none of them is either. http-status-checker has both, behind an analyzeHtml switch that is on by default. The chain is necessary, and crawlee.dev is a real documentation site with a large sitemap, which makes it a fair test.

I gave that question, word for word, to four fresh agents with both Actors exposed through the Apify MCP server, and no other instructions. I did not tell them to be careful, and I did not mention inputs or defaults - in an earlier experiment I learned that asking an agent to count its rows is the same as telling it where to look.

Then I ignored what they told me and read what the platform stored.

stage-2 run	`urls` passed	first entry
`p5qwOESMQB0Vwj0ls`	500	`https://crawlee.dev/js/api/3.11/playwright-crawler/...`
`NPM7mW5oC848wn2ZL`	174	`https://crawlee.dev/js/api/3.16/puppeteer-crawler/...`
`3fO6vTbKnLlkDwGnw`	500	`https://crawlee.dev/blog`
`H0HkakmGE2bfkAdWU`	400	`https://crawlee.dev/js/api/3.15/core/interface/HttpResponse`
`2cJHKXtsrapw9aikV`	400	`https://crawlee.dev/js/api/3.14/browser-pool/class/PuppeteerPlugin`
`oGbGCrfPEM4uDuOBY`	400	`https://crawlee.dev/js/api/3.12/puppeteer-crawler`
`sDQnaUb7NxX9iz88B`	400	`https://crawlee.dev/js/api/3.11/core/class/SystemStatus`
`QDpb6PG685ptVthUN`	400	`https://crawlee.dev/blog`
`bB20ynAk9ymZafaiY`	724	`https://crawlee.dev/js/api/3.14/utils/function/downloadListOfUrls`

Nine stage-2 runs. Every one carries real crawlee.dev URLs. Not one carries https://apify.com/store. They also batched - 174 to 724 URLs per call - rather than making one call per URL.

The trap I had spent a day setting never closed.

Why it never closed

The mechanism is in the result once you stop arguing with it. A default stands in for a value the caller never had. Inside a chain, the agent produced that value itself one step earlier and is still holding it. The omission the default is waiting for does not occur.

That reconciles this with the case where the same defect does bite. When an agent calls one of these Actors cold - no preceding step, nothing in hand - the missing field is real, the platform fills it, and the run proceeds on a target nobody chose. The danger lives at the entry point of a workflow, not in the joins.

I would rather publish that than the article I planned, because it changes where you should spend attention. If you are worried about agents mangling your Actor, the risk is concentrated in the first call, not in the handoffs.

What is wrong with my own experiment

A reviewer would find this, so I will say it first.

The question I picked has a null answer. Here is the summary record from one 500-URL batch:

{"checked":500,"severity":{"ok":500,"info":0,"warning":0,"error":0},
 "headline":{"broken":0,"soft404s":0,"noindexOnLivePages":0,"metaRefreshRedirects":0}}

crawlee.dev has no soft 404s and no noindex pages in those URLs. "Which ones are noindex?" answers to "none". An agent that had quietly audited apify.com/store instead would probably also have answered "none".

So the agents' answers cannot tell a careful run from a careless one here. Only the stored inputs can. That is the one thing I got right by accident of habit: I built the measurement on records the platform keeps, not on what the agents said about themselves. It mattered more than I expected - all four agents finished without returning a report to me at all, and every number above survived that because none of them came from an agent.

A stronger version of this trial would use a site with known noindex pages, so a substituted target produces a different answer rather than the same one. If you repeat this, pick your target site for that property.

The fix, and what it costs

For any input field that names what to work on - a URL, a domain, a repository, an account - delete the default and keep the prefill.

"urls": {
  "title": "URLs",
  "type": "array",
  "editor": "stringList",
  "prefill": ["https://apify.com/store"]
}

with "required": ["urls"] alongside it. The Console form still opens with an example filled in, so nothing changes for a human clicking through your Actor. What changes is that a caller who omits the field now gets Field input.urls is required instead of a clean dataset about somebody else's website.

The specification anticipated this too. Its guidance for required reads:

Use for fields that don't have a reasonable default and MUST be entered by the user (e.g., API token, password)

Fields that don't have a reasonable default. Seven of mine had both, which is a combination the sentence quietly rules out. Marking a field required and then supplying the answer yourself is not a stricter setting; it is two instructions that cancel.

Keep default for settings that describe how to work - concurrency, timeouts, depth limits, feature switches. Those are genuinely safe to assume, and an empty call that inherits them is behaving correctly.

The distinction is worth stating as a rule: default on a method is a convenience; default on a subject is a substitution.

Checking your own Actors

I wrote the catalogue sweep as a script so I would stop trusting my memory of my own schemas. It reads every local .actor/ directory, classifies each input and output field into an identifier kind, and reports which of your Actors can be a chain stage and which of them will accept an empty call:

node composability.mjs           # summary and pair matrix
node composability.mjs --rules   # print the classifier so you can audit it

The --rules flag exists because the classifier is regular expressions over field names, and a regex table you cannot see is a number you should not trust. Mine misfired the first time I ran it: it reported that github-repository-audit requires no identifier, which is nonsense for an Actor that audits repositories. The cause was real and boring - that schema has no required key at all, so every field is optional and the defaults decide everything.

If you want a single check with no script, call your own Actor with an empty input and see what happens:

apify call <your-actor> --input '{}'

If it succeeds, look at what it audited. That is what an agent gets when it forgets.

What I changed and what I did not

I have not stripped the defaults from all seven Actors yet. The measurement says the harm is concentrated in cold calls, and I want to measure that path on these specific Actors before I change seven live listings - the last time I acted on a confident prediction here, the prediction was wrong twice.

What I have changed is the rule I apply to new fields, and the check I run before publishing. Both of them came out of an experiment that failed to prove its own thesis, which is becoming a habit I am not entirely unhappy about.

Local market intelligence: Google Maps, reviews, and news with Apify

Crawler Bros — Thu, 10 Sep 2026 06:30:03 +0000

Local market intelligence: Google Maps, reviews, and news with Apify

Local competitors leave useful traces everywhere: map rankings, review velocity, star ratings, local press, opening hours, categories, photos, and website links.

The problem is that those signals rarely live in one clean report. When a client asks, "Which dental clinics are gaining ground in Austin?", the answer is scattered across Google Maps, review pages, news results, and local search results.

Local market intelligence is the practice of pulling those scattered public signals — Google Maps listings, customer reviews, and local press coverage — into one structured, weekly dataset an agency, franchise operator, or founder can use for decisions, instead of piecing it together from browser tabs and screenshots.

Quick answer

You can build a local market intelligence pipeline on Apify by combining a Google Maps scraper for business discovery, a reviews scraper for customer sentiment, and a Google News scraper for local press mentions. CrawlerBros Actors return structured datasets that can be exported to Google Sheets, BI tools, or an internal CRM.

The local intelligence stack

The stack combines three CrawlerBros Actors:

Google Maps Scraper for business discovery, ratings, locations, categories, phone numbers, websites, and coordinates.
Google Maps Reviews Scraper for review text, ratings, dates, reviewer metadata, and owner response tracking.
Google News Scraper for brand mentions, competitor mentions, article snippets, full-text extraction, and publication metadata.

The data collection runs in three passes:

\Google Maps search -> competitor and location dataset Google Maps reviews -> customer complaint and praise dataset Google News search -> local press and reputation dataset \\

For a local SEO agency, this gives one repeatable view of a market. Instead of asking an analyst to search "plumber Dallas", open every competitor, copy ratings, and read reviews manually, the Actor run produces a structured export.

Key facts

3 Actors, 3 passes: Google Maps Scraper for discovery, Google Maps Reviews Scraper for sentiment, Google News Scraper for reputation.
Maps search caps around 120 results per query: keep queries narrow (category + neighborhood) to avoid noisy, unusable output.
Ratings tell you who's winning; reviews tell you why: a 4.8-rating business with 200 reviews and a 4.2-rating business with 4,000 reviews are different competitive stories.
Narrow queries beat broad ones: "vegan restaurants in Williamsburg" produces a decision-ready dataset; "restaurants in New York" produces noise.

Actor configuration that matters

The Google Maps Scraper supports two modes: search for a text query and placeUrls for direct Google Maps place URLs. For broad market discovery, start with searchQuery, location, maxResults, and language. The public input schema notes that search mode typically returns up to 120 places for a query, so keep queries narrow enough to avoid noisy results.

A small first run looks like this:

\json { "mode": "search", "searchQuery": "orthodontists", "location": "Austin, TX", "maxResults": 50, "language": "en", "proxyConfiguration": { "useApifyProxy": true } } \\

For Google News, the important fields are queries, maxResultsPerQuery, language, country, dateRange, optional siteFilter, optional excludeWords, and extractFullText. Keep extractFullText off during discovery and enable it only for the articles you actually want to analyze.

What the output looks like

The reason this workflow works is that every stage produces rows you can join later. For the Maps stage, the fields that matter are name, category, rating, reviewsCount, address, website, phone, location, and googleMapsUrl.

The review and news stages add the context the base listing cannot explain. Join by business name, website domain, and location, then keep the original URLs beside every insight. That way, when a client asks why you flagged a competitor, you can point back to the review or article instead of hand-waving from a summary.

In a small test run, Google Maps Scraper was asked for five Austin coffee shops. The Actor returned five rows with name, totalScore, reviewsCount, categoryName, address, phone, website, priceLevel, location, placeId, serviceOptions, hours, ratingHistogram, popularTimesSummary, photosSample, googleMapsUrl, and scrapedAt. That's enough for the first market snapshot before you even scrape reviews.

Why Google Maps is the starting point

Google Maps is usually the best first source for local market research because it captures the businesses that customers actually discover. A local competitor might have a weak website and still dominate discovery because its Maps profile has 2,000 reviews and strong proximity coverage.

The Google Maps Scraper gives you the base table:

business name
category
address
phone number
website
rating
review count
Google Maps URL
latitude and longitude

From there, sort competitors by review volume, distance, category, or rating. For agencies, this is useful before a sales call. For franchise teams, it helps compare location density across neighborhoods.

Reviews turn rankings into explanations

Ratings tell you who is winning. Reviews help explain why.

A restaurant with a 4.8 rating and 200 reviews is different from a restaurant with a 4.2 rating and 4,000 reviews. The second business may still dominate demand because of location, brand awareness, or delivery coverage. The review text shows the operational pattern behind the score.

Use the Google Maps Reviews Scraper to pull recent reviews from each competitor, then group comments into recurring themes:

slow service
pricing complaints
staff mentions
cleanliness
appointment availability
product quality
parking or location issues

This is where the dataset becomes useful. A local dentist can see that competitors get repeated complaints about long wait times. A gym can see that customers praise flexible hours. A home services company can spot which competitors are praised for same-day availability.

News adds reputation context

The Google News Scraper completes the picture. Local businesses often show up in news for openings, lawsuits, awards, acquisitions, health inspections, funding, or community events.

The Actor supports keyword searches, date filters, country and language settings, optional full-text extraction, and deduplication across queries. That means you can search for:

the client brand
competitor names
category keywords
neighborhood + service terms

The output gives article title, source, URL, publication date, snippet, and optionally full article text. News volume isn't a perfect popularity metric, but it's a useful reputation signal.

Production notes

Keep the market definition tight. "Restaurants in New York" produces noise. "Vegan restaurants in Williamsburg" produces a dataset someone can act on.

Store historical snapshots. A single scrape is useful. Weekly snapshots are better. Review count growth, rating changes, and new competitor appearances matter more over time.

Separate business discovery from review scraping. Run Maps discovery first, inspect the dataset, then pass selected URLs into the reviews stage. That keeps review scraping focused on relevant competitors.

Use review text carefully. Reviews can include personal data. Collect only the fields needed for analysis and avoid unnecessary storage of sensitive reviewer details.

Broad queries return datasets, not decisions. An early test query of "restaurants in Chicago" returned a dataset, but not a decision. Narrowing the search to a category, neighborhood, and use case made the output useful — a smaller, sharper input usually beats a giant scrape.

Cost comparison

Approach	Monthly cost	Freshness	Custom workflow	Raw data ownership
Manual research	Analyst time	Inconsistent	Yes	Spreadsheet only
Local SEO SaaS	$50-$300+	Varies	Limited	Limited export
CrawlerBros on Apify	Pay per run	On demand	Full	Yes

For the Apify row, check the current Pricing tab on each Actor page before running at scale. These Actors may be priced by compute usage, result/event count, or a platform pricing model that changes over time — the point is you only pay for the runs you need and keep the raw dataset.

FAQ

Can Apify scrape local business data from Google Maps?

Yes. CrawlerBros provides Google Maps Actors that extract structured business and review data from public Google Maps pages.

What can I do with local market intelligence data?

You can monitor competitors, find weak review themes, prioritize sales outreach, compare locations, and feed local SEO dashboards.

Do I need to code this workflow?

No. You can run each Actor from Apify Console and export the datasets. Developers can also automate the workflow with the Apify API.

What's the fastest way to try this?

Start with a small query such as "orthodontists in Austin," run Google Maps Scraper, pick 10 competitors, then run the reviews and news stages.

Try it yourself

Start with a small query such as "orthodontists in Austin" or "coffee shops in Denver." Run Google Maps Scraper, choose 10 competitors, then run Google Maps Reviews Scraper and Google News Scraper. You'll know very quickly whether the market is crowded, who owns attention, and what customers are saying.