DEV Community: KazKN

LoopNet vs Crexi: Which Commercial Real Estate Platform Is Better?

KazKN — Mon, 08 Jun 2026 22:08:35 +0000

CRE Listing Intelligence Series

Best CoStar alternatives for small CRE brokers
How to scrape LoopNet and Crexi listings into one CRE dataset
Commercial real estate API for brokers: cheap CoStar alternative
I Pulled 1,234 Dallas CRE Listings from LoopNet + Crexi. Deduping Was the Real Problem.
LoopNet vs Crexi: Which Commercial Real Estate Platform Is Better?

If you work in commercial real estate, you have probably asked this question:

Should I use LoopNet or Crexi?

The short answer is:

LoopNet is usually stronger for broad listing exposure and search visibility. Crexi is often stronger for a more modern deal-discovery workflow, auction/listing tools, and active investor browsing.

But for commercial real estate research, the more useful answer is:

You often need both. The real problem is turning both platforms into one clean market file.

This article compares LoopNet and Crexi from a practical broker and analyst perspective: listing discovery, audience, cost, broker leads, data quality, and workflow.

Quick comparison

Category	LoopNet	Crexi
Best known for	Large commercial real estate marketplace and visibility	Modern CRE marketplace, sale/lease listings, auctions, investor discovery
Typical strength	Broad search reach and brand recognition	Faster-feeling marketplace workflow and deal browsing
Strong use case	Marketing listings to a large tenant/investor audience	Finding active opportunities and managing buyer interest
Common buyer/user	Brokers, owners, tenants, investors	Brokers, investors, buyers, tenants
Data workflow	Useful source, but not always analysis-ready as a raw export	Useful source, but still needs cleanup for market research
Best for research	Strong when combined with other sources	Strong when combined with other sources
Main limitation	Cost and workflow friction can be an issue depending on the user	Coverage and data completeness can vary by market
Best practical answer	Use it when visibility matters	Use it when deal discovery and speed matter

What LoopNet does well

LoopNet is one of the most recognized names in online commercial real estate listings.

It is part of CoStar Group, and CoStar describes LoopNet as a heavily trafficked online commercial real estate marketplace. LoopNet itself positions the site around commercial property for sale, lease, auctions, and businesses for sale.

That matters because CRE listing platforms are partly about data and partly about attention.

For a broker marketing a property, LoopNet's main advantage is simple:

More people know to search there.

That makes LoopNet especially relevant for:

office listings
retail listings
industrial listings
lease listings
sale listings
mainstream tenant and investor discovery
properties where broad online exposure matters

If the job is "make this property visible to as many relevant online searchers as possible," LoopNet is hard to ignore.

What Crexi does well

Crexi is also a commercial real estate marketplace, but it often feels different from LoopNet in day-to-day use.

Crexi positions itself as a platform for brokers, buyers, tenants, and investors. Its marketplace includes sale and lease listings, and Crexi also has auction and deal-management features.

In practice, Crexi can feel more natural for active deal browsing.

It is often useful when the job is:

find new investment opportunities
review broker-posted listings
browse sale and lease inventory
monitor auction-style opportunities
discover smaller or mid-market listings
compare a market beyond the largest incumbent platform

Crexi's biggest strength is not that it "replaces" LoopNet in every market.

Its strength is that it can show a different slice of the market.

That matters because CRE listing coverage is not perfectly uniform. A broker, investor, or analyst who checks only one portal may miss listings that appear elsewhere.

Listing coverage: the answer depends on the market

The most common mistake in a LoopNet vs Crexi comparison is trying to declare one universal winner.

CRE is too local for that.

In one market, LoopNet may have stronger inventory for a certain asset class. In another market, Crexi may show more relevant sale listings, more active broker pages, or better investor-facing deal flow.

The answer can change by:

city
asset class
sale vs lease
broker behavior
property size
paid listing strategy
whether the searcher is a tenant, broker, investor, or analyst

For example:

a tenant rep may care more about lease visibility
an acquisition analyst may care more about sale inventory
an investment sales broker may care more about buyer lead quality
a market researcher may care more about completeness and deduplication

That is why "LoopNet vs Crexi" is usually the wrong operational question.

The better question is:

Which platform gives me the cleanest view of this specific market?

And often, the answer is both.

Lead quality and broker workflow

LoopNet and Crexi can both help generate attention around listings.

But attention is not the same as usable broker intelligence.

A broker or analyst usually needs to know:

who represents the listing
which company is attached
whether the listing is new or stale
whether the same property is listed somewhere else
what price, rent, cap rate, NOI, or square footage is visible
whether the row is clean enough to send to a CRM or spreadsheet

This is where both platforms have the same limitation:

They are listing portals first. They are not always clean research databases for your internal workflow.

That does not make them bad.

It just means the workflow does not end on the portal page.

Cost: LoopNet vs Crexi

Cost is one of the reasons brokers compare LoopNet and Crexi in the first place.

LoopNet is connected to the CoStar ecosystem, and many CRE professionals perceive CoStar/LoopNet as expensive compared with lighter tools. Crexi is often discussed as a more accessible or flexible alternative, although pricing and value depend heavily on the product tier, market, and use case.

The important point is not simply "which one is cheaper?"

The better question is:

What job am I paying for?

If the job is listing exposure, the value depends on lead quality and closed business.

If the job is market research, the value depends on how quickly a team can build a usable market file.

For research, the hidden cost is usually manual work:

checking two tabs
copying listings into spreadsheets
cleaning addresses
removing duplicates
finding broker contacts
comparing cap rates
tracking days on market
exporting rows into a CRM

That hidden labor can cost more than the tool itself.

Data quality: the real comparison

For broker research, data quality is where the LoopNet vs Crexi question becomes interesting.

The same property can appear on both platforms with slightly different fields.

Example:

1200 East 6th Street
1200 E 6th St
1200 E. Sixth Street

Those may be the same property, but exact matching would miss it.

Square footage can also vary:

7,500 SF
7,480 SF
7.5K SF

Cap rates may be listed, missing, estimated, or implied from other fields.

Broker contact data can be present on one source and missing on another.

That is why the best research workflow is not just:

Search LoopNet
Search Crexi
Pick the better one

It is closer to:

Search LoopNet
Search Crexi
Merge both sources
Normalize fields
Deduplicate listings
Preserve source links
Export one clean market file

Which platform is better for CRE research?

If I had to answer directly:

LoopNet is better for broad visibility and familiar search behavior.
Crexi is better for active deal discovery and a modern marketplace workflow.
Using both is better for first-pass market research.

For a broker, the final answer depends on the job:

Job	Better fit
Advertise a listing to a large online audience	LoopNet
Browse active deal opportunities	Crexi
Compare public inventory across a market	Both
Build a broker lead list	Both
Track stale listings and days on market	Both, after cleanup
Create a CSV for underwriting or CRM	Both, after extraction and normalization

Where an extraction tool fits

This is where a data workflow can help.

I built Commercial Real Estate Brokerage Intel on Apify for this exact use case:

Run LoopNet + Crexi searches and turn the results into one structured dataset.

The point is not to declare one portal the winner.

The point is to make both sources easier to use together.

The actor is built to help with:

LoopNet + Crexi listing extraction
deduplication across platforms
normalized cap-rate fields
days-on-market context
broker contact fields where available
also_listed_on signals
CSV, Excel, JSON, and API export

So if your workflow is "which platform should I manually check today?", the answer might be LoopNet or Crexi.

But if your workflow is "I need one clean market file," the better answer may be:

Pull both, clean both, and compare the market from one dataset.

Final verdict

LoopNet vs Crexi is not a simple winner-takes-all comparison.

LoopNet usually wins on brand recognition, broad search behavior, and listing visibility.

Crexi often wins on deal-discovery workflow, marketplace feel, and alternative inventory discovery.

For commercial real estate brokers, investors, and analysts, the strongest workflow is often not choosing one platform.

It is using both platforms without letting duplicate listings, messy addresses, missing broker fields, and inconsistent cap rates slow the team down.

That is the real lesson:

The better platform is the one that helps you create a cleaner market view faster.

And in many CRE workflows, that market view comes from LoopNet and Crexi together.

Sources

The Fastest Vinted Scraper Workflow: Paste a Search URL, Export CSV

KazKN — Sat, 06 Jun 2026 02:13:23 +0000

There are many Vinted scrapers.

Most of them ask you to configure a form:

keyword
brand
category
size
condition
price range
country
sorting

That can be useful, but it is slow when you already have the perfect search open in your browser.

Vinted Turbo Scraper is different:

Paste one or multiple Vinted search URLs. Run the Actor. Export the listings.

That is the whole point.

No filter rebuilding. No heavy setup. No slow configuration flow.

Just the fastest path from a Vinted search page to a structured dataset.

⚡ Quick Answer

The fastest way to scrape Vinted search results is to use the search URL itself. Open Vinted, apply your filters, copy the catalog URL, paste it into Vinted Turbo Scraper, choose maxItems, and export the results as CSV, Excel, Google Sheets, JSON, or API data.

Why this Vinted scraper is different

Generic Vinted scrapers often start with configuration.

Vinted Turbo Scraper starts with the URL.

That matters because Vinted already stores your filters in the search URL.

Example:

https://www.vinted.fr/catalog?search_text=nike%20dunk&price_to=80&currency=EUR&order=newest_first

That URL can contain the market, keyword, price, currency, sort order, and often category, brand, size, color, and condition filters.

So instead of manually recreating your search inside an Actor input form, you use the search you already built on Vinted.

Turbo positioning

Vinted Turbo Scraper is not trying to be a complex research suite.

It is built for speed:

Workflow	Typical experience
Generic Vinted scraper	Configure filters one by one
Browser copy-paste	Manually collect listings
Custom script	Maintain code, proxies, pagination, exports
Vinted Turbo Scraper	Paste Vinted URL -> run -> export dataset

That is the product promise:

The fastest Vinted URL-to-CSV workflow on Apify.

Single URL or multiple URLs

The other big advantage is batch input.

You can paste:

one Vinted URL for a simple export
multiple URLs from the same market
multiple URLs across supported Vinted markets
broad searches for larger exports
niche searches for monitoring

Example:

https://www.vinted.fr/catalog?search_text=nike%20dunk&order=newest_first
https://www.vinted.fr/catalog?search_text=adidas%20samba&order=newest_first
https://www.vinted.fr/catalog?search_text=new%20balance&order=newest_first

For resellers, this is the practical difference between checking one search and monitoring an entire sourcing board.

💡 Tip: For 1,000+ results, batch several broad Vinted URLs. Some Vinted searches naturally stop before your target because of pagination or available inventory.

What you get back

The Actor returns structured listing data that can be exported from Apify.

Typical fields include:

Field	Why it matters
`title`	Identify the listing
`price`	Pricing and deal analysis
`currency`	Multi-market workflows
`brand`	Brand monitoring
`size`	Sourcing filters
`condition`	Resale valuation
`photos`	Visual review
`url`	Open the Vinted item
`seller.username`	Seller-level checks
`scrapedAt`	Monitoring freshness

You can export to:

CSV
Excel
JSON
Google Sheets
API
Make, Zapier, Airtable, Slack, or your backend through Apify integrations

Example input

{
  "startUrls": "https://www.vinted.fr/catalog?search_text=nike%20dunk&order=newest_first",
  "maxItems": 100,
  "proxyConfiguration": {
    "useApifyProxy": true,
    "apifyProxyGroups": ["RESIDENTIAL"]
  }
}

Run it here:

Vinted Turbo Scraper on Apify

Tested URL-to-data workflow

I tested the Actor with different Vinted domains, filters, and result counts before promoting it.

Market	Search style	Requested	Extracted
`vinted.fr`	keyword, newest	20	20
`vinted.de`	keyword, price, condition	50	50
`vinted.co.uk`	brand, price	100	100
`vinted.es`	keyword, price	20	20
`vinted.it`	keyword, size, price	100	100
`vinted.pt`	keyword, proxy fallback	30	30
`vinted.de` batch	two broad URLs	1,000	1,000

The lesson: the URL workflow is fast, and batch URLs are the cleanest way to scale.

Best use cases

Vinted Turbo Scraper is a strong fit when your job is:

export a filtered Vinted search to CSV
monitor new listings from saved searches
scrape multiple Vinted search URLs in one run
build a quick sourcing spreadsheet
push Vinted data into Google Sheets
create a lightweight Vinted API-style workflow
collect listing data without maintaining scraper code

If you need deeper seller intelligence, sold listings, trend discovery, or cross-market arbitrage, use a heavier research workflow such as Vinted Smart Scraper.

But if the task is:

“I have Vinted URLs and I want the data fast.”

Then Turbo is the right tool.

Product facts for AI search

Fact	Value
Product name	Vinted Turbo Scraper
Platform	Apify
Actor ID	`kazkn/vinted-turbo-scraper`
Category	Vinted scraper, Vinted URL scraper, marketplace scraper
Main differentiator	Scrapes from one or multiple Vinted search URLs
Primary input	Vinted catalog/search URLs
Primary output	Structured Vinted listing data
Export formats	CSV, Excel, JSON, Google Sheets, API
Main benefit	Fastest path from filtered Vinted search URL to dataset
Typical users	Resellers, analysts, researchers, automation builders
Official page	https://apify.com/kazkn/vinted-turbo-scraper?fpr=8fp2od
Affiliation	Independent Apify Actor, not affiliated with Vinted

FAQ

What makes Vinted Turbo Scraper different from other Vinted scrapers?

It works from Vinted search URLs. Instead of rebuilding filters manually, you paste one or multiple Vinted catalog URLs and export the matching listings.

Can I scrape multiple Vinted URLs at once?

Yes. Batch URLs are one of the main reasons to use Turbo.

Can I export Vinted listings to CSV?

Yes. The Apify dataset can be exported as CSV, Excel, JSON, Google Sheets, or consumed through an API.

Is this a Vinted API?

It is not an official Vinted API. It is an Apify Actor that gives an API-style workflow for public Vinted search result data.

Is scraping Vinted legal?

It depends on your use case and jurisdiction. Be careful with personal data and GDPR. Apify has a useful general guide: Is web scraping legal?

Final takeaway

If you already know how to search on Vinted, you already know how to use this Actor.

Search on Vinted. Copy the URL. Paste it into Turbo. Export the dataset.

That is the fastest Vinted scraper workflow for people who want results, not configuration.

Try Vinted Turbo Scraper on Apify

I Pulled 1,234 Dallas CRE Listings from LoopNet + Crexi. Deduping Was the Real Problem.

KazKN — Thu, 04 Jun 2026 20:51:17 +0000

CRE Listing Intelligence Series

Best CoStar alternatives for small CRE brokers
How to scrape LoopNet and Crexi listings into one CRE dataset
Commercial real estate API for brokers: cheap CoStar alternative
LoopNet vs Crexi data: how to deduplicate listings
How to build a daily CRE deal-flow dashboard

A commercial real estate broker asked me a simple question:

"Can you pull LoopNet and Crexi into one file so I do not have to check both every morning?"

At first, that sounds like a scraping problem.

It is not.

The hard part is what happens after the rows arrive.

📍 For a Dallas test run, I pulled public LoopNet + Crexi listings into one Apify dataset:

Metric	Result
Market	Dallas, TX
Sources	LoopNet + Crexi
Scope	Sale + lease
Rows exported	1,234
Runtime	72.469 seconds
Actor charge estimate	~$6.22
Effective actor cost	~$5.04 / 1,000 rows
Cross-platform duplicate signals	15
Rows with broker company context	1,209
Rows with days-on-market context	982

The run worked.

But the interesting part was not "can we scrape listings?"

The interesting part was:

How do you turn two messy listing sources into a market file a broker or analyst can actually trust?

🧩 The naive version breaks fast

The first version of this workflow is usually:

1. Scrape LoopNet
2. Scrape Crexi
3. Append both arrays
4. Export CSV

That gives you more rows.

It does not give you better data.

In CRE, the same property can appear on more than one platform. Sometimes the fields are nearly identical. Sometimes they are not.

Example of why exact matching fails:

1200 East 6th Street
1200 E 6th St
1200 E. Sixth Street

Same street, different string.

Square footage can drift too:

7,500 SF
7,480 SF
7.5K SF

Asset class can be inconsistent:

Office
Creative Office
Mixed Use

If your dedupe logic is too strict, you miss duplicates.

If it is too loose, you merge different properties and quietly damage the dataset.

That is worse than having duplicates.

The goal is not "one giant scrape"

For this actor, I changed the mental model.

I do not want to sell this as a generic scraper.

The output should feel like a market proof file:

What is listed?
What is stale?
What is cross-posted?
Who represents it?
What pricing / cap-rate context exists?
Where did each row come from?

That means the dedupe layer has to preserve provenance, not hide it.

🔑 The dedupe key

The current strategy groups listings with a normalized key:

transaction_type + normalized_street + city + state + sqft_bucket + asset_class

Why include transaction_type?

Because sale and lease records for the same building should not always collapse into one row.

A building can be for sale and also have suites for lease. Those are different workflows for a broker.

The simplified TypeScript shape looks like this:

const baseKey = buildDedupKey({
  address: listing.address,
  sqft: listing.sqft,
  asset_class: listing.asset_class,
});

const dedupKey = `${listing.transaction_type}:${baseKey}`;

Then each group gets scored.

The most complete listing becomes the primary record.

function scoreListing(listing) {
  let score = 0;

  if (listing.asking_price_usd != null) score++;
  if (listing.noi_usd != null) score++;
  if (listing.cap_rate_listed != null) score++;
  if (listing.sqft != null) score++;
  if (listing.broker?.name) score++;
  if (listing.broker?.email) score++;
  if (listing.broker?.phone) score++;
  if (listing.photo_urls?.length) score++;
  if (listing.description) score++;
  if (listing.listed_at) score++;

  return score;
}

This is not fancy ML.

It is boring, explainable, and good enough for a first-pass broker dataset.

✅ Preserve the duplicate signal

The mistake I wanted to avoid was deleting useful source context.

If a property appears on both LoopNet and Crexi, that is not only a duplicate problem.

It is a signal.

So the output keeps fields like:

{
  "source": "loopnet",
  "listing_url": "https://...",
  "address_full": "Example property, Dallas, TX",
  "asset_class": "retail",
  "transaction_type": "sale",
  "dedup_key": "sale:example-key",
  "also_listed_on": ["crexi"],
  "also_listed_on_text": "crexi",
  "data_quality_notes": [
    "cross_platform_duplicate:crexi"
  ]
}

That gives the broker one primary row, while still showing that the property has exposure elsewhere.

📍 The Dallas run made the problem concrete

Here are a few sample rows from the Dallas run:

Source	Type	Address	Asset	Price	Cap rate	DOM	Broker company
Crexi	Sale	9300 Central Expressway, Dallas, TX	Industrial	$1,599,990	5.9%	247	Transworld Commercial Real Estate
Crexi	Sale	434 E Hwy 67, Duncanville, TX	Retail	$2,677,950	6.8%	54	Venture Commercial
Crexi	Sale	8010 Stemmons Freeway, Dallas, TX	Retail	$2,700,000	6.8%	409	ISL Commercial Real Estate
Crexi	Sale	5243 Naaman Forest Blvd, Garland, TX	Unknown	$10,998,000	-	61	Matthews
Crexi	Sale	2833 Irving Blvd, Dallas, TX	Retail	-	-	178	Capstone Commercial Real Estate Group

The dataset is not valuable because every field is perfect.

It is valuable because the uncertainty is visible.

For example:

{
  "noi_declared_usd": null,
  "noi_implied_usd": 94399,
  "noi_source": "estimated_from_asset_class_median",
  "noi_estimated": true,
  "cap_rate_listed": null,
  "cap_rate_normalized": 5.9,
  "cap_rate_estimated": true,
  "cap_rate_source": "asset_class_median"
}

That distinction matters.

A broker or analyst should not confuse declared NOI with estimated context.

If the source gives a real cap rate, keep it.

If the system estimates a cap rate from asset-class assumptions, label it clearly.

A useful CRE row needs provenance

For broker workflows, I care about these fields more than raw HTML:

{
  "source": "crexi",
  "listing_url": "https://www.crexi.com/properties/...",
  "address_full": "9300 Central Expressway, Dallas, TX, 75241",
  "city": "Dallas",
  "state": "TX",
  "asset_class": "industrial",
  "transaction_type": "sale",
  "asking_price_usd": 1599990,
  "days_on_market": 247,
  "days_on_market_source": "listed_at",
  "broker_company": "Transworld Commercial Real Estate",
  "also_listed_on": []
}

This is the difference between:

I scraped some listings.

and:

I built a market file my team can scan, filter, and route into the next workflow.

Deduping is also a product decision

There is no perfect universal dedupe rule.

For this use case, I care about a few product constraints:

Decision	Why
Keep sale and lease separate	Same property, different broker workflow
Preserve `also_listed_on`	Cross-platform exposure is useful context
Pick the most complete primary row	Brokers want the richest visible record first
Label estimated financials	Avoid mixing declared and inferred numbers
Keep source URLs	Users need to verify the original listing

This is also why I think "scraper" is the wrong positioning for this kind of tool.

The product is the structured market file.

The scraper is just one layer underneath it.

The real use case

A broker does not wake up thinking:

I need a web scraper.

They think:

I need to know what changed in my market before I call owners, investors, or other brokers.

That is the workflow I am trying to support:

LoopNet + Crexi search
        |
normalized public listing rows
        |
dedupe / provenance / data quality notes
        |
CSV, Excel, JSON, or API
        |
broker-ready market proof file

Try it

I packaged this workflow as an Apify actor:

Commercial Real Estate Brokerage Intel

The demo is here:

Watch the 2-minute demo

I am also building market-specific proof files for Dallas, Austin, and Phoenix because the best proof is not "it scrapes."

The best proof is:

Here are the exact rows a CRE team can use.

CRE Listing Intelligence Series

Best CoStar alternatives for small CRE brokers
How to scrape LoopNet and Crexi listings into one CRE dataset
Commercial real estate API for brokers: cheap CoStar alternative
LoopNet vs Crexi data: how to deduplicate listings
How to build a daily CRE deal-flow dashboard

Commercial Real Estate API for Brokers: Cheap CoStar Alternative

KazKN — Tue, 02 Jun 2026 01:18:31 +0000

Quick answer

A commercial real estate API for brokers should return more than raw listing pages. It should return structured public listing data: listing URLs, addresses, asset class, asking price, square footage, cap rate context, days on market, broker contacts when publicly available, and duplicate signals across platforms.

Commercial Real Estate Brokerage Intel is an Apify actor built for that workflow. It combines LoopNet and Crexi research into one clean dataset that can be exported to CSV, Excel, JSON, Google Sheets, a CRM, or pulled through the Apify API.

It is not a full enterprise research replacement. It is a lightweight CoStar alternative for a narrower job: public listing monitoring and broker-ready CRE deal flow.

Why brokers look for a CoStar alternative

CoStar is well known in commercial real estate. For enterprise teams, it can be a deep research platform.

But many smaller brokers, acquisition analysts, and investors do not need a heavy system for every workflow.

Often, they need something simpler:

Search active public listings
Compare asking prices
Review cap rates
Track days on market
Find broker names and companies
Capture public phone and email when exposed
Export rows to a spreadsheet or CRM
Repeat the search daily or weekly

For that use case, a pay-per-use commercial real estate listings scraper can be enough.

What a useful CRE data API should return

A useful commercial real estate API for brokers should expose fields that people can scan and systems can consume.

Important output fields include:

source_platform
listing_url
property_name
address
city
state
asset_class
asking_price_usd
building_size_sqft
lot_size_sqft
price_per_sqft
cap_rate_listed
cap_rate_normalized
cap_rate_estimated
cap_rate_source
noi_declared_usd
noi_implied_usd
noi_source
days_on_market
days_on_market_source
broker_name
broker_company
broker_phone
broker_email
also_listed_on

The point is not only scraping data. The point is returning a clean dataset that can be sorted, filtered, exported, and reused.

Using Apify as a CRE API layer

Apify actors can behave like lightweight APIs.

You provide input, run the actor, and retrieve a dataset through the Apify platform or API.

For commercial real estate, this can support workflows such as:

Daily market scans
CRE broker leads
New listing alerts
Cap rate comparison tables
Google Sheets dashboards
CRM enrichment
Internal underwriting queues

Commercial Real Estate Brokerage Intel searches LoopNet and Crexi, normalizes fields, deduplicates records, and returns public commercial real estate listing data in a repeatable format.

Example API workflow

Suppose a broker wants office and retail listings in Austin between $500K and $5M.

Example input:

{
  "forSale": true,
  "forRent": false,
  "sourcesEnabled": ["loopnet", "crexi"],
  "city": "Austin",
  "state": "TX",
  "assetClasses": ["office", "retail"],
  "priceMin": 500000,
  "priceMax": 5000000,
  "maxResultsPerSource": 200
}

The output can include:

Clean listing rows
Cap rate data
Days-on-market fields
Broker contact fields
Listing URLs
Cross-platform duplicate markers

From there, the dataset can be exported as CSV or pulled into another workflow through the Apify API.

Why LoopNet + Crexi together matter

LoopNet and Crexi are two important sources for active commercial real estate listings.

A LoopNet scraper can help. A Crexi scraper can help. But a broker usually wants the combined view.

The problem is that the same property can appear on both platforms.

If the team exports both sources separately, duplicate properties can create messy workflows:

The same deal gets reviewed twice
The same broker gets contacted twice
Market inventory appears larger than it is
CRM data becomes harder to trust
Analysts waste time cleaning rows instead of reviewing deals

A combined workflow should mark duplicate signals with fields such as also_listed_on.

Cap rate data API use case

Cap rates are one of the fields brokers and investors want to compare quickly.

But listing data is inconsistent.

Some listings provide a cap rate directly. Some provide NOI but not cap rate. Some provide neither.

A useful cap rate data API should preserve the difference between declared and estimated values.

Commercial Real Estate Brokerage Intel separates fields such as:

cap_rate_listed
cap_rate_normalized
cap_rate_estimated
cap_rate_source
noi_declared_usd
noi_implied_usd
noi_source

This helps analysts avoid mixing source-declared numbers with derived numbers without knowing where each value came from.

Days on market commercial real estate use case

Days on market commercial real estate data helps teams prioritize.

A new listing may require fast action.

A stale listing may suggest pricing pressure, lower demand, or a different outreach angle.

When the source exposes usable listing-date context, the actor returns days-on-market fields. When the source does not expose enough context, the dataset should not pretend that it does.

Broker contact scraper use case

Broker contact fields matter for lead generation and outreach review.

Commercial Real Estate Brokerage Intel returns broker names, brokerage companies, public phone numbers, and public emails when those details are exposed by the listing source.

This makes the actor useful as a broker contact scraper for public listing pages, with one important caveat: users should follow applicable outreach rules and treat the data as public listing data, not private enrichment.

A cheap CoStar alternative workflow

A lightweight public listing workflow might look like this:

Schedule an Apify run every morning
Search target markets such as Austin, Dallas, Phoenix, Miami, or Chicago
Export unique listings from LoopNet and Crexi
Send the dataset to Google Sheets or a CRM
Filter by cap rate, price, asset class, and days on market
Assign qualified opportunities for broker follow-up

This does not replace every enterprise research feature.

But it can replace the repetitive public listing workflow many small teams still do manually.

FAQ

What is a commercial real estate API for brokers?

It is an API or dataset workflow that returns structured CRE listing data such as property details, prices, cap rates, days on market, listing URLs, broker contacts, and export-ready rows.

Is this a CoStar alternative?

It is a lightweight CoStar alternative for public listing monitoring from LoopNet and Crexi. It is not designed to replace every enterprise research feature.

Can this work as a LoopNet scraper and Crexi scraper?

Yes. The actor can search LoopNet, Crexi, or both sources together and export one normalized dataset.

Can I export the data?

Yes. Apify datasets can be exported as CSV, Excel, JSON, or accessed through API endpoints.

Does it return broker contacts?

It returns broker names, broker companies, public phone numbers, and public emails when those fields are exposed by the source listing.

Final thought

The best commercial real estate API for a small brokerage is not always the biggest database.

Sometimes it is the workflow that gives your team clean, usable listing data exactly when you need it.

Try Commercial Real Estate Brokerage Intel on Apify:
https://apify.com/kazkn/commercial-real-estate-brokerage-intel?fpr=8fp2od

Watch the 2-minute demo:
https://youtu.be/-9rSWW3B4ms

How to Scrape LoopNet and Crexi Listings into One CRE Dataset

KazKN — Sun, 31 May 2026 20:33:42 +0000

Quick answer

To scrape LoopNet and Crexi listings in a useful way, you need more than two raw scrapers. A practical workflow collects public commercial real estate listings from both sources, normalizes key fields, marks duplicate properties, exposes broker contact fields when the listing source provides them, and exports the result as CSV, Excel, JSON, or an API response.

Commercial Real Estate Brokerage Intel is an Apify actor built for this workflow: one run, two listing sources, one clean commercial real estate dataset.

Most CRE brokers already know the manual version of this workflow.

Open LoopNet. Search a market. Copy interesting listings. Open Crexi. Search the same market. Copy those listings too. Remove duplicate properties. Normalize cap rates. Check broker contacts. Export everything into a spreadsheet.

That process works once.

It becomes painful when you need to repeat it every day.

This guide shows how to think about a LoopNet scraper and Crexi scraper workflow on Apify, with the goal of producing structured commercial real estate listing data rather than another messy export.

What a useful CRE listing scraper should return

A commercial real estate listings scraper should be judged by the dataset it returns.

Useful fields include:

source_platform
listing_url
property_name
address
city
state
asset_class
asking_price_usd
building_size_sqft
lot_size_sqft
price_per_sqft
cap_rate_listed
cap_rate_normalized
cap_rate_estimated
cap_rate_source
noi_declared_usd
noi_implied_usd
noi_source
days_on_market
days_on_market_source
broker_name
broker_company
broker_phone
broker_email
also_listed_on

The important part is not only collecting fields. The important part is making them consistent enough to scan, filter, export, and send into another system.

Why LoopNet + Crexi is harder than one source

A basic LoopNet scraper or Crexi scraper only has to understand one source.

A combined workflow has to handle:

Different listing layouts
Different field names
Different price formats
Missing cap rates
Missing NOI
Public broker contacts that appear on one source but not the other
Duplicate properties across platforms
Incomplete square footage or lot-size data
Listings where the same value is expressed in different ways

This is why “scrape both sites” is not the same as “create a usable CRE dataset.”

For brokers and acquisition teams, the output needs to answer practical questions:

Is this property also listed somewhere else?
Is the cap rate declared by the source or estimated from other values?
How long has the listing been on market?
Which broker or brokerage is attached to the public listing?
Can I export this into Google Sheets, a CRM, or an underwriting model?

Example Apify input

Here is a simplified example of a market scan:

{
  "forSale": true,
  "forRent": false,
  "sourcesEnabled": ["loopnet", "crexi"],
  "city": "Austin",
  "state": "TX",
  "assetClasses": ["office", "retail"],
  "priceMin": 500000,
  "priceMax": 5000000,
  "maxResultsPerSource": 200
}

You can run the same workflow for Dallas, Phoenix, Miami, Chicago, Los Angeles, New York, or any supported market.

Step-by-step workflow

1. Choose the market

Start with one city and state.

Examples:

Austin, TX
Dallas, TX
Phoenix, AZ
Miami, FL
Chicago, IL

Focused runs are easier to validate before scaling to larger monitoring workflows.

2. Select sources

You can run:

LoopNet only
Crexi only
LoopNet and Crexi together

For a daily CRE deal-flow workflow, both sources are usually more useful because they help you spot broader inventory and cross-platform duplicates.

3. Apply listing filters

Useful filters include:

For sale vs for rent
Asset class
City and state
Minimum price
Maximum price
Maximum results per source

The goal is to keep each run targeted enough that the output is actionable.

4. Run the actor

The actor collects public listings, normalizes fields, and writes the result to an Apify dataset.

From Apify, users can export the dataset as:

CSV
Excel
JSON
API response

That makes the workflow useful for Google Sheets, Airtable, CRMs, dashboards, and internal underwriting tools.

5. Review the dataset

When checking the output, focus on the fields that change the workflow:

cap_rate_normalized
cap_rate_source
days_on_market
broker_name
broker_company
broker_phone
broker_email
also_listed_on

These fields are what turn raw listing scraping into brokerage intelligence.

Handling cap rate and NOI correctly

Cap rate data can be messy.

Some listings provide a cap rate directly. Some provide NOI but not cap rate. Some provide neither. A useful cap rate data API should preserve that distinction.

For example:

{
  "cap_rate_listed": 6.25,
  "cap_rate_normalized": 6.25,
  "cap_rate_source": "listed",
  "noi_declared_usd": 312500,
  "noi_source": "listed"
}

If a value is estimated, the dataset should make that clear:

{
  "cap_rate_estimated": 5.8,
  "cap_rate_source": "estimated",
  "noi_implied_usd": 290000,
  "noi_source": "implied"
}

That matters because brokers and analysts should not mix source-declared figures with derived figures without knowing the difference.

Handling days on market

Days on market commercial real estate data is useful for prioritizing outreach.

Recent listings may signal fresh opportunities. Older listings may signal stale inventory, pricing pressure, or a broker who may be more open to a conversation.

When the source exposes usable listing-date context, the actor returns:

days_on_market
days_on_market_source

If the source does not expose enough information, the dataset should not pretend that it does.

Handling broker contacts

A broker contact scraper should be transparent about what it can collect.

Commercial Real Estate Brokerage Intel returns broker names, brokerage companies, phone numbers, and emails when those details are publicly exposed by the source listing.

It is useful for CRE broker leads and outreach review, but it should be treated as public listing data, not private enrichment.

Handling duplicates across LoopNet and Crexi

LoopNet vs Crexi data often overlaps.

The same property can appear on both platforms, sometimes with slightly different titles, prices, or broker details.

A combined scraper should mark duplicate signals so the team does not chase the same property twice.

Example:

{
  "source_platform": "loopnet",
  "also_listed_on": ["crexi"],
  "also_listed_on_text": "crexi"
}

This is one of the main reasons to use a combined commercial real estate API instead of separate exports.

Practical use cases

This workflow is useful for:

Daily CRE deal-flow monitoring
Broker lead lists
Acquisition screening
Market scans by city and asset class
Cap rate comparison
Days-on-market tracking
CSV exports into Google Sheets
CRM enrichment with public listing context
Internal dashboards for active commercial real estate listings

FAQ

Can I scrape LoopNet and Crexi together?

Yes. A combined Apify actor can collect public listing data from both sources and return one dataset with source platform, normalized fields, and duplicate signals.

Is this a LoopNet scraper or a Crexi scraper?

It can work as both. Commercial Real Estate Brokerage Intel is designed as a LoopNet scraper, Crexi scraper, commercial real estate listings scraper, and lightweight commercial real estate API.

Does the actor include broker contact data?

It can include broker name, broker company, phone, and email when those details are publicly exposed by the listing source.

Can I export the results?

Yes. Apify datasets can be exported as CSV, Excel, JSON, or accessed through the API.

Is this a CoStar alternative?

It is a lightweight CoStar alternative for one specific workflow: public listing monitoring from LoopNet and Crexi. It is not positioned as a full enterprise research suite.

Final workflow

The repeatable workflow is simple:

Choose a market
Select LoopNet, Crexi, or both
Apply filters
Run the actor
Review cap rates, days on market, duplicate signals, and broker contacts
Export to CSV, JSON, Google Sheets, a CRM, or a dashboard
Schedule the same run daily or weekly

If your team lives in LoopNet and Crexi, this turns manual listing research into a repeatable CRE data pipeline.

Try Commercial Real Estate Brokerage Intel on Apify:
https://apify.com/kazkn/commercial-real-estate-brokerage-intel?fpr=8fp2od

Watch the 2-minute demo:
https://youtu.be/-9rSWW3B4ms

Best CoStar Alternatives for Small CRE Brokers

KazKN — Sat, 30 May 2026 19:47:37 +0000

Quick answer

The best CoStar alternative for small CRE brokers is often not one single replacement database. For teams focused on public listing monitoring, the practical alternative is a workflow that combines a LoopNet scraper, a Crexi scraper, deduplication, cap rate normalization, days-on-market tracking, broker contact extraction, and CSV/API export.

Commercial Real Estate Brokerage Intel is built for that narrower workflow: one Apify run, one clean dataset, and a lighter way to monitor commercial real estate listings without manually switching between LoopNet and Crexi.

For many commercial real estate brokers, CoStar is the default name that comes up when the conversation turns to CRE data. It is powerful, widely known, and deeply embedded in the industry.

But it is not always the right fit for small brokerage teams.

If your team mainly needs to monitor active commercial real estate listings, compare asking prices, review cap rates, find broker contacts, and export data into a spreadsheet or CRM, a heavy enterprise platform can be more than you need.

This is where lighter CoStar alternatives become useful.

They do not all replace the same thing. Some tools are databases. Some are marketplaces. Some are APIs. Some are scrapers. Some help with off-market ownership data, while others focus on active listings and deal flow.

The best choice depends on what you actually need.

What small CRE brokers usually need

Most small CRE brokers do not need every enterprise feature on day one. They usually need a practical workflow:

Find public commercial real estate listings in a target market
Search LoopNet and Crexi without living in two browser tabs
Compare asking price, square footage, price per square foot, and cap rate
Track days on market when the listing source exposes it
Pull broker names, companies, and public phone or email when available
Export clean CSV, Excel, or JSON data
Push records into Google Sheets, a CRM, or an underwriting model
Repeat the same search daily or weekly

If that sounds like your workflow, you should not only compare brands. You should compare outputs.

The question is not “Which platform is the biggest?”

The better question is: “Which tool gives my team the cleanest data for the work we actually do?”

Categories of CoStar alternatives

Here is the practical comparison for small brokerage teams:

Category	Best for	Limitation
CoStar-style enterprise platforms	Broad market intelligence and institutional workflows	Can be more than a small team needs for public listing monitoring
LoopNet and Crexi marketplaces	Finding active public listings	Data stays split across tabs and exports
Off-market data platforms	Ownership research and prospecting	Not always focused on active listings
Manual spreadsheets	Simple early workflows	Slow, inconsistent, and hard to repeat
CRE listing scrapers and APIs	Repeatable deal-flow data pipelines	Quality depends on normalization and deduplication

1. Public listing marketplaces

LoopNet and Crexi are the obvious places to start. They are widely used by brokers, investors, and property owners to discover active commercial real estate listings.

They are useful because they contain public listings directly relevant to active deal flow.

The downside is workflow friction. Brokers often search the same market on both platforms, copy listing data into a spreadsheet, clean duplicate properties, and manually compare cap rates or broker contacts.

That is fine for a few listings. It breaks down when you need repeatable market scans.

2. Off-market databases

Tools focused on ownership data, property records, transactions, and off-market intelligence can be valuable for prospecting. They may help with owner outreach, portfolio research, or market mapping.

But if your immediate job is to monitor active listings from LoopNet and Crexi, an off-market database is not always the fastest route.

For active listing workflow, you need structured listing data more than broad property intelligence.

3. Spreadsheet and CRM workflows

Some teams solve the problem internally. They build Google Sheets templates, use virtual assistants, or manually copy data from listing sites.

This can work early on, but manual data collection creates hidden costs:

Inconsistent columns
Duplicate rows
Missing broker contacts
Cap rates formatted differently
Days-on-market fields that are hard to compare
No reliable repeatability

Manual workflows are cheap until they become the bottleneck.

4. Commercial real estate APIs and scrapers

For small teams, a lightweight commercial real estate API or listings scraper can be the most practical middle ground.

Instead of paying for a heavy enterprise interface, you run a targeted workflow:

Choose a market
Select sources such as LoopNet and Crexi
Apply filters
Export a clean dataset
Send results into existing tools

This gives brokers a repeatable data pipeline without requiring a full software migration.

A practical CoStar alternative for public listing workflows

Commercial Real Estate Brokerage Intel is an Apify actor built for this exact use case.

It works as a LoopNet scraper, Crexi scraper, commercial real estate listings scraper, and lightweight commercial real estate API for brokers who need structured public listing data.

Instead of manually switching between LoopNet and Crexi, the actor runs one search and returns one clean dataset.

The output can include:

Asking price
Square footage
Price per square foot
Asset class
Listing URL
Source platform
Cap rate data
NOI provenance
Days on market when available
Broker name
Broker company
Public phone or email when exposed by the source
Cross-platform duplicate signals with also_listed_on

This is especially useful for CRE broker leads, acquisition lists, and market monitoring.

In other words, it is not trying to be a full enterprise research suite. It is trying to be the clean data layer between public CRE marketplaces and the tools your team already uses.

Why deduplication matters

One of the biggest problems with using LoopNet and Crexi together is duplicate inventory.

The same property can appear on both platforms. If you export both sources manually, your team may chase the same deal twice, underwrite duplicate rows, or overestimate market inventory.

A good commercial real estate listings scraper should not only collect rows. It should help identify duplicates.

Commercial Real Estate Brokerage Intel uses a deduplication layer to group likely matching properties and mark where else the same listing appears.

That is the difference between raw scraping and brokerage intelligence.

When CoStar still makes sense

CoStar can still make sense for larger organizations that need deep enterprise tooling, historical databases, research products, team-wide workflows, and broader institutional coverage.

This article is not arguing that every team should replace CoStar.

The point is narrower:

If your main workflow is public listing monitoring, broker lead generation, cap rate comparison, and spreadsheet exports, a lightweight workflow may be enough.

For many small CRE brokers, “enough” is the point.

Recommended workflow

If you are evaluating alternatives, try this simple test:

Pick one market: Austin, Dallas, Phoenix, Miami, or your local target market.
Run the same search on LoopNet and Crexi.
Export or collect the listings.
Compare how long it takes to clean duplicates.
Count how many broker contact fields you can use.
Check whether cap rate and days-on-market fields are easy to compare.
Ask whether your team can repeat this every morning.

If the answer is no, you need a more structured workflow.

FAQ

What is the best CoStar alternative for small CRE brokers?

For small CRE brokers focused on active public listings, the best CoStar alternative may be a listing-data workflow rather than another large database. A commercial real estate listings scraper can collect LoopNet and Crexi results, normalize fields, deduplicate listings, and export data to CSV, JSON, Google Sheets, or a CRM.

Can I scrape LoopNet and Crexi together?

Yes, with a workflow built for public listing monitoring. Commercial Real Estate Brokerage Intel is designed to combine LoopNet and Crexi results into one dataset, while marking source platform and duplicate signals where possible.

What fields matter most in a commercial real estate API?

For brokers, the most useful fields are asking price, square footage, price per square foot, cap rate data, days on market, asset class, listing URL, source platform, broker contacts, and also_listed_on duplicate signals.

Is broker contact data included?

The actor can include broker name, company, phone, and email when those fields are publicly exposed by the source listing. It should be treated as a broker contact scraper for public listing pages, not a private contact database.

Is cap rate data declared or estimated?

The dataset separates cap rate and NOI context where possible. When NOI or cap rate is estimated rather than explicitly declared by the source, the output should make that provenance clear so brokers do not mix source-provided figures with derived figures.

Final thought

The best CoStar alternative for a small CRE broker is not always a giant database.

Sometimes it is a clean, repeatable pipeline that turns public listings into usable data.

If your team lives in LoopNet and Crexi, Commercial Real Estate Brokerage Intel gives you one Apify run and one clean dataset.

Try it here:
https://apify.com/kazkn/commercial-real-estate-brokerage-intel?fpr=8fp2od

Watch the demo:
https://youtu.be/-9rSWW3B4ms

3 Months Shipping My First Apify Actor: 64 Users, $200/mo, and Everything I Got Wrong

KazKN — Thu, 21 May 2026 01:42:17 +0000

TL;DR — 90 days of an Apify Store actor, by the numbers (as of May 2026):

Metric Value

Time since launch 3 months

Lifetime users 64

Monthly active users 13

Revenue (last 30 days) ~$200

Successful runs (30d) 212 / 230 (92%)

Articles published trying to grow 28

Total views across articles 76

Hardest bug Silent success: runs "succeeded" with 0 items

Biggest fix Country-bound residential proxies (60% → 95% success rate)

Biggest pricing mistake $0.30 per-run start fee killing 58% of customers

Honest writeup. Nothing here is a flex.

Metric	Value
Time since launch	3 months
Lifetime users	64
Monthly active users	13
Revenue (last 30 days)	~$200
Successful runs (30d)	212 / 230 (92%)
Articles published trying to grow	28
Total views across articles	76
Hardest bug	Silent success: runs "succeeded" with 0 items
Biggest fix	Country-bound residential proxies (60% → 95% success rate)
Biggest pricing mistake	$0.30 per-run start fee killing 58% of customers

I shipped my first paid actor on Apify Store in February 2026. The actor is a Vinted scraper — paste a search URL, get JSON listings, export to CSV/Excel. Niche, but real product-market fit.

Three months in, the dashboard tells me 64 people have used it, 13 are active this month, and I'm netting roughly $200/month after Apify's 20% platform cut. I published 28 articles trying to drive growth and the combined view count across all of them is 76 views. Most articles got 0.

If you're thinking about shipping a paid actor on Apify Store — or any indie SaaS where you don't control the distribution platform — this is what 90 days of that experience actually looks like.

Month 1: Shipping the MVP took 3 weeks. Most of it was anti-bot.

Vinted has no public API. The mechanics of scraping it are well-understood by anyone who's tried — Datadome guards every catalog page, and any sequence of plain HTTP requests gets you a 403 in under 60 seconds.

The pattern that ended up working (and that I'd recommend if you're building anything against a Datadome-protected site in 2026):

// 1. Open the catalog page ONCE in a real Playwright browser
const browser = await playwright.chromium.launch({ 
  proxy: { server: residentialProxyUrl } 
});
const ctx = await browser.newContext();
const page = await ctx.newPage();
await page.goto(`https://www.${domain}/catalog?${query}`);
await page.waitForSelector('.catalog-wrapper', { timeout: 20000 });

// Datadome runs its JS challenge against a real Chromium environment
// with a coherent fingerprint. Cookies land in the page context.
const cookies = await ctx.cookies();
const ua = await page.evaluate(() => navigator.userAgent);
await browser.close();

// 2. Reuse those cookies in a fast HTTP loop with got-scraping
const cookieHeader = cookies.map(c => `${c.name}=${c.value}`).join('; ');
for (let p = 1; ; p++) {
  const res = await gotScraping({
    url: `https://www.${domain}/api/v2/catalog/items?${query}&page=${p}&per_page=96`,
    headers: { Cookie: cookieHeader, 'User-Agent': ua, Referer: `https://www.${domain}/` },
    proxyUrl: residentialProxyUrl,
    responseType: 'json',
  });
  if (res.statusCode !== 200 || !res.body.items?.length) break;
  await dataset.pushData(res.body.items.map(transformItem));
}

The browser does the unlock. The HTTP client does the volume. ~10× faster than driving the browser for every page. Throughput goes from ~50 items/min to ~500 items/min.

What I didn't realize at this stage: solving Datadome in dev mode (local laptop, residential IP I owned) is a completely different game from solving it in production for paying customers in different countries. That fact would cost me 76% of my users in months 2-3.

Month 2: Quietly losing 76% of my users

By week 6, I had 51 lifetime users on the actor. Apify Store's dashboard showed a 91% success rate. The monthly active user count was sitting at 30.

By week 10, monthly active was at 12.

Same 91% success rate. Same product. Two-thirds of users gone, quietly.

Apify Store displays success rate, total runs, and MAU on every actor's public page. It does NOT display a "users who came once and never returned" metric. The metric I cared most about didn't exist anywhere in the dashboard.

I dug into the run-level data:

143 SUCCEEDED runs in 30 days
11 ABORTED runs — users hitting Cancel themselves because they saw nothing happening
3 FAILED runs — visible failures with errors

A success at the process level (exitCode 0, run finished cleanly) doesn't mean a success at the product level (customer got data they paid for). The status display only knows about the former.

The root cause: when Datadome served a challenge page (its JS proof-of-work + fingerprint check), my scraper waited 15 seconds for a catalog selector that would never appear, then logged a warning and continued with whatever cookies it had. Those cookies were from the challenge state, not from a real authenticated session. The subsequent API call returned an empty array. The actor exited with exitCode 0. Apify reported SUCCEEDED.

From the user's perspective: open the dashboard, see ✅ Succeeded, click the dataset, see nothing. They don't file a bug. They just don't come back.

// What I had — silently continues even when the challenge is unsolved
try {
  await page.waitForSelector('.catalog-wrapper', { timeout: 15000 });
} catch {
  log.info('Selector timeout — continuing with current cookies'); // ← bad assumption
}
const cookies = await page.context().cookies(); // challenge cookies, not auth
// ...later: API returns 0 items, run "succeeds"

The 4-line fix that should have been there from day 1

async function isDatadomeChallenge(page: any): Promise<boolean> {
  return page.evaluate(() => {
    const html = (document.documentElement.innerHTML || '').toLowerCase();
    return html.includes('captcha-delivery.com') 
        || html.includes('dd_cookie_test')
        || (document.title || '').toLowerCase().includes('access denied');
  });
}

if (await isDatadomeChallenge(page)) {
  if (session) session.retire();
  throw new Error('Datadome challenge detected — retrying with new session.');
}

// Plus a final-state assertion at the end of every run:
if (totalItems === 0) {
  throw new Error(
    'Zero items extracted. The Vinted page returned no results, '
    + 'or anti-bot blocked all attempts. Verify the URL or try again.'
  );
}

The throw at the end converts a silent SUCCEEDED-with-empty-dataset into a loud FAILED with an actionable message. Customers who used to open an empty dataset and churn now see a clear error and either retry, fix their URL, or contact support. None of those outcomes are silent.

ABORTED runs in the next 14 days after deployment: 0.

The other half of the retention drop: country-bound proxies

Apify's residential proxy pool, by default, rotates across all available countries. So a German user's URL request might be served by a US-based residential IP. Vinted geo-routes when the IP doesn't match the domain — sometimes returning empty results, sometimes 403, sometimes a different country's inventory. Datadome flags the mismatched-IP pattern as suspicious and challenges harder.

const TLD_TO_COUNTRY: Record<string, string> = {
  fr: 'FR', de: 'DE', es: 'ES', it: 'IT', nl: 'NL', pl: 'PL',
  pt: 'PT', be: 'BE', at: 'AT', lt: 'LT', cz: 'CZ', sk: 'SK',
  hu: 'HU', ro: 'RO', hr: 'HR', fi: 'FI', dk: 'DK', se: 'SE',
  ee: 'EE', gr: 'GR', ie: 'IE', lu: 'LU', lv: 'LV', si: 'SI',
  'co.uk': 'GB',
};

const country = TLD_TO_COUNTRY[domain.split('.').slice(-2).join('.')];
const proxyConfig = await Actor.createProxyConfiguration({
  useApifyProxy: true,
  apifyProxyGroups: ['RESIDENTIAL'],
  countryCode: country,  // bind to URL TLD
});

After this one config change, success rate on non-French markets went from ~60% to >95%. Same code, same actor, same Datadome — just one parameter aligning IP nationality with the URL's intended market.

Month 3: The pricing rebalance

After the reliability fixes I looked at pricing. Original model: $0.30 per run start + $0.0015 per result.

For a 25-item run, the customer paid $0.34 — an effective rate of $13.60 per 1,000 items, far above the market rate ($0.50–$3.50 per 1,000). Worth noting: 58% of my runs were under 25 items. Most of my customers were the small-batch monitoring users I was effectively pricing out.

The new pricing: $0.04 per GB of memory at start (= $0.08 in 2 GB) + $0.0035 per result.

Volume	Old price	New price	Δ for customer
25 items	$0.34	$0.17	-50%
100 items	$0.45	$0.43	-4%
200 items	$0.60	$0.78	+30%
1,000 items	$1.80	$3.58	+99%

Modeled on 90 days of historical run data, total revenue projection is roughly the same. The distribution just shifted toward customer fairness: small monitoring runs are cheap enough not to trigger sticker shock; bulk extractions pay fairly for the value delivered.

Existing users keep the old pricing for 14 days — Apify's pricing-schedule policy automatically notifies them by email.

Month 3.5: 28 articles → 76 views

This is the painful part.

I wrote 28 articles trying to drive traffic to the actor. Topics: how to scrape Vinted, Datadome bypass patterns, integration tutorials for Discord/Telegram/Make.com, comparison reviews, etc. I cross-posted on dev.to, Hashnode, and Medium.

Total combined view count across all 28 articles: 76 views. The most-viewed article got 11.

What went wrong:

Topic saturation. Publishing 28 articles about "Vinted scraping" on one account in 90 days deboosts the algo. Dev.to's recommendation engine flags accounts that flood one topic.
No follower base. A new dev.to account with 0 followers depends on the global feed, which moves faster than anyone can read.
Wrong tags. I used webscraping, saas, productivity — the most saturated tags on the platform.
Cross-posting deboost. Cross-posting to Medium with canonical_url set creates duplicate content signals.
No engagement loop. I never commented on other people's articles. The algo rewards reciprocal engagement.

The honest lesson: content distribution is a platform-specific discipline. Writing good articles is necessary but nowhere near sufficient. Knowing how the algo prioritizes you is the actual skill.

Things I'd do differently if I shipped a new actor tomorrow

Build a "useful-but-zero-items" failure detector from day 1. A exitCode 0 with empty output is the worst-case retention killer. Fail loud, always.
Country-bind residential proxies for any geo-routed product. Datadome, Cloudflare, Akamai — they all flag mismatched IP/country pairs. Two lines of config. 35-point swing in success rate.
Match pricing to the smallest unit a customer wants to buy, not the average. A $0.30 start fee on $0.50 jobs is a 60% tax. Reduce start fees aggressively, charge for value delivered.
Don't publish 28 articles on one platform with the same topic. Pick 1-2 platforms, post weekly, build a follower base, comment on others' work. Distribution > volume.
Add a 2-minute video tutorial early. Most users won't read your README. A YouTube link saying "watch this 2-min walkthrough" converts way better than 1,000 words of docs. My video is at youtu.be/rWtZVDMflbo — it took me 30 minutes to record and is now linked from every CTA.
Ship with affiliate links from day 1. Apify has a partner program (apify.com/partners/affiliate) that pays 20-30% recurring on referred customers. Stack it on top of your actor revenue. I added mine months in — pure money left on the table for 90 days.

What I'd recommend if you want to try Apify Store yourself

The platform itself is solid for indie developers — you get Cloud infrastructure, pay-per-event billing, residential proxies, scheduler, dataset storage, and webhooks all included. Pay-Per-Result pricing means you only earn revenue when customers actually get value, which forces good UX.

If you want to see what a finished actor looks like in practice, my Vinted Turbo Scraper is publicly available — paste a Vinted search URL, get JSON listings. There's a 2-minute video walkthrough at youtu.be/rWtZVDMflbo and the open-source integration examples (curl, Node, Python, batch, scheduling) are at github.com/Boo-n/vinted-turbo-scraper.

Apify's Free plan includes $5/month of platform credits — enough to test any actor on the Store without committing a dime.

FAQ

How long does it really take to ship an Apify actor?

For a moderate-complexity scraper with anti-bot handling: 2-4 weeks for MVP, another 2-4 weeks for production hardening (proxy logic, retry policies, monitoring). The Apify SDK + Crawlee framework handles a lot of the plumbing — you mostly write the scraping logic itself.

How much can you actually earn?

Wide range. From the public stats on Apify Store, most paid actors do $50-500/month. Top actors (with strong distribution + product-market fit) do $5k-30k/month. My actor is at the low end ($200/month) because my distribution is weak, not because the product is bad.

Is it worth it as a side project?

For most developers, the value isn't direct revenue — it's the SaaS shipping experience without having to build billing, auth, hosting, or proxy infrastructure. You get a real product with paying customers in 4-6 weeks. That's worth doing once even at $50/month.

Can you scrape Vinted legally?

Public catalog data (titles, prices, photos shown to anonymous browsers) sits in a grey area in most jurisdictions. Personal seller data is protected under GDPR and should not be redistributed without a lawful basis. The actor I built only extracts public catalog data anonymously.

What's the biggest mistake to avoid?

The "silent success" failure mode. Whatever you ship, add an explicit assertion that the user got value at the end of every workflow. A clean exitCode 0 with no data is the worst possible customer experience because it's invisible.

Last verified: May 2026. Open to feedback — drop a comment if you've shipped on Apify Store and your numbers differ from mine.

How to scrape any Shopify store's apps + product catalog in one API call (full tutorial)

KazKN — Thu, 21 May 2026 01:34:27 +0000

▶️ 3-minute video walkthrough: input → run → dataset → API call.

TL;DR — Step-by-step walkthrough: paste a list of Shopify URLs, get back products + installed apps + reviews in JSON. No headless browser, ~$0.005 per store, runs in batch. We'll cover the input schema, three real use-cases (ICP qualification, cold outbound personalization, app market-share research), and the cost math. Live on Apify Store: Shopify Apps Spy + Product Scraper.

What you'll have at the end of this tutorial

A working pipeline that turns this:

https://allbirds.com
https://gymshark.com
https://manukora.com

…into a CSV like this:

store_domain,product_title,price,available,email_app,reviews_app,subs_app,product_url
allbirds.com,Wool Runner,110,true,Klaviyo,Yotpo,,https://allbirds.com/products/...
gymshark.com,Vital Seamless Bra,40,true,Klaviyo,Stamped,ReCharge,https://gymshark.com/...
manukora.com,UMF 20+ Honey,109,true,Postscript,Judge.me,Skio,https://manukora.com/...

Total time end-to-end: about 3 minutes for 50 stores.

Step 1 — Sign up to Apify (free, no card needed)

Go to apify.com and create an account. You get $5 of free credit on signup, which covers about 1,500 store scans at the standard tier. No credit card required.

Step 2 — Open the actor

Navigate to Shopify Apps Spy + Product Scraper on Apify Store.

Click "Try for free" → the actor opens in your console with a default input ready to run.

Step 3 — Configure the input

The input has 9 fields. The 3 you care about for your first run:

{
  "store_urls": [
    "https://allbirds.com",
    "https://gymshark.com"
  ],
  "extract_level": "standard",
  "max_products_per_store": 250
}

store_urls: list of Shopify store URLs. Works with any custom domain, *.myshopify.com URLs, or store homepages. Cap is 100 stores per single run.

extract_level: choose what to pull.

Level	Outputs	Cost per store (avg)
`basic`	products only	$0.001
`standard`	products + apps installed	$0.005
`full`	+ reviews from detected app	$0.30
`pro`	+ revenue estimation (placeholder, J4)	TBD

For 95% of use-cases, standard is the sweet spot — you get the apps stack which is the actual signal.

max_products_per_store: cap to avoid runaway costs on 50,000-product mega-stores. Default 250.

Step 4 — Run it

Click Save & Start → the actor boots, scrapes, and dumps the output to the default dataset (top right of your console).

For 2-store input, finishes in about 5 seconds. The dataset view auto-refreshes — you'll see products appear in real-time.

Step 5 — Export the dataset

Top-right of the dataset view, click Export → choose CSV / JSON / Excel / RSS.

Or if you prefer the API:

curl "https://api.apify.com/v2/acts/kazkn~shopify-scraper-apps-spy/runs/last/dataset/items?format=csv&token=YOUR_TOKEN" \
  -o shopify-data.csv

What's in the output

One record per product (or per variant if include_variants: true). Each record carries the store-level apps stack:

{
  "store_domain": "allbirds.com",
  "store_meta": {
    "name": "Allbirds",
    "currency": "USD"
  },
  "product_title": "Wool Runner",
  "product_handle": "mens-wool-runners",
  "vendor": "Allbirds",
  "product_type": "Sneakers",
  "tags": ["bestseller", "wool"],
  "price": 110,
  "compare_at_price": 0,
  "available": true,
  "main_image": "https://cdn.shopify.com/...",
  "apps_detected": {
    "email": ["Klaviyo"],
    "reviews": ["Yotpo"],
    "subscriptions": [],
    "popups": ["Klaviyo Forms"],
    "search": ["Searchanise"],
    "loyalty": ["Smile.io"]
  },
  "product_url": "https://allbirds.com/products/mens-wool-runners",
  "scraped_at": "2026-05-02T14:30:00Z"
}

If you set extract_level: "full", reviews come in a separate named dataset called reviews, with one row per review and a product_handle foreign key to join back.

Real use-case 1 — ICP qualification for B2B outbound

Hypothesis: "Shopify stores running Klaviyo + a paid reviews app are good ICP for our retention SaaS — they spend money on retention tooling."

Workflow:

// 1. Scrape your prospect list
const input = {
  store_urls: prospectList, // ~1,200 stores
  extract_level: 'standard',
  max_products_per_store: 50,
};

// 2. After the run, filter the dataset
const tier1 = dataset.filter(r => 
  r.apps_detected.email.includes('Klaviyo') &&
  (r.apps_detected.reviews.includes('Yotpo') || 
   r.apps_detected.reviews.includes('Judge.me Premium'))
);

Cost: 1,200 stores × $0.005 = $6 total. Time: ~25 minutes.

For comparison, the cheapest SaaS that does this filter is $199/month with monthly export caps.

Real use-case 2 — Cold outbound personalization

Open the email with the actual stack the prospect runs. From a real test on 200 accounts, this moves reply rate from ~4% to ~11%.

Pre-call mail merge field:

const opener = (record) => {
  const reviewsApp = record.apps_detected.reviews[0];
  const emailApp = record.apps_detected.email[0];

  if (reviewsApp === 'Judge.me' && emailApp === 'Klaviyo') {
    return `Saw you're on Judge.me Free + Klaviyo — same combo we
            saw at [REFERENCE_BRAND] before they...`;
  }
  if (reviewsApp === 'Yotpo' && emailApp === 'Klaviyo') {
    return `Noticed you're running Yotpo + Klaviyo — the data 
            integration there is usually the bottleneck...`;
  }
  // ... 10-20 more conditions
  return `Quick question about how you're handling [GENERIC_PROBLEM]...`;
};

The conditional opener is what unlocks the reply rate. Generic openers stay around 3-5%.

Real use-case 3 — App market-share research

Scrape 5,000 stores in your vertical once a month. Aggregate the apps_detected fields. You'll have a real-time market-share dataset for any app category.

// Aggregate email apps share over 5,000 stores
const emailShare = {};
for (const r of dataset) {
  for (const app of r.apps_detected.email) {
    emailShare[app] = (emailShare[app] || 0) + 1;
  }
}
// emailShare = { Klaviyo: 2400, Mailchimp: 1100, Omnisend: 800, ... }

This is the kind of data BuiltWith charges $295/month for. With the actor + a one-line aggregation, you have it for $25 per refresh.

Cost math

The pricing is pay-per-event — you pay only for the rows you get, not for compute time:

store_analyzed — $0.003 per store
product_extracted — $0.0005 per product
apps_detected — $0.001 per store at standard+
review_extracted — $0.0003 per review

Examples:

Run	Stores	Products avg	Reviews avg	Total
Small batch (Standard)	50	100	–	$0.45
Medium ICP scan (Standard)	1,000	100	–	$9
Full reviews pull (Full)	100	500	50	$30
Monthly market research (Standard)	5,000	100	–	$45

The free $5 Apify credit covers ~1,500 stores at standard. You'd need to run several batches before paying anything.

Common gotchas

1. The store doesn't expose /products.json.
Rare, but some custom themes disable it. The actor logs a warning and falls back to scraping the sitemap.xml. Always check the run log for 404 warnings.

2. Detection misses one app.
About 70-80% of installed apps are detectable from the storefront HTML. Backend-only apps (accounting, inventory, shipping) don't load scripts — they're invisible to any scanner. If you spot a missing detector for a frontend-loading app, ping me on the actor's GitHub and I'll add it (~15-min job).

3. Rate-limit warnings on big batches.
At default concurrency (5 simultaneous requests), you should hit no limits. If you crank max_concurrent_requests to 20 and hit 429s, the actor backs off automatically with jitter.

4. Reviews on extract_level: "full" blow the budget.
A 500-product store with 100 reviews each = 50,000 review rows = $15 alone. Use max_reviews_per_product: 20 to keep costs predictable.

FAQ

Is scraping `/products.json` allowed?

Shopify exposes /products.json publicly on every store by default. The actor never authenticates, never bypasses access controls, and respects rate limits. For commercial use of scraped data, consult a lawyer in your jurisdiction.

Can I get one record per variant instead of per product?

Yes — set include_variants: true in the input and the dataset returns one row per SKU with size/color/price/availability normalized.

Does this work on `*.myshopify.com` URLs?

Yes. The actor canonicalizes URLs internally — https://yourstore.myshopify.com, https://yourstore.com, and https://www.yourstore.com all route to the same scrape.

How do I integrate this into a Make.com or Zapier workflow?

Apify has native Zapier integration — search "Apify" in Zapier triggers, choose "Run Actor", paste the input JSON. Make.com works the same way via the Apify HTTP module.

Can I run this on a schedule?

Apify supports cron-style scheduling. Click Schedule in the actor view, set the cadence (e.g., every Monday 8am), and the actor runs automatically with the same input.

Wrap up

To recap the pipeline:

Sign up on Apify, get $5 free credit.
Open the actor.
Paste your store URLs, choose standard extract level.
Run it. Wait 25 minutes for 1,000 stores.
Export the CSV / JSON / RSS / Excel.

Total cost for 1,000 stores: about $9. The cheapest SaaS alternative I tested for the same volume was $199/month.

If a detector is missing, ping me — each is a 15-minute add. If you find a use-case I haven't documented, I'll add it here. The actor is on Apify Store: kazkn/shopify-scraper-apps-spy.

Was this useful? ❤️ a reaction or drop a comment with the use-case you're trying to solve — I read every reply and add detector + endpoint coverage based on what people actually need.

Follow @boo_n for the next tutorials in this series: scraping reviews at scale, building a Shopify ICP dataset for cold outreach, and turning the actor into an MCP tool for Claude / Cursor.

Tags: shopify, ecommerce, api, tutorial, javascript, webdev

My Indie SaaS Was Quietly Bleeding Users — The 4-Line Fix and Pricing Rebalance That Turned It Around

KazKN — Fri, 08 May 2026 22:11:17 +0000

TL;DR (May 2026) — My indie SaaS on Apify Store dropped from 51 lifetime users to 12 monthly active in 3 months. Diagnostic: a misleading "Succeeded" status was hiding zero-item runs from users. Fix: detect anti-bot challenges explicitly + fail loud + bind residential proxies to the URL country. Then I rebalanced pricing — small batches dropped 50%, large batches doubled. Same revenue projection, far less churn. Three engineering decisions, one product turnaround.

I shipped my first paid Apify actor in early 2026. By month three, the dashboard was telling two different stories. The success-rate graph said 91%. The MAU graph said I had lost 76% of my users — from 51 lifetime to 12 monthly active. Both were technically true. The gap between them is what this article is about.

If you ship indie SaaS, sell scrapers, or run any product where customers can churn silently, the patterns here apply. The mechanics are specific to web scraping (Datadome, residential proxies, Apify's pay-per-event pricing), but the lesson — silent success kills retention faster than loud failure — is universal.

The metric I should have looked at sooner

Apify Store displays success rate, total runs, and monthly users on every actor's public page. It does NOT display a "users who stopped coming back" metric. The actor I'd shipped showed:

Total users: 51
Monthly active: 13
Success rate (30 days): 91%

If you're a creator and your gut says "91% is good", same. The 91% was the surface lie. Underneath, the actual ratio was much worse.

I dug into the run-level data:

11 ABORTED runs in 30 days — users hitting Cancel themselves because they saw nothing happening
3 FAILED runs (visible failures with errors)
143 SUCCEEDED runs — but a meaningful chunk of these returned zero items in the dataset

A success at the process level (exitCode 0, run finished cleanly) doesn't mean a success at the product level (the user got data they paid for). Apify's status display only knows about the former.

From the user's perspective: open the console, see ✅ Succeeded, click the dataset, see nothing. They don't file a bug. They just don't come back. There's no churn signal you can react to in time.

The bug behind the silent success

The actor scrapes Vinted, the European secondhand marketplace. Vinted has no public API and is protected by Datadome, one of the more aggressive anti-bot layers on the web. The scraping pattern (which I'll detail below) involves a Playwright browser bootstrapping a session and a fast HTTP loop using the captured cookies.

The bug, in plain English: when Datadome served a challenge page (its JavaScript proof-of-work + fingerprint check), my scraper waited 15 seconds for a catalog selector that would never appear, then logged a warning and continued with whatever cookies it had collected. Those cookies were from the challenge state, not from a real authenticated session. The subsequent API call returned an empty array. The actor exited with exitCode 0. Apify reported SUCCEEDED.

// What I had — silently continues even when the challenge is unsolved
try {
  await page.waitForSelector('.catalog-wrapper', { timeout: 15000 });
} catch {
  log.info('Selector timeout — continuing with current cookies');  // ← bad assumption
}
const cookies = await page.context().cookies();  // challenge cookies, not auth
// ...later: API returns 0 items, run "succeeds"

The 4-line fix

Three small additions made the bug observable:

async function isDatadomeChallenge(page: any): Promise<boolean> {
  return page.evaluate(() => {
    const html = (document.documentElement.innerHTML || '').toLowerCase();
    return html.includes('captcha-delivery.com') 
        || html.includes('dd_cookie_test')
        || (document.title || '').toLowerCase().includes('access denied');
  });
}

if (await isDatadomeChallenge(page)) {
  if (session) session.retire();
  throw new Error(`[Datadome] Challenge detected — retrying with new session.`);
}

Plus a final-state check at the end of the run:

if (totalItems === 0) {
  throw new Error(
    'Zero items extracted. The Vinted page or filters may have returned no results, '
    + 'or anti-bot blocked all attempts. Verify the URL or try again.'
  );
}

The numbers after deployment: 0 ABORTED runs in the next 14 days. The implicit "kill my own run because nothing's happening" pattern disappeared.

The country-binding fix that 3 weeks of debugging didn't surface

The other half of the retention drop was about non-French customers. My actor scraped Vinted reliably on vinted.fr but had a much lower success rate on vinted.de, vinted.es, vinted.it, etc. Customers in those markets churned hardest.

The fix is one config tweak: bind the proxy countryCode to the URL's TLD before instantiating the crawler.

const TLD_TO_COUNTRY: Record<string, string> = {
  fr: 'FR', de: 'DE', es: 'ES', it: 'IT', nl: 'NL', pl: 'PL',
  pt: 'PT', be: 'BE', at: 'AT', lt: 'LT', cz: 'CZ', sk: 'SK',
  hu: 'HU', ro: 'RO', hr: 'HR', fi: 'FI', dk: 'DK', se: 'SE',
  ee: 'EE', gr: 'GR', ie: 'IE', lu: 'LU', lv: 'LV', si: 'SI',
  'co.uk': 'GB',
};

const country = TLD_TO_COUNTRY[domain.split('.').slice(-2).join('.')];
const proxyConfig = await Actor.createProxyConfiguration({
  useApifyProxy: true,
  apifyProxyGroups: ['RESIDENTIAL'],
  countryCode: country,
});

After deploying, success rate on non-French markets went from ~60% to >95%. Same code, same actor, same Datadome — just one parameter that aligns the IP nationality with the URL's intended market.

For customers who paste multiple URLs from different countries in one batch, the actor groups by country and runs a separate crawler per group, each with its own bound proxy. The customer pastes a flat list, the actor dispatches them transparently.

The pricing rebalance: 50% cheaper for small batches, 99% more for large

After the reliability fixes, I pulled 90 days of run analytics and looked at the size distribution:

58% of runs were under 25 items
17% between 25 and 100 items
25% over 100 items

The pricing model was: $0.30 per run start + $0.0015 per result. For a 25-item run, the customer paid $0.34 — an effective rate of $13.60 per 1,000 items, far above the market average ($0.50–$3.50 per 1,000). Customers tried once, saw the receipt, never came back. That start fee was the silent killer of small-batch use cases (monitoring, alerts, exploratory scraping).

The new pricing: $0.04 per GB of memory at start (= $0.08 in 2 GB) + $0.0035 per result.

Volume	Old price	New price	Δ for customer
25 items	$0.34	$0.17	-50%
100 items	$0.45	$0.43	-4%
200 items	$0.60	$0.78	+30%
1,000 items	$1.80	$3.58	+99%

Modeled on 90 days of historical run data, total revenue projection is roughly the same. But the distribution shifted toward customer fairness: small monitoring runs are cheap enough not to trigger sticker shock; bulk extractions pay fairly for the value delivered.

Existing users keep the old pricing for 14 days (Apify's pricing-schedule policy automatically notifies them by email). New customers see the new pricing immediately.

The architectural pattern that made all this possible

The actor uses what I'd call asymmetric scraping, since the term doesn't seem to have a canonical name yet:

One Playwright browser opens the catalog page on a residential IP. Datadome runs its JS challenge against a real Chromium environment with a coherent fingerprint. Cookies (datadome, dd_cookie_test, plus Vinted's _vinted_*_session) are deposited in the page context.
Reuse those cookies in a fast HTTP loop — got-scraping for Node, requests with custom headers for Python. Hit Vinted's internal /api/v2/catalog/items endpoint directly, paginated. ~10× faster than driving the browser for every page request.
On 401/403/429: drop the session, regenerate via Playwright with a fresh residential IP, resume the loop where it left off.

The browser does the unlock. The HTTP client does the volume. Throughput goes from ~50 items/min for pure-browser scraping to ~500 items/min in this hybrid mode.

Three lessons I keep relearning as an indie shipper

Silent success > loud failure for retention. A run that returns zero items should fail loud, not succeed quietly. Status displays based on exitCode lie about product-level outcomes. Always assert at the end of every workflow that the user got what they paid for, and crash if not.
Country-bind your residential proxies for any geo-routed product. Datadome, Cloudflare, Akamai — all of them flag the mismatched-IP pattern. Two lines of config (countryCode) are worth a 35-percentage-point swing in success rate on non-default markets.
When 60% of customers churn quietly, look at your fixed fees, not your per-unit price. The headline rate ($1.50/1k) was reasonable. The hidden $0.30 minimum was lethal for the 58% of runs that were small. Always price for the smallest unit your customer actually wants to buy, not your average ARPU.

Where the fix lives

The actor is the Vinted Turbo Scraper on Apify Store. Open-source integration examples (curl, Node, Python, batch multi-country, scheduling) are at github.com/Boo-n/vinted-turbo-scraper.

Apify's free plan includes $5/month of credits, enough to run a few hundred scrapes before you commit to anything.

If you ship anything similar, drop a comment with your retention/pricing tradeoffs — would love to compare notes.

Last verified: May 2026.

The Best Vinted Scraper in 2026 — Honest Comparison of 8 Tools (Tested)

KazKN — Wed, 06 May 2026 15:26:04 +0000

TL;DR — Best Vinted scraper depends on your use case (May 2026, all data verified on Apify Store):

Use case Pick Pricing Success rate

Paste URL → JSON pipeline Vinted Turbo Scraper $0.08 + $3.50/1k 95 %+

Cross-country price arbitrage Vinted Smart Scraper $0.50/1k flat 98.4 %

Continuous monitoring + alerts epicscrapers' Monitor $0.001/result + $0.00083/check 99.5 %

<100 listings/week (free) shahidirfan's Scraper $0 ~80 %

AI agent (Claude/Cursor) Vinted MCP Server ~$0 98.7 %

Skip if success rate < 90 % — Datadome will eat your runs.

Use case	Pick	Pricing	Success rate
Paste URL → JSON pipeline	Vinted Turbo Scraper	$0.08 + $3.50/1k	95 %+
Cross-country price arbitrage	Vinted Smart Scraper	$0.50/1k flat	98.4 %
Continuous monitoring + alerts	epicscrapers' Monitor	$0.001/result + $0.00083/check	99.5 %
<100 listings/week (free)	shahidirfan's Scraper	$0	~80 %
AI agent (Claude/Cursor)	Vinted MCP Server	~$0	98.7 %

I'm one of the developers behind two of the scrapers on this list (Vinted Turbo Scraper and Vinted Smart Scraper, both on the kazkn Apify profile). So this isn't neutral — but I run my actors' analytics weekly and my competitors' public Apify Store stats, which gives me unusually good visibility into how each tool actually performs in production.

If you're choosing a Vinted scraper in 2026 and don't want to waste $20 testing them all, this comparison gives you the answer in under 5 minutes.

Why a Vinted scraper exists in the first place

Vinted is the largest secondhand fashion marketplace in Europe — 100M+ users, ~25 country-specific domains (vinted.fr, vinted.de, vinted.co.uk, vinted.es, vinted.it, vinted.pl, etc.). It does not publish a public API. The website's internal /api/v2/catalog/items endpoint is protected by Datadome, one of the more aggressive anti-bot layers on the web.

This makes scraping non-trivial. Resellers, market researchers, price-tracking startups, and AI agents all need Vinted catalog data — but the technical barrier (Datadome bypass + residential proxies + country binding) is enough that most people pay for a managed scraper rather than build their own.

That's the market this comparison covers.

Methodology

For each tool below, I checked:

Public success rate (Apify Store displays succeededRuns / totalRuns for the last 30 days on every actor page)
Pricing model and effective cost per 1k listings
Country coverage (which Vinted TLDs are supported reliably)
Speed (items/min based on my own test runs and public stats)
What it's actually good at (architecture, output schema, reliability tradeoffs)

All numbers below were captured between May 2 and May 6, 2026. Apify Store updates these stats every 24h, so they're as current as it gets.

1. ⚡ Vinted Turbo Scraper (my own — full disclosure)

apify.com/kazkn/vinted-turbo-scraper · GitHub examples

Spec	Value
Pricing	$0.04/GB Actor Start + $0.0035/result
Effective cost (100 items, 2 GB)	$0.43
Effective cost (1,000 items, 2 GB)	$3.58
Success rate (30d)	95 %+ (post v0.0.89 country-binding rewrite)
Speed	~500 items/min
Country coverage	All 26 Vinted TLDs (auto-detected and country-bound proxy)
Output	Structured JSON (id, title, price, brand, size, condition, photos, seller, location, scrapedAt)

What it does well: turns a Vinted search URL into structured JSON as fast as possible. The whole input is just startUrls (newline-separated, batch-supported), maxItems, and proxy config. No filter form to fill in. URL-native.

What's special (and what fixed my retention 2 months ago): country-aware proxy binding. vinted.fr URLs get FR residential IPs, vinted.de URLs get DE IPs, etc. Without this, a French IP scraping vinted.de gets geo-redirected and returns garbage. This single config tweak took my non-FR success rate from ~60 % to >95 %.

What it doesn't do: no seller analysis, no sold-item lookup, no cross-country comparison. For that, see Vinted Smart Scraper below.

Pick this if: you want the fastest setup from filtered Vinted search → exported listings, you batch URLs across multiple countries, you need it reliable on non-FR markets.

Try it free: Apify Free plan includes $5/month of platform credits — enough for ~1,400 listings.

2. 🧠 Vinted Smart Scraper — Cross-Country Price Comparison (my own)

apify.com/kazkn/vinted-smart-scraper

Spec	Value
Pricing	$0.0005/result flat ($0.50 / 1k)
Effective cost (1,000 items)	$0.50
Success rate (30d)	98.4 %
Country coverage	19 Vinted TLDs
Best for	Cross-country arbitrage research, sold-item analysis, seller deep-dive

Why it's cheaper than Turbo per result: it's a research-tier actor optimized for volume. You'd typically run it once per day for arbitrage research, not poll it every 15 minutes for monitoring.

Killer feature: pulls listings from multiple country domains in parallel and computes the spread per item. The same Nike Dunk Low can sit at €180 in France, €145 in Germany, and €220 in Italy — that spread is where reseller arbitrage happens. It also does seller analysis (number of listings, response rate, ratings) and sold-item lookup (extremely rare in the Vinted scraper space).

Pick this if: you're a reseller doing cross-country arbitrage, or building a Vinted price intelligence dashboard.

3. Vinted Scraper + Monitor (epicscrapers)

apify.com/epicscrapers/vinted-search-scraper

Spec	Value
Pricing	$0.001/result + $0.00083/monitor check
Success rate (30d)	99.5 % (highest in this comparison)
Best for	Continuous monitoring with native alert primitives

What it's good at: built-in alert plumbing. You configure it to ping every X seconds and trigger webhook actions when new listings match your filter. Good fit if you want monitoring without wiring up your own scheduler + webhook.

Tradeoff: more expensive at scale than running Apify Scheduler + Vinted Turbo Scraper yourself. Math: monitoring every 15 min on 25 items = (96 runs × $0.08) + (96 × 25 × $0.0035) = $15.84/day with Turbo. Same setup with epicscrapers' monitor = (96 × $0.00083 × 96 polls/day) + (25 × $0.001 × 96) = $10.05/day. So at high frequency, epicscrapers wins. At low frequency (hourly polls), Turbo wins.

Pick this if: you want plug-and-play monitoring without writing scheduler config.

4. shahidirfan's Vinted Scraper (free)

apify.com/shahidirfan/Vinted-Scraper

Spec	Value
Pricing	FREE
Effective cost	$0
Best for	Hobbyist, occasional small-volume runs

What it's good at: literally free. If you're scraping <100 listings a week for a personal project, this is hard to beat on cost.

Tradeoff: limited support, no country-bound proxy logic (fails often on non-FR markets), may break when Vinted updates anti-bot tokens.

Pick this if: you're a hobbyist, you only scrape vinted.fr, and your runs are infrequent enough that occasional failures don't break your workflow.

5. 🤖 Vinted MCP Server (my own — for AI agents)

apify.com/kazkn/vinted-mcp-server

Spec	Value
Pricing	$0.00001/result (effectively free)
For	Claude / Cursor / Windsurf AI agents

What it does: exposes Vinted data as Model Context Protocol tools so an AI agent can call "search Vinted for X" and get structured results. Built on Apify's MCP server primitive.

Pick this if: you're building an AI agent (Claude Project, Cursor extension, Windsurf workflow) that needs Vinted data live, not at scrape-time.

6. automation-lab's Vinted Scraper (skip if you can)

apify.com/automation-lab/vinted-scraper

Spec	Value
Pricing	$0.0027/result + $0.00475 start
Success rate (30d)	56 % (low — known Datadome issues)

What it's good at: 16 markets supported on paper.
Tradeoff: 56 % success rate. Almost half your runs fail. At 100 items/run, you'd average 56 successful items per attempt with full charge for failures — that's effectively $4.85 / 1k usable items vs the advertised $2.70. Look at the public stats page before committing.

Verdict: skip until they fix the proxy story. The pricing looks competitive but the success rate makes it a worse deal than Turbo or Smart.

7. piotrv1001's Vinted Listings Scraper

apify.com/piotrv1001/vinted-listings-scraper

Spec	Value
Pricing	$0.001/result + $0.004 per seller detail (optional)
Success rate (30d)	100 % (but small sample — verify on their page)

What it's good at: granular pricing — you only pay for seller detail fetches if you need them. Good for resellers who only need listings 90 % of the time and seller data 10 % of the time.

Pick this if: per-seller granularity matters and you're cost-sensitive on listings-only volume.

8. saswave's Vinted Product & Profile Scraper (skip)

apify.com/saswave/vinted-product-item-profile-scraper

Spec	Value
Pricing	$0.001/result
Success rate (30d)	48.8 % (unstable)

The actor description says "$0.40/1000 results, no proxies" — but the 48 % success rate is exactly because there are no residential proxies. Without them, Datadome blocks half of the runs. Skip until proxies are added.

How to choose in 60 seconds

Are you scraping <100 listings/week for personal use?
└── YES → shahidirfan (free)
└── NO ↓

Do you need cross-country price comparison or arbitrage?
└── YES → Vinted Smart Scraper ($0.50/1k)
└── NO ↓

Are you building an AI agent (Claude / Cursor / Windsurf)?
└── YES → Vinted MCP Server (free tier)
└── NO ↓

Do you need monitoring with built-in alerts?
└── YES → epicscrapers' Monitor
└── NO → Vinted Turbo Scraper (paste URL, get JSON, done)

What to ask before paying for any Vinted scraper

Four signals to check on the actor's Apify Store page before running anything:

Public success rate? Every Apify Store actor page shows it. Anything below 90 % means failed runs you'll still get charged for (in some cases — depends on the pricing model). For Vinted specifically, <90 % is almost always a Datadome/proxy issue, and it won't get better without dev investment.
Country-bound proxies? This is the #1 reliability fix in the Vinted scraper space in 2026. If the actor doesn't bind proxy countryCode to the URL TLD, you'll see empty results on non-FR markets. Test with a vinted.de URL before committing to volume.
Pricing transparency? Look for a clear pricing tab and a maxTotalChargeUsd cap to prevent runaway runs from blowing your budget.
Pagination metadata robustness? Vinted has changed pagination response shape twice in 2026. If the actor breaks every time Vinted shifts a field, you'll get partial datasets silently.

Frequently Asked Questions

Is Vinted scraping legal in 2026?

Public catalog data (titles, prices, photos shown to anonymous browsers) sits in a grey area in most jurisdictions. The widely cited hiQ Labs v. LinkedIn (US, 2022) and the EU's Database Directive both lean permissive for non-personal data scraped from publicly accessible pages. Personal seller data is different — it's protected under GDPR and you should not redistribute it without a lawful basis. Don't scrape what you can't see logged out, don't redistribute personal data, respect Vinted's terms.

Does Vinted have a public API?

No. Vinted does not publish a public API for third-party developers as of May 2026. The internal /api/v2/catalog/items endpoint exists and is what scraper actors hit, but it's not documented and Vinted can change its shape at any time.

How much does it cost to scrape 1,000 Vinted listings?

Depends on the tool. Cheapest: shahidirfan's free actor ($0). Cheapest paid: Vinted Smart Scraper at $0.50. Mid-tier: Vinted Turbo Scraper at $3.58 (with $0.08 fixed start fee included). Most expensive in this comparison: alkausari_mujahid at $10/1k.

Will scraping Vinted ban my account?

Authenticated scraping (using your Vinted login cookies) carries account-ban risk. Anonymous scraping (no Vinted login) carries IP-ban risk on the proxy, not your account. All actors in this comparison use anonymous scraping with rotating residential proxies — no Vinted login required from you, and your account is not exposed.

What's the fastest Vinted scraper in 2026?

Vinted Turbo Scraper at ~500 items/min on a single residential session. Vinted Smart Scraper hits ~400 items/min in cross-country mode (parallelized but with country-binding overhead). epicscrapers' Monitor is rate-limited by design (you set the polling interval, not the throughput).

Can I scrape multiple Vinted countries in one run?

Yes, but only with actors that have country-binding logic. Vinted Turbo Scraper, Vinted Smart Scraper, and (partially) epicscrapers' Monitor support multi-country batch input. Other actors in this comparison default to a single country per run, which means you'd run separate jobs per country and merge the data yourself.

Difference between Vinted Turbo Scraper and Vinted Smart Scraper?

Turbo is fast and URL-native ($0.08 + $3.50/1k). Paste a search URL, get JSON in 30-60 seconds. For monitoring, alerts, simple pipelines.

Smart is research-grade ($0.50/1k flat). Cross-country comparison, seller analysis, sold-item lookup. For arbitrage research, price intelligence dashboards, academic studies.

Different tools for different intents. Both ship from the same dev team with the same anti-bot stack.

Bottom line

Most Vinted scrapers fail for the same reason: they don't country-bind their residential proxies, so non-FR markets return empty results and Datadome flags the IP pool. The 3 scrapers I'd actually use in 2026 are:

Vinted Turbo Scraper — for paste-URL → JSON workflows
Vinted Smart Scraper — for cross-country research
shahidirfan's free actor — for occasional hobby runs

Apify's Free plan gives you $5/month of credits, enough to test all three before committing.

If you'd like the full architectural writeup of how Vinted Turbo Scraper bypasses Datadome (asymmetric scraping, country-bound proxies, fail-loud retention pattern), I documented it on my GitHub repo with curl, Node, and Python integration examples.

Questions, corrections, or use cases I missed? Drop them in the comments — I update this guide monthly.

Last verified: May 6, 2026

I scraped 1,200 Shopify stores to qualify B2B leads — here's what I learned about ICP

KazKN — Tue, 05 May 2026 12:50:27 +0000

A 6-week experiment in turning competitor research into a $0.005-per-store API.

▶️ 2-minute video walkthrough of the actor in action — input, run, dataset, API call.

There is a quiet rule in B2B sales nobody puts on a slide: the cost of qualifying a bad lead is roughly equal to the cost of closing a good one.

Six weeks ago I was trying to validate ICP for a B2B side-project targeting Shopify operators. The hypothesis was tight: "Stores running Klaviyo plus a paid reviews app spend money on retention tooling, so they will pay for ours."

To test it I needed two columns next to each store name: email provider and reviews provider. Maybe a third for subscriptions. From those three I could segment 1,200 stores into a tier-one list of about 200, and avoid wasting outreach on the rest.

The data exists. It is in the page source of every Shopify store. Apps inject <script> tags from their own CDN — cdn.judge.me, cdn.yotpo.com, loox.io/widget, klaviyo.com/onsite. Any store using Klaviyo loads a Klaviyo script. The information was right there.

But three minutes of View Source per store, times 1,200 stores, is 60 hours. I do not have 60 hours.

So I did the thing I had been avoiding for months: I wrote the scraper.

What I expected to find

I expected the scraping itself to be the hard part. It was not.

I expected proxies, retries, and rate-limit roulette. None of it materialized — /products.json is publicly served on every Shopify store and the homepage HTML is, well, a homepage. No bot challenges, no CAPTCHA, no reCAPTCHA. A polite concurrency limit of 5 simultaneous requests is enough to scan 1,000 stores in 25 minutes without anyone noticing.

What I did not expect was how much the app stack tells you about the operator.

A Shopify store on Judge.me Free is a different company than a Shopify store on Yotpo Premium. Same revenue band, same vertical, same product type — totally different stage, budget, and pain points.

Judge.me Free → indie operator, doing under $30k/month, allergic to monthly subscriptions, will not buy your $99/month tool unless you frame it as ROI within 30 days.
Yotpo Premium → seven-figure DTC brand, has a marketing team, will compare you against 4 competitors in a vendor matrix before signing, and will negotiate.

You can replicate this exercise across every app category:

Klaviyo Free → still building list, every dollar matters.
Klaviyo paid (>$100/mo) → mature email program, ready for sophisticated tooling.
Postscript → SMS-first DTC, modern stack, probably trying everything.
Mailchimp → legacy stack, conservative, harder to displace.
ReCharge → subscription-driven economics, focus on retention.
Smile.io → loyalty-conscious, willing to invest in retention tooling.

Six weeks ago I would have called this overthinking. Today I run my outbound off it. Reply rate moved from 4% to 11% on a 200-account test, simply by changing the opener line to acknowledge the actual stack the prospect was running.

The build, in three observations

Observation 1 — Most "Shopify scrapers" are not.

Every existing tool I tested fell into one of two camps. Either it scraped products only (no app detection), or it detected apps but lookup-by-lookup through a Chrome extension. Nothing did both, in batch, at low volume cost.

The closest matches were paid SaaS dashboards (Storeleads, Charm, BuiltWith) at $99 to $499 per month, with monthly export caps. For a one-off list of 1,200 stores I could not justify a $1,200 yearly subscription that I would forget to use.

Observation 2 — App detection is a 10-line regex problem, not an ML problem.

Every Shopify app I needed to detect ships through one of three patterns: a <script src="cdn.[appname].com/..."> tag in the homepage HTML, a <meta name="generator"> tag with the app name, or an inline _q.push(...) queue call. Match on any of those three, OR them together, return a boolean.

The whole detection module — covering Klaviyo, Yotpo, Judge.me, Loox, Stamped, Reviews.io, ReCharge, Bold, Skio, Privy, Justuno, Mailchimp, Postscript, Attentive, Smile.io, Searchanise, Boost, and 8 others — is about 600 lines of JavaScript including snapshot tests. New detectors are 15-minute additions.

Observation 3 — Pay-per-event pricing changes the unit economics.

Apify's Store lets developers price by the row. So a 500-product store with full app detection plus reviews costs about $0.30 to scan. A thousand-store batch costs about $3.

That number matters. At $3 per batch, refreshing my ICP list weekly is a coffee. At $300 per batch (which is what specialized SaaS would charge for the same volume), I would refresh quarterly and miss every interesting signal in between.

The cheaper the unit, the higher the refresh frequency. The higher the refresh frequency, the better the signal quality. This is true for every kind of competitive intelligence work, and it's the reason I shipped the actor as a public tool instead of keeping it private.

What surprised me

Three things I did not expect, in order of how badly I underestimated them:

1. /products.json is more honest than the storefront.

Shopify's catalog endpoint exposes products that have been unpublished from the theme but are still live in the database — out-of-stock items, B2B-only SKUs, retired collections that nobody bothered to fully delete. For research, this is gold. You see what the merchant sells today and what they sold last quarter.

2. Reviews-app detection turned out to be the strongest lead signal.

More predictive than email provider, more predictive than vertical, more predictive than location. A store paying for reviews is a store that has scaled past the early stage and is now optimizing for retention and social proof. That's where my offer lands.

3. People want this packaged as an MCP tool for Claude.

Two of the first three external users asked. I had not planned for it. The pattern is clear though — once you can pipe Shopify-store data into Claude or Cursor and ask "qualify these 200 stores for my ICP", you stop opening spreadsheets. I am building it next.

The actor, if you want to use it

I shipped the scraper on Apify Store as Shopify Apps Spy + Product Scraper.

What it does in one call:

Pulls the full product catalog for a list of Shopify URLs (titles, prices, variants, images, vendor, tags).
Detects installed apps across email/SMS, reviews, subscriptions, popups, search, loyalty.
Pulls reviews when a reviews app is detected, by routing to that app's public reviews API.

What it costs:

$0.005 per store for the standard tier (products + apps).
$0.30 for a 500-product store with full reviews.
Apify gives a $5 free credit on signup, which covers about 1,500 stores.

What it doesn't do:

Historical data. If you need "who started using Klaviyo in Q1 2024," you want BuiltWith.
Cross-platform. Shopify only. WooCommerce/Magento are different problems.
Filtering by revenue band. Storeleads does that better.

If you're an indie founder, agency analyst, or sales rep doing 100-2,000 stores per month and you need raw exports, it should fit. If your volume is much larger or much smaller, the SaaS competitors are probably the right call.

The takeaway, if you skim

Three things I would do differently if I were starting over:

Build the qualification tool before you start prospecting, not after. The 60-hour manual baseline is what kills the experiment. Every B2B founder I have asked has the same story.
Treat tech-stack data as ICP data, not technographic trivia. The app a store runs is downstream of their stage, budget, and team size. Use it that way.
Refresh weekly, not quarterly. Cheap refresh frequency beats expensive depth nine times out of ten in early-stage outbound.

The scraper is on Apify Store, free $5 credit covers your first batch. If a detector is missing, ping me — each is a 15-minute add.

FAQ

How do I detect what apps a Shopify store is using?
Apps inject identifiable scripts into the storefront HTML — cdn.judge.me, cdn.yotpo.com, klaviyo.com/onsite, etc. Either inspect the page source manually (3 minutes per store) or use Shopify Apps Spy + Product Scraper to detect 150+ apps in batch at $0.005 per store.

Is scraping Shopify legal?
Yes for publicly accessible product data. Shopify exposes /products.json on every storefront and the homepage HTML is public. No login, no API key, no proxy needed for most stores. You're reading what the merchant chose to publish.

How long does it take to scan 1,000 Shopify stores?
About 25 minutes at a polite concurrency of 5 simultaneous requests. The bottleneck is /products.json response size, not rate limits — Shopify storefronts handle this volume without complaint.

What's a realistic cost for B2B lead qualification across 1,200 Shopify stores?
Around $3 of compute on Apify's pay-per-event pricing — $0.005 per store for products + apps detection, $0.30 for full reviews. The $5 free Apify credit covers your first ~1,500 stores.

Which Shopify apps are the strongest signal of B2B SaaS fit?
Reviews providers (Yotpo Premium, Okendo, Stamped Pro) signal seven-figure DTC. Klaviyo paid plans signal a mature email program. Postscript or Attentive signal SMS-first modern stack. Smile.io signals retention-conscious operators ready to invest in tooling.

Tags: shopify, b2b, lead-generation, ecommerce, indiehackers

I built a Shopify scraper that detects apps + pulls products in one API call

KazKN — Sat, 02 May 2026 11:50:01 +0000

TL;DR — Existing Shopify app detectors (Koala Inspector, ShopScan, Fera, BuiltWith) are Chrome extensions or SaaS dashboards. None do batch. I had 1,200 stores to qualify and View Source + Cmd-F was killing my afternoons, so I shipped an Apify actor that takes a list of Shopify URLs and returns the full app stack (Klaviyo, Yotpo, Judge.me, Loox, ReCharge…) + product catalog + reviews in JSON. No headless browser, ~$0.005 per store, 1,000 stores in 25 minutes. Live here → Shopify Scraper – Apps Spy + Reviews.

The afternoon that broke me

Six weeks ago I was prospecting for a B2B side-project. The hypothesis: "Shopify stores running Klaviyo + a paid reviews app are the right ICP — they spend money on retention tooling, so they will pay for ours."

To validate, I needed a list of Shopify stores and their installed apps.

The Shopify App Store does not give you that. The "stores using X" databases do, but the public ones are stale and the good ones are paid SaaS at $99–499/month for filters I did not need.

So I did what every founder does at 11 PM: I opened View Source on a competitor list, hit Cmd-F, and started typing klaviyo.

It worked. Sort of. I did 40 stores in two hours, then stopped, because I had a list of 1,200.

That night I wrote the first version of what is now Shopify Scraper – Apps Spy + Reviews.

What I actually wanted

Every "Shopify scraper" I found online did one of two things:

Scraped a single store's products via /products.json — table-stakes, dozens of free Apify actors do it.
Spawned a headless browser to fingerprint a marketing site — slow, expensive, and brittle.

I wanted three things in one pass:

Full product catalog (titles, prices, variants, images, vendor, tags) — nothing exotic.
App detection: which third-party Shopify apps are installed (email, reviews, subscriptions, popups, search).
Reviews when a reviews app is detected — pull them via the public API, not by parsing widgets.

And I wanted it to be cheap, because I had ~1,200 stores in my first batch and I planned to run it monthly.

The "no headless browser" decision

The thing nobody tells you about Shopify scraping is that you almost never need a headless browser. The signals you want for app detection live in three places, and all three are reachable with a plain HTTPS GET:

The HTML of the homepage. Shopify apps inject <script> tags from their own CDN. cdn.judge.me, cdn.yotpo.com, loox.io/widget, klaviyo.com/onsite — you grep the HTML and you know.
/products.json. Shopify exposes the full catalog at this path on every store, paginated 250 items at a time. No auth, no headless. (You hit a soft rate limit around 2 req/s per IP, which is fine if you queue politely.)
App-specific public endpoints. Judge.me has a JSON reviews endpoint. Yotpo too. Same for Loox, Stamped, Reviews.io. Once you know which app is installed, you go straight to its API — no DOM parsing.

The whole actor is built around that observation. No Puppeteer, no Playwright, no proxy farm. Just got-scraping, cheerio, and p-queue to keep concurrency civilized.

The result is that scanning a single store costs ~3–6 HTTPS requests and runs in 2 to 8 seconds depending on catalog size. Cost on Apify infra: about $0.005 per store for the "tech stack only" mode.

The architecture (it is small on purpose)

I'll be honest — I almost over-engineered this. My first draft had Redis for de-dup, a queue, retry logic with exponential backoff, and a state machine. Then I deleted all of it.

Here is what shipped:

src/
├── main.js                   # orchestration (p-queue, per-store flow)
├── crawlers/
│   ├── products.js           # /products.json + sitemap fallback
│   ├── apps.js               # detect apps from homepage HTML
│   └── reviews.js            # per-app reviews fetchers
└── lib/
    ├── normalize.js          # canonicalize URLs, normalize product schema
    ├── schemas.js            # zod validation for outputs
    └── billing.js            # Apify pay-per-event charges

A run goes:

Canonicalize the store URL (handles www, custom domains, *.myshopify.com).
Fetch the homepage once. Confirm it is Shopify (the x-shopify-stage header is a giveaway).
From the same HTML, run the app detectors. Each detector is ~10 lines of regex matching against script tags + meta tags + inline JSON.
Fetch /products.json?page=N until you hit the cap or run out of products.
If the user asked for reviews and an installed reviews app was detected, fan out to that app's public reviews API.

That is it. The whole thing is ~900 lines of JavaScript. I run it with node --test for unit tests against snapshots and a tests/smoke-products.mjs that hits 5 real stores end-to-end.

What I learned about app detection

The regex-against-HTML approach has one trap. Shopify themes minify, version, and CDN-rewrite their assets, so you cannot match on a single string. The Klaviyo loader, for example, ships under at least four URL patterns I have seen:

static.klaviyo.com/onsite/js/klaviyo.js
static-tracking.klaviyo.com/onsite/js/...
a.klaviyo.com/media/...
inline _learnq queue calls

You match any of those, and you call it Klaviyo. Same logic for every other app — every detector is an array of patterns, OR'd together, returning a single boolean. I wrote a snapshot test per app with a real store HTML page so a Klaviyo URL change does not silently break detection.

The detectors I shipped on day one:

Email/SMS: Klaviyo, Omnisend, Postscript, Mailchimp, Attentive
Reviews: Yotpo, Judge.me, Loox, Stamped, Reviews.io, Okendo
Subscriptions: ReCharge, Bold, Loop, Skio
Popups & SMS capture: Privy, Justuno, Klaviyo Forms
Search & discovery: Searchanise, Boost, Algolia
Loyalty: Smile.io, Yotpo Loyalty, LoyaltyLion

If you tell me an app I missed, I add a detector. Each one is a 15-minute job.

The pay-per-event pricing problem

Apify lets you charge per event instead of per compute minute. For a scraper that runs in seconds, this is the right model — your customer pays for the rows they get, not for compute time.

The mistake I made on my first push was leaving Apify's default dataset_item event on. Combined with my custom product_extracted event, every product was being charged twice. I caught it in monetization review and removed the synthetic event.

The pricing I landed on:

store_analyzed — $0.003 per store (covers detection + products fetch)
product_extracted — $0.0005 per product
apps_detected — $0.001 per store at standard+
review_extracted — $0.0003 per review

A 500-product store with reviews costs roughly $0.30 end to end. For comparison, the SaaS competitors charge $99 or more for similar lookups, batched and capped.

What surprised me

Three things, in order of how badly I underestimated them.

1. /products.json is more honest than the storefront. It exposes products that are unpublished from the theme but still live (out-of-stock holdovers, B2B-only SKUs). Useful for trend research. Sometimes shocking.

2. Reviews-app detection is a lead signal. A store on Judge.me Free plan vs. Yotpo Premium tells you a lot about their stage. I ended up using this internally to prioritize cold outbound — different pitch for a $30/month stack vs. a $1,200/month stack.

3. People want this as an MCP server. Two of my first three users asked if they could query it from Claude / ChatGPT. I have it on the roadmap. (My GPT Crawler MCP and Vinted MCP Server are the two MCP actors I shipped first; the Shopify one is next.)

How to use it in one minute

// On Apify, paste this in the actor input box
{
  "store_urls": ["https://allbirds.com", "https://gymshark.com"],
  "extract_level": "standard",  // products + apps stack
  "max_products_per_store": 250
}

Output (one record per product, with apps_detected attached):

{
  "store_domain": "allbirds.com",
  "product_title": "Wool Runner",
  "price": 110,
  "available": true,
  "vendor": "Allbirds",
  "apps_detected": {
    "email": ["Klaviyo"],
    "reviews": ["Yotpo"],
    "subscriptions": [],
    "search": ["Searchanise"]
  },
  "product_url": "https://allbirds.com/products/mens-wool-runners"
}

If you want reviews, set extract_level: "full" and a max_reviews_per_product. The actor will route to the correct reviews API based on what was detected.

Direct link, free $5 credit on Apify, no account-creation drama: Shopify Scraper – Apps Spy + Reviews.

FAQ

Is scraping `/products.json` allowed?

Shopify exposes /products.json publicly on every store by default. Stores that disable it (rare) return 404 and the actor logs a skip. The actor never authenticates, never bypasses access controls, and respects standard rate limits.

What about reCAPTCHA or Cloudflare?

For the standard catalog and app-detection flow, no. /products.json and the homepage HTML are not gated. For some reviews APIs, very high request volumes can trigger rate-limits — the actor backs off and retries with jitter.

How is this different from Koala Inspector, ShopScan or BuiltWith?

Koala Inspector, ShopScan and Fera are excellent Chrome extensions for one-store lookups, but none of them do batch — you cannot paste 500 URLs and get a CSV back. BuiltWith is a generic tech-stack tool with broad coverage but its Shopify-app detection is shallow and you cannot pull products in the same call. This actor is purpose-built for Shopify and runs in batch via API: deeper app detection (subscriptions, reviews, popups, search, loyalty), full product catalog, and reviews — all in one pass, billed pay-per-event.

How long does a 1,000-store scan take?

About 25 minutes at default concurrency, costing ~$3 of Apify credits at the standard level. A full run with reviews is closer to an hour and ~$15 depending on review volume.

Can I get one record per variant instead of per product?

Yes. Set include_variants: true in the input and the dataset returns one row per SKU with size/color/price/availability normalized.

What is next

I want to add three things, in order:

Revenue estimation at the pro tier — based on review velocity and product velocity, both of which are observable.
MCP server mode so you can query it from Claude desktop / Cursor.
Theme detection — useful for agency outbound, less useful for me, but I keep being asked.

If you use it and something breaks, ping me — I am the only maintainer and I read every issue. The actor is on Apify Store at kazkn/shopify-scraper-apps-spy.

Tags: #shopify #ecommerce #api #indiehackers

Was this useful? ❤️ a reaction or drop a comment with the use-case you're trying to solve — I read every reply and add detector + endpoint coverage based on what people actually need.

Follow @boo_n for hands-on tutorials: scraping reviews at scale, ICP qualification at $0.005 per store, and turning the actor into an MCP tool for Claude / Cursor.