DEV Community: Scrape-Smith

How to Scrape Bluesky Posts, Hashtags and Followers with the Official AT Protocol API

Scrape-Smith — Fri, 23 Oct 2026 18:00:00 +0000

How to Scrape Bluesky Posts, Hashtags and Followers

Bluesky has grown into a real place for tech, journalism, science and politics conversations, and unlike some other networks, it has an open, documented public API: the AT Protocol. That makes it one of the friendliest platforms for social listening, research and audience analysis.

This tutorial covers what the public API offers, then uses the Bluesky Scraper to export posts, profiles and followers to a spreadsheet without code, plus a short Python example.

What you can get from the public API

Bluesky's AppView (public.api.bsky.app) serves, without login:

Post search by keyword, hashtag, phrase or author
Author feeds: everything a user posted
Profiles: display name, bio, follower/following/post counts
Followers and follows lists
User search by name or bio

The actor wraps all of these, handles cursor pagination and flattens the nested records into rows with like, repost, reply and quote counts, hashtags, links and image URLs.

Step 1: Search a hashtag

Open the actor and click Try for free. It's prefilled with #buildinpublic and 20 items. Start it and you get rows like:

type	authorHandle	text	likeCount	createdAt
post	someone.bsky.social	Build log, day 42: shipped...	12	2026-10-22

Search supports Bluesky's syntax: #tag, "exact phrase", from:handle.bsky.social, and plain keywords. Use sort=top for the most engaged posts or latest for a chronological stream. lang=en narrows by language.

Step 2: Profiles and followers

Fill any of these fields (one handle per line):

authorFeeds: export a user's posts (includeReplies to add replies).
profiles: full profile rows.
followersOf / followsOf: one profile row per follower or followed account.
userSearch: find accounts by keyword, e.g. data journalist.

maxItems caps each query or handle, so a typo can't run up a bill.

Step 3: Monitor a topic daily

from apify_client import ApifyClient
import datetime

since = (datetime.date.today() - datetime.timedelta(days=1)).isoformat()
client = ApifyClient("YOUR_APIFY_TOKEN")
run = client.actor("scrape-smith/bluesky-posts-profiles").call(run_input={
    "searchQueries": ["\"your brand\"", "#yourproduct"],
    "sort": "latest",
    "onlyNewSince": since,
    "maxItems": 500,
})
posts = [p for p in client.dataset(run["defaultDatasetId"]).iterate_items() if p.get("type") == "post"]
posts.sort(key=lambda p: p.get("likeCount", 0), reverse=True)
for p in posts[:10]:
    print(p["likeCount"], p["authorHandle"], p["text"][:100], p["url"])

That's a daily "top mentions" digest in about fifteen lines.

Do I need a Bluesky login?

Usually not. Bluesky has at times restricted unauthenticated post search. If that happens, the actor can fall back to your handle plus an app password (Settings > Privacy and security > App passwords). Never use your main password. Without one, a refused request returns a free error row explaining what to add, rather than failing silently.

Use cases

Social listening and brand monitoring.
Hashtag and trend research for content and community teams.
Influencer discovery: search users by bio, then export their follower counts.
Academic research and NLP datasets.
Archiving your own posts.

Honest limitations and etiquette

Search results come from Bluesky's index, which may not include every post ever made, and very old posts can be harder to reach through search than through author feeds.
Respect deletions. If you store data, refresh it rather than keeping posts people deleted, and follow Bluesky's developer guidelines and applicable privacy law.
Follower exports of very large accounts are large; use maxItems.
Not affiliated with Bluesky.

Cost

$0.0005 per post or profile row. The 20-post test is a cent; 1,000 posts is $0.50; 10,000 followers is $5. Error rows are free.

Wrap-up

Bluesky's open API is a gift for anyone who does social research. The Bluesky Scraper turns it into CSVs and scheduled feeds with no code. Start with the prefilled hashtag and swap in your own topics.

Scrape-Smith builds small, cheap data tools on public and official APIs.

How to Check US Trademark Status in Bulk with the USPTO TSDR API

Scrape-Smith — Thu, 22 Oct 2026 16:42:02 +0000

How to Check US Trademark Status in Bulk with the USPTO TSDR API

If you manage more than a handful of US trademarks, for clients or your own brands, you know the routine: open TSDR, paste a serial number, read the status, copy it into a spreadsheet, repeat. It's fine for five marks. It's a lost afternoon for two hundred.

The USPTO has an official API for exactly this, called TSDR (Trademark Status & Document Retrieval). This tutorial explains how to get access, what it returns, and how to run bulk lookups with the USPTO Trademark Status Lookup actor without writing any code.

Step 1: Get a free USPTO API key

TSDR requires an API key. It is free:

Go to https://account.uspto.gov/api-manager and sign in with (or create) a USPTO.gov account.
Request a TSDR API key.
Copy the key somewhere safe.

The key is personal and rate limited to 60 requests per minute. That's why this actor is "bring your own key": you stay in control of your own access and quota, and nothing is shared.

Step 2: Collect serial numbers

TSDR looks up marks by 8-digit serial number (the application number), such as 78787878. If you only have a registration number, look it up once on tsdr.uspto.gov to find the serial. Most docketing exports already include serials.

Step 3: Run the lookup

Open the actor, paste your serial numbers one per line, and paste your key into the API key field (it's stored as a secret input). Click Start.

Each successful row contains:

Field	Example meaning
`serialNumber`	78787878
`markText`	The word mark
`status`	e.g. "Registered", "Abandoned", "Published for opposition"
`statusDate`	When the status last changed
`filingDate`, `registrationDate`, `registrationNumber`	Key dates and number
`owner`	Current owner name
`goodsServices`	Goods & services descriptions

If you run without a key (or with a wrong one), each row comes back as a free error row saying HTTP 401 - API key required/invalid. That's deliberate: you can test the tool safely before setting up the key, and you're never charged for failures.

Step 4: Automate a weekly status report

Schedule the actor weekly and export to Google Sheets, or from Python:

from apify_client import ApifyClient

client = ApifyClient("YOUR_APIFY_TOKEN")
serials = [line.strip() for line in open("portfolio.txt") if line.strip()]
run = client.actor("scrape-smith/uspto-trademark-status").call(run_input={
    "serialNumbers": serials,
    "apiKey": "YOUR_USPTO_KEY",
})
for r in client.dataset(run["defaultDatasetId"]).iterate_items():
    if "error" in r:
        print("CHECK", r["serialNumber"], r["error"])
    else:
        print(r["serialNumber"], r["status"], r["statusDate"])

Store last week's output and diff it: any row whose status or statusDate changed is something a human should look at (an office action, publication, registration, or abandonment).

Who this is for

Small IP practices that want a status sheet without a docketing system.
Brand owners with a portfolio across product lines.
Due diligence before acquiring a business or licensing a brand.
Competitive watch: track competitors' pending applications as they move through examination.

Honest limitations

You need your own USPTO key. There's no way around that, and that's a good thing.
The USPTO limit is 60 requests per minute, so the actor paces itself at about one lookup per second. 1,000 marks take about 17 minutes.
It looks up known serial numbers. It is not a clearance search by mark name.
Status text comes straight from USPTO. It is not legal advice; deadlines should still be checked by an attorney.
Not affiliated with the USPTO.

Cost

$0.004 per successful lookup; errors are free. 100 marks is $0.40, 1,000 marks is $4. Apify platform usage is included.

Wrap-up

The TSDR API turns trademark status checks from a copy-paste chore into a scheduled report. If you'd rather not write the plumbing, the USPTO Trademark Status Lookup does it with your own key in a couple of clicks.

Scrape-Smith builds small, cheap data tools on public and official APIs.

How to Export Steam Game Reviews (with Playtime) for Player Sentiment Analysis

Scrape-Smith — Thu, 22 Oct 2026 16:38:40 +0000

How to Export Steam Game Reviews for Player Sentiment Analysis

Steam reviews are unusually rich feedback. Every review is a thumbs up or down, it comes with how many hours the reviewer had played, and other players vote on whether it was helpful. For game developers, publishers and analysts, that's a sentiment dataset most industries would envy.

Steam shows reviews nicely on the store page, but not in a form you can analyse. This tutorial uses Valve's public reviews endpoint through the Steam Reviews Scraper to export them, then looks at a couple of analyses worth doing.

The endpoint

Valve documents a public endpoint:

https://store.steampowered.com/appreviews/440?json=1&filter=recent&language=all&num_per_page=100

It returns reviews in pages with a cursor for the next page, plus a summary like "Very Positive". The actor handles cursor pagination, throttles to roughly one request per second, and flattens each review into a row.

Step 1: Run the example

Open the actor and click Try for free. It's prefilled with app 440 (Team Fortress 2) and 20 reviews:

gameName	recommended	playtimeAtReviewMinutes	votesUp	language
Team Fortress 2	true	1236	0	spanish

Each row includes review text, recommended, votesUp, votesFunny, weightedVoteScore, playtimeForeverMinutes, playtimeAtReviewMinutes, playtimeLastTwoWeeksMinutes, steamPurchase, receivedForFree, writtenDuringEarlyAccess, primarilySteamDeck, author stats and any developerResponse.

Step 2: Pick your game and filters

appIds: the number in the store URL (store.steampowered.com/app/1091500/...) or the whole URL.
filter: recent (newest first; the way to page through full history), updated, or all (most helpful first, optionally with dayRange).
language: english, german, schinese... or all.
reviewType: positive or negative only.
purchaseType: Steam purchases vs. keys bought elsewhere.

Step 3: Two analyses worth doing

from apify_client import ApifyClient
import pandas as pd

client = ApifyClient("YOUR_APIFY_TOKEN")
run = client.actor("scrape-smith/steam-reviews-scraper").call(run_input={
    "appIds": ["1091500"],
    "maxReviewsPerApp": 5000,
    "filter": "recent",
    "language": "english",
})
df = pd.DataFrame(client.dataset(run["defaultDatasetId"]).iterate_items())
df["created"] = pd.to_datetime(df["createdAt"])
df["hours"] = df["playtimeAtReviewMinutes"] / 60

# 1) Sentiment over time: did the last patch help?
print(df.set_index("created").resample("W")["recommended"].mean().tail(8))

# 2) Do experienced players feel differently?
df["bucket"] = pd.cut(df["hours"], [0, 2, 10, 50, 200, 10000])
print(df.groupby("bucket", observed=True)["recommended"].agg(["mean", "count"]))

The first table shows the weekly share of positive reviews, the fastest way to see a patch landing well or badly. The second often reveals a pattern: players under two hours (refund window) and players over 200 hours can feel very differently about the same game. Both are useful, for different reasons.

From there, feed negative reviews from heavy players into an LLM or topic model to find the issues your most committed players care about.

Use cases

Live-ops and patch monitoring for studios.
Competitor and genre research before greenlighting a project.
Publisher due diligence on games they might sign.
Sentiment and NLP datasets with a built-in label (recommended or not).
Daily alerts for new negative reviews.

Honest limitations

Some orderings (like all) don't go back through the full history. Use recent to page through everything.
Huge games have hundreds of thousands of reviews; at one request per second, exporting all of them takes a while. Set maxReviewsPerApp.
Review text is user content. If you publish it, attribute and respect Steam's terms.
Not affiliated with or endorsed by Valve.

Cost

$0.0003 per review. The 20-review test is well under a cent; 10,000 reviews is $3. Error rows are free and you can cap spend per run.

Wrap-up

Steam gives you labelled, playtime-weighted feedback for every game on the platform. The Steam Reviews Scraper gets it into a DataFrame in a minute. Try the prefilled example, then point it at your game or the competitor you keep comparing yourself to.

Scrape-Smith builds small, cheap data tools on public and official APIs.

How to Export iOS App Store Reviews by Country (for Sentiment Analysis and ASO)

Scrape-Smith — Thu, 22 Oct 2026 16:35:09 +0000

How to Export iOS App Store Reviews by Country

App reviews are the most honest product feedback you will ever get, and also the most annoying to collect. App Store Connect only shows reviews for your own apps, one storefront at a time, and third-party ASO suites are priced for agencies. If you just want a spreadsheet of recent reviews for your app or a competitor's, there is a simpler route: Apple's own public customer-reviews RSS feed.

This tutorial walks through that feed, exports reviews with the Apple App Store Reviews Scraper, and ends with a quick sentiment pass in Python.

The feed behind it

Apple publishes recent reviews for every app at a URL like:

https://itunes.apple.com/us/rss/customerreviews/page=1/id=284882215/sortby=mostrecent/json

us is the storefront, 284882215 is the app ID (Facebook here), and each page holds up to 50 reviews. There are up to 10 pages, so roughly the 500 most recent reviews per app per country.

The JSON is awkward (every value is wrapped in a label object, and the first entry is sometimes app metadata rather than a review), which is why a small wrapper helps.

Step 1: Find the app ID

Open the app on apps.apple.com. The ID is the number after id in the URL: apps.apple.com/us/app/facebook/id284882215. You can also paste the full URL into the actor.

Step 2: Run it

Open the actor and click Try for free. The prefilled input is one app, the us store and one page, so you get 50 real reviews in a few seconds:

rating	title	appVersion	updated
3	Finding post	584.0.0	2026-10-20
1	...	584.0.0	...

Each row has appId, country, reviewId, title, text, rating (1–5), appVersion, author, updated and voteCount.

Step 3: Add countries

Reviews are per storefront. A German user's review only shows up in the de store. Add the markets you care about, one per line: us, gb, de, fr, jp, br... Set maxPages to 10 to get everything Apple exposes.

Tip: the appVersion column is gold. Group by version and you can see whether a release made ratings better or worse.

Step 4: A five-minute sentiment check

from apify_client import ApifyClient
import pandas as pd

client = ApifyClient("YOUR_APIFY_TOKEN")
run = client.actor("scrape-smith/apple-app-store-reviews").call(run_input={
    "appIds": ["284882215"],
    "countries": ["us", "gb", "ca", "au"],
    "maxPages": 10,
})
df = pd.DataFrame(client.dataset(run["defaultDatasetId"]).iterate_items())

print(df.groupby("appVersion")["rating"].agg(["mean", "count"]).tail(5))
low = df[df.rating <= 2]
print(low["text"].str.lower().str.extractall(r"(crash\w*|login|ads?|slow|bug\w*)")[0].value_counts())

The first print shows average rating per recent version. The second counts common complaint words in 1–2 star reviews. Crude, but it gets you 80% of the insight before you reach for an LLM. If you do want an LLM, the reviews CSV is a perfect prompt input ("cluster these complaints into themes").

Use cases

Release monitoring: schedule it daily and alert on new 1-star reviews.
Competitor research: mine competitors' reviews for missing features.
ASO keyword research: real users' words are the words people search.
NLP datasets for sentiment models.
Support triage: route review complaints to the right team.

Honest limitations

Apple's feed only exposes about 500 recent reviews per app per country. For an app with millions of reviews, this is a rolling window, not the full history. Schedule regular runs if you want to build history over time.
Ratings without written reviews aren't included; Apple doesn't publish them in the feed.
No developer replies; the feed doesn't contain them.
This is App Store only, not Google Play.
Not affiliated with Apple. It reads a public feed and doesn't log in.

Cost

$0.0003 per review. The 50-review test costs 1.5 cents; 1,000 reviews is $0.30. Apify platform usage is included.

Wrap-up

For most "what are users saying?" questions, Apple's RSS feed plus a spreadsheet is enough. Try the App Store Reviews Scraper with the prefilled app, then replace it with yours or a competitor's.

Scrape-Smith builds small, cheap data tools on public and official APIs.

How to Scrape Job Postings from Workable, Personio, SmartRecruiters and More (One Clean Feed)

Scrape-Smith — Thu, 22 Oct 2026 16:32:26 +0000

How to Scrape Job Postings from Workable, Personio, SmartRecruiters and More

Building a niche job board, tracking which competitors are hiring, or finding companies that just opened a "Head of Data" role all start the same way: you need job postings from many company career pages, in one consistent format.

The good news is that most applicant tracking systems (ATS) publish official public job feeds so their customers' jobs can be syndicated. The bad news is that every ATS does it differently: Personio speaks XML, Teamtailor RSS, Workable and Recruitee JSON with different field names, SmartRecruiters a paginated REST API, and BambooHR its own list format.

This tutorial shows how to pull all six into one normalized table with the ATS Jobs Scraper.

Supported ATS and how to identify them

Look at a company's careers page URL:

ATS	Careers URL looks like	Input
Workable	apply.workable.com/huggingface	`workable:huggingface`
Recruitee	bunq.recruitee.com	`recruitee:bunq`
Personio	acme.jobs.personio.de	`personio:acme`
Teamtailor	polestar.teamtailor.com	`teamtailor:polestar`
SmartRecruiters	jobs.smartrecruiters.com/Ubisoft2	`smartrecruiters:Ubisoft2`
BambooHR	bitly.bamboohr.com/careers	`bamboohr:bitly`

You can type ats:slug or just paste the careers URL. Mixing ATS in one run is fine.

Tip: if a company's careers page is on its own domain, click any job. The apply link usually reveals which ATS is behind it.

Step 1: Run the example

Open the actor and click Try for free. The prefill includes a few real companies and a limit of 20 jobs per company. Within seconds:

ats	company	title	location	remote	postedAt
teamtailor	Polestar	Head of Customer Engagement Benelux	Beesd, Netherlands	false	2026-10-15
workable	Hugging Face	...	Remote	true	...

Every row has the same fields regardless of source: ats, companySlug, company, id, title, location, city, country, remote, department, employmentType, experienceLevel, postedAt, url, applyUrl, description. If an ATS doesn't expose a field, it's null rather than missing.

Step 2: Filter

keyword: case-insensitive title filter, e.g. engineer or sales.
onlyNewSince: skip jobs posted before a date. Combine with a daily schedule to get only new jobs each day.
includeDescription: on by default. Turn it off for faster, title-only runs.

Step 3: Daily "new jobs" feed

from apify_client import ApifyClient
import datetime

yesterday = (datetime.date.today() - datetime.timedelta(days=1)).isoformat()
client = ApifyClient("YOUR_APIFY_TOKEN")
run = client.actor("scrape-smith/multi-ats-jobs-feed").call(run_input={
    "companies": [line.strip() for line in open("companies.txt") if line.strip()],
    "onlyNewSince": yesterday,
    "keyword": "data",
    "includeDescription": False,
})
for job in client.dataset(run["defaultDatasetId"]).iterate_items():
    if "error" not in job:
        print(f'{job["company"]}: {job["title"]} ({job["location"]}) {job["url"]}')

Pipe that into Slack, email, or a static site generator and you have a niche job board backend.

Use cases

Niche job boards (remote, climate, Rust, EU startups) from hundreds of career pages.
Competitive intelligence: hiring is a leading indicator. A competitor posting ten sales roles in Germany is telling you something.
Sales prospecting: companies hiring for a role often need tools for that role.
Recruiting research and labour-market analytics.

Honest limitations

Only these six ATS. Greenhouse, Lever, Ashby and Workday aren't covered by this actor.
You need the company slug. The tool doesn't discover companies for you; you bring the list.
Some ATS don't publish a posted date (so onlyNewSince can't filter those), and some don't expose remote flags.
If a company disables its public feed or the slug is wrong, you get a free error row for that company and the rest continue.
These are official public feeds meant for syndication, but check each company's terms if you republish job content, and always link to the original apply URL.

Cost

Pay per result: $0.005 per company feed fetched plus $0.001 per job. The test run costs about three cents. 1,000 jobs from 20 companies is $1.10. Platform usage is included.

Wrap-up

Six ATS, six formats, one table. If you're building anything on top of job data, try the ATS Jobs Scraper with the prefilled companies and then paste in your own list.

Scrape-Smith builds small, cheap data tools on public and official APIs.

How to Get the Full Wayback Machine History of Any Website (No Code or Python)

Scrape-Smith — Thu, 22 Oct 2026 16:24:13 +0000

How to Get the Full Wayback Machine History of Any Website

If you have ever tried to answer "when did this competitor change their pricing page?" or "what did this domain look like before I bought it?", you have probably clicked around web.archive.org one calendar dot at a time. It works, but it does not scale past a handful of pages, and you cannot put it in a spreadsheet.

This tutorial shows how to pull the complete capture history of a URL or domain into a table you can filter, chart and diff. We will use the Internet Archive's public CDX API through a small hosted tool, the Wayback Machine Scraper on Apify, and then look at the same thing from Python.

What the CDX API actually gives you

The Wayback Machine keeps an index of every capture it has made. Each entry has:

the original URL and capture timestamp
the HTTP status code the archive saw (200, 301, 404...)
the MIME type
a content digest (a hash of the page body)

The digest is the interesting bit. If two consecutive captures have the same digest, the page did not change. If it differs, something on the page changed. That turns the archive from a photo album into a change log.

The raw API is free and public, but it has quirks: timestamps like 20200109130705, pagination by line, and collapse rules that are easy to get wrong. The actor wraps all of that and returns clean rows.

Step 1: Run it with the example input

Open the actor page and click Try for free. The input is prefilled with apify.com, monthly collapse, from 2020, limit 20. Click Start. In a few seconds you get rows like:

capturedAt	statusCode	contentChanged	snapshotUrl
2020-01-09T13:07:05Z	200	null	https://web.archive.org/web/20200109130705/https://apify.com/
2020-02-03T...	200	true	...

contentChanged is null for the first row (nothing to compare against) and true/false after that.

Step 2: Choose the right collapse mode

The collapse field decides how many rows you get back:

month: at most one capture per month. Best for a quick overview of a site over years.
day: one per day. Good for a period you care about, like a product launch.
digest: only captures where the content changed. This is the one to use for change tracking.
none: every single capture. Popular sites can have tens of thousands, so set a limit.

Step 3: Pick a match type

exact looks at one URL, such as example.com/pricing.
prefix covers everything under a path, such as example.com/blog/. Handy for finding deleted blog posts.
domain covers the whole domain including subdomains.

A practical example: to see every blog URL a competitor ever published, use matchType=prefix, collapse=digest and a high limit. Then sort by URL and look for posts that return 404 today.

Step 4: Export or automate

Every run produces a dataset you can download as CSV, Excel or JSON. You can also schedule the actor weekly and get only new captures by setting from to last week's date.

From Python with the official client:

from apify_client import ApifyClient

client = ApifyClient("YOUR_APIFY_TOKEN")
run = client.actor("scrape-smith/wayback-snapshots").call(run_input={
    "urls": ["example.com/pricing"],
    "from": "2022",
    "collapse": "digest",
    "limit": 200,
})
for row in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(row["capturedAt"], row["statusCode"], row["snapshotUrl"])

That prints one line per version of the pricing page since 2022. Open the snapshotUrl of two neighbours to see exactly what changed.

Use cases that come up a lot

SEO audits. When did a page start redirecting? When did the title change? Status-code history answers both.
Competitor monitoring. Pricing, plans and positioning pages, version by version.
Expired-domain due diligence. See whether a domain you are about to buy used to host spam.
Content recovery. Find old copies of pages you deleted by accident.
Research and journalism. Show how an organisation's public statements evolved.

Honest limitations

The archive only has what it crawled. Small sites may have a few captures a year.
The tool returns snapshot metadata and links, not the page HTML. If you need the HTML, fetch the snapshotUrl yourself (add id_ after the timestamp for the raw page).
The digest changes on any byte change, so pages with rotating timestamps or ads will show "changed" more often than a human would say.
This is not an official Internet Archive product. Please be kind to their servers; the actor makes a modest number of requests.

What it costs

Pricing is pay-per-result: $0.0005 per snapshot plus $0.003 per run. A monthly history of one domain since 2020 is around 80 rows, so about four cents. 1,000 snapshots is $0.50. Apify's free plan credit easily covers experimenting.

Wrap-up

The Wayback Machine is one of the most useful free datasets on the web, and the CDX index makes it queryable. If you want to try it without writing any code, the Wayback Machine Scraper has a prefilled example you can run in one click. If you build something with it, I would love to hear what.

Scrape-Smith builds small, cheap data tools on public and official APIs.

How to Export Any Shopify Store's Product Catalog and Prices to CSV

Scrape-Smith — Thu, 22 Oct 2026 16:22:40 +0000

How to Export Any Shopify Store's Product Catalog and Prices to CSV

Roughly a quarter of online stores run on Shopify, and almost all of them expose something many people don't know about: a public JSON product feed. If you do price monitoring, dropshipping research or competitor analysis, this feed is the cleanest data source you will find, and you don't need a headless browser or proxies to use it.

In this tutorial we'll look at how the feed works, then export a full catalog with the Shopify Products Scraper on Apify, and finish with a small price-tracking workflow.

The /products.json trick

Open any Shopify store and add /products.json to the domain, for example https://www.allbirds.com/products.json. You will see a JSON list of products with titles, handles, vendors, product types, tags, images and every variant with its price, compare-at price and availability.

Two catches:

It is paginated (?limit=250&page=2 ...), so large stores need many requests.
Prices and availability live inside variants, so you have to flatten them to get a useful table.

That is exactly the boring part a tool should do for you.

Step 1: Check the store is on Shopify

Visit https://thestore.com/products.json. JSON means you're good. A 404 or an HTML page means the store isn't on Shopify or has disabled the feed (rare, but it happens).

Step 2: Run the scraper

Open the Shopify Products Scraper and click Try for free. The input is prefilled with https://www.allbirds.com and a limit of 20 products, so the first run takes a few seconds:

title	vendor	minPrice	onSale	anyAvailable
Women's Allbirds Flip Flop - Dusty Pink	Allbirds	25	true	true
Men's Canvas Runner NZ - Deep Navy Stripes	Allbirds	100	false	true

Each product row includes minPrice, maxPrice, onSale (a compare-at price exists), anyAvailable, variantCount, tags, image URLs and timestamps.

Step 3: Products or variants?

The includeVariants switch changes the shape of the output:

Off (default): one row per product with min/max price. Best for catalog overviews.
On: one row per variant (each size/colour) with its own SKU, price, compare-at price and availability. Best for price and stock monitoring.

Raise maxProductsPerStore to export the full catalog. You can paste many stores at once, one per line.

Step 4: Build a simple price tracker

Schedule the actor daily on Apify (Schedules > Create), then compare today's dataset with yesterday's. A minimal Python version:

from apify_client import ApifyClient
import csv, datetime

client = ApifyClient("YOUR_APIFY_TOKEN")
run = client.actor("scrape-smith/shopify-products-feed").call(run_input={
    "storeUrls": ["https://www.allbirds.com", "https://www.gymshark.com"],
    "maxProductsPerStore": 1000,
    "includeVariants": True,
})
rows = list(client.dataset(run["defaultDatasetId"]).iterate_items())
fname = f"prices-{datetime.date.today()}.csv"
with open(fname, "w", newline="") as f:
    w = csv.DictWriter(f, fieldnames=sorted({k for r in rows for k in r}))
    w.writeheader(); w.writerows(rows)
print(len(rows), "rows saved to", fname)

Join two days on the variant ID and you have price drops, new products and sold-out items. Or skip code entirely: connect the dataset to Google Sheets with Apify's integration and use a formula.

What people use this for

Competitor price monitoring, especially catching sales via compare-at prices.
Dropshipping and product research across dozens of stores in a niche.
Catalog backup or migration when moving platforms.
Brand and market research: which product types and tags dominate a category.
Feeding AI agents: the actor works through Apify's MCP server, so an assistant can pull a catalog on request.

Honest limitations

The public feed shows availability (in stock / out of stock), not exact inventory counts.
Some stores hide products from the feed or disable it; you can't get around that, and you shouldn't try.
Prices are in the store's default currency and market.
Respect each store's terms of service. This tool reads a public endpoint at a gentle rate; it does not log in or bypass anything.

Cost

$0.0008 per row. The 20-product test is under two cents; 1,000 products is $0.80; a 10,000-variant monitoring run is $8. Apify platform usage is included, and you can cap spend per run.

Wrap-up

Shopify's product feed is one of the easiest structured data sources on the web, as long as something handles pagination and flattening. Try the Shopify Products Scraper with the prefilled example, then swap in the stores you actually care about.

Scrape-Smith builds small, cheap data tools on public and official APIs.