DEV Community: Ethan Teague

How to Scrape Shopify App Store Reviews (and Track Competitor Churn) in 2026

Ethan Teague — Mon, 20 Jul 2026 12:21:26 +0000

Note: this uses a tool I built, but the parsing approach works for any Shopify App Store scraping project.

If you build Shopify apps — or invest in them — the App Store is a goldmine of competitive intelligence. The trick most people miss: reviews come with the reviewing merchant's store name, country, and how long they used the app. That turns "3 new 1-star reviews" into "here are the named stores leaving my competitor this week, and why."

The good news: it's server-rendered HTML

Unlike a lot of modern sites, Shopify App Store pages render on the server. No headless browser needed.

App details live in a JSON-LD block (<script type="application/ld+json">): name, aggregateRating, category. Developer, launch date, and pricing plans are in the surrounding markup.
Reviews are chunked by a data-merchant-review marker. Each chunk has the star rating (aria-label="X out of 5 stars"), date, review text, and — three lines above the "using the app" tenure string — the store name and country.

The reviews endpoint even supports filtering:

https://apps.shopify.com/<slug>/reviews?page=1&sort_by=newest&ratings[]=1&ratings[]=2

That ratings[] filter plus sort_by=newest is the whole churn-monitoring machine: only new 1–2 star reviews, newest first.

The monitoring pattern

Scrape newest-first, stop when reviews get older than your last run's date, and you have a cheap weekly "who's unhappy with competitor X" pipeline. Pair it with star filters and you're watching exactly the merchants most likely to switch.

The no-code way

I published a Shopify App Store Scraper on Apify that returns flat JSON: one item per app, one per review, with store/country/tenure attached.

curl -X POST "https://api.apify.com/v2/acts/ethanteague~shopify-app-store-scraper/run-sync-get-dataset-items?token=YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"appUrls": ["stockie"], "maxReviewsPerApp": 200, "reviewsNewerThan": "2026-01-01", "starFilters": ["1","2"]}'

Example review item:

{
  "appSlug": "stockie",
  "store": "Woolyn",
  "country": "United States",
  "tenure": "4 days using the app",
  "stars": 4,
  "dateIso": "2026-07-15",
  "text": "Pros: The most robust PO ordering system I've found..."
}

Pricing is $2 per 1,000 reviews, $5 per 1,000 apps — schedule it weekly and you've got a standing competitive-intel feed.

What to build

Churn-watch on competitors (the reviewers are named stores → warm-lead signal)
App-idea validation: mine every complaint in a category before writing code
Portfolio monitoring: ratings and pricing across dozens of apps
A market-data feed for an AI agent via MCP

Comments open for edge cases.

How to Scrape Bluesky Data in 2026 (Posts, Profiles, and Search)

Ethan Teague — Sun, 19 Jul 2026 14:13:50 +0000

Note: this walkthrough uses a tool I built, but the API details apply to any Bluesky scraping project.

Bluesky is built on the AT Protocol, which means a lot of its data is available through a clean, documented, public API — no login required for most of it. Here's how to get posts, profiles, followers, and threads in 2026, plus the one thing that does need authentication.

What's open without a login

Bluesky's public AppView (https://public.api.bsky.app) serves these unauthenticated:

app.bsky.actor.getProfile — bio, follower/following/post counts
app.bsky.feed.getAuthorFeed — a user's posts (with reply/repost filters)
app.bsky.graph.getFollowers / getFollows — the social graph
app.bsky.feed.getPostThread — a post and its full reply tree

For example, a profile is one GET away:

curl "https://public.api.bsky.app/xrpc/app.bsky.actor.getProfile?actor=bsky.app"

The one catch: keyword search needs auth

app.bsky.feed.searchPosts returns 403 unauthenticated. Any tool claiming anonymous Bluesky keyword search is quietly routing your queries through someone else's account. The correct way is to create a free app password (Bluesky → Settings → Privacy and security → App passwords — never your main password), call com.atproto.server.createSession on your PDS to get a token, and pass it as a Bearer header. App passwords can't manage your account and are revocable, so this is safe.

Handling the edge cases

Some profiles show handle.invalid (broken handle verification) — fall back to the DID for building URLs.
Post URLs use the record key: at://did/app.bsky.feed.post/<rkey> maps to bsky.app/profile/<handle>/post/<rkey>.
Paginate with the cursor field; back off on 429s.

The no-code way

I published a Bluesky Scraper on Apify that wraps all of the above into flat JSON/CSV — profiles, feeds, followers, threads, and (with your app password) keyword search.

curl -X POST "https://api.apify.com/v2/acts/ethanteague~bluesky-scraper/run-sync-get-dataset-items?token=YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"userHandles": ["bsky.app"], "maxPostsPerUser": 100}'

It's $0.50 per 1,000 posts, and it's callable from Python/JS, on a schedule, or as an AI-agent tool via MCP.

What to build

Track a topic or brand across Bluesky
Export a follower graph for community analysis
Feed fresh social data to an AI agent
Archive threads before they're deleted

Questions or edge cases? Comments are open.

How to Scrape Airbnb Listings and Prices in 2026 (No Code Required)

Ethan Teague — Fri, 17 Jul 2026 15:21:31 +0000

Heads-up: this post references a tool I built. It's a genuinely useful walkthrough either way — the technique applies to any Airbnb scraping project.

If you've ever tried to scrape Airbnb, you already know the two walls you hit: the pages are rendered by JavaScript, and Airbnb aggressively blocks datacenter IPs. Below is the reliable way to get clean Airbnb data in 2026 — listing prices, ratings, coordinates, and discounts — without running a headless browser or babysitting proxies.

The key insight: Airbnb ships its data in the HTML

You don't need to render the page. Every Airbnb search response embeds the full result set as JSON inside a <script id="data-deferred-state-0"> tag. Parse that and you get structured data straight away — no DOM scraping, no selectors that break on the next redesign.

The path to the results is:

niobeClientData[*][1].data.presentation.staysSearch.results
  ├── searchResults[]      // ~18 listings per page
  └── paginationInfo.pageCursors[]   // all page cursors, upfront

Each listing carries a base64-encoded ID in demandStayListing.id (decode it, take the segment after the last colon, and you have the numeric listing ID for airbnb.com/rooms/<id>), a price line with discounts, avgRatingLocalized ("4.95 (123)"), and GPS coordinates.

The two gotchas

Datacenter IPs get blocked. You need residential proxies. If a response comes back without the data-deferred-state marker, you've been served a bot check — rotate to a fresh IP and retry.
~270 result cap per search. Airbnb won't paginate past ~15 pages. To cover a whole market, split into narrower searches (by price band or neighborhood) and dedupe by listing ID.

The no-code way

If you'd rather not maintain proxy pools and parsers, I published an Airbnb Scraper on Apify that does exactly the above. Paste a location or a full Airbnb search URL (every filter is honored), and get flat JSON/CSV back.

curl -X POST "https://api.apify.com/v2/acts/ethanteague~airbnb-scraper/run-sync-get-dataset-items?token=YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"locations": ["Austin, Texas"], "checkIn": "2026-08-16", "checkOut": "2026-08-21", "maxListingsPerSearch": 180}'

Example output per listing:

{
  "listingId": "1652957843916333450",
  "url": "https://www.airbnb.com/rooms/1652957843916333450",
  "name": "Stylish Pool Home 4BR Near Siesta Key",
  "rating": 5.0,
  "reviewsCount": 8,
  "badges": ["Guest favorite"],
  "priceLabel": "$3,190 for 5 nights",
  "latitude": 27.30754,
  "longitude": -82.52108
}

It's pay-per-result ($4 per 1,000 listings), residential proxies included, and callable from Python/JS/Make/Zapier or as an AI-agent tool via MCP.

What you can build with this

Nightly price tracking across a market for revenue management
Supply/rating analysis by neighborhood (coordinates make it map-ready)
Discount hunting at scale
A data feed for an AI travel agent

Happy scraping. If you hit an edge case, drop it in the comments.