DEV Community: NexGenData

Bulk Apple App Store + Google Play Review Monitoring (2026 Guide)

NexGenData — Tue, 15 Sep 2026 18:41:31 +0000

Bulk Apple App Store + Google Play Review Monitoring (2026 Guide)

Every mobile product manager eventually hits the same wall. App Store Connect shows you ratings in aggregate, maybe a scrolling feed of the latest reviews, but nothing that answers "what are users complaining about this week compared to last week" across both iOS and Android. Google Play Console is marginally better for Android-only shops. Neither tool lets you monitor competitor apps. Neither lets you export 10,000 reviews to a notebook for serious sentiment work. Neither gives you webhooks when a new 1-star hits.

The result is that indie devs and small PM teams end up manually copy-pasting reviews into a spreadsheet every Monday, or paying $199/month for AppFollow, Appfigures, or Sensor Tower — tools that give you everything except the raw, structured review stream you can actually pipe into your own analytics.

This post walks through building that stream yourself: a dual-platform scraper stack that pulls reviews from both Apple App Store and Google Play, runs sentiment classification, clusters complaints by theme, and lands you with a weekly digest you can share in Slack. The goal is not to replicate AppFollow's UI. The goal is to own the raw data so you can do things AppFollow will never support — custom sentiment models, competitor diffs, cohort analysis by app version, correlation between review spikes and App Store ranking changes.

Grounding Numbers

A few numbers to frame the problem. Apple's App Store has roughly 1.8 million active apps as of Q4 2025 (Apple's own Services report, reconciled with data.ai). Google Play has about 2.3 million. Combined, users leave approximately 65 million reviews per month across both stores, per Sensor Tower's 2025 State of Mobile. The median app gets 0.4 reviews per 1,000 downloads; top-decile apps hit 3.2 reviews per 1,000.

For a single mid-sized consumer app doing 500k MAU and 10k weekly downloads, you can expect 30-60 fresh reviews per week, with spikes of 500+ when a release goes badly or a viral moment hits. For a competitor-tracking setup covering 20 apps in a category, you are looking at 600-1,200 reviews per week to ingest and classify.

On the classification side, modern transformer-based sentiment models (DistilBERT multilingual fine-tuned on app reviews, or any of the OpenAI/Anthropic small models) hit 88-92% agreement with human labels on the standard app-review benchmark (GAR, Maalej et al., 2016). That is good enough for weekly digests, not good enough for legal or HR consequences. Keep that framing.

Why This Is Hard

Four things make bulk review monitoring harder than it looks.

Neither store offers a public reviews API. Apple's App Store Connect API covers only your own apps, and even that is rate-limited to ~50 calls per hour. Google Play Developer API has a reviews endpoint but it only returns the last 7 days and only for apps you own. Competitor tracking is officially unsupported.
Pagination is awful on both. Apple's RSS feed caps at 500 reviews per country and doesn't go further back. The iTunes Lookup API gives you ratings counts but not review text. Google Play's web interface paginates with a continuation token that changes format every few months.
Localization fragments everything. Apple reviews are partitioned by country (155 storefronts). A US-only pull misses your Japanese, German, and Brazilian review streams entirely. Google Play is language-partitioned rather than country-partitioned, which is different but equally fragmenting.
Review text is messy. Emojis, transliterations, auto-translated reviews (Google Play auto-translates into the viewer's language, which means the same review appears twice if you query in two languages), and spam from incentivized-review farms. Cleaning this before you classify is a real step.

None of these are unsolvable. They are just not a weekend project if you start from scratch.

Architecture

Here is the end-to-end pipeline:

  [app ID list]
  (iOS + Android, competitors included)
        |
        v
  +---------------------+     +----------------------+
  | apple-app-store-    |     | google-play-reviews- |
  | reviews-scraper     |     | scraper              |
  +---------------------+     +----------------------+
        |                            |
        +------------+---------------+
                     |
                     v
             [normalize schema]
             (app_id, platform, rating,
              text, lang, version, date)
                     |
                     v
         [dedupe + language filter]
                     |
                     v
         [sentiment classifier]
         (DistilBERT or LLM call)
                     |
                     v
         [theme clustering]
         (BERTopic or LLM-guided)
                     |
                     v
         [Postgres + weekly digest]

The two scrapers run in parallel. Output lands in a normalized schema that hides the per-platform differences. Sentiment and theme clustering happen in a separate step, which lets you swap classifiers without re-scraping.

Code: Pull 200 Reviews Across Both Stores

The apple-app-store-reviews-scraper and google-play-reviews-scraper actors share a compatible output schema. You can fire both from one script.

from apify_client import ApifyClient
import pandas as pd

client = ApifyClient("APIFY_TOKEN")

ios_apps = ["544007664", "310633997"]      # YouTube, WhatsApp
android_apps = ["com.google.android.youtube", "com.whatsapp"]

ios_run = client.actor("nexgendata/apple-app-store-reviews-scraper").call(run_input={
    "app_ids": ios_apps,
    "countries": ["us", "gb", "de", "jp", "br"],
    "max_reviews_per_app": 200,
    "sort": "most_recent",
})

android_run = client.actor("nexgendata/google-play-reviews-scraper").call(run_input={
    "app_ids": android_apps,
    "languages": ["en", "de", "ja", "pt"],
    "max_reviews_per_app": 200,
    "sort": "newest",
})

ios_reviews = list(client.dataset(ios_run["defaultDatasetId"]).iterate_items())
android_reviews = list(client.dataset(android_run["defaultDatasetId"]).iterate_items())

df = pd.DataFrame([
    {"platform": "ios", "app": r["app_id"], "rating": r["rating"],
     "text": r["text"], "lang": r.get("country"), "version": r.get("app_version"),
     "date": r["submitted_at"]}
    for r in ios_reviews
] + [
    {"platform": "android", "app": r["app_id"], "rating": r["rating"],
     "text": r["text"], "lang": r.get("language"), "version": r.get("app_version"),
     "date": r["submitted_at"]}
    for r in android_reviews
])

print(df.groupby(["platform", "rating"]).size().unstack(fill_value=0))

A typical output cross-tab for the two apps above:

rating       1    2    3    4     5
platform
android     43   18   31   55   353
ios         27    9   22   40   402

The rating skew is expected — reviewers self-select toward extremes. What you are interested in is the 1- and 2-star tail, because that is where actionable product complaints live.

Sentiment Classification

Once the reviews are in a dataframe, run a classifier. For a weekly digest at this volume, a small local model is cheaper than an LLM API call per review. Here is a minimal transformers pass:

from transformers import pipeline

clf = pipeline(
    "sentiment-analysis",
    model="nlptown/bert-base-multilingual-uncased-sentiment",
    device=0,  # set to -1 for CPU
)

# Truncate to 512 chars — BERT tokenizer limit
df["text_trunc"] = df["text"].fillna("").str.slice(0, 512)
df["sentiment"] = clf(df["text_trunc"].tolist(), batch_size=32)
df["sentiment_stars"] = df["sentiment"].apply(lambda x: int(x["label"][0]))
df["sentiment_score"] = df["sentiment"].apply(lambda x: x["score"])

This model outputs 1 star through 5 stars. You now have two signals per review: the user-provided rating and the model-predicted rating. When they disagree by 2+ stars it usually means sarcasm (5-star review trashing the app) or a rating-without-comment (1-star with empty text). Both are worth flagging.

For theme clustering, BERTopic works out of the box on a few thousand reviews. For smaller volumes, an LLM with a clustering prompt is faster to ship:

import openai
from collections import Counter

negative = df[df["rating"] <= 2].sample(min(150, len(df)))
prompt = "Cluster these app reviews into 5 themes. Return JSON: {theme: [review_ids]}\n\n"
prompt += "\n".join(f"{i}: {t[:200]}" for i, t in enumerate(negative["text"]))

resp = openai.chat.completions.create(
    model="gpt-4o-mini",
    messages=[{"role": "user", "content": prompt}],
    response_format={"type": "json_object"},
)
themes = resp.choices[0].message.content
print(themes)

For 150 reviews, this is a ~$0.01 call and returns something like {"crashes_on_launch": [3,7,22,41], "ads_too_aggressive": [1,5,12,18,33,...], ...}.

Worked Example: Weekly Competitor Digest

Say you are PM on a meditation app. You track three competitors plus your own app across iOS and Android. Every Monday morning you want a Slack post that lists, per app: total reviews this week, average rating, top 3 complaint themes, and any rating drop of more than 0.3 stars week-over-week.

from datetime import datetime, timedelta

# Pull last 7 days across 4 apps × 2 platforms
apps = [
    {"name": "Calm", "ios": "571800810", "android": "com.calm.android"},
    {"name": "Headspace", "ios": "493145008", "android": "com.getsomeheadspace.android"},
    {"name": "Insight Timer", "ios": "337472899", "android": "com.spotlightsix.zentimerlite2"},
    {"name": "YourApp", "ios": "...", "android": "..."},
]

cutoff = datetime.utcnow() - timedelta(days=7)

weekly = df[pd.to_datetime(df["date"]) >= cutoff]

digest = weekly.groupby(["app", "platform"]).agg(
    n=("rating", "count"),
    avg_rating=("rating", "mean"),
    neg_pct=("rating", lambda x: (x <= 2).mean()),
).round(2)

print(digest)

Feed the negative reviews for each app into the LLM theme clusterer, dump the results into a templated Slack message, and you have a digest that would cost $199/month from AppFollow. Running it as an Apify scheduled task pushes the cost under $5/month at this volume.

Over six months of running this, a PM on the team spots a pattern: whenever a release contains a meaningful UI change, 1-star reviews spike within 72 hours. They start shipping UI changes behind a feature flag and rolling out to 10% first. Review sentiment during the subsequent staged rollout becomes a release signal. That is the actual point of owning the data stream.

Gotchas

Things we have watched trip people up:

Apple's RSS cap is 500 reviews per country. You cannot page deeper. For apps with thousands of reviews, you either ingest continuously from day one or accept that historical backfill is partial.
Google Play auto-translation double-counts. If you query com.whatsapp with languages=["en", "de"], a German review may appear once in German and once translated-to-English. Dedupe by review ID, not text.
Version numbers are unreliable. Apple's app_version field reflects the version the user was on when reviewing. Google Play sometimes drops it entirely for older reviews. Do not use this for cohort analysis without spot-checking.
Spam and incentivized reviews exist. Both stores have anti-fraud teams, but clusters of 5-star reviews with near-identical phrasing show up regularly. Flagging them is a classifier problem — usually low perplexity + short length + recent account.
Localization scoring varies wildly. Japanese users rate 0.5 stars lower than US users on average for the same app; German users 0.3 lower. Cross-country average-rating comparisons are meaningless without normalization.
Rate limits. The actors handle backoff, but if you self-roll the scrapers: Apple RSS is soft-capped around 10 requests/sec per IP; Google Play's web UI starts serving CAPTCHAs around 30 requests/min per IP.
Reviews disappear. Apple removes reviews the developer successfully appeals. Google Play removes reviews that violate policy. If you are doing longitudinal analysis, snapshot reviews as you ingest; do not assume they will still be there next quarter.
Emoji-only reviews. About 2% of reviews are emoji-only. Most sentiment classifiers handle them poorly. Either route them to a separate classifier or drop them.

FAQ

Can I track my competitors legally?
App Store reviews are public data under both Apple's and Google's terms of service for normal users. Automated collection is technically against Apple's Developer Agreement, but the enforcement target has historically been scraping the storefront for listings and pricing, not reviews. Consult your own counsel for anything commercial; for internal research, the risk is low.

How often should I run the scrape?
For a weekly digest: once per week is enough. For crisis monitoring (post-release or incident): hourly for the first 48 hours after a release, then daily. The actors are designed to run idempotently — running twice in an hour will mostly return cached data.

How far back does the history go?
Apple: about 500 most recent reviews per country, no deeper. Google Play: roughly 12 months with cooperative pagination. For older reviews, you need snapshots you collected previously.

What if my app is in 40 countries?
Pull the top 10 by install volume for weekly digests. Run a monthly fuller pull across all countries for trend analysis. Running all 155 Apple storefronts weekly is overkill and expensive.

Does the sentiment classifier handle non-English reviews?
nlptown/bert-base-multilingual-uncased-sentiment handles 6 languages natively (English, Dutch, German, French, Spanish, Italian). For Japanese, Chinese, Korean, Portuguese, Arabic, you want a different model per language, or just use an LLM.

Can I correlate reviews with my install numbers?
Yes, if you pull installs from App Store Connect and Play Console. The Apify scrapers do not pull install data (it is private to the developer), but you can join on date and app_version after the fact. The signal-to-noise is noisy at weekly granularity, good at monthly.

What about in-app review prompts skewing the data?
The SKStoreReviewController and Google's in-app review API skew ratings positive because they are shown after success events. If you run these, expect your ratings to trend 0.5-1.0 stars higher than organic, which makes competitor comparison harder. Note it in your dashboards.

How do I handle developer responses?
Both scrapers return the developer reply text if present. Treat replies as a separate signal — response rate, median response time, and reply length all correlate with rating trends. Teams that reply within 24 hours see measurably lower churn on 2-3 star reviews.

Conclusion

Mobile review monitoring does not require a $199/month subscription or a dedicated analyst. Two Apify actors, a normalized schema, a small sentiment classifier, and a weekly cron give you most of what AppFollow and Appfigures sell, plus the freedom to do custom analysis neither tool supports.

The bigger win is not cost savings — it is that owning the raw review stream lets you correlate reviews with things AppFollow does not know about: your release cadence, your feature flags, your marketing spikes, your outages. That correlation is where product insight actually lives.

Start with the apple-app-store-reviews-scraper and google-play-reviews-scraper on Apify. Both run pay-per-use, handle pagination and localization, and return a consistent schema you can feed into whatever analytics stack you already own.

Crunchbase Killed Its Free API. Here's How to Rebuild It (2026)

NexGenData — Tue, 08 Sep 2026 15:08:46 +0000

Crunchbase Killed Its Free API. Here's How to Rebuild It (2026)

In 2023, Crunchbase quietly deprecated its free Basic API. A generation of indie hackers, academic researchers, and early-stage founders piping structured company data into dashboards and Jupyter notebooks suddenly had nowhere to go. The paid replacement starts at roughly $500/month — a rounding error for a BD team, a non-starter for a solo developer or a graduate student building a thesis dataset.

The good news: Crunchbase is mostly aggregating public information anyway. Company domains, tech stacks, team-size signals, infrastructure footprints — it's all sitting in public DNS, GitHub, certificate transparency logs, and registrar records. Nobody compiled it into one neat JSON endpoint for free, but the raw material is.

Stitch together the right eight sources and you can reconstruct roughly 60–70% of what Crunchbase Basic used to offer. You won't get funding rounds or cap tables — those still require paid data. You will get domain, founding-era proxy, tech stack, CDN, hosting provider, subdomain footprint, GitHub activity, email infrastructure, brand assets, and a reasonable team-size estimate. This post walks through each of the eight sources, the parallel-fanout architecture, a worked VC screening example, and the gotchas that will bite you at scale.

The Data Gap: What Crunchbase Basic Used to Give You

Before the shutdown, a free Crunchbase Basic call returned: company name, domain, logo URL, description, employee_count_range, funding_rounds with amounts and investors, industry tags, location, and founded_year.

Roughly half of that is derivable from public sources. The other half — funding data, precise employee counts, curated industry taxonomy — requires Crunchbase, PitchBook, or a human research team.

What you can reconstruct for free:

Domain, logo, tagline — company site, favicon, Open Graph tags
Founding-era proxy — WHOIS domain creation date (imperfect; domains are sometimes registered years before or after incorporation)
Tech stack — HTTP headers, robots.txt hints, npm org scopes
Hosting + CDN — DNS A records and HTTP headers (CF-Ray, X-Amz-Cf-Id, Via)
Team-size proxy — GitHub org member + repo count, plus subdomain footprint
Security posture — SSL cert issuer (Let's Encrypt vs. DigiCert vs. internal CA)
SaaS stack — DNS TXT records (SPF/DMARC verification tokens leak vendor relationships)

What you basically cannot derive: exact employee count, funding rounds, investors, cap table, ARR/MRR, board composition. For those, pay Crunchbase/PitchBook or do journalism.

The Eight Free Data Sources

Let's go through each source in order, including what you get, what it costs (in API calls and ethical risk), and where it breaks.

1. WHOIS — The Domain Registry Record

WHOIS is the oldest piece of internet infrastructure still relevant for company-data work. Every one of the ~362M registered domains (Verisign DNIB 2024) has a WHOIS record. Query whois stripe.com and you get registrar (MarkMonitor), creation date (2009-09-10), expiration, and nameservers.

What's useful:

Creation date as a founding-era proxy. Stripe's domain was registered in 2009; the company was founded in 2010. Close enough for most analysis.
Registrar choice. MarkMonitor, CSC, or Com Laude skew enterprise with IP-protection budgets. GoDaddy, Namecheap, Porkbun skew indie.
Registrant organization — when it isn't redacted. Post-GDPR, most registrars hide this behind "Whois Privacy," but pre-2018 .com registrations still expose org names surprisingly often.
Nameservers — ns-cloud-a1.googledomains.com means GCP DNS. pdns*.ultradns.net means enterprise DNS. ns1.cloudflare.com means Cloudflare.

Gotcha: WHOIS rate limits are aggressive and per-TLD. ccTLDs (.io, .ai, .de) have their own registry endpoints with tighter limits. Use whoisxmlapi.com free tier (500/month) if you need volume.

2. DNS — The Email and Infrastructure Fingerprint

DNS is where a huge amount of company intelligence lives, and it's all free to query. For any domain, pull:

A records — where the apex points. Is it a Cloudflare IP (104.21.x.x, 172.67.x.x)? A direct EC2 IP? A Fastly anycast range? This tells you about scale and sophistication.
MX records — the email provider. aspmx.l.google.com means Google Workspace (usually 1–500 employees, startup-heavy). *.mail.protection.outlook.com means Microsoft 365 (enterprise-skewed). Self-hosted MX on the company's own mail server means either a security company or a 90s holdout.
NS records — authoritative nameservers. Paired with A records, confirms the DNS provider.
TXT records — the real gold mine. SPF records (v=spf1 include:_spf.google.com include:mailgun.org ~all) reveal every email-sending service they use: Mailgun, SendGrid, Postmark, Mailchimp, Customer.io. DMARC records signal security maturity. And the weird ones — google-site-verification=..., atlassian-domain-verification=..., stripe-verification=..., intercom-site-verification=..., facebook-domain-verification=... — each one is a confirmed SaaS relationship.

A single dig TXT stripe.com can tell you Stripe uses Google for email, has strict DMARC, and verifies with Atlassian, Segment, and a dozen other vendors. That's a SaaS-stack leak that would cost $300/month from BuiltWith's paid tier.

3. SSL Certificates via crt.sh — The Subdomain Leak

Certificate Transparency logs are a post-2013 mandate requiring every publicly-trusted TLS cert to be logged. Combined CT logs indexed by crt.sh contain roughly 10B certificates as of 2026.

For any domain, crt.sh?q=%25.stripe.com&output=json returns every cert ever issued for a subdomain. This leaks:

Subdomain list — api.stripe.com, dashboard.stripe.com, files.stripe.com, checkout.stripe.com, and dozens more. Internal and staging hosts sometimes leak when someone accidentally requests a public cert for them.
Cert issuer — Let's Encrypt signals startup/small team (free, automated). DigiCert, Sectigo, GlobalSign signal mid-to-large with procurement budgets. Internal CAs signal enterprise with a dedicated PKI team.
Cert cadence — aggressive 90-day rotation signals modern DevOps; 2-year certs signal legacy ops.

Subdomain count alone is a useful size proxy. A 5-person startup has 3–5 subdomains. A Series B SaaS has 20–40. A Stripe-scale company has hundreds.

4. GitHub Org API — The Engineering-Team Signal

GitHub's REST API is free and generous: 5,000 requests/hour authenticated. GET /orgs/{org} returns public member count, repo count, creation date, description, and location. GET /orgs/{org}/repos paginates through public repos.

What you learn:

Public repo count — 5 repos means a small shop. 100+ repos means a real engineering org. Many companies put serious work in private repos, so this is a floor.
Primary languages — top 3 being Python/Go/TypeScript signals a modern stack. Java/Ruby at the top hints legacy.
Member count — public org members. Most companies hide members by default, so this drastically undercounts. Stripe has probably 300+ engineers on GitHub but shows maybe 40 public members.
Activity level — commit frequency, open-issue count, star count on flagship projects. A dead org with 500 repos and no recent commits is very different from a live one.

Note: the company-data-aggregator actor currently returns first-page repos only; a full-org walk with pagination is on the roadmap. For ~95% of companies, first page (30 repos) is enough signal.

5. Tech Headers — The Lightweight BuiltWith

Make a single HEAD or GET request to the company's homepage and inspect the response headers. You'll see:

Server — Apache, nginx, Caddy, IIS, LiteSpeed, or a proxy like Cloudflare/CloudFront that masks the origin.
X-Powered-By — PHP, Express, ASP.NET, Next.js. Modern shops often strip this for security; its absence is itself a signal.
Via — usually 1.1 varnish, 1.1 google, or similar; reveals intermediate proxies.
CF-Ray — Cloudflare presence and data center (e.g., 8a3f2c... -SJC = San Jose POP).
X-Amz-Cf-Id — CloudFront.
X-Served-By — Fastly.
X-Akamai-* — Akamai.
X-Vercel-* — Vercel.
X-Github-Request-Id — GitHub Pages.

From five headers you can usually identify the CDN, the front-end host, and sometimes the framework (Next.js, Remix, SvelteKit, Nuxt all leak through X-Nextjs-Prerender, X-Nuxt-Renderer, or distinctive Link prefetch patterns).

6. robots.txt + sitemap.xml — The Internal-Map Leak

Every well-behaved website publishes /robots.txt and usually /sitemap.xml. These are free to fetch and contain remarkable amounts of internal structure.

robots.txt tells you what the company doesn't want indexed, which is often what's interesting:

User-agent: *
Disallow: /admin/
Disallow: /api/
Disallow: /dashboard/
Disallow: /internal/
Disallow: /beta/
Disallow: /_next/
Disallow: /staging/
Sitemap: https://example.com/sitemap.xml

Just from that, you know they have an admin panel, a dashboard, a beta product, internal tooling paths, a Next.js frontend, and a staging environment. A lot of architectural intel for one HTTP GET.

sitemap.xml gives you size:

50-URL sitemap: tiny marketing site
500-URL sitemap: real product with docs and blog
5,000-URL sitemap: content-heavy SaaS with docs + changelogs + customer stories
50,000+ URLs split across sitemap index files: scale (think Notion, Webflow, marketplace-style)

Sitemap <lastmod> dates also hint at content velocity — is this a live, actively-maintained site, or abandoned?

7. npm Organization Scope — The Internal-Library Fingerprint

npm has 3M+ public packages and a scoping mechanism (@stripe/, @vercel/, @shopify/) that lets companies publish under an org namespace. Most serious JavaScript-shipping companies claim their scope.

Querying registry.npmjs.org/-/v1/search?text=scope:stripe gives you every package published under the scope. What this reveals:

Which products they ship. @stripe/stripe-js is the browser SDK. @stripe/react-stripe-js is the React wrapper. Each package is a confirmed product surface area.
Which internal libraries they've extracted. When a company open-sources @acme/ui-primitives, they're signaling internal practices and often recruiting (the README usually links to careers).
Download volume as a crude user-base proxy. @stripe/stripe-js gets 8M+ weekly downloads — a real distribution footprint.
Publication frequency. Monthly releases across many packages = active engineering. Last publish 2 years ago = maintenance mode.

Gotcha: some companies publish under personal names rather than org scopes (early-stage founders who never migrated). You'll miss those unless you also search packages where repository.url contains the company's GitHub org.

8. Favicon + Open Graph Images — The Brand Fingerprint

Finally, the branding layer. Fetch the homepage HTML and parse:

<link rel="icon" href="..."> — the favicon. High-res with multiple sizes = company cares about brand. A 16x16 Rails/Next placeholder = they don't.
<meta property="og:image"> — the social-share card, usually containing logo + tagline. Perfect for a visual directory.
<meta property="og:title"> and <meta property="og:description"> — official tagline and description, usually higher-quality than a scraped H1.

Fallback: Google's favicon service at https://www.google.com/s2/favicons?domain=stripe.com&sz=128 returns an icon for almost any indexed domain.

OG images are gold for company directories — pre-designed 1200x630, include logo and tagline, explicitly intended for third-party consumption. No ethical concerns.

Grounding Numbers

To put this in perspective: Crunchbase has ~11M companies (most with shallow data). GitHub has ~60M public repos across ~100M users (2024 Octoverse). WHOIS covers ~362M registered domains (Verisign DNIB 2024) — essentially every company with a web presence. npm has 3M+ public packages. CT logs indexed by crt.sh hold ~10B certificates.

You are not dealing with scarcity — you're dealing with an aggregation-and-normalization problem. Every relevant company is in these sources somewhere; the hard part is hitting them in parallel, merging results, and surviving rate limits.

Architecture: Parallel Fanout

Here's the architecture for pulling all eight sources per domain without taking forever:

[Domain list]
     |
     v
[company-data-aggregator]
     |
     +-> WHOIS        -> registrar, age, registrant org
     +-> DNS          -> MX stack, verification tokens
     +-> SSL/crt.sh   -> subdomains, cert issuer
     +-> GitHub org   -> repos, languages, size proxy
     +-> tech headers -> CDN, hosting, server
     +-> robots/sitemap -> scale + internal paths
     +-> npm          -> internal packages
     +-> favicon/og   -> brand assets
     |
     v
[merged JSON profile per domain]

The company-data-aggregator actor on Apify hits all eight in parallel per domain via asyncio.gather, with per-source error isolation (if WHOIS rate-limits you, DNS and GitHub still succeed), a configurable per-source timeout (default 10s), and a unified merged JSON output. For a list of 100 domains, the whole job finishes in under 2 minutes on Apify's standard compute.

Code Example

from apify_client import ApifyClient

client = ApifyClient("APIFY_TOKEN")

targets = ["stripe.com", "vercel.com", "fly.io", "railway.app", "render.com"]
run = client.actor("nexgendata/company-data-aggregator").call(run_input={
    "domains": targets,
    "sources": ["whois", "dns", "ssl", "github", "tech_headers", "robots", "npm", "favicon"],
    "timeout_per_source_s": 10,
})

for d in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(f"\n{d['domain']}: GH repos={d.get('github', {}).get('repo_count')}, "
          f"MX={d.get('dns', {}).get('mx_provider')}, "
          f"CDN={d.get('tech_headers', {}).get('cdn')}")

Expected output for the five targets above:

stripe.com: GH repos=250, MX=Google Workspace, CDN=Cloudflare
vercel.com: GH repos=180, MX=Google Workspace, CDN=Vercel
fly.io: GH repos=120, MX=Google Workspace, CDN=Fastly
railway.app: GH repos=45, MX=Google Workspace, CDN=Cloudflare
render.com: GH repos=60, MX=Google Workspace, CDN=Cloudflare

At a glance: all five are modern infrastructure companies on Google Workspace, three on Cloudflare, one on Vercel's own CDN, one on Fastly. Stripe has 5x the public GitHub presence of Railway — consistent with their respective company sizes.

Worked Example: VC Associate Screens 100 AI-Infra Startups

A VC associate at a seed-stage fund is screening 100 "AI infra" startups pulled from a Twitter thread, a Substack, and a conference attendee list. She needs to narrow to ~10 worth a deeper call. Manually visiting 100 sites is a full day.

Instead, she feeds the list to the aggregator and gets back, per company:

Tech stack sophistication: AWS vs GCP vs self-hosted, Kubernetes usage (via api.k8s.* patterns), Vercel (prototype-y) vs bare EC2 (more production-grade for infra plays).
Team-size proxy: GitHub member count + subdomain count. 5 subdomains and 3 GitHub members = probably 1–3 people. 30+ subdomains and 20+ members = 15+ people.
Age: WHOIS creation dates. Founded 2024 is a different conversation than founded 2019 with no traction.
Subdomain footprint: a production-maturity proxy. An "enterprise-ready" AI-infra startup with just www. and app. is pre-product. One with api., docs., status., console., plus regional endpoints, has real infrastructure.
SaaS stack: TXT records reveal Segment, Intercom, Linear, Notion usage — commercial intent vs. pure research.

From 100 companies she narrows to 12 based on: founded 2022–2024, at least 10 GitHub public members OR 15+ subdomains, hosted on real cloud infra (not Wix/Webflow), and not already funded past seed (checked against Crunchbase Pro for the 12 finalists only — paying for 12 lookups is fine, 100 wasn't).

Total cost: ~$5 of Apify credits + 12 Crunchbase Pro lookups (~$40). Total time: 45 minutes instead of a day.

Augmenting With Paid Sources Where Free Isn't Enough

Free sources cover 60–70%. The rest requires paying. The honest hierarchy:

OpenCorporates (free tier, ~500 calls/month): incorporation records, legal entity names, officer lists. Genuinely useful for due diligence.
SEC EDGAR (fully free): 10-K, 10-Q, 8-K, S-1 filings for US public companies. Unbeatable for public-company financials.
Clearbit (RIP): acquired by HubSpot in 2023, rebranded as HubSpot Insights, free tier deprecated. Not a replacement anymore.
PeopleDataLabs / Proxycurl: LinkedIn-adjacent. Contact info, titles, seniority. Paid, cheaper per-record than Apollo.
Crunchbase Pro / PitchBook: for funding data you cannot derive elsewhere. Use surgically on your shortlist.
BuiltWith Pro: tech stack at scale without maintaining your own header parsing.

The play: free aggregation screens the top of the funnel, paid data enriches the shortlist. That's how you replicate Crunchbase Basic workflows without $500/month.

OSINT Ethics and Legal Limitations

All eight sources above are public data. That doesn't make bulk aggregation unconditionally fine.

GDPR Article 14 covers "information from sources other than the data subject." Aggregating EU-company data with personal info about employees (e.g., GitHub names) technically requires notice. Enforcement tends to target bulk sellers, not internal tools.
CCPA has similar provisions, plus specific rules around data brokers.
State-level data-broker laws (Vermont, Oregon, Texas, Delaware as of 2025) require registration if you re-sell aggregated personal data.
Crawl politeness: honor robots.txt, respect Crawl-Delay, use a User-Agent that identifies your bot with a contact email.

Good practice: use aggregated data for internal research, don't re-sell it as a "contact list," delete on request, and don't merge it with contact-level data (emails, phone numbers) without explicit consent pathways.

Gotchas

Things that will bite you at scale:

CDN-masked origins. Behind Cloudflare, you see Cloudflare, not the real origin. Workarounds: historical DNS records (securitytrails, viewdns.info), or MX records which usually sit on the real infra.
WHOIS privacy. GoDaddy redacts by default on all new .com registrations post-2018. You'll see "Redacted for Privacy" a lot.
GitHub orgs that hide members. Most enterprises have private membership. Member count will be null or drastically underreported. Repo count is more reliable.
npm under personal names. Founders publish early libraries under personal handles and never migrate. Search by repository.url pointing at the company's GitHub org to catch these.
DNS resolver leakage. Querying 8.8.8.8 for 10,000 targets tells Google exactly who you're researching. For sensitive OSINT, use DoH/DoT through a resolver you control.
Rate limits. crt.sh 429s around 10 req/sec. GitHub: 5,000/hour authenticated. WHOIS varies wildly per TLD. The aggregator handles backoff per-source; roll-your-own needs to plan for it.
CT noise. crt.sh returns every cert including expired, pre-cert, and duplicates. Dedupe by SAN, not cert serial.

FAQ

Is this legal?
Aggregating public WHOIS, DNS, CT logs, GitHub, npm, and HTTP headers is legal everywhere we're aware of. Re-selling aggregated data combined with personal info crosses into data-broker regulation. Stay on the internal-research side and you're fine.

How often does the data update?
DNS propagates in minutes. WHOIS updates weekly at registrars. CT logs are near-real-time. GitHub and npm are real-time. Re-run the aggregator weekly for fresh data.

Can I use this for lead generation?
For targeting and qualification: yes (this is what BuiltWith/HG Insights are for). For cold-email lists with personal info: no, not without a separate consent-compliant enrichment step.

How does accuracy compare to Crunchbase?
Domain, tech stack, hosting, CDN, subdomain count, GitHub metrics: more accurate (Crunchbase doesn't even track most of this). Funding, employee counts, investors: drastically less accurate — Crunchbase wins and it's not close.

What if the company uses Cloudflare — do I get real IPs?
No. Cloudflare hides the origin. Infer it from historical DNS (viewdns.info, securitytrails free tier) or non-proxied subdomains like mail and staging.

How do I combine this with Apollo/Hunter for contact enrichment?
Run the aggregator first for company-level profiles. Pass qualified domains to Apollo/Hunter for person-level enrichment. Much cheaper than running Apollo on your full list.

Can I run it on 10,000 domains at once?
Yes. At 10 concurrent with 10s per-source timeouts, 10,000 domains take ~3 hours and about $15–25 in Apify credits.

Where's the rate-limit bottleneck?
Usually crt.sh and WHOIS. GitHub is generous if authenticated. DNS has no practical limit with a good resolver. The aggregator backs off per-source independently.

Conclusion

Crunchbase killing its free Basic API hurt, but it wasn't fatal. Roughly two-thirds of what it provided is reconstructable from public WHOIS, DNS, CT logs, GitHub, npm, HTTP headers, robots/sitemap, and favicon/OG metadata. The rest — funding rounds, exact headcount, investor lists — still requires paid data, but you can now pay for it surgically on a pre-qualified shortlist instead of blanket-licensing it for your entire pipeline.

If you want the ready-made version rather than wiring up eight scrapers yourself, try the company-data-aggregator on Apify — it handles parallel fanout, per-source error isolation, rate-limit backoff, and unified JSON output across all eight sources. Pay per run, not per month.

Alexa Rank Is Dead. Here's What Replaces It in 2026 (Free API)

NexGenData — Thu, 03 Sep 2026 15:03:03 +0000

Alexa Rank Is Dead. Here's What Replaces It in 2026 (Free API)

On May 1, 2022, Amazon shut down Alexa.com. For roughly 25 years it had been the default shorthand for "how big is this site" — a single integer between 1 and ~30 million that every SEO deck and media kit cited as if it were ground truth. Then it was gone. Nearly four years later, in 2026, we still don't have a direct replacement.

Instead, the space fragmented. If you want to know how popular a domain is today, you pick from five or six datasets with different methodologies, coverage, cadences, and prices. Some are free but cover only the top 10,000. Some are paywalled at $1,000+ per month. Some aren't really "rank" at all — they're Core Web Vitals dressed up in a ranking shirt.

This post is a practical guide for people who still need a number to plug into a spreadsheet: SEO analysts, competitive intelligence teams, affiliate marketers, VCs doing diligence, and compliance teams deduping blocklists. It explains what replaced Alexa Rank, why Tranco is the closest true successor for most free-tier workflows, and how to pull Tranco ranks via an Apify actor we maintain.

Why This Matters (The Gap Alexa Left Behind)

The absence of a free, open, broadly-covered ranking API creates real friction in workflows that used to run on autopilot:

Affiliate marketers negotiating ad rates need a third-party popularity signal that publishers can't self-report. "We're a top 50,000 site" used to be an objective claim.
VCs and M&A analysts sizing up a target want a sanity check on claimed traffic numbers. A founder can hand you a Google Analytics screenshot; a Tranco rank is harder to forge.
SEO agencies doing competitive audits need to rank a client against peers. Without a common yardstick, every comp analysis turns into a subjective argument about which paid tool to trust.
Journalists fact-checking "the #X site on the internet" claims need a citable source. Alexa was a Wikipedia-grade reference; its absence has degraded the quality of such citations.
Compliance teams deduping domain blocklists by popularity want to know whether example.cn is a global top-10k domain before they sinkhole it.
Security researchers studying phishing rely on "popularity of the impersonated domain" as a feature in detection models. Alexa was the default corpus in hundreds of papers until 2019.

None of these workflows died when Alexa died. They just got worse, hackier, and more expensive.

The Grounding Numbers

A quick reality check on scale:

Alexa Rank at peak tracked ~30 million domains, daily updates from toolbar telemetry and crawl data.
Tranco covers a top 1 million list, updated daily, aggregated from four independent sources.
Cloudflare Radar exposes a ranked API but caps at the top 10,000 globally.
Similarweb tracks 100 million+ domains with engagement metrics, API starts at ~$1,000/month.
CrUX covers millions of origins but surfaces Core Web Vitals (LCP, CLS, etc.) rather than rank.
OpenPageRank is still online but deprecated; scores haven't refreshed in years.

The closest direct substitute for Alexa — daily-updated, programmatically accessible, free, with hundreds of thousands of domains — is Tranco. Everything else covers a narrower slice, charges a lot more, or measures something different.

Comparison: The Five Alexa Alternatives That Actually Matter

Source	Price	Coverage	Update Cadence	Primary Signal	Commercial Use?
Tranco	Free	Top 1M	Daily (30-day averaged)	DNS queries + browser telemetry + web graph	Yes, with attribution
Similarweb API	$1,000+/mo	100M+	Monthly aggregates	Panel + ISP + scraping	Yes (commercial license)
Cloudflare Radar	Free API	Top 10k	Daily	1.1.1.1 DNS resolver traffic	Yes (fair use)
CrUX BigQuery	Free (GCP egress)	Millions of origins	Monthly	Real-user Chrome telemetry	Yes
SEMrush / Ahrefs Rank	$100-$500+/mo	Tens of millions	Weekly-ish	Organic traffic estimates	Yes (commercial tier)

A few calls on when each makes sense:

Pure "how big is this site" for free? Tranco.
Need engagement metrics (visits, bounce rate, avg session duration)? Similarweb. There is no free equivalent that is remotely close.
Only care about the top few thousand globally famous domains? Cloudflare Radar is fine and often fresher than Tranco.
Optimizing Core Web Vitals and want a popularity weight so you don't waste budget on obscure origins? CrUX.
Doing keyword-level SEO competitive work? Ahrefs/SEMrush — but note their "rank" is traffic-weighted, not popularity-weighted, and will disagree with Tranco on many domains.

Tranco in Depth

Tranco deserves its own section because it's the one most people should start with, and because its methodology is unusually well-documented.

Tranco was introduced in a 2019 NDSS paper by Le Pochat et al.: "Tranco: A Research-Oriented Top Sites Ranking Hardened Against Manipulation." The motivating observation was that every public domain ranking at the time — Alexa, Cisco Umbrella, Majestic Million, Quantcast — was manipulable, volatile, or both. Security researchers were publishing results on "the Alexa top 1M" that didn't reproduce a week later because the underlying list had churned by 30%.

Tranco fixes this by doing two things:

Aggregating four independent sources using Dowdall's rule, a positional voting method that weights top positions more heavily than tail positions. The four sources are:
- Cisco Umbrella 1M — ranks based on DNS queries to Cisco's OpenDNS resolvers (billions/day).
- Chrome User Experience Report — real-user browser telemetry from opted-in Chrome users.
- Majestic Million — based on the number of unique referring subnets linking to a domain.
- Farsight DNSDB — passive DNS observations across a large sensor network.
Applying a 30-day rolling average so a domain's rank reflects sustained popularity rather than a single-day spike. This makes Tranco very hard to "game" by flooding DNS queries for a day — the primary failure mode of Cisco Umbrella and Alexa.

Tranco gives you an integer between 1 and 1,000,000 for the vast majority of domains you'd care about, updated every 24 hours, via a stable list ID you can pin for reproducibility.

Main caveats: global aggregate only (no country sub-rankings), long-tail cutoff at 1M (anything below doesn't appear), and a ~24-hour publishing lag.

The tranco-rank-lookup Actor

Tranco itself publishes CSV files and a simple HTTP endpoint, but the ergonomics for day-to-day work are rough: you have to download the daily list, parse 1M rows, do your own historical diffing, and write your own similar-domain logic if you want peer suggestions.

We built tranco-rank-lookup on Apify to wrap all of that into a single actor call. It does three things on top of the raw Tranco data:

Point-in-time and historical ranks. Pass a list of domains, get back current rank plus a 30-day history series per domain. Useful for MoM deltas without managing your own storage.
Similar-domain suggestions. For each input domain, surface a curated list of peer domains in the same category band. (Honest caveat: the similarity path uses a curated category mapping today rather than parsing the full 1M CSV for nearest-rank neighbors — see limitations below.)
Bulk lookup with caching. The actor caches daily snapshots so repeat calls in the same day don't re-download the full Tranco list. For a 1,000-domain comp set this takes a run from minutes to seconds.

Architecture

┌───────────────┐    ┌───────────────────────┐    ┌─────────────────────┐    ┌────────────────────┐
│  Domain list  │───▶│  tranco-rank-lookup   │───▶│  ranks + history +  │───▶│  BigQuery / Sheets │
│ (CSV / array) │    │   (Apify actor)       │    │  similar domains    │    │  / Airtable sink   │
└───────────────┘    └───────────────────────┘    └─────────────────────┘    └────────────────────┘
                              │
                              ▼
                     ┌──────────────────┐
                     │  Tranco CSV +    │
                     │  30-day cache    │
                     └──────────────────┘

The sink layer is optional — you can read from the dataset directly via the Apify client — but most teams push to a warehouse or a Google Sheet for dashboarding.

Code Example

from apify_client import ApifyClient
client = ApifyClient("APIFY_TOKEN")

competitors = ["competitor-a.com", "competitor-b.com", "competitor-c.com"]
run = client.actor("nexgendata/tranco-rank-lookup").call(run_input={
    "domains": competitors,
    "include_history": True,
    "history_days": 30,
    "include_similar": True,
})
for d in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(f"{d['domain']}: rank {d['current_rank']:,} (Δ30d: {d['rank_delta_30d']:+,})")

Output for a typical comp run looks like this:

competitor-a.com: rank 47,812 (Δ30d: -3,104)
competitor-b.com: rank 112,490 (Δ30d: +18,772)
competitor-c.com: rank 9,988 (Δ30d: -214)

Negative deltas mean the rank number got smaller — i.e., the domain moved up. This convention matches Alexa's original interpretation and is what most analysts expect.

Worked Example: SEO Agency Quarterly Competitive Brief

A realistic use case: an SEO agency is preparing a quarterly brief for a direct-to-consumer skincare client with ~50 meaningful competitors. They need to identify who's gaining ground, who's losing it, and where to focus next quarter's content investment.

Pre-2022 this would have been a 20-minute job in Alexa. Post-2022 it's been a recurring headache — the agency was paying for Similarweb just to run this one report.

With Tranco via the actor, the flow is:

Define the comp set. A Google Sheet with ~50 domains tagged by category (direct / aspirational / marketplace).
Run the actor weekly. A scheduled Apify run pulls current rank plus 30-day history; results land in BigQuery via a small ETL step.
Compute deltas in SQL. MoM rank delta, QoQ delta, and a volatility score (stdev of daily rank).
Surface the outliers. In the most recent quarter the agency identifies:
- 3 fast risers whose rank improved by >500,000 positions MoM. All three launched sustained influencer campaigns in Q1 — confirmed by cross-referencing Facebook Ad Library.
- 2 decliners whose rank dropped by >1,000,000 positions. One was acquired and had traffic redirected to the parent brand; the other had a technical SEO regression after a site migration.
- 5 steady-state leaders whose rank fluctuated by <5,000 positions — durable category leaders worth benchmarking against.
Build the brief. The deck leads with a competitive landscape chart (rank over time for the top 10 comps), drills into each fast riser's tactics, and recommends content and paid investments.

The whole pipeline costs ~$3-5/month in Apify compute plus a trivial BigQuery bill, versus $1,000+/month for Similarweb. Operational note: the 30-day trend is more useful than any single-day rank. Single-day Tranco rank is noisy for mid-tail domains (200k-800k). Show trend lines, not point estimates.

What Alexa Rank Measured vs What Tranco Measures

A common trap when migrating from Alexa to Tranco is assuming the two numbers are interchangeable. They are not, and the methodological differences explain most of the disagreements you'll see when you look up a domain in both services (via Alexa's historical archives).

Alexa's signal was primarily the Alexa Toolbar, which created two well-known biases:

English-speaking, technical power users. The toolbar was disproportionately installed by SEO professionals and webmasters. Sites popular with those audiences ranked higher than real global traffic justified; sites popular with non-English audiences ranked lower.
Manipulation via toolbar botnets. Each install contributed a meaningful fraction of observed traffic for long-tail domains. A coordinated install campaign could push an Alexa rank up by hundreds of thousands of positions — a visible cottage industry from 2010-2016.

Tranco's signal is fundamentally different:

Authoritative DNS query volume (Cisco Umbrella, Farsight) measures how often anyone resolves a domain, regardless of browser or language.
Chrome User Experience Report adds real-user browser telemetry at massive scale — Chrome has ~65% global browser share.
Web graph data (Majestic) captures importance via unique referring subnets, which is slower-moving and harder to spoof.
30-day averaging defeats single-day manipulation. Sustained attack over a month stops being manipulation and starts being actual traffic.

Practically: domains inflated on Alexa because they catered to Western webmasters have more conservative Tranco ranks. Domains serving non-English audiences tend to rank higher on Tranco than on Alexa. Migrating a historical report from Alexa to Tranco numbers? Expect some of the "rankings changed" to be "measurement got better," not real movement.

Other Sources Worth Combining with Tranco

Tranco is the spine, but for a complete competitive-intelligence stack you'll want to pair it with one or two of these:

Cloudflare Radar API

Free, rate-limited. Covers the top 10,000 domains based on 1.1.1.1 DNS resolver traffic. Value-add over Tranco is traffic category data — share-of-traffic breakdowns by category over time, useful for "is this category growing?" questions. Useless below top 10k.

CrUX (via BigQuery or the API)

Free with GCP egress costs. Not a rank source, but the chrome-ux-report.all.origin table tells you which domains are in the dataset — a coarse popularity threshold. Also gives Core Web Vitals per domain for technical-SEO benchmarking.

OpenPageRank

Historical archive only. Useful for pre-2022 longitudinal studies. Don't use it for anything current.

DataForSEO

Paid but pennies per lookup instead of thousands per month. Reasonable middle ground between free Tranco and enterprise Similarweb if you need commercial-grade data at scale without engagement metrics.

Ahrefs Rank

Traffic-weighted organic search rank. Conceptually different from Tranco — a domain can rank well on Tranco (direct/DNS traffic) while ranking poorly on Ahrefs (little organic search), and vice versa. Use Ahrefs when the question is specifically about SEO performance.

Limitations of Any Ranking Approach

Whichever source you pick, keep these caveats in mind:

Regional bias. Tranco is global-aggregate and under-weights country-concentrated traffic. A top-100 Indonesian site may rank in the hundreds of thousands on Tranco. Cloudflare Radar and CrUX publish country breakdowns; Tranco doesn't.
Freshness lag. Tranco publishes today's list tomorrow — 24-48 hour blind spot for viral spikes.
Long-tail dropout. Below rank 1M, no data. Hard cutoff. Niche players get partial coverage.
Bot traffic contamination. DNS methods include bot queries. Multi-source aggregation mitigates this but doesn't eliminate it.
Ecosystem gaps. App-first sites (TikTok, Instagram) and aggressive Cloudflare-proxied domains are under-represented in web-graph signals.

Report ranks as ranges or trends, not GPS coordinates.

FAQ

Is Tranco legally usable for commercial purposes?

Yes, with attribution. The Tranco list is distributed under a permissive academic license that explicitly allows commercial use. The standard practice is to cite the paper (Le Pochat et al., NDSS 2019) and include a link to the list URL used, along with the list ID for reproducibility. Talk to your counsel if you're building a paid product on top of it, but the license does not prohibit that.

How accurate is the 30-day history?

The history reflects Tranco's own 30-day rolling aggregation, so each day's data point is already a smoothed value. For domains in the top 100k, day-over-day volatility is typically under 5%. For mid-tail domains (rank 200k-800k), daily noise is much larger — plus or minus 20-30k positions is normal. Trust the trend, not the point.

Does the actor cache results?

Yes. The actor pulls the latest Tranco list once per day and caches it internally. Subsequent lookups in the same 24-hour window use the cached snapshot. This keeps runs fast and minimizes load on Tranco's infrastructure. If you need a specific historical list ID (for reproducible research), you can pass it via input.

Can I pull the full 1M list?

Not through the current actor input — it's designed for targeted lookups of a user-supplied domain set. If you need the full list, pull it directly from the Tranco website. We've considered adding a bulk-export mode; if you have a use case, open an issue on the actor page.

How do I convert Tranco rank to estimated traffic?

You can't, cleanly. Rank and traffic are not linearly related — the top 10 sites have orders of magnitude more traffic than rank 100, which has orders of magnitude more than rank 10,000. Any published conversion heuristic (Zipf-based or otherwise) has huge error bars. If you need traffic numbers, pay for Similarweb or triangulate with SimilarWeb's free digital overview + Ahrefs.

Is there a free tier for the actor?

Yes — Apify's free tier gives you enough compute to run small batches (up to a few hundred domains/month) without paying. Scale beyond that and you're in the low single digits of dollars per month for most realistic workloads. See the actor page for current pricing.

How does Cloudflare Radar compare for top-10 analysis?

For the genuinely top-10 global domains (the Googles, YouTubes, Facebooks of the world), Cloudflare Radar is often more useful because it gives you traffic share percentages and category context. Tranco will just tell you Google is #1 — Radar tells you Google accounts for roughly X% of all DNS queries observed by 1.1.1.1 in the last 24 hours, broken down by category. For anything below the top 100, Tranco is better.

When should I just pay for Similarweb?

When you need engagement metrics (sessions, session duration, pages per visit, bounce rate) alongside rank. Tranco gives you popularity; Similarweb gives you behavior. If your workflow requires "which of my competitors has stickier users?" or "how is on-site conversion behavior trending?" — Similarweb is the answer and there is no free substitute. Also worth it if you need country-level breakdowns at scale.

Conclusion

Alexa Rank is dead, but the need it served — a single, quotable, third-party popularity number for any domain — is not. For 80% of the use cases that used to rely on Alexa, Tranco is the right successor: free, daily, methodologically defensible, and commercially usable. For the other 20% (engagement metrics, country-level depth, real-time data), you'll still end up paying someone.

If you just want to get ranks into a spreadsheet without downloading CSVs or writing your own cache layer, grab the tranco-rank-lookup actor and run the Python example above against your comp set. Most teams are running useful competitive dashboards within an afternoon, and the whole stack costs less per month than a single Similarweb seat.

Heroku Is Getting Expensive: A 2026 Cost Calculator + Migration Guide

NexGenData — Tue, 01 Sep 2026 17:00:36 +0000

Heroku Is Getting Expensive: A 2026 Cost Calculator + Migration Guide

Heroku is not dead. But for a non-trivial slice of its user base, it has crossed the line from "obviously worth it" to "we should probably run the numbers." This post is for that second group.

A short timeline for context. In November 2022, Heroku killed its free tier, which had been the on-ramp for a generation of side projects and prototypes. What replaced it was Eco dynos at $5/month (shared across an account, sleeps after 30 minutes of inactivity) and Basic dynos at $7/month. The professional tiers stayed roughly where they were: Standard-1X at $25/month, Standard-2X at $50/month, Performance-M at $250/month, Performance-L at $500/month. Heroku Postgres got repriced into Essential-0 at $5/month (with 10k row limit), Essential-1 at $9/month, Essential-2 at $20/month, then Standard-0 at $50/month as the first "production" tier. Heroku Redis Mini is $15/month. Add a Papertrail add-on, a Scheduler, a few preview apps, and a staging environment, and a single-team SaaS is paying $200-400/month before anyone has written a line of code for scale.

Salesforce's 2024 annual report pegs Heroku at roughly 9 million active apps and 13 million developer accounts. Stack Overflow's 2025 developer survey reports that 38% of current Heroku users are "planning or actively evaluating" a migration within 12 months, up from 24% in the 2023 survey. The migration market is real and Heroku knows it — the 2025 Heroku "Fir" runtime (Kubernetes-based, AMD64 + ARM, OCI images) was an acknowledgment that the cheaper, more flexible alternatives had pulled far enough ahead that the product needed a structural response, not just a pricing one.

This post is a cost calculator and a migration guide. It is skeptical of the "Heroku is dead" framing — for a lot of teams Heroku is still the right answer — but it takes the bill seriously and does the actual math against Railway, Render, Fly.io, Cloudflare Workers + D1, and DigitalOcean App Platform. The companion tool is the heroku-cost-calculator Apify actor, which takes your current Heroku setup as input and spits out recommended targets, projected bills, and a migration outline.

Pricing data as of Q1 2026; check vendor pages for live rates before you commit to anything.

Why Heroku bills balloon

The sticker prices above are misleading in two directions. Good news first: a single Basic dyno at $7/month with an Essential-0 Postgres at $5/month is genuinely $12/month, and for a toy SaaS that sees 100 requests a day, that is still an excellent deal. There is a reason Heroku exists.

The bad news is that the bill structure is aggressively additive. A realistic "we have some paying customers" Heroku stack typically looks like this:

2x Standard-1X web dynos (for horizontal redundancy): $50/month
1x Standard-1X worker dyno (Sidekiq, Celery, whatever): $25/month
Heroku Postgres Standard-0: $50/month
Heroku Redis Mini: $15/month
Heroku Scheduler: $0/month (free, but limited to 10-minute precision and no retry logic)
Papertrail Choklad plan: $7/month
SSL for custom domains: included on Performance dynos, manual ACM setup on Standard, which most teams bypass by staying on *.herokuapp.com and eating the branding hit
Staging environment (same shape as prod, scaled down): ~$75/month
3 preview apps for PR reviews: 3 * ~$12 = $36/month

Running total: $258/month, and we have not added Sentry, a mail provider, a CDN, or Heroku Connect. This is the "$200-400/month before you blink" scenario. And it scales roughly linearly for a while — two Performance-M dynos, a Standard-2 Postgres, and a Premium-0 Redis is closer to $800/month for the same architecture shape.

The deeper problem is that Heroku's pricing is opaque at scale. You don't pay for bandwidth, but you do pay for dyno-hours. You don't pay for database connections, but you pay for row counts on Essential tiers. Add-ons are billed by a partner ecosystem with its own pricing logic. You can scrape together heroku ps and heroku pg:info output to build an accurate inventory, but very few teams have ever done it.

The alternatives, by the numbers

Here is the 2026 landscape for PaaS-style hosting, scoped to the "I just want to ship a web app + worker + database + Redis" use case.

Provider	Compute pricing	Database	Redis/KV	Free tier	Best for
Heroku	$7-500/mo/dyno (tiered)	Postgres $5-$3500+/mo	$15-1400/mo	None (Eco starts $5)	Salesforce integrations, compliance, zero-ops premium
Railway	$0.000231/GB-hr RAM + $0.000463/vCPU-hr, usage-based	Postgres included as a service (RAM+CPU metered)	Redis same	$5 credit on Hobby	Rails, Django, Node — Heroku feel with better pricing
Render	$7/mo Starter, $25/mo Standard, $85/mo Pro	Postgres $7/mo (256MB) to $95/mo (4GB)	Key-Value $10/mo+	Free web services (spin down after 15 min idle)	Predictable pricing, strong Docker support
Fly.io	$0.0000022/s per shared-cpu-1x-256mb (~$1.94/mo), scales by machine	Postgres via Fly Postgres, $1.94/mo baseline	Upstash Redis via extensions	3x shared-cpu-1x-256mb machines + 3GB storage free	Global edge, low-latency, per-region replicas
Cloudflare Workers + D1	$5/mo Workers Paid + $0.50/M requests after 10M	D1 Postgres-lite: $5/mo + $1/M reads, $1/M writes	KV $0.50/M reads	100k req/day + 5GB D1 storage free	Stateless APIs, edge-first, cheap for spiky traffic
DigitalOcean App Platform	Basic $5-12/mo, Pro $12-48/mo	Managed Postgres $15/mo (1GB) to $60/mo+	Managed Redis $15/mo+	3 static sites free	Teams who want DO's full stack (Droplets + Spaces nearby)

A few honest caveats on the table:

Railway is usage-based, which is cheaper if your traffic is spiky and more expensive if you're always-on at high RAM. Do the math for your actual profile.
Render's starter services are single-instance and restart on deploys with a few seconds of downtime. Not a deal-breaker, but not Heroku-parity.
Fly.io is the cheapest at small scale, but their Postgres offering is "you're responsible for it" — more like a managed VM than a managed service. Production-grade HA Postgres on Fly requires real effort.
Cloudflare Workers + D1 is a completely different architecture, not a Heroku drop-in. You are rewriting from Rails/Django/Express to Workers-flavored JavaScript or Python. Massive wins for the right shape of app, total wall for the wrong shape.
DigitalOcean App Platform is the closest to Heroku in feel, but their managed database minimums ($15/mo) are higher than the starter tiers on Render or Fly.

The architecture

The tool this post describes is an Apify actor that takes a Heroku inventory, applies a cost model, and recommends a target. High level:

+---------------------------+
|  Heroku app inventory     |
|  (dynos, DBs, add-ons,    |
|   team size, priority)    |
+------------+--------------+
             |
             v
+---------------------------+
|  heroku-cost-calculator   |
|  (Apify actor)            |
|                           |
|  - normalize inventory    |
|  - compute Heroku bill    |
|  - compute each target    |
|    bill (Railway, Render, |
|    Fly, Cloudflare, DO)   |
+------------+--------------+
             |
             v
+---------------------------+
|  Recommendation engine    |
|                           |
|  priority: cost | speed | |
|  ops-simplicity | scale   |
+------------+--------------+
             |
             v
+---------------------------+
|  Migration playbook       |
|  - target host + sizing   |
|  - projected monthly cost |
|  - step-by-step commands  |
|  - gotchas for this stack |
+---------------------------+

The calculator is not trying to replace your engineering judgment. It's trying to replace the spreadsheet you would otherwise spend an afternoon building, and replace the half-informed recommendations you would otherwise get from a Reddit thread where the top comment mentions their SvelteKit side project.

Using the calculator

The actor is public and free to run on Apify's platform. You can invoke it from the UI or from any Apify SDK. Here is the Python version.

from apify_client import ApifyClient

client = ApifyClient("APIFY_TOKEN")

run = client.actor("nexgendata/heroku-cost-calculator").call(run_input={
    "web_dynos": [{"size": "Standard-1X", "count": 2}],
    "worker_dynos": [{"size": "Standard-1X", "count": 1}],
    "postgres_tier": "Standard-0",
    "redis_tier": "Mini",
    "preview_apps": 3,
    "team_size": 5,
    "priority": "cost",
})

result = client.dataset(run["defaultDatasetId"]).iterate_items().__next__()
print(result)

The priority parameter is the load-bearing one. Options:

cost — minimize monthly bill, willing to absorb more ops work
simplicity — minimize ops work (Heroku-like feel), willing to pay more
speed — prioritize performance / global latency (tends toward Fly.io)
scale — prioritize headroom for 10x growth (tends toward DO or self-managed)

Output includes the computed Heroku bill, a per-provider bill for the equivalent setup, a recommended target based on priority, and a migration outline specific to that target.

Worked example: typical Rails/Django prod+staging

Let's run a real-shaped app through it. Our hypothetical SaaS:

Prod: 2x Standard-1X web + 1x Standard-1X worker = $75/month
Postgres: Standard-0 = $50/month
Redis: Mini = $15/month
Scheduler: free
Preview apps: 3 (ephemeral, auto-created for each PR) ~ $12/month
Staging: 1x Basic web + 1x Basic worker + Essential-1 Postgres = ~$25/month

Monthly Heroku bill: $177/month, or ~$2,120/year. Add a Papertrail plan at $7/month, a Sentry Business at $26/month, and Bugsnag at $40/month (common defaults) and the full prod stack is $250/month. For this example, we'll stick to just what Heroku bills ($177) to keep the comparison apples-to-apples.

Running the calculator with priority: "cost":

{
  "heroku_monthly_usd": 177,
  "targets": [
    {
      "provider": "railway",
      "monthly_usd_estimate": 68,
      "confidence": "high",
      "notes": "Usage-based; estimate assumes 60% CPU, 512MB avg per service."
    },
    {
      "provider": "fly.io",
      "monthly_usd_estimate": 54,
      "confidence": "medium",
      "notes": "Requires managing Fly Postgres HA manually for prod-grade setup."
    },
    {
      "provider": "render",
      "monthly_usd_estimate": 96,
      "confidence": "high",
      "notes": "Cleanest Heroku-feel; preview apps built in."
    },
    {
      "provider": "digitalocean",
      "monthly_usd_estimate": 108,
      "confidence": "high",
      "notes": "Managed Postgres minimum is $15/mo; raises baseline."
    },
    {
      "provider": "cloudflare",
      "monthly_usd_estimate": null,
      "confidence": "not_applicable",
      "notes": "Rails app does not map to Workers runtime; rewrite required."
    }
  ],
  "recommended": "railway",
  "recommended_reason": "Closest Heroku feel + ~61% cost reduction vs current. Fly.io is cheaper but requires more ops investment on Postgres HA.",
  "projected_savings_monthly_usd": 109,
  "projected_savings_annual_usd": 1308,
  "migration_outline": [
    "Export Heroku config: heroku config -s --app myapp > .env.prod",
    "Export Postgres: heroku pg:backups:capture --app myapp && heroku pg:backups:download",
    "Convert Procfile web + worker processes to railway.json services",
    "Provision Railway Postgres and Redis services",
    "Restore Postgres dump: psql $RAILWAY_DATABASE_URL < latest.dump",
    "Point staging DNS at Railway domain; smoke-test",
    "Cut over prod DNS with 60s TTL pre-lowered 24h in advance",
    "Monitor for 7 days; decommission Heroku after sign-off"
  ]
}

The recommendation is Railway with a ~61% cost reduction. Fly.io would be cheaper again, but the calculator flags the Postgres HA cost-in-engineering-time and down-weights it when priority is cost (not scale or simplicity).

If we rerun with priority: "simplicity", Render moves to the top — preview apps are built-in, the deploy feel is closest to Heroku, and the ops burden is lowest. If we rerun with priority: "speed", Fly.io wins because of its multi-region primitives. The point of the priority parameter is that the calculator will not pretend there is a single right answer.

Gotchas when actually migrating

A cost estimate is the easy part. The actual migration has a dozen small traps. In rough order of appearance:

DNS cutover

Lower your DNS TTL to 60 seconds at least 24-48 hours before the cutover. Heroku's DNS for custom domains requires a CNAME (you don't get an A record because the edge IP rotates), which is fine unless you are using a naked apex domain with a provider that doesn't support ALIAS or ANAME. Cloudflare's CNAME flattening solves this on Cloudflare DNS. Route 53 handles it with ALIAS records. Classic BIND does not.

Environment variables

Export with:

heroku config -s --app myapp > .env.prod

The -s flag gives you KEY=value lines, which most target hosts can import directly (Railway has a paste-in env import, Render does too, Fly.io uses fly secrets import < .env.prod). Don't forget to rotate anything that was leaked to logs or build output during the migration.

Buildpack to Dockerfile

Most targets strongly prefer (Railway, Render, Fly.io, DO) or require (Cloudflare Workers) explicit build definitions rather than Heroku buildpacks. The heroku/builder project will still produce an OCI image from a Heroku-style buildpack stack, and both Railway and Render will auto-detect Rails/Django/Node and produce a reasonable Dockerfile-equivalent for you. But for anything non-trivial (native gem compilation, custom system packages, specific Python versions), plan on writing a real Dockerfile. It is an afternoon of work, and you will be glad to have it under source control afterward.

Postgres dump and restore

heroku pg:backups:capture then heroku pg:backups:download gives you a latest.dump pg_dump archive. Restore with:

pg_restore --verbose --clean --no-acl --no-owner \
  -h TARGET_HOST -U TARGET_USER -d TARGET_DB latest.dump

The --no-acl --no-owner flags are critical — Heroku's Postgres creates roles and grants that don't exist on the target, and pg_restore will error loudly without those flags. For very large databases (>20GB), pg_dump is slow and pgcopyonly or a logical replication approach (Bucardo, pglogical, or a native PG16+ logical subscription) is worth the setup cost to get a zero-downtime cutover.

Add-on replacements

Heroku add-on	Common replacement
Heroku Redis	Upstash Redis (serverless, pay-per-request) or provider-managed
Papertrail	Loki + Grafana (self-host) or Better Stack / Axiom
Sentry	Sentry (portable, stays)
Bugsnag	Bugsnag (portable, stays)
SendGrid	SendGrid, Postmark, Resend (portable)
Heroku Scheduler	Railway Cron, Render Cron, Fly.io machines + schedules, GitHub Actions cron
Heroku Connect	No good replacement; see the "staying on Heroku" section

The portable add-ons (Sentry, Bugsnag, SendGrid, Datadog, New Relic) migrate trivially — they're just API keys. The Heroku-specific ones (Heroku Redis, Heroku Postgres, Heroku Scheduler, Heroku Connect) need replacements, and Heroku Connect is the hardest.

Scheduled jobs

Heroku Scheduler supports 10-minute, hourly, and daily jobs. Railway Cron and Render Cron both support full cron expressions. Fly.io doesn't have a native scheduler product but lets you use machines with schedules — it's a few lines of fly.toml. For the occasional "run this once a day" job, GitHub Actions cron is free and dead simple.

Worker queues

If you're using Sidekiq or Celery with Heroku Redis, plan to do the queue drain carefully. Stop enqueueing new jobs, let the queue empty on the old environment, then cut over. Running two workers pointing at two Redis instances is how you end up double-processing payments.

When staying on Heroku still makes sense

I have spent most of this post explaining why teams leave Heroku. Here is the honest counter-case, because there are still real reasons to stay.

Salesforce integrations. If your app is deeply tied to Salesforce via Heroku Connect (bi-directional data sync between Postgres and Salesforce objects), there is no clean replacement. You can build your own with the Salesforce Bulk API, but "build your own Heroku Connect" is a project, not a sprint.
Compliance-heavy workloads. Heroku Shield (HIPAA, PCI-DSS high-trust, SOC 2 Type 2 with audit trails) is a real product. The alternatives have compliance stories (Fly.io has HIPAA tiers, Render has SOC 2), but if you already have your BAA signed and your QSA comfortable with Heroku, the cost of re-auditing is not trivial. The dollar savings on compute rarely justify a new compliance workstream.
Zero-ops as a stated value. Some teams — often small, profitable, non-technical-founder teams — assign real value to "we never touch the infrastructure." If your company is profitable, you have 5 engineers, and your Heroku bill is $500/month, the migration is probably not worth the engineering time. $500/month is less than a week of engineer salary. The calculator surfaces this: if priority: "simplicity" and the savings are under ~$200/month, the recommendation string is "probably stay on Heroku and revisit in 12 months."
Early-stage apps with very bursty traffic. Eco dynos ($5/month, sleep when idle) are still an excellent deal for something you want to keep online but aren't actively promoting. The free-tier equivalents on Fly.io and Cloudflare are also good, but if you already have an account, inertia is fine.

There is no shame in staying on Heroku. There is also no moral victory in leaving. Do the math.

FAQ

Does the calculator include Performance tiers?

Yes. It models Performance-M ($250/month) and Performance-L ($500/month) dynos, and for each it computes equivalent targets. For Performance-L especially, the savings are larger in absolute dollars but smaller in percentage — at that tier you're competing against less-abstracted infrastructure (DO Droplets, AWS ECS) where Heroku's margin is still defensible.

What about Heroku Connect pricing?

Heroku Connect pricing starts at $2,100/month for the Standard tier (5M rows, 10 mappings) and climbs fast. The calculator treats Heroku Connect as a "sticky" component — if your inventory includes it, the recommendation is weighted heavily toward "stay on Heroku or plan a multi-quarter replacement project." There is no drop-in replacement at the PaaS layer.

How does Railway compare to Fly.io for Rails apps?

Both work. Railway is closer to Heroku's mental model: you push code, it builds, it runs. Fly.io asks you to think about machines, regions, and a Fly-specific fly.toml. For a standard Rails app with a single region and a standard Postgres, Railway is less friction. For a Rails app where you actually care about edge latency (global users), Fly.io's region primitives are better. Cost-wise Fly.io is usually cheaper at the same spec, but you'll pay back some of that savings in ops time unless your team already knows the Fly.io mental model.

Is Cloudflare Workers viable for a Postgres-backed Django app?

Not really. Workers runs a V8 isolate or Python WASI runtime with strict CPU-time limits, and Django's synchronous ORM is a poor fit. You can proxy Postgres through Hyperdrive (Cloudflare's connection pooler) and run a subset of Django in Python Workers, but realistically you are rewriting to Workers-native patterns. Workers + D1 is a superb fit for new projects designed for that architecture — edge APIs, stateless services, JAMstack backends. It is a poor fit for existing monoliths.

What about Supabase for the DB only?

Supabase Postgres is legitimately one of the best-priced managed Postgres options in 2026. Pro tier at $25/month gives you 8GB storage, daily backups, and read replicas as an add-on. Many teams migrate just the database to Supabase and leave compute on Heroku or Railway, which often saves more than moving compute. The calculator has a postgres_only_migration flag that models this scenario.

How does pricing change with autoscaling?

Heroku's autoscaling is only available on Performance dynos and adds complexity to the bill because you are paying for the peak of the scaling window. Railway's usage-based model and Fly.io's per-second billing are both more efficient for autoscaling workloads — you pay for what you used, not what you provisioned. The calculator has a traffic_profile parameter (steady, business_hours, spiky) that adjusts the target estimates for each provider's billing model. Spiky workloads see the largest relative savings on Railway and Fly.io.

Can I run the calculator for free?

Yes. The actor uses Pay-Per-Event pricing with the first run free, and typical runs cost $0.00 to $0.02. You'll need an Apify account (free to create). The actor's input form is usable from the Apify web UI without any API keys.

Where does the pricing data come from?

The actor pulls published pricing from each provider's pricing page, normalizes it to a common schema (dollars per vCPU-hour, dollars per GB-RAM-hour, dollars per GB storage-month), and caches the normalized values. The cache is refreshed nightly and the actor output includes a pricing_data_freshness timestamp. For enterprise pricing (Heroku Enterprise, Render Enterprise, Railway Pro), the calculator uses public list prices and flags the estimate as "enterprise discount not modeled." If you're on a negotiated contract, the real numbers on your side will be different.

Conclusion

Heroku is not dead, but for a lot of teams it is quietly over-priced. Running the numbers for 20 minutes is worth doing at least once a year — and if the answer is "we'd save $100/month but it'd cost us $20k in engineering time," that is also a useful result. The heroku-cost-calculator is free to run and does the math in under a minute. Plug in your current setup, pick a priority, and see what the spreadsheet says before you spend a sprint on the migration.

Court Records Research for Legal & Due Diligence Workflows (2026)

NexGenData — Thu, 27 Aug 2026 15:08:57 +0000

Court Records Research for Legal & Due Diligence Workflows (2026)

When you are evaluating a company for acquisition, a candidate for a sensitive hire, or a counterparty for a multi-million-dollar contract, "have they been sued before?" is often the most informative single question you can ask. Public court records answer it — if you can extract them at scale.

Some grounding numbers for 2026: PACER (the federal court records system) contains 1.2 billion documents across 450+ million dockets. State courts collectively file around 83 million new cases per year (National Center for State Courts, 2024 data), of which approximately 21 million are civil. Most of this is public, but less than 8% is freely searchable through any unified API. The DOJ's 2025 report on fraud prosecutions showed that private-sector litigation checks detect roughly 3.4x more red flags per target than a public-records-only check — but only when the search is both federal-and-state AND does proper name disambiguation. A superficial "Google the name" check catches about 12% of material cases. A PACER-only check catches federal but misses the bulk of state civil and all county matters. A real diligence pipeline crosses all three tiers, which is exactly what this post builds.

The problem: US court records are scattered across PACER (federal), fifty state systems, and hundreds of county-level courts, each with their own auth scheme, rate limits, and data model. Pulling a comprehensive record for a single subject can take a paralegal half a day. Pulling it for 200 subjects (due-diligence scale) is a project that used to justify hiring a boutique diligence firm at $250/subject. In 2026 it becomes a scripted step that completes overnight and costs roughly 1/40th of that per subject.

This post walks through how to build an automated court records research pipeline in 2026 using Apify, with use cases spanning M&A diligence, litigation tracking, KYC, and background checks. A brief note on framing: this is not about turning your laptop into a credit bureau. Any decision with FCRA implications (hiring, credit, insurance, housing) requires compliant processes, proper adverse-action procedures, and often a licensed CRA. What you can build is a research tool — a way to know what public records exist about an entity before you sign a contract, partner with a company, or fund a deal.

Why this is hard

No unified index. PACER covers federal. State systems are individually gated. County courts often have terrible interfaces or none.
PACER costs real money. $0.10/page for search results, capped at $3.00 per document. 100 searches can cost $30+.
Name matching is fuzzy. "John A. Smith" and "Smith, John A." and "J. A. Smith" are the same person — or three different people. Disambiguation requires DOB, location, or case context.
Authentication and CAPTCHAs. Many state systems throw CAPTCHAs on anything that looks automated.
Compliance. FCRA (in the US) restricts how court records can be used for employment or credit decisions. Your pipeline needs an audit trail.
Sealed and expunged records. Some states automatically seal juvenile, sealed, or expunged cases; others seal upon petition. Absence of a hit in a search does not mean absence of a case — it means absence of a visible case.
Historical depth varies. Some jurisdictions have digitized back to the 1980s; others only 2005+. For older cases, physical court-house archives may be the only option.

The architecture

[List of subjects (name, location)]
          |
          v
 [court-records-search actor] --> federal + state + county hits
          |
          v
  [Dedup & entity resolution]
          |
          v
    [Case detail enrichment]
          |
          v
 [Postgres + audit log]
          |
          v
  [Report generator (PDF/CSV)]

The court-records-search actor abstracts over the major federal and state court systems, handles CAPTCHAs via residential proxies, and returns normalized JSON.

Step 1: Search by subject

from apify_client import ApifyClient
client = ApifyClient("APIFY_TOKEN")

subjects = [
    {"name": "Acme Holdings LLC", "state": "DE", "type": "entity"},
    {"name": "John A. Smith", "state": "NY", "dob_year": 1978, "type": "person"},
]

run = client.actor("nexgendata/court-records-search").call(run_input={
    "subjects": subjects,
    "jurisdictions": ["federal", "NY", "DE", "CA"],
    "case_types": ["civil", "criminal", "bankruptcy"],
    "date_from": "2015-01-01",
})

cases = list(client.dataset(run["defaultDatasetId"]).iterate_items())

Each case:

{
  "subject_query": "Acme Holdings LLC",
  "case_number": "1:23-cv-04578",
  "court": "S.D.N.Y.",
  "jurisdiction": "federal",
  "case_type": "civil",
  "filed_date": "2023-08-12",
  "case_status": "Terminated",
  "parties": [
    {"role": "plaintiff", "name": "Bright Path Partners"},
    {"role": "defendant", "name": "Acme Holdings LLC"}
  ],
  "causes_of_action": ["Breach of contract", "Fraudulent inducement"],
  "disposition": "Settled",
  "pacer_url": "https://ecf.nysd.uscourts.gov/..."
}

Step 2: Dedupe and disambiguate

The same case may appear under both the plaintiff and the defendant name in your result set. Dedupe on case_number:

seen = set(); unique = []
for c in cases:
    k = (c["jurisdiction"], c["case_number"])
    if k not in seen:
        seen.add(k); unique.append(c)

For person subjects, apply a secondary filter — filter out cases where the party name matches but the DOB or state is known to be a mismatch. Entity subjects (LLCs) are easier because names are more unique.

Step 3: Classify the risk signal

Not every case is a red flag. A breach-of-contract case where the subject won is very different from a fraud case they settled. Tag each case:

RED_FLAGS = {"fraud", "securities fraud", "wire fraud",
             "embezzlement", "ponzi", "misappropriation"}

def risk_level(case):
    coas = [c.lower() for c in case.get("causes_of_action", [])]
    if any(flag in " ".join(coas) for flag in RED_FLAGS):
        return "high"
    if case.get("case_type") == "criminal":
        return "high"
    if case.get("disposition") == "Judgment for plaintiff" and any("defendant" == p["role"] for p in case["parties"]):
        return "medium"
    return "low"

for c in unique:
    c["risk_level"] = risk_level(c)

Step 4: Summarize per subject

from collections import defaultdict
summary = defaultdict(lambda: {"total": 0, "high": 0, "medium": 0, "low": 0, "cases": []})
for c in unique:
    s = summary[c["subject_query"]]
    s["total"] += 1
    s[c["risk_level"]] += 1
    s["cases"].append(c)

for subject, s in summary.items():
    print(f"{subject}: {s['total']} cases ({s['high']} high-risk)")

Step 5: Generate a report

For M&A and diligence, deliverable = PDF. Render a Jinja template with case summaries, risk levels, and links back to source documents. Audit log every query — including time, operator, and subject — for FCRA compliance.

Here is a minimal but realistic report-generation pipeline that writes both the audit log and the PDF. Uses Jinja2 + WeasyPrint, both of which are ~2 minute installs:

import datetime, hashlib, json
from jinja2 import Template
from weasyprint import HTML
import psycopg

AUDIT_DB = psycopg.connect("postgresql://.../audit")

def audit(operator, subject, query_hash, n_hits):
    with AUDIT_DB.cursor() as cur:
        cur.execute("""
            INSERT INTO diligence_audit
            (ts, operator, subject_name, query_hash, n_hits)
            VALUES (%s, %s, %s, %s, %s)
        """, (datetime.datetime.utcnow(), operator, subject, query_hash, n_hits))
    AUDIT_DB.commit()

REPORT_TEMPLATE = Template("""
<html>
<head><style>
  body { font-family: -apple-system, Helvetica, Arial; }
  .high { background: #ffe5e5; padding: 8px; margin: 4px 0; border-left: 4px solid #c00; }
  .medium { background: #fff4d6; padding: 8px; margin: 4px 0; border-left: 4px solid #d88; }
  .low { background: #f0f0f0; padding: 8px; margin: 4px 0; border-left: 4px solid #999; }
  table { width: 100%; border-collapse: collapse; margin-top: 12px; }
  td, th { padding: 6px 10px; border-bottom: 1px solid #ddd; text-align: left; }
</style></head>
<body>
<h1>Diligence Report — {{ subject }}</h1>
<p>Generated {{ generated_at }} by {{ operator }}</p>
<p>Jurisdictions searched: {{ jurisdictions|join(', ') }}</p>

<h2>Summary</h2>
<ul>
  <li>Total cases: {{ s.total }}</li>
  <li><b>High risk:</b> {{ s.high }}</li>
  <li>Medium risk: {{ s.medium }}</li>
  <li>Low risk: {{ s.low }}</li>
</ul>

<h2>Case details</h2>
{% for c in s.cases|sort(attribute='risk_level', reverse=True) %}
  <div class="{{ c.risk_level }}">
    <b>{{ c.case_number }}</b> — {{ c.court }} ({{ c.case_type }})<br/>
    Filed: {{ c.filed_date }} | Status: {{ c.case_status }}<br/>
    Causes: {{ c.causes_of_action|join(', ') }}<br/>
    Disposition: {{ c.disposition or 'Pending' }}<br/>
    <a href="{{ c.pacer_url }}">Source</a>
  </div>
{% endfor %}
</body></html>
""")

for subject, s in summary.items():
    query_hash = hashlib.sha256(json.dumps(s, default=str, sort_keys=True).encode()).hexdigest()[:16]
    audit(operator="steve@example.com", subject=subject,
          query_hash=query_hash, n_hits=s["total"])

    html = REPORT_TEMPLATE.render(
        subject=subject, s=s,
        jurisdictions=["federal", "NY", "DE", "CA"],
        operator="steve@example.com",
        generated_at=datetime.datetime.utcnow().isoformat(),
    )
    HTML(string=html).write_pdf(f"report_{subject.replace(' ','_')}.pdf")
    print(f"Wrote report for {subject}: {s['total']} cases, {s['high']} high-risk")

The audit log is the piece most diligence pipelines skip and regret. For any case that might later need to support a decision, you want a record of exactly what was queried, when, by whom, and what was returned — ideally hashed so you can prove non-tampering later.

Use cases

1. M&A legal diligence. An acquirer running a $40M deal pulled 8 years of litigation history for the target plus its officers. Found undisclosed wage-and-hour class action. Renegotiated escrow terms.

2. Vendor KYC. A fintech onboards 50 merchant partners/month. The court-records search is automated, and any hit triggers human review before signing.

3. Journalism / investigative research. A reporter investigating a local politician pulls state and federal court records to build a timeline of civil disputes.

4. Pre-hire screening (regulated, FCRA-compliant). Background-check vendors pull criminal and civil records with signed consent, using the Apify dataset for structured data, not just screenshots.

5. Insurance underwriting pre-check. A commercial insurance broker runs court records on prospective business clients before quoting. A pattern of prior employment claims against a restaurant chain, for example, informs the EPLI premium. The broker reports catching approximately 18% of material risk indicators that would otherwise have surfaced only after a claim.

6. Investor diligence on founders. A family office running later-stage venture checks runs every founder in a target cap table through the pipeline. They have twice killed deals based on undisclosed prior fraud-adjacent litigation that would have been embarrassing to surface post-investment. The cost per founder checked is about $0.40; the cost of a blown-up investment is in the millions.

Pricing comparison

Service	Monthly cost (200 subjects)	Federal + state?	Structured JSON?
PACER direct	~$60 page fees + eng time	Federal only	No (HTML)
Trellis.law	$399+/mo	State-focused	Yes
CourtListener	Free API (limited)	Federal only	Yes
LexisNexis / Westlaw	$500+/user/mo	Yes	Yes
Apify actor	~$30	Yes	Yes

Pay-per-search pricing is the killer feature for occasional users who do not need a $500/mo legal database seat.

Common pitfalls

Court records at scale is a surprisingly nuanced domain. These are the pitfalls that matter:

FCRA compliance is your problem. The data is public, but using it for employment or credit decisions has legal requirements: written consent, adverse-action notices, dispute procedures, and usually a licensed CRA. The Apify actor gives you structured data. Turning that data into a hiring decision without proper process is where the liability is. Consult counsel, especially for anything touching employment.
PACER fee caps. PACER has per-quarter fee caps (currently $30/quarter waived if you spend under that threshold). Budget accordingly for federal-heavy searches. Note that PACER reform proposals in 2024-2025 may move toward free access for most docket metadata; keep an eye on the federal calendar.
Seal / redaction. Sealed and juvenile cases are not returned. Absence of a record does not equal exoneration — it may equal successful expungement. Your report templates should explicitly state what was searched and what was excluded by design.
Recency lag. County systems can be 2-4 weeks behind real filings. Federal PACER updates within hours of filing. If you are doing due diligence with a critical cutoff date, schedule a re-scrape close to signing to catch recent filings.
Common-name problem. "John Smith" in Texas will return hundreds of matches. Any search without DOB, middle name, or location context is essentially useless for disambiguation. Collect at least two identifiers before running.
Corporate-entity name drift. "Acme Holdings LLC" may be the parent, but the lawsuits are against "Acme Operating Co." or "Acme Acquisition Sub 7." Cross-reference with corporate registries (e.g., Delaware Secretary of State, OpenCorporates) to enumerate related entities before searching.
Inbound vs. outbound litigation. A case where your subject is the plaintiff is very different from a case where they are the defendant. Early-stage scripts sometimes conflate these and flag every case as a risk. Always filter by role.
Criminal record nuances. "Arrested" is not "convicted." "Charged" is not "guilty." "Pleaded guilty to a reduced charge" is different from "acquitted after trial." For criminal records specifically, the disposition field is more informative than the charge field.
Class action participation. If your subject is a member of a class, they may appear as a "class member" in a large case but not be individually named. Whether that counts as "the subject has been involved in litigation" depends on your context. Most diligence treats class-member status differently from individually-named parties.
Foreign jurisdictions. If the subject has operated internationally, US-only court searches miss UK court records, EU court records, and others. For cross-border diligence, add OFAC and international litigation sources.

How NexGenData handles this

Court records are the kind of domain where shortcuts in the actor cost the end user real money. A few specific design choices:

Unified federal + state + county schema. Every jurisdiction's output normalizes to the same JSON shape. You write one parser, not fifty.

Entity-type-aware search. Pass type: "entity" for companies and type: "person" for individuals, and the actor routes queries differently. Entity searches cross-reference with Secretary of State data; person searches leverage DOB and location filters.

Proxy and CAPTCHA handling. State and county systems that throw CAPTCHAs are handled via residential proxies and CAPTCHA-solving middleware. You rarely need to intervene.

Disposition normalization. Raw court systems use wildly different disposition language ("Dismissed with prejudice", "Disposed — Settled", "Discontinued"). The actor normalizes to a canonical set (settled, judgment-for-plaintiff, judgment-for-defendant, dismissed, pending) while preserving the raw string.

Incremental re-search. Run the same subject weekly and the actor returns only new/changed cases. For ongoing monitoring (litigation watch), this keeps costs bounded.

Audit-ready output. Every result includes source URL, retrieval timestamp, and a content hash. Perfect for FCRA audit trails and for proving a given case was retrieved on a given date.

Pay-per-result. 200 subjects with full federal + state search typically costs $25-40. Compare to Trellis ($399+/month seat) or LexisNexis ($500+/user/month).

Conclusion

Court-records research used to require a paralegal or a $500/mo legal database. In 2026, with pay-per-result APIs, it becomes a normal scripted step in any diligence pipeline. The main constraint is now legal compliance, not data access — which is the right problem to have.

Three actors to start with:

Court Records Search — federal + state + county case search.
Domain WHOIS Lookup — cross-reference corporate web presence.
AP News Scraper — layer in news coverage of subjects for context.

FAQ

Is scraping court records legal?
Court records are public information by design. Accessing them via the court's own systems or via API aggregators is generally legal. Using them for FCRA-regulated purposes (employment, credit, insurance, housing) requires a compliant process and often a licensed CRA. Using them for non-FCRA purposes (M&A diligence, journalism, KYC, litigation research) is broadly permissible. Consult counsel for your specific use case.

How does this compare to PACER direct?
PACER is the authoritative federal source with per-page fees ($0.10/page, capped $3.00/document). The Apify actor uses PACER and other sources and normalizes output. You pay Apify's per-result fee and avoid the overhead of managing a PACER account. For very high federal-only volumes, direct PACER access may be cheaper; for mixed federal + state, the actor wins on total cost.

What about CourtListener?
CourtListener (from the Free Law Project) is an excellent free resource for federal opinions and PACER-adjacent docket metadata. Its coverage of state court data is limited. The actor complements CourtListener by adding state and county coverage.

How fresh is the data?
Federal PACER: near-real-time, typically within hours of filing. State courts: usually 1-7 days behind. County courts: 1-4 weeks behind, highly variable. If you need real-time monitoring, schedule a re-scrape cadence that matches the worst-case jurisdiction you care about.

Can I get full case documents (complaints, motions, judgments)?
The actor returns docket metadata and document URLs. Fetching full-text documents typically requires a separate PACER fee (for federal) or a document-retrieval fee for state systems. For bulk document retrieval, the Free Law Project's RECAP service is a free alternative that lets community-contributed documents be reused without fees.

What about non-US court records?
The actor focuses on US jurisdictions. For UK court records, use Courts and Tribunals Judiciary or Judiciary.uk. For EU, ECLI-indexed searches vary by country. Cross-border diligence typically combines multiple national sources.

Do I need an attorney to interpret these results?
For high-stakes decisions, yes. A paralegal or attorney can distinguish nuisance claims from material ones, weigh the significance of a settlement vs. a verdict, and identify omissions. The actor surfaces data; humans still provide judgment.

How do I handle false positives (wrong person with the same name)?
Always collect at least two identifying data points (DOB, city, employer) before running a search. Tag any match without a strong secondary identifier as "unconfirmed" in your output, and require human review before treating it as a hit.

Related tools

Page Speed Monitoring at Scale: Lighthouse API Alternatives (2026)

NexGenData — Tue, 25 Aug 2026 16:50:55 +0000

Page Speed Monitoring at Scale: Lighthouse API Alternatives (2026)

Google's PageSpeed Insights API has a rate limit of 25,000 queries per day per project — generous until you actually try to monitor a real e-commerce catalog with 40,000 product pages. At that point you are either batching over three days (stale data) or setting up 10 GCP projects to fan out keys (a compliance nightmare). Self-hosting Lighthouse works but turns into a $500/month EC2 bill the minute you want parallelism.

The reason this matters more in 2026 than it did three years ago is that Core Web Vitals are now a ranking signal with documented economic impact. Google's March 2024 INP (Interaction to Next Paint) rollout replaced FID as a CWV metric, and a 2025 Shopify Plus study of 3,400 stores found that moving from the "needs improvement" to "good" bucket on mobile LCP correlated with a 7.8% conversion-rate lift at the p75 percentile. The same study reported that stores which monitored CWV continuously (rather than quarterly audits) caught 83% of regressions before they affected organic traffic for more than 72 hours. A quarterly Lighthouse audit is not monitoring — it is archaeology. If you care about SEO, ad quality scores, or conversion rate, continuous monitoring is the bar. The mental model to adopt: page speed is a time-series, not a snapshot. Treat it like APM for user-facing latency — Datadog for your frontend.

This post compares the viable Lighthouse API alternatives in 2026 and walks through a production pipeline for Core Web Vitals monitoring across thousands of URLs. We cover lab vs. field data, how to correlate lab regressions with CrUX (Chrome User Experience Report) field data, how to wire alerts that do not cry wolf, and where the commercial tools (SpeedCurve, Calibre, DebugBear) are worth the money vs. where you are paying for a dashboard you could build in a day.

Why this is hard

Lighthouse itself is open source and fine for one-off runs. Scaling it is the problem:

Each Lighthouse run costs 5-15 seconds of CPU. A 10k-URL daily audit is 25-40 hours of sequential compute.
Chromium instance management. Parallel runs require careful container orchestration — memory leaks are common past 50 parallel workers.
Network variability. Run the same URL 5 times and you get 5 different LCPs. Proper monitoring needs median-of-N runs.
Headers and auth. Staging environments often require basic auth or custom cookies. PageSpeed Insights does not support this; only a custom Lighthouse can.
Core Web Vitals vs. field data. Lighthouse reports lab data. CrUX reports field data. You need both for a true picture.
Geographic variance. A page that loads in 1.2s from a US datacenter may take 4.8s from Jakarta. If your audience is global, single-region monitoring tells a flattering lie.
Alert noise. Naive threshold alerts ("LCP > 2.5s = page") fire on every minor variance. Production alerting needs percentile-based regressions with proper baselines.

The architecture

[URL list (e.g. sitemap.xml)]
          |
          v
 [page-speed-analyzer actor] --> Lighthouse audits, parallel, with proxies
          |
          v
 [Postgres / ClickHouse]
          |
          v
 [Grafana dashboard]
 [Slack alerts on regressions]

The page-speed-analyzer actor runs headless Chrome with Lighthouse, supports custom cookies/headers, parallelizes automatically on the Apify platform, and costs a fraction of a self-hosted fleet at low/medium volume.

Step 1: Collect URLs

The simplest way: point at a sitemap.

import xml.etree.ElementTree as ET, requests, re

r = requests.get("https://example.com/sitemap.xml", timeout=15)
urls = re.findall(r"<loc>(.*?)</loc>", r.text)
print(f"Found {len(urls)} URLs")

For e-commerce, pull collection sitemaps plus product sitemaps. For content sites, pull the main sitemap.

Step 2: Run Lighthouse at scale

from apify_client import ApifyClient
client = ApifyClient("APIFY_TOKEN")

run = client.actor("nexgendata/page-speed-analyzer").call(run_input={
    "urls": urls[:2000],
    "strategy": "mobile",           # "desktop" or "mobile"
    "categories": ["performance", "seo", "accessibility"],
    "runs_per_url": 3,              # median-of-3
    "throttling": "simulated-3g",
    "headers": {"Cookie": "staging_auth=abc123"},
})

results = list(client.dataset(run["defaultDatasetId"]).iterate_items())

Each result:

{
  "url": "https://example.com/products/widget",
  "strategy": "mobile",
  "run_timestamp": "2026-04-17T10:00:00Z",
  "performance_score": 67,
  "lcp_ms": 3120,
  "fcp_ms": 1480,
  "cls": 0.18,
  "tbt_ms": 410,
  "ttfb_ms": 680,
  "speed_index_ms": 3890,
  "opportunities": [
    {"id":"unused-javascript","wasted_ms":1200},
    {"id":"offscreen-images","wasted_ms":450}
  ]
}

Step 3: Persist and diff

import psycopg
with psycopg.connect("postgresql://...") as con, con.cursor() as cur:
    for r in results:
        cur.execute("""
        INSERT INTO pagespeed (url, run_ts, lcp, fcp, cls, tbt, ttfb, perf_score, strategy)
        VALUES (%s,%s,%s,%s,%s,%s,%s,%s,%s)
        """, (r["url"], r["run_timestamp"], r["lcp_ms"], r["fcp_ms"],
              r["cls"], r["tbt_ms"], r["ttfb_ms"],
              r["performance_score"], r["strategy"]))

Daily regression detection:

WITH today AS (
  SELECT url, AVG(lcp) AS lcp FROM pagespeed
  WHERE run_ts > now() - INTERVAL '24 hours' GROUP BY url
),
last_week AS (
  SELECT url, AVG(lcp) AS lcp FROM pagespeed
  WHERE run_ts BETWEEN now() - INTERVAL '8 days' AND now() - INTERVAL '1 day'
  GROUP BY url
)
SELECT t.url, l.lcp AS lcp_last_week, t.lcp AS lcp_today,
       round((t.lcp - l.lcp) / l.lcp * 100, 1) AS pct_change
FROM today t JOIN last_week l USING (url)
WHERE t.lcp > l.lcp * 1.15
ORDER BY pct_change DESC;

That query surfaces URLs that got 15% slower week-over-week. Post the top 10 to Slack every morning.

Step 4: Field data cross-check

Lab data is reproducible but synthetic. Real users experience something different. Combine with CrUX (Chrome User Experience Report) via its BigQuery public dataset or REST API, and align by URL or origin. If your lab LCP is 2.1s but CrUX p75 is 4.8s, something in the real user journey (3rd-party scripts, bad CDN cache hit ratio) is not reproduced in lab.

A practical way to pull CrUX field data alongside your lab runs:

import requests, os

CRUX_KEY = os.environ["CRUX_API_KEY"]

def crux_for_url(url):
    r = requests.post(
        f"https://chromeuxreport.googleapis.com/v1/records:queryRecord?key={CRUX_KEY}",
        json={"url": url, "formFactor": "PHONE",
              "metrics": ["largest_contentful_paint", "interaction_to_next_paint", "cumulative_layout_shift"]},
        timeout=10,
    )
    if r.status_code == 404:
        return None  # not enough traffic for CrUX data
    return r.json().get("record", {}).get("metrics", {})

enriched = []
for r in results:
    crux = crux_for_url(r["url"]) or {}
    lcp_field_p75 = crux.get("largest_contentful_paint", {}).get("percentiles", {}).get("p75")
    inp_field_p75 = crux.get("interaction_to_next_paint", {}).get("percentiles", {}).get("p75")
    enriched.append({
        **r,
        "lcp_field_p75": lcp_field_p75,
        "inp_field_p75": inp_field_p75,
        "lab_field_gap": (lcp_field_p75 - r["lcp_ms"]) if lcp_field_p75 else None,
    })

# flag URLs where field is >40% worse than lab
suspects = [e for e in enriched if e["lab_field_gap"] and e["lab_field_gap"] > 0.4 * e["lcp_ms"]]
print(f"{len(suspects)} URLs where real users experience >40% worse LCP than lab")

The lab_field_gap column is where the interesting engineering happens. Lab-only dashboards miss this entirely. A typical cause: a CDN with a 40% cache hit ratio from Asia (vs. 95% from North America), making your lab runs from Virginia look fantastic while half your actual users experience origin-pull latency.

Use cases

1. E-commerce catalog monitoring. A DTC brand watches 8,000 product URLs. Any URL with LCP > 4s gets flagged for the frontend team; a weekly report ranks offenders by pageview volume.

2. Pre-deploy regression gate. A CI pipeline runs Lighthouse on 20 critical URLs against a staging URL. Fails the build if performance score drops >10 points.

3. Agency client reporting. An SEO agency runs weekly audits for 40 clients, exports PDF snapshots, and bills on improvement metrics.

4. SaaS SLA enforcement. A B2B SaaS tracks TTFB across tenants. Tenants on specific cloud regions are flagged when TTFB exceeds SLA.

5. A/B test variant performance. A media site A/B tests layout changes with Optimizely. A scheduled Lighthouse run against each variant's control URL surfaces when a variant degrades LCP enough to offset its engagement win. The team killed three variants that looked like wins on click-through but were net losses on revenue because of CLS spikes.

6. Third-party script impact audit. A marketing team was pushed to add an analytics vendor. A pre/post Lighthouse comparison showed the vendor's tag adds 340ms to LCP p75 and 0.12 to CLS on key product pages. That became the basis for negotiating a hosted-on-CDN edition of the tag instead of the default JS snippet.

Pricing comparison

Service	Cost (10k audits/day)	Custom headers?	Field data?
Google PageSpeed Insights	Free (25k/day limit)	No	Via CrUX
SpeedCurve	$174+/mo	Yes	Yes
Calibre	$125+/mo	Yes	No
DebugBear	$69+/mo	Yes	Yes
Self-hosted Lighthouse + k8s	$300-800/mo	Yes	No
Apify actor	~$25	Yes	Via CrUX

Pay-per-result at scale wipes out the monthly seat model, and custom headers unlock staging/authenticated monitoring that PageSpeed Insights can't do.

Common pitfalls

Page speed monitoring is easy to set up and hard to set up correctly. These are the traps:

Throttling matters. Lab runs without throttling underreport real-world slowness. Default to simulated-3g or simulated-4g for mobile audits. Desktop audits should still throttle to a mid-tier broadband profile — unthrottled desktop Lighthouse is essentially useless as a production signal because your office internet is not the world's internet.
Median-of-N. A single Lighthouse run is noisy. Use runs_per_url: 3 or 5 and take the median. Standard deviation across runs is itself a useful signal — a URL with 3.2s mean and 1.8s stddev tells you something very different than a URL with 3.2s mean and 0.2s stddev.
Cache state. First-run (cold cache) vs. repeat-view are very different. Decide which matters for your use case. For SEO and new-visitor experience, cold cache is what matters. For logged-in SaaS dashboards, warm cache is what most users actually see.
Third-party scripts. Your LCP regression might be a vendor script update, not your code. Lighthouse's diagnostics.mainThreadWorkBreakdown and network waterfall can localize which third party caused the hit. Tag management systems (GTM, Segment) make this harder, not easier — they hide which vendor is responsible.
Device-class assumptions. "Mobile" throttling defaults in Lighthouse assume a low-to-mid tier Android device. If your audience skews iOS high-end (say, a fashion-focused DTC brand), the default throttling profile overstates your problem. If your audience skews emerging-market Android, the defaults understate.
CLS attribution. A page can have a "good" CLS score in lab but terrible CLS in field because user interactions (ads loading, infinite scroll) only fire in real sessions. Always cross-reference field CLS, especially for ad-supported content.
INP requires user interaction. Lighthouse's INP proxy (total blocking time, max potential FID) is only an approximation. Real INP comes from RUM (Real User Monitoring) tools like web-vitals.js, not lab runs.
Server warm-up on cold deploys. If you run Lighthouse right after a deploy, the server's caches are cold and TTFB is inflated. Either warm the server with a pre-scrape or skip the first few minutes after deploy.
Bot detection on Lighthouse runs. Some sites (especially those behind Cloudflare Super Bot Fight Mode) treat headless Chrome as a bot and serve it an interstitial. The interstitial's load time ends up in your data. Test a few URLs manually before trusting the output.
LCP element changes. When you make layout changes, the LCP element itself can change (hero image → hero text → hero video). LCP regressions that look like "the site got slower" may actually be "LCP is now measuring a different element." Always log the largestContentfulPaint.element selector so you can tell them apart.

How NexGenData handles this

The page-speed-analyzer actor is built specifically to close the gaps between free tools and $200/month commercial alternatives:

Parallel execution built in. Runs up to 32 Lighthouse instances concurrently on the Apify platform with automatic container scaling. A 2,000-URL audit completes in roughly 15 minutes, compared to 8+ hours sequential on a single worker.

Custom headers and cookies. Auth-gated staging environments, A/B test cookies, geo-spoofing headers — all supported natively. PageSpeed Insights has none of this.

Median-of-N built in. Pass runs_per_url: 3 (or more) and the actor handles dispatching, waiting, and median computation. You get a single median result per URL without running a statistics library locally.

Throttling presets. Ships with named profiles (simulated-3g, simulated-4g, desktop-cable, mobile-lte) so you don't have to hand-tune bandwidth/latency numbers.

Structured output for diff pipelines. Every row is a flat JSON record ready for Postgres, BigQuery, or DuckDB — no nested structures to unpack, no ad-hoc parsing.

Field-data hydration optional. Pass a CrUX API key and the actor enriches each result with field-data p75 LCP and INP, producing the lab-vs-field gap in a single dataset.

Pay-per-result pricing. A 10,000-URL daily audit runs ~$25/month, roughly 1/7th the cost of SpeedCurve's entry tier with no seat limits.

Conclusion

Page speed monitoring at scale has three hard pieces: parallel execution, clean storage, and useful regression alerts. Apify gives you the first, Postgres plus a few SQL queries gives you the rest. For the cost of a cheap lunch per month you get Lighthouse coverage across thousands of URLs with none of the container-management overhead.

Start with:

Page Speed Analyzer — Lighthouse at scale with custom headers.
Website Content Crawler — discover all URLs on a site if no sitemap exists.
Tech Stack Detector — correlate perf regressions with third-party script adoption.

FAQ

Is Lighthouse still the right tool in 2026?
Yes, for lab-based synthetic monitoring. For RUM (Real User Monitoring), complement Lighthouse with web-vitals.js or a tool like Datadog RUM, Splunk APM, or Cloudflare Web Analytics. Lab and field each answer different questions.

How does this compare to SpeedCurve or Calibre?
SpeedCurve and Calibre add pretty dashboards, annotation of deploys, and some RUM correlation for $125-200/month and up. If you want turnkey, they are great. If you are technical enough to wire Grafana to Postgres, you get 80% of the value for 1/10th the cost with Apify.

What about DebugBear?
DebugBear is closer to the Apify approach — API-first, scriptable, with Core Web Vitals tracking. It is a good product and fairly priced. If you want a fully-managed dashboard, DebugBear is a reasonable choice. If you want to own the data in your own warehouse and integrate with other pipelines, Apify wins.

How accurate are Lighthouse scores vs. real user data?
Lighthouse scores are a compact summary of many signals. The 0-100 score itself is a weighted roll-up; the individual metrics (LCP, CLS, INP) are what you should alert on. Lab scores consistently run "better" than field because lab runs use a clean cache and a dedicated CPU. Expect 20-40% gap on most properties.

Can I audit authenticated pages?
Yes, pass the session cookie in the headers parameter. For token-based auth that rotates, either run a pre-step that fetches a fresh token or use long-lived service-account tokens specifically for monitoring.

What should my alert thresholds be?
Avoid absolute thresholds. Use rolling baselines: alert when today's p50 LCP is more than 15% above the previous 7-day median. Absolute thresholds create alert fatigue because they fire on every modest regression and miss true issues that sneak in under the line.

Does this work for JavaScript-heavy SPAs?
Yes. Lighthouse renders the full page after JS execution. For SPAs with heavy client-side routing, also audit the individual routes (not just the shell), because the shell's LCP is often meaningless.

Can I audit non-public URLs (e.g., staging, behind VPN)?
Staging with basic auth or cookie auth: yes, pass auth via headers. Behind VPN with no internet route: no, the actor needs internet access to your staging. Work around it with a reverse tunnel or by running the actor from a machine with VPN access.

Related tools

H1B Salary Research Toolkit for Job Seekers (2026)

NexGenData — Thu, 20 Aug 2026 16:33:42 +0000

H1B Salary Research Toolkit for Job Seekers (2026)

If you are on (or considering) an H1B visa in 2026, the single highest-ROI hour you can spend is pulling your own compensation benchmark data. Employers know the prevailing-wage floor for your LCA. Recruiters know the tier bands at your company. The only person who usually does not know is you — which is exactly why you end up $25k under market.

Some context on the 2026 landscape: USCIS received roughly 442,000 H1B registrations for FY2026, with an 85,000-visa annual cap (65,000 regular + 20,000 US-masters). The cap hit in March 2025 with a 5.2x oversubscription, which means the tightness of the H1B market has real leverage effects on compensation — both good and bad. Employers who genuinely want specific candidates are willing to pay premium multiples above prevailing wage. Employers who rely on high-volume LCA filings (the classic "body shops") pay closer to the prevailing-wage floor. Knowing which side of that line your target employer falls on, before you walk into the negotiation, is worth anywhere from $15k to $80k on a senior-engineer offer. And unlike most career advice, the data for this is 100% public — the Department of Labor releases quarterly LCA disclosure datasets, every filing by every employer, searchable by job title and location. The problem is nobody outside specialty firms and immigration lawyers actually parses it.

This post builds a personal salary research toolkit using public H1B filing data plus general compensation scrapers. You will end up with a spreadsheet showing what your target role pays by company, location, and seniority — the same data recruiters use to anchor their first offer. The mental model to adopt: you are not asking "what should I make?" — you are answering "what has this company actually paid people with my profile, as proven by their own federal filings, in the last 24 months?" That shift in framing alone changes how the conversation goes.

Why this is hard

Public H1B data is technically free. The DOL publishes Labor Condition Applications quarterly. But getting to usable insight is rough:

Raw DOL data is 2-3 GB of CSV per quarter with inconsistent column names across years.
Employer names are messy. "Amazon.com Services LLC", "Amazon Web Services Inc.", and "Amazon Dev Center U.S., Inc." are all Amazon.
Prevailing wage != actual wage. The LCA lists the floor. Real TC (base + bonus + equity) is 20-60% higher.
Location adjustments. An SF offer at $180k base is different from a Seattle $180k base when equity and cost-of-living differ.

You need to join LCA data with actual compensation data (Levels.fyi, Glassdoor, Blind) and salary benchmark APIs.

SOC code ambiguity. The Standard Occupational Classification is granular, but many employers file under a generic "Software Developers, Applications" (15-1252) regardless of actual role. A data scientist and a backend engineer can appear identical in the LCA.
Wage unit confusion. Most filings are annual, but a minority are filed hourly or monthly. Group-by or aggregation without normalizing to annual throws off your bands by a factor of 10.

The architecture

[Target company + role]
          |
          v
 [h1b-visa-salary-search] --> LCA history, prevailing wages, counts
          |
          v
 [salary-data-search]     --> market compensation, bands, percentiles
          |
          v
     [Google Sheet]
          |
          v
 [Negotiation cheat sheet]

Step 1: Pull H1B filings for a target company

The h1b-visa-salary-search actor aggregates DOL LCA filings with fuzzy employer matching already done.

from apify_client import ApifyClient
client = ApifyClient("APIFY_TOKEN")

run = client.actor("nexgendata/h1b-visa-salary-search").call(run_input={
    "employers": ["Stripe", "Databricks", "Anthropic"],
    "job_titles": ["Software Engineer", "Senior Software Engineer",
                   "Staff Software Engineer", "Machine Learning Engineer"],
    "years": [2024, 2025, 2026],
    "locations": ["San Francisco, CA", "New York, NY", "Seattle, WA"],
})

filings = list(client.dataset(run["defaultDatasetId"]).iterate_items())

Each record:

{
  "employer": "Stripe, Inc.",
  "job_title": "Software Engineer II",
  "base_salary": 210000,
  "wage_unit": "Year",
  "location": "San Francisco, CA",
  "filing_date": "2025-03-14",
  "case_status": "Certified",
  "soc_code": "15-1252",
  "prevailing_wage": 186300
}

Note both base_salary (what the employer committed to pay) and prevailing_wage (the DOL floor for the role/location). The gap is informative — a tight gap means the employer is paying close to floor; a wide gap means they are comfortable paying well above.

Step 2: Build band distributions

import pandas as pd
df = pd.DataFrame(filings)
bands = (df.groupby(["employer","job_title","location"])
           .base_salary.quantile([0.25,0.5,0.75])
           .unstack())
bands.columns = ["p25","p50","p75"]
print(bands)

Output:

                                                         p25      p50      p75
employer  job_title              location
Stripe    Software Engineer II   San Francisco, CA    205000   215000   228000
Stripe    Senior Engineer        San Francisco, CA    245000   260000   280000

That p50 is your floor. Anything below is an instant-counter.

Step 3: Cross-check with market comp

H1B base salary does not include bonus or equity. For total comp, pull market data via salary-data-search:

comp_run = client.actor("nexgendata/salary-data-search").call(run_input={
    "roles": ["Senior Software Engineer"],
    "companies": ["Stripe", "Databricks", "Anthropic"],
    "locations": ["San Francisco, CA"],
    "include_bonus": True,
    "include_equity": True,
})

comp = list(client.dataset(comp_run["defaultDatasetId"]).iterate_items())

Returns ranges like:

{
  "company": "Stripe",
  "role": "Senior Software Engineer",
  "level": "L4",
  "location": "San Francisco, CA",
  "base_range": [210000, 260000],
  "bonus_range": [20000, 40000],
  "equity_4yr": [400000, 700000],
  "sample_size": 82
}

Step 4: Compare offer vs. band

Quick negotiation sanity check:

def evaluate_offer(offer_base, offer_bonus, offer_equity_4yr, market):
    m = market
    tc_offer = offer_base + offer_bonus + offer_equity_4yr/4
    tc_market_median = m["base_range"][0] + (m["base_range"][1]-m["base_range"][0])/2 \
                     + m["bonus_range"][0] + (m["bonus_range"][1]-m["bonus_range"][0])/2 \
                     + ((m["equity_4yr"][0]+m["equity_4yr"][1])/2)/4
    return round((tc_offer/tc_market_median - 1)*100, 1)

pct = evaluate_offer(225000, 25000, 400000, comp[0])
print(f"Offer is {pct}% of market median")

If the result is negative, you have concrete data to push back with.

Here is a more complete end-to-end script that produces a Google-Sheets-ready CSV combining LCA + market comp for a list of target companies. Designed to be copy-pasted before a round of final-round interviews:

import csv, statistics
from apify_client import ApifyClient
from collections import defaultdict

client = ApifyClient("APIFY_TOKEN")

TARGETS = [
    ("Stripe", ["Software Engineer", "Senior Software Engineer", "Staff Software Engineer"]),
    ("Databricks", ["Software Engineer", "Senior Software Engineer"]),
    ("Anthropic", ["Member of Technical Staff"]),
]
LOCS = ["San Francisco, CA", "New York, NY", "Seattle, WA"]

rows = []
for company, titles in TARGETS:
    lca = client.actor("nexgendata/h1b-visa-salary-search").call(run_input={
        "employers": [company], "job_titles": titles,
        "years": [2024, 2025, 2026], "locations": LOCS,
    })
    comp = client.actor("nexgendata/salary-data-search").call(run_input={
        "companies": [company], "roles": titles, "locations": LOCS,
        "include_bonus": True, "include_equity": True,
    })

    lca_by_key = defaultdict(list)
    for f in client.dataset(lca["defaultDatasetId"]).iterate_items():
        if f.get("case_status") != "Certified" or f.get("wage_unit") != "Year":
            continue
        key = (f["employer"], f["job_title"], f["location"])
        lca_by_key[key].append(f["base_salary"])

    comp_by_key = {(c["company"], c["role"], c["location"]): c
                   for c in client.dataset(comp["defaultDatasetId"]).iterate_items()}

    for (emp, title, loc), salaries in lca_by_key.items():
        if len(salaries) < 3:
            continue
        c = comp_by_key.get((company, title, loc), {})
        rows.append({
            "company": emp, "title": title, "location": loc, "n_filings": len(salaries),
            "lca_p25": int(statistics.quantiles(salaries, n=4)[0]),
            "lca_p50": int(statistics.median(salaries)),
            "lca_p75": int(statistics.quantiles(salaries, n=4)[2]),
            "market_base_low": c.get("base_range", [None, None])[0],
            "market_base_high": c.get("base_range", [None, None])[1],
            "market_tc_midpoint": (
                c["base_range"][1] + c.get("bonus_range", [0,0])[1]
                + c.get("equity_4yr", [0,0])[1] / 4
            ) if c else None,
        })

with open("negotiation_prep.csv", "w", newline="") as f:
    w = csv.DictWriter(f, fieldnames=list(rows[0].keys()))
    w.writeheader(); w.writerows(rows)
print(f"Wrote {len(rows)} rows")

Open that CSV in Google Sheets, conditional-format the columns, and you are walking into the negotiation with a spreadsheet that recruiters literally cannot argue with — it's their employer's own federal filings.

Step 5: Track employers with healthy sponsorship

For H1B visa holders, approval history matters as much as salary. The actor also returns per-employer counts:

-- Top sponsors in your field (proxy for: they actually file)
SELECT employer, COUNT(*) AS filings, AVG(base_salary) AS avg_pay
FROM filings
WHERE job_title ILIKE '%machine learning%'
  AND case_status = 'Certified'
  AND filing_date > '2024-01-01'
GROUP BY employer
ORDER BY filings DESC
LIMIT 30;

Use cases

1. Pre-offer prep. Before signing, a senior engineer pulled their potential employer's last 24 months of LCAs. Found peers filed at +15% of their offered base. Negotiated and got it.

2. Targeting companies that actually sponsor. A mid-career PM filters for employers who have filed 20+ successful LCAs in the past 2 years. Saves months of dead-end applications.

3. Location arbitrage. A senior dev compared NYC vs. Austin vs. Seattle LCAs for the same role at the same employer. Austin came out ahead on effective comp.

4. Salary review for green card cases. Pulling your employer's LCA filings for people at your level is a strong anchor for "am I being underpaid" conversations.

5. PERM/I-140 prevailing wage awareness. For green card sponsorship, the employer files a PERM labor certification that references the DOL prevailing wage. Knowing your target prevailing wage level (I, II, III, IV) before conversations with your immigration attorney can flag potential downgrading, where an employer files at a lower level than your actual responsibilities warrant.

6. Relocation decision support. A senior engineer weighing a cross-country transfer used the toolkit to quantify the effective comp delta. LCA data showed her company paid equivalent titles 18% less in the Raleigh office than in the NYC office. She renegotiated the transfer offer upward by 12% before accepting.

Pricing comparison

Service	Monthly cost	LCA history	Market comp	Sponsor counts
myvisajobs.com	Free (limited)	Yes	No	Yes
h1bdata.info	Free	Yes	No	Limited
Levels.fyi	Free + paid	No	Yes	No
Payscale	$29/month	No	Yes	No
Apify combo	~$5	Yes	Yes	Yes

You get the DOL data that myvisajobs paywalls plus Levels-grade comp data in one pipeline for <$5/month at typical job-search volume.

Common pitfalls

LCA data is rich but full of edge cases. Here are the ones that regularly mislead people using it for negotiation:

LCA != offer letter. Companies file LCAs for ranges or multiple candidates. Treat as floor, not ceiling. The filed wage is often the minimum the employer is committing to — the real offer may be 10-30% higher, especially at senior levels.
Case_status filter. Only "Certified" filings are valid; "Denied" and "Withdrawn" are noise. Some public tools aggregate all statuses and skew distributions. Always filter WHERE case_status = 'Certified' or 'Certified-Withdrawn' (the employer got the cert but withdrew it, usually because the candidate chose another offer — still a valid data point).
Location field is messy. "San Francisco, CA" and "SAN FRANCISCO, CALIFORNIA" both appear. Normalize before grouping. Also beware metro-level filings: some employers file with a metro MSA (e.g., "San Jose-Sunnyvale-Santa Clara, CA") that needs a crosswalk to the city you care about.
Remote work complicates location. Post-pandemic, some LCAs list the employer's HQ location while the actual work is remote. Cross-reference with the employer's job postings if the location seems off.
Amendment filings inflate counts. An employer amending an existing H1B (title change, location change, material job-duty change) files a new LCA. These show up as new filings but do not represent a new hire. For "is this employer actively hiring" analysis, watch for duplicate beneficiary fingerprints.
Level of experience confounds title. "Software Engineer" at Company A might mean "L3 new-grad" while at Company B it means "L5 staff-engineer equivalent." Always cross-reference with levels.fyi or Blind mappings before concluding a company pays more/less.
The 2026 H1B wage floor rule changes. In late 2025, DOL proposed raising prevailing wage levels across the board. As of this writing the rule is partially implemented. Filings from Q1 2026 onward may reflect higher floors. Do not compare 2026 filings directly to 2023 without accounting for this.
OPT STEM salaries aren't in LCA data. If you are on F1 OPT, your salary is not in this data. LCAs only cover H1B/H1B1/E-3 workers. For OPT negotiation, rely on levels.fyi and Blind instead.
Prevailing wage levels I-IV. Level I is "entry"; Level IV is "fully competent." The level determines the minimum. An employer filing a senior engineer at Level II is legally questionable — that is a downgrading pattern flag.
Secondary filings (H1B transfers) look like new hires. They are not. A transfer from Company A to Company B generates a new LCA at Company B. For hiring-volume metrics, filter by visa_class = H1B new-employment vs. continuing-employment.

How NexGenData handles this

The h1b-visa-salary-search actor is built to solve these problems rather than expose them:

Pre-normalized employer names. We run every raw DOL employer string through an entity-resolution pipeline that collapses variations. "AMAZON.COM SERVICES LLC", "Amazon Web Services, Inc.", and "Amazon Dev Center U.S., Inc." resolve to a canonical Amazon entity with a stable ID.

Wage-unit normalization. Hourly and monthly filings are converted to annualized equivalents with a standard assumption (2080 hours/year for hourly, 12 months for monthly) and flagged with a wage_source field so you can filter if you prefer.

Location normalization. All location strings are mapped to canonical MSA + state. You can group by MSA without messy string matching.

Status filtering by default. Denied and Withdrawn filings are excluded from default output. A flag lets you include them for specific analyses (e.g., "what employers have a high denial rate").

Fresh data weekly. We pull the quarterly DOL release and interpolate with weekly OFLC datasets. The data is rarely more than 5-7 days stale.

Pay-per-result pricing. A single company's 24-month history is typically under $1. Compare with myvisajobs.com's $99/year subscription — the actor is cheaper at any volume under 100 company queries per year.

Conclusion

The information asymmetry in hiring is entirely on the employer's side by default. A three-hour afternoon with these two actors flips it. You walk into the negotiation with percentiles, not vibes.

Start the toolkit with:

H1B Visa Salary Search — LCA filings and sponsorship history.
Salary Data Search — market compensation with bonus + equity.
GitHub Repo Stats — evaluate the engineering org you are joining.

FAQ

Is this legal to use for negotiation?
Absolutely. LCA data is public information the employer files with the federal government. You are allowed to cite it in negotiation. Most recruiters will not be surprised — some will be impressed you did the work.

What if my target company doesn't sponsor H1B?
Then LCA data will not help you, but salary-data-search still will. Use the market-comp side of the pipeline on its own.

Does H1B salary data reflect total compensation?
No. LCA only covers base salary. Bonus and equity are not disclosed. That is why you must cross-reference with market-comp data for total compensation.

How do I identify my "level" at a target company?
Use levels.fyi's role mapping. Most major tech companies have well-documented leveling guides on levels.fyi, Blind, or Team Blind's anonymous comp sheets. Match your current scope of work and years of experience against their level descriptions, then filter LCAs for the corresponding title.

Can I use this data in a salary dispute with my current employer?
Yes, and it's a useful tool. If your current employer has filed LCAs for peers at your level at a higher wage than you make, that is a reasonable conversation to have in a comp review — though the cultural norms vary by company.

What about non-tech H1B roles?
The actor works for any SOC code. Filings are not tech-specific. Pharmaceutical scientists, financial analysts, civil engineers — all in the data. Adjust your SOC code filter accordingly.

How recent is the data?
Quarterly releases are about 45-60 days behind. Weekly OFLC updates are 7-14 days behind. For negotiation purposes, data within 3 months is considered current.

Is there an equivalent for L1, O1, or TN visas?
LCA data is specific to H1B/H1B1/E-3. L1 visa wages are not published. For O1 and TN, there is no equivalent public filing. These visa types require different research strategies (Glassdoor, levels.fyi, peer networks).

Related tools

How to Build a Sentiment Dashboard from Reddit + Hacker News (2026)

NexGenData — Wed, 19 Aug 2026 21:02:53 +0000

How to Build a Sentiment Dashboard from Reddit + Hacker News (2026)

Developer sentiment is the leading indicator every dev-tools founder wishes they had. Twitter/X is noise, LinkedIn is performance art, but Hacker News and Reddit's technical subs are where people say what they actually think about your product, your stack, or the launch you fumbled last week. The problem: aggregating that signal manually is a soul-crushing job.

Some 2026 context on why this matters more than it did three years ago. Hacker News publishes roughly 900-1,200 stories per day with average thread lengths north of 60 comments. Reddit crossed 100M daily active users in 2024 and shows ~3 million comments per day across the 2,000+ technical subreddits most relevant to a dev-tools company. A developer-survey by JetBrains (2025, n=26,348) found that 71% of respondents named Hacker News and Reddit as their "first three sources" when evaluating new infrastructure or tooling. That is a bigger information-gathering role than product pages, vendor documentation, or even YouTube. If you ship a developer product and you are not listening to both, you are letting competitors shape the narrative unchallenged. The mental model most founders carry — "I'll just check HN when I remember" — produces exactly the right amount of information to confirm whatever mood you are in when you check.

In this post we build a sentiment dashboard that monitors Hacker News plus a configurable set of subreddits, runs every comment through an AI sentiment analyzer, and surfaces the themes trending up or down. Runs entirely on Apify plus a Postgres database plus a Grafana dashboard. The end state: a single pane of glass that shows, at any moment, "what is the internet saying about my product and my category?" with drill-down by topic, time, and source. You can replace three $300/month social-listening subscriptions with a stack that costs about $25/month in compute and owns its own data.

Why this is hard

Reddit API restrictions post-2023. Reddit's pricing changes killed most free Reddit scrapers. You need a route that handles throttling and auth renewal.
Hacker News has no native sentiment scores. Upvotes are not sentiment. A 300-point story with 400 comments can be 90% "this is garbage."
Comment threading. A comment at depth 4 disagreeing with its parent reads differently than one at depth 1. Flat sentiment averages lie.
AI sentiment quality. Off-the-shelf sentiment models (VADER, TextBlob) fail on sarcasm, tech jargon, and mixed polarity. You need an LLM-grade analyzer.
Topic extraction drift. "Postgres" as a topic at the string level is easy. "Postgres 17's new logical replication bug that affects read replicas but only under logical decoding with streaming enabled" is what you actually need to cluster on. Naive keyword extraction gets you buried in overlapping topics.
Temporal drift in baselines. What sentiment looks like on HN in December is not what it looks like in June (fewer comments, different moods, more end-of-year layoffs chatter). Alerts need seasonally-adjusted baselines, not hardcoded thresholds.

The architecture

[hacker-news-scraper]  --> stories + comments
         +
[reddit scraper / keyword watcher] (external)
         |
         v
   [Raw comments JSONL]
         |
         v
[ai-sentiment-analyzer] --> per-comment sentiment + topic
         |
         v
     [Postgres]
         |
         v
    [Grafana dashboard]

Step 1: Pull Hacker News

The hacker-news-scraper pulls stories and their comment trees with engagement metadata.

from apify_client import ApifyClient
client = ApifyClient("APIFY_TOKEN")

run = client.actor("nexgendata/hacker-news-scraper").call(run_input={
    "feeds": ["top", "new", "best"],
    "keywords": ["postgres", "rust", "openai", "apify", "anthropic"],
    "max_stories": 100,
    "include_comments": True,
    "max_comments_per_story": 200,
})

stories = list(client.dataset(run["defaultDatasetId"]).iterate_items())

Each story contains nested comment objects. Flatten:

def flatten(story):
    out = []
    def walk(c, depth, parent_id, story_id):
        out.append({
            "source": "hn",
            "story_id": story_id,
            "comment_id": c["id"],
            "parent_id": parent_id,
            "depth": depth,
            "text": c["text"],
            "author": c["by"],
            "timestamp": c["time"],
        })
        for child in c.get("children", []):
            walk(child, depth+1, c["id"], story_id)
    for c in story.get("comments", []):
        walk(c, 0, story["id"], story["id"])
    return out

comments = [row for s in stories for row in flatten(s)]

Step 2: Run sentiment analysis

The ai-sentiment-analyzer handles sarcasm and mixed polarity better than classic sentiment libraries. It returns polarity (-1 to +1), subjectivity, and a one-line topic summary.

sent_run = client.actor("nexgendata/ai-sentiment-analyzer").call(run_input={
    "texts": [{"id": c["comment_id"], "text": c["text"]} for c in comments],
    "include_topic": True,
    "language": "en",
})
sentiments = {s["id"]: s for s in client.dataset(sent_run["defaultDatasetId"]).iterate_items()}

Result per comment:

{
  "id": "39847112",
  "polarity": -0.62,
  "subjectivity": 0.8,
  "label": "negative",
  "topic": "postgres 17 performance regressions",
  "confidence": 0.91
}

Step 3: Persist to Postgres

import psycopg
with psycopg.connect("postgresql://...") as con:
    with con.cursor() as cur:
        for c in comments:
            s = sentiments.get(c["comment_id"], {})
            cur.execute("""
                INSERT INTO comments
                (source, story_id, comment_id, text, polarity, label, topic, ts)
                VALUES (%s,%s,%s,%s,%s,%s,%s,to_timestamp(%s))
                ON CONFLICT (comment_id) DO UPDATE
                SET polarity=EXCLUDED.polarity, label=EXCLUDED.label
            """, (c["source"], c["story_id"], c["comment_id"], c["text"],
                  s.get("polarity"), s.get("label"), s.get("topic"), c["timestamp"]))

Step 4: Build the dashboard

Grafana queries against Postgres:

Top negative topics (last 24h):

SELECT topic, COUNT(*) AS negatives, AVG(polarity) AS avg_polarity
FROM comments
WHERE label = 'negative' AND ts > now() - INTERVAL '24 hours'
GROUP BY topic
ORDER BY negatives DESC
LIMIT 20;

Sentiment over time for "apify":

SELECT date_trunc('hour', ts) AS hour, AVG(polarity) AS sentiment
FROM comments
WHERE text ILIKE '%apify%'
GROUP BY 1 ORDER BY 1;

Pin both panels to a dashboard and you have an early-warning system. When sentiment dips 2 standard deviations below baseline, trigger a PagerDuty alert. (Yes, for real.)

Here is a more sophisticated rolling-z-score alert that actually works in production — it uses a 14-day trailing baseline per keyword, so seasonal noise doesn't trigger pages:

import psycopg, os, json, urllib.request

PAGERDUTY_KEY = os.environ["PAGERDUTY_INTEGRATION_KEY"]
KEYWORDS = ["apify", "nexgendata", "postgres 17"]

def page(keyword, z, current, baseline):
    event = {
        "routing_key": PAGERDUTY_KEY,
        "event_action": "trigger",
        "payload": {
            "summary": f"Sentiment anomaly on '{keyword}' (z={z:.2f})",
            "severity": "warning",
            "source": "sentiment-dashboard",
            "custom_details": {
                "keyword": keyword,
                "current_polarity": current,
                "14d_baseline": baseline,
            },
        },
    }
    req = urllib.request.Request(
        "https://events.pagerduty.com/v2/enqueue",
        data=json.dumps(event).encode(),
        headers={"Content-Type": "application/json"},
    )
    urllib.request.urlopen(req)

with psycopg.connect(os.environ["DATABASE_URL"]) as con, con.cursor() as cur:
    for kw in KEYWORDS:
        cur.execute("""
            WITH recent AS (
              SELECT AVG(polarity) AS mean FROM comments
              WHERE text ILIKE %s AND ts > now() - INTERVAL '24 hours'
            ),
            baseline AS (
              SELECT AVG(polarity) AS mean, STDDEV(polarity) AS sd FROM comments
              WHERE text ILIKE %s AND ts > now() - INTERVAL '14 days'
                AND ts < now() - INTERVAL '24 hours'
            )
            SELECT recent.mean, baseline.mean, baseline.sd FROM recent, baseline
        """, (f"%{kw}%", f"%{kw}%"))
        cur_mean, base_mean, base_sd = cur.fetchone()
        if None in (cur_mean, base_mean, base_sd) or base_sd == 0:
            continue
        z = (cur_mean - base_mean) / base_sd
        if z < -2:
            page(kw, z, cur_mean, base_mean)
            print(f"ALERT: {kw} z={z:.2f}")

Run this on a 2-hour schedule from GitHub Actions or a cron container. The 24h / 14d baseline is a reasonable default, but if your keyword's chatter is sparse, widen the window to 72h / 30d.

Step 5: Extend to Reddit

Reddit's API is locked down post-2023, but two paths work:

Use Reddit's official Data API with an OAuth client (free for <100 req/min).
Use Apify's Reddit actor (not ours — this is a commodity one on the store) to scrape public subreddits.

For the sentiment pipeline, only the comment text and timestamp matter. Feed the output through the same ai-sentiment-analyzer call and write to the comments table with source = 'reddit'.

Use cases

1. Dev-tool founder listening post. A database company monitors five competitor names plus their own. Weekly, the dashboard surfaces the top five complaints per product — direct feature-roadmap input.

2. Release post-mortem. A framework team launched a v2 and saw sentiment crater. Grafana showed 70% of negativity clustered around a breaking change nobody mentioned in the blog post. Addressed in a follow-up patch.

3. Hiring pulse. A recruiter monitors "remote work" and "layoffs" threads, detecting sentiment swings that correlate with candidate availability windows.

4. Crypto/macro signal. A trader runs the pipeline over /r/cryptocurrency and WSB to front-run retail sentiment shifts.

5. Community manager priority queue. A DevRel team for an open-source framework uses the dashboard to route themselves toward the threads most worth engaging with. Instead of reading all 40 daily mentions, they look at the 5 with polarity < -0.4 and confidence > 0.8. Response time to legitimate criticism dropped from 3 days to 6 hours, and community sentiment metrics recovered measurably over the following quarter.

6. PR crisis early warning. A consumer SaaS company was quietly getting clobbered on /r/SideProject over a pricing change. The sentiment dashboard flagged the spike 9 hours before it hit their Twitter mentions. They rolled back, posted a clear statement, and contained what would have been a 48-hour news cycle into a morning.

Pricing comparison

Service	Monthly cost (100k comments)	Reddit?	HN?	AI analysis?
Brand24	$149	Limited	No	Basic
Brandwatch	$1000+	Yes	Yes	Yes
Mention	$41+	Limited	Limited	Basic
Custom (Reddit API + OpenAI)	$60 + eng	Yes	Yes	Yes
Apify combo	~$25	Via add-on	Yes	Yes

Common pitfalls

Sentiment pipelines look simple and are not. These are the traps most teams fall into:

Comment depth matters. Weight top-level comments more than replies in rollups — replies pile on and amplify whatever the top comment said, creating an echo-chamber effect in your averages. A common heuristic: weight by 1/(depth+1) or only aggregate top-level + depth-1 replies.
Sarcasm on HN is peak. Manually spot-check 20 "negative" comments weekly; if polarity calibration drifts, tune the analyzer prompt. The canonical failure: "oh great, another JavaScript framework" gets tagged positive by naive models because "great" is a strong positive signal.
Deletion handling. HN comments get deleted. Re-run your fetcher on a 48-hour lag to capture final state. Reddit deletions are even messier — a deleted user's entire history may vanish, creating gaps in your time series. Store a first_seen_ts and treat comments as immutable once captured.
Shills and sock puppets. Any subreddit above 100k subs has them. A rash of positive comments from 12-hour-old accounts is a signal, not sentiment. Track account age and karma alongside the comment, and flag suspicious clusters.
Brigades and cross-posts. A thread linked from Twitter/X or another subreddit can flood with off-topic noise. Watch for sudden comment-volume spikes that are out of proportion to upvote ratio — those are usually brigades, not organic conversation.
Jargon and acronyms. Tech subreddits have massive in-group vocabulary. "Skill issue" is a pejorative, "NGMI" is negative, "chad" is positive. A general-purpose sentiment model will misread all three. Either fine-tune or prompt-engineer for your domain.
Language mixing. Subs like /r/india and /r/brasil have heavy code-mixing (English + Hindi/Portuguese). Your analyzer should either handle multilingual or you need language detection + filtering upstream.
Quote extraction. A comment that quotes another ("> the GC pauses are unacceptable") may be sarcastic agreement or genuine criticism. Naive analyzers treat the quote as the comment's own text. Strip or tag quoted lines.
Sentiment vs. engagement. High-engagement threads skew negative (controversy drives comments). If you are tracking rollups, normalize sentiment by expected baseline for that story's score, not the flat average.
Timezone biases. HN comments peak in US business hours; a sentiment average skewed by time-of-day tells you more about who is online than what they think. Aggregate by rolling 24h window, not calendar day.

How NexGenData handles this

The hacker-news-scraper and ai-sentiment-analyzer were designed to work together, which drives some specific choices:

Full comment tree with depth metadata. Every comment in the output includes its depth, parent ID, and the path from root. You can weight, filter, or flatten however your analysis requires — no restructuring needed.

Incremental fetching. Pass a since timestamp and the scraper returns only new comments since your last run. You are not paying to re-fetch the same 10,000 comments every hour.

Deletion-aware re-runs. A follow-up mode: "refresh" re-examines comments you already have and updates their status if they have been deleted or edited. Keeps your dataset honest.

LLM-grade sentiment with topic extraction. The ai-sentiment-analyzer uses a prompted LLM rather than VADER/TextBlob, which handles sarcasm, tech jargon, and mixed polarity far better. It also returns a topic string, not just polarity, so clustering by theme is immediate.

Batch cost efficiency. The analyzer batches up to 100 texts per API call and uses a small, fast model (the equivalent of GPT-4o-mini class), so per-comment cost is fractions of a cent.

Pay-per-result pricing for both actors. 10,000 comments scraped and sentiment-analyzed costs roughly $3-5 all-in. A comparable Brandwatch subscription that covers the same volume is $1,000+/month with multi-seat minimums.

Conclusion

A sentiment dashboard is one of the highest-leverage projects a solo founder can build. It takes half a day, costs <$30/month, and replaces a surprising amount of anecdotal "what is the vibe" anxiety with actual data.

Build it with:

Hacker News Scraper — stories, comments, engagement history.
AI Sentiment Analyzer — LLM-grade polarity + topic extraction.
AP News Scraper — layer in mainstream news signal alongside developer chatter.

FAQ

Is scraping Hacker News allowed?
Yes, within reason. HN has an official Firebase-backed API at hn.algolia.com and hacker-news.firebaseio.com that anyone can use for free. The Apify actor uses these endpoints and adds structure, pagination, and dedup. HN does not require an API key. Be polite — don't hammer at 100 req/sec.

Can I still scrape Reddit in 2026?
Yes, but carefully. Reddit's Data API is free for <100 req/min with OAuth; higher volumes are paid. For public subreddits, the official API is the safest route. Third-party scrapers exist but carry more risk as Reddit tightens enforcement. For this use case (sentiment over a few focused subs), the Data API is plenty.

Which LLM should I use for sentiment?
For this use case, a small fast model (GPT-4o-mini, Claude Haiku, or Llama 3.1 8B hosted on Groq or Cerebras) is more than enough. The ai-sentiment-analyzer actor uses a prompt-engineered equivalent. The task is not "write Shakespeare," it's "classify this 200-character text reliably." Cheap models win on cost per comment without a meaningful accuracy loss.

How do I handle ambiguous polarity?
Always store the confidence score alongside polarity. In dashboards, filter to confidence > 0.7 for rollups. Comments with low confidence are often genuinely ambiguous (sarcasm, mixed signals) and should go in a "needs human review" bucket, not a statistic.

What about non-English posts?
The ai-sentiment-analyzer supports most major languages out of the box. For subreddits with heavy code-mixing, detect language per-comment and process only those above a confidence threshold.

Can I share this dashboard with my team?
Grafana supports multi-user dashboards with role-based permissions. A common pattern: give the CEO read-only access to a "Sentiment overview" board, give product managers access to the per-topic drilldowns, and keep the alert-config board locked to the ops team.

How accurate is LLM sentiment really?
On developer-heavy text, LLM sentiment with a domain-tuned prompt runs 85-92% agreement with human labels on a hand-coded test set. VADER and TextBlob on the same corpus score 58-65%. The gap is enormous for sarcasm and technical jargon, which is where developer chatter lives.

Related tools

Scraping Google Scholar for Literature Reviews (Python + Apify, 2026)

NexGenData — Thu, 13 Aug 2026 22:35:10 +0000

Scraping Google Scholar for Literature Reviews (Python + Apify, 2026)

Every grad student hits the wall around week three of a lit review: you have 400 open tabs, 60% of which are duplicates, and Zotero crashed again. The manual workflow — search, skim, cite, repeat — was already broken before LLMs 10x'd paper output. In 2026, with ~5 million new papers indexed by Scholar annually, anything you are not automating is actively costing you sanity.

To put some real numbers on the problem: a 2025 STM report estimated 3.3 million peer-reviewed articles were published that year across 47,000+ journals, a 7.2% year-over-year increase. ArXiv alone received 23,400 new submissions in January 2026, up from about 15,000 monthly in 2022. The National Institutes of Health PubMed Central added 1.7 million records in 2025. Even in a narrow subfield — say "retrieval augmented generation for clinical NLP" — a naive Scholar query returns 800+ results now where five years ago it would have returned 40. The bottleneck in research has quietly shifted from discovery to filtering. The researcher's mental model can no longer be "I read every relevant paper"; it has to be "I have a system that surfaces the 20 papers most worth my attention this week." That system is what this post builds.

This post walks through an Apify-based literature-review pipeline: automatic paper collection from Google Scholar and ArXiv, citation graph construction, duplicate dedup, and PRISMA-style export for systematic reviews. Code included, including a minimal LLM-based triage step for the cases where you want to narrow a 1,000-paper harvest to the 50 worth reading.

Why this is hard

Google Scholar is notorious among scrapers for good reason:

Aggressive bot detection. Scholar's anti-scraping rivals Cloudflare enterprise. A direct Python requests loop gets blocked after 5-15 queries.
No public API. Unlike PubMed, Scholar has no official API. Semantic Scholar is a partial alternative but lacks some long-tail papers.
Ambiguous deduplication. The same paper shows up three ways: a preprint on ArXiv, a conference version, a journal version. Collapsing them correctly is non-trivial.
Citation counts drift. Citation counts update daily, so pipelines need to be idempotent and store snapshots.

On top of all this, research integrity standards (PRISMA 2020, for example) require reproducible search queries and documented exclusion counts. Your scraper has to log its query history.

Preprint vs. published attribution. The same paper may be indexed three times (ArXiv preprint, conference proceedings, journal publication) with different DOIs and different citation counts. Scholar sometimes collapses these with a "[PDF] from..." link and sometimes doesn't.
Cross-language coverage. Scholar does index non-English papers, but its coverage of Chinese, Japanese, and Portuguese literature is uneven. For systematic reviews in medicine or engineering, excluding those corpora is a real bias source.

The architecture

[Query strings + filters]
          |
          v
 [google-scholar-scraper] --> papers with citations, DOIs, authors
          |
          v
 [arxiv-scraper]          --> preprint full text + metadata
          |
          v
 [dedup + embedding cluster]
          |
          v
 [Citation graph (Neo4j or simple DuckDB)]
          |
          v
 [PRISMA flowchart + CSV export]

Step 1: Search Scholar

The google-scholar-scraper handles Scholar's bot detection and returns structured papers with citation counts and author h-index data.

from apify_client import ApifyClient

client = ApifyClient("APIFY_TOKEN")

run = client.actor("nexgendata/google-scholar-scraper").call(run_input={
    "queries": [
        "large language model hallucination detection",
        "retrieval augmented generation evaluation"
    ],
    "year_from": 2023,
    "year_to": 2026,
    "max_results_per_query": 500,
    "include_patents": False,
    "include_citations": True,
})

papers = list(client.dataset(run["defaultDatasetId"]).iterate_items())

Each paper:

{
  "title": "Evaluating Hallucinations in LLMs: A Meta-Analysis",
  "authors": ["Smith, J.", "Chen, L."],
  "venue": "NeurIPS 2024",
  "year": 2024,
  "citation_count": 142,
  "doi": "10.48550/arXiv.2404.12345",
  "abstract": "...",
  "pdf_url": "https://arxiv.org/pdf/2404.12345",
  "scholar_id": "xYz123"
}

Step 2: Pull ArXiv full text

Many Scholar results are behind paywalls. If a paper has a preprint on ArXiv, use the arxiv-scraper to fetch the full-text PDF link and structured abstract.

arxiv_ids = [extract_arxiv_id(p["doi"]) for p in papers if "arxiv" in p.get("doi","").lower()]

run = client.actor("nexgendata/arxiv-scraper").call(run_input={
    "arxiv_ids": arxiv_ids,
    "include_pdf_url": True,
    "include_latex_source": False,
})
arxiv_papers = list(client.dataset(run["defaultDatasetId"]).iterate_items())

Step 3: Deduplicate across sources

Scholar and ArXiv overlap heavily. Dedup on DOI first, fall back to title similarity:

from rapidfuzz import fuzz

def is_dupe(a, b):
    if a.get("doi") and a["doi"] == b.get("doi"):
        return True
    return fuzz.token_set_ratio(a["title"], b["title"]) > 92

seen, unique = [], []
for p in papers + arxiv_papers:
    if not any(is_dupe(p, s) for s in seen):
        seen.append(p); unique.append(p)

print(f"{len(papers)+len(arxiv_papers)} raw -> {len(unique)} unique")

Step 4: Build a citation graph

For snowball sampling — finding papers that cite (or are cited by) a seed set — run forward and backward from high-citation anchor papers. The scraper's include_citations flag returns Scholar's "Cited by" links.

Load into DuckDB:

import duckdb
con = duckdb.connect("lit_review.duckdb")
con.execute("CREATE TABLE papers (scholar_id VARCHAR, title VARCHAR, year INT, citations INT)")
con.executemany(
    "INSERT INTO papers VALUES (?, ?, ?, ?)",
    [(p["scholar_id"], p["title"], p["year"], p["citation_count"]) for p in unique]
)

Then ask questions:

-- Most-cited papers in the corpus
SELECT title, year, citations FROM papers ORDER BY citations DESC LIMIT 20;

-- Papers on the topic from the last 12 months with >50 citations (fast movers)
SELECT title, authors FROM papers WHERE year = 2025 AND citations > 50;

Step 5: Export a PRISMA-compliant CSV

Systematic reviews require documented inclusion/exclusion counts. Export:

import csv
with open("prisma.csv", "w") as f:
    w = csv.writer(f)
    w.writerow(["id","title","year","authors","venue","doi","included","exclusion_reason"])
    for p in unique:
        w.writerow([p["scholar_id"], p["title"], p["year"], "; ".join(p["authors"]),
                    p.get("venue",""), p.get("doi",""), "", ""])

Now fill in inclusion/exclusion manually (or via an LLM screening pass) and you have a PRISMA-ready audit trail.

Step 6: LLM triage (optional but transformative)

If you have harvested 800 papers and realistically only need 50, an LLM screening pass can save a week of skimming. The pattern: feed each paper's title+abstract to a small model with strict inclusion criteria, get a yes/no/maybe, and have a human review only the maybes. This is the same pattern used in Cochrane-style reviews.

import os, json
from openai import OpenAI
oai = OpenAI(api_key=os.environ["OPENAI_API_KEY"])

CRITERIA = """
Include a paper if ALL are true:
- Published 2023 or later
- Involves retrieval augmented generation (RAG) as primary method
- Reports quantitative evaluation on a named benchmark
- Is a peer-reviewed paper or widely-cited preprint (>20 citations)
Exclude if it is a survey, workshop abstract, or tutorial.
"""

def screen(paper):
    prompt = f"""{CRITERIA}

Title: {paper['title']}
Abstract: {paper.get('abstract','')[:1500]}

Respond with JSON: {{"decision": "include" | "exclude" | "maybe", "reason": "..."}}"""
    resp = oai.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": prompt}],
        response_format={"type": "json_object"},
    )
    return json.loads(resp.choices[0].message.content)

screened = []
for p in unique:
    try:
        s = screen(p)
        screened.append({**p, **s})
    except Exception as e:
        screened.append({**p, "decision": "maybe", "reason": f"screening error: {e}"})

included = [p for p in screened if p["decision"] == "include"]
maybes = [p for p in screened if p["decision"] == "maybe"]
print(f"Included: {len(included)}, Needs human: {len(maybes)}, Excluded: {len(screened)-len(included)-len(maybes)}")

A 2025 study in the Journal of Medical Internet Research showed LLM-assisted screening with human review of maybes achieved 96% concordance with a fully-manual Cochrane review while reducing time-to-screen by 73%. The key is the "maybe" category — you do not trust the model to exclude confidently, only to pre-filter and flag uncertainty.

Use cases

1. PhD lit review. A computational biology student runs Scholar + ArXiv nightly for 30 queries. Wakes up to 200 new papers/week already deduped and ranked by citation velocity.

2. Patent landscape analysis. A biotech analyst uses the include_patents=True flag to pull both papers and patents for CRISPR+gene therapy, surfacing IP overlaps.

3. Grant writing. A PI needs "key references" for a NIH R01. Automates top-20 papers per aim, by recency, in 10 minutes.

4. Research trend forecasting. A VC analyst counts yearly papers per topic to identify emerging subfields 12-18 months before they trend.

5. Competitor-lab monitoring. A pharma R&D team tracks publication output from 12 competing research groups. Weekly Scholar queries by author surface new papers, and an LLM summarization pass compresses them into a one-page Monday briefing. The team reports catching a competitor's methodology pivot roughly 4 months before it appeared in any industry press.

6. Teaching updated syllabi. A university professor auto-refreshes reading lists for their graduate seminar each semester. The pipeline pulls the top 30 most-cited papers matching course topics from the past 18 months. Students get a current reading list instead of one last updated in 2019.

Pricing comparison

Service	Cost for 10k papers	Full text?	Citation graph?
Scopus API	Institutional only	Partial	Yes
Web of Science	$300-1000/user/yr	Partial	Yes
Semantic Scholar API	Free (rate-limited)	Abstract	Yes
scholarly (Python)	Free (blocked fast)	No	Limited
Apify actors	~$15	Yes (ArXiv)	Yes

For anyone without an academic institutional login, Apify is the most economical path to structured Scholar data.

Common pitfalls

These are the failure modes that turn a promising pipeline into a week of debugging:

Rate-limit yourself. Even with the actor's proxy rotation, set max_results_per_query sensibly. 500 is fine; 5000 will be slow and approaches Scholar's deep-pagination ceiling, where results past page 100 become unreliable.
Citation counts are snapshots. Re-run weekly if you care about trajectory. A paper with 5 citations in January might have 200 by June if it catches on — that velocity is often the signal you actually want, not the absolute count.
Author disambiguation. Scholar merges authors with the same name. "J. Smith" could be 8 different researchers across Scholar's index. Join on author+affiliation+ORCID when possible. ORCID is the gold standard for disambiguation and is present in about 45% of post-2019 papers.
Scholar's "include citations" toggle. The checkbox that includes citations and patents in search results can wildly inflate counts. Decide early whether you want scholarly articles only or everything.
Preprint vs. final version citation counts. ArXiv preprint citations and journal-version citations are often tracked separately. A paper might show 50 citations as a preprint and 200 as the NeurIPS version. Semantic Scholar merges these; Google Scholar sometimes does not.
Retracted papers. Scholar does not always flag retractions. Cross-reference with Retraction Watch's database if you are doing medical or biomedical reviews. A retracted paper cited as supporting evidence is worse than not finding the paper at all.
Year drift in metadata. A paper might have year: 2024 in Scholar and year: 2025 in CrossRef because one uses "submitted" and the other uses "published." When you dedup across sources, trust the DOI-resolved year from CrossRef over Scholar's heuristic year.
PDF URL freshness. Scholar caches PDF links. Many point to author homepages that 404 a year later. If you need archival access, fetch the PDF at scrape-time and store it in your own bucket, not rely on the URL later.
Scholar's "sciting" vs. "citing". Citations can be positive or negative. For meta-analysis, consider using scite.ai's signal layer on top of your Scholar data to weight supporting vs. contradicting citations.

How NexGenData handles this

Scraping Scholar reliably is a full-time job for someone. We built the google-scholar-scraper actor to absorb that work:

Anti-bot hardening. We rotate residential proxies by geography, use realistic browser fingerprints, and handle Scholar's rotating captcha challenges automatically. You will get substantially more queries through without blocks than a raw Python script.

Structured author + citation output. The actor returns authors as separate objects with h-index where available, not as a single flat string. Citation counts, cited-by links, and related-papers links are all preserved.

DOI extraction and normalization. Scholar embeds DOIs inconsistently — sometimes as links, sometimes in snippets, sometimes not at all. The actor extracts and normalizes them so dedup-by-DOI works cleanly.

Snapshotting support. Every run can be tagged with a query fingerprint, so reproducibility for PRISMA audits is built-in. You can replay any past run's exact input.

Native pairing with the ArXiv actor. The two actors share DOI conventions, so merging their outputs is a straightforward DuckDB join. No impedance mismatch.

Pay-per-result. 10,000 papers costs roughly $15. Compare that to Scopus institutional access at $20k+/year, and it is the clear choice for independent researchers and teams without a university affiliation.

Conclusion

Literature review in 2026 is an information-retrieval problem, not a reading problem. Automate the collection, deduplication, and ranking steps, then spend your actual brain cycles on the hard part: reading the 30 papers that matter.

Three actors to start:

Google Scholar Scraper — bot-detection-proof Scholar access.
ArXiv Scraper — full preprint metadata and PDF links.
Wikipedia Scraper — for background context on unfamiliar fields.

FAQ

Is scraping Google Scholar allowed?
Scholar's ToS prohibit automated access, and Google does actively rate-limit. That said, academic use of search data has been treated leniently in practice, and the aggregation of public bibliometric data is broadly considered fair use. If you publish research based on scraped data, cite Scholar as your source. For true systematic reviews with publication requirements, supplement Scholar with official databases (PubMed, Scopus) that have APIs and documented ToS.

How does this compare to Semantic Scholar's API?
Semantic Scholar has a clean API and free tier (100 req/5min). Its coverage is strong for CS, medicine, and physics but weaker for humanities and some regional journals. For thorough systematic reviews, use both: Scholar for coverage breadth, Semantic Scholar for citation graphs and clean metadata.

What about OpenAlex?
OpenAlex is an open bibliometric database covering 240M+ works with a free, unlimited API. It is excellent for citation graphs and author profiles, often better than Scholar for those specific tasks. Scholar still wins for "find me the most-read recent paper on X," because its ranking is informed by actual reader behavior.

Can I get full-text PDFs reliably?
For ArXiv papers, yes. For anything else, it depends on open-access status. The pipeline can fetch PDFs where URLs are public, but paywalled papers need either your institutional login, an interlibrary loan, or (for some reviews) contacting authors directly for a copy.

How do I cite papers I find this way in my own writing?
Export the deduped set to BibTeX via pandoc-citeproc or the bibtexparser Python library. The doi field in the output is what most reference managers key on.

What about non-English literature?
The actor passes through whatever Scholar returns, which includes non-English papers if your query matches them. For comprehensive non-English coverage, pair with regional databases (CNKI for Chinese, CiNii for Japanese, SciELO for Latin American journals).

How do I handle the "forward snowball" (papers that cite my seed papers)?
Use the include_citations=True flag and the "cited by" URL in the Scholar response. Run a second pass with those URLs as seeds. Two iterations of forward snowballing typically saturates a well-defined subfield.

Related tools

Bulk Domain WHOIS Lookup — API Alternatives to Whois.com (2026)

NexGenData — Tue, 11 Aug 2026 21:34:44 +0000

Bulk Domain WHOIS Lookup — API Alternatives to Whois.com (2026)

If you have ever tried to look up WHOIS info for more than 50 domains in a row, you know the drill: whois.com throws a captcha, the whois CLI stops resolving halfway through, and GoDaddy's WHOIS widget silently starts returning empty responses after some undocumented throttle. Running proper bulk WHOIS in 2026 is harder than it should be — thanks in part to GDPR redactions, thick vs. thin registrar models, and a fragmented TLD landscape.

A quick 2026 snapshot to set the stage: there are roughly 359 million registered domains globally (Verisign Domain Name Industry Brief, Q4 2025), spread across 1,500+ TLDs, with .com still dominant at ~157M and newer TLDs like .ai, .xyz, .io, and .dev growing 20-40% year-over-year. The WHOIS ecosystem used to be a single TCP-port-43 protocol with reasonably uniform output, but ICANN's 2024 mandate requiring all registries to support RDAP (Registration Data Access Protocol) means you are now living in a hybrid world: some registries serve RDAP JSON, some still serve legacy WHOIS text, and a surprising number serve both with subtly different field names. GDPR redactions, in effect since May 2018, hide registrant personal data by default for EU-registered domains and in practice for most registrars globally. The mental model of "WHOIS is a phonebook for domains" is outdated. The accurate model is "WHOIS is a fragmented, rate-limited, partially-redacted dataset that requires an aggregation layer to use at scale."

This post compares the main options for bulk WHOIS in 2026, shows how to build a production pipeline in Python, and covers the use cases that actually justify the effort: domain monitoring, brand protection, cybersecurity triage, and M&A diligence. If you are doing any of those at more than handful-of-domains scale, you need an automated pipeline, not a browser tab.

Why this is hard

WHOIS looks simple — one TCP connection to port 43, one record back. Reality:

Every TLD has its own WHOIS server and response format. .com uses Verisign, .io uses NIC.IO, country-code TLDs each run their own registry. Parsing is a mess.
GDPR redactions hide most registrant data. Since 2018, EU-registered domains no longer expose owner names or contact emails. Some registrars redact all domains, not just EU ones.
Rate limits vary wildly. Verisign thin WHOIS tolerates ~20 queries/sec, but query the same .com registrar's thick WHOIS and you will see 5 queries/sec before blocks.
RDAP is slowly replacing WHOIS. Some registries return only RDAP JSON now. Your parser needs both code paths.

Rolling this yourself means maintaining per-TLD parsers, proxy pools, and RDAP fallbacks. Most teams burn a week on it and then the parser breaks again when .ai changes its format.

Intermittent TLDs. Some ccTLDs (.ke, .pk, .sa) are only available via web-form WHOIS with no programmatic endpoint. Others require paid registry access.
Stale cache semantics. Registry responses include a varying TTL. If you query a domain right after a transfer, you may get old registrar data for hours. Knowing when to retry vs. accept staleness is a heuristic, not a rule.

The architecture

[Domain list (CSV, 10k rows)]
           |
           v
    [Apify WHOIS actor] -- rotating proxies, RDAP fallback, TLD-aware parsing
           |
           v
    [Structured JSON dataset]
           |
           v
 [DuckDB / Postgres] -- dedupe + expiry alerting
           |
           v
     [Slack / email] -- "10 domains expire in 30 days"

The core building block is the domain-whois-lookup actor, which handles TLD routing, RDAP fallback, rate-limit evasion, and parsing into a normalized schema.

Step 1: One-shot bulk lookup

from apify_client import ApifyClient

client = ApifyClient("APIFY_TOKEN")

domains = [l.strip() for l in open("domains.csv") if l.strip()]

run = client.actor("nexgendata/domain-whois-lookup").call(run_input={
    "domains": domains,
    "include_raw": False,
    "rdap_fallback": True,
})

results = list(client.dataset(run["defaultDatasetId"]).iterate_items())
print(f"Processed {len(results)} domains")

A typical result looks like:

{
  "domain": "example.com",
  "registrar": "MarkMonitor Inc.",
  "registrar_iana_id": 292,
  "creation_date": "1995-08-14T04:00:00Z",
  "expiry_date": "2028-08-13T04:00:00Z",
  "last_updated": "2025-08-14T07:01:23Z",
  "nameservers": ["a.iana-servers.net", "b.iana-servers.net"],
  "dnssec": "signedDelegation",
  "status": ["clientDeleteProhibited", "clientTransferProhibited"],
  "registrant_country": "US",
  "redacted": true
}

Notice the redacted: true flag. You still get registrar, nameservers, and dates — which is 80% of what most use cases need.

Step 2: Watch for expiries

Once results land in your dataset, a 10-line SQL query surfaces domains expiring soon:

SELECT domain, registrar, expiry_date,
       date_diff('day', current_date, expiry_date) AS days_left
FROM whois_results
WHERE expiry_date < current_date + INTERVAL 60 DAY
ORDER BY expiry_date ASC;

Wire this into a scheduled Apify run (daily), then a cron job that posts the result to Slack:

import requests, json
slack = "https://hooks.slack.com/services/..."
rows = [r for r in results if days_until_expiry(r) < 60]
if rows:
    text = "Domains expiring soon:\n" + "\n".join(
        f"{r['domain']} -> {r['expiry_date']}" for r in rows)
    requests.post(slack, json={"text": text})

Step 3: Join with DNS

WHOIS alone is often not enough. Pair it with DNS records and you get a much richer picture: who owns it, where it resolves, and whether it is parked.

The dns-propagation-checker actor resolves A, AAAA, MX, and TXT across 15+ resolvers. Combined with WHOIS you can answer:

Which competitor domains changed nameservers this week?
Which newly-registered typosquats already have MX records (meaning they are phishing-ready)?
Which domains in our portfolio are pointed at the wrong CDN?

Here is a reasonably complete typosquat-detection snippet that fans out permutations of a brand, runs WHOIS on each, and flags any that were registered in the last 30 days. This is the bones of a real brand-protection pipeline:

from apify_client import ApifyClient
from datetime import datetime, timezone, timedelta
import itertools

BRAND = "acmecorp"
TLDS = [".com", ".net", ".org", ".co", ".io", ".ai", ".app", ".dev"]

def permutations(brand):
    # character-swap, leet-speak, and homoglyph basics
    swaps = {"a": ["4", "@"], "e": ["3"], "o": ["0"], "i": ["1", "l"]}
    out = {brand, brand + "s", brand + "hq", brand + "-app", "get" + brand}
    for i, ch in enumerate(brand):
        for alt in swaps.get(ch, []):
            out.add(brand[:i] + alt + brand[i+1:])
    return out

candidates = [b + t for b, t in itertools.product(permutations(BRAND), TLDS)]

client = ApifyClient("APIFY_TOKEN")
run = client.actor("nexgendata/domain-whois-lookup").call(run_input={
    "domains": candidates,
    "rdap_fallback": True,
})

cutoff = datetime.now(timezone.utc) - timedelta(days=30)
new_squats = []
for r in client.dataset(run["defaultDatasetId"]).iterate_items():
    if not r.get("creation_date"):
        continue
    created = datetime.fromisoformat(r["creation_date"].replace("Z", "+00:00"))
    if created > cutoff:
        new_squats.append(r)

print(f"Found {len(new_squats)} newly-registered typosquat candidates")
for r in new_squats:
    print(f"- {r['domain']} registered {r['creation_date']} via {r['registrar']}")

A similar pipeline in Node.js for teams more comfortable in JavaScript:

import { ApifyClient } from 'apify-client';
const client = new ApifyClient({ token: process.env.APIFY_TOKEN });

const domains = (await Deno.readTextFile('portfolio.csv')).split('\n').filter(Boolean);
const run = await client.actor('nexgendata/domain-whois-lookup').call({
  domains,
  rdap_fallback: true,
});

const ALERT_DAYS = 45;
const now = Date.now();
const expiring = [];
for await (const item of client.dataset(run.defaultDatasetId).iterate()) {
  if (!item.expiry_date) continue;
  const days = (new Date(item.expiry_date).getTime() - now) / 86400000;
  if (days < ALERT_DAYS) expiring.push({ ...item, days: Math.round(days) });
}
expiring.sort((a, b) => a.days - b.days);
console.table(expiring.map(({ domain, registrar, days }) => ({ domain, registrar, days })));

Use cases

1. Brand protection / typosquatting monitoring. A SaaS company monitors 300 domain permutations of their brand. Daily WHOIS runs flag newly-registered lookalikes within 24 hours, triggering legal takedowns.

2. M&A tech diligence. An acquirer runs WHOIS on every domain owned by the target. Finds three domains registered to an ex-founder's personal email — caught before close.

3. Cybersecurity triage. An incident responder receives 500 suspicious URLs. Bulk WHOIS reveals 240 were registered in the past 14 days at the same registrar — strong phishing-campaign signal.

4. Affiliate / SEO portfolio management. An affiliate marketer owns 120 domains. Monthly WHOIS batches feed a spreadsheet of expiries, saving at least one "oh no I forgot to renew" event per year.

5. Drop-catching research. A domain investor monitors expiring high-value domains by running WHOIS against a curated 10k watchlist weekly. When a domain moves into the 30-day redemption period, a Slack alert routes it to a review queue. In 2025 the same investor caught a 4-letter .com that would have gone to auction for low five figures.

6. Phishing kit attribution. A security vendor scrapes newly observed phishing URLs from public feeds (URLHaus, OpenPhish), runs WHOIS on the domains, and clusters by registrar + nameserver + creation date. Patterns emerge quickly: 60% of a week's phishing campaigns might trace back to 3 registrars. That kind of clustering is not possible without a WHOIS pipeline.

Pricing comparison

Service	10k lookups cost	Historical?	RDAP?	Bulk CSV?
WhoisXML API	~$100	Paid add-on	Yes	Yes
DomainTools Iris	$395+/mo	Yes	Yes	Yes
JsonWhoisAPI	~$80	No	No	Limited
whois.com manual	Free (captcha)	No	No	No
NexGenData actor	~$15	No	Yes	Yes

For the vast majority of use cases — bulk triage, expiry monitoring, diligence — you do not need $400/mo DomainTools. Pay-per-result at $0.0015/domain is comfortably within hobby-budget range.

Common pitfalls

WHOIS is one of those protocols where every shortcut has a hidden cost. These are the ones that catch people:

.ai and .io change formats. Expect occasional parsing misses on exotic ccTLDs. The actor updates parsers monthly, but if you are parsing raw responses yourself, budget for format drift every quarter.
Registrar WHOIS vs. registry WHOIS. Thick TLDs (.org, .info) return full data from the registry. Thin TLDs (.com) require a second hop to the registrar. The actor handles this transparently, but raw WHOIS CLIs do not. If you are using the whois CLI and seeing minimal data for .com, you are probably reading the Verisign thin response and missing the registrar follow-up.
Queries are cached. Most registries serve cached data. Same-second repeated queries will not reflect updates in real time. For change detection, poll every 6-24 hours, not every minute.
Premium WHOIS is its own thing. Some registrars charge extra for "WHOIS history" — time-series data of how WHOIS records changed. If you need that (common for cyber threat intel), only WhoisXML or DomainTools offer it at scale, though you can build a historical dataset yourself by snapshotting daily.
Punycode and IDN domains. A domain like bücher.de shows up as xn--bcher-kva.de in WHOIS. If your input list has UTF-8 brand names, convert to punycode before lookup, or the query will fail silently. Python has idna.encode() for this.
Glue records. WHOIS returns nameserver hostnames, but not their IPs. If you are tracking infrastructure migration, you also want the nameserver's A/AAAA records, which requires a follow-up DNS query.
Contact email redactions. GDPR redacts the registrant email, but sometimes the registrar exposes a forwarding address like contact@domainsbyproxy.com. These are useless for direct contact but reveal the proxy service, which is an attribution signal in itself.
Registrar name inconsistencies. "GoDaddy", "GoDaddy.com, LLC", and "GoDaddy Online Services Cayman Islands Ltd." are all the same company. Normalize on IANA ID (the registrar_iana_id field), not the string name, if you are clustering.
Expiry vs. grace period vs. redemption. A domain past its expiry_date is not immediately available. It enters a 30-day grace, then a 30-day redemption, then a 5-day pending-delete window. Drop-catching pipelines need to model all four phases correctly.

How NexGenData handles this

The domain-whois-lookup actor was built after we watched too many teams reinvent the same broken parser. Specific design choices:

Dual WHOIS + RDAP by default. Every lookup tries RDAP first (faster, structured JSON, consistent schema) and falls back to legacy WHOIS if RDAP is unavailable. You get the best data the registry will give, without having to code two pipelines.

1,500+ TLD parsers maintained. We monitor TLD response-format changes and push parser updates on a rolling basis. If .ai changes its format next month, you do not need to do anything.

Automatic thick-TLD follow-through. For .com/.net, we follow Verisign's referral to the registrar and merge both responses into a single row. No double-query logic on your side.

IANA-ID based registrar normalization. The output always includes both the raw string and the canonical IANA ID, so clustering and reporting work out of the box.

Proxy rotation and polite rate-limiting. We fingerprint each TLD's tolerance and adjust concurrency automatically. You can pass 10k domains at once and the actor will schedule them appropriately.

No API key, no seat fees, pay per result. $0.0015 per lookup. 10,000 domains costs $15. There is no monthly commitment.

Conclusion

Bulk WHOIS is a classic example of "seems trivial, actually has 15 edge cases." Rather than maintain per-TLD parsers yourself, point Apify at your domain list, schedule it, and wire the dataset into your alerting pipeline. You get coverage across 1,500+ TLDs, RDAP fallback, and normalized JSON out of the box.

Start here:

Domain WHOIS Lookup — core WHOIS + RDAP with pay-per-result pricing.
DNS Propagation Checker — resolve across 15 global DNS servers.
Tech Stack Detector — fingerprint the websites behind those domains.

FAQ

Is bulk WHOIS legal?
Yes. WHOIS is a public protocol mandated by ICANN for most generic TLDs. You cannot use WHOIS data for spam or direct marketing to registrants (that violates most registrar AUPs and GDPR). You can use it for security research, portfolio management, diligence, and trademark enforcement.

Does GDPR prevent me from getting any useful data?
No. GDPR redacts personal registrant info (name, email, address for individuals), but registrar, nameservers, creation date, expiry, and status codes are still public. For most operational use cases (monitoring, expiry, security triage) that is plenty.

What if I need the redacted data — say, for a trademark claim?
Use the registrar's abuse or legal contact channel with a formal request. Most registrars will disclose registrant info to a court order, UDRP filing, or legitimate IP complaint. The WHOIS API cannot magic it through.

RDAP vs. WHOIS — which should I use?
RDAP is the future and should be your default. It returns structured JSON, supports HTTPS, includes authenticated access paths for LEA, and is what ICANN is pushing. Legacy WHOIS should be a fallback. The actor handles both automatically.

How often should I re-lookup a domain?
For expiry monitoring: weekly is plenty. For change detection (nameserver flips, registrar transfers): daily. For active incident response: on-demand is fine. There is rarely a reason to poll more than once per 24h per domain.

Can I bulk-register domains via this pipeline?
No. WHOIS is read-only. Registration happens at a registrar (GoDaddy, Namecheap, Porkbun, Google Domains) via their API. You can use the WHOIS pipeline to find available domains and then register them separately.

What about new gTLDs like .xyz, .app, .dev?
All supported. The newer gTLDs almost all serve RDAP, which actually makes them easier to handle than some legacy ccTLDs.

Related tools

Building a Real-Time FX Dashboard with Apify + Google Sheets (2026)

NexGenData — Thu, 06 Aug 2026 20:53:01 +0000

Building a Real-Time FX Dashboard with Apify + Google Sheets (2026)

Every international SaaS founder eventually hits the same wall: "what is our real MRR in USD this week?" You multiply your EUR, GBP, CAD, and INR invoices by whatever Stripe happened to use on the settlement date, and the number drifts 3-5% every month depending on EUR/USD. Multiply that by a currency basket of payouts, vendor invoices, and contractor pay, and a proper FX dashboard stops being optional.

In 2026, FX volatility is still a meaningful P&L factor — the DXY (US Dollar Index) moved a cumulative 14% over the trailing 12 months, and emerging-market pairs like USD/TRY and USD/ARS have shifted by 40%+ in single quarters. A BIS Triennial Survey (April 2025 preliminary data) pegged daily global FX turnover at $8.3 trillion, with the retail and SMB slice growing roughly 11% year-over-year as more small businesses operate cross-border through platforms like Stripe, Wise, and Payoneer. The mental model for a small international business used to be "we will true up once a quarter with the accountant." That model is now a quiet tax of 2-4% of gross revenue for any business with meaningful non-home-currency flow. You want a live dashboard because it surfaces the moments where hedging, timing a wire, or repricing in a market actually moves the needle.

Commercial FX APIs (Fixer, OpenExchangeRates, CurrencyLayer) charge $50-300/month for real-time rates once you exceed the hobbyist tier. More expensive options like XE or Bloomberg's FX datafeed run $800-$2400/month and assume you have a Bloomberg Terminal already. In this guide we build the same thing for the cost of a coffee: an Apify-powered pipeline that writes directly into a Google Sheet your CFO, accountant, or dashboard can read. The end state is a spreadsheet that refreshes every 15 minutes during market hours, writes historical snapshots to a second tab for trend analysis, and fires a conditional-formatted cell red when a pair moves more than 1% intraday. You will spend about an hour setting it up and effectively zero minutes maintaining it.

Why this is hard

Free FX data has three traps:

Staleness. exchangerate.host and similar free endpoints cache aggressively, so rates lag market by 30+ minutes.
Midpoint vs. market. ECB midpoints are not what your bank actually pays. For realistic P&L you want the rate Yahoo Finance, Google Finance, or XE show — which are retail-skewed.
Rate limits on free tiers. 1,000 requests/month sounds like a lot until you poll 30 currency pairs every 15 minutes (that is 86k requests/month).

We will dodge all three by combining two Apify actors with Google Sheets' IMPORTDATA pattern plus a scheduled refresh.

Ticker-symbol mismatches across providers. Fixer uses ISO-4217 codes only. Yahoo Finance uses a XXXYYY=X convention. TradingView uses FX:EURUSD. If you ever need to cross-reference, plan for a symbol-mapping table from day one, not day 30.
Forward rates and spreads. Spot rates are what every free API serves. If you actually need forward rates (common for treasury hedging or SaaS with annual contracts), free sources are useless. We flag this below so you know when to escalate to a paid feed.

The architecture

[currency-exchange-rates actor] --> FX midpoint rates (150+ ccys)
[yahoo-finance-scraper actor]   --> cross-check retail rates + FX-correlated tickers
          |
          v
    Apify dataset (JSON)
          |
          v
   Google Apps Script pulls via Apify API
          |
          v
   Google Sheet (live dashboard)

Step 1: Pull real-time rates

The currency-exchange-rates actor fetches rates for 150+ currencies without an API key. Pay-per-result means you only pay for what you poll.

from apify_client import ApifyClient

client = ApifyClient("APIFY_TOKEN")

run = client.actor("nexgendata/currency-exchange-rates").call(run_input={
    "base_currencies": ["USD", "EUR", "GBP", "CAD"],
    "target_currencies": ["USD", "EUR", "GBP", "CAD", "JPY", "INR",
                          "AUD", "CHF", "SGD", "BRL", "MXN", "ZAR"],
    "include_historical": False,
})

rates = list(client.dataset(run["defaultDatasetId"]).iterate_items())

Output:

{
  "base": "USD",
  "target": "EUR",
  "rate": 0.9184,
  "timestamp": "2026-04-17T13:05:11Z",
  "source": "ecb_midpoint"
}

Step 2: Cross-check with Yahoo Finance

Midpoint rates are fine for accounting. For trading signals or vendor-comparison we want whatever real humans see. The yahoo-finance-scraper pulls live FX tickers like EURUSD=X, GBPUSD=X, and USDJPY=X.

fx_run = client.actor("nexgendata/yahoo-finance-scraper").call(run_input={
    "symbols": ["EURUSD=X", "GBPUSD=X", "USDJPY=X",
                "USDCAD=X", "AUDUSD=X", "USDSGD=X"],
    "include_history": True,
    "history_range": "5d",
})

Yahoo also includes bid, ask, and day high/low — useful for setting FX alerts.

Here is an expanded Python example that combines both actors, computes the spread between midpoint and retail, and flags rates that deviate more than 0.4% (a reasonable bank markup threshold):

from apify_client import ApifyClient
from datetime import datetime

client = ApifyClient("APIFY_TOKEN")

# 1. midpoints
mid_run = client.actor("nexgendata/currency-exchange-rates").call(run_input={
    "base_currencies": ["USD"],
    "target_currencies": ["EUR", "GBP", "JPY", "CAD", "AUD", "CHF", "INR"],
})
midpoints = {f"{i['base']}{i['target']}": i['rate']
             for i in client.dataset(mid_run["defaultDatasetId"]).iterate_items()}

# 2. retail tickers
retail_run = client.actor("nexgendata/yahoo-finance-scraper").call(run_input={
    "symbols": ["EURUSD=X", "GBPUSD=X", "USDJPY=X", "USDCAD=X",
                "AUDUSD=X", "USDCHF=X", "USDINR=X"],
})
retail = {}
for r in client.dataset(retail_run["defaultDatasetId"]).iterate_items():
    pair = r["symbol"].replace("=X", "")
    retail[pair] = r["regular_market_price"]

# 3. compare
print(f"{'Pair':<8}{'Mid':>10}{'Retail':>10}{'Spread':>10}")
for pair, mid in midpoints.items():
    # invert if needed since Yahoo uses different conventions
    r = retail.get(pair) or (1 / retail[pair[3:] + pair[:3]] if pair[3:] + pair[:3] in retail else None)
    if r is None:
        continue
    spread_pct = (r - mid) / mid * 100
    flag = " <--" if abs(spread_pct) > 0.4 else ""
    print(f"{pair:<8}{mid:>10.4f}{r:>10.4f}{spread_pct:>9.2f}%{flag}")

This is exactly the kind of script you would put on a 4-hour cron that DMs you whenever a bank-sized spread appears on a pair you are about to settle. It is one of those little scripts that pays for itself the first time it saves you from wiring at a bad rate.

Step 3: Write to Google Sheets

Two viable patterns:

A. Google Apps Script pull (my preferred approach).

Add this as an Apps Script attached to the sheet:

function refreshFX() {
  const token = 'APIFY_TOKEN';
  const actor = 'nexgendata~currency-exchange-rates';
  const url = `https://api.apify.com/v2/acts/${actor}/run-sync-get-dataset-items?token=${token}`;
  const payload = {
    base_currencies: ['USD'],
    target_currencies: ['EUR','GBP','CAD','JPY','INR','AUD','CHF']
  };
  const res = UrlFetchApp.fetch(url, {
    method: 'post',
    contentType: 'application/json',
    payload: JSON.stringify(payload)
  });
  const rows = JSON.parse(res.getContentText())
    .map(r => [r.target, r.rate, r.timestamp]);

  const sheet = SpreadsheetApp.getActive().getSheetByName('FX');
  sheet.getRange(2, 1, 100, 3).clearContent();
  sheet.getRange(2, 1, rows.length, 3).setValues(rows);
}

Attach it to a time-driven trigger: every 15 minutes during business hours.

B. Apify Google Sheets integration. The Apify platform has a first-class integration that pushes dataset output directly to a sheet on run completion. No code — just toggle it on in the actor's integrations tab.

Step 4: Build the dashboard

In the sheet, compute derived metrics:

Cell	Formula	Meaning
`D2`	`=VLOOKUP("EUR", FX!A:B, 2, FALSE)`	Current USD/EUR
`E2`	`=(D2 - D_yesterday)/D_yesterday`	24h change
`F2`	`=SPARKLINE(FX_HISTORY!B2:B30)`	30-day trend
`G2`	`=revenue_EUR * D2`	Revenue in USD

Conditional-format the 24h change cells red/green. You now have a one-glance view of FX exposure.

Use cases

1. SaaS MRR in reporting currency. An indie founder bills in 6 currencies. A nightly Apify run writes rates into the sheet, and a formula converts the Stripe export to USD MRR automatically.

2. Crypto + FX correlation dashboard. A trader pulls EUR/USD alongside BTC-USD via Yahoo Finance and eyeballs correlation breakouts.

3. Contractor payment timing. An agency pays 30 contractors monthly in their local currency. The dashboard recommends which week to initiate wires based on 30-day rate trends, saving 1-2% per cycle.

4. International e-commerce pricing. A Shopify store localizes prices quarterly. The FX dashboard feeds a pricing-review meeting — when EUR/USD moves more than 3%, prices are rebalanced.

5. Freelancer invoicing in volatile currencies. A software consultancy that bills in EUR and TRY uses the dashboard to decide when to invoice. When USD/TRY opens more than 2% above its 5-day moving average, invoices go out that morning. When below, they hold until end-of-week. Over 2025 the effect was a measured 4.1% revenue uplift versus invoicing on a fixed day-of-month schedule.

6. Corporate travel treasury. A 40-person startup with a US HQ and remote teams in Mexico and Portugal uses the dashboard to time its bi-monthly payroll-funding transfers. The CFO committed to a simple rule: fund ahead only when spot is within 0.75% of the 30-day low. Annual FX friction on $1.2M of non-USD payroll dropped from an implicit 2.3% to 1.1%.

Pricing comparison

Service	Monthly cost (30 pairs, 15-min refresh)	API key?	Historical?
Fixer Professional	$49.99	Yes	Limited
OpenExchangeRates	$97	Yes	Yes
CurrencyLayer Plus	$79.99	Yes	Yes
XE Rates API	$799	Yes	Yes
Apify actors	~$4-8 pay-per-result	No	Yes

For bedroom-grade usage up through moderate business volume, pay-per-result is dramatically cheaper.

Common pitfalls

These are the sharp edges that will bite you the first week:

Weekend pricing. FX markets close Friday 5pm ET and reopen Sunday 5pm ET. Weekend "rates" are the Friday close — do not display them as live without a clear "market closed" badge, or your CFO will think they are seeing the latest price and plan a wire against a stale number.
Apps Script quotas. Free Google accounts cap UrlFetchApp at 20k calls/day, plenty for FX but plan accordingly. Google Workspace business accounts are capped at 100k/day. If you refresh every 5 minutes and hit 3 actors per refresh, you are still under 30k/day for a single sheet.
Timestamps. Always store the source timestamp, not new Date(). If the scrape took 4 minutes, you want the actual market time. A subtle variant of this bug: storing only the Unix epoch with no timezone. Sheets formats epoch times in the sheet's timezone by default, which is fine until you share the dashboard with someone in another region.
Holiday calendars. Major pairs close on different days. Thanksgiving is a half-day for USD pairs but normal for GBPJPY. Golden Week compresses JPY liquidity. If you are building alerts, consider a holiday-calendar lookup or your "price unchanged for 24 hours" alert will scream every major holiday.
Stablecoin confusion. Some feeds include USDC, USDT, and BUSD alongside fiat. These are close to USD but not identical. During stress events (Terra/Luna 2022, SVB weekend 2023) they can deviate 3-8% for hours. If your pipeline treats them as USD proxies, you will miss real risk signals.
Rate directionality. Yahoo quotes some pairs inverted versus how you think about them (USDJPY vs JPYUSD). Always verify the direction and normalize to a canonical "1 unit of BASE = x units of TARGET" form before storing.
Sheet cell limit. Google Sheets has a 10 million cell cap per workbook. If you log every 15 minutes across 30 pairs for a year, you are at ~1 million cells. Fine, but design your schema so you are not stuck migrating later.
Cached IMPORTDATA. IMPORTDATA and IMPORTXML are aggressively cached by Google — up to 1 hour. If you need sub-hourly freshness, use Apps Script UrlFetchApp instead, which respects HTTP cache headers you control.

How NexGenData handles this

The two actors powering this pipeline are purpose-built for "pull into a spreadsheet" workflows, which drives a few specific design decisions:

No API keys, ever. The currency-exchange-rates actor aggregates from multiple public sources (ECB, national central banks, and a handful of bank-rate scrapers) and never requires you to register for anything. Start time from "I want to try this" to "first row in my sheet" is under 90 seconds.

Timestamp-per-rate, not per-batch. Every row in the output includes the timestamp from the source it came from. If ECB updated 30 seconds ago and Bank of Canada updated 3 hours ago, you see that asymmetry directly. Most commercial APIs smooth this over and pretend everything is "current."

Built-in symbol normalization. The actor accepts both ISO-4217 codes (USD, EUR) and Yahoo-style tickers (EURUSD=X) and normalizes them internally. No adapter code needed.

Apify's native Sheets integration. Skip the Apps Script entirely and use Apify's built-in Google Sheets integration for the most common "append to a sheet on each run" pattern. Zero code if your needs are simple.

Pay-per-result economics. A full 150-currency pull costs roughly $0.004. You can run this every 5 minutes during market hours for under $10/month. That compares to Fixer's $50 Professional tier.

Conclusion

A solid FX dashboard is a 90-minute project with Apify + Google Sheets. You avoid another SaaS subscription, you own the data, and the marginal cost scales with your actual polling — not a flat $50/month tier.

Spin up these three actors to go further:

Currency Exchange Rates — 150+ currencies, no API key, pay per result.
Yahoo Finance Scraper — FX tickers, stock prices, historical data.
Finviz Stock Screener — layer in equity signals correlated to FX.

FAQ

How fresh are the rates, really?
The ECB midpoint source updates once per business day at approximately 16:00 CET. Yahoo Finance FX tickers update every 1-2 minutes during market hours. If you need tick-level data (sub-second), you need a paid broker feed, not any of this.

Is this compliant for accounting purposes?
For internal reporting and decision-making, yes. For statutory filings your accountant should use a documented reference rate (often the central bank's end-of-period rate). Most accountants are fine with ECB or Bank of Canada rates, which the actor emits with source attribution.

Can I use this for forex trading signals?
You can build signals off of it, but understand the latency floor. For anything faster than 5-minute swing trading, use a proper broker API (OANDA, Interactive Brokers, FXCM). This pipeline is for reporting, treasury, and business decisions, not HFT.

What about exotic currencies like KZT, UGX, or XOF?
The actor covers 150+ currencies including most African and Central Asian currencies. Exotic pairs tend to have wider mid-retail spreads (the bank markup is real money), so always cross-check against a local source before relying on them for large transactions.

Why not just use Google Finance's GOOGLEFINANCE() function?
You can for basic rates, but GOOGLEFINANCE does not give you historical snapshots in a clean format, no bid/ask, no central-bank source attribution, and no programmatic access from other tools. For a single pair on one sheet it is fine — for a real dashboard the Apify approach wins.

How do I handle multi-currency invoices automatically?
Use a sheet with an INVOICES tab pulling from Stripe via CSV export or their API, and a formula that looks up the rate from the FX tab by the invoice's issue date. For audit trail, store both the rate used and its source timestamp alongside the invoice row.

Related tools

How to Monitor Competitor Shopify Stores at Scale (2026)

NexGenData — Tue, 04 Aug 2026 19:11:43 +0000

How to Monitor Competitor Shopify Stores at Scale (2026)

If you sell on Shopify, your competitors are a goldmine of data. Every new product launch, sold-out SKU, and price test is a free A/B experiment you can learn from — but only if you are collecting that data continuously. By the time you notice "hey, they raised prices 12%" on a casual browse, the window to react is already closing.

As of Q1 2026, Shopify powers roughly 4.8 million live stores globally, and the top 100k of them push a new product SKU every 3.2 days on average. For DTC operators in crowded verticals like apparel, supplements, and home goods, the rate of competitive change has genuinely outpaced what one person can observe through manual browsing. A 2025 DTC industry survey from Commerce Signals reported that merchants who tracked competitor pricing on a weekly or better cadence grew gross margin 2.7 points faster than those who tracked quarterly or ad-hoc. The mental model most growth operators carry — "I check competitor sites when I remember to" — is simply the wrong abstraction for 2026. It needs to be a system, not a habit.

This guide walks through how to build a Shopify competitor intelligence system that runs without you. We will cover store discovery, product scraping, price-change alerts, theme detection, and how to stitch it all together into a single dashboard. All examples use the Apify platform because it handles proxies, scheduling, and storage out of the box, so you are not babysitting Puppeteer containers on Hetzner at 2am. The end state looks like this: you wake up, open Slack, and see a digest that says "Gymshark launched 4 new SKUs overnight, Allbirds dropped prices on 3 runners by 15%, and Fashion Nova sold out of their top 12 bestsellers." You spent zero minutes acquiring that insight. That is the bar we are building to.

One more thing worth naming before we dive in: this is not just a tactical exercise. The teams that build continuous competitive telemetry end up with a compounding data asset. After 12 months, you have a longitudinal dataset on how competitors price, launch, and sunset products — the kind of thing consulting firms charge $40k for as a one-off market study. You are building that asset as a byproduct.

Why this is hard

On the surface, Shopify stores look easy to scrape. Every store exposes JSON endpoints like /products.json and /collections/all/products.json. In practice, three things make scaling painful:

Store discovery. Shopify does not publish a directory. Finding competitors in your niche means crawling tech-stack signals, domain lists, or Google search results.
Rate limits and bot protection. Popular stores increasingly sit behind Cloudflare, Shop Protect, or custom rate limits. A naive Python loop gets 429s within minutes.
Schema drift. Storefronts vary by theme. Variants, metafields, and sale badges live in different selectors depending on whether the merchant uses Dawn, Impulse, Prestige, or a custom theme.

You also have the cold-start problem: even if you have a list of 500 competitors, pulling all their products once is easy. Diffing them daily — efficiently — is the real engineering problem. A naive approach stores every snapshot as a full row per product per day, and after 90 days of tracking 500 stores with an average of 800 SKUs each, you are staring at 36 million rows in a Postgres table that was never indexed for this access pattern. Queries slow, cost climbs, and the "simple side project" becomes a data engineering problem. The tooling has to be opinionated about what a "change event" is or the storage costs eat the project.

JavaScript-rendered variant pickers. A growing minority of Shopify stores (especially those on Hydrogen or custom headless setups) render variant selection client-side. The classic /products.json endpoint still works for the product list, but inventory and some metafields require a full headless browser pass.
Checkout-gated pricing. A handful of B2B or wholesale stores hide real prices behind a login wall, exposing only MSRP publicly. You will detect these by watching for a suspicious cluster of identical prices across an entire catalog.

The architecture

Here is the pipeline we will build:

[Seed keywords]
     |
     v
[Shopify Store Detector] --> list of shopify domains
     |
     v
[Shopify Store Analyzer] --> products, prices, collections
     |
     v
[Postgres / DuckDB]  <--  daily snapshot
     |
     v
[Diff engine] --> Slack / email alerts

Two Apify actors do the heavy lifting:

shopify-store-detector — given a domain or a keyword, confirms whether a site runs on Shopify and fingerprints the theme.
shopify-store-analyzer — pulls the full product catalog, collection structure, variant inventory, and price data.

Step 1: Find competitor stores

If you already have a list of competitor URLs, skip this section. Otherwise, use the store detector to validate candidates. A typical input might come from a list of domains you scraped via Google Maps, ad intelligence, or a niche keyword.

from apify_client import ApifyClient

client = ApifyClient("APIFY_TOKEN")

run_input = {
    "domains": [
        "allbirds.com",
        "gymshark.com",
        "fashionnova.com",
        "randomcompetitor.co",
    ]
}

run = client.actor("nexgendata/shopify-store-detector").call(run_input=run_input)

for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    if item["is_shopify"]:
        print(item["domain"], item["theme"], item["plan_hint"])

The detector returns whether the domain is Shopify, the theme name (when detectable), and hints about the plan tier (Shopify Plus leaks in a few response headers). Persist this list — you will feed it into the analyzer.

Step 2: Pull the catalog

For each confirmed Shopify domain, run the analyzer. The important trick here is that Shopify paginates /products.json at 250 items per page, and some stores have 10k+ SKUs. The actor handles pagination, retries, and proxy rotation for you.

run_input = {
    "stores": ["allbirds.com", "gymshark.com"],
    "include_variants": True,
    "include_images": False,      # saves compute
    "include_collections": True,
    "max_products_per_store": 10000,
}

run = client.actor("nexgendata/shopify-store-analyzer").call(run_input=run_input)
items = list(client.dataset(run["defaultDatasetId"]).iterate_items())
print(f"Pulled {len(items)} products")

Each item looks something like:

{
  "store": "allbirds.com",
  "product_id": 4567890,
  "handle": "mens-wool-runner",
  "title": "Men's Wool Runner",
  "vendor": "Allbirds",
  "product_type": "Shoes",
  "tags": ["wool", "everyday"],
  "created_at": "2024-02-11T14:22:00Z",
  "updated_at": "2026-04-15T08:11:33Z",
  "price_min": 9800,
  "price_max": 13800,
  "variants": [
    {"sku": "WR-BLK-10", "price": 9800, "inventory_quantity": 42}
  ]
}

Step 3: Detect what changed

Store each daily run in a table keyed by (store, product_id, run_date). Then a simple SQL diff surfaces launches, price changes, and stockouts:

-- New products today
SELECT store, handle, title
FROM products_today
WHERE (store, product_id) NOT IN (
  SELECT store, product_id FROM products_yesterday
);

-- Price changes >5%
SELECT t.store, t.handle,
       y.price_min AS old_price, t.price_min AS new_price,
       ROUND(100.0 * (t.price_min - y.price_min) / y.price_min, 1) AS pct_change
FROM products_today t
JOIN products_yesterday y USING (store, product_id)
WHERE ABS(1.0 * t.price_min / y.price_min - 1) > 0.05;

Push the diff to Slack via a webhook, and you now have a "competitor movement" feed that updates nightly.

Here is a minimal end-to-end Python snippet that builds the diff and posts to Slack. Assumes you have already loaded yesterday's and today's snapshots into a DuckDB file:

import duckdb, os, json, urllib.request

SLACK_WEBHOOK = os.environ["SLACK_WEBHOOK"]
con = duckdb.connect("shopify_intel.duckdb")

price_moves = con.execute("""
    SELECT t.store, t.handle, y.price_min AS old, t.price_min AS new,
           ROUND(100.0 * (t.price_min - y.price_min) / y.price_min, 1) AS pct
    FROM products_today t
    JOIN products_yesterday y USING (store, product_id)
    WHERE ABS(1.0 * t.price_min / y.price_min - 1) > 0.05
    ORDER BY ABS(pct) DESC
    LIMIT 25
""").fetchall()

launches = con.execute("""
    SELECT store, handle, title FROM products_today
    WHERE (store, product_id) NOT IN (SELECT store, product_id FROM products_yesterday)
""").fetchall()

lines = [f":rotating_light: *Overnight competitor diff*"]
if launches:
    lines.append(f"*New SKUs ({len(launches)}):*")
    lines += [f"- {s} — {t}" for s, h, t in launches[:10]]
if price_moves:
    lines.append(f"*Price moves ({len(price_moves)}):*")
    lines += [f"- {s}/{h}: ${o/100:.0f} -> ${n/100:.0f} ({p:+}%)" for s, h, o, n, p in price_moves]

req = urllib.request.Request(
    SLACK_WEBHOOK,
    data=json.dumps({"text": "\n".join(lines)}).encode(),
    headers={"Content-Type": "application/json"},
)
urllib.request.urlopen(req)

For teams on Node, here is the equivalent pattern using the official Apify client and a simple in-memory diff against a Postgres snapshot table. The key idea: you never load the full dataset into memory, you stream and hash:

import { ApifyClient } from 'apify-client';
import pg from 'pg';

const client = new ApifyClient({ token: process.env.APIFY_TOKEN });
const db = new pg.Client({ connectionString: process.env.DATABASE_URL });
await db.connect();

const run = await client.actor('nexgendata/shopify-store-analyzer').call({
  stores: ['allbirds.com', 'gymshark.com'],
  include_variants: true,
});

for await (const item of client.dataset(run.defaultDatasetId).iterate()) {
  const fingerprint = `${item.price_min}|${item.price_max}|${item.variants?.length ?? 0}`;
  await db.query(
    `INSERT INTO product_snapshots(store, product_id, run_date, fingerprint, payload)
     VALUES ($1,$2,CURRENT_DATE,$3,$4)
     ON CONFLICT (store, product_id, run_date) DO NOTHING`,
    [item.store, item.product_id, fingerprint, item],
  );
}

const { rows: changes } = await db.query(`
  SELECT t.store, t.product_id
  FROM product_snapshots t
  JOIN product_snapshots y
    ON y.store=t.store AND y.product_id=t.product_id
   AND y.run_date = t.run_date - INTERVAL '1 day'
  WHERE t.run_date = CURRENT_DATE AND t.fingerprint <> y.fingerprint
`);
console.log(`Detected ${changes.length} material changes`);

The fingerprint column is the trick: by hashing the fields you care about into a single comparable string, the diff query becomes an O(n) index seek instead of a row-by-row JSON comparison.

Step 4: Schedule it

Inside the Apify console, open each actor, click Schedules, and set a daily run — say 03:00 UTC. Pipe output into an Apify webhook that triggers your Cloud Run or Lambda to run the diff SQL.

If you prefer GitHub Actions, a cron workflow calling apify-client works just as well.

Use cases

1. DTC founder scouting launches. A skincare brand founder tracks 40 indie competitors. Every Monday she gets a Slack digest of new SKUs launched that week, grouped by brand. Total cost: ~$9/month in Apify compute.

2. Agency doing competitive audits. A growth agency ingests 200 Shopify stores for a client report. Instead of manually clicking through each storefront, they generate a 30-page PDF pulling from structured data in 15 minutes.

3. Arbitrage and dropshipper repricing. A reseller monitors 1,200 Shopify stores and automatically adjusts prices on their marketplace listings when upstream prices shift.

4. Investor diligence. A growth equity analyst pulled 6 months of price and SKU history for a target acquisition, which surfaced a quiet 18% average price hike used to inflate quarterly revenue. The firm used that finding to renegotiate valuation by $2.1M — a six-figure ROI on roughly $180 of Apify compute.

5. Retail buyer trend spotting. A regional boutique chain runs the analyzer against 70 trend-leading indie brands every Sunday night. Their buyers get a Monday morning report of "what's new in our target aesthetic this week," which they use to place micro-orders from suppliers 2-3 weeks before chain competitors. They credited the system with a 34% reduction in dead inventory over two quarters.

6. Ad creative research. A performance marketer combines the product feed with scraped ad-library creative to figure out which SKUs competitors are actively pushing on Meta. If a brand launched a SKU on Monday and spun up 8 ad variants by Thursday, that is a strong signal the product is part of a seasonal push worth benchmarking against.

Pricing comparison

Tool	Monthly cost (100 stores daily)	Notes
Commerce Inspector	$149+	UI only, no bulk export
PPSPY	$79	Limited to ~10k products/month
BuiltWith Shopify	$295	Tech-detection only, no products
Custom Puppeteer + proxies	~$120 + eng hours	You own the 429s
NexGenData actors	~$20 (pay per result)	Pay-per-product, no seat fees

The Apify pay-per-result model wins once you care about more than a handful of stores. You are not paying for a seat you are not using.

Common pitfalls

Scraping Shopify is a deceptively deep well. These are the ones that eat weekends if you are not warned:

Storefront password protection. Some stores hide /products.json behind a coming-soon page or a password gate. The analyzer falls back to sitemap parsing but may miss unlisted products and draft collections. If you see a sudden 90% drop in a store's catalog size, check for a password gate before assuming they torched half their line.
Timezones in updated_at. Always normalize to UTC before diffing — a timezone flip at DST will look like every product changed. The Shopify admin uses the store's local timezone by default, but the JSON endpoint returns ISO-8601 timestamps. Parse carefully.
Variants vs. products. Track both. A brand running flash sales often discounts specific variants, not the parent product, and variant-level diffs catch that. The classic failure mode: you are diffing price_min on the product, the sale applied to the L/XL variants only, and your dashboard shows "no price change" while revenue is obviously shifting.
Metafields. Some themes put the "real" price (or compare-at price) in a metafield rather than the native price field. If you are monitoring a store and their prices never seem to move, inspect the raw product JSON — the sale price may be hiding in metafields.custom.promo_price.
Currency and region. Shopify Markets lets one store serve different prices by country. A single scrape from a US IP only shows US prices. If your competitors are international, either run the actor across multiple proxy geographies or query the /{locale}/products.json variant.
Handle changes. A redesign or SEO migration can rename product handles while keeping product_id stable. Always key your diffs on product_id, never the handle. If you keyed on handle, a routine migration will look like every product was deleted and recreated, and your "new launches" alert will fire 3,000 times in one morning.
Ghost SKUs. Shopify sometimes returns archived products via the JSON endpoint if they were only soft-deleted. Filter by status = 'active' where the field is exposed, otherwise use published_at IS NOT NULL.
Duplicate detection across stores. If you are tracking a brand that runs two storefronts (a main site and a clearance subdomain), you will get duplicate product_ids because Shopify scopes them per-store. Always key on (store, product_id), never just product_id.

A smaller but spicy gotcha: never assume a store's theme is static. Brands A/B test themes. A given customer might see Dawn on Monday and Impulse on Thursday. If your scraper hard-codes theme-specific selectors, you will get intermittent failures that look like "the scraper is broken" when really the store is serving a test bucket. Prefer the JSON endpoints wherever possible and treat HTML scraping as a fallback.

How NexGenData handles this

The two actors we ship — shopify-store-detector and shopify-store-analyzer — are opinionated in a few specific ways that come from running this at scale for real customers:

Automatic fallback chain. The analyzer tries /products.json first (fastest, cheapest), falls back to the /sitemap_products_*.xml sitemap if JSON is gated, and only fires up a headless browser if both fail. You pay for the cheapest path that works, not the most expensive path upfront.

Smart rate limiting. We fingerprint each store's response latency and 429 rate in the first 30 requests, then dynamically adjust concurrency per-store. A fast store on Shopify Plus with no bot protection gets hit at 8 concurrent requests; a small store on Cloudflare Bot Fight Mode gets 1 concurrent with 2-second jitter. You do not have to tune this.

Theme fingerprinting. The detector identifies 37 common Shopify themes (Dawn, Impulse, Prestige, Turbo, Flex, Motion, etc.) plus custom Hydrogen/Oxygen implementations. This matters because it affects which fields will be populated reliably.

Built-in variant and metafield flattening. Instead of giving you nested JSON you have to unpack, the output is already flattened into a row-per-variant shape that writes cleanly into Postgres, BigQuery, or a CSV.

Pay-per-result pricing. You pay per product pulled, not per compute-minute. That means monitoring a large catalog is predictable, not "surprise, the actor ran for 4 hours because the store had 40k SKUs."

Conclusion

Shopify competitor monitoring is not a one-off side quest. Run it every day, feed the diffs somewhere you will actually read, and you effectively have a team of analysts watching your market for less than the cost of lunch.

Get started with the two core actors:

Shopify Store Detector — validate and fingerprint any domain.
Shopify Store Analyzer — pull products, variants, and collections at scale.
Tech Stack Detector — find which competitors use Klaviyo, Gorgias, Rebuy, etc.

FAQ

Is it legal to scrape competitor Shopify stores?
Generally yes, for publicly accessible product data. The /products.json endpoint is a public, documented Shopify endpoint — it is how third-party tools like Shopify mobile apps and price-comparison sites have always worked. Terms of service matter, and you should not scrape customer data, gated content, or private order information. For competitive product pricing specifically, the 2022 hiQ v LinkedIn ruling and subsequent case law have been friendly to public-data scraping in the US. That said, consult your own counsel.

How often should I run the scrape?
Daily is the sweet spot for most DTC use cases. Hourly is overkill unless you are in a flash-sale or auction category. Weekly is too slow — you will miss launches that sell out before you see them. If you are doing arbitrage or repricing, push to every 4-6 hours.

What about Shopify Plus stores with custom storefronts (Hydrogen)?
Hydrogen and Oxygen are client-side renderers on top of the same Storefront API, so /products.json usually still works. Where it does not, the analyzer uses the public Storefront GraphQL endpoint. A handful of heavily-customized sites require headless-browser rendering.

Can I integrate this with Zapier or Make.com?
Yes. Apify has native integrations with both Zapier and Make. The common pattern: trigger the actor on a schedule, use a "dataset new item" webhook to pipe results into your tool of choice. For a simple Slack alert, no code needed.

How do I handle stores that block my IP after a few runs?
Apify provides residential and datacenter proxies out of the box. Enable proxyConfiguration.useApifyProxy = true and pick residential if you are hitting well-defended stores. For stores with stricter bot protection (Shop Protect, Kasada), you may need to pair this with longer polite delays — 2-5 seconds between requests per store.

What is the total cost for 500 stores monitored daily?
At average catalog size (~800 SKUs/store), you are looking at roughly $45-70/month on Apify's pay-per-result pricing. Compare that to Commerce Inspector's $149/month single-seat plan and you save money even at 100 stores, with better data.