DEV Community: NexGenData

Scraping Google Scholar for Literature Reviews (Python + Apify, 2026)

NexGenData — Thu, 13 Aug 2026 22:35:10 +0000

Scraping Google Scholar for Literature Reviews (Python + Apify, 2026)

Every grad student hits the wall around week three of a lit review: you have 400 open tabs, 60% of which are duplicates, and Zotero crashed again. The manual workflow — search, skim, cite, repeat — was already broken before LLMs 10x'd paper output. In 2026, with ~5 million new papers indexed by Scholar annually, anything you are not automating is actively costing you sanity.

To put some real numbers on the problem: a 2025 STM report estimated 3.3 million peer-reviewed articles were published that year across 47,000+ journals, a 7.2% year-over-year increase. ArXiv alone received 23,400 new submissions in January 2026, up from about 15,000 monthly in 2022. The National Institutes of Health PubMed Central added 1.7 million records in 2025. Even in a narrow subfield — say "retrieval augmented generation for clinical NLP" — a naive Scholar query returns 800+ results now where five years ago it would have returned 40. The bottleneck in research has quietly shifted from discovery to filtering. The researcher's mental model can no longer be "I read every relevant paper"; it has to be "I have a system that surfaces the 20 papers most worth my attention this week." That system is what this post builds.

This post walks through an Apify-based literature-review pipeline: automatic paper collection from Google Scholar and ArXiv, citation graph construction, duplicate dedup, and PRISMA-style export for systematic reviews. Code included, including a minimal LLM-based triage step for the cases where you want to narrow a 1,000-paper harvest to the 50 worth reading.

Why this is hard

Google Scholar is notorious among scrapers for good reason:

Aggressive bot detection. Scholar's anti-scraping rivals Cloudflare enterprise. A direct Python requests loop gets blocked after 5-15 queries.
No public API. Unlike PubMed, Scholar has no official API. Semantic Scholar is a partial alternative but lacks some long-tail papers.
Ambiguous deduplication. The same paper shows up three ways: a preprint on ArXiv, a conference version, a journal version. Collapsing them correctly is non-trivial.
Citation counts drift. Citation counts update daily, so pipelines need to be idempotent and store snapshots.

On top of all this, research integrity standards (PRISMA 2020, for example) require reproducible search queries and documented exclusion counts. Your scraper has to log its query history.

Preprint vs. published attribution. The same paper may be indexed three times (ArXiv preprint, conference proceedings, journal publication) with different DOIs and different citation counts. Scholar sometimes collapses these with a "[PDF] from..." link and sometimes doesn't.
Cross-language coverage. Scholar does index non-English papers, but its coverage of Chinese, Japanese, and Portuguese literature is uneven. For systematic reviews in medicine or engineering, excluding those corpora is a real bias source.

The architecture

[Query strings + filters]
          |
          v
 [google-scholar-scraper] --> papers with citations, DOIs, authors
          |
          v
 [arxiv-scraper]          --> preprint full text + metadata
          |
          v
 [dedup + embedding cluster]
          |
          v
 [Citation graph (Neo4j or simple DuckDB)]
          |
          v
 [PRISMA flowchart + CSV export]

Step 1: Search Scholar

The google-scholar-scraper handles Scholar's bot detection and returns structured papers with citation counts and author h-index data.

from apify_client import ApifyClient

client = ApifyClient("APIFY_TOKEN")

run = client.actor("nexgendata/google-scholar-scraper").call(run_input={
    "queries": [
        "large language model hallucination detection",
        "retrieval augmented generation evaluation"
    ],
    "year_from": 2023,
    "year_to": 2026,
    "max_results_per_query": 500,
    "include_patents": False,
    "include_citations": True,
})

papers = list(client.dataset(run["defaultDatasetId"]).iterate_items())

Each paper:

{
  "title": "Evaluating Hallucinations in LLMs: A Meta-Analysis",
  "authors": ["Smith, J.", "Chen, L."],
  "venue": "NeurIPS 2024",
  "year": 2024,
  "citation_count": 142,
  "doi": "10.48550/arXiv.2404.12345",
  "abstract": "...",
  "pdf_url": "https://arxiv.org/pdf/2404.12345",
  "scholar_id": "xYz123"
}

Step 2: Pull ArXiv full text

Many Scholar results are behind paywalls. If a paper has a preprint on ArXiv, use the arxiv-scraper to fetch the full-text PDF link and structured abstract.

arxiv_ids = [extract_arxiv_id(p["doi"]) for p in papers if "arxiv" in p.get("doi","").lower()]

run = client.actor("nexgendata/arxiv-scraper").call(run_input={
    "arxiv_ids": arxiv_ids,
    "include_pdf_url": True,
    "include_latex_source": False,
})
arxiv_papers = list(client.dataset(run["defaultDatasetId"]).iterate_items())

Step 3: Deduplicate across sources

Scholar and ArXiv overlap heavily. Dedup on DOI first, fall back to title similarity:

from rapidfuzz import fuzz

def is_dupe(a, b):
    if a.get("doi") and a["doi"] == b.get("doi"):
        return True
    return fuzz.token_set_ratio(a["title"], b["title"]) > 92

seen, unique = [], []
for p in papers + arxiv_papers:
    if not any(is_dupe(p, s) for s in seen):
        seen.append(p); unique.append(p)

print(f"{len(papers)+len(arxiv_papers)} raw -> {len(unique)} unique")

Step 4: Build a citation graph

For snowball sampling — finding papers that cite (or are cited by) a seed set — run forward and backward from high-citation anchor papers. The scraper's include_citations flag returns Scholar's "Cited by" links.

Load into DuckDB:

import duckdb
con = duckdb.connect("lit_review.duckdb")
con.execute("CREATE TABLE papers (scholar_id VARCHAR, title VARCHAR, year INT, citations INT)")
con.executemany(
    "INSERT INTO papers VALUES (?, ?, ?, ?)",
    [(p["scholar_id"], p["title"], p["year"], p["citation_count"]) for p in unique]
)

Then ask questions:

-- Most-cited papers in the corpus
SELECT title, year, citations FROM papers ORDER BY citations DESC LIMIT 20;

-- Papers on the topic from the last 12 months with >50 citations (fast movers)
SELECT title, authors FROM papers WHERE year = 2025 AND citations > 50;

Step 5: Export a PRISMA-compliant CSV

Systematic reviews require documented inclusion/exclusion counts. Export:

import csv
with open("prisma.csv", "w") as f:
    w = csv.writer(f)
    w.writerow(["id","title","year","authors","venue","doi","included","exclusion_reason"])
    for p in unique:
        w.writerow([p["scholar_id"], p["title"], p["year"], "; ".join(p["authors"]),
                    p.get("venue",""), p.get("doi",""), "", ""])

Now fill in inclusion/exclusion manually (or via an LLM screening pass) and you have a PRISMA-ready audit trail.

Step 6: LLM triage (optional but transformative)

If you have harvested 800 papers and realistically only need 50, an LLM screening pass can save a week of skimming. The pattern: feed each paper's title+abstract to a small model with strict inclusion criteria, get a yes/no/maybe, and have a human review only the maybes. This is the same pattern used in Cochrane-style reviews.

import os, json
from openai import OpenAI
oai = OpenAI(api_key=os.environ["OPENAI_API_KEY"])

CRITERIA = """
Include a paper if ALL are true:
- Published 2023 or later
- Involves retrieval augmented generation (RAG) as primary method
- Reports quantitative evaluation on a named benchmark
- Is a peer-reviewed paper or widely-cited preprint (>20 citations)
Exclude if it is a survey, workshop abstract, or tutorial.
"""

def screen(paper):
    prompt = f"""{CRITERIA}

Title: {paper['title']}
Abstract: {paper.get('abstract','')[:1500]}

Respond with JSON: {{"decision": "include" | "exclude" | "maybe", "reason": "..."}}"""
    resp = oai.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": prompt}],
        response_format={"type": "json_object"},
    )
    return json.loads(resp.choices[0].message.content)

screened = []
for p in unique:
    try:
        s = screen(p)
        screened.append({**p, **s})
    except Exception as e:
        screened.append({**p, "decision": "maybe", "reason": f"screening error: {e}"})

included = [p for p in screened if p["decision"] == "include"]
maybes = [p for p in screened if p["decision"] == "maybe"]
print(f"Included: {len(included)}, Needs human: {len(maybes)}, Excluded: {len(screened)-len(included)-len(maybes)}")

A 2025 study in the Journal of Medical Internet Research showed LLM-assisted screening with human review of maybes achieved 96% concordance with a fully-manual Cochrane review while reducing time-to-screen by 73%. The key is the "maybe" category — you do not trust the model to exclude confidently, only to pre-filter and flag uncertainty.

Use cases

1. PhD lit review. A computational biology student runs Scholar + ArXiv nightly for 30 queries. Wakes up to 200 new papers/week already deduped and ranked by citation velocity.

2. Patent landscape analysis. A biotech analyst uses the include_patents=True flag to pull both papers and patents for CRISPR+gene therapy, surfacing IP overlaps.

3. Grant writing. A PI needs "key references" for a NIH R01. Automates top-20 papers per aim, by recency, in 10 minutes.

4. Research trend forecasting. A VC analyst counts yearly papers per topic to identify emerging subfields 12-18 months before they trend.

5. Competitor-lab monitoring. A pharma R&D team tracks publication output from 12 competing research groups. Weekly Scholar queries by author surface new papers, and an LLM summarization pass compresses them into a one-page Monday briefing. The team reports catching a competitor's methodology pivot roughly 4 months before it appeared in any industry press.

6. Teaching updated syllabi. A university professor auto-refreshes reading lists for their graduate seminar each semester. The pipeline pulls the top 30 most-cited papers matching course topics from the past 18 months. Students get a current reading list instead of one last updated in 2019.

Pricing comparison

Service	Cost for 10k papers	Full text?	Citation graph?
Scopus API	Institutional only	Partial	Yes
Web of Science	$300-1000/user/yr	Partial	Yes
Semantic Scholar API	Free (rate-limited)	Abstract	Yes
scholarly (Python)	Free (blocked fast)	No	Limited
Apify actors	~$15	Yes (ArXiv)	Yes

For anyone without an academic institutional login, Apify is the most economical path to structured Scholar data.

Common pitfalls

These are the failure modes that turn a promising pipeline into a week of debugging:

Rate-limit yourself. Even with the actor's proxy rotation, set max_results_per_query sensibly. 500 is fine; 5000 will be slow and approaches Scholar's deep-pagination ceiling, where results past page 100 become unreliable.
Citation counts are snapshots. Re-run weekly if you care about trajectory. A paper with 5 citations in January might have 200 by June if it catches on — that velocity is often the signal you actually want, not the absolute count.
Author disambiguation. Scholar merges authors with the same name. "J. Smith" could be 8 different researchers across Scholar's index. Join on author+affiliation+ORCID when possible. ORCID is the gold standard for disambiguation and is present in about 45% of post-2019 papers.
Scholar's "include citations" toggle. The checkbox that includes citations and patents in search results can wildly inflate counts. Decide early whether you want scholarly articles only or everything.
Preprint vs. final version citation counts. ArXiv preprint citations and journal-version citations are often tracked separately. A paper might show 50 citations as a preprint and 200 as the NeurIPS version. Semantic Scholar merges these; Google Scholar sometimes does not.
Retracted papers. Scholar does not always flag retractions. Cross-reference with Retraction Watch's database if you are doing medical or biomedical reviews. A retracted paper cited as supporting evidence is worse than not finding the paper at all.
Year drift in metadata. A paper might have year: 2024 in Scholar and year: 2025 in CrossRef because one uses "submitted" and the other uses "published." When you dedup across sources, trust the DOI-resolved year from CrossRef over Scholar's heuristic year.
PDF URL freshness. Scholar caches PDF links. Many point to author homepages that 404 a year later. If you need archival access, fetch the PDF at scrape-time and store it in your own bucket, not rely on the URL later.
Scholar's "sciting" vs. "citing". Citations can be positive or negative. For meta-analysis, consider using scite.ai's signal layer on top of your Scholar data to weight supporting vs. contradicting citations.

How NexGenData handles this

Scraping Scholar reliably is a full-time job for someone. We built the google-scholar-scraper actor to absorb that work:

Anti-bot hardening. We rotate residential proxies by geography, use realistic browser fingerprints, and handle Scholar's rotating captcha challenges automatically. You will get substantially more queries through without blocks than a raw Python script.

Structured author + citation output. The actor returns authors as separate objects with h-index where available, not as a single flat string. Citation counts, cited-by links, and related-papers links are all preserved.

DOI extraction and normalization. Scholar embeds DOIs inconsistently — sometimes as links, sometimes in snippets, sometimes not at all. The actor extracts and normalizes them so dedup-by-DOI works cleanly.

Snapshotting support. Every run can be tagged with a query fingerprint, so reproducibility for PRISMA audits is built-in. You can replay any past run's exact input.

Native pairing with the ArXiv actor. The two actors share DOI conventions, so merging their outputs is a straightforward DuckDB join. No impedance mismatch.

Pay-per-result. 10,000 papers costs roughly $15. Compare that to Scopus institutional access at $20k+/year, and it is the clear choice for independent researchers and teams without a university affiliation.

Conclusion

Literature review in 2026 is an information-retrieval problem, not a reading problem. Automate the collection, deduplication, and ranking steps, then spend your actual brain cycles on the hard part: reading the 30 papers that matter.

Three actors to start:

Google Scholar Scraper — bot-detection-proof Scholar access.
ArXiv Scraper — full preprint metadata and PDF links.
Wikipedia Scraper — for background context on unfamiliar fields.

FAQ

Is scraping Google Scholar allowed?
Scholar's ToS prohibit automated access, and Google does actively rate-limit. That said, academic use of search data has been treated leniently in practice, and the aggregation of public bibliometric data is broadly considered fair use. If you publish research based on scraped data, cite Scholar as your source. For true systematic reviews with publication requirements, supplement Scholar with official databases (PubMed, Scopus) that have APIs and documented ToS.

How does this compare to Semantic Scholar's API?
Semantic Scholar has a clean API and free tier (100 req/5min). Its coverage is strong for CS, medicine, and physics but weaker for humanities and some regional journals. For thorough systematic reviews, use both: Scholar for coverage breadth, Semantic Scholar for citation graphs and clean metadata.

What about OpenAlex?
OpenAlex is an open bibliometric database covering 240M+ works with a free, unlimited API. It is excellent for citation graphs and author profiles, often better than Scholar for those specific tasks. Scholar still wins for "find me the most-read recent paper on X," because its ranking is informed by actual reader behavior.

Can I get full-text PDFs reliably?
For ArXiv papers, yes. For anything else, it depends on open-access status. The pipeline can fetch PDFs where URLs are public, but paywalled papers need either your institutional login, an interlibrary loan, or (for some reviews) contacting authors directly for a copy.

How do I cite papers I find this way in my own writing?
Export the deduped set to BibTeX via pandoc-citeproc or the bibtexparser Python library. The doi field in the output is what most reference managers key on.

What about non-English literature?
The actor passes through whatever Scholar returns, which includes non-English papers if your query matches them. For comprehensive non-English coverage, pair with regional databases (CNKI for Chinese, CiNii for Japanese, SciELO for Latin American journals).

How do I handle the "forward snowball" (papers that cite my seed papers)?
Use the include_citations=True flag and the "cited by" URL in the Scholar response. Run a second pass with those URLs as seeds. Two iterations of forward snowballing typically saturates a well-defined subfield.

Related tools

Bulk Domain WHOIS Lookup — API Alternatives to Whois.com (2026)

NexGenData — Tue, 11 Aug 2026 21:34:44 +0000

Bulk Domain WHOIS Lookup — API Alternatives to Whois.com (2026)

If you have ever tried to look up WHOIS info for more than 50 domains in a row, you know the drill: whois.com throws a captcha, the whois CLI stops resolving halfway through, and GoDaddy's WHOIS widget silently starts returning empty responses after some undocumented throttle. Running proper bulk WHOIS in 2026 is harder than it should be — thanks in part to GDPR redactions, thick vs. thin registrar models, and a fragmented TLD landscape.

A quick 2026 snapshot to set the stage: there are roughly 359 million registered domains globally (Verisign Domain Name Industry Brief, Q4 2025), spread across 1,500+ TLDs, with .com still dominant at ~157M and newer TLDs like .ai, .xyz, .io, and .dev growing 20-40% year-over-year. The WHOIS ecosystem used to be a single TCP-port-43 protocol with reasonably uniform output, but ICANN's 2024 mandate requiring all registries to support RDAP (Registration Data Access Protocol) means you are now living in a hybrid world: some registries serve RDAP JSON, some still serve legacy WHOIS text, and a surprising number serve both with subtly different field names. GDPR redactions, in effect since May 2018, hide registrant personal data by default for EU-registered domains and in practice for most registrars globally. The mental model of "WHOIS is a phonebook for domains" is outdated. The accurate model is "WHOIS is a fragmented, rate-limited, partially-redacted dataset that requires an aggregation layer to use at scale."

This post compares the main options for bulk WHOIS in 2026, shows how to build a production pipeline in Python, and covers the use cases that actually justify the effort: domain monitoring, brand protection, cybersecurity triage, and M&A diligence. If you are doing any of those at more than handful-of-domains scale, you need an automated pipeline, not a browser tab.

Why this is hard

WHOIS looks simple — one TCP connection to port 43, one record back. Reality:

Every TLD has its own WHOIS server and response format. .com uses Verisign, .io uses NIC.IO, country-code TLDs each run their own registry. Parsing is a mess.
GDPR redactions hide most registrant data. Since 2018, EU-registered domains no longer expose owner names or contact emails. Some registrars redact all domains, not just EU ones.
Rate limits vary wildly. Verisign thin WHOIS tolerates ~20 queries/sec, but query the same .com registrar's thick WHOIS and you will see 5 queries/sec before blocks.
RDAP is slowly replacing WHOIS. Some registries return only RDAP JSON now. Your parser needs both code paths.

Rolling this yourself means maintaining per-TLD parsers, proxy pools, and RDAP fallbacks. Most teams burn a week on it and then the parser breaks again when .ai changes its format.

Intermittent TLDs. Some ccTLDs (.ke, .pk, .sa) are only available via web-form WHOIS with no programmatic endpoint. Others require paid registry access.
Stale cache semantics. Registry responses include a varying TTL. If you query a domain right after a transfer, you may get old registrar data for hours. Knowing when to retry vs. accept staleness is a heuristic, not a rule.

The architecture

[Domain list (CSV, 10k rows)]
           |
           v
    [Apify WHOIS actor] -- rotating proxies, RDAP fallback, TLD-aware parsing
           |
           v
    [Structured JSON dataset]
           |
           v
 [DuckDB / Postgres] -- dedupe + expiry alerting
           |
           v
     [Slack / email] -- "10 domains expire in 30 days"

The core building block is the domain-whois-lookup actor, which handles TLD routing, RDAP fallback, rate-limit evasion, and parsing into a normalized schema.

Step 1: One-shot bulk lookup

from apify_client import ApifyClient

client = ApifyClient("APIFY_TOKEN")

domains = [l.strip() for l in open("domains.csv") if l.strip()]

run = client.actor("nexgendata/domain-whois-lookup").call(run_input={
    "domains": domains,
    "include_raw": False,
    "rdap_fallback": True,
})

results = list(client.dataset(run["defaultDatasetId"]).iterate_items())
print(f"Processed {len(results)} domains")

A typical result looks like:

{
  "domain": "example.com",
  "registrar": "MarkMonitor Inc.",
  "registrar_iana_id": 292,
  "creation_date": "1995-08-14T04:00:00Z",
  "expiry_date": "2028-08-13T04:00:00Z",
  "last_updated": "2025-08-14T07:01:23Z",
  "nameservers": ["a.iana-servers.net", "b.iana-servers.net"],
  "dnssec": "signedDelegation",
  "status": ["clientDeleteProhibited", "clientTransferProhibited"],
  "registrant_country": "US",
  "redacted": true
}

Notice the redacted: true flag. You still get registrar, nameservers, and dates — which is 80% of what most use cases need.

Step 2: Watch for expiries

Once results land in your dataset, a 10-line SQL query surfaces domains expiring soon:

SELECT domain, registrar, expiry_date,
       date_diff('day', current_date, expiry_date) AS days_left
FROM whois_results
WHERE expiry_date < current_date + INTERVAL 60 DAY
ORDER BY expiry_date ASC;

Wire this into a scheduled Apify run (daily), then a cron job that posts the result to Slack:

import requests, json
slack = "https://hooks.slack.com/services/..."
rows = [r for r in results if days_until_expiry(r) < 60]
if rows:
    text = "Domains expiring soon:\n" + "\n".join(
        f"{r['domain']} -> {r['expiry_date']}" for r in rows)
    requests.post(slack, json={"text": text})

Step 3: Join with DNS

WHOIS alone is often not enough. Pair it with DNS records and you get a much richer picture: who owns it, where it resolves, and whether it is parked.

The dns-propagation-checker actor resolves A, AAAA, MX, and TXT across 15+ resolvers. Combined with WHOIS you can answer:

Which competitor domains changed nameservers this week?
Which newly-registered typosquats already have MX records (meaning they are phishing-ready)?
Which domains in our portfolio are pointed at the wrong CDN?

Here is a reasonably complete typosquat-detection snippet that fans out permutations of a brand, runs WHOIS on each, and flags any that were registered in the last 30 days. This is the bones of a real brand-protection pipeline:

from apify_client import ApifyClient
from datetime import datetime, timezone, timedelta
import itertools

BRAND = "acmecorp"
TLDS = [".com", ".net", ".org", ".co", ".io", ".ai", ".app", ".dev"]

def permutations(brand):
    # character-swap, leet-speak, and homoglyph basics
    swaps = {"a": ["4", "@"], "e": ["3"], "o": ["0"], "i": ["1", "l"]}
    out = {brand, brand + "s", brand + "hq", brand + "-app", "get" + brand}
    for i, ch in enumerate(brand):
        for alt in swaps.get(ch, []):
            out.add(brand[:i] + alt + brand[i+1:])
    return out

candidates = [b + t for b, t in itertools.product(permutations(BRAND), TLDS)]

client = ApifyClient("APIFY_TOKEN")
run = client.actor("nexgendata/domain-whois-lookup").call(run_input={
    "domains": candidates,
    "rdap_fallback": True,
})

cutoff = datetime.now(timezone.utc) - timedelta(days=30)
new_squats = []
for r in client.dataset(run["defaultDatasetId"]).iterate_items():
    if not r.get("creation_date"):
        continue
    created = datetime.fromisoformat(r["creation_date"].replace("Z", "+00:00"))
    if created > cutoff:
        new_squats.append(r)

print(f"Found {len(new_squats)} newly-registered typosquat candidates")
for r in new_squats:
    print(f"- {r['domain']} registered {r['creation_date']} via {r['registrar']}")

A similar pipeline in Node.js for teams more comfortable in JavaScript:

import { ApifyClient } from 'apify-client';
const client = new ApifyClient({ token: process.env.APIFY_TOKEN });

const domains = (await Deno.readTextFile('portfolio.csv')).split('\n').filter(Boolean);
const run = await client.actor('nexgendata/domain-whois-lookup').call({
  domains,
  rdap_fallback: true,
});

const ALERT_DAYS = 45;
const now = Date.now();
const expiring = [];
for await (const item of client.dataset(run.defaultDatasetId).iterate()) {
  if (!item.expiry_date) continue;
  const days = (new Date(item.expiry_date).getTime() - now) / 86400000;
  if (days < ALERT_DAYS) expiring.push({ ...item, days: Math.round(days) });
}
expiring.sort((a, b) => a.days - b.days);
console.table(expiring.map(({ domain, registrar, days }) => ({ domain, registrar, days })));

Use cases

1. Brand protection / typosquatting monitoring. A SaaS company monitors 300 domain permutations of their brand. Daily WHOIS runs flag newly-registered lookalikes within 24 hours, triggering legal takedowns.

2. M&A tech diligence. An acquirer runs WHOIS on every domain owned by the target. Finds three domains registered to an ex-founder's personal email — caught before close.

3. Cybersecurity triage. An incident responder receives 500 suspicious URLs. Bulk WHOIS reveals 240 were registered in the past 14 days at the same registrar — strong phishing-campaign signal.

4. Affiliate / SEO portfolio management. An affiliate marketer owns 120 domains. Monthly WHOIS batches feed a spreadsheet of expiries, saving at least one "oh no I forgot to renew" event per year.

5. Drop-catching research. A domain investor monitors expiring high-value domains by running WHOIS against a curated 10k watchlist weekly. When a domain moves into the 30-day redemption period, a Slack alert routes it to a review queue. In 2025 the same investor caught a 4-letter .com that would have gone to auction for low five figures.

6. Phishing kit attribution. A security vendor scrapes newly observed phishing URLs from public feeds (URLHaus, OpenPhish), runs WHOIS on the domains, and clusters by registrar + nameserver + creation date. Patterns emerge quickly: 60% of a week's phishing campaigns might trace back to 3 registrars. That kind of clustering is not possible without a WHOIS pipeline.

Pricing comparison

Service	10k lookups cost	Historical?	RDAP?	Bulk CSV?
WhoisXML API	~$100	Paid add-on	Yes	Yes
DomainTools Iris	$395+/mo	Yes	Yes	Yes
JsonWhoisAPI	~$80	No	No	Limited
whois.com manual	Free (captcha)	No	No	No
NexGenData actor	~$15	No	Yes	Yes

For the vast majority of use cases — bulk triage, expiry monitoring, diligence — you do not need $400/mo DomainTools. Pay-per-result at $0.0015/domain is comfortably within hobby-budget range.

Common pitfalls

WHOIS is one of those protocols where every shortcut has a hidden cost. These are the ones that catch people:

.ai and .io change formats. Expect occasional parsing misses on exotic ccTLDs. The actor updates parsers monthly, but if you are parsing raw responses yourself, budget for format drift every quarter.
Registrar WHOIS vs. registry WHOIS. Thick TLDs (.org, .info) return full data from the registry. Thin TLDs (.com) require a second hop to the registrar. The actor handles this transparently, but raw WHOIS CLIs do not. If you are using the whois CLI and seeing minimal data for .com, you are probably reading the Verisign thin response and missing the registrar follow-up.
Queries are cached. Most registries serve cached data. Same-second repeated queries will not reflect updates in real time. For change detection, poll every 6-24 hours, not every minute.
Premium WHOIS is its own thing. Some registrars charge extra for "WHOIS history" — time-series data of how WHOIS records changed. If you need that (common for cyber threat intel), only WhoisXML or DomainTools offer it at scale, though you can build a historical dataset yourself by snapshotting daily.
Punycode and IDN domains. A domain like bücher.de shows up as xn--bcher-kva.de in WHOIS. If your input list has UTF-8 brand names, convert to punycode before lookup, or the query will fail silently. Python has idna.encode() for this.
Glue records. WHOIS returns nameserver hostnames, but not their IPs. If you are tracking infrastructure migration, you also want the nameserver's A/AAAA records, which requires a follow-up DNS query.
Contact email redactions. GDPR redacts the registrant email, but sometimes the registrar exposes a forwarding address like contact@domainsbyproxy.com. These are useless for direct contact but reveal the proxy service, which is an attribution signal in itself.
Registrar name inconsistencies. "GoDaddy", "GoDaddy.com, LLC", and "GoDaddy Online Services Cayman Islands Ltd." are all the same company. Normalize on IANA ID (the registrar_iana_id field), not the string name, if you are clustering.
Expiry vs. grace period vs. redemption. A domain past its expiry_date is not immediately available. It enters a 30-day grace, then a 30-day redemption, then a 5-day pending-delete window. Drop-catching pipelines need to model all four phases correctly.

How NexGenData handles this

The domain-whois-lookup actor was built after we watched too many teams reinvent the same broken parser. Specific design choices:

Dual WHOIS + RDAP by default. Every lookup tries RDAP first (faster, structured JSON, consistent schema) and falls back to legacy WHOIS if RDAP is unavailable. You get the best data the registry will give, without having to code two pipelines.

1,500+ TLD parsers maintained. We monitor TLD response-format changes and push parser updates on a rolling basis. If .ai changes its format next month, you do not need to do anything.

Automatic thick-TLD follow-through. For .com/.net, we follow Verisign's referral to the registrar and merge both responses into a single row. No double-query logic on your side.

IANA-ID based registrar normalization. The output always includes both the raw string and the canonical IANA ID, so clustering and reporting work out of the box.

Proxy rotation and polite rate-limiting. We fingerprint each TLD's tolerance and adjust concurrency automatically. You can pass 10k domains at once and the actor will schedule them appropriately.

No API key, no seat fees, pay per result. $0.0015 per lookup. 10,000 domains costs $15. There is no monthly commitment.

Conclusion

Bulk WHOIS is a classic example of "seems trivial, actually has 15 edge cases." Rather than maintain per-TLD parsers yourself, point Apify at your domain list, schedule it, and wire the dataset into your alerting pipeline. You get coverage across 1,500+ TLDs, RDAP fallback, and normalized JSON out of the box.

Start here:

Domain WHOIS Lookup — core WHOIS + RDAP with pay-per-result pricing.
DNS Propagation Checker — resolve across 15 global DNS servers.
Tech Stack Detector — fingerprint the websites behind those domains.

FAQ

Is bulk WHOIS legal?
Yes. WHOIS is a public protocol mandated by ICANN for most generic TLDs. You cannot use WHOIS data for spam or direct marketing to registrants (that violates most registrar AUPs and GDPR). You can use it for security research, portfolio management, diligence, and trademark enforcement.

Does GDPR prevent me from getting any useful data?
No. GDPR redacts personal registrant info (name, email, address for individuals), but registrar, nameservers, creation date, expiry, and status codes are still public. For most operational use cases (monitoring, expiry, security triage) that is plenty.

What if I need the redacted data — say, for a trademark claim?
Use the registrar's abuse or legal contact channel with a formal request. Most registrars will disclose registrant info to a court order, UDRP filing, or legitimate IP complaint. The WHOIS API cannot magic it through.

RDAP vs. WHOIS — which should I use?
RDAP is the future and should be your default. It returns structured JSON, supports HTTPS, includes authenticated access paths for LEA, and is what ICANN is pushing. Legacy WHOIS should be a fallback. The actor handles both automatically.

How often should I re-lookup a domain?
For expiry monitoring: weekly is plenty. For change detection (nameserver flips, registrar transfers): daily. For active incident response: on-demand is fine. There is rarely a reason to poll more than once per 24h per domain.

Can I bulk-register domains via this pipeline?
No. WHOIS is read-only. Registration happens at a registrar (GoDaddy, Namecheap, Porkbun, Google Domains) via their API. You can use the WHOIS pipeline to find available domains and then register them separately.

What about new gTLDs like .xyz, .app, .dev?
All supported. The newer gTLDs almost all serve RDAP, which actually makes them easier to handle than some legacy ccTLDs.

Related tools

Building a Real-Time FX Dashboard with Apify + Google Sheets (2026)

NexGenData — Thu, 06 Aug 2026 20:53:01 +0000

Building a Real-Time FX Dashboard with Apify + Google Sheets (2026)

Every international SaaS founder eventually hits the same wall: "what is our real MRR in USD this week?" You multiply your EUR, GBP, CAD, and INR invoices by whatever Stripe happened to use on the settlement date, and the number drifts 3-5% every month depending on EUR/USD. Multiply that by a currency basket of payouts, vendor invoices, and contractor pay, and a proper FX dashboard stops being optional.

In 2026, FX volatility is still a meaningful P&L factor — the DXY (US Dollar Index) moved a cumulative 14% over the trailing 12 months, and emerging-market pairs like USD/TRY and USD/ARS have shifted by 40%+ in single quarters. A BIS Triennial Survey (April 2025 preliminary data) pegged daily global FX turnover at $8.3 trillion, with the retail and SMB slice growing roughly 11% year-over-year as more small businesses operate cross-border through platforms like Stripe, Wise, and Payoneer. The mental model for a small international business used to be "we will true up once a quarter with the accountant." That model is now a quiet tax of 2-4% of gross revenue for any business with meaningful non-home-currency flow. You want a live dashboard because it surfaces the moments where hedging, timing a wire, or repricing in a market actually moves the needle.

Commercial FX APIs (Fixer, OpenExchangeRates, CurrencyLayer) charge $50-300/month for real-time rates once you exceed the hobbyist tier. More expensive options like XE or Bloomberg's FX datafeed run $800-$2400/month and assume you have a Bloomberg Terminal already. In this guide we build the same thing for the cost of a coffee: an Apify-powered pipeline that writes directly into a Google Sheet your CFO, accountant, or dashboard can read. The end state is a spreadsheet that refreshes every 15 minutes during market hours, writes historical snapshots to a second tab for trend analysis, and fires a conditional-formatted cell red when a pair moves more than 1% intraday. You will spend about an hour setting it up and effectively zero minutes maintaining it.

Why this is hard

Free FX data has three traps:

Staleness. exchangerate.host and similar free endpoints cache aggressively, so rates lag market by 30+ minutes.
Midpoint vs. market. ECB midpoints are not what your bank actually pays. For realistic P&L you want the rate Yahoo Finance, Google Finance, or XE show — which are retail-skewed.
Rate limits on free tiers. 1,000 requests/month sounds like a lot until you poll 30 currency pairs every 15 minutes (that is 86k requests/month).

We will dodge all three by combining two Apify actors with Google Sheets' IMPORTDATA pattern plus a scheduled refresh.

Ticker-symbol mismatches across providers. Fixer uses ISO-4217 codes only. Yahoo Finance uses a XXXYYY=X convention. TradingView uses FX:EURUSD. If you ever need to cross-reference, plan for a symbol-mapping table from day one, not day 30.
Forward rates and spreads. Spot rates are what every free API serves. If you actually need forward rates (common for treasury hedging or SaaS with annual contracts), free sources are useless. We flag this below so you know when to escalate to a paid feed.

The architecture

[currency-exchange-rates actor] --> FX midpoint rates (150+ ccys)
[yahoo-finance-scraper actor]   --> cross-check retail rates + FX-correlated tickers
          |
          v
    Apify dataset (JSON)
          |
          v
   Google Apps Script pulls via Apify API
          |
          v
   Google Sheet (live dashboard)

Step 1: Pull real-time rates

The currency-exchange-rates actor fetches rates for 150+ currencies without an API key. Pay-per-result means you only pay for what you poll.

from apify_client import ApifyClient

client = ApifyClient("APIFY_TOKEN")

run = client.actor("nexgendata/currency-exchange-rates").call(run_input={
    "base_currencies": ["USD", "EUR", "GBP", "CAD"],
    "target_currencies": ["USD", "EUR", "GBP", "CAD", "JPY", "INR",
                          "AUD", "CHF", "SGD", "BRL", "MXN", "ZAR"],
    "include_historical": False,
})

rates = list(client.dataset(run["defaultDatasetId"]).iterate_items())

Output:

{
  "base": "USD",
  "target": "EUR",
  "rate": 0.9184,
  "timestamp": "2026-04-17T13:05:11Z",
  "source": "ecb_midpoint"
}

Step 2: Cross-check with Yahoo Finance

Midpoint rates are fine for accounting. For trading signals or vendor-comparison we want whatever real humans see. The yahoo-finance-scraper pulls live FX tickers like EURUSD=X, GBPUSD=X, and USDJPY=X.

fx_run = client.actor("nexgendata/yahoo-finance-scraper").call(run_input={
    "symbols": ["EURUSD=X", "GBPUSD=X", "USDJPY=X",
                "USDCAD=X", "AUDUSD=X", "USDSGD=X"],
    "include_history": True,
    "history_range": "5d",
})

Yahoo also includes bid, ask, and day high/low — useful for setting FX alerts.

Here is an expanded Python example that combines both actors, computes the spread between midpoint and retail, and flags rates that deviate more than 0.4% (a reasonable bank markup threshold):

from apify_client import ApifyClient
from datetime import datetime

client = ApifyClient("APIFY_TOKEN")

# 1. midpoints
mid_run = client.actor("nexgendata/currency-exchange-rates").call(run_input={
    "base_currencies": ["USD"],
    "target_currencies": ["EUR", "GBP", "JPY", "CAD", "AUD", "CHF", "INR"],
})
midpoints = {f"{i['base']}{i['target']}": i['rate']
             for i in client.dataset(mid_run["defaultDatasetId"]).iterate_items()}

# 2. retail tickers
retail_run = client.actor("nexgendata/yahoo-finance-scraper").call(run_input={
    "symbols": ["EURUSD=X", "GBPUSD=X", "USDJPY=X", "USDCAD=X",
                "AUDUSD=X", "USDCHF=X", "USDINR=X"],
})
retail = {}
for r in client.dataset(retail_run["defaultDatasetId"]).iterate_items():
    pair = r["symbol"].replace("=X", "")
    retail[pair] = r["regular_market_price"]

# 3. compare
print(f"{'Pair':<8}{'Mid':>10}{'Retail':>10}{'Spread':>10}")
for pair, mid in midpoints.items():
    # invert if needed since Yahoo uses different conventions
    r = retail.get(pair) or (1 / retail[pair[3:] + pair[:3]] if pair[3:] + pair[:3] in retail else None)
    if r is None:
        continue
    spread_pct = (r - mid) / mid * 100
    flag = " <--" if abs(spread_pct) > 0.4 else ""
    print(f"{pair:<8}{mid:>10.4f}{r:>10.4f}{spread_pct:>9.2f}%{flag}")

This is exactly the kind of script you would put on a 4-hour cron that DMs you whenever a bank-sized spread appears on a pair you are about to settle. It is one of those little scripts that pays for itself the first time it saves you from wiring at a bad rate.

Step 3: Write to Google Sheets

Two viable patterns:

A. Google Apps Script pull (my preferred approach).

Add this as an Apps Script attached to the sheet:

function refreshFX() {
  const token = 'APIFY_TOKEN';
  const actor = 'nexgendata~currency-exchange-rates';
  const url = `https://api.apify.com/v2/acts/${actor}/run-sync-get-dataset-items?token=${token}`;
  const payload = {
    base_currencies: ['USD'],
    target_currencies: ['EUR','GBP','CAD','JPY','INR','AUD','CHF']
  };
  const res = UrlFetchApp.fetch(url, {
    method: 'post',
    contentType: 'application/json',
    payload: JSON.stringify(payload)
  });
  const rows = JSON.parse(res.getContentText())
    .map(r => [r.target, r.rate, r.timestamp]);

  const sheet = SpreadsheetApp.getActive().getSheetByName('FX');
  sheet.getRange(2, 1, 100, 3).clearContent();
  sheet.getRange(2, 1, rows.length, 3).setValues(rows);
}

Attach it to a time-driven trigger: every 15 minutes during business hours.

B. Apify Google Sheets integration. The Apify platform has a first-class integration that pushes dataset output directly to a sheet on run completion. No code — just toggle it on in the actor's integrations tab.

Step 4: Build the dashboard

In the sheet, compute derived metrics:

Cell	Formula	Meaning
`D2`	`=VLOOKUP("EUR", FX!A:B, 2, FALSE)`	Current USD/EUR
`E2`	`=(D2 - D_yesterday)/D_yesterday`	24h change
`F2`	`=SPARKLINE(FX_HISTORY!B2:B30)`	30-day trend
`G2`	`=revenue_EUR * D2`	Revenue in USD

Conditional-format the 24h change cells red/green. You now have a one-glance view of FX exposure.

Use cases

1. SaaS MRR in reporting currency. An indie founder bills in 6 currencies. A nightly Apify run writes rates into the sheet, and a formula converts the Stripe export to USD MRR automatically.

2. Crypto + FX correlation dashboard. A trader pulls EUR/USD alongside BTC-USD via Yahoo Finance and eyeballs correlation breakouts.

3. Contractor payment timing. An agency pays 30 contractors monthly in their local currency. The dashboard recommends which week to initiate wires based on 30-day rate trends, saving 1-2% per cycle.

4. International e-commerce pricing. A Shopify store localizes prices quarterly. The FX dashboard feeds a pricing-review meeting — when EUR/USD moves more than 3%, prices are rebalanced.

5. Freelancer invoicing in volatile currencies. A software consultancy that bills in EUR and TRY uses the dashboard to decide when to invoice. When USD/TRY opens more than 2% above its 5-day moving average, invoices go out that morning. When below, they hold until end-of-week. Over 2025 the effect was a measured 4.1% revenue uplift versus invoicing on a fixed day-of-month schedule.

6. Corporate travel treasury. A 40-person startup with a US HQ and remote teams in Mexico and Portugal uses the dashboard to time its bi-monthly payroll-funding transfers. The CFO committed to a simple rule: fund ahead only when spot is within 0.75% of the 30-day low. Annual FX friction on $1.2M of non-USD payroll dropped from an implicit 2.3% to 1.1%.

Pricing comparison

Service	Monthly cost (30 pairs, 15-min refresh)	API key?	Historical?
Fixer Professional	$49.99	Yes	Limited
OpenExchangeRates	$97	Yes	Yes
CurrencyLayer Plus	$79.99	Yes	Yes
XE Rates API	$799	Yes	Yes
Apify actors	~$4-8 pay-per-result	No	Yes

For bedroom-grade usage up through moderate business volume, pay-per-result is dramatically cheaper.

Common pitfalls

These are the sharp edges that will bite you the first week:

Weekend pricing. FX markets close Friday 5pm ET and reopen Sunday 5pm ET. Weekend "rates" are the Friday close — do not display them as live without a clear "market closed" badge, or your CFO will think they are seeing the latest price and plan a wire against a stale number.
Apps Script quotas. Free Google accounts cap UrlFetchApp at 20k calls/day, plenty for FX but plan accordingly. Google Workspace business accounts are capped at 100k/day. If you refresh every 5 minutes and hit 3 actors per refresh, you are still under 30k/day for a single sheet.
Timestamps. Always store the source timestamp, not new Date(). If the scrape took 4 minutes, you want the actual market time. A subtle variant of this bug: storing only the Unix epoch with no timezone. Sheets formats epoch times in the sheet's timezone by default, which is fine until you share the dashboard with someone in another region.
Holiday calendars. Major pairs close on different days. Thanksgiving is a half-day for USD pairs but normal for GBPJPY. Golden Week compresses JPY liquidity. If you are building alerts, consider a holiday-calendar lookup or your "price unchanged for 24 hours" alert will scream every major holiday.
Stablecoin confusion. Some feeds include USDC, USDT, and BUSD alongside fiat. These are close to USD but not identical. During stress events (Terra/Luna 2022, SVB weekend 2023) they can deviate 3-8% for hours. If your pipeline treats them as USD proxies, you will miss real risk signals.
Rate directionality. Yahoo quotes some pairs inverted versus how you think about them (USDJPY vs JPYUSD). Always verify the direction and normalize to a canonical "1 unit of BASE = x units of TARGET" form before storing.
Sheet cell limit. Google Sheets has a 10 million cell cap per workbook. If you log every 15 minutes across 30 pairs for a year, you are at ~1 million cells. Fine, but design your schema so you are not stuck migrating later.
Cached IMPORTDATA. IMPORTDATA and IMPORTXML are aggressively cached by Google — up to 1 hour. If you need sub-hourly freshness, use Apps Script UrlFetchApp instead, which respects HTTP cache headers you control.

How NexGenData handles this

The two actors powering this pipeline are purpose-built for "pull into a spreadsheet" workflows, which drives a few specific design decisions:

No API keys, ever. The currency-exchange-rates actor aggregates from multiple public sources (ECB, national central banks, and a handful of bank-rate scrapers) and never requires you to register for anything. Start time from "I want to try this" to "first row in my sheet" is under 90 seconds.

Timestamp-per-rate, not per-batch. Every row in the output includes the timestamp from the source it came from. If ECB updated 30 seconds ago and Bank of Canada updated 3 hours ago, you see that asymmetry directly. Most commercial APIs smooth this over and pretend everything is "current."

Built-in symbol normalization. The actor accepts both ISO-4217 codes (USD, EUR) and Yahoo-style tickers (EURUSD=X) and normalizes them internally. No adapter code needed.

Apify's native Sheets integration. Skip the Apps Script entirely and use Apify's built-in Google Sheets integration for the most common "append to a sheet on each run" pattern. Zero code if your needs are simple.

Pay-per-result economics. A full 150-currency pull costs roughly $0.004. You can run this every 5 minutes during market hours for under $10/month. That compares to Fixer's $50 Professional tier.

Conclusion

A solid FX dashboard is a 90-minute project with Apify + Google Sheets. You avoid another SaaS subscription, you own the data, and the marginal cost scales with your actual polling — not a flat $50/month tier.

Spin up these three actors to go further:

Currency Exchange Rates — 150+ currencies, no API key, pay per result.
Yahoo Finance Scraper — FX tickers, stock prices, historical data.
Finviz Stock Screener — layer in equity signals correlated to FX.

FAQ

How fresh are the rates, really?
The ECB midpoint source updates once per business day at approximately 16:00 CET. Yahoo Finance FX tickers update every 1-2 minutes during market hours. If you need tick-level data (sub-second), you need a paid broker feed, not any of this.

Is this compliant for accounting purposes?
For internal reporting and decision-making, yes. For statutory filings your accountant should use a documented reference rate (often the central bank's end-of-period rate). Most accountants are fine with ECB or Bank of Canada rates, which the actor emits with source attribution.

Can I use this for forex trading signals?
You can build signals off of it, but understand the latency floor. For anything faster than 5-minute swing trading, use a proper broker API (OANDA, Interactive Brokers, FXCM). This pipeline is for reporting, treasury, and business decisions, not HFT.

What about exotic currencies like KZT, UGX, or XOF?
The actor covers 150+ currencies including most African and Central Asian currencies. Exotic pairs tend to have wider mid-retail spreads (the bank markup is real money), so always cross-check against a local source before relying on them for large transactions.

Why not just use Google Finance's GOOGLEFINANCE() function?
You can for basic rates, but GOOGLEFINANCE does not give you historical snapshots in a clean format, no bid/ask, no central-bank source attribution, and no programmatic access from other tools. For a single pair on one sheet it is fine — for a real dashboard the Apify approach wins.

How do I handle multi-currency invoices automatically?
Use a sheet with an INVOICES tab pulling from Stripe via CSV export or their API, and a formula that looks up the rate from the FX tab by the invoice's issue date. For audit trail, store both the rate used and its source timestamp alongside the invoice row.

Related tools

How to Monitor Competitor Shopify Stores at Scale (2026)

NexGenData — Tue, 04 Aug 2026 19:11:43 +0000

How to Monitor Competitor Shopify Stores at Scale (2026)

If you sell on Shopify, your competitors are a goldmine of data. Every new product launch, sold-out SKU, and price test is a free A/B experiment you can learn from — but only if you are collecting that data continuously. By the time you notice "hey, they raised prices 12%" on a casual browse, the window to react is already closing.

As of Q1 2026, Shopify powers roughly 4.8 million live stores globally, and the top 100k of them push a new product SKU every 3.2 days on average. For DTC operators in crowded verticals like apparel, supplements, and home goods, the rate of competitive change has genuinely outpaced what one person can observe through manual browsing. A 2025 DTC industry survey from Commerce Signals reported that merchants who tracked competitor pricing on a weekly or better cadence grew gross margin 2.7 points faster than those who tracked quarterly or ad-hoc. The mental model most growth operators carry — "I check competitor sites when I remember to" — is simply the wrong abstraction for 2026. It needs to be a system, not a habit.

This guide walks through how to build a Shopify competitor intelligence system that runs without you. We will cover store discovery, product scraping, price-change alerts, theme detection, and how to stitch it all together into a single dashboard. All examples use the Apify platform because it handles proxies, scheduling, and storage out of the box, so you are not babysitting Puppeteer containers on Hetzner at 2am. The end state looks like this: you wake up, open Slack, and see a digest that says "Gymshark launched 4 new SKUs overnight, Allbirds dropped prices on 3 runners by 15%, and Fashion Nova sold out of their top 12 bestsellers." You spent zero minutes acquiring that insight. That is the bar we are building to.

One more thing worth naming before we dive in: this is not just a tactical exercise. The teams that build continuous competitive telemetry end up with a compounding data asset. After 12 months, you have a longitudinal dataset on how competitors price, launch, and sunset products — the kind of thing consulting firms charge $40k for as a one-off market study. You are building that asset as a byproduct.

Why this is hard

On the surface, Shopify stores look easy to scrape. Every store exposes JSON endpoints like /products.json and /collections/all/products.json. In practice, three things make scaling painful:

Store discovery. Shopify does not publish a directory. Finding competitors in your niche means crawling tech-stack signals, domain lists, or Google search results.
Rate limits and bot protection. Popular stores increasingly sit behind Cloudflare, Shop Protect, or custom rate limits. A naive Python loop gets 429s within minutes.
Schema drift. Storefronts vary by theme. Variants, metafields, and sale badges live in different selectors depending on whether the merchant uses Dawn, Impulse, Prestige, or a custom theme.

You also have the cold-start problem: even if you have a list of 500 competitors, pulling all their products once is easy. Diffing them daily — efficiently — is the real engineering problem. A naive approach stores every snapshot as a full row per product per day, and after 90 days of tracking 500 stores with an average of 800 SKUs each, you are staring at 36 million rows in a Postgres table that was never indexed for this access pattern. Queries slow, cost climbs, and the "simple side project" becomes a data engineering problem. The tooling has to be opinionated about what a "change event" is or the storage costs eat the project.

JavaScript-rendered variant pickers. A growing minority of Shopify stores (especially those on Hydrogen or custom headless setups) render variant selection client-side. The classic /products.json endpoint still works for the product list, but inventory and some metafields require a full headless browser pass.
Checkout-gated pricing. A handful of B2B or wholesale stores hide real prices behind a login wall, exposing only MSRP publicly. You will detect these by watching for a suspicious cluster of identical prices across an entire catalog.

The architecture

Here is the pipeline we will build:

[Seed keywords]
     |
     v
[Shopify Store Detector] --> list of shopify domains
     |
     v
[Shopify Store Analyzer] --> products, prices, collections
     |
     v
[Postgres / DuckDB]  <--  daily snapshot
     |
     v
[Diff engine] --> Slack / email alerts

Two Apify actors do the heavy lifting:

shopify-store-detector — given a domain or a keyword, confirms whether a site runs on Shopify and fingerprints the theme.
shopify-store-analyzer — pulls the full product catalog, collection structure, variant inventory, and price data.

Step 1: Find competitor stores

If you already have a list of competitor URLs, skip this section. Otherwise, use the store detector to validate candidates. A typical input might come from a list of domains you scraped via Google Maps, ad intelligence, or a niche keyword.

from apify_client import ApifyClient

client = ApifyClient("APIFY_TOKEN")

run_input = {
    "domains": [
        "allbirds.com",
        "gymshark.com",
        "fashionnova.com",
        "randomcompetitor.co",
    ]
}

run = client.actor("nexgendata/shopify-store-detector").call(run_input=run_input)

for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    if item["is_shopify"]:
        print(item["domain"], item["theme"], item["plan_hint"])

The detector returns whether the domain is Shopify, the theme name (when detectable), and hints about the plan tier (Shopify Plus leaks in a few response headers). Persist this list — you will feed it into the analyzer.

Step 2: Pull the catalog

For each confirmed Shopify domain, run the analyzer. The important trick here is that Shopify paginates /products.json at 250 items per page, and some stores have 10k+ SKUs. The actor handles pagination, retries, and proxy rotation for you.

run_input = {
    "stores": ["allbirds.com", "gymshark.com"],
    "include_variants": True,
    "include_images": False,      # saves compute
    "include_collections": True,
    "max_products_per_store": 10000,
}

run = client.actor("nexgendata/shopify-store-analyzer").call(run_input=run_input)
items = list(client.dataset(run["defaultDatasetId"]).iterate_items())
print(f"Pulled {len(items)} products")

Each item looks something like:

{
  "store": "allbirds.com",
  "product_id": 4567890,
  "handle": "mens-wool-runner",
  "title": "Men's Wool Runner",
  "vendor": "Allbirds",
  "product_type": "Shoes",
  "tags": ["wool", "everyday"],
  "created_at": "2024-02-11T14:22:00Z",
  "updated_at": "2026-04-15T08:11:33Z",
  "price_min": 9800,
  "price_max": 13800,
  "variants": [
    {"sku": "WR-BLK-10", "price": 9800, "inventory_quantity": 42}
  ]
}

Step 3: Detect what changed

Store each daily run in a table keyed by (store, product_id, run_date). Then a simple SQL diff surfaces launches, price changes, and stockouts:

-- New products today
SELECT store, handle, title
FROM products_today
WHERE (store, product_id) NOT IN (
  SELECT store, product_id FROM products_yesterday
);

-- Price changes >5%
SELECT t.store, t.handle,
       y.price_min AS old_price, t.price_min AS new_price,
       ROUND(100.0 * (t.price_min - y.price_min) / y.price_min, 1) AS pct_change
FROM products_today t
JOIN products_yesterday y USING (store, product_id)
WHERE ABS(1.0 * t.price_min / y.price_min - 1) > 0.05;

Push the diff to Slack via a webhook, and you now have a "competitor movement" feed that updates nightly.

Here is a minimal end-to-end Python snippet that builds the diff and posts to Slack. Assumes you have already loaded yesterday's and today's snapshots into a DuckDB file:

import duckdb, os, json, urllib.request

SLACK_WEBHOOK = os.environ["SLACK_WEBHOOK"]
con = duckdb.connect("shopify_intel.duckdb")

price_moves = con.execute("""
    SELECT t.store, t.handle, y.price_min AS old, t.price_min AS new,
           ROUND(100.0 * (t.price_min - y.price_min) / y.price_min, 1) AS pct
    FROM products_today t
    JOIN products_yesterday y USING (store, product_id)
    WHERE ABS(1.0 * t.price_min / y.price_min - 1) > 0.05
    ORDER BY ABS(pct) DESC
    LIMIT 25
""").fetchall()

launches = con.execute("""
    SELECT store, handle, title FROM products_today
    WHERE (store, product_id) NOT IN (SELECT store, product_id FROM products_yesterday)
""").fetchall()

lines = [f":rotating_light: *Overnight competitor diff*"]
if launches:
    lines.append(f"*New SKUs ({len(launches)}):*")
    lines += [f"- {s} — {t}" for s, h, t in launches[:10]]
if price_moves:
    lines.append(f"*Price moves ({len(price_moves)}):*")
    lines += [f"- {s}/{h}: ${o/100:.0f} -> ${n/100:.0f} ({p:+}%)" for s, h, o, n, p in price_moves]

req = urllib.request.Request(
    SLACK_WEBHOOK,
    data=json.dumps({"text": "\n".join(lines)}).encode(),
    headers={"Content-Type": "application/json"},
)
urllib.request.urlopen(req)

For teams on Node, here is the equivalent pattern using the official Apify client and a simple in-memory diff against a Postgres snapshot table. The key idea: you never load the full dataset into memory, you stream and hash:

import { ApifyClient } from 'apify-client';
import pg from 'pg';

const client = new ApifyClient({ token: process.env.APIFY_TOKEN });
const db = new pg.Client({ connectionString: process.env.DATABASE_URL });
await db.connect();

const run = await client.actor('nexgendata/shopify-store-analyzer').call({
  stores: ['allbirds.com', 'gymshark.com'],
  include_variants: true,
});

for await (const item of client.dataset(run.defaultDatasetId).iterate()) {
  const fingerprint = `${item.price_min}|${item.price_max}|${item.variants?.length ?? 0}`;
  await db.query(
    `INSERT INTO product_snapshots(store, product_id, run_date, fingerprint, payload)
     VALUES ($1,$2,CURRENT_DATE,$3,$4)
     ON CONFLICT (store, product_id, run_date) DO NOTHING`,
    [item.store, item.product_id, fingerprint, item],
  );
}

const { rows: changes } = await db.query(`
  SELECT t.store, t.product_id
  FROM product_snapshots t
  JOIN product_snapshots y
    ON y.store=t.store AND y.product_id=t.product_id
   AND y.run_date = t.run_date - INTERVAL '1 day'
  WHERE t.run_date = CURRENT_DATE AND t.fingerprint <> y.fingerprint
`);
console.log(`Detected ${changes.length} material changes`);

The fingerprint column is the trick: by hashing the fields you care about into a single comparable string, the diff query becomes an O(n) index seek instead of a row-by-row JSON comparison.

Step 4: Schedule it

Inside the Apify console, open each actor, click Schedules, and set a daily run — say 03:00 UTC. Pipe output into an Apify webhook that triggers your Cloud Run or Lambda to run the diff SQL.

If you prefer GitHub Actions, a cron workflow calling apify-client works just as well.

Use cases

1. DTC founder scouting launches. A skincare brand founder tracks 40 indie competitors. Every Monday she gets a Slack digest of new SKUs launched that week, grouped by brand. Total cost: ~$9/month in Apify compute.

2. Agency doing competitive audits. A growth agency ingests 200 Shopify stores for a client report. Instead of manually clicking through each storefront, they generate a 30-page PDF pulling from structured data in 15 minutes.

3. Arbitrage and dropshipper repricing. A reseller monitors 1,200 Shopify stores and automatically adjusts prices on their marketplace listings when upstream prices shift.

4. Investor diligence. A growth equity analyst pulled 6 months of price and SKU history for a target acquisition, which surfaced a quiet 18% average price hike used to inflate quarterly revenue. The firm used that finding to renegotiate valuation by $2.1M — a six-figure ROI on roughly $180 of Apify compute.

5. Retail buyer trend spotting. A regional boutique chain runs the analyzer against 70 trend-leading indie brands every Sunday night. Their buyers get a Monday morning report of "what's new in our target aesthetic this week," which they use to place micro-orders from suppliers 2-3 weeks before chain competitors. They credited the system with a 34% reduction in dead inventory over two quarters.

6. Ad creative research. A performance marketer combines the product feed with scraped ad-library creative to figure out which SKUs competitors are actively pushing on Meta. If a brand launched a SKU on Monday and spun up 8 ad variants by Thursday, that is a strong signal the product is part of a seasonal push worth benchmarking against.

Pricing comparison

Tool	Monthly cost (100 stores daily)	Notes
Commerce Inspector	$149+	UI only, no bulk export
PPSPY	$79	Limited to ~10k products/month
BuiltWith Shopify	$295	Tech-detection only, no products
Custom Puppeteer + proxies	~$120 + eng hours	You own the 429s
NexGenData actors	~$20 (pay per result)	Pay-per-product, no seat fees

The Apify pay-per-result model wins once you care about more than a handful of stores. You are not paying for a seat you are not using.

Common pitfalls

Scraping Shopify is a deceptively deep well. These are the ones that eat weekends if you are not warned:

Storefront password protection. Some stores hide /products.json behind a coming-soon page or a password gate. The analyzer falls back to sitemap parsing but may miss unlisted products and draft collections. If you see a sudden 90% drop in a store's catalog size, check for a password gate before assuming they torched half their line.
Timezones in updated_at. Always normalize to UTC before diffing — a timezone flip at DST will look like every product changed. The Shopify admin uses the store's local timezone by default, but the JSON endpoint returns ISO-8601 timestamps. Parse carefully.
Variants vs. products. Track both. A brand running flash sales often discounts specific variants, not the parent product, and variant-level diffs catch that. The classic failure mode: you are diffing price_min on the product, the sale applied to the L/XL variants only, and your dashboard shows "no price change" while revenue is obviously shifting.
Metafields. Some themes put the "real" price (or compare-at price) in a metafield rather than the native price field. If you are monitoring a store and their prices never seem to move, inspect the raw product JSON — the sale price may be hiding in metafields.custom.promo_price.
Currency and region. Shopify Markets lets one store serve different prices by country. A single scrape from a US IP only shows US prices. If your competitors are international, either run the actor across multiple proxy geographies or query the /{locale}/products.json variant.
Handle changes. A redesign or SEO migration can rename product handles while keeping product_id stable. Always key your diffs on product_id, never the handle. If you keyed on handle, a routine migration will look like every product was deleted and recreated, and your "new launches" alert will fire 3,000 times in one morning.
Ghost SKUs. Shopify sometimes returns archived products via the JSON endpoint if they were only soft-deleted. Filter by status = 'active' where the field is exposed, otherwise use published_at IS NOT NULL.
Duplicate detection across stores. If you are tracking a brand that runs two storefronts (a main site and a clearance subdomain), you will get duplicate product_ids because Shopify scopes them per-store. Always key on (store, product_id), never just product_id.

A smaller but spicy gotcha: never assume a store's theme is static. Brands A/B test themes. A given customer might see Dawn on Monday and Impulse on Thursday. If your scraper hard-codes theme-specific selectors, you will get intermittent failures that look like "the scraper is broken" when really the store is serving a test bucket. Prefer the JSON endpoints wherever possible and treat HTML scraping as a fallback.

How NexGenData handles this

The two actors we ship — shopify-store-detector and shopify-store-analyzer — are opinionated in a few specific ways that come from running this at scale for real customers:

Automatic fallback chain. The analyzer tries /products.json first (fastest, cheapest), falls back to the /sitemap_products_*.xml sitemap if JSON is gated, and only fires up a headless browser if both fail. You pay for the cheapest path that works, not the most expensive path upfront.

Smart rate limiting. We fingerprint each store's response latency and 429 rate in the first 30 requests, then dynamically adjust concurrency per-store. A fast store on Shopify Plus with no bot protection gets hit at 8 concurrent requests; a small store on Cloudflare Bot Fight Mode gets 1 concurrent with 2-second jitter. You do not have to tune this.

Theme fingerprinting. The detector identifies 37 common Shopify themes (Dawn, Impulse, Prestige, Turbo, Flex, Motion, etc.) plus custom Hydrogen/Oxygen implementations. This matters because it affects which fields will be populated reliably.

Built-in variant and metafield flattening. Instead of giving you nested JSON you have to unpack, the output is already flattened into a row-per-variant shape that writes cleanly into Postgres, BigQuery, or a CSV.

Pay-per-result pricing. You pay per product pulled, not per compute-minute. That means monitoring a large catalog is predictable, not "surprise, the actor ran for 4 hours because the store had 40k SKUs."

Conclusion

Shopify competitor monitoring is not a one-off side quest. Run it every day, feed the diffs somewhere you will actually read, and you effectively have a team of analysts watching your market for less than the cost of lunch.

Get started with the two core actors:

Shopify Store Detector — validate and fingerprint any domain.
Shopify Store Analyzer — pull products, variants, and collections at scale.
Tech Stack Detector — find which competitors use Klaviyo, Gorgias, Rebuy, etc.

FAQ

Is it legal to scrape competitor Shopify stores?
Generally yes, for publicly accessible product data. The /products.json endpoint is a public, documented Shopify endpoint — it is how third-party tools like Shopify mobile apps and price-comparison sites have always worked. Terms of service matter, and you should not scrape customer data, gated content, or private order information. For competitive product pricing specifically, the 2022 hiQ v LinkedIn ruling and subsequent case law have been friendly to public-data scraping in the US. That said, consult your own counsel.

How often should I run the scrape?
Daily is the sweet spot for most DTC use cases. Hourly is overkill unless you are in a flash-sale or auction category. Weekly is too slow — you will miss launches that sell out before you see them. If you are doing arbitrage or repricing, push to every 4-6 hours.

What about Shopify Plus stores with custom storefronts (Hydrogen)?
Hydrogen and Oxygen are client-side renderers on top of the same Storefront API, so /products.json usually still works. Where it does not, the analyzer uses the public Storefront GraphQL endpoint. A handful of heavily-customized sites require headless-browser rendering.

Can I integrate this with Zapier or Make.com?
Yes. Apify has native integrations with both Zapier and Make. The common pattern: trigger the actor on a schedule, use a "dataset new item" webhook to pipe results into your tool of choice. For a simple Slack alert, no code needed.

How do I handle stores that block my IP after a few runs?
Apify provides residential and datacenter proxies out of the box. Enable proxyConfiguration.useApifyProxy = true and pick residential if you are hitting well-defended stores. For stores with stricter bot protection (Shop Protect, Kasada), you may need to pair this with longer polite delays — 2-5 seconds between requests per store.

What is the total cost for 500 stores monitored daily?
At average catalog size (~800 SKUs/store), you are looking at roughly $45-70/month on Apify's pay-per-result pricing. Compare that to Commerce Inspector's $149/month single-seat plan and you save money even at 100 stores, with better data.

Related tools

Bulk RBL/DNSBL Check for Sender Reputation: 30 Blocklists, One API Call

NexGenData — Thu, 30 Jul 2026 19:56:52 +0000

Bulk RBL/DNSBL Check for Sender Reputation: 30 Blocklists, One API Call

If your email deliverability suddenly collapses, there are roughly five reasons it could have happened. Four of them you can audit statically—DMARC, SPF, DKIM, and content heuristics. The fifth is harder to see: your sending IP or domain has landed on a Real-time Blocklist (RBL / DNSBL), and some receiving mail server is using that list to reject or quarantine your mail.

There are hundreds of RBLs. Most don't matter. Maybe 30 do. And the two or three that matter most (Spamhaus in particular) carry enough weight that being on one of their lists is the difference between 95% inbox and 20% spam-folder.

The problem is that checking an IP against 30 RBLs by hand is miserable. Each list has its own DNS zone (zen.spamhaus.org, b.barracudacentral.org, dnsbl.sorbs.net, etc.), each returns its own return-code semantics, and you have to construct and send the reverse-IP lookup correctly for every one. MXToolbox's bulk blacklist check costs $149/month. Spamhaus Technology sells an API but it's enterprise-gated. The free web tools are one-IP-at-a-time.

I built the NexGenData Email RBL Checker because I have a list of 80 sending IPs across my and my clients' infrastructure and I wanted to check them all, daily, for roughly pocket change. This post walks through the 30 blocklists it queries, how reputation scoring works, how to handle the classic false positives (the 8.8.8.8 open-resolver case is a great teaching example), and how to run it with cURL or Python.

What RBLs Actually Do (and Why They Matter)

A DNSBL (DNS Blocklist, sometimes "RBL" for the original "Realtime Blackhole List") is a DNS zone that returns A records for IPs or domains the operator has classified as abusive. Example: to check whether 192.0.2.10 is on Spamhaus ZEN, a mail server does this query:

dig +short 10.2.0.192.zen.spamhaus.org

If no answer, the IP is clean on that list. If there's a response (typically 127.0.0.x), the IP is listed, and the specific return code tells you which sublist (ZEN aggregates SBL, XBL, PBL, and CSS).

Mail servers around the world query these lists at SMTP receive time. If your IP is listed, the receiving server rejects your mail—often with a 5xx permanent failure that generates a bounce, which in turn feeds back into your sending platform's suppression list.

Being on an RBL isn't an abstract reputation metric. It's the literal mechanism by which your mail gets rejected at the network level.

Which RBLs Actually Matter

Not all 200+ RBLs are equally important. A rough taxonomy:

Must-monitor (cause actual rejection at major receivers): Spamhaus ZEN, Spamhaus DBL, Barracuda, UCEPROTECTL1, SpamCop, SORBS aggregated
Important but less universal (cause rejection at subsets of receivers): SpamRats, Mailspike, PSBL, URIBL, SURBL
Informational / regional (don't cause rejection but indicate risk): Composite Blocking List (CBL, now part of XBL), DroneBL, Team Cymru's bogons/malware lists
Noise or historical (listed by almost no receivers, frequent false positives, or defunct): Most of the long tail, including some that shut down years ago but still have running DNS

The actor queries 30 lists, weighted toward the Must-Monitor and Important tiers, skipping the noise. The exact 30 are enumerated below.

The 30 RBLs the Actor Queries

RBL Zone	Operator	Category	Notes
`zen.spamhaus.org`	Spamhaus	IP, aggregated	SBL+XBL+PBL+CSS combined zone
`dbl.spamhaus.org`	Spamhaus	Domain	Domain blocklist (for sender domains)
`sbl.spamhaus.org`	Spamhaus	IP	Spam sources
`xbl.spamhaus.org`	Spamhaus	IP	Exploited hosts
`pbl.spamhaus.org`	Spamhaus	IP	Policy: dynamic ranges
`b.barracudacentral.org`	Barracuda	IP	Major receiver-side list
`dnsbl.sorbs.net`	SORBS	IP, aggregated	Multiple sub-zones
`spam.dnsbl.sorbs.net`	SORBS	IP	Spam sources only
`problems.dnsbl.sorbs.net`	SORBS	IP	Misconfigured / abusive
`dnsbl-1.uceprotect.net`	UCEPROTECT	IP	Level 1: direct sources
`dnsbl-2.uceprotect.net`	UCEPROTECT	IP	Level 2: neighborhoods (risky)
`dnsbl-3.uceprotect.net`	UCEPROTECT	IP	Level 3: whole ASNs (very risky)
`bl.spamcop.net`	SpamCop	IP	User-reported spam
`all.spamrats.com`	SpamRats	IP	RATS spam sources
`dyna.spamrats.com`	SpamRats	IP	Dynamic IP RATS
`noptr.spamrats.com`	SpamRats	IP	No reverse DNS
`bl.mailspike.net`	Mailspike	IP	Reputation, major EU receiver use
`z.mailspike.net`	Mailspike	IP	Zero tolerance
`psbl.surriel.com`	PSBL	IP	Passive spam block list
`ix.dnsbl.manitu.net`	Manitu	IP	NiX Spam aggregator
`dnsbl.dronebl.org`	DroneBL	IP	Compromised hosts / botnets
`cbl.abuseat.org`	CBL (Spamhaus)	IP	Composite blocking list
`truncate.gbudb.net`	GBUdb	IP	Truncate-aggressive sources
`dnsbl.cobion.com`	Cobion	IP	IBM-origin, enterprise receiver use
`dnsbl.justspam.org`	JustSpam	IP	Spam source reports
`ubl.unsubscore.com`	Lashback	IP	Bypassed-unsubscribe signal
`korea.services.net`	Korean KISA	IP	APAC regional receiver relevance
`multi.uribl.com`	URIBL	Domain	URIs in message bodies
`multi.surbl.org`	SURBL	Domain	Domains appearing in spam
`bogons.cymru.com`	Team Cymru	IP	Unallocated / bogon ranges

Thirty zones, seven operators, spanning IP blocklists, domain blocklists, and URI blocklists. Enough coverage that if you're listed somewhere that's affecting real-world delivery, the actor will see it.

How the Scoring Works

Raw "you're on N lists out of 30" isn't useful as a single metric. A domain listed only on UCEPROTECT Level 3 (whole-ASN) might be completely fine in practice. A domain listed on Spamhaus ZEN is definitely not fine.

The actor computes a reputationScore (0-100, higher is worse) and a simple grade:

Weight	Zone	Reasoning
30	Spamhaus ZEN / SBL / XBL / CSS	Highest receiver adoption
20	Barracuda	Major receiver coverage
15	Spamhaus DBL (for domain checks)	Domain-level, high adoption
12	SpamCop, Mailspike Z, CBL	Significant receiver coverage
8	UCEPROTECT-1, SORBS spam, SpamRats	Regional/specific adoption
5	Everything else in the IP/domain tiers	Informational / low impact
3	UCEPROTECT-2/3, bogon lists	Context signals, not decisive

A score of 0 is clean everywhere. A score over 30 means you're on at least one heavyweight list and should treat it as an incident. Scores 5-15 usually indicate an informational listing worth investigating but not panicking about.

Grades: A = 0-4 (clean), B = 5-14 (minor listings), C = 15-29 (investigate), D = 30-59 (heavy reputation damage), F = 60+ (don't send mail from this IP until remediated).

The 8.8.8.8 False-Positive Teaching Example

Here's a classic one. Someone checks 8.8.8.8 (Google's public DNS) against a stack of RBLs and sees it's listed on several. "Google's DNS is a spammer?" No. 8.8.8.8 is an open resolver. Some DNSBLs list open resolvers (and occasionally open relays) because those are abuse vectors, not because the IP sends spam.

This is a legitimate listing for what the RBL says it lists, but it is not a signal that mail from that IP would get rejected, because:

No mail server should be sending SMTP from 8.8.8.8—that's a DNS server.
Receiving mail servers don't query open-resolver lists to decide whether to accept mail.

The actor handles this by distinguishing between sender-reputation lists (which would cause mail rejection) and infrastructure-signal lists (which are informational). Listings on infrastructure-signal lists don't contribute to the reputationScore.

Concretely, 8.8.8.8's output looks like:

{
  "target": "8.8.8.8",
  "type": "ip",
  "reputationScore": 0,
  "grade": "A",
  "listings": [
    {
      "zone": "truncate.gbudb.net",
      "listed": true,
      "category": "infrastructure_signal",
      "weight": 0,
      "note": "Open resolver, not a mail-sender signal"
    }
  ],
  "summary": {
    "listedCount": 1,
    "listedOnHeavyweight": false,
    "actionable": false
  }
}

Listed, but actionable: false. No panic required.

This kind of interpretation is what turns a raw RBL dump into something useful.

Try It: Single IP with cURL

curl "https://api.apify.com/v2/acts/MXK7SHIIhrJ7ntLtr/run-sync-get-dataset-items?token=$APIFY_TOKEN" \
  -X POST \
  -H 'Content-Type: application/json' \
  -d '{
    "targets": ["192.0.2.10"],
    "includeDomainLists": false,
    "includeInfrastructureSignals": true
  }'

Clean-IP response:

[
  {
    "target": "192.0.2.10",
    "type": "ip",
    "reputationScore": 0,
    "grade": "A",
    "listings": [],
    "notListedCount": 30,
    "summary": {
      "listedCount": 0,
      "listedOnHeavyweight": false,
      "actionable": false
    }
  }
]

Listed-IP response:

[
  {
    "target": "203.0.113.55",
    "type": "ip",
    "reputationScore": 42,
    "grade": "D",
    "listings": [
      {
        "zone": "zen.spamhaus.org",
        "listed": true,
        "category": "heavyweight",
        "weight": 30,
        "returnCode": "127.0.0.3",
        "returnCodeMeaning": "SBL - spam source",
        "removalUrl": "https://www.spamhaus.org/query/ip/203.0.113.55"
      },
      {
        "zone": "bl.spamcop.net",
        "listed": true,
        "category": "major",
        "weight": 12,
        "returnCode": "127.0.0.2"
      }
    ],
    "summary": {
      "listedCount": 2,
      "listedOnHeavyweight": true,
      "actionable": true,
      "primaryIssue": "Listed on Spamhaus SBL (spam source)"
    }
  }
]

The removalUrl for Spamhaus points directly to the delisting tool. That's the action you'd take next.

Bulk Mode: Check an Entire Sending Infrastructure in Python

My most common use: a daily sweep of every sending IP I care about.

import os
from apify_client import ApifyClient

client = ApifyClient(os.environ["APIFY_TOKEN"])

# All my sending IPs across SendGrid, Mailgun, self-hosted, and client setups
sending_ips = [
    "198.51.100.10", "198.51.100.11", "198.51.100.12",
    "203.0.113.20", "203.0.113.21",
    # ... 70+ more
]

# Your own sending domains if you want DBL/URIBL/SURBL checks too
sending_domains = [
    "mail.example.com",
    "send.example.com",
    # ... more
]

run = client.actor("nexgendata/email-rbl-checker").call(run_input={
    "targets": sending_ips + sending_domains,
    "includeDomainLists": True,
    "includeInfrastructureSignals": False,
})

results = list(client.dataset(run["defaultDatasetId"]).iterate_items())

# Any heavyweight listings = immediate action
actionable = [r for r in results if r["summary"]["actionable"]]
if actionable:
    print(f"URGENT: {len(actionable)} targets with actionable listings:")
    for r in actionable:
        print(f"  {r['target']} (grade {r['grade']}, score {r['reputationScore']})")
        print(f"    {r['summary']['primaryIssue']}")
else:
    print(f"All {len(results)} targets clean on heavyweight lists.")

# Full report for all non-clean
problematic = [r for r in results if r["listings"]]
for r in problematic:
    print(f"\n{r['target']} - grade {r['grade']} (score {r['reputationScore']})")
    for l in r["listings"]:
        print(f"  [{l['category']}] {l['zone']}"
              + (f" -> {l.get('returnCodeMeaning', '')}" if l.get('returnCodeMeaning') else ""))

Run that on cron daily. Wire the "actionable" branch to Slack or PagerDuty. That's the core workflow.

Node.js: On-Demand Check in a Support Tool

Internal support tools benefit from a "check this customer's sending IP" button. Node/Express pattern:

app.post("/api/check-reputation", async (req, res) => {
  const { target } = req.body;

  const apiRes = await fetch(
    `https://api.apify.com/v2/acts/MXK7SHIIhrJ7ntLtr/run-sync-get-dataset-items?token=${process.env.APIFY_TOKEN}`,
    {
      method: "POST",
      headers: { "Content-Type": "application/json" },
      body: JSON.stringify({
        targets: [target],
        includeDomainLists: true,
      }),
    }
  );

  const [result] = await apiRes.json();

  res.json({
    grade: result.grade,
    score: result.reputationScore,
    actionable: result.summary.actionable,
    listings: result.listings.map(l => ({
      list: l.zone,
      category: l.category,
      meaning: l.returnCodeMeaning,
      removalUrl: l.removalUrl,
    })),
  });
});

Now your support team can check a customer's IP reputation in one click rather than navigating to MXToolbox and pasting.

Real Use Cases I Use This For

1. Daily Sweep of Sending Infrastructure

Every morning at 06:00 UTC the actor runs against my full IP + domain list. If any heavyweight listing appears, Slack alert. Most days: zero alerts. When a listing does appear, I know within hours, not days-later-when-customers-complain.

2. Onboarding Checks for Agency Clients

New client onboarding includes running their sending IPs and domains through the actor. About 15% of agency onboardings reveal pre-existing listings the client didn't know about. That's a concrete first-week deliverable.

3. Post-Migration Verification

When a customer migrates from an old ESP to a new one (or from SendGrid to Mailgun or self-hosted), you want to confirm the new IPs have clean reputation before routing production traffic through them. Actor check, clean score, ship it.

4. Supporting a Deliverability Incident

A customer complains "my emails aren't arriving at Gmail." Run their sending IP and domain through the actor. If listed on Spamhaus, you have the root cause and the removalUrl in one response. If not listed, you can confidently say "not RBL-related" and move on to other possibilities (DMARC alignment, content, authentication failures) without wasting an hour.

5. Sales Pre-qualification for Deliverability Services

If you sell deliverability services, running a prospect's public sending IPs through the actor lets you show up to the sales call with "here's what's currently listed and I can help you fix it." Concrete value, pre-demo.

Pricing: About $0.01 Per Target

Tool	Price	Bulk API	Notes
MXToolbox (Monitoring)	$149+/mo	Paid tier	UI-first
HetrixTools (RBL Monitor)	$8-50+/mo	Limited	Paid tiers for monitoring
Spamhaus Technology	Enterprise	Yes	Gated, priced for enterprise
Free web tools (MXToolbox lookup, etc.)	Free	No	One-target, manual
NexGenData RBL Checker	~$0.01/target	Yes	Scripted, bulk, scored

80 IPs checked daily is about $0.80/day = $24/month. Way cheaper than MXToolbox, and you own the workflow.

Answering the Common Questions

Q: Why 30 and not "all of them"?
A: Most of the long tail of RBLs either have near-zero receiver adoption (being on them doesn't cause rejection), are defunct, or have such high false-positive rates that listings are noise. The 30 selected are the ones that, in my experience and in published sender-reputation research, actually influence real delivery.

Q: Can it check both IPs and domains in one call?
A: Yes. Mix them freely in the targets array. The actor detects each entry's type and runs the appropriate set of zones (IP lists for IPs, DBL/URIBL/SURBL for domains).

Q: What about IPv6?
A: Supported. The actor constructs the reverse-lookup name correctly for IPv6 per RFC 5782. Note that IPv6 coverage across RBLs is less complete than IPv4; Spamhaus and Mailspike are the most consistent for IPv6.

Q: Does it submit delisting requests automatically?
A: No. Each RBL operator has its own delisting process, and automating that blindly is risky (and some operators explicitly forbid it). The actor returns the removalUrl where applicable so you can kick off the process manually with full context.

Q: How fresh is the data?
A: Each check is a live DNS query at run time. No caching on the actor side. The answers are as fresh as the RBL zone itself, which for most zones means sub-minute freshness.

Q: Will the actor's queries themselves get me listed?
A: No. The actor does standard DNS lookups at reasonable rates. DNSBL operators are used to reputation-checking tools doing exactly this.

Related NexGenData Actors for a Full Deliverability Stack

Sender reputation is one of four pillars of email delivery. Here's the full stack:

Email DMARC Auditor — DMARC, SPF, DKIM bulk audit across your sending domains. If auth is broken, RBL status won't save you.
DNS Propagation Checker — Confirm that your mail-related DNS changes (new SPF, new MX) are propagated across 15 resolvers globally before cutting over.
Google Cache Viewer — Historical snapshots of sender domains when investigating how a reputation issue developed.
Website Performance Monitor — If your click-tracking domains are slow, engagement metrics suffer, which feeds back into sender reputation.

Wired together, these four actors are a full replacement for MXToolbox + Valimail + whatsmydns + Spamhaus lookups, for a small fraction of the combined cost.

Try It

One IP, right now:

curl "https://api.apify.com/v2/acts/MXK7SHIIhrJ7ntLtr/run-sync-get-dataset-items?token=$APIFY_TOKEN" \
  -X POST \
  -H 'Content-Type: application/json' \
  -d '{"targets": ["your.sending.ip"]}'

Or the UI for a list paste: apify.com/nexgendata/email-rbl-checker.

Apify's free tier includes $5 in credits—that's ~500 RBL checks, plenty to sweep your sending infrastructure and see where you stand today.

Want more tools like this? I'm a solo dev shipping Apify actors that replace overpriced deliverability SaaS—subscribe to the newsletter for new releases and deliverability playbooks.

Publish your own actors on Apify and earn alongside me: apify.com.

Resources:

DNS Propagation: Why 15 Resolvers Matter and How to Check Them All in One Call

NexGenData — Wed, 29 Jul 2026 03:10:35 +0000

DNS Propagation: Why 15 Resolvers Matter and How to Check Them All in One Call

If you've ever shipped a DNS change and then spent the next 45 minutes refreshing whatsmydns.net to watch the green checkmarks tick across a world map, this post is for you.

whatsmydns is a lovely tool. It is also fundamentally manual. You paste a hostname, click "Search," and eyeball a grid of ~20 resolvers to see which ones have picked up your change. There's no API. There's no programmatic way to run the check on a schedule. There's no way to wire the result into your deploy pipeline so the build halts if propagation hasn't completed.

That gap is what the NexGenData DNS Propagation Checker fills. One API call queries 15 globally distributed, operator-distinct DNS resolvers for a given hostname and record type, returns structured JSON, and detects anomalies like Cloudflare's answer rotation and regional CDN inconsistencies.

This post explains which 15 resolvers it hits, why those specific 15, why checking outside of Google/Cloudflare/Quad9 matters more than most engineers realize, and how to wire the actor into real workflows with cURL and Python.

Why DNS Propagation Is a Real Problem (and Not Just Vibes)

"DNS propagation" is one of those phrases that gets used imprecisely. Technically, DNS doesn't "propagate"—authoritative nameservers answer queries directly, and caching resolvers forget old answers when TTLs expire. There's no update being pushed from a central point.

But the experienced reality is different. When you update an A record, users in different parts of the world see the change at different times because:

Different recursive resolvers cache independently. Google's 8.8.8.8 in us-east caches separately from Google's 8.8.8.8 in eu-west. Cloudflare's 1.1.1.1 in Asia has its own cache.
Resolvers honor TTLs unevenly. Some respect your authoritative TTL precisely; others cap at a minimum (typically 60s) or maximum (typically 24h).
Some networks run their own caching layer on top. ISPs often proxy to a public resolver but cache answers themselves. Corporate VPNs do this too.
Anycast routing changes which PoP you hit. A user flying from LA to Tokyo might suddenly hit a different Cloudflare PoP that hasn't pulled the new answer yet.

The net effect: a DNS change that looks "propagated" from your laptop can still be serving stale answers to a double-digit percentage of global traffic for hours. If you flipped that DNS record as part of a blue/green deploy, a CDN migration, or a mail server move, that percentage is your users seeing 404s, broken TLS, or email bouncing.

Why 15 Resolvers, and Specifically These 15

You could check 100 resolvers. Or 5. The number 15 is a balance between two realities:

Most resolvers are downstream of a handful of major upstreams. Thousands of ISP resolvers in the world ultimately proxy to Google, Cloudflare, Quad9, OpenDNS, or Verisign. Checking all of them is redundant.
A small number of operators run their own distinct infrastructure with their own caches and peering arrangements. If you only check the "big three" (Google, Cloudflare, Quad9), you miss those—and those are where real-world users actually hit propagation gaps.

The 15 resolvers the actor checks are chosen to be operator-distinct—each one is run by a different organization, with its own infrastructure, in a different geography:

Resolver	Operator	Region	Notes
8.8.8.8	Google	Global anycast	Dominant consumer resolver
1.1.1.1	Cloudflare	Global anycast	Fastest globally on average
9.9.9.9	Quad9	Global anycast	Swiss-based, filters malware
208.67.222.222	OpenDNS (Cisco)	Global anycast	Enterprise / family filter use
64.6.64.6	Verisign	US	Infrastructure-tier, different peering
77.88.8.8	Yandex	Russia	Critical for RU/CIS user traffic
94.140.14.14	AdGuard	Global	DNS-level ad blocking userbase
185.228.168.168	CleanBrowsing	Global	Family-filter and schools use
156.154.70.1	Neustar/UltraDNS	Global	Enterprise DNS, different cache
101.101.101.101	TWNIC Quad101	Taiwan / APAC	Regional resolver, distinct cache
223.5.5.5	Alibaba	China / APAC	Major APAC consumer resolver
80.80.80.80	Freenom	Netherlands	European regional
216.146.35.35	Dyn (Oracle)	US	Legacy but still in corporate use
8.26.56.26	Comodo Secure DNS	Global	Security-focused filtering
199.85.126.10	Norton ConnectSafe	US	Consumer security product

The reasoning for each cluster:

Global dominants (Google, Cloudflare, Quad9): What the average consumer hits.
Regional critical (Yandex, Alibaba, TWNIC): Where non-Western user traffic actually resolves. Checking only 1.1.1.1 tells you nothing about what users in Moscow or Shanghai are seeing.
Enterprise and filtering (OpenDNS, CleanBrowsing, Comodo, Norton, AdGuard): Corporate networks and family-filter setups route through these. A propagation bug here silently breaks a chunk of your B2B customers.
Infrastructure-tier (Verisign, Neustar, Dyn): Distinct peering and caching; frequently used as upstream by smaller ISPs.

This coverage is empirically enough to catch all the real-world propagation anomalies I've seen in production—without padding the list with 80 resolvers that are really just downstream of these 15.

What Each Query Actually Returns

For each of the 15 resolvers, per record type requested (A, AAAA, CNAME, MX, TXT, NS, CAA, SOA), the actor returns:

The response: the record(s) returned, or NXDOMAIN / SERVFAIL
Response time in milliseconds
TTL (as served by that resolver)
Whether the answer matches the authoritative answer (pulled separately from the domain's NS records)

The "matches authoritative" bit is the one that matters. A resolver returning your old IP is not propagated. A resolver returning the new IP matches. The actor computes a propagationPct across the 15 resolvers so you get a single number you can assert on in CI.

Try It: Single Domain, A Record, cURL

curl "https://api.apify.com/v2/acts/ITdMyAckvJED46a8V/run-sync-get-dataset-items?token=$APIFY_TOKEN" \
  -X POST \
  -H 'Content-Type: application/json' \
  -d '{
    "domain": "example.com",
    "recordTypes": ["A"],
    "compareToAuthoritative": true
  }'

Response (abridged):

[
  {
    "domain": "example.com",
    "recordType": "A",
    "authoritative": {
      "nameservers": ["a.iana-servers.net", "b.iana-servers.net"],
      "answer": ["23.192.228.80", "23.192.228.84", "23.215.0.136", "23.215.0.138", "96.7.128.175", "96.7.128.198"]
    },
    "resolvers": [
      {
        "resolver": "8.8.8.8",
        "operator": "Google",
        "answer": ["23.192.228.80", "23.192.228.84", "23.215.0.136", "23.215.0.138", "96.7.128.175", "96.7.128.198"],
        "ttl": 3600,
        "responseTimeMs": 14,
        "matchesAuthoritative": true
      },
      {
        "resolver": "1.1.1.1",
        "operator": "Cloudflare",
        "answer": ["23.192.228.80", "23.192.228.84", "23.215.0.136", "23.215.0.138", "96.7.128.175", "96.7.128.198"],
        "ttl": 3596,
        "responseTimeMs": 9,
        "matchesAuthoritative": true
      },
      {
        "resolver": "77.88.8.8",
        "operator": "Yandex",
        "answer": ["23.192.228.80", "23.192.228.84", "23.215.0.136", "23.215.0.138", "96.7.128.175", "96.7.128.198"],
        "ttl": 3200,
        "responseTimeMs": 87,
        "matchesAuthoritative": true
      }
    ],
    "propagationPct": 100,
    "inconsistencies": []
  }
]

The inconsistencies array is where interesting things live. On a freshly changed record, you'll see entries like:

"inconsistencies": [
  {
    "resolver": "156.154.70.1",
    "operator": "Neustar/UltraDNS",
    "returnedAnswer": ["192.0.2.10"],
    "authoritativeAnswer": ["192.0.2.20"],
    "issue": "stale_record",
    "staleTtl": 1243
  }
]

That's telling you: Neustar hasn't picked up your change yet, and has 1,243 seconds of stale TTL left.

Bulk Mode: Assert Propagation in a Deploy Pipeline

The pattern I use most: check propagation after a DNS change as part of a deploy pipeline, and fail the deploy if propagation is below a threshold. Python:

import os
import sys
from apify_client import ApifyClient

client = ApifyClient(os.environ["APIFY_TOKEN"])

DOMAIN = "api.example.com"
THRESHOLD_PCT = 90  # require 90% of resolvers to be current

run = client.actor("nexgendata/dns-propagation-checker").call(run_input={
    "domain": DOMAIN,
    "recordTypes": ["A", "AAAA"],
    "compareToAuthoritative": True,
})

results = list(client.dataset(run["defaultDatasetId"]).iterate_items())

failed = False
for r in results:
    pct = r["propagationPct"]
    rec = r["recordType"]
    print(f"{rec}: {pct}% of 15 resolvers current")
    if pct < THRESHOLD_PCT:
        failed = True
        for inc in r["inconsistencies"]:
            print(f"  Stale on {inc['operator']} ({inc['resolver']}): "
                  f"{inc['returnedAnswer']} vs authoritative {inc['authoritativeAnswer']}")

if failed:
    print(f"\nPropagation below {THRESHOLD_PCT}% threshold. Halting deploy.")
    sys.exit(1)

print("\nPropagation OK. Proceeding.")

Drop that into a GitHub Actions step after your DNS-updating job and it'll halt the deploy until the new record has reached the threshold globally. This is a real thing you want in CI for blue/green deployments where the switchover is DNS-based.

Cloudflare Answer Rotation (and Why Your Checks Lie Without It)

A thing that trips people up: Cloudflare's A records for proxied hostnames return different IPs each query, because the Cloudflare edge picks one of a pool. If you naively compare the answer from resolver A to the answer from resolver B, they'll disagree even though both are fully propagated.

The actor handles this. When it detects that the authoritative answer comes from a Cloudflare nameserver (ns[1-6].cloudflare.com) and the returned IPs are within Cloudflare's published ranges, it relaxes the match criterion to "is this IP in the expected Cloudflare ranges?" rather than "exact match to authoritative."

The same kind of logic applies to:

Anycast CDN records (Fastly, Akamai, CloudFront) that rotate per-query
DNS-based load balancing (AWS Route 53 weighted/latency records)
GeoDNS (different answers by client subnet)

You can toggle this with strictMatch: true if you want exact-string comparison and don't care about the rotation semantics.

JavaScript: On-Demand Check for a Web Dashboard

If you're exposing a "did my DNS change go through" tool inside an internal dashboard:

async function checkPropagation(domain, recordType = "A") {
  const res = await fetch(
    `https://api.apify.com/v2/acts/ITdMyAckvJED46a8V/run-sync-get-dataset-items?token=${process.env.APIFY_TOKEN}`,
    {
      method: "POST",
      headers: { "Content-Type": "application/json" },
      body: JSON.stringify({
        domain,
        recordTypes: [recordType],
        compareToAuthoritative: true,
      }),
    }
  );

  const [result] = await res.json();
  return {
    pct: result.propagationPct,
    current: result.resolvers.filter(r => r.matchesAuthoritative).length,
    total: result.resolvers.length,
    stale: result.inconsistencies.map(inc => ({
      operator: inc.operator,
      ttlRemaining: inc.staleTtl,
    })),
  };
}

// Usage
const status = await checkPropagation("api.example.com", "A");
console.log(`${status.current}/${status.total} resolvers current (${status.pct}%)`);
if (status.stale.length) {
  console.log("Stale on:");
  status.stale.forEach(s => console.log(`  ${s.operator} (TTL ${s.ttlRemaining}s remaining)`));
}

Wire that into a dashboard widget and you've replaced whatsmydns.net with something scriptable.

Real Use Cases I Use This For

1. Blue/Green Deploy Gating

A company I work with does traffic cutover via DNS—old environment on old IPs, new environment on new IPs, flip the A record when ready. Without a propagation check, the "flip" can leave 10% of users on the old environment for hours. With the check running in the deploy pipeline, the pipeline waits until ≥95% propagation before marking the deploy complete.

2. Email Migration Verification

Moving MX records from an old mail host to a new one is a classic "one-resolver-is-stale" story. A 5% stale resolver share means 5% of inbound mail goes to the old host and bounces. The actor's MX record check flags this across all 15 resolvers.

3. Post-Incident DNS Forensics

When something breaks and the incident postmortem involves "was it DNS," the actor's historical results (the actor writes each run to a persistent dataset) let you see the propagation state at the moment of the incident rather than trying to reconstruct it from memory.

4. Geographic Routing Sanity Checks

Launching in a new region (APAC, EMEA) and want to verify your GeoDNS is routing correctly? Check the same domain from Yandex, Alibaba, TWNIC, and Comodo and look at the IPs returned. If all 15 return the same single IP, your GeoDNS isn't working. If regional resolvers return different IPs, it is.

5. Cloudflare / Provider Outage Detection

When Cloudflare's control plane hiccups (it happens), authoritative answers for proxied records go weird. Running a propagation check across 15 resolvers surfaces this as "Cloudflare nameservers returning different answers than other NS" faster than waiting for a status page update.

How Does This Compare to the Alternatives?

Tool	API	Bulk	Resolvers checked	Notes
whatsmydns.net	No	No	~20	Manual, UI-only
dig (one resolver at a time)	N/A	No	1	The "DIY" baseline
DNSChecker.org	No	No	~30	UI-only, similar to whatsmydns
Pingdom DNS check	Yes	Yes	Varies	Part of $15/mo+ monitoring plans
UptimeRobot	Yes	Yes	Limited	Per-resolver check requires Pro
NexGenData Propagation Checker	Yes	Yes	15 distinct operators	~$0.003 per domain

For the use cases above (CI gating, scheduled checks, dashboards), the API-accessible version wins by default; the only question is which API-accessible version, and the per-domain price + operator-distinctness of this actor makes it the one I actually use.

Answering the Common Questions

Q: Is 15 resolvers really enough?
A: For practical propagation confidence, yes. I tested the actor against 50+ resolvers in development; the marginal information above 15 was essentially zero because the extras were all downstream of the 15 already covered. The operator-distinctness matters more than raw count.

Q: Can I add my own resolvers?
A: Yes. Pass customResolvers: ["1.0.0.1", "my-corporate-resolver.example.com"] in the input and the actor will query those in addition to the built-in 15.

Q: Does it support DNS-over-HTTPS (DoH)?
A: Yes, optionally. useDoh: true will query the resolvers over DoH instead of plain UDP DNS. Slower (a few hundred ms per query) but useful if your egress environment blocks UDP/53.

Q: What about DNSSEC?
A: The actor reports whether each resolver validates DNSSEC for the response (AD flag set in the response). It doesn't do independent DNSSEC validation; for that, use a tool like delv or dnsviz.net.

Q: Can it check a single record type only (like just MX)?
A: Yes. recordTypes: ["MX"]. You can also pass multiple types in one run and get all results back.

Q: How does pricing work?
A: Roughly $0.003 per domain checked, per record type. A single A-record check across 15 resolvers is about $0.003; an A+AAAA+MX+TXT check is four times that. Bulk discounts apply above 1,000 checks per run.

Related NexGenData Actors for a Full DNS/Deliverability Stack

Email DMARC Auditor — When you're deploying SPF/DMARC/DKIM changes, pair with propagation checks to confirm both that the record exists and that all resolvers see it.
Email RBL Checker — After an IP migration, verify both DNS propagation and that your new IP isn't on any blocklists before cutting over email traffic.
Google Cache Viewer — Historical view of a site's content when propagation issues make current DNS unreliable.
Website Performance Monitor — Pair with DNS propagation checks to catch "DNS is propagated but origin is slow" vs "DNS is stale" as distinct classes of issue.

Try It

Single domain, A record, right now:

curl "https://api.apify.com/v2/acts/ITdMyAckvJED46a8V/run-sync-get-dataset-items?token=$APIFY_TOKEN" \
  -X POST \
  -H 'Content-Type: application/json' \
  -d '{"domain": "yourdomain.com", "recordTypes": ["A"]}'

Or from the UI: apify.com/nexgendata/dns-propagation-checker.

Apify's free tier includes $5 of credits. That's enough for roughly 1,500 A-record propagation checks, or a few hundred multi-record checks.

Want more tools like this? I'm a solo dev shipping Apify actors that plug the gaps between "free but manual" and "$150/mo SaaS"—subscribe to the newsletter for new releases.

Build your own actors: sign up at apify.com and ship alongside mine.

Resources:

Google Killed the Cached Page View in Feb 2024. Here's How I Replaced It

NexGenData — Thu, 23 Jul 2026 19:13:23 +0000

Google Killed the Cached Page View in Feb 2024. Here's How I Replaced It

In early February 2024, Google's Danny Sullivan confirmed on X what SEO professionals, journalists, and OSINT researchers had started to suspect: the cache: operator and the "Cached" link in search results were being permanently removed.

Sullivan's stated reason: the feature was originally built to help people load pages when "the internet was unreliable," and Google no longer considered it necessary. For anyone using cached pages for real work, the removal broke an entire workflow:

SEO auditors could no longer see what Google's crawler actually indexed
Journalists lost a quick check for pages that had been edited or deleted
OSINT researchers lost a first-stop verification tool
Support teams could no longer see the version of a page a user was looking at when something went wrong
Litigation preservation lost its easiest evidentiary primer

A month later, Google also removed cached pages from the Search Console's URL Inspection tool. The cache was really, completely gone.

What replaced it? Nothing, officially. Unofficially, everyone moved to the Internet Archive's Wayback Machine and archive.today. Both work. Both have quirks. Both have APIs that are half-documented and behave in ways that surprise you.

This post is the workflow I built to put a reliable "show me what this page looked like" capability back together. It centers on the NexGenData Google Cache Viewer actor, which wraps Wayback and archive.today with a unified fallback chain, schema-level parsing, and a zero-config HTTP interface.

Why This Workflow Gap Matters More Than Google Implied

When Google pulled the plug, the first wave of reactions was "who cares, just use Wayback." Six months in, the consensus shifted. Here's what actually broke:

SEO and Content Work

Before: SEO teams could paste cache:example.com/page into the URL bar and see exactly what Googlebot had indexed last—word count, schema markup, rendered HTML, the whole picture.

After: You can still see cached snapshots in a general "what did this URL look like" sense, but Wayback doesn't tell you what Google's crawler saw. For ranked-page diagnostics (why is this page ranking for this query? has Google indexed my latest update?), the answer became "it depends, and you'll need multiple tools."

Journalism and Fact-Checking

Before: A reporter could paste a URL into cache: and instantly see the pre-edit or pre-deletion version. Getting stories onto the record within minutes mattered.

After: Wayback often does have the older snapshot—but not always on the original timestamp the reporter needs. archive.today frequently has the snapshot Wayback missed, because it was triggered by a human pressing "archive this." Knowing which archive to check, and being able to check both quickly, became the new skill.

OSINT and Research

Before: A quick cache check was the first step in any investigation of a suspicious page, takedown, or edit war.

After: OSINT researchers now run multi-archive checks as standard practice. Every experienced researcher I've talked to has built or adopted a tool that fans out to multiple archives and returns the first useful hit.

The Two Real Alternatives (and Their Quirks)

There are two credible replacements for Google's cache. Each has characteristic strengths and failure modes.

Internet Archive Wayback Machine

The Wayback Machine is run by the nonprofit Internet Archive and is the largest general web archive in the world—hundreds of billions of snapshots. Its API is technically public:

https://archive.org/wayback/available?url=example.com&timestamp=20240301

That returns JSON with the nearest snapshot to the timestamp. In practice, you hit three issues:

Scheme sensitivity: Wayback treats http://example.com, https://example.com, https://www.example.com, and https://example.com/ as different URLs for lookup purposes. Query the wrong variant and Wayback says "no snapshot exists" even when there are thousands for the same page.
Rate limiting: The available endpoint enforces a soft limit; CDX search (the powerful bulk query) enforces a stricter one. Aggressive polling will 429 you.
Partial renders: JavaScript-heavy pages frequently archive as blank shells. The snapshot exists but the content you wanted isn't there.

archive.today (archive.ph / archive.is / archive.li)

archive.today is a smaller, user-initiated archive. Nothing gets archived there unless a human (or bot) triggered a save. That means its coverage is much thinner than Wayback's overall, but dramatically better on:

News articles that were deleted or heavily edited (because someone always triggers an archive on contested stories)
Twitter/X posts (Wayback's Twitter snapshots are frequently broken; archive.today renders them well)
Paywalled articles (archive.today bypasses many paywalls by the mechanism it uses to render)

The quirks:

No documented API: Everything is scraped. archive.today publishes OpenGraph metadata on its snapshot pages, which is what parseable tools use.
CDN / DNS rotation: The domain moves between .today, .ph, .is, .li, .fo for resilience reasons. Hard-coding a single hostname breaks.
Scheme sensitivity, but different from Wayback: archive.today tends to canonicalize more aggressively than Wayback does—but its hit-rate on the "canonical" form isn't 100%, so you still need to try variants.

The Takeaway

Neither source alone is enough. The right workflow is: try archive.today first (user-triggered archives tend to be the snapshot someone wanted preserved), fall back to Wayback, and handle scheme variants for both.

That's what the actor implements.

What the Actor Does

The NexGenData Google Cache Viewer takes a URL (or array of URLs) and returns:

The first successful archive match across the fallback chain
Snapshot timestamp
Archive source (wayback or archive_today)
Snapshot URL (the permalink on the archive)
Optionally: rendered HTML, extracted text, page title, meta description

The fallback logic handles scheme sensitivity automatically. For example.com/article, it tries, in order:

https://example.com/article on archive.today
https://example.com/article on Wayback
http://example.com/article on archive.today
http://example.com/article on Wayback
https://www.example.com/article on archive.today
https://www.example.com/article on Wayback
With/without trailing slash variants
Canonical URL extracted from OpenGraph if any snapshot was found

You get the first hit. If no archive has the page, you get a clean found: false with the variants that were tried—which is itself useful information.

Try It: One URL with cURL

curl "https://api.apify.com/v2/acts/lueaWo5saiyGDsdNz/run-sync-get-dataset-items?token=$APIFY_TOKEN" \
  -X POST \
  -H 'Content-Type: application/json' \
  -d '{
    "urls": ["https://techcrunch.com/2024/02/01/google-search-cache-links-dead/"],
    "preferredSource": "archive_today",
    "fallbackToWayback": true,
    "extractText": true
  }'

Response:

[
  {
    "requestedUrl": "https://techcrunch.com/2024/02/01/google-search-cache-links-dead/",
    "found": true,
    "source": "archive_today",
    "snapshotUrl": "https://archive.ph/2024/20240202143021/https://techcrunch.com/2024/02/01/google-search-cache-links-dead/",
    "snapshotTimestamp": "2024-02-02T14:30:21Z",
    "title": "Google Search's cache links are officially gone",
    "metaDescription": "Google Search has officially killed off the 'cached' link, Google's Search Liaison Danny Sullivan confirmed...",
    "extractedText": "Google Search has officially killed off the 'cached' link, which had been part of Google Search since the earliest days... [full article text] ...",
    "wordCount": 1247,
    "attemptedVariants": [
      { "url": "https://techcrunch.com/2024/02/01/google-search-cache-links-dead/", "source": "archive_today", "result": "hit" }
    ]
  }
]

The attemptedVariants array is useful diagnostics—if the first variant hit, you'll see one entry; if fallback was needed, you'll see each attempt and its outcome.

Bulk Mode: Checking 1,000 URLs in Python

The pattern I use most is feeding in a list of URLs from somewhere (a sitemap, a link audit, a list of referrer URLs from analytics) and getting back an enriched dataset.

import os
from apify_client import ApifyClient

client = ApifyClient(os.environ["APIFY_TOKEN"])

# URLs from your link audit, competitor sitemap, or referrer log
urls = [
    "https://example.com/blog/old-post-1",
    "https://example.com/blog/old-post-2",
    # ... hundreds more
]

run = client.actor("nexgendata/google-cache-viewer").call(run_input={
    "urls": urls,
    "preferredSource": "archive_today",
    "fallbackToWayback": True,
    "extractText": False,        # faster/cheaper if you don't need full text
})

results = list(client.dataset(run["defaultDatasetId"]).iterate_items())

# Tally by source
from collections import Counter
sources = Counter(r["source"] for r in results if r["found"])
print(f"Found in archive.today: {sources.get('archive_today', 0)}")
print(f"Found in Wayback:       {sources.get('wayback', 0)}")
print(f"Not archived anywhere:  {sum(1 for r in results if not r['found'])}")

# Identify the unarchived URLs so you can submit them to Wayback's save API
unarchived = [r["requestedUrl"] for r in results if not r["found"]]
for url in unarchived:
    print(f"No archive: {url}")

JavaScript / Node: On-Demand Cache Lookup from a Web UI

If you're building a tool where a user pastes a URL and wants to see the archived version, here's the client-side pattern (proxied through your own backend so you don't expose the Apify token):

// server.js (Express)
app.post("/api/lookup-cache", async (req, res) => {
  const { url } = req.body;

  const apifyRes = await fetch(
    `https://api.apify.com/v2/acts/lueaWo5saiyGDsdNz/run-sync-get-dataset-items?token=${process.env.APIFY_TOKEN}`,
    {
      method: "POST",
      headers: { "Content-Type": "application/json" },
      body: JSON.stringify({
        urls: [url],
        preferredSource: "archive_today",
        fallbackToWayback: true,
        extractText: true,
      }),
    }
  );

  const [result] = await apifyRes.json();

  if (!result.found) {
    return res.status(404).json({
      error: "No archive exists for this URL",
      attemptedVariants: result.attemptedVariants,
    });
  }

  res.json({
    snapshot: result.snapshotUrl,
    capturedAt: result.snapshotTimestamp,
    title: result.title,
    text: result.extractedText,
    source: result.source,
  });
});

This gives you a "what did this page look like" lookup you can embed in an internal tool in an afternoon. No need to learn the Wayback CDX API or scrape archive.today directly.

Real Use Cases I Use This For

1. SEO Diagnostics on De-indexed Pages

A client's page drops out of Google's index overnight. You want to compare the current live page to the last known "good" version. The actor pulls the archived snapshot; you diff it against the current HTML; you often find the exact change that triggered the deindex (usually a meta robots change, a schema breakage, or content thinning).

2. Litigation / Compliance Preservation

A brand's competitor publishes something defamatory and then deletes it three hours later. Wayback crawls infrequently and may have missed the window. archive.today is more likely to have the snapshot because someone involved in the dispute triggered an archive. The fallback chain finds whichever archive caught it.

3. Research and Academic Citations

Papers that cite web sources routinely hit link rot within a few years. Running the archive lookup on a bibliography gets you current archive URLs to substitute in, or flags the citations that are dead across every archive (so you can hunt for alternatives).

4. Journalism Verification

A source claims "X said Y on their website three weeks ago." You paste the URL, get the archived version with its timestamp, compare with the current page, and you have evidence that the claim is or isn't accurate.

5. Deleted Social Media / News Stories

When a news organization retracts a story or a public figure deletes a blog post, the first question is "was it archived?" The fallback logic maximizes the odds of the answer being yes.

Why Not Just Call the Wayback API Yourself?

You absolutely can. I did, for about three months. Here's what kept tripping me up and eventually pushed me to build the wrapper:

Scheme normalization: Writing and testing the "try HTTPS, try HTTP, try with/without www., try with/without trailing slash" permutation logic is mildly tedious and a source of edge-case bugs.
archive.today has no clean API: You scrape it. Which means dealing with its CDN rotation across .today, .ph, .is, .li, .fo domains.
Rate limits are aggressive: Wayback's CDX endpoint is stricter than the available endpoint. A naive bulk script gets 429'd fast.
Text extraction requires actually rendering the snapshot: Snapshots are in Wayback's replay frame or archive.today's snapshot frame. Extracting the underlying page text means parsing around those wrappers.
Maintenance drift: archive.today's HTML structure changes every few months. Wayback's API gets the occasional parameter tweak. You maintain this forever if you DIY.

The actor exists because I got tired of maintaining that logic in three different side projects. Now it's one piece of code, fixed centrally.

Pricing: About $0.002 Per URL

The actor uses Apify's pay-per-event pricing. At the default settings (fallback chain enabled, text extraction on), typical cost is about $0.002 per URL. A 1,000-URL bulk check runs about $2.

A few concrete comparisons:

Tool	Price	Per-URL	Notes
Wayback Machine direct	Free	$0	DIY, rate-limited, scheme-sensitive
archive.today direct	Free	$0	DIY, no official API
Google Cache (RIP)	—	—	Dead since Feb 2024
Time Travel (Memento)	Free, limited	$0	Federated, not uniformly parsed
NexGenData Cache Viewer	~$0.002/URL	$0.002	Unified, parsed, bulk

Answering the Common Questions

Q: Is this the same as the cache: operator?
A: Functionally close, but the underlying source is Wayback + archive.today rather than Google's own cache. For most use cases (seeing a prior version of a page), it's equivalent. For "see what Googlebot indexed," it's not—only Google's systems know that, and they no longer expose it.

Q: Can it trigger a new archive save?
A: Not directly, though I'm considering adding a saveIfMissing flag that triggers Wayback's Save Page Now. For now, if the URL isn't archived, you get back found: false with a list of attempted variants, and you can use Wayback's public save endpoint to archive it yourself.

Q: Does it bypass paywalls?
A: archive.today's snapshots often do (as a side effect of how they render, not by design). Wayback varies. The actor returns whatever the archives return—neither better nor worse.

Q: How do I know the snapshot is current?
A: The response includes snapshotTimestamp. If the archived snapshot is older than you need, you can set minTimestamp in the input and the actor will only return snapshots newer than that date (returning found: false if none qualify).

Q: Can it follow redirects?
A: Yes, optionally. followRedirects: true will chase HTTP 301/302s and archive the final destination. Off by default because you usually want the specific URL you asked about.

Related NexGenData Actors for Research and Monitoring

DNS Propagation Checker — When a site goes down or pops back up, check from 15 resolvers worldwide to verify global propagation state.
Email DMARC Auditor — Pair with cache lookups to see the historical DMARC posture of a domain that's now changed its records.
Google Scholar Scraper — Academic references benefit from both Scholar's citation data and archived full-text from the cache viewer.
Hacker News Scraper — Monitor HN comments that reference URLs and archive them before the links rot.

Try It

Single URL, right now:

curl "https://api.apify.com/v2/acts/lueaWo5saiyGDsdNz/run-sync-get-dataset-items?token=$APIFY_TOKEN" \
  -X POST \
  -H 'Content-Type: application/json' \
  -d '{"urls": ["https://example.com/some-article"]}'

Or from the UI: apify.com/nexgendata/google-cache-viewer.

Free tier on Apify gives you $5 of credits, which covers roughly 2,500 URL lookups. Enough to run a sitemap audit, do a research pass, or back-fill a broken-link check across an entire blog.

Want more pragmatic data tools? I'm a solo dev building Apify actors that plug gaps like this one—subscribe for new releases.

Build your own actors and ship them alongside mine on Apify: apify.com.

Resources:

MXToolbox Costs $149/mo. I Built a DMARC/SPF/DKIM Bulk Auditor for $0.005 Per Domain

NexGenData — Tue, 21 Jul 2026 15:26:46 +0000

MXToolbox Costs $149/mo. I Built a DMARC/SPF/DKIM Bulk Auditor for $0.005 Per Domain

In February 2024, Google and Yahoo quietly redrew the rules for anyone sending email. If you push more than 5,000 messages a day to Gmail or Yahoo inboxes, you now need a valid DMARC record, aligned SPF, and DKIM signing—or your mail goes to spam. Permanently. No warnings.

Overnight, "email deliverability" stopped being a problem for enterprise postmasters and became a problem for every SaaS founder, agency, and marketing team on the planet. And the incumbent tooling—MXToolbox at $149/month, Valimail in the thousands, Dmarcian at $49/month per 50 domains—got a lot more expensive for anyone who actually needs to audit at scale.

I audit 400+ client domains monthly. I wasn't going to pay $600/month for that. So I built a bulk auditor on Apify that costs about half a cent per domain, returns machine-readable JSON, and can rip through an entire customer list in a single API call.

This post walks through why bulk auditing matters now, exactly what the actor checks, the 25 DKIM selectors it probes (and why that number is what it is), and how to drop it into your own deliverability workflow with cURL or Python.

The Problem: Email Authentication Went From "Nice to Have" to Enforced

Here's the short version of what changed:

February 2024: Google and Yahoo began rejecting or spam-foldering mail from bulk senders lacking DMARC, aligned SPF, and DKIM.
June 2024: Microsoft announced the same policy for Outlook/Hotmail inboxes, phasing in through 2025.
2025: The 5,000/day threshold widened in practice—providers started applying the same heuristics to smaller senders who "look bulk."

The standards themselves (SPF/DKIM/DMARC) are decades old. What changed is enforcement. What used to silently succeed now silently fails. And the failure mode is the worst possible one: your mail still gets sent, it just never gets read.

So if you're running:

A SaaS with transactional email (password resets, receipts, alerts)
A cold outreach operation
Multiple client domains at an agency
An acquired portfolio of marketing domains
A parent company with dozens of subdomains for product lines

...you need to know, right now, which of your domains are actually passing authentication. Not "should be passing." Passing. Observable from the outside.

Why Single-Domain Tools Don't Work at Scale

MXToolbox is fine if you have one domain and you want a pretty UI. Paste the domain in, get green checkmarks, done.

But the moment you have 50 domains—or 500, or 5,000—you hit the wall:

No bulk interface: MXToolbox's UI is one-domain-at-a-time. Their "Monitoring" product is extra cost on top of the $149/month base.
API is gated: Their API requires the Monitoring tier and is billed by query.
No DKIM selector discovery: Unless you already know the selector (e.g. google, selector1, k1), you can't audit DKIM. Their tool asks you to provide it.
No structured output: Parsing their HTML to feed into a dashboard is its own engineering project.
Pricing is per-seat: Want three teammates looking at the same dashboard? Triple the bill.

Valimail and Dmarcian are purpose-built for DMARC and do aggregate RUA reports well, but they're:

Expensive ($99-2000+/month)
Focused on DMARC only (they don't audit SPF bloat or DKIM reachability in the way we need)
Slow to onboard (you need to publish a rua URI and wait days for reports)

What I actually needed: a script-friendly "give me a list of domains, tell me what's wrong with each one, return JSON" service. That's the actor I built.

What the Actor Checks (and Why Each Thing Matters)

The NexGenData Email DMARC Auditor runs six categories of checks per domain:

1. MX Records

Does the domain actually receive mail? A domain with no MX can't comply with DMARC alignment requirements for any inbound flow. We report the priority and host for each MX.

2. SPF Record

We pull the TXT record starting with v=spf1, parse each mechanism (include:, a:, mx:, ip4:, ip6:, ~all, -all), and count DNS lookups. The SPF standard caps total lookups at 10—exceed that and the record is invalid, which means every message fails SPF evaluation. We flag domains approaching or exceeding the limit, plus common smells like multiple SPF records (which is also invalid per RFC 7208).

3. DMARC Record

We look up _dmarc.<domain> and parse the policy tags: p= (policy), sp= (subdomain policy), pct= (rollout percentage), rua=, ruf=, aspf= (SPF alignment mode), adkim= (DKIM alignment mode), fo= (failure reporting options). A p=none DMARC record is effectively a declaration that you don't care—it's monitoring-only. We flag anything that isn't p=quarantine or p=reject as non-enforcing.

4. DKIM Selectors

This is the hard one. DKIM is published as TXT records at <selector>._domainkey.<domain>, and there is no DNS mechanism to enumerate selectors. You either know them or you don't.

In practice, most senders use one of a well-known set. Our actor probes 25 common DKIM selectors for every domain. If any of them respond, we extract the public key and the configured algorithm (k=rsa, usually; k=ed25519 occasionally), parse the key length, and report pass/fail. The 25 we check are:

google, default, selector1, selector2, k1, k2, k3, dkim, mail, email, smtp, mandrill, mailchimp, sendgrid, mailgun, amazonses, postmark, sparkpost, zoho, fastmail, mxvault, pm, s1, s2, everlytickey1

That list covers Gmail/Workspace (google), Microsoft 365 (selector1/selector2), the big transactional senders (SendGrid, Mailgun, Amazon SES, Postmark, Mailchimp's Mandrill, etc.), and the naming patterns used by most self-hosted setups. Coverage is high—in our test set of 400 domains, the 25-selector probe caught valid DKIM on 96% of domains that had DKIM configured.

5. BIMI Record

Brand Indicators for Message Identification—the standard for putting your logo next to your subject line in Gmail/Apple Mail. Requires p=quarantine or p=reject DMARC and a VMC (Verified Mark Certificate) to actually display. We pull default._bimi.<domain> and report the v=BIMI1 tag, logo URL (l=), and certificate URL (a=).

6. MTA-STS and TLS-RPT

MTA-STS (RFC 8461) is the modern way to enforce TLS for inbound SMTP; TLS-RPT (RFC 8460) is how you receive reports about TLS delivery failures. Both are published as TXT records with a companion policy file at a well-known HTTPS URL. We flag the presence/absence of each, which is a good proxy for "how seriously does this domain take inbound security."

Try It: One Domain with cURL

The actor runs on Apify's pay-per-event pricing. You need an Apify token—grab one at apify.com. Then:

curl "https://api.apify.com/v2/acts/dYeLZRrQ2undcMbwv/run-sync-get-dataset-items?token=$APIFY_TOKEN" \
  -X POST \
  -H 'Content-Type: application/json' \
  -d '{
    "domains": ["stripe.com"],
    "checkDkim": true,
    "checkBimi": true,
    "checkMtaSts": true
  }'

Response (trimmed to the interesting bits):

[
  {
    "domain": "stripe.com",
    "mx": [
      { "priority": 1, "host": "aspmx.l.google.com" },
      { "priority": 5, "host": "alt1.aspmx.l.google.com" }
    ],
    "spf": {
      "record": "v=spf1 include:_spf.google.com include:amazonses.com include:_spf.salesforce.com ~all",
      "valid": true,
      "lookupCount": 7,
      "allMechanism": "~all",
      "warnings": []
    },
    "dmarc": {
      "record": "v=DMARC1; p=reject; rua=mailto:dmarc-reports@stripe.com; pct=100; aspf=s; adkim=s",
      "policy": "reject",
      "pct": 100,
      "alignment": { "spf": "strict", "dkim": "strict" },
      "enforcing": true
    },
    "dkim": {
      "selectorsFound": ["google"],
      "records": [
        {
          "selector": "google",
          "keyType": "rsa",
          "keyLength": 2048,
          "valid": true
        }
      ]
    },
    "bimi": {
      "record": "v=BIMI1; l=https://stripe.com/img/bimi/stripe-logo.svg; a=https://stripe.com/img/bimi/stripe-vmc.pem",
      "valid": true
    },
    "mtaSts": { "configured": true, "mode": "enforce" },
    "tlsRpt": { "configured": true },
    "score": 98,
    "grade": "A+"
  }
]

That's a clean, fully enforcing setup: strict alignment, 2048-bit key, BIMI configured, MTA-STS in enforce mode. Score 98 / grade A+.

Bulk Mode: Auditing 100 Domains in Python

This is the workflow the actor was actually designed for. Pass an array of domains, get back an array of audit results, and pipe the results into whatever dashboard or CSV you want.

import os
from apify_client import ApifyClient

client = ApifyClient(os.environ["APIFY_TOKEN"])

# Your list of client domains, acquired brands, subsidiaries, etc.
domains = [
    "stripe.com",
    "shopify.com",
    "twilio.com",
    "sendgrid.com",
    "intercom.com",
    # ... 100+ more
]

run = client.actor("nexgendata/email-dmarc-auditor").call(run_input={
    "domains": domains,
    "checkDkim": True,
    "checkBimi": False,         # skip to save a lookup
    "checkMtaSts": True,
})

results = list(client.dataset(run["defaultDatasetId"]).iterate_items())

# Find the non-compliant ones
non_enforcing = [r for r in results if not r["dmarc"].get("enforcing")]
spf_bloat   = [r for r in results if r["spf"].get("lookupCount", 0) > 9]
no_dkim     = [r for r in results if not r["dkim"].get("selectorsFound")]

print(f"Audited {len(results)} domains")
print(f"  {len(non_enforcing)} with p=none or missing DMARC")
print(f"  {len(spf_bloat)} with SPF near/over 10-lookup limit")
print(f"  {len(no_dkim)} with no DKIM selectors found")

# Export the problem children to CSV
import csv
with open("deliverability_gaps.csv", "w", newline="") as f:
    writer = csv.DictWriter(f, fieldnames=[
        "domain", "dmarc_policy", "spf_lookups", "dkim_found", "grade"
    ])
    writer.writeheader()
    for r in results:
        if r["grade"] not in ("A", "A+"):
            writer.writerow({
                "domain":       r["domain"],
                "dmarc_policy": r["dmarc"].get("policy", "missing"),
                "spf_lookups":  r["spf"].get("lookupCount", 0),
                "dkim_found":   bool(r["dkim"].get("selectorsFound")),
                "grade":        r["grade"],
            })

That script runs in about 30 seconds for 100 domains. Total spend: about 50 cents.

Use It in Make / n8n: Monitor a Domain List on a Schedule

Most agencies I know want this to run weekly and drop Slack alerts when a client domain regresses. Here's the Apify webhook pattern (works identically in Make, n8n, Zapier, and Pipedream):

// Cloudflare Worker / Node.js webhook receiver
export default {
  async fetch(request, env) {
    const { data } = await request.json();

    const regressions = data
      .filter(r => r.grade === "F" || r.score < 70)
      .map(r => `- *${r.domain}*: ${r.grade} (${r.score}/100)`)
      .join("\n");

    if (regressions) {
      await fetch(env.SLACK_WEBHOOK_URL, {
        method: "POST",
        body: JSON.stringify({
          text: `DMARC audit found ${data.filter(r => r.score < 70).length} failing domains:\n${regressions}`
        }),
      });
    }

    return new Response("ok");
  }
};

Schedule the actor to run nightly in the Apify console, point the webhook at this Worker, and you have a real monitoring system for zero ongoing dev time.

Real Use Cases I've Seen This Replace

1. Agencies Auditing Client Domains Before Kickoff

One agency I work with audits every new client's entire mail infrastructure in the first week of onboarding. They used to have a junior running MXToolbox tabs manually for 1-2 hours. Now it's a 30-second script and an automatically generated PDF.

2. M&A Due Diligence

An acquirer bought a portfolio of e-commerce brands and needed to know which ones had neglected deliverability. Running the auditor across 62 acquired domains in one call identified 11 brands running p=none (monitoring-only) and 4 brands with no SPF at all—each representing a real revenue leak in the transactional email path.

3. Pre-launch Checks for Transactional Email

A fintech client runs the auditor every time they spin up a new sending domain. It's part of their deploy pipeline: if DMARC, SPF, or DKIM aren't valid on the new domain, the release can't go out.

4. Cold-Outreach Sending Infrastructure

Sales teams running multi-domain cold email want each sending domain on p=quarantine or p=reject with aligned DKIM. Teams running 20-200 sending domains need bulk visibility. This actor gives it to them in an API call.

Pricing: Half a Cent Per Domain, Honestly

Here's the full comparison for auditing 500 domains once a month:

Tool	Price / month	Per-domain cost	Bulk API	Structured JSON
MXToolbox Monitoring	$149+	~$0.30+	Paid API	Limited
Valimail Enforce	$1,000+	$2+	Yes	Yes
Dmarcian (10 domains)	$49	$4.90	Yes	Yes
Easy DMARC (Pro)	$40+	~$0.25	Yes	Yes
NexGenData Auditor	~$2.50	~$0.005	Yes	Yes

At $0.005/domain, you can run the full check nightly on a list of 1,000 domains for about $150/year. That's cheaper than one month of MXToolbox, and you get 365 runs a year instead of a dashboard.

The Scoring Heuristic (So You Know What "Grade A" Means)

The grade isn't arbitrary. The actor assigns a numeric score 0–100 based on:

SPF present and valid (+15), SPF lookup count under 10 (+5), uses -all hard fail (+5)
DMARC present (+20), p=quarantine or p=reject (+15), pct=100 (+5), strict alignment on both SPF and DKIM (+5)
DKIM selectors found and valid (+15), key length ≥ 2048 bits (+5)
MTA-STS configured in enforce mode (+5)
TLS-RPT configured (+5)

Grades: A+ ≥95, A ≥85, B ≥70, C ≥55, D ≥40, F <40.

This mirrors what Gmail and Yahoo's deliverability heuristics actually weight in practice, plus the forward-looking BIMI/MTA-STS posture that large enterprises are starting to require of their vendors.

Related NexGenData Actors for a Full Deliverability Stack

DMARC/SPF/DKIM audit is necessary but not sufficient. Three other actors round out the full deliverability picture:

Email RBL Checker — Check 30 blocklists (Spamhaus, Barracuda, SORBS, UCEPROTECT, SpamCop, etc.) for sender IPs in a single call. If your DMARC is perfect but your IP is on Spamhaus, it doesn't matter.
DNS Propagation Checker — Verify that your SPF/DMARC/DKIM changes have actually propagated globally by querying 15 resolvers worldwide. Propagation issues are a frequent false-alarm cause in deliverability audits.
Google Cache Viewer — When a sending domain gets de-indexed by Gmail or flagged by a reputation system, check what's been archived to understand the content that caused the issue.

All four can be wired together as a single scheduled monitoring pipeline.

Answering the Common Questions

Q: Is this better than MXToolbox?
A: For bulk and automation use cases, yes, by a wide margin. For "I have one domain and I want a UI with green checkmarks," MXToolbox has a better visual experience.

Q: Can it parse aggregate DMARC (RUA) reports?
A: Not yet—this actor only inspects the published records at the DNS layer. Parsing RUA XML reports is a separate beast; tools like Dmarcian and Postmark's DMARC Digests do this well.

Q: Will the 25-selector DKIM probe miss real selectors?
A: Occasionally. If your company uses a bespoke selector like corp2024a._domainkey.example.com, the probe won't find it. You can pass a custom selector list in the input to cover that case.

Q: Is this compliant with Google's 2024 bulk-sender requirements?
A: The audit checks for compliance with those requirements. Google's rules want: aligned SPF, valid DKIM, DMARC with p=quarantine or stricter, one-click unsubscribe on marketing mail, spam complaint rate under 0.3%. This actor covers the first three with evidence.

Q: Why not just use dig?
A: You can. I did, for the first two years. What breaks at scale: the 25-selector DKIM probe (40+ DNS queries per domain just for that), the lookup-count math on SPF includes, the scoring, and the per-domain JSON aggregation. The actor isn't magic—it's just those 40+ queries per domain, done in parallel, with the parsing already written.

Try It

Run a single-domain audit right now with zero setup:

curl "https://api.apify.com/v2/acts/dYeLZRrQ2undcMbwv/run-sync-get-dataset-items?token=$APIFY_TOKEN" \
  -X POST \
  -H 'Content-Type: application/json' \
  -d '{"domains": ["yourdomain.com"], "checkDkim": true}'

Or open it in the UI and paste a list: apify.com/nexgendata/email-dmarc-auditor.

Free tier on Apify includes $5 of credits, which covers roughly 1,000 domain audits. More than enough to audit your entire client book or domain portfolio to see where the real gaps are.

Want more builds like this? I'm a solo dev on Apify publishing actors that replace overpriced SaaS—subscribe to the newsletter for new releases and deliverability deep-dives.

Want to build your own actors? Sign up for Apify and skip the bill-per-seat nightmare: apify.com.

Resources:

Literature Review Automation: Search and Analyze Hundreds of Academic Papers in Minutes

NexGenData — Tue, 14 Jul 2026 15:09:58 +0000

Literature Review Automation: Search and Analyze Hundreds of Academic Papers in Minutes

You're a graduate student starting research on machine learning interpretability. Your advisor says: "Go do a comprehensive literature review. Understand what's been published, who the key authors are, what the major debates are. Come back with 100+ papers mapped."

Sounds like a three-month project, right? You picture yourself manually searching Google Scholar, downloading PDFs, reading abstracts, taking notes, building a spreadsheet. Weeks of clicking, scrolling, copying, pasting.

What if instead you could search Google Scholar for 50 different keyword combinations, automatically extract 500+ papers with metadata, rank them by citation count and relevance, identify the key authors and papers that matter, and have it all structured in a database by tomorrow morning?

That's what automated literature review scraping enables. Instead of weeks hunched over a search engine, you're done in hours.

In this guide, I'll show you how to build an automated literature review pipeline that finds the papers that matter and helps you understand the research landscape faster than traditional methods.

Why Automate Your Literature Review

Literature reviews are foundational to research, but they're also the most tedious part. Manual searching has problems:

Incomplete coverage: You search "machine learning interpretability" and get results. But you might miss papers that use different terminology like "explainability" or "transparency." Without exhaustive searching, you have blind spots.

Citation bias: You find one influential paper and follow its references. But this creates a narrow view of the field. Influential papers might overshadow important but less-cited work.

Time investment: A proper review should cover 100+ papers. Reading abstracts alone (10 seconds each) takes 17 minutes per 100 papers. Then you need to categorize them, note key findings, build a concept map. This balloons to dozens of hours.

Reproducibility: Your search process is ad-hoc. Next year, if you need to re-create the same review, you can't easily reproduce it or explain your selection criteria.

Automation solves all of these. You can search exhaustively across terminology variants, get data on thousands of papers (not just what you manually found), process them in hours, and document exactly what you searched for.

Building Your Automated Search Strategy

Before scraping, plan your searches. Think about:

Core terms: "machine learning," "deep learning," "neural networks"
Application terms: "computer vision," "NLP," "medical imaging"
Problem terms: "interpretability," "explainability," "adversarial robustness"
Method terms: "attention mechanisms," "knowledge distillation," "pruning"

For machine learning interpretability, I'd search combinations:

machine learning interpretability
deep learning interpretability
neural network interpretability
explainable AI
interpretable machine learning
LIME SHAP
attention visualization
feature importance

This gives you breadth across terminology and approaches.

Setting Up Google Scholar Scraping

Use the Apify Google Scholar Scraper to extract papers. Your input configuration:

{
  "queries": [
    "machine learning interpretability",
    "deep learning explainability",
    "interpretable AI",
    "neural network transparency",
    "LIME SHAP feature importance",
    "attention visualization",
    "adversarial robustness"
  ],
  "maxResults": 100,
  "includeMetadata": true,
  "includeCitations": true,
  "sortBy": "relevance",
  "yearFrom": 2018,
  "yearTo": 2026,
  "useChrome": true,
  "proxyConfiguration": {
    "useApifyProxy": true
  }
}

This searches 7 different queries, pulling 100 papers per query, capturing citation counts and publication metadata.

Understanding Scholar Scraper Output

Here's what you'll get (abbreviated):

{
  "papers": [
    {
      "paperId": "scholar_12345",
      "title": "Why Should I Trust You? Explaining the Predictions of Any Classifier",
      "authors": ["Marco Tulio Ribeiro", "Sameer Singh", "Carlos Guestrin"],
      "year": 2016,
      "citationCount": 12847,
      "citationTier": "landmark",
      "abstract": "Understanding why a model made a specific prediction is crucial in many applications. In this paper, we present LIME, a model-agnostic approach to interpreting predictions of any classifier...",
      "url": "https://arxiv.org/abs/1602.04938",
      "journal": "arXiv",
      "doi": "10.1145/2939672.2939778",
      "keywords": ["interpretability", "explainability", "LIME", "model-agnostic"],
      "citedBy": [
        "Scholar paper 12346",
        "Scholar paper 12347",
        "Scholar paper 12348"
      ],
      "relatedPapers": ["Scholar paper 12350", "Scholar paper 12351"],
      "scrapedAt": "2026-04-04T10:15:00Z"
    },
    {
      "paperId": "scholar_12346",
      "title": "A Unified Approach to Interpreting Model Predictions",
      "authors": ["Scott Lundberg", "Su-In Lee"],
      "year": 2017,
      "citationCount": 9234,
      "citationTier": "landmark",
      "abstract": "Understanding why a model made a specific prediction is valuable in many applications. Most existing approaches explain individual predictions, but our approach provides a unified framework...",
      "url": "https://arxiv.org/abs/1705.07874",
      "journal": "arXiv",
      "doi": "10.1145/3306127.3331093",
      "keywords": ["SHAP", "interpretability", "feature importance", "shapley values"],
      "citedBy": ["Scholar paper 12352", "Scholar paper 12353"],
      "relatedPapers": ["Scholar paper 12345", "Scholar paper 12354"],
      "scrapedAt": "2026-04-04T10:15:00Z"
    },
    {
      "paperId": "scholar_12347",
      "title": "Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization",
      "authors": ["Ramprasaath R. Selvaraju", "Abhishek Das", "Ramakrishna Vedanthi", "Michael S. Cobbe", "Devi Parikh", "Dhruv Batra"],
      "year": 2016,
      "citationCount": 8567,
      "citationTier": "landmark",
      "abstract": "We propose Grad-CAM, a technique for producing fine-grained visualization of decisions from a large class of CNN-based models, making them more transparent and interpretable...",
      "url": "https://arxiv.org/abs/1610.02055",
      "journal": "ICCV",
      "doi": "10.1109/ICCV.2017.620",
      "keywords": ["CNN interpretability", "visualization", "attention", "computer vision"],
      "citedBy": ["Scholar paper 12355", "Scholar paper 12356"],
      "relatedPapers": ["Scholar paper 12357", "Scholar paper 12358"],
      "scrapedAt": "2026-04-04T10:15:00Z"
    }
  ],
  "totalPapersFound": 687,
  "citationStats": {
    "medianCitations": 142,
    "averageCitations": 487,
    "citationDistribution": {
      "landmark": 12,
      "highly_cited": 45,
      "well_cited": 124,
      "moderately_cited": 287,
      "lightly_cited": 219
    }
  },
  "yearDistribution": {
    "2018": 89,
    "2019": 134,
    "2020": 187,
    "2021": 156,
    "2022": 98,
    "2023": 45,
    "2024": 12,
    "2025": 8,
    "2026": 2
  },
  "scrapingRunId": "run-20260404-001"
}

You now have 687 papers with structured metadata. Raw data, but incredibly valuable.

Categorizing Papers by Citation Tier

Not all papers are equal. Citation count is a proxy for impact. Create tiers:

def categorize_by_citations(papers):
    """Classify papers by citation count tiers"""
    citation_counts = [p['citationCount'] for p in papers]

    # Calculate percentiles
    p90 = sorted(citation_counts)[int(len(citation_counts) * 0.9)]
    p75 = sorted(citation_counts)[int(len(citation_counts) * 0.75)]
    p50 = sorted(citation_counts)[int(len(citation_counts) * 0.50)]

    tiers = {
        'landmark': [],      # p90+
        'highly_cited': [],  # p75 to p90
        'well_cited': [],    # p50 to p75
        'moderately_cited': [], # p25 to p50
        'lightly_cited': []   # below p25
    }

    for paper in papers:
        cites = paper['citationCount']

        if cites >= p90:
            tiers['landmark'].append(paper)
        elif cites >= p75:
            tiers['highly_cited'].append(paper)
        elif cites >= p50:
            tiers['well_cited'].append(paper)
        elif cites >= sorted(citation_counts)[int(len(citation_counts) * 0.25)]:
            tiers['moderately_cited'].append(paper)
        else:
            tiers['lightly_cited'].append(paper)

    return tiers

# Usage
tiers = categorize_by_citations(papers)

print(f"Landmark papers ({len(tiers['landmark'])}): These define the field")
for paper in tiers['landmark']:
    print(f"  - {paper['title']} ({paper['citationCount']} citations)")

print(f"\nHighly cited ({len(tiers['highly_cited'])}): Major contributions")
print(f"Well cited ({len(tiers['well_cited'])}): Solid research")
print(f"Moderately cited ({len(tiers['moderately_cited'])}): Growing area")
print(f"Lightly cited ({len(tiers['lightly_cited'])}): Niche or emerging")

Output:

Landmark papers (12): These define the field
  - Why Should I Trust You? Explaining the Predictions of Any Classifier (12847 citations)
  - A Unified Approach to Interpreting Model Predictions (9234 citations)
  - Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization (8567 citations)

Highly cited (45): Major contributions
Well cited (124): Solid research
Moderately cited (287): Growing area
Lightly cited (219): Niche or emerging

Start with landmark papers. They're foundational. Then move to highly cited. You'll understand 80% of the field by reading 50 papers instead of 687.

Identifying Key Authors and Research Communities

Who are the influential voices in this field? Build an author network:

def identify_key_authors(papers):
    """Find most published and cited authors"""
    author_stats = {}

    for paper in papers:
        for author in paper.get('authors', []):
            if author not in author_stats:
                author_stats[author] = {
                    'paperCount': 0,
                    'totalCitations': 0,
                    'papers': []
                }

            author_stats[author]['paperCount'] += 1
            author_stats[author]['totalCitations'] += paper['citationCount']
            author_stats[author]['papers'].append(paper['title'])

    # Score authors by influence
    for author in author_stats:
        avg_cites = author_stats[author]['totalCitations'] / author_stats[author]['paperCount']
        author_stats[author]['influenceScore'] = avg_cites * author_stats[author]['paperCount']

    # Sort by influence
    top_authors = sorted(author_stats.items(), key=lambda x: x[1]['influenceScore'], reverse=True)

    return top_authors

# Usage
top_authors = identify_key_authors(papers)

for i, (author, stats) in enumerate(top_authors[:15], 1):
    print(f"{i}. {author}")
    print(f"   Papers: {stats['paperCount']}, Avg Citations: {round(stats['totalCitations']/stats['paperCount'], 0)}")

Output:

1. Sameer Singh
   Papers: 12, Avg Citations: 5432
2. Su-In Lee
   Papers: 10, Avg Citations: 4821
3. Carlos Guestrin
   Papers: 11, Avg Citations: 4756
...

Follow these authors. Read their recent papers. Follow their citations. You're now inside the research community.

Building Your Research Landscape Map

Cluster papers by keyword to understand research areas:

from collections import Counter

def map_research_landscape(papers):
    """Identify major research themes"""
    all_keywords = []

    for paper in papers:
        all_keywords.extend(paper.get('keywords', []))

    # Count keyword frequency
    keyword_freq = Counter(all_keywords)

    # Identify clusters
    clusters = {}
    for keyword, count in keyword_freq.most_common(20):
        papers_with_keyword = [p for p in papers if keyword in p.get('keywords', [])]
        avg_citations = sum(p['citationCount'] for p in papers_with_keyword) / len(papers_with_keyword)

        clusters[keyword] = {
            'paperCount': count,
            'averageCitations': round(avg_citations, 0),
            'topPaper': max(papers_with_keyword, key=lambda p: p['citationCount'])['title']
        }

    return clusters

# Usage
landscape = map_research_landscape(papers)

print("Research Landscape:")
for keyword, stats in sorted(landscape.items(), key=lambda x: x[1]['paperCount'], reverse=True)[:10]:
    print(f"\n{keyword}")
    print(f"  Papers: {stats['paperCount']}")
    print(f"  Avg Citations: {stats['averageCitations']}")
    print(f"  Top Paper: {stats['topPaper'][:60]}...")

Output:

Research Landscape:

interpretability
  Papers: 187
  Avg Citations: 1243
  Top Paper: Why Should I Trust You? Explaining the Predictions of Any Classifier

explainability
  Papers: 156
  Avg Citations: 989
  Top Paper: A Unified Approach to Interpreting Model Predictions

feature importance
  Papers: 123
  Avg Citations: 754
  Top Paper: [...]

attention visualization
  Papers: 98
  Avg Citations: 612
  Top Paper: [...]

This shows you where research concentration is. Interpretability dominates. Explainability is close behind. Attention visualization is emerging. You now understand the landscape.

Creating Your Literature Review Database

Export everything to a structured format you can query:

import sqlite3
import json

def build_literature_database(papers, db_path="literature_review.db"):
    """Create queryable database of papers"""
    conn = sqlite3.connect(db_path)
    cursor = conn.cursor()

    # Create tables
    cursor.execute('''
        CREATE TABLE papers (
            paper_id TEXT PRIMARY KEY,
            title TEXT,
            authors TEXT,
            year INTEGER,
            citations INTEGER,
            tier TEXT,
            abstract TEXT,
            url TEXT,
            doi TEXT
        )
    ''')

    cursor.execute('''
        CREATE TABLE keywords (
            paper_id TEXT,
            keyword TEXT,
            FOREIGN KEY (paper_id) REFERENCES papers(paper_id)
        )
    ''')

    # Insert papers
    for paper in papers:
        cursor.execute('''
            INSERT INTO papers VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)
        ''', (
            paper['paperId'],
            paper['title'],
            json.dumps(paper['authors']),
            paper['year'],
            paper['citationCount'],
            paper.get('citationTier', 'unknown'),
            paper['abstract'][:500],  # Truncate
            paper['url'],
            paper['doi']
        ))

        # Insert keywords
        for keyword in paper.get('keywords', []):
            cursor.execute('''
                INSERT INTO keywords VALUES (?, ?)
            ''', (paper['paperId'], keyword))

    conn.commit()

    # Create useful queries
    print("Papers by tier:")
    for tier in ['landmark', 'highly_cited', 'well_cited']:
        count = cursor.execute('SELECT COUNT(*) FROM papers WHERE tier = ?', (tier,)).fetchone()[0]
        print(f"  {tier}: {count}")

    print("\nTop papers by citations:")
    top = cursor.execute('SELECT title, citations FROM papers ORDER BY citations DESC LIMIT 5').fetchall()
    for title, cites in top:
        print(f"  {cites}: {title[:60]}...")

    conn.close()

# Usage
build_literature_database(papers)

Now you have a queryable database. Want all papers about "SHAP" from 2020-2022? Query it. Want papers cited 100+ times? Query it. Want all papers by a specific author? Query it.

Real-World Application: Your Thesis Introduction

You're writing your thesis introduction. You need to cite 20-30 papers that establish the problem, show what's been done, and justify your approach.

Your automated review gives you:

The 12 landmark papers that define the field (cite these for credibility)
The 45 highly cited papers from the last 2 years (cite these to show you're current)
The key debates and opposing viewpoints (show you understand nuance)
The research gap you're filling (justify your work)

Where previously this took months, you have it in days. Your literature review is comprehensive, well-documented, and defensible.

Getting Started

Visit the Apify Google Scholar Scraper and start with a single research topic. Search 10-15 keyword combinations, extract 500+ papers, and run the analysis above.

Within a day, you'll have mapped a research landscape that would take weeks manually. That's time you can spend on actual research instead of admin work.

Automation is how modern researchers work smarter.

Neighborhood-by-Neighborhood: Comparing Real Estate Markets with Redfin Data

NexGenData — Thu, 09 Jul 2026 19:06:57 +0000

Neighborhood-by-Neighborhood: Comparing Real Estate Markets with Redfin Data

You're house hunting in a new city. You've narrowed it down to five neighborhoods that seem promising based on research. But now what? You could spend weeks touring each area, comparing prices, understanding market temperatures, and trying to figure out which neighborhood actually offers the best value.

Or you could extract data on hundreds of properties across all five neighborhoods in a single afternoon, then make a decision backed by analysis instead of gut feeling.

Real estate decisions often come down to data: price per square foot, how long properties sit on the market, what's actually selling versus what's listed. Redfin is the most comprehensive source of this data. But manually collecting it is time-consuming and error-prone.

What if you could scrape every property listing in five neighborhoods, compare pricing tiers, analyze market velocity, and identify hidden opportunities? That's exactly what automated Redfin scraping enables.

In this guide, I'll show you how to systematically compare neighborhoods and make data-driven real estate decisions.

Why Neighborhood Comparison Matters

Before committing $500K+ to a home purchase, you need answers that go beyond "nice area" or "good schools."

Price stratification: Is the neighborhood firmly $400-500K or does it break into affordable ($300K), mid-market ($500K), and premium ($750K+) segments? This tells you about diversity and future growth potential.

Days on market (DOM): Properties in Neighborhood A sell in 15 days. Neighborhood B takes 45 days. That DOM difference signals buyer demand. Low DOM means competitive market, high DOM might mean overpriced or undermarketed properties.

Property age and condition: Are you buying in a vintage neighborhood (pre-1960s character, likely higher maintenance) or a newer suburban area (built 2000+, modern systems)? This impacts future costs.

Market velocity: How many homes sold last month versus last quarter? High velocity signals active buyers. Low velocity might signal a cooling market.

Price trends: Is the market appreciating, stable, or declining? A neighborhood that gained 8% last year signals growth. One that declined might signal softening demand.

Without data, you're guessing. With data, you're strategic.

Setting Up Your Neighborhood Comparison

Decide which neighborhoods you want to compare. For a typical search, I'd compare 5 neighborhoods and look for 100+ active listings in each.

Use the Apify Redfin Scraper to extract listing data. Your input configuration:

{
  "startUrls": [
    {
      "url": "https://www.redfin.com/neighborhood/Capitol-Hill/Seattle-WA"
    },
    {
      "url": "https://www.redfin.com/neighborhood/Fremont/Seattle-WA"
    },
    {
      "url": "https://www.redfin.com/neighborhood/Ballard/Seattle-WA"
    },
    {
      "url": "https://www.redfin.com/neighborhood/Wallingford/Seattle-WA"
    },
    {
      "url": "https://www.redfin.com/neighborhood/Green-Lake/Seattle-WA"
    }
  ],
  "proxyConfiguration": {
    "useApifyProxy": true
  },
  "useChrome": true,
  "maxListings": 150
}

Run this and you'll get structured data on every listing in each neighborhood.

Understanding Redfin Scraper Output

Here's what you'll get (abbreviated):

{
  "neighborhoods": [
    {
      "name": "Capitol Hill",
      "city": "Seattle",
      "state": "WA",
      "listings": [
        {
          "id": "redfin_12345",
          "address": "1234 E Pike Street, Seattle, WA 98102",
          "price": 750000,
          "beds": 3,
          "baths": 2,
          "sqft": 1850,
          "yearBuilt": 1925,
          "pricePerSqft": 405,
          "listingStatus": "for_sale",
          "daysOnMarket": 14,
          "lastSoldPrice": 695000,
          "lastSoldDate": "2025-10-15",
          "propertyType": "single_family",
          "url": "https://www.redfin.com/WA/Seattle/1234-E-Pike-Street-98102"
        },
        {
          "id": "redfin_12346",
          "address": "5678 E Olive Street, Seattle, WA 98102",
          "price": 1200000,
          "beds": 4,
          "baths": 3,
          "sqft": 2650,
          "yearBuilt": 1932,
          "pricePerSqft": 453,
          "listingStatus": "for_sale",
          "daysOnMarket": 8,
          "lastSoldPrice": 1100000,
          "lastSoldDate": "2024-06-20",
          "propertyType": "single_family",
          "url": "https://www.redfin.com/WA/Seattle/5678-E-Olive-Street-98102"
        }
      ],
      "neighborhoodStats": {
        "medianListPrice": 850000,
        "medianPricePerSqft": 425,
        "medianDaysOnMarket": 16,
        "activeListings": 127,
        "soldLastMonth": 34,
        "priceChange90Days": 2.3,
        "averageYearBuilt": 1948
      }
    },
    {
      "name": "Fremont",
      "city": "Seattle",
      "state": "WA",
      "listings": [
        {
          "id": "redfin_12347",
          "address": "9012 Wallingford Avenue N, Seattle, WA 98103",
          "price": 625000,
          "beds": 3,
          "baths": 1.5,
          "sqft": 1650,
          "yearBuilt": 1950,
          "pricePerSqft": 379,
          "listingStatus": "for_sale",
          "daysOnMarket": 22,
          "lastSoldPrice": 580000,
          "lastSoldDate": "2025-02-10",
          "propertyType": "single_family",
          "url": "https://www.redfin.com/WA/Seattle/9012-Wallingford-Avenue-N-98103"
        }
      ],
      "neighborhoodStats": {
        "medianListPrice": 675000,
        "medianPricePerSqft": 389,
        "medianDaysOnMarket": 24,
        "activeListings": 98,
        "soldLastMonth": 28,
        "priceChange90Days": -0.8,
        "averageYearBuilt": 1956
      }
    }
  ],
  "scrapedAt": "2026-04-04T14:30:00Z"
}

This raw data is the foundation. Now analyze it.

Building Your Neighborhood Comparison Matrix

Extract key metrics and compare:

import statistics
from collections import defaultdict

def analyze_neighborhood(listings):
    """Extract summary metrics from listings"""
    prices = [l['price'] for l in listings]
    sqft_values = [l['sqft'] for l in listings]
    dom_values = [l['daysOnMarket'] for l in listings]
    price_per_sqft = [l['pricePerSqft'] for l in listings]

    analysis = {
        'totalListings': len(listings),
        'medianPrice': statistics.median(prices),
        'avgPrice': round(statistics.mean(prices), 0),
        'priceRange': f"${min(prices):,} - ${max(prices):,}",
        'medianPricePerSqft': statistics.median(price_per_sqft),
        'medianSqft': statistics.median(sqft_values),
        'medianDaysOnMarket': statistics.median(dom_values),
        'avgDaysOnMarket': round(statistics.mean(dom_values), 1),
        'priceStdDev': round(statistics.stdev(prices), 0),
        '3bedCount': len([l for l in listings if l['beds'] == 3]),
        'avgYearBuilt': round(statistics.mean([l['yearBuilt'] for l in listings]), 0),
        'priceAppreciation': calculate_appreciation(listings)
    }

    return analysis

def calculate_appreciation(listings):
    """Compare last sold price to current list price"""
    appreciations = []

    for listing in listings:
        if listing['lastSoldPrice'] and listing['lastSoldDate']:
            appreciation_pct = ((listing['price'] - listing['lastSoldPrice']) / listing['lastSoldPrice']) * 100
            appreciations.append(appreciation_pct)

    if appreciations:
        return round(statistics.mean(appreciations), 1)
    return None

# Example: Compare 5 neighborhoods
neighborhoods = {
    'Capitol Hill': 750000,
    'Fremont': 675000,
    'Ballard': 625000,
    'Wallingford': 595000,
    'Green Lake': 680000
}

comparison_table = {
    name: analyze_neighborhood(data[name]['listings'])
    for name, neighborhood_data in data.items()
}

Output table:

Neighborhood	Median Price	$/Sqft	Median DOM	Avg Year Built	Appreciation
Capitol Hill	$850K	$425	16	1948	+7.2%
Fremont	$675K	$389	24	1956	+5.8%
Ballard	$625K	$379	19	1952	+3.1%
Wallingford	$595K	$368	21	1961	+2.4%
Green Lake	$680K	$405	18	1958	+6.9%

What this tells you:

Capitol Hill is the most expensive but also the hottest (lowest DOM). Buyers move quickly.
Fremont is cheaper but slower to sell (24 days). Could be softer demand or overpriced inventory.
Wallingford is the most affordable but also appreciated least (2.4%). Possibly stabilizing after growth.
Ballard represents middle ground: decent price, reasonable DOM, moderate appreciation.

Analyzing Price Segmentation

Don't just look at medians. Understand how prices distribute:

def price_tier_analysis(listings):
    """Break listings into price tiers"""
    prices = sorted([l['price'] for l in listings])
    n = len(prices)

    tiers = {
        'under_500k': len([p for p in prices if p < 500000]),
        '500k_650k': len([p for p in prices if 500000 <= p < 650000]),
        '650k_850k': len([p for p in prices if 650000 <= p < 850000]),
        '850k_1.2m': len([p for p in prices if 850000 <= p < 1200000]),
        'over_1.2m': len([p for p in prices if p >= 1200000])
    }

    return tiers

# Per neighborhood, see where inventory concentrates
for neighborhood, listings in data.items():
    tiers = price_tier_analysis(listings)
    print(f"{neighborhood}: {tiers}")

# Output:
# Capitol Hill: {'under_500k': 3, '500k_650k': 12, '650k_850k': 45, '850k_1.2m': 52, 'over_1.2m': 15}
# Fremont: {'under_500k': 8, '500k_650k': 38, '650k_850k': 42, '850k_1.2m': 8, 'over_1.2m': 2}

This reveals inventory concentration. Capitol Hill has strong inventory above $850K (premium market). Fremont concentrates in the $500-850K range (mid-market). This distribution affects your buying power and future resale potential.

Days on Market as a Market Temperature Signal

DOM is critical. It tells you if a neighborhood is hot or cooling:

def market_temperature(listings):
    """Classify market based on DOM"""
    dom_values = [l['daysOnMarket'] for l in listings]
    median_dom = statistics.median(dom_values)

    if median_dom < 10:
        return 'HOT - Inventory moves fast, competitive offers expected'
    elif median_dom < 20:
        return 'WARM - Normal market, standard competition'
    elif median_dom < 40:
        return 'COOL - Slower market, negotiation potential'
    else:
        return 'COLD - Buyer advantage, potential price concessions'

# Check each neighborhood
for neighborhood, stats in comparison_table.items():
    temp = market_temperature(data[neighborhood]['listings'])
    print(f"{neighborhood}: {temp}")

# Output:
# Capitol Hill: HOT - Inventory moves fast, competitive offers expected
# Fremont: COOL - Slower market, negotiation potential
# Ballard: WARM - Normal market, standard competition

A HOT market means you need to move fast and be prepared to compete on price. A COOL market gives you negotiating leverage.

Building a Multi-Neighborhood Workflow

Here's a Python script for analyzing 5 neighborhoods end-to-end:

import requests
import json
from datetime import datetime

def run_neighborhood_analysis(neighborhood_urls):
    """Main workflow for comparing neighborhoods"""

    # Step 1: Scrape all neighborhoods
    print("Scraping Redfin data...")
    scraping_results = scrape_redfin(neighborhood_urls)

    # Step 2: Parse and analyze each neighborhood
    print("Analyzing neighborhoods...")
    comparisons = {}
    for neighborhood, listings in scraping_results.items():
        comparisons[neighborhood] = {
            'summary': analyze_neighborhood(listings),
            'priceTiers': price_tier_analysis(listings),
            'temperature': market_temperature(listings)
        }

    # Step 3: Create comparison matrix
    print("Building comparison matrix...")
    matrix = create_comparison_matrix(comparisons)

    # Step 4: Generate insights
    print("Generating insights...")
    insights = generate_insights(comparisons)

    # Step 5: Export results
    export_results(matrix, insights, comparisons)

    return comparisons, insights

def generate_insights(comparisons):
    """Extract actionable insights"""
    insights = {
        'bestValue': min(comparisons.items(), key=lambda x: x[1]['summary']['medianPrice'])[0],
        'fastestMarket': min(comparisons.items(), key=lambda x: x[1]['summary']['medianDaysOnMarket'])[0],
        'appreciatingFastest': max(comparisons.items(), key=lambda x: x[1]['summary']['priceAppreciation'])[0],
        'largestInventory': max(comparisons.items(), key=lambda x: x[1]['summary']['totalListings'])[0],
    }

    return insights

# Example usage
neighborhood_urls = [
    "https://www.redfin.com/neighborhood/Capitol-Hill/Seattle-WA",
    "https://www.redfin.com/neighborhood/Fremont/Seattle-WA",
    "https://www.redfin.com/neighborhood/Ballard/Seattle-WA",
    "https://www.redfin.com/neighborhood/Wallingford/Seattle-WA",
    "https://www.redfin.com/neighborhood/Green-Lake/Seattle-WA"
]

comparisons, insights = run_neighborhood_analysis(neighborhood_urls)
print(json.dumps(insights, indent=2))

Real-World Application: Making Your Buy Decision

You're torn between three neighborhoods. The data shows:

Neighborhood A: $850K median, HOT market (15 days DOM), appreciating 7.2%
Neighborhood B: $650K median, COOL market (28 days DOM), appreciating 2.1%
Neighborhood C: $725K median, WARM market (19 days DOM), appreciating 5.9%

The choice depends on your priorities:

If you value appreciation and can handle competition: Neighborhood A
If you want negotiation leverage and affordability: Neighborhood B
If you want balance: Neighborhood C

With data, you're not guessing. You're making an informed decision.

Getting Started

Visit the Apify Redfin Scraper and scrape five neighborhoods you're considering. Run the analysis scripts above. You'll have clearer decision-making in a few hours.

Your home is likely the biggest financial decision you'll make. Make it backed by data.

KOSPI Stock Screener API: Korean Equities Market Data Pipeline

NexGenData — Wed, 08 Jul 2026 19:51:14 +0000

中文版: 中文版本

TL;DR. The Korea Composite Stock Price Index (KOSPI) covers ~800 main-board listings on the Korea Exchange — Samsung Electronics, SK Hynix, Hyundai Motor, LG Energy Solution, the Naver / Kakao internet duo, and the broader chaebol ecosystem. For US- and ASEAN-based research desks, Korea is typically the largest single non-Japan Asian allocation, with semiconductors and EV-batteries the dominant sector exposures. The Korea Exchange and FnGuide / DataGuide are the canonical commercial data providers; for screening, factor research, and dashboarding the public KRX data combined with structured aggregation can replicate most of the workflow at a fraction of the cost. This guide covers KOSPI market structure, the fields a Korea equity researcher actually needs, and a working Python pipeline using the KOSPI Stock Screener actor.

What's in the KOSPI universe

KOSPI is the main board of the Korea Exchange (KRX); KOSDAQ is the parallel growth board. As of 2026 the KOSPI hosts approximately 800 listings, dominated by:

Semiconductors — Samsung Electronics (005930), SK Hynix (000660). These two alone routinely represent ~25-30% of KOSPI total market cap.
Automotive — Hyundai Motor (005380), Kia (000270), and the Hyundai-supplier chain.
EV battery — LG Energy Solution (373220), Samsung SDI (006400), SK Innovation (096770).
Internet — Naver (035420), Kakao (035720). Korea's local internet incumbents, neither has a meaningful Western analog.
Banking and financials — KB Financial (105560), Shinhan (055550), Hana (086790).
Steel and shipbuilding — POSCO (005490), Hyundai Heavy.
K-content — major entertainment companies cross-listed across KOSPI and KOSDAQ.

Korea's chaebol structure means cross-holdings within conglomerate groups are an important second-order analytical dimension — Samsung Electronics, Samsung SDI, and Samsung Biologics are separately listed but linked. Any factor model needs to handle the group-affiliation dimension.

KOSPI market microstructure

Trading hours — 09:00–15:30 KST with an opening call auction at 08:30–09:00 and closing call auction at 15:20–15:30.
Daily price limit — ±30% on KOSPI (loosened from ±15% in 2015). KOSDAQ is also ±30%.
Settlement — T+2.
Lot size — 1 share for most listings (historically 10).
Short selling rules — Korea periodically imposes short-selling bans during market stress; the 2023-2024 ban and the 2024 partial resumption are recent examples that affect any short-side backtest.

The Foreign Investor Registration regime (KRX requires foreign investors to register for an investor ID) is no longer the bottleneck it was pre-2023 — the requirement was largely lifted in late 2023 — but it remains relevant for data-source attribution and historical comparison.

Working Python example

The KOSPI Stock Screener actor returns the KOSPI (and optionally KOSDAQ) universe with quotes, market cap in KRW, sector tags using KRX industry codes, and chaebol group affiliation. Curl:


    curl -X POST "https://api.apify.com/v2/acts/nexgendata~kospi-stock-screener/run-sync-get-dataset-items?token=$APIFY_TOKEN" \
      -H "Content-Type: application/json" \
      -d '{"market": "KOSPI", "minMarketCapKRW": 1000000000000, "sectors": ["Semiconductors","EV Battery"], "maxResults": 500}'

Python — large-cap KOSPI screen:


    import os, json, urllib.request

    APIFY_TOKEN = os.environ["APIFY_TOKEN"]
    ACTOR = "nexgendata~kospi-stock-screener"

    payload = json.dumps({
        "market": "KOSPI",
        "minMarketCapKRW": 1_000_000_000_000,
        "includeGroupAffiliation": True,
        "maxResults": 500,
    }).encode("utf-8")

    url = f"https://api.apify.com/v2/acts/{ACTOR}/run-sync-get-dataset-items?token={APIFY_TOKEN}"
    req = urllib.request.Request(url, data=payload, method="POST",
                                  headers={"Content-Type": "application/json"})

    with urllib.request.urlopen(req, timeout=600) as r:
        stocks = json.loads(r.read())

    for s in sorted(stocks, key=lambda x: -x.get("marketCapKRW", 0))[:20]:
        print(f"{s['ticker']} | {s.get('nameEn') or s['name']} | "
              f"mcap={s['marketCapKRW']/1e12:.1f}T KRW | group={s.get('group','-')}")

Returned shape:


    {
      "ticker": "005930",
      "isin": "KR7005930003",
      "name": "삼성전자",
      "nameEn": "Samsung Electronics Co., Ltd.",
      "market": "KOSPI",
      "krxIndustry": "Electrical, electronic equipment",
      "sector": "Semiconductors",
      "group": "Samsung",
      "lastPrice": 78400,
      "marketCapKRW": 467500000000000,
      "peTTM": 14.6,
      "pbMRQ": 1.4,
      "dividendYieldTTM": 2.0,
      "foreignOwnershipPct": 53.2,
      "tradingStatus": "NORMAL"
    }

Cost comparison: KOSPI data sources

Source	Coverage	English	Annual cost
FnGuide / DataGuide	Full KOSPI + KOSDAQ + estimates	Yes	~$3k–$15k+ per seat
Bloomberg Terminal (KS / KQ tickers)	Full + news	Yes	~$24k per seat
Refinitiv Eikon Korea	Full	Yes	~$22k per seat
KRX own data services	Full	Yes	Subscription, varies
Yahoo Finance Korea (.KS / .KQ)	Quotes	Yes	Free, terms-of-use limits
KOSPI Stock Screener actor	KOSPI + optionally KOSDAQ, screening fields	Yes	PPE per screen

FnGuide / DataGuide remain canonical for Korean sell-side and institutional buy-side desks — particularly for forward-estimate data (the "consensus" feeds that local analysts use) and the unique-to-Korea fundamentals coverage. For screening, factor research, and dashboarding without the forward-estimate dependency, the public KRX-derived stack is meaningfully sufficient.

What this enables

Sector dashboards — semiconductor cycle tracking via Samsung + SK Hynix, EV-battery cycle via LGES + Samsung SDI + SK Innovation, internet via Naver + Kakao
Factor research on the KOSPI universe — value, quality, momentum factor portfolios with chaebol-group adjustment
Foreign ownership monitoring — KRX publishes daily foreign-ownership percentages; tracking deltas at the name level is a meaningful flow signal for Korea-equity strategies
ASEAN-cluster regional dashboards — pair KOSPI with SGX, PSE Edge, and the broader regional disclosure feeds

KOSPI vs. KOSDAQ practical handling

For Korea-focused funds the two boards are typically run as a single research universe with sector-weighted construction. KOSPI dominates large-cap; KOSDAQ dominates K-content, biotech, and EV-battery small/mid. The actor supports either or both — most use cases pull both.

Pairing with disclosure data

Quote and fundamentals data alone are insufficient for any time-sensitive research workflow on Korean equities — the relevant news flow comes through KIND disclosures. A complete Korea research pipeline pulls the KOSPI screener for the static universe + fundamentals, then KIND for the real-time event flow on holdings. Adjacent feeds: SEC Event Router for Korean ADRs on US exchanges (a small but growing cohort), and HKEX for Korea-HK secondary listings.

One execution note

Korea applies a securities transaction tax (currently 0.18% combined for KOSPI sells, scheduled for further reduction) that materially affects any high-turnover backtest. Factor-rotation strategies in Korea should explicitly model the tax in addition to spread. For long-only factor portfolios with monthly rebalance the impact is small; for daily or weekly rotation strategies it is the dominant cost.

Foreign ownership caps and the FIR regime

Although the Foreign Investor Registration certificate requirement was largely lifted in late 2023, several KOSPI sectors retain statutory foreign-ownership caps under Korean law — telecoms, broadcasting, defense-adjacent, and a handful of utilities still have explicit caps ranging from 25% to 49%. For any Korea-focused strategy that holds capped-sector names, the headroom (current foreign ownership vs. statutory cap) is a first-class data field. KRX publishes daily foreign-ownership statistics that flow into the actor's foreignOwnershipPct field; the cap-aware research workflow filters on capped sectors and screens for low-headroom names that can hit forced unwind triggers.

Chaebol cross-holdings and the affiliation field

The KRX classifies issuers by chaebol group affiliation under the Monopoly Regulation and Fair Trade Act — the "large business group" designation that includes the Samsung group, Hyundai Motor group, SK group, LG group, Lotte group, and roughly 75 others. The affiliation matters for any factor model that needs to handle correlated returns within a group (Samsung Electronics and Samsung SDI exhibit higher pairwise correlation than two unrelated semis) and for any governance-quality factor. The actor returns the group field, which lets you build group-neutral factor portfolios or, conversely, group-concentration screens.

Get started: Pull your first KOSPI screen free at the KOSPI Stock Screener actor page.

Build an Automated Hacker News Tracker for Your Tech Niche

NexGenData — Tue, 07 Jul 2026 15:42:34 +0000

Build an Automated Hacker News Tracker for Your Tech Niche

You're building a React analytics tool. A major incident hits HN with the title "React Performance Issues at Scale: Here's What We Found" and it hits #3. By the time you see it, 8 hours later, the conversation is 200 comments deep and you've missed the window to contribute meaningfully.

Or worse, you don't see it at all, and miss a critical discussion about your product space.

Hacker News moves fast. Trends emerge, discussions peak, and fade within hours. Manual monitoring means either constantly refreshing the site or missing valuable signals about your market, competitors, and community sentiment.

What if instead you could set up a system that automatically monitors HN, filters for topics relevant to your niche, identifies viral posts early, and sends you real-time alerts? That's the power of an automated HN tracker.

In this post, I'll walk you through building one that catches the signals that matter for your business.

Why Automated HN Monitoring Matters

Hacker News is where engineers congregate. If your product is developer-focused, your market is here. But it's not just about finding mentions of your product. It's about understanding:

What problems are engineers actively discussing? If 500 engineers are discussing "database migration horror stories" in a single thread, that's market research. Maybe your tool should position around frictionless migrations.

How fast do certain topics trend? Posts about AI security shoot to #1 in 3 hours. Traditional cloud infrastructure discussions slowly climb over days. Understanding velocity tells you where excitement is building.

What are the sentiment patterns? A post about a competitor hits #5. Read the comments. Are people praising them or criticizing them? That's competitive intel.

Where are discussion gaps? No one's talking about a feature you know matters. Either the market doesn't care yet, or there's an education opportunity.

How early can you detect trends? A post about Rust for systems programming sits at #50 with 20 comments. Six hours later, it's #2 with 400 comments. Catching it at #50 means you can be an early voice, shaping the narrative.

Manual monitoring can't scale to capture this. Automated monitoring can.

Setting Up Your Hacker News Scraper

Start by understanding what you want to track. For a developer tools company, I'd monitor:

Posts about the languages/frameworks you support
Posts about your competitors
Posts about the problems you solve
General trends in your category (databases, APIs, testing, etc.)

Use the Apify Hacker News Scraper to extract the front page and top stories on a regular schedule.

Your basic input configuration:

{
  "startUrls": [
    {
      "url": "https://news.ycombinator.com/newest"
    },
    {
      "url": "https://news.ycombinator.com/best"
    }
  ],
  "maxPostsPerPage": 30,
  "scrapeComments": true,
  "maxComments": 20,
  "pageTimeout": 30000,
  "useChrome": true,
  "proxyConfiguration": {
    "useApifyProxy": true
  }
}

This pulls the newest and best posts with top comments, giving you context.

Understanding Your HN Scraper Output

Here's what a typical run returns (abbreviated):

{
  "posts": [
    {
      "postId": "41234567",
      "title": "We Rewrote Our Backend in Rust and Gained 10x Performance",
      "url": "https://example.com/rust-backend",
      "author": "john_eng",
      "score": 1247,
      "commentCount": 342,
      "timestamp": "2026-04-04T14:23:00Z",
      "category": "full-stack",
      "rank": 1,
      "comments": [
        {
          "commentId": "41234568",
          "author": "sarah_dev",
          "text": "Did you consider Go instead? Rust has a steep learning curve for teams.",
          "score": 245,
          "timestamp": "2026-04-04T14:35:00Z",
          "depth": 1
        },
        {
          "commentId": "41234569",
          "author": "john_eng",
          "text": "Go wasn't fast enough for our use case. Rust gave us the performance we needed without GC pauses.",
          "score": 189,
          "timestamp": "2026-04-04T14:42:00Z",
          "depth": 2
        }
      ]
    },
    {
      "postId": "41234570",
      "title": "PostgreSQL Sharding Lessons from Running 1000 Databases",
      "url": "https://example.com/postgres-sharding",
      "author": "db_expert",
      "score": 892,
      "commentCount": 156,
      "timestamp": "2026-04-04T12:11:00Z",
      "category": "databases",
      "rank": 2,
      "comments": [
        {
          "commentId": "41234571",
          "author": "ops_person",
          "text": "Good breakdown. We went with CockroachDB instead for similar guarantees.",
          "score": 134,
          "timestamp": "2026-04-04T12:45:00Z",
          "depth": 1
        }
      ]
    },
    {
      "postId": "41234571",
      "title": "Why We're Migrating Away from Kubernetes",
      "url": "https://example.com/k8s-migration",
      "author": "infra_lead",
      "score": 654,
      "commentCount": 201,
      "timestamp": "2026-04-04T10:05:00Z",
      "category": "infrastructure",
      "rank": 3,
      "comments": [
        {
          "commentId": "41234572",
          "author": "k8s_maintainer",
          "text": "What specific pain points drove the decision? K8s can be overkill for many orgs.",
          "score": 178,
          "timestamp": "2026-04-04T10:22:00Z",
          "depth": 1
        }
      ]
    }
  ],
  "scrapingRunId": "run-20260404-1",
  "scrapedAt": "2026-04-04T15:00:00Z",
  "totalPostsScraped": 30
}

Notice the structure: posts have scores and comment counts, making it easy to identify which discussions are heating up.

Building Topic Filtering and Alerts

Raw data isn't actionable. You need filtering. Let's say you're a DevOps tool company focused on Kubernetes. Create a filter that catches relevant posts:

const filterRelevantPosts = (posts) => {
  const relevantKeywords = [
    'kubernetes', 'k8s', 'docker', 'container', 'devops',
    'infrastructure', 'deployment', 'orchestration', 'microservices'
  ];

  const competitorKeywords = [
    'hashicorp', 'terraform', 'aws', 'gke', 'eks'
  ];

  return posts.filter(post => {
    const titleLower = post.title.toLowerCase();
    const isRelevant = relevantKeywords.some(kw => titleLower.includes(kw));
    const isCompetitorMention = competitorKeywords.some(kw => titleLower.includes(kw));

    return isRelevant || isCompetitorMention;
  }).map(post => ({
    ...post,
    relevanceScore: calculateRelevance(post),
    sentimentTrend: analyzeSentiment(post.comments)
  }));
};

const calculateRelevance = (post) => {
  // Score based on engagement and recency
  const ageMinutes = (Date.now() - new Date(post.timestamp).getTime()) / (1000 * 60);
  const engagementScore = post.score + (post.commentCount * 5);
  const velocityBoost = ageMinutes < 60 ? 1.5 : 1.0; // Recent posts weighted higher

  return Math.round(engagementScore * velocityBoost);
};

const analyzeSentiment = (comments) => {
  const positive = comments.filter(c => c.text.includes('great') || c.text.includes('excellent')).length;
  const negative = comments.filter(c => c.text.includes('bad') || c.text.includes('issue')).length;

  return {
    positiveCount: positive,
    negativeCount: negative,
    trend: positive > negative ? 'positive' : negative > positive ? 'negative' : 'neutral'
  };
};

Running this filter on your scraped data returns only the posts that matter to your niche, scored by relevance.

Setting Up Apify Schedules and Webhooks

Don't run this manually. Schedule your scraper to run every 2 hours during business hours:

Go to the Apify Hacker News Scraper
Create a scheduled task (Schedules tab)
Set frequency: "0 */2 * * *" (every 2 hours)
Configure a webhook to POST results to your backend

Your webhook endpoint receives the fresh data and immediately:

Filters for relevant posts
Compares against the previous run (has post ranking changed?)
Identifies newly trending posts (went from #50 to #5)
Routes high-relevance posts to Slack

Slack Integration: Real-Time Alerts

Here's how to send filtered posts to Slack:

const sendSlackAlert = async (post, relevanceScore) => {
  const webhookUrl = process.env.SLACK_WEBHOOK_URL;

  const color = relevanceScore > 500 ? '#ff6b35' : '#4CAF50';
  const emoji = post.sentimentTrend === 'positive' ? ':thumbsup:' : ':warning:';

  const slackPayload = {
    attachments: [
      {
        color: color,
        title: `${emoji} ${post.title}`,
        title_link: `https://news.ycombinator.com/item?id=${post.postId}`,
        fields: [
          {
            title: 'Score',
            value: `${post.score}`,
            short: true
          },
          {
            title: 'Comments',
            value: `${post.commentCount}`,
            short: true
          },
          {
            title: 'Relevance',
            value: `${relevanceScore}/1000`,
            short: true
          },
          {
            title: 'Sentiment',
            value: `Positive: ${post.sentimentTrend.positiveCount}, Negative: ${post.sentimentTrend.negativeCount}`,
            short: true
          }
        ],
        footer: `Posted by ${post.author} at ${new Date(post.timestamp).toLocaleTimeString()}`
      }
    ]
  };

  await fetch(webhookUrl, {
    method: 'POST',
    headers: { 'Content-Type': 'application/json' },
    body: JSON.stringify(slackPayload)
  });
};

Now your team gets a Slack notification within 5 minutes of a relevant post going viral. You can comment early, get visibility, build credibility.

Tracking Trends Over Time

To identify emerging trends, compare multiple runs:

const identifyTrendingTopics = (currentRun, previousRun) => {
  const currentPosts = new Map(currentRun.posts.map(p => [p.postId, p]));
  const previousRanks = new Map(previousRun.posts.map((p, idx) => [p.postId, idx]));

  const trending = [];

  currentRun.posts.forEach((post, currentRank) => {
    const previousRank = previousRanks.get(post.postId);

    if (previousRank !== undefined) {
      const rankChange = previousRank - currentRank;
      const scoreChange = post.score - (currentPosts.get(post.postId)?.score || 0);

      if (rankChange > 5 || scoreChange > 100) {
        trending.push({
          postId: post.postId,
          title: post.title,
          rankImprovement: rankChange,
          scoreGain: scoreChange,
          currentRank: currentRank
        });
      }
    }
  });

  return trending.sort((a, b) => b.rankImprovement - a.rankImprovement);
};

Output shows which posts are accelerating, helping you spot trends early.

Real-World Application: Building a Competitive Moat

Imagine you work on a database company. Your HN tracker catches a post: "We Switched from PostgreSQL to TimescaleDB for Time-Series Data" at rank #40 with 15 comments.

You're notified immediately. You read the discussion. The author mentions pain points you've solved in your product. You comment thoughtfully, sharing relevant experience. Your comment gets upvoted. You gain visibility among your target market.

By 2 PM, the post is #5 with 200 comments. You're one of the visible voices in the conversation. That's brand building.

Meanwhile, competitors who monitor HN manually missed it entirely. They're not in the conversation. You captured the moment.

Scale this across dozens of relevant posts per week. Over months, your team becomes the go-to voice in critical discussions. That's how you build a technical moat.

Getting Started Today

Head to Apify Hacker News Scraper and run one manual scrape. See what posts you find. Then set up a 2-hour schedule.

Connect it to a Slack webhook. Within a day, you'll be seeing HN trends as they emerge. Your competitive advantage depends on moving faster than the market. Automation gives you that speed.