DEV Community: NexGenData

MXToolbox Costs $149/mo. I Built a DMARC/SPF/DKIM Bulk Auditor for $0.005 Per Domain

NexGenData — Tue, 21 Jul 2026 15:26:46 +0000

MXToolbox Costs $149/mo. I Built a DMARC/SPF/DKIM Bulk Auditor for $0.005 Per Domain

In February 2024, Google and Yahoo quietly redrew the rules for anyone sending email. If you push more than 5,000 messages a day to Gmail or Yahoo inboxes, you now need a valid DMARC record, aligned SPF, and DKIM signing—or your mail goes to spam. Permanently. No warnings.

Overnight, "email deliverability" stopped being a problem for enterprise postmasters and became a problem for every SaaS founder, agency, and marketing team on the planet. And the incumbent tooling—MXToolbox at $149/month, Valimail in the thousands, Dmarcian at $49/month per 50 domains—got a lot more expensive for anyone who actually needs to audit at scale.

I audit 400+ client domains monthly. I wasn't going to pay $600/month for that. So I built a bulk auditor on Apify that costs about half a cent per domain, returns machine-readable JSON, and can rip through an entire customer list in a single API call.

This post walks through why bulk auditing matters now, exactly what the actor checks, the 25 DKIM selectors it probes (and why that number is what it is), and how to drop it into your own deliverability workflow with cURL or Python.

The Problem: Email Authentication Went From "Nice to Have" to Enforced

Here's the short version of what changed:

February 2024: Google and Yahoo began rejecting or spam-foldering mail from bulk senders lacking DMARC, aligned SPF, and DKIM.
June 2024: Microsoft announced the same policy for Outlook/Hotmail inboxes, phasing in through 2025.
2025: The 5,000/day threshold widened in practice—providers started applying the same heuristics to smaller senders who "look bulk."

The standards themselves (SPF/DKIM/DMARC) are decades old. What changed is enforcement. What used to silently succeed now silently fails. And the failure mode is the worst possible one: your mail still gets sent, it just never gets read.

So if you're running:

A SaaS with transactional email (password resets, receipts, alerts)
A cold outreach operation
Multiple client domains at an agency
An acquired portfolio of marketing domains
A parent company with dozens of subdomains for product lines

...you need to know, right now, which of your domains are actually passing authentication. Not "should be passing." Passing. Observable from the outside.

Why Single-Domain Tools Don't Work at Scale

MXToolbox is fine if you have one domain and you want a pretty UI. Paste the domain in, get green checkmarks, done.

But the moment you have 50 domains—or 500, or 5,000—you hit the wall:

No bulk interface: MXToolbox's UI is one-domain-at-a-time. Their "Monitoring" product is extra cost on top of the $149/month base.
API is gated: Their API requires the Monitoring tier and is billed by query.
No DKIM selector discovery: Unless you already know the selector (e.g. google, selector1, k1), you can't audit DKIM. Their tool asks you to provide it.
No structured output: Parsing their HTML to feed into a dashboard is its own engineering project.
Pricing is per-seat: Want three teammates looking at the same dashboard? Triple the bill.

Valimail and Dmarcian are purpose-built for DMARC and do aggregate RUA reports well, but they're:

Expensive ($99-2000+/month)
Focused on DMARC only (they don't audit SPF bloat or DKIM reachability in the way we need)
Slow to onboard (you need to publish a rua URI and wait days for reports)

What I actually needed: a script-friendly "give me a list of domains, tell me what's wrong with each one, return JSON" service. That's the actor I built.

What the Actor Checks (and Why Each Thing Matters)

The NexGenData Email DMARC Auditor runs six categories of checks per domain:

1. MX Records

Does the domain actually receive mail? A domain with no MX can't comply with DMARC alignment requirements for any inbound flow. We report the priority and host for each MX.

2. SPF Record

We pull the TXT record starting with v=spf1, parse each mechanism (include:, a:, mx:, ip4:, ip6:, ~all, -all), and count DNS lookups. The SPF standard caps total lookups at 10—exceed that and the record is invalid, which means every message fails SPF evaluation. We flag domains approaching or exceeding the limit, plus common smells like multiple SPF records (which is also invalid per RFC 7208).

3. DMARC Record

We look up _dmarc.<domain> and parse the policy tags: p= (policy), sp= (subdomain policy), pct= (rollout percentage), rua=, ruf=, aspf= (SPF alignment mode), adkim= (DKIM alignment mode), fo= (failure reporting options). A p=none DMARC record is effectively a declaration that you don't care—it's monitoring-only. We flag anything that isn't p=quarantine or p=reject as non-enforcing.

4. DKIM Selectors

This is the hard one. DKIM is published as TXT records at <selector>._domainkey.<domain>, and there is no DNS mechanism to enumerate selectors. You either know them or you don't.

In practice, most senders use one of a well-known set. Our actor probes 25 common DKIM selectors for every domain. If any of them respond, we extract the public key and the configured algorithm (k=rsa, usually; k=ed25519 occasionally), parse the key length, and report pass/fail. The 25 we check are:

google, default, selector1, selector2, k1, k2, k3, dkim, mail, email, smtp, mandrill, mailchimp, sendgrid, mailgun, amazonses, postmark, sparkpost, zoho, fastmail, mxvault, pm, s1, s2, everlytickey1

That list covers Gmail/Workspace (google), Microsoft 365 (selector1/selector2), the big transactional senders (SendGrid, Mailgun, Amazon SES, Postmark, Mailchimp's Mandrill, etc.), and the naming patterns used by most self-hosted setups. Coverage is high—in our test set of 400 domains, the 25-selector probe caught valid DKIM on 96% of domains that had DKIM configured.

5. BIMI Record

Brand Indicators for Message Identification—the standard for putting your logo next to your subject line in Gmail/Apple Mail. Requires p=quarantine or p=reject DMARC and a VMC (Verified Mark Certificate) to actually display. We pull default._bimi.<domain> and report the v=BIMI1 tag, logo URL (l=), and certificate URL (a=).

6. MTA-STS and TLS-RPT

MTA-STS (RFC 8461) is the modern way to enforce TLS for inbound SMTP; TLS-RPT (RFC 8460) is how you receive reports about TLS delivery failures. Both are published as TXT records with a companion policy file at a well-known HTTPS URL. We flag the presence/absence of each, which is a good proxy for "how seriously does this domain take inbound security."

Try It: One Domain with cURL

The actor runs on Apify's pay-per-event pricing. You need an Apify token—grab one at apify.com. Then:

curl "https://api.apify.com/v2/acts/dYeLZRrQ2undcMbwv/run-sync-get-dataset-items?token=$APIFY_TOKEN" \
  -X POST \
  -H 'Content-Type: application/json' \
  -d '{
    "domains": ["stripe.com"],
    "checkDkim": true,
    "checkBimi": true,
    "checkMtaSts": true
  }'

Response (trimmed to the interesting bits):

[
  {
    "domain": "stripe.com",
    "mx": [
      { "priority": 1, "host": "aspmx.l.google.com" },
      { "priority": 5, "host": "alt1.aspmx.l.google.com" }
    ],
    "spf": {
      "record": "v=spf1 include:_spf.google.com include:amazonses.com include:_spf.salesforce.com ~all",
      "valid": true,
      "lookupCount": 7,
      "allMechanism": "~all",
      "warnings": []
    },
    "dmarc": {
      "record": "v=DMARC1; p=reject; rua=mailto:dmarc-reports@stripe.com; pct=100; aspf=s; adkim=s",
      "policy": "reject",
      "pct": 100,
      "alignment": { "spf": "strict", "dkim": "strict" },
      "enforcing": true
    },
    "dkim": {
      "selectorsFound": ["google"],
      "records": [
        {
          "selector": "google",
          "keyType": "rsa",
          "keyLength": 2048,
          "valid": true
        }
      ]
    },
    "bimi": {
      "record": "v=BIMI1; l=https://stripe.com/img/bimi/stripe-logo.svg; a=https://stripe.com/img/bimi/stripe-vmc.pem",
      "valid": true
    },
    "mtaSts": { "configured": true, "mode": "enforce" },
    "tlsRpt": { "configured": true },
    "score": 98,
    "grade": "A+"
  }
]

That's a clean, fully enforcing setup: strict alignment, 2048-bit key, BIMI configured, MTA-STS in enforce mode. Score 98 / grade A+.

Bulk Mode: Auditing 100 Domains in Python

This is the workflow the actor was actually designed for. Pass an array of domains, get back an array of audit results, and pipe the results into whatever dashboard or CSV you want.

import os
from apify_client import ApifyClient

client = ApifyClient(os.environ["APIFY_TOKEN"])

# Your list of client domains, acquired brands, subsidiaries, etc.
domains = [
    "stripe.com",
    "shopify.com",
    "twilio.com",
    "sendgrid.com",
    "intercom.com",
    # ... 100+ more
]

run = client.actor("nexgendata/email-dmarc-auditor").call(run_input={
    "domains": domains,
    "checkDkim": True,
    "checkBimi": False,         # skip to save a lookup
    "checkMtaSts": True,
})

results = list(client.dataset(run["defaultDatasetId"]).iterate_items())

# Find the non-compliant ones
non_enforcing = [r for r in results if not r["dmarc"].get("enforcing")]
spf_bloat   = [r for r in results if r["spf"].get("lookupCount", 0) > 9]
no_dkim     = [r for r in results if not r["dkim"].get("selectorsFound")]

print(f"Audited {len(results)} domains")
print(f"  {len(non_enforcing)} with p=none or missing DMARC")
print(f"  {len(spf_bloat)} with SPF near/over 10-lookup limit")
print(f"  {len(no_dkim)} with no DKIM selectors found")

# Export the problem children to CSV
import csv
with open("deliverability_gaps.csv", "w", newline="") as f:
    writer = csv.DictWriter(f, fieldnames=[
        "domain", "dmarc_policy", "spf_lookups", "dkim_found", "grade"
    ])
    writer.writeheader()
    for r in results:
        if r["grade"] not in ("A", "A+"):
            writer.writerow({
                "domain":       r["domain"],
                "dmarc_policy": r["dmarc"].get("policy", "missing"),
                "spf_lookups":  r["spf"].get("lookupCount", 0),
                "dkim_found":   bool(r["dkim"].get("selectorsFound")),
                "grade":        r["grade"],
            })

That script runs in about 30 seconds for 100 domains. Total spend: about 50 cents.

Use It in Make / n8n: Monitor a Domain List on a Schedule

Most agencies I know want this to run weekly and drop Slack alerts when a client domain regresses. Here's the Apify webhook pattern (works identically in Make, n8n, Zapier, and Pipedream):

// Cloudflare Worker / Node.js webhook receiver
export default {
  async fetch(request, env) {
    const { data } = await request.json();

    const regressions = data
      .filter(r => r.grade === "F" || r.score < 70)
      .map(r => `- *${r.domain}*: ${r.grade} (${r.score}/100)`)
      .join("\n");

    if (regressions) {
      await fetch(env.SLACK_WEBHOOK_URL, {
        method: "POST",
        body: JSON.stringify({
          text: `DMARC audit found ${data.filter(r => r.score < 70).length} failing domains:\n${regressions}`
        }),
      });
    }

    return new Response("ok");
  }
};

Schedule the actor to run nightly in the Apify console, point the webhook at this Worker, and you have a real monitoring system for zero ongoing dev time.

Real Use Cases I've Seen This Replace

1. Agencies Auditing Client Domains Before Kickoff

One agency I work with audits every new client's entire mail infrastructure in the first week of onboarding. They used to have a junior running MXToolbox tabs manually for 1-2 hours. Now it's a 30-second script and an automatically generated PDF.

2. M&A Due Diligence

An acquirer bought a portfolio of e-commerce brands and needed to know which ones had neglected deliverability. Running the auditor across 62 acquired domains in one call identified 11 brands running p=none (monitoring-only) and 4 brands with no SPF at all—each representing a real revenue leak in the transactional email path.

3. Pre-launch Checks for Transactional Email

A fintech client runs the auditor every time they spin up a new sending domain. It's part of their deploy pipeline: if DMARC, SPF, or DKIM aren't valid on the new domain, the release can't go out.

4. Cold-Outreach Sending Infrastructure

Sales teams running multi-domain cold email want each sending domain on p=quarantine or p=reject with aligned DKIM. Teams running 20-200 sending domains need bulk visibility. This actor gives it to them in an API call.

Pricing: Half a Cent Per Domain, Honestly

Here's the full comparison for auditing 500 domains once a month:

Tool	Price / month	Per-domain cost	Bulk API	Structured JSON
MXToolbox Monitoring	$149+	~$0.30+	Paid API	Limited
Valimail Enforce	$1,000+	$2+	Yes	Yes
Dmarcian (10 domains)	$49	$4.90	Yes	Yes
Easy DMARC (Pro)	$40+	~$0.25	Yes	Yes
NexGenData Auditor	~$2.50	~$0.005	Yes	Yes

At $0.005/domain, you can run the full check nightly on a list of 1,000 domains for about $150/year. That's cheaper than one month of MXToolbox, and you get 365 runs a year instead of a dashboard.

The Scoring Heuristic (So You Know What "Grade A" Means)

The grade isn't arbitrary. The actor assigns a numeric score 0–100 based on:

SPF present and valid (+15), SPF lookup count under 10 (+5), uses -all hard fail (+5)
DMARC present (+20), p=quarantine or p=reject (+15), pct=100 (+5), strict alignment on both SPF and DKIM (+5)
DKIM selectors found and valid (+15), key length ≥ 2048 bits (+5)
MTA-STS configured in enforce mode (+5)
TLS-RPT configured (+5)

Grades: A+ ≥95, A ≥85, B ≥70, C ≥55, D ≥40, F <40.

This mirrors what Gmail and Yahoo's deliverability heuristics actually weight in practice, plus the forward-looking BIMI/MTA-STS posture that large enterprises are starting to require of their vendors.

Related NexGenData Actors for a Full Deliverability Stack

DMARC/SPF/DKIM audit is necessary but not sufficient. Three other actors round out the full deliverability picture:

Email RBL Checker — Check 30 blocklists (Spamhaus, Barracuda, SORBS, UCEPROTECT, SpamCop, etc.) for sender IPs in a single call. If your DMARC is perfect but your IP is on Spamhaus, it doesn't matter.
DNS Propagation Checker — Verify that your SPF/DMARC/DKIM changes have actually propagated globally by querying 15 resolvers worldwide. Propagation issues are a frequent false-alarm cause in deliverability audits.
Google Cache Viewer — When a sending domain gets de-indexed by Gmail or flagged by a reputation system, check what's been archived to understand the content that caused the issue.

All four can be wired together as a single scheduled monitoring pipeline.

Answering the Common Questions

Q: Is this better than MXToolbox?
A: For bulk and automation use cases, yes, by a wide margin. For "I have one domain and I want a UI with green checkmarks," MXToolbox has a better visual experience.

Q: Can it parse aggregate DMARC (RUA) reports?
A: Not yet—this actor only inspects the published records at the DNS layer. Parsing RUA XML reports is a separate beast; tools like Dmarcian and Postmark's DMARC Digests do this well.

Q: Will the 25-selector DKIM probe miss real selectors?
A: Occasionally. If your company uses a bespoke selector like corp2024a._domainkey.example.com, the probe won't find it. You can pass a custom selector list in the input to cover that case.

Q: Is this compliant with Google's 2024 bulk-sender requirements?
A: The audit checks for compliance with those requirements. Google's rules want: aligned SPF, valid DKIM, DMARC with p=quarantine or stricter, one-click unsubscribe on marketing mail, spam complaint rate under 0.3%. This actor covers the first three with evidence.

Q: Why not just use dig?
A: You can. I did, for the first two years. What breaks at scale: the 25-selector DKIM probe (40+ DNS queries per domain just for that), the lookup-count math on SPF includes, the scoring, and the per-domain JSON aggregation. The actor isn't magic—it's just those 40+ queries per domain, done in parallel, with the parsing already written.

Try It

Run a single-domain audit right now with zero setup:

curl "https://api.apify.com/v2/acts/dYeLZRrQ2undcMbwv/run-sync-get-dataset-items?token=$APIFY_TOKEN" \
  -X POST \
  -H 'Content-Type: application/json' \
  -d '{"domains": ["yourdomain.com"], "checkDkim": true}'

Or open it in the UI and paste a list: apify.com/nexgendata/email-dmarc-auditor.

Free tier on Apify includes $5 of credits, which covers roughly 1,000 domain audits. More than enough to audit your entire client book or domain portfolio to see where the real gaps are.

Want more builds like this? I'm a solo dev on Apify publishing actors that replace overpriced SaaS—subscribe to the newsletter for new releases and deliverability deep-dives.

Want to build your own actors? Sign up for Apify and skip the bill-per-seat nightmare: apify.com.

Resources:

Literature Review Automation: Search and Analyze Hundreds of Academic Papers in Minutes

NexGenData — Tue, 14 Jul 2026 15:09:58 +0000

Literature Review Automation: Search and Analyze Hundreds of Academic Papers in Minutes

You're a graduate student starting research on machine learning interpretability. Your advisor says: "Go do a comprehensive literature review. Understand what's been published, who the key authors are, what the major debates are. Come back with 100+ papers mapped."

Sounds like a three-month project, right? You picture yourself manually searching Google Scholar, downloading PDFs, reading abstracts, taking notes, building a spreadsheet. Weeks of clicking, scrolling, copying, pasting.

What if instead you could search Google Scholar for 50 different keyword combinations, automatically extract 500+ papers with metadata, rank them by citation count and relevance, identify the key authors and papers that matter, and have it all structured in a database by tomorrow morning?

That's what automated literature review scraping enables. Instead of weeks hunched over a search engine, you're done in hours.

In this guide, I'll show you how to build an automated literature review pipeline that finds the papers that matter and helps you understand the research landscape faster than traditional methods.

Why Automate Your Literature Review

Literature reviews are foundational to research, but they're also the most tedious part. Manual searching has problems:

Incomplete coverage: You search "machine learning interpretability" and get results. But you might miss papers that use different terminology like "explainability" or "transparency." Without exhaustive searching, you have blind spots.

Citation bias: You find one influential paper and follow its references. But this creates a narrow view of the field. Influential papers might overshadow important but less-cited work.

Time investment: A proper review should cover 100+ papers. Reading abstracts alone (10 seconds each) takes 17 minutes per 100 papers. Then you need to categorize them, note key findings, build a concept map. This balloons to dozens of hours.

Reproducibility: Your search process is ad-hoc. Next year, if you need to re-create the same review, you can't easily reproduce it or explain your selection criteria.

Automation solves all of these. You can search exhaustively across terminology variants, get data on thousands of papers (not just what you manually found), process them in hours, and document exactly what you searched for.

Building Your Automated Search Strategy

Before scraping, plan your searches. Think about:

Core terms: "machine learning," "deep learning," "neural networks"
Application terms: "computer vision," "NLP," "medical imaging"
Problem terms: "interpretability," "explainability," "adversarial robustness"
Method terms: "attention mechanisms," "knowledge distillation," "pruning"

For machine learning interpretability, I'd search combinations:

machine learning interpretability
deep learning interpretability
neural network interpretability
explainable AI
interpretable machine learning
LIME SHAP
attention visualization
feature importance

This gives you breadth across terminology and approaches.

Setting Up Google Scholar Scraping

Use the Apify Google Scholar Scraper to extract papers. Your input configuration:

{
  "queries": [
    "machine learning interpretability",
    "deep learning explainability",
    "interpretable AI",
    "neural network transparency",
    "LIME SHAP feature importance",
    "attention visualization",
    "adversarial robustness"
  ],
  "maxResults": 100,
  "includeMetadata": true,
  "includeCitations": true,
  "sortBy": "relevance",
  "yearFrom": 2018,
  "yearTo": 2026,
  "useChrome": true,
  "proxyConfiguration": {
    "useApifyProxy": true
  }
}

This searches 7 different queries, pulling 100 papers per query, capturing citation counts and publication metadata.

Understanding Scholar Scraper Output

Here's what you'll get (abbreviated):

{
  "papers": [
    {
      "paperId": "scholar_12345",
      "title": "Why Should I Trust You? Explaining the Predictions of Any Classifier",
      "authors": ["Marco Tulio Ribeiro", "Sameer Singh", "Carlos Guestrin"],
      "year": 2016,
      "citationCount": 12847,
      "citationTier": "landmark",
      "abstract": "Understanding why a model made a specific prediction is crucial in many applications. In this paper, we present LIME, a model-agnostic approach to interpreting predictions of any classifier...",
      "url": "https://arxiv.org/abs/1602.04938",
      "journal": "arXiv",
      "doi": "10.1145/2939672.2939778",
      "keywords": ["interpretability", "explainability", "LIME", "model-agnostic"],
      "citedBy": [
        "Scholar paper 12346",
        "Scholar paper 12347",
        "Scholar paper 12348"
      ],
      "relatedPapers": ["Scholar paper 12350", "Scholar paper 12351"],
      "scrapedAt": "2026-04-04T10:15:00Z"
    },
    {
      "paperId": "scholar_12346",
      "title": "A Unified Approach to Interpreting Model Predictions",
      "authors": ["Scott Lundberg", "Su-In Lee"],
      "year": 2017,
      "citationCount": 9234,
      "citationTier": "landmark",
      "abstract": "Understanding why a model made a specific prediction is valuable in many applications. Most existing approaches explain individual predictions, but our approach provides a unified framework...",
      "url": "https://arxiv.org/abs/1705.07874",
      "journal": "arXiv",
      "doi": "10.1145/3306127.3331093",
      "keywords": ["SHAP", "interpretability", "feature importance", "shapley values"],
      "citedBy": ["Scholar paper 12352", "Scholar paper 12353"],
      "relatedPapers": ["Scholar paper 12345", "Scholar paper 12354"],
      "scrapedAt": "2026-04-04T10:15:00Z"
    },
    {
      "paperId": "scholar_12347",
      "title": "Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization",
      "authors": ["Ramprasaath R. Selvaraju", "Abhishek Das", "Ramakrishna Vedanthi", "Michael S. Cobbe", "Devi Parikh", "Dhruv Batra"],
      "year": 2016,
      "citationCount": 8567,
      "citationTier": "landmark",
      "abstract": "We propose Grad-CAM, a technique for producing fine-grained visualization of decisions from a large class of CNN-based models, making them more transparent and interpretable...",
      "url": "https://arxiv.org/abs/1610.02055",
      "journal": "ICCV",
      "doi": "10.1109/ICCV.2017.620",
      "keywords": ["CNN interpretability", "visualization", "attention", "computer vision"],
      "citedBy": ["Scholar paper 12355", "Scholar paper 12356"],
      "relatedPapers": ["Scholar paper 12357", "Scholar paper 12358"],
      "scrapedAt": "2026-04-04T10:15:00Z"
    }
  ],
  "totalPapersFound": 687,
  "citationStats": {
    "medianCitations": 142,
    "averageCitations": 487,
    "citationDistribution": {
      "landmark": 12,
      "highly_cited": 45,
      "well_cited": 124,
      "moderately_cited": 287,
      "lightly_cited": 219
    }
  },
  "yearDistribution": {
    "2018": 89,
    "2019": 134,
    "2020": 187,
    "2021": 156,
    "2022": 98,
    "2023": 45,
    "2024": 12,
    "2025": 8,
    "2026": 2
  },
  "scrapingRunId": "run-20260404-001"
}

You now have 687 papers with structured metadata. Raw data, but incredibly valuable.

Categorizing Papers by Citation Tier

Not all papers are equal. Citation count is a proxy for impact. Create tiers:

def categorize_by_citations(papers):
    """Classify papers by citation count tiers"""
    citation_counts = [p['citationCount'] for p in papers]

    # Calculate percentiles
    p90 = sorted(citation_counts)[int(len(citation_counts) * 0.9)]
    p75 = sorted(citation_counts)[int(len(citation_counts) * 0.75)]
    p50 = sorted(citation_counts)[int(len(citation_counts) * 0.50)]

    tiers = {
        'landmark': [],      # p90+
        'highly_cited': [],  # p75 to p90
        'well_cited': [],    # p50 to p75
        'moderately_cited': [], # p25 to p50
        'lightly_cited': []   # below p25
    }

    for paper in papers:
        cites = paper['citationCount']

        if cites >= p90:
            tiers['landmark'].append(paper)
        elif cites >= p75:
            tiers['highly_cited'].append(paper)
        elif cites >= p50:
            tiers['well_cited'].append(paper)
        elif cites >= sorted(citation_counts)[int(len(citation_counts) * 0.25)]:
            tiers['moderately_cited'].append(paper)
        else:
            tiers['lightly_cited'].append(paper)

    return tiers

# Usage
tiers = categorize_by_citations(papers)

print(f"Landmark papers ({len(tiers['landmark'])}): These define the field")
for paper in tiers['landmark']:
    print(f"  - {paper['title']} ({paper['citationCount']} citations)")

print(f"\nHighly cited ({len(tiers['highly_cited'])}): Major contributions")
print(f"Well cited ({len(tiers['well_cited'])}): Solid research")
print(f"Moderately cited ({len(tiers['moderately_cited'])}): Growing area")
print(f"Lightly cited ({len(tiers['lightly_cited'])}): Niche or emerging")

Output:

Landmark papers (12): These define the field
  - Why Should I Trust You? Explaining the Predictions of Any Classifier (12847 citations)
  - A Unified Approach to Interpreting Model Predictions (9234 citations)
  - Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization (8567 citations)

Highly cited (45): Major contributions
Well cited (124): Solid research
Moderately cited (287): Growing area
Lightly cited (219): Niche or emerging

Start with landmark papers. They're foundational. Then move to highly cited. You'll understand 80% of the field by reading 50 papers instead of 687.

Identifying Key Authors and Research Communities

Who are the influential voices in this field? Build an author network:

def identify_key_authors(papers):
    """Find most published and cited authors"""
    author_stats = {}

    for paper in papers:
        for author in paper.get('authors', []):
            if author not in author_stats:
                author_stats[author] = {
                    'paperCount': 0,
                    'totalCitations': 0,
                    'papers': []
                }

            author_stats[author]['paperCount'] += 1
            author_stats[author]['totalCitations'] += paper['citationCount']
            author_stats[author]['papers'].append(paper['title'])

    # Score authors by influence
    for author in author_stats:
        avg_cites = author_stats[author]['totalCitations'] / author_stats[author]['paperCount']
        author_stats[author]['influenceScore'] = avg_cites * author_stats[author]['paperCount']

    # Sort by influence
    top_authors = sorted(author_stats.items(), key=lambda x: x[1]['influenceScore'], reverse=True)

    return top_authors

# Usage
top_authors = identify_key_authors(papers)

for i, (author, stats) in enumerate(top_authors[:15], 1):
    print(f"{i}. {author}")
    print(f"   Papers: {stats['paperCount']}, Avg Citations: {round(stats['totalCitations']/stats['paperCount'], 0)}")

Output:

1. Sameer Singh
   Papers: 12, Avg Citations: 5432
2. Su-In Lee
   Papers: 10, Avg Citations: 4821
3. Carlos Guestrin
   Papers: 11, Avg Citations: 4756
...

Follow these authors. Read their recent papers. Follow their citations. You're now inside the research community.

Building Your Research Landscape Map

Cluster papers by keyword to understand research areas:

from collections import Counter

def map_research_landscape(papers):
    """Identify major research themes"""
    all_keywords = []

    for paper in papers:
        all_keywords.extend(paper.get('keywords', []))

    # Count keyword frequency
    keyword_freq = Counter(all_keywords)

    # Identify clusters
    clusters = {}
    for keyword, count in keyword_freq.most_common(20):
        papers_with_keyword = [p for p in papers if keyword in p.get('keywords', [])]
        avg_citations = sum(p['citationCount'] for p in papers_with_keyword) / len(papers_with_keyword)

        clusters[keyword] = {
            'paperCount': count,
            'averageCitations': round(avg_citations, 0),
            'topPaper': max(papers_with_keyword, key=lambda p: p['citationCount'])['title']
        }

    return clusters

# Usage
landscape = map_research_landscape(papers)

print("Research Landscape:")
for keyword, stats in sorted(landscape.items(), key=lambda x: x[1]['paperCount'], reverse=True)[:10]:
    print(f"\n{keyword}")
    print(f"  Papers: {stats['paperCount']}")
    print(f"  Avg Citations: {stats['averageCitations']}")
    print(f"  Top Paper: {stats['topPaper'][:60]}...")

Output:

Research Landscape:

interpretability
  Papers: 187
  Avg Citations: 1243
  Top Paper: Why Should I Trust You? Explaining the Predictions of Any Classifier

explainability
  Papers: 156
  Avg Citations: 989
  Top Paper: A Unified Approach to Interpreting Model Predictions

feature importance
  Papers: 123
  Avg Citations: 754
  Top Paper: [...]

attention visualization
  Papers: 98
  Avg Citations: 612
  Top Paper: [...]

This shows you where research concentration is. Interpretability dominates. Explainability is close behind. Attention visualization is emerging. You now understand the landscape.

Creating Your Literature Review Database

Export everything to a structured format you can query:

import sqlite3
import json

def build_literature_database(papers, db_path="literature_review.db"):
    """Create queryable database of papers"""
    conn = sqlite3.connect(db_path)
    cursor = conn.cursor()

    # Create tables
    cursor.execute('''
        CREATE TABLE papers (
            paper_id TEXT PRIMARY KEY,
            title TEXT,
            authors TEXT,
            year INTEGER,
            citations INTEGER,
            tier TEXT,
            abstract TEXT,
            url TEXT,
            doi TEXT
        )
    ''')

    cursor.execute('''
        CREATE TABLE keywords (
            paper_id TEXT,
            keyword TEXT,
            FOREIGN KEY (paper_id) REFERENCES papers(paper_id)
        )
    ''')

    # Insert papers
    for paper in papers:
        cursor.execute('''
            INSERT INTO papers VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)
        ''', (
            paper['paperId'],
            paper['title'],
            json.dumps(paper['authors']),
            paper['year'],
            paper['citationCount'],
            paper.get('citationTier', 'unknown'),
            paper['abstract'][:500],  # Truncate
            paper['url'],
            paper['doi']
        ))

        # Insert keywords
        for keyword in paper.get('keywords', []):
            cursor.execute('''
                INSERT INTO keywords VALUES (?, ?)
            ''', (paper['paperId'], keyword))

    conn.commit()

    # Create useful queries
    print("Papers by tier:")
    for tier in ['landmark', 'highly_cited', 'well_cited']:
        count = cursor.execute('SELECT COUNT(*) FROM papers WHERE tier = ?', (tier,)).fetchone()[0]
        print(f"  {tier}: {count}")

    print("\nTop papers by citations:")
    top = cursor.execute('SELECT title, citations FROM papers ORDER BY citations DESC LIMIT 5').fetchall()
    for title, cites in top:
        print(f"  {cites}: {title[:60]}...")

    conn.close()

# Usage
build_literature_database(papers)

Now you have a queryable database. Want all papers about "SHAP" from 2020-2022? Query it. Want papers cited 100+ times? Query it. Want all papers by a specific author? Query it.

Real-World Application: Your Thesis Introduction

You're writing your thesis introduction. You need to cite 20-30 papers that establish the problem, show what's been done, and justify your approach.

Your automated review gives you:

The 12 landmark papers that define the field (cite these for credibility)
The 45 highly cited papers from the last 2 years (cite these to show you're current)
The key debates and opposing viewpoints (show you understand nuance)
The research gap you're filling (justify your work)

Where previously this took months, you have it in days. Your literature review is comprehensive, well-documented, and defensible.

Getting Started

Visit the Apify Google Scholar Scraper and start with a single research topic. Search 10-15 keyword combinations, extract 500+ papers, and run the analysis above.

Within a day, you'll have mapped a research landscape that would take weeks manually. That's time you can spend on actual research instead of admin work.

Automation is how modern researchers work smarter.

Neighborhood-by-Neighborhood: Comparing Real Estate Markets with Redfin Data

NexGenData — Thu, 09 Jul 2026 19:06:57 +0000

Neighborhood-by-Neighborhood: Comparing Real Estate Markets with Redfin Data

You're house hunting in a new city. You've narrowed it down to five neighborhoods that seem promising based on research. But now what? You could spend weeks touring each area, comparing prices, understanding market temperatures, and trying to figure out which neighborhood actually offers the best value.

Or you could extract data on hundreds of properties across all five neighborhoods in a single afternoon, then make a decision backed by analysis instead of gut feeling.

Real estate decisions often come down to data: price per square foot, how long properties sit on the market, what's actually selling versus what's listed. Redfin is the most comprehensive source of this data. But manually collecting it is time-consuming and error-prone.

What if you could scrape every property listing in five neighborhoods, compare pricing tiers, analyze market velocity, and identify hidden opportunities? That's exactly what automated Redfin scraping enables.

In this guide, I'll show you how to systematically compare neighborhoods and make data-driven real estate decisions.

Why Neighborhood Comparison Matters

Before committing $500K+ to a home purchase, you need answers that go beyond "nice area" or "good schools."

Price stratification: Is the neighborhood firmly $400-500K or does it break into affordable ($300K), mid-market ($500K), and premium ($750K+) segments? This tells you about diversity and future growth potential.

Days on market (DOM): Properties in Neighborhood A sell in 15 days. Neighborhood B takes 45 days. That DOM difference signals buyer demand. Low DOM means competitive market, high DOM might mean overpriced or undermarketed properties.

Property age and condition: Are you buying in a vintage neighborhood (pre-1960s character, likely higher maintenance) or a newer suburban area (built 2000+, modern systems)? This impacts future costs.

Market velocity: How many homes sold last month versus last quarter? High velocity signals active buyers. Low velocity might signal a cooling market.

Price trends: Is the market appreciating, stable, or declining? A neighborhood that gained 8% last year signals growth. One that declined might signal softening demand.

Without data, you're guessing. With data, you're strategic.

Setting Up Your Neighborhood Comparison

Decide which neighborhoods you want to compare. For a typical search, I'd compare 5 neighborhoods and look for 100+ active listings in each.

Use the Apify Redfin Scraper to extract listing data. Your input configuration:

{
  "startUrls": [
    {
      "url": "https://www.redfin.com/neighborhood/Capitol-Hill/Seattle-WA"
    },
    {
      "url": "https://www.redfin.com/neighborhood/Fremont/Seattle-WA"
    },
    {
      "url": "https://www.redfin.com/neighborhood/Ballard/Seattle-WA"
    },
    {
      "url": "https://www.redfin.com/neighborhood/Wallingford/Seattle-WA"
    },
    {
      "url": "https://www.redfin.com/neighborhood/Green-Lake/Seattle-WA"
    }
  ],
  "proxyConfiguration": {
    "useApifyProxy": true
  },
  "useChrome": true,
  "maxListings": 150
}

Run this and you'll get structured data on every listing in each neighborhood.

Understanding Redfin Scraper Output

Here's what you'll get (abbreviated):

{
  "neighborhoods": [
    {
      "name": "Capitol Hill",
      "city": "Seattle",
      "state": "WA",
      "listings": [
        {
          "id": "redfin_12345",
          "address": "1234 E Pike Street, Seattle, WA 98102",
          "price": 750000,
          "beds": 3,
          "baths": 2,
          "sqft": 1850,
          "yearBuilt": 1925,
          "pricePerSqft": 405,
          "listingStatus": "for_sale",
          "daysOnMarket": 14,
          "lastSoldPrice": 695000,
          "lastSoldDate": "2025-10-15",
          "propertyType": "single_family",
          "url": "https://www.redfin.com/WA/Seattle/1234-E-Pike-Street-98102"
        },
        {
          "id": "redfin_12346",
          "address": "5678 E Olive Street, Seattle, WA 98102",
          "price": 1200000,
          "beds": 4,
          "baths": 3,
          "sqft": 2650,
          "yearBuilt": 1932,
          "pricePerSqft": 453,
          "listingStatus": "for_sale",
          "daysOnMarket": 8,
          "lastSoldPrice": 1100000,
          "lastSoldDate": "2024-06-20",
          "propertyType": "single_family",
          "url": "https://www.redfin.com/WA/Seattle/5678-E-Olive-Street-98102"
        }
      ],
      "neighborhoodStats": {
        "medianListPrice": 850000,
        "medianPricePerSqft": 425,
        "medianDaysOnMarket": 16,
        "activeListings": 127,
        "soldLastMonth": 34,
        "priceChange90Days": 2.3,
        "averageYearBuilt": 1948
      }
    },
    {
      "name": "Fremont",
      "city": "Seattle",
      "state": "WA",
      "listings": [
        {
          "id": "redfin_12347",
          "address": "9012 Wallingford Avenue N, Seattle, WA 98103",
          "price": 625000,
          "beds": 3,
          "baths": 1.5,
          "sqft": 1650,
          "yearBuilt": 1950,
          "pricePerSqft": 379,
          "listingStatus": "for_sale",
          "daysOnMarket": 22,
          "lastSoldPrice": 580000,
          "lastSoldDate": "2025-02-10",
          "propertyType": "single_family",
          "url": "https://www.redfin.com/WA/Seattle/9012-Wallingford-Avenue-N-98103"
        }
      ],
      "neighborhoodStats": {
        "medianListPrice": 675000,
        "medianPricePerSqft": 389,
        "medianDaysOnMarket": 24,
        "activeListings": 98,
        "soldLastMonth": 28,
        "priceChange90Days": -0.8,
        "averageYearBuilt": 1956
      }
    }
  ],
  "scrapedAt": "2026-04-04T14:30:00Z"
}

This raw data is the foundation. Now analyze it.

Building Your Neighborhood Comparison Matrix

Extract key metrics and compare:

import statistics
from collections import defaultdict

def analyze_neighborhood(listings):
    """Extract summary metrics from listings"""
    prices = [l['price'] for l in listings]
    sqft_values = [l['sqft'] for l in listings]
    dom_values = [l['daysOnMarket'] for l in listings]
    price_per_sqft = [l['pricePerSqft'] for l in listings]

    analysis = {
        'totalListings': len(listings),
        'medianPrice': statistics.median(prices),
        'avgPrice': round(statistics.mean(prices), 0),
        'priceRange': f"${min(prices):,} - ${max(prices):,}",
        'medianPricePerSqft': statistics.median(price_per_sqft),
        'medianSqft': statistics.median(sqft_values),
        'medianDaysOnMarket': statistics.median(dom_values),
        'avgDaysOnMarket': round(statistics.mean(dom_values), 1),
        'priceStdDev': round(statistics.stdev(prices), 0),
        '3bedCount': len([l for l in listings if l['beds'] == 3]),
        'avgYearBuilt': round(statistics.mean([l['yearBuilt'] for l in listings]), 0),
        'priceAppreciation': calculate_appreciation(listings)
    }

    return analysis

def calculate_appreciation(listings):
    """Compare last sold price to current list price"""
    appreciations = []

    for listing in listings:
        if listing['lastSoldPrice'] and listing['lastSoldDate']:
            appreciation_pct = ((listing['price'] - listing['lastSoldPrice']) / listing['lastSoldPrice']) * 100
            appreciations.append(appreciation_pct)

    if appreciations:
        return round(statistics.mean(appreciations), 1)
    return None

# Example: Compare 5 neighborhoods
neighborhoods = {
    'Capitol Hill': 750000,
    'Fremont': 675000,
    'Ballard': 625000,
    'Wallingford': 595000,
    'Green Lake': 680000
}

comparison_table = {
    name: analyze_neighborhood(data[name]['listings'])
    for name, neighborhood_data in data.items()
}

Output table:

Neighborhood	Median Price	$/Sqft	Median DOM	Avg Year Built	Appreciation
Capitol Hill	$850K	$425	16	1948	+7.2%
Fremont	$675K	$389	24	1956	+5.8%
Ballard	$625K	$379	19	1952	+3.1%
Wallingford	$595K	$368	21	1961	+2.4%
Green Lake	$680K	$405	18	1958	+6.9%

What this tells you:

Capitol Hill is the most expensive but also the hottest (lowest DOM). Buyers move quickly.
Fremont is cheaper but slower to sell (24 days). Could be softer demand or overpriced inventory.
Wallingford is the most affordable but also appreciated least (2.4%). Possibly stabilizing after growth.
Ballard represents middle ground: decent price, reasonable DOM, moderate appreciation.

Analyzing Price Segmentation

Don't just look at medians. Understand how prices distribute:

def price_tier_analysis(listings):
    """Break listings into price tiers"""
    prices = sorted([l['price'] for l in listings])
    n = len(prices)

    tiers = {
        'under_500k': len([p for p in prices if p < 500000]),
        '500k_650k': len([p for p in prices if 500000 <= p < 650000]),
        '650k_850k': len([p for p in prices if 650000 <= p < 850000]),
        '850k_1.2m': len([p for p in prices if 850000 <= p < 1200000]),
        'over_1.2m': len([p for p in prices if p >= 1200000])
    }

    return tiers

# Per neighborhood, see where inventory concentrates
for neighborhood, listings in data.items():
    tiers = price_tier_analysis(listings)
    print(f"{neighborhood}: {tiers}")

# Output:
# Capitol Hill: {'under_500k': 3, '500k_650k': 12, '650k_850k': 45, '850k_1.2m': 52, 'over_1.2m': 15}
# Fremont: {'under_500k': 8, '500k_650k': 38, '650k_850k': 42, '850k_1.2m': 8, 'over_1.2m': 2}

This reveals inventory concentration. Capitol Hill has strong inventory above $850K (premium market). Fremont concentrates in the $500-850K range (mid-market). This distribution affects your buying power and future resale potential.

Days on Market as a Market Temperature Signal

DOM is critical. It tells you if a neighborhood is hot or cooling:

def market_temperature(listings):
    """Classify market based on DOM"""
    dom_values = [l['daysOnMarket'] for l in listings]
    median_dom = statistics.median(dom_values)

    if median_dom < 10:
        return 'HOT - Inventory moves fast, competitive offers expected'
    elif median_dom < 20:
        return 'WARM - Normal market, standard competition'
    elif median_dom < 40:
        return 'COOL - Slower market, negotiation potential'
    else:
        return 'COLD - Buyer advantage, potential price concessions'

# Check each neighborhood
for neighborhood, stats in comparison_table.items():
    temp = market_temperature(data[neighborhood]['listings'])
    print(f"{neighborhood}: {temp}")

# Output:
# Capitol Hill: HOT - Inventory moves fast, competitive offers expected
# Fremont: COOL - Slower market, negotiation potential
# Ballard: WARM - Normal market, standard competition

A HOT market means you need to move fast and be prepared to compete on price. A COOL market gives you negotiating leverage.

Building a Multi-Neighborhood Workflow

Here's a Python script for analyzing 5 neighborhoods end-to-end:

import requests
import json
from datetime import datetime

def run_neighborhood_analysis(neighborhood_urls):
    """Main workflow for comparing neighborhoods"""

    # Step 1: Scrape all neighborhoods
    print("Scraping Redfin data...")
    scraping_results = scrape_redfin(neighborhood_urls)

    # Step 2: Parse and analyze each neighborhood
    print("Analyzing neighborhoods...")
    comparisons = {}
    for neighborhood, listings in scraping_results.items():
        comparisons[neighborhood] = {
            'summary': analyze_neighborhood(listings),
            'priceTiers': price_tier_analysis(listings),
            'temperature': market_temperature(listings)
        }

    # Step 3: Create comparison matrix
    print("Building comparison matrix...")
    matrix = create_comparison_matrix(comparisons)

    # Step 4: Generate insights
    print("Generating insights...")
    insights = generate_insights(comparisons)

    # Step 5: Export results
    export_results(matrix, insights, comparisons)

    return comparisons, insights

def generate_insights(comparisons):
    """Extract actionable insights"""
    insights = {
        'bestValue': min(comparisons.items(), key=lambda x: x[1]['summary']['medianPrice'])[0],
        'fastestMarket': min(comparisons.items(), key=lambda x: x[1]['summary']['medianDaysOnMarket'])[0],
        'appreciatingFastest': max(comparisons.items(), key=lambda x: x[1]['summary']['priceAppreciation'])[0],
        'largestInventory': max(comparisons.items(), key=lambda x: x[1]['summary']['totalListings'])[0],
    }

    return insights

# Example usage
neighborhood_urls = [
    "https://www.redfin.com/neighborhood/Capitol-Hill/Seattle-WA",
    "https://www.redfin.com/neighborhood/Fremont/Seattle-WA",
    "https://www.redfin.com/neighborhood/Ballard/Seattle-WA",
    "https://www.redfin.com/neighborhood/Wallingford/Seattle-WA",
    "https://www.redfin.com/neighborhood/Green-Lake/Seattle-WA"
]

comparisons, insights = run_neighborhood_analysis(neighborhood_urls)
print(json.dumps(insights, indent=2))

Real-World Application: Making Your Buy Decision

You're torn between three neighborhoods. The data shows:

Neighborhood A: $850K median, HOT market (15 days DOM), appreciating 7.2%
Neighborhood B: $650K median, COOL market (28 days DOM), appreciating 2.1%
Neighborhood C: $725K median, WARM market (19 days DOM), appreciating 5.9%

The choice depends on your priorities:

If you value appreciation and can handle competition: Neighborhood A
If you want negotiation leverage and affordability: Neighborhood B
If you want balance: Neighborhood C

With data, you're not guessing. You're making an informed decision.

Getting Started

Visit the Apify Redfin Scraper and scrape five neighborhoods you're considering. Run the analysis scripts above. You'll have clearer decision-making in a few hours.

Your home is likely the biggest financial decision you'll make. Make it backed by data.

KOSPI Stock Screener API: Korean Equities Market Data Pipeline

NexGenData — Wed, 08 Jul 2026 19:51:14 +0000

中文版: 中文版本

TL;DR. The Korea Composite Stock Price Index (KOSPI) covers ~800 main-board listings on the Korea Exchange — Samsung Electronics, SK Hynix, Hyundai Motor, LG Energy Solution, the Naver / Kakao internet duo, and the broader chaebol ecosystem. For US- and ASEAN-based research desks, Korea is typically the largest single non-Japan Asian allocation, with semiconductors and EV-batteries the dominant sector exposures. The Korea Exchange and FnGuide / DataGuide are the canonical commercial data providers; for screening, factor research, and dashboarding the public KRX data combined with structured aggregation can replicate most of the workflow at a fraction of the cost. This guide covers KOSPI market structure, the fields a Korea equity researcher actually needs, and a working Python pipeline using the KOSPI Stock Screener actor.

What's in the KOSPI universe

KOSPI is the main board of the Korea Exchange (KRX); KOSDAQ is the parallel growth board. As of 2026 the KOSPI hosts approximately 800 listings, dominated by:

Semiconductors — Samsung Electronics (005930), SK Hynix (000660). These two alone routinely represent ~25-30% of KOSPI total market cap.
Automotive — Hyundai Motor (005380), Kia (000270), and the Hyundai-supplier chain.
EV battery — LG Energy Solution (373220), Samsung SDI (006400), SK Innovation (096770).
Internet — Naver (035420), Kakao (035720). Korea's local internet incumbents, neither has a meaningful Western analog.
Banking and financials — KB Financial (105560), Shinhan (055550), Hana (086790).
Steel and shipbuilding — POSCO (005490), Hyundai Heavy.
K-content — major entertainment companies cross-listed across KOSPI and KOSDAQ.

Korea's chaebol structure means cross-holdings within conglomerate groups are an important second-order analytical dimension — Samsung Electronics, Samsung SDI, and Samsung Biologics are separately listed but linked. Any factor model needs to handle the group-affiliation dimension.

KOSPI market microstructure

Trading hours — 09:00–15:30 KST with an opening call auction at 08:30–09:00 and closing call auction at 15:20–15:30.
Daily price limit — ±30% on KOSPI (loosened from ±15% in 2015). KOSDAQ is also ±30%.
Settlement — T+2.
Lot size — 1 share for most listings (historically 10).
Short selling rules — Korea periodically imposes short-selling bans during market stress; the 2023-2024 ban and the 2024 partial resumption are recent examples that affect any short-side backtest.

The Foreign Investor Registration regime (KRX requires foreign investors to register for an investor ID) is no longer the bottleneck it was pre-2023 — the requirement was largely lifted in late 2023 — but it remains relevant for data-source attribution and historical comparison.

Working Python example

The KOSPI Stock Screener actor returns the KOSPI (and optionally KOSDAQ) universe with quotes, market cap in KRW, sector tags using KRX industry codes, and chaebol group affiliation. Curl:


    curl -X POST "https://api.apify.com/v2/acts/nexgendata~kospi-stock-screener/run-sync-get-dataset-items?token=$APIFY_TOKEN" \
      -H "Content-Type: application/json" \
      -d '{"market": "KOSPI", "minMarketCapKRW": 1000000000000, "sectors": ["Semiconductors","EV Battery"], "maxResults": 500}'

Python — large-cap KOSPI screen:


    import os, json, urllib.request

    APIFY_TOKEN = os.environ["APIFY_TOKEN"]
    ACTOR = "nexgendata~kospi-stock-screener"

    payload = json.dumps({
        "market": "KOSPI",
        "minMarketCapKRW": 1_000_000_000_000,
        "includeGroupAffiliation": True,
        "maxResults": 500,
    }).encode("utf-8")

    url = f"https://api.apify.com/v2/acts/{ACTOR}/run-sync-get-dataset-items?token={APIFY_TOKEN}"
    req = urllib.request.Request(url, data=payload, method="POST",
                                  headers={"Content-Type": "application/json"})

    with urllib.request.urlopen(req, timeout=600) as r:
        stocks = json.loads(r.read())

    for s in sorted(stocks, key=lambda x: -x.get("marketCapKRW", 0))[:20]:
        print(f"{s['ticker']} | {s.get('nameEn') or s['name']} | "
              f"mcap={s['marketCapKRW']/1e12:.1f}T KRW | group={s.get('group','-')}")

Returned shape:


    {
      "ticker": "005930",
      "isin": "KR7005930003",
      "name": "삼성전자",
      "nameEn": "Samsung Electronics Co., Ltd.",
      "market": "KOSPI",
      "krxIndustry": "Electrical, electronic equipment",
      "sector": "Semiconductors",
      "group": "Samsung",
      "lastPrice": 78400,
      "marketCapKRW": 467500000000000,
      "peTTM": 14.6,
      "pbMRQ": 1.4,
      "dividendYieldTTM": 2.0,
      "foreignOwnershipPct": 53.2,
      "tradingStatus": "NORMAL"
    }

Cost comparison: KOSPI data sources

Source	Coverage	English	Annual cost
FnGuide / DataGuide	Full KOSPI + KOSDAQ + estimates	Yes	~$3k–$15k+ per seat
Bloomberg Terminal (KS / KQ tickers)	Full + news	Yes	~$24k per seat
Refinitiv Eikon Korea	Full	Yes	~$22k per seat
KRX own data services	Full	Yes	Subscription, varies
Yahoo Finance Korea (.KS / .KQ)	Quotes	Yes	Free, terms-of-use limits
KOSPI Stock Screener actor	KOSPI + optionally KOSDAQ, screening fields	Yes	PPE per screen

FnGuide / DataGuide remain canonical for Korean sell-side and institutional buy-side desks — particularly for forward-estimate data (the "consensus" feeds that local analysts use) and the unique-to-Korea fundamentals coverage. For screening, factor research, and dashboarding without the forward-estimate dependency, the public KRX-derived stack is meaningfully sufficient.

What this enables

Sector dashboards — semiconductor cycle tracking via Samsung + SK Hynix, EV-battery cycle via LGES + Samsung SDI + SK Innovation, internet via Naver + Kakao
Factor research on the KOSPI universe — value, quality, momentum factor portfolios with chaebol-group adjustment
Foreign ownership monitoring — KRX publishes daily foreign-ownership percentages; tracking deltas at the name level is a meaningful flow signal for Korea-equity strategies
ASEAN-cluster regional dashboards — pair KOSPI with SGX, PSE Edge, and the broader regional disclosure feeds

KOSPI vs. KOSDAQ practical handling

For Korea-focused funds the two boards are typically run as a single research universe with sector-weighted construction. KOSPI dominates large-cap; KOSDAQ dominates K-content, biotech, and EV-battery small/mid. The actor supports either or both — most use cases pull both.

Pairing with disclosure data

Quote and fundamentals data alone are insufficient for any time-sensitive research workflow on Korean equities — the relevant news flow comes through KIND disclosures. A complete Korea research pipeline pulls the KOSPI screener for the static universe + fundamentals, then KIND for the real-time event flow on holdings. Adjacent feeds: SEC Event Router for Korean ADRs on US exchanges (a small but growing cohort), and HKEX for Korea-HK secondary listings.

One execution note

Korea applies a securities transaction tax (currently 0.18% combined for KOSPI sells, scheduled for further reduction) that materially affects any high-turnover backtest. Factor-rotation strategies in Korea should explicitly model the tax in addition to spread. For long-only factor portfolios with monthly rebalance the impact is small; for daily or weekly rotation strategies it is the dominant cost.

Foreign ownership caps and the FIR regime

Although the Foreign Investor Registration certificate requirement was largely lifted in late 2023, several KOSPI sectors retain statutory foreign-ownership caps under Korean law — telecoms, broadcasting, defense-adjacent, and a handful of utilities still have explicit caps ranging from 25% to 49%. For any Korea-focused strategy that holds capped-sector names, the headroom (current foreign ownership vs. statutory cap) is a first-class data field. KRX publishes daily foreign-ownership statistics that flow into the actor's foreignOwnershipPct field; the cap-aware research workflow filters on capped sectors and screens for low-headroom names that can hit forced unwind triggers.

Chaebol cross-holdings and the affiliation field

The KRX classifies issuers by chaebol group affiliation under the Monopoly Regulation and Fair Trade Act — the "large business group" designation that includes the Samsung group, Hyundai Motor group, SK group, LG group, Lotte group, and roughly 75 others. The affiliation matters for any factor model that needs to handle correlated returns within a group (Samsung Electronics and Samsung SDI exhibit higher pairwise correlation than two unrelated semis) and for any governance-quality factor. The actor returns the group field, which lets you build group-neutral factor portfolios or, conversely, group-concentration screens.

Get started: Pull your first KOSPI screen free at the KOSPI Stock Screener actor page.

Build an Automated Hacker News Tracker for Your Tech Niche

NexGenData — Tue, 07 Jul 2026 15:42:34 +0000

Build an Automated Hacker News Tracker for Your Tech Niche

You're building a React analytics tool. A major incident hits HN with the title "React Performance Issues at Scale: Here's What We Found" and it hits #3. By the time you see it, 8 hours later, the conversation is 200 comments deep and you've missed the window to contribute meaningfully.

Or worse, you don't see it at all, and miss a critical discussion about your product space.

Hacker News moves fast. Trends emerge, discussions peak, and fade within hours. Manual monitoring means either constantly refreshing the site or missing valuable signals about your market, competitors, and community sentiment.

What if instead you could set up a system that automatically monitors HN, filters for topics relevant to your niche, identifies viral posts early, and sends you real-time alerts? That's the power of an automated HN tracker.

In this post, I'll walk you through building one that catches the signals that matter for your business.

Why Automated HN Monitoring Matters

Hacker News is where engineers congregate. If your product is developer-focused, your market is here. But it's not just about finding mentions of your product. It's about understanding:

What problems are engineers actively discussing? If 500 engineers are discussing "database migration horror stories" in a single thread, that's market research. Maybe your tool should position around frictionless migrations.

How fast do certain topics trend? Posts about AI security shoot to #1 in 3 hours. Traditional cloud infrastructure discussions slowly climb over days. Understanding velocity tells you where excitement is building.

What are the sentiment patterns? A post about a competitor hits #5. Read the comments. Are people praising them or criticizing them? That's competitive intel.

Where are discussion gaps? No one's talking about a feature you know matters. Either the market doesn't care yet, or there's an education opportunity.

How early can you detect trends? A post about Rust for systems programming sits at #50 with 20 comments. Six hours later, it's #2 with 400 comments. Catching it at #50 means you can be an early voice, shaping the narrative.

Manual monitoring can't scale to capture this. Automated monitoring can.

Setting Up Your Hacker News Scraper

Start by understanding what you want to track. For a developer tools company, I'd monitor:

Posts about the languages/frameworks you support
Posts about your competitors
Posts about the problems you solve
General trends in your category (databases, APIs, testing, etc.)

Use the Apify Hacker News Scraper to extract the front page and top stories on a regular schedule.

Your basic input configuration:

{
  "startUrls": [
    {
      "url": "https://news.ycombinator.com/newest"
    },
    {
      "url": "https://news.ycombinator.com/best"
    }
  ],
  "maxPostsPerPage": 30,
  "scrapeComments": true,
  "maxComments": 20,
  "pageTimeout": 30000,
  "useChrome": true,
  "proxyConfiguration": {
    "useApifyProxy": true
  }
}

This pulls the newest and best posts with top comments, giving you context.

Understanding Your HN Scraper Output

Here's what a typical run returns (abbreviated):

{
  "posts": [
    {
      "postId": "41234567",
      "title": "We Rewrote Our Backend in Rust and Gained 10x Performance",
      "url": "https://example.com/rust-backend",
      "author": "john_eng",
      "score": 1247,
      "commentCount": 342,
      "timestamp": "2026-04-04T14:23:00Z",
      "category": "full-stack",
      "rank": 1,
      "comments": [
        {
          "commentId": "41234568",
          "author": "sarah_dev",
          "text": "Did you consider Go instead? Rust has a steep learning curve for teams.",
          "score": 245,
          "timestamp": "2026-04-04T14:35:00Z",
          "depth": 1
        },
        {
          "commentId": "41234569",
          "author": "john_eng",
          "text": "Go wasn't fast enough for our use case. Rust gave us the performance we needed without GC pauses.",
          "score": 189,
          "timestamp": "2026-04-04T14:42:00Z",
          "depth": 2
        }
      ]
    },
    {
      "postId": "41234570",
      "title": "PostgreSQL Sharding Lessons from Running 1000 Databases",
      "url": "https://example.com/postgres-sharding",
      "author": "db_expert",
      "score": 892,
      "commentCount": 156,
      "timestamp": "2026-04-04T12:11:00Z",
      "category": "databases",
      "rank": 2,
      "comments": [
        {
          "commentId": "41234571",
          "author": "ops_person",
          "text": "Good breakdown. We went with CockroachDB instead for similar guarantees.",
          "score": 134,
          "timestamp": "2026-04-04T12:45:00Z",
          "depth": 1
        }
      ]
    },
    {
      "postId": "41234571",
      "title": "Why We're Migrating Away from Kubernetes",
      "url": "https://example.com/k8s-migration",
      "author": "infra_lead",
      "score": 654,
      "commentCount": 201,
      "timestamp": "2026-04-04T10:05:00Z",
      "category": "infrastructure",
      "rank": 3,
      "comments": [
        {
          "commentId": "41234572",
          "author": "k8s_maintainer",
          "text": "What specific pain points drove the decision? K8s can be overkill for many orgs.",
          "score": 178,
          "timestamp": "2026-04-04T10:22:00Z",
          "depth": 1
        }
      ]
    }
  ],
  "scrapingRunId": "run-20260404-1",
  "scrapedAt": "2026-04-04T15:00:00Z",
  "totalPostsScraped": 30
}

Notice the structure: posts have scores and comment counts, making it easy to identify which discussions are heating up.

Building Topic Filtering and Alerts

Raw data isn't actionable. You need filtering. Let's say you're a DevOps tool company focused on Kubernetes. Create a filter that catches relevant posts:

const filterRelevantPosts = (posts) => {
  const relevantKeywords = [
    'kubernetes', 'k8s', 'docker', 'container', 'devops',
    'infrastructure', 'deployment', 'orchestration', 'microservices'
  ];

  const competitorKeywords = [
    'hashicorp', 'terraform', 'aws', 'gke', 'eks'
  ];

  return posts.filter(post => {
    const titleLower = post.title.toLowerCase();
    const isRelevant = relevantKeywords.some(kw => titleLower.includes(kw));
    const isCompetitorMention = competitorKeywords.some(kw => titleLower.includes(kw));

    return isRelevant || isCompetitorMention;
  }).map(post => ({
    ...post,
    relevanceScore: calculateRelevance(post),
    sentimentTrend: analyzeSentiment(post.comments)
  }));
};

const calculateRelevance = (post) => {
  // Score based on engagement and recency
  const ageMinutes = (Date.now() - new Date(post.timestamp).getTime()) / (1000 * 60);
  const engagementScore = post.score + (post.commentCount * 5);
  const velocityBoost = ageMinutes < 60 ? 1.5 : 1.0; // Recent posts weighted higher

  return Math.round(engagementScore * velocityBoost);
};

const analyzeSentiment = (comments) => {
  const positive = comments.filter(c => c.text.includes('great') || c.text.includes('excellent')).length;
  const negative = comments.filter(c => c.text.includes('bad') || c.text.includes('issue')).length;

  return {
    positiveCount: positive,
    negativeCount: negative,
    trend: positive > negative ? 'positive' : negative > positive ? 'negative' : 'neutral'
  };
};

Running this filter on your scraped data returns only the posts that matter to your niche, scored by relevance.

Setting Up Apify Schedules and Webhooks

Don't run this manually. Schedule your scraper to run every 2 hours during business hours:

Go to the Apify Hacker News Scraper
Create a scheduled task (Schedules tab)
Set frequency: "0 */2 * * *" (every 2 hours)
Configure a webhook to POST results to your backend

Your webhook endpoint receives the fresh data and immediately:

Filters for relevant posts
Compares against the previous run (has post ranking changed?)
Identifies newly trending posts (went from #50 to #5)
Routes high-relevance posts to Slack

Slack Integration: Real-Time Alerts

Here's how to send filtered posts to Slack:

const sendSlackAlert = async (post, relevanceScore) => {
  const webhookUrl = process.env.SLACK_WEBHOOK_URL;

  const color = relevanceScore > 500 ? '#ff6b35' : '#4CAF50';
  const emoji = post.sentimentTrend === 'positive' ? ':thumbsup:' : ':warning:';

  const slackPayload = {
    attachments: [
      {
        color: color,
        title: `${emoji} ${post.title}`,
        title_link: `https://news.ycombinator.com/item?id=${post.postId}`,
        fields: [
          {
            title: 'Score',
            value: `${post.score}`,
            short: true
          },
          {
            title: 'Comments',
            value: `${post.commentCount}`,
            short: true
          },
          {
            title: 'Relevance',
            value: `${relevanceScore}/1000`,
            short: true
          },
          {
            title: 'Sentiment',
            value: `Positive: ${post.sentimentTrend.positiveCount}, Negative: ${post.sentimentTrend.negativeCount}`,
            short: true
          }
        ],
        footer: `Posted by ${post.author} at ${new Date(post.timestamp).toLocaleTimeString()}`
      }
    ]
  };

  await fetch(webhookUrl, {
    method: 'POST',
    headers: { 'Content-Type': 'application/json' },
    body: JSON.stringify(slackPayload)
  });
};

Now your team gets a Slack notification within 5 minutes of a relevant post going viral. You can comment early, get visibility, build credibility.

Tracking Trends Over Time

To identify emerging trends, compare multiple runs:

const identifyTrendingTopics = (currentRun, previousRun) => {
  const currentPosts = new Map(currentRun.posts.map(p => [p.postId, p]));
  const previousRanks = new Map(previousRun.posts.map((p, idx) => [p.postId, idx]));

  const trending = [];

  currentRun.posts.forEach((post, currentRank) => {
    const previousRank = previousRanks.get(post.postId);

    if (previousRank !== undefined) {
      const rankChange = previousRank - currentRank;
      const scoreChange = post.score - (currentPosts.get(post.postId)?.score || 0);

      if (rankChange > 5 || scoreChange > 100) {
        trending.push({
          postId: post.postId,
          title: post.title,
          rankImprovement: rankChange,
          scoreGain: scoreChange,
          currentRank: currentRank
        });
      }
    }
  });

  return trending.sort((a, b) => b.rankImprovement - a.rankImprovement);
};

Output shows which posts are accelerating, helping you spot trends early.

Real-World Application: Building a Competitive Moat

Imagine you work on a database company. Your HN tracker catches a post: "We Switched from PostgreSQL to TimescaleDB for Time-Series Data" at rank #40 with 15 comments.

You're notified immediately. You read the discussion. The author mentions pain points you've solved in your product. You comment thoughtfully, sharing relevant experience. Your comment gets upvoted. You gain visibility among your target market.

By 2 PM, the post is #5 with 200 comments. You're one of the visible voices in the conversation. That's brand building.

Meanwhile, competitors who monitor HN manually missed it entirely. They're not in the conversation. You captured the moment.

Scale this across dozens of relevant posts per week. Over months, your team becomes the go-to voice in critical discussions. That's how you build a technical moat.

Getting Started Today

Head to Apify Hacker News Scraper and run one manual scrape. See what posts you find. Then set up a 2-hour schedule.

Connect it to a Slack webhook. Within a day, you'll be seeing HN trends as they emerge. Your competitive advantage depends on moving faster than the market. Automation gives you that speed.

New: Norway Company Registry — Brønnøysund Data API — look up any Norwegian company from the official Brønnøysund register

NexGenData — Mon, 06 Jul 2026 16:16:40 +0000

What it does

This actor queries Norway's official Brønnøysund company register (Brreg) by company name or organization number and returns the full public record — organization form, industry, employee count, registered address and status. It's a clean API over the authoritative Norwegian registry for KYB and prospecting.

Who it's for

KYB and onboarding teams verifying Norwegian counterparties, B2B sales teams prospecting into Norway, compliance and due-diligence analysts, and data teams enriching records with authoritative registry data.

Sample fields / output

name — registered company name
orgnr — organization number
org_form — legal organization form
industry — NACE industry code / description
employees — employee count
address — registered address
status — registration status

Example use cases

Verify a Norwegian company's legal form, address and status for KYB onboarding.
Enrich CRM records with authoritative Brønnøysund registry data.
Build prospect lists of Norwegian companies filtered by industry or size.

Try it now: Norway Company Registry — Brønnøysund Data API on Apify.

Related actors

FAQ

What's the data source?

Norway's official Brønnøysund Register Centre (Brreg) — the authoritative public company register.

Can I search by name and org number?

Yes — look up by company name or by organization number (orgnr).

What's in the full record?

Organization form, industry, employee count, registered address and registration status.

New: Financial Adviser Directory — BrokerCheck & IAPD Records API — search FINRA BrokerCheck & SEC IAPD adviser records

NexGenData — Mon, 06 Jul 2026 16:15:59 +0000

What it does

This actor searches the public FINRA BrokerCheck and SEC IAPD registers and returns structured records for financial advisers and brokers — name, CRD number, current firm, state and registration status. It turns two public regulatory registers into a queryable dataset for recruiting and vendor prospecting.

Who it's for

Financial-advisory recruiters and RIA aggregators sourcing advisers, wealth-management BD teams, compliance and due-diligence researchers verifying registrations, and data teams building adviser directories.

Sample fields / output

name — adviser / broker name
crd — CRD number
firm — current registered firm
state — registration state
status — registration status

Example use cases

Build a sourcing list of advisers at target firms for recruiting outreach.
Verify an adviser's CRD, firm and registration status during due diligence.
Track adviser registrations by state or firm for market mapping.

Try it now: Financial Adviser Directory — BrokerCheck & IAPD Records API on Apify.

Related actors

FAQ

Where does the data come from?

Public FINRA BrokerCheck and SEC IAPD registers — official regulatory sources.

Is this suitable for recruiting?

Yes — it's designed for recruiting and vendor prospecting using public adviser records.

What identifiers are returned?

Each record includes the adviser's name, CRD number, current firm, state and registration status.

New: US Government Contracts — Open Data Procurement API — track awarded US government contracts in one feed

NexGenData — Mon, 06 Jul 2026 16:15:11 +0000

What it does

This actor aggregates awarded US government contracts published across city and county open-data portals into a single structured feed. For each award you get the vendor, contract value, awarding agency and dates — turning scattered municipal procurement data into a dataset you can prospect and analyze against. Built for vendor prospecting and bid intelligence.

Who it's for

Government-focused sales and BD teams, GovTech vendors, bid and proposal analysts, market researchers sizing public-sector spend, and anyone who needs awarded-contract data without visiting dozens of portals by hand.

Sample fields / output

vendor — awarded contractor / supplier
value — contract / award amount
agency — awarding government body
award_date — date of award
description — contract scope / title

Example use cases

Build a prospect list of vendors already winning public-sector contracts.
Track a competitor's public-sector wins across multiple jurisdictions.
Size and segment municipal spend by agency, vendor or contract value.

Try it now: US Government Contracts — Open Data Procurement API on Apify.

Related actors

FAQ

Which governments are covered?

Awarded contracts from US city and county open-data procurement portals, aggregated into one feed.

What can I do with the data?

It's built for vendor prospecting and bid intelligence — find who's winning work, with whom, and for how much.

What fields do I get per contract?

Vendor, contract value, awarding agency, award date and a scope/description.

New: UK Trademark Search — UKIPO Brand Clearance API — clear UK brand names against the GB register

NexGenData — Mon, 06 Jul 2026 16:14:49 +0000

What it does

This actor searches the UK (GB) trademark register maintained by the UKIPO, so you can run brand-clearance checks post-Brexit without the EU register. Query by mark text, Nice class and status, and get structured records back — each with an official UKIPO deep-link to the source entry. You pay per record.

Who it's for

Trademark attorneys and paralegals running clearance searches, brand and product teams naming new offerings, IP watch services, and founders checking a name is free before they file in the UK.

Sample fields / output

mark — the trademark text
nice_class — Nice classification class(es)
status — application/registration status
owner — applicant / proprietor
application_number — UKIPO reference
link — official UKIPO deep-link

Example use cases

Run a UK clearance search before filing a new trademark application.
Monitor a Nice class for newly filed marks that conflict with your brand.
Bulk-check a list of candidate product names against the GB register.

Try it now: UK Trademark Search — UKIPO Brand Clearance API on Apify.

Related actors

FAQ

Does this cover the EU register too?

No — this searches the UK (GB) register at the UKIPO, which is the one that matters for UK protection after Brexit. Use the EUIPO actor for the EU register.

Can I search by Nice class?

Yes — you can filter by mark text, Nice class and status.

Are results linked to the official source?

Every record includes an official UKIPO deep-link to the underlying register entry.

New: PR TIMES Scraper — Japan Press Releases · プレスリリース API — monitor Japan's dominant PR wire by company or keyword

NexGenData — Mon, 06 Jul 2026 16:14:12 +0000

What it does

PR TIMES is Japan's dominant press-release wire, and this actor turns it into a clean, structured feed. Monitor releases by company or keyword and get back one record per release — headline, issuing company, publish date, canonical link and a summary — with native Japanese (日本語) text preserved. You pay per release returned.

Who it's for

PR and comms teams tracking Japanese-market announcements, market and equity researchers following Japanese issuers, competitive-intelligence analysts, and anyone building a Japan news feed who needs structured data instead of scraping HTML.

Sample fields / output

headline — release title (Japanese)
issuer — issuing company name
date — publish date
link — canonical PR TIMES URL
summary — release summary text

Example use cases

Monitor a watchlist of Japanese companies for new announcements in near-real time.
Keyword-track a product, brand or theme across the entire PR TIMES wire.
Feed structured Japanese press releases into a translation or LLM summarization pipeline.

Try it now: PR TIMES Scraper — Japan Press Releases · プレスリリース API on Apify.

Related actors

FAQ

Is the text in Japanese?

Yes — headlines and summaries are returned in native Japanese (日本語) exactly as published, ready for translation or LLM processing.

How is it billed?

Pay per release returned — you're only charged for the structured records you actually get back.

Can I filter by company or keyword?

Yes, you can monitor by specific issuing company or by keyword across the whole wire.

China Social Media Trends API: Tracking Weibo, Douyin, and Baidu in Real Time

NexGenData — Mon, 06 Jul 2026 14:18:03 +0000

中文版: 中文版本

TL;DR. Mainland Chinese consumer attention concentrates on three platforms with almost no overlap to Western analogs: Weibo (~600M MAU, the closest functional equivalent to early Twitter for trending public discourse), Douyin (the domestic version of TikTok with ~750M DAU and a distinct content profile), and Baidu (the dominant Chinese search engine — Google has minimal China share). For consumer-brand research, China consumer-equity due diligence, and competitive intelligence on China-market entrants, you need real-time access to what's trending on these three platforms. Each exposes some public surface area but none publishes a comprehensive English-language API. This guide covers what each platform actually reveals about Chinese consumer attention, the structured trend data you can pull, and a working Python pipeline using the China Trends Tracker actor.

Why these three platforms

The Chinese internet operates on a parallel stack to the global Western one — Google, Facebook, and X have negligible mainland presence. For a non-Chinese researcher trying to understand what consumers are paying attention to in China, the three platforms above are the canonical sources, supplemented by RED / Xiaohongshu for lifestyle and Bilibili for younger-skewing video.

Weibo (微博) — public posts and trending topics. The closest cultural analog is "what's trending on X/Twitter today." The Weibo Hot Search (微博热搜) list updates throughout the day and is heavily watched as a real-time signal of public attention, including on consumer-brand topics.
Douyin (抖音) — the dominant short-video platform. Distinct from international TikTok (different content moderation, different creator economy, no overlap of creator accounts). Trending videos and hashtag challenges concentrate consumer-product attention faster than any other surface.
Baidu (百度) — search interest. Baidu Index (百度指数) publishes search-volume data with weekly and daily granularity for keywords. There is no public Baidu Index API; the index is browser-only.

What trend data is actually useful

For consumer-research and brand-monitoring use cases the structured fields that matter:

Trending topic with rank — Weibo Hot Search position over time. Rank movement is more informative than absolute rank.
Topic category — Weibo flags topics as entertainment, current affairs, sports, etc. Filter heavily for any business research.
Engagement signal — Weibo post count for a topic; Douyin video count and aggregate views per hashtag.
Time series — same topic across multiple snapshots to compute lifetime, peak, and decay.
Associated brands / entities — named-entity extraction over the trend text and top-engagement posts.

For Baidu Index specifically: the canonical signal is search-volume index for a keyword, with overall and Daily-Active-User splits. Higher noise than Weibo trending but lower bot contamination.

Working Python example

The China Trends Tracker actor consolidates Weibo Hot Search, Douyin trending hashtags, and Baidu search-interest signals into a single normalized JSON feed. Curl:


    curl -X POST "https://api.apify.com/v2/acts/nexgendata~china-trends-tracker/run-sync-get-dataset-items?token=$APIFY_TOKEN" \
      -H "Content-Type: application/json" \
      -d '{"platforms": ["weibo","douyin","baidu"], "category": "consumer", "maxResults": 200}'

Python — brand mention monitoring:


    import os, json, urllib.request

    APIFY_TOKEN = os.environ["APIFY_TOKEN"]
    ACTOR = "nexgendata~china-trends-tracker"

    brand_terms = ["蜜雪冰城","瑞幸","喜茶","茅台","小米","比亚迪","蔚来","SHEIN"]

    payload = json.dumps({
        "platforms": ["weibo","douyin","baidu"],
        "keywords": brand_terms,
        "includeAssociatedPosts": True,
    }).encode("utf-8")

    url = f"https://api.apify.com/v2/acts/{ACTOR}/run-sync-get-dataset-items?token={APIFY_TOKEN}"
    req = urllib.request.Request(url, data=payload, method="POST",
                                  headers={"Content-Type": "application/json"})

    with urllib.request.urlopen(req, timeout=600) as r:
        trends = json.loads(r.read())

    for t in trends:
        print(f"{t['platform']} | rank={t.get('rank','-')} | {t['topic']}")
        print(f"  engagement={t.get('engagement','-')} | sentiment={t.get('sentiment','-')}")

Returned shape:


    {
      "platform": "weibo",
      "snapshotAt": "2026-05-29T11:00:00+08:00",
      "rank": 7,
      "topic": "蜜雪冰城新品",
      "topicEn": "Mixue new product launch",
      "category": "BRAND_CONSUMER",
      "engagement": 4280000,
      "engagementType": "discussion_count",
      "associatedEntities": ["蜜雪冰城","Mixue Bingcheng"],
      "topPostUrl": "https://weibo.com/...",
      "snapshotUrl": "https://s.weibo.com/top/summary"
    }

Cost comparison: China social listening tools

Source	Coverage	Sentiment / NER	Annual cost
Baidu Index (web)	Search interest, by keyword	No, requires Baidu login	Free, no API
Weibo official open platform	Limited, app-only API for approved partners	No	Partner program required
Sina Weibo public hot search	Top trending topics	No	Free, web only
Brandwatch China module	Multi-platform aggregation	Yes, with English UI	Enterprise quote, $10k+ year
Talkwalker China module	Multi-platform aggregation	Yes	Enterprise quote
Local CN social-listening vendors (CMNet, etc.)	Deep CN coverage	Yes	Variable, often $5k–$30k/yr
China Trends Tracker actor	Weibo + Douyin + Baidu signals	Basic NER, sentiment optional	PPE per snapshot

For enterprise-scale brand surveillance with compliance-grade audit trails, Brandwatch and Talkwalker remain the right choice — they integrate with marketing-team workflows, retain historical archives, and handle multi-region monitoring. For research-team and investment-team use cases that just need "what's trending in China today on the platforms my consumers actually use," the lighter PPE-priced approach is materially better fit.

What this enables

Consumer-brand due diligence for VC and PE deals in China consumer — pre-deal sentiment baseline plus post-deal tracking
Equity research on China consumer names — Mixue, Luckin, BYD, Xiaomi, NIO. Real-time attention signals as a feature for fundamental analysis
Competitive intelligence — track non-Chinese brands' Chinese-market attention (Apple, Tesla, Nike all have heavy CN signal but no equivalent US public data)
Crisis monitoring for global brands operating in China — Weibo trending is typically the first signal of brewing reputational issues

Sampling cadence and storage

Weibo Hot Search rotates quickly — the average lifetime of a trending topic is hours, not days. For meaningful trend research you want 15-30 minute snapshots stored as a time series, then aggregated nightly. Daily-only snapshots miss the majority of relevant topics. The actor exposes batch and scheduled modes; for serious time-series use, schedule snapshots at 15-minute granularity.

Bot contamination and trend authenticity

One realistic caveat for any Chinese social-trend research: paid promotion and coordinated inauthentic activity are well-documented features of the Weibo and Douyin trend ecosystems. Brand-paid trending topics (the platforms run formal trending-buy ad products), MCN-coordinated creator promotions, and outright bot amplification all show up in the raw trend feed. For investment-grade research, the trend rank alone is insufficient — you need engagement-velocity profile (organic trends accelerate gradually; paid trends spike then collapse), creator-account quality scoring, and cross-platform corroboration (a trend that appears organically on Weibo, Douyin, and Baidu simultaneously is materially more credible than a single-platform spike). The actor exposes the raw feed; downstream credibility scoring is the analyst's responsibility.

Censorship and trend disappearance

A trend that appears on the Weibo Hot Search and then disappears within minutes is itself a signal — sensitive topics get scrubbed faster than organic decay would explain. For research workflows the practical handling is to sample at high frequency (15-minute intervals minimum) and to retain the snapshot history independently of the platform. The disappearance window is occasionally a useful research feature; conversely, a trend that survives multiple snapshots is meaningfully more durable. The actor's time-series mode captures appearance and disappearance timestamps for downstream analysis.

What this does not cover

WeChat (微信) — the substantial majority of WeChat content is private (1:1 messaging and closed Moments). There is no scrapable public Weibo-equivalent surface.
RED / Xiaohongshu (小红书) — lifestyle content, distinct platform with its own dynamics. Worth a separate feed for any beauty / fashion / lifestyle research.
Toutiao (今日头条) — news-aggregator with editorial curation, structurally different from Weibo trending.

For consumer-equity research linking attention data to share prices, pair this feed with A-share consumer-sector data via Eastmoney for mainland-listed names and Chinese ADRs for the US-listed consumer cohort.

Get started: Pull your first China trends batch free at the China Trends Tracker actor page.

New: HK Centaline Property Index — the Hong Kong Centaline (CCL) property price index as clean JSON

NexGenData — Sun, 05 Jul 2026 19:47:52 +0000

What it does

HK Centaline Property Index tracks the Hong Kong Centaline property price index (CCL) and returns it as clean JSON — one record per index reading, with the value, period and month-on-month change. It is a monitoring play built for analysts, quant researchers, fintech builders and AI agents.

Who it's for

Property and macro analysts tracking Hong Kong home prices, quant and fintech builders wiring the CCL into a model, and AI agents that need the latest index value and its monthly change on demand.

Sample fields / output

index_name, index_full_name — the CCL series
index_value — the index reading
change_pct, change_pct_mom — change and month-on-month change
change_period, date, data_type
region, publisher

Example use cases

Pull the latest CCL reading and its month-on-month change on a schedule
Wire the Hong Kong price index into a macro or property model
Have an AI agent report the current CCL value and monthly move on demand

Try HK Centaline Property Index on Apify →

FAQ

What index does this cover?

The Hong Kong Centaline property price index (CCL / 中原城市指數), returned as clean JSON with value, period and month-on-month change.

How is it priced?

$0.10 per index record (about 10 records per $1), pay-per-event.

Can I run it on a schedule?

Yes — it is a monitoring play. Schedule a daily or weekly recurring pull to keep a standing CCL time series.

DEV Community: NexGenData

MXToolbox Costs $149/mo. I Built a DMARC/SPF/DKIM Bulk Auditor for $0.005 Per Domain

MXToolbox Costs $149/mo. I Built a DMARC/SPF/DKIM Bulk Auditor for $0.005 Per Domain

The Problem: Email Authentication Went From "Nice to Have" to Enforced

Why Single-Domain Tools Don't Work at Scale

What the Actor Checks (and Why Each Thing Matters)

1. MX Records

2. SPF Record

3. DMARC Record

4. DKIM Selectors

5. BIMI Record

6. MTA-STS and TLS-RPT

Try It: One Domain with cURL

Bulk Mode: Auditing 100 Domains in Python

Use It in Make / n8n: Monitor a Domain List on a Schedule

Real Use Cases I've Seen This Replace

1. Agencies Auditing Client Domains Before Kickoff

2. M&A Due Diligence

3. Pre-launch Checks for Transactional Email

4. Cold-Outreach Sending Infrastructure

Pricing: Half a Cent Per Domain, Honestly

The Scoring Heuristic (So You Know What "Grade A" Means)

Related NexGenData Actors for a Full Deliverability Stack

Answering the Common Questions

Try It

Literature Review Automation: Search and Analyze Hundreds of Academic Papers in Minutes

Literature Review Automation: Search and Analyze Hundreds of Academic Papers in Minutes

Why Automate Your Literature Review

Building Your Automated Search Strategy

Setting Up Google Scholar Scraping

Understanding Scholar Scraper Output

Categorizing Papers by Citation Tier

Identifying Key Authors and Research Communities

Building Your Research Landscape Map

Creating Your Literature Review Database

Real-World Application: Your Thesis Introduction

Getting Started

Neighborhood-by-Neighborhood: Comparing Real Estate Markets with Redfin Data

Neighborhood-by-Neighborhood: Comparing Real Estate Markets with Redfin Data

Why Neighborhood Comparison Matters

Setting Up Your Neighborhood Comparison

Understanding Redfin Scraper Output

Building Your Neighborhood Comparison Matrix

Analyzing Price Segmentation

Days on Market as a Market Temperature Signal

Building a Multi-Neighborhood Workflow

Real-World Application: Making Your Buy Decision

Getting Started

KOSPI Stock Screener API: Korean Equities Market Data Pipeline

What's in the KOSPI universe

KOSPI market microstructure

Working Python example

Cost comparison: KOSPI data sources

What this enables

KOSPI vs. KOSDAQ practical handling

Pairing with disclosure data

One execution note

Foreign ownership caps and the FIR regime

Chaebol cross-holdings and the affiliation field

Related Reading

Build an Automated Hacker News Tracker for Your Tech Niche

Build an Automated Hacker News Tracker for Your Tech Niche

Why Automated HN Monitoring Matters

Setting Up Your Hacker News Scraper

Understanding Your HN Scraper Output

Building Topic Filtering and Alerts

Setting Up Apify Schedules and Webhooks

Slack Integration: Real-Time Alerts

Tracking Trends Over Time

Real-World Application: Building a Competitive Moat

Getting Started Today

New: Norway Company Registry — Brønnøysund Data API — look up any Norwegian company from the official Brønnøysund register

What it does

Who it's for

Sample fields / output

Example use cases

Related actors

FAQ

What's the data source?

Can I search by name and org number?