DEV Community: NexGenData

Public Real Estate Data Scrapers for Regional Market Research

NexGenData — Fri, 26 Jun 2026 05:45:21 +0000

Real estate research used to mean buying a Bloomberg terminal seat and hoping the vendor licensed the right region. In 2026 the data is mostly public — Zillow, Redfin, Rightmove, Singapore's Urban Redevelopment Authority (URA), Hong Kong's Centaline Property Index, India's MagicBricks — but it sits behind five different websites, four different languages, and zero unified APIs. The result: most analyst comps stop at the US border, REIT screens lean heavily on one country's listings, and PropTech VCs evaluating a Jakarta or Mumbai thesis end up paying a regional aggregator $40K/year for data the source publishes for free.

This guide is a playbook for the alternative: region-specific public real estate data scrapers , run on demand, paid by usage, exported to CSV or piped into a BI stack. We will walk through the actors that cover the US, UK, Singapore, Hong Kong, Denmark, and India; how to combine them into market-entry briefs and cap-rate comparisons; and where the regulatory and methodological landmines sit. If you run a REIT desk, a relocation practice, an urban-planning model, or a PropTech competitive intelligence function, this is the new stack.

The Problem: Real Estate Data Is Hyper-Regional and Hyper-Fragmented

Property markets are local in a way that, say, equities are not. A NASDAQ ticker trades the same way from Tokyo or Toronto. A two-bedroom condo in Tanjong Pagar trades against entirely different rules, taxes, leasehold structures, and buyer pools than a two-bedroom in Brooklyn or Battersea. The data infrastructure reflects that fragmentation. Zillow and Redfin dominate the US Multiple Listing Service (MLS) ecosystem. Rightmove owns roughly 80% of UK listing traffic. Singapore splits its disclosures across the Housing Development Board (HDB) for public housing and the URA for private and commercial transactions. Hong Kong's Centaline Property Index is the de facto Case-Shiller of the territory. India's listings live primarily on MagicBricks and 99acres.

The structural problem: no single vendor covers all of these well. Western real estate data platforms barely touch Asia. The Asian government feeds rarely structure data in a form Western analysts can plug into a model. Aggregators that claim global coverage are typically thin everywhere except their home market. And legacy enterprise vendors charge five- or six-figure annual contracts for what is, at the source, freely published. The pragmatic answer is per-region scrapers — small, cheap, focused tools that pull each market's authoritative source and hand you a tidy CSV. That is the playbook below.

Why Structured Real Estate Data Matters Right Now

Several research workflows are quietly being rebuilt on this kind of bottom-up listing data:

PropTech VC research. When a Series A pitch claims a $2.8B TAM for short-term rentals in Southeast Asia, the diligence team needs to back-check against actual listing counts, median rents per sqm, and absorption rates. Public scrapers give you that without a vendor procurement cycle.
REIT analysis. Listed REITs trade on net asset value (NAV) and funds from operations (FFO), but the underlying property comps that justify NAV come from listings. Independent scrapers let buy-side analysts build their own comp set rather than trust the manager's reported cap rates.
Relocation pricing. Corporate relocation desks need credible 90th-percentile rent estimates for executive housing in 20+ cities. Aggregating Zillow plus Rightmove plus Singapore HDB plus Boliga gives you a defensible policy benchmark.
Urban-planning research. City planners and academic researchers use listing data to model gentrification, affordability gaps, and the price-to-income ratio shift over five-year windows.
Mortgage and lending models. Lenders running loan-to-value (LTV) and default models need fresh comparable sales by ZIP/postcode. Listing data is a leading indicator before recorded deed data closes.
Regional property arbitrage. Family offices comparing Lisbon vs. Athens vs. Kuala Lumpur for yield need consistent price-per-sqft (sqm) numbers across markets. Scrapers normalize the inputs.

What the Actors Extract: Source Coverage at a Glance

Here is a quick coverage map of the actors we will use in the rest of this guide. All linked actors below are public on Apify; the affiliate parameter helps support this site.

Source	Region	Coverage	Key Fields	Update Frequency
Zillow Scraper	US	Residential for-sale & for-rent	Zestimate, list price, beds/baths, lot size, days on market, price history	Daily-fresh listings
Redfin Real Estate Scraper	US	Residential MLS-grade comps	Sold price, $/sqft, school score, hotness, last-sold date	Near real-time on MLS push
Apartments.com Scraper	US	Multi-family & SFR rental	Asking rent, sqft, amenities, availability, concessions	Daily
Rightmove UK Scraper	UK	Residential sales & lettings	Asking price, EPC rating, tenure (freehold/leasehold), postcode	Daily
Singapore HDB Resale Tracker	Singapore	Public housing resale transactions	Block, flat type, floor area sqm, resale price, lease commence date	Monthly (gov publish cycle)
Singapore URA Private (catalog)	Singapore	Private residential transactions	Project, district, tenure, $/psf, sale date	Weekly
Singapore URA Commercial (catalog)	Singapore	Office & retail transactions	Building, $/psf, transaction date, use class	Quarterly
Hong Kong Centaline Index (catalog)	Hong Kong	Residential index & transactions	CCL index, district, sqft, $/sqft, transaction direction	Weekly index, daily txn
India MagicBricks (catalog)	India	Residential sales & rent listings	Locality, BHK, super built-up area, asking price, possession status	Daily
Boliga Denmark Real Estate	Denmark	Residential listings & sales history	Address, kvm (m²), asking price, days on market, price changes	Daily

A note on the "catalog" links: a handful of Asian-specific actors are either in private beta or being rebuilt. The catalog links above route you to the current public Apify actors that cover those data sources — substitute the equivalent vendor as needed.

Example Workflow: Building a Singapore PropTech Competitor Brief

Let us run through a concrete brief — the kind of deliverable a Series B PropTech founder or a regional REIT analyst might commission. Goal: a one-page market brief on Singapore residential plus a Hong Kong comparison, suitable for an investment committee memo.

Pull HDB resale transactions (public housing). Run the Singapore HDB Resale Tracker for the last 12 months across all towns. Output: ~25,000 rows with block, flat type, floor area sqm, lease balance, and resale price. Compute median $/sqm by town and 12-month price CAGR.
Layer URA private residential transactions. Run the URA private transactions actor (or catalog equivalent) for the same period. This gives you the private condo side: project, district, tenure (freehold vs 99-year leasehold), $/psf. Critical for any thesis touching the private market, which trades at a 2–4× multiple to HDB.
Add URA commercial. Use the URA commercial transactions actor for Grade-A office and retail. This unlocks cap-rate analysis — divide net operating income estimates by sale price to ballpark commercial yields by district.
Cross-reference Hong Kong with Centaline. Run the Centaline actor for the Centaline City Leading (CCL) index plus district-level transactions. This lets you anchor Singapore numbers against HK's larger but more volatile market — useful for "Asia luxury residential" pitches.
Normalize and export. Pipe each actor's dataset to CSV (Apify supports this natively), or push directly to BigQuery / Snowflake via the Apify integrations. Normalize sqft vs sqm (1 sqm = 10.764 sqft) and convert prices to USD using a snapshot FX rate so cross-market comps are apples-to-apples.
Build the BI dashboard. A simple Metabase or Looker Studio dashboard with three views: median $/sqm by district, 12-month CAGR by segment, and rental yield estimates (asking rent ÷ asking price × 12). That is your competitor brief.

Total cost on Apify pay-per-event pricing for one full refresh: typically under $20 for all four markets. Compare with a $35K/year regional aggregator subscription and the build-vs.-buy math is uncomfortable for the incumbents.

Use Cases: Who Actually Uses This Data

REIT research desks build independent NAV models with first-party comp sets rather than trusting manager-reported cap rates.
Market-entry consultants compile city-by-city affordability and yield reports for institutional clients evaluating new geographies.
Buy-side comp pulls for private real estate funds doing diligence on a portfolio acquisition — independent verification of seller-supplied comparables.
Real-estate journalism uses the same data to fact-check developer claims about "record-breaking" prices and to chart 5-year affordability shifts.
Mortgage pricing teams calibrate LTV cutoffs and default-probability models with fresh, granular listing data by postcode.
Relocation consulting firms produce defensible executive housing benchmarks for global mobility programs.
PropTech competitor intel — track which markets a rival listings platform is adding inventory in, what their median price is, and which features (virtual tours, EPC ratings) they are standardizing on.
Urban-planning research models gentrification, displacement risk, and the housing affordability gap with primary-source data instead of decennial census snapshots.
Mortgage default research at academic and policy institutions backtests stress scenarios against actual listing-derived price trajectories.
Family-office property arbitrage compares yield, tax treatment, and currency risk across 8–10 candidate markets on a quarterly cadence.

Run It Yourself: Start with the Redfin Real Estate Scraper

If you want a single starting point that demonstrates the workflow end-to-end — fresh comps, $/sqft, ARV (after-repair value) calculations, neighborhood-level filters — the Redfin Real Estate Scraper on Apify is the cleanest first run. Drop in a city or ZIP, pick sold or for-sale, hit run, and have a CSV of MLS-grade comps in under five minutes. Use it to validate the workflow on a market you know cold, then layer in the regional actors above for the geographies you do not.

Run the Redfin Real Estate Scraper on Apify

Frequently Asked Questions

Is real estate listing data public?

Listings displayed on public-facing real estate portals are generally accessible to anyone with a browser, and government transaction registries (URA, HDB, UK Land Registry, US county recorders) explicitly publish them. Terms of service vary by site, so review each portal's terms and applicable local law before bulk collection, and prefer government sources where they exist.

Can I bulk-export results to CSV or a data warehouse?

Yes. Every actor on Apify writes its results to a dataset that exports natively to CSV, JSON, Excel, or RSS. Native integrations let you push directly to BigQuery, Snowflake, S3, Google Sheets, Airtable, or a webhook — no glue code required.

How fresh is the data?

Most listing actors re-fetch on demand, so freshness is determined by when you run them. Daily-scheduled runs are typical for active analyst workflows; government feeds (HDB, URA, Centaline index) refresh weekly or monthly, matching the source publishing cadence.

Do you cover commercial real estate?

Yes — URA commercial transactions, Centaline commercial subsets, and the commercial filters on Rightmove and Zillow cover office, retail, and industrial assets. For pure-play commercial data (CoStar-style), expect to combine these with broker-published quarterly reports to back out cap rates and absorption.

What about Asian markets beyond Singapore and Hong Kong?

India is best served via MagicBricks and 99acres scrapers; Japan typically via SUUMO or LIFULL HOME'S; Malaysia via PropertyGuru and iProperty. The general pattern: identify the dominant national portal, run a per-portal actor, and normalize fields downstream. Catalog search on Apify is the fastest way to find current public actors for any country.

Can I track price changes over time on the same listing?

Yes. The Redfin, Zillow, Rightmove, and Boliga actors all surface price-change history on a listing. Schedule a daily run and store snapshots in your warehouse to build your own price-change time series — useful for spotting motivated sellers (multiple price drops) or detecting cap-rate compression in a submarket.

How does this compare to paid vendors like CoStar or REIS?

Enterprise vendors offer richer commercial datasets, valuation models, and analyst support, and they are appropriate when those features pay for themselves. Public-data scrapers excel at coverage breadth (any country with a major portal), cost (pay-per-run instead of seat licenses), and customization. Most sophisticated teams run both: a paid vendor for their primary market and scrapers for the long tail.

Are these actors suitable for AI agents and LLM pipelines?

Yes — the Real Estate MCP Server and Redfin MCP Server wrap these actors as Model Context Protocol tools, so Claude, Cursor, or any MCP-compatible agent can query property data directly. This is increasingly how analyst desks expose internal datasets to AI assistants.

Using Apify Actors for Business Intelligence Workflows

NexGenData — Thu, 25 Jun 2026 19:05:21 +0000

BI teams that rely only on internal data — CRM, billing, product analytics — are working with half a map. Your dashboards tell you what you are doing, not what the market is doing around you. Competitor price changes, new 13F filings, fresh Y Combinator batches, Shopify app rankings, press releases — all of it is public, all of it is structured, and all of it can land in your warehouse on a nightly schedule if you wire it up correctly.

This tutorial walks through exactly that wiring. By the end you'll have a working pattern for: scheduling an Apify actor to run nightly, exporting the dataset to S3 (or GCS, or directly to Snowflake), loading it into a warehouse table, building a dashboard view on top, and firing a Slack alert when a metric moves. We'll use real BI tooling (Snowflake, BigQuery, dbt, Airflow, Google Sheets, Looker) and real actors from the NexGenData catalog. Code samples are copy-pasteable. If you maintain a competitive-intel dashboard, a lead-gen pipeline, or a market-monitoring report, you should be able to ship something useful from this post in an afternoon.

Anatomy of an Apify actor in a BI context

An actor is a serverless scraper. You pass it JSON input (URLs, search terms, filters), it runs in Apify's cloud, and it writes results to a dataset — a key-value store that exposes the rows as JSON, CSV, XLSX, JSONL, RSS, or HTML via a stable URL.

For BI purposes, three properties matter:

Output is structured. Each actor publishes a schema. Records come back as flat JSON objects, easy to COPY INTO a warehouse table or pd.read_json in a notebook.
Billing is pay-per-event (PPE). Most NexGenData actors charge $0.05–$0.50 per result, not per minute. Your cost scales with rows ingested, which makes monthly forecasting trivial: rows_per_run × runs_per_month × $/row.
Live-running beats batch downloads. You can hit the run-sync-get-dataset-items endpoint, block until the actor finishes, and stream the dataset back in one HTTP call — ideal for ad-hoc analyst queries. Or you schedule a batch run and have Apify webhook the dataset URL to your ingest layer.

Three patterns for ingesting actor data

Pattern A: Direct API call from your ETL

Simplest pattern. Your Airbyte custom source, Fivetran function, or hand-rolled Python job calls the actor and writes results to the warehouse. Good for low-frequency runs (daily/weekly) and small-to-medium volumes.


    curl -X POST \
      "https://api.apify.com/v2/acts/nexgendata~saas-pricing-tracker/run-sync-get-dataset-items?token=$APIFY_TOKEN" \
      -H "Content-Type: application/json" \
      -d '{
        "vendors": ["notion.so", "airtable.com", "monday.com"],
        "plans": ["all"]
      }' \
      > pricing_$(date +%Y-%m-%d).json

From Python, the equivalent using the official SDK:


    from apify_client import ApifyClient
    import pandas as pd

    client = ApifyClient("apify_api_xxx")
    run = client.actor("nexgendata/saas-pricing-tracker").call(
        run_input={"vendors": ["notion.so", "airtable.com"]}
    )
    items = list(client.dataset(run["defaultDatasetId"]).iterate_items())
    df = pd.DataFrame(items)
    df.to_sql("stg_competitor_pricing", warehouse_engine, if_exists="append")

Pattern B: Scheduled run + webhook to object storage

The production pattern. Apify's scheduler triggers the actor; on ACTOR.RUN.SUCCEEDED a webhook posts to your endpoint with the dataset URL. Your endpoint (Lambda, Cloud Function, or a small Flask container) streams the dataset to S3/GCS, and Snowpipe/BigQuery auto-ingest picks it up.

A webhook payload looks like this:


    {
      "userId": "abc123",
      "createdAt": "2026-05-24T02:00:00.000Z",
      "eventType": "ACTOR.RUN.SUCCEEDED",
      "eventData": {
        "actorId": "nexgendata~serp-rank-tracker-lite",
        "actorRunId": "xYzRunId789",
        "actorTaskId": "nightlyRankCheck"
      },
      "resource": {
        "id": "xYzRunId789",
        "status": "SUCCEEDED",
        "defaultDatasetId": "dsId456",
        "stats": { "computeUnits": 0.02, "datasetItemCount": 240 }
      }
    }

Your receiver fetches https://api.apify.com/v2/datasets/dsId456/items?format=json&clean;=true and writes the body to s3://bi-raw/apify/serp/2026-05-24.json. A Snowpipe definition pointed at that prefix loads it automatically.

Pattern C: Google Sheets via IMPORTDATA

For analysts who live in Sheets, you don't need a warehouse at all. Apify exposes dataset items as a CSV URL that IMPORTDATA can consume. Drop this in cell A1:


    =IMPORTDATA("https://api.apify.com/v2/acts/nexgendata~yc-companies-directory-scraper/run-sync-get-dataset-items?token=APIFY_TOKEN&format;=csv&clean;=true")

Sheets refreshes the formula on edit and on a one-hour interval. Wrap it in QUERY() to filter, or feed it into a Looker Studio data source for a free dashboard layer.

Worked example: a daily competitor pricing BI dashboard

Goal: a Snowflake-backed Looker dashboard that, every morning at 7am, shows yesterday's pricing changes across your top 10 SaaS competitors, their SERP rank movement on five priority keywords, and any new music-gear pricing trends if you happen to sell music gear. We'll use three actors:

Step 1: schedule actor runs nightly

In the Apify Console, go to Schedules -> Create new. Set cron to 0 2 * * * (2am UTC). Add three "actions", one per actor, each pointing at a saved task with its input pre-filled. Save. You're done with scheduling.

Step 2: configure dataset export via webhook

On each actor task, open Integrations -> Webhooks. Add a webhook for ACTOR.RUN.SUCCEEDED pointing at your ingest endpoint:


    https://ingest.yourcompany.com/apify?source={{eventData.actorId}}&run;={{eventData.actorRunId}}

A minimal Python receiver (FastAPI on Cloud Run):


    @app.post("/apify")
    async def ingest(source: str, run: str, payload: dict):
        dataset_id = payload["resource"]["defaultDatasetId"]
        url = f"https://api.apify.com/v2/datasets/{dataset_id}/items?format=jsonl&clean;=true"
        data = httpx.get(url).content
        key = f"apify/{source.replace('~','/')}/{date.today()}.jsonl"
        s3.put_object(Bucket="bi-raw", Key=key, Body=data)
        return {"status": "ok", "rows": payload["resource"]["stats"]["datasetItemCount"]}

Step 3: load to data warehouse

Snowflake DDL for the raw and curated layers:


    -- Raw landing table, schemaless
    CREATE TABLE raw.apify_pricing (
      load_ts TIMESTAMP_NTZ DEFAULT CURRENT_TIMESTAMP,
      source_file STRING,
      payload VARIANT
    );

    -- Snowpipe pointed at s3://bi-raw/apify/nexgendata/saas-pricing-tracker/
    CREATE PIPE raw.pipe_apify_pricing AS
    COPY INTO raw.apify_pricing (source_file, payload)
    FROM (SELECT METADATA$FILENAME, $1 FROM @bi_raw_stage/apify/nexgendata/saas-pricing-tracker/)
    FILE_FORMAT = (TYPE = JSON);

    -- Curated view, one row per vendor-plan-day
    CREATE OR REPLACE VIEW analytics.competitor_pricing AS
    SELECT
      payload:vendor::STRING       AS vendor,
      payload:plan_name::STRING    AS plan_name,
      payload:price_usd::FLOAT     AS price_usd,
      payload:billing_period::STRING AS billing_period,
      payload:scraped_at::DATE     AS observation_date
    FROM raw.apify_pricing;

BigQuery equivalent (for teams on GCP): create an external table on the GCS prefix, then a scheduled query that materializes the curated view nightly at 3am.

Step 4: build the dashboard view

The query the dashboard sits on top of — day-over-day price deltas plus a flag for movers:


    WITH ranked AS (
      SELECT
        vendor, plan_name, price_usd, observation_date,
        LAG(price_usd) OVER (PARTITION BY vendor, plan_name ORDER BY observation_date) AS prev_price
      FROM analytics.competitor_pricing
      WHERE observation_date >= CURRENT_DATE - 30
    )
    SELECT
      vendor, plan_name, observation_date,
      prev_price, price_usd,
      price_usd - prev_price AS delta_usd,
      ROUND((price_usd - prev_price) / NULLIF(prev_price, 0) * 100, 2) AS delta_pct,
      CASE WHEN ABS(price_usd - prev_price) > 0 THEN TRUE ELSE FALSE END AS price_changed
    FROM ranked
    WHERE prev_price IS NOT NULL
    ORDER BY observation_date DESC, ABS(delta_pct) DESC;

Point Looker, Mode, Hex, Tableau, or Power BI at that view. A "Price Movers" tile filtering on price_changed = TRUE AND observation_date = CURRENT_DATE - 1 gives you the morning briefing.

Step 5: alert on changes via Slack

A 50-line cron job converts the same query into a Slack ping:


    import snowflake.connector, requests, os
    conn = snowflake.connector.connect(...)
    rows = conn.cursor().execute("""
      SELECT vendor, plan_name, prev_price, price_usd, delta_pct
      FROM analytics.competitor_pricing_changes
      WHERE observation_date = CURRENT_DATE - 1 AND ABS(delta_pct) >= 5
    """).fetchall()

    if rows:
        text = "*Competitor price moves (>=5%) overnight:*\n" + "\n".join(
            f"• {v} {p}: ${pp} -> ${cp} ({d:+.1f}%)" for v, p, pp, cp, d in rows
        )
        requests.post(os.environ["SLACK_WEBHOOK"], json={"text": text})

Schedule it with Airflow, Prefect, GitHub Actions, or a plain crontab. You now have a closed loop: scrape -> load -> transform -> alert.

Cost calibration

NexGenData actors price between $0.05 and $0.50 per result. A realistic monthly bill for the pipeline above:

saas-pricing-tracker: 10 vendors × 4 plans each = 40 rows/night × 30 nights = 1,200 rows/month × $0.10 = $120/mo
serp-rank-tracker-lite: 5 keywords × 10 SERP positions = 50 rows/night × 30 = 1,500 × $0.05 = $75/mo
reverb-musical-instrument-scraper: 500 listings/night × 30 × $0.05 = $750/mo (only worth it if music gear is core to your business; otherwise drop it)

So the lean pricing-and-SERP pipeline lands around $200/month — cheaper than a single seat of most competitive-intel SaaS tools, and you own the data. If you sign up for Apify through the NexGenData referral link 30% of your usage flows back as affiliate credit, which effectively rebates the cost.

Rule of thumb for estimating before you commit: prototype with a 1-day run, multiply datasetItemCount by 30, multiply by the actor's per-result price (visible on the actor page), and that's your monthly floor.

Best actors for BI use cases

Use case	Actor	What it returns	Typical run cost
Market intel	pr-newswire-press-releases-scraper	Press release feed by company/topic	$0.05–$0.10/release
eastmoney-china-stock-screener	A-share screener data, fundamentals	$0.05/ticker
sec-form-13f-tracker-pro	Institutional holdings from 13F filings	$0.10/position
Competitor intel	saas-pricing-tracker	Vendor pricing tiers, plan features	$0.10/plan
serp-rank-tracker-lite	Keyword SERP positions	$0.05/result
shopify-app-store-scraper	App listings, ratings, install counts	$0.05/app
Lead-gen feeds	contact-info-scraper	Emails, phones, social handles from any URL	$0.05/contact
yc-companies-directory-scraper	YC batch directory: companies, founders, status	$0.05/company

For deeper dives on specific verticals, see our sister posts on YC data for VC workflows, Eastmoney for China equity research, and sanctions/compliance scraping. Browse the full category collections at /market-intelligence-tools/, /lead-generation-data-tools/, and /financial-data-tools/, or hit the full resources hub.

Limitations and when NOT to use actors

Actors are powerful, but they're not always the right tool:

If a vendor has a real API, use the API. Salesforce, HubSpot, Stripe, Segment — these have first-party connectors in Fivetran/Airbyte. Don't scrape what you can query.
If a SaaS already aggregates the dataset, weigh the math. SimilarWeb, Ahrefs, Sensor Tower, Pitchbook all charge $1k–$10k/month but give you historical depth that's expensive to backfill via scraping. Use actors when you need a specific narrow slice (10 competitors, 5 keywords) at 1/10th the price, or when the SaaS doesn't cover your niche.
Respect ToS and robots.txt. Public data is fair game in most jurisdictions, but a target site's terms may prohibit automated access. LinkedIn, Glassdoor, and a few others are litigious. Check the actor's documentation for compliance notes, and never scrape gated/authenticated content you don't have rights to.
If you need sub-minute freshness, scraping isn't the architecture. Actors run on a schedule. For real-time price changes or fraud signals, you need streaming sources, not nightly batches.

FAQ

Can I run actors from a Jupyter notebook? Yes — install apify-client, call client.actor("name").call(run_input={...}), iterate the dataset with client.dataset(run["defaultDatasetId"]).iterate_items(). Results come back as Python dicts, ready for pandas.

Does Apify support webhooks? Yes. Every actor run can fire webhooks on STARTED, SUCCEEDED, FAILED, TIMED_OUT, or ABORTED. Payload includes the dataset ID so your receiver can pull results immediately.

How do I refresh nightly? Apify Console -> Schedules -> Create new, cron syntax. Or trigger from Airflow's SimpleHttpOperator against the run-sync endpoint if you want orchestration in one place.

What about long-running actors ( >1h)? Don't use run-sync. Use the async POST /acts/{id}/runs endpoint, then either poll GET /actor-runs/{id} or wait for the SUCCEEDED webhook. Apify runs can go up to 168 hours with the right memory configuration.

Can I pipe data to dbt/Airflow? Yes. Land actor results in a raw warehouse table (Pattern A or B), then dbt models the curated layer. For Airflow, a DAG with a PythonOperator that calls the Apify SDK, followed by a SnowflakeOperator that runs the dbt build, is the canonical shape.

Is the JSON format stable? Each actor publishes a JSON Schema on its detail page and versions it. NexGenData actors include a schema_version field on every row so your downstream models can branch on changes. Subscribe to actor update notifications in the console to catch breaking changes before they hit production.

Operational tips from production deployments

A few hard-won lessons from teams that have run this pattern in production for 6+ months:

Idempotency keys matter. Set ?clean=true&fields;=id,vendor,plan_name,price_usd,scraped_at on dataset pulls and dedupe in your raw layer on the actor's natural key. Webhooks can fire twice during Apify platform incidents; without dedup you'll double-count.
Land raw, model later. Resist the urge to flatten in the ingest layer. Drop the full payload as VARIANT/JSON and let dbt or your transformation layer pick fields. When the actor schema evolves (it will), you won't have to backfill — the raw rows are intact.
Tag every run with a build_id. Pass {"build": "1.2"} in run_input.metadata so you can join scraped rows back to the actor version that produced them. Useful when investigating why a metric jumped one Tuesday.
Monitor actor health, not just data freshness. Apify exposes GET /v2/users/me/usage/monthly — alert when daily compute units drift more than 30% from baseline. A silently broken actor returns zero rows, which looks like "no price changes" on the dashboard. Set a freshness SLA in dbt with dbt source freshness and page on stale data.
Backfill on first deploy. Run the actor 7–14 times with date inputs (where supported) before going live, so day-over-day deltas have a baseline.

Airflow DAG sketch

The canonical orchestration shape, for teams already running Airflow:


    from airflow import DAG
    from airflow.operators.python import PythonOperator
    from airflow.providers.snowflake.operators.snowflake import SnowflakeOperator
    from apify_client import ApifyClient
    from datetime import datetime

    def run_actor(actor_id, **ctx):
        client = ApifyClient(Variable.get("APIFY_TOKEN"))
        run = client.actor(actor_id).call(run_input={"vendors": COMPETITORS})
        ctx["ti"].xcom_push(key="dataset_id", value=run["defaultDatasetId"])

    with DAG("competitor_pricing", schedule="0 2 * * *", start_date=datetime(2026,1,1)) as dag:
        scrape = PythonOperator(task_id="scrape", python_callable=run_actor,
                                op_kwargs={"actor_id":"nexgendata/saas-pricing-tracker"})
        load = SnowflakeOperator(task_id="load", sql="CALL sp_load_apify_pricing('{{ ti.xcom_pull(key=\"dataset_id\") }}')")
        transform = SnowflakeOperator(task_id="dbt", sql="CALL run_dbt_models('competitor_pricing')")
        scrape >> load >> transform

Three tasks, one DAG, runs in under a minute end-to-end for the volumes in our worked example. Plug the same shape into Prefect, Dagster, or Mage if Airflow isn't your stack.

How to Build a Startup Lead List from YC Company Data

NexGenData — Thu, 25 Jun 2026 17:05:15 +0000

If you sell to startups, recruit engineers, or build a book of business off freshly funded teams, the Y Combinator company directory is one of the highest-signal lead sources on the public internet. Every company in YC has been vetted, funded, and pushed through a structured program — they have budget, urgency, and a habit of buying tools quickly. The catch is that the directory was built for browsing, not outbound. Pulling a clean, segmented lead list out of it is where most GTM teams stall. This post is a tactical playbook for turning YC into a working pipeline asset.

The Problem: YC's Directory Is Built for Browsing, Not Pipeline

Anyone who has tried to source startups manually from ycombinator.com/companies knows the pain. The UI lets you filter by batch, industry, region, and a handful of tags, but you cannot export. You cannot save a segment. You cannot stack a "founded in 2024, hiring engineers, based in SF, NOT in stealth" filter and walk away with a CSV. Copy-paste works for ten rows. By row fifty, you have lost an afternoon and your data is already stale because S25 just dropped.

The other failure mode is static lists. People download a one-off CSV, load it into Apollo or HubSpot, and run sequences against it for six months. The YC directory is a living dataset: companies change status (Active, Acquired, Public, Inactive), team sizes move, new batches launch twice a year. A list clean in March is full of zombies by September, and your reply rates quietly tank.

The third gap is segmentation depth. The native filters are coarse. You cannot easily ask "show me every Series A-stage YC company in fintech, US-based, 11-50 employees, with an open engineering role, excluding stealth." That is a textbook ICP query for an SDR team, and you cannot run it in the UI.

Why YC Company Data Matters for Outbound

YC companies are unusually attractive buyers for B2B sellers. They are well-capitalized — even a fresh-batch company typically closes on $500K from YC plus a SAFE round shortly after Demo Day. They move fast on tooling because the founders are usually the buyers, with no procurement gauntlet. And they have a multi-year window where the stack is still being chosen. If you land a YC company in months 0-12, you are usually in before they pick an incumbent, which means you ride the expansion curve from 5 to 50 to 500 employees.

For recruiters, the math is even better. YC companies hire aggressively in years 1-3 and pay above market for senior engineering, product, and GTM talent. A founder posting their first three engineering roles is a warm intro waiting to happen. For VCs and corp dev, the directory is a real-time map of who is shipping in any vertical. For agencies, journalists, and BD professionals, it is the cleanest public list of which startups are alive, who runs them, and what they do. The value of a YC lead decays over time — the earlier you reach a company, the higher the conversion.

What the YC Companies Directory Scraper Extracts

The YC Companies Directory Scraper on Apify pulls a structured row for every company in the directory, from batch S05 forward through the latest cohort. For each company you get roughly thirty fields:

Identity: company name, slug, YC profile URL, primary website, logo URL
Batch metadata: batch code (W25, S25, W24, etc.), batch year, top-company badge
Positioning: one-line tagline, long description, primary industry, sub-industries, regions, locations
Operational signals: current status (Active, Acquired, Public, Inactive), team size, hiring flag, careers/jobs URL
People: founder names and titles where listed on the YC profile
Geo: primary city, country, remote flag

That is enough surface area to build a tight ICP filter. Stack conditions like "batch in (W25, S25) AND status = Active AND team_size 5-50 AND hiring = true AND region = US" and you end up with a few hundred companies that are exactly your target — not a few thousand junk rows to triage.

Example Workflow: From Directory Dump to Live Sequence

Here is the concrete five-step playbook GTM teams run on this data. End-to-end it takes about an hour the first time, roughly ten minutes per refresh after that.

Step 1 — Pull the batch slice. Run the YC scraper with the batches you care about (typically the two most recent, plus the prior year for longer cycles). Export to CSV or push to Google Sheets. For most SDR teams the right slice is the last 18 months, which gives you 600-1,200 companies.

Step 2 — Apply your ICP filter. Filter by status (drop Inactive and Acquired unless those fit your persona), team size, industry, and hiring flag. A typical SDR ICP cuts the raw list by 60-80%. You are now sitting on the qualified subset.

Step 3 — Enrich for contacts. The YC profile gives you founder names but not direct emails. Pipe the qualified list through contact-info-scraper against each company website to surface published emails, phones, and socials. For deeper coverage on founder and exec emails, run lead-list-enricher to append titles, LinkedIn URLs, and verified work emails.

Step 4 — Personalize on the YC tagline. The one-liner and long description fields are gold for first-touch personalization. Use them as the merge variable: "Saw [Company] is building [one-liner]. We help YC-stage teams in [industry] solve [pain]." This is the single highest-leverage step for reply rate. The tagline already tells you what the founder cares about this quarter.

Step 5 — Push to outbound and set cadence. Upload the enriched list to Apollo, Outreach, Salesloft, or HubSpot. Build a YC-specific sequence with a three-touch cadence over ten days (email, LinkedIn connect, email). Tag by batch so you can measure reply rate by cohort — fresh batches almost always outperform older ones. Re-run the scraper monthly to catch new batches and status changes.

Use Cases Across GTM, Recruiting, and Capital

SDR / BDR outbound: batch-specific sequences for the freshest cohort each quarter, with founder-direct email as the primary channel. Highest reply rates land in months 2-6 post-batch.
Account executive territory planning: assign new YC accounts by industry or region so AEs own a clean named-account book with intent signals built in.
RevOps list hygiene: use the refreshed scrape as source of truth to auto-deactivate stale CRM records. Inactive and Acquired companies should not be in active sequences.
Recruiter pipeline sourcing: filter for hiring = true, cross-reference founder LinkedIn for warm intros. YC companies hire engineers at a multiple of the broader market rate.
VC competitive intel: map every new YC company in your thesis areas the day each batch is announced. Decide who to chase before the herd notices.
Agency lead generation: design, dev, and growth agencies use YC as a primary ICP. Filter by batch and team size to find the sweet spot where companies have budget but no in-house function yet.
Founder peer outreach: founders selling to founders use the directory to find peers for partnership, beta testing, and design-partner conversations.
Journalist and analyst sources: reporters use the scrape to surface trends and find founders to interview before they are saturated with press requests.
Corp dev and BD targeting: enterprises identify M&A; and partnership targets filtered by stage and team size.
Investor relations: VCs map portfolio peers, comps, and follow-on opportunities in the YC ecosystem.

Get the Data: Run the YC Companies Directory Scraper

The fastest way to turn this playbook into pipeline is to run the actor and pull your first batch. The scraper is pay-per-result, so you only pay for the rows you actually use, and the output drops straight into CSV, JSON, Excel, or Google Sheets. Set it on a monthly schedule and you have a self-refreshing lead source that beats any static list you can buy.

Run the YC Companies Directory Scraper on Apify->

Related Actors for Building a Complete Lead Stack

The YC scraper is the seed list. To turn it into a contactable, segmented outbound dataset, pair it with these:

Contact Info Scraper — pulls emails, phones, and social profiles from any company website.
Company Enrichment Tool — appends firmographic details (industry, size, tech stack hints, social presence) to the YC base record.
Lead List Enricher — converts company rows into person-level records with titles, LinkedIn URLs, and verified work emails.
B2B Leads Finder — discovery layer for decision-makers when YC profiles list only founders and you need VPs or directors.
Website Email Extractor — lightweight bulk email harvester for fast sweeps across YC website URLs.
Indie Hackers product trackers — complementary feed for bootstrapped founders outside the YC universe.
Founders Fund Portfolio Scraper — parallel feed for sellers targeting tier-one VC-backed startups beyond YC.
Lightspeed Portfolio Scraper — same pattern for the Lightspeed portfolio, useful for layering multiple VC feeds into one startup ICP.

For deeper workflows see our guide on exporting YC company directory data for VC sourcing, the startup funding data playbook, and our walkthrough on extracting contact information for lead-gen workflows.

FAQ

Is YC company data public?

Yes. The directory is published on ycombinator.com/companies and is publicly browsable. Scraping the same fields visible in the UI for research, sales, and recruiting is standard practice. Respect rate limits, do not republish the raw dataset, and use the data to inform outreach.

How fresh is the data?

The scraper pulls live from the directory each run. A monthly schedule catches new batches within weeks of launch and picks up status changes as companies get acquired, go public, or wind down.

Can I filter by batch?

Yes — every row includes batch code (W25, S25, etc.) and batch year. Most outbound teams slice the two most recent batches plus the prior year to focus on the high-conversion window.

What about Demo Day-only listings?

Companies that present at Demo Day but stay off the public directory will not appear in this scrape, since the actor mirrors what is published. For Demo Day-specific intel you typically need an investor login or press partnership.

Can I get founder emails directly from the YC profile?

The YC profile lists founder names but not direct emails. Pipe the company list through contact-info-scraper or use lead-list-enricher to append verified work emails.

Is this against YC's terms of service?

Scraping publicly available pages for research is generally permitted under standard web norms, and the YC directory is a public marketing asset. The actor accesses only what is rendered to any anonymous visitor. Consult your own legal counsel for your specific use case.

How does this compare to Crunchbase Pro or Apollo?

Crunchbase and Apollo carry YC tags, but their batch-level metadata often lags the official directory by weeks or months. Scraping the source gives you the canonical, freshest version of every field at a fraction of the per-record cost. Most teams use the YC scrape as the authoritative base and layer Apollo or Crunchbase for person-level enrichment.

How often should I refresh the list?

Monthly is the sweet spot for most outbound teams. Weekly is overkill unless you run a high-velocity SDR motion or VC scout program. Quarterly is too slow and silently degrades reply rate.

New: Japan BoJ Macro Rates & JGB Auctions — BoJ policy rate, JGB yield curve, MoF auction calendar & policy statements

NexGenData — Thu, 25 Jun 2026 16:18:13 +0000

What it does

This actor is a JPY source-of-truth feed for Japan rates — Bank of Japan macro rates, the JGB yield curve, the MoF JGB auction calendar, money supply, FX reserves, and BoJ Policy Board monetary policy statements. It consolidates several official Japanese sources into one structured output.

Who it's for

Japan-rates desks, macro PMs, JGB and swaps traders, and AI-agent integrations that need consolidated BoJ/MoF data without scraping multiple Japanese-language sites.

Sample fields / output

indicator / series
date / period
value
JGB tenor
yield
auction date
bid-to-cover
money supply
FX reserves
policy statement text

Example use cases

Track the JGB yield curve and BoJ policy rate together for a Japan-rates dashboard.
Pull the MoF JGB auction calendar and bid-to-cover history for supply analysis.
Feed BoJ Policy Board statements into an NLP pipeline for tone/stance scoring.

▶ Try the Japan BoJ Macro Rates & JGB Auctions actor on Apify

Related actors

FAQ

What sources does this consolidate?

Bank of Japan macro rates and policy statements plus MoF JGB auction and yield-curve data, in one structured feed.

Does it include monetary policy statements?

Yes — BoJ Policy Board monetary policy statements are included alongside the rates data.

Is auction data covered?

Yes — the MoF JGB auction calendar with details such as bid-to-cover is included.

New: Japan MoF FX Intervention Tracker — every MoF yen FX intervention back to 1991 with JPY + USD estimates

NexGenData — Thu, 25 Jun 2026 16:17:34 +0000

What it does

This actor delivers a structured feed of Japan's Ministry of Finance (MoF) yen FX intervention disclosures — every USD/JPY, EUR/JPY, and GBP/JPY operation going back to 1991, including the 2022 and 2024 yen-defense episodes. Amounts are normalized in JPY with USD estimates and bilingual (Japanese/English) descriptions.

Who it's for

FX desks, macro PMs, and options-vol quants who need a clean, historical record of MoF intervention rather than reconstructing it from press releases and monthly PDFs.

Sample fields / output

operation date
currency pair
direction (yen buy / sell)
amount (JPY, normalized)
USD estimate
reporting period
bilingual description (JP / EN)

Example use cases

Backtest USD/JPY behavior around historical MoF intervention dates.
Build an alert for new intervention disclosures during yen-defense episodes.
Quantify cumulative intervention size across the 2022 and 2024 episodes for a macro note.

▶ Try the Japan MoF FX Intervention Tracker actor on Apify

Related actors

FAQ

How far back does the intervention data go?

Back to 1991, covering every disclosed MoF operation including the 2022 and 2024 yen-defense episodes.

Are amounts available in USD?

Yes — operations are normalized in JPY with USD estimates alongside.

Which currency pairs are included?

USD/JPY, EUR/JPY, and GBP/JPY operations as disclosed by Japan's Ministry of Finance.

New: Korea BoK ECOS — Base Rate & Macro — Bank of Korea base rate, M1/M2, GDP, CPI & FX reserves as structured macro records

NexGenData — Thu, 25 Jun 2026 16:16:34 +0000

What it does

This actor wraps the Bank of Korea (BoK) ECOS Open API into clean, structured macro records — the Korea base rate, monetary aggregates (M1/M2), GDP, CPI, FX reserves, and trade balance. It turns ECOS series into consistent JSON you can feed straight into a model or dashboard.

Who it's for

KRW/USD FX desks, EM-rates PMs, Korean-equity strategists, macro researchers, and AI-agent integrations that need Bank of Korea series without hand-managing ECOS series codes and pagination.

Sample fields / output

series / statistic code
indicator name
period (date)
value
unit
frequency
base rate %
monetary aggregate (M1 / M2)
CPI / GDP / FX-reserves value

Example use cases

Track the BoK base rate alongside CPI to model the Korean rate path for an FX or rates book.
Pull M1/M2 and FX reserves into a macro dashboard for KRW positioning.
Give an AI agent grounded Bank of Korea macro data instead of stale or hallucinated figures.

▶ Try the Korea BoK ECOS — Base Rate & Macro actor on Apify

Related actors

FAQ

Which Bank of Korea series are covered?

Core macro series including the base rate, M1/M2, GDP, CPI, FX reserves, and trade balance from the ECOS Open API.

Is the data official?

Yes — it comes directly from the Bank of Korea's ECOS Open API, normalized into structured records.

Can I use this for AI agents?

Yes — the structured output is designed to ground FX/rates agents and macro models in real BoK data.

New: Korea KRX Market Stats — daily KOSPI/KOSDAQ/KONEX prices, foreign-ownership ratios & market-cap rankings

NexGenData — Thu, 25 Jun 2026 16:15:33 +0000

What it does

This actor returns daily market statistics from KRX (Korea Exchange) across KOSPI, KOSDAQ, and KONEX — settlement prices, foreign-ownership ratios, trading volumes, and market-cap rankings. Query by market or by a single ticker over a date range and get structured Korean-equity reference data back.

Who it's for

Korean-equity strategists, EM and Asia-focused PMs, quant teams modeling foreign-investor flows, and AI agents that need clean KRX reference data without parsing the exchange's Korean-language portal.

Sample fields / output

ticker code
issuer name
market (KOSPI / KOSDAQ / KONEX)
settlement price (KRW)
foreign-ownership ratio %
trading volume
market capitalization
market-cap rank
trade date

Example use cases

Track foreign-ownership ratios over time to model foreign-investor positioning in Korean names.
Build a daily KOSPI/KOSDAQ reference table for a screener or factor model.
Pull a single ticker's price and market-cap history across a date range for backtests.

▶ Try the Korea KRX Market Stats actor on Apify

Related actors

FAQ

Is this real-time market data?

No — it's daily reference/statistics data (settlement prices, ownership ratios, volumes), not a live tick feed.

Can I query a single ticker over a date range?

Yes — you can request by market or by an individual ticker across a date range.

What does the foreign-ownership ratio represent?

It's the share of a stock held by foreign investors as published in KRX statistics — useful for tracking foreign flow.

New: Sri Lanka CSE — Colombo Stock Exchange Data — live Colombo Stock Exchange prices, gainers/losers & company announcements

NexGenData — Thu, 25 Jun 2026 16:15:28 +0000

What it does

This actor pulls structured market data from the Colombo Stock Exchange (CSE) in Sri Lanka — today's share prices, the day's top gainers and losers, and company announcements. Instead of scraping the exchange site by hand or stitching together PDFs, you get clean JSON ready to drop into a model, dashboard, or index pipeline.

Who it's for

Frontier- and emerging-market fund managers, index providers, quant researchers, and fintech apps that need reliable Sri Lanka equity coverage where mainstream data vendors are thin or expensive.

Sample fields / output

ticker / symbol
company name
last price (LKR)
change & change %
trading volume
turnover
gainer / loser rank
announcement title
announcement date

Example use cases

Maintain a daily CSE price snapshot for an emerging-markets index or factor model.
Surface the day's biggest movers on the Colombo exchange for a research desk or newsletter.
Feed company announcements into an alerting workflow for Sri Lanka-listed names.

▶ Try the Sri Lanka CSE — Colombo Stock Exchange Data actor on Apify

Related actors

FAQ

Is the data real-time?

No — it provides end-of-day / reference market data suitable for research, indexing, and screening rather than live trading.

What currency are prices in?

Prices and turnover are reported in Sri Lankan rupees (LKR), as published by the Colombo Stock Exchange.

Can I get the day's top gainers and losers directly?

Yes — gainers and losers are returned with rank, price, and change %, so you don't have to compute them yourself.

Building a Research Pipeline: From Google Scholar Search to Citation Network Analysis

NexGenData — Thu, 25 Jun 2026 15:09:38 +0000

Building a Research Pipeline: From Google Scholar Search to Citation Network Analysis

If you've ever tried to stay current in a fast-moving research field, you know the problem: there's too much being published to read everything, but missing key papers means missing critical context. You end up doing what researchers have always done—manually searching Google Scholar, reading abstracts, following citation trails, and hoping you find the important work before it's obsoleted by the next breakthrough.

What if you automated that entire workflow? What if you could systematically extract papers, analyze their citation networks, identify the most influential authors and venues, and automatically classify emerging vs. established research?

That's the power of a research pipeline. Let's build one.

The Research Pipeline Architecture

A complete research system has five stages:

Stage 1: Query and Extraction
Search for papers matching your research interest. Collect metadata: title, authors, publication date, abstract, citation count.

Stage 2: Retrieval and Enrichment
Get the full citation details for each paper. Extract references cited by each paper. Build a bidirectional citation graph.

Stage 3: Classification
Categorize papers by research area, methodology, or stage of maturity (foundational vs. incremental vs. applied).

Stage 4: Network Analysis
Identify key papers (high in-degree citations), influential authors (frequently cited across papers), core venues (conferences/journals where key work is published).

Stage 5: Trend Detection
Compare recent papers vs. older papers. Which topics are accelerating? Which are becoming established? Which are declining?

Let's work through each stage with practical code.

Stage 1: Query and Extraction

The Google Scholar Scraper is your foundation. Configure it to search for your research topic and capture all papers meeting your criteria.

Sample extracted data:

{
  "papers": [
    {
      "title": "Attention Is All You Need",
      "authors": ["Ashish Vaswani", "Noam Shazeer", "Parmar N.", "Uszkoreit J."],
      "publication_year": 2017,
      "venue": "NIPS 2017",
      "abstract": "The dominant sequence transduction models are based on complex recurrent or convolutional neural networks in an encoder-decoder configuration. The best performing models also connect the encoder and decoder through an attention mechanism. We propose a new simple network architecture, the Transformer, based solely on attention mechanisms, dispensing with recurrence and convolutions entirely.",
      "citation_count": 84320,
      "pdf_url": "https://arxiv.org/pdf/1706.03762.pdf",
      "google_scholar_url": "https://scholar.google.com/scholar?q=Attention+Is+All+You+Need"
    },
    {
      "title": "Language Models are Unsupervised Multitask Learners",
      "authors": ["Tom B. Brown", "Benjamin Mann", "Nick Reiley"],
      "publication_year": 2019,
      "venue": "OpenAI Blog",
      "abstract": "Recent work has demonstrated that transfer learning can greatly improve performance on natural language tasks. We demonstrate that language models begin to learn these tasks without any explicit supervision when trained on a new dataset of Internet text...",
      "citation_count": 34290,
      "pdf_url": "https://d4mucfpksywv.cloudfront.net/better-language-models/language-models.pdf"
    },
    {
      "title": "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding",
      "authors": ["Jacob Devlin", "Ming-Wei Chang", "Kenton Lee", "Kristina Toutanova"],
      "publication_year": 2018,
      "venue": "NAACL 2019",
      "abstract": "We introduce BERT, a new method of pre-training language representations that obtains state-of-the-art results on a wide array of Natural Language Processing (NLP) tasks.",
      "citation_count": 67450,
      "pdf_url": "https://arxiv.org/pdf/1810.04805.pdf"
    }
  ],
  "metadata": {
    "search_query": "transformer language models",
    "total_results": 1247,
    "results_extracted": 3,
    "extraction_date": "2026-04-05T10:30:00Z"
  }
}

Stage 2: Citation Network Construction

Here's the critical part. For each paper, you need to extract what it cites and what cites it. This creates a citation graph where nodes are papers and edges are citation relationships.

import json
from collections import defaultdict
from datetime import datetime

class CitationNetworkBuilder:
    def __init__(self):
        self.papers = {}  # paper_id -> paper_data
        self.citations = defaultdict(list)  # paper_id -> list of cited paper_ids
        self.cited_by = defaultdict(list)  # paper_id -> list of papers citing it

    def add_paper(self, paper_data):
        """Store paper metadata"""
        paper_id = self._create_paper_id(paper_data['title'])
        self.papers[paper_id] = {
            'title': paper_data['title'],
            'authors': paper_data['authors'],
            'year': paper_data['publication_year'],
            'venue': paper_data['venue'],
            'citation_count': paper_data.get('citation_count', 0),
            'abstract': paper_data.get('abstract', '')
        }
        return paper_id

    def add_citation(self, citing_paper_id, cited_paper_id):
        """Record that paper A cites paper B"""
        self.citations[citing_paper_id].append(cited_paper_id)
        self.cited_by[cited_paper_id].append(citing_paper_id)

    def get_influential_papers(self, min_citations=5):
        """Find papers cited by many others in the network"""
        influential = []

        for paper_id, citing_papers in self.cited_by.items():
            in_degree = len(citing_papers)

            if in_degree >= min_citations:
                influential.append({
                    'paper_id': paper_id,
                    'title': self.papers[paper_id]['title'],
                    'in_degree': in_degree,
                    'year': self.papers[paper_id]['year'],
                    'cited_by': citing_papers[:5]  # First 5 citers
                })

        return sorted(influential, key=lambda x: x['in_degree'], reverse=True)

    def get_influential_authors(self, top_n=10):
        """Find authors whose work is most cited in the network"""
        author_citation_score = defaultdict(int)

        for paper_id, citing_count in self.cited_by.items():
            if paper_id in self.papers:
                authors = self.papers[paper_id]['authors']
                score = len(citing_count)  # How many papers in our network cite this

                for author in authors:
                    author_citation_score[author] += score

        top_authors = sorted(
            author_citation_score.items(),
            key=lambda x: x[1],
            reverse=True
        )[:top_n]

        return [
            {'author': name, 'network_citation_score': score}
            for name, score in top_authors
        ]

    def get_key_venues(self):
        """Identify conferences/journals where influential papers are published"""
        venue_scores = defaultdict(int)

        for paper_id, in_degree in self.cited_by.items():
            if paper_id in self.papers and in_degree:
                venue = self.papers[paper_id]['venue']
                venue_scores[venue] += in_degree

        return sorted(
            venue_scores.items(),
            key=lambda x: x[1],
            reverse=True
        )

    def classify_paper_maturity(self, paper_id):
        """Classify paper as foundational, core, or emerging"""
        paper = self.papers[paper_id]
        in_degree = len(self.cited_by[paper_id])
        years_published = datetime.now().year - paper['year']

        # Heuristic: old papers with high citations are foundational
        # New papers with any citations are emerging
        # Middle ground are core

        if years_published >= 5 and in_degree >= 10:
            return 'Foundational'
        elif years_published <= 2 and in_degree >= 1:
            return 'Emerging'
        elif in_degree >= 3:
            return 'Core'
        else:
            return 'Peripheral'

    def _create_paper_id(self, title):
        """Create deterministic ID from title"""
        return title.lower().replace(' ', '_')[:50]

# Usage example
network = CitationNetworkBuilder()

# Add papers from your Google Scholar export
papers_data = json.load(open('google_scholar_papers.json'))

for paper in papers_data['papers']:
    paper_id = network.add_paper(paper)

# Simulate citation relationships (in reality, you'd extract these from papers)
# This would come from parsing paper PDFs or Google Scholar citation links
network.add_citation('attention_is_all_you_need', 'bert_pre_training_of_deep_bidirectional')
network.add_citation('language_models_are_unsupervised', 'attention_is_all_you_need')
network.add_citation('bert_pre_training_of_deep_bidirectional', 'attention_is_all_you_need')

# Analyze the network
influential = network.get_influential_papers(min_citations=1)
top_authors = network.get_influential_authors()
key_venues = network.get_key_venues()

print("Most Influential Papers in Your Research Area:")
for paper in influential[:5]:
    print(f"  {paper['title']} ({paper['year']}) - cited {paper['in_degree']} times")

print("\nMost Influential Authors:")
for author, score in top_authors[:5]:
    print(f"  {author} - score: {score}")

print("\nKey Venues:")
for venue, score in key_venues[:5]:
    print(f"  {venue} - score: {score}")

Stage 3: Paper Classification

Not all papers are equally important for your understanding. Some are seminal foundational work. Some are recent applications. Some are incremental extensions.

Classify papers automatically based on multiple signals:

def classify_paper(paper_data, network_context):
    """Multi-factor paper classification"""
    title = paper_data['title'].lower()
    abstract = paper_data.get('abstract', '').lower()
    year = paper_data['publication_year']
    citations = paper_data.get('citation_count', 0)

    # Topic classification
    if any(term in title + abstract for term in ['survey', 'review', 'overview']):
        topic_type = 'Survey'
    elif any(term in title + abstract for term in ['benchmark', 'dataset', 'corpus']):
        topic_type = 'Resource'
    elif any(term in title + abstract for term in ['application', 'implementation', 'case study']):
        topic_type = 'Application'
    elif any(term in title + abstract for term in ['novel', 'new', 'method', 'algorithm']):
        topic_type = 'Method'
    else:
        topic_type = 'General'

    # Maturity classification
    years_old = datetime.now().year - year
    if years_old >= 5:
        maturity = 'Established'
    elif years_old <= 1:
        maturity = 'Emerging'
    else:
        maturity = 'Established-Recent'

    # Impact classification (based on citations)
    if citations >= 1000:
        impact = 'Landmark'
    elif citations >= 100:
        impact = 'High-Impact'
    elif citations >= 20:
        impact = 'Moderate-Impact'
    else:
        impact = 'Low-Impact'

    return {
        'topic_type': topic_type,
        'maturity': maturity,
        'impact': impact,
        'citations': citations,
        'year': year
    }

# Classify your papers
papers = json.load(open('google_scholar_papers.json'))['papers']
classifications = []

for paper in papers:
    classification = classify_paper(paper, network)
    classifications.append({
        'title': paper['title'],
        'classification': classification
    })

# Analyze distribution
from collections import Counter
maturity_dist = Counter([c['classification']['maturity'] for c in classifications])
print("Paper Maturity Distribution:", dict(maturity_dist))

impact_dist = Counter([c['classification']['impact'] for c in classifications])
print("Paper Impact Distribution:", dict(impact_dist))

Stage 4 & 5: Trend Analysis

Now the real intelligence emerges. Analyze trends across your research area:

def analyze_trends(papers):
    """Identify what's accelerating, what's established, what's declining"""
    by_year = defaultdict(lambda: {'count': 0, 'avg_citations': 0, 'papers': []})

    for paper in papers:
        year = paper['publication_year']
        by_year[year]['count'] += 1
        by_year[year]['avg_citations'] += paper.get('citation_count', 0)
        by_year[year]['papers'].append(paper['title'])

    # Calculate averages
    for year in by_year:
        count = by_year[year]['count']
        if count > 0:
            by_year[year]['avg_citations'] = int(by_year[year]['avg_citations'] / count)

    # Sort by year
    sorted_years = sorted(by_year.items())

    # Calculate growth rates
    trends = []
    for i in range(1, len(sorted_years)):
        prev_year, prev_data = sorted_years[i-1]
        curr_year, curr_data = sorted_years[i]

        growth = ((curr_data['count'] - prev_data['count']) / prev_data['count'] * 100
                  if prev_data['count'] > 0 else 0)

        trends.append({
            'year': curr_year,
            'papers_published': curr_data['count'],
            'yoy_growth': f"{growth:+.1f}%",
            'avg_citations_per_paper': curr_data['avg_citations']
        })

    return trends

trends = analyze_trends(papers)
print("Research Trend Analysis:")
for trend in trends[-5:]:  # Last 5 years
    print(f"  {trend['year']}: {trend['papers_published']} papers " +
          f"(YoY growth: {trend['yoy_growth']}) " +
          f"avg citations: {trend['avg_citations_per_paper']}")

# Identify emerging topics
recent_papers = [p for p in papers if p['publication_year'] >= datetime.now().year - 2]
recent_keywords = extract_keywords(recent_papers)
older_keywords = extract_keywords([p for p in papers if p['publication_year'] < datetime.now().year - 5])

emerging = set(recent_keywords) - set(older_keywords)
declining = set(older_keywords) - set(recent_keywords)

print("\nEmerging Topics:", emerging)
print("Declining Topics:", declining)

Putting It All Together

Here's your complete workflow:

Week 1: Set up the Google Scholar Scraper to monitor your research area

Configure search queries for your field
Extract all papers matching your criteria
Store results as JSON

Week 2-3: Build the citation network

For each paper, extract references (manual parsing or use citation APIs)
Build the citation graph
Identify influential papers and authors

Week 4: Classify and analyze

Classify papers by type, maturity, impact
Analyze trends
Identify emerging topics

Ongoing: Run weekly or monthly

Re-run the scraper for new papers
Update citation counts
Track trend changes

Practical Use Cases

Once your pipeline is built, you can:

Stay Ahead of Your Field

Set alerts for when a new influential paper is published
Know when your domain shifts before competitors do
Identify which authors to follow for emerging trends

Inform Product Development

Product team wants to know what's technically feasible? Check if there's recent published work.
Are you solving a solved problem? The citation network tells you.
What's actually novel in your approach? Compare against foundational and recent work.

Build Competitive Intelligence

Which research venues are competitors focused on?
What problems is the academic community solving that might become products?
Which authors are most influential in your space?

Research Direction

Where are the gaps in published work? (Low citation count despite relevance)
Which methods are becoming standard vs. exploratory?
What adjacent fields should you be monitoring?

Getting Started

Use the Google Scholar Scraper to extract papers systematically. Set it up to monitor your research area continuously. Then build the analysis layers on top—citation networks, classifications, trend analysis.

The advantage of automating this process is that you see patterns humans miss. After running this pipeline for 3-6 months, you'll have better market intelligence about your research domain than most researchers who are doing this manually.

Start with your core research area. Run the pipeline. Let the data patterns guide your next steps.

Scraping China A-Share Stock Data from Eastmoney

NexGenData — Thu, 25 Jun 2026 15:04:54 +0000

If you cover Chinese equities for a living, you already know the data problem. Bloomberg's mainland coverage is exhaustive but priced for the buy-side. Refinitiv and FactSet are similar. Wind and Choice are excellent if you can read Chinese, negotiate a domestic license, and route payment onshore. And the free Western feeds -- Yahoo Finance, Google Finance, Alpha Vantage -- have spotty Shenzhen coverage and almost no Beijing Stock Exchange tickers at all. For a quant team trying to build a multi-factor model across the full A-share universe, that gap is a real research blocker.

This post walks through the Eastmoney China A-Shares Screener actor on Apify -- what it pulls, how to wire it into a screening workflow, and how to combine it with adjacent China and Hong Kong data sources to build a coherent EM research stack without paying terminal-class fees.

1. The problem: China A-share data coverage is broken outside China

The structural issue is simple. Eastmoney (东方财富, eastmoney.com) is the de facto retail and semi-professional data portal for Chinese equities. It aggregates exchange data from the Shanghai Stock Exchange (SSE), Shenzhen Stock Exchange (SZSE), and the Beijing Stock Exchange (BSE), plus derived analytics, sector classifications, northbound flow snapshots, and consensus estimates. If you ask a Chinese sell-side analyst where they check a quote intraday, the answer is usually Eastmoney or Tonghuashun.

The catch: Eastmoney's UI is Chinese-only, there is no documented public REST API, and the underlying JSON endpoints rotate, paginate inconsistently, and gzip-encode with non-standard headers. Western vendors either skip the full A-share universe or charge enterprise prices. Bloomberg with mainland entitlements runs $24,000+ per seat-year. Free vendors like Yahoo Finance and Alpha Vantage carry the SSE Composite and a curated slice of large caps but miss the Shenzhen Main Board long tail, almost all of ChiNext, most of the STAR Market, and effectively none of the Beijing Stock Exchange.

MSCI's progressive A-share inclusion has dragged passive flow into mainland names since 2018, but coverage of the long tail -- Shenzhen mid-caps, ChiNext growth names, STAR semis, and the smaller BSE listings -- remains expensive or unavailable through Western channels. Scraping Eastmoney directly is the pragmatic workaround, and an Apify actor is the cleanest way to do it without maintaining your own headless-browser pipeline.

2. Why this data matters for global allocators

China's A-share market is the world's second-largest equity market by capitalization, behind only the United States. Combined market cap across SSE, SZSE, and BSE sits in the USD 10-12 trillion range depending on the day, with roughly 5,300 listed companies as of mid-2026. For comparison, that is more than four times the listed universe of Hong Kong and roughly twice the listed count of Japan.

Several investor archetypes need clean A-share data:

EM and global allocators rebalancing against MSCI EM, MSCI ACWI, and FTSE Global All Cap need security-master and fundamental data for every A-share constituent. MSCI's A-share inclusion factor sits at 20% for large caps; further upweights will mechanically move billions of passive flow.
Stock Connect strategists running northbound and southbound flows need eligibility flags and daily quota awareness. Northbound daily quota is RMB 52 billion per channel; southbound RMB 42 billion. Quota utilization is a tracked positioning signal.
QFII and RQFII funds holding A-shares directly need fundamentals normalized against international accounting concepts (IFRS / US GAAP) rather than Chinese Accounting Standards.
China-focused hedge funds running long-short books need the full universe, not the curated index slice. Alpha in China concentrates outside the largest 300 names where sell-side coverage is thinnest.
Quant researchers backtesting factor strategies (value, quality, momentum, low-vol) need historical fundamentals across the full listed history, including delisted tickers to avoid survivorship bias.

In every case, missing the bottom half of the cap distribution -- where alpha is most likely to live -- is a non-starter.

3. What the Eastmoney actor extracts

The Eastmoney China A-Shares Screener returns a normalized JSON record per ticker with both Chinese and English-language fields. Coverage spans all three mainland exchanges (Shanghai, Shenzhen including Main Board and ChiNext, and Beijing) plus the STAR Market segment of Shanghai. The current field set includes:

ticker -- six-digit code (e.g. 600519 Kweichow Moutai, 000858 Wuliangye, 300750 CATL, 688981 SMIC)
name_cn / name_en -- Chinese and best-effort English company name
exchange -- SH (Shanghai), SZ (Shenzhen), BJ (Beijing)
board -- Main, ChiNext (创业板), STAR (科创板), BSE (北交所)
sector / industry -- CSRC top-level sector and Eastmoney sub-industry classification
market_cap_rmb -- total market cap in CNY
free_float_market_cap_rmb -- float-adjusted cap, relevant for index-weighted exposure
pe_ttm, pb, ps_ttm -- valuation ratios
dividend_yield -- trailing twelve-month dividend yield
roe_ttm, net_margin_ttm -- quality factors
revenue_growth_yoy, eps_growth_yoy -- growth factors
eps_ttm, bvps -- per-share metrics
price, change_pct, volume, turnover_rmb -- live quote fields
week_52_high, week_52_low -- trailing range
stock_connect_eligible -- boolean for northbound (HK -> mainland) eligibility

A sample output record looks like this:


    {
      "ticker": "600519",
      "name_cn": "贵州茅台",
      "name_en": "Kweichow Moutai",
      "exchange": "SH",
      "board": "Main",
      "sector": "Consumer Staples",
      "industry": "Baijiu",
      "market_cap_rmb": 1932000000000,
      "free_float_market_cap_rmb": 1816000000000,
      "pe_ttm": 22.4,
      "pb": 7.8,
      "ps_ttm": 11.2,
      "dividend_yield": 0.0341,
      "roe_ttm": 0.342,
      "net_margin_ttm": 0.521,
      "revenue_growth_yoy": 0.151,
      "eps_growth_yoy": 0.139,
      "eps_ttm": 68.42,
      "bvps": 197.30,
      "price": 1538.20,
      "change_pct": -0.0042,
      "volume": 1820000,
      "turnover_rmb": 2802000000,
      "week_52_high": 1812.50,
      "week_52_low": 1428.10,
      "stock_connect_eligible": true
    }

Pricing is being repriced to $0.10 per stock record effective May 26, 2026, under Apify's Pay-Per-Event model. A full A-share universe pull of roughly 5,300 names lands at about $530 -- a small fraction of any single seat of a Western terminal that covers the same universe, and orders of magnitude below the multi-year contracts demanded by Wind or Choice for offshore institutions.

4. Example workflow: a deep-value screen across the A-share universe

A canonical use case for this dataset is a classical Graham-style value screen. Most Western screeners can't even express this query against the full A-share universe because half the tickers are missing.

The recipe:

Run the Eastmoney actor with no ticker filter to pull the full universe.
Filter for pe_ttm < 15, pb < 1.5, and dividend_yield > 0.03.
Drop names with negative trailing EPS, negative free float, or market cap below RMB 5 billion (basic liquidity screen).
Tag each survivor with its board (Main / ChiNext / STAR / BSE) using the board field, and segment results by board to see where value is concentrated.
Optionally enrich with the China ETF Flow Tracker to see whether sector ETFs covering survivors are seeing accumulation or redemption -- a useful confirmation signal for sector-level value rotation.
Cross-reference survivors against the China A-Share Insider Trades actor to flag names where the executive cohort is buying (高管增持). Insider buys are a high-information signal in the mainland market because mandatory disclosure thresholds are tighter than in many comparable jurisdictions.
For each survivor with a Hong Kong dual listing, join the H-share equivalent and compute the AH Premium spread -- useful both as a relative-value entry signal and as a hedge mechanism.
Export the joined dataset to CSV and load into your portfolio tooling -- Portfolio Visualizer, a custom risk model, or a Jupyter notebook running pandas, numpy, and statsmodels.

A typical screen against May 2026 data returns 180-220 names. Banks, steel, coal, highways, and selected consumer staples dominate. ChiNext and STAR names are largely absent -- growth-oriented boards rarely trade below book. Invert the screen -- revenue_growth_yoy > 0.25, roe_ttm > 0.15, pe_ttm < 30, restricted to ChiNext and STAR -- and you get a very different list: semi equipment, EV supply chain, biotech, industrial robotics. Both screens take a single actor run plus a few lines of pandas.

5. Use cases across the EM research stack

China A-share quant research: build value, quality, momentum, and low-vol factor portfolios against the full mainland universe with monthly rebalances and CSRC sector neutralization.
EM portfolio rebalancing: reconcile your benchmark weights when MSCI updates A-share inclusion factors or when index providers reclassify names between the Main, ChiNext, and STAR boards.
Stock Connect through-channel analysis: filter on stock_connect_eligible to isolate the Hong Kong-accessible subset, then overlay HKEX disclosed northbound positioning to track foreign flow concentration by name.
Sector rotation backtests: group by CSRC sector and build long-only or long-short sector momentum strategies; the industry field gives finer granularity for sub-sector trades like baijiu within consumer staples or rare-earth processors within materials.
QFII and RQFII strategy backtests: foreign institutional investors operating under the QFII/RQFII regime need fundamental data normalized for international comparison, particularly around accounting differences between CAS and IFRS.
China-focused factor models: construct Fama-French-style factor portfolios (SMB, HML, RMW, CMA) calibrated to A-share data rather than re-using developed-market loadings, which empirically misprice the value premium in China.
Cross-listing arbitrage with H-shares: pair the A-share series with the corresponding H-share Hong Kong listing and trade the Hang Seng AH Premium spread mechanically, with the A-share fundamentals as your anchor.
Sentiment overlays: combine fundamentals with retail sentiment from the China Trends Tracker (Weibo, Baidu, Douyin) to detect retail-driven momentum on small-cap names before it shows up in price.
Insider-flow event studies: use the insider-trades actor to test whether executive buying predicts forward returns on A-share names, conditioning on board and market-cap quintile.
Investigative journalism: reporters covering Chinese listed companies -- particularly around accounting concerns, related-party transactions, or sanctioned entities -- need a quick way to pull fundamentals and ownership color without a Bloomberg seat.

6. Run it on Apify

The fastest way to start is to run the actor directly against the Shanghai or Shenzhen ticker space and inspect the output before wiring it into a pipeline. The interface is straightforward: paste a ticker list (or leave blank to pull the full universe), pick the fields you want, hit run.

Run the Eastmoney China A-Shares Screener on Apify->

The actor runs on Apify's standard infrastructure -- no proxy setup, no headless-browser maintenance, no Chinese-language UI to navigate. Pull a single ticker for a sanity check, then scale to the full universe. Results land in Apify's dataset format and can export to JSON, CSV, Excel, S3, Google Sheets, BigQuery, Snowflake, or a webhook. Schedule end-of-day for daily snapshots and you have a self-maintaining A-share data archive.

For broader context see our Chinese-language deep dive 免费抓取中国A股数据, the Best Free Stock Market APIs guide, and the FX Dashboard with Apify and Google Sheets.

7. Related actors for an integrated China and Asia stack

The Eastmoney actor is the foundation, but a serious China research workflow usually combines several feeds. The following are all publicly available on Apify under the same publisher and stitch together cleanly via shared ticker keys (mainland six-digit codes for A-shares, four-digit codes for HKEX, exchange-specific codes for KRX and NSE).

China ETF Flow Tracker (东方财富ETF资金流向) -- daily ETF subscription and redemption flows. Useful as a sector-level demand signal that can lead price by hours to days, particularly for thematic ETFs covering semis, EV, and biotech.
China A-Share Insider Trades (高管增减持) -- executive and major-shareholder buying and selling disclosed to CSRC. The canonical insider-flow dataset for the mainland market.
HKEX Insider Trades and Short Interest Tracker -- pairs naturally with the A-share insider feed when you are trading the A/H cross-listing spread or hedging A-share exposure via HKEX shorts.
HKEX IPO Calendar -- Hong Kong new listings, including secondary listings of mainland names returning home via H-share dual listings and the increasing flow of US-listed Chinese ADRs converting to Hong Kong primary listings.
KOSPI Stock Screener -- Korea is the obvious adjacency for a North Asia equity strategy; structure is similar to the Eastmoney actor and Korean semis are often correlated with Chinese supply chain names.
NSE India Stock Indices Screener -- the other big EM Asia market; useful for cross-region factor work and for any allocator running a barbelled EM Asia book.

8. FAQ

How fresh is the data?

Eastmoney refreshes intraday quotes in near real time during mainland trading hours (09:30-11:30 and 13:00-15:00 CST). The actor pulls live values at run time. Fundamentals refresh daily for ratios and quarterly for underlying statements per CSRC deadlines.

Does this include Shenzhen ChiNext and the Beijing Stock Exchange?

Yes. The actor covers all three mainland exchanges: Shanghai (SH), Shenzhen Main Board and ChiNext (SZ), the STAR Market on Shanghai, and the Beijing Stock Exchange (BJ). BSE coverage is the main differentiator versus most Western vendors, which historically skipped BSE entirely. BSE launched in 2021 for innovation-oriented SMEs and remains effectively invisible to most non-Chinese feeds.

Can I get historical price data?

The current actor focuses on snapshot screener fields. For historical OHLCV, schedule the actor end-of-day and build a time series in object storage or your warehouse. A dedicated historical-price actor with adjusted prices and corporate-action handling is on the roadmap.

Are foreign-language names included?

Yes. Each record includes name_cn (official Chinese name) and name_en (best-effort English name). English names are not always present for the smallest BSE listings, in which case fall back to a transliteration or the company's own annual-report English name.

How does pricing compare to Bloomberg or Wind?

At $0.10 per stock record, a full A-share universe pull is roughly $530. Bloomberg seats covering mainland China run $24,000+ per year. Wind pricing typically lands in the high four to low five figures per year in RMB and requires onshore payment routing. The Apify approach is two orders of magnitude cheaper.

Can I screen by Shanghai-Hong Kong Stock Connect eligibility?

Yes. The stock_connect_eligible boolean flags whether a name is northbound-accessible across both Shanghai and Shenzhen Connect channels. Re-pull after each quarterly Stock Connect list revision. Current northbound daily quota is RMB 52 billion per channel.

Does the actor handle Chinese character encoding correctly?

All Chinese fields are returned as UTF-8 strings. Python pandas, JavaScript, Excel (with UTF-8 BOM), and BigQuery handle the output natively. Use Excel's Data -> From Text/CSV wizard with explicit UTF-8 to avoid character mangling.

Can I use this dataset for production trading?

Yes, with caveats. Reconcile against your prime broker's security master and official exchange feeds (SSE Datafeed, SZSE Datafeed) where regulatory requirements apply. Best suited to research, backtesting, screening, and reporting.

See also: New -- Kuaishou Trending Tracker

Asian Market Data Scrapers for Public Business Research

NexGenData — Thu, 25 Jun 2026 14:43:52 +0000

Ask any Western fund manager what their best source is for Bombay Stock Exchange equities, Hong Kong insider trades, or Singapore HDB resale prices, and the answer is usually a wince. Bloomberg covers headline indices well, Refinitiv has decent A-share fundamentals if your firm pays the seven-figure subscription, and Crunchbase's APAC startup coverage thins the moment you cross south of Tokyo or west of Mumbai. For everyone else — emerging-markets equity analysts, market-entry consultants, OSINT researchers, regional VC associates — Asia-Pacific public data is a patchwork of native-language portals, government registries, and exchange microsites that no single vendor stitches together affordably.

This post catalogues the Asian market data scrapers we build at NexGenData. They pull from East Money, the HK Companies Registry, MCA21, ACRA BizFile+, EDINET, the Korea Exchange, IDX, SET, PSE, Bursa Malaysia, HOSE, and the major regional social platforms — turning fragmented APAC sources into clean JSON or CSV. Whether you're tracking northbound Stock Connect flows, building a Vietnam consumer-electronics market map, or watching for SEBI enforcement, the goal is structured, refresh-on-demand data that doesn't require a Mandarin-speaking analyst and a Selenium farm.

Why APAC public data stays underserved

The reasons Western vendors under-index on Asia are structural. East Money, the de facto retail terminal for mainland Chinese investors, publishes mostly in Simplified Chinese behind aggressive bot mitigation. The HK Companies Registry charges per-document fees and gates structured exports. India's MCA21 requires CAPTCHA solves for bulk lookups. ACRA BizFile+ is priced for one-off lookups, not analyst workflows. EDINET returns XBRL bundles needing parsing, and Korea's DART lives behind a Korean-language UI. Add regional CDNs that block non-APAC IPs and the operational cost of running headless browsers across a dozen jurisdictions, and the math stops working for generalist vendors. The result: APAC equity coverage on the major Western platforms is a mile wide and an inch deep — Nikkei 225, Hang Seng, Sensex, KOSPI 200, and not much else with the same depth as the S&P 500 dataset.

That gap is where targeted scrapers shine. Each actor below focuses on one source, handles the local-language fields, the anti-bot dance, and the API or HTML quirks specific to that exchange or registry, and returns a normalised payload.

Why this data matters now

APAC weight in global equity benchmarks keeps climbing, MSCI inclusion factors for China A-shares have ratcheted up, and India's float-adjusted market cap crossed $5T in 2024. Practical use cases driving demand for structured APAC data:

China A-share research — northbound flow analysis, A/H premium tracking, CSI 300 vs ChiNext vs STAR Market rotation, SOE insider activity.
India market-entry diligence — MCA company master data on local competitors, director-network mapping, SEBI filings, MagicBricks for physical footprint.
Singapore property and family-office research — URA transactions, HDB resale, MAS-licensed institutions, ACRA UEN counterparty lookups.
Southeast Asian e-commerce intel — JD.com, Made-in-China, Alibaba B2B catalogues for supplier discovery and category share estimates.
Regional VC sourcing — IPO calendars across HKEX, SGX, KOSPI plus the APAC sweep for pre-listing diligence and comp sets.
OSINT and forensic research — HK Companies Registry shell-tracing, Land Registry asset attribution, SFC enforcement, India MCA director-DIN graphs.
M &A targeting — public filings, ownership disclosures, insider trade clusters, CNIPA patent grants for IP-adjacent deals.
Macro and policy tracking — RBI MPC statements, MAS and SFC enforcement, exchange-level IPO pipelines as leading indicators.

What's covered, grouped by sub-region

The roster below is organised by geography. Public actors link straight through; specialised regional actors not on the public marketplace are reachable from the NexGenData Apify catalog — message us or trigger a custom run.

China — financial markets

Actor	Source	What it returns
Eastmoney A-Shares Screener	East Money / 东方财富	Full Shanghai + Shenzhen A-share universe with quotes, PE, PB, market cap, sector tags.
China ETF Flow Tracker	East Money	Daily inflows, outflows, AUM, premium/discount across mainland-listed ETFs.
China A-Share Insider Trades	SSE / SZSE disclosures	Executive 增减持 records — buyer, role, share count, price, transaction window.
HKEX Hang Seng Screener	HKEX	Hang Seng constituents, sector breakdown, fundamentals, A/H pair flagging.
STAR Market Screener	SSE STAR / 科创板	Innovation-board listings with R&D intensity, lockup status, sponsor banks.
ChiNext Screener	SZSE ChiNext / 创业板	Growth-board names with fundamentals, IPO date, suspension flags.
Chinese ADRs Screener	NYSE / NASDAQ	US-listed China names with ADR ratio, sector, regulator status, delisting risk.

China — social and e-commerce signal

Actor	Source	What it returns
China Trends Tracker	Weibo + Baidu + Douyin	Unified daily trending topics across the three biggest discovery surfaces.
Bilibili Video Search	Bilibili / B站	Keyword search results — title, uploader, views, likes, danmaku count.
RedNote (Xiaohongshu) Scraper	Xiaohongshu / 小红书	Posts, engagement, hashtags from the dominant Gen-Z product-discovery platform.
Weibo Hot Search Tracker	Weibo / 微博	Hourly hot-search board snapshots with rank, topic, view count.
Douyin Trending Tracker	Douyin / 抖音	Trending video board with creator handle, view count, hashtag set.
Zhihu Q&A Tracker	Zhihu / 知乎	Hot questions and answer engagement — useful for B2B and pro-consumer themes.
Made-in-China B2B Suppliers	Made-in-China.com	Supplier directory with category, location, certifications, capacity notes.
Alibaba B2B Supplier Finder	Alibaba.com	Wholesale supplier intel — Gold member status, transaction volume, MOQ.

Hong Kong & Taiwan

Actor	Source	What it returns
HKEX IPO Calendar	HKEX	Upcoming Hong Kong listings — pricing range, sponsor, lockup terms, debut date.
HKEX Insider & Short Tracker	HKEX CCASS + disclosures	Substantial-shareholder filings, director dealings, short-interest aggregates.
HK SFC Enforcement Tracker	Securities & Futures Commission	Enforcement notices, sanctions, license suspensions across HK financial sector.
HK Land Registry Transactions	HK Land Registry	Property transaction records with consideration, parties, date, address.
HK Companies Registry	HK Companies Registry	CR number, directors, officers, registered address, dissolution status.
HK Centaline Property Index	Centaline CCL	Weekly residential index — citywide and by sub-market for HK property cycle.
HK Trademark Search	HK IP Department	Trademark registry lookups — applicant, class, status, conflict checks.
Taiwan TWSE Screener	Taiwan Stock Exchange	TWSE-listed universe with fundamentals, dividend yield, foreign holdings.

India

Actor	Source	What it returns
NSE India Indices Screener	National Stock Exchange	Nifty 50, Nifty Next 50, sector indices — constituents and live fundamentals.
BSE India Stock Screener	Bombay Stock Exchange	Sensex and full BSE universe with quotes, fundamentals, group classification.
OGD India Companies Lookup	data.gov.in MCA master	Company master records — CIN, status, category, paid-up capital, address.
India MCA Companies	MCA21	CIN-keyed director lookups, charges, recent filings, signatory history.
India MCA INC-22 / INC-32 Filings	MCA21	Registered-office and incorporation filings — useful for fresh-funded leads.
India SEBI Filings Tracker	SEBI	Listed-company filings, takeover-code disclosures, enforcement orders.
India RBI Monetary Policy	Reserve Bank of India	MPC statements, repo decisions, governor speeches with date and text payload.
India MagicBricks Real Estate	MagicBricks	Listings — price, sqft, locality, builder, possession date across metros.

Singapore

Actor	Source	What it returns
Singapore ACRA Companies	ACRA BizFile+	UEN lookup, directors, registered address, entity type, status.
SG HDB Resale Prices	data.gov.sg HDB	Flat resale transactions — block, sqm, lease, price, town, transaction month.
SG URA Property Transactions	URA REALIS	Private property caveats — project, type, area, price, tenure.
SG MAS Financial Institutions	MAS register	Licensed banks, capital-markets services holders, insurance, payment firms.
SG Rental Market Tracker	HDB + URA rental	HDB and private rental contracts with median rent by district and unit type.
SG SGX Stock Screener	SGX	STI constituents, sector splits, REIT yield, mainboard vs Catalist.
SG MAS Enforcement	MAS enforcement	Fines, prohibition orders, and regulator notices with date and party.
SG MyCareersFuture Jobs	MyCareersFuture.gov.sg	Job postings — employer, title, salary band, EP eligibility flag.

Japan & Korea

Actor	Source	What it returns
TSE Japan Stock Screener	Tokyo Stock Exchange	Nikkei 225 + Prime constituents, fundamentals, foreign ownership ratio.
Japan EDINET Insider Filings	EDINET	Insider trading disclosures, large-shareholder reports, parsed XBRL fields.
KOSPI Stock Screener	Korea Exchange	KOSPI listings — market cap, quotes, foreign holding %, sector classification.

Southeast Asia

Actor	Source	What it returns
HOSE Vietnam Stock Screener	Ho Chi Minh Exchange	VN30 + full HOSE universe with quotes, foreign room, sector tags.
IDX Indonesia Stock Screener	Indonesia Stock Exchange	LQ45 and full IDX list, fundamentals, ownership data.
SET Thailand Stock Screener	Stock Exchange of Thailand	SET50 constituents, ETF list, dividend yield, sector breakdown.
PSE Philippines Stock Screener	Philippine Stock Exchange	PSEi constituents, fundamentals, board lots, sector classification.
Bursa Malaysia Stock Screener	Bursa Malaysia	KLCI universe — quotes, syariah flag, sector splits, dividends.
APAC IPO Calendar Sweep	HKEX + SGX + KOSPI + others	Pan-Asia upcoming listings consolidated in one table with venue, sponsor, date.

Example workflow: building an India market-entry brief

Suppose you're a strategy consultant scoping India entry for a European industrial-automation client. Crunchbase is thin past the funded startups, LinkedIn is noisy, and the official MCA portal won't let you bulk-query. Here's a 90-minute workflow that produces a defensible diligence pack.

Step 1 — define the competitor set. Run the BSE India Stock Screener filtered for Capital Goods / Industrial Manufacturing. Cross-reference with the NSE India Indices Screener. You now have a ranked universe of listed competitors with market cap, sector, and ticker.

Step 2 — pull company master data. For every listed competitor plus unlisted private rivals, run the OGD India Companies Lookup or India MCA Companies for CIN, registered address, paid-up capital, directors, charges, last filing date. Spine of the diligence pack.

Step 3 — overlay regulatory signal. Pipe the same CIN list through the India SEBI Filings Tracker to flag takeover-code disclosures, insider trades, and enforcement orders.

Step 4 — physical-footprint check. Use the India MagicBricks Real Estate actor to sweep commercial listings in competitor HQ cities — useful for facility-size and rental benchmarks.

Step 5 — macro overlay. Append the latest RBI Monetary Policy statement for rate-environment context.

Step 6 — export. Every actor returns CSV / JSON. Drop outputs into BigQuery, Snowflake, or a single workbook. The pipeline runs unattended on a weekly schedule, so the brief stays current through the deal cycle.

The same shape works for a China A-share thesis (East Money + A-share insider trades + Trends tracker), a Singapore property pitch (URA + HDB + Centaline), or a Vietnam consumer scan (HOSE + B2B suppliers + social signal).

Who uses these

EM equity analysts building China A-share, India, or ASEAN coverage without a Bloomberg APAC seat.
Fund managers tracking northbound Stock Connect flows, A/H premium, and KOSPI foreign ownership shifts.
Market-entry consultants producing diligence packs for European or US clients scoping APAC expansion.
Family offices and wealth platforms in Singapore and Hong Kong needing licensed-counterparty checks via MAS and SFC registers.
VC associates sourcing pre-IPO and recently-listed APAC names from HKEX, SGX, KOSPI calendars and the consolidated APAC IPO sweep.
OSINT and due-diligence firms running shell-company traces through HK Companies Registry, Land Registry, and India MCA director graphs.
Trade-finance and supply-chain teams validating Chinese suppliers through Made-in-China, Alibaba B2B, and CNIPA patent records.
Journalists covering APAC business needing primary-source structured data instead of vendor screenshots.
Recruiters and talent intelligence mapping the Singapore tech market via MyCareersFuture and EP-flagged listings.
Macro and policy desks tracking RBI MPC decisions, MAS and SFC enforcement, and regional IPO pipelines as leading indicators.

Start here

Browse the full set of Asian market actors on the NexGenData Apify catalog, or jump straight into the workhorse — the Eastmoney A-Shares Screener — to see the data shape before committing to a workflow. Most actors run on per-event pricing, so you can pull a few hundred records to validate fit before scaling up.

Related actors worth a look

Japan EDINET Insider Filings — for Japan-focused activist or M&A workflows.
HK Land Registry Transactions — asset attribution and HK property research.
RedNote (Xiaohongshu) Scraper — Gen-Z product-discovery signal for consumer brands targeting mainland China.
APAC IPO Calendar Sweep — single pan-regional IPO feed for syndicate desks.
Made-in-China B2B Suppliers — sourcing and supplier-risk research.
China Trends Tracker — combined Weibo / Baidu / Douyin signal for consumer thesis work.

Related reading: Free A-Share data scraping (Chinese-language guide) and Building a real-time FX dashboard for the currency overlay any APAC workflow needs.

FAQ

Why do Western data vendors miss so much APAC data?

Three structural reasons. Language — most APAC primary sources publish in CJK, Bahasa, Thai, or Vietnamese, and parsing requires per-source engineering. Anti-bot — mainland and Indian government sites block non-local IPs and rotate CAPTCHA. Economics — bespoke pipelines for ACRA, MCA21, EDINET, DART, IDX don't pencil out for generalist vendors.

Do these handle Chinese, Japanese, and Korean characters cleanly?

Yes. Every actor that targets a CJK source emits UTF-8 with the original character set preserved alongside any Romanised or English fields the source provides. East Money, EDINET, KOSPI, and HKEX actors have been hardened against the usual encoding pitfalls (mojibake, half-width vs full-width punctuation, traditional vs simplified mixing).

Can I monitor multiple regions in one workflow?

Yes — the most common pattern is a scheduled Apify task that fires three or four region-specific actors in sequence and writes a combined output to a single dataset, Google Sheet, or webhook. The APAC IPO Calendar Sweep is the in-house example of that pattern, pulling HKEX, SGX, KOSPI, and adjacent venues into one stream.

How fresh is the China A-share data?

The Eastmoney A-Shares Screener refreshes within seconds of East Money's own page render — effectively real-time during mainland trading hours, with end-of-day snapshots persisted to your dataset. ETF flows and insider trades follow the underlying exchange disclosure cadence (daily for most fields, intra-day for flow tracking).

Are these compliant with PRC and APAC data laws?

The actors target only public, publicly-disclosed data — exchange filings, government registries, public social platforms — without bypassing login walls or paywalls. PRC PIPL, India DPDPA, Singapore PDPA, and similar regimes principally regulate personal data; structured corporate, listing, and market data is generally outside those scopes. That said, downstream use is your responsibility — if you're republishing or selling derived datasets, get local counsel involved.

How do you handle anti-bot defences on Chinese sites?

A mix of rotating residential proxy pools sized for each region, fingerprint randomisation, request pacing tuned per target, and per-actor fall-through logic that retries with a different network path on classification failure. The infrastructure is shared across actors, so reliability improvements on one Chinese-source actor lift the rest.

What if I need a custom field, region, or source not listed?

Most actors expose configuration that already covers common variations (ticker lists, date ranges, region filters). If you need a genuinely new source — say a specific provincial registry or a niche exchange — message via the Apify console or the NexGenData catalog page and we'll spec it. Many of the listed actors started life as custom builds for a single customer.

Can I export to BI tools or warehouses?

Yes — every Apify run writes to a dataset you can pull as CSV, JSON, JSONL, or Excel, or push via webhook into BigQuery, Snowflake, Postgres, Google Sheets, Airtable, or any system that accepts a POST. The dashboard guides on this site walk through the FX and sentiment-pipeline patterns that work just as well for APAC data.

Company Registry Data Tools for Business Intelligence

NexGenData — Thu, 25 Jun 2026 07:04:56 +0000

Company registry data is the backbone of every credible KYC, KYB, and M &A workflow — yet it lives in 200+ fragmented government portals, each with its own login flow, CAPTCHA, and download cap. This guide walks BD teams, due-diligence analysts, sanctions investigators, and corporate-credit underwriters through the structured company-registry data tools we publish on Apify, which jurisdictions they cover, and how to assemble them into a production-grade entity-resolution pipeline.

1. The Problem: Company Registries Are Fragmented Across 200+ Jurisdictions

Every country runs its own corporate registry. The UK has Companies House. France has Pappers (a private aggregator over INPI/RCS). India has the Ministry of Corporate Affairs (MCA). Singapore has ACRA's BizFile+. Hong Kong has the Companies Registry's ICRIS portal. Australia has ASIC Connect. The United States has a fifty-state patchwork of Secretary-of-State filings, plus Delaware as the de facto incorporation hub. Cayman, BVI, Jersey, and Guernsey each run their own opaque, often-paywalled systems.

For a KYB analyst trying to onboard a single multinational supplier, this means logging into six portals, solving three CAPTCHAs, paying a £3 fee for one PDF in Hong Kong, and copy-pasting director names into a spreadsheet because nothing exports cleanly. For a sanctions investigator mapping the ultimate beneficial owner (UBO) of a shell-company chain across three jurisdictions, it means days of manual cross-referencing. For a sales operations team trying to enrich 50,000 inbound leads with corporate registration numbers (CRN, CIN, UEN, ABN), it means the project never ships.

The fragmentation isn't going away. Even where APIs exist (UK Companies House publishes one of the best), rate limits, schema drift, and bulk-download caps make them unworkable for production. The result: KYC/KYB workflows that can't scale, EDD that takes weeks instead of hours, and procurement teams approving vendors with no UBO visibility because lookup cost is too high. The fix is a thin, consistent layer of registry scrapers — one per jurisdiction, normalized, billed per result, runnable from CLI, n8n, Zapier, or any HTTP client.

2. Why This Data Matters: KYC, KYB, BD, M&A, Sanctions, Credit, OSINT

Structured company registry data sits underneath nearly every regulated and unregulated B2B workflow:

KYC onboarding — verify legal entity name, registration number, registered address, and incorporation date before opening a corporate account. A bank or fintech that skips this fails its AML program audit.
KYB vendor due diligence — procurement teams confirm a supplier is a real, active company with disclosed directors before signing a master services agreement.
BD account enrichment — sales teams append CRN, industry SIC codes, employee count, and director names to inbound leads to route them correctly and personalize outbound.
M &A target screening — corporate development teams build target lists by filtering registries on jurisdiction, SIC code, incorporation year, share-capital range, and director overlap with existing portfolio companies.
Sanctions enrichment and UBO mapping — investigators chain registry data with sanctions watchlists to surface entities whose beneficial owners appear on OFAC SDN, UK HMT, EU Consolidated, or UN sanctions lists.
Credit underwriting — trade credit insurers and B2B lenders pull filed accounts, charges, and director histories to score default risk.
OSINT and investigative journalism — reporters trace shell-company networks across the UK, France, Delaware, and offshore jurisdictions to expose money laundering, tax evasion, or political corruption.
Regulatory horizon scanning — compliance teams monitor enforcement registers (FCA, ASIC, MAS, SFC) for early warning that a counterparty or competitor is under investigation.

The common thread: every one of these workflows needs the data in JSON, in bulk, on a schedule, with predictable latency and cost. None of them are well-served by manual portal lookups.

3. What the Actors Extract — Registry Coverage Matrix

Below is the coverage map across the public NexGenData actor fleet. Identifiers are listed in the format each registry actually uses, because no two jurisdictions agree on what to call a company number.

Country	Source	Identifier	Key fields extracted	Best for
United Kingdom	Companies House	CRN (8-digit)	Officers, appointments, dates of birth (partial), nationality, occupation, resigned/active status	KYB, director-overlap mapping, EDD
United Kingdom	Companies House PSC register	CRN	People with Significant Control (PSC), nature of control, share %, voting %, corporate PSCs	UBO mapping, sanctions enrichment
France	Pappers (RCS / INPI)	SIREN (9-digit) / SIRET (14)	Officers (dirigeants), share capital, NAF/APE code, registered address, RCS filings	French KYB, M&A screening in EU
India	MCA / OGD	CIN (21-char)	Master data, registered address, paid-up capital, ROC, listing status, directors (DIN), date of incorporation	India KYB, supplier vetting, group-structure mapping
India	MCA filings (INC-22/32)	CIN / SRN	Registered-office changes (INC-22), director appointments/changes (INC-32/DIR-12), filing dates	Change detection, EDD trigger events
Singapore	ACRA BizFile+	UEN (9-10 char)	Entity name, status, address, directors, shareholders, business activities (SSIC), paid-up capital	Singapore KYB, regional HQ verification
Hong Kong	Companies Registry (ICRIS)	CR number (7-digit)	Name, status, directors, secretary, registered office, charges, annual return dates	HK KYB, offshore-structure investigation
Australia	ASIC Connect	ACN (9-digit) / ABN (11)	Entity status, registration date, type, jurisdiction, address, EX/AX flags	AU KYB, ASX-adjacent due diligence
United States	Multi-state Secretary of State	State filing number / EIN	Entity name, status, registered agent, formation date, jurisdiction, principal address	US KYB across all 50 states
United States	Delaware Division of Corporations	File number	Entity name, file number, incorporation date, status, registered agent	Delaware-domiciled entity verification
China	CNIPA	Patent number / applicant name	Applicant entity, address, IPC classification, grant/publication dates	China entity discovery via IP filings (adjacent)

Output schemas are normalized across actors where possible: entity_name, jurisdiction, identifier, status, incorporation_date, registered_address, officers[], beneficial_owners[]. Source-specific fields (PSC nature of control, NAF code, SIC code, SSIC, etc.) are preserved in a raw object so downstream parsers don't lose fidelity.

4. Example Workflow — Building a UK Supplier KYB Pipeline

Imagine you're the head of procurement compliance at a mid-market UK SaaS company. You've inherited a vendor master list of 4,200 active suppliers. Internal audit wants UBO disclosure on every supplier by end of quarter, plus an exception report for anyone with adverse regulatory history. Here's the pipeline:

Step 1 — Normalize identifiers. Your ERP exports company names and VAT numbers. For UK suppliers, resolve to a CRN by name+postcode lookup or VAT-to-CRN cross-reference. Aim for >95% match rate; flag the rest for manual review.

Step 2 — Enrich officers via Companies House. Run the Business Registration Lookup actor (UK Companies House Officers actor available on request) over all 4,200 CRNs in batches of 500. You get a director-level dataset: name, role, appointed date, resigned date, date of birth (month/year), nationality, occupation. Materialize this into a suppliers_officers table.

Step 3 — Pull PSC / beneficial ownership. Run the OGD India Companies Master Data Lookup for India suppliers, and a comparable UK PSC actor over the same CRNs. PSC records give you the legally-disclosed beneficial owners with >25% ownership/voting/control, plus corporate PSCs (where a holding company is itself the PSC). For corporate PSCs, recurse: pull their PSC register, and again, until you bottom out at a natural person or hit a non-UK jurisdiction (Cayman, BVI, Jersey — flag for manual EDD).

Step 4 — Cross-check against regulatory enforcement. Run the resolved officer and beneficial-owner names against the Australia ASIC Enforcement Tracker (and the equivalent UK FCA Enforcement Tracker) for financial-services regulatory history, and against the Delaware Corporations Search for US-incorporated counterparties (and the OFAC SDN Watchlist scraper from the catalog) for US sanctions exposure. Fuzzy-match on name + DOB to avoid false positives on common names.

Step 5 — Score risk. Compute a per-supplier risk score: base score from entity status (dissolved/in-liquidation = high), director-overlap with known-bad entities, PSC concealment (corporate PSCs in opaque jurisdictions), and any enforcement hit. Anything above a threshold goes to enhanced due diligence (EDD).

Step 6 — Feed procurement tooling. Push the enriched dataset into your procurement system (Coupa, Ariba, Ivalua) as supplier attributes, and set up a weekly delta job: re-run steps 2–4 only for suppliers whose Companies House last_updated timestamp has changed, plus any new suppliers added in the past 7 days.

End-to-end this is a 1–2 day build, versus 4–6 weeks of analyst time manually. Marginal cost per supplier is well under $0.10 including all enrichment and screening calls.

5. Use Cases at a Glance

KYC onboarding for fintechs and banks — verify corporate customers at account opening, populate AML/CDD records, log evidence for regulator audit.
KYB vendor due diligence — procurement and TPRM (third-party risk management) teams baseline every new vendor before contracting.
Sales operations account enrichment — append registration data to inbound leads for routing, scoring, and outbound personalization.
M &A target screening — corp dev teams filter on jurisdiction, SIC code, age, and director patterns to build long lists.
Beneficial-owner mapping — investigators trace ownership chains across PSC, UEN, CIN, and offshore jurisdictions.
Sanctions exposure screening — chain UBO data with OFAC, HMT, EU, UN watchlists to surface indirect sanctions risk.
OSINT and investigative journalism — map shell-company networks, expose hidden directorships, surface political exposure (PEP-adjacent).
Trade credit underwriting — pull filed accounts, charges, and director histories to predict default risk.
Regulatory horizon scanning — monitor FCA, ASIC, MAS, SFC, and SEBI enforcement actions tied to counterparty entities.
Director-overlap analytics — surface directors who sit on the boards of competitors, suppliers, or distressed entities.

6. Run It Yourself — Business Registration Lookup

The fastest way to feel the difference between portal-by-portal lookups and a normalized API is to actually run one. Start with the highest-volume use case: enriching a list of US-incorporated companies with their state filing records and registered-agent data.

Run the Business Registration Lookup on Apify →

Paste a CSV of entity names or filing numbers, pick output format (JSON / CSV / Excel), and run. Results stream into your Apify dataset and can be pulled via API, webhook, n8n, Zapier, or downloaded as a single file. Pricing is pay-per-result with no monthly minimum, so you can validate the workflow on 50 companies before committing to 50,000.

For Delaware-domiciled holdcos, chain it with the Delaware Corporations Search actor — the schemas are designed to join on entity_name + state with no munging.

7. Related Actors for Cross-Jurisdiction and Risk Enrichment

Real KYB/KYC programs span jurisdictions and data domains. These actors slot into the same pipeline:

Singapore ACRA / BizFile+ Company Lookup — UEN-based entity verification for Singapore-registered suppliers, regional HQs, and APAC fund structures.
OGD India Companies Master Data Lookup — CIN lookup over India's Open Government Data master file, ideal for India KYB and group-structure mapping.
Business Registration Lookup (US multi-state) — Secretary-of-State filing records across the US, including Delaware, California, Texas, New York, and Florida.
Delaware Corporations Search — dedicated Delaware Division of Corporations lookup for the world's most common incorporation jurisdiction.
Australia ASIC Enforcement Tracker — regulatory-action cross-reference for Australian entities and directors.
YC Companies Directory — private-market cross-reference for early-stage US-incorporated entities.

8. FAQ

Are company registries public data?

Most national corporate registries are statutorily public — the UK Companies House, French Pappers (over INPI/RCS), Singapore ACRA, India MCA, Australia ASIC, and US state Secretary-of-State systems all publish entity and officer data because incorporation law requires it. Some registries charge fees for individual document downloads (HK Companies Registry charges per filing; Cayman and BVI are largely paywalled), but the existence and basic data of a registered company is public almost everywhere. Scraping is legally distinct from the data being public — always review each registry's terms of service and applicable jurisdictional law before automating at scale.

What's UBO and how is it captured?

UBO stands for Ultimate Beneficial Owner — the natural person who ultimately owns or controls a legal entity, typically defined as >25% ownership, voting rights, or control. The UK captures this through the PSC (People with Significant Control) register, mandated since 2016. Singapore captures it via the ACRA Register of Registrable Controllers. France captures it via the RBE (Registre des Bénéficiaires Effectifs). Hong Kong via the SCR (Significant Controllers Register). Coverage and accessibility vary widely; the UK PSC register is the most open and machine-readable, which is why it's the starting point for most cross-jurisdiction UBO investigations.

Can I bulk-screen 10,000 suppliers?

Yes. The actors are designed for bulk inputs. For 10,000 UK CRNs, batch into runs of 500–1,000 and stream results via webhook into your data warehouse. End-to-end runtime for a 10K CRN officers-and-PSC pull is typically a few hours, with marginal cost in the low tens of dollars depending on the actor. For ongoing monitoring, set up a weekly delta job keyed on last_updated.

Do you cover the Cayman Islands or BVI?

Not yet for the offshore jurisdictions (Cayman, BVI, Jersey, Guernsey, Bermuda) — these registries are paywalled per-document and largely don't expose bulk lookup. The pragmatic workflow for offshore exposure: identify the offshore entity via PSC chains in covered jurisdictions (UK PSC will name a Cayman holding company as the corporate PSC), then escalate to a paid lookup service (Companies House International, OpenCorporates Premium, or a regulated EDD provider) for the offshore leg. Coverage of offshore registries is on the roadmap.

How fresh is the data?

Each actor pulls live from the source registry at run time, so freshness equals registry freshness. UK Companies House updates within minutes of a filing. Singapore ACRA and Australia ASIC update within hours to a day. India MCA can lag 24–72 hours. France Pappers refreshes daily from INPI. The actors don't cache stale data — each run is a fresh query against the source.

Can I export to my CRM or KYC platform?

Yes. Apify exposes datasets via REST API, webhooks, and direct integrations with n8n, Zapier, Make, and dozens of warehouse connectors. Common patterns: webhook to a Lambda/Cloud Function that upserts into Salesforce, HubSpot, or your KYC platform (Onfido, Sumsub, ComplyAdvantage); scheduled run with output to S3 or Google Cloud Storage for warehouse ingestion (Snowflake, BigQuery, Databricks); direct CSV download for ad-hoc analyst use.

How do you handle name fuzzy-matching across jurisdictions?

Entity names are messy — "Acme Holdings Limited" in the UK might be "Acme Holdings (HK) Ltd" in Hong Kong and "Acme Holdings Pte Ltd" in Singapore. The actors return canonical names from each registry; cross-jurisdiction resolution is the caller's responsibility. We recommend a two-stage match: (1) exact match on registration number where available (LEI, CRN, UEN, CIN), (2) fuzzy match on normalized name (lowercase, strip suffixes, strip punctuation) + address + director-overlap as a tiebreaker.

Is this compliant with GDPR?

Officer and PSC data published by national registries is public by statute, and processing it for KYC, KYB, AML, sanctions screening, and due diligence is generally a legitimate-interest or legal-obligation basis under GDPR Article 6. You remain controller for onward processing — document lawful basis, honor data-subject rights, and consult your DPO before launch.

Published as part of the NexGenData public registry data tools series. Explore the fullPublic Registry Data Tools category for jurisdiction-specific deep dives, or browse related coverage on sanctions data tools, ASIC enforcement tracking, and court records due diligence.

DEV Community: NexGenData

Public Real Estate Data Scrapers for Regional Market Research

The Problem: Real Estate Data Is Hyper-Regional and Hyper-Fragmented

Why Structured Real Estate Data Matters Right Now

What the Actors Extract: Source Coverage at a Glance

Example Workflow: Building a Singapore PropTech Competitor Brief

Use Cases: Who Actually Uses This Data

Run It Yourself: Start with the Redfin Real Estate Scraper

Related Actors and Internal Reading

Frequently Asked Questions

Is real estate listing data public?

Can I bulk-export results to CSV or a data warehouse?

How fresh is the data?

Do you cover commercial real estate?

What about Asian markets beyond Singapore and Hong Kong?

Can I track price changes over time on the same listing?

How does this compare to paid vendors like CoStar or REIS?

Are these actors suitable for AI agents and LLM pipelines?

Using Apify Actors for Business Intelligence Workflows

Anatomy of an Apify actor in a BI context

Three patterns for ingesting actor data

Pattern A: Direct API call from your ETL

Pattern B: Scheduled run + webhook to object storage

Pattern C: Google Sheets via IMPORTDATA

Worked example: a daily competitor pricing BI dashboard

Step 1: schedule actor runs nightly

Step 2: configure dataset export via webhook

Step 3: load to data warehouse

Step 4: build the dashboard view

Step 5: alert on changes via Slack

Cost calibration

Best actors for BI use cases

Limitations and when NOT to use actors

FAQ

Operational tips from production deployments

Airflow DAG sketch

How to Build a Startup Lead List from YC Company Data

The Problem: YC's Directory Is Built for Browsing, Not Pipeline

Why YC Company Data Matters for Outbound

What the YC Companies Directory Scraper Extracts

Example Workflow: From Directory Dump to Live Sequence

Use Cases Across GTM, Recruiting, and Capital

Get the Data: Run the YC Companies Directory Scraper

Related Actors for Building a Complete Lead Stack

FAQ

New: Japan BoJ Macro Rates & JGB Auctions — BoJ policy rate, JGB yield curve, MoF auction calendar & policy statements

What it does

Who it's for

Sample fields / output

Example use cases

Related actors

FAQ

What sources does this consolidate?

Does it include monetary policy statements?

Is auction data covered?

New: Japan MoF FX Intervention Tracker — every MoF yen FX intervention back to 1991 with JPY + USD estimates

What it does

Who it's for

Sample fields / output

Example use cases

Related actors

FAQ

How far back does the intervention data go?

Are amounts available in USD?

Which currency pairs are included?

New: Korea BoK ECOS — Base Rate & Macro — Bank of Korea base rate, M1/M2, GDP, CPI & FX reserves as structured macro records

What it does

Who it's for

Sample fields / output

Example use cases

Related actors

FAQ

Which Bank of Korea series are covered?

Is the data official?

Can I use this for AI agents?

New: Korea KRX Market Stats — daily KOSPI/KOSDAQ/KONEX prices, foreign-ownership ratios & market-cap rankings

What it does

Who it's for

Sample fields / output

Example use cases