DEV Community

Cover image for I Stopped Scraping Business Directories and Built an MCP Server on Official Registry Data
Oaida Adrian
Oaida Adrian

Posted on

I Stopped Scraping Business Directories and Built an MCP Server on Official Registry Data

#webdev #opensource #mcp #scraping

I Stopped Scraping Business Directories and Built an MCP Server on Official Registry Data

For a long time I built B2B lead-generation tools the way everyone does: scrape a directory, fight the anti-bot, re-scrape when the layout changes, pray the data is current. Then I found out the Romanian government publishes the entire company registry as open data — 4.2 million firms, refreshed monthly, with legal status, activity codes and directors. No scraping required.

This post is the story of that switch: how I turned an official open-data snapshot into a self-hosted MCP server with three tools that actually do work for an AI agent.

The problem

If you want to build a list of Romanian companies — say, every SRL in Cluj with a CAEN code for software services — you have three options:

  1. Scrape a directory site. Fragile, slow, legally grey, and almost always stale. Directories are themselves scraped from the registry, so you're scraping a copy of a copy.
  2. Buy a database. Expensive, opaque provenance, and you still have to build the pipeline.
  3. Use the official source. The ONRC (Romania's Trade Register) publishes a full snapshot of every registered company on the national open-data portal, data.gov.ro, which runs CKAN.

Option three is the one nobody seems to talk about.

Why official registry data beats scraping

The ONRC open-data programme publishes monthly CSV snapshots:

  • firme — 4.2M companies: name, CUI (tax ID), registration code, legal form, address, website
  • reprezentanti legali — legal representatives (directors) per company
  • caen autorizat — CAEN activity codes per company
  • stare firma — company status history
  • nomenclatoare — the decode tables: status codes and CAEN activity names

That's the whole registry, structured, and licensed for reuse. No login, no API key, no rate limit, no anti-bot. The fragility is gone: the government keeps the source fresh, and I just re-download the monthly snapshot.

The catch is that "open data" is not "clean data". Those CSVs are enormous (the firms file alone is ~690 MB), use ^ as a delimiter, carry a BOM, encode Romanian diacritics, and store statuses and activities as codes that mean nothing without the nomenclator tables. That's the real engineering work.

The pipeline: CKAN → SQLite → MCP

The pattern I landed on works for any CKAN-backed government portal:

  1. Find the dataset via the CKAN API. GET {portal}/api/3/action/package_search finds the ONRC organization; package_show returns the resource file URLs.
  2. Inspect the schema cheaply. A single HTTP range request (curl -r 0-1200) reads the CSV header before committing to a download.
  3. Stream-load into SQLite. Never read a multi-hundred-MB CSV into memory. csv.reader + executemany in batches of 5,000 rows, with errors="replace" for diacritics.
  4. Load the nomenclators into their own tables, then LEFT JOIN at query time to decode status and CAEN codes.
  5. Expose as MCP tools that open the database read-only.

The whole thing lives in a small Python package — a loader script, a nomenclator loader, and a FastMCP server. The MCP server (Streamable HTTP) runs on a homelab box and exposes three tools.

The three tools

1. lookup_business — the registry search

Search by name or CUI (tax ID). Digits hit the exact CUI index; anything else is a case-insensitive LIKE on the name. Each result is enriched with decoded CAEN activities, directors, and status — the nomenclator joins are what make it useful. A real call:

lookup_business("Dedeman")
Enter fullscreen mode Exit fullscreen mode 
{
  "query": "Dedeman",
  "total": 2,
  "results": [
    {
      "companyName": "DEDEMAN SRL",
      "cui": "2816464",
      "registrationCode": "J1992002621040",
      "registrationDate": "05/11/1992",
      "legalForm": "SRL",
      "euid": "ROONRC.J1992002621040",
      "address": "Municipiul Bacău, Bacău, Str. ALEXEI TOLSTOI, 8, 600093",
      "county": "Bacău",
      "website": "www.dedeman.ro",
      "caenActivities": [
        { "code": "0125", "activity": "Cultivarea altor pomi fructiferi, a arbuștilor fructiferi, căpșunilor și a nuciferelor" },
        { "code": "1610", "activity": "Tăierea și rindeluirea lemnului" }
      ],
      "directors": ["PAVAL I. DRAGOS", "BRINZEA S. STEFAN"],
      "status": [{ "code": "1048", "name": "funcțiune" }],
      "source": "onrc"
    }
  ]
}
Enter fullscreen mode Exit fullscreen mode

Everything is decoded: 1048 is "funcțiune" (active), the CAEN codes come back as readable activities. An agent can ask "what does this company actually do?" and get a straight answer.

2. extract_contacts — find the humans

Once you know a company exists, you need the contact points. This tool crawls the company website (bounded to a few pages, prioritising contact/about pages) and extracts emails, phone numbers and social profiles. It filters aggressively — no image files, no example.com placeholders, no noreply@ — and matches emails against the site's own domain to cut the noise:

extract_contacts("https://www.bitdefender.ro")
Enter fullscreen mode Exit fullscreen mode 
{
  "url": "https://www.bitdefender.ro",
  "domain": "bitdefender.ro",
  "pagesCrawled": 3,
  "emails": null,
  "phones": null,
  "facebook": "https://www.facebook.com/bitdefender",
  "twitter": "https://twitter.com/bitdefender",
  "instagram": "https://www.instagram.com/bitdefender",
  "linkedin": "https://www.linkedin.com/company/bitdefender",
  "youtube": "https://www.youtube.com/c/Bitdefender",
  "socialLinks": [
    "https://www.facebook.com/bitdefender",
    "https://www.twitter.com/bitdefender",
    "https://www.instagram.com/bitdefender",
    "https://www.linkedin.com/company/bitdefender",
    "https://www.youtube.com/c/Bitdefender"
  ],
  "error": null
}
Enter fullscreen mode Exit fullscreen mode

Honest limitations: corporate homepages often carry no public email (hence emails: null here), and obfuscated emails (Cloudflare's data-cfemail, name [at] domain [dot] com) need decoders. The tool handles both, but you learn to expect gaps on big corporate sites — the SMB sites are where the gold is.

3. lookup_domain — verify before you call

Email validation is a lead-gen step most people skip. This tool wraps WHOIS + DNS + SPF/DMARC so an agent can check a domain before adding it to a list — is it registered, who owns it, does it even have mail?

lookup_domain("dedeman.ro")
Enter fullscreen mode Exit fullscreen mode 
{
  "domain": "dedeman.ro",
  "whois": {
    "registrar": "ICI - Registrar",
    "creationDate": "2001-04-23",
    "nameServers": ["ns1.dedeman.ro", "ns2.orange.ro"]
  },
  "dns": {
    "A": ["52.16.150.45"],
    "MX": ["5 mx.dedeman.ro."]
  },
  "security": {
    "hasSPF": true,
    "spf": "v=spf1 mx ip4:91.216.225.16/32 ... -all",
    "hasDMARC": true
  }
}
Enter fullscreen mode Exit fullscreen mode

That single call tells you the domain is 24 years old, points at a real mail server, and has both SPF and DMARC — a company that takes email seriously. For the ones that fail, you've just saved a bounced email.

Lessons learned

  • The decode tables are the real work. The registry itself is just codes; the nomenclators turn 1048 into "funcțiune". Budget real time for them — they're in a separate CKAN dataset and it's easy to miss.
  • The join key isn't the public ID. ONRC joins internally on the registration code (COD_INMATRICULARE) while everyone searches by CUI (tax ID). Get that mapping right or every lookup "silently fails".
  • CSVs from governments are hostile by accident. Caret delimiters, BOMs, .CSV-suffixed resource names, errors="replace" for diacritics. A range request to read the header first saves hours.
  • Monthly snapshots beat real-time scraping. The data is a month stale at worst and complete — no pagination war, no missing pages, no layout break. If a registry publishes open data, use it.
  • Agents need tools, not scrapers. An LLM can't responsibly scrape 4M companies, but it can absolutely call lookup_business("Dedeman") and reason about the result. MCP is the right seam between "registry data" and "agent capability".

What's next

I'm packaging the same pipeline as an Apify actor so it can run on demand (Ro Business Data MCP, coming soon to my Apify account) — but the whole pattern is portable to any CKAN portal. Poland, France, Germany and most EU states publish similar company registries. If your lead-gen data is scraped from a directory, check whether your government already publishes the real thing. It probably does.

More from me

While you're here, these might be worth a read:

Top comments (0)