DEV Community: Oaida Adrian

Building a Romanian Business Registry API: 4.2M Companies as Queryable JSON

Oaida Adrian — Wed, 26 Aug 2026 11:39:48 +0000

Every Romanian company is registered in the ONRC (the national trade register), and the state publishes the whole thing as open data on data.gov.ro. That's 4.2 million companies, 19.3 million CAEN activity records, 3.68 million legal representatives and 4.64 million status entries — a genuinely useful corpus for lead generation, market analysis and "who actually owns this company" lookups.

The catch: it ships as a pile of enormous CSVs, and the moment you try to query it, Romanian orthography punches you in the face. I built a small free API around it, and this is how.

The source

No scraping, no vendor. The official snapshot is a set of CKAN files, the biggest of which is OD_FIRME.csv at ~690 MB and OD_CAEN_AUTORIZAT.csv at ~425 MB. Combined, raw, that's well over a gigabyte of flat files. Loaded into SQLite with a full-text index it becomes a 3.6 GB database that answers a search in milliseconds.

The build

The load is boring and that's the point: stream the CSVs into SQLite, then create an FTS5 virtual table so lookup_business("dedeman") is a real full-text query rather than a LIKE '%dedeman%' scan. Two tables matter most:

firme — companies, keyed by CUI (the tax identifier) and registration code
reprezentanti_legali — the legal representatives, so you can search by person to find every company a director is attached to

On top of that sits a thin FastAPI wrapper exposing two read-only endpoints — no auth, no keys:

# by company name or CUI
curl "https://onrc-api.adrianhomelab.com/lookup_business?query=DEDEMAN&max_results=3"

# by director / legal-representative name
curl "https://onrc-api.adrianhomelab.com/lookup_director?name=popescu"

The traps (this is the real content)

Three things will bite you, in order of pain:

1. Diacritics. Romanian uses ă â î ș ț, and the registry stores them faithfully. Popescu and Popéscu are different strings to a naive matcher — but a human searching for "Popescu" expects both. The FTS index has to be diacritic-insensitive: normalize ș→s, ț→t, ă→a on both the indexed text and the query. Miss this and your "obvious" search returns nothing for half the surnames in the country.

2. CUI vs registration code. Every firm has a CUI (digits, sometimes zero-padded) and a registration code like J1992002621040 (court + year + serial). They are not interchangeable, and newcomers constantly pass one where the API expects the other. Supporting both on a single query parameter means testing numeric input against the CUI column first, then falling back to the registration code.

3. CSV scale and encoding. The raw dumps are ;-delimited and large enough that csv.DictReader in a tight loop will keep you up all night. Stream with csv.reader, batch your executemany inserts, and watch your transaction size — a single 690 MB file has to be committed incrementally or SQLite's WAL balloons.

What came out of it

A live, free endpoint:

GET /lookup_business?query=<name|CUI>&max_results=N
GET /lookup_director?name=<name>&max_results=N
GET /docs — interactive OpenAPI docs
GET /health

The same data is also available as an MCP server (for AI agents) at https://hermes.adrianhomelab.com/mcp, and the code is open. If you're doing anything with Romanian business data — or you just want a worked example of turning a big government CSV dump into a queryable service — the endpoint is live and free to hit. Source code and loaders are on GitHub: https://github.com/darksider4all/leadgen-mcp.

Building a SEC EDGAR Filings Scraper: 10-K Risk Factors and 13F Holdings as Clean JSON

Oaida Adrian — Wed, 19 Aug 2026 11:11:51 +0000

US public companies file everything with the SEC, and EDGAR is free to use. But the raw archive is hostile to analysis: filings are HTML documents with table-heavy layouts, and the full-text search UI is built for humans, not pipelines. I built an actor that turns a ticker into a clean corpus of JSON records — one per filing — ready for financial alt-data, risk analysis, RAG corpora, or LLM fine-tuning.

The source

No API key, no data vendor. Three official SEC endpoints:

company_tickers.json — ticker to CIK resolution
the EDGAR full-text search JSON API (efts.sec.gov/LATEST/search-index) — query by ticker, form type, and date window
the SEC Archives for the documents themselves

Two traps surfaced immediately. First, SEC fair-access enforcement is real: a bare curl or python-requests User-Agent gets a 403, while a descriptive UA (Darknezz Research admin@…) gets a 200. The actor sets a descriptive UA, paces at ~5 requests/second (well under the 10 rps limit), and backs off on 429s. Second, the ciks search parameter requires the 10-digit zero-padded CIK — 320193 returns zero hits, 0000320193 returns the filing. Padding is mandatory.

The structuring

Raw HTML is only the beginning. Each filing becomes:

Metadata — accession number, ticker, company, CIK, form type, filing date, period ending, 8-K event items, document and index URLs
Sections — split on Item headings (Item 1., Item 1A., Item 7., Item 2.02…), per-section capped so one enormous filing can't blow a record
Risk factors — the full 10-K/10-Q Item 1A text, verified on a real Apple 10-K at the 60k-character cap
Holdings — 13F-HR InfoTable XML parsed into issuer, CUSIP, value, shares, and voting authority. The XML is namespace-prefixed (<ns1:infoTable>), so every regex is namespace-tolerant
Summary — a deterministic, LLM-ready markdown digest: metadata, section map, risk excerpt, holdings preview. No external LLM call, no extra cost

The smoke test

AAPL 8-K ×3 — sections Item 2.02 and 9.01, summaries 476 chars each
AAPL 10-K ×1 — 10+ sections, Item 1A risk factors 60,038 chars, summary 3,267 chars
JPM 13F-HR — 34,064 positions parsed locally; the mega-manager cap (20,000 positions) proven, with holdingsCount and holdingsTruncated flags so the record stays under Apify's item size limit

All cloud runs succeeded with non-zero, well-formed output.

The honest bits

Each filing yields its primary document only; exhibits (press releases, contracts) stay one click away via the filing index URL.
The summary is deterministic, not generative — great for pipelines, not a substitute for an LLM pass.
Mega-manager 13Fs are truncated at 20,000 positions (flagged), because a 9 MB JSON record doesn't ship.

Try it

👉 SEC EDGAR Filings Scraper on Apify Store

More from me

While you're here, these might be worth a read:

Building a Price & Competitor Monitor That Diffs Shopify and AliExpress Prices

Oaida Adrian — Wed, 19 Aug 2026 11:11:47 +0000

Most price watching is a spreadsheet: paste a URL, check it today, forget it tomorrow. The moment you want a history — what did this product cost a month ago, did it dip last Tuesday — snapshots are useless. I built an actor that keeps a real per-URL price timeline, diffs every run against the last one, and pings a webhook the instant something changes.

The design

For each URL the actor:

Fetches the product — Shopify stores expose /products/<handle>.json plus /cart.js for the store currency; AliExpress item pages are client-rendered, so the price comes from the search page matched back by product id
Reads the previous snapshot from a durable key-value store
Computes the diff — NEW, PRICE_CHANGED, AVAILABILITY_CHANGED, or UNCHANGED, with absolute and percentage change
Appends the new point to history, pruning anything older than historyDays
Emits one row per URL with snapshot + diff + recent history, and POSTs a compact JSON alert to your webhook if the row changed

The first run of a URL is the baseline (NEW, no alert). Alerts fire only on real changes after that.

The persistence trap

This is the part that bit: Apify's default key-value store is per-run for API-started runs — it's purged at run start and deleted after the run. A naive implementation would lose its history between runs and re-emit everything as NEW every time. The fix is a named store: Actor.open_key_value_store(name='price-competitor-monitor-history') (the name= argument is keyword-only in SDK 3.4). That store persisted across three cloud runs in testing, which is what turns scheduled runs into a real timeline.

The smoke test

Three URLs — two Shopify stores and an AliExpress dock:

Run 1: allbirds NEW 91.00 USD, colourpop NEW 29.00 USD, non-zero ✓ (the AliExpress item page was x5sec-challenged from the datacenter egress — honest error row, no false price)
Run 2: both Shopify products UNCHANGED — durable history proven in the cloud
Run 3 (with a modified snapshot): colourpop PRICE_CHANGED, previous 36.25 → 29.00, −20%, direction down, alert true ✓

A fetch that fails cleanly (blocked page, deleted product) emits a status: "error" row and leaves the previous snapshot untouched — a transient block never masquerades as a price crash.

The honest bits

AliExpress item pages are fully client-rendered: the price is recovered from the search page by product id, so an item absent from search is reported without a price until it reappears.
Availability on AliExpress isn't reported yet (available: null).
History is capped at 500 points per URL as a safety limit.

Try it

👉 Price & Competitor Monitor on Apify Store

More from me

While you're here, these might be worth a read:

Building a Vertical Corpus Builder: Clean JSONL Datasets for LLM Fine-Tuning

Oaida Adrian — Wed, 19 Aug 2026 11:11:46 +0000

Raw web pages are terrible training data. Nav bars, cookie banners, "related articles" and ads drown the signal, and near-identical syndicated text pollutes the corpus. If you're fine-tuning a domain model — legal reasoning, medical QA, financial analysis — you want clean vertical text with provenance, not a pile of HTML. I built an actor that goes from seed URLs to a token-aware JSONL dataset in one run.

The pipeline

seed URLs → crawl (same-domain BFS) → clean (boilerplate strip)
  → dedupe (exact + near) → token-aware chunk → JSONL with provenance

Three steps matter most:

Boilerplate strip — drop script/style/nav/footer/header/aside and utility blocks (breadcrumbs, shares, comments, menus). One detail that earned its keep: paragraphs where more than half the links are anchor text get dropped too — those are link farms, not content.
Dedupe, exact and near — exact duplicates collapse by normalised hash; near-duplicates by 6-gram Jaccard similarity (default 0.95), so syndicated copies of the same opinion appear once.
Token-aware chunking — chunks split at paragraph boundaries and hard-split on sentence/word boundaries only when a single paragraph exceeds the budget (default 512 tokens). Each chunk stays coherent, which is what fine-tuning actually wants.

Every record carries source (hostname), url, domain (your vertical), title, and chunk_index — so you can filter, cite, or re-weight the corpus later.

The seed-source reality check

Not every legal text source works from a datacenter IP. CourtListener sits behind an AWS WAF JavaScript challenge, and Justia 403s datacenter egress — both unusable as plain-HTTP seeds. Cornell LII (law.cornell.edu/supremecourt/text/…) serves full opinions as clean HTML and works beautifully. The honest move was documenting that in the README and defaulting the examples to LII rather than pretending every source is reachable.

The smoke test

Three LII Supreme Court opinions (Dobbs, Bruen, WV v EPA):

Local: 559 chunks, all 6 provenance keys present, chunk sizes 63–2,054 chars, zero duplicates after global dedupe
Cloud (2 seeds): SUCCEEDED, 443 items, every record with non-empty text, domain=legal, source law.cornell.edu, 0 duplicate texts
KV store: output.jsonl — 443 lines / ~811 KB, plus a SUMMARY record: 443 chunks, 2 pages, ~173,000 estimated tokens, no failed sources

The honest bits

Seed URLs must serve server-rendered HTML. JS-only SPAs and WAF'd sites need a browser-rendering actor instead.
Chunk size is an estimate (~4 chars/token), not a byte-exact tokeniser count.
Near-dedupe is O(n²) in chunk count — cap large crawls with maxChunksPerPage.
The crawler follows same-domain links only; cross-domain citations aren't chased.

Try it

👉 Vertical Corpora Builder on Apify Store

More from me

While you're here, these might be worth a read:

Building a Change & Alert Engine That Webhooks Any Diff on Any URL

Oaida Adrian — Wed, 19 Aug 2026 11:11:44 +0000

Most data jobs re-pull the whole source every run and waste time and money on data that didn't move. A change-detection engine wakes up, checks what actually changed, and pushes only the diff. I built a generic one: point it at any URL — a JSON feed, an RSS/Atom feed, or a plain HTML page — and it emits only the changes, with before and after values, to the dataset and/or your webhook.

How it works

Poll — fetch sourceUrl once per run (the interval is your schedule; the actor is a single-shot poll)
Parse — content is sniffed automatically: JSON arrays used directly; object feeds auto-detect common list keys (items, results, data, records…) or take an itemsPath dot path; RSS/Atom parsed with stable ids (guid or link); HTML becomes one monitored item keyed on a content hash
Diff — each item gets a stable item_id (from idField, natural keys like guid/link/url/title, or a content-hash fallback) and a content hash. New id → added; same id, different hash → modified with full before/after; id missing → removed (only with includeRemovals: true)
Emit — changed items go to the dataset, one per item, plus one batched webhook POST per run that has changes. Nothing is emitted on a clean poll.

The delivery guarantee

At-least-once is the detail I care most about: state is persisted only after a successful webhook POST (or immediately when no webhook is configured). If the POST fails, the run FAILS and the next run re-emits the same changes — an alert is never silently lost. Within a run, duplicate item ids collapse to the last occurrence, so a feed that repeats entries won't double-notify.

The persistence trap

Same lesson as the sibling price monitor: Apify's default key-value store is per-run for API-started runs, so state kept there vanishes between runs and the second run re-emits everything as added. The fix is a named store — Actor.open_key_value_store(name='change-alert-state') (keyword-only name= in SDK 3.4, account-scoped, persists across runs).

The smoke test

BBC News RSS: run 1 → 41 added (real titles and links); run 2 → 0 items, zero changes ✓
Controlled changed source (an Apify dataset with a pre-signed public URL — gist raw CDN served stale content to datacenter IPs, so a dataset is the deterministic route): run A → 2 added; append a modified g2 and a new g3 → run B → modified g2 (price 20 → 25) + added g3; run C → 0 ✓

The honest bits

The engine compares current fetch vs last persisted fetch — items that leave the feed are only reported with includeRemovals: true (off by default, because first-page rotation is often noise).
JSON items without any natural id fall back to a content hash; any edit then looks like an add+remove pair — set idField for clean modified detection.
Feeds with per-item volatile timestamps can look "modified" every poll — list those fields in ignoreFields to hash only what matters.

Try it

👉 Change & Alert Engine on Apify Store

More from me

While you're here, these might be worth a read:

Building a Romania Mall Store Directory: Every Store in 29 Malls, One Schema

Oaida Adrian — Wed, 19 Aug 2026 11:11:43 +0000

Romania's shopping malls publish their store directories, but they publish them in eight different ways: one operator runs a headless CMS, another a WordPress store list, another a Laravel JSON blob. Nobody publishes the whole country. So I built an actor that crawls every major operator and emits every store as one consistent record.

The gap

If you want the store mix of every mall in Romania — for retail analytics, site selection, or leasing research — you'd normally stitch together NEPI's site, AFI's site, Iulius's site, and so on, and still end up with per-operator formats that don't join. Worse, nobody flags the anchor tenants: the hypermarkets, cinemas and department stores that actually drive foot traffic. That's the gap: a country-wide directory with one schema and anchor flags built in.

The sources

Each operator publishes its own public store data, and each needed a different reverse-engineering pass:

NEPI Rockcastle (19 centres, ~2,434 tenants) — their own Strapi CMS JSON API (cms.<site>/api/tenants), including Mega Mall's 215 stores and Promenada's 182
AFI Europe (Cotroceni, Ploiești) — the evx_retailers sitemap
Iulius Group (Cluj, Iași, Suceava, Iulius Town) — a shopping index mixing category and store links, disambiguated by nesting
Sun Plaza, Veranda, Colosseum — WordPress store sitemaps
Băneasa — a Laravel/Inertia data-page JSON payload across 11 pages

The robots.txt lesson deserves a callout: fetching it with Python's default RobotFileParser gets you 403'd by Sun Plaza and Veranda's WAFs — empty rules, everything disallowed. Fetching it with requests and a browser User-Agent works, and the actor then honours the real rules.

The smoke test

Input {"country":"ro","malls":["all"],"includeAnchor":true}:

HTTP 201, 3,868 well-formed records in ~86 seconds
0 empty store names, 0 malformed key sets
382 anchor tenants flagged, 2,495 records with floor/unit, 1,499 with phone
29 of 30 registered malls covered
All 10 identity fields (mall, store, brand, category, url, …) 100% filled

Run it again and the first_seen / last_seen / is_closed fields turn the directory into a vacancy and churn time-series — which stores opened, which closed, which anchors moved.

The honest bits

ParkLake is WAF-gapped from the cloud. Sonae Sierra serves an HTTP 202 JavaScript-challenge page to datacenter IPs for its API, sitemap and archive alike; even curl_cffi impersonation hits the wall. From a residential or proxy IP the REST API returns ~223 stores with full detail. The adapter tries hard, then logs the challenge honestly instead of faking data — attach a residential proxy if you need ParkLake from the cloud.
is_closed only becomes meaningful from the second run — the first run is your baseline.
Fields an operator doesn't publish come back as empty strings, so exports stay clean but sometimes thin.

Try it

👉 Romania Mall Store Directory on Apify Store

More from me

While you're here, these might be worth a read:

Scraping Romanian Public Contracts: A Native-Language Tender & Awarded Deals Scraper

Oaida Adrian — Sun, 16 Aug 2026 19:30:10 +0000

Romania publishes every public tender and awarded contract through e-licitatie.ro — the successor to the old SEAP system. The data is public, the opportunities are real (millions of RON in awards every month), and almost nobody outside Romania is building on it. Being native-language gives us an edge no English-first competitor has.

The gap

The niche's existing actors are thin: the top reference has single-digit runs. Tenders come with deadlines and award values, but most scrapers stop at the listing page. Ours goes deeper — tenders and awarded contracts, with detail enrichment, CPV classification, and winner + CUI (company ID) extraction.

The source

The Angular SPA at e-licitatie.ro calls its own JSON API — but it 403s without the right headers:

POST api-pub/NoticeCommon/GetCNoticeListFiltered/   # tenders
POST api-pub/NoticeCommon/GetCANoticeList/          # awarded contracts

The trick: the API requires a Referer of https://e-licitatie.ro/pub plus Origin and X-Requested-With. Missing referer, instant 403 "Access Denied: Referrer cannot be null". Detail enrichment needs GetSection21View (description, CPV, estimated value — keyed off the item's own sysNoticeTypeId, not the noticeId) and GetSection1View for the contracting authority. Winners come from GetCANoticeContracts — and its filter payload is strict: a bare payload 400s.

The native-language advantage

Romanian titles, Romanian descriptions, CPV codes joined to EU TED. The reference actors are English-first and miss the nuance — e.g., distinguishing licitație deschisă (open tender) from achiziție directă (direct award). Our output carries the original Romanian fields untouched, so downstream users can do their own analysis without a translation layer.

The smoke test

Tenders: keyword infrastructura, dateFrom 2026-01-01 → 5 items, 5/5 complete — real titles, authorities, CPV names, estimated RON values, deadlines, status
Awarded: keyword drumuri, dateFrom 2026-06-01 → 4 items, including winner ROUTE CENTER CONSTRUCT (CUI 29170569) with a contract value of 4,822,401.88 RON and an award date

The honest bits

Authority filtering is client-side containment — the API ignores authority text server-side
Some CAN (award) records show zero value and no winner — that's data-side, the tender was annulled
Publication dates are what the portal publishes; actual contract signatures can lag

Try it

👉 Romanian Public Contracts Scraper on Apify Store

More from me

While you're here, these might be worth a read:

Building a 12-City US Building Permits Scraper With Python

Oaida Adrian — Sun, 16 Aug 2026 19:30:09 +0000

Building permits are the earliest public signal of construction activity — new housing, commercial builds, renovation waves. Every major US city publishes them, but each through a different portal with a different schema. I built one scraper that covers twelve of them with a single output shape.

The gap

The reference actor in this niche covers the big cities but has blind spots: in its own all-cities run it scanned zero rows for Washington DC and never even attempted West Sacramento. Ours covers all twelve: ten Socrata portals (Chicago, NYC, LA, San Francisco, Austin, Seattle, Cincinnati, Mesa, Montgomery County, Baton Rouge) plus two ArcGIS servers (DC, West Sacramento).

The reverse-engineering

Two very different API families to reconcile:

Socrata — the well-documented resource endpoint family, clean JSON
ArcGIS — the old FeatureServer contract, where field names vary per server and some servers reject modern ordering parameters outright

The resilience ladder for the old ArcGIS servers: try TIMESTAMP + orderBy first, fall back to where-only queries, then to unfiltered pulls with client-side date filtering. West Sacramento's server rejects TIMESTAMP entirely — the ladder is what makes it work.

One schema to rule them all

Eighteen canonical keys: city, permit_number, permit_type, status, issue_date, application_date, address, work_description, valuation, contractor_name, applicant, latitude, longitude, source, source_type, source_url, scraped_at, cityKey. NYC's "Permit is not yet issued" placeholder rows get filtered. Adding a city is one entry in the config table.

The smoke test

24/24 non-zero — two runs per city across all twelve, full 18-key schema on every item, plus local unit tests covering ten Socrata normalisations, both ArcGIS servers, and live fetches.

The honest bits

Coverage is US-only and city-scoped — counties and rural areas aren't in this pass
Valuation can be null on older records; some cities simply don't publish it
ArcGIS servers rate-limit; the ladder paces requests per city

Try it

👉 Building Permits Scraper on Apify Store

More from me

While you're here, these might be worth a read:

Building an EU Safety Gate (RAPEX) Product Recall Scraper With Python

Oaida Adrian — Sun, 16 Aug 2026 19:30:08 +0000

Every week the EU publishes the product recalls that keep dangerous goods off the shelves — the Safety Gate (RAPEX) portal. It's open data, but it's buried behind a JavaScript SPA, so most teams never build on it. I did.

The gap

The reference actor in this niche couldn't even be trialled: it carries a $5 minimum charge and our account had $4.69 of budget left. One dollar short, locked out. So instead of paying to watch someone else's implementation, I reverse-engineered the upstream directly — a strictly better outcome.

The source

The EU Safety Gate portal exposes a pure JSON API:

POST /public/api/search — the search contract
GET /public/api/notification/{id}?language=en — per-alert detail

No auth, no cookies, no browser. I captured the exact endpoints and payloads from the live SPA's network traffic, then ground-truthed the date semantics against real notifications.

The smoke test

Input {category: "Electrical appliances and equipment", country: "DE", dateFrom: "2026-07-01"}:

SUCCEEDED, 17 items, 17/17 complete
Every record has reference, productName, riskTypes, riskDescription
Unique reference numbers, date window exactly 2026-07-01..2026-08-11
Run cost: $0.007 — seven-tenths of a cent for 17 recalls

The honest bits

The API covers consumer products, vehicles, and some professional equipment — not medical devices or food
Search is a POST contract, so you need the right payload shape; a bare GET returns nothing
Risk descriptions are free text in the EU's own taxonomy — normalise carefully

Try it

👉 EU Safety Gate (RAPEX) Product Recall Scraper on Apify Store

More from me

While you're here, these might be worth a read:

How I Built a Water Utility Risk Intelligence Tool With Python and MCP

Oaida Adrian — Sun, 16 Aug 2026 19:30:07 +0000

Water utilities fail slowly: aging pipes, drought pressure, unaffordable rates, crumbling infrastructure. The data to assess all of it is public — you just need to pull nine different government APIs and score what comes back. That's the tool I built.

The gap

The niche's reference actor is MCP-standby-only: its batch runs emit zero dataset items. It's an MCP server you call by hand — fine for a demo, useless for monitoring a portfolio of utilities. Our build fixes that: batch mode returns a full assessment item per utility, and the engine is also exposed as eight fastmcp MCP tools for interactive use.

The scoring engine

Four transparent models, one composite score, five verdicts:

Water Vulnerability — contamination and source risk
Infrastructure — pipe age and condition signals
Drought/Climate — precipitation and heat trends
Affordability — rate and income pressure

Composite weights: vulnerability 0.30, infrastructure 0.25, drought 0.25, affordability 0.20. Verdicts: CRITICAL ≥80, HIGH_RISK ≥60, ELEVATED ≥40, MANAGEABLE ≥20, LOW_RISK <20. Every signal that moved a score is returned in the output — no black-box scoring.

Nine keyless upstreams

USGS earthquakes, FEMA disaster declarations (v2 — the v1 endpoint 404s), NOAA weather alerts (custom UA required), Federal Register water-scoped notices, CFPB complaints (default UA only — a custom UA gets 403, a lovely asymmetry), BLS unemployment + CPI, World Bank precipitation/GINI indicators, and Open-Meteo geocoding. OpenAQ v3 needs a key, so it's an optional input that degrades gracefully.

The smoke test

Local engine: NYC → 26 MANAGEABLE, Flint → 21 MANAGEABLE, Sacramento → 26 MANAGEABLE with an aging-infrastructure signal. Cloud: NYC full assessment → composite 26 with all four dimensions populated; Phoenix drought → LOW_RISK 8 with a real NOAA heat signal. The MCP server boots via fastmcp, lists all eight tools, and executes end-to-end.

The honest bits

It's a screening tool, not an engineering audit — a composite score won't tell you which valve is rusting
OpenAQ is keyed and optional; the other eight sources are keyless by design
Some upstreams rate-limit aggressively; the batch mode paces requests

Try it

👉 Water Utility Risk Intelligence on Apify Store

More from me

While you're here, these might be worth a read:

Building an Aviation Hub API: Airports, Airlines, Live Flights & Weather From Six Keyless Sources

Oaida Adrian — Sun, 16 Aug 2026 19:29:56 +0000

The aviation niche looks impenetrable at first: proprietary flight feeds, licensing fees, and a wall of "contact sales" forms. But most of the data people actually pay for is already public — it's just scattered across six or seven government and community sources that nobody bothered to stitch together.

So I built an aviation hub actor that does exactly that: one input, six keyless upstreams, six output modes.

The gap

The reference actor in this niche did the same job, but its live mode returned zero rows in our trial — its single ADS-B feed was empty at the time. One feed, no fallback, no resilience. Ours ships six modes:

Airports — 85K+ entries from OurAirports (public domain), including runways
Airlines — the OpenFlights registry with country and callsign data
Routes — geocoded endpoints plus great-circle distance in kilometres
Live — adsb.lol community ADS-B with an OpenSky anonymous fallback, so live never returns empty when either feed has coverage
Weather — NOAA METAR JSON from aviationweather.gov
Digest — a market summary: busiest airports, biggest airlines, most common aircraft types

Reverse-engineering

The trickiest part was the route mode: OpenFlights gives you source/destination pairs, but not coordinates. I geocode both ends and compute great-circle distance — one formula, no API key, works for every pair in the registry.

The live mode needed the fallback architecture: try adsb.lol first, and if the feed is thin or empty, flip to OpenSky's anonymous endpoint. The reference's single-feed design was its weakness; the fallback is the whole point.

The smoke test

Six modes, six cloud runs, all non-zero:

airports — Romanian airports (Bacău, Belfast shapes correct)
airlines — Tarom returned with country + callsign
routes — OTP→LHR, American Airlines at 5,539.8 km great-circle
live — BAW416 (G-DBCA, A319) tracked near EGLL via adsb.lol
weather — LROP METAR, VFR conditions
digest — 85,892 airports, matching the reference digest total exactly

The honest bits

Live coverage depends on community ADS-B volunteers; the OpenSky fallback helps but isn't a paid-quality feed
METAR is aviation weather only — no TAF long-range forecasts in this mode
Route distances are great-circle, not flight-path

Try it

👉 Aviation Hub on Apify Store

More from me

While you're here, these might be worth a read:

I Stopped Scraping Business Directories and Built an MCP Server on Official Registry Data

Oaida Adrian — Thu, 13 Aug 2026 17:05:25 +0000

I Stopped Scraping Business Directories and Built an MCP Server on Official Registry Data

For a long time I built B2B lead-generation tools the way everyone does: scrape a directory, fight the anti-bot, re-scrape when the layout changes, pray the data is current. Then I found out the Romanian government publishes the entire company registry as open data — 4.2 million firms, refreshed monthly, with legal status, activity codes and directors. No scraping required.

This post is the story of that switch: how I turned an official open-data snapshot into a self-hosted MCP server with three tools that actually do work for an AI agent.

The problem

If you want to build a list of Romanian companies — say, every SRL in Cluj with a CAEN code for software services — you have three options:

Scrape a directory site. Fragile, slow, legally grey, and almost always stale. Directories are themselves scraped from the registry, so you're scraping a copy of a copy.
Buy a database. Expensive, opaque provenance, and you still have to build the pipeline.
Use the official source. The ONRC (Romania's Trade Register) publishes a full snapshot of every registered company on the national open-data portal, data.gov.ro, which runs CKAN.

Option three is the one nobody seems to talk about.

Why official registry data beats scraping

The ONRC open-data programme publishes monthly CSV snapshots:

firme — 4.2M companies: name, CUI (tax ID), registration code, legal form, address, website
reprezentanti legali — legal representatives (directors) per company
caen autorizat — CAEN activity codes per company
stare firma — company status history
nomenclatoare — the decode tables: status codes and CAEN activity names

That's the whole registry, structured, and licensed for reuse. No login, no API key, no rate limit, no anti-bot. The fragility is gone: the government keeps the source fresh, and I just re-download the monthly snapshot.

The catch is that "open data" is not "clean data". Those CSVs are enormous (the firms file alone is ~690 MB), use ^ as a delimiter, carry a BOM, encode Romanian diacritics, and store statuses and activities as codes that mean nothing without the nomenclator tables. That's the real engineering work.

The pipeline: CKAN → SQLite → MCP

The pattern I landed on works for any CKAN-backed government portal:

Find the dataset via the CKAN API. GET {portal}/api/3/action/package_search finds the ONRC organization; package_show returns the resource file URLs.
Inspect the schema cheaply. A single HTTP range request (curl -r 0-1200) reads the CSV header before committing to a download.
Stream-load into SQLite. Never read a multi-hundred-MB CSV into memory. csv.reader + executemany in batches of 5,000 rows, with errors="replace" for diacritics.
Load the nomenclators into their own tables, then LEFT JOIN at query time to decode status and CAEN codes.
Expose as MCP tools that open the database read-only.

The whole thing lives in a small Python package — a loader script, a nomenclator loader, and a FastMCP server. The MCP server (Streamable HTTP) runs on a homelab box and exposes three tools.

The three tools

1. `lookup_business` — the registry search

Search by name or CUI (tax ID). Digits hit the exact CUI index; anything else is a case-insensitive LIKE on the name. Each result is enriched with decoded CAEN activities, directors, and status — the nomenclator joins are what make it useful. A real call:

lookup_business("Dedeman")

{
  "query": "Dedeman",
  "total": 2,
  "results": [
    {
      "companyName": "DEDEMAN SRL",
      "cui": "2816464",
      "registrationCode": "J1992002621040",
      "registrationDate": "05/11/1992",
      "legalForm": "SRL",
      "euid": "ROONRC.J1992002621040",
      "address": "Municipiul Bacău, Bacău, Str. ALEXEI TOLSTOI, 8, 600093",
      "county": "Bacău",
      "website": "www.dedeman.ro",
      "caenActivities": [
        { "code": "0125", "activity": "Cultivarea altor pomi fructiferi, a arbuștilor fructiferi, căpșunilor și a nuciferelor" },
        { "code": "1610", "activity": "Tăierea și rindeluirea lemnului" }
      ],
      "directors": ["PAVAL I. DRAGOS", "BRINZEA S. STEFAN"],
      "status": [{ "code": "1048", "name": "funcțiune" }],
      "source": "onrc"
    }
  ]
}

Everything is decoded: 1048 is "funcțiune" (active), the CAEN codes come back as readable activities. An agent can ask "what does this company actually do?" and get a straight answer.

2. `extract_contacts` — find the humans

Once you know a company exists, you need the contact points. This tool crawls the company website (bounded to a few pages, prioritising contact/about pages) and extracts emails, phone numbers and social profiles. It filters aggressively — no image files, no example.com placeholders, no noreply@ — and matches emails against the site's own domain to cut the noise:

extract_contacts("https://www.bitdefender.ro")

{
  "url": "https://www.bitdefender.ro",
  "domain": "bitdefender.ro",
  "pagesCrawled": 3,
  "emails": null,
  "phones": null,
  "facebook": "https://www.facebook.com/bitdefender",
  "twitter": "https://twitter.com/bitdefender",
  "instagram": "https://www.instagram.com/bitdefender",
  "linkedin": "https://www.linkedin.com/company/bitdefender",
  "youtube": "https://www.youtube.com/c/Bitdefender",
  "socialLinks": [
    "https://www.facebook.com/bitdefender",
    "https://www.twitter.com/bitdefender",
    "https://www.instagram.com/bitdefender",
    "https://www.linkedin.com/company/bitdefender",
    "https://www.youtube.com/c/Bitdefender"
  ],
  "error": null
}

Honest limitations: corporate homepages often carry no public email (hence emails: null here), and obfuscated emails (Cloudflare's data-cfemail, name [at] domain [dot] com) need decoders. The tool handles both, but you learn to expect gaps on big corporate sites — the SMB sites are where the gold is.

3. `lookup_domain` — verify before you call

Email validation is a lead-gen step most people skip. This tool wraps WHOIS + DNS + SPF/DMARC so an agent can check a domain before adding it to a list — is it registered, who owns it, does it even have mail?

lookup_domain("dedeman.ro")

{
  "domain": "dedeman.ro",
  "whois": {
    "registrar": "ICI - Registrar",
    "creationDate": "2001-04-23",
    "nameServers": ["ns1.dedeman.ro", "ns2.orange.ro"]
  },
  "dns": {
    "A": ["52.16.150.45"],
    "MX": ["5 mx.dedeman.ro."]
  },
  "security": {
    "hasSPF": true,
    "spf": "v=spf1 mx ip4:91.216.225.16/32 ... -all",
    "hasDMARC": true
  }
}

That single call tells you the domain is 24 years old, points at a real mail server, and has both SPF and DMARC — a company that takes email seriously. For the ones that fail, you've just saved a bounced email.

Lessons learned

The decode tables are the real work. The registry itself is just codes; the nomenclators turn 1048 into "funcțiune". Budget real time for them — they're in a separate CKAN dataset and it's easy to miss.
The join key isn't the public ID. ONRC joins internally on the registration code (COD_INMATRICULARE) while everyone searches by CUI (tax ID). Get that mapping right or every lookup "silently fails".
CSVs from governments are hostile by accident. Caret delimiters, BOMs, .CSV-suffixed resource names, errors="replace" for diacritics. A range request to read the header first saves hours.
Monthly snapshots beat real-time scraping. The data is a month stale at worst and complete — no pagination war, no missing pages, no layout break. If a registry publishes open data, use it.
Agents need tools, not scrapers. An LLM can't responsibly scrape 4M companies, but it can absolutely call lookup_business("Dedeman") and reason about the result. MCP is the right seam between "registry data" and "agent capability".

What's next

I'm packaging the same pipeline as an Apify actor so it can run on demand (Ro Business Data MCP, coming soon to my Apify account) — but the whole pattern is portable to any CKAN portal. Poland, France, Germany and most EU states publish similar company registries. If your lead-gen data is scraped from a directory, check whether your government already publishes the real thing. It probably does.

More from me

While you're here, these might be worth a read: