DEV Community: Oaida Adrian

Scraping Romanian Public Contracts: A Native-Language Tender & Awarded Deals Scraper

Oaida Adrian — Sun, 16 Aug 2026 19:30:10 +0000

Romania publishes every public tender and awarded contract through e-licitatie.ro — the successor to the old SEAP system. The data is public, the opportunities are real (millions of RON in awards every month), and almost nobody outside Romania is building on it. Being native-language gives us an edge no English-first competitor has.

The gap

The niche's existing actors are thin: the top reference has single-digit runs. Tenders come with deadlines and award values, but most scrapers stop at the listing page. Ours goes deeper — tenders and awarded contracts, with detail enrichment, CPV classification, and winner + CUI (company ID) extraction.

The source

The Angular SPA at e-licitatie.ro calls its own JSON API — but it 403s without the right headers:

POST api-pub/NoticeCommon/GetCNoticeListFiltered/   # tenders
POST api-pub/NoticeCommon/GetCANoticeList/          # awarded contracts

The trick: the API requires a Referer of https://e-licitatie.ro/pub plus Origin and X-Requested-With. Missing referer, instant 403 "Access Denied: Referrer cannot be null". Detail enrichment needs GetSection21View (description, CPV, estimated value — keyed off the item's own sysNoticeTypeId, not the noticeId) and GetSection1View for the contracting authority. Winners come from GetCANoticeContracts — and its filter payload is strict: a bare payload 400s.

The native-language advantage

Romanian titles, Romanian descriptions, CPV codes joined to EU TED. The reference actors are English-first and miss the nuance — e.g., distinguishing licitație deschisă (open tender) from achiziție directă (direct award). Our output carries the original Romanian fields untouched, so downstream users can do their own analysis without a translation layer.

The smoke test

Tenders: keyword infrastructura, dateFrom 2026-01-01 → 5 items, 5/5 complete — real titles, authorities, CPV names, estimated RON values, deadlines, status
Awarded: keyword drumuri, dateFrom 2026-06-01 → 4 items, including winner ROUTE CENTER CONSTRUCT (CUI 29170569) with a contract value of 4,822,401.88 RON and an award date

The honest bits

Authority filtering is client-side containment — the API ignores authority text server-side
Some CAN (award) records show zero value and no winner — that's data-side, the tender was annulled
Publication dates are what the portal publishes; actual contract signatures can lag

Try it

👉 Romanian Public Contracts Scraper on Apify Store

More from me

While you're here, these might be worth a read:

Building a 12-City US Building Permits Scraper With Python

Oaida Adrian — Sun, 16 Aug 2026 19:30:09 +0000

Building permits are the earliest public signal of construction activity — new housing, commercial builds, renovation waves. Every major US city publishes them, but each through a different portal with a different schema. I built one scraper that covers twelve of them with a single output shape.

The gap

The reference actor in this niche covers the big cities but has blind spots: in its own all-cities run it scanned zero rows for Washington DC and never even attempted West Sacramento. Ours covers all twelve: ten Socrata portals (Chicago, NYC, LA, San Francisco, Austin, Seattle, Cincinnati, Mesa, Montgomery County, Baton Rouge) plus two ArcGIS servers (DC, West Sacramento).

The reverse-engineering

Two very different API families to reconcile:

Socrata — the well-documented resource endpoint family, clean JSON
ArcGIS — the old FeatureServer contract, where field names vary per server and some servers reject modern ordering parameters outright

The resilience ladder for the old ArcGIS servers: try TIMESTAMP + orderBy first, fall back to where-only queries, then to unfiltered pulls with client-side date filtering. West Sacramento's server rejects TIMESTAMP entirely — the ladder is what makes it work.

One schema to rule them all

Eighteen canonical keys: city, permit_number, permit_type, status, issue_date, application_date, address, work_description, valuation, contractor_name, applicant, latitude, longitude, source, source_type, source_url, scraped_at, cityKey. NYC's "Permit is not yet issued" placeholder rows get filtered. Adding a city is one entry in the config table.

The smoke test

24/24 non-zero — two runs per city across all twelve, full 18-key schema on every item, plus local unit tests covering ten Socrata normalisations, both ArcGIS servers, and live fetches.

The honest bits

Coverage is US-only and city-scoped — counties and rural areas aren't in this pass
Valuation can be null on older records; some cities simply don't publish it
ArcGIS servers rate-limit; the ladder paces requests per city

Try it

👉 Building Permits Scraper on Apify Store

More from me

While you're here, these might be worth a read:

Building an EU Safety Gate (RAPEX) Product Recall Scraper With Python

Oaida Adrian — Sun, 16 Aug 2026 19:30:08 +0000

Every week the EU publishes the product recalls that keep dangerous goods off the shelves — the Safety Gate (RAPEX) portal. It's open data, but it's buried behind a JavaScript SPA, so most teams never build on it. I did.

The gap

The reference actor in this niche couldn't even be trialled: it carries a $5 minimum charge and our account had $4.69 of budget left. One dollar short, locked out. So instead of paying to watch someone else's implementation, I reverse-engineered the upstream directly — a strictly better outcome.

The source

The EU Safety Gate portal exposes a pure JSON API:

POST /public/api/search — the search contract
GET /public/api/notification/{id}?language=en — per-alert detail

No auth, no cookies, no browser. I captured the exact endpoints and payloads from the live SPA's network traffic, then ground-truthed the date semantics against real notifications.

The smoke test

Input {category: "Electrical appliances and equipment", country: "DE", dateFrom: "2026-07-01"}:

SUCCEEDED, 17 items, 17/17 complete
Every record has reference, productName, riskTypes, riskDescription
Unique reference numbers, date window exactly 2026-07-01..2026-08-11
Run cost: $0.007 — seven-tenths of a cent for 17 recalls

The honest bits

The API covers consumer products, vehicles, and some professional equipment — not medical devices or food
Search is a POST contract, so you need the right payload shape; a bare GET returns nothing
Risk descriptions are free text in the EU's own taxonomy — normalise carefully

Try it

👉 EU Safety Gate (RAPEX) Product Recall Scraper on Apify Store

More from me

While you're here, these might be worth a read:

How I Built a Water Utility Risk Intelligence Tool With Python and MCP

Oaida Adrian — Sun, 16 Aug 2026 19:30:07 +0000

Water utilities fail slowly: aging pipes, drought pressure, unaffordable rates, crumbling infrastructure. The data to assess all of it is public — you just need to pull nine different government APIs and score what comes back. That's the tool I built.

The gap

The niche's reference actor is MCP-standby-only: its batch runs emit zero dataset items. It's an MCP server you call by hand — fine for a demo, useless for monitoring a portfolio of utilities. Our build fixes that: batch mode returns a full assessment item per utility, and the engine is also exposed as eight fastmcp MCP tools for interactive use.

The scoring engine

Four transparent models, one composite score, five verdicts:

Water Vulnerability — contamination and source risk
Infrastructure — pipe age and condition signals
Drought/Climate — precipitation and heat trends
Affordability — rate and income pressure

Composite weights: vulnerability 0.30, infrastructure 0.25, drought 0.25, affordability 0.20. Verdicts: CRITICAL ≥80, HIGH_RISK ≥60, ELEVATED ≥40, MANAGEABLE ≥20, LOW_RISK <20. Every signal that moved a score is returned in the output — no black-box scoring.

Nine keyless upstreams

USGS earthquakes, FEMA disaster declarations (v2 — the v1 endpoint 404s), NOAA weather alerts (custom UA required), Federal Register water-scoped notices, CFPB complaints (default UA only — a custom UA gets 403, a lovely asymmetry), BLS unemployment + CPI, World Bank precipitation/GINI indicators, and Open-Meteo geocoding. OpenAQ v3 needs a key, so it's an optional input that degrades gracefully.

The smoke test

Local engine: NYC → 26 MANAGEABLE, Flint → 21 MANAGEABLE, Sacramento → 26 MANAGEABLE with an aging-infrastructure signal. Cloud: NYC full assessment → composite 26 with all four dimensions populated; Phoenix drought → LOW_RISK 8 with a real NOAA heat signal. The MCP server boots via fastmcp, lists all eight tools, and executes end-to-end.

The honest bits

It's a screening tool, not an engineering audit — a composite score won't tell you which valve is rusting
OpenAQ is keyed and optional; the other eight sources are keyless by design
Some upstreams rate-limit aggressively; the batch mode paces requests

Try it

👉 Water Utility Risk Intelligence on Apify Store

More from me

While you're here, these might be worth a read:

Building an Aviation Hub API: Airports, Airlines, Live Flights & Weather From Six Keyless Sources

Oaida Adrian — Sun, 16 Aug 2026 19:29:56 +0000

The aviation niche looks impenetrable at first: proprietary flight feeds, licensing fees, and a wall of "contact sales" forms. But most of the data people actually pay for is already public — it's just scattered across six or seven government and community sources that nobody bothered to stitch together.

So I built an aviation hub actor that does exactly that: one input, six keyless upstreams, six output modes.

The gap

The reference actor in this niche did the same job, but its live mode returned zero rows in our trial — its single ADS-B feed was empty at the time. One feed, no fallback, no resilience. Ours ships six modes:

Airports — 85K+ entries from OurAirports (public domain), including runways
Airlines — the OpenFlights registry with country and callsign data
Routes — geocoded endpoints plus great-circle distance in kilometres
Live — adsb.lol community ADS-B with an OpenSky anonymous fallback, so live never returns empty when either feed has coverage
Weather — NOAA METAR JSON from aviationweather.gov
Digest — a market summary: busiest airports, biggest airlines, most common aircraft types

Reverse-engineering

The trickiest part was the route mode: OpenFlights gives you source/destination pairs, but not coordinates. I geocode both ends and compute great-circle distance — one formula, no API key, works for every pair in the registry.

The live mode needed the fallback architecture: try adsb.lol first, and if the feed is thin or empty, flip to OpenSky's anonymous endpoint. The reference's single-feed design was its weakness; the fallback is the whole point.

The smoke test

Six modes, six cloud runs, all non-zero:

airports — Romanian airports (Bacău, Belfast shapes correct)
airlines — Tarom returned with country + callsign
routes — OTP→LHR, American Airlines at 5,539.8 km great-circle
live — BAW416 (G-DBCA, A319) tracked near EGLL via adsb.lol
weather — LROP METAR, VFR conditions
digest — 85,892 airports, matching the reference digest total exactly

The honest bits

Live coverage depends on community ADS-B volunteers; the OpenSky fallback helps but isn't a paid-quality feed
METAR is aviation weather only — no TAF long-range forecasts in this mode
Route distances are great-circle, not flight-path

Try it

👉 Aviation Hub on Apify Store

More from me

While you're here, these might be worth a read:

I Stopped Scraping Business Directories and Built an MCP Server on Official Registry Data

Oaida Adrian — Thu, 13 Aug 2026 17:05:25 +0000

I Stopped Scraping Business Directories and Built an MCP Server on Official Registry Data

For a long time I built B2B lead-generation tools the way everyone does: scrape a directory, fight the anti-bot, re-scrape when the layout changes, pray the data is current. Then I found out the Romanian government publishes the entire company registry as open data — 4.2 million firms, refreshed monthly, with legal status, activity codes and directors. No scraping required.

This post is the story of that switch: how I turned an official open-data snapshot into a self-hosted MCP server with three tools that actually do work for an AI agent.

The problem

If you want to build a list of Romanian companies — say, every SRL in Cluj with a CAEN code for software services — you have three options:

Scrape a directory site. Fragile, slow, legally grey, and almost always stale. Directories are themselves scraped from the registry, so you're scraping a copy of a copy.
Buy a database. Expensive, opaque provenance, and you still have to build the pipeline.
Use the official source. The ONRC (Romania's Trade Register) publishes a full snapshot of every registered company on the national open-data portal, data.gov.ro, which runs CKAN.

Option three is the one nobody seems to talk about.

Why official registry data beats scraping

The ONRC open-data programme publishes monthly CSV snapshots:

firme — 4.2M companies: name, CUI (tax ID), registration code, legal form, address, website
reprezentanti legali — legal representatives (directors) per company
caen autorizat — CAEN activity codes per company
stare firma — company status history
nomenclatoare — the decode tables: status codes and CAEN activity names

That's the whole registry, structured, and licensed for reuse. No login, no API key, no rate limit, no anti-bot. The fragility is gone: the government keeps the source fresh, and I just re-download the monthly snapshot.

The catch is that "open data" is not "clean data". Those CSVs are enormous (the firms file alone is ~690 MB), use ^ as a delimiter, carry a BOM, encode Romanian diacritics, and store statuses and activities as codes that mean nothing without the nomenclator tables. That's the real engineering work.

The pipeline: CKAN → SQLite → MCP

The pattern I landed on works for any CKAN-backed government portal:

Find the dataset via the CKAN API. GET {portal}/api/3/action/package_search finds the ONRC organization; package_show returns the resource file URLs.
Inspect the schema cheaply. A single HTTP range request (curl -r 0-1200) reads the CSV header before committing to a download.
Stream-load into SQLite. Never read a multi-hundred-MB CSV into memory. csv.reader + executemany in batches of 5,000 rows, with errors="replace" for diacritics.
Load the nomenclators into their own tables, then LEFT JOIN at query time to decode status and CAEN codes.
Expose as MCP tools that open the database read-only.

The whole thing lives in a small Python package — a loader script, a nomenclator loader, and a FastMCP server. The MCP server (Streamable HTTP) runs on a homelab box and exposes three tools.

The three tools

1. `lookup_business` — the registry search

Search by name or CUI (tax ID). Digits hit the exact CUI index; anything else is a case-insensitive LIKE on the name. Each result is enriched with decoded CAEN activities, directors, and status — the nomenclator joins are what make it useful. A real call:

lookup_business("Dedeman")

{
  "query": "Dedeman",
  "total": 2,
  "results": [
    {
      "companyName": "DEDEMAN SRL",
      "cui": "2816464",
      "registrationCode": "J1992002621040",
      "registrationDate": "05/11/1992",
      "legalForm": "SRL",
      "euid": "ROONRC.J1992002621040",
      "address": "Municipiul Bacău, Bacău, Str. ALEXEI TOLSTOI, 8, 600093",
      "county": "Bacău",
      "website": "www.dedeman.ro",
      "caenActivities": [
        { "code": "0125", "activity": "Cultivarea altor pomi fructiferi, a arbuștilor fructiferi, căpșunilor și a nuciferelor" },
        { "code": "1610", "activity": "Tăierea și rindeluirea lemnului" }
      ],
      "directors": ["PAVAL I. DRAGOS", "BRINZEA S. STEFAN"],
      "status": [{ "code": "1048", "name": "funcțiune" }],
      "source": "onrc"
    }
  ]
}

Everything is decoded: 1048 is "funcțiune" (active), the CAEN codes come back as readable activities. An agent can ask "what does this company actually do?" and get a straight answer.

2. `extract_contacts` — find the humans

Once you know a company exists, you need the contact points. This tool crawls the company website (bounded to a few pages, prioritising contact/about pages) and extracts emails, phone numbers and social profiles. It filters aggressively — no image files, no example.com placeholders, no noreply@ — and matches emails against the site's own domain to cut the noise:

extract_contacts("https://www.bitdefender.ro")

{
  "url": "https://www.bitdefender.ro",
  "domain": "bitdefender.ro",
  "pagesCrawled": 3,
  "emails": null,
  "phones": null,
  "facebook": "https://www.facebook.com/bitdefender",
  "twitter": "https://twitter.com/bitdefender",
  "instagram": "https://www.instagram.com/bitdefender",
  "linkedin": "https://www.linkedin.com/company/bitdefender",
  "youtube": "https://www.youtube.com/c/Bitdefender",
  "socialLinks": [
    "https://www.facebook.com/bitdefender",
    "https://www.twitter.com/bitdefender",
    "https://www.instagram.com/bitdefender",
    "https://www.linkedin.com/company/bitdefender",
    "https://www.youtube.com/c/Bitdefender"
  ],
  "error": null
}

Honest limitations: corporate homepages often carry no public email (hence emails: null here), and obfuscated emails (Cloudflare's data-cfemail, name [at] domain [dot] com) need decoders. The tool handles both, but you learn to expect gaps on big corporate sites — the SMB sites are where the gold is.

3. `lookup_domain` — verify before you call

Email validation is a lead-gen step most people skip. This tool wraps WHOIS + DNS + SPF/DMARC so an agent can check a domain before adding it to a list — is it registered, who owns it, does it even have mail?

lookup_domain("dedeman.ro")

{
  "domain": "dedeman.ro",
  "whois": {
    "registrar": "ICI - Registrar",
    "creationDate": "2001-04-23",
    "nameServers": ["ns1.dedeman.ro", "ns2.orange.ro"]
  },
  "dns": {
    "A": ["52.16.150.45"],
    "MX": ["5 mx.dedeman.ro."]
  },
  "security": {
    "hasSPF": true,
    "spf": "v=spf1 mx ip4:91.216.225.16/32 ... -all",
    "hasDMARC": true
  }
}

That single call tells you the domain is 24 years old, points at a real mail server, and has both SPF and DMARC — a company that takes email seriously. For the ones that fail, you've just saved a bounced email.

Lessons learned

The decode tables are the real work. The registry itself is just codes; the nomenclators turn 1048 into "funcțiune". Budget real time for them — they're in a separate CKAN dataset and it's easy to miss.
The join key isn't the public ID. ONRC joins internally on the registration code (COD_INMATRICULARE) while everyone searches by CUI (tax ID). Get that mapping right or every lookup "silently fails".
CSVs from governments are hostile by accident. Caret delimiters, BOMs, .CSV-suffixed resource names, errors="replace" for diacritics. A range request to read the header first saves hours.
Monthly snapshots beat real-time scraping. The data is a month stale at worst and complete — no pagination war, no missing pages, no layout break. If a registry publishes open data, use it.
Agents need tools, not scrapers. An LLM can't responsibly scrape 4M companies, but it can absolutely call lookup_business("Dedeman") and reason about the result. MCP is the right seam between "registry data" and "agent capability".

What's next

I'm packaging the same pipeline as an Apify actor so it can run on demand (Ro Business Data MCP, coming soon to my Apify account) — but the whole pattern is portable to any CKAN portal. Poland, France, Germany and most EU states publish similar company registries. If your lead-gen data is scraped from a directory, check whether your government already publishes the real thing. It probably does.

More from me

While you're here, these might be worth a read:

I Built a Canada Product Recalls & Safety Alerts Scraper That Reads Open Government Data

Oaida Adrian — Thu, 13 Aug 2026 14:45:52 +0000

I Built a Canada Product Recalls & Safety Alerts Scraper That Reads Open Government Data

Every few weeks another consumer product gets recalled — a battery that overheats, a fan that catches fire, a vehicle part that fails. For compliance teams, importers and retailers, staying on top of those announcements is a job in itself. Canada publishes every recall on a government website, but the site has no JSON API and no RSS feed. It is plain HTML, served by Drupal.

So I built an Apify actor that reads it directly — no API key, no login, no proxy budget.

The niche, and why build our own

The product-recall scraper niche on Apify is small but telling. The reference implementation that dominates it is dromb/canada-recalls-safety-alerts — a reliable, well-shaped actor that turns Health Canada's recall database into clean JSON. When I looked at the niche, that actor was the reliability benchmark. It worked, it had a sensible schema, and it had traction.

Our house rule is simple: when a niche is worth entering, build our own implementation rather than depend on someone else's actor — same source, same schema, same promises, our code. That way the data pipeline we sell isn't hostage to another developer's maintenance schedule. This post is the story of that build.

The data source

The database lives at recalls-rappels.canada.ca (part of the open.canada.ca family) — Health Canada's official recall, alert and safety advisory database covering consumer products, vehicles, food, drugs and health products. It's Drupal, which means:

No public JSON API. The structured data exists, but you have to parse it out of HTML.
A full-text search index at /en/search?search_api_fulltext=<keyword>.
Detail pages at /en/alert-recall/<slug> with structured fields (Drupal machine-names).
A polite 418 for plain script clients — it wants a browser User-Agent.

The whole design flows from those four facts: one source, one job, HTML scraping, and a browser User-Agent.

First contact: the 418

The very first request to the search page came back 418 I'm a teapot. Not a 403, not a captcha — a teapot. The site is configured to serve plain-scripted clients nothing. The fix is one line of honesty:

HTTP_HEADERS = {
    "User-Agent": (
        "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
        "(KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36"
    ),
    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
    "Accept-Language": "en-CA,en;q=0.9,fr-CA;q=0.8,fr;q=0.7",
}

The search page also 301-redirects to itself, so redirects must be followed. Once you look like a browser and follow redirects, the site is completely open — no login anywhere.

The flow

Run a full-text search against the Drupal search index (/en/search?search_api_fulltext=<keyword>).
Page through the results (page=,0,N pagination).
Visit each recall detail page (/en/alert-recall/<slug>).
Parse the structured Drupal fields into clean records.
Apply the category and date post-filters, cap at maxItems.

Archived recalls are excluded — the search surface excludes them, and the reference actor behaves the same way, so every row is an active recall.

Detail fetches are paced (DETAIL_DELAY_SECONDS = 0.25) with exponential backoff on 4xx/5xx — it's a government site, be polite.

The schema — reverse-engineered from the reference

The hardest part wasn't scraping. It was matching the reference actor's schema exactly, so that anyone using the community actor can switch to ours with zero changes. I pulled a 50-item sample of the reference's dataset and reverse-engineered the semantics field by field.

The final record has 17 fields:

{
  "recallId": 82426,
  "title": "Portable Clip-On Rechargeable Mini Fan recalled due to fire hazard",
  "dateUpdated": "2026-08-12",
  "recallClass": null,
  "category": "Household items",
  "summary": "Fire hazard",
  "affectedProducts": "Portable Clip-On Rechargeable Mini Fan",
  "companies": "Consumer product safety",
  "hazard": null,
  "issue": "Fire hazard",
  "whatToDo": "Consumers should immediately stop using the recalled product and return it to CTG Brands Inc. for a refund...",
  "archived": false,
  "status": "active",
  "language": "en",
  "sourceUrl": "https://recalls-rappels.canada.ca/en/alert-recall/portable-clip-rechargeable-mini-fan-recalled-due-fire-hazard",
  "dataSource": "open.canada.ca",
  "rawSourceType": "recall_record"
}

The interesting semantics, all verified against the reference dataset:

category strips the top-level taxonomy prefix. The site's raw value is Consumer products - Electronics; the reference emits just Electronics. Multi-item categories join differently (Light Truck & Van - SUV stays as-is because the top-level prefix doesn't apply).
summary and issue always mirror each other — the last - segment of the issue-type label.
companies is the issuing organization from publisher metadata (Consumer product safety, CFIA, TC), not the manufacturer.
recallClass is Type I/II/III for health-product recalls, Class 1/2/3 for food recalls, else null.
hazard is always null in practice — the field exists in the reference schema but the live pages don't populate it.
archived is always false and status always active, because the search surface excludes archived recalls.

Getting this right was a weekend of diffing: parse a page, compare with the reference row for the same recall, adjust, repeat across consumer/vehicle/drug/CFIA recall types.

Filters

All input fields are optional — {} returns the most recent active recalls:

Field	Type	Description
`keyword`	string	Full-text search term (`battery`, `croissant`, `vehicle`). Empty = browse latest recalls.
`category`	string	Case-insensitive partial match against the parsed category (`Electronics`, `Household items`, `Vehicles`).
`dateFrom`	string	Only recalls last updated on or after this date (`YYYY-MM-DD`).
`maxItems`	integer	Maximum records to extract (default 50, max 500).

The keyword filter runs server-side through Drupal's search index; category and dateFrom are post-filters on the parsed records. The dateFrom filter is what makes an incremental feed possible: run it weekly with your last-run date, and you get only the new recalls.

The smoke test

The cloud smoke run that proved the actor:

{
  "keyword": "battery",
  "category": "Electronics",
  "dateFrom": "2026-06-01"
}

Result: SUCCEEDED, 4 well-formed Electronics items, all past the date filter, all schema keys exact with native types:

recallId	Title	Date
82213	Super Off-Road power bank recalled due to fire hazard	2026-08-12
82139	Wyze Cam v3 recalled — battery may overheat	2026-08-06
82210	Arizer vaporizer battery recalled	2026-08-08
82150	Steambow AR-6 recalled — arrow rest may break	2026-07-10

Every field parses, every type is native (recallId is an int, archived a bool — not strings), and the reference comparison is clean.

Pitfalls worth naming

Three Apify-specific traps bit during this build:

Input schema requires editor on every property. Apify's input schema validation rejects any field without an editor — add it to all four inputs or the push fails.
actor.json dataset fields are string-only, and enforced on push. Declaring storages.dataset.fields with type: "integer" fails at build with Schema validation failed. The fix is to drop the storages block entirely — native types are then stored, exactly like the reference actor.
run-sync-get-dataset-items?timeout=N sets the RUN's timeout, not the API wait. A 20-page crawl needs N >= 480. And the sync socket read can time out client-side while the run keeps going — poll /v2/actor-runs/{id} and fetch defaultDatasetId when that happens.

Run it

The actor is darknezz/canada-recalls-safety-alerts. From the API:

curl -X POST "https://api.apify.com/v2/acts/darknezz~canada-recalls-safety-alerts/run-sync-get-dataset-items?token=YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"keyword":"battery","category":"Electronics","dateFrom":"2026-06-01"}'

Or with the Python SDK:

from apify_client import ApifyClient

client = ApifyClient("YOUR_TOKEN")
run = client.actor("darknezz~canada-recalls-safety-alerts").call(
    run_input={"keyword": "battery", "category": "Electronics", "dateFrom": "2026-06-01"},
)
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item["recallId"], item["title"])

Use cases

Compliance monitoring — weekly cron on a category (electronics, toys, food) and alert on new recalls via Telegram or webhook.
Importer / retailer screening — check a product line against active recalls before ordering or listing.
Safety trend analysis — pull all recalls in a category over a window and analyse hazard mix, issuing agency and product types.
Data enrichment — join recall IDs against internal product SKUs for a live exposure report.

Limitations, honestly

HTML scraping — the parser targets stable Drupal machine-names; a site redesign would need a rebuild.
English only — records come from the /en/ surface; French (/fr/) records are not covered.
Archived recalls excluded — matching the reference actor; historical records are not returned.
Politeness pacing — very large result sets (500+ items) take proportionally longer by design.

Why this pattern wins

One source, one job. The Canadian government publishes this data for public consumption; the actor just makes it machine-readable at a predictable price. No third-party aggregator, no licensing ambiguity, no API key to rotate — and because the schema matches the established reference exactly, the actor slots into pipelines that already consume recall data.

The teapot was the only wall. Everything else was parsing.

Try it

👉 Canada Product Recalls & Safety Alerts on Apify Store

I Built a Telegram Members Scraper That Reads Public Chat Stats Without Login

Oaida Adrian — Tue, 11 Aug 2026 22:15:10 +0000

I Built a Telegram Members Scraper That Reads Public Chat Stats Without Login

When I shipped my Telegram Channel Scraper, the same question kept coming back: "can you get the member count too?"

Turns out you can — without a Telegram account, without API keys, without a phone number, and without a proxy budget. So I built Telegram Members Scraper — Group & Channel Stats, and it is free to use on the Apify Store.

The problem

Telegram hosts some of the largest communities on the internet — public groups and channels show their size right on their public info page. Community managers, growth teams and researchers all want the same thing: how big is this community, and is it growing?

The official Telegram API needs api_id / api_hash plus a logged-in account. Telegram deliberately hides member rosters from the unauthenticated web. The result: an oddly empty niche for a very common question.

The extrapolation pattern: reuse the winner

Rather than write a fresh scraper, I copied my proven telegram-channel-scraper codebase and adapted it — the extrapolation play. Same httpx + BeautifulSoup stack, same no-login design philosophy, same one-item-per-chat output shape. The hardest part wasn't the code; it was being honest about what the public web surface can and cannot give you.

What the public web actually exposes (the honest finding)

Telegram does not expose member lists on its public pages. A group's /s/ URL redirects to a bare info card — title and member count only. Channels expose a /s/ post preview, but only the channel itself as author.

So the honest scope of a no-login Telegram scraper is:

Groups → title, description, member count, online count, photo
Channels → title, description, subscriber count, photo, message preview availability

Not per-user rosters. Anyone promising usernames/IDs/join dates without auth is using a different (authenticated) surface. I documented that limitation right in the README instead of burying it.

What it does

No Telegram account needed — reads the public web profile (t.me/<chat>), nothing to authenticate or configure
Groups and channels, one input — detects the chat type automatically and returns the right size metric (members for groups, subscribers for channels)
One dataset item per chat — clean JSON, ready for pipelines
Optional preview probe — for channels, collects visible author accounts from the /s/ post preview

Example output

{
  "chatUsername": "DatascienceChats",
  "chatType": "group",
  "chatTitle": "Data Science,ML & AI Nugget Chats",
  "chatDescription": "Discussion community for DS, ML, IOT, AI, DEEP LEARNING and much more…",
  "memberCount": 13871,
  "onlineCount": 380,
  "subscriberCount": null,
  "photoUrl": "https://cdn4.telegram-cdn.org/file/…",
  "profileUrl": "https://t.me/DatascienceChats",
  "messagePreviewAvailable": false,
  "previewAuthors": [],
  "scrapedAt": "2026-08-10T12:00:00+00:00"
}

Smoke test (before shipping)

Three chats, all public, no login:

Chat	Type	Size
`DatascienceChats`	group	~13.9K members, ~340 online
`durov`	channel	~11.2M subscribers
`python_community`	group	51 members

3/3 extracted non-zero, titles and counts verified against the live pages.

Input

{
  "chats": ["@DatascienceChats", "https://t.me/durov"],
  "checkPreview": true
}

Field	Type	Description
`chats`	array	Public group/channel usernames (`@name`, `name`) or links (`t.me/name`, `t.me/s/name`) — required
`checkPreview`	boolean	Probe the `/s/` message preview for channels and collect visible authors (default `true`)

Use cases

Growth tracking — chart member/subscriber counts of competitor or topic communities over time
Community segmentation — classify public groups vs broadcast channels and their relative sizes
Market research — size the Telegram audience for a niche before launching a channel or product
Lead-list hygiene — confirm a group exists, is public, and roughly how many people it reaches before outreach
AI / datasets — build metadata corpora of public Telegram communities for classification and analytics

Try it

The tool is live and free on the Apify Store: Telegram Members Scraper

Pass a few chat usernames, get back their public profile and membership size as JSON. No account, no API keys, no phone number, no proxy.

Limitations (stated plainly)

Public chats only — private groups/channels are not accessible without authentication
No member roster — Telegram hides member lists from the web; you get the membership size and chat profile, not per-user rows
Counts are as displayed — Telegram rounds large numbers on the public page (e.g. 13.9K → 13900)

If you need an actual member roster, that requires the authenticated Telegram API — a different architecture, legal posture and price point. This tool deliberately stays on the no-login web surface.

More from me

While you're here, these might be worth a read:

Building a WHOIS & DNS Lookup Tool: Domain Intelligence in One...

Oaida Adrian — Thu, 09 Jul 2026 16:23:49 +0000

Why Domain Intelligence Matters

Whether you're doing security research, monitoring your own infrastructure, or evaluating a potential acquisition target, domain data tells a story. WHOIS records reveal ownership history. DNS records expose infrastructure topology. SPF/DMARC configs show email security posture.

The problem? Gathering all this data usually means jumping between multiple tools — whois CLI, dig, nslookup, online MX lookup tools... It's tedious and doesn't scale.

So I built a WHOIS & DNS Lookup Tool on Apify that returns everything in a single API call.

What It Does

Give it a domain (or a list of domains), and you get:

WHOIS Data: Registrar, creation date, expiration date, registrant info, status flags
DNS Records: A, AAAA, MX, NS, TXT, CNAME, SOA, CAA, SRV
Email Security: SPF and DMARC record detection
IP Resolution: Quick A-record IP lookup

All in clean, structured JSON.

Example Output

{
  "domain": "example.com",
  "whois": {
    "registrar": "RESERVED-Internet Assigned Numbers Authority",
    "creationDate": "1995-08-14",
    "expirationDate": "2026-08-13",
    "nameServers": ["a.iana-servers.net", "b.iana-servers.net"]
  },
  "dns": {
    "A": ["93.184.216.34"],
    "MX": ["0 ."],
    "TXT": ["v=spf1 -all"],
    "SOA": ["ns.icann.org. noc.icann.org. 2025081403 7200 3600 1209600 3600"]
  },
  "security": {
    "spf": "v=spf1 -all",
    "dmarc": "v=DMARC1;p=reject;sp=reject;adkim=s;aspf=s"
  },
  "resolvedIP": "93.184.216.34"
}

How It Works

The tool uses two battle-tested Python libraries:

python-whois for WHOIS registration data
dnspython for DNS record resolution

It wraps them in the Apify SDK for cloud execution, automatic dataset storage, and pay-per-event pricing.

SPF/DMARC Detection

Email security is often overlooked in domain analysis. The tool checks TXT records for SPF policies and queries _dmarc.{domain} for DMARC configurations — two critical indicators of email deliverability posture.

Use Cases

Scenario	What You Get
Security Audit	Full DNS topology + WHOIS ownership trail
Domain Monitoring	Track expiry dates and DNS changes over time
Email Deliverability	SPF/DMARC validation for sender domains
Competitive Intel	Infrastructure discovery via DNS records
SEO Analysis	Name server patterns, CDN detection

Try It

You can run the tool directly on Apify:

👉 WHOIS & DNS Lookup Tool

The full source code is available on GitHub:

📦 multi-tool-content-api

Pricing

$0.01 per domain looked up. No subscriptions, no minimums — pure pay-per-use.

What's Next

This is tool #10 in a growing portfolio of content and data extraction APIs. Each one solves a specific data gathering need:

RSS Feed Aggregator
llms.txt Generator
RO Business Scraper
Sitemap Content Extractor
Product Hunt Scraper
Sentiment Analyzer
Property Listing Scraper
Press Release Monitor
AI Web Crawler
WHOIS & DNS Lookup Tool ← this one

All available individually on Apify, or unified through a single Apify endpoint.

Built with Python, Apify SDK, and a passion for clean data APIs.

More from me

While you're here, these might be worth a read:

Building an AI Web Crawler That Outputs LLM-Ready Content Chunks

Oaida Adrian — Wed, 08 Jul 2026 06:34:02 +0000

Building an AI Web Crawler That Outputs LLM-Ready Content Chunks

The biggest challenge in building RAG (Retrieval-Augmented Generation) pipelines isn't the vector database or the embeddings — it's getting clean, structured content from web pages in the first place. Raw HTML is noisy, full of navigation, ads, and boilerplate that pollutes your embeddings.

I built an AI Web Crawler that extracts clean, chunked content from any website — ready to feed directly into LLMs and RAG systems.

The Problem with Web Content for AI

When you feed raw HTML to an LLM, you waste tokens on navigation menus, cookie banners, and footer links. When you feed it poorly cleaned text, retrieval quality drops because the chunks don't align with semantic boundaries.

The solution: a crawler that understands document structure and outputs content chunks designed for AI consumption.

Key Features

Smart Content Extraction

The crawler strips boilerplate (nav, footer, scripts, styles) and extracts only the main content — the actual article, documentation, or page text that matters.

Semantic Chunking

Content is split into chunks that respect heading boundaries:

def chunk_content(text, headings, max_chunk=500):
    chunks = []
    current = []
    for heading in headings:
        section = extract_section(text, heading)
        if len(section) > max_chunk:
            chunks.extend(split_long(section, max_chunk))
        else:
            chunks.append(section)
    return chunks

Token Estimation

Every chunk includes an estimated token count, so you know exactly how much context you're working with before sending to your LLM:

tokens = len(chunk_text) // 4  # rough estimate

Structured Data Extraction

Beyond plain text, the crawler captures:

Headings — document structure/hierarchy
Tables — structured data preserved as markdown
Internal/External Links — relationship mapping
JSON-LD — structured data from schema.org
OpenGraph — social metadata

Use Cases

RAG Pipelines: Feed chunks directly to your vector database
Knowledge Base Building: Crawl documentation sites systematically
Content Monitoring: Track changes on competitor pages
Dataset Creation: Build training data from web sources
AI Agent Tools: Give your autonomous agents a web reader

Try It

Apify Store: AI Web Crawler
GitHub: multi-tool-content-api

The Apify actor uses pay-per-event pricing — you only pay for pages successfully crawled.

Conclusion

The future of web scraping is AI-ready output. By chunking content semantically, estimating tokens, and preserving structure, we bridge the gap between raw web pages and LLM-ready knowledge.

Built with Python, BeautifulSoup4, httpx, and the Apify platform.

More from me

While you're here, these might be worth a read:

Building a Real-Time Press Release Monitor with Python and RSS...

Oaida Adrian — Tue, 07 Jul 2026 06:09:17 +0000

Building a Real-Time Press Release Monitor with Python and RSS Aggregation

Staying on top of company announcements is hard. Press releases are scattered across PRWeb, Business Wire, GlobeNewswire, and dozens of smaller wires. Each has its own RSS feed, and none of them offer keyword filtering out of the box.

I built a Press Release Monitor that aggregates all major PR wires into a single, searchable stream with keyword filtering and full-text extraction.

The Problem

PR professionals, journalists, and analysts need to track company announcements as they happen. But the current workflow is manual: visit each PR wire site, search for your keywords, copy results. Slow and error-prone.

The Solution: Multi-Source RSS Aggregation

The monitor pulls from multiple PR distribution feeds simultaneously:

PRWeb — broad press release distribution
Google News PR Search — catches releases picked up by news aggregators
Dow Jones MarketWatch — financial market announcements

PR_FEEDS = {
    "prweb_all": "https://www.prweb.com/rss2/daily.xml",
    "google_news_pr": "https://news.google.com/rss/search?q=press+release+announces",
    "dowjones_markets": "https://feeds.content.dowjones.io/public/rss/RSSMarketsMain",
}

Keyword Filtering

Once all releases are aggregated, the monitor applies keyword filters:

if keywords:
    text = f"{release['title']} {release['summary']}".lower()
    matching = [r for r in releases if any(kw.lower() in text for kw in keywords)]

This lets you track specific companies, products, or topics across all wires at once.

Full-Text Extraction

For deeper analysis, the monitor optionally visits each release page and extracts the full text using BeautifulSoup:

soup = BeautifulSoup(resp.text, "html.parser")
for tag in soup(["script", "style", "nav", "footer"]):
    tag.decompose()
full_text = soup.get_text(" ")[:5000]

Auto-Detected Company Names

A smart parser extracts the issuing company from each title using common PR naming conventions:

def extract_company(title):
    parts = title.split(":")
    if len(parts) >= 2:
        return parts[0].strip()  # "Company Name: Announces..."
    if " announces " in title.lower():
        return title[:title.lower().index(" announces ")].strip()

Try It

You can use the Press Release Monitor right now:

Apify Store: Press Release Monitor
GitHub: multi-tool-content-api

The Apify actor uses pay-per-event pricing at $0.01 per press release extracted.

Use Cases

Media Monitoring: Track mentions of your company across all PR wires
Competitive Intelligence: Monitor competitor announcements in real-time
Investment Research: Watch for market-moving press releases
Lead Generation: Find companies announcing funding, product launches, or expansion

Conclusion

By aggregating multiple RSS feeds with keyword filtering and full-text extraction, we turned scattered press releases into a single, searchable stream. The future of media monitoring is automated, real-time, and keyword-aware.

Built with Python, feedparser, BeautifulSoup4, httpx, and the Apify platform.

More from me

While you're here, these might be worth a read:

Building a Universal Property Listing Scraper with Python and ...

Oaida Adrian — Mon, 06 Jul 2026 06:15:02 +0000

Building a Universal Property Listing Scraper with Python and JSON-LD

Ever wanted to extract structured data from real estate listings across multiple sites without writing custom parsers for each one? I built a Property Listing Scraper that works with any property website — from Zillow to Rightmove to Imobiliare — using a combination of JSON-LD, OpenGraph metadata, and smart HTML pattern matching.

The Problem

Every real estate site structures its data differently. Zillow uses one schema, Rightmove uses another, and smaller sites like Imobiliare.ro have their own format entirely. Building separate scrapers for each is a maintenance nightmare.

The Solution: Multi-Layer Extraction

The scraper uses three extraction layers, falling back gracefully:

1. JSON-LD (Structured Data)

Many modern property sites embed structured data using Schema.org vocabulary. This is the gold standard — clean, machine-readable, and standardised.

for script in soup.find_all("script", type="application/ld+json"):
    data = json.loads(script.string)
    if "Product" in str(data.get("@type", "")):
        # Extract price, address, coordinates, images

2. OpenGraph Meta Tags

When JSON-LD isn't available, we fall back to OpenGraph metadata that most sites provide for social sharing.

og_title = soup.find("meta", property="og:title")
og_image = soup.find("meta", property="og:image")

3. Regex Pattern Matching

As a final fallback, we scan the page text for common price and property patterns:

price_match = re.search(r"([$£€]\s*[\d,]+(?:\.\d+)?)", text)
bed_match = re.search(r"(\d+)\s*(?:bed|bedroom|camera)", text)

Extracted Data Fields

Each property listing yields:

Price & Currency — parsed from any format (USD, EUR, GBP, RON)
Property Type — apartment, house, studio, land, office
Bedrooms & Bathrooms — multi-language support (English, Italian, Romanian)
Area — square metres detection
Address Components — street, city, region, postal code, country
Coordinates — latitude/longitude from geo data
Images — up to 20 high-quality images per listing

Search Page Support

The scraper automatically detects whether a URL is:

An individual listing — extracts data directly
A search results page — discovers listing links, then scrapes each one concurrently (5 parallel requests)

Try It

You can use the scraper right now:

Apify Store: Property Listing Scraper
GitHub: multi-tool-content-api

The Apify actor uses pay-per-event pricing at $0.01 per property extracted — you only pay for actual results.

Use Cases

Market Analysis: Track property prices across multiple markets
Lead Generation: Build databases of available properties
Price Monitoring: Watch specific neighbourhoods for price changes
Investment Research: Compare yields across countries and currencies

Conclusion

By combining JSON-LD extraction with meta tags and regex fallbacks, we achieve universal compatibility without sacrificing data quality. The scraper handles the messy reality of real estate websites so you can focus on analysing the data.

Built with Python, BeautifulSoup4, httpx, and the Apify platform.

More from me

While you're here, these might be worth a read:

DEV Community: Oaida Adrian

Scraping Romanian Public Contracts: A Native-Language Tender & Awarded Deals Scraper

The gap

The source

The native-language advantage

The smoke test

The honest bits

Try it

More from me

Building a 12-City US Building Permits Scraper With Python

The gap

The reverse-engineering

One schema to rule them all

The smoke test

The honest bits

Try it

More from me

Building an EU Safety Gate (RAPEX) Product Recall Scraper With Python

The gap

The source

The smoke test

The honest bits

Try it

More from me

How I Built a Water Utility Risk Intelligence Tool With Python and MCP

The gap

The scoring engine

Nine keyless upstreams

The smoke test

The honest bits

Try it

More from me

Building an Aviation Hub API: Airports, Airlines, Live Flights & Weather From Six Keyless Sources

The gap

Reverse-engineering

The smoke test

The honest bits

Try it

More from me

I Stopped Scraping Business Directories and Built an MCP Server on Official Registry Data

I Stopped Scraping Business Directories and Built an MCP Server on Official Registry Data

The problem

Why official registry data beats scraping

The pipeline: CKAN → SQLite → MCP

The three tools

1. lookup_business — the registry search

2. extract_contacts — find the humans

3. lookup_domain — verify before you call

Lessons learned

What's next

More from me

I Built a Canada Product Recalls & Safety Alerts Scraper That Reads Open Government Data

I Built a Canada Product Recalls & Safety Alerts Scraper That Reads Open Government Data

The niche, and why build our own

The data source

First contact: the 418

The flow

The schema — reverse-engineered from the reference

Filters

The smoke test

Pitfalls worth naming

Run it

Use cases

Limitations, honestly

Why this pattern wins

Try it

More from me

I Built a Telegram Members Scraper That Reads Public Chat Stats Without Login

I Built a Telegram Members Scraper That Reads Public Chat Stats Without Login

The problem

The extrapolation pattern: reuse the winner

What the public web actually exposes (the honest finding)

What it does

Example output

Smoke test (before shipping)

Input

Use cases

Try it

Limitations (stated plainly)

More from me

1. `lookup_business` — the registry search

2. `extract_contacts` — find the humans

3. `lookup_domain` — verify before you call