DEV Community: Oaida Adrian

VAT validation: three answers, not two

Oaida Adrian — Fri, 06 Nov 2026 22:51:59 +0000

A VAT validator that answers "invalid" when the registry is down is worse than one that refuses to answer. It takes a good customer, marks them a compliance risk, and nobody notices until an invoice is queried.

I built one against VIES — the European Commission's own VAT registry — and the interesting part was not the HTTP call. It was the three-way status, and the fact that VIES makes it easy to get wrong.

The endpoint is free and keyless

GET https://ec.europa.eu/taxation_customs/vies/rest-api/ms/{country}/vat/{number}

For RO14399840 it answers:

{
  "isValid": true,
  "userError": "VALID",
  "name": "DANTE INTERNATIONAL SA",
  "address": "MUNICIPIUL BUCUREŞTI, SECTOR 2 …"
}

Note userError: "VALID". The verdict is duplicated across two fields, and they disagree in meaning: isValid is the answer, userError carries a code. My first classifier read userError first, found a non-empty string, and reported every valid number as invalid. The fix is one line — trust isValid when it is true, and only look at userError when it is false:

def classify(payload):
    if payload.get("isValid") is True:
        return {"status": "valid", "reason": None}
    error = str(payload.get("userError") or "").strip().upper()
    if error in UNAVAILABLE or "CONCURRENT" in error:
        return {"status": "unknown", "reason": error}
    return {"status": "invalid", "reason": error or None}

The status that matters is the third one

On the day I wrote this, two member states were not answering:

DE811569869   -> unknown   MS_UNAVAILABLE
FR40303265045 -> unknown   MS_MAX_CONCURRENT_REQ   (then valid, twenty minutes later)

MS_UNAVAILABLE, MS_MAX_CONCURRENT_REQ, GLOBAL_MAX_CONCURRENT_REQ, TIMEOUT are all "I could not answer". A two-state validator folds them into invalid, and there is no way for the caller to tell the difference between "this company is not VAT registered" and "Germany is having a bad afternoon".

So the output carries status: valid | invalid | unknown and the raw reason for the third. unknown rows are never billed and never counted as a change — an outage should not look like movement.

Monitoring beats dumping

The second half is a diff. Take a list of VAT numbers, validate them, and on the next run emit only the ones whose answer changed:

{
  "vatNumber": "RO14399840",
  "status": "valid",
  "name": "DANTE INTERNATIONAL SA",
  "changed": {"status": false, "name": true, "address": false},
  "isChange": true
}

That needs state outside the run. On Apify the default key-value store is per run, so a scheduled run always looks like a first run and never diffs anything — the state has to live in a named store:

store = await Actor.open_key_value_store(name="eu-vat-validator-state")
cached = await store.get_value(f"{country}{number}")   # {ts, payload, row}

A quiet day then costs a handful of calls and produces an empty dataset, which is itself the answer.

The parse trap nobody expects

VAT numbers are two letters plus a number — so the string Dedeman parses cleanly as country DE and number DEMAN. A company name silently becomes a query. Real VAT numbers always carry digits (ATU12345678, ESA1234567, NL123456789B01), so the guard is that at least three digits must survive:

if sum(ch.isdigit() for ch in num) < 3:
    return None

That one is a test now, along with the userError: "VALID" case, so neither can come back.

Numbers

Concurrency 3, small delay: VIES throttles per member state, and a burst earns MS_MAX_CONCURRENT_REQ.
24 h cache per number by default; cacheHours: 0 forces a live check.
13 unit tests, pure logic, no network, run inside the Docker build so a wrong verdict cannot ship.

The validator is published as an Apify actor (darknezz/eu-vat-validator). If you are writing your own, the single most useful thing I can tell you is the one above: VIES answers "valid" with userError: "VALID", and it answers "Germany is down" with isValid: false. Those two must not end up in the same bucket.

Metering an MCP server by answers, not requests: what Leadgen 2.0 changed

Oaida Adrian — Wed, 28 Oct 2026 18:40:22 +0000

Every MCP server that gets listed in a directory learns the same lesson within a week: the most frequent caller is not a customer. It is a crawler doing initialize followed by tools/list, deciding whether you deserve to be indexed, and moving on.

Leadgen 2.0 went live this week. It is not a price change and not a promo — it is the list of things that got in our own way when we used the platform, fixed.

The most interesting one is the metering, because it is a category of bug every MCP server will hit.

Probing should not spend a lookup

Our allowances went up in this release: a free key now gives 50 lookups a day (it was 25), and without a key the MCP server answers 30 calls a day per IP (it was 5), with the REST door counted separately at 5. The two doors never spend each other's allowance, and every limit message quotes the door it came from.

The part that needed fixing was what counts as a call: the counter was incremented on every request to /mcp, including the ones that never reach a tool.

Two days of production logs, same window:

2,333 requests to /mcp          ~100 distinct client IPs
  612  Uptime-Kuma              /health
  390  mcpbeat/0.1              directory crawler
  339  SentinelOracle/0.1       rank probe
  246  curl/7.88.1
  112  mcptry.com indexer
   64  AgentRankProbe/0.1

and, in the usage table: 4 quota rows at 0 calls, 1 real keyed lookup

A crawler doing initialize + tools/list and nothing else was opening the counter. A day full of robots could leave a real visitor without their free lookups — on a shared egress IP, they would even collide with each other.

The rule we landed on: only answers that carry data (or a genuine not-found) count. The fix is a rule, not a heuristic: only answers that carry data (or a genuine not-found) count. Everything else — initialize, tools/list, ping, notifications, input errors, refusals — is handed back. One owner for the counter (record_call()), one function for the give-back (refund_call()), and the enforcement point decides by the shape of the answer.

We deliberately did not key it on the User-Agent. A rule like "don't charge mcpbeat" is a rule you rewrite every month, and it teaches nothing: the next crawler is one header away from looking like a browser. Meter the protocol, not the caller.

Passwordless sign-in that survives mail scanners

Email plus a 6-digit code, in the app and on the site. No password to reset, no "confirm your email" step that fights with link prefetching.

The trick is that the mailed link only opens a page — it never confirms anything by being fetched. The address is confirmed by that page's POST, so a scanner that walks every link in an inbox confirms nothing. Same pattern we use for watchlist alerts: pending → confirmed on a form submit, never on a GET.

An expired or already-used sign-in link now says so in plain words and offers a fresh one to the same address. The API key waits behind the same door: masked, with a copy button, and replaceable without losing plan, usage or watchlists. That last part is a database decision, not a UI one — the key is a row's identity, not the account.

One search box

Company name, registration number (CUI), or website, in the same input. The front end classifies what you typed and routes it; the API keeps its explicit parameters. Users do not read parameter names, and a search box that requires you to know which field you are filling in is a form wearing a search box costume.

Phone numbers, written the way you dial them

Formatted to dial (0744 700 293), with an international column. Each number carries its provenance — declared to ANAF, found on the company's own site, or listed elsewhere — and a number found on a website is rated high only when it matches the one the company declared. In a registry with millions of rows, "we have a number" and "this number is still that company's" are different claims, and the UI should not blur them.

Big lists: compute in the database, not in the page

Revenue and headcount bands are computed in the database, not in the page, so lists of thousands of companies stop being a progress bar. The bands no longer overlap either — a company lands in exactly one.

Developer door and user door

On the site, "Start free" goes to the app and "Get the free key" goes to the developer page — which is what /start now is. The MCP server carries 18 tools, and scripts can still self-serve a key with POST /register. Machines and people have different first clicks; pretending otherwise serves neither.

Try it in one call

Streamable HTTP is a stateful protocol: you initialize, keep the session id, then call. Two requests, no key:

# 1. open a session — the reply header carries Mcp-Session-Id
curl -s -D - https://leadgen-mcp.adrianhomelab.com/mcp \
  -H 'Content-Type: application/json' \
  -H 'Accept: application/json, text/event-stream' \
  -d '{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2025-06-18","capabilities":{},"clientInfo":{"name":"curl","version":"1"}}}'

# 2. call a tool with that id (a real one: this CUI is my own company)
curl -s https://leadgen-mcp.adrianhomelab.com/mcp \
  -H 'Content-Type: application/json' \
  -H 'Accept: application/json, text/event-stream' \
  -H 'Mcp-Session-Id: <id from step 1>' \
  -d '{"jsonrpc":"2.0","id":2,"method":"tools/call",
       "params":{"name":"lookup_business","arguments":{"query":"55685084"}}}'

{"query":"55685084","total":1,"results":[{"companyName":"OAIDA A. IOAN-ADRIAN
 PERSOANĂ FIZICĂ AUTORIZATĂ","cui":"55685084","registrationCode":"F2026045821000",
 "legalForm":"PFA","county":"Bucureşti",...}]}

Eighteen tools sit behind that door — lookup_business, lookup_companies,
find_company_by_domain, lookup_financials, extract_contacts, watch_company,
poll_watchlist, get_usage and the rest. Docs: https://leadgen.adrianhomelab.com/en/docs.

What 2.0 does not fix yet

Being honest about the tail, because these are the parts still on the board:

Authorization: Bearer and X-API-Key both work, but the docs do not say so — a cold caller guesses.
A request for /docs with no Accept-Language still defaults to Romanian. English lives at /en/docs.
The anonymous wall is per source IP, so callers behind one egress share a bucket. That is a design limit, not a bug we can paper over.

If you build an MCP server, the metering item is the one I would copy first. It is a small amount of code and it decides whether your free tier is a loss leader or a denial of service against your own users.

Leadgen serves Romanian company data over the ONRC registry and ANAF filings — 4.2M companies, lookups by CUI, name or website, from Python, curl or an agent. Prices are charged in euro; on the Romanian pages the amount in lei is shown next to them at the National Bank of Romania's reference rate, with the rate's date. Server: https://leadgen-mcp.adrianhomelab.com/mcp.

Romania's company registry, now callable from Claude and Cursor (MCP + REST)

Oaida Adrian — Fri, 16 Oct 2026 21:20:42 +0000

Last month I wrote about building a free API over Romania's company registry — 4.2 million firms from ONRC, plus ANAF data, because there was no free way to look a company up by CUI without opening four government portals.

The part people actually replied about was the MCP server. So this post is the how-to: wiring Romanian company data into Claude, Cursor, or any MCP-capable client in about three minutes.

Why an agent wants this

If you do anything with Romanian companies — B2B sales, invoicing, compliance, KYB, market research — the workflow is always the same shape. Someone hands you a name or a CUI. You go to the registry. You copy fields into a doc. You check whether the firm is still active. You try to find the phone number.

That is exactly the loop an agent should close on its own. The only reason it can't is that the registry speaks Romanian HTML tables and CAPTCHA, not JSON.

The config

Point your client at the streamable HTTP endpoint:

{
  "mcpServers": {
    "leadgen": {
      "url": "https://leadgen-mcp.adrianhomelab.com/mcp"
    }
  }
}

That's it for Cursor and Claude Code. For a client that only reads command-line servers, wrap it with mcp-remote:

npx -y mcp-remote https://leadgen-mcp.adrianhomelab.com/mcp

No key is required to start. Anonymous access is 5 queries a day per IP — enough to try it.

What the agent gets

Sixteen tools, all read-only. The ones I use most:

Tool	For
`lookup_business`	One firm by CUI or name: status, address, CAEN codes, ANAF VAT state
`lookup_companies`	Batch up to 100 CUIs, one row out per row in, same order
`match_companies`	Fuzzy match up to 500 rows (name + county + domain) against the registry
`find_company_by_domain`	Which company owns this website — with SPF/DMARC audit
`lookup_director`	Every firm a person is registered at — useful for due diligence
`lookup_financials`	Turnover, profit, employees, year by year
`watch_company` / `watch_caen`	Monitor a firm, or an entire CAEN activity code, for changes

A real exchange looks like this:

You: which company runs emag.ro, and is its VAT still active?

Agent: (calls find_company_by_domain, then lookup_business) — Dante International SA, CUI 14399840, Bucharest, VAT-active as of this month.

The watch_* pair is the sleeper feature: it turns the registry into a feed. Watch a competitor's CUI and you hear about a director change, a branch opening, or a new CAEN code without polling anything yourself.

If you'd rather not use MCP

Same data over plain REST:

curl "https://leadgen.adrianhomelab.com/lookup_business?query=14399840" \
  -H "X-API-Key: lg_your_key"

Every response carries RateLimit-Limit, RateLimit-Remaining and RateLimit-Reset, and GET /usage tells you where you stand without spending a query. Interactive docs with try-it-out: leadgen.adrianhomelab.com/docs.

The limits, stated plainly

No key: 5 queries/day per IP. Free key from /start: 25/day. Resets 00:00 UTC.
Every endpoint is free during launch. When paid plans arrive, Pro (€9/month) takes the full 2008–2025 financial history, ratios, health scores, revenue/employee/VAT filters, CSV export and lists over 25 rows. The free tier keeps the company profile, status and risk, ANAF data, last-year figures, and lists up to 25 rows. Nothing free today becomes paid without warning.
Batches count per row, and a batch larger than your remaining quota is refused whole rather than half-executed.

Why I'm giving it away

The registry data is public. What isn't public is a sane interface to it, and every Romanian developer has now paid the same tax of scraping HTML tables or buying a €200/month KYB subscription for data that came out of a government PDF.

So the free tier is real, not a trial: it exists because the alternative for most people is not paying me, it's giving up on the lookup. The paid tier is for teams who need depth — a decade of financials, ratios, bulk export.

Getting started

Two minutes, no card:

Get a free key at leadgen.adrianhomelab.com/start
Add the MCP config above, or call the REST endpoint with X-API-Key
Ask your agent about a Romanian company

Landing page (EN): leadgen.adrianhomelab.com/en · MCP server card: leadgen-mcp.adrianhomelab.com

It's a solo project, so if something is missing — a field you need, an endpoint that doesn't exist, a county your data is wrong for — reply here or open an issue from the /feedback page. I read both.

Romania's company registry has no free API — so I built one (4.2M firms, MCP + REST)

Oaida Adrian — Mon, 07 Sep 2026 09:33:43 +0000

Romania's company registry has no free API — so I built one (4.2M firms, MCP + REST)

I run a side hustle building data tools for AI agents. A few weeks ago I needed
Romanian company data — the kind of thing every lead-gen tool wants: who's
registered, their CAEN activity codes, legal representatives, status, address,
website.

Turns out that data has no free API. Romania's official business registry
(ONRC) publishes everything as open data — a monthly CSV dump on
data.gov.ro — but there's no programmatic way to query it. The only access is
commercial scrapers, or scraping the registry website yourself.

So I built the missing API. Here's the whole thing, from a 1.5 GB government
CSV dump to a live MCP server.

The data is already free — it's just not queryable

Romania publishes its entire registry as open data. Every month, data.gov.ro
drops a snapshot: 4,196,860 companies, 19.3M CAEN activity records, 3.68M
legal representatives, 4.64M status entries. For 2025, it also publishes full
financial statements — balance sheets, revenue, headcount, debt.

The problem is ergonomics, not availability. The files are pipe-delimited CSVs,
a gigabyte and a half each, with no index and no API. You can download them,
but then you have a pile of text.

Turning a CSV dump into a queryable registry

The whole pipeline is three loaders:

load_onrc.py — streams the CKAN CSVs straight into SQLite (no pandas loading 1.5 GB into RAM).
load_nomenclatoare.py — pulls the CAEN activity + status code decode tables, so 0125 renders as "cultivation of fruit trees" instead of a number.
load_financiare.py — loads the 2025 financial statements, keyed by CUI (tax code).

Then the important bit: search. Romanian names are full of diacritics —
PAVĂL, ȘTEFAN, ţ. If you search paval, a naive LIKE misses PAVĂL.
So I built an FTS5 full-text index with diacritic-insensitive folding:
ă â î ș ț fold to a a i s t on both the index and the query. Search
popescu and you match Popéscu; search paval and you match PAVĂL.

The result is a 3.5 GB SQLite database that answers "who is this company?" in
milliseconds.

Two surfaces, same data

I exposed it two ways:

A REST API (leadgen.adrianhomelab.com) — plain JSON over HTTP, with Swagger docs. GET /lookup_business?query=DEDEMAN, done.
An MCP server (leadgen-mcp.adrianhomelab.com — endpoint /mcp) — Streamable HTTP, so any MCP-capable agent (Claude Desktop, Cursor, etc.) can query it natively with 16 tools (as of October 2026), among them lookup_business, lookup_companies, match_companies, lookup_by_category, lookup_director, lookup_financials, extract_contacts and lookup_domain. The full list is in the docs.

Both are free: 5 lookups a day with no key at all, or 25 a day with a free API
key from leadgen.adrianhomelab.com/start
— one email address, no payment.

The freemium shape

Registry identity (name, CUI, address, CAEN, directors, status) is free. The
financial statements — revenue, profit, employees, assets, debt — are the
paid tier. The scaffolding is already in the repo (monetization.py), held
behind a single MONETIZATION_ENABLED flag. Flip it and paid tools require a
bearer key; free tools get rate-limited.

The point of the flag: prove agents actually want this before I wire billing.

What I learned

Open data ≠ accessible data. The value isn't the data — it's the diacritic-insensitive search, the join across five tables, and making it one HTTP call away.
MCP is a distribution channel, not just a protocol. Wrapping the same SQLite DB as an MCP server instantly made it usable by any agent. That's the wedge: the registry is one country, but the pattern generalises.
Ship the flag before the billing. Monetization scaffolding costs nothing to build and keeps the door open.

Try it at leadgen.adrianhomelab.com — a free key takes one email.

The data is Romania's — the queryability is the product.

Companion posts in the same build series:

Building a Romanian Business Registry API: 4.2M Companies as Queryable JSON — the REST API deep-dive and the diacritics/FTS5 traps
I Stopped Scraping Business Directories and Built an MCP Server on Official Registry Data — why official open data beats directory scraping

Run it on Apify without hosting anything: Ro Business Data MCP.

Building a Romanian Business Registry API: 4.2M Companies as Queryable JSON

Oaida Adrian — Wed, 26 Aug 2026 11:39:48 +0000

Every Romanian company is registered in the ONRC (the national trade register), and the state publishes the whole thing as open data on data.gov.ro. That's 4.2 million companies, 19.3 million CAEN activity records, 3.68 million legal representatives and 4.64 million status entries — a genuinely useful corpus for lead generation, market analysis and "who actually owns this company" lookups.

The catch: it ships as a pile of enormous CSVs, and the moment you try to query it, Romanian orthography punches you in the face. I built a small free API around it, and this is how.

The source

No scraping, no vendor. The official snapshot is a set of CKAN files, the biggest of which is OD_FIRME.csv at ~690 MB and OD_CAEN_AUTORIZAT.csv at ~425 MB. Combined, raw, that's well over a gigabyte of flat files. Loaded into SQLite with a full-text index it becomes a 3.6 GB database that answers a search in milliseconds.

The build

The load is boring and that's the point: stream the CSVs into SQLite, then create an FTS5 virtual table so lookup_business("dedeman") is a real full-text query rather than a LIKE '%dedeman%' scan. Two tables matter most:

firme — companies, keyed by CUI (the tax identifier) and registration code
reprezentanti_legali — the legal representatives, so you can search by person to find every company a director is attached to

On top of that sits a thin FastAPI wrapper exposing two read-only endpoints — no auth, no keys:

# by company name or CUI
curl "https://leadgen.adrianhomelab.com/lookup_business?query=DEDEMAN&max_results=3"

# by director / legal-representative name
curl "https://leadgen.adrianhomelab.com/lookup_director?name=popescu%20ion"

The traps (this is the real content)

Three things will bite you, in order of pain:

1. Diacritics. Romanian uses ă â î ș ț, and the registry stores them faithfully. Popescu and Popéscu are different strings to a naive matcher — but a human searching for "Popescu" expects both. The FTS index has to be diacritic-insensitive: normalize ș→s, ț→t, ă→a on both the indexed text and the query. Miss this and your "obvious" search returns nothing for half the surnames in the country.

2. CUI vs registration code. Every firm has a CUI (digits, sometimes zero-padded) and a registration code like J1992002621040 (court + year + serial). They are not interchangeable, and newcomers constantly pass one where the API expects the other. Supporting both on a single query parameter means testing numeric input against the CUI column first, then falling back to the registration code.

3. CSV scale and encoding. The raw dumps are ;-delimited and large enough that csv.DictReader in a tight loop will keep you up all night. Stream with csv.reader, batch your executemany inserts, and watch your transaction size — a single 690 MB file has to be committed incrementally or SQLite's WAL balloons.

What came out of it

A live, free endpoint:

GET /lookup_business?query=<name|CUI>&max_results=N
GET /lookup_director?name=<name>&max_results=N
GET /docs — interactive OpenAPI docs
GET /start — free API key: 25 lookups/day for an email address
GET /health

The same data is also available as an MCP server (for AI agents) at leadgen-mcp.adrianhomelab.com — Streamable HTTP at the /mcp path. If you're doing anything with Romanian business data — or you just want a worked example of turning a big government CSV dump into a queryable service — the endpoint is live and free to hit. Try it at leadgen.adrianhomelab.com — a free key takes one email.

Run it on Apify without hosting anything: Ro Business Data MCP.

Building a SEC EDGAR Filings Scraper: 10-K Risk Factors and 13F Holdings as Clean JSON

Oaida Adrian — Wed, 19 Aug 2026 11:11:51 +0000

US public companies file everything with the SEC, and EDGAR is free to use. But the raw archive is hostile to analysis: filings are HTML documents with table-heavy layouts, and the full-text search UI is built for humans, not pipelines. I built an actor that turns a ticker into a clean corpus of JSON records — one per filing — ready for financial alt-data, risk analysis, RAG corpora, or LLM fine-tuning.

The source

No API key, no data vendor. Three official SEC endpoints:

company_tickers.json — ticker to CIK resolution
the EDGAR full-text search JSON API (efts.sec.gov/LATEST/search-index) — query by ticker, form type, and date window
the SEC Archives for the documents themselves

Two traps surfaced immediately. First, SEC fair-access enforcement is real: a bare curl or python-requests User-Agent gets a 403, while a descriptive UA (Darknezz Research admin@…) gets a 200. The actor sets a descriptive UA, paces at ~5 requests/second (well under the 10 rps limit), and backs off on 429s. Second, the ciks search parameter requires the 10-digit zero-padded CIK — 320193 returns zero hits, 0000320193 returns the filing. Padding is mandatory.

The structuring

Raw HTML is only the beginning. Each filing becomes:

Metadata — accession number, ticker, company, CIK, form type, filing date, period ending, 8-K event items, document and index URLs
Sections — split on Item headings (Item 1., Item 1A., Item 7., Item 2.02…), per-section capped so one enormous filing can't blow a record
Risk factors — the full 10-K/10-Q Item 1A text, verified on a real Apple 10-K at the 60k-character cap
Holdings — 13F-HR InfoTable XML parsed into issuer, CUSIP, value, shares, and voting authority. The XML is namespace-prefixed (<ns1:infoTable>), so every regex is namespace-tolerant
Summary — a deterministic, LLM-ready markdown digest: metadata, section map, risk excerpt, holdings preview. No external LLM call, no extra cost

The smoke test

AAPL 8-K ×3 — sections Item 2.02 and 9.01, summaries 476 chars each
AAPL 10-K ×1 — 10+ sections, Item 1A risk factors 60,038 chars, summary 3,267 chars
JPM 13F-HR — 34,064 positions parsed locally; the mega-manager cap (20,000 positions) proven, with holdingsCount and holdingsTruncated flags so the record stays under Apify's item size limit

All cloud runs succeeded with non-zero, well-formed output.

The honest bits

Each filing yields its primary document only; exhibits (press releases, contracts) stay one click away via the filing index URL.
The summary is deterministic, not generative — great for pipelines, not a substitute for an LLM pass.
Mega-manager 13Fs are truncated at 20,000 positions (flagged), because a 9 MB JSON record doesn't ship.

Try it

👉 SEC EDGAR Filings Scraper on Apify Store

Building a Price & Competitor Monitor That Diffs Shopify and AliExpress Prices

Oaida Adrian — Wed, 19 Aug 2026 11:11:47 +0000

Most price watching is a spreadsheet: paste a URL, check it today, forget it tomorrow. The moment you want a history — what did this product cost a month ago, did it dip last Tuesday — snapshots are useless. I built an actor that keeps a real per-URL price timeline, diffs every run against the last one, and pings a webhook the instant something changes.

The design

For each URL the actor:

Fetches the product — Shopify stores expose /products/<handle>.json plus /cart.js for the store currency; AliExpress item pages are client-rendered, so the price comes from the search page matched back by product id
Reads the previous snapshot from a durable key-value store
Computes the diff — NEW, PRICE_CHANGED, AVAILABILITY_CHANGED, or UNCHANGED, with absolute and percentage change
Appends the new point to history, pruning anything older than historyDays
Emits one row per URL with snapshot + diff + recent history, and POSTs a compact JSON alert to your webhook if the row changed

The first run of a URL is the baseline (NEW, no alert). Alerts fire only on real changes after that.

The persistence trap

This is the part that bit: Apify's default key-value store is per-run for API-started runs — it's purged at run start and deleted after the run. A naive implementation would lose its history between runs and re-emit everything as NEW every time. The fix is a named store: Actor.open_key_value_store(name='price-competitor-monitor-history') (the name= argument is keyword-only in SDK 3.4). That store persisted across three cloud runs in testing, which is what turns scheduled runs into a real timeline.

The smoke test

Three URLs — two Shopify stores and an AliExpress dock:

Run 1: allbirds NEW 91.00 USD, colourpop NEW 29.00 USD, non-zero ✓ (the AliExpress item page was x5sec-challenged from the datacenter egress — honest error row, no false price)
Run 2: both Shopify products UNCHANGED — durable history proven in the cloud
Run 3 (with a modified snapshot): colourpop PRICE_CHANGED, previous 36.25 → 29.00, −20%, direction down, alert true ✓

A fetch that fails cleanly (blocked page, deleted product) emits a status: "error" row and leaves the previous snapshot untouched — a transient block never masquerades as a price crash.

The honest bits

AliExpress item pages are fully client-rendered: the price is recovered from the search page by product id, so an item absent from search is reported without a price until it reappears.
Availability on AliExpress isn't reported yet (available: null).
History is capped at 500 points per URL as a safety limit.

Try it

👉 Price & Competitor Monitor on Apify Store

Building a Vertical Corpus Builder: Clean JSONL Datasets for LLM Fine-Tuning

Oaida Adrian — Wed, 19 Aug 2026 11:11:46 +0000

Raw web pages are terrible training data. Nav bars, cookie banners, "related articles" and ads drown the signal, and near-identical syndicated text pollutes the corpus. If you're fine-tuning a domain model — legal reasoning, medical QA, financial analysis — you want clean vertical text with provenance, not a pile of HTML. I built an actor that goes from seed URLs to a token-aware JSONL dataset in one run.

The pipeline

seed URLs → crawl (same-domain BFS) → clean (boilerplate strip)
  → dedupe (exact + near) → token-aware chunk → JSONL with provenance

Three steps matter most:

Boilerplate strip — drop script/style/nav/footer/header/aside and utility blocks (breadcrumbs, shares, comments, menus). One detail that earned its keep: paragraphs where more than half the links are anchor text get dropped too — those are link farms, not content.
Dedupe, exact and near — exact duplicates collapse by normalised hash; near-duplicates by 6-gram Jaccard similarity (default 0.95), so syndicated copies of the same opinion appear once.
Token-aware chunking — chunks split at paragraph boundaries and hard-split on sentence/word boundaries only when a single paragraph exceeds the budget (default 512 tokens). Each chunk stays coherent, which is what fine-tuning actually wants.

Every record carries source (hostname), url, domain (your vertical), title, and chunk_index — so you can filter, cite, or re-weight the corpus later.

The seed-source reality check

Not every legal text source works from a datacenter IP. CourtListener sits behind an AWS WAF JavaScript challenge, and Justia 403s datacenter egress — both unusable as plain-HTTP seeds. Cornell LII (law.cornell.edu/supremecourt/text/…) serves full opinions as clean HTML and works beautifully. The honest move was documenting that in the README and defaulting the examples to LII rather than pretending every source is reachable.

The smoke test

Three LII Supreme Court opinions (Dobbs, Bruen, WV v EPA):

Local: 559 chunks, all 6 provenance keys present, chunk sizes 63–2,054 chars, zero duplicates after global dedupe
Cloud (2 seeds): SUCCEEDED, 443 items, every record with non-empty text, domain=legal, source law.cornell.edu, 0 duplicate texts
KV store: output.jsonl — 443 lines / ~811 KB, plus a SUMMARY record: 443 chunks, 2 pages, ~173,000 estimated tokens, no failed sources

The honest bits

Seed URLs must serve server-rendered HTML. JS-only SPAs and WAF'd sites need a browser-rendering actor instead.
Chunk size is an estimate (~4 chars/token), not a byte-exact tokeniser count.
Near-dedupe is O(n²) in chunk count — cap large crawls with maxChunksPerPage.
The crawler follows same-domain links only; cross-domain citations aren't chased.

Try it

👉 Vertical Corpora Builder on Apify Store

Building a Change & Alert Engine That Webhooks Any Diff on Any URL

Oaida Adrian — Wed, 19 Aug 2026 11:11:44 +0000

Most data jobs re-pull the whole source every run and waste time and money on data that didn't move. A change-detection engine wakes up, checks what actually changed, and pushes only the diff. I built a generic one: point it at any URL — a JSON feed, an RSS/Atom feed, or a plain HTML page — and it emits only the changes, with before and after values, to the dataset and/or your webhook.

How it works

Poll — fetch sourceUrl once per run (the interval is your schedule; the actor is a single-shot poll)
Parse — content is sniffed automatically: JSON arrays used directly; object feeds auto-detect common list keys (items, results, data, records…) or take an itemsPath dot path; RSS/Atom parsed with stable ids (guid or link); HTML becomes one monitored item keyed on a content hash
Diff — each item gets a stable item_id (from idField, natural keys like guid/link/url/title, or a content-hash fallback) and a content hash. New id → added; same id, different hash → modified with full before/after; id missing → removed (only with includeRemovals: true)
Emit — changed items go to the dataset, one per item, plus one batched webhook POST per run that has changes. Nothing is emitted on a clean poll.

The delivery guarantee

At-least-once is the detail I care most about: state is persisted only after a successful webhook POST (or immediately when no webhook is configured). If the POST fails, the run FAILS and the next run re-emits the same changes — an alert is never silently lost. Within a run, duplicate item ids collapse to the last occurrence, so a feed that repeats entries won't double-notify.

The persistence trap

Same lesson as the sibling price monitor: Apify's default key-value store is per-run for API-started runs, so state kept there vanishes between runs and the second run re-emits everything as added. The fix is a named store — Actor.open_key_value_store(name='change-alert-state') (keyword-only name= in SDK 3.4, account-scoped, persists across runs).

The smoke test

BBC News RSS: run 1 → 41 added (real titles and links); run 2 → 0 items, zero changes ✓
Controlled changed source (an Apify dataset with a pre-signed public URL — gist raw CDN served stale content to datacenter IPs, so a dataset is the deterministic route): run A → 2 added; append a modified g2 and a new g3 → run B → modified g2 (price 20 → 25) + added g3; run C → 0 ✓

The honest bits

The engine compares current fetch vs last persisted fetch — items that leave the feed are only reported with includeRemovals: true (off by default, because first-page rotation is often noise).
JSON items without any natural id fall back to a content hash; any edit then looks like an add+remove pair — set idField for clean modified detection.
Feeds with per-item volatile timestamps can look "modified" every poll — list those fields in ignoreFields to hash only what matters.

Try it

👉 Change & Alert Engine on Apify Store

Building a Romania Mall Store Directory: Every Store in 29 Malls, One Schema

Oaida Adrian — Wed, 19 Aug 2026 11:11:43 +0000

Romania's shopping malls publish their store directories, but they publish them in eight different ways: one operator runs a headless CMS, another a WordPress store list, another a Laravel JSON blob. Nobody publishes the whole country. So I built an actor that crawls every major operator and emits every store as one consistent record.

The gap

If you want the store mix of every mall in Romania — for retail analytics, site selection, or leasing research — you'd normally stitch together NEPI's site, AFI's site, Iulius's site, and so on, and still end up with per-operator formats that don't join. Worse, nobody flags the anchor tenants: the hypermarkets, cinemas and department stores that actually drive foot traffic. That's the gap: a country-wide directory with one schema and anchor flags built in.

The sources

Each operator publishes its own public store data, and each needed a different reverse-engineering pass:

NEPI Rockcastle (19 centres, ~2,434 tenants) — their own Strapi CMS JSON API (cms.<site>/api/tenants), including Mega Mall's 215 stores and Promenada's 182
AFI Europe (Cotroceni, Ploiești) — the evx_retailers sitemap
Iulius Group (Cluj, Iași, Suceava, Iulius Town) — a shopping index mixing category and store links, disambiguated by nesting
Sun Plaza, Veranda, Colosseum — WordPress store sitemaps
Băneasa — a Laravel/Inertia data-page JSON payload across 11 pages

The robots.txt lesson deserves a callout: fetching it with Python's default RobotFileParser gets you 403'd by Sun Plaza and Veranda's WAFs — empty rules, everything disallowed. Fetching it with requests and a browser User-Agent works, and the actor then honours the real rules.

The smoke test

Input {"country":"ro","malls":["all"],"includeAnchor":true}:

HTTP 201, 3,868 well-formed records in ~86 seconds
0 empty store names, 0 malformed key sets
382 anchor tenants flagged, 2,495 records with floor/unit, 1,499 with phone
29 of 30 registered malls covered
All 10 identity fields (mall, store, brand, category, url, …) 100% filled

Run it again and the first_seen / last_seen / is_closed fields turn the directory into a vacancy and churn time-series — which stores opened, which closed, which anchors moved.

The honest bits

ParkLake is WAF-gapped from the cloud. Sonae Sierra serves an HTTP 202 JavaScript-challenge page to datacenter IPs for its API, sitemap and archive alike; even curl_cffi impersonation hits the wall. From a residential or proxy IP the REST API returns ~223 stores with full detail. The adapter tries hard, then logs the challenge honestly instead of faking data — attach a residential proxy if you need ParkLake from the cloud.
is_closed only becomes meaningful from the second run — the first run is your baseline.
Fields an operator doesn't publish come back as empty strings, so exports stay clean but sometimes thin.

Try it

👉 Romania Mall Store Directory on Apify Store

Scraping Romanian Public Contracts: A Native-Language Tender & Awarded Deals Scraper

Oaida Adrian — Sun, 16 Aug 2026 19:30:10 +0000

Romania publishes every public tender and awarded contract through e-licitatie.ro — the successor to the old SEAP system. The data is public, the opportunities are real (millions of RON in awards every month), and almost nobody outside Romania is building on it. Being native-language gives us an edge no English-first competitor has.

The gap

The niche's existing actors are thin: the top reference has single-digit runs. Tenders come with deadlines and award values, but most scrapers stop at the listing page. Ours goes deeper — tenders and awarded contracts, with detail enrichment, CPV classification, and winner + CUI (company ID) extraction.

The source

The Angular SPA at e-licitatie.ro calls its own JSON API — but it 403s without the right headers:

POST api-pub/NoticeCommon/GetCNoticeListFiltered/   # tenders
POST api-pub/NoticeCommon/GetCANoticeList/          # awarded contracts

The trick: the API requires a Referer of https://e-licitatie.ro/pub plus Origin and X-Requested-With. Missing referer, instant 403 "Access Denied: Referrer cannot be null". Detail enrichment needs GetSection21View (description, CPV, estimated value — keyed off the item's own sysNoticeTypeId, not the noticeId) and GetSection1View for the contracting authority. Winners come from GetCANoticeContracts — and its filter payload is strict: a bare payload 400s.

The native-language advantage

Romanian titles, Romanian descriptions, CPV codes joined to EU TED. The reference actors are English-first and miss the nuance — e.g., distinguishing licitație deschisă (open tender) from achiziție directă (direct award). Our output carries the original Romanian fields untouched, so downstream users can do their own analysis without a translation layer.

The smoke test

Tenders: keyword infrastructura, dateFrom 2026-01-01 → 5 items, 5/5 complete — real titles, authorities, CPV names, estimated RON values, deadlines, status
Awarded: keyword drumuri, dateFrom 2026-06-01 → 4 items, including winner ROUTE CENTER CONSTRUCT (CUI 29170569) with a contract value of 4,822,401.88 RON and an award date

The honest bits

Authority filtering is client-side containment — the API ignores authority text server-side
Some CAN (award) records show zero value and no winner — that's data-side, the tender was annulled
Publication dates are what the portal publishes; actual contract signatures can lag

Try it

👉 Romanian Public Contracts Scraper on Apify Store

Building a 12-City US Building Permits Scraper With Python

Oaida Adrian — Sun, 16 Aug 2026 19:30:09 +0000

Building permits are the earliest public signal of construction activity — new housing, commercial builds, renovation waves. Every major US city publishes them, but each through a different portal with a different schema. I built one scraper that covers twelve of them with a single output shape.

The gap

The reference actor in this niche covers the big cities but has blind spots: in its own all-cities run it scanned zero rows for Washington DC and never even attempted West Sacramento. Ours covers all twelve: ten Socrata portals (Chicago, NYC, LA, San Francisco, Austin, Seattle, Cincinnati, Mesa, Montgomery County, Baton Rouge) plus two ArcGIS servers (DC, West Sacramento).

The reverse-engineering

Two very different API families to reconcile:

Socrata — the well-documented resource endpoint family, clean JSON
ArcGIS — the old FeatureServer contract, where field names vary per server and some servers reject modern ordering parameters outright

The resilience ladder for the old ArcGIS servers: try TIMESTAMP + orderBy first, fall back to where-only queries, then to unfiltered pulls with client-side date filtering. West Sacramento's server rejects TIMESTAMP entirely — the ladder is what makes it work.

One schema to rule them all

Eighteen canonical keys: city, permit_number, permit_type, status, issue_date, application_date, address, work_description, valuation, contractor_name, applicant, latitude, longitude, source, source_type, source_url, scraped_at, cityKey. NYC's "Permit is not yet issued" placeholder rows get filtered. Adding a city is one entry in the config table.

The smoke test

24/24 non-zero — two runs per city across all twelve, full 18-key schema on every item, plus local unit tests covering ten Socrata normalisations, both ArcGIS servers, and live fetches.

The honest bits

Coverage is US-only and city-scoped — counties and rural areas aren't in this pass
Valuation can be null on older records; some cities simply don't publish it
ArcGIS servers rate-limit; the ladder paces requests per city

Try it

👉 Building Permits Scraper on Apify Store

DEV Community: Oaida Adrian

VAT validation: three answers, not two

The endpoint is free and keyless

The status that matters is the third one

Monitoring beats dumping

The parse trap nobody expects

Numbers

More from me

Metering an MCP server by answers, not requests: what Leadgen 2.0 changed

Probing should not spend a lookup

Passwordless sign-in that survives mail scanners

One search box

Phone numbers, written the way you dial them

Big lists: compute in the database, not in the page

Developer door and user door

Try it in one call

What 2.0 does not fix yet

More from me

Romania's company registry, now callable from Claude and Cursor (MCP + REST)

Why an agent wants this

The config

What the agent gets

If you'd rather not use MCP

The limits, stated plainly

Why I'm giving it away

Getting started

More from me

Romania's company registry has no free API — so I built one (4.2M firms, MCP + REST)

Romania's company registry has no free API — so I built one (4.2M firms, MCP + REST)

The data is already free — it's just not queryable

Turning a CSV dump into a queryable registry

Two surfaces, same data

The freemium shape

What I learned

The data is Romania's — the queryability is the product.

More from me

Building a Romanian Business Registry API: 4.2M Companies as Queryable JSON

The source

The build

The traps (this is the real content)

What came out of it

More from me

Building a SEC EDGAR Filings Scraper: 10-K Risk Factors and 13F Holdings as Clean JSON

The source

The structuring

The smoke test

The honest bits

Try it

More from me

Building a Price & Competitor Monitor That Diffs Shopify and AliExpress Prices

The design

The persistence trap

The smoke test

The honest bits

Try it

More from me

Building a Vertical Corpus Builder: Clean JSONL Datasets for LLM Fine-Tuning

The pipeline

The seed-source reality check

The smoke test

The honest bits

Try it

More from me

Building a Change & Alert Engine That Webhooks Any Diff on Any URL

How it works

The delivery guarantee

The persistence trap

The smoke test

The honest bits

Try it

More from me

Building a Romania Mall Store Directory: Every Store in 29 Malls, One Schema

The gap

The sources

The smoke test

The honest bits

Try it

More from me

Scraping Romanian Public Contracts: A Native-Language Tender & Awarded Deals Scraper

The gap