DEV Community

Cover image for I Scraped 40,000 Government Grants into Clean JSON. Here's What Broke.
Gala
Gala

Posted on

I Scraped 40,000 Government Grants into Clean JSON. Here's What Broke.

There are billions of euros in European NextGen funds and regional government grants sitting in Spain's National Subsidies Database (BDNS).

Finding them through the official portal, however, feels like navigating an intranet built in 2004 that actively wants you to give up.

Session timeouts every twelve minutes. Mystery search filters that return zero results if you misplace an accent mark. And tabular views where a single grant spans three nested IFRAMEs.

I wanted to make this data actually searchable for humans. So I set out to scrape the entire portal, normalize 40,000+ public grants into clean JSON, and serve them on a lightweight web interface: Subvenciones BDNS.

Naturally, almost every assumption I had about public web data immediately blew up in production.

Here is what broke, why modern APIs still struggle with legacy government systems, and how I built a declarative parsing engine in Python and PHP that runs reliably without constant maintenance.

TL;DR:

  • Public sector databases are not designed like REST APIs: they mix ISO-8859-1 encodings, non-standard dates, and decimal commas.
  • Writing custom procedural scrapers for each portal creates a brittle maintenance nightmare.
  • I replaced imperative scraping scripts with a declarative JSON specification that handles retries, CSS extraction, regex transformations, and entity deduplication.
  • We parsed 40,000+ grants down to structured feeds, but not before hitting five distinct failure modes.

Clean searchable portal built on normalized government grant data
The result: a minimalist directory parsing 40k+ grants into indexed hubs by province, sector, and beneficiary.

Mistake 1: Trusting the HTTP Content-Type Header

Have you ever seen an HTTP header claim Content-Type: text/html; charset=utf-8, only to discover that the payload is secretly encoded in ISO-8859-1?

Government servers in Southern Europe love doing this.

When parsing Spanish administrative bodies—especially regional entities in Galicia, Catalonia, or Andalusia—characters like ñ, á, and ç frequently turned into mojibake (ñ, €).

If your scraper naïvely trusts the response header or defaults to UTF-8 without byte-sniffing, you corrupt your database silently. You don't get a 500 error; you just get a database full of garbage strings like "Subvención para pymes".

# The quick fix: inspect raw byte markers before decoding
def normalize_encoding(raw_bytes: bytes, header_encoding: str) -> str:
    # Many portals declare utf-8 but serve windows-1252 or iso-8859-1
    if b'\xc3\xa9' in raw_bytes or b'\xc3\xb3' in raw_bytes:
        return raw_bytes.decode('utf-8', errors='replace')
    try:
        return raw_bytes.decode('utf-8')
    except UnicodeDecodeError:
        return raw_bytes.decode('iso-8859-1', errors='ignore')
Enter fullscreen mode Exit fullscreen mode

Never assume the server knows what encoding it is emitting. Always inspect the raw bytes.

Mistake 2: The "Decimal Comma" Trap

If you scrape data from US or UK portals, numerical amounts look like 150000.50.

In Spain and throughout Continental Europe, thousands are separated by dots and decimals by commas: 150.000,50 €.

Worse, raw government tables often contain inconsistent spacing:

  • 150.000,00 €
  • 150 000,00EUR
  • 150.000 € (Sin IVA)
  • A determinar (undetermined)

If your parser casts this with float(val.replace('€', '')), your pipeline will crash on the 43rd record. Or worse, 150.000 gets parsed as 150.0. Imagine telling an applicant their €150,000 grant is worth 150 bucks.

To solve this across thousands of disparate fields, I moved away from bespoke regex strings in Python and created a declarative transformer in our scraping engine:

{
  "presupuesto_raw": "td.importe",
  "presupuesto": {
    "path": "td.importe",
    "transform": "decimal_comma"
  }
}
Enter fullscreen mode Exit fullscreen mode

The decimal_comma transformer strips currency symbols, removes thousand-separator dots, swaps commas for points, and returns null if the amount is pending. Simple, testable, and zero regex leakage in business logic.

Mistake 3: Pagination Queues that Disappear Mid-Crawl

How do you paginate through 40,000 items on a portal where page 150 times out after 60 seconds?

Legacy stateful portals often track sessions in cookies. If you request page 1, the backend stores a cursor in its memory. If you jump to page 120, the database performs a full table scan and drops your connection.

When I started, I tried a simple sequential loop:

Page 1 -> Page 2 -> Page 3 ... -> Page 184 (Timeout / 504) -> Pipeline dies.
Enter fullscreen mode Exit fullscreen mode

If your scraper crashes at page 184, do you start over from page 1? If you do, you risk getting rate-limited or banned before you ever reach page 185.

The solution was two-fold:

  1. Incremental ingestion by date: Instead of asking for all 40,000 grants in one run, we query date slices using dynamic URL placeholders ({date:YYYY-MM-DD}). Each day's batch contains 80 to 200 items.
  2. Key-based deduplication: We define a primary entity key (convocatoria_id) and an update fingerprint. If the scraper sees an ID that already exists with the same hash, it halts pagination early (stopAtKnown).
{
  "url": "https://sede.gob.es/convocatorias?desde={date-7:YYYY-MM-DD}",
  "list": {
    "dedupeBy": "convocatoria_id",
    "stopAtKnown": "convocatoria_id"
  }
}
Enter fullscreen mode Exit fullscreen mode

Now, instead of a 4-hour crawl that fails 80% of the time, our worker runs in under 45 seconds every morning.

Detail view showing extracted legal basis, deadline, and normalized budget
Normalized grant view: clean deadline badges, computed days remaining, and direct links to the official bulletin.

Mistake 4: Imperative Code vs Declarative Configs

In my early projects, every new scraper was its own 300-line Python script with BeautifulSoup, custom loops, and bespoke error handling.

By the time I had six scrapers running across different regional portals, maintaining them was a nightmare. A CSS class changed on a regional bulletin? Go find the script, write a test, fix the selector, commit, and deploy.

I finally threw out the imperative scripts and built a unified engine (scrapper) configured entirely via JSON files.

Every scraper configuration defines four things:

  1. Source & engine: HTTP or headless Chromium for SPA pages.
  2. List extraction: CSS/XPath selectors and field mappings.
  3. Derived fields: Slugs, date formatting, and canonical lookups.
  4. Collection settings: Retention windows and change detection keys.

Here is an actual snippet of how we derive geographic slugs safely:

"derived": {
  "provincia_slug": {
    "from": "provincia",
    "map": {
      "ARABA/ÁLAVA": "alava",
      "BALEARS (ILLES)": "illes-balears",
      "BIZKAIA": "vizcaya"
    },
    "default": "estatal"
  }
}
Enter fullscreen mode Exit fullscreen mode

When an official portal updates its markup, I don't touch any Python or PHP code. I edit a single JSON mapping, run our test suite (249 tests passing in scrapper), and apply it via API.

Mistake 5: The Danger of Dead External Links

A government grant without a working link to the official gazette (BOE, DOGC, BOCM) is useless.

Many portals generate links using JavaScript postbacks: <a href="javascript:doSubmit('009482')">.

If your scraper blindly saves javascript:..., your frontend users hit dead ends.

We had to add an automated resolution step that transforms portal postback IDs into canonical permalinks:

def resolve_gazette_url(postback_arg: str, gazette_code: str) -> str:
    # Map internal IDs to durable government permalinks
    if gazette_code == "BOE":
        return f"https://www.boe.es/diario_boe/xml.php?id={postback_arg}"
    if gazette_code == "BDNS":
        return f"https://www.infosubvenciones.es/bdnstrans/GE/es/convocatoria/{postback_arg}"
    return ""
Enter fullscreen mode Exit fullscreen mode

In addition, because public contracts and grants frequently overlap, we built automated crosslinks connecting related subsidies to relevant public tenders on Licitaciones Alerta. If a company qualifies for an innovation grant, they are likely looking for municipal tech contracts as well.

What I Learned After 40,000 Records

Building scrapers for legacy government infrastructure taught me one fundamental lesson:

The real challenge of web scraping is not extracting HTML. It is handling degradation gracefully.

Portals will go down with 503s on Sunday nights. Encodings will flip randomly between UTF-8 and ISO-8859-1. Table layouts will change without warning.

If your architecture is declarative and resilient to empty runs, unexpected formats won't crash your server:

  • Decouple your extraction rules into declarative configs so you can adapt in seconds.
  • Treat every date, number, and text string as untrusted data until it passes a strict normalizer.
  • Use incremental date slicing instead of massive sequential crawls.

What is the weirdest encoding or pagination bug you have ever encountered in a public dataset?


I write these guides with help from AI agents and verify every number, test, and code snippet myself against production systems.

Top comments (0)