If you've ever tried to build a tool that surfaces grants, SBIR awards, or other government funding opportunities, you've hit the same wall everyone hits: the crawl is the easy part. Normalizing what you crawl is the actual job.
The core problem
Government funding data isn't one dataset — it's dozens of agencies, each publishing in whatever format their internal tooling happens to produce. Two functionally identical programs — same funding mechanism, same eligibility rules, same review process — can show up looking nothing alike:
One agency posts a clean JSON feed with typed fields.
Another exports an HTML table that changes column order every quarter.
A third buries the actual solicitation in a linked PDF, and the "structured" listing just has a title and a link.
There's no shared schema across sources, so step one of any pipeline is building a canonical model before you've normalized a single record.
Eligibility lives in prose, not fields
This is the part that breaks naive scrapers. Eligibility criteria — company size, revenue caps, citizenship requirements, prior-award restrictions — usually isn't a field you can extract. It's a paragraph inside a PDF, written in legal/regulatory language, often referencing a different PDF (an agency-wide policy doc) by section number instead of restating the rule.
Practical implication: you can't treat eligibility as a boolean you scrape. You need either an LLM-assisted extraction step with a human-reviewable confidence score, or you accept that eligibility is advisory text you surface as-is rather than a filter you compute on.
Deadlines: high value, low quality
Deadlines are the single field users care about most and the field agencies are worst at expressing consistently:
Static dates ("proposals due March 15, 2026")
Rolling deadlines ("reviewed on a rolling basis")
Cycle-based dates ("first Friday of each quarter")
Relative dates buried in a PDF ("within 90 days of the pre-application")
Silent deadline changes — the page updates, but nothing marks it as revised
A pipeline that just regexes for date patterns will produce garbage silently. You need a deadline model that can represent "unknown," "recurring," and "relative-to-X" as first-class states — not just null when parsing fails.
What actually works
A few patterns hold up in production:
Separate ingestion from normalization. Store the raw source (HTML, PDF, JSON) untouched, then run normalization as its own versioned step. When an agency changes its format, you re-run normalization, not the whole crawl.
Treat PDFs as a first-class source, not an edge case. If a meaningful share of your programs live behind PDFs, your architecture should assume PDF extraction from day one, not bolt it on later.
Score confidence, don't binary-classify. For fields like eligibility and deadlines, store an extraction confidence alongside the value so downstream consumers (including end users) know when to double-check.
Diff, don't just overwrite. Funding pages change without notice. Snapshotting and diffing lets you detect a moved deadline or narrowed eligibility window instead of silently serving stale data.
Keep a human in the loop for ambiguous cases. Full automation on eligibility parsing will get you a plausible-looking wrong answer often enough to matter. A lightweight review queue for low-confidence extractions is cheaper than the trust you lose from bad data.
The takeaway
Crawling government funding sites is a solved problem — it's HTTP requests and HTML parsing. The actual engineering work is building a data model expressive enough to hold "we don't really know" as a valid state, because for eligibility and deadlines, that's the honest answer more often than any dataset will admit.
Top comments (0)