"Which grants do I qualify for?" sounds like a search problem. It isn't. We built a pipeline that turns that question into an answer automatically, and the lesson that surprised us most: crawling thousands of sources was the easy 20%. Eligibility matching was the hard 80%.
Here's how the system actually works, and why matching — not scraping — is where the real engineering lives.
The Problem, Restated
"Which programs do I qualify for" is really three separate problems wearing a trench coat:
Discovery — find every program that exists, across federal, state, local, and private sources.
Normalization — turn wildly inconsistent program descriptions into a single structured schema.
Matching — evaluate a specific applicant against each program's eligibility rules and produce a ranked, defensible answer.
Most people who haven't built this assume step 1 is the bottleneck. It's not.
Step 1: Crawl — Necessary, Not Novel
We ingest from thousands of sources: federal agency solicitation pages, SBIR.gov, state economic development portals, foundation RFP pages, PDF notices of funding opportunity (NOFOs), and more. The crawling itself is standard engineering:
Source-specific scrapers where structure is stable (agency APIs, SBIR.gov feeds)
Headless-browser crawlers for JS-heavy state/local portals
PDF extraction pipelines for NOFOs that only exist as scanned or generated documents
Scheduled re-crawls with diffing, since programs open, close, and get amended constantly
This layer is genuinely hard in the "many annoying edge cases" sense — broken pagination, inconsistent PDF layouts, portals that change their DOM weekly. But it's a well-understood category of problem. Throw enough engineering hours at it and it converges. It doesn't require judgment calls about correctness.
Step 2: Normalization — Where the Format Chaos Lives
Every source describes eligibility differently:
One agency writes "small business" and means a specific SBA size standard tied to NAICS code and revenue.
Another writes "startup" and means "founded within the last 5 years," full stop.
A state program says "must be headquartered in-state" — but "headquartered" sometimes means legal entity address, sometimes means where employees physically work.
A foundation RFP embeds eligibility criteria in prose buried in paragraph four of a PDF, with no structured field at all.
We normalize all of this into a common schema — entity size, ownership structure, sector/NAICS mapping, company stage, geography, prior award history, and so on — using a mix of rule-based extraction for structured sources and LLM-assisted extraction for prose-heavy ones, with human-reviewed spot checks on the highest-value programs. The goal isn't just "extract text," it's "extract text into a representation you can actually run logic against."
This is also where source disagreement shows up. Two programs both claim to fund "AI startups" — one requires majority U.S. ownership and under 500 employees, the other has no size cap but requires a university research partner. Normalization has to preserve that specificity instead of flattening it into a generic tag.
Step 3: Matching — The Actual Engineering Problem
This is the part that made us rethink the whole system.
Eligibility isn't a single boolean. It's a set of interacting constraints, several of which are ambiguous, jurisdiction-specific, or genuinely contestable:
Size standards aren't one number — SBA size standards vary by NAICS code, and a company can be "small" under one code and not another.
Ownership rules get complicated fast: majority U.S.-owned and controlled, foreign ownership disclosure thresholds, cap table structure for venture-backed companies.
Sector fit requires mapping a company's actual technology to program scope — "AI applied to diagnostics" needs to match against a program written for "health-related biomedical research," which is a semantic match, not a keyword match.
Stage requirements are often written loosely ("early-stage," "pre-revenue," "seeking to commercialize") and need company-specific interpretation, not just a lookup.
Compound rules stack: a program might require SBA small-business status AND majority ownership by U.S. citizens AND a specific NAICS sector AND no more than one prior Phase II award in that topic area.
We model this as a rules engine, not a single classifier. Each program gets compiled into a structured eligibility graph: hard constraints (disqualifying if violated), soft constraints (reduce fit score but don't disqualify), and ambiguous constraints (flagged for the applicant to confirm rather than silently assumed). A company profile gets evaluated against that graph, and the output isn't just "match" or "no match" — it's a ranked, explainable fit score with the specific reasons attached.
That explainability turned out to be non-negotiable. "You're 73% eligible" is useless to a founder. "You qualify, but this program requires majority U.S. ownership and your cap table shows 40% foreign investment — confirm before applying" is something they can actually act on.
Why Matching Beats Crawling in Difficulty
Three reasons this layer dominates the engineering effort:
Ground truth is inconsistent by nature. Eligibility criteria aren't written by engineers for machine consumption — they're written by program officers for human readers, with all the ambiguity that implies. There's no clean spec to parse against.
Correctness has real stakes. A false positive means a founder wastes weeks writing a proposal they were never eligible for. A false negative means they miss real money. Both failure modes are expensive, so precision matters more than recall in a way that changes the whole design.
The rules change underneath you. NAICS codes get revised. Agencies update size standards. A program that excluded foreign-owned companies last cycle drops that requirement this cycle. Matching logic has to be versioned and re-validated continuously, not built once and left alone.
What We'd Tell Anyone Building Something Similar
If you're building a discovery-plus-matching system in any regulated or rules-heavy domain — grants, benefits, compliance, eligibility of any kind — the lesson generalizes: don't budget engineering time proportional to "how many sources," budget it proportional to "how many distinct rule structures those sources encode." Ten thousand pages that all express the same five eligibility patterns is a much smaller problem than five hundred pages that each express a slightly different one.
Crawling gets you data. Normalization gets you structure. Matching is the part that has to be right — and it's the part nobody budgets enough time for.
Top comments (0)