DEV Community

Cover image for How we built a crawler for thousands of funding sources
Fundn A.I
Fundn A.I

Posted on

How we built a crawler for thousands of funding sources

When we started building Fundn, we thought the hardest part would be the product itself.

It wasn't.

The UI was relatively straightforward. The difficult part was building the data infrastructure behind it.

Non-dilutive funding is scattered across thousands of different sources: federal portals, state programs, agency SBIR pages, foundations, universities, nonprofits, banks, and corporate programs.

There is no single database containing all of this information.

So we had to build one.

**
The data problem**

A funding opportunity might look simple on the surface:

"Small businesses in the technology sector can apply for up to $100,000."

But turning that sentence into structured data is surprisingly difficult.

To determine whether a company actually qualifies, we may need to know:

Industry or sector
Company stage
Number of employees
Annual revenue
Location
Ownership structure
Whether the company is for-profit or nonprofit
Whether it is minority-, women-, or veteran-owned
Whether it has previously received certain types of funding
Whether the applicant is a specific type of organization
Whether a Principal Investigator (PI) meets employment requirements
Whether the company meets agency-specific definitions of "small business"

And those requirements aren't standardized.

One program might say "U.S. small business."

Another might reference an SBA size standard.

Another might require the applicant to have fewer than 500 employees.

They can all sound similar while representing very different eligibility rules.

Building the crawler

The first challenge was collecting the data.

We built a crawler that continuously discovers and monitors funding sources across the web.

It needs to deal with a huge variety of websites.

Some sources are well structured. Others aren't.

Some have dedicated funding databases. Others publish opportunities as individual pages. Some rely heavily on PDFs or documents. Others change their page structure regularly.

This means the crawler can't assume that every website behaves the same way.

We built the system to identify funding opportunities, extract the relevant information, and normalize it into a consistent format that the rest of Fundn can work with.

But crawling is only the beginning.

**
The real challenge: understanding eligibility**

This is where most of the complexity lives.

Imagine we find a $250,000 grant for technology companies.

At first glance, it looks relevant to a software startup.

But after looking at the eligibility requirements, we might discover that the program is only available to companies with fewer than 50 employees, majority U.S. ownership, and a specific type of research activity.

Now the question isn't:

"Is this a technology grant?"

It's:

"Does this particular company satisfy all of the requirements?"

That requires understanding both sides of the problem.

We need to understand the funding opportunity, but we also need a structured understanding of the company.

The matching layer connects the two.

From search to matching

A traditional search engine might match a company with a funding opportunity because both mention "climate technology."

That's not enough.

We want to determine whether the company actually qualifies.

For every opportunity, we try to understand the requirements and turn them into structured eligibility criteria.

Then we compare those criteria against the company's profile.

Some matches are straightforward.

Others are conditional.

And some require information we don't have yet.

That's why we don't think about eligibility as simply "yes" or "no."

Sometimes the correct answer is:

"This looks like a strong match, but we need to know one more thing."

That distinction is important.

We'd rather tell a company that we need more information than confidently recommend a program it can't apply for.

Keeping the data fresh

There is another problem: funding opportunities constantly change.

Deadlines pass.

Programs reopen.

Funding amounts change.

Eligibility requirements get updated.

Programs disappear.

A database that was accurate three months ago can already be outdated.

So the crawler isn't a one-time data collection system.

It's an ongoing process.

We continuously revisit sources, detect changes, update existing opportunities, and remove or mark programs that are no longer active.

This is especially important for deadlines.

A great funding opportunity with an expired deadline isn't useful to anyone.

What surprised us

The biggest lesson from building Fundn was that the crawler isn't actually the hardest part.

Collecting thousands of funding opportunities is a difficult engineering problem, but it's a fairly well-defined one.

The much harder problem is understanding what those opportunities actually mean.

"Small business."

"U.S.-based company."

"Technology company."

"Qualified researcher."

"Majority-owned."

These phrases sound simple until you have to turn them into rules that a computer can evaluate consistently.

That's where most of the interesting engineering work happens.

Where we're going next

We're continuing to improve both sides of the system: finding more funding sources and getting better at determining which opportunities are actually relevant to a company.

The long-term goal isn't to build another directory of grants.

It's to make funding discovery feel more like a personalized search engine:

Tell us about your company, and we'll figure out which funding opportunities are worth your time.

The web crawling gets us the raw material.

The matching layer is what makes that data useful.

Top comments (0)