How I Turned the SAM.gov Public Bulk Dataset Into a Searchable Contractor Database

Bitrixa Limited — Sun, 13 Sep 2026 15:59:08 +0000

Working with public government datasets often sounds straightforward until you actually try to turn them into something useful.

That was my experience with the SAM.gov public bulk dataset.

The data is valuable because it contains a large amount of publicly available information about registered entities and federal contractors. But if your goal is something practical — for example, building a targeted contractor list by NAICS code, geography, small-business status, CAGE availability or other criteria — the workflow can become surprisingly technical.

You can absolutely solve the problem with Python, SQL, a database import, the SAM.gov API, or a commercial GovCon platform.

But I wanted a workflow that was:

local
repeatable
easy to filter
export-friendly
independent of API limits
usable by non-developers

So I built the SAM.gov Public Contractor & NAICS Database Compiler.

The problem

A researcher might want to answer questions like:

Which active contractors operate under NAICS 541512?
Which small businesses are registered in a particular state?
Which entities have a CAGE code?
Which records belong to particular SBA classifications?
Which registrations include websites?
Which contractors match particular PSC codes?
Which results should be treated as individual registrations versus unique entities?

The raw public dataset contains the information, but turning it into a clean research database usually requires some kind of processing layer.

For developers, that might mean writing a reusable script.

For analysts, it might mean Power Query or a database import.

For less technical users, it can mean relying on third-party platforms.

I wanted to reduce that setup.

Local processing instead of an API-first workflow

One of the main design decisions was to process the official public bulk extract locally.

That means the workflow does not depend on repeatedly calling an external API just to filter data that already exists in a downloadable public dataset.

The basic process is:

SAM.gov public bulk extract
        ↓
Local compiler
        ↓
Apply filters
        ↓
Select output structure
        ↓
Excel / CSV

This approach has a few advantages.

There is no API dependency during processing.

There is no need to upload the source dataset to another service.

And once the source file is available locally, the user can run different searches against it repeatedly.

Filters

The current compiler supports filtering around fields that are useful in contractor research, including:

registration status
NAICS codes
primary vs any declared NAICS
state/province
country
small-business indicator
business type
SBA classifications
Product or Service Codes
CAGE availability
entity website availability

There are also different research modes depending on whether someone wants a guided workflow or more control over the criteria.

Entity-level vs registration-level output

This was one of the more interesting parts of the workflow.

Sometimes you want every matching registration/CAGE variation.

Other times, you really want one result per unique entity.

So the compiler can produce either:

All matching registration records

or:

One row per Unique Entity Identifier

That distinction can significantly change the usefulness of an exported contractor list.

Excel and CSV output

The tool can export:

Excel
CSV
both formats

There is also a choice between a smaller set of curated research fields and a broader output based on the supported public source fields.

For business users, Excel is usually convenient.

For developers or larger downstream workflows, CSV often makes more sense.

Why I included a Python version as well

The product includes a standalone Windows EXE, but I also wanted a Python/BAT version available.

The EXE is aimed at users who do not want to set up a programming environment.

The Python version is more useful for technical users who prefer a script-driven workflow or want to understand the processing approach more directly.

I like the idea of the same product being usable by both:

Analyst → guided Windows workflow
Developer → Python-based workflow

Why not just use SAM.gov search?

SAM.gov itself is obviously the authoritative source and is excellent for many individual searches.

The use case here is different.

If you need to produce a repeatable dataset involving potentially hundreds or thousands of matching entities, bulk processing becomes useful.

For example:

Find active small businesses in California matching several NAICS codes, requiring a CAGE code and website, then export the results.

That is more of a data-processing problem than an individual record lookup.

Why not use a commercial GovCon platform?

There are excellent commercial platforms that provide far more functionality than this tool.

They may include opportunity intelligence, bid tracking, historical awards, competitive analysis, alerts and CRM-style features.

I wasn't trying to reproduce that.

The idea was much narrower:

Give people a practical way to process the underlying public contractor dataset without requiring them to build their own data pipeline or take on another recurring subscription.

Performance considerations

Large public datasets create some basic engineering decisions that don't matter with small CSV files.

For example:

avoiding unnecessary repeated parsing
applying filters efficiently
limiting memory overhead
handling Excel row limits
deciding when CSV is the better output format
preserving identifiers correctly
dealing with blank or inconsistent classification fields
avoiding accidental duplication at entity level

Even when the final interface is simple, those details matter if the software is expected to process large datasets reliably.

The commercial version

I eventually packaged the workflow into a commercial product:

SAM.gov Public Contractor & NAICS Database Compiler

https://www.bitrixa.com/product/sam-gov-contractor-database-compiler/

It is a one-time purchase rather than a monthly subscription, and includes both the Windows EXE and Python/BAT editions.

The underlying contractor information still comes from the official SAM.gov public dataset. The software is the processing and filtering layer around that data.

What I’m considering next

A few features I’m thinking about for future versions:

SQLite export
DuckDB export
saved filter profiles
incremental monthly updates
command-line automation
more advanced combinations of NAICS/PSC criteria
reusable research presets
summary statistics before export

I’m particularly interested in the SQLite/DuckDB question.

For developers working with large public datasets, would you rather receive:

CSV
SQLite
DuckDB
Parquet

instead of Excel?

And if you regularly work with SAM.gov or other government bulk datasets, what part of the workflow usually causes the most friction?

Interested to hear how other developers and data people approach this.

DEV Community: Bitrixa Limited