DEV Community: CBLU2005

Federal contracts: see who actually WON — post-award intel from USAspending's free API

CBLU2005 — Wed, 12 Aug 2026 18:04:21 +0000

SAM.gov tells you what the government wants to buy (I covered tracking it without an API key earlier). But the pre-award side is only half the picture. The other half — the half your competitors' business developers are quietly living in — is who won, for how much, from which agency, and when that contract runs out.

That data is USAspending.gov: every federal contract, grant, loan, and direct payment, published under the DATA Act. It answers questions SAM.gov can't:

Competitive intel — who actually wins IT-services contracts in Texas? At what dollar sizes? For which agencies?
Recompete pipeline — an award with an end date 9 months out is a solicitation waiting to happen. Incumbents know. Now you do too.
Teaming targets — small primes winning work in your NAICS are potential partners (or acquisition targets).
Grant prospecting — universities and nonprofits: see exactly which programs fund organizations like yours, and how much.

And unlike SAM.gov's official API, USAspending's API needs no key, no account, no registration. It's genuinely open. Here's the honest DIY, where it bites, and the shortcut.

The DIY: one POST endpoint does most of it

The workhorse is spending_by_award — POST your filters, get awards back:

curl -s -X POST "https://api.usaspending.gov/api/v2/search/spending_by_award/" \
  -H "Content-Type: application/json" \
  -d '{
    "filters": {
      "award_type_codes": ["A", "B", "C", "D"],
      "time_period": [{"start_date": "2026-05-01", "end_date": "2026-08-12"}],
      "naics_codes": ["541511"],
      "place_of_performance_locations": [{"country": "USA", "state": "TX"}]
    },
    "fields": ["Award ID", "Recipient Name", "Award Amount", "Awarding Agency",
               "Start Date", "End Date", "Description", "generated_internal_id"],
    "sort": "Award Amount", "order": "desc", "limit": 25, "page": 1
  }'

That call works today, unauthenticated (custom-programming contracts performed in Texas, biggest first — the top hit as I write this is a $37.7M DoD award). Every result's generated_internal_id gives you a shareable page: https://www.usaspending.gov/award/<id>.

Where it bites

The API is free and stable — and its query grammar was clearly designed for USAspending's own frontend, not for you:

award_type_codes is mandatory and cryptic. Contracts are A–D, contract IDVs are IDV_A–IDV_E, grants are 02/03/04/05, direct payments 06/10, loans 07/08. Send an empty filter set and you get a 422, not "everything."
Contracts and grants can't be queried together properly. Several filters (naics_codes, PSC) only apply to contracts; CFDA program numbers only to assistance. Real coverage means separate queries per award group, merged and deduplicated yourself.
fields are magic display strings. You request columns by their UI names — "Recipient Name", "Award Amount" — and misspellings just silently vanish from the response. Contract and grant responses also name overlapping things differently (Award Type vs Contract Award Type).
The good stuff needs a second call per award. Recipient mailing address, business categories, executive compensation — all on the award detail endpoint (/api/v2/awards/<generated_internal_id>/), one round trip each. A 500-award pull becomes 501 requests with polite pacing.
Occasional nested surprises — a field that's a string on one award is an object on the next. Your flattener finds out in production.

None of this is a dealbreaker; it's a long afternoon plus perpetual maintenance. If you enjoy that afternoon, the official docs are decent.

The shortcut: filters in, flat records out

I maintain an Apify actor that wraps all of the above — award-group fan-out, field normalization, optional per-award enrichment, paging, dedupe:

USAspending Federal Awards Scraper

{
    "awardTypes": ["contracts"],
    "naicsCodes": ["541511"],
    "placeOfPerformanceStates": ["TX"],
    "timePeriodStart": "2026-05-01",
    "minAwardAmount": 250000,
    "includeRecipientAddress": true,
    "maxResults": 500
}

Every record comes out flat and consistently named — recipientName, recipientUei, awardAmount, awardingAgency, startDate, endDate, naics, usaspendingUrl, plus the recipient's address and business categories when you flip includeRecipientAddress — with JSON/CSV/Excel export and pricing per record returned (a 500-award competitive-landscape pull costs about a dollar and a half).

Grants work the same way — swap awardTypes to ["grants"] and filter by cfdaNumbers (e.g. 93.778 for Medicaid) or awardingAgency.

The play that pays: pre-award + post-award together

The two datasets are a pincer:

USAspending (this post): find every contract in your NAICS ending in the next 12 months. That's your recompete target list, with the incumbent, the agency, and the dollar value attached.
SAM.gov: watch for the recompete solicitation to drop — with the contracting officer's name and email on it.

Schedule both weekly on Apify Schedules and you've rebuilt the core of a $300/month GovCon intelligence subscription for a few dollars of data charges.

Recap

USAspending.gov's API is free, keyless, and covers every federal award — the post-award half of GovCon intel.
DIY is one POST endpoint plus real friction: cryptic mandatory type codes, split contract/grant querying, magic field names, N+1 enrichment.
The actor flattens all of it behind one input; pair it with SAM.gov monitoring for the full pre-award + post-award picture.

If there's a specific slice you're trying to pull — a NAICS, an agency, a grant program — drop a comment and I'll sketch the exact input.

Verify Florida contractor licenses in bulk — DBPR's 46 MB mystery CSV, decoded

CBLU2005 — Wed, 12 Aug 2026 18:00:12 +0000

If your business touches Florida construction — insurance, lending, building materials, a marketplace, a permit-pulling SaaS — sooner or later you need to answer, at scale: is this contractor actually licensed, and is that license current, active, and not about to expire?

Florida makes this genuinely answerable. Unlike states with no statewide general-contractor license (looking at you, Texas), Florida's DBPR licenses contractors statewide and publishes the entire license roll as public records. The catch: the official ways to read it are either painfully slow or painfully cryptic. This post shows both DIY routes honestly, then the shortcut, and finally how to turn a one-off check into standing compliance alerts.

Route 1: the search portal (fine for one, hopeless for a list)

DBPR's Verify a License portal works one search at a time: pick a board, type a name or license number, click through to a detail page. For a single sub you're about to hire, perfect. For the 400 subcontractors on your book, that's an afternoon of copy-paste — every week, if you care about status changes. Nobody does this twice.

Route 2: the bulk extract files (free, complete, and user-hostile)

The real data lives on DBPR's instant public records page: full statewide extract files per profession, regenerated on a daily/weekly cadence. The construction file is one URL away:

curl -O "https://www2.myfloridalicense.com/sto/file_download/extracts/CONSTRUCTIONLICENSE_1.csv"

That's every licensed contractor in Florida — 46 MB of it (the cosmetology file is ~74 MB). Now the friction starts:

No header row. The file opens with data. There are 22 columns and DBPR doesn't label them in the file.
Everything is coded. Counties are numbers (23 = Miami-Dade, 60 = Palm Beach — the mapping is on DBPR's "understanding DBPR codes" page). Statuses are letters: primary C/P/S/N/D = Current / Probation / Suspended / Null-and-Void / Delinquent, secondary A/I = Active / Inactive. A usable license is C + A — most rows aren't.
Two license-number columns. Column 13 is a bare sequence (0015061); the full number you actually want (CBC015061) is column 21.
Rows that aren't licenses. Qualified-business entries ride along with no license number and have to be skipped.
The CDN blocks datacenter IPs. The download works from your laptop and then 403s from AWS/GitHub Actions — precisely where your cron job lives. (Ask me how I know.)
What's missing matters. DBPR excludes null-and-void, delinquent, and involuntarily-inactive licenses from the extracts. A contractor disappearing from the file is itself a signal.

Here's a working decoder for the construction file (tested against the live extract):

import pandas as pd

# 22 columns, no header row - mapping reverse-engineered from DBPR's code tables
COLS = ["board", "license_type", "licensee_name", "dba_name", "_4",
        "addr1", "addr2", "addr3", "city", "state", "zip", "county_code",
        "license_seq", "primary_status", "secondary_status",
        "original_license_date", "status_effective_date", "expiration_date",
        "_18", "_19", "license_number", "notes"]

PRIMARY = {"C": "Current", "P": "Probation", "S": "Suspended",
           "N": "Null and Void", "D": "Delinquent"}
SECONDARY = {"A": "Active", "I": "Inactive"}

df = pd.read_csv("CONSTRUCTIONLICENSE_1.csv", names=COLS, dtype=str,
                 keep_default_na=False)
df["primary_status"] = df["primary_status"].map(PRIMARY)
df["secondary_status"] = df["secondary_status"].map(SECONDARY)

# Every active certified general contractor in Palm Beach County (code 60):
cgc = df[(df.license_type == "CGC") & (df.county_code == "60")
         & (df.primary_status == "Current") & (df.secondary_status == "Active")]
print(len(cgc), "active Palm Beach CGCs")

To verify your list, inner-join it on license_number and flag anything that isn't Current + Active — plus anything from your list that's not in the file at all (see point 6).

If you only need this once, the DIY route is genuinely fine. The pain compounds when you need it fresh: re-downloading 46-74 MB files, babysitting the 403s, re-checking the column layout, decoding county/status/type codes for eight different boards (electrical, home inspectors, mold, cosmetology... each its own file).

The shortcut: one input, clean records

I maintain an Apify actor that streams the official extracts and hands back decoded, filterable records — county names instead of codes, ISO dates, human-readable statuses and license types, QB rows skipped, proxy business handled:

Florida License Records Scraper (DBPR)

{
    "profession": "construction",
    "licenseTypes": ["CGC"],
    "counties": ["Palm Beach"],
    "status": "current-active",
    "expiresBefore": "2026-12-31",
    "maxResults": 5000
}

Statewide pulls take about a minute, export as JSON/CSV/Excel, and pricing is per record returned — a 1,000-contractor county list costs a couple of dollars. Eight DBPR boards are covered: construction, electrical, home inspectors, mold services, cosmetology, barbers, veterinary, architecture.

The part that actually solves compliance: monitoring mode

Bulk verification has a shelf life of exactly one day — DBPR regenerates the files that often. What compliance teams really want isn't a snapshot, it's an alarm: tell me when something changes.

The actor has a monitor mode for that. Give it a watch list (or a filter set), run it on an Apify Schedule, and each run returns only what changed since the last one:

{
    "mode": "monitor",
    "profession": "construction",
    "licenseNumbers": ["CGC058548", "CCC1330911", "CFC1425030"],
    "alertOnStatusChange": true,
    "expiringWithinDays": 90
}

The first run saves a baseline. Every scheduled run after that emits change records only: status-changed (Current → Suspended is the one you're paying attention for), renewed, expiring-soon (once per license as it enters your 90-day window, with a daysUntilExpiration field), new-license, and removed-from-extract — that disappearing-contractor signal from point 6, delivered instead of silently missed.

Because pricing is per record returned, a run where nothing changed costs roughly a cent of compute and zero record charges. Watching 500 subcontractors costs approximately nothing until the week one of them gets suspended — which is the week it pays for itself a few hundred times over. Wire the run into a Slack/email integration and you're done: set-and-forget license compliance.

Recap

Florida publishes its complete contractor license roll as free public extract files — the data access problem is solved.
The files are honest work to use: headerless 22-column CSVs, coded everything, datacenter-IP blocking, and meaningful absences.
Decode it yourself with ~25 lines of pandas for a one-off; use the actor for clean filtered pulls; use monitor mode on a schedule when what you actually need is "alert me when a license changes."

Fighting with a different state's license data? Drop a comment — Texas TDLR and California CSLB are next on my list.

Query every dentist (or physician) in a metro without hitting NPPES's 1,200-record wall

CBLU2005 — Wed, 12 Aug 2026 14:07:29 +0000

Every US healthcare provider — every physician, dentist, nurse practitioner, physical therapist, pharmacy, and clinic — has a National Provider Identifier, and the whole registry is public. CMS runs it as NPPES, and it has a free, keyless API. If you sell to providers (dental supplies, medical devices, EHR/billing software, staffing), this is the authoritative list of your entire market, with practice addresses and phone numbers, for free.

There's one catch, and it's the reason so many "NPI scrapers" quietly return garbage: the NPPES API hard-caps every search at 1,200 records — and past that cap it doesn't error, it silently repeats the same page. If a metro has 4,000 dentists and you page naively, you'll get 1,200 real ones followed by duplicates of the last page, feel like you got "a lot of data," and never realize two-thirds of your market is missing.

This post explains the wall, shows the manual way through it, and gives you a copy-paste Python fix.

The API, and the wall

The base call is simple. Dentists in Florida:

curl "https://npiregistry.cms.hhs.gov/api/?version=2.1&\
taxonomy_description=Dentist&state=FL&limit=200"

limit maxes out at 200 per call. To go deeper you add skip:

curl "https://npiregistry.cms.hhs.gov/api/?version=2.1&\
taxonomy_description=Dentist&state=FL&limit=200&skip=1000"

And here's the wall: skip maxes out at 1,000. 1,000 skipped + 200 returned = 1,200 records, full stop. Ask for skip=1200 and the API doesn't say "out of range" — it hands you the skip=1000 page again. Naive pagers treat that as more data and append duplicates forever. (You'll also notice very broad searches — a whole state with no other filter — get rejected outright; NPPES requires at least one criterion beyond state.)

So the real problem isn't fetching data. It's fetching complete data for any search that legitimately has more than 1,200 matches — which is most useful ones (dentists in Miami, PTs in Houston, family medicine in all of California).

The manual fix: subdivide until every slice fits under 1,200

The trick is to split one too-big search into several small-enough searches, then merge and de-duplicate. The cleanest axis to split on is ZIP code prefix, because NPPES supports trailing wildcards on postal_code. "All Miami dentists" (too big) becomes "dentists in 331xx," then if that's still over 1,200, "3310x, 3311x, 3312x…," and so on. You recurse only into the slices that are actually full, and you de-duplicate the final set by NPI (a provider with a practice and a mailing address in different ZIPs can match twice).

Here's the whole thing in Python — no dependencies beyond requests:

import requests

API = "https://npiregistry.cms.hhs.gov/api/"
CAP = 1200          # NPPES hard cap per search
PAGE = 200          # max limit per call

def _page_all(params):
    """Fetch up to the 1,200 cap for one exact search."""
    out, skip = [], 0
    while skip <= 1000:
        r = requests.get(API, params={**params, "version": "2.1",
                                       "limit": PAGE, "skip": skip}, timeout=30)
        r.raise_for_status()
        results = r.json().get("results", [])
        if not results:
            break
        out.extend(results)
        if len(results) < PAGE:
            break
        skip += PAGE
    return out

def search(base, zip_prefix=""):
    """Recursively subdivide by ZIP prefix so no slice exceeds the cap."""
    params = dict(base)
    if zip_prefix:
        params["postal_code"] = zip_prefix + "*"
    hits = _page_all(params)
    # If we came back pinned at the cap, this slice is probably truncated —
    # split it into 10 narrower ZIP prefixes and recurse.
    if len(hits) >= CAP and len(zip_prefix) < 5:
        deeper = []
        for d in "0123456789":
            deeper.extend(search(base, zip_prefix + d))
        return deeper
    return hits

def dedupe(records):
    seen, out = set(), []
    for rec in records:
        npi = rec.get("number")
        if npi and npi not in seen:
            seen.add(npi)
            out.append(rec)
    return out

# Every dentist in greater Miami, complete:
raw = search({"taxonomy_description": "Dentist", "state": "FL"}, zip_prefix="331")
providers = dedupe(raw)
print(f"{len(providers)} unique dentists (naive paging would have stopped at 1,200)")

Run that against a dense metro and you'll see it sail past 1,200 — a Miami-area dentist search returns close to 2,000 unique providers, not the 1,200 a single search reports. That gap is exactly the providers a naive scraper never sees.

A few things to add before you rely on it:

Be polite. Add a small delay between calls and a retry/backoff — NPPES will throttle a tight loop. The recursion above can fire dozens of calls for a dense metro.
Pull the fields you actually need out of each record: number (the NPI), the taxonomy marked primary for specialty and license, and the address where address_purpose == "LOCATION" for the practice phone. NPPES nests these in arrays, so a naive record["phone"] won't exist.
Emails aren't in here. NPPES publishes practice phone, fax, and address, plus the authorized official's name and phone for organizations — but not provider email. Any tool selling you "NPI emails" is enriching from somewhere else; treat those with the skepticism they deserve.

The maintained shortcut

I wrapped this exact approach — the cap detection, the ZIP fan-out, the de-duplication, the field extraction, the backoff — into an Apify actor so you don't have to own the recursion:

NPI Registry Scraper (NPPES Healthcare Providers)

The complete-Miami-dentists query becomes:

{
    "taxonomyDescription": "Dentist",
    "state": "FL",
    "postalCode": "331*",
    "maxResults": 2000
}

It returns clean, flat records — npi, name, credential, specialty, licenseNumber, licenseState, practiceAddress (with phone and fax), authorizedOfficial for organizations — and hands you JSON/CSV/Excel, scheduling, and webhooks. You're charged per record returned, so a ~2,000-provider metro list costs about four dollars. The point of paying isn't access to the data (it's free and keyless); it's that the 1,200-cap fan-out, the paging, and the field-unnesting are already correct and maintained.

Which route should you take?

One search that's clearly under 1,200 (a single small city and specialty) → just page it yourself with the first snippet. No fan-out needed.
Any dense metro or statewide specialty, on a schedule, or feeding a CRM or AI agent → use the fan-out. The actor is the least-effort version of it (and it's callable as an MCP tool via Apify, so "how many pediatric dentists are in the Dallas metro?" becomes a one-line agent query).
A one-off count for a single ZIP → honestly, the NPPES website is fine; don't automate a thing you'll do once.

The registry is public infrastructure paid for with tax dollars. The only thing standing between you and a complete provider list is knowing that 1,200 isn't the end — it's where the naive scrapers give up.

Hit a specialty or metro where the fan-out behaves oddly? Drop a comment — the taxonomy descriptions have some sharp edges (there are three different flavors of "nurse practitioner" alone) and I'm happy to compare notes.

Build a fresh trucking-insurance lead feed: every new US carrier with phone + email, updated weekly

CBLU2005 — Wed, 12 Aug 2026 14:01:58 +0000

When a trucking company registers with the FMCSA and gets its authority, the clock starts on a bunch of things it now legally has to buy: primary liability and cargo insurance, a BOC-3 process agent, an ELD, often factoring to survive the 30–60 day payment cycle. Whoever reaches that carrier first — in the days after it registers, while it's still shopping — wins the account.

The good news for anyone selling into that market: the list of who just registered is a public federal record, published on an official US DOT open-data portal, with phone and email on file for essentially every carrier registered in the last couple of years. You do not need to scrape SAFER, solve captchas, or buy a $300/month lead list. You need one HTTPS request and a filter.

This post shows how to pull it yourself, where it gets fiddly, and how to turn it into a scheduled weekly feed.

Where the data lives

Everything SAFER shows you about a carrier — legal name, DBA, address, phone, email, company officers, fleet size, cargo, hazmat flag, safety rating — is rendered from FMCSA's Company Census File. FMCSA publishes that census on data.transportation.gov, the US DOT's open-data portal, which runs on Socrata. That means the same SODA API you'd use for any city open-data set: plain HTTPS in, JSON out, SQL-ish query params, no API key for moderate use.

First, find the current dataset. Portal dataset IDs occasionally change when an agency re-publishes, so don't hard-code one you found in a blog post — ask the catalog:

curl "https://api.us.socrata.com/api/catalog/v1?domains=data.transportation.gov&q=motor%20carrier%20census&only=datasets"

That returns the dataset's id (a four-four Socrata token like az4n-8mr2) and its resource URL. From there the census is queryable at:

https://data.transportation.gov/resource/<dataset-id>.json

Pull the five most-recently-added carriers:

curl "https://data.transportation.gov/resource/az4n-8mr2.json?\
\$order=add_date%20DESC&\$limit=5"

The annoying parts (there are three)

1. The column names are not what you'd guess. The census predates every naming convention you like. Depending on the published extract, the phone field is telephone or phone, email is email_address, physical state is phy_state, the power-unit count is nbr_power_unit (not power_units), and the "when did this carrier first appear" date is add_date. Before you build anything, hit the dataset's own API-docs page (linked from the portal page) or just pull one row and read its keys:

curl "https://data.transportation.gov/resource/az4n-8mr2.json?\$limit=1" | python3 -m json.tool

2. "New carrier" is a date field, and dates on Socrata are strings. Filter server-side with $where against add_date, ISO-formatted:

curl "https://data.transportation.gov/resource/az4n-8mr2.json?\
\$where=add_date%20%3E=%20'2026-06-01T00:00:00'&\
\$order=add_date%20DESC&\$limit=1000"

3. Contact completeness varies by age. Carriers registered recently almost always have both phone and email on file; older records have gaps. If you're building a contactable lead feed, filter for the fields you actually need (WHERE email_address IS NOT NULL) rather than assuming.

DIY: a weekly new-carrier feed in ~30 lines

Here's a self-contained Node script: every active carrier added since a given date, in your states, with phone and email, newest first.

const DATASET = 'az4n-8mr2'; // verify via the catalog query above
const BASE = `https://data.transportation.gov/resource/${DATASET}.json`;

async function newCarriers({ states, addedAfter, limit = 1000 }) {
  const stateList = states.map((s) => `'${s}'`).join(',');
  const where = [
    `add_date >= '${addedAfter}T00:00:00'`,
    `phy_state in (${stateList})`,
    `telephone IS NOT NULL`,
    `email_address IS NOT NULL`,
  ].join(' AND ');

  const params = new URLSearchParams({
    $where: where,
    $order: 'add_date DESC',
    $limit: String(limit),
  });

  const rows = await (await fetch(`${BASE}?${params}`)).json();
  return rows.map((r) => ({
    dotNumber: r.dot_number,
    legalName: r.legal_name,
    dbaName: r.dba_name || null,
    phone: r.telephone,
    email: r.email_address,
    state: r.phy_state,
    powerUnits: r.nbr_power_unit ? Number(r.nbr_power_unit) : null,
    addedDate: (r.add_date || '').slice(0, 10),
    safer: `https://safer.fmcsa.dot.gov/query.asp?searchtype=ANY&query_type=queryCarrierSnapshot&query_param=USDOT&query_string=${r.dot_number}`,
  }));
}

const leads = await newCarriers({ states: ['TX', 'OK'], addedAfter: '2026-06-01' });
console.log(`${leads.length} contactable new carriers`);
console.log(leads.slice(0, 5));

Cron that weekly, diff against last week's DOT numbers so you only email each carrier once, and you have a real lead pipeline for the cost of nothing.

One data-source gotcha that will bite you if you don't know it: FMCSA populates a carrier's cargo-classification flags about two months after it first registers. So if you filter for, say, "refrigerated" carriers and "added in the last 30 days," you'll get almost nothing — not because they don't exist, but because that field hasn't been backfilled yet. Filter on cargo type for established carriers, and on add_date for fresh ones; don't combine the two on a tight recent window.

Things you'll end up building on top of it

Within a week of using the raw feed seriously you'll want: fleet-size banding (nbr_power_unit between 1 and 10 is the factoring/fuel-card sweet spot), interstate-vs-intrastate filtering, hazmat and safety-rating fields, company-officer extraction (the decision-maker's name is in there), de-duplication across weekly runs, the ~1,000-row Socrata page limit handled with $offset paging, and retry/backoff. None of it is hard. All of it is maintenance you now own.

The maintained shortcut

I package exactly this — the census, streamed and filtered server-side, with all the fields above normalized into clean names and the fleet/cargo/hazmat/date filters as simple inputs — as an Apify actor:

FMCSA Motor Carrier Scraper

The weekly-insurance-lead query becomes:

{
    "states": ["TX", "OK"],
    "statuses": ["active"],
    "addedAfter": "2026-06-01",
    "requireEmail": true,
    "requirePhone": true,
    "maxResults": 1000
}

Every record comes back in the same clean shape — dotNumber, legalName, dbaName, phone, email, companyOfficers, powerUnits, cargoTypes, safetyRating, saferUrl — and Apify gives you JSON/CSV/Excel export, a scheduler (run it every Monday at 6am), and webhooks into your CRM. You're charged per record returned, so a weekly pull of a few hundred fresh carriers costs about a dollar. It also handles the paging, the cargo-lag warning, and re-discovers the dataset ID if DOT re-publishes it — the babysitting I described above.

Which route should you take?

One state, personal prospecting → the script above. It's 30 lines and the data is free.
Multiple states, weekly scheduling, fleet/cargo filters, CRM webhooks, or feeding an AI agent → the actor (it's also callable as an MCP tool via Apify, so "any new reefer carriers in Texas this week?" becomes a one-line agent query).
Enterprise BD with dialer integration and territory routing → that's when the dedicated trucking-lead SaaS earns its subscription.

Either way the underlying record is a public federal registration — official, free, and yours to use. That's the part most people selling "exclusive carrier leads" would rather you didn't know.

Questions about a specific census field or a state's registration volume? Drop a comment — I've spent more time in this dataset's column names than I'd like to admit.

Track SAM.gov contract opportunities without an API key (or a $300/month GovCon subscription)

CBLU2005 — Wed, 12 Aug 2026 13:56:01 +0000

If you sell anything to the US federal government, you live and die by SAM.gov's Contract Opportunities — every proposed contract action over $25,000 gets posted there. The problem is consuming it:

The website is slow, aggressively bot-protected, and unusable for bulk work.
The official API requires a SAM.gov account, an API key that expires every 90 days, and has rate limits that hurt (Google "sam.gov api key" and enjoy the pain threads).
The GovCon SaaS tools that repackage this data start around $100/month and run into the thousands.

Here's the thing almost nobody seems to know: the sam.gov website's own search backend is a public, keyless JSON API. No account. No API key. No 90-day key rotation. It's the same endpoint your browser hits when you search Contract Opportunities — it's just never advertised as an API.

(You may have read older posts pointing at SAM.gov's daily CSV extract on S3 — ContractOpportunitiesFullCSV.csv. That was the best free path for years, but since late July 2026 SAM.gov has been regenerating it as a header-only stub: 47 column names, zero rows. If your pipeline was built on it, that's why it's suddenly empty. The search API below is the working replacement — and unlike the 200+ MB daily download, it filters server-side.)

The endpoint

https://sam.gov/api/prod/sgs/v1/search/

Standard query parameters, JSON out. The important ones:

Param	Meaning
`index=opp&mode=search&responseType=json`	boilerplate — always send these
`page` / `size`	paging; `size` max 100
`sort=-modifiedDate`	newest first
`is_active=true`	only still-open notices
`naics`	comma-separated NAICS codes (prefixes work)
`pop_state`	place-of-performance states, e.g. `TX,FL`
`set_aside`	set-aside codes (`SBA`, `8A`, `WOSB`, `SDVOSBC`...)
`publish_date.from` / `.to`	needs an explicit UTC offset, e.g. `2026-07-01-04:00`
`q` + `qMode=ALL`	keyword search

Check it yourself — active custom-programming solicitations, newest first:

curl "https://sam.gov/api/prod/sgs/v1/search/?index=opp&mode=search&responseType=json&sort=-modifiedDate&size=5&page=0&is_active=true&naics=541511" \
  -H "Accept: application/hal+json"

Two things the search results don't carry: the contracting officer's contacts and the full description. Those live on a second keyless endpoint, one call per notice:

https://sam.gov/api/prod/opps/v2/opportunities/<noticeId>

DIY: a NAICS watcher in ~35 lines of Python

import requests

SEARCH = "https://sam.gov/api/prod/sgs/v1/search/"
DETAIL = "https://sam.gov/api/prod/opps/v2/opportunities/{}"
HEADERS = {"Accept": "application/hal+json"}

params = {
    "index": "opp", "mode": "search", "responseType": "json",
    "sort": "-modifiedDate", "size": 100, "page": 0,
    "is_active": "true",
    "naics": "541511,541512",            # custom programming, systems design
    "publish_date.from": "2026-07-01-04:00",
}

while True:
    data = requests.get(SEARCH, params=params, headers=HEADERS, timeout=60).json()
    results = data.get("_embedded", {}).get("results", [])
    if not results:
        break
    for r in results:
        # contacts + description live on the detail endpoint
        d = requests.get(DETAIL.format(r["_id"]), headers=HEADERS, timeout=60).json()
        data2 = d.get("data2", {})
        poc = next(iter(data2.get("pointOfContact") or []), {})
        due = (data2.get("solicitation", {}).get("deadlines", {}).get("response") or "?")[:10]
        print(f'{(r.get("publishDate") or "")[:10]}  {r["title"][:70]}  '
              f'due={due}  poc={poc.get("email", "")}  '
              f'https://sam.gov/opp/{r["_id"]}/view')
    if (params["page"] + 1) * params["size"] >= data["page"]["totalElements"]:
        break
    params["page"] += 1

That's a working daily bid-pipeline feed with zero accounts and zero dollars. Cron it, pipe it to Slack, done.

Things you'll add within a week of using it seriously: the per-notice detail calls are 1:1 with results, so you'll want concurrency with a cap and retry/backoff (sam.gov will throttle a tight loop — it's the same bot protection the website has); the API stops paging at 10,000 records per query, so broad searches need date-windowing; the date params silently return wrong windows without that UTC offset; set-aside and notice-type filtering take internal codes; and you'll want dedupe across days. None of it is hard; all of it is maintenance.

The maintained shortcut

I package exactly this — the same search API the sam.gov site uses, paged, enriched with the per-notice contact details, all filters server-side, with the (currently broken) CSV extract kept as an automatic fallback source — as an Apify actor:

SAM.gov Contract Opportunities Scraper

Everything still biddable in construction trades, small-business set-aside, in Texas or Florida:

{
    "naicsCodes": ["236", "237", "238"],
    "setAsides": ["SBA"],
    "popStates": ["TX", "FL"],
    "postedAfter": "2026-06-01",
    "responseDueAfter": "2026-07-11",
    "maxResults": 1000
}

Out comes clean JSON/CSV/Excel with structured contacts, award data on award notices, and direct SAM.gov links. Schedule it daily in Apify, add a webhook, and you have the core of the $300/month GovCon alert products for about the price of a coffee per month (pricing is per record returned).

Which route should you take?

One NAICS code, personal use → the Python script above. Seriously, it's ~35 lines.
Multiple filters, scheduling, teammates who want CSVs, or feeding an AI agent → the actor (it's also callable as an MCP tool via Apify, which makes "any new cybersecurity RFPs today?" a one-line agent query).
Enterprise BD with pipeline analytics → that's when the SaaS subscriptions earn their keep.

The data itself is public, official, and free either way — which is exactly how government contracting data should be.

Building permits are the best construction leads nobody uses — here's how to pull them from official city open data

CBLU2005 — Wed, 12 Aug 2026 13:55:56 +0000

Every building permit is a public record of someone about to spend money on construction: the address, the type of work, often the declared project value and the contractor doing it. If you sell building materials, dumpster rentals, insurance, solar, HVAC service plans — or you are a contractor watching your competitors — permits are the earliest buying signal that exists.

And most US cities publish them, daily, for free, on official open-data APIs. Not scraped, not resold, not 30 days stale: the city's own database.

This post shows how to pull them yourself with nothing but curl, what the data looks like, where it gets annoying (every city names its columns differently), and how to automate the whole thing.

Where permits live: the SODA API

Most large US cities run their open-data portals on Socrata, which exposes every dataset through the SODA API: plain HTTPS, JSON out, SQL-ish query parameters in. No API key required for moderate use.

Chicago's building permits, newest first:

curl "https://data.cityofchicago.org/resource/ydr8-5enu.json?\
\$order=issue_date%20DESC&\$limit=5"

Filter server-side with $where — say, permits issued this month:

curl "https://data.cityofchicago.org/resource/ydr8-5enu.json?\
\$where=issue_date%20%3E=%20'2026-07-01T00:00:00'&\
\$order=issue_date%20DESC&\$limit=100"

You get structured JSON with the permit number, work description, fees, reported cost, and — in Chicago's case — up to five named contacts including the contractor.

A few dataset IDs to get you started (all verified working as of July 2026):

City	Portal	Dataset ID
New York City (DOB NOW)	data.cityofnewyork.us	`rbx6-tga4`
Chicago	data.cityofchicago.org	`ydr8-5enu`
Los Angeles (2020+)	data.lacity.org	`pi9x-tg5x`
Austin	data.austintexas.gov	`3syk-w9eu`
San Francisco	data.sfgov.org	`i98e-djp9`
New Orleans	data.nola.gov	`72f9-bi28`

To find any other city's dataset, query Socrata's catalog API:

curl "https://api.us.socrata.com/api/catalog/v1?domains=data.brla.gov&q=building%20permits&only=datasets"

The annoying part: every city is a special snowflake

Here's where the DIY route gets expensive. The issue date is issue_date in Chicago, issued_date in San Francisco, issuedate in New Orleans, and issueddate in Baton Rouge. Valuation is reported_cost, estimated_cost, valuation, estprojectcost, or declaredvaluation depending on who you ask. NYC's feed includes rows for permits that aren't issued yet, with the date simply missing. One city publishes latitude/longitude as columns, another nests them in a GeoJSON point, a third uses a geolocation object.

If you only care about one city, write the mapping once and move on. If you want a multi-city feed, you're maintaining a normalization layer — and re-verifying dataset IDs whenever a city migrates portals (NYC's old ipu4-2q9a dataset silently stopped updating in 2020; the live one is rbx6-tga4; Seattle and Dallas left Socrata entirely).

Here's a minimal normalizer for two cities to show the shape of the problem:

const CITIES = {
  chicago: {
    url: 'https://data.cityofchicago.org/resource/ydr8-5enu.json',
    dateField: 'issue_date',
    map: (r) => ({
      permitNumber: r.permit_,
      permitType: r.permit_type,
      description: r.work_description,
      issuedDate: r.issue_date?.slice(0, 10),
      address: [r.street_number, r.street_direction, r.street_name].filter(Boolean).join(' '),
      valuation: r.reported_cost ? Number(r.reported_cost) : null,
    }),
  },
  'san-francisco': {
    url: 'https://data.sfgov.org/resource/i98e-djp9.json',
    dateField: 'issued_date',
    map: (r) => ({
      permitNumber: r.permit_number,
      permitType: r.permit_type_definition,
      description: r.description,
      issuedDate: r.issued_date?.slice(0, 10),
      address: [r.street_number, r.street_name, r.street_suffix].filter(Boolean).join(' '),
      valuation: r.estimated_cost ? Number(r.estimated_cost) : null,
    }),
  },
};

async function fetchPermits(cityKey, issuedAfter, limit = 100) {
  const city = CITIES[cityKey];
  const params = new URLSearchParams({
    $limit: String(limit),
    $order: `${city.dateField} DESC`,
    $where: `${city.dateField} >= '${issuedAfter}T00:00:00'`,
  });
  const rows = await (await fetch(`${city.url}?${params}`)).json();
  return rows.map(city.map);
}

console.log(await fetchPermits('chicago', '2026-07-01', 10));

The shortcut: one normalized API for 9 cities

I maintain an Apify actor that does exactly this across nine cities/counties (NYC, Chicago, LA, Austin, SF, New Orleans, Baton Rouge, Montgomery County MD, Norfolk VA), with one shared schema, server-side date filtering and full-text search, and the dataset-ID babysitting handled for you:

US Building Permits Scraper

One input gets you fresh, high-value leads across cities:

{
    "cities": ["austin", "chicago"],
    "issuedAfter": "2026-07-01",
    "minValuation": 50000,
    "maxResultsPerCity": 1000
}

Every record comes back in the same shape — permitNumber, permitType, description, issuedDate, address, valuation, contractorName where the city publishes it, coordinates — and Apify hands you JSON/CSV/Excel export, scheduling (run it every Monday morning), and webhooks into your CRM for free. Pricing is per record returned, so a weekly 1,000-lead pull costs a few dollars.

Bonus: turn permits into qualified contractor lists

Permits tell you who's building. State license records tell you who's licensed — with status, expiration date, and mailing address. Cross-referencing the two is a genuinely underrated play: match permit contractor names against the state's license roll and you get, e.g., "active Florida GCs who pulled zero permits this quarter" (prime targets for lead-gen services) or "contractors whose license expires within 90 days" (renewal/CE marketing).

Florida publishes its entire license database as public CSV extracts, and there's an actor for that too: Florida Contractor & Professional License Search (DBPR). A join on normalized business names between the two datasets is an afternoon of pandas and surprisingly good fun.

Recap

City open-data portals publish building permits daily through the SODA API — free, official, structured.
The pain is normalization and dataset churn, not access.
DIY one city with 30 lines of code; use a maintained multi-city actor when you want breadth and scheduling without the babysitting.

Questions about a specific city's dataset? Drop a comment — I've probably already fought with its column names.