DEV Community: Steven Carleton

My Actor worked for humans and failed for agents: a bug postmortem in four acts

Steven Carleton — Tue, 01 Sep 2026 18:06:37 +0000

On August 12, 2026, I looked at the public stats for my SAM.gov Contract Opportunities Scraper (an Apify Actor that searches US federal contract notices) and saw 22 failed runs in the last 30 days. My own runs? 7 lifetime, all green. My local test harness? Passing for weeks.

The Actor had been quietly broken since late July. The platform's health checks had been failing daily since roughly July 21, and I hadn't looked. Meanwhile my Florida license Actor, the only Actor in my portfolio earning money, had failed 9 of its 48 runs in 30 days while its daily health check kept passing. Sit with that ratio for a second: the check that ran every day was green, and one run in five was not.

A note if you click that listing today: its 30-day stats still show the failures from this incident. All 12 of them are from a single day, August 6, on the build this article is about. Every run since the August 12 fix has succeeded, including one from a stranger this morning. The number rolls back on its own as the window moves, which is its own small lesson about what public quality metrics actually measure.

Both Actors "worked" in every test I ran. Both were failing the callers that mattered. And increasingly, those callers are not humans clicking around Apify Console. They're AI agents hitting the Actor through an API, whether via the Model Context Protocol (MCP), Apify's MCP server, or my own us-govdata-mcp wrapper. Agents don't retry creatively. They don't eyeball suspicious output. They trust your schema, your pricing, and your exit code.

Here is what broke, what I changed, and what I'd do differently. Every build number, run ID, and dollar amount below is real.

Act one: the dataset schema that lied

My SAM.gov Actor declares a dataset schema, the platform-validated contract for every record it outputs. placeOfPerformance.city is a string or null. primaryContact.email is a string or null. And so on.

SAM.gov's notice-detail API had other plans. For some notices it returns nested objects where strings should be: a city as {}, or as {"code": "0"} instead of a name. My mapping code passed those objects straight through, and I never questioned it, because nothing I ran ever complained.

On the Apify platform, pushing a record that violates the dataset schema doesn't just drop that record. It aborts the whole pushData batch with "Schema validation failed." Runs died mid-flight. The platform's automated health checks started failing daily, and the Actor got auto-flagged as broken.

Here's the part that stung: local runs never validate the dataset schema. My local harness ran the same code against the same API and passed every time. The only environment where the bug existed was the one with paying users in it.

The fix, shipped in build 0.1.4 on August 12, is a one-function idea: no value reaches a string-typed output field without being coerced. From src/main.js:

/**
 * Coerce an API value to a trimmed string or null - never an object/array.
 * The SAM.gov detail API sometimes returns nested objects where a string is
 * expected (e.g. placeOfPerformance.city as {} or {"code":"0"} instead of a
 * name). Objects leaking into string fields fail the platform's dataset
 * schema validation, which aborts the whole pushData batch ("Schema
 * validation failed") - the bug that got this actor auto-flagged as broken
 * (local runs never validate the dataset schema, so they passed).
 */
const asText = (value) => {
    if (typeof value === 'string') return clean(value);
    if (typeof value === 'number' && Number.isFinite(value)) return String(value);
    return null;
};

(clean(), referenced above, is a two-line helper: trim the string, return null when nothing is left.)

And every mapping line now goes through it, with fallbacks for the API's shape-shifting:

record.placeOfPerformance = {
    street: asText(pop.streetAddress) ?? asText(pop.street),
    city: asText(pop.city?.name) ?? asText(pop.city),
    state: asText(pop.state?.code) ?? asText(pop.state),
    zip: asText(pop.zip?.code) ?? asText(pop.zip),
    country: asText(pop.country?.code) ?? asText(pop.country),
};

Boring code. That's the point. The dataset schema is a promise, and for an agent consumer it's the only promise. An agent building a pipeline on my output fields has no human in the loop to notice a city that's suddenly an object. Coerce at the boundary, always, and treat upstream APIs as adversarial about types even when they're government-official.

The lesson, in one line: your dataset schema is validated by the platform, not by your laptop. If your output can be shaped by an upstream API, run schema validation yourself in CI, or ship the coercion layer from day one.

Act two: charging 700 for 500

While reproducing the schema bug I found something worse in the billing data. My Actor uses pay-per-event (PPE) pricing: one contract-opportunity event, at $0.004, per record delivered.

I ran two reproduction runs on build 0.1.3 with maxResults: 500. Both delivered exactly 500 records. Both charged 700 events. So I did the arithmetic a customer would do: 700 events at $0.004 is $2.80 billed for $2.00 of delivered data, an extra $0.80 (40%) per run, hiding in the interaction between charging, the aborted pushData batches from act one, and the Actor's CSV fallback path. And per the failure mode above, a run could charge events and then die, leaving a user paying for a failed run.

It's worth walking the incentive structure here, because it's what makes this bug a different animal from a crash. On PPE pricing, every event my code posts is me billing a stranger's account. Apify takes its platform cut and I keep the rest. The user's only protection is that my accounting is honest. Mine wasn't. Not by intent, but the user's card can't tell the difference.

The chargedEventCounts on my own repro runs made the signature undeniable:

Run	Build	Delivered items	Charged events
`vaIzPUYUmZbzzuM4I`	0.1.3	500	700
`1W2kINLpaI2miHz1m`	0.1.3	500	700
`Z7b0J8yEFuT2XCeSq`	0.1.4	500	500
`cnr4Y0rP0kUIlXTPN`	0.1.4	44	44

The fix in 0.1.4 centralizes the invariant in one function that everything routes through. Charge first, then push only what was actually charged. Delivered items can then never exceed charged events, and the pushed count is whatever the platform confirms it charged, regardless of what my own bookkeeping expected:

/**
 * Push dataset items, charging one event per item when pay-per-event is active.
 * Never pushes more items than were successfully charged.
 */
async function pushCharged(items, eventName) {
    if (items.length === 0) return { pushed: 0, limitReached: false };
    if (!isPayPerEvent()) {
        await Actor.pushData(items);
        return { pushed: items.length, limitReached: false };
    }
    const { chargedCount, eventChargeLimitReached } = await Actor.charge({ eventName, count: items.length });
    if (chargedCount > 0) await Actor.pushData(items.slice(0, chargedCount));
    return { pushed: chargedCount, limitReached: eventChargeLimitReached };
}

(isPayPerEvent() is a one-liner around the SDK's charging manager; it returns false whenever the run isn't on PPE pricing, so the same code works on free and rental runs.)

A footnote I owe you, because this article is about billing honesty. This 0.1.4 version later turned out to have its own edge case. At the run's charge ceiling the SDK grants a partial charge, and pushData can then silently drop the batch, billing records that never land: the same harm, arriving by a different door. Build 0.1.8 replaced the pair with the SDK's atomic Actor.pushData(items, eventName), which charges exactly what it delivers. If you copy a pattern out of this article, copy that one.

Then came the uncomfortable part: making affected users whole. I learned that the developer API cannot enumerate other users' runs. GET /v2/acts/{actorId}/runs returns only runs started by my own account. 7 runs, all mine, while the public stats showed 41. The affected run IDs exist only on Apify's side.

So I wrote support a charge signature instead of a run list: any run of my Actor in the window, by any user but me, that either failed with billed PPE events, or succeeded with more billed contract-opportunity events than clean dataset items. I asked for the refunds to be debited from my payout. I sized the exposure at $1–$7, with an absolute ceiling around $45. Trivial money. But a pay-per-result model is the trust, and being prompt and boring about making people whole is part of the product, not a gesture on top of it.

Support answered the next day, and the answer is the reason I asked instead of guessed. Most of those failed runs were Apify's own test system, which runs every Store Actor daily on its default input. The failures I had read as burned customers were largely the platform telling me my Actor was broken, in the one channel I wasn't reading. The real user exposure was a handful of runs from a single free account: $0.17, which support credited directly. Two lessons, and I'd rank them in this order. My blast radius was small because I caught this while I had almost no users, not because I was careful. And the numbers a developer can see (22 failed runs, a 19% failure rate) do not distinguish a robot's daily probe from a paying customer's ruined afternoon. I sized my exposure at up to $45 and was off by two orders of magnitude, in the lucky direction.

The lesson: on PPE pricing, your charge accounting is part of your public interface. Test it with the same rigor as your output. chargedEventCounts on your own runs is the ground truth; diff it against your dataset item count in CI if you can.

Act three: the configuration that lives only in Console

Two smaller incidents, same root cause: some Actor properties don't live in your repo.

The swapped billing events. Earlier in the summer, my NPI and FMCSA Actors (National Provider Identifier registry, and the Federal Motor Carrier Safety Administration's carrier census) had their PPE events swapped in the Console monetization config (NPI charging FMCSA's event and vice versa). The code was correct. The config, which lives only in Console and can't be pushed from the repo, wasn't. The fix was manual, followed by a paid smoke-test run of all the Actors, watching the logs for 'unknown event' warnings. Event names in Actor.charge() must match the Console config character for character, and nothing in your repo will tell you they don't.

The title that wouldn't ship. In an SEO audit I'd rewritten all my Actor titles around what buyers actually search ("lookup", "verification", "no API key"). I updated every .actor/actor.json, pushed, verified the builds succeeded. The live store titles didn't change. For a published Actor, apify push updates code and versions but does not propagate title or description to the store listing; those are Publication-tab metadata, edited in Console. I verified this the hard way, via the public store API after pushing: modifiedAt updated, title unchanged.

And when the titles finally went in through Console, they had to shrink. My approved SEO title for the SAM.gov Actor was 77 characters; what's live is 57. The Publication title field caps at 63 characters, and every title in my portfolio now sits at or under that line. The 63-character reality annoyed me and then improved me: agents discovering Actors as tools see the title and description first, and a title that states the capability in 10 words beats one that lists every keyword.

The lesson: know which parts of your Actor are code and which are Console state. Monetization events, published titles and descriptions, and SEO fields won't version, diff, or deploy with your repo. Keep a checklist for them, because your repo's green build says nothing about them.

Act four: the proxy that cuts the stream at 60 seconds

My Florida license Actor streams the official license extracts of Florida's Department of Business and Professional Regulation (DBPR), filters rows, and charges per record returned. The construction file alone runs 46.3 MB. These are public-records files the state publishes for this very purpose, and the Actor fetches each one once per run at normal download rates; the proxy business below is about reachability, not volume. In August the Actor kept passing its daily platform health check while 9 of its 48 runs failed. Call it one in five. Whose runs those were, I can't tell you. The developer API shows me my own runs and aggregate counts, never whose run died. It's the same blind spot that had me overestimating the SAM.gov exposure by two orders of magnitude in act two.

I reproduced it with a whale-shaped input: Miami-Dade county, certified general contractors (license type CGC), maxResults: 1000, which needs a multi-megabyte scan of the file. DBPR's content delivery network (CDN) blocks datacenter IPs, so the Actor escalates to Apify Proxy. And in every one of my repro runs, the proxied connection died at roughly 60 seconds mid-stream. My code treated the truncated stream as end-of-file. The CSV parser (csv-parse) hit the truncation mid-quoted-field and crashed the run: Quote Not Closed ... line 15263 (repro run kI1VkPKBOJcYs3JqB, failed at 1m05s). Worse: 400 of the 1,000 records had already been pushed and charged. A paying user, paying for a failed run. The worst outcome available on this pricing model.

Why did the health checks pass? They fetch 25 records, match early, and close the stream well before the 60-second cut. A small-input health check is the one kind of run guaranteed never to see a large-input failure. And there was a nastier latent case: if the cut landed on a row boundary, the run would succeed with silently truncated data. A human might notice a suspiciously short list. An agent would report it as complete.

The fix (build 0.1.5, pushed the same night) was to make the download resumable. The DBPR CDN honors HTTP Range requests (I verified the 206 responses and Content-Range headers), so on any mid-stream cut the Actor resumes from the exact byte offset on a fresh connection, feeding one continuous stream to the parser:

function requestFromOffset(url, proxyUrl, offset) {
    return new Promise((resolve, reject) => {
        // Uncompressed body so byte offsets line up with what we already consumed.
        const headers = { Accept: 'text/csv,*/*', 'Accept-Encoding': 'identity' };
        if (offset > 0) headers.Range = `bytes=${offset}-`;
        const stream = gotScraping.stream({
            url,
            proxyUrl,
            timeout: { response: RESPONSE_TIMEOUT_MS },
            headers,
            https: { rejectUnauthorized: true },
        });
        stream.on('response', (res) => {
            if (res.statusCode === 206) {
                const match = /^bytes (\d+)-\d+\/(\d+)$/.exec(res.headers['content-range'] ?? '');
                if (!match || Number(match[1]) !== offset) {
                    stream.destroy();
                    reject(new Error(`Server returned an unexpected Content-Range "${res.headers['content-range']}" for offset ${offset}`));
                    return;
                }
                resolve({ stream, totalBytes: Number(match[2]), skipBytes: 0 });
                return;
            }
            if (res.statusCode === 200) {
                // Server ignored the Range header and replayed the file:
                // skip the bytes we already have instead of double-counting them.
                const totalBytes = Number(res.headers['content-length'] ?? 0) || null;
                resolve({ stream, totalBytes, skipBytes: offset });
                return;
            }
            stream.destroy();
            reject(new Error(`HTTP ${res.statusCode} while downloading ${url}`));
        });
        stream.once('error', reject);
    });
}

(gotScraping is the got-scraping HTTP client; RESPONSE_TIMEOUT_MS is the Actor's response-header timeout constant.)

Details that mattered in practice:

Accept-Encoding: identity matters. With compression on, your byte offsets and the server's don't line up, and Range resumption corrupts silently.
Check that the 206's Content-Range starts at your offset. Some servers answer 206 with the wrong window.
Handle the server ignoring Range entirely (a 200 replay) by skipping the bytes you already consumed.
The stream is only ended once all bytes have arrived, which kills both the crash and the silent-truncation case with one invariant.
An idle watchdog destroys a stalled socket after 60 seconds of no data. Node's pipe() does not forward source errors; without the watchdog, a dead connection hangs until the run timeout.
Progress (rowsProcessed, pushed) checkpoints to the run's key-value store at every flush. After a platform migration the Actor fast-forwards and never double-pushes or double-charges. I verified this locally with a seeded checkpoint: it resumed at 500 pushed and delivered exactly the remaining 500.

Verification on build 0.1.5, same night:

Run	Input shape	Result	Time	Platform cost
`du5uozEtuf72Tycjx`	Miami-Dade + CGC, 1,000 records (the shape that failed)	1,000 records, 1 cut resumed	1m 37s	$0.011
`nq26dTSyyVsGEBRPh`	Full 46.3 MB scan to end-of-file (worst case)	241 records, matches the local count, 4 cuts resumed	4m 16s	$0.018
`6bgjK0P7MBhzIrNjO`	Health-check input, 25 records	25 records	6s	$0.001

Four cuts on a single full-file scan. For any serious input this was the everyday case, and every small test was blind to it.

The lesson: small health checks are structurally blind to large-input failures. My 6-second check could never see a cut that happens at 60.

The input schema, or: leaving the agent nothing to guess

The incidents above are about output, billing, and transport. The remaining surface is input, and here the work is preventative. My rule after this month is blunt: an agent should be able to construct a correct call from the schema alone, with nothing to guess.

Concretely, from the Florida Actor's input schema:

Closed inputs are enums with labels. profession is an enum of 8 licensing boards with enumTitles like "Construction contractors (CILB: general, building, residential, roofing, plumbing, HVAC, pool...)". An agent cannot invent a board I don't support.
Formats are machine-checkable. Dates carry "pattern": "^\\d{4}-\\d{2}-\\d{2}$". A malformed date fails validation with a message, instead of silently matching nothing.
Costs are stated where the number is set. The maxResults description says "You are only charged for records actually returned." That sentence is doing billing-anxiety work for humans and budget arithmetic for agents.
Semantics that could surprise are spelled out. The monitoring mode's licenseNumbers watch list deliberately overrides the status filters, because a license that gets suspended must not fall out of its own watch list. The description spells that out, and says why. An agent that reads it configures compliance alerts correctly on the first try.

What I'd do differently

If I were starting the portfolio again:

Validate output against the dataset schema in CI. The platform validates; my laptop didn't. That asymmetry cost me 3 weeks of silent failures. It's the single highest-leverage fix on this list.
Treat charge counts as testable output. Assert chargedEventCounts equals delivered items on a real platform run before every release of a PPE Actor.
Test at the scale users buy, not the scale that's convenient. My health checks were 25-record runs; my customer's were ~1,000. Everything interesting happened past the 60-second mark that small runs never reach.
Keep a Console-state checklist. Event names, published title, description, SEO fields, all reviewed by hand, because no diff will catch them.
Assume every long-lived connection will be cut, and design downloads to resume rather than restart. Range plus identity encoding plus an idle watchdog is a reusable pattern; I've since carried it into my Texas and California license Actors.

The framing I keep coming back to: humans forgive an Actor its quirks because they can see them. Agents can't see anything you didn't put in the schema, the pricing events, or the error message. Making an Actor "AI-friendly" turned out to mean making it precise. And every user, human or not, got a better Actor out of it.

If you're monetizing Actors, pull your own chargedEventCounts tonight. I thought mine were fine too.

Trial site selection & investigator mapping with ClinicalTrials.gov's free API (CRO edition)

Steven Carleton — Mon, 17 Aug 2026 13:46:21 +0000

Every interventional study on ClinicalTrials.gov lists its sites — facility, city, recruitment status, and, very often, the named principal investigator at each one. Which means the world's most complete map of who runs clinical trials, where, on what is a free public API. If you work in or sell to clinical research, this is your market map:

CRO / sponsor site selection — which facilities in Florida are actively enrolling MASH patients right now? Those sites have the patients, the equipment, and a competing protocol.
Investigator / KOL mapping — the named PIs on recruiting oncology trials are the exact people a sponsor, CRO, or medical-affairs team needs on a list.
Competitive enrollment intel — before you commit a site, count the trials already recruiting the same population in the same metro. Enrollment risk is quantifiable.
BD prospecting — labs, imaging vendors, ePRO/eCOA platforms, patient-recruitment firms: active sites are your buyers, sorted by indication.

Here's the DIY route against the real API, the three places it gets annoying, and the shortcut.

The DIY: API v2 is genuinely good

No key, no account. Recruiting MASH studies with at least one Florida site:

curl "https://clinicaltrials.gov/api/v2/studies?\
query.cond=NASH&query.locn=Florida&filter.overallStatus=RECRUITING&\
pageSize=100&fields=protocolSection.identificationModule,\
protocolSection.contactsLocationsModule,protocolSection.designModule"

Works today, returns JSON, pages cleanly with pageToken. The contactsLocationsModule is where the value lives — per-site facility, city, state, status, and a contacts array where entries with "role": "PRINCIPAL_INVESTIGATOR" are your named investigators.

The three annoyances

Location filters return studies, not sites. query.locn=Florida gives you every study that has a Florida site — with the full worldwide locations array attached (large multi-center trials list hundreds of sites). Your "Florida site list" requires walking every study's location array and keeping only the Florida rows yourself. This is the big one: the question is site-shaped, the API is study-shaped.
The JSON is deeply nested and inconsistently populated. Everything hides under protocolSection.<something>Module, and a "simple" flat row means stitching identification, status, design, sponsor, and contacts modules together. PI naming also varies by sponsor type: academic centers usually name investigators (Mayo Clinic trials list PIs with credentials), while some industry sponsors anonymize sites to "89bio Clinical Study Site" with a call-center contact — your pipeline needs to handle both without pretending one is the other.
Field selection is its own language. The fields parameter wants exact dotted module paths; get one wrong and it errors (better than silence, but still a docs-diving session). Filters similarly split across query.* (search-y) and filter.* (exact) families with different semantics — query.cond, query.spons, filter.overallStatus, aggregate filters, and so on.

A workable Python flattener for site-level rows:

import requests

def florida_sites(condition, state="Florida"):
    params = {
        "query.cond": condition, "query.locn": state,
        "filter.overallStatus": "RECRUITING", "pageSize": 100,
    }
    url = "https://clinicaltrials.gov/api/v2/studies"
    while True:
        data = requests.get(url, params=params).json()
        for study in data.get("studies", []):
            ps = study["protocolSection"]
            nct = ps["identificationModule"]["nctId"]
            title = ps["identificationModule"].get("briefTitle")
            for loc in ps.get("contactsLocationsModule", {}).get("locations", []):
                if loc.get("state") != state:
                    continue  # the study-shaped-vs-site-shaped problem
                pis = [c["name"] for c in loc.get("contacts", [])
                       if c.get("role") == "PRINCIPAL_INVESTIGATOR"]
                yield {"nctId": nct, "title": title,
                       "facility": loc.get("facility"), "city": loc.get("city"),
                       "siteStatus": loc.get("status"), "investigators": pis}
        if not (token := data.get("nextPageToken")):
            break
        params["pageToken"] = token

for row in florida_sites("NASH"):
    print(row)

Fine for one query. The maintenance shows up when you want this across indications, sponsor classes, and phases, refreshed weekly, in a spreadsheet your feasibility team can actually open.

The shortcut: one input, flat study records with sites and PIs attached

I maintain an Apify actor that wraps the v2 API — module-stitching, paging, and contact extraction included:

ClinicalTrials.gov Scraper

{
    "condition": "NASH",
    "statuses": ["RECRUITING"],
    "phases": ["PHASE2", "PHASE3"],
    "sponsorClasses": ["INDUSTRY"],
    "state": "Florida",
    "maxResults": 500
}

Each record is one study, flattened: identification, phase, enrollment, lead sponsor, centralContacts, overallOfficials, and a clean locations array where every site carries its status and a ready-made principalInvestigators list. Export JSON/CSV/Excel, schedule it weekly on Apify Schedules for a standing feasibility feed, and pricing is per study returned — a 500-study indication landscape costs about a dollar and a half.

Filter by sponsor (e.g. Pfizer) to map a competitor's entire active footprint, or by intervention (e.g. semaglutide) to watch a mechanism.

Recap

ClinicalTrials.gov's v2 API is free, keyless, and contains the industry's site + investigator map.
The DIY tax: study-shaped responses for site-shaped questions, deep module-stitching, and per-sponsor contact inconsistency.
Thirty lines of Python covers one query; the actor covers the recurring, multi-indication version with named PIs already extracted.

Working a specific feasibility question — an indication, a geography, a competitor? Drop a comment and I'll sketch the exact input.

Federal contracts: see who actually WON — post-award intel from USAspending's free API

Steven Carleton — Wed, 12 Aug 2026 18:04:21 +0000

SAM.gov tells you what the government wants to buy (I covered tracking it without an API key earlier). But the pre-award side is only half the picture. The other half — the half your competitors' business developers are quietly living in — is who won, for how much, from which agency, and when that contract runs out.

That data is USAspending.gov: every federal contract, grant, loan, and direct payment, published under the DATA Act. It answers questions SAM.gov can't:

Competitive intel — who actually wins IT-services contracts in Texas? At what dollar sizes? For which agencies?
Recompete pipeline — an award with an end date 9 months out is a solicitation waiting to happen. Incumbents know. Now you do too.
Teaming targets — small primes winning work in your NAICS are potential partners (or acquisition targets).
Grant prospecting — universities and nonprofits: see exactly which programs fund organizations like yours, and how much.

And unlike SAM.gov's official API, USAspending's API needs no key, no account, no registration. It's genuinely open. Here's the honest DIY, where it bites, and the shortcut.

The DIY: one POST endpoint does most of it

The workhorse is spending_by_award — POST your filters, get awards back:

curl -s -X POST "https://api.usaspending.gov/api/v2/search/spending_by_award/" \
  -H "Content-Type: application/json" \
  -d '{
    "filters": {
      "award_type_codes": ["A", "B", "C", "D"],
      "time_period": [{"start_date": "2026-05-01", "end_date": "2026-08-12"}],
      "naics_codes": ["541511"],
      "place_of_performance_locations": [{"country": "USA", "state": "TX"}]
    },
    "fields": ["Award ID", "Recipient Name", "Award Amount", "Awarding Agency",
               "Start Date", "End Date", "Description", "generated_internal_id"],
    "sort": "Award Amount", "order": "desc", "limit": 25, "page": 1
  }'

That call works today, unauthenticated (custom-programming contracts performed in Texas, biggest first — the top hit as I write this is a $37.7M DoD award). Every result's generated_internal_id gives you a shareable page: https://www.usaspending.gov/award/<id>.

Where it bites

The API is free and stable — and its query grammar was clearly designed for USAspending's own frontend, not for you:

award_type_codes is mandatory and cryptic. Contracts are A–D, contract IDVs are IDV_A–IDV_E, grants are 02/03/04/05, direct payments 06/10, loans 07/08. Send an empty filter set and you get a 422, not "everything."
Contracts and grants can't be queried together properly. Several filters (naics_codes, PSC) only apply to contracts; CFDA program numbers only to assistance. Real coverage means separate queries per award group, merged and deduplicated yourself.
fields are magic display strings. You request columns by their UI names — "Recipient Name", "Award Amount" — and misspellings just silently vanish from the response. Contract and grant responses also name overlapping things differently (Award Type vs Contract Award Type).
The good stuff needs a second call per award. Recipient mailing address, business categories, executive compensation — all on the award detail endpoint (/api/v2/awards/<generated_internal_id>/), one round trip each. A 500-award pull becomes 501 requests with polite pacing.
Occasional nested surprises — a field that's a string on one award is an object on the next. Your flattener finds out in production.

None of this is a dealbreaker; it's a long afternoon plus perpetual maintenance. If you enjoy that afternoon, the official docs are decent.

The shortcut: filters in, flat records out

I maintain an Apify actor that wraps all of the above — award-group fan-out, field normalization, optional per-award enrichment, paging, dedupe:

USAspending Federal Awards Scraper

{
    "awardTypes": ["contracts"],
    "naicsCodes": ["541511"],
    "placeOfPerformanceStates": ["TX"],
    "timePeriodStart": "2026-05-01",
    "minAwardAmount": 250000,
    "includeRecipientAddress": true,
    "maxResults": 500
}

Every record comes out flat and consistently named — recipientName, recipientUei, awardAmount, awardingAgency, startDate, endDate, naics, usaspendingUrl, plus the recipient's address and business categories when you flip includeRecipientAddress — with JSON/CSV/Excel export and pricing per record returned (a 500-award competitive-landscape pull costs about a dollar and a half).

Grants work the same way — swap awardTypes to ["grants"] and filter by cfdaNumbers (e.g. 93.778 for Medicaid) or awardingAgency.

The play that pays: pre-award + post-award together

The two datasets are a pincer:

USAspending (this post): find every contract in your NAICS ending in the next 12 months. That's your recompete target list, with the incumbent, the agency, and the dollar value attached.
SAM.gov: watch for the recompete solicitation to drop — with the contracting officer's name and email on it.

Schedule both weekly on Apify Schedules and you've rebuilt the core of a $300/month GovCon intelligence subscription for a few dollars of data charges.

Recap

USAspending.gov's API is free, keyless, and covers every federal award — the post-award half of GovCon intel.
DIY is one POST endpoint plus real friction: cryptic mandatory type codes, split contract/grant querying, magic field names, N+1 enrichment.
The actor flattens all of it behind one input; pair it with SAM.gov monitoring for the full pre-award + post-award picture.

If there's a specific slice you're trying to pull — a NAICS, an agency, a grant program — drop a comment and I'll sketch the exact input.

Verify Florida contractor licenses in bulk — DBPR's 46 MB mystery CSV, decoded

Steven Carleton — Wed, 12 Aug 2026 18:00:12 +0000

If your business touches Florida construction — insurance, lending, building materials, a marketplace, a permit-pulling SaaS — sooner or later you need to answer, at scale: is this contractor actually licensed, and is that license current, active, and not about to expire?

Florida makes this genuinely answerable. Unlike states with no statewide general-contractor license (looking at you, Texas), Florida's DBPR licenses contractors statewide and publishes the entire license roll as public records. The catch: the official ways to read it are either painfully slow or painfully cryptic. This post shows both DIY routes honestly, then the shortcut, and finally how to turn a one-off check into standing compliance alerts.

Route 1: the search portal (fine for one, hopeless for a list)

DBPR's Verify a License portal works one search at a time: pick a board, type a name or license number, click through to a detail page. For a single sub you're about to hire, perfect. For the 400 subcontractors on your book, that's an afternoon of copy-paste — every week, if you care about status changes. Nobody does this twice.

Route 2: the bulk extract files (free, complete, and user-hostile)

The real data lives on DBPR's instant public records page: full statewide extract files per profession, regenerated on a daily/weekly cadence. The construction file is one URL away:

curl -O "https://www2.myfloridalicense.com/sto/file_download/extracts/CONSTRUCTIONLICENSE_1.csv"

That's every licensed contractor in Florida — 46 MB of it (the cosmetology file is ~74 MB). Now the friction starts:

No header row. The file opens with data. There are 22 columns and DBPR doesn't label them in the file.
Everything is coded. Counties are numbers (23 = Miami-Dade, 60 = Palm Beach — the mapping is on DBPR's "understanding DBPR codes" page). Statuses are letters: primary C/P/S/N/D = Current / Probation / Suspended / Null-and-Void / Delinquent, secondary A/I = Active / Inactive. A usable license is C + A — most rows aren't.
Two license-number columns. Column 13 is a bare sequence (0015061); the full number you actually want (CBC015061) is column 21.
Rows that aren't licenses. Qualified-business entries ride along with no license number and have to be skipped.
The CDN blocks datacenter IPs. The download works from your laptop and then 403s from AWS/GitHub Actions — precisely where your cron job lives. (Ask me how I know.)
What's missing matters. DBPR excludes null-and-void, delinquent, and involuntarily-inactive licenses from the extracts. A contractor disappearing from the file is itself a signal.

Here's a working decoder for the construction file (tested against the live extract):

import pandas as pd

# 22 columns, no header row - mapping reverse-engineered from DBPR's code tables
COLS = ["board", "license_type", "licensee_name", "dba_name", "_4",
        "addr1", "addr2", "addr3", "city", "state", "zip", "county_code",
        "license_seq", "primary_status", "secondary_status",
        "original_license_date", "status_effective_date", "expiration_date",
        "_18", "_19", "license_number", "notes"]

PRIMARY = {"C": "Current", "P": "Probation", "S": "Suspended",
           "N": "Null and Void", "D": "Delinquent"}
SECONDARY = {"A": "Active", "I": "Inactive"}

df = pd.read_csv("CONSTRUCTIONLICENSE_1.csv", names=COLS, dtype=str,
                 keep_default_na=False)
df["primary_status"] = df["primary_status"].map(PRIMARY)
df["secondary_status"] = df["secondary_status"].map(SECONDARY)

# Every active certified general contractor in Palm Beach County (code 60):
cgc = df[(df.license_type == "CGC") & (df.county_code == "60")
         & (df.primary_status == "Current") & (df.secondary_status == "Active")]
print(len(cgc), "active Palm Beach CGCs")

To verify your list, inner-join it on license_number and flag anything that isn't Current + Active — plus anything from your list that's not in the file at all (see point 6).

If you only need this once, the DIY route is genuinely fine. The pain compounds when you need it fresh: re-downloading 46-74 MB files, babysitting the 403s, re-checking the column layout, decoding county/status/type codes for eight different boards (electrical, home inspectors, mold, cosmetology... each its own file).

The shortcut: one input, clean records

I maintain an Apify actor that streams the official extracts and hands back decoded, filterable records — county names instead of codes, ISO dates, human-readable statuses and license types, QB rows skipped, proxy business handled:

Florida License Records Scraper (DBPR)

{
    "profession": "construction",
    "licenseTypes": ["CGC"],
    "counties": ["Palm Beach"],
    "status": "current-active",
    "expiresBefore": "2026-12-31",
    "maxResults": 5000
}

Statewide pulls take about a minute, export as JSON/CSV/Excel, and pricing is per record returned — a 1,000-contractor county list costs a couple of dollars. Eight DBPR boards are covered: construction, electrical, home inspectors, mold services, cosmetology, barbers, veterinary, architecture.

The part that actually solves compliance: monitoring mode

Bulk verification has a shelf life of exactly one day — DBPR regenerates the files that often. What compliance teams really want isn't a snapshot, it's an alarm: tell me when something changes.

The actor has a monitor mode for that. Give it a watch list (or a filter set), run it on an Apify Schedule, and each run returns only what changed since the last one:

{
    "mode": "monitor",
    "profession": "construction",
    "licenseNumbers": ["CGC058548", "CCC1330911", "CFC1425030"],
    "alertOnStatusChange": true,
    "expiringWithinDays": 90
}

The first run saves a baseline. Every scheduled run after that emits change records only: status-changed (Current → Suspended is the one you're paying attention for), renewed, expiring-soon (once per license as it enters your 90-day window, with a daysUntilExpiration field), new-license, and removed-from-extract — that disappearing-contractor signal from point 6, delivered instead of silently missed.

Because pricing is per record returned, a run where nothing changed costs roughly a cent of compute and zero record charges. Watching 500 subcontractors costs approximately nothing until the week one of them gets suspended — which is the week it pays for itself a few hundred times over. Wire the run into a Slack/email integration and you're done: set-and-forget license compliance.

Recap

Florida publishes its complete contractor license roll as free public extract files — the data access problem is solved.
The files are honest work to use: headerless 22-column CSVs, coded everything, datacenter-IP blocking, and meaningful absences.
Decode it yourself with ~25 lines of pandas for a one-off; use the actor for clean filtered pulls; use monitor mode on a schedule when what you actually need is "alert me when a license changes."

Fighting with a different state's license data? Drop a comment — Texas TDLR and California CSLB are next on my list.

Query every dentist (or physician) in a metro without hitting NPPES's 1,200-record wall

Steven Carleton — Wed, 12 Aug 2026 14:07:29 +0000

Every US healthcare provider — every physician, dentist, nurse practitioner, physical therapist, pharmacy, and clinic — has a National Provider Identifier, and the whole registry is public. CMS runs it as NPPES, and it has a free, keyless API. If you sell to providers (dental supplies, medical devices, EHR/billing software, staffing), this is the authoritative list of your entire market, with practice addresses and phone numbers, for free.

There's one catch, and it's the reason so many "NPI scrapers" quietly return garbage: the NPPES API hard-caps every search at 1,200 records — and past that cap it doesn't error, it silently repeats the same page. If a metro has 4,000 dentists and you page naively, you'll get 1,200 real ones followed by duplicates of the last page, feel like you got "a lot of data," and never realize two-thirds of your market is missing.

This post explains the wall, shows the manual way through it, and gives you a copy-paste Python fix.

The API, and the wall

The base call is simple. Dentists in Florida:

curl "https://npiregistry.cms.hhs.gov/api/?version=2.1&\
taxonomy_description=Dentist&state=FL&limit=200"

limit maxes out at 200 per call. To go deeper you add skip:

curl "https://npiregistry.cms.hhs.gov/api/?version=2.1&\
taxonomy_description=Dentist&state=FL&limit=200&skip=1000"

And here's the wall: skip maxes out at 1,000. 1,000 skipped + 200 returned = 1,200 records, full stop. Ask for skip=1200 and the API doesn't say "out of range" — it hands you the skip=1000 page again. Naive pagers treat that as more data and append duplicates forever. (You'll also notice very broad searches — a whole state with no other filter — get rejected outright; NPPES requires at least one criterion beyond state.)

So the real problem isn't fetching data. It's fetching complete data for any search that legitimately has more than 1,200 matches — which is most useful ones (dentists in Miami, PTs in Houston, family medicine in all of California).

The manual fix: subdivide until every slice fits under 1,200

The trick is to split one too-big search into several small-enough searches, then merge and de-duplicate. The cleanest axis to split on is ZIP code prefix, because NPPES supports trailing wildcards on postal_code. "All Miami dentists" (too big) becomes "dentists in 331xx," then if that's still over 1,200, "3310x, 3311x, 3312x…," and so on. You recurse only into the slices that are actually full, and you de-duplicate the final set by NPI (a provider with a practice and a mailing address in different ZIPs can match twice).

Here's the whole thing in Python — no dependencies beyond requests:

import requests

API = "https://npiregistry.cms.hhs.gov/api/"
CAP = 1200          # NPPES hard cap per search
PAGE = 200          # max limit per call

def _page_all(params):
    """Fetch up to the 1,200 cap for one exact search."""
    out, skip = [], 0
    while skip <= 1000:
        r = requests.get(API, params={**params, "version": "2.1",
                                       "limit": PAGE, "skip": skip}, timeout=30)
        r.raise_for_status()
        results = r.json().get("results", [])
        if not results:
            break
        out.extend(results)
        if len(results) < PAGE:
            break
        skip += PAGE
    return out

def search(base, zip_prefix=""):
    """Recursively subdivide by ZIP prefix so no slice exceeds the cap."""
    params = dict(base)
    if zip_prefix:
        params["postal_code"] = zip_prefix + "*"
    hits = _page_all(params)
    # If we came back pinned at the cap, this slice is probably truncated —
    # split it into 10 narrower ZIP prefixes and recurse.
    if len(hits) >= CAP and len(zip_prefix) < 5:
        deeper = []
        for d in "0123456789":
            deeper.extend(search(base, zip_prefix + d))
        return deeper
    return hits

def dedupe(records):
    seen, out = set(), []
    for rec in records:
        npi = rec.get("number")
        if npi and npi not in seen:
            seen.add(npi)
            out.append(rec)
    return out

# Every dentist in greater Miami, complete:
raw = search({"taxonomy_description": "Dentist", "state": "FL"}, zip_prefix="331")
providers = dedupe(raw)
print(f"{len(providers)} unique dentists (naive paging would have stopped at 1,200)")

Run that against a dense metro and you'll see it sail past 1,200 — a Miami-area dentist search returns close to 2,000 unique providers, not the 1,200 a single search reports. That gap is exactly the providers a naive scraper never sees.

A few things to add before you rely on it:

Be polite. Add a small delay between calls and a retry/backoff — NPPES will throttle a tight loop. The recursion above can fire dozens of calls for a dense metro.
Pull the fields you actually need out of each record: number (the NPI), the taxonomy marked primary for specialty and license, and the address where address_purpose == "LOCATION" for the practice phone. NPPES nests these in arrays, so a naive record["phone"] won't exist.
Emails aren't in here. NPPES publishes practice phone, fax, and address, plus the authorized official's name and phone for organizations — but not provider email. Any tool selling you "NPI emails" is enriching from somewhere else; treat those with the skepticism they deserve.

The maintained shortcut

I wrapped this exact approach — the cap detection, the ZIP fan-out, the de-duplication, the field extraction, the backoff — into an Apify actor so you don't have to own the recursion:

NPI Registry Scraper (NPPES Healthcare Providers)

The complete-Miami-dentists query becomes:

{
    "taxonomyDescription": "Dentist",
    "state": "FL",
    "postalCode": "331*",
    "maxResults": 2000
}

It returns clean, flat records — npi, name, credential, specialty, licenseNumber, licenseState, practiceAddress (with phone and fax), authorizedOfficial for organizations — and hands you JSON/CSV/Excel, scheduling, and webhooks. You're charged per record returned, so a ~2,000-provider metro list costs about four dollars. The point of paying isn't access to the data (it's free and keyless); it's that the 1,200-cap fan-out, the paging, and the field-unnesting are already correct and maintained.

Which route should you take?

One search that's clearly under 1,200 (a single small city and specialty) → just page it yourself with the first snippet. No fan-out needed.
Any dense metro or statewide specialty, on a schedule, or feeding a CRM or AI agent → use the fan-out. The actor is the least-effort version of it (and it's callable as an MCP tool via Apify, so "how many pediatric dentists are in the Dallas metro?" becomes a one-line agent query).
A one-off count for a single ZIP → honestly, the NPPES website is fine; don't automate a thing you'll do once.

The registry is public infrastructure paid for with tax dollars. The only thing standing between you and a complete provider list is knowing that 1,200 isn't the end — it's where the naive scrapers give up.

Hit a specialty or metro where the fan-out behaves oddly? Drop a comment — the taxonomy descriptions have some sharp edges (there are three different flavors of "nurse practitioner" alone) and I'm happy to compare notes.

Build a fresh trucking-insurance lead feed: every new US carrier with phone + email, updated weekly

Steven Carleton — Wed, 12 Aug 2026 14:01:58 +0000

When a trucking company registers with the FMCSA and gets its authority, the clock starts on a bunch of things it now legally has to buy: primary liability and cargo insurance, a BOC-3 process agent, an ELD, often factoring to survive the 30–60 day payment cycle. Whoever reaches that carrier first — in the days after it registers, while it's still shopping — wins the account.

The good news for anyone selling into that market: the list of who just registered is a public federal record, published on an official US DOT open-data portal, with phone and email on file for essentially every carrier registered in the last couple of years. You do not need to scrape SAFER, solve captchas, or buy a $300/month lead list. You need one HTTPS request and a filter.

This post shows how to pull it yourself, where it gets fiddly, and how to turn it into a scheduled weekly feed.

Where the data lives

Everything SAFER shows you about a carrier — legal name, DBA, address, phone, email, company officers, fleet size, cargo, hazmat flag, safety rating — is rendered from FMCSA's Company Census File. FMCSA publishes that census on data.transportation.gov, the US DOT's open-data portal, which runs on Socrata. That means the same SODA API you'd use for any city open-data set: plain HTTPS in, JSON out, SQL-ish query params, no API key for moderate use.

First, find the current dataset. Portal dataset IDs occasionally change when an agency re-publishes, so don't hard-code one you found in a blog post — ask the catalog:

curl "https://api.us.socrata.com/api/catalog/v1?domains=data.transportation.gov&q=motor%20carrier%20census&only=datasets"

That returns the dataset's id (a four-four Socrata token like az4n-8mr2) and its resource URL. From there the census is queryable at:

https://data.transportation.gov/resource/<dataset-id>.json

Pull the five most-recently-added carriers:

curl "https://data.transportation.gov/resource/az4n-8mr2.json?\
\$order=add_date%20DESC&\$limit=5"

The annoying parts (there are three)

1. The column names are not what you'd guess. The census predates every naming convention you like. Depending on the published extract, the phone field is telephone or phone, email is email_address, physical state is phy_state, the power-unit count is nbr_power_unit (not power_units), and the "when did this carrier first appear" date is add_date. Before you build anything, hit the dataset's own API-docs page (linked from the portal page) or just pull one row and read its keys:

curl "https://data.transportation.gov/resource/az4n-8mr2.json?\$limit=1" | python3 -m json.tool

2. "New carrier" is a date field, and dates on Socrata are strings. Filter server-side with $where against add_date, ISO-formatted:

curl "https://data.transportation.gov/resource/az4n-8mr2.json?\
\$where=add_date%20%3E=%20'2026-06-01T00:00:00'&\
\$order=add_date%20DESC&\$limit=1000"

3. Contact completeness varies by age. Carriers registered recently almost always have both phone and email on file; older records have gaps. If you're building a contactable lead feed, filter for the fields you actually need (WHERE email_address IS NOT NULL) rather than assuming.

DIY: a weekly new-carrier feed in ~30 lines

Here's a self-contained Node script: every active carrier added since a given date, in your states, with phone and email, newest first.

const DATASET = 'az4n-8mr2'; // verify via the catalog query above
const BASE = `https://data.transportation.gov/resource/${DATASET}.json`;

async function newCarriers({ states, addedAfter, limit = 1000 }) {
  const stateList = states.map((s) => `'${s}'`).join(',');
  const where = [
    `add_date >= '${addedAfter}T00:00:00'`,
    `phy_state in (${stateList})`,
    `telephone IS NOT NULL`,
    `email_address IS NOT NULL`,
  ].join(' AND ');

  const params = new URLSearchParams({
    $where: where,
    $order: 'add_date DESC',
    $limit: String(limit),
  });

  const rows = await (await fetch(`${BASE}?${params}`)).json();
  return rows.map((r) => ({
    dotNumber: r.dot_number,
    legalName: r.legal_name,
    dbaName: r.dba_name || null,
    phone: r.telephone,
    email: r.email_address,
    state: r.phy_state,
    powerUnits: r.nbr_power_unit ? Number(r.nbr_power_unit) : null,
    addedDate: (r.add_date || '').slice(0, 10),
    safer: `https://safer.fmcsa.dot.gov/query.asp?searchtype=ANY&query_type=queryCarrierSnapshot&query_param=USDOT&query_string=${r.dot_number}`,
  }));
}

const leads = await newCarriers({ states: ['TX', 'OK'], addedAfter: '2026-06-01' });
console.log(`${leads.length} contactable new carriers`);
console.log(leads.slice(0, 5));

Cron that weekly, diff against last week's DOT numbers so you only email each carrier once, and you have a real lead pipeline for the cost of nothing.

One data-source gotcha that will bite you if you don't know it: FMCSA populates a carrier's cargo-classification flags about two months after it first registers. So if you filter for, say, "refrigerated" carriers and "added in the last 30 days," you'll get almost nothing — not because they don't exist, but because that field hasn't been backfilled yet. Filter on cargo type for established carriers, and on add_date for fresh ones; don't combine the two on a tight recent window.

Things you'll end up building on top of it

Within a week of using the raw feed seriously you'll want: fleet-size banding (nbr_power_unit between 1 and 10 is the factoring/fuel-card sweet spot), interstate-vs-intrastate filtering, hazmat and safety-rating fields, company-officer extraction (the decision-maker's name is in there), de-duplication across weekly runs, the ~1,000-row Socrata page limit handled with $offset paging, and retry/backoff. None of it is hard. All of it is maintenance you now own.

The maintained shortcut

I package exactly this — the census, streamed and filtered server-side, with all the fields above normalized into clean names and the fleet/cargo/hazmat/date filters as simple inputs — as an Apify actor:

FMCSA Motor Carrier Scraper

The weekly-insurance-lead query becomes:

{
    "states": ["TX", "OK"],
    "statuses": ["active"],
    "addedAfter": "2026-06-01",
    "requireEmail": true,
    "requirePhone": true,
    "maxResults": 1000
}

Every record comes back in the same clean shape — dotNumber, legalName, dbaName, phone, email, companyOfficers, powerUnits, cargoTypes, safetyRating, saferUrl — and Apify gives you JSON/CSV/Excel export, a scheduler (run it every Monday at 6am), and webhooks into your CRM. You're charged per record returned, so a weekly pull of a few hundred fresh carriers costs about a dollar. It also handles the paging, the cargo-lag warning, and re-discovers the dataset ID if DOT re-publishes it — the babysitting I described above.

Which route should you take?

One state, personal prospecting → the script above. It's 30 lines and the data is free.
Multiple states, weekly scheduling, fleet/cargo filters, CRM webhooks, or feeding an AI agent → the actor (it's also callable as an MCP tool via Apify, so "any new reefer carriers in Texas this week?" becomes a one-line agent query).
Enterprise BD with dialer integration and territory routing → that's when the dedicated trucking-lead SaaS earns its subscription.

Either way the underlying record is a public federal registration — official, free, and yours to use. That's the part most people selling "exclusive carrier leads" would rather you didn't know.

Questions about a specific census field or a state's registration volume? Drop a comment — I've spent more time in this dataset's column names than I'd like to admit.

Track SAM.gov contract opportunities without an API key (or a $300/month GovCon subscription)

Steven Carleton — Wed, 12 Aug 2026 13:56:01 +0000

If you sell anything to the US federal government, you live and die by SAM.gov's Contract Opportunities — every proposed contract action over $25,000 gets posted there. The problem is consuming it:

The website is slow, aggressively bot-protected, and unusable for bulk work.
The official API requires a SAM.gov account, an API key that expires every 90 days, and has rate limits that hurt (Google "sam.gov api key" and enjoy the pain threads).
The GovCon SaaS tools that repackage this data start around $100/month and run into the thousands.

Here's the thing almost nobody seems to know: the sam.gov website's own search backend is a public, keyless JSON API. No account. No API key. No 90-day key rotation. It's the same endpoint your browser hits when you search Contract Opportunities — it's just never advertised as an API.

(You may have read older posts pointing at SAM.gov's daily CSV extract on S3 — ContractOpportunitiesFullCSV.csv. That was the best free path for years, but since late July 2026 SAM.gov has been regenerating it as a header-only stub: 47 column names, zero rows. If your pipeline was built on it, that's why it's suddenly empty. The search API below is the working replacement — and unlike the 200+ MB daily download, it filters server-side.)

The endpoint

https://sam.gov/api/prod/sgs/v1/search/

Standard query parameters, JSON out. The important ones:

Param	Meaning
`index=opp&mode=search&responseType=json`	boilerplate — always send these
`page` / `size`	paging; `size` max 100
`sort=-modifiedDate`	newest first
`is_active=true`	only still-open notices
`naics`	comma-separated NAICS codes (prefixes work)
`pop_state`	place-of-performance states, e.g. `TX,FL`
`set_aside`	set-aside codes (`SBA`, `8A`, `WOSB`, `SDVOSBC`...)
`publish_date.from` / `.to`	needs an explicit UTC offset, e.g. `2026-07-01-04:00`
`q` + `qMode=ALL`	keyword search

Check it yourself — active custom-programming solicitations, newest first:

curl "https://sam.gov/api/prod/sgs/v1/search/?index=opp&mode=search&responseType=json&sort=-modifiedDate&size=5&page=0&is_active=true&naics=541511" \
  -H "Accept: application/hal+json"

Two things the search results don't carry: the contracting officer's contacts and the full description. Those live on a second keyless endpoint, one call per notice:

https://sam.gov/api/prod/opps/v2/opportunities/<noticeId>

DIY: a NAICS watcher in ~35 lines of Python

import requests

SEARCH = "https://sam.gov/api/prod/sgs/v1/search/"
DETAIL = "https://sam.gov/api/prod/opps/v2/opportunities/{}"
HEADERS = {"Accept": "application/hal+json"}

params = {
    "index": "opp", "mode": "search", "responseType": "json",
    "sort": "-modifiedDate", "size": 100, "page": 0,
    "is_active": "true",
    "naics": "541511,541512",            # custom programming, systems design
    "publish_date.from": "2026-07-01-04:00",
}

while True:
    data = requests.get(SEARCH, params=params, headers=HEADERS, timeout=60).json()
    results = data.get("_embedded", {}).get("results", [])
    if not results:
        break
    for r in results:
        # contacts + description live on the detail endpoint
        d = requests.get(DETAIL.format(r["_id"]), headers=HEADERS, timeout=60).json()
        data2 = d.get("data2", {})
        poc = next(iter(data2.get("pointOfContact") or []), {})
        due = (data2.get("solicitation", {}).get("deadlines", {}).get("response") or "?")[:10]
        print(f'{(r.get("publishDate") or "")[:10]}  {r["title"][:70]}  '
              f'due={due}  poc={poc.get("email", "")}  '
              f'https://sam.gov/opp/{r["_id"]}/view')
    if (params["page"] + 1) * params["size"] >= data["page"]["totalElements"]:
        break
    params["page"] += 1

That's a working daily bid-pipeline feed with zero accounts and zero dollars. Cron it, pipe it to Slack, done.

Things you'll add within a week of using it seriously: the per-notice detail calls are 1:1 with results, so you'll want concurrency with a cap and retry/backoff (sam.gov will throttle a tight loop — it's the same bot protection the website has); the API stops paging at 10,000 records per query, so broad searches need date-windowing; the date params silently return wrong windows without that UTC offset; set-aside and notice-type filtering take internal codes; and you'll want dedupe across days. None of it is hard; all of it is maintenance.

The maintained shortcut

I package exactly this — the same search API the sam.gov site uses, paged, enriched with the per-notice contact details, all filters server-side, with the (currently broken) CSV extract kept as an automatic fallback source — as an Apify actor:

SAM.gov Contract Opportunities Scraper

Everything still biddable in construction trades, small-business set-aside, in Texas or Florida:

{
    "naicsCodes": ["236", "237", "238"],
    "setAsides": ["SBA"],
    "popStates": ["TX", "FL"],
    "postedAfter": "2026-06-01",
    "responseDueAfter": "2026-07-11",
    "maxResults": 1000
}

Out comes clean JSON/CSV/Excel with structured contacts, award data on award notices, and direct SAM.gov links. Schedule it daily in Apify, add a webhook, and you have the core of the $300/month GovCon alert products for about the price of a coffee per month (pricing is per record returned).

Which route should you take?

One NAICS code, personal use → the Python script above. Seriously, it's ~35 lines.
Multiple filters, scheduling, teammates who want CSVs, or feeding an AI agent → the actor (it's also callable as an MCP tool via Apify, which makes "any new cybersecurity RFPs today?" a one-line agent query).
Enterprise BD with pipeline analytics → that's when the SaaS subscriptions earn their keep.

The data itself is public, official, and free either way — which is exactly how government contracting data should be.

Building permits are the best construction leads nobody uses — here's how to pull them from official city open data

Steven Carleton — Wed, 12 Aug 2026 13:55:56 +0000

Every building permit is a public record of someone about to spend money on construction: the address, the type of work, often the declared project value and the contractor doing it. If you sell building materials, dumpster rentals, insurance, solar, HVAC service plans — or you are a contractor watching your competitors — permits are the earliest buying signal that exists.

And most US cities publish them, daily, for free, on official open-data APIs. Not scraped, not resold, not 30 days stale: the city's own database.

This post shows how to pull them yourself with nothing but curl, what the data looks like, where it gets annoying (every city names its columns differently), and how to automate the whole thing.

Where permits live: the SODA API

Most large US cities run their open-data portals on Socrata, which exposes every dataset through the SODA API: plain HTTPS, JSON out, SQL-ish query parameters in. No API key required for moderate use.

Chicago's building permits, newest first:

curl "https://data.cityofchicago.org/resource/ydr8-5enu.json?\
\$order=issue_date%20DESC&\$limit=5"

Filter server-side with $where — say, permits issued this month:

curl "https://data.cityofchicago.org/resource/ydr8-5enu.json?\
\$where=issue_date%20%3E=%20'2026-07-01T00:00:00'&\
\$order=issue_date%20DESC&\$limit=100"

You get structured JSON with the permit number, work description, fees, reported cost, and — in Chicago's case — up to five named contacts including the contractor.

A few dataset IDs to get you started (all verified working as of July 2026):

City	Portal	Dataset ID
New York City (DOB NOW)	data.cityofnewyork.us	`rbx6-tga4`
Chicago	data.cityofchicago.org	`ydr8-5enu`
Los Angeles (2020+)	data.lacity.org	`pi9x-tg5x`
Austin	data.austintexas.gov	`3syk-w9eu`
San Francisco	data.sfgov.org	`i98e-djp9`
New Orleans	data.nola.gov	`72f9-bi28`

To find any other city's dataset, query Socrata's catalog API:

curl "https://api.us.socrata.com/api/catalog/v1?domains=data.brla.gov&q=building%20permits&only=datasets"

The annoying part: every city is a special snowflake

Here's where the DIY route gets expensive. The issue date is issue_date in Chicago, issued_date in San Francisco, issuedate in New Orleans, and issueddate in Baton Rouge. Valuation is reported_cost, estimated_cost, valuation, estprojectcost, or declaredvaluation depending on who you ask. NYC's feed includes rows for permits that aren't issued yet, with the date simply missing. One city publishes latitude/longitude as columns, another nests them in a GeoJSON point, a third uses a geolocation object.

If you only care about one city, write the mapping once and move on. If you want a multi-city feed, you're maintaining a normalization layer — and re-verifying dataset IDs whenever a city migrates portals (NYC's old ipu4-2q9a dataset silently stopped updating in 2020; the live one is rbx6-tga4; Seattle and Dallas left Socrata entirely).

Here's a minimal normalizer for two cities to show the shape of the problem:

const CITIES = {
  chicago: {
    url: 'https://data.cityofchicago.org/resource/ydr8-5enu.json',
    dateField: 'issue_date',
    map: (r) => ({
      permitNumber: r.permit_,
      permitType: r.permit_type,
      description: r.work_description,
      issuedDate: r.issue_date?.slice(0, 10),
      address: [r.street_number, r.street_direction, r.street_name].filter(Boolean).join(' '),
      valuation: r.reported_cost ? Number(r.reported_cost) : null,
    }),
  },
  'san-francisco': {
    url: 'https://data.sfgov.org/resource/i98e-djp9.json',
    dateField: 'issued_date',
    map: (r) => ({
      permitNumber: r.permit_number,
      permitType: r.permit_type_definition,
      description: r.description,
      issuedDate: r.issued_date?.slice(0, 10),
      address: [r.street_number, r.street_name, r.street_suffix].filter(Boolean).join(' '),
      valuation: r.estimated_cost ? Number(r.estimated_cost) : null,
    }),
  },
};

async function fetchPermits(cityKey, issuedAfter, limit = 100) {
  const city = CITIES[cityKey];
  const params = new URLSearchParams({
    $limit: String(limit),
    $order: `${city.dateField} DESC`,
    $where: `${city.dateField} >= '${issuedAfter}T00:00:00'`,
  });
  const rows = await (await fetch(`${city.url}?${params}`)).json();
  return rows.map(city.map);
}

console.log(await fetchPermits('chicago', '2026-07-01', 10));

The shortcut: one normalized API for 9 cities

I maintain an Apify actor that does exactly this across nine cities/counties (NYC, Chicago, LA, Austin, SF, New Orleans, Baton Rouge, Montgomery County MD, Norfolk VA), with one shared schema, server-side date filtering and full-text search, and the dataset-ID babysitting handled for you:

US Building Permits Scraper

One input gets you fresh, high-value leads across cities:

{
    "cities": ["austin", "chicago"],
    "issuedAfter": "2026-07-01",
    "minValuation": 50000,
    "maxResultsPerCity": 1000
}

Every record comes back in the same shape — permitNumber, permitType, description, issuedDate, address, valuation, contractorName where the city publishes it, coordinates — and Apify hands you JSON/CSV/Excel export, scheduling (run it every Monday morning), and webhooks into your CRM for free. Pricing is per record returned, so a weekly 1,000-lead pull costs a few dollars.

Bonus: turn permits into qualified contractor lists

Permits tell you who's building. State license records tell you who's licensed — with status, expiration date, and mailing address. Cross-referencing the two is a genuinely underrated play: match permit contractor names against the state's license roll and you get, e.g., "active Florida GCs who pulled zero permits this quarter" (prime targets for lead-gen services) or "contractors whose license expires within 90 days" (renewal/CE marketing).

Florida publishes its entire license database as public CSV extracts, and there's an actor for that too: Florida Contractor & Professional License Search (DBPR). A join on normalized business names between the two datasets is an afternoon of pandas and surprisingly good fun.

Recap

City open-data portals publish building permits daily through the SODA API — free, official, structured.
The pain is normalization and dataset churn, not access.
DIY one city with 30 lines of code; use a maintained multi-city actor when you want breadth and scheduling without the babysitting.

Questions about a specific city's dataset? Drop a comment — I've probably already fought with its column names.