Quick answer
USAspending.gov's Award Search API is public, keyless, and quietly hostile to anything that doesn't look like a browser. The USAspending Federal Awards Scraper sends the exact POST body the API demands, pages until it says there's nothing left, and flattens every federal contract award — recipient, amount, agency, dates — into one billable row, for govcon lead generation and competitive intelligence, at $3.05 per 1,000 awards.
The block that shows up as a 500, not a 403 🛡️
A self-identifying User-Agent gets you a fake server error. api.usaspending.gov sits behind a WAF, and when we probed it with a plain, honest User-Agent identifying the client, it answered with an HTTP 500 and a "Web Page Blocked!" interstitial carrying attack_ID 20000051. Not a 403. Not a 429. A 500 — the status code every retry loop in existence treats as "the server hiccuped, try again." So a naive client burns all five of its retries against a page that was never going to succeed, and the run finishes reporting whatever happened to land before that page, with nobody the wiser that a WAF, not an outage, ate the rest of the request.
That's the worst kind of failure: the silent kind. A 403 at least tells you that you've been blocked. A 500 tells you the target is having a bad day and you should be patient — which is exactly backwards, and exactly why a job can finish "SUCCEEDED" with a fraction of the rows it should have collected. We found this by comparing two otherwise-identical requests: one with a browser-impersonated TLS/H2 fingerprint carrying its own native User-Agent (chrome131, in this case) came back 200 with real award data; the same fingerprint with our own honest UA layered on top came back the block page. Overriding the User-Agent on top of an impersonated fingerprint doesn't make the request look more legitimate — it makes it look like exactly the mismatch anti-bot stacks are built to catch. So we let curl-cffi's impersonation supply a consistent fingerprint end to end, and we keep our attribution in the Store listing, where the target is happy to let it sit, instead of in a header it refuses to serve.
The API also insists on a field most naive clients skip. award_type_codes is a required, enum-validated filter — send an empty list and the whole request 422s before you get a single row. This Actor validates every code against the live-confirmed enum before your run is charged, so a typo in an award type never turns into a wasted, billed request.
Pagination has no total-count field. You only get page_metadata.hasNext. Stop on an assumed page count and you either quit early or loop forever past the last real page; this Actor pages strictly on that flag, on maxResults, or on an empty results array — never on a guess.
The judgement call: an empty first page is a failure, not a success 🔍
Once a run has emitted at least one row, a page that can't be fetched stops the run cleanly and ships what's already collected — a narrow filter combination legitimately runs dry partway through, and that's not an error.
But the first page is different, and we treat it differently on purpose. If page one never lands — WAF block, network fault, whatever — the run has delivered zero rows, and letting it exit "SUCCEEDED" with an empty dataset would be worse than useless: every health dashboard scores that as 100%, and you'd have been charged the start fee for nothing. So a first-page failure raises loudly instead. A narrow filter that legitimately matches nothing still finishes SUCCEEDED with zero rows and a status message — that's a real answer, not a fault, and it shouldn't look like one.
What you get per row 📋
Award ID and a stable internal ID used to build the profile link, recipient name, award amount, top-tier awarding agency plus the specific awarding sub-agency, top-tier funding agency, contract/award type, period-of-performance start and end dates, free-text description, and a ready-made usaspending.gov/award/... profile URL for every row.
Filter by free-text keyword, recipient name, awarding agency, award type code (defaults to the four prime-contract types, A/B/C/D), and an award-action date window — USAspending's documented data floor is 2007-10-01.
What it costs
Pay-per-event: a $0.05 actor-start fee plus $0.003 per award row. A thousand awards runs about $3.05. No API key to request, no subscription, and Apify gives every new account $5 of free credit to try it with no card on file.
Top comments (0)