Tracking Municipal Equipment Auctions Across States Without Web Scraping Logic

#beginners #devops #webdev #webscraping

Government surplus inventory is fragmented across thousands of municipal, county, and state agencies. When public utilities or school districts retire vehicles, heavy machinery, or office hardware, they frequently list these assets on GovDeals. For data engineers building automated procurement pipelines or valuation models, parsing this data manually or building custom scrapers against dynamic marketplace search pages introduces brittle selector maintenance and rate-limiting issues.

The GovDeals Scraper Actor extracts structured data directly from GovDeals public listing endpoints without requiring custom browser automation scripts, login sessions, or dedicated proxy management.

Query Modes for Surplus Datasets

The scraper operates in three distinct modes specified by the mode parameter:

search: Free-text search matching listing titles and descriptions via the searchText field.
byCategory: Extracts listings from GovDeals' top-level category taxonomy using a category identifier.
byState: Pulls active or closed inventory located within a specific US state code (for example, TX, CA, or OH).

Using target filters at the query level reduces unnecessary output events. If you only need equipment from specific manufacturers, you can pass structured fields like vehicleMake (e.g., Caterpillar or Ford) and vehicleModelYear directly in the payload rather than filtering records post-ingestion.

Filtering by Auction State and Price

By default, the Actor looks for live listings with auctionStatus: "open". However, if your goal is training price-prediction models or running historical valuation benchmarks, setting auctionStatus to "closed" returns historical sold listings.

You can also bound extraction runs using minCurrentBid and maxCurrentBid to filter out low-value incidental items or restrict items by sale format using auctionType (such as Online Auction, Sealed Bid, Buy Now, or Make Offer).

{
  "mode": "search",
  "searchText": "excavator",
  "auctionStatus": "open",
  "minCurrentBid": 5000,
  "maxCurrentBid": 50000,
  "maxItems": 100
}

Running the Extraction via Python

The scraper outputs normalized records containing asset identifiers (assetId, auctionId), auction terms (currentBid, buyNowPrice, bidIncrement), seller metadata (sellerName), location details (locationCity, locationState, locationZip), and asset parameters (makeBrand, model, modelYear). Empty attributes are omitted from individual records.

Here is a minimal implementation using the official Apify Python SDK to fetch closed commercial vehicle auctions in Ohio:

from apify_client import ApifyClient

client = ApifyClient("YOUR_APIFY_TOKEN")

run_input = {
    "mode": "byState",
    "state": "OH",
    "auctionStatus": "closed",
    "vehicleMake": "Ford",
    "maxItems": 50,
    "sortBy": "endingSoonest"
}

# Start the Actor run and wait for completion
run = client.actor("crawlerbros/govdeals-scraper").call(run_input=run_input)

# Fetch results from the default dataset
dataset_items = client.dataset(run["defaultDatasetId"]).list_items().items

for item in dataset_items:
    title = item.get("title")
    seller = item.get("sellerName")
    closing_bid = item.get("currentBid", "No bids")
    print(f"{title} | Seller: {seller} | Final Bid: {closing_bid}")

Parsing Output Records

The returned dataset normalizes dates and numeric pricing fields, allowing direct insertion into downstream analytical stores. A standard payload item emitted under recordType: "asset" contains the following structure:

{
  "assetId": "12345",
  "accountId": "6789",
  "auctionId": "54321",
  "title": "2018 Ford F-250 Super Duty",
  "makeBrand": "Ford",
  "model": "F-250",
  "modelYear": 2018,
  "categoryId": "6",
  "categoryName": "Automobiles, Trucks, Vans, SUVs",
  "auctionTypeId": "1",
  "auctionTypeName": "Online Auction",
  "currentBid": 14500,
  "currencyCode": "USD",
  "sellerName": "City of Columbus",
  "locationCity": "Columbus",
  "locationState": "OH",
  "locationZip": "43215",
  "auctionEndDate": "2024-03-15T18:00:00Z",
  "auctionStatus": "open",
  "isReserveNotMet": false,
  "imageUrl": "https://www.govdeals.com/photos/12345/full.jpg",
  "sourceUrl": "https://www.govdeals.com/index.cfm?fa=Main.Item&itemid=12345&acctid=6789",
  "recordType": "asset",
  "scrapedAt": "2024-03-12T10:15:30.123Z"
}

If an item receives no bids, the currentBid key is omitted from the object rather than emitted as zero or null. Similarly, makeBrand and modelYear only appear when parsing vehicle or equipment taxonomies.

Execution Walkthrough

Define your criteria: Select either search, byCategory, or byState mode based on whether you are tracking a specific inventory type or monitoring a geographic jurisdiction.
Apply constraints: Populate vehicleMake, price thresholds (minCurrentBid/maxCurrentBid), or auctionStatus inside the input configuration to limit the result set.
Set the output limit: Provide an integer for maxItems (between 1 and 2000) to control dataset volume.
Execute and consume: Trigger the Actor using the API or SDK, then stream the dataset items into your downstream staging tables or data lake.

Event-Based Pricing Structure

Billing for this Actor follows a pay-per-event pricing model rather than traditional compute allocation models. Charges are assessed strictly on two event types:

Actor Start (apify-actor-start): $0.005 per GB of memory allocated to the run, charged once per run.
Result (apify-default-dataset-item): $0.005 per emitted item on the FREE tier ($0.00433 on BRONZE, $0.00367 on SILVER, and $0.003 on GOLD, PLATINUM, and DIAMOND tiers).

Extracting a batch of 200 listing records on the standard FREE tier incurs one start event at $0.005 (assuming 1 GB memory) plus 200 result events at $0.005 each ($1.00), for a total run cost of $1.005.

This tool does not provide continuous websocket streaming or sub-second notification of live bids; it captures point-in-time snapshots of auction metadata when executed. Pipelines requiring real-time outbid notifications within the final seconds of an auction must implement continuous polling schedules or look to direct bidding integrations.

GovDeals Scraper is what these steps drive. The README covers the inputs this article skipped, including the ones that change how much a run costs.

DEV Community