Extracting US Bill-of-Lading Records Past the 25-Request Limit

#webdev #programming #beginners #webscraping

Tracking overseas supply chains or sourcing vetted manufacturers typically requires digging through US Customs bill-of-lading filings. These public records document commercial ocean freight entering the United States, linking US buyers directly to international suppliers. ImportYeti indexes this ocean freight data into a searchable profile system, but querying their public search endpoints directly hits a hard ceiling: anonymous IP addresses are limited to roughly 25 search requests before the platform requires authentication.

Building an internal pipeline to extract these records at scale requires handling session caps, paginating through search results, and parsing unstandardized importer and supplier entities. The ImportYeti US Trade & Supplier Data Scraper handles these constraints by running HTTP queries across rotated proxy pools to extract structured company profiles.

Query Modes: Importers vs. Suppliers

The scraper provides two distinct operational targets via the mode parameter:

searchImporters: Queries US buyers importing goods through maritime freight.
searchSuppliers: Queries overseas manufacturers and exporters shipping goods into the US.

The search mechanism handles both company names and raw product keywords inside the single query field. For example, querying searchImporters with "lithium battery" extracts US entities actively receiving battery shipments, while querying "Home Depot" returns the specific import profile and consolidated alias footprint for that entity.

Example Search Payload

{
  "mode": "searchImporters",
  "query": "lithium battery",
  "maxItems": 100,
  "proxyConfiguration": {
    "useApifyProxy": true
  }
}

The maxItems property accepts integer values from 1 to 500, setting a hard cap on the number of results returned per run. When paginating large result sets, the actor steps through result pages until it hits maxItems or exhausts available public search results.

Output Schema and Normalized Fields

The scraper strips empty fields and outputs normalized records to the default dataset. A standard extracted record includes the following properties:

{
  "companyName": "ACME ENERGY STORAGE LLC",
  "type": "importer",
  "country": "US",
  "address": "123 INDUSTRIAL PARKWAY, SUITE 400, DALLAS, TX",
  "totalShipments": 1420,
  "mostRecentShipment": "2024-01-15",
  "otherAddressesCount": 4,
  "otherNamesCount": 12,
  "trademarks": [
    "ACME POWER",
    "GRIDSAFE"
  ],
  "companyUrl": "https://www.importyeti.com/company/acme-energy-storage",
  "scrapedAt": "2024-03-30T10:14:22.123Z"
}

Key Analytical Fields

totalShipments: Represents lifetime shipment volume attributed to the profile by ImportYeti, acting as a historical scale metric.
mostRecentShipment: Indicates recency of supply chain activity. Customs filings publish with a lag of several weeks, so this date reflects the latest processed filing rather than real-time arrival.
otherNamesCount and otherAddressesCount: Large multinational importers often file under dozens of corporate subsidiaries or slight variations of their legal names. These counts reveal how many discrete entity records ImportYeti has consolidated into the profile.
trademarks: Lists registered trademarks tied to the importer (capped upstream at 20 items to keep payload sizes manageable).

How to Configure and Run the Scraper

Executing a search run involves setting the target category, defining proxy settings to avoid request limits, and capturing the output dataset.

Select the target mode: Define either searchImporters to find domestic buyers or searchSuppliers to map overseas manufacturers.
Define the search query: Input a specific company name (e.g., "Foshan furniture") or an industry keyword (e.g., "solar panel").
Set the extraction volume: Configure maxItems (between 1 and 500) based on your sampling requirements.
Enable Proxy Configuration: Set proxyConfiguration to {"useApifyProxy": true}. The default Apify Proxy AUTO group supplies the necessary IP rotation to cycle around the 25-request anonymous limit.
Run the actor and consume the dataset: Query the generated dataset using standard Apify API endpoints or integrate the run directly into a data pipeline using the Apify Python or Node.js client.

from apify_client import ApifyClient

client = ApifyClient("YOUR_APIFY_TOKEN")

run_input = {
    "mode": "searchSuppliers",
    "query": "textile manufacturing",
    "maxItems": 50,
    "proxyConfiguration": {"useApifyProxy": True},
}

run = client.actor("crawlerbros/import-yeti-scraper").call(run_input=run_input)

for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(
        f"{item.get('companyName')} ({item.get('country')}): {item.get('totalShipments')} shipments"
    )

Pricing and Cost Model

The actor operates strictly on a pay-per-event pricing model. There are no compute duration fees; costs are tied to flat event triggers:

Actor Start (apify-actor-start): $0.005 per GB of memory allocated to the run (charged once at execution start, minimum one event).
Result Output (apify-default-dataset-item): $0.005 per item written to the default dataset under the standard FREE tier.

For higher volume tiers, the per-result event price discounts as follows:

BRONZE: $0.00433 per result
SILVER: $0.00367 per result
GOLD / PLATINUM / DIAMOND: $0.003 per result

Extracting a full batch of 100 importer records on the standard tier with 1 GB memory costs $0.005 (actor start) + $0.50 (100 results × $0.005), totaling $0.505.

Scope Boundaries

This approach only covers ocean freight entering the United States documented on public customs bills of lading; air cargo, land border trucking, and outbound US exports are out of scope and will not appear in results.

The Actor used throughout this walkthrough is ImportYeti US Trade & Supplier Data Scraper. Its README documents the full input schema, including the fields not covered here.

DEV Community