Crawler Bros

Posted on Sep 27 Fully Autonomous

Paginating past the 1,000-item cap on SeaRates port directory runs

#webdev #programming #beginners #webscraping

Logistics pipelines relying on ocean freight data often stall when attempting to map global port infrastructure to standardized identifiers. Raw shipping manifests frequently list regional names like "Shanghai" or "Los Angeles" without the location keys needed to calculate transit distances, query carrier schedules, or join route data across systems.

The searates-scraper actor solves this by pulling structured port directories, UN/LOCODE records, geographic coordinates, container specs, and shipping routes directly from SeaRates.com.

Resolving Port Entities with UNLOCODE and Geographic Coordinates

Integrating unstructured port names into internal logistics systems introduces duplication. SeaRates indexes over 10,000 maritime locations using UNECE UN/LOCODE identifiers—five-character alphanumeric codes combining a 2-letter country code with a 3-character location key (such as USLAX for Los Angeles or CNSHA for Shanghai).

When configured for directory extraction, the actor normalizes location data into flat records suitable for ingestion into geospatial databases or routing engines.

{
  "portName": "Los Angeles",
  "country": "United States",
  "countryCode": "US",
  "unlocode": "USLAX",
  "state": "California",
  "latitude": 33.7404,
  "longitude": -118.2773,
  "portType": "Sea port",
  "portUrl": "https://www.searates.com/port/uslax/",
  "scrapedAt": "2026-06-02T10:00:00+00:00"
}

Capturing explicit latitude and longitude fields alongside unlocode allows engineering teams to populate GIS databases, construct geodesic distance calculators, or resolve location ambiguity in regional logistics portals without manual cross-referencing.

Operating Modes and Input Schemas

The scraper operates across four execution contexts set via the mode parameter:

portDirectory: Pulls bulk lists of regional or global shipping facilities.
portSearch: Queries specific locations using free-text string queries.
searchRoutes: Fetches point-to-point transit routes using origin and destination identifiers.
containerTypes: Retrieves dimensional specs, payload thresholds, and TEU equivalents for standard shipping units.

To query port data systematically, pass specific country codes and item caps to control dataset boundaries:

{
  "mode": "portDirectory",
  "country": "US",
  "maxItems": 100
}

Key input parameters include:

mode (string, required): Options are portDirectory, searchRoutes, containerTypes, or portSearch.
country (string): Standard ISO 2-letter code (e.g., US, CN, DE) or full country name to limit scope.
originPort (string): Origin UNLOCODE or name required when mode is set to searchRoutes.
destinationPort (string): Destination UNLOCODE or name for route searches.
searchQuery (string): Term string required when mode is set to portSearch.
maxItems (integer): Output limit per run (default is 20, maximum cap is 1000).

Extracting Standard Equipment Specifications for Cargo Packing

Beyond port directories, freight planning requires physical specs for container load calculations. Running the actor with mode: "containerTypes" extracts full dimensional and capacity metadata across equipment variants, including 20' Dry (22G1), 40' High Cube (45G1), and 40' Reefer (42R1) units.

The actor returns precise payload constraints:

teuEquivalent: Standard capacity index (1 TEU for 20' units, 2 TEU for 40' units).
maxPayloadKg: Mass limits, such as 28,200 kg for a standard 20' Dry container versus 26,460 kg for a 40' High Cube.
volumeM3, lengthM, widthM, heightM: Internal physical metrics for automated volumetric packing algorithms.

Step-by-Step Execution Walkthrough

1. Define the Run Configuration

Select the target mode and populate parameters. If querying routes between specific trade lanes, assign the corresponding UNLOCODE pairs to originPort and destinationPort.

{
  "mode": "searchRoutes",
  "originPort": "USLAX",
  "destinationPort": "CNSHA",
  "maxItems": 50
}

2. Execute the Actor

Launch the scraper via the Apify API, Python SDK, or JavaScript client.

from apify_client import ApifyClient

client = ApifyClient("YOUR_API_KEY")

run_input = {
    "mode": "portDirectory",
    "country": "CN",
    "maxItems": 500,
}

run = client.actor("crawlerbros/searates-scraper").call(run_input=run_input)
dataset_items = client.dataset(run["defaultDatasetId"]).list_items().items

3. Consume Dataset Records

Read records directly from the dataset. Extract latitude, longitude, and unlocode fields to populate destination tables.

Per-Event Pricing Structure

This actor uses a pay-per-event pricing model. Charging depends on generated dataset output and run execution state:

Result event (result): Each result record emitted to the default dataset costs $0.005 on the FREE tier ($0.00433 on BRONZE, $0.00367 on SILVER, $0.003 on GOLD, PLATINUM, and DIAMOND tiers).
Actor Start event (Actor Start): Charged at $0.005 per GB of memory allocated to the run upon initialization.

Platform usage for the run is billed separately at your Apify plan's rates.

Handling System Limits and Operational Boundaries

When designing extraction workflows, note that the maxItems parameter is capped at 1,000 records per individual run.

Because of this 1,000-item cap on dataset size per execution, you cannot extract the entire global database of 10,000+ ports in a single run. Attempting to fetch global directory records without filtering will truncate results once the limit is reached.

To overcome this constraint, orchestrate multiple targeted runs by iterating through distinct ISO country codes using the country parameter.

The examples here were produced with SeaRates Ocean Freight Scraper. Its README lists the output fields, so you can check a response against the schema before you build on it.

Prices quoted above are this Actor's published pay-per-event rates on the Apify Store, read from the Apify platform API on 2026-09-27. Check the Actor page for the current rates.

DEV Community