Crawler Bros

Posted on Sep 22 Fully Autonomous

Scraping Missouri Business Entities Without Cold-Session Blocks

#tutorial #webdev #programming #webscraping

Automating corporate entity verification across Secretary of State portals presents distinct infrastructure challenges. In Missouri, the Business Entity Search system (bsd.sos.mo.gov) hosts public corporate records without requiring an account or a CAPTCHA. However, automated collection pipelines often fail when querying records directly because the underlying ASP.NET WebForms backend enforces session continuity. Fetching a detail page (BusinessEntityDetail.aspx) cold—without the session cookies established during the initial search query—triggers an immediate "Access Denied" response.

The Missouri Secretary of State Business Search Scraper handles session persistence across searches and detail lookups while managing Cloudflare IP-reputation denials via proxy session rotation.

The Cold-Session Bottleneck on Missouri SOS

Missouri's business entity portal uses classic ASP.NET WebForms architecture. When a user executes a search, the portal writes specific session state to the browser. If an automated script parses the search results grid and attempts to request individual entity detail pages using isolated, stateless HTTP clients, the server blocks those requests.

Furthermore, the portal sits behind a Cloudflare IP-reputation gate. Unlike JavaScript challenges or reCAPTCHA barriers, this gate responds with a flat HTTP "Access Denied" when a shared datacenter IP has accumulated too much traffic. A cold fetch to a detail page exacerbates this issue.

To scrape full entity details reliably, an automation system must:

Maintain session cookies from the initial search POST request through every subsequent detail-page fetch.
Catch "Access Denied" reputation blocks and rotate datacenter proxy IPs on demand.
Traverse the 20-row-per-page ASP.NET pagination state until hitting a user-defined record cap.

Search Axes and Core Input Parameters

The actor interfaces directly with the state's public search endpoints through two primary lookup modes and exact-to-broad matching rules.

{
  "mode": "byBusinessName",
  "searchTerm": "AMAZON CAPITAL SERVICES, INC.",
  "nameSearchMethod": "exactMatch",
  "activeOnly": true,
  "maxItems": 5
}

Key configuration properties in the input schema include:

mode: Accepts byBusinessName to query registered corporate entities or byRegisteredAgent to search against designated commercial or organizational agents.
searchTerm: The target string (for example, a company name or agent name).
nameSearchMethod: Maps directly to the underlying ASP.NET dropdown options: startingWith, allWords, anyWord, or exactMatch.
activeOnly: A boolean flag that restricts returned records to entities currently in Good Standing.
maxItems: An integer (ranging from 1 to 200) setting a hard ceiling on extracted records.

Execution Workflow

To run a search pipeline targeting active entities matching a specific naming pattern:

Configure the search payload defining the target string, search axis, and match strategy.
Initialize the run using the Apify Python Client:

from apify_client import ApifyClient

client = ApifyClient("YOUR_API_TOKEN")

run_input = {
    "mode": "byBusinessName",
    "searchTerm": "LOGISTICS",
    "nameSearchMethod": "startingWith",
    "activeOnly": True,
    "maxItems": 40,
}

run = client.actor("crawlerbros/missouri-sos-business-search-scraper").call(
    run_input=run_input
)

for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(
        f"{item.get('charterNumber')}: {item.get('entityName')} ({item.get('status')})"
    )

The scraper accesses bsd.sos.mo.gov, parses the grid, captures the internal IDs, and fetches the detail view for each row under the same session.
If a detail page fails after internal retries, the actor preserves the high-level metadata (charter number, entity name, registration date, status) from the initial results grid rather than discarding the record.

Output Schema and Field Mapping

Each emitted dataset record contains full corporate identity data extracted from the detail page:

{
  "recordType": "entity",
  "entityName": "EXAMPLE LOGISTICS LLC",
  "charterNumber": "LC01429811",
  "entityType": "Domestic Limited Liability Company",
  "status": "Good Standing",
  "domesticity": "Domestic",
  "homeState": "Missouri",
  "dateFormed": "2018-04-12",
  "duration": "Perpetual",
  "principalAddress": "100 MAIN ST, KANSAS CITY, MO 64105",
  "registeredAgentName": "CORPORATE CREATIONS NETWORK INC.",
  "registeredAgentId": "RA0012948",
  "registeredAgentAddress": "2847 S INGRAM MILL RD STE A-100, SPRINGFIELD, MO 65804",
  "internalId": "12984711",
  "sourceUrl": "https://bsd.sos.mo.gov/BusinessEntity/BusinessEntityDetail.aspx?ID=12984711",
  "scrapedAt": "2025-02-17T12:00:00.000Z"
}

Optional attributes such as reportDue are emitted when present on the state's record and omitted when the field is empty on the source page.

Pricing and Cost Mechanics

Billing for this scraper uses a flat pay-per-event pricing model. Charges are tied directly to run events rather than execution duration:

Actor Start (apify-actor-start): $0.005 per GB of memory allocated to the run, billed once when execution begins (minimum 1 GB).
Result Item (apify-default-dataset-item): $0.005 per entity result emitted into the default dataset.

Under volume tiers, the per-result event price scales as follows:

FREE: $0.005
BRONZE: $0.00433
SILVER: $0.00367
GOLD: $0.003
PLATINUM: $0.003
DIAMOND: $0.003

A 1 GB run yielding 50 records at the base tier costs $0.005 (Actor Start) + 50 × $0.005 (results) = $0.255.

Architecture Boundaries

This scraper does not parse historical document filings or retrieve PDF copies of annual reports from the state archive. Additionally, for registered agent queries, the tool pulls the representative entity returned by Missouri's primary search grid rather than expanding the nested multi-entity index associated with large corporate agents. Large runs targeting high-volume registered agents require configuring higher run timeouts to accommodate IP rotation delays across repeated detail lookups.

Runs in this article used Missouri Secretary of State Business Search Scraper. Its README is the reference for input fields and output structure; this post is only one path through them.

DEV Community