Handling Imperva Cookie Handshakes for Massachusetts Entity Data

Automating business verification for Massachusetts entities often runs into a specific infrastructural barrier. The Massachusetts Corporations Division search (corp.sec.state.ma.us) hosts registry data—entity type, registered agent name, principal office address, and active officers. However, the search form sits behind Imperva/Incapsula protection.

Unlike registry portals that deploy full JavaScript execution walls (such as Nevada's SilverFlume portal, which returns a static 1 KB _Incapsula_Resource iframe on raw HTTP requests), the Massachusetts registry operates protection primarily at the cookie level. Standard headless browser setups add unnecessary overhead when pure HTTP session warming can solve the access problem directly.

The massachusetts-sos-scraper actor bypasses this barrier over HTTP, collecting registry data and structuring it for downstream data pipelines.

Bypassing Cookie Challenges Over Plain HTTP

When a standard HTTP client hits the Massachusetts search portal for the first time, Imperva returns an interstitial response. If your scraper does not maintain the session state and evaluate cookie headers, subsequent search requests fail or get dropped.

The scraper handles this protocol interaction automatically:

1. It sends an initial warm-up HTTP request to capture the seeded visid_incap_* and incap_ses_* session cookies.
2. It re-uses that initialized session for subsequent form submissions against corp.sec.state.ma.us.
3. It queries the registry search endpoints and parses each entity summary page (CorpSummary.aspx) without requiring a headless browser.

Because every search match requires an independent detail-page request to extract complete records (such as officer tables and merger history), the actor uses deliberate delays between requests and falls back to free datacenter (AUTO) proxies if the target endpoint applies temporary rate limits.

Search Axes and Filtering Modes

The scraper accepts four operational modes via the mode parameter: byEntityName, byIndividualName, byIdentificationNumber, and byFilingNumber.

{
  "mode": "byEntityName",
  "entityName": "Acme Logistics",
  "entityMatchMode": "M",
  "maxItems": 10
}

Entity Name Matching

When querying with mode: "byEntityName", the entityMatchMode parameter controls how the Massachusetts backend filters company records:

• B (Begins with): Returns entities starting with the search string.
• M (Exact match): Restricts results to exact legal name matches.
• F (Full text): Finds the query substring anywhere in the registered legal name.
• S (Soundex): Matches phonetically similar company names.

Individual Lookups for Officers and Agents

When auditing individuals or mapping corporate networks, setting mode: "byIndividualName" queries the registry's officer and registered agent index. The search requires lastName, with optional narrowing via firstName and middleName.

{
  "mode": "byIndividualName",
  "lastName": "SMITH",
  "firstName": "JOHN",
  "individualMatchMode": "M",
  "maxItems": 25
}

Because individual search results return an entry for every distinct appointment across all registered entities, common surnames yield dozens of separate corporate records. Using individualMatchMode: "M" prevents unbounded result lists.

Derived Inactivity Status and Data Structure

The Massachusetts portal does not provide a standard status: "Active" string on its summary page. Instead, inactive corporations display an explicit inactivity label (such as Date of Involuntary Dissolution by Court Order or by the SOC) alongside a dissolution date, while active entities leave this section empty.

The scraper normalizes this quirk into structured fields:

• status: Outputs either "Active" or "Inactive".
• statusReason: Contains the exact dissolution or revocation string provided by the state when inactive.
• statusDate: Formatted ISO date (YYYY-MM-DD) when inactive.
• officers: Array of objects containing { title, name, address }.
• mergedWith: Array of absorbed entities ({ entityName, date, sourceUrl }), linking directly to the merged entity's historical summary page.
• previousNames: Historical record of legal name changes ({ previousName, date }).

Empty fields are omitted from the resulting dataset items to keep payloads compact.

Running the Scraper via the Apify Python Client

You can run this extraction pipeline programmatically using Python to pull data directly into your ETL flow.

1. Install the client library:

pip install apify-client

1. Initialize the client and run a targeted search:

from apify_client import ApifyClient

client = ApifyClient("YOUR_APIFY_TOKEN")

run_input = {
    "mode": "byIdentificationNumber",
    "identificationNumber": "000445089",
    "maxItems": 1
}

run = client.actor("crawlerbros/massachusetts-sos-scraper").call(run_input=run_input)
dataset_items = client.dataset(run["defaultDatasetId"]).list_items().items

for entity in dataset_items:
    print(f"Entity: {entity.get('entityName')}")
    print(f"Status: {entity.get('status')} ({entity.get('statusReason', 'N/A')})")
    print(f"Agent: {entity.get('registeredAgentName')}")
    for officer in entity.get("officers", []):
        print(f"  - {officer.get('title')}: {officer.get('name')}")

Pricing and Cost Calculation

The actor uses a pay-per-event pricing model based strictly on executed events.

• Actor Start (apify-actor-start): $0.005 per GB of memory allocated to the run (billed once upon run initialization).
• Result (apify-default-dataset-item): Billed per record written to the dataset.
  • FREE: $0.005 per result
  • BRONZE: $0.00433 per result
  • SILVER: $0.00367 per result
  • GOLD: $0.003 per result
  • PLATINUM: $0.003 per result
  • DIAMOND: $0.003 per result

For example, on the FREE tier, running a batch job that initializes with 1 GB of memory ($0.005) and extracts 100 entity records ($0.50) costs a total of $0.505.

Operational Constraints

This scraper does not download original PDF filings or annual report documents, as raw document retrieval requires the Secretary of the Commonwealth's paid certificate portal. Furthermore, the maxItems property is strictly capped at 100 items per run to prevent registry endpoint blocks during multi-page entity traversals. For large corporate directories, batch your inputs by distinct entity numbers or narrow prefixes rather than executing broad open-ended queries.

---

Massachusetts Secretary of State Business Search Scraper is the Actor behind these examples. If a selector in your own version breaks, compare your output against the fields listed in its README first.