4 states, 1 open-data API, and the pagination bug that splits a company in half

Four US states publish new-business filings on the same open-data platform. Three of them give you one row per company. Oregon gives you one row per fact about a company — and a single page of results can cut a company's rows in half.

Quick answer

New York, Colorado, Connecticut and Oregon all publish their Secretary-of-State registries through Socrata's SODA API — same query language, wildly different field names per state. Oregon is the outlier structurally: it publishes "long format" rows, one per associated-name type (PRINCIPAL PLACE OF BUSINESS, MAILING ADDRESS, REGISTERED AGENT) per registry_number, so one real entity is several raw rows. Page that with plain $offset pagination and you will regularly split one entity's rows across two pages — miss the second page and you silently drop its address or agent. The fix is a tail-group re-fetch: treat the last group in a full page as possibly incomplete and re-request starting from that group's first row.

Why does Oregon need special pagination and NY/CO/CT don't?

Because NY, CO and CT are one-row-per-entity already:

async def _paginate(
    session, adapter, where, order, max_results,
) -> list[dict[str, Any]]:
    """REQ-4: $offset pagination for NY/CO/CT."""
    collected = []
    offset = 0
    while len(collected) < max_results:
        limit = min(client.SODA_PAGE_SIZE, max_results - len(collected))
        page = await client.soda_range_query(session, adapter.resource_url,
                                              where=where, order=order, limit=limit, offset=offset)
        if not page:
            break
        collected.extend(page)
        offset += len(page)
        if len(page) < limit:
            break
    return collected[:max_results]

Oregon needs _paginate_oregon() instead, because a $limit/$offset window has no concept of "don't split this entity's rows":

def _consume_oregon_page(page, limit, offset, complete):
    """Merge one OR page's usable groups into ``complete``; return the next $offset."""
    groups = parser.group_oregon_rows(page)
    keys = list(groups.keys())
    if len(page) == limit and len(keys) > 1:
        tail_key = keys[-1]
        tail_offset = offset + parser.first_row_index(page, tail_key, OR_REGISTRY_NUMBER_FIELD)
        usable_keys, next_offset = keys[:-1], tail_offset
    else:
        usable_keys, next_offset = keys, offset + len(page)
    for key in usable_keys:
        complete.setdefault(key, groups[key])
    return next_offset

When a page comes back full and holds more than one registry_number group, the last group is assumed incomplete — its rows are held back and the next fetch starts exactly at that group's first row, so its remaining associated-name rows arrive intact instead of being cut off mid-entity.

Why does a New York filing sometimes have a blank principal address?

Because plenty of small NY filers only ever fill in their process-agent contact, not a separate principal address column. Rather than ship a null and call it done, the adapter falls back:

def _principal_address(raw: dict[str, Any], adapter: JurisdictionAdapter) -> str | None:
    """Primary address_field_map, falling back to NY's dos_process_* columns when blank."""
    primary = _address_from_map(raw, adapter.address_field_map)
    if primary:
        return primary
    return _address_from_map(raw, adapter.address_fallback_field_map)

address_fallback_field_map only exists on the NY adapter — it points at dos_process_address_1/2/city/state/zip, the columns NY actually populates for small filers when location_* is empty.

Why does Colorado's registered-agent name need its own function?

Because Colorado's open-data export represents an agent as either an organization name or a full person-name broken into four columns — never both, and there's no single column to just read:

def _co_agent_name(raw: dict[str, Any]) -> str | None:
    """CO: agentorganizationname if present, else joined first/middle/last/suffix."""
    org = raw.get("agentorganizationname")
    if org:
        return org
    parts = (raw.get("agentfirstname"), raw.get("agentmiddlename"),
             raw.get("agentlastname"), raw.get("agentsuffix"))
    joined = " ".join(p for p in parts if p)
    return joined or None

Every other jurisdiction reads one named column for the agent; CO is the one deliberate exception to the fleet's otherwise fully data-driven per-state mapping.

Is scraping state Secretary-of-State open-data portals legal?

All four sources are official government open-data endpoints (data.ny.gov, data.colorado.gov, data.ct.gov, data.oregon.gov) published specifically for public reuse via the same keyless SODA REST API — this is open civic data by design, not a private database.

FAQ

What happens if my date range matches zero filings across all four states?
The run fails with a non-zero exit and a status message naming the exact dateFrom/dateTo and jurisdictions queried — so you can tell instantly whether the window was empty or the run actually broke, rather than guessing from a silent empty dataset.

Why is status null for some rows and populated for others?
Only Colorado and Connecticut publish an entity-status column in their open-data export; New York and Oregon don't expose one at all. The Actor reads status: null for NY/OR rather than guessing a value the source never gave it.

Do I need an API key for any of the four states?
No — all four are keyless, unauthenticated SODA endpoints. curl-cffi with Chrome TLS impersonation is still used per house default even though none of the four showed anti-bot behavior during live probing.

What does it cost?
$0.20 to start a run, then $0.005 per entity row written — about $5.00 per 1,000 rows, and a bad jurisdiction fetch is skipped with a warning rather than failing the whole run.

---

Try it: New Business Filings Leads Scraper — a daily feed of newly-formed US companies across NY, CO, CT and OR, one clean row per entity regardless of how each state actually structures its data.

We do the dirty work so your dataset stays clean. 😈