DEV Community: Brad Ju

12.4 Million US Business Registrations Are Sitting on State Open-Data Portals, Free

Brad Ju — Wed, 08 Jul 2026 07:00:30 +0000

Five US states — New York, Colorado, Pennsylvania, Oregon, and Connecticut — publish their entire business registries as open data with commercial use explicitly allowed: ~12.4 million entities via documented Socrata APIs. Here's the verified dataset list, the fetcher we built, the measured rate limits nobody documents, and the gotchas that silently corrupt naive pulls.

If you've ever needed US company registration data — who registered what LLC, when, where — you've probably hit the same walls we did. OpenCorporates has great coverage, but the free tier is heavily capped and commercial use requires a license that starts in "contact sales" territory. Aggregator sites are mostly portals that link back to each state's search page — no bulk data. And Delaware, where a huge share of US companies actually incorporate, offers zero bulk access and zero API; selling that data is part of the state's business model.

So we assumed this data was simply locked up. It isn't. Several states publish their entire business registry — every LLC, corporation, and nonprofit ever registered — as open data on Socrata portals, explicitly in the public domain or licensed for commercial use. Here's what we verified, what we built to pull it, and the rate-limit realities we measured along the way.

The five states that just give you everything

Each of these is a Socrata SODA endpoint. All record counts are from live count(*) queries against the datasets; all license terms were verified on the dataset pages (June 2026).

State	Dataset	Records	License
New York	n9v6-gdp6 (active corps, beginning 1800)	4.22M	NY Open Data, commercial OK
Colorado	4ykn-tg5h	3.06M	Public Domain
Pennsylvania	xvd7-5r2c (officer-level rows)	2.31M entities	Public Domain
Oregon	tckn-sxa6 (row per associated name)	1.56M	Public record
Connecticut	n7gp-d28j (master table)	1.28M	Public Domain

Total: ~12.4 million business entities, with names, entity types, formation dates, addresses, and (in most states) registered agents. New York deserves a special mention: the active-corporations file goes back to 1800, and there's a companion dataset (63wc-4exh) with 20.6 million raw filing records if you want full filing history rather than current state.

One script, config-driven

Every state names its columns differently (entityname vs business_name vs current_entity_name), so the fetcher is a single Python-stdlib script driven by a SOURCES dict — one entry per state, mapping onto a normalized schema:

"co": {
    "name": "Colorado",
    "resource": "https://data.colorado.gov/resource/4ykn-tg5h.json",
    "license": "Public Domain",
    "fields": {
        "entity_id":      "entityid",
        "name":           "entityname",
        "entity_type":    "entitytype",
        "status":         "entitystatus",
        "formation_date": "entityformdate",
        "city":           "principalcity",
        "region":         "principalstate",
        "postal":         "principalzipcode",
    },
},

Adding a new Socrata state is one more dict entry. Output is one JSONL file per state with a unified schema (state / entity_id / name / entity_type / status / formation_date / city / region / postal / agent_name), plus the raw row preserved under _raw so nothing is lost in normalization. Pagination is $limit/$offset ordered by :id, with resume-by-line-count: if the pull dies at row 650,000, rerunning counts the lines already on disk and continues from that offset in append mode.

The rate-limit findings nobody documents

This is where we burned the most time, so here are actual measurements from anonymous (no app token) requests in June 2026:

Plain offset pagination: ~500 rows/sec — the best you'll do anonymously. Counter-intuitively, deep offsets are not the problem: a query at offset 1,000,000 returns in about 3 seconds, barely slower than offset 0. The bottleneck is per-page transfer throughput, not offset depth — so the classic "keyset beats offset" instinct is wrong here.
Keyset pagination via $select=:, is a dead end. Asking for :* forces the server to compute system metadata for every row; a single 50k-row page took 200+ seconds and then timed out.
The CSV bulk export endpoint (/api/views/{id}/rows.csv) is even slower: files are generated server-side on demand, and we measured 1,229 rows in 30 seconds — roughly 12× slower than JSON offset paging.

At ~500 rows/sec, the full ~13M rows across these states is a 7–8 hour pull. Register a free Socrata app token and send it as X-App-Token for a higher rate-limit tier — worth doing for the full pull; the anonymous numbers above are the floor.

Gotchas that will silently corrupt your data

Row granularity differs per state. Oregon's dataset is one row per associated name, and Pennsylvania's is one row per officer — so multiple rows share one entity ID. Count rows naively and you'll overcount entities by 2–3×. Dedup on registry_number / filing_number is mandatory.
CSV column labels ≠ API field names. Connecticut's CSV export has Title_Case labels like Business_City, while the SODA API uses billingcity. If you mix export formats, map columns through the dataset metadata (/api/views/{id}.json → columns[].fieldName), never by header string.
Connecticut splits agents into companion datasets: the master table has no agent columns; registered agents and principals live in separate Agent Details / Principal Details datasets joined on accountnumber.

The states you can't get (and why)

California, Texas, Delaware: bulk registry data is paid. Delaware in particular has no bulk product and no API at any price we could find — remarkable given it's the incorporation capital of the country.
Florida: free! But it's a fixed-width flat file on an FTP server (Sunbiz), so it needs its own parser rather than the Socrata adapter.
Ohio: publishes monthly bulk files, but the SoS site sits behind an aggressive bot wall.
Iowa: was on Socrata (CC BY 4.0), then migrated to a new "Iowa Data Hub" platform — all the documented legacy endpoints now 404.
Hawaii: the full statewide registry (~442k) is online, but the dataset carries no explicit license tag, so we left it disabled until commercial terms are confirmed.
Washington, Illinois: publish business license data, not the registration registry.

On legality and ethics

Everything enabled here is official government open data with explicit public-domain or commercial-OK terms — no scraping of search UIs, no ToS gray zones. The one dataset without a clear license tag stays off. If you build on this, the same discipline applies: a state publishing data on an open-data portal is an invitation; a state putting its registry behind a paywall or bot wall is an answer, and the answer is no.

What this is good for — and where to get the script

Business registries are the skeleton of the US SMB economy: formation trends by county, entity-type shifts (the LLC-ification of everything), registered-agent market share, business survival analysis by cohort. We pulled this for company matching in deciqAI's analysis agents — but honestly the dataset is more interesting than our use case.

The whole thing ships as an open-source Agent Skill

The verified dataset registry, the ~300-line dependency-free fetcher, the measured rate limits, and every gotcha above are packaged as us-business-registry-open-data in deciqAI Knowledge Skills — MIT licensed. Install: npx skills add deciqAI/knowledge-skills

FAQ

Is US business registration data free to use commercially?

For five states, yes: New York, Colorado, Pennsylvania, Oregon, and Connecticut publish their full registries as open data with explicit public-domain or commercial-OK licenses. Most other states either charge for bulk access or don't offer it at all.

Why isn't Delaware in the list?

Delaware offers no bulk registry product and no API at any price — selling that data is part of the state's business model. California and Texas also charge for bulk registry access.

How many records can you get in total?

About 12.4 million deduplicated business entities across the five states, plus New York's companion dataset of 20.6 million raw filing records going back to 1800.

How long does a full pull take?

At the measured anonymous rate of ~500 rows/sec, roughly 7–8 hours for all ~13M rows. A free Socrata app token (sent as X-App-Token) raises the rate-limit tier.

What's the biggest mistake when pulling this data?

Counting rows as entities. Oregon is one row per associated name and Pennsylvania is one row per officer, so naive counts overcount by 2–3×. Deduplicate on the state's entity identifier before any analysis.

Where do I get the fetcher script?

It ships inside the us-business-registry-open-data skill in deciqAI Knowledge Skills (MIT, on GitHub) — ~300 lines of Python stdlib, no dependencies, one config entry per state.

Originally published at deciqai.com.

How We Made 163 Mental Models Executable for AI Agents

Brad Ju — Wed, 08 Jul 2026 07:00:08 +0000

Mental models like first principles and inversion usually live in books as prose an agent can't run. We turned 163 of them into open-source Agent Skills — each with explicit trigger conditions, a step-by-step process with hard gates, and worked historical case studies cited to primary sources. MIT-licensed, 10K+ installs on ClawHub.

Every founder has read about first-principles thinking. Almost no one — human or AI — actually runs it. The method lives in books and blog posts as prose: inspiring to read, impossible to execute. Ask an LLM to "use first principles" and you get vibes — a paragraph that name-checks Musk and then does whatever it was going to do anyway.

We wanted our agents to genuinely run these methods before acting — tear assumptions to bedrock before pricing a product, invert before shipping a launch plan, update like a Bayesian when evidence lands. So we rebuilt 163 mental models as Agent Skills: self-contained SKILL.md files an agent loads and executes. Then we open-sourced all of them under MIT. Here's what "executable" actually took.

A mental model becomes executable when it can fail

The difference between a description and a process is that a process can halt. Each skill has three load-bearing parts most prompt libraries skip:

Explicit triggers — every skill's description states "Activate when: …" with concrete user phrasings, and most add "Do NOT activate when: …". This is what lets an agent load the right method at the right moment instead of applying its favorite hammer to everything.
A numbered process with hard gates — steps like "Gate: no quantified baseline → stop." If the input doesn't meet the bar, the skill refuses to proceed rather than producing confident filler.
A Common Rationalizations table — the fake moves people (and models) use to dodge the method, each paired with the reality check. "Everyone feels more aligned" is not a velocity change. "Value exists but is hard to quantify" is the standard defense of programs that don't work.

That last one matters more than it looks. LLMs are excellent at producing text that sounds like the method. The rationalization table is a list of exactly those moves, so the agent can catch itself mid-dodge.

Every skill ships a real case, cited to a primary source

Abstract methods don't transfer. So every skill includes worked historical examples — real cases walked through the skill's own steps, with the primary source cited at the bottom. Not invented "imagine a startup…" scenarios. These:

Skill	Case	The move
checklist	Van Halen's brown-M&M rider clause (1980s)	A trivial, instantly verifiable item embedded in a safety checklist proves whether the counterparty read the rest
black-swan	Fukushima vs. Onagawa tsunami design basis (2011)	The 869 AD paleotsunami record was in the literature; one plant's design basis treated the tail as real, one didn't
inversion	Abraham Wald's bomber armor analysis (1943)	Armor where returning planes show no damage — survivor data is silent exactly where hits are fatal
second-order-thinking	The Hanoi rat bounty (1902)	Pay per severed rat tail and you don't get fewer rats — you get tail-less rats released to breed, and rat farms
bayesian-reasoning	The search for Air France 447 (2009–2011)	Discounting negative search evidence by the probability both acoustic beacons failed kept the posterior near the crash point — found in a week
first-principles	SpaceX's rocket cost teardown (2002)	Decompose launch prices into commodity bill-of-materials; the citable industry number was inherited belief, not bedrock

40 of the 163 skills now ship two cases from different domains, and we're extending that to the whole library. The sourcing rule is hard: cite only documents that exist, paraphrase rather than fabricate quotes, and when a number is a reconstruction, label it as one.

The numbers

deciqAI Knowledge Skills, today

164 skills · MIT licensed · 10K+ installs on ClawHub · worked examples cited to primary sources · the same skills wired into every deciqAI agent. Install: npx skills add deciqAI/knowledge-skills

The library covers reasoning methods (first principles, inversion, Occam's razor, Bayesian updating), cognitive biases (anchoring, sunk cost, confirmation bias, hindsight), strategy frameworks (Porter's five forces, blue ocean, disruption), and operating tools (checklists, OKRs, OODA, pre-mortems). Each is a folder: SKILL.md plus examples/ plus references/ — copy one into your agent's skills directory, or install all 164 with one command.

Why we open-sourced the thinking layer

These are the same skills every deciqAI agent runs before it acts — when it drafts your pricing, plans an outreach campaign, or tears down a business plan. Open-sourcing them costs us nothing competitive: the moat isn't the method, it's the execution loop around it — the data, the agents, and the company operating system they run inside. Publishing the thinking layer just means anyone can check our agents' reasoning against the source, and any builder can give their own agent the same rigor.

Contributions are open — the repo ships a skill template and quality bars (real triggers, hard gates, primary-sourced cases, no placeholders). The easiest first contribution is an [O] entry: a real failure pattern you observed using a skill, pasted into its rationalization table.

FAQ

What is an Agent Skill?

A self-contained SKILL.md file (an open format introduced by Anthropic and now supported across Claude Code, Codex, Cursor, Gemini CLI and others) that an AI agent loads to execute a specific capability — here, a thinking method with triggers, steps, gates, and worked examples.

How is this different from a prompt library?

Prompts describe; skills execute and can halt. Each skill has explicit activation conditions, numbered steps with hard gates that stop the process when inputs don't meet the bar, and a table of the fake moves that imitate the method — so the agent can catch itself producing method-flavored filler.

Are the case studies real?

Yes — every worked example is a documented historical case walked through the skill's own steps, with the primary source cited (e.g., Wald's survivability memoranda, the NAIIC Fukushima report, Stone et al.'s AF 447 Bayesian search paper). No invented scenarios.

Can I use them outside deciqAI?

That's the point — they're MIT-licensed and work in any agent that supports the SKILL.md format. Run npx skills add deciqAI/knowledge-skills, install from ClawHub, or copy a single folder from GitHub.

Originally published at deciqai.com.