The Largest Schema Extension Experiment in History Has Been Running Since 2009

Somewhere in your codebase there is a column named custom_field_3, a JSONB blob called metadata, or a plugin API that lets integrators define their own event types. Every extensible system eventually runs into the same question, and most of us only ever guess at the answer: what happens to the consumers downstream once everybody starts extending? We don't actually have to guess. The answer has been measured continuously, at national scale, in public, for over fifteen years — and it matters more now than it did, because the evidence gathered on why losing a single analyst quietly reprices an entire company shows the human interpretation layer thinning out fastest at exactly the organizations least able to absorb the loss. What's left reading your output is a query. This is about whether your data shows up in it.

The Setup, in Terms a Backend Engineer Will Recognize

Since 2009, every public company in the United States has had to submit its financial statements in XBRL, an XML-based markup format. Strip away the accounting and the architecture is familiar: roughly eight thousand independent organizations write records against one shared, versioned schema — the US GAAP taxonomy, republished annually by the FASB — and the whole corpus is exposed through a free public REST API with no authentication and no keys.

The endpoints are worth knowing even if you never touch finance. The SEC's own API documentation covers three: companyconcept for one field from one filer, companyfacts for everything one filer has ever reported, and frames, which is the interesting one — it slices a single concept across every reporting entity for a given calendar period, snapping each filer's ragged fiscal calendar onto a common grid. Fair-access rules cap you at ten requests per second and require a User-Agent header that identifies you by name and email. Bulk ZIPs rebuild nightly around 3am ET.

And buried in that documentation is the sentence this whole article hangs on. The aggregation APIs only include facts that use a non-custom taxonomy — us-gaap, ifrs-full, dei, srt — and that apply to the entire filing entity. Everything else is silently absent.

Companies are permitted to extend the schema. The query layer just doesn't carry the extensions.

The Rule Nobody Follows

The regulation governing extensions is unusually blunt for a legal text. Under Regulation S-T, a filer must create a custom element if and only if no appropriate standard tag exists. Not "if it fits better." Not "if the label reads nicer." The rule explicitly rejects an inappropriate standard label as sufficient justification on its own.

So how often does a system with a legally-mandated "extend only when strictly necessary" clause actually get extended? The SEC's Division of Economic and Risk Analysis has published the number every year for a decade. Across annual reports from 2016 through 2024, average custom tag rates hover in a narrow band of roughly 16 to 22 percent depending on filer category and year, with a visible bump around 2020 and a modest decline through 2024.

Read that again in engineering terms. Roughly one in five line items in the corpus is a bespoke field, in a system where bespoke fields are supposed to be the exception. If a product manager told you 20% of your API's payload keys were vendor-specific extensions, you would not describe that schema as standardized. You would describe it as forked eight thousand ways.

Extension Rate Is a Property of Your Tooling

Here's where the data stops being merely interesting and starts being a lesson.

The averages hide the tail. When DERA staff went looking in 2014 for filers whose custom tag rate exceeded 50% — meaning more than half of their line items were invented fields — smaller filers made up 96% of that group. Staff review of the large filers with high extension rates generally found the extensions justified. Among the small ones, they found systematic evidence of custom tags created where a perfectly good standard tag already existed. And the staff pointed at a cause that has nothing to do with accounting: many of those filers had routed their submissions through the same handful of third-party filing vendors.

The trend lines say the same thing more quietly. Through 2019 and 2020, large accelerated filers' extension rates moved around while smaller reporting companies climbed steadily year over year. The gap wasn't driven by small companies having genuinely stranger businesses than large ones. Large companies had teams and software that pushed back. Small companies had a vendor whose importer made "create new element" the path of least resistance.

Any engineer who has inherited a database where half the columns are attr_1 through attr_12 already knows this story. Extension rate is not a measure of how unique your domain is. It is a measure of how cheap your tooling made the escape hatch relative to the effort of finding the right existing field. Make the escape hatch one click and it becomes the default. Ship a search that surfaces the correct standard field in under three seconds and it stops being the default.

What It Costs to Be in Your Own Namespace

The consequence is concrete and easy to demonstrate:

UA='Jane Dev [e-mail]'

# Every filer that reported the standard Revenues concept for CY2024Q1
curl -s -H "User-Agent: $UA" \
  'https://data.sec.gov/api/xbrl/frames/us-gaap/Revenues/USD/CY2024Q1.json' \
  | jq '.data | length'

# Now inspect which namespaces the API will even acknowledge for a filer
curl -s -H "User-Agent: $UA" \
  'https://data.sec.gov/api/xbrl/companyfacts/CIK0000320193.json' \
  | jq '.facts | keys'
# -> ["dei","us-gaap"]   ...and that is the entire universe.

That second call is the punchline. No matter how many bespoke elements a company defined, the response keys are a short list of standard taxonomies. The company's own namespace is not there and never will be. The number was disclosed. It was filed correctly. It passed validation. And it is invisible to every consumer working through the aggregation layer, which increasingly means every consumer at all.

For a large company this is a rounding error, because a dozen analysts read the raw exhibit anyway and translate it by hand. For a company with thin coverage and no analysts left, the aggregation layer is the only reader. The organizations extending hardest are precisely the ones with the least human interpretation left to compensate.

Europe Shipped a Fix Worth Stealing

The EU hit the same wall with its own mandate and responded with a design pattern rather than a prohibition. Under the European Single Electronic Format, an issuer may still create extension elements, but each one must be anchored to the closest standard taxonomy concept — a relationship recorded in the definition linkbase, so a consumer encountering an unknown element can walk the edge to a canonical one it does understand. When ESMA published its 2024 update to the ESEF Reporting Manual it tightened this further, clarifying that anchors must point to a core element sharing the same data type.

That is a genuinely good piece of schema engineering, and it generalizes far beyond filings:

• Don't ban extension, require a mapping — a custom field with a declared edge to a canonical field is queryable; one without is a dead end
• Enforce type compatibility on the mapping — an anchor from a monetary value to a text block is worse than no anchor, because it looks valid
• Store the mapping in the schema, not the documentation — a relationship a machine can traverse survives; a wiki note explaining what acme_adj_ebitda_v2 means does not
• Instrument your extension rate and watch it by customer segment — the segment with the highest rate is telling you where your standard fields or your UI are failing
• Promote recurring extensions into the standard on a schedule — the FASB adds elements annually precisely because a static core schema guarantees a growing fork

The Reader You're Actually Writing For

The financial story and the engineering story converge on the same point. When a market loses the analysts who used to read a company's filings and translate the idiosyncratic bits into something comparable, the surviving reader is a program that queries standard fields and moves on. Anything sitting in a private namespace isn't judged harshly — it simply isn't seen. That is a worse outcome than being criticized, because it produces no signal at all.

Your API has the same structure. Somewhere downstream, a consumer is running the equivalent of a frames query against your standard fields, and whatever your users buried in metadata is not in the result set. The question worth asking on your own schema this week isn't whether extension is allowed. It's what percentage of the payload is currently extension, whether that number is being driven by real domain complexity or by an autocomplete that's easier to ignore than to use, and whether anything custom carries an edge back to something canonical. Those three numbers are cheap to compute and, on the evidence of eight thousand companies filing for fifteen years, nobody computes them until the humans who used to paper over the mess are already gone.