In the last year I built, alone, a quantitative research data platform: about 1.8 billion rows across 764 tables of alternative data, SEC filings, government records and market data, engineered to be trusted in a backtest. I am self-taught. The discipline came from the work itself: 206 architecture decision records and 680 research notes, one for every problem worth not solving twice.
The hard problems were not the ones anyone warns you about. They were data problems.
The stack, in the terms the field uses
- Point-in-time (as-of) data. Every record carries when it became public, not just when it happened, and 103 tables are bitemporal (valid time and recorded time). That is the defence against look-ahead bias, the most common way a backtest flatters itself.
- Survivorship-bias-free history. Delisted and renamed companies stay in the data with their history, and a window in which a stock dies is scored with its delisting return instead of being dropped.
- Entity resolution and a security master. Record linkage across tickers, CIKs, LEIs and CUSIPs, with point-in-time symbology: tickers get renamed, reused and delisted, so "which company was this on that date" is answered as of the date.
- Corporate actions. Split and dividend adjustment validated against an independent source rather than against itself; the split-adjustment tool is open source.
- Information extraction (NLP) from regulatory filings. 8-K events, earnings guidance and reported results, press releases, IPO prospectuses, SEC comment letters, material contracts, Schedule 13D/13G beneficial ownership, insider Forms 3/4/5, foreign-issuer reports and XBRL financial statements, turned into structured, dated facts.
- A medallion lakehouse. Bronze, silver and gold layers on Parquet, PostgreSQL for the serving tables, about 456 data contracts guarding them, and data lineage that answers "what breaks downstream if this table changes" in one command.
- MLOps and model validation. Feature pipelines, walk-forward and purged cross-validation, backtesting and paper trading, because a model that cannot fail a test is a horoscope.
- Domain-specific language models. Small models fine-tuned for narrow financial extraction tasks on a desktop GPU.
- Agentic workflows. Model Context Protocol (MCP) servers that let AI agents query the data, and agent-to-agent orchestration around them.
- All of it on one workstation, with a resource broker that schedules heavy jobs so the machine does not eat itself.
By the numbers: 4,776 commits since 21 November 2025, 5,281 Python files, 738 of them tests.
Three lessons in data quality
1. Labels lie; specifications do not. The SEC's fund-holdings data (Form N-PORT) has a field called "Ticker". Measured across every quarter the SEC has published, about 28% of its 17.4 million values are not stock tickers: mortgage-pool codes, bond issuers' symbols, futures contracts. One pool code, FG, is also an NYSE ticker, so a naive join turns Freddie Mac mortgage pools into shares of an insurance company. The fix was metadata management, not cleverer code: the SEC's own form instructions say the field is a fallback identifier. I now build a data dictionary (a codebook, in the ISO/IEC 11179 sense) from each agency's official documentation before touching its data. The first seven dictionaries turned up twelve defects in the SEC's own published files.
2. A test that compares data with itself proves nothing. One evening every data-quality check on prices passed. The next morning an independent reconciliation found 236 days of prices rewritten onto the wrong side of a stock split. Every check had been comparing the pipeline with its own output. Now a data-quality claim needs an external witness.
3. Coverage is not precision. The easy metric in entity resolution is match rate. The useful one is precision. A shortcut that raised one match rate by ten points was right only 38% of the time against an independent identifier. It never shipped.
What is public
Two open-source releases so far:
- split-adjustment-tool - verify, repair and apply stock-split adjustments against your own as-traded price tape. Vendor-neutral Python (Polars), Apache 2.0, release 0.4.0: https://github.com/AlexBocio/split-adjustment-tool
- us-gov-data-dictionary - machine-readable data dictionaries for seven SEC data sets: 1,469 fields, each with the SEC's own definition and citation, official code lists, and a profile of what the published files actually contain. W3C CSVW and SKOS formats, CC BY 4.0: https://github.com/AlexBocio/us-gov-data-dictionary
More agencies are next. If you work with public financial data and it has ever quietly lied to you, I would like to hear the story.
Top comments (4)
Lesson 2 is the one I'd underline: a check that compares the pipeline with its own output can't fail. One related thing from our side on "when it became public": EDGAR's submissions-file acceptanceDateTime, which is labelled UTC, is shifted by an extra Eastern offset in some companies' files and correct in others. We measured 450 filings from 25 companies' files on 2026-10-11. Each file was either right for every filing in it (MSFT, GOOGL, TSLA) or shifted for every filing in it (AAPL, NVDA, JPM). JPMorgan's 06:30 ET filing reads 14:30Z instead of 10:30Z. The same filing also reads differently in Apple's file than in its director's file. So a bitemporal "recorded time" taken from that field can sit 4-5 hours late, which matters for anything intraday around an 8-K.
Do your 103 bitemporal tables take their public-time from acceptanceDateTime, or from the filing index page? And is the N-PORT ticker audit repeatable per quarter, so a new quarter's non-ticker share gets flagged automatically?
Disclosure: I built canli-mcp, a free MIT SEC/market-data MCP server with no key that runs locally:
npx -y canli-mcp, repos at github.com/arhancanli/canlicapital and github.com/arhancanli/alphac.Thanks, and that's a sharp measurement. We hit the same thing: acceptanceDateTime in the submissions file is often an Eastern wall-clock time wearing a Z, so we don't use it. Our public time comes from the ACCEPTANCE-DATETIME in each filing's own EDGAR index header, checked to the second against a sample (200 of 200 matched). Your per-file pattern, every filing right or every filing shifted, is a detail I hadn't seen stated that cleanly.
On N-PORT: the audit is a script that re-reads every published quarter, so a new quarter is one re-run. It doesn't flag automatically yet. You've just talked me into scheduling it.
What do you use canli-mcp for most: agents, or your own research?
The Form N-PORT
Tickerfallback trap is brutal because the column name invites a naive equi-join before anyone reads the schema spec. I hit a similar wall wiring MCP tools over Postgres tables for agent workflows, where an LLM sees a column namedtickerordateand immediately writes aJOINwithout checking whether the field holds CUSIP fallbacks or unadjusted effective dates. Exposing the ISO/IEC 11179 codebook definitions and valid value domains directly inside the MCP tool schema descriptions is the only way I have found to stop agents from writing plausible, look-ahead-tainted queries that pass syntax checks and silently corrupt the result set.Agreed: putting the definitions where the model reads them is the point. The dictionaries are machine-readable for exactly that reason (CSVW and SKOS, plus a small Python loader), and an MCP lookup over them is next. Have you found agents actually honour constraints written into tool descriptions, or do you still need a hard check after the query?