Data Contracts for Quant Pipelines: Schema Enforcement at Ingest

By Shakti Tiwari · Engineering note · Educational only · Not investment advice

A quant pipeline is a chain of trust. Raw bytes arrive from an exchange feed, a vendor API, an internal log, or a partner export. They travel through parsing, normalization, feature construction, labeling, and finally into a model or a backtest. The weakest link in that chain is almost never the clever part — the gradient, the regime detector, the volatility surface fit. The weakest link is the very first step: ingest. If the data that enters the pipeline is silently wrong, every downstream computation inherits that wrongness and compounds it. A model trained on corrupted inputs does not fail loudly; it fails plausibly. That is the most dangerous failure mode in quantitative work, and it is the one a data contract is designed to prevent.

This article is about data contracts for quant pipelines, and specifically about enforcing them at the point of ingest rather than deep inside the pipeline. We will build the concept from first principles, look at the failure modes that contracts exist to stop, write real code for a schema-enforcing ingest gate, and derive the simple statistics that make enforcement decisions objective instead of vibes. There are no live market numbers here; everything is structural and reproducible against your own data.

What a data contract actually is

A data contract is a formal, machine-checkable agreement between a producer of data and a consumer of data. It states, in a form a program can verify, what the data promises to look like: which columns exist, what types they carry, which are nullable, which ranges are legal, what units apply, how fresh the data must be, and what volume to expect. In a quant context the producer might be a market-data collector or a vendor, and the consumer is the feature store, the backtest engine, or the live inference path.

The contract is not documentation. Documentation is a paragraph someone wrote and nobody re-reads. A contract is code that runs on every record, every batch, every file, before that data is allowed to touch anything downstream. If the data violates the contract, it does not get a free pass because the dashboard looks okay. It is rejected, quarantined, or flagged, by rule.

There are four layers that a mature contract stack usually separates:

1. Structural / schema contract — column names, types, nullability, primary keys. This is the layer most people mean when they say "schema enforcement," and it is the focus of this article.
2. Semantic contract — the meaning of a field. A column called price must be a positive number in the instrument's trading currency; a column called ts must be epoch milliseconds, not seconds, not a date string.
3. SLA contract — freshness and volume. The batch must arrive within a window; the row count must fall inside an expected band.
4. Distribution contract — the statistical shape. The null rate of price must stay below a threshold; the distribution of return must not suddenly shift. This is where drift detection lives.

Schema enforcement at ingest means you check layers one and two (and ideally three) the instant data crosses the boundary into your system, before it is persisted, joined, or modeled.

Why ingest is the right place, not the middle

It is tempting to validate data "somewhere downstream," after it has been transformed a few times, when it is convenient to write a test. That temptation is a trap. Three reasons make ingest the only correct enforcement point.

First, cost of correction grows downstream. A single mistyped column that slips through ingest can propagate into a feature table, then into a label, then into a trained model artifact, then into a published backtest. Finding the root cause after the fact means recomputing every stage. Catching it at ingest means dropping or fixing one batch.

Second, trust is binary at the boundary. Once data is in your warehouse, other engineers and other jobs will assume it is "your clean data." If you let a malformed batch in, you have implicitly signed off on it. The boundary is where the signature belongs.

Third, reproducibility depends on it. A backtest is only reproducible if the input it consumed is exactly the input you think it consumed. A schema gate that records what was accepted (and what was rejected) gives you an auditable ledger of inputs. Without that ledger, "I reran the backtest" is not reproducible; it is a hope.

The failure modes contracts exist to stop

Un-contracted ingest fails in predictable ways. Naming them makes the contract design obvious.

Type drift. A vendor silently changes a field from integer to string, or from float to string-encoded number. Your parser does not crash; it coerces, or it stores the string, and three weeks later a feature computed as float(price) throws at two in the morning during a live run.

Unit drift. A feed switches a price from paise to rupees, or a timestamp from seconds to milliseconds. Nothing errors. Your features are just wrong by a factor of one hundred or one thousand. This is the silent killer.

Missing columns. A schema change removes a column your labels depend on. If you tolerate missing columns, your label computation produces NaN rows that you may or may not notice before they poison training.

Null injection. A new release starts emitting nulls for a field that was previously always populated. Your model treats null as a real value via imputation and drifts quietly.

Volume anomalies. A partial file arrives — only a slice of the symbols, or only the first hour of the session. If you ingest it without a volume check, your features describe a half-day and your backtest describes a market that never existed.

Duplicate keys. Reconnects and retries resend rows. Without a primary-key contract, you double-count, and double-counting inflates sample sizes and corrupts statistics. (This dovetails with the idempotent ingest work covered in the linked pipeline article.)

Every one of these is cheap to detect at ingest and expensive to discover later.

A minimal contract, expressed as code

The cleanest way to make a contract real is to express it as a typed schema object and validate incoming records against it before persistence. Below is a self-contained validator written in plain Python with no heavy dependencies, so you can drop it into any pipeline. It shows the shape of the idea; in production you would back it with a library such as a Pandas schema validator or a JSON Schema validator, but the logic is identical.

from __future__ import annotations
import re
from dataclasses import dataclass, field
from typing import Any, Callable, Optional


class ContractViolation(Exception):
    """Raised when a record or batch breaks the data contract."""


@dataclass
class FieldSpec:
    name: str
    dtype: type                 # int, float, str, bool
    nullable: bool = False
    unit: Optional[str] = None  # semantic guard, e.g. "ms", "INR"
    min_value: Optional[float] = None
    max_value: Optional[float] = None
    regex: Optional[str] = None  # format guard for strings


@dataclass
class SchemaContract:
    name: str
    fields: list[FieldSpec] = field(default_factory=list)
    primary_key: list[str] = field(default_factory=list)
    min_rows: int = 1
    max_rows: int = 10_000_000   # guard against absurd batches

    def validate_record(self, rec: dict) -> None:
        for spec in self.fields:
            value = rec.get(spec.name, None)
            if value is None:
                if not spec.nullable:
                    raise ContractViolation(f"non-nullable field '{spec.name}' is missing/Null")
                continue
            if not isinstance(value, spec.dtype):
                # strict type check; coercion is a separate explicit step
                raise ContractViolation(
                    f"field '{spec.name}' expected {spec.dtype.__name__}, got {type(value).__name__}"
                )
            if spec.min_value is not None and value < spec.min_value:
                raise ContractViolation(f"field '{spec.name}' below min {spec.min_value}")
            if spec.max_value is not None and value > spec.max_value:
                raise ContractViolation(f"field '{spec.name}' above max {spec.max_value}")
            if spec.regex is not None and not re.match(spec.regex, str(value)):
                raise ContractViolation(f"field '{spec.name}' fails format {spec.regex}")

    def validate_batch(self, rows: list[dict]) -> None:
        n = len(rows)
        if n < self.min_rows or n > self.max_rows:
            raise ContractViolation(f"row count {n} outside [{self.min_rows}, {self.max_rows}]")
        seen_keys: set[tuple] = set()
        for i, rec in enumerate(rows):
            self.validate_record(rec)
            if self.primary_key:
                key = tuple(rec.get(k) for k in self.primary_key)
                if key in seen_keys:
                    raise ContractViolation(f"duplicate primary key {key} at row {i}")
                seen_keys.add(key)

This contract does four concrete things per record: it enforces non-nullability, it enforces the exact Python type (no silent coercion), it enforces value ranges, and it enforces a string format. Per batch it enforces a row-count band and primary-key uniqueness. That single object, run before persistence, stops type drift, unit drift (via the unit annotation used in a fuller implementation), null injection, volume anomalies, and duplicate keys.

Coercion versus rejection: a deliberate choice

A tempting "helpful" behavior is to coerce. See a string "123.45" where a float is expected? Coerce it. See an integer 100 where a float is expected? Coerce it. The problem is that coercion hides the contract breach. If the producer is sending strings where they promised floats, that is a bug on their side, and you want it to be loud, not laundered into a "successful" ingest.

The disciplined pattern is: reject or quarantine on type mismatch, coerce only along explicit, declared widening paths. Widening from int to float is safe and reversible in the sense that no information is lost; coercing str to float is not safe and must be treated as a violation. The validator above takes the strict stance. In a real system you would add an explicit coerce set to the contract, e.g. {("int", "float")}, and reject everything else. The point is that the choice is made once, in the contract, not ad hoc in a dozen parsing functions.

Range and null-rate thresholds as formulas

Two numeric guards turn "looks fine" into "objectively fine": a per-field range check and a per-field null-rate check. Both have simple formulas you can compute in a single pass.

For a field x over a batch of N records, the observed null rate is:

null_rate(x) = (count of records where x is None) / N

The contract accepts the batch only if null_rate(x) <= T_x, where T_x is the maximum tolerable null fraction for that field. For a field that the label depends on, T_x is typically zero; for an auxiliary field it might be a small value such as a few percent. Notice that T_x is a contract parameter, not a guess — it is decided when the contract is written, and changing it is a reviewed change, not a silent drift.

For a numeric field, the observed range is the pair (min_x, max_x) across the batch:

accept iff  L_x <= min_x  and  max_x <= U_x

where (L_x, U_x) is the contract's declared legal interval. This catches unit drift immediately: if price is supposed to be in rupees and a batch arrives in paise, max_x jumps by a factor of one hundred and the batch is rejected before it can corrupt a feature.

A useful companion metric is the freshness of a batch, measured as the gap between ingest time and the newest record timestamp:

freshness = ingest_ts - max(record_ts)

The SLA contract declares freshness <= F. A batch that is too old is quarantined: it may be perfectly typed and yet stale, and stale data in a live model is its own kind of corruption.

Fail-fast versus quarantine

Once a breach is detected, you have two sane responses and one insane one.

The insane response is ingest-and-warn: write the bad data anyway and emit a log line nobody reads. This is how silent corruption enters production.

The two sane responses are fail-fast and quarantine:

• Fail-fast rejects the entire batch and halts the pipeline stage. Use this when a breach means the data is unusable and there is a human or an upstream owner who will fix and resend. Backtests and offline feature builds usually want fail-fast: a partial or malformed input should not produce a "result."
• Quarantine writes the offending records (or batch) to a side location and lets the clean portion proceed. Use this in live ingest where dropping a whole session because of one bad symbol is worse than routing the bad symbol aside. Quarantine must still be loud: it feeds the same alerting as fail-fast.

A practical implementation tags each rejected batch with the violation type and the count, so monitoring can tell you whether breaches are sporadic (a flaky vendor) or systematic (a real schema change you must adapt the contract to).

Schema versioning and compatibility

Producers change. That is not a bug; it is life. The contract must version so that a legitimate, reviewed change does not look identical to a silent breach. The standard discipline borrows from API versioning:

• Backward compatibility: a new contract version can add optional fields and relax constraints, but must not remove or narrow fields that old consumers depend on. Old consumers keep working.
• Forward compatibility: a consumer written against a newer contract can tolerate older data by ignoring unknown optional fields.

At ingest you store the contract version alongside the data. If incoming data declares contract_version = 3 but your pipeline only knows 1 and 2, that is a hard stop: someone shipped a change nobody told the consumer about. If data declares 2 and you know 3, you accept with a deprecation note. The key is that version is a first-class field, checked exactly like any other.

A clean way to encode compatibility is a small compatibility matrix evaluated at startup:

def compatible(consumer_version: int, producer_version: int) -> bool:
    # backward compatible: consumer accepts <= its own version
    # forward tolerant: consumer accepts one minor ahead via optional-field handling
    return producer_version <= consumer_version

In a real deployment you would separate major and minor versions and apply the stricter major-compatibility rule, but the principle holds: version is checked, not assumed.

Distribution contracts: when structure is not enough

Schema checks confirm the data is shaped correctly. They do not confirm it is behaving correctly. A column can be perfectly typed, non-null, in-range, and still be statistically wrong — for example, a return field that used to be centered near zero now sits at a large constant because of a bug upstream. This is where a distribution contract earns its place.

The workhorse metric is the Population Stability Index (PSI), which quantifies how much a feature's distribution has shifted between a reference period and the current batch:

PSI = sum over bins i of  (actual_i - expected_i) * ln(actual_i / expected_i)

where actual_i and expected_i are the fraction of records falling in bin i for the current and reference distributions respectively. A PSI below a small threshold (commonly a low single-digit percentage in practice, but the exact threshold is a contract parameter you set) means the distribution is stable; a larger PSI means it has moved enough to warrant investigation before the data feeds a model.

For a two-sample test on a continuous field you can also use the Kolmogorov–Smirnov statistic:

D = sup_x | F_actual(x) - F_expected(x) |

the maximum vertical distance between the two empirical cumulative distribution functions. A large D rejects the hypothesis that the two samples come from the same distribution. Either metric turns "the numbers feel off" into a numeric, contract-enforced gate.

These distribution checks sit one layer above schema enforcement. You should still run schema enforcement first — there is no point computing PSI on a column that failed the type check.

A schema registry pattern

As you accumulate contracts, you do not want them scattered across services as hardcoded dictionaries. The registry pattern centralizes them:

• Contracts live in version-controlled files (YAML or JSON) in a dedicated repository.
• An ingest service fetches the contract for a given dataset_id and version from the registry at startup.
• Validation uses the fetched contract, so changing a threshold is a reviewed pull request, not a silent edit on a production box.
• The registry also stores the reference distributions used by distribution contracts, so PSI baselines are themselves versioned.

Centralization has a second benefit: you can run the same contract in three places — the producer's pre-publish test, the ingest gate, and a nightly audit — and they cannot drift apart because they read the same file.

Wiring it into CI and monitoring

A contract that only runs in production is half a contract. The producer should validate against the contract in continuous integration before shipping a data change, using a sample of real records. If the sample would breach the contract, the change is blocked. This moves the failure from "three a.m. page" to "merge-time error," which is where you want it.

On the consumer side, every ingest run should emit metrics:

• batches accepted and rejected, by violation type;
• null rates per critical field, compared to T_x;
• row counts compared to the expected band;
• freshness compared to F.

These feed dashboards and alerts. A sudden spike in type_mismatch violations is the earliest signal of a vendor change; a creeping null_rate on a label field is the earliest signal of a slow upstream regression. Both are caught days before they would surface as a mysteriously degraded model.

How this connects to the rest of the pipeline

Schema enforcement at ingest is not a standalone ritual; it is the foundation the rest of the quant stack stands on. The idempotent collector article explains how to stop duplicate ticks at the socket layer — the primary-key clause in the contract above is the persistence-layer echo of that same idea, defense in depth. The broader data-pipeline article shows how clean ticks become a feature store; a feature store is only trustworthy if the ticks entering it passed a contract, because the feature code assumes the columns and types it was written against.

For backtesting the link is even sharper. A backtest is a claim about how a strategy would have performed. That claim is only as good as the inputs. If a malformed batch slipped past ingest and biased a feature, the backtest reports a result that never could have happened — and you would not know, because the numbers would look reasonable. The contract is what makes "I trust this backtest" a defensible statement rather than a hope. This is the same epistemic discipline that governs every article on this site: verify the input before you trust the output.

Common mistakes to avoid

The errors teams make around data contracts are consistent, which means they are avoidable if named.

The first mistake is treating the contract as documentation. A paragraph in a wiki is not a contract. If a program does not evaluate it on every record, it is decoration.

The second is coercing instead of rejecting. Coercion converts a loud, fixable breach into a silent, compounding one. Widen explicitly or reject; never launder.

The third is versioning by hope — changing a schema in place and assuming downstream consumers will cope. Version the contract, check the version at ingest, and make incompatibility a hard stop.

The fourth is forgetting the distribution layer. Type-correct data can still be statistically broken. Schema is necessary, not sufficient; add PSI or KS gates for critical fields.

The fifth is no metric emit. A contract that fails silently in a log file nobody reads has failed in spirit. Rejection counts and null rates must be visible.

Avoid these five and you have already out-engineered most pipelines you will encounter, not because the math is exotic but because the discipline is consistent.

Conclusion

Data contracts for quant pipelines are the cheapest insurance in the entire stack. The idea is simple: state, in code, what your data promises to be, and check that promise at the boundary where data enters your world — before it is persisted, joined, modeled, or backtested. Schema enforcement handles shape and type; semantic and SLA clauses handle units, freshness, and volume; distribution clauses handle statistical behavior through PSI and KS. Together they convert "the data looks okay" into "the data passed an auditable, versioned, numeric gate."

The payoff is not a single prevented outage. The payoff is that every downstream result — every feature, every label, every backtest, every live inference — rests on inputs you can name and defend. In quantitative work, where a silent wrong number can masquerade as an edge, that defensibility is the whole game. Write the contract once, enforce it everywhere, version it deliberately, and let the metrics tell you when the world changed. Everything else in the pipeline is easier when the floor is solid.

About the Author

Shakti Tiwari writes about AI, local AI agents, XGBoost, and systematic quantitative trading for Indian markets — code-first, no-hype, governed by an epistemic firewall against fabricated numbers. Educational, reproducible, and source-attributed. Follow on X, LinkedIn, GitHub, and DEV via the links below.

• 🐦 X: https://x.com/shaktitiwari
• 💼 LinkedIn: https://linkedin.com/in/shakti-tiwari-a3b22a38b
• 💻 GitHub: https://github.com/shaktitiwari715-ai
• 📝 DEV.to: https://dev.to/shaktitiwari
• 🌐 Site: https://optiontradingwithai.in

Educational only. Not investment advice.

Continue Reading (Authority OS series)

• Nifty Options Data Pipeline: From WebSocket Tick to Clean Feature Store
• Idempotent WebSocket Market Data Collector
• Leakage-Free XGBoost Pipeline
• AI Trading Myth-Buster
• Interactive Tools Hub

Tags

ShaktiTiwariOnAI #QuantFinance #DataContracts #DataEngineering #Python #MachineLearning #TradingAIBharat #Backtesting