The most expensive bugs in data engineering are the ones that don't error. A column gets renamed, a type gets widened, a null starts showing up where the business said "never null" — and for weeks, nothing crashes. The dashboard just quietly drifts. Then one Tuesday, the report is missing a month, and suddenly a "small schema change" from April is your incident.
That's schema drift, and it's the difference between pipelines that feel stable and ones that feel haunted.
What a data contract actually is
A data contract is the API contract idea applied to data. When you change a REST endpoint you version it, document it, and give consumers a migration path. Data rarely gets that courtesy — producers change tables and consumers find out in prod.
A contract makes the producer's promise explicit. The useful ones carry four things:
| Field | What it answers |
|---|---|
| Schema | What fields exist, their types, nullability, allowed values |
| Quality rules | Completeness, uniqueness, range checks, accepted values |
| Freshness / SLA | How often data updates and by when |
| Ownership | Who produces, who consumes, who gets paged |
The point isn't the document. It's that a change to the data now has a reviewable surface instead of being an invisible commit.
Schema evolution vs breaking changes
The core skill is telling safe changes from breaking ones. A pragmatic, semver-flavored split:
| Category | Example | Verdict |
|---|---|---|
| Additive | New nullable column | Usually safe (minor) |
| Additive | New optional field in JSON | Usually safe (minor) |
| Widening | INT → BIGINT | Safe if consumers handle it |
| Breaking | Rename or drop a column | Breaking (major) |
| Breaking | Type change, tighter nullability | Breaking (major) |
| Breaking | Reorder columns for positional reads | Breaking (major) |
The subtle one: Spark, Delta Lake, and most modern engines read by name, not position. Renaming a column doesn't fail — the old name just resolves to nothing, so consumers get NULLs instead of errors. A broken contract that fails loudly in CI is a gift; the same break discovered by a dashboard is a fire.
Versioning contracts
Put a version on the contract and treat it like an API version. Bump the minor for additive changes, the major for breaking ones. Consumers pin to a major version, and a major bump becomes a coordinated, visible event — not a surprise in the morning batch.
Versioning buys you the most important thing: a conversation. "We're moving to v3 of orders" beats "why is my report empty?"
Enforce it in CI
A contract nobody checks is a wish. Enforcement belongs in the pipeline's CI, before anything lands in prod. The lightweight shape:
- Keep the contract next to the producer code
- On each PR, diff the proposed schema against the current contract
- Fail the build on breaking changes unless the major version was bumped
- Run a sample of real data through the quality rules
A minimal check in Python:
def assert_compatible(old: dict, new: dict) -> None:
"""Fail on breaking changes unless the major version bumps."""
if new["major"] == old["major"]:
old_names = {f["name"] for f in old["fields"]}
for field in new["fields"]:
if field["name"] not in old_names and not field.get("nullable"):
raise SystemExit(
f"Breaking change: non-nullable '{field['name']}' added "
"without a major version bump"
)
for field in old["fields"]:
if field["name"] not in {f["name"] for f in new["fields"]}:
raise SystemExit(
f"Breaking change: field '{field['name']}' removed "
"without a major version bump"
)
print("contract compatible")
Crude, but the shape is right: small, scriptable, and impossible to skip once it's a CI step.
Producers and consumers
Contracts also fix a social problem. Most teams have no roster of who consumes a table — "I wonder who uses this" is a real production skill. Put the consumer list in the contract, and a schema change becomes a notification instead of an archaeology project. Producers own the contract; consumers get pinged on change; the contract is where the two sides negotiate instead of discovering each other's needs in prod.
Tooling landscape (honestly)
You don't need a vendor to start. The honest landscape:
- Quality testing: dbt tests, Great Expectations, Soda — mature, good at row-level checks
- Schema/serialization: JSON Schema, Pydantic, Avro — good for validating shapes
- Platform-native: Delta Live Tables expectations, Unity Catalog — good if you're on Databricks
- Contract platforms: data-contract-spec and managed tools — powerful once you have many teams; overkill at the start
Start with a YAML contract and a CI script. Buy the platform when coordination hurts more than the price.
A lightweight contract you can adopt today
Something you can write before lunch:
dataset: prod.sales.orders
version: 1.2.0
owner: billing-team
schema:
fields:
- name: order_id
type: string
nullable: false
- name: total
type: decimal
nullable: false
- name: note
type: string
nullable: true
quality:
- total >= 0
- order_id is unique
freshness:
sla_hours: 24
consumers:
- analytics
- finance-recon
Check it into the producer repo, wire the diff check into CI, and you've got drift insurance for the cost of a YAML file.
Where to go deeper
If you're on Databricks or Spark, there's ready-made tooling for exactly this. The DataStack Pro store carries a Schema Evolution Toolkit that detects, validates, migrates, and analyzes schema changes across Delta Lake tables automatically, plus a Data Quality Framework with pluggable completeness, accuracy, consistency, and timeliness checks, a Data Pipeline Testing Kit, and PySpark utilities covering schema evolution and lineage. For the heavier end — a full contract implementation with YAML spec, CLI generator, schema validation, SLA monitoring, and breaking-change detection — the Data Contract Framework covers it.
The Data Engineering Bundle puts all 17 DataStack tools in one purchase for $199 instead of $523 separately.
Schema drift is a process problem, not a code problem. A contract, a version, and a CI check turn "who broke the data?" into "here's the change, here's who it affects." That's a much better way to spend a Tuesday.
Top comments (0)