Schema registries are a waste of time if they don't break your build.
If you are still relying on a Confluent Schema Registry to "warn" your downstream consumers about a breaking change, you have already lost the war. You are treating data integrity as a runtime request for permission rather than a pre-compilation requirement. In six years of building data platforms for healthcare and fintech, I’ve seen enough production outages caused by "harmless" field deprecations to know that human communication is not a data strategy.
If it isn't in the CI pipeline, it doesn't exist.
Why the common approach falls short
The industry standard is currently "passive enforcement." You publish an Avro schema to a registry, and maybe, if you’ve configured your Kafka producers correctly, you get a compatibility check. But what about the downstream SQL transformation layer? What about the dbt models that expect a user_id to be a STRING but suddenly receive a BIGINT because someone in billing decided to clean up their database?
Passive registries are decoupled from the lifecycle of the actual code. A producer can change their schema, wait for the registry to return a 200 OK, and deploy their service. The consumer—often a different team entirely—finds out when their Pydantic models explode at 3:00 AM because the total_amount field is now a nested object instead of a float.
We treat data like a stream of bytes that might change, rather than an API that must be versioned. If your data pipeline doesn't treat a schema drift as a failed unit test, you aren't doing data engineering; you’re doing data archaeology.
Moving contracts into the repository
My team moved away from centralized registry-only workflows toward a "Contract-as-Code" model. We define our schemas in YAML or Protobuf within the producer's repository, but we also maintain a contracts/ directory in a shared monorepo that our CI pipelines reference.
When a producer team wants to make a change, they don't just update their code. They open a PR that includes a change to the contract file. Our CI pipeline runs a tool like data-contract-cli (or a custom wrapper around jsonschema and dbt-checkpoint) to validate that the new schema doesn't break the existing downstream expectations.
# example-contract.yaml
dataContract: 1.0.0
servers:
kafka:
type: kafka
bootstrapServers: broker.prod.svc
models:
transactions:
fields:
transaction_id: { type: string, required: true }
amount: { type: decimal, required: true }
currency: { type: string, required: true }
When a developer tries to remove the currency field, the CI job executes a diff against the current production-deployed contract. If the diff indicates a breaking change—like removing a required field—the build fails. The developer is stopped before they even hit the deploy button. They are forced to either negotiate the change with the consumers or version the topic to transactions_v2.
Building the gatekeeper in CI
The magic happens in the .github/workflows/contract-check.yaml. You need to treat this job with the same severity as a terraform plan or a pytest suite. If the contract check fails, the PR cannot be merged.
jobs:
validate-contract:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Install contract tool
run: npm install -g @data-contract/cli
- name: Validate schema compatibility
run: |
datacontract test ./contracts/transactions.yaml \
--source ./schema/transactions.json \
--compatibility backward
By setting --compatibility backward, we ensure that any new deployment is strictly additive. If a developer accidentally changes a type, the CI pipeline throws an exit code 1. This isn't just a linter; this is an automated negotiation.
In fintech, we dealt with PCI-DSS compliance. If a producer changed a field name that inadvertently exposed PII, our contract checks—which included regex patterns for field names—would catch the leak. By enforcing these rules at the PR level, we moved security and data quality from a reactive "ops" problem to a proactive "dev" problem.
Photo by Brecht Corbeel on Unsplash
The objections (and my answers)
"This will slow down our velocity," is the most common pushback I hear. Peer engineers argue that they should be able to iterate on their data structures as fast as they iterate on their services.
My answer: Velocity without reliability is just moving fast toward a cliff. When you break a consumer in production, you aren't just "moving fast"—you’re spending three days in an incident response war room fixing the downstream dependency chain. That is a massive tax on your velocity. A five-minute wait for a CI check is an investment, not a speed bump.
"What if we need to make a breaking change?" they ask. Then you version your stream. If you are changing the meaning of your data, you are fundamentally creating a new product. If you change a field type, you are breaking the contract. Create a new topic, publish the new schema, and run them in parallel for a migration window. Yes, it takes more work. That’s because data is a persistent asset, not a transient cache. If you aren't willing to manage the lifecycle of your data, you shouldn't be allowed to expose it to other teams.
"This is too much overhead for small teams." If you are a team of two, you might think you don't need this. You’re wrong. You’re just smaller, which means the cost of a catastrophic data failure is more likely to kill your startup. Automate the contract check early. It costs almost nothing to set up when you have three services. It’s nearly impossible to retrofit when you have three hundred.
Conclusion
We have to stop treating data producers as kings who can change the shape of the world whenever they feel like it. Data is the foundation of every financial ledger and healthcare record we manage. Treating it like an afterthought is negligence.
By moving your data contracts into your repository and enforcing them with hard failures in CI, you move the friction to where it belongs: the moment the change is proposed, not the moment it hits production. You don't need a fancy data mesh or a million-dollar governance platform. You need a YAML file, a basic CLI tool, and the backbone to make your CI pipeline block bad code.
Stop trusting the registry. Start breaking the build. Your downstream consumers will thank you—or at least, they won't be calling you at 3:00 AM.
Cover photo by Fons Heijnsbroek on Unsplash.
Top comments (1)
The CI gate catches declared breaking changes, which is where most of the 3AM pages come from. What it can't see is semantic drift: currency stays a string, billing quietly switches from "EUR" to "978", and every check stays green while downstream conversion goes wrong. Do you push value-level constraints into the contract as well (enums, regex, ranges), or is that left to runtime assertions on the consumer side?