You ship the job. It passes CI. Your dbt tests show green. You push to production, grab a coffee, and feel like a competent engineer. Then, at 3:00 AM, the downstream dashboard team wakes up to a PagerDuty alert because your service decided that user_id should now be a string instead of an integer. The schema registry was updated, but nobody read the Slack announcement, and your pipeline just broke three critical financial reporting models.
This is the state of most data engineering teams today. We call it "coordination," but it’s really just a polite term for guessing what the other team did. Data contracts aren't just fancy YAML files; they are the only thing standing between you and a weekend of debugging JSON parsing errors.
The real question isn't whether you need contracts, but how you enforce them. You have two real paths: you either integrate the contract into your CI/CD pipeline using schema-as-code, or you rely on runtime checks that catch the fire after the house is already burning down.
The contenders
On one side, you have Schema Registry-based enforcement (e.g., Confluent/Avro or AWS Glue Schema Registry). This approach treats the contract as a gatekeeper at the ingestion layer. If the producer doesn't match the registry, the message is dropped or rejected at the broker level.
On the other, you have CI-level contract testing (e.g., Great Expectations, dbt-checkpoint, or custom JSON Schema validation in GitHub Actions). This approach forces the producer to validate their data against a versioned contract before the code is even allowed to merge. You catch the breaking change at the PR stage, not in the Kafka topic.
Photo by Eyasu Etsub on Unsplash
The cost of failure
If you use a Schema Registry, your failure mode is "data loss." A producer updates their service, the registry rejects the message because of a minor type mismatch, and your data is effectively incinerated at the source. In a healthcare context, this is a HIPAA nightmare—you are failing to capture data that might be clinically significant.
With CI-level enforcement, the failure mode is "developer friction." The CI build fails, the PR stays blocked, and the developer has to fix the contract or the code before they can ship. It’s annoying, but it’s predictable. I’d take a blocked PR over a production outage any day.
In terms of cost, Schema Registries incur a per-message latency tax. If you’re high-frequency trading or doing real-time IoT ingestion, that millisecond overhead adds up. CI checks are "free" in terms of production overhead because they execute during development. You pay in compute minutes on GitHub Actions or GitLab CI, but those costs are negligible compared to the time spent on manual schema reconciliation.
Operational burden and complexity
Schema Registries are a massive operational lift. You are now managing a stateful service. If your registry goes down or experiences a partition, your entire ingestion pipeline stalls. I’ve spent countless hours debugging Kafka producer configurations where the registry URL was misconfigured in the environment variables, leading to weird SerializationException errors that are nearly impossible to trace in production logs.
CI-level enforcement is significantly cleaner because it’s stateless. You define a contract in a contracts/ directory in your repository. In your .github/workflows/main.yml, you run:
- name: Validate Contract
run: |
npm install -g ajv-cli
ajv validate -s contracts/user-event-v1.json -d data/sample-payload.json
If the JSON doesn't match, the build fails. No state, no registry, no infrastructure to patch. You are just using the same tools you use for linting code. It’s portable, it’s version-controlled, and it’s transparent.
The "Version 2.0" trap
The biggest issue with the Schema Registry approach is the "version bump" dance. You have to ensure that every downstream consumer is ready to handle the new schema version, or you have to support backward compatibility forever. The Schema Registry forces you into a specific evolution strategy—usually AVRO's BACKWARD_TRANSITIVE compatibility.
When you enforce contracts in CI, you can implement custom logic that the registry can't handle. For example, you can write a script that scans your downstream dbt models to see if a field is actually being used before allowing a breaking change.
I once worked on a team where we used a simple Python script in our CI pipeline to check for field deletions. If a field was marked deprecated in the contract, we’d allow the removal. If it wasn't, the CI build would fail with a link to the downstream data warehouse impact report. This level of context-aware enforcement is impossible if you’re just relying on a Kafka registry.
What I'd pick, and why
If you’re building a greenfield platform today, skip the heavy-weight Schema Registries unless you are working in a highly regulated, high-throughput environment where the broker must be the source of truth. For 90% of data teams, CI-based contract enforcement is the superior choice.
My recommendation: use JSON Schema or Protobuf definitions stored in the same repository as the producer code. Add a CI step that validates the sample payloads against these schemas. Use a tool like dbt-checkpoint if you’re deep in the warehouse stack.
The caveat? This requires discipline. If your team doesn't maintain the contracts/ directory, the system decays immediately. You have to treat the contract like production code. If a test fails, you don't "force push" to bypass it; you update the contract and notify the downstream teams.
Treating contracts as a shared library is the only way to scale. When you move the enforcement point to the PR, you turn "data quality" from a frantic, reactive firefighting exercise into a boring, routine part of the development lifecycle. And boring is exactly what you want when you’re managing mission-critical data.
Don’t wait for the schema registry to tell you that you’ve broken the world. Catch it when you commit. Your on-call self will thank you.
Top comments (0)