CDC Schema Changes: What Breaks and How to Ship Them

On Friday at 17:40, someone renamed one column in the orders table. The pipeline stayed green all weekend. On Monday, the revenue dashboard showed zero for Saturday and Sunday. Nothing failed, and that was the problem.

This is Part 5 of CDC in Production. In Part 4 the sink learned to survive duplicates, late events and deletes. This time the table itself changes. We'll follow a schema change through the pipeline, look at the registry rules that decide what's allowed, see what Snowflake and the lakehouse can absorb, and finish with a migration pattern that never breaks either side.

Prefer to watch? Here's the 7-minute video version with animated diagrams:

Video link: CDC Schema Changes: What Breaks and How to Ship Them

How a schema change travels

Postgres logical decoding (the feature that turns the write-ahead log into row changes) doesn't carry DDL. So Debezium never sees the ALTER TABLE itself. It notices the new shape on the next row change for that table and refreshes its in-memory schema. That event goes out with the new structure.

If you use Avro, the converter then registers the new structure as a new schema version in Schema Registry, a service that stores every version of each message schema. Versions are grouped under a subject; with the default naming, the value schema for topic shop.public.orders lives under shop.public.orders-value. Before accepting a new version, the registry runs a compatibility check against what's already there.

The converter is configured on the connector (or the Connect worker):

"value.converter": "io.confluent.connect.avro.AvroConverter",
"value.converter.schema.registry.url": "http://schema-registry:8081"

Registry compatibility modes

Compatibility is the rule the registry uses to decide whether a new version may be registered. The default is BACKWARD.

Mode	Guarantee	Upgrade first
BACKWARD (default)	Consumers using the new schema can read data written with the old one	Consumers
FORWARD	Consumers using the old schema can read data written with the new one	Producers
FULL	Both directions	Either side, independently

By default the check compares only against the latest version. The _TRANSITIVE variants (BACKWARD_TRANSITIVE and so on) compare against every version, which matters if a consumer might replay old data from the start of a topic.

What does that allow in practice, with Avro?

Change	BACKWARD	FORWARD	FULL
Add an optional field (has a default)	✅	✅	✅
Remove an optional field	✅	✅	✅
Add a required field (no default)	❌	✅	❌
Remove a required field	✅	❌	❌

All of it comes from one rule in Avro's schema resolution: if the reader's schema has a field with no default, and the data was written without that field, the read fails. With Debezium, a nullable Postgres column becomes an optional field with a null default. A NOT NULL column becomes a required field, unless Debezium can carry the column's default value into the schema (it does this for many defaults; for function defaults, see mistake 4 below).

The rename, two ways

amount_cents became total_cents. To Debezium, that's one field gone and a new one added.

If the column is NOT NULL, the new field is required, with no default. Under BACKWARD, the registry rejects it. The converter throws, and with Kafka Connect's default errors.tolerance=none, the connector task fails. That's loud, and loud is good: someone gets paged on Friday.

In the real incident, the column was nullable. Both fields are optional, so the registry accepts the change and nothing turns red. Snowflake, with schema evolution switched on, adds a TOTAL_CENTS column. The old AMOUNT_CENTS column gets NULL on every new row. Every dashboard still sums AMOUNT_CENTS, so weekend revenue reads zero.

A failed connector is the good outcome. The dangerous schema changes are the ones every system accepts.

What Snowflake absorbs

Schema evolution lets a load add columns to the target table automatically. It needs three things:

ALTER TABLE analytics.orders SET ENABLE_SCHEMA_EVOLUTION = TRUE;

-- The loading role needs EVOLVE SCHEMA (or OWNERSHIP) on the table
GRANT EVOLVE SCHEMA ON TABLE analytics.orders TO ROLE loader;

and a supported loading method: COPY INTO (with MATCH_BY_COLUMN_NAME), Snowpipe, or Snowpipe Streaming. Plain INSERT can't evolve a table.

And it does only two things:

• it adds new columns, and
• it drops NOT NULL from columns that are missing in new data.

It never renames and never changes a type.

Type changes through ALTER TABLE ... ALTER COLUMN are stricter still:

Change	Allowed?
Make a VARCHAR longer	✅
Raise a number's precision	✅
Change a number's scale	❌
Make a VARCHAR shorter	❌
Turn a number into a VARCHAR	❌

So if Postgres moves price from numeric(10,2) to numeric(12,4), Snowflake can't follow in place. The scale changed. You add a new column, backfill it, and switch readers over, which is the pattern at the end of this post.

The lakehouse: more forgiving, not magic

Apache Iceberg tracks every column by a unique field ID, not by its name. Renames and drops are metadata changes, so no data files are rewritten. A new column gets a fresh ID, so it never reads old data by mistake, even if it reuses an old name. Type changes are limited to safe widening: int to long, float to double, or a decimal with more precision and the same scale.

Delta Lake can rename and drop columns only with column mapping enabled:

ALTER TABLE orders SET TBLPROPERTIES (
  'delta.columnMapping.mode' = 'name',
  'delta.minReaderVersion'   = '2',
  'delta.minWriterVersion'   = '5'
);

That's a protocol upgrade you can't undo, and after a rename or drop it can break readers of the table's change data feed.

Here's the catch for both formats: a CDC sink still sees a rename as a dropped field and a new one, because that's what the events say. Unless you rename the table column yourself, you get the same split as in Snowflake.

Expand, then contract

The safe pattern is expand/contract: make the schema bigger first, move everything over, and only then make it smaller. A rename becomes four steps:

-- 1. Expand: add the new column as nullable (or with a constant default)
ALTER TABLE orders ADD COLUMN total_cents bigint;

-- 2. Backfill in small batches, and have the application write both columns
UPDATE orders
SET    total_cents = amount_cents
WHERE  total_cents IS NULL
  AND  id BETWEEN 1 AND 50000;

-- 3. Move every reader (views, models, dashboards) to total_cents

-- 4. Contract, weeks later, once nothing reads the old column
ALTER TABLE orders DROP COLUMN amount_cents;

Because amount_cents was nullable, every step only adds or removes an optional field, which the registry accepts in any mode. (Dropping a NOT NULL column removes a required field, which passes BACKWARD but not FORWARD or FULL.) Two practical notes:

• The backfill is CDC traffic. Every row it updates becomes a change event, which is why it runs in batches.
• Snowflake won't drop the old column for you. Schema evolution never removes columns. Drop AMOUNT_CENTS in the warehouse yourself once readers have moved.

Two guardrails

Test schemas in CI. Schema Registry has a compatibility endpoint that tells you whether a schema would be accepted, without registering it:

# Wrap the new Avro schema in the request body the registry expects
python3 -c 'import json, sys; print(json.dumps({"schema": open(sys.argv[1]).read()}))' \
  orders-value.avsc > payload.json

curl -X POST \
  -H 'Content-Type: application/vnd.schemaregistry.v1+json' \
  --data @payload.json \
  'http://schema-registry:8081/compatibility/subjects/shop.public.orders-value/versions/latest?verbose=true'
# {"is_compatible":false,"messages":[...]}

Fail the build when is_compatible is false. Remember that a nullable rename passes this check, so it catches only the loud kind of break.

Watch the warehouse. Alert when a new column appears on a captured table, and when a column's null rate jumps:

SELECT DATE_TRUNC('hour', updated_at)              AS hour,
       COUNT(*)                                    AS rows_loaded,
       COUNT_IF(amount_cents IS NULL) / COUNT(*)   AS null_rate
FROM   analytics.orders
WHERE  updated_at >= DATEADD(day, -2, CURRENT_TIMESTAMP())
GROUP  BY 1
HAVING null_rate > 0.2
ORDER  BY 1;

The threshold is yours to pick. Either alert would have caught Friday's rename on Friday night.

Five common mistakes

1. Renaming a column with a single ALTER. Use expand/contract.
2. Changing a numeric column's scale in place. Snowflake can't follow.
3. Setting compatibility to NONE just to get a failed connector running again. You've switched off the only alarm.
4. Trusting function defaults. Debezium exports a default like CURRENT_TIMESTAMP(6) as a zero value (1970-01-01) in the schema.
5. Changing a schema during an incremental snapshot. The Postgres connector doesn't support it.

Recap

• Postgres doesn't send DDL, so the next event carries the new schema.
• The registry checks compatibility, BACKWARD by default. Optional fields are safe to add or remove; required fields are not.
• Snowflake adds columns, but never renames or changes types.
• In the lakehouse, your sink still sees a rename as a drop and an add.
• So expand, then contract.

This week, find the last schema change on a captured table and check what it did in the warehouse.

The CDC in Production series

• Part 1: Change Data Capture (CDC) Explained (article · video)
• Part 2: Debezium Postgres to Kafka: Every Setting Explained (video)
• Part 3: The Silent CDC Bug: Data Loss With No Errors (video)
• Previous, Part 4: CDC Deletes, Tombstones and the Idempotent MERGE (video)
• Part 5: CDC Schema Changes: What Breaks and How to Ship Them (this post · video)
• Next, Part 6: Running CDC in Production: The WAL Slot Trap (video, out November 2)

---

If this was useful, the video version has the animated diagrams for every step: watch it on YouTube. Subscribe to Pipeline Whiteboard for more diagram-first data engineering explainers. Next up, in the final part: the WAL slot trap, lag, heartbeats, alerting and runbooks.