A deleted CDC row still needs a version: a Python counterexample

A current-state projection removes a merchant record. Later, an older update for that key arrives and the record comes back.

The delete worked. The consumer forgot enough ordering information to accept stale state afterward.

Here is a small Python experiment that separates the visible value from the latest version applied to its key.

The source order and delivery order differ

Assume a single source namespace and a strictly ordered integer version per key. Higher versions describe newer source state. Payloads at a given version are immutable.

Source version	Operation	Value
1	upsert merchant-7	10
2	upsert merchant-7	20
3	delete merchant-7	absent

Now deliver versions in the order 1, 3, 2. This could represent reordering in a downstream pipeline; the example does not claim that a particular source connector reorders its own log.

A consumer that applies every arrival ends with merchant-7=20. The source’s newest state says the key is deleted.

Adding version checks only to visible rows still leaves a hole if deleting the row also discards the version. When version 2 arrives, the consumer finds no row and no evidence of version 3.

Keep a versioned deletion marker

The experiment stores the latest tuple for each key: version, operation, and value. Deletes keep ordering metadata while the visible projection excludes deleted keys.

• An older version is stale.
• An identical repeat at the same version is harmless.
• A different payload at the same version is a conflict.
• A higher version replaces the previous state, including a legitimate newer upsert after a delete.

The code uses plain dictionaries and has no external dependencies. Save it as cdc_delete_demo.py and run it with Python.

from itertools import permutations


def apply(latest, seq, key, op, value=None):
    # Teaching contract: comparable integer versions per key; immutable payloads.
    if op not in {"upsert", "delete"} or (op == "delete" and value is not None):
        raise ValueError("invalid operation")
    previous = latest.get(key)
    incoming = (seq, op, value)
    if previous is not None:
        if seq < previous[0]:
            return "stale"
        if seq == previous[0]:
            if incoming != previous:
                raise ValueError("conflicting payload at the same version")
            return "repeat"
    latest[key] = incoming
    return "applied"


def visible(latest):
    return {key: value for key, (_, op, value) in latest.items() if op == "upsert"}


changes = [(1, "merchant-7", "upsert", 10),
           (3, "merchant-7", "delete", None),
           (2, "merchant-7", "upsert", 20)]

naive = {}
for seq, key, op, value in changes:
    if op == "delete":
        naive.pop(key, None)
    else:
        naive[key] = value
assert naive == {"merchant-7": 20}

latest = {}
statuses = [apply(latest, *change) for change in changes]
assert statuses == ["applied", "applied", "stale"]
assert visible(latest) == {}
assert latest["merchant-7"] == (3, "delete", None)
assert apply(latest, 3, "merchant-7", "delete") == "repeat"
before = latest.copy()
try:
    apply(latest, 3, "merchant-7", "upsert", 999)
except ValueError:
    pass
else:
    raise AssertionError("same-version conflict accepted")
assert latest == before

for delivery_order in permutations(changes):
    trial = {}
    for change in delivery_order:
        apply(trial, *change)
    assert trial == {"merchant-7": (3, "delete", None)}

unknown = {}
apply(unknown, 3, "merchant-7", "delete")
assert apply(unknown, 2, "merchant-7", "upsert", 20) == "stale"
assert visible(unknown) == {}
apply(latest, 4, "merchant-7", "upsert", 30)
assert visible(latest) == {"merchant-7": 30}

print("arrival-order projection:", naive)
print("version-aware statuses:", statuses)
print("six delivery orders, repeat, conflict, unknown delete: passed")
print("newer upsert:", visible(latest))

The local run produced:

arrival-order projection: {'merchant-7': 20}
version-aware statuses: ['applied', 'applied', 'stale']
six delivery orders, repeat, conflict, unknown delete: passed
newer upsert: {'merchant-7': 30}

Notice that a delete for an unknown key still records its version. Dropping it as a no-op would let an older upsert recreate the key later.

The six permutations test the same three immutable changes under every delivery order. They all end at the version-3 deletion marker. That establishes the result for this fixture; it does not establish correctness for arbitrary source schemas or ordering rules.

This marker is not a Kafka tombstone

The dictionary’s deletion marker is local consumer state that retains a version. A Kafka tombstone has a different wire-level meaning. Debezium’s PostgreSQL documentation describes a delete event followed, by default, by a same-key record with a null value for log compaction. See the official delete-event documentation.

Do not assume a null Kafka value carries the version tuple used in this Python example. A real consumer needs to handle the connector’s delete envelope, key, metadata, configuration, and subsequent tombstone according to its actual contract.

The ordering contract does the work

The integer seq here is a teaching device. It must not be replaced casually with a wall-clock timestamp. Equal timestamps, separate source partitions, source resets, and key reuse can require additional ordering or identity information.

Likewise, the dictionary is not durable. If values and version metadata are stored separately, both must survive failure consistently. Otherwise a restart can recreate the same bug even when the in-memory implementation is correct.

Removing deletion markers also needs a replay boundary. If an old update can still arrive, removing the only evidence of a newer delete makes the key look unseen again. This example does not choose a safe retention duration, model source epochs, or handle concurrent writers.

A current-state delete also does not establish that a payment was refunded. Financial events and customer or merchant profile state need separate business rules.

Start with a finite exercise, then change one assumption

I maintain a free fintech data engineering roadmap. Its E14 CDC current projection exercise has a smaller contract: take a finite input, sort by seq, then apply upserts and deletes.

Sorting the complete finite input resolves that exercise’s delivery-order problem. The incremental version-aware consumer above is an extension, not a claim that E14 requires a streaming service.

The lesson on deletes and tombstones asks learners to distinguish current-state deletion from financial reversal and historical handling.

For a next counterexample, clear only the version metadata after processing version 3, then deliver version 2. Write down which recovery invariant the consumer lost before adding another feature.