Equality deletes have no address, and that's why your Iceberg reads crawl

Here's a failure I've watched happen more than once, and it always looks the same from the outside.

A table is fed by CDC off a transactional database. Flink sink, upsert mode, a primary key, commits every minute. The write path is healthy: no backpressure, no checkpoint failures, no small-file explosion because compaction runs nightly and does its job. Then over about three weeks, one dashboard query goes from roughly four seconds to roughly half a minute. The query didn't change. Row count is flat. The files look fine — a few hundred megabytes each, well above the target size.

Somebody checks the delete file count, sees a few hundred, decides that's normal, and the ticket drifts. Then the table gets upgraded to format version 3 for deletion vectors, because deletion vectors are the fix for merge-on-read overhead, and the read latency doesn't move at all.

It doesn't move because deletion vectors were never the problem on this table. The problem is equality deletes, and v3 left them exactly where they were.

One of these names a file and the other doesn't

The Iceberg spec says when a delete applies to a data file. For deletion vectors, one of the conditions is that the data file's path matches the vector's reference, and the spec is blunt that the reference is mandatory: "Setting the referenced file is required for deletion vectors." A vector is bound to one data file. At most one per data file per snapshot. A reader that opens a data file either loads that file's bitmap or loads nothing, and the bitmap is a position lookup — no key comparison, no join.

Equality deletes carry no such reference. The condition the spec gives is on sequence numbers: "The data file's data sequence number is strictly less than the delete's data sequence number," plus the same-partition rule. And if an equality delete file was written under an unpartitioned spec, "Equality delete files stored with an unpartitioned spec are applied as global deletes."

So an equality delete file saying id = 8831 doesn't say which data file holds id = 8831. It can't — the writer doesn't know, which is the whole reason streaming writers use them. The reader has to assume the worst: every live data file in that partition with a lower sequence number might contain it. There's no pruning available beyond partition and sequence number, and the spec notes separately that the file sequence number can't be used for pruning deletes at all.

That's the asymmetry. Deletion vectors cost you one bitmap per file you were already reading. Equality deletes cost you a key probe against every row of every older file in the partition.

Delete file count is the wrong metric

Here's what the reader actually does with an equality delete. It reads the delete file, materialises the key columns into a set, and then every candidate data file is scanned with a membership check per row. Cheap per row, but the work multiplies: number of delete keys, times number of data files the delete is allowed to touch.

Run the arithmetic on two tables. The first has 300 equality delete files of 50 keys each, all landing in one partition that holds 200 data files: 300 × 50 × 200, call it three million key probes. The second has 2,000 delete files of the same size, but spread across partitions holding three data files each: 2,000 × 50 × 3, about three hundred thousand. The second table has nearly seven times as many delete files and does a tenth of the work. If you're alerting on delete file count, you'd have paged on the healthy one.

The number worth watching is the product. Call it probe units: delete keys multiplied by data files touched. On a CDC table it grows quadratically between compactions, because each new delete file is older-than nothing and newer-than everything that came before it.

This also explains why the two instinctive fixes don't work. Committing less often makes each delete file bigger without reducing how many data files it reaches, so probe units stay where they were and your freshness gets worse. Compacting more often does help, but only by resetting the counter — it shortens the window, it doesn't change the slope, and you're paying a full data-file rewrite to clear deletes that only ever needed resolving into positions.

The worst version is an unpartitioned table. An equality delete written under an unpartitioned spec applies globally, so "data files in the partition" becomes "data files in the table" and fan-out is unbounded by construction. That argues for partitioning the CDC table, which is usually right, but it brings a constraint: for a partitioned upsert table, the source columns of the partition fields must be included in the equality fields. Partition on a column that isn't part of your natural key and you've widened the key your writer compares on, which changes what "the same row" means.

Measure it on your own table

You don't need a profiler for this. The entries metadata table exposes the manifest entries for both data and delete files, including the sequence number, so you can compute the fan-out directly in SQL.

WITH live AS (
  SELECT data_file.partition    AS part,
         data_file.content      AS content,       -- 0 data, 1 pos delete, 2 eq delete
         data_file.file_path    AS path,
         data_file.record_count AS record_count,
         sequence_number        AS seq
  FROM prod.cdc.orders.entries
  WHERE status < 2                                -- 0 existing, 1 added
),
eq AS (
  SELECT part, path, seq, record_count FROM live WHERE content = 2
),
dat AS (
  SELECT part, seq FROM live WHERE content = 0
),
fanout AS (
  SELECT eq.path,
         eq.part,
         eq.record_count AS delete_keys,
         COUNT(dat.seq)  AS data_files_touched
  FROM eq
  LEFT JOIN dat
    ON dat.part = eq.part
   AND dat.seq  < eq.seq
  GROUP BY eq.path, eq.part, eq.record_count
)
SELECT part,
       COUNT(*)                              AS eq_delete_files,
       SUM(delete_keys)                      AS delete_keys,
       MAX(data_files_touched)               AS worst_fanout,
       SUM(delete_keys * data_files_touched)  AS probe_units
FROM fanout
GROUP BY part
ORDER BY probe_units DESC
LIMIT 20;

Run it, then run it again tomorrow. On a healthy table probe_units is flat and boring. On the table in the opening it climbs all day and resets at 2am when compaction runs, which is why nobody looking at a weekly average ever sees it.

worst_fanout is the one to put on a dashboard. It's the number of data files a single delete file is permitted to touch, and it is an upper bound on how badly one minute of CDC traffic can degrade a full-partition scan.

Treat equality deletes as a buffer with an expiry

Equality deletes are not a mistake. They're the only way a streaming writer can emit a delete without first finding the row, and they survive in v3 for exactly that reason. The mistake is treating them as storage rather than as a buffer — something that must be drained on a schedule you've actually chosen.

Iceberg 1.12.0, released on 5 October 2026, ships the draining mechanism as a Flink maintenance task. ConvertEqualityDeletes resolves equality deletes into deletion vectors, so the key-probe work happens once in a maintenance job instead of on every scan. It requires format version 3 or later, validated at job startup, runs on Flink 1.20, 2.1, 2.2 and 2.3, and attaches as post-commit maintenance on the newer IcebergSink:

IcebergSink.forRowData(dataStream)
    .table(table)
    .tableLoader(tableLoader)
    .upsert(true)
    .equalityFieldColumns(List.of("id"))
    .rewriteDataFiles()
    .convertEqualityDeletes()
    .append();

The branch arrangement is the part worth understanding. The sink stages new data files and equality deletes on a source branch; the task reads the equality deletes from there and commits resolved data files plus deletion vectors to the target branch, which defaults to main. Readers on main never see an equality delete at all. Two constraints come with it: don't run compaction on the staging branch, because the planner needs those data files to still exist, and the staging branch must not contain v2 position delete files.

Everything the builder methods do is also reachable as write options under the flink-maintenance. prefix, which is what you want if the job is defined in SQL rather than in Java. The scheduling keys are the ones to think about:

INSERT INTO orders /*+ OPTIONS(
  'upsert-enabled' = 'true',
  'flink-maintenance.convert-equality-deletes.enabled' = 'true',
  -- default is 1, which converts after every single commit
  'flink-maintenance.convert-equality-deletes.schedule.commit-count' = '10'
) */
SELECT * FROM orders_cdc;

Run the task standalone through TableMaintenance and you get a better trigger than either of those: scheduleOnEqDeleteFileCount. Schedule on the backlog itself. Commit count is a proxy that drifts the moment your CDC volume changes shape, and an interval guarantees nothing about what piled up inside it.

If you're on Spark-side maintenance instead, the knob is delete-file-threshold on rewrite_data_files. Check its default before you assume you're covered — it is 2147483647. A data file will never be rewritten on delete-count grounds unless you set that number yourself:

CALL prod.system.rewrite_data_files(
  table   => 'cdc.orders',
  options => map(
    'delete-file-threshold', '8',
    'min-input-files',       '2',
    'partial-progress.enabled', 'true'
  )
);

That still rewrites whole data files, which is the heavier option. It's the right fallback, not the destination.

The argument

Deletion vectors are a real improvement, and the migration to v3 is worth doing. But they fixed the positional half of merge-on-read — the half where the writer already knew which file and which row. The expensive half on a CDC table is the half where it didn't, and no format version fixes that, because the information genuinely isn't available at write time. Only a maintenance pass can supply it.

So the upgrade to v3 doesn't buy you read performance on an upsert table. It buys you the right to drain the buffer, because ConvertEqualityDeletes won't run below format version 3. Doing the upgrade and stopping there gets you a longer spec version string and the same slow queries.

Something to check this week: run the query above against your busiest CDC-fed table, note worst_fanout, then run it again eight hours later. If that number has grown, you have a buffer with no expiry, and the clock on your next mystery regression is already running.