On a Tuesday in October the carrier collection for our Leeds warehouse left half empty, and about six thousand picked orders sat on the dock overnight. They had been picked, packed and announced on Kafka exactly as normal. Our carrier integration had read every one of those announcements and parked each of them on its dead letter topic as unreadable.
The topic belongs to the warehouse team. Every message is Avro, registered in a Confluent Schema Registry, and our consumer uses classes generated from the schema as it stood when we last built, so it reads each record through that schema. That afternoon the warehouse team released a new version that dropped depot_code. In our generated schema depot_code had no default. Avro can resolve a record that is missing a field only if the reader's schema says what to fill in, and ours did not, so every message written after two o'clock failed to deserialise.
The warehouse team had done what they were supposed to do. The registry checks every new schema before accepting it, and it accepted this one. The subject used the registry's default compatibility mode, BACKWARD, and backward compatibility asks one specific question: can a consumer using the new schema read data written with the old one? Removing a field passes that easily. What it assumes is that consumers move to the new schema before producers start writing it. Their own consumers had. We were in another team, on another release cycle, and nothing in the check knew we existed.
The question we needed answered was the opposite one, whether a consumer still on the old schema can read what the new one writes. That is FORWARD compatibility, and under it the registry would have rejected removing a field with no default. Every subject with a consumer outside the team that owns it is now set to FULL_TRANSITIVE, which checks both directions against every earlier version. The warehouse team put depot_code back with a default and we replayed the dead letter topic that evening. Our generated schemas now give every field we read a default, and the dead letter topic alerts on its first message rather than its thousandth.
A compatibility check is only as useful as the question it asks. Ours was asking about deploy order, and assumed we would go first.
– Sergey Shinder
Top comments (0)