DEV Community

Cover image for How Kafka
Darshan Turakhia
Darshan Turakhia

Posted on Originally published at darshanturakhia.com

How Kafka

#architecture #programming #backend #devops

03:14 UTC. Security's on-call channel gets a message from the fraud-scanning job: a customer API key flagged and revoked four days earlier is still authorizing requests, all of them from ap-south-1. The revoke ticket has been closed as resolved since Monday.

the setup

authz-gateway sits in front of every API route and checks incoming keys against a local cache before anything touches the database. It used to call Postgres on every request; that was a flat 40ms added to p99 regardless of what the request actually needed. Six months earlier, someone replaced it with a Kafka Streams state store hydrated from a compacted topic, api-key-status, keyed by api_key_id. p99 for the auth check dropped to 4ms. An active key is a normal record. A revoked key is a tombstone, a record with the key present and the value set to null, which is how compacted topics represent deletion.



gateway/authz.js
Enter fullscreen mode Exit fullscreen mode

function isKeyValid(apiKeyId) {
const record = keyStatusStore.get(apiKeyId); // local RocksDB state store
return record !== undefined && record.status === 'active';
}
Enter fullscreen mode Exit fullscreen mode

The consumer group backing that store runs with group.instance.id set, static membership, turned on months earlier after a rolling deploy triggered a full rebalance and briefly spiked auth latency across every pod, not just the ones restarting. Static membership means a pod that drops out during a deploy keeps its partition assignment reserved instead of forcing an immediate reassignment. It was a good fix for that problem. It also meant nobody was watching for what happens when a pod goes quiet for reasons that have nothing to do with a deploy.

the scramble

First move: check the source of truth. Postgres's api_keys table shows the key correctly marked revoked_at four days ago. Whatever's wrong isn't the revoke itself, it happened, it's recorded, it's just not everywhere it needs to be.

Second theory: the tombstone never made it onto the topic. A console consumer against api-key-status from the revoke's approximate timestamp finds it immediately, a null-value record for that exact key, committed and replicated. The event exists. The dead end didn't cost much, five minutes to rule out, but it mattered: it meant the problem was downstream of Kafka, not upstream of it.

Third theory, on-call's: a stale edge cache. This service doesn't have one on the authz path, ruled out by reading its own deployment topology rather than by testing anything.

the hunt

The fraud job's flagged requests all traced to three pods, all in ap-south-1, all part of the same consumer group. kafka-consumer-groups.sh --describe against that group showed those three partitions current, no lag, caught up to the log's high-water mark. Not stuck. Not behind. Just wrong.



kafka-consumer-groups.sh --describe --group authz-gateway
Enter fullscreen mode Exit fullscreen mode

PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID
4 88213 88213 0 authz-gateway-7f2a...
7 61940 61940 0 authz-gateway-7f2a...
9 73301 73301 0 authz-gateway-7f2a...
Enter fullscreen mode Exit fullscreen mode

Caught up with a wrong answer is a different bug than stuck behind. The next thing to check wasn't the topic, it was those three pods' history. Kubernetes events showed the answer: an ap-south-1-scoped NetworkPolicy change went out five days earlier as part of a node pool migration, and a rule ordering mistake in it blocked egress from those three pods to the Kafka broker subnet for roughly 30 hours before someone caught it in an unrelated ticket and reverted it. Nothing restarted those pods. Their liveness probe was an HTTP ping against the gateway's own port, which had nothing to do with whether it could reach Kafka, so it stayed green the entire time.

With no broker access, the consumer's background heartbeat thread stopped reaching the group coordinator. Kafka correctly detected that after session.timeout.ms and reassigned those three partitions to healthy members elsewhere, who kept the cache correct for everyone else. The revoke tombstone was produced and consumed by those healthy members within seconds, exactly as designed. It's what happened to the three isolated pods, once the network policy reverted and they reconnected, that broke.

the find

Kafka Streams persists its state store to local disk and checkpoints the offset it last processed. On reconnect, if that checkpoint is still within the log, it doesn't replay from scratch, it does a delta restore: read forward from the checkpointed offset, apply whatever's new. That's the whole point of a local store, cheap recovery. The three pods' checkpoints were roughly 6 hours stale when the network came back, right around when the revoke had been written.

The topic's delete.retention.ms was left at Kafka's default, 86400000ms, 24 hours. That setting exists specifically so a lagging consumer gets a grace window to see a tombstone before the log cleaner physically removes it. The three pods were unreachable for roughly 30 hours, past that window. By the time they reconnected and asked to resume from their checkpoint, the tombstone that checkpoint needed to see had already been compacted away. The delta restore read forward past where the tombstone used to be and found nothing for that key, so it never touched the in-memory record still sitting there from before the isolation: active.

Nothing in that path throws an error. The consumer isn't lagging, the topic isn't corrupt, the restore completes successfully. It just completes with a state store that is now permanently wrong for one key, with no future event on the topic ever going to fix it, because as far as the topic is concerned that key's history now starts after the revoke.

the fix

Immediate: force the three pods to drop their local state directory and rebuild from the earliest offset rather than delta-restoring from a checkpoint that could no longer be trusted.



recovery, run once per affected pod
Enter fullscreen mode Exit fullscreen mode

kubectl exec authz-gateway-7f2a-xyz -- rm -rf /data/kafka-streams/authz-gateway
kubectl delete pod authz-gateway-7f2a-xyz
Enter fullscreen mode Exit fullscreen mode

Then the two changes meant to keep this from being invisible again. First, a health check that actually reflects Kafka connectivity instead of just the local HTTP server:



gateway/health.js, after
Enter fullscreen mode Exit fullscreen mode

let lastPollAt = Date.now();

consumer.on('poll', () => {
lastPollAt = Date.now();
});

app.get('/healthz', (req, res) => {
const staleMs = Date.now() - lastPollAt;
if (staleMs > 60_000) {
return res.status(503).json({ error: kafka poll stale for \${staleMs}ms\ });
}
res.status(200).json({ ok: true });
});
Enter fullscreen mode Exit fullscreen mode

Second, delete.retention.ms on api-key-status went from 24 hours to 7 days, trading disk for a grace window wide enough to cover a multi-day network isolation, not just a routine restart. It doesn't fix the underlying risk, a consumer down longer than the retention window can still miss a tombstone, but it moves the failure mode from "a botched network policy" to "something has been broken for a week and nobody noticed," which is a bar this team was willing to accept.

the aftermath

96 hrs Time between the revoke and the fraud job catching it in ap-south-1

3 Requests authorized with the revoked key before detection

24h → 7d delete.retention.ms on api-key-status

2 Other Kafka-backed services found with the same HTTP-only healthcheck

  • Static group membership solves rebalance storms during deploys, but it also means a consumer that's gone quiet for an unrelated reason keeps its assignment and its local state without raising anything from outside. It needs its own signal, not the assumption that a healthy pod implies a healthy consumer.
  • delete.retention.ms is a grace window, not a guarantee. It's sized against how long a consumer might lag, not against how long one might be completely unreachable, and those are different failure modes with very different durations.
  • A local state store that restores by delta from a checkpoint is fast exactly because it trusts the checkpoint. That trust is only as good as the log's willingness to still contain everything between the checkpoint and now, and compaction doesn't make that promise indefinitely.
  • A healthcheck that only proves a process is running, not that it's connected to the one dependency that determines whether its answers are correct, will report green through the exact failure it exists to catch.

Kafka never logged an error either. It deleted exactly what its retention setting told it to delete, on schedule, while the only three pods that needed to see it first were on the wrong side of a firewall rule.

Top comments (0)