KafkaStreams.state() says REBALANCING. Every thread is dead.

We found this one the embarrassing way: a seeded demo stream sat inREBALANCING for a day. Not failing — rebalancing. The UI said so. The Streams client said so. The only place the truth existed was a server log nobody was reading.

The trap

Deploy a Kafka Streams topology whose source topic doesn't exist , and here's the exact sequence (Kafka clients 4.x, but the behaviour is old):

1. The group leader's assignment fails withINCOMPLETE_SOURCE_TOPIC_METADATA; the member receives the error in its assignment.
2. The StreamThread logs MissingSourceTopicException, transitionsPENDING_SHUTDOWN → DEAD, and does not retry. This is deliberate — a missing source topic is not a transient condition Kafka Streams can wait out.
3. Crucially, the client-level state machine never follows. TheStreamsUncaughtExceptionHandler isn't consulted (the thread shut down; it didn't throw), so there's no PENDING_ERROR → ERROR transition.KafkaStreams.state() last saw a rebalance start, and that's where it stays.

The result: a KafkaStreams instance with zero live threads that reportsREBALANCING indefinitely. It will never process a record, never error, and never change state again.

Why every Kafka UI has this bug

If your tool renders KafkaStreams.state() — and that's the obvious, documented thing to render — you have this bug. The state enum simply has no value for "all my threads are dead but nobody told the coordinator layer."REBALANCING is the truthful answer to the wrong question.

The signal that does exist is one call away:metadataForLocalThreads() returns per-thread metadata including each thread's state. A client reporting REBALANCING whose thread set is empty — or whose threads are all DEAD — is not rebalancing. It's gone.

The fix, in two halves

Surface it. We derive the displayed state instead of trusting the raw one:

static KafkaStreams.State effectiveState(KafkaStreams.State state,
                                         Collection<ThreadMetadata> threads) {
    if (state == KafkaStreams.State.REBALANCING
            && (threads.isEmpty()
                || threads.stream().allMatch(t -> "DEAD".equals(t.threadState())))) {
        return KafkaStreams.State.ERROR;
    }
    return state;
}

Genuine rebalances have live threads (STARTING, PARTITIONS_ASSIGNED, …) and pass through untouched. In our end-to-end test, deleting a running stream's source topic flips the reported state to ERROR within seconds — where before it showed REBALANCING until someone read the log.

Prevent the common case. The most frequent way to hit this is a typo'd topic name at deploy time. Since a missing source topic is unrecoverable by design, we now validate every source node's topics against the cluster before building the topology, and fail the deploy with the missing names — one listTopics() round-trip. (Fail-open if the listing itself errors: a broker hiccup shouldn't block a deploy that would have succeeded; the state derivation above is the backstop.)

Neither half needs anything from the broker that isn't already public API.

Takeaways

• KafkaStreams.state() is the state of the coordinator conversation, not the health of your processing. Dead threads don't move it.
• If you're operating Kafka Streams with your own dashboards: alert on thread liveness (metadataForLocalThreads(), or the alive-stream-threadsmetric), not on state() != RUNNING.
• If you're building a tool: derive, don't relay. The raw state is truthful and useless at the same time.

Both fixes shipped in Alginte 0.7.0. The stuck demo that taught us this now recovers in seconds — and deploying against a typo'd topic tells you the topic's name instead of miming a rebalance.