How Apache SeaTunnel Handles Kafka Topic Partition Expansion Without a Job Restart

1. Scenario

When using an Apache SeaTunnel streaming job to consume data from a Kafka topic, what happens if the topic is expanded from 2 partitions to 4?

By default, the new partitions are not consumed, and none of the data written to them will be processed unless the SeaTunnel job is restarted.

The reason is that SeaTunnel’s Kafka Source scans the available partitions when the job starts. Without additional configuration, it does not automatically detect partitions added later.

Restarting the job can force SeaTunnel to scan the topic again, but this approach introduces several issues:

• The entire job is paused during the restart. If checkpoints are used for recovery, there is also a risk of missing data depending on the recovery and offset state.
• When start_mode = EARLIEST, restarting may cause the newly assigned partitions to be consumed from the beginning, which can replay historical data and result in duplicate records.

SeaTunnel provides the partition-discovery.interval-millis parameter specifically for this scenario. It periodically scans Kafka for newly added partitions and automatically adds them to the consumer, so the streaming job does not need to be restarted.

2. Hands-On Guide

2.1 Prerequisites

• A Kafka cluster
• Apache SeaTunnel 2.3.12

2.2 Create a Topic with Two Partitions

Create a Kafka topic named ksource with two partitions:

bin/kafka-topics.sh --create --topic ksource \
  --bootstrap-server ip:9092 \
  --partitions 2 --replication-factor 1

2.3 Configure the SeaTunnel Streaming Job

Create a SeaTunnel configuration file at job/k.conf.

There are three key settings to pay attention to:

• Set job.mode to STREAMING.
• Enable checkpointing explicitly with checkpoint.interval.
• Set partition-discovery.interval-millis to a positive value.

env {
  parallelism = 2
  job.mode = "STREAMING"
  checkpoint.interval = 5000
}

source {
  Kafka {
    bootstrap.servers = "ip:9092"
    topic = "ksource"
    consumer.group = "seatunnel_k_group"
    partition-discovery.interval-millis = 5000
    start_mode = "EARLIEST"
    format = "json"
    schema = {
      fields {
        id = INT
        cusname = STRING
        amount = DOUBLE
      }
    }
  }
}

sink {
  Console {}
}

The key settings are:

• partition-discovery.interval-millis = 5000: SeaTunnel scans Kafka for new partitions every 5 seconds.
• start_mode = EARLIEST: When a new partition is discovered, SeaTunnel starts consuming from the earliest available offset.
• Schema field types such as INT, STRING, and DOUBLE are basic types and can be specified directly.

2.4 Start the Job and Verify Normal Consumption

Start the SeaTunnel job in local mode:

bin/seatunnel.sh --config job/k.conf -m local

Open another terminal and produce three messages to ksource. Because the topic currently has only two partitions and no key is specified, the messages will be distributed across partitions 0 and 1:

bin/kafka-console-producer.sh --broker-list ip:9092 --topic ksource

Enter the following messages, one per line:

{"id":1,"cusname":"Zhang San","amount":100.50}
{"id":2,"cusname":"Li Si","amount":200.00}
{"id":3,"cusname":"Wang Wu","amount":300.00}

The corresponding records should appear in the SeaTunnel job console, confirming that the source is consuming data normally.

2.5 Expand the Topic from 2 Partitions to 4

Now increase the number of partitions from 2 to 4:

bin/kafka-topics.sh --alter --topic ksource \
  --bootstrap-server ip:9092 \
  --partitions 4

Verify the result:

bin/kafka-topics.sh --describe --topic ksource \
  --bootstrap-server ip:9092

The topic should now contain four partitions: 0, 1, 2, and 3.

2.6 Produce More Data and Check the New Partitions

Continue producing six messages to ksource. Because no key is specified, Kafka's default partitioning behavior distributes the records across all four partitions. Some of them should therefore be written to the newly added partitions 2 and 3.

bin/kafka-console-producer.sh --broker-list ip:9092 --topic ksource

Enter:

{"id":4,"cusname":"Zhao Liu","amount":400.00}
{"id":5,"cusname":"Sun Qi","amount":500.00}
{"id":6,"cusname":"Zhou Ba","amount":600.00}
{"id":7,"cusname":"Wu Jiu","amount":700.00}
{"id":8,"cusname":"Zheng Shi","amount":800.00}
{"id":9,"cusname":"Qian Shiyi","amount":900.00}

The six records should gradually appear in the SeaTunnel console, including those written to the newly added partitions 2 and 3.

The job continues running without a restart, and data from the new partitions is consumed automatically.

2.7 Comparison: What Happens Without Dynamic Partition Discovery?

Now remove partition-discovery.interval-millis from the configuration and repeat Steps 2.4 through 2.6.

The results are:

• Messages produced before the topic expansion: consumed normally.
• Messages produced after the expansion and written to new partitions 2 and 3: not consumed.
• Restarting the SeaTunnel job: the new partitions are detected and their data can then be consumed.

The difference between the two configurations is summarized below:

3. Important Considerations

3.1 The Starting Offset of New Partitions Depends on start_mode

New partitions do not have existing checkpoint offsets. The starting position therefore depends on the configured start_mode.

With EARLIEST, SeaTunnel consumes the new partition from its earliest available offset. If the partition already contains historical data, that data will be consumed as well.

With LATEST, SeaTunnel only consumes new records written after the partition is discovered.

3.2 Avoid Setting the Discovery Interval Too Low

Each discovery scan sends metadata requests to the Kafka cluster. A shorter interval allows SeaTunnel to detect partition changes more quickly, but it also increases the request frequency and therefore the load on Kafka.

A 5-second interval is a reasonable starting point. If partition changes are infrequent, you can consider increasing the interval to 30–60 seconds.

3.3 Kafka Partitions Can Only Be Increased, Not Decreased

Kafka supports increasing the number of partitions for a topic, but does not support reducing the partition count. Reducing partitions could result in data loss.

3.4 Partition Expansion Triggers Consumer Group Rebalancing

When the number of partitions changes, Kafka rebalances the consumer group and redistributes partitions among consumers.

A brief fluctuation in consumption during the rebalance is expected behavior.

3.5 Existing Partitions Are Not Affected

Dynamic partition discovery only handles newly added partitions.

Existing partitions continue consuming from their checkpointed offsets, so enabling partition discovery does not cause SeaTunnel to reprocess data from existing partitions.

3.6 Types with Precision or Generic Parameters Must Be Quoted

Types containing commas or angle brackets must be enclosed in double quotes, for example:

"decimal(10,2)"
"array<int>"
"map<string, int>"

Otherwise, the HOCON parser will report a syntax error.

4. Summary

The partition-discovery.interval-millis parameter solves a specific problem: automatically detecting newly added Kafka partitions while a SeaTunnel streaming job is already running.

Set the parameter to a positive interval, and SeaTunnel periodically checks Kafka for new partitions and automatically starts consuming from them without requiring a job restart.

There are three key points to remember:

Streaming mode + checkpointing + a positive partition discovery interval.

The starting offset for a newly discovered partition depends on start_mode, while the discovery interval should be chosen based on the balance between detection latency and Kafka cluster load.

With this configuration in place, expanding a Kafka topic no longer requires manually restarting the SeaTunnel job just to make the new partitions visible to the consumer.