I have a Kafka setup running on Linux. I have set the num of partitions in server.properties to 5. I also have set the num of partitions for the topic I am using (let us call it topic1) to 5.
Now, I have 5 consumers (implemented in Java) subscribing to the above topic. I am expecting that all 5 partitions should split the traffic equally among all the 5 consumers.
I tried the following:
ensured that the num of partitions is set to 5 at the topic level and in server.properties.
I ran to ensure that there are 5 partitions for this topic:
**kafka-topics.sh --zookeeper localhost:2181 --describe --topic topic1 Topic: topic1 TopicId: 4kX9oP3ARA2uHQ1_nVGY-Q PartitionCount: 5 ReplicationFactor: 1 Configs: Topic: topic1 Partition: 0 Leader: 0 Replicas: 0 Isr: 0 Topic: topic1 Partition: 1 Leader: none Replicas: 1 Isr: 1 Topic: topic1 Partition: 2 Leader: none Replicas: 2 Isr: 2 Topic: topic1 Partition: 3 Leader: none Replicas: 3 Isr: 3 Topic: topic1 Partition: 4 Leader: none Replicas: 4 Isr: 4**
I also ran a similar config in another setup with 5 partitions and 5 consumers on a Windows setup. But there, I could see that all 5 consumers are equally getting the data.
But on the Linux setup I mentioned above, it always goes to just 1 consumer.
Any inputs on how I can debug this issue? From the Kafka-topics describe output, it is clear that this topic has 5 partitions. But still those 5 partitions are not splitting among the 5 consumers available.
Is there a way to check either via command line or from the Kafka logs, how many partitions are actually used during the test run? If Kafka-topics describe says 5 partitions, does it actually mean that 5 partitions are also getting used when we run a test?
Note that I am running a load test on this machine - with more than 100 users firing data.
Top comments (0)