Confluent’s Latest Funding Round Is One of Many Indicators Apache Kafka Is Here to Stay

tags: BeSafe kafka ecosystem round series valuation

With Confluent recently raising a $250 million Series E round on a $4.5 billion valuation, it’s safe to say Apache Kafka and its ecosystem is here to stay.

We celebrate the fact so many developers from Confluent and other companies have contributed over the years to make Kafka better and easier to use. For example, Colin McCabe at Confluent is currently working on KIP-500 which aims to eliminate Apache Zookeeper as a dependency to store Kafka’s metadata. This will make Kafka easier to use.

In the latest Kafka 2.5 release, some progress on KIP-500 was made with the completion of KIP-555 and KIP-543. KIP-555 is about the deprecation process leading up to KIP-500. Now, whenever the –-zookeeper flag is used with either kafka-configs.sh or kafka-message-partititions.sh, you should get a warning encouraging you to use -–bootstrap-server instead of the --zookeeper flag. The goal of KIP-543 is to handle more operations without direct Zookeeper access. This was accomplished with some modifications to Kafka’s ConfigCommand object. Arguably Kafka’s closest “competition”, Apache Pulsar, also uses Apache Zookeeper, so, once KIP-500 is complete, this would provide an additional point of differentiation between the two technologies.

At this point, there aren’t many questions lingering for this technology. One of the few questions that needs to be answered is one that needed to be answered for many of the technologies as prominent as this one. That is defining the boundaries of when Kafka should be used and shouldn’t. It’s easy to lose sight of this when you have incredible add-on technologies like ksqlDB that allows you to use language-neutral SQL to author stream processing programs.

For example, we often associate the concept of stream-table duality with Kafka. Some have gone so far as to say this is misleading. The argument is this may persuade development teams into thinking Kafka can be used as a relational database. The counter-argument is this is not listed as one of Kafka’s typical use cases.

References

1. What every software engineer should know about Apache Kafka. https://news.ycombinator.com/item?id=23206566. Last accessed: 7/7/2020.
2. Confluent Raises $250M and Kicks Off Project Metamorphosis. https://www.confluent.io/blog/series-e-round-metamorphosis/. Last accessed: 7/7/2020.
3. Apache Kafka Needs No Keeper: Removing the Apache ZooKeeper Dependency. https://www.confluent.io/blog/removing-zookeeper-dependency-in-kafka/. Last accessed: 7/7/2020.
4. Announcing ksqlDB 0.10.0. https://www.confluent.io/blog/ksqldb-0-10-0-latest-features-updates/. Last accessed: 7/16/2020.