Well I'm surprised to hear "could not scale by design". We are in the works of implementing it for our enterprise and have seen the exact opposite. Straight from the Apache website: "The disk structures Kafka uses scale well—Kafka will perform the same whether you have 50 KB or 50 TB of persistent data on the server." The logs are written to disk in conjunction of using RocksDB for certain cached data storage.
Kafka partitions use only one thread. To me the best situation where Kafka starts to gain some interest is distribution (horizontal scale). But what about vertical scaling ? The post questioning was supposed to bring up a serious debate.
For further actions, you may consider blocking this person and/or reporting abuse
We're a place where coders share, stay up-to-date and grow their careers.
Well I'm surprised to hear "could not scale by design". We are in the works of implementing it for our enterprise and have seen the exact opposite. Straight from the Apache website: "The disk structures Kafka uses scale well—Kafka will perform the same whether you have 50 KB or 50 TB of persistent data on the server." The logs are written to disk in conjunction of using RocksDB for certain cached data storage.
Yes, I was surprised as well.
Kafka partitions use only one thread. To me the best situation where Kafka starts to gain some interest is distribution (horizontal scale). But what about vertical scaling ? The post questioning was supposed to bring up a serious debate.