Running Kafka-style systems on Kubernetes is possible, but it often feels like fighting the model rather than working with it.
Stateful brokers, disks, tight coupling between compute and storage, painful scaling events, and recovery paths that are harder than they should be are common operational pain points. This becomes especially visible when clusters grow, traffic patterns fluctuate, or upgrades are frequent.
We started experimenting with an alternative design that keeps Kafka protocol compatibility but changes the underlying assumptions.
The core ideas are simple:
- Brokers are stateless and disposable
- Message segments are stored in object storage (S3 or compatible)
- Scaling brokers becomes a compute concern, not a data migration problem
- Retention and durability are handled by object storage lifecycle policies
KafScale was initially released yesterday and is not meant to replace Kafka everywhere. It’s an DevOps-friendly drop-in with minimal ops in mind.
We’ve been building this as an open-source project, licensed under Apache 2.0 and designed to be fully self-hosted.
Repository and technical details:
https://github.com/novatechflow/kafscale
Architecture and Docs:
https://kafscale.io
At this stage, the most valuable thing for us is feedback from people who operate streaming systems in production: where this model makes sense, where it breaks down, and what tradeoffs matter most in practice.
Top comments (0)