I work on KafScale, so take this with appropriate salt. But I've also spent time looking at WarpStream, AutoMQ, and Bufstream, and the marketing pages don't tell you what you actually need to know.
They all store data in S3. They all claim to be cheaper than Kafka. Here's what's actually different.
WarpStream
Confluent bought them in September 2024. The agents run in your VPC, but metadata and coordination run in Confluent's cloud.
Latency is 400-600ms p99. That's the cost of writing directly to S3 with no local buffer.
If you're already a Confluent shop and want S3 pricing without running infrastructure, this makes sense. If you don't want a cloud dependency, look elsewhere.
AutoMQ
Fork of Kafka with a new storage layer. Uses EBS as a write-ahead log, then tiers to S3.
Latency is around 10ms p99 because of the EBS buffer. That's close to real Kafka.
The catch: you're still managing EBS volumes. It's simpler than Kafka, but it's not stateless. Also BSL licensed, so read the terms if you're building a platform.
Bufstream
From the Buf/Protobuf people. S3 for storage, PostgreSQL for metadata. Native Iceberg output.
Latency around 500ms p99. Similar to WarpStream.
If you're building on Iceberg and want Kafka-compatible ingestion, this is purpose-built for that. If you're not in the lakehouse world, the PostgreSQL dependency is extra infrastructure for no benefit.
KafScale
Stateless Go brokers, S3 for storage, etcd for coordination. Apache 2.0 license.
Latency around 400ms p99. Same ballpark as WarpStream.
No transactions. No compacted topics. If you need those, use something else.
What's different: the segment format is documented and open. You can write processors that read directly from S3 without hitting brokers. That matters if you have analytical workloads (batch replay, Iceberg materialization, AI agents pulling context) that you want to keep separate from your streaming traffic.
The tradeoff is coupling. Your processors depend on the .kfs format, not just the Kafka protocol.
When to use what
Need low latency (<100ms)? AutoMQ or stick with Kafka/Redpanda.
Want managed S3 streaming with Confluent ecosystem? WarpStream.
Building on Iceberg? Bufstream.
Want Apache 2.0 license and direct S3 reads? KafScale.
Need transactions? Not KafScale. Kafka or Bufstream.
The latency thing
400-500ms is fine for most workloads. Log aggregation, ETL, async events, audit trails. If you're honest about your actual requirements, you probably don't need 10ms.
But if you do need it, the pure S3 options won't work for you. AutoMQ with EBS is the compromise.
The license thing
WarpStream is proprietary (Confluent). AutoMQ is BSL. Bufstream is proprietary. KafScale is Apache 2.0.
If you care about this, you already know why. If you don't, it probably won't matter until it does.
Why I built another one
After looking at all of these, I still wrote KafScale. Here's why.
WarpStream got the architecture right: stateless brokers, S3 storage, no disk ops. But it's proprietary and now owned by Confluent. I wanted that architecture without the dependency.
More importantly, I wanted processors that bypass brokers entirely. Kubernetes pods that read directly from S3, process historical data, write to Iceberg, feed AI agents. No connector framework. No fighting for broker resources. Just pods and object storage.
Someone pointed out that this makes processors "fat clients" coupled to the storage format. Fair. But Kafka's message format has had three versions in 15 years. V2 has been stable since 2017. The entire ecosystem depends on it not changing. That's a bet I'm willing to make.
The alternative is routing everything through brokers. Then your batch replay jobs compete with your real-time consumers. Your AI training pipeline spikes latency for everyone. That's the problem I was trying to solve.
Open format. Open license. Processors that scale independently from brokers. That's the gap none of the others filled.
More on the architecture: Streaming Data Becomes Storage-Native
Top comments (0)