Introduction
AutoMQ is a new generation Kafka distribution redesigned for cloud-native. While maintaining 100% compatibility with Apache Kafka, AutoMQ offers users up to 10 times cost advantage and a hundred times the elasticity advantage. It also supports move partition in seconds and automatic traffic rebalancing, addressing operational pain points.
Star AutoMQ on Github to learn more.
Why Do You Need AutoMQ?
When applying Apache Kafka clusters on a large scale in production scenarios, enterprises will inevitably be troubled by the following issues:
- High and rapidly inflating costs: Big data scenarios often accompany high throughput and store a large amount of data, requiring reserved disks, which are not flexible and costly.
- Slow partition migration: Scaling up or down requires partition migration and relocation of historical data, which can take several hours, during which cold reads of data can also affect services.
- Lack of traffic rebalancing: Different nodes have varying partition loads and lack a self-balancing mechanism, easily leading to partial node idleness or hotspot failures.
The above pain points are attributed to Apache Kafka's design philosophy oriented towards IDC and integrated storage-compute, which cannot be fundamentally resolved. Now, cloud computing has redefined the earliest hardware and software with deterministic services. At this point, it is necessary to redesign Kafka based on the cloud, fully leveraging the service-oriented capabilities and infinite resource pools of underlying cloud products to thoroughly solve the pain points of excessive costs, inability to scale elastically, and low operational efficiency.
Architecture Philosophy and Innovation
- From hardware-dependent to service-dependent
- From reserving resources to pay-as-you-go
- Separating storage to services, rather than software
- Shared storage is superior to non-shared architecture
- Architectural design oriented towards cloud billing items
Core Advantages
10 times cost optimization, scientifically reducing costs
- AutoMQ's brand-new cloud-native architecture fully utilizes the data high availability and elastic supply capabilities of cloud services such as object storage and Spot instances, bringing customers a 10 times cost advantage compared to Apache Kafka.
- Using object storage as the core primary storage, significantly reducing storage unit prices. Single replica high availability architecture, saving 2/3 of traffic replication costs.
- Fully utilizing Spot instances, combined with elastic scaling strategies, reducing computing costs.
Move partition in seconds and automatic traffic rebalancing
AutoMQ completely separates storage state to object storage services, with a completely stateless business logic layer. AutoMQ clusters can complete partition migration and traffic rebalancing in seconds, thoroughly solving the pain points of slow rebalancing and difficult partition migration in Apache Kafka. Paired with cloud vendor's elastic scaling group strategies, it easily achieves adaptive elastic scaling of the cluster.
100% Compatible with Apache Kafka
Unlike other vendors that reimplement the Kafka protocol, AutoMQ chooses a minimal aspect replacement of the storage layer, only modifying the underlying LogSegment implementation, while still reusing the main code of various Apache Kafka versions. AutoMQ can easily achieve 100% compatibility with Apache Kafka and quickly adapt to new versions. It has passed validation with 130+ functional use cases.
The functional scope covers 1000+ KIPs.
Compatible with versions 0.9.0 ~ 3.4.
Top comments (0)