We Ditched RabbitMQ for Kafka 4.0 and Cut Our Message Processing Latency by 55%

After 18 months of battling 2.1-second p99 message processing latency with RabbitMQ 3.12, our team migrated to Kafka 4.0 and slashed that metric by 55% to 945ms, while reducing infrastructure costs by 32% in the first quarter post-migration. This isn't a marketing pitch—it's a data-backed breakdown of why we made the switch, how we executed the migration without downtime, and the exact benchmarks that prove Kafka 4.0's performance gains for high-throughput, low-latency workloads. We initially chose RabbitMQ in 2021 for its ease of setup and strong support for complex routing rules, but as our order processing volume grew from 1k msg/s to 12k msg/s, we hit Erlang VM bottlenecks that RabbitMQ's architecture couldn't overcome. Our RabbitMQ cluster suffered 3 unplanned outages in Q1 2024 alone, each costing an average of $22k in lost revenue, forcing us to evaluate alternatives. We benchmarked Kafka 4.0, Redpanda 2.4, and Pulsar 2.10 over 6 weeks, and Kafka 4.0 emerged as the clear winner for our workload.

📡 Hacker News Top Stories Right Now

• GTFOBins (167 points)
• Talkie: a 13B vintage language model from 1930 (355 points)
• Microsoft and OpenAI end their exclusive and revenue-sharing deal (877 points)
• Can You Find the Comet? (31 points)
• Is my blue your blue? (531 points)

Key Insights

• Kafka 4.0 reduced our p99 message processing latency by 55% (from 2.1s to 945ms) for 12k msg/s workloads
• RabbitMQ 3.12's per-node throughput peaked at 4.2k msg/s vs Kafka 4.0's 18.7k msg/s on identical EC2 c7g.2xlarge instances
• Migration to Kafka 4.0 cut our monthly messaging infrastructure costs by 32% ($14.7k/month savings) by reducing node count from 12 to 4
• Kafka 4.0's native KRaft mode eliminates ZooKeeper dependencies, reducing operational overhead by 40% for small teams

Why We Stuck with RabbitMQ for 3 Years (Then Outgrew It)

When we first adopted RabbitMQ in 2021, we were a 6-person startup processing 1k order events per second. RabbitMQ's Erlang-based architecture was a perfect fit: it took one engineer 2 days to set up a 3-node cluster with mirrored queues, dead letter exchanges, and TTL-based message expiration. We relied heavily on RabbitMQ's routing keys to direct different event types to dedicated consumer groups, and its management UI made it easy to debug message flow without deep infrastructure knowledge.

By 2023, we had grown to 4 backend engineers supporting 12k msg/s, and cracks started to show. RabbitMQ's Erlang VM uses a global lock for message routing, which caps per-node throughput at ~4.2k msg/s regardless of instance size. To handle growth, we scaled out to 12 c7g.2xlarge nodes, but inter-node synchronization overhead pushed our p99 latency from 400ms in 2022 to 2.1s by Q1 2024. Mirrored queue failover took 28 seconds on average, during which we saw 1-2% message loss. We also spent 18 hours per week on RabbitMQ maintenance: managing Erlang versions, troubleshooting cluster partitions, and tuning queue depths.

We evaluated three alternatives in Q2 2024: Redpanda 2.4, Pulsar 2.10, and Kafka 4.0. Redpanda offered 10% lower latency than Kafka but had a smaller ecosystem and limited support for our existing Spring Boot tooling. Pulsar's separate bookie and broker nodes added operational complexity our 4-person team couldn't support. Kafka 4.0's stable KRaft mode (removing ZooKeeper dependencies) cut operational overhead by 40%, and its sequential I/O-based log storage delivered 4x higher per-node throughput than RabbitMQ. After 6 weeks of benchmarking, we committed to migrating.

Head-to-Head Benchmark Results

We ran all benchmarks on identical EC2 c7g.2xlarge instances (16 vCPU, 32GB RAM, 10Gbps network) over 72 hours of steady load. We used 1KB message payloads matching our production order event schema, and measured latency from producer send to consumer commit. Below are the final results comparing our 12-node RabbitMQ 3.12 cluster to a 4-node Kafka 4.0 cluster:

Metric

RabbitMQ 3.12.0 (12 nodes)

Kafka 4.0.0 (4 nodes)

% Change

p99 Message Processing Latency

2100ms

945ms

-55%

Max Throughput (total cluster)

42k msg/s

74.8k msg/s

+78%

Throughput per Node

4.2k msg/s

18.7k msg/s

+345%

Monthly Infrastructure Cost

$46,000

$31,300

-32%

Operational Overhead (hours/week)

18

11

-39%

Max Connections per Node

12k

45k

+275%

Failover Time (node failure)

28s

4s

-86%

Kafka 4.0 Producer: Idempotent, Low-Latency Writes

Our first step was replacing RabbitMQ producers with Kafka 4.0 idempotent producers. Idempotency is enabled by default in Kafka 4.0, but we explicitly configured it to avoid duplicate messages during retries. We added Micrometer Prometheus metrics to track end-to-end producer latency, which was critical for validating our 55% latency reduction claim. All code examples are available in our public migration repo: https://github.com/streaming-eng/kafka-rabbitmq-migration.

import org.apache.kafka.clients.producer.*;
import org.apache.kafka.common.serialization.StringSerializer;
import io.micrometer.core.instrument.MeterRegistry;
import io.micrometer.core.instrument.Timer;
import io.micrometer.prometheus.PrometheusConfig;
import io.micrometer.prometheus.PrometheusMeterRegistry;
import java.time.Duration;
import java.util.Properties;
import java.util.concurrent.ExecutionException;
import java.util.concurrent.atomic.AtomicLong;

/**
 * Kafka 4.0 Idempotent Producer with latency tracking and error handling.
 * Configured for our production workload: 12k msg/s, 1KB message size, us-east-1.
 */
public class Kafka4LatencyTrackedProducer {
    private static final String BOOTSTRAP_SERVERS = \"kafka-broker-1:9092,kafka-broker-2:9092,kafka-broker-3:9092\";
    private static final String TOPIC = \"order-events-v2\";
    private static final int MAX_RETRIES = 5;
    private static final AtomicLong totalMessagesSent = new AtomicLong(0);
    private static final PrometheusMeterRegistry registry = new PrometheusMeterRegistry(PrometheusConfig.DEFAULT);
    private static final Timer producerLatencyTimer = Timer.builder(\"kafka.producer.latency\")
            .description(\"End-to-end producer latency for Kafka 4.0 writes\")
            .register(registry);

    public static void main(String[] args) {
        Properties props = new Properties();
        // Kafka 4.0 bootstrap servers (KRaft mode, no ZooKeeper)
        props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, BOOTSTRAP_SERVERS);
        // Idempotent producer enabled by default in Kafka 4.0, but explicit for clarity
        props.put(ProducerConfig.ENABLE_IDEMPOTENCE_CONFIG, true);
        // Required for idempotence: acks=all, max.in.flight.requests.per.connection=5 (Kafka 4.0 supports >1)
        props.put(ProducerConfig.ACKS_CONFIG, \"all\");
        props.put(ProducerConfig.MAX_IN_FLIGHT_REQUESTS_PER_CONNECTION, 5);
        // Retry config for retriable errors (e.g., network blips, leader elections)
        props.put(ProducerConfig.RETRIES_CONFIG, MAX_RETRIES);
        props.put(ProducerConfig.RETRY_BACKOFF_MS_CONFIG, 100);
        // Serializers for key (String) and value (String, but we use JSON in prod)
        props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());
        props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());
        // Batching config to optimize throughput: 16KB batch size, 100ms linger
        props.put(ProducerConfig.BATCH_SIZE_CONFIG, 16384);
        props.put(ProducerConfig.LINGER_MS_CONFIG, 100);
        // Compression for network efficiency: Kafka 4.0 supports zstd, lz4, gzip, snappy
        props.put(ProducerConfig.COMPRESSION_TYPE_CONFIG, \"zstd\");
        // Request timeout: 30s to handle broker leader elections
        props.put(ProducerConfig.REQUEST_TIMEOUT_MS_CONFIG, 30000);
        props.put(ProducerConfig.DELIVERY_TIMEOUT_MS_CONFIG, 120000);

        KafkaProducer producer = null;
        try {
            producer = new KafkaProducer<>(props);
            // Sample messages: simulate order events with 1KB payload
            String sampleMessage = \"{\\\"orderId\\\":\\\"%s\\\",\\\"userId\\\":\\\"%s\\\",\\\"amount\\\":%.2f,\\\"timestamp\\\":%d}\";
            for (int i = 0; i < 12000; i++) { // Simulate 12k messages for our workload
                String key = \"order-\" + i;
                String value = String.format(sampleMessage, \"ORD-\" + i, \"USR-\" + (i % 1000), 99.99, System.currentTimeMillis());
                // Track latency via Micrometer Timer
                producerLatencyTimer.recordCallable(() -> {
                    try {
                        producer.send(new ProducerRecord<>(TOPIC, key, value), (metadata, exception) -> {
                            if (exception != null) {
                                System.err.println(\"Failed to send message \" + key + \": \" + exception.getMessage());
                                // Log retriable vs non-retriable errors separately
                                if (exception instanceof RetriableException) {
                                    System.err.println(\"Retriable error, will retry automatically\");
                                } else {
                                    System.err.println(\"Non-retriable error, manual intervention required\");
                                }
                            } else {
                                totalMessagesSent.incrementAndGet();
                            }
                        }).get(); // Block for latency measurement (use async in prod, but sync for benchmark)
                    } catch (InterruptedException | ExecutionException e) {
                        Thread.currentThread().interrupt();
                        System.err.println(\"Producer send interrupted: \" + e.getMessage());
                    }
                    return null;
                });
            }
            System.out.println(\"Total messages sent: \" + totalMessagesSent.get());
            System.out.println(\"Prometheus metrics: \" + registry.scrape());
        } catch (KafkaException e) {
            System.err.println(\"Fatal Kafka producer error: \" + e.getMessage());
            e.printStackTrace();
        } finally {
            if (producer != null) {
                producer.flush();
                producer.close(Duration.ofSeconds(5));
            }
        }
    }
}

Kafka 4.0 Consumer: Manual Offset Acking for Data Consistency

RabbitMQ's auto-ack model caused intermittent message loss during consumer restarts, so we adopted manual offset acking for Kafka consumers. This ensures offsets are only committed after business logic completes successfully. We disabled auto-commit, set a 5-minute max poll interval to handle long-running processing, and added latency tracking for end-to-end consumer processing time.

import org.apache.kafka.clients.consumer.*;
import org.apache.kafka.common.serialization.StringDeserializer;
import io.micrometer.core.instrument.MeterRegistry;
import io.micrometer.core.instrument.Timer;
import io.micrometer.prometheus.PrometheusConfig;
import io.micrometer.prometheus.PrometheusMeterRegistry;
import java.time.Duration;
import java.util.Collections;
import java.util.Properties;
import java.util.concurrent.atomic.AtomicLong;

/**
 * Kafka 4.0 Consumer with manual offset acking, latency tracking, and error handling.
 * Configured for our production order-events-v2 topic, 4 consumers per node.
 */
public class Kafka4ManualAckConsumer {
    private static final String BOOTSTRAP_SERVERS = \"kafka-broker-1:9092,kafka-broker-2:9092,kafka-broker-3:9092\";
    private static final String TOPIC = \"order-events-v2\";
    private static final String GROUP_ID = \"order-processors-v2\";
    private static final AtomicLong totalMessagesProcessed = new AtomicLong(0);
    private static final AtomicLong processingErrors = new AtomicLong(0);
    private static final PrometheusMeterRegistry registry = new PrometheusMeterRegistry(PrometheusConfig.DEFAULT);
    private static final Timer consumerProcessingTimer = Timer.builder(\"kafka.consumer.processing.latency\")
            .description(\"End-to-end consumer processing latency for Kafka 4.0\")
            .register(registry);

    public static void main(String[] args) {
        Properties props = new Properties();
        props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, BOOTSTRAP_SERVERS);
        props.put(ConsumerConfig.GROUP_ID_CONFIG, GROUP_ID);
        // Disable auto-commit: we manually ack offsets after processing
        props.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, false);
        // Earliest offset only if no committed offset exists (first run)
        props.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, \"earliest\");
        // Deserializers for key and value
        props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getName());
        props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getName());
        // Max poll records: 500 per poll to balance throughput and processing time
        props.put(ConsumerConfig.MAX_POLL_RECORDS_CONFIG, 500);
        // Session timeout: 45s, max poll interval: 5m to handle long processing
        props.put(ConsumerConfig.SESSION_TIMEOUT_MS_CONFIG, 45000);
        props.put(ConsumerConfig.MAX_POLL_INTERVAL_MS_CONFIG, 300000);
        // Heartbeat interval: 3s to detect consumer failures quickly
        props.put(ConsumerConfig.HEARTBEAT_INTERVAL_MS_CONFIG, 3000);

        KafkaConsumer consumer = null;
        try {
            consumer = new KafkaConsumer<>(props);
            consumer.subscribe(Collections.singletonList(TOPIC));
            System.out.println(\"Subscribed to topic: \" + TOPIC);

            while (true) {
                // Poll for 100ms if no records available
                ConsumerRecords records = consumer.poll(Duration.ofMillis(100));
                if (records.isEmpty()) {
                    continue;
                }
                // Process each record with latency tracking
                for (ConsumerRecord record : records) {
                    consumerProcessingTimer.recordCallable(() -> {
                        try {
                            // Simulate business logic: process order event (10ms avg)
                            Thread.sleep(10);
                            totalMessagesProcessed.incrementAndGet();
                            // Manual offset ack: commit after processing
                            consumer.commitSync(Collections.singletonMap(
                                    new TopicPartition(record.topic(), record.partition()),
                                    new OffsetAndMetadata(record.offset() + 1)
                            ));
                        } catch (InterruptedException e) {
                            Thread.currentThread().interrupt();
                            processingErrors.incrementAndGet();
                            System.err.println(\"Processing interrupted for record \" + record.key() + \": \" + e.getMessage());
                        } catch (CommitFailedException e) {
                            processingErrors.incrementAndGet();
                            System.err.println(\"Offset commit failed for record \" + record.key() + \": \" + e.getMessage());
                            // Retry commit once
                            try {
                                consumer.commitSync();
                            } catch (CommitFailedException ex) {
                                System.err.println(\"Retry commit failed: \" + ex.getMessage());
                            }
                        } catch (Exception e) {
                            processingErrors.incrementAndGet();
                            System.err.println(\"Unexpected error processing record \" + record.key() + \": \" + e.getMessage());
                        }
                        return null;
                    });
                }
                // Log metrics every 1000 records
                if (totalMessagesProcessed.get() % 1000 == 0) {
                    System.out.println(\"Processed \" + totalMessagesProcessed.get() + \" messages, errors: \" + processingErrors.get());
                    System.out.println(\"Consumer processing latency metrics: \" + registry.scrape());
                }
            }
        } catch (KafkaException e) {
            System.err.println(\"Fatal Kafka consumer error: \" + e.getMessage());
            e.printStackTrace();
        } finally {
            if (consumer != null) {
                consumer.close(Duration.ofSeconds(5));
            }
        }
    }
}

Zero-Downtime Migration: Dual-Write with Feature Flags

We avoided downtime by implementing a dual-write phase where all messages were sent to both RabbitMQ and Kafka, with a feature flag controlling what percentage of traffic was processed by Kafka consumers. We started with 1% Kafka traffic, increased to 10% after 1 week, 50% after 2 weeks, and 100% after 4 weeks. This allowed us to catch issues early without impacting production users.

import org.springframework.beans.factory.annotation.Value;
import org.springframework.stereotype.Service;
import com.rabbitmq.client.Channel;
import com.rabbitmq.client.Connection;
import com.rabbitmq.client.ConnectionFactory;
import org.apache.kafka.clients.producer.KafkaProducer;
import org.apache.kafka.clients.producer.ProducerRecord;
import org.apache.kafka.clients.producer.ProducerConfig;
import org.apache.kafka.common.serialization.StringSerializer;
import java.io.IOException;
import java.time.Duration;
import java.util.Properties;

/**
 * Dual-write service for zero-downtime migration from RabbitMQ to Kafka 4.0.
 * Writes to both systems during transition, controlled by feature flag.
 */
@Service
public class DualWriteMigrationService {
    private final boolean kafkaEnabled;
    private final Channel rabbitChannel;
    private final KafkaProducer kafkaProducer;
    private final String rabbitQueue = \"order-events-v1\";
    private final String kafkaTopic = \"order-events-v2\";

    public DualWriteMigrationService(
            @Value(\"${feature.flag.kafka-enabled:false}\") boolean kafkaEnabled,
            @Value(\"${rabbitmq.host}\") String rabbitHost,
            @Value(\"${kafka.bootstrap-servers}\") String kafkaBootstrapServers) throws Exception {
        this.kafkaEnabled = kafkaEnabled;
        // Initialize RabbitMQ connection (legacy)
        ConnectionFactory rabbitFactory = new ConnectionFactory();
        rabbitFactory.setHost(rabbitHost);
        rabbitFactory.setPort(5672);
        rabbitFactory.setUsername(\"rabbit-user\");
        rabbitFactory.setPassword(\"rabbit-pass\");
        Connection rabbitConnection = rabbitFactory.newConnection();
        this.rabbitChannel = rabbitConnection.createChannel();
        this.rabbitChannel.queueDeclare(rabbitQueue, true, false, false, null);
        // Initialize Kafka 4.0 producer (new)
        Properties kafkaProps = new Properties();
        kafkaProps.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, kafkaBootstrapServers);
        kafkaProps.put(ProducerConfig.ENABLE_IDEMPOTENCE_CONFIG, true);
        kafkaProps.put(ProducerConfig.ACKS_CONFIG, \"all\");
        kafkaProps.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());
        kafkaProps.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());
        this.kafkaProducer = new KafkaProducer<>(kafkaProps);
    }

    /**
     * Publishes order event to RabbitMQ (always) and Kafka (if feature flag enabled).
     * Ensures no data loss during migration.
     */
    public void publishOrderEvent(OrderEvent event) {
        String message = event.toJson();
        String key = event.getOrderId();
        // Always write to RabbitMQ (legacy path)
        try {
            rabbitChannel.basicPublish(\"\", rabbitQueue, null, message.getBytes());
        } catch (IOException e) {
            System.err.println(\"Failed to write to RabbitMQ: \" + e.getMessage());
            throw new RuntimeException(\"RabbitMQ write failed\", e);
        }
        // Write to Kafka only if feature flag is enabled
        if (kafkaEnabled) {
            try {
                kafkaProducer.send(new ProducerRecord<>(kafkaTopic, key, message), (metadata, exception) -> {
                    if (exception != null) {
                        System.err.println(\"Failed to write to Kafka: \" + exception.getMessage());
                    }
                });
            } catch (Exception e) {
                System.err.println(\"Unexpected Kafka write error: \" + e.getMessage());
            }
        }
    }

    /**
     * Sample order event POJO for migration testing.
     */
    static class OrderEvent {
        private String orderId;
        private String userId;
        private double amount;
        private long timestamp;

        public OrderEvent(String orderId, String userId, double amount) {
            this.orderId = orderId;
            this.userId = userId;
            this.amount = amount;
            this.timestamp = System.currentTimeMillis();
        }

        public String getOrderId() { return orderId; }
        public String toJson() {
            return String.format(\"{\\\"orderId\\\":\\\"%s\\\",\\\"userId\\\":\\\"%s\\\",\\\"amount\\\":%.2f,\\\"timestamp\\\":%d}\",
                    orderId, userId, amount, timestamp);
        }
    }
}

Migration Case Study: Order Processing Team

• Team size: 4 backend engineers
• Stack & Versions: RabbitMQ 3.12.0 on EC2 c7g.2xlarge (12 nodes), Java 17, Spring Boot 3.2.0, PostgreSQL 16; migrated to Kafka 4.0.0 on same instance type (4 nodes), Java 17, Spring Boot 3.2.0, PostgreSQL 16
• Problem: p99 message processing latency was 2.1s, throughput plateaued at 42k msg/s across 12 nodes, monthly infrastructure cost $46k, frequent RabbitMQ cluster outages during peak traffic (3 outages/month)
• Solution & Implementation: Dual-write phase with feature flag to route 10% traffic to Kafka initially, incrementally increase to 100%, decommission RabbitMQ after 4 weeks of zero errors, enabled Kafka 4.0 KRaft mode, idempotent producers, consumer offset auto-commit disabled with manual ack
• Outcome: p99 latency dropped to 945ms (55% reduction), throughput increased to 74.8k msg/s across 4 nodes, monthly cost reduced to $31.3k (32% savings), zero outages in 6 months post-migration

Developer Tips for Kafka 4.0 Migrations

1. Enable Native Latency Metrics Before You Migrate

You cannot optimize what you do not measure, and this is doubly true for messaging migrations where latency is the primary success metric. Before sending a single message to Kafka 4.0, enable its native JMX metrics or integrate with a metrics library like Micrometer to export latency, throughput, and error rates to a monitoring system like Prometheus or Datadog. We made the mistake of starting our migration without baseline Kafka metrics, which forced us to re-run 2 weeks of benchmarks to validate our latency claims. Kafka 4.0 exposes hundreds of metrics out of the box, but the three most critical for migration validation are kafka.producer.latency (end-to-end producer write time), kafka.consumer.processing.latency (time from poll to offset commit), and kafka.server:type=BrokerTopicMetrics,name=MessagesInPerSec (throughput per topic). Use these metrics to set alert thresholds before cutover: we set a p99 latency alert at 1s, and a throughput drop alert at 10% below baseline. For tooling, we recommend the Micrometer Prometheus registry to export metrics in a format Grafana can natively ingest, which cut our dashboard setup time from 8 hours to 1 hour. Below is a snippet of adding a latency timer to a Kafka producer:

// Add to producer config
private static final Timer producerLatencyTimer = Timer.builder(\"kafka.producer.latency\")
        .description(\"End-to-end producer latency\")
        .register(registry);

// Wrap send in timer
producerLatencyTimer.recordCallable(() -> {
    producer.send(new ProducerRecord<>(topic, key, value));
    return null;
});

Always run metrics for 72 hours of steady load before making migration decisions—short benchmarks miss garbage collection pauses and network jitter that only appear under sustained load. We also recommend running nightly metric reports to share progress with stakeholders, which helped us secure buy-in from product teams worried about migration risk.

2. Use Dual-Write with Feature Flags for Zero Downtime

Cutting over 100% of traffic from RabbitMQ to Kafka in a single deployment is a recipe for disaster, even for teams with extensive messaging experience. We learned this the hard way in a 2022 migration that caused 2 hours of downtime when Kafka consumers failed to process a legacy message format. Dual-write with feature flags eliminates this risk by sending all messages to both systems, then gradually shifting consumer traffic to Kafka. Start by routing 1% of consumer traffic to Kafka (with RabbitMQ as the primary source of truth), validate that latency and error rates match baseline, then increase to 10%, 50%, and finally 100% over 4-6 weeks. Feature flags allow you to roll back to RabbitMQ instantly if issues arise, which we had to do twice during our migration when Kafka consumers hit an unexpected serialization error. For feature flag tooling, we used Spring Cloud Config to manage flags via properties files, but commercial tools like LaunchDarkly or Unleash add percentage-based rollout and instant rollback capabilities that are worth the cost for large teams. The dual-write producer must be idempotent to avoid duplicate processing: Kafka's idempotent producer handles this automatically, but you should also add a message ID to your payload and deduplicate in consumers during the transition phase. We kept the dual-write phase running for 4 weeks after 100% cutover to handle any edge cases, then decommissioned RabbitMQ 2 weeks later. Below is a snippet of a feature flag check in a dual-write service:

@Value(\"${feature.flag.kafka-enabled:false}\")
private boolean kafkaEnabled;

public void publishEvent(Event event) {
    // Always write to RabbitMQ
    rabbitTemplate.convertAndSend(queue, event);
    // Write to Kafka only if flag enabled
    if (kafkaEnabled) {
        kafkaTemplate.send(topic, event);
    }
}

Never skip the dual-write phase for revenue-critical workloads—our 4-week phase caught 3 bugs that would have caused customer impact, saving an estimated $45k in lost revenue.

3. Disable Auto-Commit and Use Manual Offset Acking

RabbitMQ's auto-ack model is convenient for simple workloads, but it is dangerous for Kafka migrations where data consistency is non-negotiable. Auto-commit in Kafka commits offsets every 5 seconds (by default) regardless of whether processing succeeded, which leads to message loss if a consumer crashes after processing a batch but before the next auto-commit. Manual offset acking ensures that offsets are only committed after your business logic completes successfully, even if that logic takes minutes. We disabled auto-commit by setting enable.auto.commit=false, then committed offsets manually after processing each batch, with a retry for commit failures. This eliminated the 0.2% message loss we saw during our initial testing with auto-commit. For Spring Kafka users, this means setting AckMode.MANUAL in your container properties, then calling Acknowledgment.acknowledge() after processing. Manual acking adds slight complexity, but it is worth it for the peace of mind that no messages are lost during consumer restarts or crashes. We also recommend setting max.poll.interval.ms to a value higher than your longest processing time (we used 5 minutes) to avoid consumer group rebalances while processing large batches. Below is a snippet of manual offset acking in a Kafka consumer:

@KafkaListener(topics = \"order-events-v2\")
public void processOrder(OrderEvent event, Acknowledgment ack) {
    try {
        orderService.process(event);
        ack.acknowledge(); // Commit offset only after processing succeeds
    } catch (Exception e) {
        log.error(\"Failed to process order\", e);
        // Do not ack: offset will not be committed, message will be re-processed
    }
}

We also added a dead letter topic (DLT) for messages that fail processing 3 times, which caught 12 invalid messages per day that would have otherwise blocked the consumer group. Manual acking combined with DLTs reduced our message loss rate to 0.001% post-migration.

Join the Discussion

We're opening this up to the engineering community to share experiences and lessons learned from messaging migrations. Whether you've migrated from RabbitMQ to Kafka, or have strong opinions on messaging architecture, we want to hear from you.

Discussion Questions

• With Kafka 4.0's KRaft mode now stable, do you think ZooKeeper will be fully deprecated in messaging stacks by 2026?
• What trade-offs have you encountered when choosing between RabbitMQ's ease of use and Kafka's throughput for latency-sensitive workloads?
• How does Kafka 4.0's performance compare to Redpanda 2.4 for your high-throughput use cases?

Frequently Asked Questions

How long did the full migration from RabbitMQ to Kafka 4.0 take?

Our team of 4 backend engineers completed the full migration in 11 weeks: 2 weeks for benchmarking, 4 weeks for dual-write phase, 3 weeks for traffic cutover, 2 weeks for RabbitMQ decommission. We allocated 20% of sprint capacity to migration work to avoid disrupting feature delivery, which kept product teams happy and prevented burnout.

Did we encounter any data loss during the migration?

Zero data loss. We used dual-write with idempotent Kafka producers and manual offset acking for consumers, combined with a 7-day retention period for both RabbitMQ and Kafka topics. We also ran nightly reconciliation jobs comparing message IDs across both systems for 4 weeks post-migration, which caught 2 duplicate messages that were automatically deduplicated.

Is Kafka 4.0 worth it for low-throughput workloads (under 1k msg/s)?

For workloads under 1k msg/s, RabbitMQ 3.12 is still a better fit: it has lower operational overhead, easier setup, and comparable latency (p99 ~300ms vs Kafka's ~280ms). Kafka 4.0's advantages only materialize at throughput above 5k msg/s, where its sequential I/O model outperforms RabbitMQ's Erlang VM-based message routing. Small teams with low throughput should stick with RabbitMQ unless they expect 3x growth in the next year.

Conclusion & Call to Action

For high-throughput (5k+ msg/s), latency-sensitive workloads, Kafka 4.0 is the clear winner over RabbitMQ 3.12. The 55% latency reduction, 4x throughput per node, and 32% cost savings we achieved are repeatable for teams willing to invest in a structured migration. If you're hitting RabbitMQ's throughput ceiling, start benchmarking Kafka 4.0 today—don't wait for outages to force your hand. We've open-sourced all our migration code and Grafana dashboards at https://github.com/streaming-eng/kafka-rabbitmq-migration to help you skip the trial and error we went through. For teams with smaller workloads, RabbitMQ remains a solid choice, but for scaling systems, Kafka 4.0's performance and operational efficiency are unmatched.

55%Reduction in p99 message processing latency after migrating to Kafka 4.0