Kafka: Distributed Event Streaming at Scale

A practical guide to Apache Kafka — the distributed event streaming platform for large-scale, real-time data pipelines — covering the log-based architecture, topics and partitions, producers and consumers, consumer groups, delivery semantics, .NET integration, and how it compares to RabbitMQ.

---

Table of Contents

1. Introduction
2. The Log: Kafka's Core Abstraction
3. Topics and Partitions
4. Brokers, Replication, and Fault Tolerance
5. Producers
6. Consumers and Consumer Groups
7. Delivery Semantics
8. .NET Integration
9. Schema Registry and Message Contracts
10. Kafka Streams and Stream Processing
11. Retention, Compaction, and Replay
12. Kafka vs. RabbitMQ vs. Managed Alternatives
13. Common Pitfalls
14. Quick Reference Table
15. Conclusion

---

Introduction

Apache Kafka is a distributed event streaming platform built around a fundamentally different model than a traditional message broker like RabbitMQ (covered in this series' companion guide): instead of a smart broker that routes and removes messages once delivered, Kafka is a distributed, append-only log — messages (events) are written sequentially, retained for a configured period regardless of whether they've been consumed, and any number of independent consumers can read through that log at their own pace, from any point they choose.

// Producer: appends an event to the log — durable, ordered within its partition, retained afterward
await producer.ProduceAsync("order-events", new Message<string, string>
{
    Key = orderId.ToString(),
    Value = JsonSerializer.Serialize(orderCreatedEvent)
});

// Consumer: reads through the log at its own pace, tracking its own position independently
var result = consumer.Consume();
Console.WriteLine($"Processing event at offset {result.Offset}: {result.Message.Value}");

This log-centric design is what makes Kafka the standard choice for high-volume event streaming, event sourcing, and building real-time data pipelines feeding multiple independent downstream systems — a genuinely different problem shape than the task-queue and routing-flexible messaging RabbitMQ excels at.

---

1. The Log: Kafka's Core Abstraction

An append-only, ordered, immutable sequence

Offset:   0        1        2        3        4        5
Events:  [order.1][order.2][order.3][order.4][order.5][order.6]
                                                              ↑ new events appended here

A Kafka partition (Section 2) is, at its core, an append-only log — new events are always added to the end, existing events are never modified or reordered, and each event gets a monotonically increasing offset identifying its position within that log. This is the single conceptual shift that explains most of what makes Kafka distinctive.

Consumers don't remove messages — they track their own position

Consumer A's position: offset 3 (has read events 0-2, will next read event 3)
Consumer B's position: offset 5 (has read events 0-4, will next read event 5)

Unlike a traditional queue, where consuming a message typically removes it, Kafka consumers simply track an offset — their own bookmark indicating how far through the log they've read. This means the same event can be read by many independent consumers, each progressing through the log at their own pace, and a consumer can be paused and later resumed exactly where it left off, or deliberately rewound to reprocess events it already handled (Section 10).

Why this model enables replay

Because events aren't deleted upon consumption (they're retained according to a configured policy, Section 10, independent of consumption), a new consumer added months after events were originally produced can still read the entire historical log from the beginning — a capability that has no direct equivalent in a traditional consume-and-remove queue model, and is central to Kafka's fit for event sourcing and reprocessing scenarios.

---

2. Topics and Partitions

Topics: named categories of events

kafka-topics.sh --create --topic order-events --partitions 6 --replication-factor 3 --bootstrap-server localhost:9092

A topic is Kafka's equivalent of a named event stream — conceptually similar to a queue's name in RabbitMQ, but representing an ongoing, retained stream of events rather than a transient work queue.

Partitions: how a topic scales

Topic: order-events (6 partitions)
  Partition 0: [event][event][event]...
  Partition 1: [event][event][event]...
  Partition 2: [event][event][event]...
  ...

Every topic is divided into one or more partitions — each partition is an independent, ordered log, and Kafka's horizontal scalability comes directly from spreading a topic's partitions across multiple broker machines (Section 3). This is the mechanism that lets Kafka handle throughput far beyond what a single machine's disk I/O could support: writes and reads are distributed across as many partitions (and therefore as many machines) as the topic is configured with.

Ordering is guaranteed within a partition, not across the whole topic

// Using the order ID as the partition key ensures all events for THIS order land in the same partition,
// and are therefore strictly ordered relative to each other
await producer.ProduceAsync("order-events", new Message<string, string>
{
    Key = orderId.ToString(),
    Value = eventJson
});

Kafka guarantees strict ordering within a single partition, but makes no ordering guarantee across different partitions of the same topic — this is a deliberate, important trade-off, and it's why choosing a good partition key matters enormously: using the order ID as the key (as shown above) guarantees every event related to a specific order lands in the same partition, and is therefore processed in the correct relative order, while still allowing different orders' events to be spread (and processed in parallel) across many partitions.

Choosing a partition count

More partitions mean more parallelism (more consumers can work concurrently, Section 5) but also more overhead (more file handles, more replication traffic, slower leader elections during a failure) — partition count is generally not something changed casually after a topic is in production use (repartitioning an existing topic changes which partition a given key hashes to, breaking the ordering guarantee for that key going forward), so it's worth deliberate upfront capacity planning rather than an arbitrary default.

---

3. Brokers, Replication, and Fault Tolerance

Brokers: the machines that actually store partition data

Kafka Cluster
  Broker 1: hosts partition 0 (leader), partition 1 (replica), partition 2 (replica)
  Broker 2: hosts partition 1 (leader), partition 2 (replica), partition 0 (replica)
  Broker 3: hosts partition 2 (leader), partition 0 (replica), partition 1 (replica)

A Kafka cluster is composed of multiple broker processes, each storing a subset of the cluster's partition data — for fault tolerance, each partition is replicated across multiple brokers, with one broker designated as the leader for that partition (handling all reads and writes) and the others as followers, continuously replicating the leader's data.

Replication factor and fault tolerance

--replication-factor 3

A replication factor of 3 means each partition's data exists on three different brokers — the cluster can tolerate up to two broker failures for that partition without losing data or availability (assuming a new leader is elected among the surviving replicas). This is directly analogous to the quorum queue replication concept covered in this series' RabbitMQ guide, adapted to Kafka's partition-leader architecture.

KRaft: Kafka's modern consensus mechanism

Older Kafka deployments relied on a separate ZooKeeper cluster for coordinating broker metadata and leader election; modern Kafka versions have moved to KRaft (Kafka Raft), an integrated Raft-based consensus mechanism built directly into Kafka itself — removing the operational burden of running and maintaining a separate ZooKeeper cluster alongside Kafka, a meaningful simplification for anyone deploying Kafka today.

---

4. Producers

Basic production

using var producer = new ProducerBuilder<string, string>(new ProducerConfig
{
    BootstrapServers = "localhost:9092"
}).Build();

var deliveryResult = await producer.ProduceAsync("order-events", new Message<string, string>
{
    Key = orderId.ToString(),
    Value = JsonSerializer.Serialize(orderEvent)
});

Console.WriteLine($"Delivered to partition {deliveryResult.Partition}, offset {deliveryResult.Offset}");

Acknowledgment levels (acks)

var config = new ProducerConfig
{
    BootstrapServers = "localhost:9092",
    Acks = Acks.All // wait for all in-sync replicas to acknowledge before considering the write successful
};

acks setting	Behavior	Trade-off
0	Fire and forget — no acknowledgment awaited at all	Fastest, but a broker failure can silently lose messages
1	Wait for the partition leader's acknowledgment only	Faster than all, but a leader failure before replication completes can lose the message
all (-1)	Wait for all in-sync replicas to acknowledge	Strongest durability guarantee, at the cost of higher latency

This mirrors the publisher-confirms trade-off covered in this series' RabbitMQ guide — the general principle (durability costs latency, and the right trade-off depends on how costly message loss would actually be for this specific data) applies identically here, just with Kafka's own specific configuration knob.

Partitioning strategy

// No key: round-robin/sticky partitioning — good throughput, no ordering guarantee across related events
await producer.ProduceAsync("metrics", new Message<Null, string> { Value = metricJson });

// With a key: same key always routes to the same partition, preserving relative order
await producer.ProduceAsync("order-events", new Message<string, string> { Key = orderId.ToString(), Value = eventJson });

As covered in Section 2, whether (and what) key you supply directly determines both partitioning distribution and ordering guarantees — this is one of the most consequential upfront design decisions when producing to a Kafka topic, since it affects correctness (ordering), not just performance.

Idempotent producers

var config = new ProducerConfig
{
    BootstrapServers = "localhost:9092",
    EnableIdempotence = true // prevents duplicate messages from producer-side retries
};

Enabling idempotence means the broker deduplicates messages that were retried due to a transient network issue on the producer side (a common source of accidental duplicates even before a message reaches any consumer) — a low-cost, broadly recommended setting for production producers, addressing one specific source of duplication distinct from the consumer-side idempotency concerns covered in Section 6.

---

5. Consumers and Consumer Groups

Basic consumption

using var consumer = new ConsumerBuilder<string, string>(new ConsumerConfig
{
    BootstrapServers = "localhost:9092",
    GroupId = "order-processing-service",
    AutoOffsetReset = AutoOffsetReset.Earliest
}).Build();

consumer.Subscribe("order-events");

while (!stoppingToken.IsCancellationRequested)
{
    var result = consumer.Consume(stoppingToken);
    await ProcessEventAsync(result.Message.Value);
    consumer.Commit(result); // records this offset as processed
}

Consumer groups: Kafka's mechanism for both scaling and independent consumption

Topic: order-events (6 partitions)

Consumer Group "order-processing":
  Consumer 1 → partitions 0, 1
  Consumer 2 → partitions 2, 3
  Consumer 3 → partitions 4, 5

Consumer Group "analytics" (entirely independent, reads the SAME topic from its own position):
  Consumer A → partitions 0, 1, 2
  Consumer B → partitions 3, 4, 5

A consumer group is a named set of consumers that split a topic's partitions among themselves — within a group, each partition is consumed by exactly one member at a time (giving the same competing-consumers scaling behavior covered in this series' RabbitMQ guide), while multiple different consumer groups can each independently consume the entire topic, each tracking its own separate offset position. This dual capability — scale out within a group, and support many independent readers across groups — is a core part of what makes Kafka a natural fit for feeding multiple, entirely unrelated downstream systems (an order-processing service, an analytics pipeline, and a fraud-detection system) from the same single stream of events, each processing at their own pace with no interference between them.

Rebalancing

A consumer in the group crashes or a new one joins →
  Kafka triggers a REBALANCE → partitions are redistributed among the group's remaining/new members

When group membership changes (a consumer instance crashes, scales up, or scales down), Kafka automatically redistributes partitions among the current members — this rebalancing is largely automatic, but it does briefly pause consumption for affected partitions during the transition, and poorly-tuned rebalance settings (or consumers that take too long to process a batch, triggering a perceived "stuck" consumer) are a common source of production consumer-group instability worth monitoring for.

Offset commit strategies

// Auto-commit (simpler, but risks reprocessing or skipping messages around a crash)
var config = new ConsumerConfig { EnableAutoCommit = true, AutoCommitIntervalMs = 5000 };

// Manual commit (more control, commit only after successful processing)
var config = new ConsumerConfig { EnableAutoCommit = false };
// ... after successfully processing a batch:
consumer.Commit(result);

Manual offset commits, performed only after a message (or batch) has been genuinely, successfully processed, give the strongest correctness guarantee against losing track of progress on a crash — auto-commit is simpler but can commit an offset for a message that was received but not yet actually finished processing, risking silent message loss (from that consumer's perspective) if a crash happens between the auto-commit and actual processing completion.

---

6. Delivery Semantics

At-most-once, at-least-once, and exactly-once

At-most-once:   commit the offset BEFORE processing — a crash after commit but before processing = message lost
At-least-once:  commit the offset AFTER processing — a crash after processing but before commit = message reprocessed
Exactly-once:    requires additional coordination (transactions, idempotent consumers) — genuinely achievable, but with real complexity cost

As with RabbitMQ, at-least-once (commit only after successful processing, accepting the small risk of reprocessing a message if a crash happens in the narrow window between processing and committing) is the standard, recommended default for most applications — the same idempotency discipline covered in this series' RabbitMQ and Background Services guides applies identically here: design consumers to safely handle processing the same event more than once.

Kafka transactions for genuine exactly-once semantics

producer.InitTransactions(TimeSpan.FromSeconds(10));
producer.BeginTransaction();
try
{
    await producer.ProduceAsync("downstream-events", newMessage);
    // ... commit the corresponding consumer offset as part of the same transaction
    producer.CommitTransaction();
}
catch
{
    producer.AbortTransaction();
    throw;
}

Kafka does support genuine exactly-once semantics for specific patterns — particularly "consume from topic A, process, produce to topic B" pipelines — via transactions that atomically tie together a consumed offset commit and produced messages. This is real and valuable for stream-processing pipelines (Section 9), but it's worth being precise about scope: exactly-once semantics apply within Kafka's own transactional guarantees; the moment a side effect touches something outside Kafka (writing to an external database, calling an external API), you're back to needing the same idempotency discipline as at-least-once delivery, since Kafka's transactions can't extend their atomicity guarantee to an arbitrary external system.

---

7. .NET Integration

Confluent's official .NET client

<PackageReference Include="Confluent.Kafka" Version="2.*" />

Confluent.Kafka is the standard, actively maintained .NET client, wrapping the high-performance native librdkafka library — the producer and consumer examples throughout this guide use this library directly.

Wrapping a consumer in a BackgroundService

public class OrderEventConsumer : BackgroundService
{
    private readonly IConsumer<string, string> _consumer;

    public OrderEventConsumer(IServiceScopeFactory scopeFactory)
    {
        _consumer = new ConsumerBuilder<string, string>(new ConsumerConfig
        {
            BootstrapServers = "localhost:9092",
            GroupId = "order-processing-service",
            EnableAutoCommit = false
        }).Build();
        _scopeFactory = scopeFactory;
    }

    protected override async Task ExecuteAsync(CancellationToken stoppingToken)
    {
        _consumer.Subscribe("order-events");
        while (!stoppingToken.IsCancellationRequested)
        {
            var result = _consumer.Consume(stoppingToken);
            using var scope = _scopeFactory.CreateScope();
            var handler = scope.ServiceProvider.GetRequiredService<IOrderEventHandler>();
            await handler.HandleAsync(result.Message.Value, stoppingToken);
            _consumer.Commit(result);
        }
    }
}

Exactly the same BackgroundService foundation and scoped-dependency pattern covered in this series' Background Services guide applies to Kafka consumers as it does to RabbitMQ consumers or any other long-running message processing loop — the broker-specific client library changes, but the hosting model and DI-scoping discipline stay consistent.

Producing from an ASP.NET Core API

app.MapPost("/orders", async (CreateOrderRequest request, IOrderService orderService, IProducer<string, string> producer) =>
{
    var order = await orderService.CreateAsync(request);
    await producer.ProduceAsync("order-events", new Message<string, string>
    {
        Key = order.Id.ToString(),
        Value = JsonSerializer.Serialize(new OrderCreatedEvent(order.Id, order.CustomerId, order.Total))
    });
    return Results.Created($"/orders/{order.Id}", order);
});

A common pattern connecting directly to this series' Minimal APIs and REST guides: an API handles a synchronous request/response for the immediate operation (creating the order), then publishes an event to Kafka for anything that can happen asynchronously afterward (updating a search index, notifying other services), decoupling the request's response time from that downstream work entirely.

---

8. Schema Registry and Message Contracts

The problem: producers and consumers need to agree on message shape

Producer (Service A, deployed Monday):    { "orderId": 1001, "total": 149.97 }
Producer (Service A, deployed Tuesday):    { "orderId": 1001, "totalAmount": 149.97 }  ← renamed field, breaks consumers

Unlike a synchronous API where a schema mismatch fails immediately and visibly (as covered in this series' REST guide's versioning discussion), an incompatible message schema change in an asynchronous event stream can silently break every downstream consumer, discovered only when they start throwing deserialization errors — often well after the producing service has already moved on.

Schema Registry

var schemaRegistryConfig = new SchemaRegistryConfig { Url = "http://localhost:8081" };
using var schemaRegistry = new CachedSchemaRegistryClient(schemaRegistryConfig);

using var producer = new ProducerBuilder<string, OrderCreatedEvent>(producerConfig)
    .SetValueSerializer(new AvroSerializer<OrderCreatedEvent>(schemaRegistry))
    .Build();

Confluent Schema Registry (or equivalents) centralizes and enforces schema definitions (commonly using Avro, Protobuf, or JSON Schema) for topics, with configurable compatibility rules (backward, forward, or full compatibility) that reject a schema change violating the configured policy before it can ever be published — directly extending the "additive, backward-compatible changes" principle covered throughout this series (in the REST, gRPC, and Database Migrations guides) to the event-streaming context specifically, where the consequences of a breaking, undetected schema change are arguably even more severe given the decoupled, asynchronous nature of who's actually consuming a given topic.

Why this matters more for Kafka than for many point-to-point integrations

Because a Kafka topic often has many independent, decoupled consumers (potentially owned by entirely different teams, as covered in Section 5's multi-consumer-group discussion) who the producing team may not even have full visibility into, schema governance becomes a genuinely load-bearing practice rather than an optional nicety — a producer team can't simply "coordinate directly" with every consumer the way they might for a smaller number of known point-to-point integrations.

---

9. Kafka Streams and Stream Processing

Beyond simple produce/consume: computing over the stream itself

While this guide has focused on Kafka as a message transport, its log-based model also enables genuine stream processing — computing aggregations, joins, and transformations directly over one or more streams, continuously, as new events arrive, rather than processing events one at a time in application code with no built-in notion of windowing or stateful aggregation.

// Kafka Streams (Java/Scala) — conceptual illustration, not a .NET-native API
KStream<String, OrderEvent> orders = builder.stream("order-events");
KTable<String, Long> orderCountsByCustomer = orders
    .groupBy((key, order) -> order.getCustomerId())
    .count();

Kafka Streams itself is a JVM-native library (Java/Scala) with no direct, first-party .NET equivalent — .NET teams needing stream-processing capabilities typically reach for a separate framework layered on top of Kafka, such as Apache Flink (which has broader language support including some .NET integration paths) or implement windowed aggregation logic directly in application code consuming from Kafka, accepting more manual responsibility for state management than a purpose-built stream-processing framework would provide.

When stream processing is (and isn't) the right layer

Simple "consume an event, take an action" processing (covered throughout Sections 4–7) doesn't need a stream-processing framework at all — a straightforward consumer loop is the right tool. Stream processing earns its additional complexity specifically for genuinely stateful, windowed, or multi-stream-joining computations (e.g., "compute a rolling 5-minute order count per customer, joined against a customer-tier lookup stream") that would otherwise require hand-building non-trivial state management and windowing logic from scratch.

---

10. Retention, Compaction, and Replay

Time/size-based retention: the default model

--config retention.ms=604800000   # 7 days
--config retention.bytes=10737418240  # 10 GB per partition

By default, Kafka retains events for a configured duration or size, regardless of whether they've been consumed — after that window, the oldest events are deleted to reclaim space. This is fundamentally different from a traditional queue's "delete on consumption" model, and it's precisely what enables replay: a new consumer group can read the entire retained history from the beginning, not just events produced after it started listening.

Log compaction: retaining only the latest value per key

--config cleanup.policy=compact

Before compaction: [key=A,v=1] [key=B,v=1] [key=A,v=2] [key=A,v=3] [key=B,v=2]
After compaction:                            [key=A,v=3]             [key=B,v=2]

Compacted topics retain only the most recent value for each distinct key indefinitely (rather than deleting based on age/size) — this turns a Kafka topic into something closer to a distributed, changelog-backed key-value store, commonly used for maintaining current-state snapshots (the latest known state of an entity) rather than a pure historical event log, and is the underlying mechanism behind Kafka Streams' KTable abstraction referenced in Section 9.

Replaying events for a new or recovering consumer

consumer.Assign(new TopicPartitionOffset("order-events", partition: 0, Offset.Beginning));

Because events remain in the log (subject to retention), a consumer can deliberately seek to the beginning of a partition (or any specific offset) and reprocess historical events — invaluable for onboarding a new downstream system that needs to build up its initial state from history, or recovering from a bug in a consumer by fixing the bug and simply replaying the affected time range, a recovery option a traditional consume-and-delete queue generally can't offer once messages have already been consumed and removed.

---

11. Kafka vs. RabbitMQ vs. Managed Alternatives

	Kafka	RabbitMQ	Azure Event Hubs / AWS Kinesis
Core model	Distributed, retained, replayable log	Smart broker with flexible exchange-based routing	Managed, Kafka-like or Kinesis-native streaming service
Message retention	Configurable, independent of consumption (replayable)	Typically consumed-and-removed	Configurable retention, replayable (similar to Kafka)
Routing flexibility	Lower — primarily topic/partition, key-based	Very high — direct/topic/fanout/headers exchanges	Lower, streaming-focused like Kafka
Throughput ceiling	Very high — purpose-built for massive event volume	High, but generally lower than Kafka's log-optimized design	High, managed-service-appropriate throughput
Multiple independent consumer groups reading the same stream	Native, core design feature	Possible via fanout exchange to multiple queues, less naturally "replay from history" oriented	Native (Event Hubs' consumer groups mirror Kafka's model closely)
Operational complexity	Higher — partition/replication/broker management (self-hosted)	Moderate — clustering for HA	Low — fully managed
Best fit	Event sourcing, high-volume pipelines, multiple independent downstream consumers, replay needs	Flexible routing, task distribution, moderate-to-high throughput	Teams wanting Kafka-like streaming semantics without operating Kafka themselves

Practical guidance

• Building an event-sourced system, a high-volume real-time data pipeline, or need multiple independent teams/systems to consume the same event stream, potentially replaying history? → Kafka (or a managed equivalent) is purpose-built for exactly this.
• Need flexible, content-based message routing for task distribution across services, without needing long-term event retention or replay? → RabbitMQ, as covered in this series' companion guide, is generally the simpler, better-fitting choice.
• Want Kafka's streaming model without operating Kafka's clustering/partition management yourself? → Azure Event Hubs (which even supports the Kafka protocol directly, easing migration) or AWS Kinesis provide managed equivalents.

As with the RabbitMQ guide's conclusion, this isn't a strict either/or across an entire organization — many real architectures use RabbitMQ for task-queue-style, routing-flexible internal messaging, and Kafka specifically for the high-volume, multi-consumer event streaming and event-sourcing portions of the same system.

---

12. Common Pitfalls

Pitfall	Why it hurts	Better approach
No or a poorly-chosen partition key	Loses ordering guarantees for logically related events	Key by an entity ID (order ID, user ID) that needs relative ordering preserved
Treating consumer-group rebalances as always instant/free	Brief processing pauses during rebalance can surprise an unprepared system	Monitor rebalance frequency/duration; tune session/heartbeat timeouts appropriately
Auto-committing offsets before processing genuinely completes	Silent message loss from that consumer's perspective on a crash	Manually commit only after successful processing
Assuming exactly-once extends to external side effects	Kafka transactions don't cover writes to external databases/APIs	Design consumers to be idempotent regardless of Kafka's own transactional guarantees
No schema governance across producer/consumer teams	A silent, breaking schema change discovered only via downstream failures	Use Schema Registry with enforced compatibility rules
Repartitioning an existing production topic casually	Breaks the key-to-partition mapping, silently breaking ordering for existing keys	Plan partition count carefully upfront; treat repartitioning as a deliberate, disruptive change
Reaching for Kafka by default for simple task-queue needs	Higher operational complexity than the problem actually requires	Use RabbitMQ (or a managed queue) for straightforward task distribution without replay needs

---

Quick Reference Table

Concept	Purpose
Topic	Named, retained stream of events
Partition	An ordered, independent log within a topic; the unit of parallelism
Offset	A consumer's position/bookmark within a partition's log
Broker / Replication factor	Distributed storage nodes; redundancy for fault tolerance
Consumer group	A set of consumers splitting a topic's partitions; multiple groups read independently
acks (producer)	Durability vs. latency trade-off for how many replicas must confirm a write
Idempotent producer	Deduplicates producer-side retries at the broker
Manual offset commit	Commits progress only after successful processing, for at-least-once correctness
Log compaction	Retains only the latest value per key, indefinitely
Schema Registry	Enforces compatible message schema evolution across producers/consumers
Replay	Rewinding a consumer to reprocess historical, still-retained events

---

Conclusion

Kafka's log-based model — retained, replayable, ordered-within-partition, independently consumable by many decoupled consumer groups — solves a genuinely different problem than a traditional message broker like RabbitMQ, and it's worth choosing deliberately rather than reaching for Kafka by default: it's the right tool specifically when high volume, replay, event sourcing, or many independent downstream consumers of the same stream are real requirements, and it brings meaningfully more operational complexity (partition planning, replication, schema governance) than a simpler task-queue broker would for workloads that don't actually need those specific capabilities.

The disciplines that make Kafka work well in production echo the same principles this series has emphasized for RabbitMQ and background processing generally — idempotent, at-least-once-aware consumers, deliberate partition key choices for ordering correctness, and (uniquely important for Kafka's decoupled, multi-consumer nature) rigorous schema governance to prevent a producer's change from silently breaking consumers it may not even have visibility into.

---

Found this useful? Feel free to star the repo, open an issue with corrections, or share the replay that saved a downstream system after a consumer bug was fixed.