Event Driven Architecture (EDA) with Kafka: Designing Events and Channels the Right Way

Enterprise organizations around the world are moving away from relying only on batch analytics that run hours or days after an event has happened. Instead, they are adopting architectures that can react when events occur. In areas such as fraud detection, real-time customer experience, IoT monitoring, payment systems, operational dashboards, and security analytics, Event Driven Architecture is no longer only a modern integration pattern. It has become one of the key building blocks of real-time data platforms.

The reason is simple: in many organizations, chronic batch analytics problems are not only technology problems; they are timing problems. Data is collected, moved, processed, and reported. But by the time the business receives the result, the decision may already be late. When EDA is designed correctly, it reduces this delay. It allows data to be processed as events happen, providing lower latency, faster action, and a more flexible integration model.

At this point, it is more accurate to position EDA not as a direct replacement for batch processing, but as a different architectural response that complements it. Large historical data processing, periodic reporting, financial closing, and heavy bulk transformations are still strong areas for batch processing. EDA creates the most value when data is continuously flowing, decisions must be made quickly, and systems are expected to react to events almost immediately.

However, poorly designed Event Driven Architecture can turn streaming flows and real-time analytics into a serious mess inside data management platforms. At the beginning, a decision like “let’s create Kafka topics and let systems write there” may look simple. A few months later, it may turn into uncontrolled topic growth, unclear event ownership, unreliable data distribution, consumer dependency problems, reprocessing issues, and data flows that are difficult to trace.

This blog series is built around two main questions: How should we design an event, and how should an event mature inside the platform? In this first article, we will focus on the event itself: the difference between events and commands, the Kafka topic/channel model, schema contracts, partition keys, producer-consumer relationships, and common design mistakes. In the second article, we will look at the event lifecycle inside the platform: the pipeline from raw to curated events, DLQ (Dead Letter Queue : It is the channel where failed or unprocessable events are sent) and alert topics, replay, monitoring, governance, and the relationship with the Medallion approach used in modern lakehouse architectures.

What Is Event Driven Architecture?

Event Driven Architecture is an architectural approach where systems communicate through events instead of direct and synchronous calls.

An event represents a meaningful business fact that has already happened in the system:

• A customer was created.
• A payment was completed.
• A card transaction failed.
• Stock level dropped below a critical threshold.
• A sensor temperature exceeded the limit.
• A user logged in to a mobile application.

The important point is this: an event is not a request. It is information about something that has already happened.

For example:

PaymentCompleted
CustomerCreated
OrderShipped
MachineTemperatureExceeded

These are events because they describe something that happened in the past.

On the other hand:

CreatePayment
UpdateCustomer
SendNotification

These are closer to commands. They express an intention to make a system do something. In EDA design, separating events from commands is a critical decision.

The Difference Between Traditional Integration and EDA

In traditional architectures, systems usually call each other directly.

Application A ---> Application B ---> Application C

This model looks simple at a small scale. However, as the number of systems increases, dependencies grow. If one service becomes slow, fails, or changes, the other systems in the chain may also be affected.

In EDA, systems do not directly depend on each other. Instead, they publish events, and the systems interested in those events consume them.

Application A ---> Event Channel ---> Application B
                               |----> Application C
                               |----> Application D

With this model, the producer system does not need to know who will consume the event. A new consumer can be added without changing the existing producer application.

Where Does Kafka Fit in EDA?

In EDA architectures, Kafka is usually positioned as an event backbone, event bus, or event streaming platform.

The core Kafka concepts are:

• Producer: The application that produces events.
• Topic: The logical channel where events are written.
• Partition: The unit that enables parallel processing within a topic.
• Consumer: The application that reads events.
• Consumer Group: A group of consumers that share the same work.
• Broker: A server inside the Kafka cluster.
• Offset: The position that shows how far a consumer has read in a topic.
• Retention: The policy that defines how long events stay in Kafka.

A simple Kafka-based EDA flow can be shown like this:

Source System
     |
     v
Kafka Topic / Event Channel
     |
     +--> Real-time Analytics
     +--> Notification Service
     +--> Data Lake Ingestion
     +--> Fraud Detection
     +--> Monitoring Dashboard

Here, Kafka topics act as event channels between systems.

What Does Channel-Based Kafka Design Mean?

A channel-based structure usually means separating event types or business domains through Kafka topics.

For example, for a payment system:

payment.transaction.created
payment.transaction.authorized
payment.transaction.completed
payment.transaction.failed
payment.transaction.reversed

For a customer domain:

customer.created
customer.updated
customer.segment.changed
customer.status.changed

For an IoT or manufacturing scenario:

machine.telemetry.raw
machine.telemetry.enriched
machine.alert.temperature
machine.maintenance.predicted

Each topic is an event channel. The producer application writes events to the related channel. Consumer applications read the channels they are interested in and perform their own work.

An Anonymized Field Example: Channel-Based Design for Energy Distribution Data

The following example is anonymized from a large-scale data platform project in the energy distribution domain. Without sharing the organization or project name, it aims to make the real architectural needs and EDA design decisions from the field more visible.

In this scenario, the main need was to centrally collect, securely transport, process, and analyze data coming from smart meters and lighting infrastructure across the country. At first glance, this may look like an integration project. However, when we consider the data volume, the number of source systems, the need for 24/7 operation, security expectations, and operational monitoring requirements, the problem becomes much bigger than classical integration. A real-time and always-on data flow architecture was needed.

One of the most critical decisions in the field was Kafka topic design. Different types of meter data were coming from multiple distribution companies: electricity consumption data, meter activity data, online/offline status information, and operational signals. Writing all data into one large topic could look simpler at first. But this would create serious complexity for consumers in terms of filtering, scaling, error handling, and monitoring.

For this reason, a source-based and data-type-based channel approach was preferred. Dozens of Kafka topics were designed based on distribution company and data type. This allowed each flow to be monitored separately, consumed separately, and scaled independently when needed.

This logic can be represented as follows:

raw.energy.meter.reading.<source>
raw.energy.meter.status.<source>
raw.energy.lighting.consumption.<source>
raw.energy.device.activity.<source>

In this design, Kafka was not only a messaging layer. It became the central event backbone that separated high-volume data from different sources into organized channels. As a result, real-time monitoring services, consumers writing to operational databases, archiving processes, and analytics platforms could all be fed independently from the same data flow.

The main lesson from this example is clear: in channel-based Kafka design, a growing number of topics is not a problem by itself. The real problem starts when it is not clear which domain, data type, ownership model, and consumption purpose each topic serves.

Event or Command?

One of the common mistakes in EDA design is mixing events and commands.

An event represents a business fact that has already happened:

PaymentCompleted
CustomerCreated
OrderShipped

A command represents an action that we want a system to perform:

CreatePayment
UpdateCustomer
SendNotification

Commands can also be carried over Kafka. However, this makes topics such as timeout, retry, correlation, response handling, and idempotency more complex. For this reason, in Kafka-based EDA design, it is usually healthier to start by modeling things that have already happened.

Business Event State and Data Pipeline Stage Should Not Be Mixed

Another important distinction in EDA design is the difference between business state and data processing stage.

A business event lifecycle may look like this:

PaymentInitiated -> PaymentAuthorized -> PaymentCompleted -> PaymentSettled

A data pipeline stage may look like this:

raw -> validated -> enriched -> curated

The first one describes the state changes of a business process. The second one describes the maturity of data inside the platform. Separating these two concepts is very important for correct topic design.

In the first article, we focus more on business events and channel design. In the second article, we will explain data pipeline stages such as raw, validated, enriched, and curated in more detail.

What to Consider in Topic Design

When designing EDA on Kafka, topic naming, partition strategy, and ownership are among the most critical decisions.

An example topic naming standard can be:

<domain>.<entity>.<event>

Examples:

payment.transaction.completed
customer.profile.updated
machine.temperature.exceeded

Some organizations also prefer to include the stage information in the topic name:

raw.payment.transaction.created
validated.payment.transaction.created
enriched.payment.transaction.created
curated.payment.transaction.completed

The important point is not that there is only one correct naming standard. The important point is having a consistent standard across the organization.

A good topic name should answer these questions:

• Which domain does it belong to?
• Which entity or business object does it represent?
• Which event does it carry?
• Which team owns this topic?
• Is this topic a stable contract, or is it a temporary processing topic?

Partition Key Selection

In Kafka, partition key selection affects both performance and ordering guarantees.

For example, if events related to the same customer must be processed in order, customer_id can be selected as the key.

Topic: payment.transaction
Key: customer_id

If transactions related to the same card must be processed in order, card_id may be a better choice.

Topic: card.transaction
Key: card_id

A poor key choice can overload some partitions while leaving others almost empty. This causes a hot partition problem.

When selecting a partition key, these questions should be asked:

• What level of ordering do we need?
• Which key provides a more balanced distribution?
• How will consumer parallelism scale?
• Should events related to the same business entity stay in the same partition?

Schema Management

In EDA, the event contract is very important. Even if producers and consumers do not know each other directly, they agree through the schema.

For this reason, each event type should have a clear schema management approach.

Key points to consider:

• Event schemas should be versioned.
• Backward compatibility rules should be defined.
• Required and optional fields should be clear.
• Metadata fields such as event timestamp, event_id, and source_system should be standardized.
• If a breaking change is needed, a new version or a new topic strategy should be defined.

Example metadata fields:

{
  "event_id": "evt-12345",
  "event_type": "PaymentCompleted",
  "event_version": "1.0",
  "event_time": "2026-05-13T10:15:00Z",
  "source_system": "payment-service",
  "correlation_id": "corr-98765"
}

If schema management is ignored, Kafka topics slowly stop being reliable event contracts. Then the answer to “who writes what and who reads it how?” becomes unclear.

Producer and Consumer Relationship

One of the most important advantages of EDA is loose coupling between producers and consumers.

The producer application publishes the event. It does not need to know how many consumers will read it.

payment.transaction.completed
        |
        +--> fraud-service
        +--> notification-service
        +--> data-lake-ingestion
        +--> realtime-dashboard
        +--> audit-service

This model allows new use cases to be added without changing the existing producer application. However, this flexibility can quickly turn into uncontrolled consumer dependency if topic ownership and schema contracts are not clear.

For every critical topic, the following information should be clear:

• Who owns the topic?
• Which application is the producer?
• Which schema versions are supported?
• Who is allowed to consume it?
• What is the retention policy?
• How are breaking changes managed?

Common EDA Design Mistakes

EDA projects usually fail not because Kafka is weak, but because architectural decisions are not clear enough.

Common mistakes include:

• Creating topics for every need without control.
• Mixing events and commands.
• Not defining topic ownership.
• Ignoring schema management.
• Choosing partition keys randomly.
• Defining retention without business requirements.
• Not documenting producer and consumer contracts.
• Mixing business event state with data pipeline stages.
• Thinking about monitoring, security, and governance too late.

Conclusion

Event Driven Architecture is a powerful approach for real-time data flows and analytics needs. However, the success of EDA does not start with installing a Kafka cluster. It starts with designing the right event model.

A good EDA design should clearly answer these questions:

• Which events will be produced?
• How will we separate events from commands?
• What will the topic/channel standard be?
• Who will own each topic?
• How will schemas be managed?
• How will partition keys be selected?
• How will producer and consumer contracts be protected?

In this article, we covered event modeling, Kafka topic/channel design, schema contracts, and producer-consumer relationships. In the next article, we will look at how these events mature inside the platform: raw, validated, enriched, and curated flows, DLQ and alert topics, replay strategy, monitoring, and the relationship with the Medallion approach used in modern lakehouse architectures.

Turkish version of this article is also available on my profile.