DEV Community: Dinesh Dunukedeniya

Event/Message Versioning in : The Complete Practical Guide

Dinesh Dunukedeniya — Fri, 13 Feb 2026 06:56:47 +0000

Event‑driven systems evolve. Your business changes, your data changes, and sooner or later your events must change too. The challenge is simple to describe but hard to get right: how do you evolve events without breaking every consumer in your system?

This guide walks through the practical patterns, trade‑offs, and real‑world techniques for versioning events in Kafka so your system can grow safely.

🧩 Why Event Versioning Matters

In a microservice architecture, events become long‑lived contracts. Once published, they may be consumed by:

Multiple services
Different teams
Different versions of the same service
External partners
Analytics pipelines
Machine learning models

A breaking change in an event can silently break all of them.

This is why event versioning is not a technical detail — it’s a core architectural responsibility.

The Golden Rule: Never Break Existing Consumers

Once an event schema is in production, you must assume:

Someone depends on every field
Someone depends on the event shape
Someone depends on the semantics
Someone depends on the ordering

This is why event versioning is about compatibility, not just schema changes.

Types of Compatibility

There are three main compatibility modes you need to understand.

Type	Meaning	Use Case
Forward	Old consumers can read new messages	Upgrade consumers later
Backward	New consumers can read old messages	Upgrade producers later
Full	Both directions work	Safest for large systems

Safe Changes (Non‑Breaking)

These changes are generally safe:

Adding a new optional field
Adding a new event type
Adding a new enum value (if consumers ignore unknowns)
Increasing field length
Adding metadata fields

These allow your system to evolve without breaking anyone.

Breaking Changes (Avoid These)

These changes will break consumers:

Renaming a field
Removing a field
Changing field type
Changing event meaning
Changing event order
Changing required fields

If you must do these, you need a new version.

Versioning Strategies

There are three practical ways to version events in Kafka.

Strategy 1: Version Inside the Event

Add a version field:

json { "eventType": "OrderPlaced", "version": 2, "orderId": "123", "customerId": "456", "items": [...] }

Consumers can switch logic based on version.

Pros

Single topic
Easy to evolve
Consumers choose when to upgrade

Cons

Consumers must handle multiple versions
Logic can get messy over time

Strategy 2: Version in the Topic Name

Example:

order-placed-v1
order-placed-v2

Pros

Clean separation
Consumers subscribe to the version they support
No branching logic

Cons

More topics to manage
Harder to migrate consumers
Analytics pipelines must merge topics

Use this when the new version is fundamentally different.

Strategy 3: Schema Registry Versioning

If you use Confluent Schema Registry or similar:

Each schema change creates a new version
Compatibility rules enforce safety
Consumers automatically get the right schema

Pros

Strong governance
Automatic compatibility checks
Great for large organisations

Cons

Requires Avro/Protobuf/JSON Schema
More tooling and process overhead

How to Evolve Events Safely

Step 1: Add new fields (non‑breaking)
Make them optional.

Step 2: Deploy producers with the new schema
They start sending both old + new fields.

Step 3: Wait until all consumers are upgraded
This may take days or weeks.

Step 4: Remove old fields (breaking)
Only after confirming no one depends on them.

This is the expand → migrate → contract pattern.

Handling Semantic Changes

Sometimes the shape stays the same but the meaning changes.

Example:

price used to mean “base price”, now it means “final price”.

This is a breaking change even if the schema is identical.

In this case:

Create a new event type
Or create a new version
Or create a new topic

Never silently change meaning.

Event Versioning Anti‑Patterns

Avoid these at all costs.

❌ Reusing event names for new meanings
This causes silent data corruption.

❌ Removing fields because “no one uses them”
Someone always does.

❌ Using timestamps as versioning
Unclear and unmaintainable.

❌ Forcing consumers to upgrade immediately
Breaks autonomy.

❌ Publishing different shapes of the same event without a version
Impossible to debug.

Conclusion

Event versioning is not just a technical detail — it’s a contract between teams, services, and the business. When done well, it allows your system to evolve safely for years. When done poorly, it creates silent failures, broken consumers, and painful migrations.

The key principles are simple:

Don’t break consumers
Version intentionally
Prefer additive changes
Use clear versioning strategies
Treat events as long‑lived contracts

Strong event versioning is a sign of a mature, well‑designed microservice architecture.

Boundaries: The Real Foundation of Any Modern Architecture (Microservices or Otherwise)

Dinesh Dunukedeniya — Sat, 17 Jan 2026 02:02:27 +0000

In 2026, the biggest lesson from the microservices era isn't about going small or fancy tooling, it's about boundaries.
A boundary is straightforward: it's the clear line defining what a service (or module) owns and controls and what it doesn't. Get this right, and whether you're in full microservices, a modular monolith, or something hybrid, things just work better.

When boundaries are weak or mismatched, you end up with a distributed mess that's harder to run than the monolith you started with.

What a strong boundary really owns

In practice, a well-drawn boundary covers:

A specific business capability - the stable piece of the domain the business values (think "Order Fulfillment" not "Order CRUD").
Exclusive data ownership - full control over schema, storage, and evolution.
Autonomous decisions— business rules and key logic that can change without cross team drama.
Exposed contracts — deliberate APIs or interfaces, nothing accidental.
Events published — clear notifications of important changes, so others can react loosely.

When these leak (sync calls everywhere, shared schemas, joint decisions), coupling creeps in. Suddenly your "independent" services aren't and the pain scales with your org.

What good boundaries deliver day-to-day

Independent deploys without release trains or endless coordination.
Schema or rule changes in one place without rippling everywhere.
Contained failures — one service slow or down? The core flows keep moving.
Easier onboarding — new team members grasp "this owns X" quickly.

The warning signs of weak boundaries

Changes always need multi-team sign-off.
Network calls turn simple operations into latency nightmares.
Downtime in one spot cascades.
Debugging feels like distributed tracing hell.

Practical ways to get boundaries right

Start simple: use Domain-Driven Design bounded contexts as your guide, map services (or modules) to real business domains, not entities or tech layers.

Then run the reality checks on any proposed boundary:

Can it handle its core decisions without constant sync calls to others?
Does it own its data fully including how it evolves?
Can the team change key logic or models independently?
If it's unavailable for 20–30 minutes, does the business keep running mostly fine?
Can someone new understand its purpose in under an hour?

Mostly yes? Solid. Mostly no? Rethink the cut.

Real-World Examples of Finding (and Refining) Boundaries

Finding the right boundaries isn't magic it's iterative discovery, usually starting with conversations, Event Storming sessions, or just mapping how the business actually works.
The gold standard is Domain Driven Design (DDD): identify bounded contexts (self-contained areas with their own language, rules, and models) that align to business capabilities or subdomains.

Here are a few practical examples from e-commerce (the classic domain) and beyond, showing how teams draw lines that stick or learn to redraw them.

Bad start (common anti-pattern):
"User Service", "Product Service", "Order Service", these are nouns/entities, not capabilities. They end up chatty (e.g., Order Service constantly asks Product for stock, User for details), leading to tight coupling.

Better way:
Map to what the business does end-to-end.
A typical online store might discover these strong boundaries:

Product Catalog & Discovery
Owns product info, search, recommendations, categories, images. Independent: Can change pricing models or add AI recommendations without touching orders.
Inventory Management
Real-time stock tracking,reservations, replenishment alerts. Owns stock data; publishes "stock low" or "reserved" events.
Shopping Cart & Checkout Experience Temporary cart state, promo application, checkout flow. Session-based; doesn't own permanent orders.
Order Fulfillment
Order creation, validation, status tracking, risk checks.
Payment Processing
Transactions, gateways, refunds. Highly autonomous; changes PCI rules without rippling.
Shipping & Logistics
Carrier integration, tracking, delivery estimates. Owns shipping rules and costs.

How teams find these:

Workshop with domain experts, ask "What changes together? What language do you use here?" In practice (like Microsoft's eShopOnContainers reference or real Zalando/Amazon setups), these boundaries allow independent teams and scaling (e.g., inventory spikes during sales without crashing checkout).

The takeaway

Boundaries aren't a microservices thing they're an architecture thing. Nail them, and the rest (tech, deployment, scale) falls into place much easier. Botch them, and no amount of service mesh, observability, or serverless magic fixes the underlying coupling.
Whether you're building greenfield or refactoring legacy, start here. Everything else is secondary.

Why Asynchronous Messaging Beats REST in Modern Architectures

Dinesh Dunukedeniya — Tue, 23 Sep 2025 09:25:05 +0000

In the early days of microservices, REST APIs became the default way for services to talk to each other. They’re simple, predictable, and well-understood. But as systems grow, REST starts showing cracks: latency, bottlenecks, and brittle dependencies.

That’s where asynchronous messaging with tools like Kafka, RabbitMQ, or Azure Service Bus takes the lead.

🔹 1. Loose Coupling vs Tight Coupling

REST APIs force a request–response relationship. If Service A calls Service B, A must wait until B replies.
Messaging flips this. A just sends a message and moves on. B (and even C, D, E…) can consume it when ready.

👉 This decoupling makes systems more resilient—one slow service doesn’t break the whole chain.

🔹 2. Push vs Pull: Control Matters

With REST APIs, the producer is in push mode. The caller must hit the server directly, regardless of whether that server is busy, overloaded, or temporarily down. This creates pressure and dependency on the consumer’s capacity.
With messaging, consumers operate in pull mode. They fetch and process events at their own pace. If a consumer slows down, the broker safely buffers messages until it catches up.

👉 This difference alone makes async systems naturally elastic. Services run as fast as they can—no more, no less.

🔹 3. Scalability at Its Core

REST requires every request to hit the server directly, which means scaling horizontally (load balancers, replicas).
Kafka can handle millions of events per second by design. Multiple consumers can process the same topic in parallel, enabling massive scale without choking upstream services.

👉 Async shines when load spikes—like flash sales or real-time data streams.

🔹 4. Built-in Reliability

If a REST endpoint is down, requests fail unless you build retry logic.
With messaging, events are stored durably in the broker. Even if consumers go offline, they can catch up later.

👉 That’s a huge win for mission-critical systems where losing events = losing money.

🔹 5. Event-Driven Power

REST is about asking:

“What’s your status right now?”

Messaging is about telling:

“This event happened.”

That small difference unlocks event-driven architectures, where systems react automatically to changes. Examples:

Order created → Inventory reserved → Notification sent
Payment succeeded → Loyalty points awarded → Email triggered

👉 This flow is natural, scalable, and reduces the need for constant polling.

🔹 6. Future-Proof Flexibility

With REST, if you add a new consumer, you must update the producer to call it.
With messaging, producers don’t care who listens. You can add new consumers (analytics, monitoring, fraud detection) without touching existing services.

👉 That’s how companies evolve from MVP to enterprise-scale without rewriting everything.

⚖️ The Trade-offs: Why Async Isn’t Always Better

While async messaging is powerful, it’s not a silver bullet:

Complexity: Designing an event-driven system requires careful planning (message schemas, idempotency, ordering).
Eventual Consistency: You can’t always get an immediate answer. Users may see a slight delay before data fully syncs.
Operational Overhead: Brokers like Kafka require infrastructure, monitoring, and scaling strategies. REST, by contrast, can run almost anywhere with minimal setup.

👉 For user-facing features that need instant confirmation (login, payment, search), REST is still the better choice. Async shines in the backend where scale, resilience, and decoupling matter most.

🚀 Final Thought

If you’re building for scale, resilience, and evolution, REST alone won’t cut it.
Async messaging is not just a tool it’s an architectural mindset that embraces decoupling, reliability, and growth.

The future is event-driven. And messaging makes it possible.

How to Choose the Right Microservice Boundaries

Dinesh Dunukedeniya — Sat, 09 Aug 2025 05:33:59 +0000

Choosing the size of a microservice isn’t just about code. it’s about team structure, business capabilities, and how the system will evolve over time.
Get it wrong, and you either end up with:

The Mini-Monolith → big, complex, slow to deploy, and hard for one team to own.
The Distributed Hairball → dozens of tiny services that constantly call each other, making debugging a nightmare.

Start with the Business Domain (Domain-Driven Design)

Most experienced teams begin with DDD to find bounded contexts areas where the business language, rules, and data make sense together.

Example – E-commerce:
Bounded contexts that naturally map to microservices:

Catalog → product data, categories, availability
Ordering → order creation, updates, status tracking
Payments → payment processing, refunds, compliance
Shipping → delivery tracking, carriers, schedules

Why it works:

Keeps each service focused
Prevents mixing unrelated rules
Data ownership is clear

Look at Change Patterns

If two features always change together, they probably belong in the same service.
If they change at different speeds or are handled by different teams, splitting makes sense.

Example:

Catalog changes once a month → stable
Checkout changes weekly → fast-moving
→ Keep them separate so catalog updates don’t block checkout releases.

Align to Team Boundaries (Two-Pizza Rule 🍕)

A service should be small enough for one small team (5–8 people) to fully own and deploy it without coordinating with other teams.

Example:
If both the Marketing and Payments teams constantly commit to the same codebase, that service is probably too big.

Data Ownership Is Non-Negotiable

Each microservice should own its own database.
If two services write to the same table, they’re still coupled, no matter what the code says.

Example:
Payments owns the Transactions table.
If Orders needs payment info, it calls Payments or subscribes to its events. it doesn’t write directly to that table.

Watch for Chatty Communication

If one user request calls more than 2–3 services synchronously, you may have gone too granular.

Fix:
Use event-driven architecture — publish an event like OrderPlaced and let other services (Payments, Notifications, Loyalty) react independently.

Start Bigger, Split Later

Most large companies don’t start with dozens of microservices.
They begin with a modular monolith or a few chunky services and split when needed:

Scaling issues
Release coordination pain
Regulatory/security isolation needs

Example:
A SaaS platform starts with Users + Billing + Analytics in one service.
When Billing needs PCI compliance, they split it into its service, with minimal disruption.

Summary

When deciding microservice boundaries:

Start with bounded contexts from Domain-Driven Design.
Group features by change patterns — if they change together, keep them together.
Align services to small, independent teams (Two-Pizza Rule).
Enforce data ownership — one service, one database.
Avoid excessive synchronous calls — prefer async/event-driven.
Start larger, split only when needed for scaling, compliance, or deployment independence.

The goal isn’t to have the smallest services.It’s to have well-defined, loosely coupled, independently deployable services that match your business needs and team structure.

💬 Over to you — Have you worked on a project where microservices were way too small or too big? How did it go?

Microservice-to-Microservice Authentication in .NET Using IdentityServer and JWT

Dinesh Dunukedeniya — Fri, 01 Aug 2025 23:06:23 +0000

In a microservices architecture, secure communication between services is just as important as securing external API calls. One of the most common and robust ways to handle this in .NET is by using IdentityServer, JWT tokens, and the Client Credentials flow.

In this article, you’ll learn how to:

Authenticate microservices using Client Credentials
Request and validate JWT tokens
Pass tokens between services securely
Optimize token management with caching
Apply best practices for real-world deployments.

Why Client Credentials Flow?

The Client Credentials flow is designed for machine-to-machine communication. Unlike user-based authentication (e.g., Authorization Code flow), there’s no interactive login. Instead, each service identifies itself using a ClientId and ClientSecret, and IdentityServer issues an access token that can be used to call other services.

When to use it:

Service A needs to call Service B directly.
No user context is involved.
Both services are trusted and run in a secure environment.

Trusted Services and Secure Environment

When we say "both services are trusted and run in a secure environment", it means:
Trusted Services

Both microservices are owned and controlled by the same organization (or within a trusted partner ecosystem).
Each service is registered in IdentityServer with its own ClientId and ClientSecret.
Services rely on IdentityServer as the trust broker — if a service presents a valid JWT from IdentityServer, other services trust it.

Example:
A Billing API (Service A) calls a Payment Processor API (Service B) within your company's internal infrastructure. Both are part of your platform and use IdentityServer for token issuance.

Secure Environment

Services communicate over a private and encrypted network (e.g., AWS VPC, Azure VNet, Kubernetes internal network).
All traffic is secured with TLS (HTTPS) to prevent token interception.
Secrets (e.g., ClientSecret) are stored securely in tools like Azure Key Vault or AWS Secrets Manager.
IdentityServer is either private or protected with firewalls, API gateways, and access controls.

Example:
Your services are deployed in Kubernetes and communicate over an internal service mesh (like Istio), where mTLS enforces encryption and mutual authentication.

⚠️ If your services are NOT in a trusted or secure environment (e.g., exposed to the public internet or involve third-party APIs), you should:

Validate the audience in the JWT token strictly.
Use an API Gateway with rate limiting, throttling, and authentication enforcement.
Consider mTLS in addition to JWT for an extra security layer.
Implement scopes and claims-based authorization to enforce least privilege.

Token Flow Diagram

Here’s how authentication works between two microservices:

+-----------------+           +---------------------+          +-----------------+
| Service A       |           | IdentityServer      |          | Service B       |
| (Caller)        |           | (Auth Provider)     |          | (Resource API)  |
+-----------------+           +---------------------+          +-----------------+
       |                              |                               |
       |   1. Request Token           |                               |
       |----------------------------->|                               |
       |                              |                               |
       |         2. JWT Token         |                               |
       |<-----------------------------|                               |
       |                              |                               |
       |  3. Call API with Token      |                               |
       |------------------------------------------------------------->|
       |                              |                               |
       |          4. Validate Token                                   |
       |                              |                               |
       |<-------------------------------------------------------------|

IdentityServer Setup

First, configure a client in IdentityServer for service-to-service communication:

new Client
{
    ClientId = "service-a",
    ClientSecrets = { new Secret("secret".Sha256()) },
    AllowedGrantTypes = GrantTypes.ClientCredentials,
    AllowedScopes = { "service-b-api" }
}

This allows Service A to request a token for the scope service-b-api and then use it when calling Service B.

Requesting a Token in .NET

Service A will use the IdentityModel library to request a token:

using IdentityModel.Client;
using System.Net.Http;

public class TokenService
{
    private readonly HttpClient _httpClient;

    public TokenService(HttpClient httpClient)
    {
        _httpClient = httpClient;
    }

    public async Task<string> GetTokenAsync()
    {
        var disco = await _httpClient.GetDiscoveryDocumentAsync("https://identityserver-url");
        if (disco.IsError) throw new Exception(disco.Error);

        var tokenResponse = await _httpClient.RequestClientCredentialsTokenAsync(new ClientCredentialsTokenRequest
        {
            Address = disco.TokenEndpoint,
            ClientId = "service-a",
            ClientSecret = "secret",
            Scope = "service-b-api"
        });

        if (tokenResponse.IsError) throw new Exception(tokenResponse.Error);

        return tokenResponse.AccessToken;
    }
}

Calling Another Microservice with JWT

Once we have a token, we can attach it to the Authorization header:

using IdentityModel.Client;

public class ServiceBClient
{
    private readonly HttpClient _httpClient;
    private readonly TokenService _tokenService;

    public ServiceBClient(HttpClient httpClient, TokenService tokenService)
    {
        _httpClient = httpClient;
        _tokenService = tokenService;
    }

    public async Task<string> GetDataAsync()
    {
        var token = await _tokenService.GetTokenAsync();
        _httpClient.SetBearerToken(token);

        var response = await _httpClient.GetAsync("https://service-b/api/data");
        response.EnsureSuccessStatusCode();

        return await response.Content.ReadAsStringAsync();
    }
}

Validating JWT in the Receiving Service

Service B must validate the JWT token. Add JWT Bearer authentication:

builder.Services.AddAuthentication("Bearer")
    .AddJwtBearer("Bearer", options =>
    {
        options.Authority = "https://identityserver-url";
        options.RequireHttpsMetadata = true;
        options.TokenValidationParameters = new TokenValidationParameters
        {
            ValidateAudience = false
        };
    });

builder.Services.AddAuthorization();

Apply authentication and authorization middleware:

var app = builder.Build();
app.UseAuthentication();
app.UseAuthorization();
app.MapControllers();
app.Run();

Optimizing with Token Caching

You should avoid requesting a token for every call. Instead, cache it until it expires:

public class CachedTokenService
{
    private readonly TokenService _tokenService;
    private string _cachedToken;
    private DateTime _expiresAt;

    public CachedTokenService(TokenService tokenService)
    {
        _tokenService = tokenService;
    }

    public async Task<string> GetTokenAsync()
    {
        if (!string.IsNullOrEmpty(_cachedToken) && DateTime.UtcNow < _expiresAt)
            return _cachedToken;

        var token = await _tokenService.GetTokenAsync();
        _cachedToken = token;
        _expiresAt = DateTime.UtcNow.AddMinutes(50); // assuming 1-hour expiry

        return _cachedToken;
    }
}

Automating Token Injection with DelegatingHandler

You can also create a custom DelegatingHandler to automatically add tokens:

public class TokenHandler : DelegatingHandler
{
    private readonly CachedTokenService _tokenService;

    public TokenHandler(CachedTokenService tokenService)
    {
        _tokenService = tokenService;
    }

    protected override async Task<HttpResponseMessage> SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
    {
        var token = await _tokenService.GetTokenAsync();
        request.SetBearerToken(token);
        return await base.SendAsync(request, cancellationToken);
    }
}

builder.Services.AddHttpClient<ServiceBClient>()
    .AddHttpMessageHandler<TokenHandler>();

Now, every call to Service B will automatically include a valid JWT.

Best Practices for Production

🔒 Secure Secrets: Store ClientSecret in a secret manager (e.g., Azure Key Vault or AWS Secrets Manager).

📜 Logging & Monitoring: Log token acquisition failures and monitor IdentityServer availability.

🚀 Use Delegating Handlers: Automate token injection for all service-to-service calls.

Summary

Use IdentityServer and Client Credentials flow for secure microservice-to-microservice communication.

Retrieve JWT tokens using IdentityModel.
Validate tokens in each service using JwtBearer.
Optimize with token caching and DelegatingHandler.
Follow security best practices for secret management and logging.

By implementing this approach, you ensure that your microservices communicate securely and efficiently in a production environment.

Fixing Microservice Dependencies with Self-Contained Events (aka ECST)

Dinesh Dunukedeniya — Sun, 27 Jul 2025 04:05:31 +0000

Microservices are supposed to be independent, autonomous, and scalable. But in reality, they often end up tightly coupled—relying on synchronous API calls, complex orchestration, and fragile dependencies.

In this post, we’ll explore how to break microservice dependencies using a powerful pattern known as Event-Carried State Transfer (ECST)—or as I like to call it, Self-Contained Events. It’s a simple yet impactful idea: put all the necessary data in the event.

The Problem: API-Based Microservice Dependency

Many microservice systems fall into a trap: they rely heavily on synchronous API calls between services to share data. This creates tight coupling, reduces resilience, and introduces latency.This API dependencies hurt reliability and scalability.

Scenario: Payment Service Needs Customer Info

Let’s say a customer places an order. The Order Service emits an event like:

{
  "eventType": "OrderPlaced",
  "orderId": "ORD-1234",
  "customerId": "CUST-5678",
  "items": [ ... ]
}

So far, so good. But now the Payment Service needs to:

Validate the customer’s billing status.

Access their preferred payment method.

Maybe even apply discounts based on customer type.

To do that, it has to call the Customer Service:

Payment Service → GET /customers/CUST-5678 → Customer Service

Here’s the problem:

If Customer Service is slow or down, the Payment Service fails.

You’ve created a runtime dependency between two services.

The system becomes fragile and hard to scale.

Despite using events, the services are still tightly coupled, relying on synchronous communication to function.

The Fix: ECST (Self-Contained Events)

Instead of making an API call, the Customer Service emits events when relevant data changes:

{
  "eventType": "CustomerUpdated",
  "customerId": "CUST-5678",
  "name": "Alice Smith",
  "email": "alice@example.com",
  "loyaltyTier": "gold",
  "preferredPaymentMethod": "visa"
}

The Payment Service subscribes to these events and stores only the fields it needs—either in a local database or an in-memory cache.

Now it looks like this:

Customer Service emits CustomerCreated and CustomerUpdated events.
Payment Service listens and stores a local copy of needed customer data.
When OrderPlaced is received, Payment uses local customer data—no API call required.

Benefits of This Approach

No runtime dependency on Customer Service.
Improved resilience—each service can function independently.
Faster, async processing using pre-synced local data.
Better boundaries—each service owns its domain and data shape.

Bonus Tip: Design Events with Purpose

Design your events with the consumer’s use case in mind.
Use versioned schemas to evolve safely.
Make handlers idempotent to handle retries.
Instead of emitting minimal "something happened" events:

{ "eventType": "CustomerUpdated", "customerId": "CUST-5678" }

Emit rich, meaningful events:

{
  "eventType": "CustomerUpdated",
  "customerId": "CUST-5678",
  "email": "alice@example.com",
  "loyaltyTier": "gold"
}

That’s how you enable autonomy.

Conclusion

Microservices shine when services can operate independently. But if every service depends on synchronous calls to others, you haven’t truly decoupled.

Event-Carried State Transfer (ECST) is a simple yet powerful technique to fix that.
✅ Push state with events
✅ Keep local, read-only data
✅ Break the web of runtime dependencies

Fixing Microservice Communication: From Fragile Calls to Resilient Systems

Dinesh Dunukedeniya — Sun, 22 Jun 2025 10:46:42 +0000

Microservices promise flexibility, scalability, and faster deployments. However, without proper communication strategies, they quickly become a tangled web of tightly coupled services, frequent downtime, and frustrating bugs. In this article, we’ll explore common microservice communication problems and how to fix them by adopting modern patterns and tools.

The Problem with Direct Service Calls

Imagine a typical e-commerce application with services like OrderService, PaymentService, and InventoryService. A direct HTTP call chain might look like this:
OrderService → PaymentService → InventoryService

Now, suppose InventoryService goes down. The entire chain breaks, and placing orders fails even though the issue is isolated.

Problems with direct service-to-service calls:

Tight Coupling
Each service depends on the availability, responsiveness, and correct behavior of the services it calls. If one service goes down, all services calling it may also fail or become slow.
Cascading Failures
One failure propagates across multiple services. This can bring down large parts of your architecture, even if only one component fails.
Increased Latency
Each call adds network delay, making the overall order process slow and frustrating.
Retry Storms and Thundering Herds
Simultaneous retries overload the failing service even more. This creates a feedback loop that makes recovery harder.
Harder to Scale and Deploy Independently
Synchronous dependencies force tight coordination between teams and services, limiting independent deployment.
Scaling one service may require scaling others to handle the load.
Harder to Test
Unit and integration tests require other services to be available.

How to Fix These Problems: Best Practices and Patterns

To build resilient, scalable, and maintainable distributed systems, it’s essential to follow sound architectural principles. Below are some best practices and design patterns that can help you tackle common challenges in microservices.

1. Favor Loose Coupling Through Asynchronous Messaging

One of the most effective ways to improve system resilience and scalability is by embracing loose coupling through asynchronous messaging. Direct, synchronous calls between services may seem convenient, but they can introduce tight dependencies, slow down performance, and increase the risk of cascading failures.

Instead, adopt a message-based/event-driven architecture using tools like Apache Kafka, Azure Service Bus, or RabbitMQ. These platforms allow services to communicate in a decoupled fashion by publishing and subscribing to events rather than calling each other directly.

Why It Works:

Decoupling: Services don’t need to know each other’s internal workings or availability.
Scalability: Consumers can scale independently based on demand.
Resilience: Fail ures in one service won’t bring down the entire system.

Example Pattern in Action:
Let’s say a customer places an order. Instead of the OrderService invoking other services directly, it publishes an OrderPlaced event to Kafka:

public class KafkaPublisher
{
    private readonly IProducer<Null, string> _producer;

    public KafkaPublisher(string bootstrapServers)
    {
        var config = new ProducerConfig { BootstrapServers = bootstrapServers };
        _producer = new ProducerBuilder<Null, string>(config).Build();
    }

    public async Task PublishOrderPlacedAsync(OrderPlacedEvent orderPlaced)
    {
        var message = JsonSerializer.Serialize(orderPlaced);
        await _producer.ProduceAsync("order-events", new Message<Null, string> { Value = message });
    }
}

Usage in OrderService:

await kafkaPublisher.PublishOrderPlacedAsync(new OrderPlacedEvent {
    OrderId = "123",
    TotalAmount = 99.99
});

From there:

PaymentService listens for OrderPlaced events and initiates payment processing.

public class KafkaConsumerService
{
    private readonly IConsumer<Ignore, string> _consumer;

    public KafkaConsumerService(string bootstrapServers, string topic, string groupId)
    {
        var config = new ConsumerConfig
        {
            BootstrapServers = bootstrapServers,
            GroupId = groupId,
            AutoOffsetReset = AutoOffsetReset.Earliest
        };

        _consumer = new ConsumerBuilder<Ignore, string>(config).Build();
        _consumer.Subscribe(topic);
    }

    public void StartConsuming(CancellationToken cancellationToken)
    {
        Task.Run(() =>
        {
            while (!cancellationToken.IsCancellationRequested)
            {
                try
                {
                    var consumeResult = _consumer.Consume(cancellationToken);
                    var orderPlaced = JsonSerializer.Deserialize<OrderPlacedEvent>(consumeResult.Message.Value);

                    if (orderPlaced != null)
                    {
                        ProcessPayment(orderPlaced);
                    }
                }
                catch (ConsumeException ex)
                {
                    // Handle consume exception (e.g. log it)
                }
            }
        }, cancellationToken);
    }

    private void ProcessPayment(OrderPlacedEvent orderPlaced)
    {
        // Simulate payment processing logic
        Console.WriteLine($"Processing payment for Order ID: {orderPlaced.OrderId}, Amount: {orderPlaced.TotalAmount}");
        // Here you could interact with a payment gateway or record payment details
    }
}

Similarly, InventoryService updates stock levels accordingly, and EmailService sends out an order confirmation email.

Each service acts independently, processing the message in its own time and retrying as needed. This pattern not only improves modularity but also creates a more robust and responsive system.

To fully harness the power of this architecture, it’s important to use the right message types for the right purpose. Choosing the correct communication pattern improves clarity, reduces unnecessary dependencies, and ensures services are loosely coupled by design.

Different scenarios call for different message styles:

Event Notifications: Inform that something has occurred, such as OrderPlaced. These are fire-and-forget and don't require a response.
Event-Carried State Transfer: Instead of triggering additional service calls, include relevant data in the event itself. For example, OrderPlaced might contain the full order details so downstream services don’t need to query OrderService.
Command Messages: Explicitly request another service to act (used carefully to avoid tight coupling).

Read my previous article to understand event driven architecture

2. Add Resilience: Timeouts, Retries, and Circuit Breakers

Even in a world leaning heavily on asynchronous messaging, sometimes synchronous calls are inevitable, particularly when dealing with legacy systems, external APIs, or tight coordination between services. When that’s the case, building resilience directly into your service communication can prevent temporary issues from turning into full-blown outages.

Timeouts
Set an upper limit on how long a service should wait for a response. This ensures your application doesn’t hang indefinitely while waiting for a slow or unresponsive dependency.

Retries with Exponential Backoff
Transient faults (like network blips or momentary service hiccups) are often self-healing. With retry policies, you can attempt the request again, waiting slightly longer each time to avoid compounding the issue.

Circuit Breakers
If a downstream service keeps failing, stop hammering it. A circuit breaker temporarily halts requests, giving the failing service time to recover and your system time to avoid a full meltdown.

Putting It All Together with Polly and .Net,

var retryPolicy = Policy
    .Handle<HttpRequestException>()
    .WaitAndRetryAsync(
        retryCount: 3,
        sleepDurationProvider: attempt => TimeSpan.FromSeconds(Math.Pow(2, attempt)),
        onRetry: (exception, timeSpan, retryCount, context) =>
        {
            Console.WriteLine($"Retry {retryCount} after {timeSpan.TotalSeconds}s due to: {exception.Message}");
        });

var circuitBreakerPolicy = Policy
    .Handle<HttpRequestException>()
    .CircuitBreakerAsync(
        exceptionsAllowedBeforeBreaking: 2,
        durationOfBreak: TimeSpan.FromSeconds(30),
        onBreak: (ex, breakDelay) =>
        {
            Console.WriteLine($"Circuit broken! Delay: {breakDelay.TotalSeconds}s");
        },
        onReset: () => Console.WriteLine("Circuit closed, operations resumed."),
        onHalfOpen: () => Console.WriteLine("Circuit half-open, testing service.")
    );

// Wrap policies
var policyWrap = Policy.WrapAsync(retryPolicy, circuitBreakerPolicy);

// Execute HTTP request with resilience
var response = await policyWrap.ExecuteAsync(() =>
    httpClient.GetAsync("https://inventory-service/api/check-stock"));

Benefits:

Improved fault tolerance: Graceful recovery from temporary service hiccups.
Stability under stress: Prevents cascading failures when a downstream service is unhealthy.
Better user experience: Avoids unnecessary timeouts and improves perceived system responsiveness.

3. Implement Graceful Fallbacks

Even with retries and circuit breakers, sometimes your dependencies simply won’t respond. In those cases, your system still needs to behave gracefully, not crash or hang indefinitely. This is where fallback strategies come in.

Fallbacks provide alternative logic or responses when primary dependencies fail, ensuring the user experience remains smooth and your service stays operational, even if at reduced functionality.

Use Case: Inventory Service Fallback
Let’s say your application relies on an external inventory service to check stock availability. If that service goes down, rather than propagating an error or timing out, you could route the request to a backup source or return a cached/default response.

public interface IInventoryService
{
    Task<string> CheckStockAsync(string productId);
}

public class PrimaryInventoryService : IInventoryService
{
    public async Task<string> CheckStockAsync(string productId)
    {
        // Simulate failure
        throw new HttpRequestException("Primary service unavailable");
    }
}

public class BackupInventoryService : IInventoryService
{
    public async Task<string> CheckStockAsync(string productId)
    {
        return await Task.FromResult("Stock from backup service: 10 units");
    }
}

Now let’s set up a fallback policy using Polly:

var backupService = new BackupInventoryService();
var primaryService = new PrimaryInventoryService();

var fallbackPolicy = Policy<string>
    .Handle<HttpRequestException>()
    .FallbackAsync(
        fallbackAction: async cancellationToken =>
        {
            // Call the backup service when the primary fails
            Console.WriteLine("Primary service failed. Using backup...");
            return await backupService.CheckStockAsync("P123");
        });

var result = await fallbackPolicy.ExecuteAsync(async () =>
{
    return await primaryService.CheckStockAsync("P123");
});

Console.WriteLine(result);

Benefits:

User continuity: Prevents error messages or app crashes.
Resilience: Keeps the service running, even if at reduced precision or reliability.
Flexibility: Allows for dynamic routing, caching, or degradation strategies when under pressure.

4. Improve Observability

Modern distributed systems are often sprawling, complex, and asynchronous, which makes understanding what's happening inside them a challenge. To avoid flying blind, it’s crucial to bake in observability from the very beginning.

Observability isn't just about logging errors; it’s about gaining real-time, actionable insight into system behavior and performance. By using tools like OpenTelemetry, Jaeger, or Zipkin, you can trace how requests flow through your services, identify bottlenecks, and debug issues with surgical precision.

Best Practices for Event Tracing and Monitoring

Tag Each Event with Correlation IDs: Generate and propagate a unique identifier with each request or event. This allows you to trace a single transaction across services and threads, even across messaging systems like Kafka or Azure Service Bus.
Trace Message Consumption and Processing Times Measure the duration and path of every event, from the moment it's published to when it's processed. This helps pinpoint slow consumers, misconfigurations, or performance regressions.
Monitor for Failing or Degraded Services: Set up alerts for failure rates, message lag, queue depth, and other signals that indicate unhealthy behavior in your system.

Benefits:

Faster troubleshooting: Identify root causes without guesswork.
Performance optimization: Locate bottlenecks and slow code paths.
Operational confidence: Know when things go wrong—and why.

Conclusion

Designing resilient distributed systems isn’t just about writing more code. It’s about writing smarter code. By favoring loose coupling through asynchronous messaging, implementing robust fault-handling mechanisms, providing graceful fallbacks, and enhancing observability, you lay the groundwork for systems that don’t just function under ideal conditions; they thrive under pressure.

The real world is messy. Services fail, networks lag, and dependencies go dark. But with the patterns and tools we've explored from Kafka and Polly to OpenTelemetry, you’ll be equipped to handle it all with confidence and control.

Start with small improvements, measure their impact, and keep refining. Resilience is a journey, not a destination, and every thoughtful design choice gets you closer.

Designing E-Commerce Order Processing: Orchestration vs Choreography

Dinesh Dunukedeniya — Wed, 11 Jun 2025 11:00:32 +0000

In the world of microservices and event-driven architecture, orchestrating business workflows is a common challenge. Two popular patterns for managing these workflows are Orchestration and Choreography, each with unique strengths and trade-offs.

In this post, we’ll design a classic use case E-Commerce Order Processing using both patterns, helping you understand how they differ in real world scenarios.

Use Case: E-Commerce Order Processing
The system must handle customer orders by:

Receiving the order
Processing payment
Updating inventory
Scheduling shipping

The flow must ensure consistency; either the entire process is completed, or appropriate compensations occur.

Let's examine the states involved in this process.

Orchestration: The Centralized Conductor

How it works:
Each microservice publishes events to Kafka. A central orchestrator subscribes to these events, determines the next step in the workflow, and then publishes a new event to Kafka to trigger the appropriate microservice to act.

The diagram below illustrates this flow with a typical e-commerce order processing example:

OrderService publishes an OrderCreated event to Kafka after a customer places an order.
OrderOrchestrator consumes the OrderCreated event and publishes a ProcessPayment event.
PaymentService subscribes to ProcessPayment, processes the payment, and publishes either:
- PaymentCompleted if successful, or
- PaymentFailed if the transaction fails.
On PaymentCompleted, the orchestrator triggers the next step by publishing ReserveStock.
InventoryService acts on ReserveStock and replies with:
- StockReserved if inventory is available, or
- OutOfStock if items are unavailable.
On successful stock reservation, the orchestrator publishes ScheduleDelivery.
ShippingService then schedules delivery and publishes DeliveryScheduled.
Finally, OrderService updates the order status to success or failure based on the outcome of each step.

Event-Driven Choreography: The Decentralized Dance

How it works:
Each microservice publishes events to Kafka and subscribes to relevant events from other services. Based on the event it receives, each service independently determines its next action and publishes a new event to Kafka, allowing the workflow to progress without a central orchestrator.

OrderService handles the customer request and publishes an OrderPlaced event to Kafka.

The diagram below illustrates this flow with a typical e-commerce order processing example:

Customer places an order via OrderService.
OrderService processes the request and publishes an OrderPlaced event to Kafka.
PaymentService subscribes to OrderPlaced, triggers the payment processing logic, and then publishes PaymentProcessed.
InventoryService subscribes to PaymentProcessed, updates the inventory, and publishes InventoryUpdated.
ShippingService subscribes to InventoryUpdated, schedules the delivery, and then publishes ShippingScheduled.
OrderService subscribes to ShippingScheduled, updates the order status, and sends the Order Confirmation to the Customer.

Orchestration vs Event Collaboration for E-commerce Applications: Which One Fits Your Needs?

E-commerce platforms must handle huge transaction volumes reliably and efficiently. Coordinating multiple microservices in such systems requires careful choice between orchestration and event collaboration (choreography).

Let’s compare both approaches in the context of e-commerce:

Orchestration: Centralized Control for Complex Workflows

When to choose:

Your order processing involves multiple dependent steps: payment, inventory reservation, shipping, notifications.
You require strict control over workflow execution order.
Auditing, retry logic, and failure recovery are critical (e.g., ensuring payment succeeds before shipping).
You want clear visibility into the entire order lifecycle.

Advantages for e-commerce:

Simplifies handling complex business rules and transactional workflows.
Easier to implement consistent retry, rollback, and compensation logic.
Centralized monitoring and logging for compliance and troubleshooting.

Challenges:

The orchestrator can become a performance bottleneck under very high load if not well designed.
Introduces tighter coupling between services and orchestrator.
Requires orchestrator to be scalable and highly available to avoid single points of failure.

Event Collaboration (Choreography): Loose Coupling for Scalability

When to choose:

Your microservices operate more independently (e.g., inventory updates, user notifications, analytics).
You can tolerate eventual consistency between services (e.g., shipping service reacts to payment success event asynchronously).
Scalability and fault tolerance are top priorities.
You want to reduce tight coupling between services.

Advantages for e-commerce:

Highly scalable as each service consumes and produces events independently.
Failure in one service doesn’t block the entire system.
Easier to extend system with new event consumers without changing existing services.

Challenges:

Complex to track end-to-end workflows and troubleshoot failures.
Potential for event storms or duplicate events that need careful handling.
Requires robust event monitoring, tracing, and alerting infrastructure.

Summary Table

Aspect	Orchestration	Event Collaboration (Choreography)
Control	Centralized, strict workflow control	Decentralized, loose coupling
Consistency	Strong consistency, sequential steps	Eventual consistency
Scalability	Can be bottleneck if not designed well	Highly scalable, services scale independently
Fault tolerance	Orchestrator failure impacts flow	Failure isolated to individual services
Monitoring & Debugging	Easier due to centralized control	Complex, needs strong tracing and monitoring

Final Thoughts

If your workflows involve multiple dependent steps like payment authorization, inventory reservation, and shipping scheduling — requiring strict control, auditing, and reliable retries — then orchestration is usually the better approach. For example, a central orchestrator managing events like OrderCreated, ProcessPayment, and ReserveStock ensures the order flow proceeds reliably. Just ensure your orchestrator is designed to be highly scalable and resilient to handle heavy transaction loads without becoming a bottleneck.

If your system benefits from loose coupling and eventual consistency—such as independent services reacting asynchronously to events like OrderPlaced, PaymentProcessed, and InventoryUpdated—and you want better scalability, then Event Collaboration (Choreography) is preferable. Be prepared to invest in strong event monitoring and tracing to handle the complexity of decentralized workflows.

Often, a hybrid approach combining orchestration for critical, multi-step processes and choreography for asynchronous, scalable interactions works best to balance reliability and scalability in e-commerce systems.

Orchestration vs. Choreography in EDA: Choosing the Right Approach

Dinesh Dunukedeniya — Mon, 02 Jun 2025 11:06:08 +0000

In my previous posts, I explored the fundamentals of orchestration and choreography within Event-Driven Architecture (EDA). These two patterns offer distinct strategies for coordinating microservice interactions.

This post cuts through the definitions and dives straight into comparing the two, with clear use cases and a side-by-side table to help you decide which pattern fits your needs best.

🕺 Choreography: The Decentralized Dance

Also known as event collaboration or broker-based topology, choreography is all about microservices reacting independently to events — no central brain calling the shots.

✅ Benefits:

Highly scalable — no central bottlenecks
Loosely coupled — services are independent
Truly event-driven — ideal for real-time systems

🔧 When to Use Choreography

Use it when you want services to respond autonomously to events, especially in reactive, scalable, and loosely coupled systems.

💡 Common Use Cases

🛒 E-Commerce Order Processing

OrderPlaced → triggers PaymentService
PaymentCompleted → triggers InventoryService
StockUpdated → triggers ShippingService

Each service listens for and reacts to events — no central coordination.

🔍 Fraud Detection

A TransactionInitiated event may trigger multiple fraud checks, logging systems, and alerting processes — all in parallel.

📡 IoT Pipelines

Sensor emits data → multiple services handle analytics, alerting, and storage asynchronously.

🎼 Orchestration: The Centralized Conductor

With orchestration, a central service — often called a Saga Orchestrator — directs the process flow step-by-step.

✅ Benefits:

Easier to manage complex workflows
Centralized error handling and rollback logic
Better observability and debugging

🔧 When to Use Orchestration

Best suited for transaction-heavy, linear, or strictly ordered workflows where the sequence of operations matters.

💡 Common Use Cases

✈️ Travel Booking System

Book flights → hotels → car rentals. If hotel booking fails, rollback flight reservation.

🏦 Loan Approval

CreditCheck → FraudCheck → FinalApproval — in order.

📦 Order Fulfillment

Centralized process: Payment → Inventory → Shipping.

🧠 Side-by-Side Comparison

⚖️ Aspect	🕺 Choreography (Event Collaboration)	🎼 Orchestration (Centralized Control)
Architecture Style	Decentralized — services own logic individually	Centralized controller sequences logic
Scalability	High — services scale independently	Limited — orchestrator can become a bottleneck
Resilience	High — one service failure doesn’t impact others	Lower — unless orchestrator is highly available
Workflow Complexity	Complex — hard to trace entire flow	Simple — easy to follow, easier to reason about
Deployment	Independent — loosely coupled	Tightly coupled — or even monolithic
Monitoring	Requires external tracking/observability tools	Easier — one place to track everything
Latency	Potentially higher due to async communication	Typically lower latency
Flexibility	Very flexible — services evolve independently	Less flexible — orchestrator updates needed for changes
Best Fit For	Long-running, complex, scalable workflows	Linear, short, transactional workflows

🧩 Final Thoughts

There's no one-size-fits-all. Choosing between choreography and orchestration depends on your system's goals:

Need scalability, loose coupling, and parallel event processing? Go with choreography.
Need order, simplicity, and centralized control? Choose orchestration.

Design smart — choose based on use case, team expertise, and operational requirements.

✍️ Have thoughts or examples from your own architecture? Drop a comment and let’s discuss!

Event Collaboration in Event-Driven Architecture (EDA)

Dinesh Dunukedeniya — Tue, 27 May 2025 12:48:12 +0000

What is Event Collaboration in EDA?

In simple terms, Event Collaboration/ Choreography is a decentralized way for services to communicate through events without a central controller. Think of it like a dance where each participant knows their steps and reacts based on cues from others there’s no single conductor telling everyone what to do.

In the context of EDA:

Service emit events upon state changes, enabling others to listen and react accordingly.
Each service listens for relevant events and reacts accordingly.
There is no central orchestrator managing the process.
Each microservice typically maintains its own state based on events it receives.

This pattern is sometimes also called Broker Topology, a term popularized by Mark Richards in Software Architecture Patterns (O’Reilly).
The name highlights that services communicate indirectly through a message broker, which routes events but does not control the workflow.

How Does Event Collaboration Work?

Imagine an online store scenario:

Order Service emits an OrderPlaced event.

Inventory Service listens to OrderPlaced, checks stock, and emits StockReserved or StockOut.
Payment Service listens to StockReserved, processes payment, and emits PaymentCompleted.
Shipping Service listens to PaymentCompleted and arranges delivery.

Each service acts independently, reacting to events and emitting new events for others. The flow emerges naturally from these interactions.

When to Use Event Collaboration in Event-Driven Architecture

Event coloboration is a powerful architectural pattern, but it’s not the perfect fit for every scenario. Consider using choreography when:

✅ You Want Loose Coupling Between Services
Each service operates independently, only knowing about events, not about the other services. This reduces dependencies and allows teams to work autonomously.

✅ Your System Is Highly Distributed
In microservices or cloud-native architectures where services are deployed separately, choreography supports scalability and flexibility.

✅ You Need Asynchronous, Event-Driven Workflows
If your business processes naturally fit event streams (e.g., order processing, payment events), choreography enables responsive and resilient flow

✅ You Expect to Scale or Evolve Services Independently
Since services produce and consume events without tight integration, you can add or update services with minimal impact on others.

✅ You Can Invest in Observability and Monitoring
Because choreography lacks a central controller, you need good tooling for tracing, logging, and monitoring event flows across services.

Avoid

⚠️ If you require strict, centralized control over business workflows, orchestration might be a better fit.

⚠️ When transactional consistency is critical across services and must be tightly managed.

⚠️ If your team is new to distributed event-driven systems and you want simpler debugging and error handling initially.

Benefits of Event Collaboration

Loose Coupling: Services only know about event formats, not about each other directly. This simplifies changes and scaling.
Scalability: Services can be developed, deployed, and scaled independently.
Flexibility: New services can join by simply subscribing to events.
Resilience: Failure in one service doesn’t block the whole process.

Challenges to Consider

Visibility & Monitoring: Without a central controller, tracking the overall process flow requires dedicated observability tools.
Complexity in Debugging: Understanding event chains can be tricky when many services interact asynchronously.
Consistency: Ensuring data consistency across services can be more challenging without orchestration.

Implementing the Choreography Pattern in .NET Microservices with Kafka

In a choreography-based event-driven system, each microservice reacts to events and, if needed, emits its own events. This allows services to be highly autonomous and loosely coupled.

Since all microservices are ASP.NET Core web apps, Kafka event consumers are implemented using BackgroundService (hosted services) within each app.

Architecture Overview

Microservices Involved:

Order Service
Initiates the workflow by publishing OrderPlaced events.
Inventory Service
Listens for OrderPlaced events, checks stock, and publishes either InventoryReserved or InventoryReservationFailed.
Payment Service
Listens for InventoryReserved and processes payment, publishing PaymentProcessed or PaymentFailed.
Shipping Service
Listens for PaymentProcessed events and handles shipping.

Each Microservice Includes:

Kafka Consumer Background Service
Subscribes to relevant Kafka topics and handles incoming events.
Kafka Producer Service
Publishes events to Kafka topics to trigger downstream service actions.
Controller/Endpoints (Optional)
For internal testing, admin access, or debug operations.
Domain Models (Shared)
All services reference a shared contracts library to ensure consistent event models.

public class OrderPlaced
{
    public string OrderId { get; set; }
    public string ProductId { get; set; }
    public int Quantity { get; set; }
    public DateTime OrderDate { get; set; }
}

public class InventoryReserved
{
    public string OrderId { get; set; }
}

public class InventoryReservationFailed
{
    public string OrderId { get; set; }
    public string Reason { get; set; }
}

public class PaymentProcessed
{
    public string OrderId { get; set; }
}

public class PaymentFailed
{
    public string OrderId { get; set; }
    public string Reason { get; set; }
}

Kafka Producer Service Each service needs a way to publish events to Kafka. This reusable service wraps the Kafka producer logic.

// KafkaProducerService.cs
public interface IKafkaProducerService
{
    Task ProduceAsync<T>(string topic, T message);
}

public class KafkaProducerService : IKafkaProducerService
{
    private readonly IProducer<Null, string> _producer;

    public KafkaProducerService(IConfiguration configuration)
    {
        var config = new ProducerConfig
        {
            BootstrapServers = configuration["Kafka:BootstrapServers"]
        };
        _producer = new ProducerBuilder<Null, string>(config).Build();
    }

    public async Task ProduceAsync<T>(string topic, T message)
    {
        var json = JsonSerializer.Serialize(message);
        await _producer.ProduceAsync(topic, new Message<Null, string> { Value = json });
    }
}

Kafka Consumer BackgroundService Each service implements a BackgroundService that listens for specific events.

Example for InventoryService consuming OrderPlaced:

// OrderPlacedConsumer.cs
public class OrderPlacedConsumer : BackgroundService
{
    private readonly IConfiguration _configuration;
    private readonly IKafkaProducerService _producer;

    public OrderPlacedConsumer(IConfiguration configuration, IKafkaProducerService producer)
    {
        _configuration = configuration;
        _producer = producer;
    }

    protected override async Task ExecuteAsync(CancellationToken stoppingToken)
    {
        var config = new ConsumerConfig
        {
            BootstrapServers = _configuration["Kafka:BootstrapServers"],
            GroupId = "inventory-service",
            AutoOffsetReset = AutoOffsetReset.Earliest
        };

        using var consumer = new ConsumerBuilder<Ignore, string>(config).Build();
        consumer.Subscribe("order-placed");

        while (!stoppingToken.IsCancellationRequested)
        {
            var result = consumer.Consume(stoppingToken);
            var order = JsonSerializer.Deserialize<OrderPlaced>(result.Message.Value);

            // Simulated logic
            var isAvailable = true; // simulate stock check

            if (isAvailable)
            {
                await _producer.ProduceAsync("inventory-reserved", new InventoryReserved { OrderId = order.OrderId });
            }
            else
            {
                await _producer.ProduceAsync("inventory-failed", new InventoryReservationFailed
                {
                    OrderId = order.OrderId,
                    Reason = "Out of stock"
                });
            }
        }
    }
}

Payment Service Consumer

public class InventoryReservedConsumer : BackgroundService
{
    private readonly IKafkaProducerService _producer;
    private readonly IConfiguration _configuration;

    public InventoryReservedConsumer(IKafkaProducerService producer, IConfiguration configuration)
    {
        _producer = producer;
        _configuration = configuration;
    }

    protected override async Task ExecuteAsync(CancellationToken stoppingToken)
    {
        var config = new ConsumerConfig
        {
            BootstrapServers = _configuration["Kafka:BootstrapServers"],
            GroupId = "payment-service",
            AutoOffsetReset = AutoOffsetReset.Earliest
        };

        using var consumer = new ConsumerBuilder<Ignore, string>(config).Build();
        consumer.Subscribe("inventory-reserved");

        while (!stoppingToken.IsCancellationRequested)
        {
            var result = consumer.Consume(stoppingToken);
            var evt = JsonSerializer.Deserialize<InventoryReserved>(result.Message.Value);

            // Simulated payment logic
            var success = true;

            if (success)
            {
                await _producer.ProduceAsync("payment-processed", new PaymentProcessed { OrderId = evt.OrderId });
            }
            else
            {
                await _producer.ProduceAsync("payment-failed", new PaymentFailed
                {
                    OrderId = evt.OrderId,
                    Reason = "Card declined"
                });
            }
        }
    }
}

Conclusion

Event Collaboration in EDA promotes a scalable, flexible, and resilient architecture by allowing services to communicate asynchronously and independently through events. While it introduces complexity in monitoring and debugging, with the right tools and practices, choreography can be a powerful pattern for modern distributed systems.

Mediator Pattern in Event-Driven Architecture (EDA)

Dinesh Dunukedeniya — Fri, 23 May 2025 12:00:24 +0000

In Event-Driven Architecture (EDA), microservices communicate by publishing and subscribing to events, enabling loose coupling and asynchronous processing. However, as systems grow in complexity, managing interactions across numerous services can become challenging.

The Mediator Pattern introduces an intermediary component — the mediator — that manages and orchestrates this communication. Rather than having services directly consume and respond to each other’s events, a dedicated mediator service listens to incoming events, applies business rules, and then routes or transforms messages as needed before passing them on to downstream services.

This pattern is sometimes referred to as the orchestration pattern or a central workflow engine, where a single component controls and coordinates the interactions between services.

This pattern is particularly useful in complex business workflows involving multiple microservices. By centralizing coordination logic within the mediator, you gain better control over the execution flow, simplify service responsibilities, reduce inter-service dependencies, and improve overall system maintainability.

When to Use the Mediator Pattern

✅ Complex workflows requiring sequencing and coordination.Especially useful in implementing the Saga Pattern or multi-step business processes involving multiple services

✅ Centralized logging, monitoring, and tracing. Gain visibility into the entire workflow execution from a single place, useful for debugging and observability.

✅ Reducing point-to-point event subscriptions between services
Avoid a tangled web of services subscribing to each other’s events, which becomes hard to manage and evolve.

✅ Business rules that span multiple services. Keep cross-cutting concerns (e.g., validation, routing decisions) in one place rather than duplicating them.

✅ Dynamic or conditional logic between steps. When the path of execution depends on business logic or the results of previous steps (e.g., retry logic, fallback strategies).

✅ Orchestrated error handling and compensations.Centralize retry strategies, failure handling, and compensating actions in distributed transactions.

✅ Auditing and compliance. A mediator can capture and persist event flows required for regulatory purposes.

Avoid

⚠️ Avoid if your system has simple, independent events.

⚠️ High-throughput, low-latency systems. Mediators can introduce bottlenecks or delays if not designed with performance in mind.

⚠️ Use cases better suited to choreography. For loosely coupled systems where services can autonomously react to events, a pure event-driven approach with no central orchestrator may be more appropriate.

Implementing the Mediator Pattern in .NET Microservices with Kafka

Let’s walk through how to implement the Mediator Pattern using Kafka in a .NET-based microservices architecture. In our setup, all microservices are web apps, meaning that to consume Kafka events, we need to use background services (hosted services) within each app to listen and process messages.

Architecture Overview

Microservices Involved:

Client Web App
Initiates the workflow by publishing OrderPlaced events.
Mediator Service
Acts as the central orchestrator. It reacts to events, applies business rules, and triggers the next step by publishing new events.
Inventory Service
Listens for ReserveInventory events and responds with either InventoryReserved or InventoryReservationFailed.
Payment Service
Listens for ProcessPayment events and responds with PaymentProcessed or PaymentFailed.
Shipping Service
Listens for OrderConfirmed events and proceeds with shipping logistics.

Design Principles

Decoupling: Services do not communicate directly; all communication flows through Kafka topics.
Orchestration: Only the Mediator knows the full business workflow. Each other service handles only its own concern.
Background Tasks: Since all services are ASP.NET Core web apps, Kafka consumers are implemented using BackgroundService.

Each Microservice Includes:

Kafka Consumer Background Task
Subscribes to relevant topics and handles incoming events.
Kafka Producer Service
Sends events to Kafka topics to trigger actions in other services.
Controller/Endpoints (if needed)
Mostly for internal APIs or admin/debug access. Not used for orchestration logic.

1. Domain Models

public record OrderPlaced(Guid OrderId, List<string> Items, PaymentInfo Payment);
public record ReserveInventory(Guid OrderId, List<string> Items);
public record InventoryReserved(Guid OrderId);
public record InventoryReservationFailed(Guid OrderId, string Reason);
public record ProcessPayment(Guid OrderId, PaymentInfo Payment);
public record PaymentProcessed(Guid OrderId);
public record PaymentFailed(Guid OrderId, string Reason);
public record OrderConfirmed(Guid OrderId);

public record PaymentInfo(string CardNumber, decimal Amount);

2. Kafka Producer Helper

public class KafkaProducer
{
    private readonly IProducer<string, string> _producer;

    public KafkaProducer(string bootstrapServers)
    {
        var config = new ProducerConfig { BootstrapServers = bootstrapServers };
        _producer = new ProducerBuilder<string, string>(config).Build();
    }

    public async Task ProduceAsync<T>(string topic, T message)
    {
        var json = JsonConvert.SerializeObject(message);
        await _producer.ProduceAsync(topic, new Message<string, string> { Key = Guid.NewGuid().ToString(), Value = json });
    }
}

3. Mediator Service Background Consumers

OrderPlacedConsumer.cs

public class OrderPlacedConsumer : BackgroundService
{
    private readonly IConsumer<string, string> _consumer;
    private readonly KafkaProducer _producer;
    private readonly ILogger<OrderPlacedConsumer> _logger;

    public OrderPlacedConsumer(KafkaProducer producer, ILogger<OrderPlacedConsumer> logger)
    {
        _producer = producer;
        _logger = logger;

        var config = new ConsumerConfig
        {
            BootstrapServers = "localhost:9092",
            GroupId = "mediator-group",
            AutoOffsetReset = AutoOffsetReset.Earliest
        };
        _consumer = new ConsumerBuilder<string, string>(config).Build();
        _consumer.Subscribe("OrderPlaced");
    }

    protected override Task ExecuteAsync(CancellationToken stoppingToken)
    {
        return Task.Run(async () =>
        {
            while (!stoppingToken.IsCancellationRequested)
            {
                var cr = _consumer.Consume(stoppingToken);
                var orderPlaced = JsonConvert.DeserializeObject<OrderPlaced>(cr.Message.Value);
                _logger.LogInformation($"OrderPlaced received: {orderPlaced.OrderId}");

                // Orchestration step: send ReserveInventory event
                var reserveInventory = new ReserveInventory(orderPlaced.OrderId, orderPlaced.Items);
                await _producer.ProduceAsync("ReserveInventory", reserveInventory);
            }
        }, stoppingToken);
    }
}

InventoryResultConsumer.cs

public class InventoryResultConsumer : BackgroundService
{
    private readonly IConsumer<string, string> _consumer;
    private readonly KafkaProducer _producer;
    private readonly ILogger<InventoryResultConsumer> _logger;

    public InventoryResultConsumer(KafkaProducer producer, ILogger<InventoryResultConsumer> logger)
    {
        _producer = producer;
        _logger = logger;

        var config = new ConsumerConfig
        {
            BootstrapServers = "localhost:9092",
            GroupId = "mediator-group",
            AutoOffsetReset = AutoOffsetReset.Earliest
        };
        _consumer = new ConsumerBuilder<string, string>(config).Build();
        _consumer.Subscribe(new[] { "InventoryReserved", "InventoryReservationFailed" });
    }

    protected override Task ExecuteAsync(CancellationToken stoppingToken)
    {
        return Task.Run(async () =>
        {
            while (!stoppingToken.IsCancellationRequested)
            {
                var cr = _consumer.Consume(stoppingToken);
                var message = cr.Message.Value;

                if (message.Contains("InventoryReservationFailed"))
                {
                    var fail = JsonConvert.DeserializeObject<InventoryReservationFailed>(message);
                    _logger.LogWarning($"Inventory reservation failed for Order {fail.OrderId}: {fail.Reason}");
                    // Handle failure (notify, rollback, etc.)
                }
                else
                {
                    var reserved = JsonConvert.DeserializeObject<InventoryReserved>(message);
                    _logger.LogInformation($"Inventory reserved for Order {reserved.OrderId}");

                    // Retrieve payment info from some store (simulate here)
                    var paymentInfo = new PaymentInfo("1234-5678-9012-3456", 100m);

                    var processPayment = new ProcessPayment(reserved.OrderId, paymentInfo);
                    await _producer.ProduceAsync("ProcessPayment", processPayment);
                }
            }
        }, stoppingToken);
    }
}

PaymentResultConsumer.cs

public class PaymentResultConsumer : BackgroundService
{
    private readonly IConsumer<string, string> _consumer;
    private readonly KafkaProducer _producer;
    private readonly ILogger<PaymentResultConsumer> _logger;

    public PaymentResultConsumer(KafkaProducer producer, ILogger<PaymentResultConsumer> logger)
    {
        _producer = producer;
        _logger = logger;

        var config = new ConsumerConfig
        {
            BootstrapServers = "localhost:9092",
            GroupId = "mediator-group",
            AutoOffsetReset = AutoOffsetReset.Earliest
        };
        _consumer = new ConsumerBuilder<string, string>(config).Build();
        _consumer.Subscribe(new[] { "PaymentProcessed", "PaymentFailed" });
    }

    protected override Task ExecuteAsync(CancellationToken stoppingToken)
    {
        return Task.Run(async () =>
        {
            while (!stoppingToken.IsCancellationRequested)
            {
                var cr = _consumer.Consume(stoppingToken);
                var message = cr.Message.Value;

                if (message.Contains("PaymentFailed"))
                {
                    var fail = JsonConvert.DeserializeObject<PaymentFailed>(message);
                    _logger.LogWarning($"Payment failed for Order {fail.OrderId}: {fail.Reason}");
                    // Handle failure (e.g., rollback inventory)
                }
                else
                {
                    var processed = JsonConvert.DeserializeObject<PaymentProcessed>(message);
                    _logger.LogInformation($"Payment processed for Order {processed.OrderId}");

                    var orderConfirmed = new OrderConfirmed(processed.OrderId);
                    await _producer.ProduceAsync("OrderConfirmed", orderConfirmed);
                }
            }
        }, stoppingToken);
    }
}

4. Program.cs (Register services and run)

var builder = WebApplication.CreateBuilder(args);

builder.Services.AddSingleton(new KafkaProducer("localhost:9092"));
builder.Services.AddHostedService<OrderPlacedConsumer>();
builder.Services.AddHostedService<InventoryResultConsumer>();
builder.Services.AddHostedService<PaymentResultConsumer>();

var app = builder.Build();
app.Run();

Summary

The Mediator service has three background consumers, each processing different event types
Each consumer applies the orchestration logic and triggers the next step by producing the next event
This keeps your microservices loosely coupled while having a central place for business workflow orchestration
You can extend this with state management (e.g., Redis, DB) for reliability and retries.

Kafka vs Traditional Message Buses: Why Kafka Wins for Highly Scalable Systems

Dinesh Dunukedeniya — Fri, 23 May 2025 10:20:24 +0000

In the world of distributed systems and microservices, choosing the right messaging infrastructure is critical for achieving scalability, reliability, and performance. While traditional message buses like RabbitMQ, ActiveMQ, or MSMQ have been popular for years, Apache Kafka has emerged as the go-to solution for building highly scalable and resilient systems.

In this post, we'll explore the fundamental differences between Kafka and traditional message buses, and why Kafka excels in high-scale environments.

What is a Traditional Message Bus?

Traditional message buses are messaging middleware designed to facilitate communication between different parts of a system, often using message queues and topics. These systems usually emphasize:

Message delivery guarantees: at-least-once or at-most-once delivery

Routing and filtering: selective delivery to subscribers

In-memory or disk-based queues with limited retention times

Broker-centric architecture: brokers manage message routing and storage

Examples include RabbitMQ, IBM MQ, and Azure Service Bus.

What is Kafka?

Kafka is a distributed event streaming platform designed for high-throughput, fault-tolerant, and scalable data pipelines. Key features include:

Partitioned logs: Kafka topics are split into partitions, which can be processed in parallel.

Durable storage: Messages are stored on disk for configurable retention periods, allowing consumers to re-read data.

Consumer-managed offsets: Consumers control their read position, enabling replay and flexible processing.

Horizontal scalability: Kafka brokers can be added to scale throughput seamlessly.

DEV Community: Dinesh Dunukedeniya

Event/Message Versioning in : The Complete Practical Guide

Boundaries: The Real Foundation of Any Modern Architecture (Microservices or Otherwise)

What a strong boundary really owns

What good boundaries deliver day-to-day

The warning signs of weak boundaries

Practical ways to get boundaries right

Real-World Examples of Finding (and Refining) Boundaries

The takeaway

Recommended Links

Why Asynchronous Messaging Beats REST in Modern Architectures

🔹 1. Loose Coupling vs Tight Coupling

🔹 2. Push vs Pull: Control Matters

🔹 3. Scalability at Its Core

🔹 4. Built-in Reliability

🔹 5. Event-Driven Power

🔹 6. Future-Proof Flexibility

⚖️ The Trade-offs: Why Async Isn’t Always Better

🚀 Final Thought

How to Choose the Right Microservice Boundaries

Start with the Business Domain (Domain-Driven Design)

Look at Change Patterns

Align to Team Boundaries (Two-Pizza Rule 🍕)

Data Ownership Is Non-Negotiable

Watch for Chatty Communication

Start Bigger, Split Later

Summary

Microservice-to-Microservice Authentication in .NET Using IdentityServer and JWT

Why Client Credentials Flow?

Trusted Services and Secure Environment

Token Flow Diagram

IdentityServer Setup

Requesting a Token in .NET

Calling Another Microservice with JWT

Validating JWT in the Receiving Service

Optimizing with Token Caching

Automating Token Injection with DelegatingHandler

Best Practices for Production

Summary

Fixing Microservice Dependencies with Self-Contained Events (aka ECST)

The Problem: API-Based Microservice Dependency

Scenario: Payment Service Needs Customer Info

The Fix: ECST (Self-Contained Events)

Now it looks like this:

Benefits of This Approach

Bonus Tip: Design Events with Purpose

Conclusion

Fixing Microservice Communication: From Fragile Calls to Resilient Systems

The Problem with Direct Service Calls

How to Fix These Problems: Best Practices and Patterns

1. Favor Loose Coupling Through Asynchronous Messaging

2. Add Resilience: Timeouts, Retries, and Circuit Breakers

3. Implement Graceful Fallbacks

4. Improve Observability

Conclusion

Designing E-Commerce Order Processing: Orchestration vs Choreography

Orchestration: The Centralized Conductor

Event-Driven Choreography: The Decentralized Dance

Orchestration vs Event Collaboration for E-commerce Applications: Which One Fits Your Needs?

Orchestration: Centralized Control for Complex Workflows

Event Collaboration (Choreography): Loose Coupling for Scalability

Summary Table

Final Thoughts

Orchestration vs. Choreography in EDA: Choosing the Right Approach

🕺 Choreography: The Decentralized Dance

✅ Benefits:

🔧 When to Use Choreography

💡 Common Use Cases

🎼 Orchestration: The Centralized Conductor

✅ Benefits:

🔧 When to Use Orchestration

💡 Common Use Cases

🧠 Side-by-Side Comparison

🧩 Final Thoughts

Event Collaboration in Event-Driven Architecture (EDA)

What is Event Collaboration in EDA?

How Does Event Collaboration Work?

When to Use Event Collaboration in Event-Driven Architecture

Avoid

Benefits of Event Collaboration