DEV Community: Dinesh Dunukedeniya

Is Your Microservice Safe from AI Agents? Rethinking Security in the Age of Agentic AI

Dinesh Dunukedeniya — Fri, 30 Oct 2026 12:02:44 +0000

For years, we've designed microservice architectures around a relatively simple assumption:

An authenticated user interacting with an application is generally the person making decisions and initiating business operations.

Our security models reflect this assumption. We authenticate users, issue access tokens, validate permissions, enforce business rules and allow authorised operations to proceed.

But what happens when an AI agent starts interacting with our applications on behalf of users?

This is not a niche concern. Agents integrated through the Model Context Protocol (MCP), agents holding delegated OAuth access, AI tool integrations embedded in platforms like Slack or GitHub, and AI assistants operating through an already authenticated browser session all expose the same architectural weakness. In each case, the agent presents valid credentials and passes every permission check and the backend has no way to tell whether the operation reflects a genuine user decision.

Some of these cases at least give us something to work with. An MCP integration, for example, gives us a registered client identity and a scoped token we can inspect and constrain though MCP's authorisation model is still maturing, and enforcing those scopes at the granularity that matters remains genuinely hard in practice. A browser session gives us none of that. There, the agent acts entirely inside the user's existing authenticated context, and the requests are indistinguishable from the user's own.

But the distinction matters less than it first appears. In every case, the token grants a user's permissions across a range of operations, and the backend executes each one on request. Permissions describe what the user is allowed to do. They do not describe what the user actually intended this time. Scoped permissions narrow the surface; they do not supply intent.

From the application's perspective, everything appears normal. The requests contain valid credentials. The user is authenticated. The requested operations are permitted.

Yet there is a fundamental problem. The challenge is not authentication failure. The challenge is not authorisation failure. The challenge is intent.

The Invisible Agent Problem

I call this the Invisible Agent Problem: a condition where an AI agent performs actions through a user's authenticated session, making the resulting requests technically indistinguishable from actions the user initiated directly. The browser case makes the agent invisible; the architectural gap it exposes authorisation without intent is broader.

Here's what it looks like in practice.

Imagine a trading application. You authenticate, open your trading account, and ask an AI assistant to help manage your investments. The AI assistant has access to your authenticated browser session and can interact with the application using the same permissions and context available to you.

You give it a straightforward instruction:

"Buy $10,000 worth of Anthropic shares."

The AI agent navigates to the trading page, enters the transaction details, and submits the request.

POST /api/trades/buy
Authorization: Bearer <user-access-token>
Content-Type: application/json

{
  "instrument": "ANTHROPIC",
  "amount": 10000
}

Your backend performs its normal validations:

Is the access token valid?
Is the user authenticated?
Does the user have permission to trade?
Does the user have sufficient available funds?

Everything works as expected until the final validation. The account does not have sufficient funds, and the backend returns:

{
  "error": "INSUFFICIENT_BALANCE",
  "message": "Insufficient available balance to complete this trade."
}

At this point, the application has done exactly what it was designed to do.

But the AI agent doesn't necessarily stop. It interprets the error and decides it needs to free up funds before completing the purchase. It navigates to the sell page, reviews the user's portfolio, identifies a SpaceX holding, and decides to sell it.

The user never requested this. The AI agent did.

It submits another request:

POST /api/trades/sell
Authorization: Bearer <user-access-token>
Content-Type: application/json

{
  "instrument": "SPACEX",
  "quantity": "ALL"
}

The backend receives the request. The access token is valid. The user has permission to sell shares. The user owns the SpaceX shares. All business validations succeed. The sale is accepted.

The AI agent has now sold an investment that the user never instructed it to sell. It may then retry the original purchase.

flowchart TD
    A["User Instruction<br/>'Buy $10k of Anthropic'"]
    B["AI Agent<br/>interprets goal"]
    C["Request: BUY ANTHROPIC"]
    D["Backend: INSUFFICIENT FUNDS"]
    E["AI Agent decides to free up funds<br/>(never instructed)"]
    F["Request: SELL SPACEX"]
    G["Backend: APPROVED<br/>valid token, valid permission"]

    A --> B --> C --> D --> E --> F --> G

    style E fill:#f8d7da,stroke:#dc3545
    style G fill:#d4edda,stroke:#28a745

The browser session makes this scenario easy to picture, but nothing about it depends on the browser. The same sequence valid credentials, valid permissions, unintended outcome plays out identically for an MCP tool call or an agent holding a delegated token.

From the backend's perspective, both transactions were properly authenticated and authorised. From the user's perspective, neither represented an explicitly approved investment decision. The backend validated permissions, but it never validated intent.

To be clear, this example is intentionally simplified. The AI agent is not assumed to be malicious. It is simply attempting to achieve the user's goal using the permissions already available through the authenticated session. The concern is not hostile behaviour. The concern is autonomous decision-making that extends beyond the user's explicit instruction.

This is where traditional microservice security assumptions start to break down.

Why Existing Controls Don't Catch This

Validation Check	Status
Is the access token valid?	✅ Yes
Is the user authenticated?	✅ Yes
Can the user buy shares?	✅ Yes
Can the user sell shares?	✅ Yes
Does the user own the SpaceX shares?	✅ Yes
Is the sell transaction valid?	✅ Yes

Every individual API request passes its security checks. So what went wrong?

The backend only sees two independent business operations: sell SpaceX shares, and buy Anthropic shares. It has no knowledge that the original instruction was "Buy $10,000 worth of Anthropic shares." It also has no visibility into the AI agent's decision to sell another asset as a workaround.

Traditional authorisation answers:

Is this user allowed to perform this operation?

But it doesn't necessarily answer:

Did the user actually intend to perform this operation?

That distinction becomes extremely important when AI agents can autonomously interpret instructions, make decisions, and execute multiple operations.

Why More Permissions Are Not the Answer

The obvious reflex is to introduce separate permissions for AI agents. Humans can buy and sell; AI agents can only view portfolios. But as the opening of this article argued, scoped permissions narrow the surface without supplying intent and in the browser based case, we often cannot even reliably identify when an agent is operating, because it acts through the user's existing authenticated identity.

The challenge is not necessarily that the user lacks permission. The challenge is that a valid user identity and a valid permission do not prove that a specific business operation was deliberately authorised by the user. We need to rethink where and how authorisation is enforced and we need to shrink the agent's agency before we ask a human to approve what remains.

Rethinking Security: Bound the Task, Then Obtain Intent

OWASP's guidance on excessive agency is a useful starting point. It breaks the problem into three root causes excessive functionality, excessive permissions, and excessive autonomy and the recommended order of remediation matters . Before you put a human approval prompt in front of an operation, you should first ensure the agent never had the capability to perform it in the first place.

This is the principle of least agency: give the agent only the tools, permissions, and autonomy the task requires, and nothing more . In our trading example, the user's instruction was "buy Anthropic." A least agency design would bound the agent to buy-only operations, with a spending limit derived from the instruction. The agent would never have had the ability to submit a sell request at all, not because a policy check blocked it, but because the tool was never exposed and the credential never carried that scope.

Least agency is prevention. Approval is the backstop for what remains. Both are necessary, but they address different failure modes. Least agency reduces the blast radius of a compromised or drifting agent; approval catches the cases where a permitted operation still exceeds what the user would have wanted.

For the backstop, the principle is simple:

A valid authenticated session should not automatically be treated as sufficient authorisation for every sensitive business operation.

Consider the trading application again.

Operation	Security Requirement
View portfolio	Existing authentication
View transaction history	Existing authentication
Buy shares	Risk-based authorisation
Sell shares	Risk-based authorisation
Transfer funds	Strong independent approval

Notice that the application does not need to determine whether the request came from a human or an AI agent. The same policy applies to both. For sensitive operations, the backend requires an additional authorisation step before execution.

That said, "additional authorisation" is a spectrum, not a single mechanism. Step-up authentication, transaction signing, policy-based re-authorisation, and out-of-band approval all sit on it. The right choice depends on how consequential the operation is and what regulatory context you operate in. For high-impact operations moving money, changing ownership, granting access the strongest option is explicit, transaction bound approval by the user.

From Mandate to Approval: Two Layers of Intent Enforcement

The standards community is converging on a two-layer model that maps cleanly onto the least-agency-then-approval sequence.

The first layer is the signed mandate. NIST's design tenets for agent identity management call for "cryptographically signed mandate statements that define intended outcomes and constraints for agentic actions," verifiable by downstream services . In the payments space, Google's Agent Payments Protocol (AP2) implements this as an Intent Mandate a signed statement of what the user authorised the agent to do, and under what conditions followed by a Cart Mandate that records what the agent actually did . The FIDO Alliance is now developing standards in this area, drawing on AP2 and Mastercard's Verifiable Intent framework .

The architectural implication is significant. If the agent's task carries a signed mandate saying "buy up to $10,000 of Anthropic, no sells," the backend can reject a sell request before it ever reaches an approval prompt. The mandate bounds the agent's authority at the task level; approval handles operations the mandate permits but that exceed risk thresholds.

The second layer is the approval mechanism itself. This article focuses on that layer, because it is where the operational gap is widest in existing microservice architectures. But the mandate is the stronger control; it encodes intent at the moment of delegation, not at the moment of execution.

Using CIBA for Independent User Authorisation

One mechanism for the approval layer is Client-Initiated Backchannel Authentication (CIBA). CIBA enables a client to initiate an authentication request through a backchannel, allowing the user to authenticate and approve actions using a separate device. Unlike traditional browser-based authentication, the approval interaction does not need to happen in the browser that initiated the transaction.

CIBA is not the only option, push notification with a signed challenge, step-up authentication with transaction binding, or a dedicated approval service can all serve the same purpose. What matters architecturally is that the approval happens on a channel the agent cannot reach, and that it is bound to the specific operation. CIBA is simply a well-standardised primitive for the backchannel part.

Example: Authorising a Share Transaction

An AI agent submits a request to sell SpaceX shares. The backend performs its normal checks:

Validate authentication.
Validate permissions.
Validate transaction details.
Evaluate authorisation requirements.

The backend determines that additional approval is required. Instead of executing the transaction immediately, it initiates an out-of-band approval process. The user receives a notification on their registered mobile device:

Transaction Approval Required

Operation: SELL
Instrument: SpaceX
Quantity: 100 Shares
Account: Trading Account
Source: Trading Application

The user can review the details and explicitly approve or reject the transaction. Only after that approval is verified does the transaction proceed.

The important point is that the user is approving a specific business operation, not simply authenticating again. We cannot infer intent from a token. For sensitive operations, we must obtain it explicitly.

A note of caution: this shifts the trust boundary to the approval channel. If the agent can reach the notification, manipulate the approval UI, or act on a compromised device, the control is defeated. The approval channel must be independently secured — out of reach of the agent, and ideally out of reach of the same browser session that initiated the request. OWASP's agentic threat model explicitly lists "Overwhelming the Human in the Loop" as a risk category, precisely because approval prompts that fire too often degrade into rubber stamps .

Transaction-Specific Authorisation Is Already Emerging

The broader identity and API security community has already recognised that some operations require more context than a generic access token can provide. Standards and approaches such as:

OAuth 2.0 Rich Authorization Requests (RAR)
Financial-grade API (FAPI) profiles
Transaction-authorisation patterns used in banking

move towards a model where authorisation can be associated with a specific business transaction rather than simply granting broad access to an application.

However, these approaches do not automatically solve the Invisible Agent Problem. If a user is authorised to buy and sell shares, an AI agent operating through the user's authenticated session can still potentially execute those actions.

The challenge is not simply expressing richer authorisation requirements. The challenge is ensuring that sensitive business operations receive explicit approval that is bound to the specific transaction being executed.

In other words, an access token may prove that a user can sell shares. It does not necessarily prove that the user intended to sell these shares, in this quantity, at this point in time.

Agentic AI makes this distinction increasingly important because autonomous systems can combine multiple individually authorised actions into outcomes that the user never explicitly approved.

Binding Approval to the Transaction

Approval alone is not a complete transaction-authorisation solution. The application must ensure that approval is securely bound to the exact transaction being executed.

For example, approval for "Sell 100 SpaceX shares" must not be reusable for "Sell 500 SpaceX shares" or "Transfer funds to another account."

The backend should validate:

Transaction identifier
Account identifier
Operation type
Instrument
Amount or quantity
Approval expiry
Single-use status

This prevents modification, replay, and misuse of previously approved transactions.

Risk-Based Authorisation

We also need to balance security and usability. Requiring explicit approval for every API call would create unnecessary friction. A more practical approach is risk-based authorisation.

Low-risk operations — view balances, view portfolios, retrieve transaction history. Existing authentication may be sufficient.

Medium-risk operations — buy shares, sell shares, create recurring trades. Additional approval may depend on transaction value or risk indicators.

High-risk operations — transfer funds, change ownership details, add payment destinations. Independent approval may always be required.

The exact policies depend on the business domain and regulatory requirements. The important point is that the policies are enforced by the backend, not by the browser or the AI agent.

Two practical cautions. First, approval fatigue is real: users who are prompted too often will tap "approve" reflexively, and the control degrades into a rubber stamp. Risk tiering exists precisely to keep the prompt rate low enough that each prompt is meaningful. Second, the policy itself is a governed artefact: who is allowed to mark an operation low-risk, and how are changes to that classification audited? An approval layer whose own configuration is unprotected simply moves the gap rather than closing it.

Unintended Behaviour Versus Compromised Behaviour

When discussing AI agents, it is useful to distinguish between two different risk categories.

Intent drift. The AI agent genuinely attempts to achieve the user's objective but independently performs actions that were never explicitly authorised. In our example, selling SpaceX shares to fund an Anthropic purchase is intent drift.

Agent compromise. The AI agent is influenced by prompt injection, malicious content, browser compromise, or another attack. Examples might include transferring funds, changing account settings, or executing unauthorised transactions.

The causes differ, but both expose the same architectural weakness:

The backend validates identity and permissions but lacks independent confirmation that the specific business operation should be executed.

Architectural Considerations for Existing Microservices

Introducing this approach into an existing microservices architecture requires thoughtful design. Rather than embedding approval logic into every controller, consider creating reusable capabilities:

Policy Decision Service (PDP) — central authorization logic, determines allow/deny/require approval
Risk Assessment Service — provides risk scores as decision input
Approval Orchestration Service — drives CIBA/push/signing workflows, including transaction binding and single-use validation
Identity Integration — OIDC/JWT validation and user context (capability, not a service)
Audit and Compliance — immutable decision and execution logs

One important observation:

The API Gateway should not be responsible for all business-level authorisation.

The gateway can enforce authentication and general access policies. However, only the business service truly understands the operation being performed. The trading service, not the gateway, understands whether a share sale requires explicit approval.

This distinction becomes even more important in distributed systems where multiple services participate in a single business transaction.

flowchart TD
    A["User / AI Agent"]
    B["API Gateway"]
    C["Trading Microservice"]
    D["Authorisation Policy"]
    E{"Sensitive action?"}
    F["Execute transaction"]
    G["CIBA / OOB approval"]
    H["Mobile device<br/>approve / deny"]
    I["Execute transaction"]

    A --> B --> C --> D --> E
    E -->|No| F
    E -->|Yes| G --> H --> I

What About Traditional Security Controls?

None of this replaces the controls we already rely on. We still need OAuth 2.0 and OpenID Connect, JWT validation, audience restrictions, fine-grained authorisation, API rate limiting, input validation, CSRF protection, fraud detection, audit logging, and distributed tracing.

These controls remain essential. But none of them answers the question agentic AI forces us to ask: does this business operation reflect the user's intended decision? That is the architectural gap exposed by agentic AI.

Sensitive operations require independent approval before execution, regardless of whether they originate from a human or an AI agent.

Potential Objections

Isn't MFA enough?

No. Multi-factor authentication strengthens identity assurance. It proves who the user is. It does not necessarily prove that the user reviewed and approved a specific business operation. An AI agent can still operate within a legitimately authenticated session.

Can't we just detect AI traffic?

Not reliably. When an AI agent operates through a legitimate browser session, its actions may be indistinguishable from those of the user. Detection approaches may help, but they should not be treated as the primary control.

Isn't this just a fraud detection problem?

Partially, but not entirely. Fraud detection is often reactive. The issue discussed here is fundamentally an authorisation design problem. The goal is to prevent unintended transactions from being executed rather than detect them afterwards.

Doesn't this just move the problem to the approval channel?

It moves the trust boundary, which is different. Out-of-band approval does not solve every threat — a compromised device or a malicious notification can defeat it. But it does close the specific gap this article addresses: an agent operating entirely inside a legitimate session, with no independent checkpoint between a permitted action and an unintended outcome. Least agency, applied first, reduces how often that checkpoint is needed.

Conclusion

AI agents are changing how users interact with applications.

For decades, we have assumed that an authenticated user interacting with a system was also the individual making decisions. Agents operating through a browser session challenge that assumption without requiring any change to our authentication infrastructure — and the same weakness runs through MCP integrations, delegated OAuth access, and AI tool integrations. An agent can use a user's authenticated session, reach legitimate functionality, and perform multiple operations that are technically permitted but never intended.

Identity does not imply intent.

In an agent-driven world, identity and permissions remain necessary — but for high-impact operations they are no longer sufficient. The next evolution of microservice security is two-layered: bound the agent's agency to the task it was given, then obtain explicit confirmation on a channel the agent cannot reach when a permitted action still requires a deliberate decision.

The Invisible Agent Problem is not limited to trading platforms. The same challenge emerges in banking, healthcare, insurance, e-commerce, and enterprise systems — anywhere AI agents can perform consequential actions using an existing authenticated session.

Disclaimer: Anthropic and SpaceX are used in the examples solely for illustration purposes. The scenarios described in this article are hypothetical and are intended to demonstrate an architectural security challenge rather than investment advice.

Event/Message Versioning in : The Complete Practical Guide

Dinesh Dunukedeniya — Fri, 13 Feb 2026 06:56:47 +0000

Event‑driven systems evolve. Your business changes, your data changes, and sooner or later your events must change too. The challenge is simple to describe but hard to get right: how do you evolve events without breaking every consumer in your system?

This guide walks through the practical patterns, trade‑offs, and real‑world techniques for versioning events in Kafka so your system can grow safely.

🧩 Why Event Versioning Matters

In a microservice architecture, events become long‑lived contracts. Once published, they may be consumed by:

Multiple services
Different teams
Different versions of the same service
External partners
Analytics pipelines
Machine learning models

A breaking change in an event can silently break all of them.

This is why event versioning is not a technical detail — it’s a core architectural responsibility.

The Golden Rule: Never Break Existing Consumers

Once an event schema is in production, you must assume:

Someone depends on every field
Someone depends on the event shape
Someone depends on the semantics
Someone depends on the ordering

This is why event versioning is about compatibility, not just schema changes.

Types of Compatibility

There are three main compatibility modes you need to understand.

Type	Meaning	Use Case
Forward	Old consumers can read new messages	Upgrade consumers later
Backward	New consumers can read old messages	Upgrade producers later
Full	Both directions work	Safest for large systems

Safe Changes (Non‑Breaking)

These changes are generally safe:

Adding a new optional field
Adding a new event type
Adding a new enum value (if consumers ignore unknowns)
Increasing field length
Adding metadata fields

These allow your system to evolve without breaking anyone.

Breaking Changes (Avoid These)

These changes will break consumers:

Renaming a field
Removing a field
Changing field type
Changing event meaning
Changing event order
Changing required fields

If you must do these, you need a new version.

Versioning Strategies

There are three practical ways to version events in Kafka.

Strategy 1: Version Inside the Event

Add a version field:

json { "eventType": "OrderPlaced", "version": 2, "orderId": "123", "customerId": "456", "items": [...] }

Consumers can switch logic based on version.

Pros

Single topic
Easy to evolve
Consumers choose when to upgrade

Cons

Consumers must handle multiple versions
Logic can get messy over time

Strategy 2: Version in the Topic Name

Example:

order-placed-v1
order-placed-v2

Pros

Clean separation
Consumers subscribe to the version they support
No branching logic

Cons

More topics to manage
Harder to migrate consumers
Analytics pipelines must merge topics

Use this when the new version is fundamentally different.

Strategy 3: Schema Registry Versioning

If you use Confluent Schema Registry or similar:

Each schema change creates a new version
Compatibility rules enforce safety
Consumers automatically get the right schema

Pros

Strong governance
Automatic compatibility checks
Great for large organisations

Cons

Requires Avro/Protobuf/JSON Schema
More tooling and process overhead

How to Evolve Events Safely

Step 1: Add new fields (non‑breaking)
Make them optional.

Step 2: Deploy producers with the new schema
They start sending both old + new fields.

Step 3: Wait until all consumers are upgraded
This may take days or weeks.

Step 4: Remove old fields (breaking)
Only after confirming no one depends on them.

This is the expand → migrate → contract pattern.

Handling Semantic Changes

Sometimes the shape stays the same but the meaning changes.

Example:

price used to mean “base price”, now it means “final price”.

This is a breaking change even if the schema is identical.

In this case:

Create a new event type
Or create a new version
Or create a new topic

Never silently change meaning.

Event Versioning Anti‑Patterns

Avoid these at all costs.

❌ Reusing event names for new meanings
This causes silent data corruption.

❌ Removing fields because “no one uses them”
Someone always does.

❌ Using timestamps as versioning
Unclear and unmaintainable.

❌ Forcing consumers to upgrade immediately
Breaks autonomy.

❌ Publishing different shapes of the same event without a version
Impossible to debug.

Conclusion

Event versioning is not just a technical detail — it’s a contract between teams, services, and the business. When done well, it allows your system to evolve safely for years. When done poorly, it creates silent failures, broken consumers, and painful migrations.

The key principles are simple:

Don’t break consumers
Version intentionally
Prefer additive changes
Use clear versioning strategies
Treat events as long‑lived contracts

Strong event versioning is a sign of a mature, well‑designed microservice architecture.

Boundaries: The Real Foundation of Any Modern Architecture (Microservices or Otherwise)

Dinesh Dunukedeniya — Sat, 17 Jan 2026 02:02:27 +0000

In 2026, the biggest lesson from the microservices era isn't about going small or fancy tooling, it's about boundaries.
A boundary is straightforward: it's the clear line defining what a service (or module) owns and controls and what it doesn't. Get this right, and whether you're in full microservices, a modular monolith, or something hybrid, things just work better.

When boundaries are weak or mismatched, you end up with a distributed mess that's harder to run than the monolith you started with.

What a strong boundary really owns

In practice, a well-drawn boundary covers:

A specific business capability - the stable piece of the domain the business values (think "Order Fulfillment" not "Order CRUD").
Exclusive data ownership - full control over schema, storage, and evolution.
Autonomous decisions— business rules and key logic that can change without cross team drama.
Exposed contracts — deliberate APIs or interfaces, nothing accidental.
Events published — clear notifications of important changes, so others can react loosely.

When these leak (sync calls everywhere, shared schemas, joint decisions), coupling creeps in. Suddenly your "independent" services aren't and the pain scales with your org.

What good boundaries deliver day-to-day

Independent deploys without release trains or endless coordination.
Schema or rule changes in one place without rippling everywhere.
Contained failures — one service slow or down? The core flows keep moving.
Easier onboarding — new team members grasp "this owns X" quickly.

The warning signs of weak boundaries

Changes always need multi-team sign-off.
Network calls turn simple operations into latency nightmares.
Downtime in one spot cascades.
Debugging feels like distributed tracing hell.

Practical ways to get boundaries right

Start simple: use Domain-Driven Design bounded contexts as your guide, map services (or modules) to real business domains, not entities or tech layers.

Then run the reality checks on any proposed boundary:

Can it handle its core decisions without constant sync calls to others?
Does it own its data fully including how it evolves?
Can the team change key logic or models independently?
If it's unavailable for 20–30 minutes, does the business keep running mostly fine?
Can someone new understand its purpose in under an hour?

Mostly yes? Solid. Mostly no? Rethink the cut.

Real-World Examples of Finding (and Refining) Boundaries

Finding the right boundaries isn't magic it's iterative discovery, usually starting with conversations, Event Storming sessions, or just mapping how the business actually works.
The gold standard is Domain Driven Design (DDD): identify bounded contexts (self-contained areas with their own language, rules, and models) that align to business capabilities or subdomains.

Here are a few practical examples from e-commerce (the classic domain) and beyond, showing how teams draw lines that stick or learn to redraw them.

Bad start (common anti-pattern):
"User Service", "Product Service", "Order Service", these are nouns/entities, not capabilities. They end up chatty (e.g., Order Service constantly asks Product for stock, User for details), leading to tight coupling.

Better way:
Map to what the business does end-to-end.
A typical online store might discover these strong boundaries:

Product Catalog & Discovery
Owns product info, search, recommendations, categories, images. Independent: Can change pricing models or add AI recommendations without touching orders.
Inventory Management
Real-time stock tracking,reservations, replenishment alerts. Owns stock data; publishes "stock low" or "reserved" events.
Shopping Cart & Checkout Experience Temporary cart state, promo application, checkout flow. Session-based; doesn't own permanent orders.
Order Fulfillment
Order creation, validation, status tracking, risk checks.
Payment Processing
Transactions, gateways, refunds. Highly autonomous; changes PCI rules without rippling.
Shipping & Logistics
Carrier integration, tracking, delivery estimates. Owns shipping rules and costs.

How teams find these:

Workshop with domain experts, ask "What changes together? What language do you use here?" In practice (like Microsoft's eShopOnContainers reference or real Zalando/Amazon setups), these boundaries allow independent teams and scaling (e.g., inventory spikes during sales without crashing checkout).

The takeaway

Boundaries aren't a microservices thing they're an architecture thing. Nail them, and the rest (tech, deployment, scale) falls into place much easier. Botch them, and no amount of service mesh, observability, or serverless magic fixes the underlying coupling.
Whether you're building greenfield or refactoring legacy, start here. Everything else is secondary.

Why Asynchronous Messaging Beats REST in Modern Architectures

Dinesh Dunukedeniya — Tue, 23 Sep 2025 09:25:05 +0000

In the early days of microservices, REST APIs became the default way for services to talk to each other. They’re simple, predictable, and well-understood. But as systems grow, REST starts showing cracks: latency, bottlenecks, and brittle dependencies.

That’s where asynchronous messaging with tools like Kafka, RabbitMQ, or Azure Service Bus takes the lead.

🔹 1. Loose Coupling vs Tight Coupling

REST APIs force a request–response relationship. If Service A calls Service B, A must wait until B replies.
Messaging flips this. A just sends a message and moves on. B (and even C, D, E…) can consume it when ready.

👉 This decoupling makes systems more resilient—one slow service doesn’t break the whole chain.

🔹 2. Push vs Pull: Control Matters

With REST APIs, the producer is in push mode. The caller must hit the server directly, regardless of whether that server is busy, overloaded, or temporarily down. This creates pressure and dependency on the consumer’s capacity.
With messaging, consumers operate in pull mode. They fetch and process events at their own pace. If a consumer slows down, the broker safely buffers messages until it catches up.

👉 This difference alone makes async systems naturally elastic. Services run as fast as they can—no more, no less.

🔹 3. Scalability at Its Core

REST requires every request to hit the server directly, which means scaling horizontally (load balancers, replicas).
Kafka can handle millions of events per second by design. Multiple consumers can process the same topic in parallel, enabling massive scale without choking upstream services.

👉 Async shines when load spikes—like flash sales or real-time data streams.

🔹 4. Built-in Reliability

If a REST endpoint is down, requests fail unless you build retry logic.
With messaging, events are stored durably in the broker. Even if consumers go offline, they can catch up later.

👉 That’s a huge win for mission-critical systems where losing events = losing money.

🔹 5. Event-Driven Power

REST is about asking:

“What’s your status right now?”

Messaging is about telling:

“This event happened.”

That small difference unlocks event-driven architectures, where systems react automatically to changes. Examples:

Order created → Inventory reserved → Notification sent
Payment succeeded → Loyalty points awarded → Email triggered

👉 This flow is natural, scalable, and reduces the need for constant polling.

🔹 6. Future-Proof Flexibility

With REST, if you add a new consumer, you must update the producer to call it.
With messaging, producers don’t care who listens. You can add new consumers (analytics, monitoring, fraud detection) without touching existing services.

👉 That’s how companies evolve from MVP to enterprise-scale without rewriting everything.

⚖️ The Trade-offs: Why Async Isn’t Always Better

While async messaging is powerful, it’s not a silver bullet:

Complexity: Designing an event-driven system requires careful planning (message schemas, idempotency, ordering).
Eventual Consistency: You can’t always get an immediate answer. Users may see a slight delay before data fully syncs.
Operational Overhead: Brokers like Kafka require infrastructure, monitoring, and scaling strategies. REST, by contrast, can run almost anywhere with minimal setup.

👉 For user-facing features that need instant confirmation (login, payment, search), REST is still the better choice. Async shines in the backend where scale, resilience, and decoupling matter most.

🚀 Final Thought

If you’re building for scale, resilience, and evolution, REST alone won’t cut it.
Async messaging is not just a tool it’s an architectural mindset that embraces decoupling, reliability, and growth.

The future is event-driven. And messaging makes it possible.

How to Choose the Right Microservice Boundaries

Dinesh Dunukedeniya — Sat, 09 Aug 2025 05:33:59 +0000

Choosing the size of a microservice isn’t just about code. it’s about team structure, business capabilities, and how the system will evolve over time.
Get it wrong, and you either end up with:

The Mini-Monolith → big, complex, slow to deploy, and hard for one team to own.
The Distributed Hairball → dozens of tiny services that constantly call each other, making debugging a nightmare.

Start with the Business Domain (Domain-Driven Design)

Most experienced teams begin with DDD to find bounded contexts areas where the business language, rules, and data make sense together.

Example – E-commerce:
Bounded contexts that naturally map to microservices:

Catalog → product data, categories, availability
Ordering → order creation, updates, status tracking
Payments → payment processing, refunds, compliance
Shipping → delivery tracking, carriers, schedules

Why it works:

Keeps each service focused
Prevents mixing unrelated rules
Data ownership is clear

Look at Change Patterns

If two features always change together, they probably belong in the same service.
If they change at different speeds or are handled by different teams, splitting makes sense.

Example:

Catalog changes once a month → stable
Checkout changes weekly → fast-moving
→ Keep them separate so catalog updates don’t block checkout releases.

Align to Team Boundaries (Two-Pizza Rule 🍕)

A service should be small enough for one small team (5–8 people) to fully own and deploy it without coordinating with other teams.

Example:
If both the Marketing and Payments teams constantly commit to the same codebase, that service is probably too big.

Data Ownership Is Non-Negotiable

Each microservice should own its own database.
If two services write to the same table, they’re still coupled, no matter what the code says.

Example:
Payments owns the Transactions table.
If Orders needs payment info, it calls Payments or subscribes to its events. it doesn’t write directly to that table.

Watch for Chatty Communication

If one user request calls more than 2–3 services synchronously, you may have gone too granular.

Fix:
Use event-driven architecture — publish an event like OrderPlaced and let other services (Payments, Notifications, Loyalty) react independently.

Start Bigger, Split Later

Most large companies don’t start with dozens of microservices.
They begin with a modular monolith or a few chunky services and split when needed:

Scaling issues
Release coordination pain
Regulatory/security isolation needs

Example:
A SaaS platform starts with Users + Billing + Analytics in one service.
When Billing needs PCI compliance, they split it into its service, with minimal disruption.

Summary

When deciding microservice boundaries:

Start with bounded contexts from Domain-Driven Design.
Group features by change patterns — if they change together, keep them together.
Align services to small, independent teams (Two-Pizza Rule).
Enforce data ownership — one service, one database.
Avoid excessive synchronous calls — prefer async/event-driven.
Start larger, split only when needed for scaling, compliance, or deployment independence.

The goal isn’t to have the smallest services.It’s to have well-defined, loosely coupled, independently deployable services that match your business needs and team structure.

💬 Over to you — Have you worked on a project where microservices were way too small or too big? How did it go?

Microservice-to-Microservice Authentication in .NET Using IdentityServer and JWT

Dinesh Dunukedeniya — Fri, 01 Aug 2025 23:06:23 +0000

In a microservices architecture, secure communication between services is just as important as securing external API calls. One of the most common and robust ways to handle this in .NET is by using IdentityServer, JWT tokens, and the Client Credentials flow.

In this article, you’ll learn how to:

Authenticate microservices using Client Credentials
Request and validate JWT tokens
Pass tokens between services securely
Optimize token management with caching
Apply best practices for real-world deployments.

Why Client Credentials Flow?

The Client Credentials flow is designed for machine-to-machine communication. Unlike user-based authentication (e.g., Authorization Code flow), there’s no interactive login. Instead, each service identifies itself using a ClientId and ClientSecret, and IdentityServer issues an access token that can be used to call other services.

When to use it:

Service A needs to call Service B directly.
No user context is involved.
Both services are trusted and run in a secure environment.

Trusted Services and Secure Environment

When we say "both services are trusted and run in a secure environment", it means:
Trusted Services

Both microservices are owned and controlled by the same organization (or within a trusted partner ecosystem).
Each service is registered in IdentityServer with its own ClientId and ClientSecret.
Services rely on IdentityServer as the trust broker — if a service presents a valid JWT from IdentityServer, other services trust it.

Example:
A Billing API (Service A) calls a Payment Processor API (Service B) within your company's internal infrastructure. Both are part of your platform and use IdentityServer for token issuance.

Secure Environment

Services communicate over a private and encrypted network (e.g., AWS VPC, Azure VNet, Kubernetes internal network).
All traffic is secured with TLS (HTTPS) to prevent token interception.
Secrets (e.g., ClientSecret) are stored securely in tools like Azure Key Vault or AWS Secrets Manager.
IdentityServer is either private or protected with firewalls, API gateways, and access controls.

Example:
Your services are deployed in Kubernetes and communicate over an internal service mesh (like Istio), where mTLS enforces encryption and mutual authentication.

⚠️ If your services are NOT in a trusted or secure environment (e.g., exposed to the public internet or involve third-party APIs), you should:

Validate the audience in the JWT token strictly.
Use an API Gateway with rate limiting, throttling, and authentication enforcement.
Consider mTLS in addition to JWT for an extra security layer.
Implement scopes and claims-based authorization to enforce least privilege.

Token Flow Diagram

Here’s how authentication works between two microservices:

+-----------------+           +---------------------+          +-----------------+
| Service A       |           | IdentityServer      |          | Service B       |
| (Caller)        |           | (Auth Provider)     |          | (Resource API)  |
+-----------------+           +---------------------+          +-----------------+
       |                              |                               |
       |   1. Request Token           |                               |
       |----------------------------->|                               |
       |                              |                               |
       |         2. JWT Token         |                               |
       |<-----------------------------|                               |
       |                              |                               |
       |  3. Call API with Token      |                               |
       |------------------------------------------------------------->|
       |                              |                               |
       |          4. Validate Token                                   |
       |                              |                               |
       |<-------------------------------------------------------------|

IdentityServer Setup

First, configure a client in IdentityServer for service-to-service communication:

new Client
{
    ClientId = "service-a",
    ClientSecrets = { new Secret("secret".Sha256()) },
    AllowedGrantTypes = GrantTypes.ClientCredentials,
    AllowedScopes = { "service-b-api" }
}

This allows Service A to request a token for the scope service-b-api and then use it when calling Service B.

Requesting a Token in .NET

Service A will use the IdentityModel library to request a token:

using IdentityModel.Client;
using System.Net.Http;

public class TokenService
{
    private readonly HttpClient _httpClient;

    public TokenService(HttpClient httpClient)
    {
        _httpClient = httpClient;
    }

    public async Task<string> GetTokenAsync()
    {
        var disco = await _httpClient.GetDiscoveryDocumentAsync("https://identityserver-url");
        if (disco.IsError) throw new Exception(disco.Error);

        var tokenResponse = await _httpClient.RequestClientCredentialsTokenAsync(new ClientCredentialsTokenRequest
        {
            Address = disco.TokenEndpoint,
            ClientId = "service-a",
            ClientSecret = "secret",
            Scope = "service-b-api"
        });

        if (tokenResponse.IsError) throw new Exception(tokenResponse.Error);

        return tokenResponse.AccessToken;
    }
}

Calling Another Microservice with JWT

Once we have a token, we can attach it to the Authorization header:

using IdentityModel.Client;

public class ServiceBClient
{
    private readonly HttpClient _httpClient;
    private readonly TokenService _tokenService;

    public ServiceBClient(HttpClient httpClient, TokenService tokenService)
    {
        _httpClient = httpClient;
        _tokenService = tokenService;
    }

    public async Task<string> GetDataAsync()
    {
        var token = await _tokenService.GetTokenAsync();
        _httpClient.SetBearerToken(token);

        var response = await _httpClient.GetAsync("https://service-b/api/data");
        response.EnsureSuccessStatusCode();

        return await response.Content.ReadAsStringAsync();
    }
}

Validating JWT in the Receiving Service

Service B must validate the JWT token. Add JWT Bearer authentication:

builder.Services.AddAuthentication("Bearer")
    .AddJwtBearer("Bearer", options =>
    {
        options.Authority = "https://identityserver-url";
        options.RequireHttpsMetadata = true;
        options.TokenValidationParameters = new TokenValidationParameters
        {
            ValidateAudience = false
        };
    });

builder.Services.AddAuthorization();

Apply authentication and authorization middleware:

var app = builder.Build();
app.UseAuthentication();
app.UseAuthorization();
app.MapControllers();
app.Run();

Optimizing with Token Caching

You should avoid requesting a token for every call. Instead, cache it until it expires:

public class CachedTokenService
{
    private readonly TokenService _tokenService;
    private string _cachedToken;
    private DateTime _expiresAt;

    public CachedTokenService(TokenService tokenService)
    {
        _tokenService = tokenService;
    }

    public async Task<string> GetTokenAsync()
    {
        if (!string.IsNullOrEmpty(_cachedToken) && DateTime.UtcNow < _expiresAt)
            return _cachedToken;

        var token = await _tokenService.GetTokenAsync();
        _cachedToken = token;
        _expiresAt = DateTime.UtcNow.AddMinutes(50); // assuming 1-hour expiry

        return _cachedToken;
    }
}

Automating Token Injection with DelegatingHandler

You can also create a custom DelegatingHandler to automatically add tokens:

public class TokenHandler : DelegatingHandler
{
    private readonly CachedTokenService _tokenService;

    public TokenHandler(CachedTokenService tokenService)
    {
        _tokenService = tokenService;
    }

    protected override async Task<HttpResponseMessage> SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
    {
        var token = await _tokenService.GetTokenAsync();
        request.SetBearerToken(token);
        return await base.SendAsync(request, cancellationToken);
    }
}

builder.Services.AddHttpClient<ServiceBClient>()
    .AddHttpMessageHandler<TokenHandler>();

Now, every call to Service B will automatically include a valid JWT.

Best Practices for Production

🔒 Secure Secrets: Store ClientSecret in a secret manager (e.g., Azure Key Vault or AWS Secrets Manager).

📜 Logging & Monitoring: Log token acquisition failures and monitor IdentityServer availability.

🚀 Use Delegating Handlers: Automate token injection for all service-to-service calls.

Summary

Use IdentityServer and Client Credentials flow for secure microservice-to-microservice communication.

Retrieve JWT tokens using IdentityModel.
Validate tokens in each service using JwtBearer.
Optimize with token caching and DelegatingHandler.
Follow security best practices for secret management and logging.

By implementing this approach, you ensure that your microservices communicate securely and efficiently in a production environment.

Fixing Microservice Dependencies with Self-Contained Events (aka ECST)

Dinesh Dunukedeniya — Sun, 27 Jul 2025 04:05:31 +0000

Microservices are supposed to be independent, autonomous, and scalable. But in reality, they often end up tightly coupled—relying on synchronous API calls, complex orchestration, and fragile dependencies.

In this post, we’ll explore how to break microservice dependencies using a powerful pattern known as Event-Carried State Transfer (ECST)—or as I like to call it, Self-Contained Events. It’s a simple yet impactful idea: put all the necessary data in the event.

The Problem: API-Based Microservice Dependency

Many microservice systems fall into a trap: they rely heavily on synchronous API calls between services to share data. This creates tight coupling, reduces resilience, and introduces latency.This API dependencies hurt reliability and scalability.

Scenario: Payment Service Needs Customer Info

Let’s say a customer places an order. The Order Service emits an event like:

{
  "eventType": "OrderPlaced",
  "orderId": "ORD-1234",
  "customerId": "CUST-5678",
  "items": [ ... ]
}

So far, so good. But now the Payment Service needs to:

Validate the customer’s billing status.

Access their preferred payment method.

Maybe even apply discounts based on customer type.

To do that, it has to call the Customer Service:

Payment Service → GET /customers/CUST-5678 → Customer Service

Here’s the problem:

If Customer Service is slow or down, the Payment Service fails.

You’ve created a runtime dependency between two services.

The system becomes fragile and hard to scale.

Despite using events, the services are still tightly coupled, relying on synchronous communication to function.

The Fix: ECST (Self-Contained Events)

Instead of making an API call, the Customer Service emits events when relevant data changes:

{
  "eventType": "CustomerUpdated",
  "customerId": "CUST-5678",
  "name": "Alice Smith",
  "email": "alice@example.com",
  "loyaltyTier": "gold",
  "preferredPaymentMethod": "visa"
}

The Payment Service subscribes to these events and stores only the fields it needs—either in a local database or an in-memory cache.

Now it looks like this:

Customer Service emits CustomerCreated and CustomerUpdated events.
Payment Service listens and stores a local copy of needed customer data.
When OrderPlaced is received, Payment uses local customer data—no API call required.

Benefits of This Approach

No runtime dependency on Customer Service.
Improved resilience—each service can function independently.
Faster, async processing using pre-synced local data.
Better boundaries—each service owns its domain and data shape.

Bonus Tip: Design Events with Purpose

Design your events with the consumer’s use case in mind.
Use versioned schemas to evolve safely.
Make handlers idempotent to handle retries.
Instead of emitting minimal "something happened" events:

{ "eventType": "CustomerUpdated", "customerId": "CUST-5678" }

Emit rich, meaningful events:

{
  "eventType": "CustomerUpdated",
  "customerId": "CUST-5678",
  "email": "alice@example.com",
  "loyaltyTier": "gold"
}

That’s how you enable autonomy.

Conclusion

Microservices shine when services can operate independently. But if every service depends on synchronous calls to others, you haven’t truly decoupled.

Event-Carried State Transfer (ECST) is a simple yet powerful technique to fix that.
✅ Push state with events
✅ Keep local, read-only data
✅ Break the web of runtime dependencies

Fixing Microservice Communication: From Fragile Calls to Resilient Systems

Dinesh Dunukedeniya — Sun, 22 Jun 2025 10:46:42 +0000

Microservices promise flexibility, scalability, and faster deployments. However, without proper communication strategies, they quickly become a tangled web of tightly coupled services, frequent downtime, and frustrating bugs. In this article, we’ll explore common microservice communication problems and how to fix them by adopting modern patterns and tools.

The Problem with Direct Service Calls

Imagine a typical e-commerce application with services like OrderService, PaymentService, and InventoryService. A direct HTTP call chain might look like this:
OrderService → PaymentService → InventoryService

Now, suppose InventoryService goes down. The entire chain breaks, and placing orders fails even though the issue is isolated.

Problems with direct service-to-service calls:

Tight Coupling
Each service depends on the availability, responsiveness, and correct behavior of the services it calls. If one service goes down, all services calling it may also fail or become slow.
Cascading Failures
One failure propagates across multiple services. This can bring down large parts of your architecture, even if only one component fails.
Increased Latency
Each call adds network delay, making the overall order process slow and frustrating.
Retry Storms and Thundering Herds
Simultaneous retries overload the failing service even more. This creates a feedback loop that makes recovery harder.
Harder to Scale and Deploy Independently
Synchronous dependencies force tight coordination between teams and services, limiting independent deployment.
Scaling one service may require scaling others to handle the load.
Harder to Test
Unit and integration tests require other services to be available.

How to Fix These Problems: Best Practices and Patterns

To build resilient, scalable, and maintainable distributed systems, it’s essential to follow sound architectural principles. Below are some best practices and design patterns that can help you tackle common challenges in microservices.

1. Favor Loose Coupling Through Asynchronous Messaging

One of the most effective ways to improve system resilience and scalability is by embracing loose coupling through asynchronous messaging. Direct, synchronous calls between services may seem convenient, but they can introduce tight dependencies, slow down performance, and increase the risk of cascading failures.

Instead, adopt a message-based/event-driven architecture using tools like Apache Kafka, Azure Service Bus, or RabbitMQ. These platforms allow services to communicate in a decoupled fashion by publishing and subscribing to events rather than calling each other directly.

Why It Works:

Decoupling: Services don’t need to know each other’s internal workings or availability.
Scalability: Consumers can scale independently based on demand.
Resilience: Fail ures in one service won’t bring down the entire system.

Example Pattern in Action:
Let’s say a customer places an order. Instead of the OrderService invoking other services directly, it publishes an OrderPlaced event to Kafka:

public class KafkaPublisher
{
    private readonly IProducer<Null, string> _producer;

    public KafkaPublisher(string bootstrapServers)
    {
        var config = new ProducerConfig { BootstrapServers = bootstrapServers };
        _producer = new ProducerBuilder<Null, string>(config).Build();
    }

    public async Task PublishOrderPlacedAsync(OrderPlacedEvent orderPlaced)
    {
        var message = JsonSerializer.Serialize(orderPlaced);
        await _producer.ProduceAsync("order-events", new Message<Null, string> { Value = message });
    }
}

Usage in OrderService:

await kafkaPublisher.PublishOrderPlacedAsync(new OrderPlacedEvent {
    OrderId = "123",
    TotalAmount = 99.99
});

From there:

PaymentService listens for OrderPlaced events and initiates payment processing.

public class KafkaConsumerService
{
    private readonly IConsumer<Ignore, string> _consumer;

    public KafkaConsumerService(string bootstrapServers, string topic, string groupId)
    {
        var config = new ConsumerConfig
        {
            BootstrapServers = bootstrapServers,
            GroupId = groupId,
            AutoOffsetReset = AutoOffsetReset.Earliest
        };

        _consumer = new ConsumerBuilder<Ignore, string>(config).Build();
        _consumer.Subscribe(topic);
    }

    public void StartConsuming(CancellationToken cancellationToken)
    {
        Task.Run(() =>
        {
            while (!cancellationToken.IsCancellationRequested)
            {
                try
                {
                    var consumeResult = _consumer.Consume(cancellationToken);
                    var orderPlaced = JsonSerializer.Deserialize<OrderPlacedEvent>(consumeResult.Message.Value);

                    if (orderPlaced != null)
                    {
                        ProcessPayment(orderPlaced);
                    }
                }
                catch (ConsumeException ex)
                {
                    // Handle consume exception (e.g. log it)
                }
            }
        }, cancellationToken);
    }

    private void ProcessPayment(OrderPlacedEvent orderPlaced)
    {
        // Simulate payment processing logic
        Console.WriteLine($"Processing payment for Order ID: {orderPlaced.OrderId}, Amount: {orderPlaced.TotalAmount}");
        // Here you could interact with a payment gateway or record payment details
    }
}

Similarly, InventoryService updates stock levels accordingly, and EmailService sends out an order confirmation email.

Each service acts independently, processing the message in its own time and retrying as needed. This pattern not only improves modularity but also creates a more robust and responsive system.

To fully harness the power of this architecture, it’s important to use the right message types for the right purpose. Choosing the correct communication pattern improves clarity, reduces unnecessary dependencies, and ensures services are loosely coupled by design.

Different scenarios call for different message styles:

Event Notifications: Inform that something has occurred, such as OrderPlaced. These are fire-and-forget and don't require a response.
Event-Carried State Transfer: Instead of triggering additional service calls, include relevant data in the event itself. For example, OrderPlaced might contain the full order details so downstream services don’t need to query OrderService.
Command Messages: Explicitly request another service to act (used carefully to avoid tight coupling).

Read my previous article to understand event driven architecture

2. Add Resilience: Timeouts, Retries, and Circuit Breakers

Even in a world leaning heavily on asynchronous messaging, sometimes synchronous calls are inevitable, particularly when dealing with legacy systems, external APIs, or tight coordination between services. When that’s the case, building resilience directly into your service communication can prevent temporary issues from turning into full-blown outages.

Timeouts
Set an upper limit on how long a service should wait for a response. This ensures your application doesn’t hang indefinitely while waiting for a slow or unresponsive dependency.

Retries with Exponential Backoff
Transient faults (like network blips or momentary service hiccups) are often self-healing. With retry policies, you can attempt the request again, waiting slightly longer each time to avoid compounding the issue.

Circuit Breakers
If a downstream service keeps failing, stop hammering it. A circuit breaker temporarily halts requests, giving the failing service time to recover and your system time to avoid a full meltdown.

Putting It All Together with Polly and .Net,

var retryPolicy = Policy
    .Handle<HttpRequestException>()
    .WaitAndRetryAsync(
        retryCount: 3,
        sleepDurationProvider: attempt => TimeSpan.FromSeconds(Math.Pow(2, attempt)),
        onRetry: (exception, timeSpan, retryCount, context) =>
        {
            Console.WriteLine($"Retry {retryCount} after {timeSpan.TotalSeconds}s due to: {exception.Message}");
        });

var circuitBreakerPolicy = Policy
    .Handle<HttpRequestException>()
    .CircuitBreakerAsync(
        exceptionsAllowedBeforeBreaking: 2,
        durationOfBreak: TimeSpan.FromSeconds(30),
        onBreak: (ex, breakDelay) =>
        {
            Console.WriteLine($"Circuit broken! Delay: {breakDelay.TotalSeconds}s");
        },
        onReset: () => Console.WriteLine("Circuit closed, operations resumed."),
        onHalfOpen: () => Console.WriteLine("Circuit half-open, testing service.")
    );

// Wrap policies
var policyWrap = Policy.WrapAsync(retryPolicy, circuitBreakerPolicy);

// Execute HTTP request with resilience
var response = await policyWrap.ExecuteAsync(() =>
    httpClient.GetAsync("https://inventory-service/api/check-stock"));

Benefits:

Improved fault tolerance: Graceful recovery from temporary service hiccups.
Stability under stress: Prevents cascading failures when a downstream service is unhealthy.
Better user experience: Avoids unnecessary timeouts and improves perceived system responsiveness.

3. Implement Graceful Fallbacks

Even with retries and circuit breakers, sometimes your dependencies simply won’t respond. In those cases, your system still needs to behave gracefully, not crash or hang indefinitely. This is where fallback strategies come in.

Fallbacks provide alternative logic or responses when primary dependencies fail, ensuring the user experience remains smooth and your service stays operational, even if at reduced functionality.

Use Case: Inventory Service Fallback
Let’s say your application relies on an external inventory service to check stock availability. If that service goes down, rather than propagating an error or timing out, you could route the request to a backup source or return a cached/default response.

public interface IInventoryService
{
    Task<string> CheckStockAsync(string productId);
}

public class PrimaryInventoryService : IInventoryService
{
    public async Task<string> CheckStockAsync(string productId)
    {
        // Simulate failure
        throw new HttpRequestException("Primary service unavailable");
    }
}

public class BackupInventoryService : IInventoryService
{
    public async Task<string> CheckStockAsync(string productId)
    {
        return await Task.FromResult("Stock from backup service: 10 units");
    }
}

Now let’s set up a fallback policy using Polly:

var backupService = new BackupInventoryService();
var primaryService = new PrimaryInventoryService();

var fallbackPolicy = Policy<string>
    .Handle<HttpRequestException>()
    .FallbackAsync(
        fallbackAction: async cancellationToken =>
        {
            // Call the backup service when the primary fails
            Console.WriteLine("Primary service failed. Using backup...");
            return await backupService.CheckStockAsync("P123");
        });

var result = await fallbackPolicy.ExecuteAsync(async () =>
{
    return await primaryService.CheckStockAsync("P123");
});

Console.WriteLine(result);

Benefits:

User continuity: Prevents error messages or app crashes.
Resilience: Keeps the service running, even if at reduced precision or reliability.
Flexibility: Allows for dynamic routing, caching, or degradation strategies when under pressure.

4. Improve Observability

Modern distributed systems are often sprawling, complex, and asynchronous, which makes understanding what's happening inside them a challenge. To avoid flying blind, it’s crucial to bake in observability from the very beginning.

Observability isn't just about logging errors; it’s about gaining real-time, actionable insight into system behavior and performance. By using tools like OpenTelemetry, Jaeger, or Zipkin, you can trace how requests flow through your services, identify bottlenecks, and debug issues with surgical precision.

Best Practices for Event Tracing and Monitoring

Tag Each Event with Correlation IDs: Generate and propagate a unique identifier with each request or event. This allows you to trace a single transaction across services and threads, even across messaging systems like Kafka or Azure Service Bus.
Trace Message Consumption and Processing Times Measure the duration and path of every event, from the moment it's published to when it's processed. This helps pinpoint slow consumers, misconfigurations, or performance regressions.
Monitor for Failing or Degraded Services: Set up alerts for failure rates, message lag, queue depth, and other signals that indicate unhealthy behavior in your system.

Benefits:

Faster troubleshooting: Identify root causes without guesswork.
Performance optimization: Locate bottlenecks and slow code paths.
Operational confidence: Know when things go wrong—and why.

Conclusion

Designing resilient distributed systems isn’t just about writing more code. It’s about writing smarter code. By favoring loose coupling through asynchronous messaging, implementing robust fault-handling mechanisms, providing graceful fallbacks, and enhancing observability, you lay the groundwork for systems that don’t just function under ideal conditions; they thrive under pressure.

The real world is messy. Services fail, networks lag, and dependencies go dark. But with the patterns and tools we've explored from Kafka and Polly to OpenTelemetry, you’ll be equipped to handle it all with confidence and control.

Start with small improvements, measure their impact, and keep refining. Resilience is a journey, not a destination, and every thoughtful design choice gets you closer.

Designing E-Commerce Order Processing: Orchestration vs Choreography

Dinesh Dunukedeniya — Wed, 11 Jun 2025 11:00:32 +0000

In the world of microservices and event-driven architecture, orchestrating business workflows is a common challenge. Two popular patterns for managing these workflows are Orchestration and Choreography, each with unique strengths and trade-offs.

In this post, we’ll design a classic use case E-Commerce Order Processing using both patterns, helping you understand how they differ in real world scenarios.

Use Case: E-Commerce Order Processing
The system must handle customer orders by:

Receiving the order
Processing payment
Updating inventory
Scheduling shipping

The flow must ensure consistency; either the entire process is completed, or appropriate compensations occur.

Let's examine the states involved in this process.

Orchestration: The Centralized Conductor

How it works:
Each microservice publishes events to Kafka. A central orchestrator subscribes to these events, determines the next step in the workflow, and then publishes a new event to Kafka to trigger the appropriate microservice to act.

The diagram below illustrates this flow with a typical e-commerce order processing example:

OrderService publishes an OrderCreated event to Kafka after a customer places an order.
OrderOrchestrator consumes the OrderCreated event and publishes a ProcessPayment event.
PaymentService subscribes to ProcessPayment, processes the payment, and publishes either:
- PaymentCompleted if successful, or
- PaymentFailed if the transaction fails.
On PaymentCompleted, the orchestrator triggers the next step by publishing ReserveStock.
InventoryService acts on ReserveStock and replies with:
- StockReserved if inventory is available, or
- OutOfStock if items are unavailable.
On successful stock reservation, the orchestrator publishes ScheduleDelivery.
ShippingService then schedules delivery and publishes DeliveryScheduled.
Finally, OrderService updates the order status to success or failure based on the outcome of each step.

Event-Driven Choreography: The Decentralized Dance

How it works:
Each microservice publishes events to Kafka and subscribes to relevant events from other services. Based on the event it receives, each service independently determines its next action and publishes a new event to Kafka, allowing the workflow to progress without a central orchestrator.

OrderService handles the customer request and publishes an OrderPlaced event to Kafka.

The diagram below illustrates this flow with a typical e-commerce order processing example:

Customer places an order via OrderService.
OrderService processes the request and publishes an OrderPlaced event to Kafka.
PaymentService subscribes to OrderPlaced, triggers the payment processing logic, and then publishes PaymentProcessed.
InventoryService subscribes to PaymentProcessed, updates the inventory, and publishes InventoryUpdated.
ShippingService subscribes to InventoryUpdated, schedules the delivery, and then publishes ShippingScheduled.
OrderService subscribes to ShippingScheduled, updates the order status, and sends the Order Confirmation to the Customer.

Orchestration vs Event Collaboration for E-commerce Applications: Which One Fits Your Needs?

E-commerce platforms must handle huge transaction volumes reliably and efficiently. Coordinating multiple microservices in such systems requires careful choice between orchestration and event collaboration (choreography).

Let’s compare both approaches in the context of e-commerce:

Orchestration: Centralized Control for Complex Workflows

When to choose:

Your order processing involves multiple dependent steps: payment, inventory reservation, shipping, notifications.
You require strict control over workflow execution order.
Auditing, retry logic, and failure recovery are critical (e.g., ensuring payment succeeds before shipping).
You want clear visibility into the entire order lifecycle.

Advantages for e-commerce:

Simplifies handling complex business rules and transactional workflows.
Easier to implement consistent retry, rollback, and compensation logic.
Centralized monitoring and logging for compliance and troubleshooting.

Challenges:

The orchestrator can become a performance bottleneck under very high load if not well designed.
Introduces tighter coupling between services and orchestrator.
Requires orchestrator to be scalable and highly available to avoid single points of failure.

Event Collaboration (Choreography): Loose Coupling for Scalability

When to choose:

Your microservices operate more independently (e.g., inventory updates, user notifications, analytics).
You can tolerate eventual consistency between services (e.g., shipping service reacts to payment success event asynchronously).
Scalability and fault tolerance are top priorities.
You want to reduce tight coupling between services.

Advantages for e-commerce:

Highly scalable as each service consumes and produces events independently.
Failure in one service doesn’t block the entire system.
Easier to extend system with new event consumers without changing existing services.

Challenges:

Complex to track end-to-end workflows and troubleshoot failures.
Potential for event storms or duplicate events that need careful handling.
Requires robust event monitoring, tracing, and alerting infrastructure.

Summary Table

Aspect	Orchestration	Event Collaboration (Choreography)
Control	Centralized, strict workflow control	Decentralized, loose coupling
Consistency	Strong consistency, sequential steps	Eventual consistency
Scalability	Can be bottleneck if not designed well	Highly scalable, services scale independently
Fault tolerance	Orchestrator failure impacts flow	Failure isolated to individual services
Monitoring & Debugging	Easier due to centralized control	Complex, needs strong tracing and monitoring

Final Thoughts

If your workflows involve multiple dependent steps like payment authorization, inventory reservation, and shipping scheduling — requiring strict control, auditing, and reliable retries — then orchestration is usually the better approach. For example, a central orchestrator managing events like OrderCreated, ProcessPayment, and ReserveStock ensures the order flow proceeds reliably. Just ensure your orchestrator is designed to be highly scalable and resilient to handle heavy transaction loads without becoming a bottleneck.

If your system benefits from loose coupling and eventual consistency—such as independent services reacting asynchronously to events like OrderPlaced, PaymentProcessed, and InventoryUpdated—and you want better scalability, then Event Collaboration (Choreography) is preferable. Be prepared to invest in strong event monitoring and tracing to handle the complexity of decentralized workflows.

Often, a hybrid approach combining orchestration for critical, multi-step processes and choreography for asynchronous, scalable interactions works best to balance reliability and scalability in e-commerce systems.

Orchestration vs. Choreography in EDA: Choosing the Right Approach

Dinesh Dunukedeniya — Mon, 02 Jun 2025 11:06:08 +0000

In my previous posts, I explored the fundamentals of orchestration and choreography within Event-Driven Architecture (EDA). These two patterns offer distinct strategies for coordinating microservice interactions.

This post cuts through the definitions and dives straight into comparing the two, with clear use cases and a side-by-side table to help you decide which pattern fits your needs best.

🕺 Choreography: The Decentralized Dance

Also known as event collaboration or broker-based topology, choreography is all about microservices reacting independently to events — no central brain calling the shots.

✅ Benefits:

Highly scalable — no central bottlenecks
Loosely coupled — services are independent
Truly event-driven — ideal for real-time systems

🔧 When to Use Choreography

Use it when you want services to respond autonomously to events, especially in reactive, scalable, and loosely coupled systems.

💡 Common Use Cases

🛒 E-Commerce Order Processing

OrderPlaced → triggers PaymentService
PaymentCompleted → triggers InventoryService
StockUpdated → triggers ShippingService

Each service listens for and reacts to events — no central coordination.

🔍 Fraud Detection

A TransactionInitiated event may trigger multiple fraud checks, logging systems, and alerting processes — all in parallel.

📡 IoT Pipelines

Sensor emits data → multiple services handle analytics, alerting, and storage asynchronously.

🎼 Orchestration: The Centralized Conductor

With orchestration, a central service — often called a Saga Orchestrator — directs the process flow step-by-step.

✅ Benefits:

Easier to manage complex workflows
Centralized error handling and rollback logic
Better observability and debugging

🔧 When to Use Orchestration

Best suited for transaction-heavy, linear, or strictly ordered workflows where the sequence of operations matters.

💡 Common Use Cases

✈️ Travel Booking System

Book flights → hotels → car rentals. If hotel booking fails, rollback flight reservation.

🏦 Loan Approval

CreditCheck → FraudCheck → FinalApproval — in order.

📦 Order Fulfillment

Centralized process: Payment → Inventory → Shipping.

🧠 Side-by-Side Comparison

⚖️ Aspect	🕺 Choreography (Event Collaboration)	🎼 Orchestration (Centralized Control)
Architecture Style	Decentralized — services own logic individually	Centralized controller sequences logic
Scalability	High — services scale independently	Limited — orchestrator can become a bottleneck
Resilience	High — one service failure doesn’t impact others	Lower — unless orchestrator is highly available
Workflow Complexity	Complex — hard to trace entire flow	Simple — easy to follow, easier to reason about
Deployment	Independent — loosely coupled	Tightly coupled — or even monolithic
Monitoring	Requires external tracking/observability tools	Easier — one place to track everything
Latency	Potentially higher due to async communication	Typically lower latency
Flexibility	Very flexible — services evolve independently	Less flexible — orchestrator updates needed for changes
Best Fit For	Long-running, complex, scalable workflows	Linear, short, transactional workflows

🧩 Final Thoughts

There's no one-size-fits-all. Choosing between choreography and orchestration depends on your system's goals:

Need scalability, loose coupling, and parallel event processing? Go with choreography.
Need order, simplicity, and centralized control? Choose orchestration.

Design smart — choose based on use case, team expertise, and operational requirements.

✍️ Have thoughts or examples from your own architecture? Drop a comment and let’s discuss!

Event Collaboration in Event-Driven Architecture (EDA)

Dinesh Dunukedeniya — Tue, 27 May 2025 12:48:12 +0000

What is Event Collaboration in EDA?

In simple terms, Event Collaboration/ Choreography is a decentralized way for services to communicate through events without a central controller. Think of it like a dance where each participant knows their steps and reacts based on cues from others there’s no single conductor telling everyone what to do.

In the context of EDA:

Service emit events upon state changes, enabling others to listen and react accordingly.
Each service listens for relevant events and reacts accordingly.
There is no central orchestrator managing the process.
Each microservice typically maintains its own state based on events it receives.

This pattern is sometimes also called Broker Topology, a term popularized by Mark Richards in Software Architecture Patterns (O’Reilly).
The name highlights that services communicate indirectly through a message broker, which routes events but does not control the workflow.

How Does Event Collaboration Work?

Imagine an online store scenario:

Order Service emits an OrderPlaced event.

Inventory Service listens to OrderPlaced, checks stock, and emits StockReserved or StockOut.
Payment Service listens to StockReserved, processes payment, and emits PaymentCompleted.
Shipping Service listens to PaymentCompleted and arranges delivery.

Each service acts independently, reacting to events and emitting new events for others. The flow emerges naturally from these interactions.

When to Use Event Collaboration in Event-Driven Architecture

Event coloboration is a powerful architectural pattern, but it’s not the perfect fit for every scenario. Consider using choreography when:

✅ You Want Loose Coupling Between Services
Each service operates independently, only knowing about events, not about the other services. This reduces dependencies and allows teams to work autonomously.

✅ Your System Is Highly Distributed
In microservices or cloud-native architectures where services are deployed separately, choreography supports scalability and flexibility.

✅ You Need Asynchronous, Event-Driven Workflows
If your business processes naturally fit event streams (e.g., order processing, payment events), choreography enables responsive and resilient flow

✅ You Expect to Scale or Evolve Services Independently
Since services produce and consume events without tight integration, you can add or update services with minimal impact on others.

✅ You Can Invest in Observability and Monitoring
Because choreography lacks a central controller, you need good tooling for tracing, logging, and monitoring event flows across services.

Avoid

⚠️ If you require strict, centralized control over business workflows, orchestration might be a better fit.

⚠️ When transactional consistency is critical across services and must be tightly managed.

⚠️ If your team is new to distributed event-driven systems and you want simpler debugging and error handling initially.

Benefits of Event Collaboration

Loose Coupling: Services only know about event formats, not about each other directly. This simplifies changes and scaling.
Scalability: Services can be developed, deployed, and scaled independently.
Flexibility: New services can join by simply subscribing to events.
Resilience: Failure in one service doesn’t block the whole process.

Challenges to Consider

Visibility & Monitoring: Without a central controller, tracking the overall process flow requires dedicated observability tools.
Complexity in Debugging: Understanding event chains can be tricky when many services interact asynchronously.
Consistency: Ensuring data consistency across services can be more challenging without orchestration.

Implementing the Choreography Pattern in .NET Microservices with Kafka

In a choreography-based event-driven system, each microservice reacts to events and, if needed, emits its own events. This allows services to be highly autonomous and loosely coupled.

Since all microservices are ASP.NET Core web apps, Kafka event consumers are implemented using BackgroundService (hosted services) within each app.

Architecture Overview

Microservices Involved:

Order Service
Initiates the workflow by publishing OrderPlaced events.
Inventory Service
Listens for OrderPlaced events, checks stock, and publishes either InventoryReserved or InventoryReservationFailed.
Payment Service
Listens for InventoryReserved and processes payment, publishing PaymentProcessed or PaymentFailed.
Shipping Service
Listens for PaymentProcessed events and handles shipping.

Each Microservice Includes:

Kafka Consumer Background Service
Subscribes to relevant Kafka topics and handles incoming events.
Kafka Producer Service
Publishes events to Kafka topics to trigger downstream service actions.
Controller/Endpoints (Optional)
For internal testing, admin access, or debug operations.
Domain Models (Shared)
All services reference a shared contracts library to ensure consistent event models.

public class OrderPlaced
{
    public string OrderId { get; set; }
    public string ProductId { get; set; }
    public int Quantity { get; set; }
    public DateTime OrderDate { get; set; }
}

public class InventoryReserved
{
    public string OrderId { get; set; }
}

public class InventoryReservationFailed
{
    public string OrderId { get; set; }
    public string Reason { get; set; }
}

public class PaymentProcessed
{
    public string OrderId { get; set; }
}

public class PaymentFailed
{
    public string OrderId { get; set; }
    public string Reason { get; set; }
}

Kafka Producer Service Each service needs a way to publish events to Kafka. This reusable service wraps the Kafka producer logic.

// KafkaProducerService.cs
public interface IKafkaProducerService
{
    Task ProduceAsync<T>(string topic, T message);
}

public class KafkaProducerService : IKafkaProducerService
{
    private readonly IProducer<Null, string> _producer;

    public KafkaProducerService(IConfiguration configuration)
    {
        var config = new ProducerConfig
        {
            BootstrapServers = configuration["Kafka:BootstrapServers"]
        };
        _producer = new ProducerBuilder<Null, string>(config).Build();
    }

    public async Task ProduceAsync<T>(string topic, T message)
    {
        var json = JsonSerializer.Serialize(message);
        await _producer.ProduceAsync(topic, new Message<Null, string> { Value = json });
    }
}

Kafka Consumer BackgroundService Each service implements a BackgroundService that listens for specific events.

Example for InventoryService consuming OrderPlaced:

// OrderPlacedConsumer.cs
public class OrderPlacedConsumer : BackgroundService
{
    private readonly IConfiguration _configuration;
    private readonly IKafkaProducerService _producer;

    public OrderPlacedConsumer(IConfiguration configuration, IKafkaProducerService producer)
    {
        _configuration = configuration;
        _producer = producer;
    }

    protected override async Task ExecuteAsync(CancellationToken stoppingToken)
    {
        var config = new ConsumerConfig
        {
            BootstrapServers = _configuration["Kafka:BootstrapServers"],
            GroupId = "inventory-service",
            AutoOffsetReset = AutoOffsetReset.Earliest
        };

        using var consumer = new ConsumerBuilder<Ignore, string>(config).Build();
        consumer.Subscribe("order-placed");

        while (!stoppingToken.IsCancellationRequested)
        {
            var result = consumer.Consume(stoppingToken);
            var order = JsonSerializer.Deserialize<OrderPlaced>(result.Message.Value);

            // Simulated logic
            var isAvailable = true; // simulate stock check

            if (isAvailable)
            {
                await _producer.ProduceAsync("inventory-reserved", new InventoryReserved { OrderId = order.OrderId });
            }
            else
            {
                await _producer.ProduceAsync("inventory-failed", new InventoryReservationFailed
                {
                    OrderId = order.OrderId,
                    Reason = "Out of stock"
                });
            }
        }
    }
}

Payment Service Consumer

public class InventoryReservedConsumer : BackgroundService
{
    private readonly IKafkaProducerService _producer;
    private readonly IConfiguration _configuration;

    public InventoryReservedConsumer(IKafkaProducerService producer, IConfiguration configuration)
    {
        _producer = producer;
        _configuration = configuration;
    }

    protected override async Task ExecuteAsync(CancellationToken stoppingToken)
    {
        var config = new ConsumerConfig
        {
            BootstrapServers = _configuration["Kafka:BootstrapServers"],
            GroupId = "payment-service",
            AutoOffsetReset = AutoOffsetReset.Earliest
        };

        using var consumer = new ConsumerBuilder<Ignore, string>(config).Build();
        consumer.Subscribe("inventory-reserved");

        while (!stoppingToken.IsCancellationRequested)
        {
            var result = consumer.Consume(stoppingToken);
            var evt = JsonSerializer.Deserialize<InventoryReserved>(result.Message.Value);

            // Simulated payment logic
            var success = true;

            if (success)
            {
                await _producer.ProduceAsync("payment-processed", new PaymentProcessed { OrderId = evt.OrderId });
            }
            else
            {
                await _producer.ProduceAsync("payment-failed", new PaymentFailed
                {
                    OrderId = evt.OrderId,
                    Reason = "Card declined"
                });
            }
        }
    }
}

Conclusion

Event Collaboration in EDA promotes a scalable, flexible, and resilient architecture by allowing services to communicate asynchronously and independently through events. While it introduces complexity in monitoring and debugging, with the right tools and practices, choreography can be a powerful pattern for modern distributed systems.

Mediator Pattern in Event-Driven Architecture (EDA)

Dinesh Dunukedeniya — Fri, 23 May 2025 12:00:24 +0000

In Event-Driven Architecture (EDA), microservices communicate by publishing and subscribing to events, enabling loose coupling and asynchronous processing. However, as systems grow in complexity, managing interactions across numerous services can become challenging.

The Mediator Pattern introduces an intermediary component — the mediator — that manages and orchestrates this communication. Rather than having services directly consume and respond to each other’s events, a dedicated mediator service listens to incoming events, applies business rules, and then routes or transforms messages as needed before passing them on to downstream services.

This pattern is sometimes referred to as the orchestration pattern or a central workflow engine, where a single component controls and coordinates the interactions between services.

This pattern is particularly useful in complex business workflows involving multiple microservices. By centralizing coordination logic within the mediator, you gain better control over the execution flow, simplify service responsibilities, reduce inter-service dependencies, and improve overall system maintainability.

When to Use the Mediator Pattern

✅ Complex workflows requiring sequencing and coordination.Especially useful in implementing the Saga Pattern or multi-step business processes involving multiple services

✅ Centralized logging, monitoring, and tracing. Gain visibility into the entire workflow execution from a single place, useful for debugging and observability.

✅ Reducing point-to-point event subscriptions between services
Avoid a tangled web of services subscribing to each other’s events, which becomes hard to manage and evolve.

✅ Business rules that span multiple services. Keep cross-cutting concerns (e.g., validation, routing decisions) in one place rather than duplicating them.

✅ Dynamic or conditional logic between steps. When the path of execution depends on business logic or the results of previous steps (e.g., retry logic, fallback strategies).

✅ Orchestrated error handling and compensations.Centralize retry strategies, failure handling, and compensating actions in distributed transactions.

✅ Auditing and compliance. A mediator can capture and persist event flows required for regulatory purposes.

Avoid

⚠️ Avoid if your system has simple, independent events.

⚠️ High-throughput, low-latency systems. Mediators can introduce bottlenecks or delays if not designed with performance in mind.

⚠️ Use cases better suited to choreography. For loosely coupled systems where services can autonomously react to events, a pure event-driven approach with no central orchestrator may be more appropriate.

Implementing the Mediator Pattern in .NET Microservices with Kafka

Let’s walk through how to implement the Mediator Pattern using Kafka in a .NET-based microservices architecture. In our setup, all microservices are web apps, meaning that to consume Kafka events, we need to use background services (hosted services) within each app to listen and process messages.

Architecture Overview

Microservices Involved:

Client Web App
Initiates the workflow by publishing OrderPlaced events.
Mediator Service
Acts as the central orchestrator. It reacts to events, applies business rules, and triggers the next step by publishing new events.
Inventory Service
Listens for ReserveInventory events and responds with either InventoryReserved or InventoryReservationFailed.
Payment Service
Listens for ProcessPayment events and responds with PaymentProcessed or PaymentFailed.
Shipping Service
Listens for OrderConfirmed events and proceeds with shipping logistics.

Design Principles

Decoupling: Services do not communicate directly; all communication flows through Kafka topics.
Orchestration: Only the Mediator knows the full business workflow. Each other service handles only its own concern.
Background Tasks: Since all services are ASP.NET Core web apps, Kafka consumers are implemented using BackgroundService.

Each Microservice Includes:

Kafka Consumer Background Task
Subscribes to relevant topics and handles incoming events.
Kafka Producer Service
Sends events to Kafka topics to trigger actions in other services.
Controller/Endpoints (if needed)
Mostly for internal APIs or admin/debug access. Not used for orchestration logic.

1. Domain Models

public record OrderPlaced(Guid OrderId, List<string> Items, PaymentInfo Payment);
public record ReserveInventory(Guid OrderId, List<string> Items);
public record InventoryReserved(Guid OrderId);
public record InventoryReservationFailed(Guid OrderId, string Reason);
public record ProcessPayment(Guid OrderId, PaymentInfo Payment);
public record PaymentProcessed(Guid OrderId);
public record PaymentFailed(Guid OrderId, string Reason);
public record OrderConfirmed(Guid OrderId);

public record PaymentInfo(string CardNumber, decimal Amount);

2. Kafka Producer Helper

public class KafkaProducer
{
    private readonly IProducer<string, string> _producer;

    public KafkaProducer(string bootstrapServers)
    {
        var config = new ProducerConfig { BootstrapServers = bootstrapServers };
        _producer = new ProducerBuilder<string, string>(config).Build();
    }

    public async Task ProduceAsync<T>(string topic, T message)
    {
        var json = JsonConvert.SerializeObject(message);
        await _producer.ProduceAsync(topic, new Message<string, string> { Key = Guid.NewGuid().ToString(), Value = json });
    }
}

3. Mediator Service Background Consumers

OrderPlacedConsumer.cs

public class OrderPlacedConsumer : BackgroundService
{
    private readonly IConsumer<string, string> _consumer;
    private readonly KafkaProducer _producer;
    private readonly ILogger<OrderPlacedConsumer> _logger;

    public OrderPlacedConsumer(KafkaProducer producer, ILogger<OrderPlacedConsumer> logger)
    {
        _producer = producer;
        _logger = logger;

        var config = new ConsumerConfig
        {
            BootstrapServers = "localhost:9092",
            GroupId = "mediator-group",
            AutoOffsetReset = AutoOffsetReset.Earliest
        };
        _consumer = new ConsumerBuilder<string, string>(config).Build();
        _consumer.Subscribe("OrderPlaced");
    }

    protected override Task ExecuteAsync(CancellationToken stoppingToken)
    {
        return Task.Run(async () =>
        {
            while (!stoppingToken.IsCancellationRequested)
            {
                var cr = _consumer.Consume(stoppingToken);
                var orderPlaced = JsonConvert.DeserializeObject<OrderPlaced>(cr.Message.Value);
                _logger.LogInformation($"OrderPlaced received: {orderPlaced.OrderId}");

                // Orchestration step: send ReserveInventory event
                var reserveInventory = new ReserveInventory(orderPlaced.OrderId, orderPlaced.Items);
                await _producer.ProduceAsync("ReserveInventory", reserveInventory);
            }
        }, stoppingToken);
    }
}

InventoryResultConsumer.cs

public class InventoryResultConsumer : BackgroundService
{
    private readonly IConsumer<string, string> _consumer;
    private readonly KafkaProducer _producer;
    private readonly ILogger<InventoryResultConsumer> _logger;

    public InventoryResultConsumer(KafkaProducer producer, ILogger<InventoryResultConsumer> logger)
    {
        _producer = producer;
        _logger = logger;

        var config = new ConsumerConfig
        {
            BootstrapServers = "localhost:9092",
            GroupId = "mediator-group",
            AutoOffsetReset = AutoOffsetReset.Earliest
        };
        _consumer = new ConsumerBuilder<string, string>(config).Build();
        _consumer.Subscribe(new[] { "InventoryReserved", "InventoryReservationFailed" });
    }

    protected override Task ExecuteAsync(CancellationToken stoppingToken)
    {
        return Task.Run(async () =>
        {
            while (!stoppingToken.IsCancellationRequested)
            {
                var cr = _consumer.Consume(stoppingToken);
                var message = cr.Message.Value;

                if (message.Contains("InventoryReservationFailed"))
                {
                    var fail = JsonConvert.DeserializeObject<InventoryReservationFailed>(message);
                    _logger.LogWarning($"Inventory reservation failed for Order {fail.OrderId}: {fail.Reason}");
                    // Handle failure (notify, rollback, etc.)
                }
                else
                {
                    var reserved = JsonConvert.DeserializeObject<InventoryReserved>(message);
                    _logger.LogInformation($"Inventory reserved for Order {reserved.OrderId}");

                    // Retrieve payment info from some store (simulate here)
                    var paymentInfo = new PaymentInfo("1234-5678-9012-3456", 100m);

                    var processPayment = new ProcessPayment(reserved.OrderId, paymentInfo);
                    await _producer.ProduceAsync("ProcessPayment", processPayment);
                }
            }
        }, stoppingToken);
    }
}

PaymentResultConsumer.cs

public class PaymentResultConsumer : BackgroundService
{
    private readonly IConsumer<string, string> _consumer;
    private readonly KafkaProducer _producer;
    private readonly ILogger<PaymentResultConsumer> _logger;

    public PaymentResultConsumer(KafkaProducer producer, ILogger<PaymentResultConsumer> logger)
    {
        _producer = producer;
        _logger = logger;

        var config = new ConsumerConfig
        {
            BootstrapServers = "localhost:9092",
            GroupId = "mediator-group",
            AutoOffsetReset = AutoOffsetReset.Earliest
        };
        _consumer = new ConsumerBuilder<string, string>(config).Build();
        _consumer.Subscribe(new[] { "PaymentProcessed", "PaymentFailed" });
    }

    protected override Task ExecuteAsync(CancellationToken stoppingToken)
    {
        return Task.Run(async () =>
        {
            while (!stoppingToken.IsCancellationRequested)
            {
                var cr = _consumer.Consume(stoppingToken);
                var message = cr.Message.Value;

                if (message.Contains("PaymentFailed"))
                {
                    var fail = JsonConvert.DeserializeObject<PaymentFailed>(message);
                    _logger.LogWarning($"Payment failed for Order {fail.OrderId}: {fail.Reason}");
                    // Handle failure (e.g., rollback inventory)
                }
                else
                {
                    var processed = JsonConvert.DeserializeObject<PaymentProcessed>(message);
                    _logger.LogInformation($"Payment processed for Order {processed.OrderId}");

                    var orderConfirmed = new OrderConfirmed(processed.OrderId);
                    await _producer.ProduceAsync("OrderConfirmed", orderConfirmed);
                }
            }
        }, stoppingToken);
    }
}

4. Program.cs (Register services and run)

var builder = WebApplication.CreateBuilder(args);

builder.Services.AddSingleton(new KafkaProducer("localhost:9092"));
builder.Services.AddHostedService<OrderPlacedConsumer>();
builder.Services.AddHostedService<InventoryResultConsumer>();
builder.Services.AddHostedService<PaymentResultConsumer>();

var app = builder.Build();
app.Run();

Summary

The Mediator service has three background consumers, each processing different event types
Each consumer applies the orchestration logic and triggers the next step by producing the next event
This keeps your microservices loosely coupled while having a central place for business workflow orchestration
You can extend this with state management (e.g., Redis, DB) for reliability and retries.

DEV Community: Dinesh Dunukedeniya

Is Your Microservice Safe from AI Agents? Rethinking Security in the Age of Agentic AI

The Invisible Agent Problem

Why Existing Controls Don't Catch This

Why More Permissions Are Not the Answer

Rethinking Security: Bound the Task, Then Obtain Intent

From Mandate to Approval: Two Layers of Intent Enforcement

Using CIBA for Independent User Authorisation

Example: Authorising a Share Transaction

Transaction-Specific Authorisation Is Already Emerging

Binding Approval to the Transaction

Risk-Based Authorisation

Unintended Behaviour Versus Compromised Behaviour

Architectural Considerations for Existing Microservices

What About Traditional Security Controls?

Potential Objections

Conclusion

Event/Message Versioning in : The Complete Practical Guide

Boundaries: The Real Foundation of Any Modern Architecture (Microservices or Otherwise)

What a strong boundary really owns

What good boundaries deliver day-to-day

The warning signs of weak boundaries

Practical ways to get boundaries right

Real-World Examples of Finding (and Refining) Boundaries

The takeaway

Recommended Links

Why Asynchronous Messaging Beats REST in Modern Architectures

🔹 1. Loose Coupling vs Tight Coupling

🔹 2. Push vs Pull: Control Matters

🔹 3. Scalability at Its Core

🔹 4. Built-in Reliability

🔹 5. Event-Driven Power

🔹 6. Future-Proof Flexibility

⚖️ The Trade-offs: Why Async Isn’t Always Better

🚀 Final Thought

How to Choose the Right Microservice Boundaries

Start with the Business Domain (Domain-Driven Design)

Look at Change Patterns

Align to Team Boundaries (Two-Pizza Rule 🍕)

Data Ownership Is Non-Negotiable

Watch for Chatty Communication

Start Bigger, Split Later

Summary

Microservice-to-Microservice Authentication in .NET Using IdentityServer and JWT

Why Client Credentials Flow?

Trusted Services and Secure Environment

Token Flow Diagram

IdentityServer Setup

Requesting a Token in .NET

Calling Another Microservice with JWT

Validating JWT in the Receiving Service

Optimizing with Token Caching

Automating Token Injection with DelegatingHandler

Best Practices for Production

Summary

Fixing Microservice Dependencies with Self-Contained Events (aka ECST)

The Problem: API-Based Microservice Dependency

Scenario: Payment Service Needs Customer Info

The Fix: ECST (Self-Contained Events)

Now it looks like this:

Benefits of This Approach

Bonus Tip: Design Events with Purpose

Conclusion

Fixing Microservice Communication: From Fragile Calls to Resilient Systems

The Problem with Direct Service Calls

How to Fix These Problems: Best Practices and Patterns

1. Favor Loose Coupling Through Asynchronous Messaging

2. Add Resilience: Timeouts, Retries, and Circuit Breakers

3. Implement Graceful Fallbacks

4. Improve Observability

Conclusion

Designing E-Commerce Order Processing: Orchestration vs Choreography

Orchestration: The Centralized Conductor

Event-Driven Choreography: The Decentralized Dance

Orchestration vs Event Collaboration for E-commerce Applications: Which One Fits Your Needs?

Orchestration: Centralized Control for Complex Workflows

Event Collaboration (Choreography): Loose Coupling for Scalability

Summary Table

Final Thoughts

Orchestration vs. Choreography in EDA: Choosing the Right Approach