DEV Community: Hossein Esmati

Event Sourcing in .NET Core

Hossein Esmati — Fri, 26 Jun 2026 01:02:16 +0000

This article is part of the Comprehensive Guide to Microservices Architecture in .NET Core, Cloud and Azure series.

Benefits of Event Sourcing

Complete Audit Trail: Every change to the system is captured as an immutable event, providing a comprehensive history of what happened, when, and why.

Time Travel Capabilities: You can reconstruct the state of any entity at any point in time by replaying events up to that moment, invaluable for debugging and compliance.

Event Replay for Debugging: Reproduce issues by replaying the exact sequence of events that caused a problem, making debugging significantly easier.

Multiple Read Models: Generate different read models (projections) from the same event stream, optimizing queries for different use cases without duplicating write logic.

Implementation with Marten

Marten is a .NET library that provides event sourcing and document database capabilities on top of PostgreSQL. Below is a complete implementation of event sourcing for an order management system.

Installing Marten

dotnet add package Marten

Defining the Aggregate Root

The aggregate root applies events to build its state:

public class Order
{
    public Guid Id { get; private set; }
    public Guid CustomerId { get; private set; }
    public OrderStatus Status { get; private set; }
    public List<OrderLine> Lines { get; private set; } = new();
    public decimal TotalAmount { get; private set; }

    // Event application methods
    public void Apply(OrderCreated @event)
    {
        Id = @event.OrderId;
        CustomerId = @event.CustomerId;
        Status = OrderStatus.Created;
    }

    public void Apply(OrderLineAdded @event)
    {
        Lines.Add(new OrderLine
        {
            ProductId = @event.ProductId,
            Quantity = @event.Quantity,
            Price = @event.Price
        });
        TotalAmount += @event.Quantity * @event.Price;
    }

    public void Apply(OrderConfirmed @event)
    {
        Status = OrderStatus.Confirmed;
    }

    public void Apply(OrderCancelled @event)
    {
        Status = OrderStatus.Cancelled;
    }
}

Defining Events

Events are immutable records that represent state changes:

public record OrderCreated(Guid OrderId, Guid CustomerId, DateTime CreatedAt);
public record OrderLineAdded(Guid ProductId, int Quantity, decimal Price);
public record OrderConfirmed(DateTime ConfirmedAt);
public record OrderCancelled(string Reason, DateTime CancelledAt);

Configuring Marten

Set up Marten with event sourcing and projection support:

builder.Services.AddMarten(options =>
{
    options.Connection(builder.Configuration.GetConnectionString("Marten"));

    // Register event types
    options.Events.AddEventType<OrderCreated>();
    options.Events.AddEventType<OrderLineAdded>();
    options.Events.AddEventType<OrderConfirmed>();
    options.Events.AddEventType<OrderCancelled>();

    // Configure projections for read models
    options.Projections.Add<OrderProjection>(ProjectionLifecycle.Inline);
});

Writing Events with Command Handlers

Command handlers append events to the event stream:

public class CreateOrderHandler : IRequestHandler<CreateOrderCommand, Guid>
{
    private readonly IDocumentSession _session;

    public CreateOrderHandler(IDocumentSession session)
    {
        _session = session;
    }

    public async Task<Guid> Handle(
        CreateOrderCommand request, 
        CancellationToken cancellationToken)
    {
        var orderId = Guid.NewGuid();

        // Start a new event stream for the order
        _session.Events.StartStream<Order>(
            orderId,
            new OrderCreated(orderId, request.CustomerId, DateTime.UtcNow)
        );

        // Append additional events to the stream
        foreach (var item in request.Items)
        {
            _session.Events.Append(
                orderId,
                new OrderLineAdded(item.ProductId, item.Quantity, item.Price)
            );
        }

        await _session.SaveChangesAsync(cancellationToken);
        return orderId;
    }
}

Reading State with Query Handlers

Query handlers rebuild aggregates by replaying events:

public class GetOrderHandler : IRequestHandler<GetOrderQuery, Order>
{
    private readonly IDocumentSession _session;

    public GetOrderHandler(IDocumentSession session)
    {
        _session = session;
    }

    public async Task<Order> Handle(
        GetOrderQuery request,
        CancellationToken cancellationToken)
    {
        // Marten automatically rebuilds the aggregate from its event stream
        var order = await _session.Events
            .AggregateStreamAsync<Order>(request.OrderId, token: cancellationToken);

        return order;
    }
}

Creating Projections for Read Models

Projections transform events into optimized read models:

public class OrderProjection : EventProjection
{
    // Create initial read model from first event
    public OrderReadModel Create(OrderCreated @event)
    {
        return new OrderReadModel
        {
            Id = @event.OrderId,
            CustomerId = @event.CustomerId,
            Status = "Created",
            CreatedAt = @event.CreatedAt
        };
    }

    // Update read model as new events occur
    public void Apply(OrderLineAdded @event, OrderReadModel model)
    {
        model.ItemCount++;
        model.TotalAmount += @event.Quantity * @event.Price;
    }

    public void Apply(OrderConfirmed @event, OrderReadModel model)
    {
        model.Status = "Confirmed";
        model.ConfirmedAt = @event.ConfirmedAt;
    }
}

Time Travel Queries

One of the most powerful features of event sourcing is the ability to query historical state:

public async Task<Order> GetOrderAtPointInTime(Guid orderId, DateTime timestamp)
{
    // Replay events only up to the specified timestamp
    return await _session.Events
        .AggregateStreamAsync<Order>(
            orderId,
            timestamp: timestamp);
}

Key Considerations

When to Use Event Sourcing:

Systems requiring complete audit trails (financial, healthcare, compliance)
Applications needing temporal queries and historical analysis
Domains with complex business logic where understanding the "why" matters
Systems that benefit from multiple read models

Challenges:

Increased complexity compared to traditional CRUD
Event schema evolution requires careful planning
Higher storage requirements due to event retention
Learning curve for developers unfamiliar with the pattern

Best Practices:

Keep events immutable and focused on business facts
Version your events to handle schema changes
Use projections for query optimization
Implement snapshots for aggregates with long event histories
Consider eventual consistency implications

Distributed Transactions in .NET Core and Azure

Hossein Esmati — Fri, 26 Jun 2026 01:02:11 +0000

This article is part of the Comprehensive Guide to Microservices Architecture in .NET Core, Cloud and Azure series.

The Saga Pattern

What is a Saga?

A sequence of local transactions where each transaction updates data within a single service. If a step fails, compensating transactions undo the changes to maintain data consistency across distributed systems.

Two Types:

Choreography: Each service produces and listens to events (decentralized)
Orchestration: A central coordinator directs the saga (centralized)

Example with Orchestration (using MassTransit)

// Saga State (using primary constructors - C# 12+)
public class OrderSagaState : SagaStateMachineInstance
{
    public required Guid CorrelationId { get; set; }
    public required string CurrentState { get; set; }
    public required Guid OrderId { get; set; }
    public Guid? PaymentId { get; set; }
    public Guid? ShipmentId { get; set; }
    public DateTimeOffset CreatedAt { get; set; } = DateTimeOffset.UtcNow;
}

// Saga State Machine
public class OrderSagaStateMachine : MassTransitStateMachine<OrderSagaState>
{
    public State OrderCreated { get; private set; } = null!;
    public State PaymentProcessing { get; private set; } = null!;
    public State PaymentCompleted { get; private set; } = null!;
    public State ShipmentScheduled { get; private set; } = null!;
    public State Completed { get; private set; } = null!;
    public State Failed { get; private set; } = null!;

    public Event<OrderCreatedEvent> OrderCreatedEvent { get; private set; } = null!;
    public Event<PaymentCompletedEvent> PaymentCompletedEvent { get; private set; } = null!;
    public Event<PaymentFailedEvent> PaymentFailedEvent { get; private set; } = null!;
    public Event<ShipmentScheduledEvent> ShipmentScheduledEvent { get; private set; } = null!;

    public OrderSagaStateMachine()
    {
        InstanceState(x => x.CurrentState);

        Initially(
            When(OrderCreatedEvent)
                .Then(context =>
                {
                    context.Saga.OrderId = context.Message.OrderId;
                    context.Saga.CreatedAt = DateTimeOffset.UtcNow;
                })
                .TransitionTo(PaymentProcessing)
                .PublishAsync(context => context.Init<ProcessPaymentCommand>(new
                {
                    context.Message.OrderId,
                    context.Message.TotalAmount,
                    CorrelationId = context.Saga.CorrelationId
                }))
        );

        During(PaymentProcessing,
            When(PaymentCompletedEvent)
                .Then(context =>
                {
                    context.Saga.PaymentId = context.Message.PaymentId;
                })
                .TransitionTo(PaymentCompleted)
                .PublishAsync(context => context.Init<ScheduleShipmentCommand>(new
                {
                    OrderId = context.Saga.OrderId,
                    CorrelationId = context.Saga.CorrelationId
                })),

            When(PaymentFailedEvent)
                .TransitionTo(Failed)
                .PublishAsync(context => context.Init<CancelOrderCommand>(new
                {
                    OrderId = context.Saga.OrderId,
                    Reason = "Payment failed",
                    CorrelationId = context.Saga.CorrelationId
                }))
        );

        During(PaymentCompleted,
            When(ShipmentScheduledEvent)
                .Then(context =>
                {
                    context.Saga.ShipmentId = context.Message.ShipmentId;
                })
                .TransitionTo(Completed)
                .Finalize()
        );

        SetCompletedWhenFinalized();
    }
}

// Registration with Azure Service Bus (in Program.cs)
builder.Services.AddMassTransit(x =>
{
    x.AddSagaStateMachine<OrderSagaStateMachine, OrderSagaState>()
        .EntityFrameworkRepository(r =>
        {
            r.ConcurrencyMode = ConcurrencyMode.Optimistic;
            r.AddDbContext<DbContext, SagaDbContext>((provider, cfg) =>
            {
                cfg.UseSqlServer(builder.Configuration.GetConnectionString("SagaDb"));
            });
        });

    x.UsingAzureServiceBus((context, cfg) =>
    {
        cfg.Host(builder.Configuration["AzureServiceBus:ConnectionString"]);
        cfg.ConfigureEndpoints(context);
    });
});

Tools for Saga Pattern:

MassTransit (v8.2+): State machine-based sagas with Azure Service Bus
Dapr Workflow (v1.14+): Actor-based workflows with state management
Azure Durable Functions: Serverless orchestration with built-in state persistence
NServiceBus (v9+): Commercial option with advanced saga features
Wolverine (v3+): Modern messaging with saga support

The Outbox Pattern

What is the Outbox Pattern?

Ensures reliable event publishing by saving events to a database table (the "outbox") in the same transaction as business data, then publishing them asynchronously. This guarantees at-least-once delivery.

Implementation with EF Core 9

// Outbox Message
public sealed record OutboxMessage
{
    public required Guid Id { get; init; }
    public required string Type { get; init; }
    public required string Payload { get; init; }
    public required DateTimeOffset CreatedAt { get; init; }
    public DateTimeOffset? ProcessedAt { get; set; }
    public int RetryCount { get; set; }
    public string? Error { get; set; }
}

// Service with Outbox
public class OrderService(AppDbContext context, ILogger<OrderService> logger)
{
    public async Task CreateOrderAsync(CreateOrderCommand command, CancellationToken ct = default)
    {
        // EF Core 9: Improved transaction handling
        var strategy = context.Database.CreateExecutionStrategy();

        await strategy.ExecuteAsync(async () =>
        {
            await using var transaction = await context.Database.BeginTransactionAsync(ct);

            try
            {
                // Save business entity
                var order = new Order(command.CustomerId, command.Items);
                context.Orders.Add(order);

                // Save event to outbox
                var evt = new OrderCreatedEvent(order.Id, order.CustomerId, order.Items);
                var outboxMessage = new OutboxMessage
                {
                    Id = Guid.NewGuid(),
                    Type = evt.GetType().AssemblyQualifiedName!,
                    Payload = JsonSerializer.Serialize(evt, JsonSerializerOptions.Default),
                    CreatedAt = DateTimeOffset.UtcNow
                };

                context.OutboxMessages.Add(outboxMessage);

                await context.SaveChangesAsync(ct);
                await transaction.CommitAsync(ct);

                logger.LogInformation("Order {OrderId} created with outbox message {MessageId}", 
                    order.Id, outboxMessage.Id);
            }
            catch (Exception ex)
            {
                await transaction.RollbackAsync(ct);
                logger.LogError(ex, "Failed to create order");
                throw;
            }
        });
    }
}

// Outbox Processor using IHostedLifecycleService (.NET 8+)
public class OutboxProcessor(
    IServiceProvider serviceProvider,
    ILogger<OutboxProcessor> logger) : BackgroundService
{
    private readonly PeriodicTimer _timer = new(TimeSpan.FromSeconds(5));

    protected override async Task ExecuteAsync(CancellationToken stoppingToken)
    {
        logger.LogInformation("Outbox processor started");

        while (!stoppingToken.IsCancellationRequested && 
               await _timer.WaitForNextTickAsync(stoppingToken))
        {
            await ProcessOutboxMessagesAsync(stoppingToken);
        }
    }

    private async Task ProcessOutboxMessagesAsync(CancellationToken ct)
    {
        await using var scope = serviceProvider.CreateAsyncScope();
        var context = scope.ServiceProvider.GetRequiredService<AppDbContext>();
        var publisher = scope.ServiceProvider.GetRequiredService<IEventPublisher>();

        // EF Core 9: ExecuteUpdate for better performance
        var messages = await context.OutboxMessages
            .Where(m => m.ProcessedAt == null && m.RetryCount < 5)
            .OrderBy(m => m.CreatedAt)
            .Take(100)
            .ToListAsync(ct);

        foreach (var message in messages)
        {
            try
            {
                await publisher.PublishAsync(message.Type, message.Payload, ct);
                message.ProcessedAt = DateTimeOffset.UtcNow;

                logger.LogInformation("Published outbox message {MessageId}", message.Id);
            }
            catch (Exception ex)
            {
                message.RetryCount++;
                message.Error = ex.Message;
                logger.LogError(ex, "Failed to publish message {MessageId}, retry {RetryCount}", 
                    message.Id, message.RetryCount);
            }
        }

        await context.SaveChangesAsync(ct);

        // Clean up old processed messages (older than 7 days)
        await context.OutboxMessages
            .Where(m => m.ProcessedAt != null && 
                        m.ProcessedAt < DateTimeOffset.UtcNow.AddDays(-7))
            .ExecuteDeleteAsync(ct);
    }

    public override void Dispose()
    {
        _timer.Dispose();
        base.Dispose();
    }
}

Azure Integration

// Using Azure Service Bus with Outbox
public class AzureServiceBusPublisher(ServiceBusClient client) : IEventPublisher
{
    public async Task PublishAsync(string eventType, string payload, CancellationToken ct = default)
    {
        var sender = client.CreateSender("events");
        var message = new ServiceBusMessage(payload)
        {
            ContentType = "application/json",
            Subject = eventType,
            MessageId = Guid.NewGuid().ToString()
        };

        await sender.SendMessageAsync(message, ct);
    }
}

// Registration
builder.Services.AddSingleton(sp =>
{
    var connectionString = builder.Configuration["AzureServiceBus:ConnectionString"]!;
    return new ServiceBusClient(connectionString);
});
builder.Services.AddSingleton<IEventPublisher, AzureServiceBusPublisher>();
builder.Services.AddHostedService<OutboxProcessor>();

Tools:

Entity Framework Core 9: ExecuteUpdate/Delete, complex types, improved performance
MassTransit (v8.2+): Built-in outbox with automatic publishing
Wolverine: Transactional messaging with PostgreSQL
Azure Service Bus: Reliable message delivery
Azure SQL Database: Outbox storage with geo-replication

Eventual Consistency

What is Eventual Consistency?

In distributed systems, data across services may be temporarily inconsistent but will eventually converge to a consistent state. This is a trade-off for availability and partition tolerance (CAP theorem).

Managing with CQRS Read Models

// Read model with complex type (EF Core 9)
public sealed record OrderReadModel
{
    public required Guid OrderId { get; init; }
    public required string Status { get; set; }
    public required decimal TotalAmount { get; init; }
    public required string PaymentStatus { get; set; }
    public required Address ShippingAddress { get; init; }
    public required DateTimeOffset LastUpdated { get; set; }
    public TimeSpan? ProcessingTime => 
        Status == "Completed" ? LastUpdated - CreatedAt : null;

    private DateTimeOffset CreatedAt { get; init; }
}

// Complex type (EF Core 8+)
public record Address
{
    public required string Street { get; init; }
    public required string City { get; init; }
    public required string PostalCode { get; init; }
}

// Event handlers with primary constructor
public class OrderReadModelUpdater(
    IRepository<OrderReadModel> repository,
    ILogger<OrderReadModelUpdater> logger) :
    IEventHandler<OrderCreatedEvent>,
    IEventHandler<PaymentCompletedEvent>,
    IEventHandler<ShipmentScheduledEvent>
{
    public async Task Handle(OrderCreatedEvent evt, CancellationToken ct = default)
    {
        var model = new OrderReadModel
        {
            OrderId = evt.OrderId,
            Status = "Created",
            TotalAmount = evt.TotalAmount,
            PaymentStatus = "Pending",
            ShippingAddress = new Address
            {
                Street = evt.ShippingAddress.Street,
                City = evt.ShippingAddress.City,
                PostalCode = evt.ShippingAddress.PostalCode
            },
            LastUpdated = DateTimeOffset.UtcNow
        };

        await repository.UpsertAsync(model, ct);
        logger.LogInformation("Order read model created for {OrderId}", evt.OrderId);
    }

    public async Task Handle(PaymentCompletedEvent evt, CancellationToken ct = default)
    {
        var model = await repository.GetByIdAsync(evt.OrderId, ct);

        if (model is null)
        {
            logger.LogWarning("Order {OrderId} not found in read model", evt.OrderId);
            return;
        }

        model.PaymentStatus = "Completed";
        model.Status = "Paid";
        model.LastUpdated = DateTimeOffset.UtcNow;

        await repository.UpdateAsync(model, ct);
    }

    public async Task Handle(ShipmentScheduledEvent evt, CancellationToken ct = default)
    {
        var model = await repository.GetByIdAsync(evt.OrderId, ct);

        if (model is null)
        {
            logger.LogWarning("Order {OrderId} not found in read model", evt.OrderId);
            return;
        }

        model.Status = "Shipped";
        model.LastUpdated = DateTimeOffset.UtcNow;

        await repository.UpdateAsync(model, ct);
    }
}

Azure Cosmos DB for Read Models

// Using Cosmos DB for eventual consistency scenarios
public class CosmosOrderReadModelRepository(CosmosClient client) : IRepository<OrderReadModel>
{
    private readonly Container _container = client
        .GetContainer("OrdersDb", "OrderReadModels");

    public async Task UpsertAsync(OrderReadModel model, CancellationToken ct = default)
    {
        await _container.UpsertItemAsync(
            model,
            new PartitionKey(model.OrderId.ToString()),
            cancellationToken: ct);
    }

    public async Task<OrderReadModel?> GetByIdAsync(Guid orderId, CancellationToken ct = default)
    {
        try
        {
            var response = await _container.ReadItemAsync<OrderReadModel>(
                orderId.ToString(),
                new PartitionKey(orderId.ToString()),
                cancellationToken: ct);

            return response.Resource;
        }
        catch (CosmosException ex) when (ex.StatusCode == HttpStatusCode.NotFound)
        {
            return null;
        }
    }
}

Idempotent Handlers

What is Idempotency?

An operation that produces the same result no matter how many times it's executed. Critical for handling message redeliveries and retries.

Read more: A practical, "ship-it" guide to idempotency keys, request hashing, persistence, PUT semantics, and de-dup in handlers/queues on .NET + Azure

Implementation with Database Tracking

// Using record for processed message tracking
public sealed record ProcessedMessage
{
    public required string MessageId { get; init; }
    public required DateTimeOffset ProcessedAt { get; init; }
    public required string HandlerType { get; init; }
}

// Idempotent event handler
public class IdempotentEventHandler(AppDbContext context, ILogger<IdempotentEventHandler> logger)
{
    public async Task Handle(OrderCreatedEvent evt, string messageId, CancellationToken ct = default)
    {
        // Check if already processed
        var exists = await context.ProcessedMessages
            .AnyAsync(m => m.MessageId == messageId, ct);

        if (exists)
        {
            logger.LogInformation("Message {MessageId} already processed, skipping", messageId);
            return;
        }

        var strategy = context.Database.CreateExecutionStrategy();

        await strategy.ExecuteAsync(async () =>
        {
            await using var transaction = await context.Database.BeginTransactionAsync(ct);

            try
            {
                // Process event
                var inventory = await context.Inventory
                    .FirstAsync(i => i.ProductId == evt.ProductId, ct);

                inventory.Reserve(evt.Quantity);

                // Record message as processed
                context.ProcessedMessages.Add(new ProcessedMessage
                {
                    MessageId = messageId,
                    ProcessedAt = DateTimeOffset.UtcNow,
                    HandlerType = nameof(IdempotentEventHandler)
                });

                await context.SaveChangesAsync(ct);
                await transaction.CommitAsync(ct);

                logger.LogInformation("Message {MessageId} processed successfully", messageId);
            }
            catch (Exception ex)
            {
                await transaction.RollbackAsync(ct);
                logger.LogError(ex, "Failed to process message {MessageId}", messageId);
                throw;
            }
        });
    }
}

Azure Redis Cache for Idempotency

// Using Azure Redis for distributed idempotency tracking
public class RedisIdempotencyChecker(
    IConnectionMultiplexer redis,
    ILogger<RedisIdempotencyChecker> logger)
{
    private readonly IDatabase _db = redis.GetDatabase();

    public async Task<bool> TryProcessAsync(
        string messageId, 
        Func<Task> handler,
        CancellationToken ct = default)
    {
        var key = $"processed:{messageId}";

        // Try to set the key with NX (only if not exists)
        var wasSet = await _db.StringSetAsync(
            key, 
            DateTimeOffset.UtcNow.ToString("O"),
            expiry: TimeSpan.FromDays(7),
            when: When.NotExists);

        if (!wasSet)
        {
            logger.LogInformation("Message {MessageId} already processed", messageId);
            return false;
        }

        try
        {
            await handler();
            return true;
        }
        catch
        {
            // Remove the key on failure to allow retry
            await _db.KeyDeleteAsync(key);
            throw;
        }
    }
}

// Usage
public class OrderEventHandler(
    RedisIdempotencyChecker idempotencyChecker,
    IOrderService orderService)
{
    public async Task Handle(OrderCreatedEvent evt, string messageId, CancellationToken ct)
    {
        await idempotencyChecker.TryProcessAsync(
            messageId,
            () => orderService.ProcessOrderAsync(evt, ct),
            ct);
    }
}

// Registration with Azure Cache for Redis
builder.Services.AddSingleton<IConnectionMultiplexer>(sp =>
{
    var configuration = builder.Configuration["Azure:Redis:ConnectionString"]!;
    return ConnectionMultiplexer.Connect(configuration);
});
builder.Services.AddSingleton<RedisIdempotencyChecker>();

Natural Idempotency with HTTP

// API endpoint with natural idempotency
[ApiController]
[Route("api/[controller]")]
public class OrdersController(IOrderService orderService) : ControllerBase
{
    // PUT is naturally idempotent
    [HttpPut("{orderId:guid}")]
    public async Task<IActionResult> CreateOrder(
        Guid orderId,
        [FromBody] CreateOrderRequest request,
        CancellationToken ct)
    {
        var result = await orderService.CreateOrUpdateOrderAsync(orderId, request, ct);
        return result.IsNew ? CreatedAtAction(nameof(GetOrder), new { orderId }, result.Order)
                            : Ok(result.Order);
    }

    // Idempotency key header approach
    [HttpPost]
    public async Task<IActionResult> CreateOrderWithIdempotencyKey(
        [FromBody] CreateOrderRequest request,
        [FromHeader(Name = "Idempotency-Key")] string? idempotencyKey,
        CancellationToken ct)
    {
        if (string.IsNullOrEmpty(idempotencyKey))
        {
            return BadRequest("Idempotency-Key header is required");
        }

        var result = await orderService.CreateOrderIdempotentAsync(idempotencyKey, request, ct);
        return result.IsNew ? CreatedAtAction(nameof(GetOrder), new { orderId = result.OrderId }, result.Order)
                            : Ok(result.Order);
    }

    [HttpGet("{orderId:guid}")]
    public async Task<IActionResult> GetOrder(Guid orderId, CancellationToken ct)
    {
        var order = await orderService.GetOrderAsync(orderId, ct);
        return order is null ? NotFound() : Ok(order);
    }
}

Compensating Actions

What are Compensating Actions?

Operations that undo the effects of previously completed transactions in a saga. Essential for maintaining consistency when a distributed transaction fails partway through.

Read more: Compensating Transaction pattern - Microsoft Azure Architecture

Implementation Example

// Payment service with compensation
public class PaymentService(
    IPaymentGateway paymentGateway,
    ILogger<PaymentService> logger)
{
    public async Task<PaymentResult> ProcessPaymentAsync(
        Guid orderId, 
        decimal amount,
        CancellationToken ct = default)
    {
        logger.LogInformation("Processing payment for order {OrderId}, amount {Amount}", 
            orderId, amount);

        var payment = await paymentGateway.ChargeAsync(amount, ct);

        return new PaymentResult 
        { 
            Success = true, 
            PaymentId = payment.Id,
            TransactionId = payment.TransactionId
        };
    }

    // Compensating action
    public async Task RefundPaymentAsync(
        Guid paymentId,
        string reason,
        CancellationToken ct = default)
    {
        logger.LogWarning("Refunding payment {PaymentId}, reason: {Reason}", 
            paymentId, reason);

        await paymentGateway.RefundAsync(paymentId, ct);
    }
}

// Inventory service with compensation
public class InventoryService(
    IRepository<Inventory> repository,
    ILogger<InventoryService> logger)
{
    public async Task ReserveInventoryAsync(
        Guid orderId, 
        IReadOnlyList<OrderLine> items,
        CancellationToken ct = default)
    {
        logger.LogInformation("Reserving inventory for order {OrderId}", orderId);

        foreach (var item in items)
        {
            var inventory = await repository.GetAsync(item.ProductId, ct);

            if (inventory is null || !inventory.CanReserve(item.Quantity))
            {
                throw new InsufficientInventoryException(item.ProductId, item.Quantity);
            }

            inventory.Reserve(orderId, item.Quantity);
        }

        await repository.SaveAsync(ct);
    }

    // Compensating action
    public async Task ReleaseInventoryAsync(Guid orderId, CancellationToken ct = default)
    {
        logger.LogWarning("Releasing inventory for order {OrderId}", orderId);

        var reservations = await repository.GetReservationsByOrderAsync(orderId, ct);

        foreach (var reservation in reservations)
        {
            var inventory = await repository.GetAsync(reservation.ProductId, ct);
            inventory?.Release(orderId);
        }

        await repository.SaveAsync(ct);
    }
}

// Saga with compensating actions
public class OrderSagaWithCompensation : MassTransitStateMachine<OrderSagaState>
{
    public Event<OrderCreatedEvent> OrderCreated { get; private set; } = null!;
    public Event<PaymentFailedEvent> PaymentFailed { get; private set; } = null!;
    public Event<InventoryReservationFailedEvent> InventoryFailed { get; private set; } = null!;

    public OrderSagaWithCompensation()
    {
        // ... other state configurations ...

        // Compensation flow when payment fails
        During(PaymentProcessing,
            When(PaymentFailed)
                .Then(ctx => Log.Warning("Payment failed for order {OrderId}", ctx.Saga.OrderId))
                .TransitionTo(Compensating)
                .PublishAsync(ctx => ctx.Init<ReleaseInventoryCommand>(new
                {
                    OrderId = ctx.Saga.OrderId,
                    Reason = "Payment failed"
                }))
                .PublishAsync(ctx => ctx.Init<CancelOrderCommand>(new
                {
                    OrderId = ctx.Saga.OrderId,
                    Reason = "Payment failed"
                }))
                .TransitionTo(Failed)
                .Finalize()
        );

        // Compensation flow when inventory reservation fails
        During(InventoryReservation,
            When(InventoryFailed)
                .Then(ctx => Log.Warning("Inventory reservation failed for order {OrderId}", 
                    ctx.Saga.OrderId))
                .TransitionTo(Compensating)
                .If(ctx => ctx.Saga.PaymentId.HasValue,
                    compensate => compensate
                        .PublishAsync(ctx => ctx.Init<RefundPaymentCommand>(new
                        {
                            PaymentId = ctx.Saga.PaymentId!.Value,
                            Reason = "Inventory unavailable"
                        })))
                .PublishAsync(ctx => ctx.Init<CancelOrderCommand>(new
                {
                    OrderId = ctx.Saga.OrderId,
                    Reason = "Inventory unavailable"
                }))
                .TransitionTo(Failed)
                .Finalize()
        );
    }
}

Azure Durable Functions for Compensation

[DurableTask(nameof(OrderOrchestrator))]
public class OrderOrchestrator : TaskOrchestrator<OrderResult, OrderRequest>
{
    public override async Task<OrderResult> RunAsync(
        TaskOrchestrationContext context, 
        OrderRequest input)
    {
        var compensations = new Stack<Func<Task>>();

        try
        {
            // Step 1: Reserve inventory
            var inventoryReserved = await context.CallActivityAsync<bool>(
                nameof(ReserveInventoryActivity), 
                input.OrderId);

            if (!inventoryReserved)
            {
                return OrderResult.Failed("Inventory unavailable");
            }

            compensations.Push(() => context.CallActivityAsync(
                nameof(ReleaseInventoryActivity), 
                input.OrderId));

            // Step 2: Process payment
            var paymentResult = await context.CallActivityAsync<PaymentResult>(
                nameof(ProcessPaymentActivity), 
                input);

            if (!paymentResult.Success)
            {
                await ExecuteCompensationsAsync(context, compensations);
                return OrderResult.Failed("Payment failed");
            }

            compensations.Push(() => context.CallActivityAsync(
                nameof(RefundPaymentActivity), 
                paymentResult.PaymentId));

            // Step 3: Schedule shipment
            await context.CallActivityAsync(
                nameof(ScheduleShipmentActivity), 
                input.OrderId);

            return OrderResult.Success(input.OrderId);
        }
        catch (Exception ex)
        {
            // Execute all compensating actions in reverse order
            await ExecuteCompensationsAsync(context, compensations);
            return OrderResult.Failed(ex.Message);
        }
    }

    private static async Task ExecuteCompensationsAsync(
        TaskOrchestrationContext context,
        Stack<Func<Task>> compensations)
    {
        while (compensations.Count > 0)
        {
            var compensation = compensations.Pop();
            try
            {
                await compensation();
            }
            catch (Exception ex)
            {
                // Log but continue with other compensations
                context.SetCustomStatus($"Compensation failed: {ex.Message}");
            }
        }
    }
}

Additional Patterns and Best Practices

Circuit Breaker with Polly

// Using Polly v8+ with resilience pipelines
builder.Services.AddHttpClient<IPaymentService, PaymentService>()
    .AddResilienceHandler("payment-resilience", static builder =>
    {
        builder
            .AddCircuitBreaker(new CircuitBreakerStrategyOptions
            {
                FailureRatio = 0.5,
                SamplingDuration = TimeSpan.FromSeconds(30),
                MinimumThroughput = 10,
                BreakDuration = TimeSpan.FromSeconds(30)
            })
            .AddTimeout(TimeSpan.FromSeconds(10))
            .AddRetry(new RetryStrategyOptions
            {
                MaxRetryAttempts = 3,
                Delay = TimeSpan.FromSeconds(1),
                BackoffType = DelayBackoffType.Exponential
            });
    });

Monitoring with Azure Application Insights

// Track saga execution
public class OrderSagaStateMachine : MassTransitStateMachine<OrderSagaState>
{
    public OrderSagaStateMachine(TelemetryClient telemetry)
    {
        Initially(
            When(OrderCreated)
                .Then(context =>
                {
                    telemetry.TrackEvent("OrderSagaStarted", new Dictionary<string, string>
                    {
                        ["OrderId"] = context.Message.OrderId.ToString(),
                        ["CorrelationId"] = context.Saga.CorrelationId.ToString()
                    });
                })
                // ... rest of saga logic
        );
    }
}

Summary

These patterns work together to handle distributed transactions in microservices:

Saga Pattern: Coordinates multi-step transactions across services
Outbox Pattern: Ensures reliable event publishing
Eventual Consistency: Accept temporary inconsistency for availability
Idempotent Handlers: Ensure operations can be safely retried
Compensating Actions: Roll back completed steps when failures occur

Advanced Topics

Event Sourcing with Marten

Event sourcing stores all changes as a sequence of events, providing a complete audit trail and enabling temporal queries.

// Event-sourced aggregate
public record OrderCreated(Guid OrderId, string CustomerId, decimal Total);
public record PaymentProcessed(Guid OrderId, Guid PaymentId, decimal Amount);
public record OrderShipped(Guid OrderId, string TrackingNumber);
public record OrderCancelled(Guid OrderId, string Reason);

public class Order
{
    public Guid Id { get; private set; }
    public string CustomerId { get; private set; } = string.Empty;
    public decimal Total { get; private set; }
    public string Status { get; private set; } = "Created";
    public Guid? PaymentId { get; private set; }
    public string? TrackingNumber { get; private set; }

    // Apply events to rebuild state
    public void Apply(OrderCreated evt)
    {
        Id = evt.OrderId;
        CustomerId = evt.CustomerId;
        Total = evt.Total;
        Status = "Created";
    }

    public void Apply(PaymentProcessed evt)
    {
        PaymentId = evt.PaymentId;
        Status = "Paid";
    }

    public void Apply(OrderShipped evt)
    {
        TrackingNumber = evt.TrackingNumber;
        Status = "Shipped";
    }

    public void Apply(OrderCancelled evt)
    {
        Status = "Cancelled";
    }
}

// Using Marten for event sourcing
public class OrderService(IDocumentSession session)
{
    public async Task<Guid> CreateOrderAsync(
        string customerId, 
        decimal total,
        CancellationToken ct = default)
    {
        var orderId = Guid.NewGuid();
        var evt = new OrderCreated(orderId, customerId, total);

        session.Events.StartStream<Order>(orderId, evt);
        await session.SaveChangesAsync(ct);

        return orderId;
    }

    public async Task ProcessPaymentAsync(
        Guid orderId, 
        Guid paymentId, 
        decimal amount,
        CancellationToken ct = default)
    {
        var evt = new PaymentProcessed(orderId, paymentId, amount);
        session.Events.Append(orderId, evt);
        await session.SaveChangesAsync(ct);
    }

    public async Task<Order?> GetOrderAsync(Guid orderId, CancellationToken ct = default)
    {
        return await session.Events.AggregateStreamAsync<Order>(orderId, token: ct);
    }

    public async Task<Order?> GetOrderAtTimeAsync(
        Guid orderId, 
        DateTimeOffset timestamp,
        CancellationToken ct = default)
    {
        return await session.Events.AggregateStreamAsync<Order>(
            orderId, 
            timestamp: timestamp.UtcDateTime,
            token: ct);
    }
}

// Marten configuration with PostgreSQL
builder.Services.AddMarten(options =>
{
    options.Connection(builder.Configuration.GetConnectionString("PostgreSQL")!);

    // Enable event sourcing
    options.Events.StreamIdentity = StreamIdentity.AsGuid;

    // Projections for read models
    options.Projections.Add<OrderProjection>(ProjectionLifecycle.Async);
})
.UseLightweightSessions()
.OptimizeArtifactWorkflow();

Distributed Tracing with OpenTelemetry

// Configure OpenTelemetry for distributed tracing
builder.Services.AddOpenTelemetry()
    .WithTracing(tracing =>
    {
        tracing
            .AddAspNetCoreInstrumentation()
            .AddHttpClientInstrumentation()
            .AddEntityFrameworkCoreInstrumentation()
            .AddSource("MassTransit")
            .AddAzureMonitorTraceExporter(options =>
            {
                options.ConnectionString = builder.Configuration["ApplicationInsights:ConnectionString"];
            });
    })
    .WithMetrics(metrics =>
    {
        metrics
            .AddAspNetCoreInstrumentation()
            .AddHttpClientInstrumentation()
            .AddMeter("MassTransit")
            .AddAzureMonitorMetricExporter(options =>
            {
                options.ConnectionString = builder.Configuration["ApplicationInsights:ConnectionString"];
            });
    });

// Custom tracing in saga
public class OrderSagaStateMachine : MassTransitStateMachine<OrderSagaState>
{
    private static readonly ActivitySource ActivitySource = new("OrderSaga");

    public OrderSagaStateMachine()
    {
        Initially(
            When(OrderCreatedEvent)
                .Then(context =>
                {
                    using var activity = ActivitySource.StartActivity("ProcessOrderCreated");
                    activity?.SetTag("order.id", context.Message.OrderId);
                    activity?.SetTag("saga.correlation_id", context.Saga.CorrelationId);

                    context.Saga.OrderId = context.Message.OrderId;
                    context.Saga.CreatedAt = DateTimeOffset.UtcNow;
                })
                .TransitionTo(PaymentProcessing)
                .PublishAsync(context => context.Init<ProcessPaymentCommand>(new
                {
                    context.Message.OrderId,
                    context.Message.TotalAmount,
                    CorrelationId = context.Saga.CorrelationId
                }))
        );
    }
}

Inbox Pattern (Complement to Outbox)

The inbox pattern ensures exactly-once processing of incoming messages by tracking received messages before processing.

// Inbox message tracking
public sealed record InboxMessage
{
    public required string MessageId { get; init; }
    public required string Source { get; init; }
    public required DateTimeOffset ReceivedAt { get; init; }
    public DateTimeOffset? ProcessedAt { get; set; }
    public string? Status { get; set; } = "Pending";
}

// Message handler with inbox
public class OrderEventConsumer(
    AppDbContext context,
    IOrderService orderService,
    ILogger<OrderEventConsumer> logger) : IConsumer<OrderCreatedEvent>
{
    public async Task Consume(ConsumeContext<OrderCreatedEvent> ctx)
    {
        var messageId = ctx.MessageId?.ToString() ?? Guid.NewGuid().ToString();

        // Check inbox for duplicate
        var inboxMessage = await context.InboxMessages
            .FirstOrDefaultAsync(m => m.MessageId == messageId, ctx.CancellationToken);

        if (inboxMessage?.ProcessedAt != null)
        {
            logger.LogInformation("Message {MessageId} already processed", messageId);
            return;
        }

        var strategy = context.Database.CreateExecutionStrategy();

        await strategy.ExecuteAsync(async () =>
        {
            await using var transaction = await context.Database.BeginTransactionAsync(ctx.CancellationToken);

            try
            {
                // Record message in inbox
                if (inboxMessage == null)
                {
                    inboxMessage = new InboxMessage
                    {
                        MessageId = messageId,
                        Source = ctx.SourceAddress?.ToString() ?? "unknown",
                        ReceivedAt = DateTimeOffset.UtcNow
                    };
                    context.InboxMessages.Add(inboxMessage);
                }

                // Process the message
                await orderService.HandleOrderCreatedAsync(ctx.Message, ctx.CancellationToken);

                // Mark as processed
                inboxMessage.ProcessedAt = DateTimeOffset.UtcNow;
                inboxMessage.Status = "Processed";

                await context.SaveChangesAsync(ctx.CancellationToken);
                await transaction.CommitAsync(ctx.CancellationToken);

                logger.LogInformation("Message {MessageId} processed successfully", messageId);
            }
            catch (Exception ex)
            {
                await transaction.RollbackAsync(ctx.CancellationToken);
                logger.LogError(ex, "Failed to process message {MessageId}", messageId);

                if (inboxMessage != null)
                {
                    inboxMessage.Status = "Failed";
                    await context.SaveChangesAsync(ctx.CancellationToken);
                }

                throw;
            }
        });
    }
}

Two-Phase Commit Alternative: Try-Confirm/Cancel (TCC)

TCC is a lightweight alternative to traditional two-phase commit, suitable for microservices.

// TCC transaction coordinator
public interface ITccParticipant
{
    Task<bool> TryAsync(Guid transactionId, CancellationToken ct);
    Task ConfirmAsync(Guid transactionId, CancellationToken ct);
    Task CancelAsync(Guid transactionId, CancellationToken ct);
}

public class PaymentTccParticipant(
    IPaymentGateway gateway,
    IRepository<PaymentReservation> repository) : ITccParticipant
{
    public async Task<bool> TryAsync(Guid transactionId, CancellationToken ct)
    {
        // Reserve funds without actually charging
        var reservation = await gateway.ReserveFundsAsync(transactionId, ct);

        if (reservation.Success)
        {
            await repository.AddAsync(new PaymentReservation
            {
                TransactionId = transactionId,
                ReservationId = reservation.Id,
                Status = "Reserved",
                ExpiresAt = DateTimeOffset.UtcNow.AddMinutes(10)
            }, ct);
        }

        return reservation.Success;
    }

    public async Task ConfirmAsync(Guid transactionId, CancellationToken ct)
    {
        var reservation = await repository.GetByTransactionIdAsync(transactionId, ct);

        if (reservation != null)
        {
            await gateway.CaptureReservedFundsAsync(reservation.ReservationId, ct);
            reservation.Status = "Confirmed";
            await repository.UpdateAsync(reservation, ct);
        }
    }

    public async Task CancelAsync(Guid transactionId, CancellationToken ct)
    {
        var reservation = await repository.GetByTransactionIdAsync(transactionId, ct);

        if (reservation != null)
        {
            await gateway.ReleaseReservedFundsAsync(reservation.ReservationId, ct);
            reservation.Status = "Cancelled";
            await repository.UpdateAsync(reservation, ct);
        }
    }
}

// TCC Coordinator
public class TccCoordinator(
    IEnumerable<ITccParticipant> participants,
    ILogger<TccCoordinator> logger)
{
    public async Task<bool> ExecuteAsync(Guid transactionId, CancellationToken ct = default)
    {
        var tryResults = new List<(ITccParticipant Participant, bool Success)>();

        // Phase 1: Try
        foreach (var participant in participants)
        {
            try
            {
                var success = await participant.TryAsync(transactionId, ct);
                tryResults.Add((participant, success));

                if (!success)
                {
                    logger.LogWarning("Try phase failed for transaction {TransactionId}", transactionId);
                    await CancelAllAsync(transactionId, tryResults.Select(r => r.Participant), ct);
                    return false;
                }
            }
            catch (Exception ex)
            {
                logger.LogError(ex, "Exception in try phase for transaction {TransactionId}", transactionId);
                await CancelAllAsync(transactionId, tryResults.Select(r => r.Participant), ct);
                return false;
            }
        }

        // Phase 2: Confirm
        foreach (var participant in participants)
        {
            try
            {
                await participant.ConfirmAsync(transactionId, ct);
            }
            catch (Exception ex)
            {
                logger.LogError(ex, "Exception in confirm phase for transaction {TransactionId}", transactionId);
                // Continue confirming others - confirmation should be idempotent
            }
        }

        return true;
    }

    private async Task CancelAllAsync(
        Guid transactionId,
        IEnumerable<ITccParticipant> participants,
        CancellationToken ct)
    {
        foreach (var participant in participants)
        {
            try
            {
                await participant.CancelAsync(transactionId, ct);
            }
            catch (Exception ex)
            {
                logger.LogError(ex, "Exception in cancel phase for transaction {TransactionId}", transactionId);
            }
        }
    }
}

Dead Letter Queue Handling

// Dead letter queue processor
public class DeadLetterProcessor(
    ServiceBusClient client,
    ILogger<DeadLetterProcessor> logger) : BackgroundService
{
    protected override async Task ExecuteAsync(CancellationToken stoppingToken)
    {
        var receiver = client.CreateReceiver(
            "orders", 
            new ServiceBusReceiverOptions 
            { 
                SubQueue = SubQueue.DeadLetter 
            });

        while (!stoppingToken.IsCancellationRequested)
        {
            try
            {
                var messages = await receiver.ReceiveMessagesAsync(
                    maxMessages: 10,
                    maxWaitTime: TimeSpan.FromSeconds(5),
                    cancellationToken: stoppingToken);

                foreach (var message in messages)
                {
                    logger.LogWarning(
                        "Dead letter message: {MessageId}, Reason: {Reason}, Description: {Description}",
                        message.MessageId,
                        message.DeadLetterReason,
                        message.DeadLetterErrorDescription);

                    // Log to external monitoring system
                    // Optionally attempt reprocessing or manual intervention

                    await receiver.CompleteMessageAsync(message, stoppingToken);
                }
            }
            catch (Exception ex)
            {
                logger.LogError(ex, "Error processing dead letter queue");
                await Task.Delay(TimeSpan.FromSeconds(30), stoppingToken);
            }
        }
    }
}

Performance Optimization: Parallel Saga Execution

// Parallel saga steps where operations are independent
public class OptimizedOrderSagaStateMachine : MassTransitStateMachine<OrderSagaState>
{
    public OptimizedOrderSagaStateMachine()
    {
        During(OrderCreated,
            When(StartProcessing)
                .Then(async context =>
                {
                    // Execute independent operations in parallel
                    var inventoryTask = context.Publish<CheckInventoryCommand>(new
                    {
                        context.Saga.OrderId
                    });

                    var customerTask = context.Publish<ValidateCustomerCommand>(new
                    {
                        context.Saga.CustomerId
                    });

                    var pricingTask = context.Publish<CalculatePricingCommand>(new
                    {
                        context.Saga.OrderId
                    });

                    // Wait for all to complete
                    await Task.WhenAll(inventoryTask, customerTask, pricingTask);
                })
                .TransitionTo(ValidationInProgress)
        );
    }
}

Testing Strategies

Unit Testing Saga State Machines

public class OrderSagaStateMachineTests
{
    [Fact]
    public async Task Should_Transition_To_PaymentProcessing_When_Order_Created()
    {
        // Arrange
        var harness = new InMemoryTestHarness();
        var sagaHarness = harness.StateMachineSaga<OrderSagaState, OrderSagaStateMachine>();

        await harness.Start();

        try
        {
            var orderId = Guid.NewGuid();
            var correlationId = Guid.NewGuid();

            // Act
            await harness.Bus.Publish(new OrderCreatedEvent
            {
                OrderId = orderId,
                CustomerId = "customer-123",
                TotalAmount = 100.00m,
                CorrelationId = correlationId
            });

            // Assert
            Assert.True(await sagaHarness.Consumed.Any<OrderCreatedEvent>());

            var saga = sagaHarness.Created.ContainsInState(
                correlationId, 
                sagaHarness.StateMachine, 
                sagaHarness.StateMachine.PaymentProcessing);

            Assert.NotNull(saga);

            Assert.True(await harness.Published.Any<ProcessPaymentCommand>());
        }
        finally
        {
            await harness.Stop();
        }
    }
}

Integration Testing with Testcontainers

public class OrderSagaIntegrationTests : IAsyncLifetime
{
    private readonly PostgreSqlContainer _postgres = new PostgreSqlBuilder()
        .WithImage("postgres:16-alpine")
        .Build();

    private readonly RabbitMqContainer _rabbitmq = new RabbitMqBuilder()
        .WithImage("rabbitmq:3-management-alpine")
        .Build();

    public async Task InitializeAsync()
    {
        await _postgres.StartAsync();
        await _rabbitmq.StartAsync();
    }

    [Fact]
    public async Task Should_Complete_Full_Order_Saga()
    {
        // Arrange
        var services = new ServiceCollection();

        services.AddMassTransit(x =>
        {
            x.AddSagaStateMachine<OrderSagaStateMachine, OrderSagaState>()
                .InMemoryRepository();

            x.UsingRabbitMq((context, cfg) =>
            {
                cfg.Host(_rabbitmq.GetConnectionString());
                cfg.ConfigureEndpoints(context);
            });
        });

        var provider = services.BuildServiceProvider();
        var bus = provider.GetRequiredService<IBus>();

        await bus.StartAsync();

        try
        {
            var orderId = Guid.NewGuid();

            // Act
            await bus.Publish(new OrderCreatedEvent
            {
                OrderId = orderId,
                CustomerId = "test-customer",
                TotalAmount = 150.00m
            });

            // Wait for saga completion
            await Task.Delay(TimeSpan.FromSeconds(5));

            // Assert - verify saga completed successfully
            // This would query your saga repository in a real test
        }
        finally
        {
            await bus.StopAsync();
        }
    }

    public async Task DisposeAsync()
    {
        await _postgres.DisposeAsync();
        await _rabbitmq.DisposeAsync();
    }
}

Recommended Resources

Official Documentation

Books

Microservices Patterns by Chris Richardson
Building Microservices by Sam Newman (3rd Edition)
Software Architecture: The Hard Parts by Neal Ford et al.

Articles & Patterns

Distributed Tracing in .NET Core and Azure

Hossein Esmati — Fri, 26 Jun 2026 00:41:33 +0000

This article is part of the Comprehensive Guide to Microservices Architecture in .NET Core, Cloud and Azure series.

This article explores distributed tracing in .NET applications, covering built-in support through the Activity API, integration with OpenTelemetry for vendor-neutral instrumentation, and deployment to Azure with Application Insights for production monitoring.

What is Distributed Tracing?

Distributed tracing is a diagnostic technique that tracks requests as they traverse different components of a distributed system, including microservices, databases, message queues, and external APIs.

Key benefits:

Pinpoint failures: Identify exactly which service or component caused an error
Measure latency: Understand time spent in each service and operation
Visualize dependencies: Map how services interact and depend on each other
Identify bottlenecks: Find slow queries, API calls, or processing steps

Example request flow:

Load balancer receives HTTP request
API Gateway routes to Order Service
Order Service calls Customer Service via HTTP
Order Service queries PostgreSQL database
Order Service publishes event to Azure Service Bus
Payment Service processes payment via external API
Response returns to client

Distributed tracing captures this entire journey as a single trace composed of multiple spans (individual operations), allowing you to see timing, errors, and context at each step.

Core Concepts

Trace: The complete journey of a request through your system, identified by a unique trace ID.

Span: A single operation within a trace (e.g., HTTP request, database query, message processing). Each span has:

Unique span ID
Parent span ID (forming a tree structure)
Start time and duration
Operation name and attributes
Status (success, error)

Context Propagation: Passing trace and span IDs between services, typically via HTTP headers (W3C Trace Context standard).

Tags/Attributes: Key-value pairs attached to spans for filtering and analysis (e.g., http.method=POST, db.statement=SELECT * FROM orders).

Built-in Support in .NET

.NET has native distributed tracing support through the System.Diagnostics namespace, centered around the Activity class.

Activity API

The Activity API represents operations in your application and automatically handles trace context propagation:

using System.Diagnostics;

public class OrderService
{
    // Create an ActivitySource for your service
    private static readonly ActivitySource ActivitySource = 
        new("CompanyName.OrderService", "1.0.0");

    public async Task<Order> CreateOrderAsync(CreateOrderRequest request)
    {
        // Start a new activity (span)
        using var activity = ActivitySource.StartActivity(
            "CreateOrder",
            ActivityKind.Server);

        // Add tags for filtering and analysis
        activity?.SetTag("order.customerId", request.CustomerId);
        activity?.SetTag("order.itemCount", request.Items.Count);
        activity?.SetTag("order.totalAmount", request.TotalAmount);

        try
        {
            // Business logic
            var order = new Order
            {
                Id = Guid.NewGuid(),
                CustomerId = request.CustomerId,
                Items = request.Items,
                TotalAmount = request.TotalAmount,
                CreatedAt = DateTime.UtcNow
            };

            await _repository.SaveAsync(order);

            // Add result information
            activity?.SetTag("order.id", order.Id);
            activity?.SetStatus(ActivityStatusCode.Ok);

            return order;
        }
        catch (Exception ex)
        {
            // Record error information
            activity?.SetStatus(ActivityStatusCode.Error, ex.Message);
            activity?.RecordException(ex);
            throw;
        }
    }

    public async Task<Customer> GetCustomerAsync(Guid customerId)
    {
        // Create a child span for external HTTP call
        using var activity = ActivitySource.StartActivity(
            "GetCustomer",
            ActivityKind.Client);

        activity?.SetTag("customer.id", customerId);

        // HTTP client automatically propagates trace context
        var response = await _httpClient.GetAsync($"/api/customers/{customerId}");

        activity?.SetTag("http.status_code", (int)response.StatusCode);

        return await response.Content.ReadFromJsonAsync<Customer>();
    }
}

Automatic Context Propagation

.NET automatically propagates trace context using W3C Trace Context headers:

traceparent: 00-0af7651916cd43dd8448eb211c80319c-b7ad6b7169203331-01
tracestate: congo=t61rcWkgMzE

Key components:

00: Version
0af7651916cd43dd8448eb211c80319c: Trace ID (128-bit)
b7ad6b7169203331: Parent span ID (64-bit)
01: Trace flags (sampled)

ActivitySource Registration

// Program.cs
var builder = WebApplication.CreateBuilder(args);

// Register ActivitySource - must match the name used in your code
builder.Services.AddSingleton(new ActivitySource("CompanyName.OrderService"));

// Built-in instrumentation for ASP.NET Core
builder.Services.AddHttpClient();

OpenTelemetry in .NET

OpenTelemetry is the CNCF standard for observability, providing vendor-neutral instrumentation that works with multiple backends.

Installing OpenTelemetry

# Core packages
dotnet add package OpenTelemetry.Extensions.Hosting
dotnet add package OpenTelemetry.Instrumentation.AspNetCore
dotnet add package OpenTelemetry.Instrumentation.Http

# Database instrumentation
dotnet add package OpenTelemetry.Instrumentation.EntityFrameworkCore
dotnet add package OpenTelemetry.Instrumentation.SqlClient

# Exporters
dotnet add package OpenTelemetry.Exporter.OpenTelemetryProtocol  # OTLP (Jaeger, etc.)
dotnet add package Azure.Monitor.OpenTelemetry.Exporter  # Application Insights

Basic Configuration

using OpenTelemetry.Resources;
using OpenTelemetry.Trace;

var builder = WebApplication.CreateBuilder(args);

builder.Services.AddOpenTelemetry()
    .ConfigureResource(resource => resource
        .AddService(
            serviceName: "order-service",
            serviceVersion: typeof(Program).Assembly.GetName().Version?.ToString() ?? "unknown",
            serviceInstanceId: Environment.MachineName))
    .WithTracing(tracing => tracing
        // Add instrumentation for ASP.NET Core
        .AddAspNetCoreInstrumentation(options =>
        {
            options.RecordException = true;
            options.Filter = context => 
                !context.Request.Path.StartsWithSegments("/health");
        })

        // Add instrumentation for HttpClient
        .AddHttpClientInstrumentation(options =>
        {
            options.RecordException = true;
            options.FilterHttpRequestMessage = request =>
                !request.RequestUri?.PathAndQuery.Contains("/health") ?? true;
        })

        // Add instrumentation for Entity Framework Core
        .AddEntityFrameworkCoreInstrumentation(options =>
        {
            options.SetDbStatementForText = true;
            options.SetDbStatementForStoredProcedure = true;
        })

        // Add instrumentation for SQL Client
        .AddSqlClientInstrumentation(options =>
        {
            options.SetDbStatementForText = true;
            options.RecordException = true;
        })

        // Register your custom ActivitySource
        .AddSource("CompanyName.*")

        // Export to OTLP endpoint (Jaeger)
        .AddOtlpExporter(options =>
        {
            options.Endpoint = new Uri(
                builder.Configuration["OpenTelemetry:Endpoint"] ?? 
                "http://localhost:4317");
        }));

var app = builder.Build();

Advanced Configuration with Multiple Exporters

Export traces to multiple backends simultaneously:

builder.Services.AddOpenTelemetry()
    .ConfigureResource(resource => resource
        .AddService("order-service")
        .AddAttributes(new Dictionary<string, object>
        {
            ["deployment.environment"] = builder.Environment.EnvironmentName,
            ["service.namespace"] = "ecommerce",
            ["service.team"] = "platform"
        }))
    .WithTracing(tracing => tracing
        .AddAspNetCoreInstrumentation(options =>
        {
            options.RecordException = true;

            // Enrich spans with additional context
            options.Enrich = (activity, eventName, rawObject) =>
            {
                if (eventName == "OnStartActivity" && rawObject is HttpRequest request)
                {
                    activity.SetTag("http.user_agent", request.Headers.UserAgent.ToString());
                    activity.SetTag("http.client_ip", request.HttpContext.Connection.RemoteIpAddress?.ToString());
                }
            };
        })

        .AddHttpClientInstrumentation()
        .AddEntityFrameworkCoreInstrumentation()
        .AddSqlClientInstrumentation()
        .AddSource("CompanyName.*")

        // Export to Jaeger for development
        .AddOtlpExporter(options =>
        {
            options.Endpoint = new Uri("http://jaeger:4317");
        })

        // Export to Azure Monitor for production
        .AddAzureMonitorTraceExporter(options =>
        {
            options.ConnectionString = 
                builder.Configuration["ApplicationInsights:ConnectionString"];
        })

        // Set sampling - only trace 10% of requests
        .SetSampler(new TraceIdRatioBasedSampler(0.1)));

Custom Instrumentation

Create detailed spans for business operations:

public class OrderProcessingService
{
    private readonly ActivitySource _activitySource;
    private readonly IOrderRepository _repository;
    private readonly IPaymentService _paymentService;
    private readonly IInventoryService _inventoryService;

    public OrderProcessingService(
        ActivitySource activitySource,
        IOrderRepository repository,
        IPaymentService paymentService,
        IInventoryService inventoryService)
    {
        _activitySource = activitySource;
        _repository = repository;
        _paymentService = paymentService;
        _inventoryService = inventoryService;
    }

    public async Task<Order> ProcessOrderAsync(
        CreateOrderRequest request,
        CancellationToken cancellationToken = default)
    {
        using var activity = _activitySource.StartActivity(
            "ProcessOrder",
            ActivityKind.Internal);

        activity?.SetTag("order.customerId", request.CustomerId);
        activity?.SetTag("order.itemCount", request.Items.Count);
        activity?.SetTag("order.totalAmount", request.TotalAmount);

        try
        {
            // Create order
            var order = await CreateOrderInternalAsync(request, cancellationToken);
            activity?.AddEvent(new ActivityEvent("OrderCreated"));

            // Reserve inventory
            using (var inventoryActivity = _activitySource.StartActivity(
                "ReserveInventory",
                ActivityKind.Client))
            {
                await _inventoryService.ReserveAsync(
                    order.Items.Select(i => new InventoryReservation
                    {
                        ProductId = i.ProductId,
                        Quantity = i.Quantity
                    }),
                    cancellationToken);

                inventoryActivity?.AddEvent(new ActivityEvent("InventoryReserved"));
            }

            // Process payment
            using (var paymentActivity = _activitySource.StartActivity(
                "ProcessPayment",
                ActivityKind.Client))
            {
                paymentActivity?.SetTag("payment.amount", request.TotalAmount);
                paymentActivity?.SetTag("payment.currency", "USD");

                var paymentResult = await _paymentService.ProcessPaymentAsync(
                    new PaymentRequest
                    {
                        OrderId = order.Id,
                        Amount = request.TotalAmount,
                        PaymentMethod = request.PaymentMethod
                    },
                    cancellationToken);

                paymentActivity?.SetTag("payment.transactionId", paymentResult.TransactionId);
                paymentActivity?.AddEvent(new ActivityEvent("PaymentProcessed"));
            }

            // Update order status
            order.Status = OrderStatus.Confirmed;
            await _repository.UpdateAsync(order, cancellationToken);

            activity?.SetTag("order.id", order.Id);
            activity?.SetTag("order.status", order.Status.ToString());
            activity?.SetStatus(ActivityStatusCode.Ok);

            return order;
        }
        catch (InsufficientInventoryException ex)
        {
            activity?.SetStatus(ActivityStatusCode.Error, "Insufficient inventory");
            activity?.RecordException(ex);
            activity?.AddEvent(new ActivityEvent("OrderFailed", 
                tags: new ActivityTagsCollection
                {
                    ["error.type"] = "InsufficientInventory"
                }));
            throw;
        }
        catch (PaymentFailedException ex)
        {
            activity?.SetStatus(ActivityStatusCode.Error, "Payment failed");
            activity?.RecordException(ex);
            activity?.AddEvent(new ActivityEvent("OrderFailed",
                tags: new ActivityTagsCollection
                {
                    ["error.type"] = "PaymentFailed"
                }));

            // Compensate: release inventory
            await _inventoryService.ReleaseAsync(order.Id, cancellationToken);

            throw;
        }
        catch (Exception ex)
        {
            activity?.SetStatus(ActivityStatusCode.Error, ex.Message);
            activity?.RecordException(ex);
            throw;
        }
    }
}

Adding Baggage for Cross-Service Context

Baggage allows you to pass custom data across service boundaries:

public class CorrelationMiddleware
{
    private readonly RequestDelegate _next;

    public async Task InvokeAsync(HttpContext context)
    {
        // Add baggage that will propagate to downstream services
        Baggage.SetBaggage("correlation.id", context.TraceIdentifier);
        Baggage.SetBaggage("user.id", context.User.FindFirst("sub")?.Value ?? "anonymous");
        Baggage.SetBaggage("tenant.id", context.Request.Headers["X-Tenant-ID"].ToString());

        await _next(context);
    }
}

// Downstream service can access baggage
public class CustomerService
{
    public async Task<Customer> GetCustomerAsync(Guid id)
    {
        using var activity = ActivitySource.StartActivity("GetCustomer");

        // Access baggage from upstream service
        var userId = Baggage.GetBaggage("user.id");
        var tenantId = Baggage.GetBaggage("tenant.id");

        activity?.SetTag("user.id", userId);
        activity?.SetTag("tenant.id", tenantId);

        // Business logic
    }
}

Application Insights Integration

Azure Application Insights provides managed distributed tracing with powerful query and visualization capabilities.

Configuration

builder.Services.AddApplicationInsightsTelemetry(options =>
{
    options.ConnectionString = builder.Configuration["ApplicationInsights:ConnectionString"];
    options.EnableAdaptiveSampling = true;
    options.EnableQuickPulseMetricStream = true;
});

// Or using OpenTelemetry exporter
builder.Services.AddOpenTelemetry()
    .WithTracing(tracing => tracing
        .AddAspNetCoreInstrumentation()
        .AddHttpClientInstrumentation()
        .AddAzureMonitorTraceExporter(options =>
        {
            options.ConnectionString = builder.Configuration["ApplicationInsights:ConnectionString"];
        }));

Application Map

Application Insights automatically generates a visual map of service dependencies based on distributed traces, showing:

Service topology
Call volumes
Success rates
Average response times
Failed dependency calls

Querying Traces with KQL

Use Kusto Query Language to analyze traces:

// Find slow requests
requests
| where timestamp > ago(1h)
| where duration > 1000  // Over 1 second
| project timestamp, name, duration, resultCode, operation_Id
| order by duration desc

// Analyze dependencies by service
dependencies
| where timestamp > ago(24h)
| summarize 
    Count = count(),
    AvgDuration = avg(duration),
    P95Duration = percentile(duration, 95),
    FailureRate = 100.0 * countif(success == false) / count()
    by target, type
| order by Count desc

// Find requests with failed dependencies
let failedDeps = dependencies
| where timestamp > ago(1h) and success == false
| distinct operation_Id;
requests
| where timestamp > ago(1h)
| where operation_Id in (failedDeps)
| project timestamp, name, operation_Id, duration, resultCode

Visualizing Traces with Jaeger

Jaeger is an open-source distributed tracing platform ideal for development and self-hosted production.

Running Jaeger Locally

# Run Jaeger all-in-one with Docker
docker run -d --name jaeger \
  -e COLLECTOR_OTLP_ENABLED=true \
  -p 4317:4317 \
  -p 4318:4318 \
  -p 16686:16686 \
  jaegertracing/all-in-one:latest

# Access Jaeger UI at http://localhost:16686

Configure OpenTelemetry to Export to Jaeger

builder.Services.AddOpenTelemetry()
    .WithTracing(tracing => tracing
        .AddAspNetCoreInstrumentation()
        .AddHttpClientInstrumentation()
        .AddSource("CompanyName.*")
        .AddOtlpExporter(options =>
        {
            options.Endpoint = new Uri("http://localhost:4317");
            options.Protocol = OtlpExportProtocol.Grpc;
        }));

Environment Variable Configuration

# Alternative: Configure via environment variables
export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317
export OTEL_SERVICE_NAME=order-service
export OTEL_RESOURCE_ATTRIBUTES=service.namespace=ecommerce,deployment.environment=development

Sampling Strategies

Tracing every request can be expensive. Use sampling to balance observability with performance:

Always On Sampler (Development)

.SetSampler(new AlwaysOnSampler())

Ratio-Based Sampler (Production)

// Sample 10% of traces
.SetSampler(new TraceIdRatioBasedSampler(0.1))

Parent-Based Sampler

// Sample based on parent decision, fallback to 10%
.SetSampler(new ParentBasedSampler(
    new TraceIdRatioBasedSampler(0.1)))

Custom Sampler

public class SmartSampler : Sampler
{
    public override SamplingResult ShouldSample(in SamplingParameters samplingParameters)
    {
        // Always sample errors
        if (samplingParameters.Tags.Any(t => 
            t.Key == "error" && (bool)t.Value))
        {
            return new SamplingResult(SamplingDecision.RecordAndSample);
        }

        // Always sample slow requests
        var duration = samplingParameters.Tags.FirstOrDefault(t => 
            t.Key == "duration").Value as TimeSpan?;
        if (duration > TimeSpan.FromSeconds(1))
        {
            return new SamplingResult(SamplingDecision.RecordAndSample);
        }

        // Sample 5% of normal requests
        return new TraceIdRatioBasedSampler(0.05)
            .ShouldSample(samplingParameters);
    }
}

.SetSampler(new SmartSampler())

Best Practices

Instrumentation:

Use meaningful span names that describe the operation
Add relevant tags for filtering (customer ID, transaction ID, etc.)
Record exceptions with full context
Use ActivityKind appropriately (Server, Client, Internal, Producer, Consumer)

Performance:

Implement sampling in production to control costs
Filter health check endpoints from tracing
Avoid logging sensitive data in spans (PII, credentials, etc.)
Use async/await to prevent blocking

Context Propagation:

Ensure HTTP clients propagate trace context automatically
Manually propagate context for message queues and non-HTTP protocols
Use baggage sparingly (it's transmitted with every request)

Organization:

Use consistent service naming across your organization
Group related services with service.namespace attribute
Tag spans with deployment.environment for filtering
Include service version for deployment tracking

Azure-Specific:

Use Application Insights for production monitoring
Enable adaptive sampling to control costs
Configure custom metrics and alerts based on traces
Use Application Map to visualize dependencies
Leverage Log Analytics for advanced querying

Troubleshooting:

Always trace error paths with detailed context
Add custom events for important business milestones
Include compensation/rollback operations in traces
Monitor trace export failures and sampling decisions

By implementing comprehensive distributed tracing, teams gain deep visibility into system behavior, enabling faster debugging, performance optimization, and confident deployment of complex distributed applications.

Deployment Strategies in .NET Core and Azure

Hossein Esmati — Fri, 26 Jun 2026 00:41:29 +0000

This article is part of the Comprehensive Guide to Microservices Architecture in .NET Core, Cloud and Azure series.

This article explores proven deployment strategies for .NET microservices, including blue-green deployments for instant rollback capability, canary deployments for gradual risk mitigation, and automated metrics guards that detect issues before they impact users.

Blue-Green Deployment

What is Blue-Green Deployment?

Blue-green deployment maintains two identical production environments: Blue (currently serving traffic) and Green (idle or running the new version). After deploying and validating the new version in Green, traffic switches instantly. If issues arise, switching back to Blue provides immediate rollback.

Benefits:

Zero-downtime deployments
Instant rollback capability
Full testing in production-like environment before cutover
Simple to understand and implement

Trade-offs:

Requires double the infrastructure resources
Database migrations need special handling
Stateful applications require session migration

Azure App Service Implementation

Azure App Service provides built-in deployment slots for blue-green deployments:

# Create a deployment slot (Green environment)
az webapp deployment slot create \
    --name my-app \
    --resource-group my-rg \
    --slot green \
    --configuration-source my-app

# Deploy new version to green slot
az webapp deployment source config-zip \
    --name my-app \
    --resource-group my-rg \
    --slot green \
    --src app.zip

# Warm up the green slot
az webapp deployment slot start \
    --name my-app \
    --resource-group my-rg \
    --slot green

# Run smoke tests on green slot before switching
curl https://my-app-green.azurewebsites.net/health
curl https://my-app-green.azurewebsites.net/api/health/ready

# Perform the swap (instant traffic switch)
az webapp deployment slot swap \
    --name my-app \
    --resource-group my-rg \
    --slot green \
    --target-slot production

# If issues detected, rollback by swapping back
az webapp deployment slot swap \
    --name my-app \
    --resource-group my-rg \
    --slot green \
    --target-slot production

Configuration with Azure CLI and ARM Templates

{
  "$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#",
  "contentVersion": "1.0.0.0",
  "resources": [
    {
      "type": "Microsoft.Web/sites/slots",
      "apiVersion": "2022-03-01",
      "name": "[concat(parameters('siteName'), '/green')]",
      "location": "[parameters('location')]",
      "properties": {
        "serverFarmId": "[parameters('appServicePlanId')]",
        "siteConfig": {
          "appSettings": [
            {
              "name": "DEPLOYMENT_SLOT",
              "value": "green"
            },
            {
              "name": "APP_VERSION",
              "value": "[parameters('appVersion')]"
            }
          ]
        }
      }
    }
  ]
}

Kubernetes Implementation

For containerized .NET applications on Azure Kubernetes Service (AKS):

# Blue deployment (current production)
apiVersion: apps/v1
kind: Deployment
metadata:
  name: order-service-blue
  namespace: production
spec:
  replicas: 3
  selector:
    matchLabels:
      app: order-service
      version: blue
  template:
    metadata:
      labels:
        app: order-service
        version: blue
    spec:
      containers:
      - name: api
        image: myacr.azurecr.io/order-service:v1.0
        ports:
        - containerPort: 8080
        env:
        - name: ASPNETCORE_ENVIRONMENT
          value: "Production"
        - name: DEPLOYMENT_VERSION
          value: "blue"
        resources:
          requests:
            memory: "256Mi"
            cpu: "250m"
          limits:
            memory: "512Mi"
            cpu: "500m"
        livenessProbe:
          httpGet:
            path: /health/live
            port: 8080
          initialDelaySeconds: 30
          periodSeconds: 10
        readinessProbe:
          httpGet:
            path: /health/ready
            port: 8080
          initialDelaySeconds: 10
          periodSeconds: 5
---
# Green deployment (new version)
apiVersion: apps/v1
kind: Deployment
metadata:
  name: order-service-green
  namespace: production
spec:
  replicas: 3
  selector:
    matchLabels:
      app: order-service
      version: green
  template:
    metadata:
      labels:
        app: order-service
        version: green
    spec:
      containers:
      - name: api
        image: myacr.azurecr.io/order-service:v2.0
        ports:
        - containerPort: 8080
        env:
        - name: ASPNETCORE_ENVIRONMENT
          value: "Production"
        - name: DEPLOYMENT_VERSION
          value: "green"
        resources:
          requests:
            memory: "256Mi"
            cpu: "250m"
          limits:
            memory: "512Mi"
            cpu: "500m"
        livenessProbe:
          httpGet:
            path: /health/live
            port: 8080
          initialDelaySeconds: 30
          periodSeconds: 10
        readinessProbe:
          httpGet:
            path: /health/ready
            port: 8080
          initialDelaySeconds: 10
          periodSeconds: 5
---
# Service initially routes to blue
apiVersion: v1
kind: Service
metadata:
  name: order-service
  namespace: production
spec:
  selector:
    app: order-service
    version: blue  # Change to 'green' to switch traffic
  ports:
  - protocol: TCP
    port: 80
    targetPort: 8080
  type: LoadBalancer

Health Check Endpoints for Validation

Implement comprehensive health checks to validate deployments before switching:

using Microsoft.Extensions.Diagnostics.HealthChecks;

[ApiController]
[Route("health")]
public class HealthController : ControllerBase
{
    private readonly IConfiguration _configuration;
    private readonly IHealthCheckService _healthCheckService;
    private readonly ILogger<HealthController> _logger;

    public HealthController(
        IConfiguration configuration,
        IHealthCheckService healthCheckService,
        ILogger<HealthController> logger)
    {
        _configuration = configuration;
        _healthCheckService = healthCheckService;
        _logger = logger;
    }

    [HttpGet("deployment")]
    public IActionResult GetDeploymentInfo()
    {
        return Ok(new
        {
            Environment = _configuration["DEPLOYMENT_SLOT"] ?? "unknown",
            Version = _configuration["APP_VERSION"] ?? "unknown",
            Hostname = Environment.MachineName,
            Timestamp = DateTime.UtcNow,
            Uptime = TimeSpan.FromMilliseconds(Environment.TickCount64)
        });
    }

    [HttpGet("live")]
    public IActionResult Liveness()
    {
        // Basic liveness check - is the application running?
        return Ok(new { status = "alive" });
    }

    [HttpGet("ready")]
    public async Task<IActionResult> Readiness()
    {
        var report = await _healthCheckService.CheckHealthAsync();

        var response = new
        {
            status = report.Status.ToString(),
            checks = report.Entries.Select(e => new
            {
                name = e.Key,
                status = e.Value.Status.ToString(),
                description = e.Value.Description,
                duration = e.Value.Duration.TotalMilliseconds
            }),
            totalDuration = report.TotalDuration.TotalMilliseconds
        };

        if (report.Status == HealthStatus.Healthy)
        {
            return Ok(response);
        }

        _logger.LogWarning(
            "Readiness check failed. Status: {Status}",
            report.Status);

        return StatusCode(503, response);
    }
}

// Configure health checks in Program.cs
builder.Services.AddHealthChecks()
    .AddCheck("self", () => HealthCheckResult.Healthy())
    .AddSqlServer(
        builder.Configuration.GetConnectionString("DefaultConnection")!,
        name: "database",
        timeout: TimeSpan.FromSeconds(5))
    .AddRedis(
        builder.Configuration.GetConnectionString("Redis")!,
        name: "cache",
        timeout: TimeSpan.FromSeconds(5))
    .AddAzureServiceBusTopic(
        builder.Configuration.GetConnectionString("ServiceBus")!,
        "orders",
        name: "servicebus",
        timeout: TimeSpan.FromSeconds(5))
    .AddCheck<ExternalApiHealthCheck>("external-api");

// Custom health check for external dependencies
public class ExternalApiHealthCheck : IHealthCheck
{
    private readonly IHttpClientFactory _httpClientFactory;
    private readonly ILogger<ExternalApiHealthCheck> _logger;

    public ExternalApiHealthCheck(
        IHttpClientFactory httpClientFactory,
        ILogger<ExternalApiHealthCheck> logger)
    {
        _httpClientFactory = httpClientFactory;
        _logger = logger;
    }

    public async Task<HealthCheckResult> CheckHealthAsync(
        HealthCheckContext context,
        CancellationToken cancellationToken = default)
    {
        try
        {
            var client = _httpClientFactory.CreateClient("external-api");
            var response = await client.GetAsync("/health", cancellationToken);

            if (response.IsSuccessStatusCode)
            {
                return HealthCheckResult.Healthy("External API is responsive");
            }

            return HealthCheckResult.Degraded(
                $"External API returned {response.StatusCode}");
        }
        catch (Exception ex)
        {
            _logger.LogError(ex, "External API health check failed");
            return HealthCheckResult.Unhealthy(
                "External API is unreachable",
                ex);
        }
    }
}

Automated Smoke Testing

Run automated tests against the green environment before switching:

public class BlueGreenSmokeTests
{
    private readonly HttpClient _client;
    private readonly IConfiguration _configuration;
    private readonly ILogger<BlueGreenSmokeTests> _logger;

    public async Task<bool> RunSmokeTestsAsync(string slotUrl)
    {
        _logger.LogInformation("Running smoke tests against {Url}", slotUrl);

        var tests = new Func<Task<bool>>[]
        {
            () => TestHealthEndpoint(slotUrl),
            () => TestApiResponsiveness(slotUrl),
            () => TestDatabaseConnectivity(slotUrl),
            () => TestCacheConnectivity(slotUrl),
            () => TestCriticalBusinessFlow(slotUrl)
        };

        var results = await Task.WhenAll(tests.Select(test => test()));
        var allPassed = results.All(r => r);

        if (allPassed)
        {
            _logger.LogInformation("All smoke tests passed");
        }
        else
        {
            _logger.LogError("Some smoke tests failed");
        }

        return allPassed;
    }

    private async Task<bool> TestHealthEndpoint(string baseUrl)
    {
        try
        {
            var response = await _client.GetAsync($"{baseUrl}/health/ready");
            return response.IsSuccessStatusCode;
        }
        catch (Exception ex)
        {
            _logger.LogError(ex, "Health endpoint test failed");
            return false;
        }
    }

    private async Task<bool> TestCriticalBusinessFlow(string baseUrl)
    {
        try
        {
            // Test a critical path like creating an order
            var orderRequest = new
            {
                customerId = Guid.NewGuid(),
                items = new[] { new { productId = Guid.NewGuid(), quantity = 1 } }
            };

            var response = await _client.PostAsJsonAsync(
                $"{baseUrl}/api/orders",
                orderRequest);

            return response.IsSuccessStatusCode;
        }
        catch (Exception ex)
        {
            _logger.LogError(ex, "Critical business flow test failed");
            return false;
        }
    }
}

Canary Deployment

What is Canary Deployment?

Canary deployment gradually rolls out changes to a small subset of users, monitors key metrics, and progressively increases traffic if the deployment proves successful. If issues are detected, traffic is immediately redirected back to the stable version.

Benefits:

Early detection of issues with minimal user impact
Real-world validation before full rollout
Data-driven deployment decisions
Reduces blast radius of failures

Typical Stages:

Deploy to 5% of infrastructure (canary)
Monitor for 5-10 minutes
If metrics healthy, increase to 25%
Continue to 50%, 75%, 100%
Rollback immediately if issues detected

Kubernetes with Argo Rollouts

Argo Rollouts extends Kubernetes with advanced deployment strategies:

# Install Argo Rollouts
kubectl create namespace argo-rollouts
kubectl apply -n argo-rollouts -f https://github.com/argoproj/argo-rollouts/releases/latest/download/install.yaml

# Install kubectl plugin
curl -LO https://github.com/argoproj/argo-rollouts/releases/latest/download/kubectl-argo-rollouts-linux-amd64
chmod +x kubectl-argo-rollouts-linux-amd64
sudo mv kubectl-argo-rollouts-linux-amd64 /usr/local/bin/kubectl-argo-rollouts

apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: order-service
  namespace: production
spec:
  replicas: 10
  revisionHistoryLimit: 3
  selector:
    matchLabels:
      app: order-service
  template:
    metadata:
      labels:
        app: order-service
    spec:
      containers:
      - name: api
        image: myacr.azurecr.io/order-service:v2.0
        ports:
        - containerPort: 8080
        resources:
          requests:
            memory: "256Mi"
            cpu: "250m"
          limits:
            memory: "512Mi"
            cpu: "500m"
  strategy:
    canary:
      # Define canary steps
      steps:
      - setWeight: 10      # Route 10% traffic to new version
      - pause: {duration: 5m}  # Monitor for 5 minutes
      - setWeight: 25      # Increase to 25%
      - pause: {duration: 5m}
      - setWeight: 50      # Increase to 50%
      - pause: {duration: 5m}
      - setWeight: 75      # Increase to 75%
      - pause: {duration: 5m}
      # Final step implicitly promotes to 100%

      # Services for canary and stable versions
      canaryService: order-service-canary
      stableService: order-service-stable

      # Traffic routing configuration
      trafficRouting:
        nginx:
          stableIngress: order-service

      # Automated analysis during rollout
      analysis:
        templates:
        - templateName: success-rate
        - templateName: latency-check
        startingStep: 1  # Start analysis after first step
        args:
        - name: service-name
          value: order-service
---
# Canary service
apiVersion: v1
kind: Service
metadata:
  name: order-service-canary
  namespace: production
spec:
  selector:
    app: order-service
  ports:
  - protocol: TCP
    port: 80
    targetPort: 8080
---
# Stable service
apiVersion: v1
kind: Service
metadata:
  name: order-service-stable
  namespace: production
spec:
  selector:
    app: order-service
  ports:
  - protocol: TCP
    port: 80
    targetPort: 8080
---
# Analysis template for success rate
apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
  name: success-rate
  namespace: production
spec:
  args:
  - name: service-name
  metrics:
  - name: success-rate
    interval: 1m
    successCondition: result >= 0.95  # 95% success rate required
    failureLimit: 3  # Fail after 3 consecutive failures
    provider:
      prometheus:
        address: http://prometheus-server.monitoring:9090
        query: |
          sum(rate(http_requests_total{
            service="{{args.service-name}}",
            status=~"2.."
          }[5m])) /
          sum(rate(http_requests_total{
            service="{{args.service-name}}"
          }[5m]))
---
# Analysis template for latency
apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
  name: latency-check
  namespace: production
spec:
  args:
  - name: service-name
  metrics:
  - name: p95-latency
    interval: 1m
    successCondition: result <= 500  # P95 latency under 500ms
    failureLimit: 3
    provider:
      prometheus:
        address: http://prometheus-server.monitoring:9090
        query: |
          histogram_quantile(0.95,
            sum(rate(http_request_duration_milliseconds_bucket{
              service="{{args.service-name}}"
            }[5m])) by (le)
          )
  - name: p99-latency
    interval: 1m
    successCondition: result <= 1000  # P99 latency under 1000ms
    failureLimit: 3
    provider:
      prometheus:
        address: http://prometheus-server.monitoring:9090
        query: |
          histogram_quantile(0.99,
            sum(rate(http_request_duration_milliseconds_bucket{
              service="{{args.service-name}}"
            }[5m])) by (le)
          )

Managing Rollouts

# Watch rollout progress
kubectl argo rollouts get rollout order-service -n production --watch

# Promote to next step manually
kubectl argo rollouts promote order-service -n production

# Abort rollout (rollback)
kubectl argo rollouts abort order-service -n production

# Retry failed rollout
kubectl argo rollouts retry order-service -n production

# Get rollout history
kubectl argo rollouts history order-service -n production

# Rollback to specific revision
kubectl argo rollouts undo order-service -n production --to-revision=3

Azure Traffic Manager for Canary

Use Azure Traffic Manager for weighted traffic distribution:

# Create Traffic Manager profile with weighted routing
az network traffic-manager profile create \
    --name order-service-canary \
    --resource-group my-rg \
    --routing-method Weighted \
    --unique-dns-name order-service-canary \
    --ttl 30 \
    --protocol HTTP \
    --port 80 \
    --path /health

# Add stable endpoint (90% weight initially)
az network traffic-manager endpoint create \
    --name stable \
    --profile-name order-service-canary \
    --resource-group my-rg \
    --type azureEndpoints \
    --target-resource-id "/subscriptions/{sub-id}/resourceGroups/my-rg/providers/Microsoft.Web/sites/order-service-stable" \
    --weight 90 \
    --endpoint-status Enabled

# Add canary endpoint (10% weight initially)
az network traffic-manager endpoint create \
    --name canary \
    --profile-name order-service-canary \
    --resource-group my-rg \
    --type azureEndpoints \
    --target-resource-id "/subscriptions/{sub-id}/resourceGroups/my-rg/providers/Microsoft.Web/sites/order-service-canary" \
    --weight 10 \
    --endpoint-status Enabled

# Gradually increase canary weight over time
az network traffic-manager endpoint update \
    --name canary \
    --profile-name order-service-canary \
    --resource-group my-rg \
    --weight 25

# Continue increasing: 50, 75, 100
az network traffic-manager endpoint update \
    --name canary \
    --profile-name order-service-canary \
    --resource-group my-rg \
    --weight 100

# Remove stable endpoint after successful rollout
az network traffic-manager endpoint delete \
    --name stable \
    --profile-name order-service-canary \
    --resource-group my-rg

Application-Level Canary with Feature Flags

Implement canary routing at the application level using Azure App Configuration and feature flags:

using Microsoft.FeatureManagement;
using Microsoft.FeatureManagement.FeatureFilters;

// Program.cs
builder.Configuration.AddAzureAppConfiguration(options =>
{
    options.Connect(builder.Configuration["AppConfiguration:ConnectionString"])
           .UseFeatureFlags(flagOptions =>
           {
               flagOptions.CacheExpirationInterval = TimeSpan.FromSeconds(30);
           });
});

builder.Services.AddFeatureManagement()
    .AddFeatureFilter<PercentageFilter>()
    .AddFeatureFilter<TargetingFilter>();

// Middleware for canary routing
public class CanaryDeploymentMiddleware
{
    private readonly RequestDelegate _next;
    private readonly IFeatureManager _featureManager;
    private readonly ILogger<CanaryDeploymentMiddleware> _logger;

    public CanaryDeploymentMiddleware(
        RequestDelegate next,
        IFeatureManager featureManager,
        ILogger<CanaryDeploymentMiddleware> logger)
    {
        _next = next;
        _featureManager = featureManager;
        _logger = logger;
    }

    public async Task InvokeAsync(HttpContext context)
    {
        // Determine if request should go to canary
        var targetingContext = new TargetingContext
        {
            UserId = context.User.Identity?.Name ?? "anonymous",
            Groups = context.User.Claims
                .Where(c => c.Type == "groups")
                .Select(c => c.Value)
                .ToList()
        };

        var isCanary = await _featureManager.IsEnabledAsync(
            "CanaryDeployment",
            targetingContext);

        if (isCanary)
        {
            context.Request.Headers.Append("X-Deployment-Target", "canary");
            context.Items["DeploymentRing"] = "Canary";
            _logger.LogDebug("Routing request to canary deployment");
        }
        else
        {
            context.Request.Headers.Append("X-Deployment-Target", "stable");
            context.Items["DeploymentRing"] = "Stable";
        }

        await _next(context);
    }

    private bool IsInternalIp(IPAddress? ipAddress)
    {
        if (ipAddress == null)
            return false;

        var bytes = ipAddress.GetAddressBytes();
        return bytes[0] == 10 ||
               (bytes[0] == 172 && bytes[1] >= 16 && bytes[1] <= 31) ||
               (bytes[0] == 192 && bytes[1] == 168);
    }
}

// Register middleware
app.UseMiddleware<CanaryDeploymentMiddleware>();

Feature Flag Configuration (appsettings.json)

{
  "FeatureManagement": {
    "CanaryDeployment": {
      "EnabledFor": [
        {
          "Name": "Percentage",
          "Parameters": {
            "Value": 10
          }
        },
        {
          "Name": "Targeting",
          "Parameters": {
            "Audience": {
              "Users": ["beta-user@company.com"],
              "Groups": ["BetaTesters", "InternalEmployees"],
              "DefaultRolloutPercentage": 10,
              "Exclusion": {
                "Users": [],
                "Groups": []
              }
            }
          }
        }
      ]
    }
  }
}

Metrics Guards and Automated Rollback

What are Metrics Guards?

Metrics guards are automated checks that continuously monitor key performance and business metrics during deployment. If thresholds are violated, they trigger automatic rollback to prevent user impact.

Key Metrics to Monitor

Technical Metrics:

Error rate (HTTP 4xx, 5xx responses)
Latency (P50, P95, P99 response times)
Throughput (requests per second)
CPU and memory utilization
Database query performance

Business Metrics:

Conversion rate
Transaction success rate
Shopping cart abandonment
Revenue per user
Feature usage rates

Implementation with Application Insights

using Microsoft.ApplicationInsights;
using Microsoft.ApplicationInsights.DataContracts;

public class MetricsGuardService : BackgroundService
{
    private readonly TelemetryClient _telemetryClient;
    private readonly ILogger<MetricsGuardService> _logger;
    private readonly IDeploymentService _deploymentService;
    private readonly IConfiguration _configuration;
    private readonly MetricsGuardConfiguration _config;

    public MetricsGuardService(
        TelemetryClient telemetryClient,
        ILogger<MetricsGuardService> logger,
        IDeploymentService deploymentService,
        IConfiguration configuration)
    {
        _telemetryClient = telemetryClient;
        _logger = logger;
        _deploymentService = deploymentService;
        _configuration = configuration;
        _config = configuration.GetSection("MetricsGuard")
            .Get<MetricsGuardConfiguration>() ?? new();
    }

    protected override async Task ExecuteAsync(CancellationToken stoppingToken)
    {
        _logger.LogInformation("Metrics guard service started");

        while (!stoppingToken.IsCancellationRequested)
        {
            try
            {
                var evaluation = await EvaluateMetricsAsync();

                if (!evaluation.IsHealthy)
                {
                    _logger.LogError(
                        "Metrics guard violations detected: {Violations}",
                        string.Join("; ", evaluation.Violations));

                    // Trigger automated rollback
                    await _deploymentService.RollbackAsync(
                        "Automated rollback due to metrics violations");

                    // Send alerts
                    await SendAlertAsync(evaluation);

                    // Stop monitoring after rollback
                    break;
                }
                else
                {
                    _logger.LogDebug("All metrics within acceptable thresholds");
                }

                await Task.Delay(_config.EvaluationInterval, stoppingToken);
            }
            catch (Exception ex)
            {
                _logger.LogError(ex, "Error evaluating metrics");
                await Task.Delay(TimeSpan.FromSeconds(30), stoppingToken);
            }
        }
    }

    private async Task<MetricsEvaluation> EvaluateMetricsAsync()
    {
        // Query Application Insights using Kusto
        var query = @"
            let timeRange = 5m;
            let deploymentRing = 'Canary';
            requests
            | where timestamp > ago(timeRange)
            | where customDimensions.DeploymentRing == deploymentRing
            | summarize 
                TotalRequests = count(),
                FailedRequests = countif(success == false),
                ErrorRate = 100.0 * countif(success == false) / count(),
                P50Latency = percentile(duration, 50),
                P95Latency = percentile(duration, 95),
                P99Latency = percentile(duration, 99)
        ";

        var results = await ExecuteAnalyticsQueryAsync(query);

        var violations = new List<string>();

        // Check error rate
        if (results.ErrorRate > _config.MaxErrorRatePercent)
        {
            violations.Add(
                $"Error rate too high: {results.ErrorRate:F2}% " +
                $"(threshold: {_config.MaxErrorRatePercent}%)");
        }

        // Check P95 latency
        if (results.P95Latency > _config.MaxP95LatencyMs)
        {
            violations.Add(
                $"P95 latency too high: {results.P95Latency:F0}ms " +
                $"(threshold: {_config.MaxP95LatencyMs}ms)");
        }

        // Check P99 latency
        if (results.P99Latency > _config.MaxP99LatencyMs)
        {
            violations.Add(
                $"P99 latency too high: {results.P99Latency:F0}ms " +
                $"(threshold: {_config.MaxP99LatencyMs}ms)");
        }

        // Check minimum traffic
        if (results.TotalRequests < _config.MinimumTrafficForEvaluation)
        {
            violations.Add(
                $"Insufficient traffic for evaluation: {results.TotalRequests} requests " +
                $"(minimum: {_config.MinimumTrafficForEvaluation})");
        }

        // Compare canary vs stable
        var comparison = await CompareCanaryToStableAsync();
        if (comparison.ErrorRateDelta > _config.MaxErrorRateDeltaPercent)
        {
            violations.Add(
                $"Error rate delta too high: +{comparison.ErrorRateDelta:F2}% " +
                $"compared to stable");
        }

        return new MetricsEvaluation
        {
            IsHealthy = violations.Count == 0,
            Violations = violations,
            Metrics = results,
            Timestamp = DateTime.UtcNow
        };
    }

    private async Task<CanaryComparison> CompareCanaryToStableAsync()
    {
        var query = @"
            let timeRange = 5m;
            requests
            | where timestamp > ago(timeRange)
            | summarize 
                ErrorRate = 100.0 * countif(success == false) / count(),
                P95Latency = percentile(duration, 95)
                by DeploymentRing = tostring(customDimensions.DeploymentRing)
        ";

        var results = await ExecuteAnalyticsQueryAsync(query);

        var canary = results.FirstOrDefault(r => r.DeploymentRing == "Canary");
        var stable = results.FirstOrDefault(r => r.DeploymentRing == "Stable");

        if (canary == null || stable == null)
        {
            return new CanaryComparison();
        }

        return new CanaryComparison
        {
            ErrorRateDelta = canary.ErrorRate - stable.ErrorRate,
            LatencyDelta = canary.P95Latency - stable.P95Latency
        };
    }

    private async Task SendAlertAsync(MetricsEvaluation evaluation)
    {
        // Send to multiple channels
        var tasks = new[]
        {
            SendSlackAlertAsync(evaluation),
            SendEmailAlertAsync(evaluation),
            SendPagerDutyAlertAsync(evaluation)
        };

        await Task.WhenAll(tasks);
    }

    private async Task SendSlackAlertAsync(MetricsEvaluation evaluation)
    {
        var slackWebhook = _configuration["Alerts:SlackWebhook"];
        if (string.IsNullOrEmpty(slackWebhook))
            return;

        var message = new
        {
            text = "🚨 Deployment Metrics Guard Violation",
            blocks = new[]
            {
                new
                {
                    type = "section",
                    text = new
                    {
                        type = "mrkdwn",
                        text = $"*Automated rollback triggered*\n{string.Join("\n", evaluation.Violations)}"
                    }
                },
                new
                {
                    type = "section",
                    fields = new[]
                    {
                        new { type = "mrkdwn", text = $"*Error Rate:*\n{evaluation.Metrics.ErrorRate:F2}%" },
                        new { type = "mrkdwn", text = $"*P95 Latency:*\n{evaluation.Metrics.P95Latency:F0}ms" },
                        new { type = "mrkdwn", text = $"*P99 Latency:*\n{evaluation.Metrics.P99Latency:F0}ms" },
                        new { type = "mrkdwn", text = $"*Timestamp:*\n{evaluation.Timestamp:u}" }
                    }
                }
            }
        };

        using var httpClient = new HttpClient();
        await httpClient.PostAsJsonAsync(slackWebhook, message);
    }
}

public class MetricsGuardConfiguration
{
    public TimeSpan EvaluationInterval { get; set; } = TimeSpan.FromMinutes(1);
    public double MaxErrorRatePercent { get; set; } = 5.0;
    public double MaxP95LatencyMs { get; set; } = 1000;
    public double MaxP99LatencyMs { get; set; } = 2000;
    public int MinimumTrafficForEvaluation { get; set; } = 100;
    public double MaxErrorRateDeltaPercent { get; set; } = 2.0;
}

public class MetricsEvaluation
{
    public bool IsHealthy { get; set; }
    public List<string> Violations { get; set; } = new();
    public MetricsSnapshot Metrics { get; set; } = new();
    public DateTime Timestamp { get; set; }
}

public class MetricsSnapshot
{
    public long TotalRequests { get; set; }
    public long FailedRequests { get; set; }
    public double ErrorRate { get; set; }
    public double P50Latency { get; set; }
    public double P95Latency { get; set; }
    public double P99Latency { get; set; }
}

public class CanaryComparison
{
    public double ErrorRateDelta { get; set; }
    public double LatencyDelta { get; set; }
}

Prometheus-Based Metrics Guards

For Kubernetes deployments, use Prometheus for metrics collection:

# PrometheusRule for automated alerts
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: canary-deployment-rules
  namespace: production
spec:
  groups:
  - name: canary-metrics
    interval: 30s
    rules:
    # High error rate alert
    - alert: CanaryHighErrorRate
      expr: |
        (
          sum(rate(http_requests_total{deployment="canary",status=~"5.."}[5m]))
          /
          sum(rate(http_requests_total{deployment="canary"}[5m]))
        ) > 0.05
      for: 2m
      labels:
        severity: critical
        deployment: canary
      annotations:
        summary: "Canary deployment has high error rate"
        description: "Error rate is {{ $value | humanizePercentage }}"
        dashboard: "https://grafana/d/canary-metrics"

    # High latency alert
    - alert: CanaryHighLatency
      expr: |
        histogram_quantile(0.95,
          sum(rate(http_request_duration_seconds_bucket{
            deployment="canary"
          }[5m])) by (le)
        ) > 1.0
      for: 2m
      labels:
        severity: critical
        deployment: canary
      annotations:
        summary: "Canary deployment has high latency"
        description: "P95 latency is {{ $value }}s"

    # Error rate comparison
    - alert: CanaryErrorRateHigherThanStable
      expr: |
        (
          sum(rate(http_requests_total{deployment="canary",status=~"5.."}[5m]))
          /
          sum(rate(http_requests_total{deployment="canary"}[5m]))
        )
        >
        (
          sum(rate(http_requests_total{deployment="stable",status=~"5.."}[5m]))
          /
          sum(rate(http_requests_total{deployment="stable"}[5m]))
        ) + 0.02
      for: 3m
      labels:
        severity: warning
        deployment: canary
      annotations:
        summary: "Canary error rate significantly higher than stable"
        description: "Canary error rate is {{ $value | humanizePercentage }} higher"
---
# AlertManager configuration for rollback webhook
apiVersion: v1
kind: ConfigMap
metadata:
  name: alertmanager-config
  namespace: monitoring
data:
  alertmanager.yml: |
    global:
      resolve_timeout: 5m

    route:
      group_by: ['alertname', 'deployment']
      group_wait: 10s
      group_interval: 10s
      repeat_interval: 12h
      receiver: 'rollback-webhook'
      routes:
      - match:
          severity: critical
          deployment: canary
        receiver: 'rollback-webhook'

    receivers:
    - name: 'rollback-webhook'
      webhook_configs:
      - url: 'http://deployment-controller.production.svc.cluster.local:8080/rollback'
        send_resolved: true

Automated Rollback Service

Implement automated rollback triggered by metrics violations:

public class AutomatedRollbackService
{
    private readonly ILogger<AutomatedRollbackService> _logger;
    private readonly IKubernetesClient _k8sClient;
    private readonly INotificationService _notificationService;
    private readonly TelemetryClient _telemetry;

    public AutomatedRollbackService(
        ILogger<AutomatedRollbackService> logger,
        IKubernetesClient k8sClient,
        INotificationService notificationService,
        TelemetryClient telemetry)
    {
        _logger = logger;
        _k8sClient = k8sClient;
        _notificationService = notificationService;
        _telemetry = telemetry;
    }

    public async Task RollbackDeploymentAsync(
        string deploymentName,
        string reason,
        CancellationToken cancellationToken = default)
    {
        var startTime = DateTime.UtcNow;

        _logger.LogWarning(
            "Initiating automated rollback for {Deployment}. Reason: {Reason}",
            deploymentName, reason);

        _telemetry.TrackEvent("AutomatedRollback", new Dictionary<string, string>
        {
            ["Deployment"] = deploymentName,
            ["Reason"] = reason,
            ["Timestamp"] = startTime.ToString("O")
        });

        try
        {
            // Using Argo Rollouts
            await _k8sClient.AbortRolloutAsync(deploymentName, cancellationToken);

            _logger.LogInformation(
                "Rollback completed successfully for {Deployment} in {Duration}ms",
                deploymentName,
                (DateTime.UtcNow - startTime).TotalMilliseconds);

            _telemetry.TrackMetric(
                "RollbackDuration",
                (DateTime.UtcNow - startTime).TotalMilliseconds,
                new Dictionary<string, string>
                {
                    ["Deployment"] = deploymentName,
                    ["Success"] = "true"
                });

            // Send success notifications
            await _notificationService.SendAsync(new Notification
            {
                Title = $"✅ Rollback Completed: {deploymentName}",
                Message = $"Deployment rolled back successfully due to: {reason}",
                Severity = NotificationSeverity.High,
                Channels = new[] { NotificationChannel.Slack, NotificationChannel.Email }
            });
        }
        catch (Exception ex)
        {
            _logger.LogError(ex,
                "Failed to rollback {Deployment}",
                deploymentName);

            _telemetry.TrackException(ex, new Dictionary<string, string>
            {
                ["Deployment"] = deploymentName,
                ["Operation"] = "Rollback"
            });

            // Escalate to on-call
            await _notificationService.SendAsync(new Notification
            {
                Title = $"🚨 URGENT: Rollback Failed for {deploymentName}",
                Message = $"Automated rollback failed. Manual intervention required. Error: {ex.Message}",
                Severity = NotificationSeverity.Critical,
                Channels = new[] 
                { 
                    NotificationChannel.PagerDuty,
                    NotificationChannel.Slack,
                    NotificationChannel.Email 
                }
            });

            throw;
        }
    }

    public async Task RollbackAzureAppServiceAsync(
        string appName,
        string resourceGroup,
        string reason)
    {
        _logger.LogWarning(
            "Initiating Azure App Service rollback for {AppName}. Reason: {Reason}",
            appName, reason);

        try
        {
            // Swap slots back to previous version
            var command = $"az webapp deployment slot swap " +
                         $"--name {appName} " +
                         $"--resource-group {resourceGroup} " +
                         $"--slot green " +
                         $"--target-slot production";

            await ExecuteAzureCliCommandAsync(command);

            _logger.LogInformation(
                "Azure App Service rollback completed for {AppName}",
                appName);
        }
        catch (Exception ex)
        {
            _logger.LogError(ex,
                "Failed to rollback Azure App Service {AppName}",
                appName);
            throw;
        }
    }
}

Canary Comparison Dashboard

Create a comparison service to visualize canary vs stable metrics:

public class CanaryComparisonService
{
    private readonly IMetricsProvider _metricsProvider;
    private readonly ILogger<CanaryComparisonService> _logger;

    public async Task<ComparisonResult> CompareVersionsAsync(
        CancellationToken cancellationToken = default)
    {
        var canaryMetrics = await _metricsProvider.GetMetricsAsync(
            "canary", 
            TimeSpan.FromMinutes(5),
            cancellationToken);

        var stableMetrics = await _metricsProvider.GetMetricsAsync(
            "stable",
            TimeSpan.FromMinutes(5),
            cancellationToken);

        var comparison = new ComparisonResult
        {
            CanaryMetrics = canaryMetrics,
            StableMetrics = stableMetrics,
            ErrorRateDelta = canaryMetrics.ErrorRate - stableMetrics.ErrorRate,
            LatencyDelta = canaryMetrics.P95Latency - stableMetrics.P95Latency,
            ThroughputDelta = canaryMetrics.Throughput - stableMetrics.Throughput,
            Timestamp = DateTime.UtcNow
        };

        // Decision logic
        if (comparison.ErrorRateDelta > 0.02) // 2% increase
        {
            comparison.Recommendation = DeploymentDecision.Rollback;
            comparison.Reason = $"Error rate increased by {comparison.ErrorRateDelta:P2}";
            comparison.Confidence = ConfidenceLevel.High;
        }
        else if (comparison.LatencyDelta > 200) // 200ms increase
        {
            comparison.Recommendation = DeploymentDecision.Rollback;
            comparison.Reason = $"P95 latency increased by {comparison.LatencyDelta:F0}ms";
            comparison.Confidence = ConfidenceLevel.High;
        }
        else if (comparison.ErrorRateDelta < -0.01 && comparison.LatencyDelta < -50)
        {
            comparison.Recommendation = DeploymentDecision.Proceed;
            comparison.Reason = "Canary shows significant improvement";
            comparison.Confidence = ConfidenceLevel.High;
        }
        else if (Math.Abs(comparison.ErrorRateDelta) < 0.005 && 
                 Math.Abs(comparison.LatencyDelta) < 50)
        {
            comparison.Recommendation = DeploymentDecision.Proceed;
            comparison.Reason = "Metrics are comparable";
            comparison.Confidence = ConfidenceLevel.Medium;
        }
        else
        {
            comparison.Recommendation = DeploymentDecision.ContinueMonitoring;
            comparison.Reason = "Metrics within acceptable range but require more data";
            comparison.Confidence = ConfidenceLevel.Low;
        }

        _logger.LogInformation(
            "Canary comparison: {Recommendation} (Confidence: {Confidence}). {Reason}",
            comparison.Recommendation,
            comparison.Confidence,
            comparison.Reason);

        return comparison;
    }
}

public class ComparisonResult
{
    public DeploymentMetrics CanaryMetrics { get; set; }
    public DeploymentMetrics StableMetrics { get; set; }
    public double ErrorRateDelta { get; set; }
    public double LatencyDelta { get; set; }
    public double ThroughputDelta { get; set; }
    public DeploymentDecision Recommendation { get; set; }
    public ConfidenceLevel Confidence { get; set; }
    public string Reason { get; set; }
    public DateTime Timestamp { get; set; }
}

public enum DeploymentDecision
{
    Proceed,
    Rollback,
    ContinueMonitoring
}

public enum ConfidenceLevel
{
    Low,
    Medium,
    High
}

Best Practices Summary

Blue-Green Deployments:

Maintain identical environments for accurate testing
Always run comprehensive smoke tests before switching
Automate the swap process to reduce human error
Plan database migrations carefully with backward compatibility
Keep both environments for quick rollback
Monitor both environments during transition

Canary Deployments:

Start with small traffic percentages (5-10%)
Define clear success criteria before deployment
Monitor business metrics, not just technical metrics
Use automated analysis to make promotion decisions
Have rollback procedures ready and tested
Communicate rollout progress to stakeholders

Metrics Guards:

Establish baseline metrics from stable deployments
Set realistic thresholds based on historical data
Monitor both absolute values and deltas
Combine multiple metrics for better accuracy
Automate rollback decisions for critical violations
Include business metrics in evaluation

General Deployment Practices:

Always deploy during low-traffic periods when possible
Implement comprehensive health checks
Use feature flags to decouple deployment from release
Maintain detailed deployment logs and audit trails
Practice rollback procedures regularly
Document incident response procedures

Azure-Specific:

Leverage Azure App Service deployment slots for blue-green
Use Azure Traffic Manager for weighted traffic distribution
Enable Application Insights for metrics collection
Configure Azure Monitor alerts for automated responses
Use Azure DevOps pipelines for deployment automation
Implement Azure Policy for deployment guardrails

Kubernetes-Specific:

Use Argo Rollouts for advanced deployment strategies
Configure Prometheus for metrics collection
Implement proper liveness and readiness probes
Use service mesh (Istio, Linkerd) for advanced traffic control
Set resource limits and requests appropriately
Monitor pod health and restart patterns

By implementing these deployment strategies with proper metrics guards and automated rollback, teams can deploy with confidence, minimize user impact from issues, and maintain high availability in production environments.

Why Avoid Service Locators in Dependency Injection?

Hossein Esmati — Fri, 26 Jun 2026 00:34:52 +0000

Why Avoid Service Locators?

Service Locator is an anti-pattern where you inject IServiceProvider and manually resolve dependencies:

// ❌ Bad - Service Locator pattern
public class OrderService
{
    private readonly IServiceProvider _serviceProvider;

    public OrderService(IServiceProvider serviceProvider)
    {
        _serviceProvider = serviceProvider;
    }

    public void ProcessOrder()
    {
        var repo = _serviceProvider.GetRequiredService<IOrderRepository>();
        var emailer = _serviceProvider.GetRequiredService<IEmailService>();
        // ...
    }
}

Problems with this approach:

Hidden dependencies - You can't tell what the class needs just by looking at its constructor. Dependencies are obscured inside the implementation.
Runtime failures - Missing dependencies only fail at runtime when that code path executes, not at application startup.
Hard to test - You have to mock IServiceProvider and set up complex mock behaviors instead of just passing in the dependencies.
Breaks IoC principle - The class is now coupled to the DI container itself, defeating the purpose of dependency injection.

Better approach - Constructor Injection:

// ✅ Good - Dependencies are explicit
public class OrderService
{
    private readonly IOrderRepository _repository;
    private readonly IEmailService _emailer;

    public OrderService(IOrderRepository repository, IEmailService emailer)
    {
        _repository = repository;
        _emailer = emailer;
    }

    public void ProcessOrder()
    {
        // Use _repository and _emailer directly
    }
}

Why Isolate Registrations Per Module?

Isolation means organizing your DI registrations by feature/module rather than dumping everything in Program.cs:

// ❌ Bad - Everything in one place
public class Program
{
    public static void Main(string[] args)
    {
        var builder = WebApplication.CreateBuilder(args);

        // 200+ lines of service registrations for all modules...
        builder.Services.AddScoped<IOrderRepository, OrderRepository>();
        builder.Services.AddScoped<IOrderService, OrderService>();
        builder.Services.AddScoped<IProductRepository, ProductRepository>();
        builder.Services.AddScoped<IUserRepository, UserRepository>();
        builder.Services.AddScoped<IAuthService, AuthService>();
        // ... many more
    }
}

Better approach - Extension methods per module:

// ✅ Good - Orders module owns its registrations
public static class OrdersServiceExtensions
{
    public static IServiceCollection AddOrdersModule(this IServiceCollection services)
    {
        services.AddScoped<IOrderRepository, OrderRepository>();
        services.AddScoped<IOrderService, OrderService>();
        services.AddScoped<IOrderValidator, OrderValidator>();
        return services;
    }
}

// Auth module
public static class AuthServiceExtensions
{
    public static IServiceCollection AddAuthModule(this IServiceCollection services)
    {
        services.AddScoped<IUserRepository, UserRepository>();
        services.AddScoped<IAuthService, AuthService>();
        return services;
    }
}

// Program.cs stays clean
builder.Services.AddOrdersModule();
builder.Services.AddAuthModule();
builder.Services.AddProductsModule();

Benefits:

Maintainability - Each module's dependencies are colocated with that module's code.
Discoverability - Easy to find what services a module provides.
Modularity - Modules can be enabled/disabled, or even moved to separate assemblies.
Testability - You can register just the modules needed for integration tests.
Clean startup - Program.cs remains readable and high-level.

Data Consistency Patterns in Distributed Systems and .NET Core

Hossein Esmati — Fri, 26 Jun 2026 00:34:49 +0000

This article is part of the Comprehensive Guide to Microservices Architecture in .NET Core, Cloud and Azure series.

Comparison of Patterns

Pattern	Complexity	Consistency	Performance	Use Case
Transactional Outbox	Medium	Strong	Good	General-purpose, reliable event publishing
CDC	Low	Strong	Excellent	Existing systems, minimal code changes
Event Sourcing	High	Strong	Excellent	Audit requirements, temporal queries
Saga Pattern	High	Eventual	Good	Complex distributed workflows

Best Practices and Decision Tree

For Transactional Outbox:

Use background services with proper error handling and retry logic
Implement idempotency in message consumers
Monitor outbox table size and clean up processed messages
Consider partitioning outbox table for high-volume scenarios

For CDC:

Regularly monitor change tracking overhead on the database
Configure appropriate retention periods
Handle schema changes carefully to avoid breaking CDC
Test CDC processor recovery after failures

For All Patterns:

Implement comprehensive logging and monitoring
Use distributed tracing to track operations across services
Design for idempotency at every level
Plan for failure scenarios and compensating actions
Consider using Azure Monitor and Application Insights for observability

Azure-Specific Considerations

When implementing these patterns on Azure:

Use Azure SQL Database with built-in CDC support
Leverage Azure Service Bus for reliable message delivery with features like dead-letter queues and scheduled messages
Implement Azure Functions as lightweight CDC processors for serverless scenarios
Use Azure Cosmos DB change feed as an alternative to CDC for NoSQL scenarios
Enable Application Insights for end-to-end transaction tracking
Consider Azure Durable Functions for implementing saga patterns with built-in state management

The Anti-Pattern

// ANTI-PATTERN: Dual write (not atomic)
public async Task CreateOrderAsync(Order order)
{
    await _database.SaveAsync(order); // Write 1
    await _serviceBus.PublishAsync(new OrderCreated(order.Id)); // Write 2

    // If the publish fails after the database save succeeds,
    // we have inconsistency - the order exists but no event was published!
}

This approach has several critical issues:

No atomicity between database write and message publication
Partial failures leave the system in an inconsistent state
Manual compensation logic is complex and error-prone
Difficult to recover from failures

Solution 1: Transactional Outbox Pattern

The Transactional Outbox pattern ensures atomicity by storing events in the same database transaction as the business data, then publishing them asynchronously. You can read more about distributed transactions in .NET core and Azure in here.

Implementation with EF Core 9 and Azure Service Bus

// Outbox message entity
public class OutboxMessage
{
    public Guid Id { get; set; }
    public string EventType { get; set; }
    public string Payload { get; set; }
    public DateTime CreatedAt { get; set; }
    public DateTime? ProcessedAt { get; set; }
    public int RetryCount { get; set; }
    public string? Error { get; set; }
}

// DbContext with outbox
public class AppDbContext : DbContext
{
    public DbSet<Order> Orders { get; set; }
    public DbSet<OutboxMessage> OutboxMessages { get; set; }

    public AppDbContext(DbContextOptions<AppDbContext> options) 
        : base(options)
    {
    }

    protected override void OnModelCreating(ModelBuilder modelBuilder)
    {
        modelBuilder.Entity<OutboxMessage>()
            .HasIndex(m => new { m.ProcessedAt, m.CreatedAt })
            .HasFilter("[ProcessedAt] IS NULL");
    }
}

// Service with transactional outbox
public class OrderService
{
    private readonly AppDbContext _context;
    private readonly ILogger<OrderService> _logger;

    public OrderService(AppDbContext context, ILogger<OrderService> logger)
    {
        _context = context;
        _logger = logger;
    }

    public async Task<Guid> CreateOrderAsync(CreateOrderRequest request)
    {
        // Use ExecuteInTransactionAsync for .NET 9
        await _context.Database.CreateExecutionStrategy().ExecuteInTransactionAsync(
            async () =>
            {
                var order = new Order
                {
                    Id = Guid.NewGuid(),
                    CustomerId = request.CustomerId,
                    TotalAmount = request.Items.Sum(i => i.Quantity * i.Price),
                    Status = OrderStatus.Created,
                    CreatedAt = DateTime.UtcNow
                };

                _context.Orders.Add(order);

                // Store event in outbox within the same transaction
                var outboxMessage = new OutboxMessage
                {
                    Id = Guid.NewGuid(),
                    EventType = nameof(OrderCreatedEvent),
                    Payload = JsonSerializer.Serialize(new OrderCreatedEvent
                    {
                        OrderId = order.Id,
                        CustomerId = order.CustomerId,
                        CreatedAt = order.CreatedAt
                    }),
                    CreatedAt = DateTime.UtcNow
                };

                _context.OutboxMessages.Add(outboxMessage);

                await _context.SaveChangesAsync();

                _logger.LogInformation(
                    "Order {OrderId} created with outbox message {MessageId}",
                    order.Id, outboxMessage.Id);

                return order.Id;
            },
            verifySucceeded: null);

        return await Task.FromResult(Guid.Empty); // This will be set by the transaction
    }
}

// Outbox processor with Azure Service Bus
public class OutboxProcessor : BackgroundService
{
    private readonly IServiceProvider _serviceProvider;
    private readonly ServiceBusSender _serviceBusSender;
    private readonly ILogger<OutboxProcessor> _logger;
    private const int BatchSize = 100;
    private const int MaxRetries = 5;

    public OutboxProcessor(
        IServiceProvider serviceProvider,
        ServiceBusClient serviceBusClient,
        ILogger<OutboxProcessor> logger)
    {
        _serviceProvider = serviceProvider;
        _serviceBusSender = serviceBusClient.CreateSender("order-events");
        _logger = logger;
    }

    protected override async Task ExecuteAsync(CancellationToken stoppingToken)
    {
        _logger.LogInformation("Outbox processor started");

        while (!stoppingToken.IsCancellationRequested)
        {
            try
            {
                await ProcessOutboxMessagesAsync(stoppingToken);
                await Task.Delay(TimeSpan.FromSeconds(5), stoppingToken);
            }
            catch (Exception ex)
            {
                _logger.LogError(ex, "Error processing outbox messages");
                await Task.Delay(TimeSpan.FromSeconds(30), stoppingToken);
            }
        }
    }

    private async Task ProcessOutboxMessagesAsync(CancellationToken cancellationToken)
    {
        using var scope = _serviceProvider.CreateScope();
        var context = scope.ServiceProvider.GetRequiredService<AppDbContext>();

        // Fetch unprocessed messages
        var messages = await context.OutboxMessages
            .Where(m => m.ProcessedAt == null && m.RetryCount < MaxRetries)
            .OrderBy(m => m.CreatedAt)
            .Take(BatchSize)
            .ToListAsync(cancellationToken);

        if (!messages.Any())
            return;

        _logger.LogInformation("Processing {Count} outbox messages", messages.Count);

        foreach (var message in messages)
        {
            try
            {
                // Publish to Azure Service Bus
                var serviceBusMessage = new ServiceBusMessage(message.Payload)
                {
                    MessageId = message.Id.ToString(),
                    Subject = message.EventType,
                    ContentType = "application/json"
                };

                await _serviceBusSender.SendMessageAsync(serviceBusMessage, cancellationToken);

                // Mark as processed
                message.ProcessedAt = DateTime.UtcNow;

                _logger.LogInformation(
                    "Outbox message {MessageId} published successfully",
                    message.Id);
            }
            catch (Exception ex)
            {
                message.RetryCount++;
                message.Error = ex.Message;

                _logger.LogWarning(ex,
                    "Failed to process outbox message {MessageId}. Retry count: {RetryCount}",
                    message.Id, message.RetryCount);
            }
        }

        await context.SaveChangesAsync(cancellationToken);
    }

    public override async Task StopAsync(CancellationToken cancellationToken)
    {
        _logger.LogInformation("Outbox processor stopping");
        await _serviceBusSender.CloseAsync();
        await base.StopAsync(cancellationToken);
    }
}

// Register services in Program.cs
builder.Services.AddDbContext<AppDbContext>(options =>
    options.UseSqlServer(builder.Configuration.GetConnectionString("DefaultConnection")));

builder.Services.AddSingleton(sp =>
    new ServiceBusClient(builder.Configuration.GetConnectionString("ServiceBus")));

builder.Services.AddHostedService<OutboxProcessor>();

Solution 2: Change Data Capture (CDC)

Change Data Capture automatically tracks changes in your database and publishes them as events, eliminating the need for application-level dual writes.

Implementation with SQL Server CDC and Azure

// Enable CDC on SQL Server
// Execute these SQL commands on your Azure SQL Database:
/*
ALTER DATABASE YourDatabase SET CHANGE_TRACKING = ON  
    (CHANGE_RETENTION = 2 DAYS, AUTO_CLEANUP = ON);

ALTER TABLE Orders ENABLE CHANGE_TRACKING  
    WITH (TRACK_COLUMNS_UPDATED = ON);
*/

// Change tracking model
public class OrderChange
{
    public Guid OrderId { get; set; }
    public Guid CustomerId { get; set; }
    public string Operation { get; set; } // I, U, D
    public long ChangeVersion { get; set; }
}

// CDC Processor with .NET 9
public class CdcProcessor : BackgroundService
{
    private readonly IServiceProvider _serviceProvider;
    private readonly ServiceBusSender _serviceBusSender;
    private readonly ILogger<CdcProcessor> _logger;
    private long _lastSyncVersion;

    public CdcProcessor(
        IServiceProvider serviceProvider,
        ServiceBusClient serviceBusClient,
        ILogger<CdcProcessor> logger)
    {
        _serviceProvider = serviceProvider;
        _serviceBusSender = serviceBusClient.CreateSender("order-events");
        _logger = logger;
    }

    protected override async Task ExecuteAsync(CancellationToken stoppingToken)
    {
        _logger.LogInformation("CDC processor started");

        // Initialize last sync version
        _lastSyncVersion = await GetCurrentChangeVersionAsync();

        while (!stoppingToken.IsCancellationRequested)
        {
            try
            {
                await ProcessChangesAsync(stoppingToken);
                await Task.Delay(TimeSpan.FromSeconds(5), stoppingToken);
            }
            catch (Exception ex)
            {
                _logger.LogError(ex, "Error processing CDC changes");
                await Task.Delay(TimeSpan.FromSeconds(30), stoppingToken);
            }
        }
    }

    private async Task ProcessChangesAsync(CancellationToken cancellationToken)
    {
        using var scope = _serviceProvider.CreateScope();
        var context = scope.ServiceProvider.GetRequiredService<AppDbContext>();

        // Query change tracking using FormattableString for safety
        var currentVersion = await GetCurrentChangeVersionAsync();

        var changes = await context.Database
            .SqlQuery<OrderChange>(
                $@"SELECT o.OrderId, o.CustomerId, 
                      CT.SYS_CHANGE_OPERATION as Operation,
                      CT.SYS_CHANGE_VERSION as ChangeVersion
                   FROM Orders o
                   RIGHT OUTER JOIN CHANGETABLE(CHANGES Orders, {_lastSyncVersion}) AS CT
                      ON o.OrderId = CT.OrderId
                   WHERE CT.SYS_CHANGE_VERSION <= {currentVersion}
                   ORDER BY CT.SYS_CHANGE_VERSION")
            .ToListAsync(cancellationToken);

        if (!changes.Any())
            return;

        _logger.LogInformation("Processing {Count} changes", changes.Count);

        foreach (var change in changes)
        {
            try
            {
                var eventMessage = change.Operation switch
                {
                    "I" => new ServiceBusMessage(JsonSerializer.Serialize(
                        new OrderCreatedEvent
                        {
                            OrderId = change.OrderId,
                            CustomerId = change.CustomerId
                        }))
                    {
                        Subject = nameof(OrderCreatedEvent)
                    },

                    "U" => new ServiceBusMessage(JsonSerializer.Serialize(
                        new OrderUpdatedEvent
                        {
                            OrderId = change.OrderId
                        }))
                    {
                        Subject = nameof(OrderUpdatedEvent)
                    },

                    "D" => new ServiceBusMessage(JsonSerializer.Serialize(
                        new OrderDeletedEvent
                        {
                            OrderId = change.OrderId
                        }))
                    {
                        Subject = nameof(OrderDeletedEvent)
                    },

                    _ => throw new InvalidOperationException(
                        $"Unknown operation: {change.Operation}")
                };

                eventMessage.MessageId = Guid.NewGuid().ToString();
                eventMessage.ContentType = "application/json";

                await _serviceBusSender.SendMessageAsync(eventMessage, cancellationToken);

                _logger.LogInformation(
                    "Published {EventType} for order {OrderId}",
                    eventMessage.Subject, change.OrderId);

                // Update last sync version
                _lastSyncVersion = Math.Max(_lastSyncVersion, change.ChangeVersion);
            }
            catch (Exception ex)
            {
                _logger.LogError(ex,
                    "Failed to publish event for order {OrderId}",
                    change.OrderId);
            }
        }
    }

    private async Task<long> GetCurrentChangeVersionAsync()
    {
        using var scope = _serviceProvider.CreateScope();
        var context = scope.ServiceProvider.GetRequiredService<AppDbContext>();

        var version = await context.Database
            .SqlQuery<long>($"SELECT CHANGE_TRACKING_CURRENT_VERSION()")
            .FirstOrDefaultAsync();

        return version;
    }

    public override async Task StopAsync(CancellationToken cancellationToken)
    {
        _logger.LogInformation("CDC processor stopping");
        await _serviceBusSender.CloseAsync();
        await base.StopAsync(cancellationToken);
    }
}

Solution 3: Event Sourcing

Event Sourcing naturally solves the dual write problem by treating events as the single source of truth. See the dedicated Event Sourcing article for a complete implementation.

Solution 4: Saga Pattern for Distributed Transactions

For complex workflows spanning multiple services, the Saga pattern coordinates distributed transactions through compensating actions.

// Saga state machine using MassTransit
public class OrderSaga : MassTransitStateMachine<OrderSagaState>
{
    public State OrderCreated { get; private set; }
    public State PaymentProcessing { get; private set; }
    public State InventoryReserving { get; private set; }
    public State OrderCompleted { get; private set; }
    public State OrderFailed { get; private set; }

    public Event<OrderSubmitted> OrderSubmitted { get; private set; }
    public Event<PaymentSucceeded> PaymentSucceeded { get; private set; }
    public Event<PaymentFailed> PaymentFailed { get; private set; }
    public Event<InventoryReserved> InventoryReserved { get; private set; }
    public Event<InventoryReservationFailed> InventoryReservationFailed { get; private set; }

    public OrderSaga()
    {
        InstanceState(x => x.CurrentState);

        Event(() => OrderSubmitted, x => x.CorrelateById(m => m.Message.OrderId));
        Event(() => PaymentSucceeded, x => x.CorrelateById(m => m.Message.OrderId));
        Event(() => PaymentFailed, x => x.CorrelateById(m => m.Message.OrderId));
        Event(() => InventoryReserved, x => x.CorrelateById(m => m.Message.OrderId));
        Event(() => InventoryReservationFailed, x => x.CorrelateById(m => m.Message.OrderId));

        Initially(
            When(OrderSubmitted)
                .Then(context =>
                {
                    context.Saga.OrderId = context.Message.OrderId;
                    context.Saga.CustomerId = context.Message.CustomerId;
                })
                .TransitionTo(OrderCreated)
                .PublishAsync(context => context.Init<ProcessPayment>(new
                {
                    context.Message.OrderId,
                    context.Message.Amount
                })));

        During(OrderCreated,
            When(PaymentSucceeded)
                .TransitionTo(PaymentProcessing)
                .PublishAsync(context => context.Init<ReserveInventory>(new
                {
                    context.Message.OrderId,
                    context.Saga.CustomerId
                })),

            When(PaymentFailed)
                .TransitionTo(OrderFailed)
                .ThenAsync(async context =>
                {
                    // Compensate: Cancel order
                    await context.PublishAsync(new OrderCancelled
                    {
                        OrderId = context.Message.OrderId,
                        Reason = "Payment failed"
                    });
                }));

        During(PaymentProcessing,
            When(InventoryReserved)
                .TransitionTo(OrderCompleted)
                .Finalize(),

            When(InventoryReservationFailed)
                .TransitionTo(OrderFailed)
                .ThenAsync(async context =>
                {
                    // Compensate: Refund payment
                    await context.PublishAsync(new RefundPayment
                    {
                        OrderId = context.Message.OrderId
                    });
                })
                .ThenAsync(async context =>
                {
                    // Cancel order
                    await context.PublishAsync(new OrderCancelled
                    {
                        OrderId = context.Message.OrderId,
                        Reason = "Inventory unavailable"
                    });
                }));
    }
}

public class OrderSagaState : SagaStateMachineInstance
{
    public Guid CorrelationId { get; set; }
    public string CurrentState { get; set; }
    public Guid OrderId { get; set; }
    public Guid CustomerId { get; set; }
}

// Configure in Program.cs
builder.Services.AddMassTransit(x =>
{
    x.AddSagaStateMachine<OrderSaga, OrderSagaState>()
        .EntityFrameworkRepository(r =>
        {
            r.ExistingDbContext<AppDbContext>();
            r.UseSqlServer();
        });

    x.UsingAzureServiceBus((context, cfg) =>
    {
        cfg.Host(builder.Configuration.GetConnectionString("ServiceBus"));
        cfg.ConfigureEndpoints(context);
    });
});

CQRS with Separate Read/Write Models in .NET Core

Hossein Esmati — Fri, 26 Jun 2026 00:29:06 +0000

This article is part of the Comprehensive Guide to Microservices Architecture in .NET Core, Cloud and Azure series.

Understanding CQRS

Command Query Responsibility Segregation (CQRS) is an architectural pattern that separates read operations (queries) from write operations (commands).

Benefits and Trade-offs

Advantages:

Independent Scaling: Scale read and write workloads separately based on demand
Optimized Performance: Denormalized read models eliminate complex joins
Flexible Querying: Create multiple read models tailored to different query patterns
Technology Diversity: Use different databases for reads (e.g., Elasticsearch) and writes (e.g., PostgreSQL)

Challenges:

Eventual Consistency: Read model updates lag behind write operations
Increased Complexity: Maintaining separate models and synchronization logic
Data Duplication: Same data exists in multiple forms across models
Debugging Difficulty: Tracking issues across distributed components

When to Use CQRS:

High-traffic systems with significantly different read and write patterns
Applications requiring multiple specialized views of the same data
Systems where read performance is critical (reporting, analytics)
Domains with complex business logic that benefits from separation of concerns

When to Avoid CQRS:

Simple CRUD applications with balanced read/write operations
Small-scale systems where the added complexity isn't justified
Teams unfamiliar with eventual consistency patterns
Projects with tight deadlines and limited resources

Write Model (Command Side)

The write model enforces business rules and maintains transactional consistency:

public class OrderWriteModel
{
    public Guid Id { get; set; }
    public Guid CustomerId { get; set; }
    public List<OrderLine> Lines { get; set; }

    // Encapsulates business logic
    public void AddLine(Guid productId, int quantity, decimal price)
    {
        if (quantity <= 0)
            throw new DomainException("Quantity must be positive");

        Lines.Add(new OrderLine(productId, quantity, price));
    }
}

Read Model (Query Side)

The read model is denormalized for optimal query performance, containing pre-computed and aggregated data:

public class OrderReadModel
{
    public Guid Id { get; set; }
    public string CustomerName { get; set; }
    public string CustomerEmail { get; set; }
    public List<OrderItemReadModel> Items { get; set; }
    public decimal TotalAmount { get; set; }
    public string Status { get; set; }
    public DateTime CreatedAt { get; set; }

    // Denormalized fields eliminate joins at query time
    public string ShippingAddress { get; set; }
    public string PaymentMethod { get; set; }
    public DateTime? ShippedAt { get; set; }
}

public class OrderItemReadModel
{
    public string ProductName { get; set; }
    public string ProductImageUrl { get; set; }
    public int Quantity { get; set; }
    public decimal UnitPrice { get; set; }
    public decimal LineTotal { get; set; }
}

Command Handler

Command handlers process write operations and update the write model:

public class CreateOrderCommandHandler : IRequestHandler<CreateOrderCommand, Guid>
{
    private readonly OrderWriteDbContext _writeDb;
    private readonly IEventPublisher _eventPublisher;

    public CreateOrderCommandHandler(
        OrderWriteDbContext writeDb,
        IEventPublisher eventPublisher)
    {
        _writeDb = writeDb;
        _eventPublisher = eventPublisher;
    }

    public async Task<Guid> Handle(
        CreateOrderCommand command,
        CancellationToken cancellationToken)
    {
        var order = new OrderWriteModel
        {
            Id = Guid.NewGuid(),
            CustomerId = command.CustomerId,
            Lines = new List<OrderLine>()
        };

        foreach (var item in command.Items)
        {
            order.AddLine(item.ProductId, item.Quantity, item.Price);
        }

        _writeDb.Orders.Add(order);
        await _writeDb.SaveChangesAsync(cancellationToken);

        // Publish integration event to synchronize read model
        await _eventPublisher.PublishAsync(
            new OrderCreatedEvent(order.Id, order.CustomerId));

        return order.Id;
    }
}

Event Handler

Event handlers listen for changes and update the read model asynchronously:

public class OrderCreatedEventHandler : IEventHandler<OrderCreatedEvent>
{
    private readonly OrderReadDbContext _readDb;
    private readonly ICustomerServiceClient _customerClient;
    private readonly IProductServiceClient _productClient;
    private readonly IOrderServiceClient _orderServiceClient;

    public OrderCreatedEventHandler(
        OrderReadDbContext readDb,
        ICustomerServiceClient customerClient,
        IProductServiceClient productClient,
        IOrderServiceClient orderServiceClient)
    {
        _readDb = readDb;
        _customerClient = customerClient;
        _productClient = productClient;
        _orderServiceClient = orderServiceClient;
    }

    public async Task Handle(OrderCreatedEvent @event)
    {
        // Fetch data from various sources for denormalization
        var customer = await _customerClient.GetCustomerAsync(@event.CustomerId);
        var order = await _orderServiceClient.GetOrderAsync(@event.OrderId);

        var productIds = order.Lines.Select(l => l.ProductId).ToList();
        var products = await _productClient.GetProductsAsync(productIds);

        // Build denormalized read model with all necessary data
        var readModel = new OrderReadModel
        {
            Id = @event.OrderId,
            CustomerName = customer.Name,
            CustomerEmail = customer.Email,
            Items = order.Lines.Select(line => new OrderItemReadModel
            {
                ProductName = products.First(p => p.Id == line.ProductId).Name,
                ProductImageUrl = products.First(p => p.Id == line.ProductId).ImageUrl,
                Quantity = line.Quantity,
                UnitPrice = line.Price,
                LineTotal = line.Quantity * line.Price
            }).ToList(),
            TotalAmount = order.Lines.Sum(l => l.Quantity * l.Price),
            Status = "Created",
            CreatedAt = DateTime.UtcNow
        };

        _readDb.Orders.Add(readModel);
        await _readDb.SaveChangesAsync();
    }
}

Query Handler

Query handlers retrieve data from the optimized read model:

public class GetOrderQueryHandler : IRequestHandler<GetOrderQuery, OrderReadModel>
{
    private readonly OrderReadDbContext _readDb;

    public GetOrderQueryHandler(OrderReadDbContext readDb)
    {
        _readDb = readDb;
    }

    public async Task<OrderReadModel> Handle(
        GetOrderQuery query,
        CancellationToken cancellationToken)
    {
        // Execute fast query on denormalized data with no joins required
        return await _readDb.Orders
            .Include(o => o.Items)
            .FirstOrDefaultAsync(o => o.Id == query.OrderId, cancellationToken);
    }
}

Database Contexts

Separate database contexts allow independent optimization strategies for reads and writes:

Write Database Context

public class OrderWriteDbContext : DbContext
{
    public DbSet<OrderWriteModel> Orders { get; set; }

    public OrderWriteDbContext(DbContextOptions<OrderWriteDbContext> options)
        : base(options)
    {
    }

    protected override void OnModelCreating(ModelBuilder modelBuilder)
    {
        // Minimal indexes optimized for write operations
        modelBuilder.Entity<OrderWriteModel>()
            .HasIndex(o => o.CustomerId);
    }
}

Read Database Context

public class OrderReadDbContext : DbContext
{
    public DbSet<OrderReadModel> Orders { get; set; }

    public OrderReadDbContext(DbContextOptions<OrderReadDbContext> options)
        : base(options)
    {
    }

    protected override void OnModelCreating(ModelBuilder modelBuilder)
    {
        // Multiple indexes optimized for various query patterns
        modelBuilder.Entity<OrderReadModel>()
            .HasIndex(o => o.CustomerEmail);

        modelBuilder.Entity<OrderReadModel>()
            .HasIndex(o => o.CreatedAt);

        modelBuilder.Entity<OrderReadModel>()
            .HasIndex(o => o.Status);

        // Mark as read-only to prevent accidental migrations
        modelBuilder.Entity<OrderReadModel>()
            .ToTable("OrderReadModels", t => t.ExcludeFromMigrations());
    }
}

Contract Stability and Versioning in .NET and Azure

Hossein Esmati — Fri, 26 Jun 2026 00:29:03 +0000

This article is part of the Comprehensive Guide to Microservices Architecture in .NET Core, Cloud and Azure series.

This article explores proven approaches to API versioning in .NET, consumer-driven contract testing with Pact.NET, and best practices for managing deprecation cycles. We'll cover modern techniques for maintaining contract stability in cloud-native applications deployed to Azure.

API Versioning Strategies

Different versioning approaches suit different scenarios. Choose based on your API consumers, organizational standards, and technical constraints.

1. URL Versioning (Recommended)

The most explicit and widely adopted approach, embedding the version directly in the URL path:

// Install package
// dotnet add package Asp.Versioning.Mvc

builder.Services.AddApiVersioning(options =>
{
    options.DefaultApiVersion = new ApiVersion(1, 0);
    options.AssumeDefaultVersionWhenUnspecified = true;
    options.ReportApiVersions = true; // Adds version info to response headers
    options.ApiVersionReader = new UrlSegmentApiVersionReader();
})
.AddApiExplorer(options =>
{
    options.GroupNameFormat = "'v'VVV"; // Format version as 'v1', 'v2', etc.
    options.SubstituteApiVersionInUrl = true;
});

[ApiController]
[Route("api/v{version:apiVersion}/[controller]")]
public class OrdersController : ControllerBase
{
    private readonly IOrderService _orderService;
    private readonly ILogger<OrdersController> _logger;

    public OrdersController(
        IOrderService orderService,
        ILogger<OrdersController> logger)
    {
        _orderService = orderService;
        _logger = logger;
    }

    // Version 1.0 - Original implementation
    [HttpGet("{id}")]
    [MapToApiVersion("1.0")]
    public async Task<ActionResult<OrderV1Dto>> GetV1(Guid id)
    {
        var order = await _orderService.GetOrderByIdAsync(id);

        if (order == null)
            return NotFound();

        return Ok(new OrderV1Dto
        {
            Id = order.Id,
            Total = order.TotalAmount,
            Status = order.Status.ToString()
        });
    }

    // Version 2.0 - Enhanced with additional fields
    [HttpGet("{id}")]
    [MapToApiVersion("2.0")]
    public async Task<ActionResult<OrderV2Dto>> GetV2(Guid id)
    {
        var order = await _orderService.GetOrderByIdAsync(id);

        if (order == null)
            return NotFound();

        return Ok(new OrderV2Dto
        {
            Id = order.Id,
            TotalAmount = order.TotalAmount,
            Currency = order.Currency,
            Status = order.Status.ToString(),
            CustomerInfo = new CustomerInfoDto
            {
                CustomerId = order.CustomerId,
                CustomerName = order.CustomerName
            },
            CreatedAt = order.CreatedAt,
            UpdatedAt = order.UpdatedAt
        });
    }

    // Version 2.0 - New endpoint not available in v1
    [HttpPost]
    [MapToApiVersion("2.0")]
    public async Task<ActionResult<OrderV2Dto>> CreateOrder(CreateOrderRequest request)
    {
        var order = await _orderService.CreateOrderAsync(request);
        return CreatedAtAction(nameof(GetV2), 
            new { id = order.Id, version = "2.0" }, 
            order);
    }
}

// DTOs for different versions
public class OrderV1Dto
{
    public Guid Id { get; set; }
    public decimal Total { get; set; }
    public string Status { get; set; }
}

public class OrderV2Dto
{
    public Guid Id { get; set; }
    public decimal TotalAmount { get; set; }
    public string Currency { get; set; }
    public string Status { get; set; }
    public CustomerInfoDto CustomerInfo { get; set; }
    public DateTime CreatedAt { get; set; }
    public DateTime? UpdatedAt { get; set; }
}

public class CustomerInfoDto
{
    public Guid CustomerId { get; set; }
    public string CustomerName { get; set; }
}

Pros:

Clear and explicit in URLs and documentation
Easy to route and cache
Simple for consumers to understand
Works well with API gateways

Cons:

URLs change between versions
Can lead to code duplication

2. Header Versioning

Version information is passed via HTTP headers, keeping URLs clean:

builder.Services.AddApiVersioning(options =>
{
    options.DefaultApiVersion = new ApiVersion(1, 0);
    options.AssumeDefaultVersionWhenUnspecified = true;
    options.ReportApiVersions = true;
    options.ApiVersionReader = new HeaderApiVersionReader("X-API-Version");
});

[ApiController]
[Route("api/[controller]")]
[ApiVersion("1.0")]
[ApiVersion("2.0")]
public class ProductsController : ControllerBase
{
    [HttpGet("{id}")]
    [MapToApiVersion("1.0")]
    public async Task<ActionResult<ProductV1>> GetV1(Guid id)
    {
        // Implementation
    }

    [HttpGet("{id}")]
    [MapToApiVersion("2.0")]
    public async Task<ActionResult<ProductV2>> GetV2(Guid id)
    {
        // Implementation
    }
}

// Client usage:
// GET /api/products/123
// X-API-Version: 2.0

Pros:

Clean, stable URLs
Good for RESTful purists
Easier URL structure

Cons:

Less discoverable
Harder to test in browsers
Caching complications

3. Media Type Versioning (Content Negotiation)

Version is specified in the Accept header using custom media types:

builder.Services.AddApiVersioning(options =>
{
    options.DefaultApiVersion = new ApiVersion(1, 0);
    options.AssumeDefaultVersionWhenUnspecified = true;
    options.ApiVersionReader = new MediaTypeApiVersionReader();
});

[ApiController]
[Route("api/[controller]")]
public class CustomersController : ControllerBase
{
    [HttpGet("{id}")]
    [MapToApiVersion("1.0")]
    [Produces("application/vnd.company.customer.v1+json")]
    public async Task<ActionResult<CustomerV1>> GetV1(Guid id)
    {
        // Implementation
    }

    [HttpGet("{id}")]
    [MapToApiVersion("2.0")]
    [Produces("application/vnd.company.customer.v2+json")]
    public async Task<ActionResult<CustomerV2>> GetV2(Guid id)
    {
        // Implementation
    }
}

// Client usage:
// GET /api/customers/123
// Accept: application/vnd.company.customer.v2+json

Pros:

RESTful and follows HTTP standards
Clean URLs
Precise control over content negotiation

Cons:

Most complex to implement
Harder for consumers to understand
Limited tooling support

4. Query Parameter Versioning

Version is passed as a query string parameter:

builder.Services.AddApiVersioning(options =>
{
    options.DefaultApiVersion = new ApiVersion(1, 0);
    options.AssumeDefaultVersionWhenUnspecified = true;
    options.ApiVersionReader = new QueryStringApiVersionReader("api-version");
});

[ApiController]
[Route("api/[controller]")]
public class OrdersController : ControllerBase
{
    [HttpGet]
    [MapToApiVersion("1.0")]
    public async Task<ActionResult<IEnumerable<OrderV1>>> GetAllV1()
    {
        // Implementation
    }

    [HttpGet]
    [MapToApiVersion("2.0")]
    public async Task<ActionResult<IEnumerable<OrderV2>>> GetAllV2()
    {
        // Implementation
    }
}

// Client usage:
// GET /api/orders?api-version=2.0

Pros:

Easy to implement
Simple for testing
Optional versioning

Cons:

Query parameters can be lost or modified
Not as clean as URL versioning
Can conflict with other query parameters

Swagger/OpenAPI Integration

Configure Swagger to document multiple API versions:

builder.Services.AddEndpointsApiExplorer();
builder.Services.AddSwaggerGen(options =>
{
    options.SwaggerDoc("v1", new OpenApiInfo
    {
        Version = "v1",
        Title = "Orders API v1",
        Description = "Original version of the Orders API",
        Contact = new OpenApiContact
        {
            Name = "API Support",
            Email = "api-support@company.com"
        }
    });

    options.SwaggerDoc("v2", new OpenApiInfo
    {
        Version = "v2",
        Title = "Orders API v2",
        Description = "Enhanced version with additional customer information",
        Contact = new OpenApiContact
        {
            Name = "API Support",
            Email = "api-support@company.com"
        }
    });

    // Filter endpoints by version
    options.DocInclusionPredicate((docName, apiDesc) =>
    {
        if (!apiDesc.TryGetMethodInfo(out var methodInfo))
            return false;

        var versions = methodInfo.DeclaringType?
            .GetCustomAttributes(true)
            .OfType<ApiVersionAttribute>()
            .SelectMany(attr => attr.Versions);

        return versions?.Any(v => $"v{v}" == docName) ?? false;
    });
});

var app = builder.Build();

app.UseSwagger();
app.UseSwaggerUI(options =>
{
    options.SwaggerEndpoint("/swagger/v1/swagger.json", "Orders API V1");
    options.SwaggerEndpoint("/swagger/v2/swagger.json", "Orders API V2");
});

Consumer-Driven Contracts (CDC) with Pact.NET

Consumer-Driven Contract testing ensures that provider services meet the expectations defined by their consumers. This prevents breaking changes from reaching production.

Setting Up Pact.NET

# Install Pact.NET packages
dotnet add package PactNet
dotnet add package PactNet.Output.Xunit

Consumer Test (Order Service)

The consumer defines its expectations for the Customer Service API:

using PactNet;
using PactNet.Matchers;
using Xunit;
using Xunit.Abstractions;

namespace OrderService.Tests.Pact
{
    public class CustomerServiceConsumerTests : IDisposable
    {
        private readonly IPactBuilderV4 _pactBuilder;
        private readonly ITestOutputHelper _output;

        public CustomerServiceConsumerTests(ITestOutputHelper output)
        {
            _output = output;

            var pactConfig = new PactConfig
            {
                PactDir = Path.Combine("..", "..", "..", "pacts"),
                LogLevel = PactLogLevel.Debug,
                Outputters = new[] { new XunitOutput(_output) }
            };

            _pactBuilder = Pact.V4("OrderService", "CustomerService", pactConfig)
                .WithHttpInteractions();
        }

        [Fact]
        public async Task GetCustomer_ReturnsCustomerDetails()
        {
            // Arrange - Define the expected interaction
            _pactBuilder
                .UponReceiving("A request for customer details")
                    .Given("Customer 123 exists")
                    .WithRequest(HttpMethod.Get, "/api/customers/123")
                    .WithHeader("Accept", "application/json")
                .WillRespond()
                    .WithStatus(HttpStatusCode.OK)
                    .WithHeader("Content-Type", "application/json")
                    .WithJsonBody(new
                    {
                        id = Match.Type("123"),
                        name = Match.Type("John Doe"),
                        email = Match.Regex("john@example.com", "^[\\w-\\.]+@([\\w-]+\\.)+[\\w-]{2,4}$"),
                        status = Match.Type("Active")
                    });

            await _pactBuilder.VerifyAsync(async ctx =>
            {
                // Act - Call the consumer code
                var httpClient = new HttpClient
                {
                    BaseAddress = ctx.MockServerUri
                };

                var customerClient = new CustomerServiceClient(httpClient);
                var customer = await customerClient.GetCustomerAsync("123");

                // Assert - Verify the consumer behaves correctly
                Assert.NotNull(customer);
                Assert.Equal("123", customer.Id);
                Assert.Equal("John Doe", customer.Name);
                Assert.Equal("john@example.com", customer.Email);
                Assert.Equal("Active", customer.Status);
            });
        }

        [Fact]
        public async Task GetCustomer_WhenNotFound_Returns404()
        {
            // Arrange
            _pactBuilder
                .UponReceiving("A request for non-existent customer")
                    .Given("Customer 999 does not exist")
                    .WithRequest(HttpMethod.Get, "/api/customers/999")
                    .WithHeader("Accept", "application/json")
                .WillRespond()
                    .WithStatus(HttpStatusCode.NotFound);

            await _pactBuilder.VerifyAsync(async ctx =>
            {
                // Act
                var httpClient = new HttpClient
                {
                    BaseAddress = ctx.MockServerUri
                };

                var customerClient = new CustomerServiceClient(httpClient);
                var customer = await customerClient.GetCustomerAsync("999");

                // Assert
                Assert.Null(customer);
            });
        }

        [Fact]
        public async Task CreateCustomer_ReturnsCreatedCustomer()
        {
            // Arrange
            _pactBuilder
                .UponReceiving("A request to create a customer")
                    .Given("No customer with email john@example.com exists")
                    .WithRequest(HttpMethod.Post, "/api/customers")
                    .WithHeader("Content-Type", "application/json")
                    .WithJsonBody(new
                    {
                        name = "John Doe",
                        email = "john@example.com"
                    })
                .WillRespond()
                    .WithStatus(HttpStatusCode.Created)
                    .WithHeader("Content-Type", "application/json")
                    .WithJsonBody(new
                    {
                        id = Match.Type("new-id"),
                        name = Match.Type("John Doe"),
                        email = Match.Type("john@example.com"),
                        status = Match.Type("Active")
                    });

            await _pactBuilder.VerifyAsync(async ctx =>
            {
                // Act
                var httpClient = new HttpClient
                {
                    BaseAddress = ctx.MockServerUri
                };

                var customerClient = new CustomerServiceClient(httpClient);
                var customer = await customerClient.CreateCustomerAsync(
                    new CreateCustomerRequest
                    {
                        Name = "John Doe",
                        Email = "john@example.com"
                    });

                // Assert
                Assert.NotNull(customer);
                Assert.NotNull(customer.Id);
                Assert.Equal("John Doe", customer.Name);
            });
        }

        public void Dispose()
        {
            _pactBuilder.Dispose();
        }
    }
}

Provider Test (Customer Service)

The provider verifies it meets the contract defined by consumers:

using Microsoft.AspNetCore.Hosting;
using Microsoft.AspNetCore.TestHost;
using PactNet;
using PactNet.Output.Xunit;
using Xunit;
using Xunit.Abstractions;

namespace CustomerService.Tests.Pact
{
    public class CustomerServiceProviderTests
    {
        private readonly ITestOutputHelper _output;
        private readonly string _pactFilePath;

        public CustomerServiceProviderTests(ITestOutputHelper output)
        {
            _output = output;
            _pactFilePath = Path.Combine("..", "..", "..", "..", 
                "OrderService.Tests", "pacts", 
                "orderservice-customerservice.json");
        }

        [Fact]
        public void EnsureCustomerServiceHonorsContract()
        {
            // Arrange - Start the provider API
            var webHost = new WebHostBuilder()
                .UseStartup<Startup>()
                .UseUrls("http://localhost:5000")
                .Build();

            webHost.Start();

            try
            {
                // Act & Assert - Verify the provider against the contract
                var config = new PactVerifierConfig
                {
                    Outputters = new[] { new XunitOutput(_output) },
                    LogLevel = PactLogLevel.Debug
                };

                new PactVerifier(config)
                    .ServiceProvider("CustomerService", new Uri("http://localhost:5000"))
                    .WithFileSource(new FileInfo(_pactFilePath))
                    .WithProviderStateUrl(new Uri("http://localhost:5000/provider-states"))
                    .Verify();
            }
            finally
            {
                webHost.StopAsync().Wait();
            }
        }
    }

    // Provider state controller for test data setup
    [ApiController]
    [Route("provider-states")]
    public class ProviderStatesController : ControllerBase
    {
        private readonly ICustomerRepository _repository;

        public ProviderStatesController(ICustomerRepository repository)
        {
            _repository = repository;
        }

        [HttpPost]
        public async Task<IActionResult> SetProviderState([FromBody] ProviderState state)
        {
            switch (state.State)
            {
                case "Customer 123 exists":
                    await _repository.SeedTestCustomer(new Customer
                    {
                        Id = "123",
                        Name = "John Doe",
                        Email = "john@example.com",
                        Status = "Active"
                    });
                    break;

                case "Customer 999 does not exist":
                    await _repository.DeleteCustomerIfExists("999");
                    break;

                case "No customer with email john@example.com exists":
                    await _repository.DeleteCustomerByEmail("john@example.com");
                    break;
            }

            return Ok();
        }
    }

    public class ProviderState
    {
        public string State { get; set; }
        public Dictionary<string, object> Params { get; set; }
    }
}

CI/CD Integration with Pact Broker

Use Pact Broker to share contracts between teams:

# Azure DevOps Pipeline
trigger:
  - main

pool:
  vmImage: 'ubuntu-latest'

steps:
- task: DotNetCoreCLI@2
  displayName: 'Run Consumer Tests'
  inputs:
    command: 'test'
    projects: '**/OrderService.Tests.csproj'

- task: Bash@3
  displayName: 'Publish Pacts to Broker'
  inputs:
    targetType: 'inline'
    script: |
      dotnet tool install --global pact-net-cli
      pact-broker publish \
        --consumer-app-version $(Build.BuildNumber) \
        --broker-base-url $(PactBrokerUrl) \
        --broker-token $(PactBrokerToken) \
        ./pacts

- task: DotNetCoreCLI@2
  displayName: 'Run Provider Tests'
  inputs:
    command: 'test'
    projects: '**/CustomerService.Tests.csproj'

Alternatives to Pact

While Pact is popular, other options exist:

OpenAPI/Swagger Schema Validation:

// Validate responses against OpenAPI schema
public class OpenApiValidationTests
{
    [Fact]
    public async Task GetCustomer_ResponseMatchesSchema()
    {
        var client = _factory.CreateClient();
        var response = await client.GetAsync("/api/customers/123");

        var schema = await File.ReadAllTextAsync("openapi.json");
        var validator = new OpenApiValidator(schema);

        await validator.ValidateResponseAsync(
            "/api/customers/{id}",
            "get",
            response);
    }
}

Postman Contract Testing:

Define contracts in Postman collections
Run with Newman in CI/CD
Good for teams already using Postman

Specmatic:

Specification-first approach
Generates tests from OpenAPI specs
Supports multiple languages

Additive Changes and Backward Compatibility

The golden rule: Always add, never remove or change (Postel's Law: "Be conservative in what you send, be liberal in what you accept").

Safe Additive Changes

// V1 - Original
public class OrderDto
{
    public Guid Id { get; set; }
    public decimal Total { get; set; }
    public string Status { get; set; }
}

// V1.1 - Adding optional fields (backward compatible)
public class OrderDto
{
    public Guid Id { get; set; }
    public decimal Total { get; set; }
    public string Status { get; set; }

    // New optional fields with defaults
    public string Currency { get; set; } = "USD";
    public DateTime? CreatedAt { get; set; }
    public List<string> Tags { get; set; } = new();
}

// Configure JSON serialization to ignore nulls
builder.Services.AddControllers()
    .AddJsonOptions(options =>
    {
        options.JsonSerializerOptions.DefaultIgnoreCondition = 
            JsonIgnoreCondition.WhenWritingNull;
        options.JsonSerializerOptions.Converters.Add(
            new JsonStringEnumConverter());
    });

Deprecation Strategy

Mark deprecated fields and provide migration paths:

public class OrderDto
{
    public Guid Id { get; set; }

    // Old field - deprecated
    [Obsolete("Use TotalAmount instead. Will be removed in v3.0 (2026-01-01)")]
    [JsonPropertyName("total")]
    public decimal Total
    {
        get => TotalAmount;
        set => TotalAmount = value;
    }

    // New field - recommended
    [JsonPropertyName("totalAmount")]
    public decimal TotalAmount { get; set; }

    public string Status { get; set; }
}

Deprecation Headers

Communicate deprecation to API consumers:

[ApiController]
[Route("api/[controller]")]
public class OrdersController : ControllerBase
{
    [HttpGet("{id}")]
    [ApiVersion("1.0", Deprecated = true)]
    [MapToApiVersion("1.0")]
    public async Task<ActionResult<OrderV1Dto>> GetV1(Guid id)
    {
        // Add deprecation headers
        Response.Headers.Append("X-API-Deprecated", "true");
        Response.Headers.Append("X-API-Sunset", "2025-12-31");
        Response.Headers.Append("X-API-Deprecation-Info", 
            "https://api.company.com/docs/migration/v1-to-v2");
        Response.Headers.Append("X-API-Deprecation-Date", "2025-06-01");
        Response.Headers.Append("Link", 
            "</api/v2/orders/{id}>; rel=\"successor-version\"");

        var order = await _orderService.GetOrderByIdAsync(id);

        if (order == null)
            return NotFound();

        return Ok(MapToV1Dto(order));
    }

    [HttpGet("{id}")]
    [ApiVersion("2.0")]
    [MapToApiVersion("2.0")]
    public async Task<ActionResult<OrderV2Dto>> GetV2(Guid id)
    {
        var order = await _orderService.GetOrderByIdAsync(id);

        if (order == null)
            return NotFound();

        return Ok(MapToV2Dto(order));
    }
}

Deprecation Policy

Establish a clear deprecation policy:

# API Deprecation Policy

1. **Announcement**: Deprecation announced at least 6 months before removal
2. **Documentation**: Migration guides published with deprecation notice
3. **Headers**: All deprecated endpoints return deprecation headers
4. **Monitoring**: Track usage of deprecated endpoints
5. **Communication**: Email notifications to registered API consumers
6. **Support Period**: Deprecated versions supported for 6 months minimum
7. **Removal**: Version removed only after support period expires

## Deprecation Timeline Example:
- 2025-01-15: Announce v1 deprecation
- 2025-01-15 to 2025-07-15: Both v1 and v2 supported
- 2025-06-01: Final reminder emails sent
- 2025-07-15: v1 removed, only v2 supported

Monitoring API Versions

Track version usage to inform deprecation decisions:

public class ApiVersionMetricsMiddleware
{
    private readonly RequestDelegate _next;
    private readonly ILogger<ApiVersionMetricsMiddleware> _logger;
    private readonly TelemetryClient _telemetry;

    public ApiVersionMetricsMiddleware(
        RequestDelegate next,
        ILogger<ApiVersionMetricsMiddleware> logger,
        TelemetryClient telemetry)
    {
        _next = next;
        _logger = logger;
        _telemetry = telemetry;
    }

    public async Task InvokeAsync(HttpContext context)
    {
        var apiVersion = context.GetRequestedApiVersion()?.ToString() ?? "unknown";
        var endpoint = context.Request.Path.Value;

        // Track metrics
        _telemetry.TrackMetric("ApiVersionUsage", 1, new Dictionary<string, string>
        {
            ["Version"] = apiVersion,
            ["Endpoint"] = endpoint,
            ["Method"] = context.Request.Method
        });

        // Add version to response headers for debugging
        context.Response.OnStarting(() =>
        {
            context.Response.Headers.Append("X-API-Version", apiVersion);
            return Task.CompletedTask;
        });

        await _next(context);
    }
}

// Register middleware
app.UseMiddleware<ApiVersionMetricsMiddleware>();

Azure API Management Integration

For enterprise scenarios, use Azure API Management for advanced versioning:

// Configure policies in Azure API Management

// Version routing policy
<choose>
    <when condition="@(context.Request.Headers.GetValueOrDefault("X-API-Version","1.0") == "2.0")">
        <set-backend-service base-url="https://api-v2.company.com" />
    </when>
    <otherwise>
        <set-backend-service base-url="https://api-v1.company.com" />
    </otherwise>
</choose>

// Deprecation warning policy
<choose>
    <when condition="@(context.Request.Headers.GetValueOrDefault("X-API-Version","1.0") == "1.0")">
        <set-header name="X-API-Deprecated" exists-action="override">
            <value>true</value>
        </set-header>
        <set-header name="X-API-Sunset" exists-action="override">
            <value>2025-12-31</value>
        </set-header>
    </when>
</choose>

Best Practices Summary

Versioning Strategy:

Use URL versioning for clarity and simplicity
Version from day one, even v1.0
Never reuse version numbers
Document all versions in OpenAPI/Swagger
Use semantic versioning (Major.Minor.Patch)

Contract Testing:

Implement consumer-driven contracts with Pact
Run contract tests in CI/CD pipeline
Use Pact Broker for contract sharing
Verify providers against all consumer contracts
Keep contracts in version control

Backward Compatibility:

Follow Postel's Law: add, never remove
Provide default values for new fields
Use optional parameters
Maintain old field names during transition
Version DTOs separately from domain models

Deprecation:

Establish clear deprecation timeline (6+ months)
Communicate through multiple channels
Use deprecation headers consistently
Monitor deprecated endpoint usage
Provide comprehensive migration guides
Never surprise consumers with breaking changes

Azure-Specific:

Use Azure API Management for enterprise versioning
Implement version routing policies
Enable analytics for version tracking
Use Azure Monitor for deprecation metrics
Leverage Azure DevOps for contract testing in CI/CD

Documentation:

Document all versions in Swagger UI
Provide migration guides for each version
Include examples for all versions
Maintain changelog with breaking changes
Link to deprecation information in API responses

Azure App Configuration vs. Azure Key Vault

Hossein Esmati — Fri, 26 Jun 2026 00:13:04 +0000

Rule of thumb:

Secrets → Key Vault
Configs → App Configuration
Use both for secure + flexible configuration management.

Azure Key Vault

Purpose: Securely store sensitive information such as secrets, keys, and certificates.

Use Cases:
- Store API keys, connection strings, tokens.
- Manage and rotate TLS/SSL certificates.
- Protect cryptographic keys used for encryption/decryption.
Strengths:
- Built-in hardware security module (HSM) support.
- Access policies and RBAC for fine-grained control.
- Automatic secret rotation with some Azure services.
- Logging and monitoring via Azure Monitor.
Limitations:
- Not designed for feature flags or configuration settings that change frequently.
- API calls can add latency if used excessively at runtime.

Azure App Configuration

Purpose: Centralized application configuration management.

Use Cases:
- Store non-sensitive app settings (feature flags, UI options, app behavior).
- Versioned configurations and labels (per environment, per region).
- Enable dynamic configuration refresh in apps.
Strengths:
- Feature flag management built-in.
- Supports key-value pairs with labels for environment separation.
- Integration with Azure Functions, App Service, AKS, and more.
- High availability and global distribution.
Limitations:
- Not designed to store secrets or keys.
- Does not provide encryption key lifecycle management.

When to Use Which

Use Key Vault when:
- Handling secrets (DB passwords, API keys).
- Managing certificates and encryption keys.
- Need secure storage with strong access policies.
Use App Configuration when:
- Handling app configs (feature flags, toggle dark mode, regional endpoints).
- Need dynamic refresh without redeployment.
- Want environment-based configuration with versioning.

How They Work Together

In most real-world solutions, you combine both:

Use Azure App Configuration for general settings and feature management.
Reference Azure Key Vault inside App Configuration for sensitive values.
Example:
- AppConfig:DbConnectionString → points to Key Vault secret.
- App reads all configs from App Configuration → securely resolves secrets from Key Vault when needed.

API Gateway Patterns in .NET Core and Azure

Hossein Esmati — Fri, 26 Jun 2026 00:13:03 +0000

This article is part of the Comprehensive Guide to Microservices Architecture in .NET Core, Cloud and Azure series.

API gateways serve as the entry point for client applications in distributed architectures, handling request routing, composition, and protocol translation. As systems grow more complex with multiple client types and backend services, choosing the right gateway pattern becomes crucial for maintaining performance, scalability, and developer productivity.

This article explores two powerful API gateway patterns in .NET: the Backend for Frontend (BFF) pattern, which creates client-specific gateways, and GraphQL, which offers flexible, query-driven data fetching. Both patterns address the challenge of efficiently serving diverse clients while maintaining clean architecture and optimal performance.

Backend for Frontend (BFF) Pattern

Web BFF Implementation

The web BFF returns detailed, enriched data suitable for desktop browsers with higher bandwidth and processing power:

[ApiController]
[Route("api/web/[controller]")]
public class OrdersController : ControllerBase
{
    private readonly IOrderServiceClient _orderClient;
    private readonly ICustomerServiceClient _customerClient;
    private readonly IInventoryServiceClient _inventoryClient;

    [HttpGet("{id}")]
    public async Task<WebOrderDto> GetOrder(Guid id)
    {
        // Execute parallel calls to improve response time
        var orderTask = _orderClient.GetOrderAsync(id);
        var customerTask = _customerClient.GetCustomerAsync(id);
        var inventoryTask = _inventoryClient.GetInventoryStatusAsync(id);

        await Task.WhenAll(orderTask, customerTask, inventoryTask);

        return new WebOrderDto
        {
            Order = orderTask.Result,
            Customer = customerTask.Result,
            InventoryStatus = inventoryTask.Result,
            // Include additional rich data for enhanced web UI experience
            RecommendedProducts = await GetRecommendationsAsync(id)
        };
    }
}

Mobile BFF Implementation

The mobile BFF provides optimized, lightweight responses to minimize bandwidth consumption and improve performance on mobile networks:

[ApiController]
[Route("api/mobile/[controller]")]
public class OrdersController : ControllerBase
{
    private readonly IOrderServiceClient _orderClient;

    [HttpGet("{id}")]
    public async Task<MobileOrderDto> GetOrder(Guid id)
    {
        var order = await _orderClient.GetOrderAsync(id);

        // Return only essential data to reduce payload size
        return new MobileOrderDto
        {
            Id = order.Id,
            Status = order.Status,
            Total = order.TotalAmount
        };
    }
}

GraphQL as API Gateway

GraphQL provides a flexible alternative to traditional REST-based BFF implementations, allowing clients to request exactly the data they need in a single query.

Setting Up GraphQL with HotChocolate

First, install the required package:

dotnet add package HotChocolate.AspNetCore

Defining Query Types

public class Query
{
    public async Task<Order> GetOrder(
        [ID] Guid id,
        [Service] IOrderRepository repository)
    {
        return await repository.GetByIdAsync(id);
    }

    public async Task<Customer> GetCustomer(
        [ID] Guid id,
        [Service] ICustomerRepository repository)
    {
        return await repository.GetByIdAsync(id);
    }
}

Extending Types for Nested Queries

Type extensions enable nested data fetching, allowing clients to retrieve related entities in a single request:

[ExtendObjectType(typeof(Order))]
public class OrderExtensions
{
    public async Task<Customer> GetCustomer(
        [Parent] Order order,
        [Service] ICustomerServiceClient client)
    {
        return await client.GetCustomerAsync(order.CustomerId);
    }

    public async Task<List<Product>> GetProducts(
        [Parent] Order order,
        [Service] IProductServiceClient client)
    {
        var productIds = order.OrderLines.Select(l => l.ProductId);
        return await client.GetProductsAsync(productIds);
    }
}

Configuring GraphQL Server

Configure the GraphQL server in Program.cs with essential features like data loaders, filtering, and sorting:

builder.Services
    .AddGraphQLServer()
    .AddQueryType<Query>()
    .AddTypeExtension<OrderExtensions>()
    .AddDataLoader<CustomerByIdDataLoader>()
    .AddFiltering()
    .AddSorting()
    .AddProjections();

Implementing DataLoader to Prevent N+1 Queries

DataLoaders batch and cache requests to prevent the common N+1 query problem in GraphQL:

public class CustomerByIdDataLoader : BatchDataLoader<Guid, Customer>
{
    private readonly ICustomerServiceClient _client;

    public CustomerByIdDataLoader(
        ICustomerServiceClient client,
        IBatchScheduler batchScheduler,
        DataLoaderOptions options = null)
        : base(batchScheduler, options)
    {
        _client = client;
    }

    protected override async Task<IReadOnlyDictionary<Guid, Customer>> 
        LoadBatchAsync(
            IReadOnlyList<Guid> keys, 
            CancellationToken cancellationToken)
    {
        var customers = await _client.GetCustomersByIdsAsync(keys);
        return customers.ToDictionary(c => c.Id);
    }
}

Benefits and Considerations

BFF Pattern Advantages:

Client-specific optimization reduces over-fetching and under-fetching
Independent evolution of client and backend APIs
Simplified client-side logic

GraphQL Advantages:

Single endpoint for all data requirements
Strongly typed schema with built-in documentation
Efficient data fetching with precise field selection
Reduced number of API requests

Trade-offs:

BFF requires maintaining multiple gateway implementations
GraphQL introduces complexity in query optimization and security
Both patterns require careful monitoring to prevent performance issues