Implement Circuit Breaker using Polly in .Net Core 8

Overview of the Circuit Breaker Pattern

The Circuit Breaker pattern is used to handle faults in distributed systems, especially in a microservices architecture. It prevents cascading failures by detecting failing services and stopping their interaction for a specified period.

Why and When to Use Circuit Breaker:

1. Fail Fast: Stops unnecessary calls to failing services and saves system resources.
2. Service Stability: Helps maintain overall application responsiveness.
3. Recovery: Provides time for failing services to recover.
4. Example Scenarios:

• External service downtime.
• Database outages.
• High latency in dependent services.

Step By Step Implementation in .NET Core 8

Step 1: Install Necessary NuGet Package
Install the Polly library:

dotnet add package Polly
dotnet add package Microsoft.Extensions.Http.Polly

Step 2: Create a Resilient HTTP Client with Circuit Breaker

Set up resilient HTTP clients for ProductService, CartService, and OrderService.

using Microsoft.Extensions.DependencyInjection;
using Polly;
using Polly.CircuitBreaker;
using Polly.Extensions.Http;
using System;
using System.Net.Http;

var builder = WebApplication.CreateBuilder(args);

// Register HTTP Clients with Circuit Breaker
builder.Services.AddHttpClient("ProductService", client =>
{
    client.BaseAddress = new Uri("https://api.example.com/products");
})
.AddPolicyHandler(GetCircuitBreakerPolicy());

builder.Services.AddHttpClient("CartService", client =>
{
    client.BaseAddress = new Uri("https://api.example.com/cart");
})
.AddPolicyHandler(GetCircuitBreakerPolicy());

builder.Services.AddHttpClient("OrderService", client =>
{
    client.BaseAddress = new Uri("https://api.example.com/orders");
})
.AddPolicyHandler(GetCircuitBreakerPolicy());

var app = builder.Build();
app.Run();

// Circuit Breaker Policy
IAsyncPolicy<HttpResponseMessage> GetCircuitBreakerPolicy()
{
    return HttpPolicyExtensions
        .HandleTransientHttpError()
        .CircuitBreakerAsync(
            handledEventsAllowedBeforeBreaking: 3,
            durationOfBreak: TimeSpan.FromSeconds(30),
            onBreak: (exception, breakDelay) =>
            {
                Console.WriteLine($"Circuit broken: {exception.Message}");
            },
            onReset: () =>
            {
                Console.WriteLine("Circuit reset");
            },
            onHalfOpen: () =>
            {
                Console.WriteLine("Circuit in half-open state");
            });
}

Step 3: Create a Service Layer to Call HTTP Clients

Encapsulate HTTP client logic in services.

public class ProductService
{
    private readonly HttpClient _httpClient;

    public ProductService(IHttpClientFactory httpClientFactory)
    {
        _httpClient = httpClientFactory.CreateClient("ProductService");
    }

    public async Task<string> GetProductsAsync()
    {
        var response = await _httpClient.GetAsync("/list");
        response.EnsureSuccessStatusCode();
        return await response.Content.ReadAsStringAsync();
    }
}

public class CartService
{
    private readonly HttpClient _httpClient;

    public CartService(IHttpClientFactory httpClientFactory)
    {
        _httpClient = httpClientFactory.CreateClient("CartService");
    }

    public async Task<string> GetCartDetailsAsync()
    {
        var response = await _httpClient.GetAsync("/details");
        response.EnsureSuccessStatusCode();
        return await response.Content.ReadAsStringAsync();
    }
}

public class OrderService
{
    private readonly HttpClient _httpClient;

    public OrderService(IHttpClientFactory httpClientFactory)
    {
        _httpClient = httpClientFactory.CreateClient("OrderService");
    }

    public async Task<string> GetOrderDetailsAsync()
    {
        var response = await _httpClient.GetAsync("/details");
        response.EnsureSuccessStatusCode();
        return await response.Content.ReadAsStringAsync();
    }
}

Step 4: Register Services in Dependency Injection
Add the service classes to the dependency injection container.

builder.Services.AddScoped<ProductService>();
builder.Services.AddScoped<CartService>();
builder.Services.AddScoped<OrderService>();

Step 5: Use Services in a Controller
Consume the services in a controller.

using Microsoft.AspNetCore.Mvc;

[ApiController]
[Route("api/[controller]")]
public class ShopController : ControllerBase
{
    private readonly ProductService _productService;
    private readonly CartService _cartService;
    private readonly OrderService _orderService;

    public ShopController(ProductService productService, CartService cartService, OrderService orderService)
    {
        _productService = productService;
        _cartService = cartService;
        _orderService = orderService;
    }

    [HttpGet("products")]
    public async Task<IActionResult> GetProducts()
    {
        var products = await _productService.GetProductsAsync();
        return Ok(products);
    }

    [HttpGet("cart")]
    public async Task<IActionResult> GetCart()
    {
        var cartDetails = await _cartService.GetCartDetailsAsync();
        return Ok(cartDetails);
    }

    [HttpGet("orders")]
    public async Task<IActionResult> GetOrders()
    {
        var orderDetails = await _orderService.GetOrderDetailsAsync();
        return Ok(orderDetails);
    }
}

Best Practices

• 1. Isolate Circuit Breakers: Use separate circuit breakers for each service.

To avoid cascading failures and ensure independent service handling, use separate circuit breakers for each service.

• Use named HTTP clients with unique circuit breaker policies for each service.
• Avoid sharing circuit breaker policies across multiple services.

builder.Services.AddHttpClient("ProductService", client =>
{
    client.BaseAddress = new Uri("https://api.example.com/products");
})
.AddPolicyHandler(GetProductServiceCircuitBreakerPolicy());

builder.Services.AddHttpClient("CartService", client =>
{
    client.BaseAddress = new Uri("https://api.example.com/cart");
})
.AddPolicyHandler(GetCartServiceCircuitBreakerPolicy());

builder.Services.AddHttpClient("OrderService", client =>
{
    client.BaseAddress = new Uri("https://api.example.com/orders");
})
.AddPolicyHandler(GetOrderServiceCircuitBreakerPolicy());

IAsyncPolicy<HttpResponseMessage> GetProductServiceCircuitBreakerPolicy() =>
    HttpPolicyExtensions
        .HandleTransientHttpError()
        .CircuitBreakerAsync(3, TimeSpan.FromSeconds(30));

IAsyncPolicy<HttpResponseMessage> GetCartServiceCircuitBreakerPolicy() =>
    HttpPolicyExtensions
        .HandleTransientHttpError()
        .CircuitBreakerAsync(2, TimeSpan.FromSeconds(20));

IAsyncPolicy<HttpResponseMessage> GetOrderServiceCircuitBreakerPolicy() =>
    HttpPolicyExtensions
        .HandleTransientHttpError()
        .CircuitBreakerAsync(5, TimeSpan.FromSeconds(60));

• 2. Monitor and Log: Continuously monitor circuit breaker events for debugging and alerting. Monitor events such as circuit breaking, resetting, and transitioning to a half-open state. This helps in debugging and tracking the health of services.

• Use onBreak, onReset, and onHalfOpen callbacks for logging.

• Leverage monitoring tools like Application Insights or ELK Stack for observability.
  

IAsyncPolicy<HttpResponseMessage> GetCircuitBreakerPolicyWithLogging()
{
    return HttpPolicyExtensions
        .HandleTransientHttpError()
        .CircuitBreakerAsync(
            handledEventsAllowedBeforeBreaking: 3,
            durationOfBreak: TimeSpan.FromSeconds(30),
            onBreak: (exception, timespan) =>
            {
                Console.WriteLine($"Circuit broken due to: {exception.Message}. Duration: {timespan}");
                // Log to Application Insights or other tools
            },
            onReset: () =>
            {
                Console.WriteLine("Circuit reset to closed state.");
            },
            onHalfOpen: () =>
            {
                Console.WriteLine("Circuit in half-open state. Testing health...");
            });
}

• 3. Graceful Fallback: Provide meaningful fallback responses when a service is unavailable. This enhances user experience and ensures the application remains functional.

• Use Polly's Fallback policy in conjunction with Circuit Breaker.

• Return cached or default data.
  

IAsyncPolicy<HttpResponseMessage> GetCircuitBreakerWithFallback()
{
    return Policy.WrapAsync(
        Policy<HttpResponseMessage>
            .Handle<Exception>()
            .FallbackAsync(new HttpResponseMessage(System.Net.HttpStatusCode.OK)
            {
                Content = new StringContent("Service temporarily unavailable. Using fallback.")
            }),
        HttpPolicyExtensions
            .HandleTransientHttpError()
            .CircuitBreakerAsync(3, TimeSpan.FromSeconds(30))
    );
}

4. Graceful Degradation and Feature Toggles

When critical services fail, gracefully degrade functionality using feature toggles. Tools like LaunchDarkly or FeatureManagement in Azure App Configuration can help.

5. Use Centralized Configuration

Store circuit breaker configurations in a centralized location for easier updates and management. Tools like Azure App Configuration or Consul are ideal for this purpose.

public class CircuitBreakerSettings
{
    public int HandledEventsAllowedBeforeBreaking { get; set; }
    public int DurationOfBreakInSeconds { get; set; }
}

builder.Services.Configure<CircuitBreakerSettings>(
    builder.Configuration.GetSection("CircuitBreaker"));

IAsyncPolicy<HttpResponseMessage> GetCircuitBreakerPolicyWithConfig(IOptionsMonitor<CircuitBreakerSettings> options)
{
    var settings = options.CurrentValue;
    return HttpPolicyExtensions
        .HandleTransientHttpError()
        .CircuitBreakerAsync(
            settings.HandledEventsAllowedBeforeBreaking,
            TimeSpan.FromSeconds(settings.DurationOfBreakInSeconds));
}

Tools for Monitoring and Testing

Application Insights:
For logging Circuit Breaker events and tracking service health.
Azure Application Insights

ELK Stack:
ElasticSearch, Logstash, and Kibana for centralized logging.

Gremlin:

• A chaos engineering tool for simulating service failures.
• Gremlin

Postman/Newman:

• For manual or automated testing of APIs under failure conditions.
• Postman

Polly Dashboard:

• Community-based dashboard for monitoring Polly policies.
• Polly Dashboard GitHub

Online Resources

Polly Documentation: Polly Official Site
Circuit Breaker Design Pattern: Microsoft Learn
Microservices Best Practices: Microservices on .NET