DEV Community

Ghassan Karwchan
Ghassan Karwchan

Posted on

1

build resilient applications accessing Azure services with no code.

When working on distributed system like systems that are running on the cloud, and microservices, then you should anticipate transient faults like temporary loss of network connection, or temporary unavailability of a service, or a service is busy.

This is why in cloud application, the term Building for resiliency become a pattern in building application, and it means properly handling these transient faults in the code.

There are two common used patterns that we can use in our code, that can help build resilience application.

  1. Retry pattern.
  2. Circuit-breaker Pattern.

The good news is there are libraries that will help you doing that, and even the SDKs for some Azure services have these two patterns built-in, and by just configure the connection properly you will get this benefits without writing your own code for those.

Retry Pattern

The retry pattern means if calling a service failed, you wait for a short time, and then retry again because you expect the problem will be resolved shortly. You try not only one time, but maybe for 3 times, and leave a short waiting time between the retries.

Services that support retry.

  1. Azure SQL Server, and Entity Framework: Entity Framework has a built-in retry technique when it comes to communicate with Azure SQL Server.
services.AddDbContextPool<ConcertDataContext>(options => options.UseSqlServer(sqlDatabaseConnectionString,
    sqlServerOptionsAction: sqlOptions =>
    {
        sqlOptions.EnableRetryOnFailure(
          maxRetryCount: 5,
          maxRetryDelay: TimeSpan.FromSeconds(3),
          errorNumbersToAdd: null);
    }));
Enter fullscreen mode Exit fullscreen mode
  1. Azure Cache for Redis If you are using the most famous Redis .NET library which is StackExchange.Redis, then the library has the mechanism for retry as follows:

var options = new ConfigurationOptions
{
    EndPoints = {"localhost"},
    ConnectRetry = 3,
    ReconnectRetryPolicy = new ExponentialRetry(
TimeSpan.FromSeconds(5).TotalMilliseconds
, TimeSpan.FromSeconds(20).TotalMilliseconds),
    ConnectTimeout = 2000
};
ConnectionMultiplexer redis = ConnectionMultiplexer.Connect(options, writer); 
Enter fullscreen mode Exit fullscreen mode
  1. Azure Service Bus The .NET package for the service bus Azure.Messaging.ServiceBus has built-in support for retry using the configuration of its client.

var options = new ServiceBusClientOptions();
options.RetryOptions = new ServiceBusRetryOptions
{
    Delay = TimeSpan.FromSeconds(10),
    MaxDelay = TimeSpan.FromSeconds(30),
    Mode = ServiceBusRetryMode.Exponential,
    MaxRetries = 3,
};
await using var client = new ServiceBusClient(connectionString, options);
Enter fullscreen mode Exit fullscreen mode

More Azure service support retry

I am not going to list the code for all services, but just check the SDK for each service, and you will find how to enable the retry features. But these services will support the feature:

  1. Azure Cosmos DB
  2. Azure Active Directory
  3. Azure Search
  4. Event Hub
  5. IoT Hub
  6. Azure Storage

Retry using Polly

If you need to connect to other API services, or you need to connect to Azure SQL but without Entity Framework, then you need to implement the code yourself.

But there is a .NET package that is already doing that: Polly.

Let us see for example how to connect to SQL using ADO.NET and Polly:

public async static Task<SqlDataReader> ExecuteReaderWithRetryAsync(this SqlCommand command)
{
    GuardConnectionIsNotNull(command);

    var policy = Policy.Handle<Exception>().WaitAndRetryAsync(
        retryCount: 3, // Retry 3 times
        sleepDurationProvider: attempt => TimeSpan.FromMilliseconds(200 * Math.Pow(2, attempt - 1)), // Exponential backoff based on an initial 200 ms delay.
        onRetry: (exception, attempt) =>
        {
            // Capture some information for logging/telemetry.
            logger.LogWarn($"ExecuteReaderWithRetryAsync: Retry {attempt} due to {exception}.");
        });

    // Retry the following call according to the policy.
    await policy.ExecuteAsync<SqlDataReader>(async token =>
    {
        // This code is executed within the Policy

        if (conn.State != System.Data.ConnectionState.Open) await conn.OpenAsync(token);
        return await command.ExecuteReaderAsync(System.Data.CommandBehavior.Default, token);

    }, cancellationToken);
}
Enter fullscreen mode Exit fullscreen mode

Circuit-breaker Pattern

There is a problem with the retry pattern. What if a service was down and stayed down for long time?

Then every time we call this service, we are going to call it three times and wait for a second between these calls.

This will add delay for our application.

The Circuit-breaker pattern will fix this, by assuming the service is down after many failed calls.

If the service marked as down, then code will immediately fail the call without even calling the service.

This state is called Open Circuit.

Then the code will wait for a while before it try again to call the service, and if it is still failing, then the state will still be Open, otherwise it is marked as healthy, and the state will be Closed Circuit.

Circuit breaker should be used in conjunction with the retry pattern.

Using Cicruit-breaker with Polly.

There is built-in implementation of circuit-breaker in SDKs of Azure services, but we can achieve it using Polly package, as follows:

private static IAsyncPolicy<HttpResponseMessage> GetCircuitBreakerPolicy()
{
    return HttpPolicyExtensions
        .HandleTransientHttpError()
        .CircuitBreakerAsync(5, TimeSpan.FromSeconds(30));
}
Enter fullscreen mode Exit fullscreen mode

Image of Timescale

🚀 pgai Vectorizer: SQLAlchemy and LiteLLM Make Vector Search Simple

We built pgai Vectorizer to simplify embedding management for AI applications—without needing a separate database or complex infrastructure. Since launch, developers have created over 3,000 vectorizers on Timescale Cloud, with many more self-hosted.

Read more

Top comments (0)

Image of Docusign

🛠️ Bring your solution into Docusign. Reach over 1.6M customers.

Docusign is now extensible. Overcome challenges with disconnected products and inaccessible data by bringing your solutions into Docusign and publishing to 1.6M customers in the App Center.

Learn more

đź‘‹ Kindness is contagious

Discover a treasure trove of wisdom within this insightful piece, highly respected in the nurturing DEV Community enviroment. Developers, whether novice or expert, are encouraged to participate and add to our shared knowledge basin.

A simple "thank you" can illuminate someone's day. Express your appreciation in the comments section!

On DEV, sharing ideas smoothens our journey and strengthens our community ties. Learn something useful? Offering a quick thanks to the author is deeply appreciated.

Okay