DEV Community

Oleksandr Viktor
Oleksandr Viktor

Posted on Edited on

WJb Resilient Actions: Designing Fault-Tolerant Background Jobs

#csharp #dotnet #programming #backend

Resilient Actions

One night, Cron was generating jobs as expected. The queue was accepting them as expected. The Worker was running as expected.

Then a single Action got stuck.

As a result:

  • Cron continued generating new jobs;
  • the queue continued growing;
  • other Actions stopped executing;
  • the issue was resolved only after a restart.

This is a classic example of how system reliability is determined not by how it behaves in a perfect world, but by how it handles failures.

An Action Must Be Designed for Failure

Every Action operates in a hostile environment:

  • the network may stop responding;
  • the database may become slow;
  • an external API may hang;
  • the server may restart;
  • the action may be cancelled by the user.

For this reason, an Action should be designed as if failure is inevitable.

Always Use CancellationToken

Bad:

await http.GetStringAsync(url);
Enter fullscreen mode Exit fullscreen mode

Good:

await http.GetStringAsync(url, ct);
Enter fullscreen mode Exit fullscreen mode

Bad:

await Task.Delay(TimeSpan.FromMinutes(5));
Enter fullscreen mode Exit fullscreen mode

Good:

await Task.Delay(TimeSpan.FromMinutes(5), ct);
Enter fullscreen mode Exit fullscreen mode

If an operation does not support cancellation, the Worker cannot stop the Action gracefully.

Always Define Timeouts

Even when a CancellationToken is used.

Bad:

var http = new HttpClient();
Enter fullscreen mode Exit fullscreen mode

Good:

var http = new HttpClient
{
    Timeout = TimeSpan.FromSeconds(30)
};
Enter fullscreen mode Exit fullscreen mode

External resources are not required to respond quickly.

Sometimes they do not respond at all.

Distinguish Timeouts from Cancellations

A timeout and a cancellation are not the same thing.

A timeout usually means:

the operation took too long and may be safe to retry.

A cancellation usually means:

execution was intentionally stopped and should not be retried.

In .NET both situations may appear as a TaskCanceledException.

Example:

try
{
    await http.GetStringAsync(url, ct);
}
catch (TaskCanceledException ex)
    when (ex.InnerException is TimeoutException)
{
    logger.LogWarning("Request timed out");

    throw;
}
catch (OperationCanceledException)
{
    logger.LogInformation("Action was cancelled");

    throw;
}
Enter fullscreen mode Exit fullscreen mode

Without this distinction, logs may show both situations as simple cancellations even though the retry decision is often the opposite.

A timeout is typically a transient failure.

A cancellation is typically an intentional stop request.

Log Execution Stages

Often the most useful log messages are the simplest ones:

ReportProgress(0, "Loading data");
Enter fullscreen mode Exit fullscreen mode 
ReportProgress(0, "Downloaded data");
Enter fullscreen mode Exit fullscreen mode 
ReportProgress(0, "Saving data");
Enter fullscreen mode Exit fullscreen mode 
ReportProgress(0, "Completed");
Enter fullscreen mode Exit fullscreen mode

If an Action gets stuck, the last message immediately identifies where execution stopped.

For example:

Loading data
Enter fullscreen mode Exit fullscreen mode

It becomes obvious that the problem occurred before the database save phase.

Make Actions Idempotent

After a failure, an Action may execute again.

If re-execution creates duplicate data, the system is not resilient.

Bad:

INSERT INTO Tokens (...)
Enter fullscreen mode Exit fullscreen mode

Better:

INSERT ...
WHERE NOT EXISTS (...)
Enter fullscreen mode Exit fullscreen mode

or

MERGE
Enter fullscreen mode Exit fullscreen mode

or

UNIQUE INDEX
Enter fullscreen mode Exit fullscreen mode

A repeated execution should be safe.

Do Not Store State in Memory

Bad:

private static int Processed;
Enter fullscreen mode Exit fullscreen mode

The value will be lost after a restart.

Instead, store state in:

  • a database;
  • a queue;
  • a persistent state store.

An Action should survive process restarts.

Break Large Operations into Smaller Actions

Bad:

DownloadAllHistory();
ProcessAllHistory();
SaveAllHistory();
Enter fullscreen mode Exit fullscreen mode

Good:

download
process
aggregate
publish
Enter fullscreen mode Exit fullscreen mode

Smaller Actions are easier to:

  • cancel;
  • retry;
  • troubleshoot.

Verify Behavior After a Timeout

It is not enough to verify that an Action was cancelled.

You must prove that the queue continues to operate.

A good integration test is:

  1. The first job becomes stuck.
  2. A timeout occurs.
  3. The second job executes successfully.
  4. The queue continues processing.

This scenario closely matches real production behavior.

A Practical Rule

Whenever an Action contains an:

await
Enter fullscreen mode Exit fullscreen mode

ask yourself:

What happens if this operation hangs forever?

If the answer is:

Nothing good.

Then the Action is not production-ready yet.

Summary

A simple, resilient Action has six characteristics:

  • supports CancellationToken;
  • defines reasonable timeouts;
  • distinguishes timeouts from cancellations;
  • logs key execution stages;
  • is idempotent;
  • safely handles re-execution.

In an ideal world, failures never happen.

In production, they happen every day.

A good Action is not one that never fails.

A good Action is one after which the system continues to operate. 🚀🐈‍⬛

Top comments (2)

Collapse
 
mehdimohseni82 profile image
Mehdi Mohseni •

Nice list, and the timeout plus CancellationToken pair has one detail that helps with the logging point. Since .NET 5, when HttpClient.Timeout fires you get a TaskCanceledException whose InnerException is a TimeoutException, so a catch can check ex.InnerException is TimeoutException and log "timed out, safe to retry" separately from "the job was cancelled, stop". Without that check they both look like the same cancellation in the log, and the retry decision is usually the opposite, so it's worth the two extra lines.

Collapse
 
ukrguru profile image
Oleksandr Viktor •

Great catch. Timeout and cancellation may surface as the same exception type, but they require different retry behavior. Thanks for pointing that out. 👍