Retry Budgets in Go: Preventing Retries from Becoming an Outage

Retries are one of the most common ways to make distributed systems more resilient.

A request fails because of a temporary network issue. The client waits briefly and tries again. The second attempt succeeds, and the failure never reaches the user.

That sounds reasonable.

But what happens when the dependency is already overloaded?

Requests start failing. Every caller retries. Those retries create additional traffic, which causes more failures, which trigger even more retries.

A temporary slowdown can turn into a retry storm.

Retries are not free. Every attempt consumes resources, occupies connections, increases queue pressure, and adds work to an already struggling dependency.

This is where retry budgets become useful.

A retry budget places a limit on how much additional traffic a service can generate through retries. Instead of allowing every failed request to retry independently, it establishes a shared budget across requests.

In this article, we'll explore:

• Why per-request retry limits are not enough
• How retry amplification creates cascading failures
• The difference between retry limits and retry budgets
• How to implement a concurrency-safe retry budget in Go
• How to combine budgets with exponential backoff and jitter
• How retries interact with timeouts, bulkheads, and circuit breakers
• What to measure and test before using retries in production

1. The Problem: Retries Can Amplify Failures

Imagine a service that sends requests to an external market data provider.

Under normal conditions:

• The service receives 1,000 requests per second.
• Each request makes one provider call.
• The provider responds successfully.

Now the provider starts experiencing problems. Its error rate increases, and requests become slower.

The service retries failed requests up to three times.

If every original request fails, the provider may receive:

Attempt	Requests
Original requests	1,000
First retries	1,000
Second retries	1,000
Third retries	1,000
Total	4,000

The service has turned 1,000 incoming requests into as many as 4,000 downstream attempts.

This is retry amplification.

The actual situation can be worse. Retries may overlap with new incoming traffic, other services may retry independently, and clients may have their own retry policies.

The dependency is already struggling, but the system keeps sending it more work.

The important distinction is that retries can improve the success rate of individual requests while making the overall system less stable.

A retry policy should not only ask:

How many times should this request retry?

It should also ask:

How much additional traffic can this service afford to generate?

2. Why Max Retries Is Not Enough

A typical retry implementation has a maximum attempt count:

for attempt := 1; attempt <= 3; attempt++ {
    err := callDependency()
    if err == nil {
        return nil
    }
}

This limits retries for an individual request.

But it does not limit retries across the entire service.

Consider a service receiving 10,000 requests per second. Even with only one retry per request, it can generate another 10,000 downstream attempts per second during an outage.

A per-request retry limit provides a local constraint, not a global traffic constraint.

There are two separate controls:

Control	What it limits
Maximum attempts	Retries for one request
Retry budget	Retries across a service or dependency

You generally want both.

A maximum attempt count prevents one request from retrying indefinitely. A shared retry budget prevents thousands of requests from independently exhausting a dependency.

3. What Is a Retry Budget?

A retry budget is a shared allowance for additional attempts.

Instead of allowing every failed request to retry automatically, the service must acquire permission from the budget before performing another attempt.

One practical approach is a token bucket.

The model is simple:

1. Successful initial requests add tokens to the budget.
2. Each retry consumes one token.
3. The budget has a maximum capacity.
4. When no tokens are available, additional retries are rejected.

For example, suppose a service uses these settings:

• Maximum budget: 20 tokens
• Credit per successful initial request: 0.1 tokens
• Cost per retry: 1 token

Every ten successful initial requests generate one retry token.

If the service experiences a sudden failure, it can spend its accumulated tokens on retries. Once the budget is exhausted, it stops generating additional retry traffic until successful initial requests replenish it.

This creates an important feedback mechanism:

• Healthy service: successful requests replenish the budget.
• Unhealthy service: fewer successful requests mean less budget replenishment.
• Outage: retries consume the remaining budget and eventually stop.

The budget is not a guarantee that a dependency can handle a particular amount of traffic. It is a mechanism for limiting retry amplification.

4. Implementing a Retry Budget in Go

Let's start with the shared budget.

We need a structure that is safe to access concurrently because multiple requests may attempt to consume or replenish tokens at the same time.

package retry

import "sync"

type RetryBudget struct {
    mu sync.Mutex

    tokens          float64
    capacity        float64
    creditPerSuccess float64
}

func NewRetryBudget(
    capacity float64,
    creditPerSuccess float64,
) *RetryBudget {
    return &RetryBudget{
        capacity:         capacity,
        creditPerSuccess: creditPerSuccess,
    }
}

func (b *RetryBudget) RecordSuccess() {
    b.mu.Lock()
    defer b.mu.Unlock()

    b.tokens += b.creditPerSuccess

    if b.tokens > b.capacity {
        b.tokens = b.capacity
    }
}

func (b *RetryBudget) TryTake() bool {
    b.mu.Lock()
    defer b.mu.Unlock()

    if b.tokens < 1 {
        return false
    }

    b.tokens--
    return true
}

The implementation deliberately keeps the budget small and predictable.

RecordSuccess adds credits without exceeding the configured capacity.

TryTake atomically checks and consumes a token. This prevents concurrent requests from spending the same token.

For example:

budget := NewRetryBudget(
    20,   // Maximum tokens
    0.1,  // Credit per successful initial request
)

A successful initial request adds 0.1 tokens. A retry requires one complete token.

The budget starts empty. This is intentional: a newly started instance does not immediately receive permission to generate retries. If a service needs an initial retry allowance, that can be configured explicitly.

5. Building a Retry Executor

A retry budget is only one part of a retry policy.

A useful retry executor should also support:

• A maximum number of attempts
• Exponential backoff
• Jitter
• Context cancellation
• Retryable error classification
• A shared retry budget

First, define the policy:

package retry

import (
    "context"
    "errors"
    "math/rand"
    "time"
)

var ErrInvalidPolicy = errors.New("invalid retry policy")

type Policy struct {
    MaxAttempts  int
    InitialDelay time.Duration
    MaxDelay     time.Duration

    Budget      *RetryBudget
    ShouldRetry func(error) bool
}

MaxAttempts includes the original attempt.

For example, MaxAttempts: 3 means one original attempt and at most two retries.

Next, calculate the exponential backoff.

func backoff(
    attempt int,
    initial time.Duration,
    maximum time.Duration,
) time.Duration {
    delay := initial

    for i := 1; i < attempt; i++ {
        if delay >= maximum/2 {
            return maximum
        }

        delay *= 2
    }

    if delay > maximum {
        return maximum
    }

    return delay
}

The delay grows exponentially until it reaches the configured maximum.

For example:

Retry	Maximum delay
1	100ms
2	200ms
3	400ms
4	800ms
5	1.6s

The next step is jitter.

Without jitter, clients that fail at approximately the same time may retry at approximately the same time. This creates synchronized traffic spikes.

Full jitter chooses a random delay between zero and the calculated maximum:

func fullJitter(maxDelay time.Duration) time.Duration {
    if maxDelay <= 0 {
        return 0
    }

    return time.Duration(
        rand.Float64() * float64(maxDelay),
    )
}

The package-level math/rand functions are safe for concurrent use. If you need reproducible tests, inject a random source rather than depending on global randomness.

Now we can implement the retry executor.

func Do(
    ctx context.Context,
    policy Policy,
    operation func(context.Context) error,
) error {
    if policy.MaxAttempts < 1 ||
        policy.InitialDelay <= 0 ||
        policy.MaxDelay < policy.InitialDelay ||
        operation == nil {
        return ErrInvalidPolicy
    }

    if policy.ShouldRetry == nil {
        return ErrInvalidPolicy
    }

    for attempt := 1; attempt <= policy.MaxAttempts; attempt++ {
        if err := ctx.Err(); err != nil {
            return err
        }

        err := operation(ctx)
        if err == nil {
            // Only successful initial attempts replenish the budget.
            if attempt == 1 && policy.Budget != nil {
                policy.Budget.RecordSuccess()
            }

            return nil
        }

        if attempt == policy.MaxAttempts {
            return err
        }

        if !policy.ShouldRetry(err) {
            return err
        }

        // Do not generate more traffic when the shared
        // retry budget has been exhausted.
        if policy.Budget != nil && !policy.Budget.TryTake() {
            return err
        }

        maxDelay := backoff(
            attempt,
            policy.InitialDelay,
            policy.MaxDelay,
        )

        delay := fullJitter(maxDelay)

        timer := time.NewTimer(delay)

        select {
        case <-ctx.Done():
            if !timer.Stop() {
                <-timer.C
            }

            return ctx.Err()

        case <-timer.C:
        }
    }

    return nil
}

A few design decisions are worth highlighting.

First, the budget is consumed before the retry.

Once the service decides to schedule another attempt, it spends a token. If the context is canceled during the backoff period, that token is not refunded.

This is a conservative choice. The budget controls permission to generate retry work, not only completed downstream calls.

Second, only successful initial attempts replenish the budget.

A successful retry does not replenish it. Otherwise, a service could keep generating retries and effectively fund its own retry traffic during a degraded period.

Third, the context is checked before each attempt and during backoff.

A canceled request should not continue generating work.

The executor returns the last operation error when the retry budget is exhausted. This preserves the underlying failure for callers, although production implementations may also expose a separate retry-denied metric or callback.

6. Choosing Which Errors Are Retryable

Not every failure should trigger a retry.

A retry policy should distinguish between transient failures and permanent failures.

Examples of potentially retryable failures:

• Temporary network errors
• Connection resets
• Selected HTTP 5xx responses
• HTTP 429 responses, subject to server guidance
• Temporary resource unavailability

Examples of failures that generally should not be retried:

• Invalid input
• Authentication or authorization failures
• Unsupported operations
• Validation errors
• Requests that have already exceeded their deadline

A simple classifier might look like this:

func shouldRetry(err error) bool {
    var netErr net.Error

    if errors.As(err, &netErr) {
        return netErr.Timeout() || netErr.Temporary()
    }

    return false
}

This is only an illustration. net.Error.Temporary() is deprecated, and real applications should classify errors according to their HTTP client, database driver, RPC framework, and dependency-specific semantics.

For HTTP clients, for example, a 429 response may include Retry-After. A robust policy should respect that server-provided delay where applicable, while still enforcing the request deadline and retry budget.

Also remember that retrying a request is not always safe.

A GET request is generally easier to retry than a payment or order-creation request. For operations with side effects, use idempotency keys or another deduplication mechanism.

A retry budget controls traffic. It does not make an operation safe to execute twice.

7. Putting It Together

Let's use the retry executor with a market data provider.

budget := retry.NewRetryBudget(
    20,
    0.1,
)

policy := retry.Policy{
    MaxAttempts:  3,
    InitialDelay: 100 * time.Millisecond,
    MaxDelay:     2 * time.Second,
    Budget:       budget,
    ShouldRetry:  shouldRetry,
}

ctx, cancel := context.WithTimeout(
    context.Background(),
    3*time.Second,
)
defer cancel()

err := retry.Do(
    ctx,
    policy,
    func(ctx context.Context) error {
        return marketDataClient.Fetch(ctx, "AAPL")
    },
)

if err != nil {
    // Handle the final operation error.
}

The behavior is now bounded by multiple independent controls:

• The request can make at most three attempts.
• The request cannot exceed its context deadline.
• Each retry uses exponential backoff with jitter.
• The service must have an available retry token.
• Only successful initial attempts replenish the shared budget.

This is much safer than allowing every request to retry independently.

8. Retry Budgets and Other Resilience Patterns

Retry budgets do not replace other resilience mechanisms. They complement them.

Mechanism	Primary responsibility
Timeout	Limits how long an operation can run
Backpressure	Controls how much work enters a system
Load shedding	Rejects work when capacity is insufficient
Bulkhead	Isolates resource consumption
Circuit breaker	Stops calls to an unhealthy dependency
Retry budget	Limits additional traffic generated by retries

Consider how these mechanisms work together.

A request enters the service. Backpressure controls admission. A bulkhead limits concurrent calls to the provider. The request has a deadline. If the provider fails, the retry policy classifies the error and checks the retry budget. If the dependency is persistently unhealthy, the circuit breaker can stop further calls.

Each mechanism addresses a different failure mode.

A subtle interaction: Bulkheads and retries

Retry placement matters.

Suppose a bulkhead permit is acquired once around the entire retry loop. The permit remains occupied during backoff.

That may be desirable if the goal is to limit the number of complete operations, but it can also waste scarce concurrency capacity while requests are sleeping.

Alternatively, acquiring a permit for each individual attempt releases capacity during backoff. However, this allows more logical requests to remain in progress simultaneously.

Neither approach is universally correct. The choice depends on what the bulkhead is protecting:

• If it protects active downstream calls, acquire a permit per attempt.
• If it protects the number of in-flight logical operations, hold the permit across the complete retry operation.

The important thing is to define the resource boundary explicitly.

9. Is a Local Retry Budget Enough?

A mutex-protected retry budget is local to one process.

If you run ten replicas, each replica has its own budget.

For example:

• 10 replicas
• 20 tokens per replica
• 200 tokens of aggregate retry capacity

This may be perfectly acceptable. Local budgets are simple, fast, and do not introduce another distributed dependency.

But if the external provider enforces a strict global quota, local budgets alone may not be sufficient.

Possible approaches include:

1. Local budgets: Simple and independent, but aggregate capacity grows with replica count.
2. Centralized budget: Coordinates retry allowance across instances, but introduces network calls and another failure boundary.
3. Provider-aware limits: Use the provider's documented quota and response headers to coordinate traffic.
4. Centralized dispatch: Route downstream work through a shared queue or dispatcher that controls the total attempt rate.

Do not introduce a distributed retry-budget service automatically. The coordination mechanism can become more expensive and less reliable than the problem it is trying to solve.

Start with local budgets when they provide sufficient protection. Move to centralized coordination only when the actual quota or failure model requires it.

10. Observability: Measure Retry Behavior

A retry policy that cannot be observed is difficult to operate.

At minimum, expose metrics that help answer:

• How often are requests retried?
• How many retries are denied by the budget?
• Which dependencies consume the most retry capacity?
• How often do requests exhaust their attempt limit?
• How much time is spent waiting between attempts?

Useful metrics include:

Metric	Type	Purpose
retry_attempts_total	Counter	Total retry attempts
retry_budget_denied_total	Counter	Retries rejected by the budget
retry_budget_tokens	Gauge	Current available tokens
retry_exhausted_total	Counter	Requests reaching the attempt limit
retry_delay_seconds	Histogram	Time spent in retry backoff

Avoid using raw URLs, request IDs, user IDs, or other high-cardinality values as metric labels.

Prefer bounded dimensions such as dependency name, operation, and outcome.

A particularly useful signal is the ratio of retries to original requests.

If the retry rate starts increasing while the initial success rate drops, the system may be entering a retry-amplification cycle.

The retry budget should make that cycle bounded, but the metrics help you understand why it happened.

11. Testing Retry Budgets

Concurrency behavior should be tested explicitly.

The first test verifies that successful initial requests replenish the budget and that retries consume tokens.

func TestRetryBudget(t *testing.T) {
    budget := NewRetryBudget(2, 1)

    budget.RecordSuccess()
    budget.RecordSuccess()

    if !budget.TryTake() {
        t.Fatal("expected first token")
    }

    if !budget.TryTake() {
        t.Fatal("expected second token")
    }

    if budget.TryTake() {
        t.Fatal("expected budget to be exhausted")
    }
}

The next test checks that concurrent callers cannot consume more tokens than are available.

func TestRetryBudgetConcurrent(t *testing.T) {
    const (
        tokens   = 100
        workers  = 1000
    )

    budget := NewRetryBudget(tokens, 1)

    for i := 0; i < tokens; i++ {
        budget.RecordSuccess()
    }

    var (
        wg       sync.WaitGroup
        accepted atomic.Int64
    )

    wg.Add(workers)

    for i := 0; i < workers; i++ {
        go func() {
            defer wg.Done()

            if budget.TryTake() {
                accepted.Add(1)
            }
        }()
    }

    wg.Wait()

    if got := accepted.Load(); got != tokens {
        t.Fatalf(
            "expected %d successful acquisitions, got %d",
            tokens,
            got,
        )
    }
}

Run the tests with the race detector:

go test -race ./...

You should also test the retry executor itself:

• A successful first attempt should not retry.
• A transient failure should retry when budget is available.
• A permanent failure should return immediately.
• An exhausted budget should prevent another attempt.
• Context cancellation should interrupt backoff.
• The maximum attempt count should be respected.
• A successful retry should not replenish the budget.
• A failed initial request should not replenish the budget.

Finally, test behavior under load. A retry policy that passes unit tests can still create unexpected traffic patterns when thousands of requests fail together.

12. Production Checklist

Before enabling retries in a production service, verify the following:

• [ ] Maximum attempts include the original request.
• [ ] Retryable errors are explicitly classified.
• [ ] Every request has a deadline or timeout.
• [ ] Backoff is exponential and bounded.
• [ ] Jitter prevents synchronized retries.
• [ ] A shared budget limits aggregate retry traffic.
• [ ] Non-idempotent operations have appropriate protection.
• [ ] Provider-specific rate limits and Retry-After are respected.
• [ ] Retry behavior is observable.
• [ ] Bulkhead permit ownership is clearly defined.
• [ ] Tests cover cancellation, concurrency, and exhaustion.
• [ ] Replica count and aggregate retry capacity are understood.

Conclusion

Retries are useful, but unlimited retry behavior can turn a temporary dependency failure into a system-wide incident.

A maximum attempt count protects individual requests. A retry budget protects the service from the combined effect of those requests.

The key idea is simple: successful traffic can fund a limited amount of additional work, but retries should not be allowed to generate unlimited traffic when the system is already unhealthy.

In Go, a concurrency-safe token bucket provides a straightforward way to implement this policy. Combined with deadlines, exponential backoff, jitter, bulkheads, and circuit breakers, retry budgets help make failure behavior more predictable.

Resilience is not about trying again forever. It's about knowing when another attempt is worth the cost.