Load Shedding in Go: Protecting Latency Under Overload

A service does not have to be completely down to be unavailable.

Sometimes every endpoint still responds, every dependency is technically reachable, and CPU utilization looks manageable. Yet requests take several seconds, queues keep growing, database connections are exhausted, and clients begin retrying failed calls.

The service is alive, but it is no longer delivering a useful experience.

This is where load shedding becomes an important reliability technique.

Instead of accepting every request and allowing the system to collapse under its own workload, a service deliberately rejects work it cannot safely process. The goal is not to maximize the number of accepted requests. It is to preserve useful throughput and predictable latency for the requests that matter most.

In this article, we will explore how to implement load shedding in Go, where to place it, how to combine it with backpressure and deadlines, and how to avoid turning overload protection into another source of failure.

1. Why accepting every request is dangerous

Imagine a Go API that can process approximately 800 requests per second under normal operating conditions.

Traffic suddenly increases to 2,000 requests per second.

If the application accepts every request, several things can happen:

• Goroutines accumulate while waiting for database connections.
• Requests exceed their deadlines while consuming resources.
• Queues grow faster than workers can drain them.
• Clients retry failed requests, increasing the incoming load.
• Downstream services become overloaded as well.

The system may eventually recover, but during the incident, almost every request can become slow or unsuccessful.

A crucial distinction is that more accepted work does not necessarily mean more completed work.

When a service operates beyond its sustainable capacity, additional requests can consume resources without producing useful results.

Load shedding limits this damage by rejecting some work early, before it consumes expensive resources.

2. Load shedding versus rate limiting and backpressure

These techniques are related, but they solve different problems.

Technique	Primary purpose
Rate limiting	Control how frequently requests are admitted over time
Concurrency limiting	Bound the number of operations running simultaneously
Backpressure	Prevent work from accumulating faster than it can be processed
Load shedding	Reject work when accepting it would threaten system stability
Circuit breaking	Stop repeatedly calling a dependency that appears unhealthy

For example, a rate limiter might allow 100 requests per second from a particular client. A concurrency limiter might allow only 50 database-heavy operations at once. A bounded queue might prevent more than 200 jobs from waiting.

Load shedding is the decision made when capacity is exhausted: reject this request now rather than let it damage the requests already in progress.

These controls are most effective when used together.

3. Start with a concurrency-based admission limit

A simple first implementation is to cap the number of active requests.

When the limit is reached, the application immediately returns 503 Service Unavailable.

This is useful when a service has a relatively stable resource budget and a clear relationship between active requests and resource consumption.

package admission

import (
    "net/http"
    "sync/atomic"
)

type Limiter struct {
    maxActive int64
    active    atomic.Int64
}

func NewLimiter(maxActive int64) *Limiter {
    if maxActive <= 0 {
        panic("maxActive must be greater than zero")
    }

    return &Limiter{
        maxActive: maxActive,
    }
}

func (l *Limiter) Middleware(next http.Handler) http.Handler {
    return http.HandlerFunc(func(
        w http.ResponseWriter,
        r *http.Request,
    ) {
        for {
            current := l.active.Load()

            if current >= l.maxActive {
                http.Error(
                    w,
                    "service temporarily overloaded",
                    http.StatusServiceUnavailable,
                )
                return
            }

            if l.active.CompareAndSwap(current, current+1) {
                break
            }
        }

        defer l.active.Add(-1)

        next.ServeHTTP(w, r)
    })
}

Register the middleware:

mux := http.NewServeMux()
mux.HandleFunc("/api/orders", createOrder)

limiter := admission.NewLimiter(500)

server := &http.Server{
    Addr:    ":8080",
    Handler: limiter.Middleware(mux),
}

The admission limit is enforced using an atomic compare-and-swap operation. Concurrent requests compete to reserve a slot, and only requests that successfully reserve a slot proceed.

The defer ensures that the active count is decremented when the handler returns, including when the handler exits through an ordinary early return.

What this implementation guarantees

It bounds the number of requests executing inside the wrapped handler to the configured limit, provided the middleware is applied consistently.

It does not guarantee that every admitted request will finish quickly. A request can still wait on a database, an external API, or another internal operation.

It also does not bound all work in the process. If a handler starts background goroutines and returns, those goroutines can continue after the admission slot is released.

That distinction matters: an HTTP request limit is not automatically a limit on the total work initiated by HTTP requests.

For long-running background operations, the work itself needs its own lifecycle and concurrency controls.

4. Why concurrency limits are not enough

Suppose two endpoints share the same global limit:

• GET /api/portfolio: a lightweight read needed by users.
• POST /api/reports: an expensive report-generation operation.

If report generation occupies every available slot, portfolio requests may be rejected even though they are inexpensive.

This is a form of resource contention between workloads with different priorities.

A global limit protects the process, but it does not guarantee fair access to capacity.

There are several ways to improve this design:

• Give expensive endpoints a smaller concurrency budget.
• Reserve some capacity for critical operations.
• Use separate worker pools for different workload classes.
• Apply per-user or per-tenant quotas where appropriate.
• Reject low-priority work before it reaches shared, expensive resources.

For example, report generation might have a concurrency limit of 20, while interactive reads have a separate limit. Both still operate under a global process-level budget.

The numbers should come from load testing and production measurements, not guesswork.

5. Make overload rejection explicit

A rejected request should communicate that the service is temporarily unable to accept work.

For an HTTP API, 503 Service Unavailable is often appropriate when the service itself is overloaded.

429 Too Many Requests is generally more appropriate when a client has exceeded a rate limit or quota.

These statuses are not interchangeable in every situation. Choose based on the actual admission policy and document the contract for clients.

A response can include Retry-After when the service has a reasonable basis for suggesting when a retry may succeed.

func rejectOverloaded(w http.ResponseWriter) {
    w.Header().Set("Retry-After", "1")

    http.Error(
        w,
        "service temporarily overloaded",
        http.StatusServiceUnavailable,
    )
}

The value above is only an example. A fixed one-second delay is not universally correct, and it does not mean the service will recover in one second.

If the system has no reliable estimate of recovery time, avoid presenting a guessed delay as a guarantee.

Also, do not tell every client to retry immediately. Coordinated retries can create a second traffic spike just as the original overload begins to subside.

6. Propagate deadlines and stop useless work

Load shedding is most effective when combined with deadlines.

Consider a request with a 500 ms deadline. If it has already spent 450 ms waiting for a downstream dependency, starting another expensive operation may be pointless.

The operation could consume resources and still produce a result that the client will never use.

Every downstream operation should receive the request context whenever possible:

func GetPortfolio(
    ctx context.Context,
    db *sql.DB,
    userID string,
) ([]Position, error) {
    rows, err := db.QueryContext(
        ctx,
        `SELECT symbol, quantity
         FROM positions
         WHERE user_id = $1`,
        userID,
    )
    if err != nil {
        return nil, err
    }
    defer rows.Close()

    var positions []Position

    for rows.Next() {
        var p Position

        if err := rows.Scan(&p.Symbol, &p.Quantity); err != nil {
            return nil, err
        }

        positions = append(positions, p)
    }

    if err := rows.Err(); err != nil {
        return nil, err
    }

    return positions, nil
}

The example assumes the Position type and database package are defined in the application.

QueryContext allows the database operation to observe cancellation and deadlines. Whether cancellation promptly stops work also depends on the database driver and the underlying system.

For HTTP clients, always use request contexts for outbound calls. For background jobs, define a separate job context and cancellation policy rather than detaching work casually with context.Background().

There is an important limit here: cancellation is a request to stop, not proof that every downstream operation has already stopped. Your application still needs bounded resource usage and appropriate cleanup.

7. Do not shed every kind of work equally

When a service is overloaded, rejecting all incoming requests may preserve process stability while still causing an unnecessarily poor user experience.

A better policy distinguishes between workload classes.

For example:

Workload	Possible overload policy
Health and readiness probes	Keep lightweight; avoid expensive dependencies in liveness checks
Interactive reads	Preserve reserved capacity where possible
Expensive analytics	Reduce concurrency or reject early
Optional recommendations	Return a documented fallback or omit the section
Durable background jobs	Queue within a defined capacity or reject submission
Administrative operations	Apply explicit priority and access controls

Priority must be implemented carefully. If every request is marked critical, the priority system provides no protection.

Similarly, a health endpoint should not bypass every security or resource control without thought. Keep probes cheap, limit their scope, and ensure they accurately represent the health signal they are intended to report.

8. Shed work before expensive resources are acquired

The placement of admission control matters.

Consider this request path:

1. Parse the request.
2. Authenticate the user.
3. Fetch data from the database.
4. Call an external API.
5. Discover that the system has reached its concurrency limit.
6. Reject the request.

The rejection came too late. The request already consumed resources.

For expensive operations, admission should happen before the expensive work begins.

At the same time, some cheap checks should happen before admission. For example, rejecting malformed requests or unauthenticated callers early can prevent invalid traffic from consuming scarce capacity.

A practical order is:

1. Apply edge-level protections and request-size limits.
2. Perform inexpensive validation and required authentication.
3. Apply the relevant admission policy.
4. Acquire resource-specific permits.
5. Perform expensive work with bounded deadlines.
6. Release permits reliably.

There is no single ordering that fits every application. The important principle is to reject work before it consumes the resource you are trying to protect.

9. Measure rejected work, not just successful requests

A service can have low latency among successful requests while rejecting most of its traffic.

If monitoring only tracks successful request duration, the system may appear healthy during a serious overload incident.

At minimum, track:

• requests_accepted_total
• requests_rejected_total
• requests_in_flight
• Request duration and timeout counts
• Queue depth and queue wait time
• Database pool usage and wait duration
• Downstream error rates
• CPU, memory, and goroutine counts

For Prometheus, counters and gauges are useful starting points:

var (
    requestsAccepted = prometheus.NewCounter(
        prometheus.CounterOpts{
            Name: "http_requests_accepted_total",
            Help: "Total number of requests admitted.",
        },
    )

    requestsRejected = prometheus.NewCounterVec(
        prometheus.CounterOpts{
            Name: "http_requests_rejected_total",
            Help: "Total number of requests rejected by admission control.",
        },
        []string{"reason"},
    )

    requestsInFlight = prometheus.NewGauge(
        prometheus.GaugeOpts{
            Name: "http_requests_in_flight",
            Help: "Number of requests currently executing.",
        },
    )
)

These metrics use the Prometheus Go client and need to be registered with your application's metrics registry. The middleware should update them at the actual admission and completion points.

Keep metric labels bounded. Avoid labels containing user IDs, raw URLs, request IDs, or other unbounded values. Use route templates such as /api/orders/{id} rather than individual paths.

Watch rejection ratios alongside latency

A useful signal is the proportion of requests rejected during a time window:

[
\text{Rejection Ratio} =
\frac{\text{Rejected Requests}}
{\text{Accepted Requests}+\text{Rejected Requests}}
]

A rising rejection ratio is not automatically a failure. During a sudden traffic spike, controlled rejection may be exactly what protects the service.

The key question is whether the service is preserving its latency and success-rate objectives for admitted requests while rejecting excess demand.

10. Test overload behavior deliberately

Load shedding should not be validated only by reading the code.

Test what happens when traffic exceeds the configured limit.

For the atomic limiter, a basic concurrent test can verify that no more than the configured number of handlers are active simultaneously:

func TestLimiterBoundsConcurrency(t *testing.T) {
    const limit = int64(5)

    limiter := admission.NewLimiter(limit)

    var active atomic.Int64
    var peak atomic.Int64

    handler := http.HandlerFunc(func(
        w http.ResponseWriter,
        r *http.Request,
    ) {
        current := active.Add(1)
        defer active.Add(-1)

        for {
            old := peak.Load()
            if current <= old || peak.CompareAndSwap(old, current) {
                break
            }
        }

        time.Sleep(20 * time.Millisecond)
        w.WriteHeader(http.StatusOK)
    })

    server := httptest.NewServer(limiter.Middleware(handler))
    defer server.Close()

    var wg sync.WaitGroup

    for range 100 {
        wg.Add(1)

        go func() {
            defer wg.Done()

            resp, err := http.Get(server.URL)
            if err != nil {
                return
            }
            defer resp.Body.Close()
        }()
    }

    wg.Wait()

    if got := peak.Load(); got > limit {
        t.Fatalf("peak concurrency = %d, want <= %d", got, limit)
    }
}

This test uses integer range syntax, available in Go 1.22 and later. It demonstrates the concurrency bound, but it does not assert the exact number of rejected requests or validate latency under sustained load.

A fuller test suite should also cover:

• Requests rejected when all slots are occupied.
• Slots released after handler errors or panics.
• Context cancellation during downstream work.
• Shutdown while requests are active.
• Different workload classes competing for capacity.
• Sustained overload and recovery after demand falls.

Run load tests with realistic dependencies and request distributions. A limiter that looks correct in isolation may still be placed too late in the request path or configured too high for the database connection pool.

11. Common mistakes

Setting the limit based only on CPU cores. Some requests are CPU-bound; others spend most of their time waiting on databases or remote services. Determine safe concurrency through load testing and resource measurements.

Returning 503 after work has already started. Admission control should protect the scarce resource, not merely change the final response.

Retrying every rejected request. Immediate retries can amplify overload. Use bounded retry budgets, exponential backoff, jitter, and retry policies that distinguish overload from permanent failures.

Using one global limit for every workload. A noisy, expensive endpoint can starve critical operations. Consider per-resource limits and reserved capacity.

Treating background work as free. If an HTTP handler starts goroutines or submits jobs to an unbounded queue, the request limit may not protect the actual workload.

Ignoring rejected-request metrics. A low error rate among accepted requests does not tell you how much demand the service is turning away.

Choosing a limit without revisiting it. Traffic patterns, payload sizes, dependency behavior, and infrastructure change. Capacity limits need operational review.

12. A production checklist

Before deploying load shedding, make sure you can answer these questions:

• What resource or service are we protecting?
• At which point do we reject excess work?
• Are concurrency and queue sizes bounded?
• Do critical endpoints have adequate capacity?
• Are deadlines propagated to downstream operations?
• Are rejected requests observable and categorized?
• Are retry policies safe during overload?
• Can the service recover automatically when demand decreases?
• Have we tested both overload and recovery?
• Are the limits documented and configurable?

Final thoughts

Load shedding is not about refusing work arbitrarily. It is about making an explicit decision before overload makes that decision for you.

A reliable service does not need to accept every request. It needs to use its available capacity effectively, protect its critical paths, and fail predictably when demand exceeds what it can safely handle.

In Go, atomic admission limits are a useful starting point. In production systems, they should be combined with bounded queues, resource-specific concurrency limits, cancellation, deadlines, observability, and realistic load testing.

The goal is not to keep every request alive. It is to keep the service useful when not every request can be served.