A service can have healthy code, reasonable timeouts, and a circuit breaker—and still suffer a cascading failure.
Imagine a Go API that serves portfolio data, market prices, and investment reports. It calls three external dependencies:
- A market data provider
- A portfolio database
- A reporting service
The reporting service becomes slow. Requests begin waiting for it, goroutines remain occupied, and incoming traffic continues to grow. Eventually, requests that do not even need reports start failing because they compete for the same resources.
The original failure was isolated to one dependency. The application's resource usage was not.
This is the problem the bulkhead pattern is designed to address.
In this article, we'll build dependency-specific bulkheads in Go, explore admission and waiting policies, examine common implementation mistakes, and discuss how to verify that isolation actually works under load.
1. What is the bulkhead pattern?
The name comes from the watertight compartments inside a ship. If one compartment floods, the others should remain intact.
In software, a bulkhead creates a resource boundary around a particular dependency, workload, or tenant. When one boundary reaches capacity, work inside that boundary is limited without automatically consuming capacity reserved for others.
Consider a service with three downstream dependencies.
Without isolation:
┌────────────────────┐
Incoming ───────►│ Shared Capacity │
└─────────┬──────────┘
│
┌──────────┼──────────┐
▼ ▼ ▼
Market Portfolio Reporting
Data Database Service
If reporting requests become slow, they can consume the shared capacity needed by market data and portfolio requests.
With dependency-specific bulkheads:
┌────────────────────┐
Incoming ───────►│ Go API │
└─────────┬──────────┘
│
┌────────────┼────────────┐
▼ ▼ ▼
┌────────┐ ┌────────┐ ┌────────┐
│ Market │ │Portfolio│ │Report │
│ Limit │ │ Limit │ │ Limit │
└───┬────┘ └───┬────┘ └───┬────┘
▼ ▼ ▼
Market API Database Report API
Each dependency receives its own concurrency budget.
If the reporting bulkhead is full, new reporting work can be rejected or briefly delayed according to policy. Market data and portfolio operations retain their own permits.
The important property is not that failures disappear. It is that one dependency cannot consume all the capacity reserved for unrelated work.
2. Bulkheads are not circuit breakers
Both patterns help prevent cascading failures, but they protect against different problems.
| Pattern | What it protects against |
|---|---|
| Timeout | Waiting too long for an operation |
| Rate limiter | Excessive request frequency |
| Load shedding | Accepting more work than the service can safely handle |
| Circuit breaker | Repeated calls to a dependency that is failing |
| Bulkhead | One workload consuming resources needed by another |
Suppose a reporting API has a 10-second response time.
A timeout limits how long a caller waits. A circuit breaker may stop future calls after repeated failures. Neither automatically prevents 100 concurrent reporting calls from consuming every available application slot.
A bulkhead limits that concurrency independently.
These patterns complement one another. A typical dependency call may use a bulkhead, a deadline, and a circuit breaker, with a carefully bounded retry policy where retries are safe.
3. Start with a bounded semaphore
Go's channel primitives provide a straightforward way to limit concurrent operations.
A buffered channel can represent available permits. Each operation acquires a permit before executing and releases it when finished.
For a dependency that should allow at most 10 concurrent calls:
var reportingSlots = make(chan struct{}, 10)
However, the acquisition policy matters.
If all permits are occupied, should a new call wait for capacity or fail immediately?
For an already-slow dependency, waiting indefinitely can simply move the queue into application memory. In many latency-sensitive paths, a non-blocking bulkhead is preferable.
Let's implement a reusable version.
4. Implementing a fail-fast bulkhead
The following implementation limits active operations and returns a recognizable error when capacity is exhausted.
package bulkhead
import (
"context"
"errors"
)
var ErrFull = errors.New("bulkhead capacity exhausted")
type Bulkhead struct {
slots chan struct{}
}
func New(maxConcurrent int) *Bulkhead {
if maxConcurrent <= 0 {
panic("maxConcurrent must be greater than zero")
}
return &Bulkhead{
slots: make(chan struct{}, maxConcurrent),
}
}
func (b *Bulkhead) Do(
ctx context.Context,
fn func(context.Context) error,
) error {
if err := ctx.Err(); err != nil {
return err
}
select {
case b.slots <- struct{}{}:
// A permit has been acquired.
case <-ctx.Done():
return ctx.Err()
default:
return ErrFull
}
defer func() {
<-b.slots
}()
// Cancellation may race with permit acquisition.
if err := ctx.Err(); err != nil {
return err
}
return fn(ctx)
}
This implementation deliberately does not queue callers waiting for a permit.
If capacity is available, the operation proceeds. If every slot is occupied, the caller receives ErrFull immediately.
The second context check handles the case where cancellation races with permit acquisition. If the context is canceled by that point, the deferred release returns the permit.
The function passed to Do should honor its context and should not start detached work that outlives the operation without an independent lifecycle policy.
Using the bulkhead
var reportBulkhead = bulkhead.New(10)
func GenerateReport(
ctx context.Context,
reporter Reporter,
) error {
return reportBulkhead.Do(ctx, func(
ctx context.Context,
) error {
return reporter.Generate(ctx)
})
}
Reporter is an application-defined interface in this example.
The reporting dependency now has an independent concurrency limit. If it becomes slow and all 10 permits are occupied, additional reporting calls fail fast rather than accumulating behind the dependency.
The limit is a policy choice, not a universal recommendation. Choose it based on load testing, downstream capacity, latency objectives, and the number of application instances.
5. Choose the right rejection behavior
A full bulkhead does not always need to produce an HTTP error directly. The appropriate response depends on the caller and the operation.
For example:
func handleReport(
w http.ResponseWriter,
r *http.Request,
) {
err := GenerateReport(r.Context(), reporter)
switch {
case errors.Is(err, bulkhead.ErrFull):
http.Error(
w,
"reporting capacity temporarily exhausted",
http.StatusServiceUnavailable,
)
return
case errors.Is(err, context.Canceled):
// The client or upstream caller canceled the request.
return
case errors.Is(err, context.DeadlineExceeded):
http.Error(
w,
"report generation timed out",
http.StatusGatewayTimeout,
)
return
case err != nil:
http.Error(
w,
"report generation failed",
http.StatusBadGateway,
)
return
}
w.WriteHeader(http.StatusNoContent)
}
This is a simplified handler; the application would need to define reporter, imports, and its actual response contract.
The important distinction is between admission rejection, caller cancellation, deadline expiration, and dependency failure. They are different events and should not be collapsed into one generic error.
For public APIs, document the overload response and decide whether a 503 or another status is appropriate for the contract. Avoid encouraging immediate retries when the dependency is already saturated.
6. A bulkhead must protect the actual bottleneck
A semaphore limits how many operations enter a code path. It does not automatically isolate every resource used by that path.
Consider a Go service that calls three downstream APIs through a single http.Client.
The client uses a shared http.Transport, which manages connection pooling and other transport resources. Even if the application has separate concurrency semaphores, shared transport configuration or other shared resources may still create contention.
A useful design starts by identifying what is being protected:
- Concurrent requests to a dependency
- Connections to a database
- Worker capacity for background jobs
- CPU-intensive computation
- Memory-intensive report generation
- Per-tenant capacity
- External provider quotas
Then enforce limits at the relevant resource boundary.
For outbound HTTP calls, use dedicated transports when separate connection-pool configuration or transport-level isolation is useful:
func newHTTPClient(maxConns int) *http.Client {
transport := &http.Transport{
MaxConnsPerHost: maxConns,
MaxIdleConns: maxConns,
MaxIdleConnsPerHost: maxConns,
}
return &http.Client{
Transport: transport,
Timeout: 2 * time.Second,
}
}
var (
marketClient = newHTTPClient(20)
reportClient = newHTTPClient(5)
)
This snippet requires the net/http and time imports.
The clients have separate transports and different connection limits. The timeout is an example; production systems should choose deadlines based on their latency budgets and may prefer request contexts plus explicit transport settings.
Connection limits are not a replacement for application-level concurrency limits. A connection can be reused, and one operation can involve more than one resource. Use both controls when they protect distinct bottlenecks.
Also remember that a custom transport changes the default transport behavior. Configure TLS, proxy handling, connection lifetimes, idle timeouts, and other relevant settings according to your environment.
7. Waiting queues: useful, but not free
Fail-fast admission is not always the best policy.
Some operations can tolerate a short wait for capacity. In that case, a bounded waiting policy may improve utilization without allowing unbounded accumulation.
There are three common approaches:
- Fail fast: Reject immediately when no permit is available.
- Wait with a deadline: Wait for a permit, but only until the caller's context expires.
- Bound active work and queued work separately: Allow a defined number of operations to run and a small number to wait.
The third approach requires particular care. A semaphore that limits active calls does not limit the number of goroutines waiting to acquire it.
A context-aware waiting version can be implemented like this:
func (b *Bulkhead) DoWait(
ctx context.Context,
fn func(context.Context) error,
) error {
if err := ctx.Err(); err != nil {
return err
}
select {
case b.slots <- struct{}{}:
case <-ctx.Done():
return ctx.Err()
}
defer func() {
<-b.slots
}()
if err := ctx.Err(); err != nil {
return err
}
return fn(ctx)
}
This method waits for a permit or cancellation. It does not bound the number of callers waiting in goroutines.
If the service can receive an unbounded number of waiting callers, use an explicit bounded admission queue, reject before creating excess work, or use a bounded worker pool.
The distinction is important: a concurrency limit without a queue bound can still leave the system with too many waiting goroutines.
8. Isolate workloads, not just services
Dependency-level bulkheads are useful, but some applications need additional boundaries.
Consider a service with the following operations:
- Interactive portfolio queries
- Historical analytics
- Exporting large reports
- Background notifications
These workloads may share dependencies while having very different resource profiles.
A report export can occupy database connections for much longer than a small portfolio query. If both share the same database pool and the same admission policy, the reporting workload may starve interactive traffic.
Possible isolation strategies include:
- Separate concurrency limits by workload.
- Dedicated worker pools for long-running jobs.
- Bounded queues for background processing.
- Database connection budgets for distinct workloads, where operationally feasible.
- Reserved capacity for latency-sensitive operations.
- Per-tenant limits to prevent one customer from dominating capacity.
Do not create a separate pool for every function by default. Each additional boundary introduces configuration, monitoring, and capacity-planning costs.
The goal is meaningful isolation around resources that have different failure modes or service-level requirements.
9. Beware of multiplicative concurrency
A local bulkhead limit is not necessarily a global limit.
Suppose a service runs 8 replicas, and each replica allows 20 concurrent calls to a reporting provider.
The potential aggregate concurrency is:
[
8 \times 20 = 160
]
If the provider can safely handle only 50 concurrent calls, the local configuration does not protect the global dependency.
This is one of the most common capacity-planning mistakes in distributed systems.
When choosing limits, account for:
- The maximum number of replicas.
- Autoscaling behavior.
- Other applications calling the same dependency.
- Provider-wide quotas.
- Deployment overlap during rolling updates.
- Traffic distribution and uneven load.
A local bulkhead is excellent for protecting each process. A global quota may require a shared admission mechanism, provider-side limits, a centralized dispatcher, or a different partitioning strategy.
Avoid introducing a distributed semaphore unless global coordination is genuinely needed. The coordination service can become another dependency whose failure must be handled.
10. Combining bulkheads with circuit breakers and retries
These patterns solve different parts of the same failure scenario.
Imagine a reporting provider starts timing out.
- The bulkhead limits how many reporting calls can be active.
- The deadline prevents each call from waiting indefinitely.
- The circuit breaker can stop repeated calls after a failure threshold is reached.
- A retry policy may retry selected transient failures, within a strict budget.
- Load shedding rejects additional work when admission capacity is exhausted.
Order and interaction matter.
If retries happen inside a bulkhead permit, the permit remains occupied across attempts and backoff periods. This can reduce pressure on the downstream service, but it also ties up local capacity while no useful work is being performed.
If retries happen outside the bulkhead, every attempt must reacquire capacity, which may make admission more responsive but can complicate retry scheduling.
Neither choice is universally correct. Define whether the permit represents a whole logical operation or one outbound attempt, then test the behavior under slow failures.
Never retry a bulkhead rejection immediately by default. The rejection indicates that local capacity is exhausted; an immediate retry simply competes for the same capacity again.
For operations with side effects, retries also require idempotency or another mechanism that makes repeated attempts safe.
11. Observability: prove that isolation works
A bulkhead without metrics is difficult to tune and nearly impossible to validate during an incident.
For each bulkhead, consider recording:
- Active operations
- Configured concurrency capacity
- Rejected acquisitions
- Wait duration, if waiting is allowed
- Operation duration
- Timeouts and cancellations
- Downstream error rates
- Circuit-breaker state, where applicable
Useful metric names might include:
dependency_bulkhead_active
dependency_bulkhead_capacity
dependency_bulkhead_rejected_total
dependency_request_duration_seconds
dependency_request_errors_total
Use bounded labels such as a dependency name or workload class. Avoid raw user IDs, request IDs, and other high-cardinality values.
Monitor the bulkheads independently. A global saturation metric can hide the fact that reporting is exhausted while market data remains healthy.
A useful operational question is not merely, “Is the service saturated?”
It is: Which resource boundary is saturated, how long has it been saturated, and are unrelated workloads still meeting their latency objectives?
12. Test the failure boundary
A bulkhead should be tested with a deliberately slow dependency.
For example, configure a dependency bulkhead with a capacity of two, start two operations that block on a test-controlled channel, and then submit a third operation.
The third operation should receive ErrFull without invoking its callback.
A simplified test:
func TestBulkheadRejectsWhenFull(t *testing.T) {
b := bulkhead.New(2)
release := make(chan struct{})
started := make(chan struct{}, 2)
operation := func(context.Context) error {
started <- struct{}{}
<-release
return nil
}
var wg sync.WaitGroup
for range 2 {
wg.Add(1)
go func() {
defer wg.Done()
_ = b.Do(context.Background(), operation)
}()
}
// Wait until both permits are occupied.
<-started
<-started
err := b.Do(
context.Background(),
func(context.Context) error {
t.Fatal("operation should not have been admitted")
return nil
},
)
if !errors.Is(err, bulkhead.ErrFull) {
t.Fatalf("got %v, want %v", err, bulkhead.ErrFull)
}
close(release)
wg.Wait()
}
This test requires the usual testing, context, sync, and errors imports, and the bulkhead package import. It assumes the test is in a separate package where the bulkhead package can be referenced.
The test uses synchronization channels rather than arbitrary sleeps to establish that both permits are occupied.
Additional tests should cover:
- Cancellation before acquisition.
- Cancellation racing with acquisition.
- Permit release after errors and panics.
- Recovery after a dependency becomes healthy.
- Waiting callers and bounded queues, if supported.
- Isolation between two independent bulkheads.
- Maximum aggregate concurrency across replicas.
- Saturation of one dependency while unrelated operations remain responsive.
For a full integration test, make one fake dependency slow or unresponsive and verify that requests to other dependencies still meet their expected latency and success-rate objectives.
13. Common mistakes
Using one shared limiter for every dependency. This can recreate the very contention the bulkhead was intended to prevent.
Bounding concurrency but not waiting callers. A semaphore can limit active operations while allowing many goroutines to accumulate while waiting.
Assuming a local limit is a global quota. Per-replica capacity multiplies across replicas and deployments.
Holding permits during unnecessary work. Long retry backoffs, unrelated processing, or detached tasks can occupy capacity longer than intended.
Creating too many isolated pools. Excessive fragmentation can leave some pools idle while others reject work. Isolation should follow real workload and resource boundaries.
Ignoring shared infrastructure. Separate application semaphores do not automatically isolate database pools, HTTP transports, CPU, memory, or downstream quotas.
Measuring only errors. A bulkhead may be working correctly by rejecting excess work. You need rejection, saturation, latency, and dependency health metrics to interpret the behavior.
14. Production checklist
Before deploying a bulkhead, verify that:
- The protected resource and isolation boundary are explicit.
- Each materially different dependency or workload has an appropriate capacity budget.
- Both active concurrency and queued work are bounded where necessary.
- Context deadlines and cancellation are propagated.
- Rejections are distinguishable from timeouts and dependency failures.
- Permits are released on every exit path.
- Retry policies do not amplify saturation.
- Per-instance limits are compatible with global dependency quotas.
- Metrics expose saturation and rejection per bulkhead.
- Tests demonstrate that one slow dependency cannot starve unrelated workloads.
Final thoughts
Bulkheads do not make dependencies reliable. They make dependency failures more contained.
In Go, channels, semaphores, dedicated clients, bounded worker pools, and resource-specific limits can establish useful isolation boundaries. The hard part is not writing a semaphore. It is deciding what must be isolated, how much capacity each workload deserves, and what should happen when that capacity is exhausted.
A reliable system should not allow one slow dependency to determine the fate of every other operation.
The goal is not to eliminate every failure. It is to keep one failure from becoming everyone else's failure.
Top comments (0)