Your service is running fine. Then a downstream dependency gets slow. Connections back up, goroutines pile up, your service's memory spikes, and suddenly you've got a full-service outage caused by a database that just had high latency for 30 seconds. This is a cascading failure — and it's entirely preventable with two patterns that belong in every production codebase: circuit breakers and bulkheads.
What circuit breakers actually do
A circuit breaker wraps calls to external dependencies. It tracks failure rates and, when things go south, opens the circuit — stopping all calls to the failing service for a cool-down period. This protects both your service (no more goroutine pile-up) and the downstream service (no more hammering a system that's struggling to recover).
The three states:
- Closed: traffic flows normally
- Open: traffic is blocked (fast-fail)
- Half-open: a probe request is allowed to test recovery
Here's a minimal circuit breaker in Go:
package circuit
import (
"errors"
"sync"
"time"
)
type State int
const (
StateClosed State = iota
StateOpen
StateHalfOpen
)
var ErrCircuitOpen = errors.New("circuit breaker is open")
type Breaker struct {
mu sync.Mutex
state State
failureCount int
threshold int
lastFailure time.Time
cooldown time.Duration
}
func New(threshold int, cooldown time.Duration) *Breaker {
return &Breaker{threshold: threshold, cooldown: cooldown}
}
func (b *Breaker) Call(fn func() error) error {
b.mu.Lock()
switch b.state {
case StateOpen:
if time.Since(b.lastFailure) > b.cooldown {
b.state = StateHalfOpen
} else {
b.mu.Unlock()
return ErrCircuitOpen
}
}
b.mu.Unlock()
err := fn()
b.mu.Lock()
defer b.mu.Unlock()
if err != nil {
b.failureCount++
b.lastFailure = time.Now()
if b.failureCount >= b.threshold {
b.state = StateOpen
}
return err
}
b.failureCount = 0
b.state = StateClosed
return nil
}
Usage:
breaker := circuit.New(5, 10*time.Second)
err := breaker.Call(func() error {
return callPaymentService()
})
if errors.Is(err, circuit.ErrCircuitOpen) {
return cachedPrice, nil
}
The threshold and cooldown values matter. For a payment service you might use 3 failures and a 30-second cooldown; for an internal cache 10 failures and 5 seconds. Tune these to your SLOs.
Bulkheads: isolating failure domains
The bulkhead pattern comes from ship design. A hull breach in one compartment doesn't sink the ship because compartments are sealed off. Applied to software: you give different clients or use cases their own resource pool.
Without bulkheads, a slow report-generation endpoint can consume all your worker goroutines and starve time-sensitive APIs. The fix is semaphore-based isolation:
package bulkhead
import (
"context"
"errors"
)
var ErrBulkheadFull = errors.New("bulkhead capacity exceeded")
type Bulkhead struct {
sem chan struct{}
}
func New(capacity int) *Bulkhead {
return &Bulkhead{sem: make(chan struct{}, capacity)}
}
func (bh *Bulkhead) Acquire(ctx context.Context) error {
select {
case bh.sem <- struct{}{}:
return nil
case <-ctx.Done():
return ctx.Err()
default:
return ErrBulkheadFull
}
}
func (bh *Bulkhead) Release() {
<-bh.sem
}
Wire this into your HTTP handlers per route group:
reportsBulkhead := bulkhead.New(10) // at most 10 concurrent report requests
apiBulkhead := bulkhead.New(200) // 200 for time-sensitive API calls
func reportsHandler(w http.ResponseWriter, r *http.Request) {
ctx, cancel := context.WithTimeout(r.Context(), 5*time.Second)
defer cancel()
if err := reportsBulkhead.Acquire(ctx); err != nil {
http.Error(w, "Service busy", http.StatusServiceUnavailable)
return
}
defer reportsBulkhead.Release()
generateReport(w, r)
}
Now a flood of report requests cannot degrade your API endpoints. When reportsBulkhead fills up, callers get a 503 immediately instead of waiting and consuming resources.
Combining both patterns
Circuit breakers and bulkheads solve different problems and belong together. The bulkhead limits concurrency within your service; the circuit breaker limits your exposure to upstream failures.
A realistic production wrapper:
type ResilientClient struct {
breaker *circuit.Breaker
bulkhead *bulkhead.Bulkhead
}
func (c *ResilientClient) Call(ctx context.Context, fn func() error) error {
if err := c.bulkhead.Acquire(ctx); err != nil {
return fmt.Errorf("bulkhead rejected: %w", err)
}
defer c.bulkhead.Release()
return c.breaker.Call(fn)
}
This works for any external call: database queries, third-party APIs, internal microservices. If you need shared circuit state across replicas, back it with Redis — but start in-process and add complexity only when you hit actual problems.
Observability you actually need
Both patterns are silent by default. Add metrics:
func (b *Breaker) recordState() {
state := "closed"
if b.state == StateOpen {
state = "open"
circuitOpenTotal.WithLabelValues(b.name).Inc()
}
circuitState.WithLabelValues(b.name).Set(float64(b.state))
_ = state
}
The key signals to track:
-
circuit_open_total— how often the breaker is tripping -
bulkhead_rejected_total— how often callers hit capacity -
bulkhead_in_flight— current concurrency level
If circuit_open_total increases steadily, your downstream dependency has a reliability problem. If bulkhead_rejected_total spikes, you either need more capacity or a client is misbehaving. These signals surface incidents before your on-call gets paged.
For teams running production services, this kind of resilience hardening fits alongside security hardening. We publish free security and infrastructure checklists that cover both areas — worth a look if you're building production systems.
The takeaway
Circuit breakers and bulkheads are not optional extras — they are the difference between a contained incident and a full outage. Circuit breakers protect you from upstream degradation; bulkheads protect your upstream services from your own traffic. Together they give you controlled failure modes instead of unpredictable cascades.
Start simple: a 50-line semaphore bulkhead and a 70-line circuit breaker cover 90% of real-world cases. Add Redis-backed shared state only when your replica count makes in-process state unreliable. The hardest part is not the code — it's calibrating the thresholds to match your SLOs and actually acting on the metrics you expose.
I run AYI NEDJIMI Consultants, a cybersecurity consulting firm. We publish free security hardening checklists — PDF and Excel.
Top comments (0)