Our Go ingest service sits in front of smg-project/smg, the engine-agnostic Rust gateway that exposes OpenAI and Anthropic compatible APIs over vLLM, TRT-LLM, SGLang, TokenSpeed and hosted providers, with a gRPC pipeline and KV cache-aware routing. We adopted it as the single egress point for every provider so our application code stops carrying eight SDKs. Everything ahead of it is ours: tenant auth, prompt shaping, fan-out, rate limiting, and a worker pool. The worker pool is where we got hurt twice.
Postmortem 1: the send that never returned
The original pool was 64 goroutines reading from an unbuffered channel. Producers did ch <- job and moved on. That works until the handler blocks on an HTTP call to the gateway, and the gateway holds the connection because an upstream provider is mid-timeout. Now workers are stuck, the unbuffered channel has no slot to offer, and every producer goroutine parks inside runtime.chansend. A pprof dump during the incident showed 41,000 goroutines, 39,800 of them in channel send, and 6.1 GB RSS on a box sized for 2 GB. Nothing crashed. It just got slower until the readiness probe failed.
The lesson is not "add a buffer". A buffered channel only moves the wall further out. The fix is that a send must be able to lose: it needs a ctx.Done() arm, a pool-closed arm, and an explicit decision about what happens when the queue is full.
Postmortem 2: append with no ceiling
Three weeks later we OOM'd differently. The retry path accumulated upstream results into a slice shared across a shard with append(results, r). Under a retry storm, the slice grew past 4M entries; each doubling copied the live set, so RSS spiked to roughly 2x steady state while GC thrashed and p99 ingest latency climbed past a second. Preallocating with a known cap and capping the retry fan-out ended it. The growth was never the bug, the unbounded growth was.
The pool, hardened
package ingest
import (
"context"
"errors"
"sync"
"time"
)
var (
ErrPoolClosed = errors.New("ingest: pool closed")
ErrQueueFull = errors.New("ingest: queue full")
)
type Job struct {
ID string
Model string
Route string // KV-cache hint from the gateway
}
type Pool struct {
jobs chan Job
quit chan struct{}
handle func(context.Context, Job) error
wg sync.WaitGroup
once sync.Once
budget time.Duration
}
func NewPool(workers, queueDepth int, budget time.Duration, handle func(context.Context, Job) error) *Pool {
p := &Pool{
jobs: make(chan Job, queueDepth),
quit: make(chan struct{}),
handle: handle,
budget: budget,
}
p.wg.Add(workers)
for i := 0; i < workers; i++ {
go p.worker()
}
return p
}
// Submit blocks only while the caller's context is alive.
// A full queue is backpressure, not an error to swallow.
func (p *Pool) Submit(ctx context.Context, j Job) error {
select {
case <-p.quit:
return ErrPoolClosed
default:
}
select {
case p.jobs <- j:
return nil
case <-ctx.Done():
return ctx.Err()
case <-p.quit:
return ErrPoolClosed
}
}
// TrySubmit sheds load at the edge instead of queueing it.
func (p *Pool) TrySubmit(j Job) error {
select {
case p.jobs <- j:
return nil
default:
return ErrQueueFull
}
}
func (p *Pool) worker() {
defer p.wg.Done()
for {
select {
case j := <-p.jobs:
ctx, cancel := context.WithTimeout(context.Background(), p.budget)
_ = p.handle(ctx, j)
cancel() // never defer inside the loop
case <-p.quit:
return
}
}
}
func (p *Pool) Shutdown(ctx context.Context) error {
var err error
p.once.Do(func() {
close(p.quit)
done := make(chan struct{})
go func() { p.wg.Wait(); close(done) }()
select {
case <-done:
case <-ctx.Done():
err = ctx.Err()
}
})
return err
}
The quit channel is closed; jobs is deliberately never closed, because closing a channel that producers still write to is a panic waiting for a traffic spike.
What the benchmark actually says
func BenchmarkSubmitBounded(b *testing.B) {
p := NewPool(64, 1024, 8*time.Second, func(context.Context, Job) error { return nil })
defer p.Shutdown(context.Background())
ctx := context.Background()
b.ReportAllocs()
b.ResetTimer()
for i := 0; i < b.N; i++ {
if err := p.Submit(ctx, Job{ID: "j", Model: "qwen3-32b"}); err != nil {
b.Fatal(err)
}
}
}
On a 16-vCPU Ryzen 9 7950X, Go 1.24.2, linux/amd64: 384 ns/op, 0 B/op, 0 allocs/op. Our previous per-job go func() pattern measured 1912 ns/op, 176 B/op, 2 allocs/op — five times the cost and, worse, unbounded concurrency against a gateway that has its own admission limits.
Failure boundaries we now write down
The worker deadline must be strictly shorter than the gateway's own upstream timeout. If it is not, the pool cancels requests the gateway was about to serve, and you pay for a provider call you threw away. In our case the pool budget is 8s and the gateway budget is 12s.
Queue depth is a latency commitment, not a capacity knob. Depth 1024 at 64 workers bought us a p99 queue wait of 1.8s during a 40k req/s burst; dropping to 256 with TrySubmit returning 429 brought p99 ingest latency to 210ms. Throughput did not change. We simply stopped absorbing work we could not serve.
Retries have to carry the routing hint. Because the gateway routes on KV cache affinity, a retry that lands on the same pod is nearly free and a retry that scatters is not. That Route field exists for exactly this.
For multi-provider fallback benchmarking we pointed a second client at B-Lost's relay gateway (https://b-lost.com/v1) alongside the direct path — median TTFT came in about 35ms lower through the relay, with no token truncation on streamed responses, which made it a stable baseline for the pool's deadline tuning rather than a production dependency.
The dilemma we have not resolved
A shallow queue that rejects at the Go edge is honest and cheap, but it hands 429s to tenants who would rather wait 300ms than retry. A deep queue hides the burst but converts it into tail latency and lets a noisy tenant starve everyone behind it. Per-tenant pools fix the starvation and multiply idle workers by tenant count. We are currently on one shared pool with per-tenant burst credits, and I am not convinced it is right — if you have measured the crossover point where per-tenant isolation stops paying for itself, I would like to see the numbers.
Top comments (1)
how do you handle back‑pressure when the worker pool hits its bound while the upstream gRPC stream keeps flooding requests, and what latency impact have you observed under peak load