DEV Community: Clint Mathews

A Zero-Allocation Worker Pool for 10kHz Sensor Ingestion in Go

Clint Mathews — Fri, 14 Aug 2026 12:00:21 +0000

Part of the build log for PhotonicOps, a fully offline, air-gapped telemetry engine for silicon photonic biosensors. No cloud APIs, no internet dependency, Apple Silicon only. This post covers the worker pool sitting between the gRPC ingestion stream and the DSP handoff.

Short version: 10,000 frames/sec, one channel send on the hot path, 10 fixed workers, zero per-frame allocations via sync.Pool, backpressure instead of drops.

The Constraints

A photonic biosensor streams resonance wavelength shift (Δλ, picometers) at 10,000 samples/sec. That's one OpticalFrame roughly every 100μs over a gRPC client-streaming call. Two rules shaped the design:

Nothing stalls the receive loop. If the goroutine reading off the stream also does the work, a slow downstream consumer pushes backpressure straight into the TCP connection and the sensor.
No GC pauses on the hot path. At 10kHz, a few milliseconds of stop-the-world GC can silently drop a sample, and a dropped sample can hide the exact anomaly (a microbubble, a cell clog) the system exists to catch.

The Pool

type FramePool struct {
    jobQueue  chan *pb.OpticalFrame
    wg        sync.WaitGroup
    forwarder *dsp.Forwarder
}

var frameSyncPool = sync.Pool{
    New: func() any { return make([]byte, 0, 1024) },
}

func NewFramePool(workers, queueSize int, forwarder *dsp.Forwarder) *FramePool {
    p := &FramePool{
        jobQueue:  make(chan *pb.OpticalFrame, queueSize),
        forwarder: forwarder,
    }
    for i := 0; i < workers; i++ {
        p.wg.Add(1)
        go p.worker(i)
    }
    return p
}

Wired up in main.go as NewFramePool(10, 50000, forwarder): 10 workers, a 50,000-slot channel. At full throughput that's 5 seconds of buffering headroom before anything blocks.

The gRPC handler's entire job:

func (p *FramePool) Enqueue(frame *pb.OpticalFrame) {
    p.jobQueue <- frame
}

One channel send. No allocation, no branching. If the channel fills, Enqueue blocks, which becomes backpressure through gRPC flow control back to the sensor. No silent drops. Non-blocking load-shedding is on the roadmap for genuine overload but isn't built yet.

Zero-Allocation Workers

func (p *FramePool) worker(id int) {
    defer p.wg.Done()
    for frame := range p.jobQueue {
        buf := frameSyncPool.Get().([]byte)

        if err := p.forwarder.Push(frame); err != nil {
            log.Printf("worker %d: dsp forwarder push error: %v", id, err)
        }

        buf = buf[:0]
        frameSyncPool.Put(buf)
    }
}

Reusable []byte scratch space, recycled by capacity instead of reallocated per frame. At 10,000 frames/sec, allocating small and letting GC reclaim is a GC storm in slow motion. sync.Pool turns it into pointer recycling.

Where the Frame Actually Goes

Each worker hands its frame to dsp.Forwarder.Push, which:

Shards frames by sensor_id into per-sensor accumulators
Buffers each sensor's stream into 100ms windows (1,000 frames each)
Flushes full windows as a FrameBatch over a Unix domain socket, via gRPC, to a local Python DSP process

No TCP, no HTTP, no cloud. The Go to Python DSP boundary stays entirely on-box.

Why Fixed Size, Not Goroutine-per-Frame

go process(frame) per frame would be simpler. Two reasons it doesn't work here:

Unbounded goroutines under load means unbounded memory. A downstream hiccup (Python DSP pausing) shouldn't turn into an OOM in an offline clinical deployment with no one watching a dashboard.
A fixed pool bounds lock contention. dsp.Forwarder.Push holds a mutex over the per-sensor accumulator map, so capping workers caps contention instead of letting it scale with load.

Worst case is "the send blocks", not "the process dies."

What's Next

Phase 5 (Ingestion Hardening) on the roadmap: per-sensor ring buffers (today's is a single global buffer, not keyed by sensor), non-blocking load-shedding for Enqueue, and a /metrics endpoint so pool saturation is observable instead of inferred from pprof.

For anything that has to sustain a hard real-time ingestion rate, keep the hot path allocation-free and non-blocking, push everything else behind a bounded channel, and let a fixed pool of workers apply steady, predictable pressure downstream.

gRPC over a Unix socket, not HTTP: a real IPC tradeoff from a HIPAA-postured edge system

Clint Mathews — Thu, 13 Aug 2026 20:32:20 +0000

I'm building PhotonicOps, an offline, air-gapped telemetry ingestion and agentic hardware-triage engine for silicon photonic biosensors. That's the kind of microfluidic optical sensor used in clinical biomarker detection. It ingests resonance wavelength shift data (Δλ, picometers) at 10,000 samples/sec, cleans and analyzes it with DSP, and (in the next phase) will use a local LLM to decide on hardware remediation. Zero cloud API calls anywhere in the stack. That last constraint isn't a preference; the target environment is HIPAA-sensitive clinical infrastructure, so "call OpenAI" was never on the table.

Most technology-choice write-ups read like marketing copy for the winner. This one is about the option that lost, and why. That reasoning is what actually transfers to your next decision, not the choice itself.

The problem

The Go ingestion engine receives sensor frames over gRPC at 10kHz and needs to hand them off to a Python process for DSP: Kalman filtering, baseline subtraction, spike/anomaly detection. Both processes run on the same host. This is an edge deployment, one Apple Silicon box per clinical site, not a distributed cloud service. The question was narrow: what's the transport for that one internal hop?

The default-looking option: HTTP + JSON

A plain HTTP service on a loopback port is the obvious first instinct. It's curl-able, needs no code generation step, and is callable from literally any language without a client library. For a lot of internal services, that's the right call. It wasn't here, for three concrete reasons tied to the actual code, not general priors about gRPC being "better."

Reason 1: the streaming shape doesn't fit request/response

The Go side doesn't send one frame at a time. Forwarder.Push accumulates frames per sensor and flushes a batch once it hits 1000 frames (roughly a 100ms window):

// services/ingestion-go/internal/dsp/forwarder.go

func (f *Forwarder) Push(frame *pb.OpticalFrame) error {
    f.mu.Lock()
    acc, ok := f.shards[frame.SensorId]
    if !ok {
        acc = &sensorAccumulator{
            frames:      make([]*pb.OpticalFrame, 0, FramesPerBatch),
            windowstart: frame.Timestamp,
        }
        f.shards[frame.SensorId] = acc
    }
    acc.frames = append(acc.frames, frame)

    if len(acc.frames) < FramesPerBatch {
        f.mu.Unlock()
        return nil
    }

    batch := &pb.FrameBatch{
        Frames:           acc.frames,
        SensorId:         frame.SensorId,
        WindowStartNs:    acc.windowstart,
        WindowDurationMs: float64(time.Duration(frame.Timestamp - acc.windowstart).Milliseconds()),
    }
    acc.frames = make([]*pb.OpticalFrame, 0, FramesPerBatch)
    acc.windowstart = frame.Timestamp
    f.mu.Unlock()

    return f.flush(batch)
}

func (f *Forwarder) flush(batch *pb.FrameBatch) error {
    ack, err := f.client.StreamBatches(context.Background())
    if err != nil {
        return fmt.Errorf("dsp: open StreamBatches: %w", err)
    }
    if err := ack.Send(batch); err != nil {
        return fmt.Errorf("dsp: send batch sensor=%s: %w", batch.SensorId, err)
    }
    reply, err := ack.CloseAndRecv()
    if err != nil {
        return fmt.Errorf("dsp: recv DSPAck: %w", err)
    }
    if !reply.Accepted {
        log.Printf("dsp: batch rejected by Python: %s", reply.RejectionReason)
    }
    return nil
}

That Send / CloseAndRecv pair is a client-streaming gRPC call. gRPC gives it to you as a primitive over HTTP/2. To get the same shape over plain HTTP/1.1 REST, you'd reach for chunked transfer encoding or a polling model on the Python side. Either is more code to build and maintain, just to reconstruct something gRPC already does natively.

Reason 2: serialization cost inside a hard latency budget

The Python DSP pipeline has a documented budget: under 10ms per frame (NFR-1.2 in the project's requirements doc). Each FrameBatch carries 1000 frames that need to be decoded on the Python side before any Kalman filtering can start. Protobuf's binary encoding is materially cheaper to decode than JSON at this frequency, and unlike a low-traffic admin API, this isn't a place where "JSON is fine, we're not that latency-sensitive" holds. Paying a JSON parsing tax here works directly against the phase gate this whole design exists to unblock.

Reason 3: exposure surface, not just performance

This is the one that's easy to skip if you're only thinking about throughput. A Unix domain socket (/tmp/photonicops-dsp.sock) is scoped by filesystem permissions. There's no port to bind, nothing to firewall, nothing to accidentally leave open. An HTTP service, even bound to 127.0.0.1, is still a listening socket that any local process can reach. For most software that distinction doesn't matter much. It matters here because the entire system is designed around an offline, zero-cloud-API, HIPAA-postured constraint (every telemetry byte stays on the local network, full stop). A smaller exposure surface for an internal hop is a small but real contribution to that posture, not a rounding error.

What HTTP would have bought — and why it didn't win anyway

To be fair to the option that lost: HTTP would have meant no protoc codegen step, and debugging with curl instead of grpcurl. Those are real costs. They didn't outweigh the above, because this is a fixed, internal, single-consumer contract. The Go side and the Python side are the only two participants, ever, by design, and the codegen pipeline (make proto, make proto-python) was already built for the primary sensor→ingestion contract. Reusing it for this hop was close to free. Building a second, HTTP-flavored contract just to keep curl in the loop would have been optimizing for a debugging convenience at the cost of the actual constraints in play.

The limit of this decision, and where it breaks

This only holds because Go and Python are co-located on one host. A Unix domain socket is a filesystem path, not a network address, so there's no way to dial it across machines. PhotonicOps is designed as a single-host edge appliance (one box per clinical site, not a horizontally-scaled service), so that's the actual production topology, not a shortcut being taken now and paid for later. But it's worth stating explicitly: if the DSP pipeline were ever split onto a separate host, a shared GPU box, say, this transport stops working entirely, not just gets slower, and would need to become TCP-based gRPC with mTLS (reusing the same certificate pattern already planned for the sensor→ingestion hop). Writing that limit down as part of the decision, rather than leaving it implicit, is the point of doing this as an ADR instead of a comment buried in the code.

Where the project stands

3 of 8 planned phases are complete: the infra harness, the Go ingestion engine (10k frames/sec, zero-allocation, sync.Pool-backed), and this DSP pipeline (Kalman filtering, baseline subtraction, spike detection, and the IPC contract described above). Next is the agentic triage layer, a local LLM (via Ollama, no cloud calls) that turns flagged anomalies into structured, schema-validated remediation decisions, with an explicit fail-safe: nothing gets auto-executed on a low-confidence or unreachable response.

Repo: PhotonicOps