Durable and HA task queues and DAG workflows in Go

Eugene Berger — Tue, 07 Jul 2026 22:34:48 +0000

ebind is an MIT-licensed Go library that gives you a persistent task queue and a durable workflow engine on a single dependency - NATS JetStream - which it can embed right inside your process.

Every background-job story in Go seems to start the same way: pick a queue library, then stand up the infrastructure it drags along. Redis for Asynq or Machinery. Postgres for River. A whole cluster if you want Temporal-style workflows. The job code is ten lines; the deployment diagram is not.

ebind is my attempt at a different trade: one dependency - NATS JetStream - carries the queue, the workflow state, and the event bus. And because NATS is a Go library as well as a server, ebind can boot it inside your process: a single embedded node for dev and small deployments, or a full 3-node JetStream cluster in-process for HA testing. No Redis. No Postgres. No sidecar.

Enqueue a function, not a string

Most queue libraries make you register a string name and hand-roll payload structs. ebind's API is function-first - you pass the function itself:

// Any function shaped like (context.Context, ...args) (T, error) or (context.Context, ...args) error.
func SendEmail(ctx context.Context, to, subject, body string) (string, error) {
    // ... actually send ...
    return "msg-id-42", nil
}

reg := task.NewRegistry()
task.MustRegister(reg, SendEmail)

// Producer and worker can be different binaries - this is a durable queue, not an RPC.
fut, _ := client.Enqueue(c, SendEmail, "alice@example.com", "hello", "world")
msgID, err := client.Await[string](ctx, fut) // typed result, not interface{}

Under the hood a registry maps canonical names (derived via runtime.FuncForPC) to reflect.Values, and argument types are validated client-side, before publish - pass an int where a string goes and Enqueue fails immediately, not on some worker at 3 a.m.

Why reflection instead of generics? Go has no variadic generics. An API where Enqueue(c, MyFunc, a, b, c) type-checks a, b, c against MyFunc's signature simply cannot be expressed with generics today. The reflect.Value.Call cost is a few hundred nanoseconds - noise next to any real handler doing I/O.

Workflows that survive restarts

The workflow layer adds durable DAGs on top: steps, dependencies, retries - with all state in a NATS KV bucket, so a workflow outlives the process that submitted it.

dag := workflow.New()
a := dag.Step("fetch",    FetchUser, userID)
b := dag.StepOpts("enrich", Enrich, []workflow.StepOption{workflow.Optional()}, userID)
c := dag.Step("combine",  Combine, a.Ref(), b.RefOrDefault(Enriched{}))

_ = dag.Submit(ctx, wf)
profile, err := workflow.Await[Profile](ctx, wf, dag.ID(), c)

a.Ref() wires data flow and dependency: combine receives fetch's result, and cascade-skips if fetch fails. RefOrDefault substitutes a fallback when an optional step fails. Handlers can add steps dynamically mid-run (workflow.FromContext(ctx).Step(...)), steps can be pinned to specific workers ("this step needs the GPU box", "run this on the same worker that ran the download"), and whole DAGs can be paused, resumed, or canceled durably.

Because results live in KV, waiting is decoupled from submitting: persist two IDs, and any process on the cluster can pick up the wait later with workflow.AwaitByID[Profile](ctx, wf, dagID, stepID) - even for results written before it started watching.

Breakpoints: a debugger for your workflows

The feature I haven't seen elsewhere: step breakpoints with debugger semantics.

upload := dag.StepOpts("upload", Upload,
    []workflow.StepOption{workflow.BreakBefore("BeforeUpload")}, parse.Ref())

// Breakpoint labels are part of the DAG's structure, but ARMING them is a runtime decision:
_ = dag.Submit(ctx, wf, workflow.WithActiveBreakpoints("BeforeUpload"))

The DAG runs until upload is about to be dispatched, then stops that line - parallel branches keep running. You inspect intermediate results (they're in KV), then continue:

n, _ := workflow.ResumeBreakpoint(ctx, wf, dag.ID(), "BeforeUpload")

ResumeBreakpoint is continue, not disable: the label stays armed, so a later step carrying it - including dynamically added ones - stops again. Blocked state lives in KV, so it survives restarts, and any process (or the bundled ebctl CLI) can list and resume breakpoints. BreakAfter is the complement: let a step finish and persist its result, but hold its dependents.

If you've ever added a "pause step" to a pipeline just so you could look at an intermediate file before the expensive part runs, this is that - as a first-class primitive.

The scheduler needs no leader

The part I'm most pleased with architecturally: every worker runs the scheduler, and correctness doesn't depend on electing a leader.

All state mutations are compare-and-swap on KV revisions (Update(key, val, expectedRevision)). Step enqueues are deduplicated by JetStream message ID (<dag_id>:<step_id>). DAG events flow through a work-queue stream with explicit acks. Stack those three and racing schedulers become harmless: two instances that both decide "step X is ready" produce one enqueue; two writers racing on a step record - one wins, the other re-reads and retries. There is a leader elector, but it's defense-in-depth for failover windows (plus a sweep that rescues stranded steps when leadership changes hands), not a correctness requirement.

The claim isn't just asserted - the repo carries a chaos end-to-end suite that runs every supported operation on a 3-node in-process cluster with replicas=3 while killing and restarting nodes mid-workflow.

Operations included

Failed steps durably record why they failed (error_kind + message on the step record itself, full error in a 7-day DLQ stream), and a cobra-based operator CLI ships in the box:

ebctl dag ls --label billing        # query workflow history by immutable labels
ebctl dag tree <dag-id>             # render the DAG with per-step status
ebctl dag step get <dag-id> <step>  # why did this step fail?
ebctl dlq ls                        # dead-lettered tasks
ebctl dag watch                     # live event feed, incl. breakpoint hits

What it isn't

ebind is young (v0.6.0) and honest about its scope. It's at-least-once delivery with dedupe - handlers should be idempotent, as in every queue system that tells you the truth. It is not Temporal: there are no signals, sagas, or compensation yet (they're on the roadmap). And it's unapologetically NATS-native - if you can't run NATS, it's not for you; if you already do, you have everything ebind needs.

Try it

go get github.com/f1bonacc1/ebind

The repo has 16 runnable examples, each self-contained with its own embedded NATS - from a basic queue to breakpoints and placement. I'd genuinely love feedback, especially on the reflection-based API and the no-leader scheduler design: github.com/F1bonacc1/ebind.

DEV Community: Eugene Berger