How to Build a Job Queue in Go Without External Dependencies

Most applications need background job processing at some point — sending emails, processing uploads, generating reports. The instinctive answer is to reach for Redis plus a worker library. But for many services, adding that infrastructure is overkill, and Go's concurrency primitives are sufficient on their own.

Why skip the external queue?

External queues (Redis, RabbitMQ, SQS) are the right answer when you need persistence across restarts, distributed workers across multiple machines, or guaranteed delivery with acknowledgment. But if your jobs can tolerate in-process execution and you're running a single instance (or even a few), a channel-based job queue gives you:

• Zero additional infrastructure
• No network round-trips between enqueue and process
• Simple deployment — it's just your binary
• Full type safety

The catch: jobs in memory are lost if the process crashes. For many workloads — sending notifications, triggering webhooks, processing user requests — that tradeoff is acceptable.

The core abstraction

A job queue in Go needs three things: a job type, a channel to pass jobs through, and workers consuming from that channel.

package jobqueue

import (
    "context"
    "log/slog"
    "sync"
)

// Job is any function that can be executed asynchronously.
type Job func(ctx context.Context) error

// Queue manages a pool of workers processing jobs from a channel.
type Queue struct {
    jobs    chan Job
    workers int
    wg      sync.WaitGroup
}

// New creates a Queue with the given worker count and buffer size.
func New(workers, bufferSize int) *Queue {
    return &Queue{
        jobs:    make(chan Job, bufferSize),
        workers: workers,
    }
}

// Start launches the worker goroutines.
func (q *Queue) Start(ctx context.Context) {
    for i := 0; i < q.workers; i++ {
        q.wg.Add(1)
        go q.worker(ctx)
    }
}

func (q *Queue) worker(ctx context.Context) {
    defer q.wg.Done()
    for {
        select {
        case job, ok := <-q.jobs:
            if !ok {
                return
            }
            if err := job(ctx); err != nil {
                slog.Error("job failed", "error", err)
            }
        case <-ctx.Done():
            return
        }
    }
}

// Enqueue adds a job to the queue. Returns false if the queue is full.
func (q *Queue) Enqueue(job Job) bool {
    select {
    case q.jobs <- job:
        return true
    default:
        return false
    }
}

// Stop closes the job channel and waits for all workers to finish.
func (q *Queue) Stop() {
    close(q.jobs)
    q.wg.Wait()
}

A few design decisions worth explaining:

• Job is func(ctx context.Context) error rather than an interface. This is simpler — you pass closures directly without defining structs for each job type.
• Enqueue returns a bool instead of blocking. If the queue is full, the caller decides what to do (log, return HTTP 429, drop) rather than blocking indefinitely.
• The ctx.Done() case in the worker handles context cancellation separately from channel close — this matters if you cancel the context before draining the queue.

Wiring it into an HTTP server

Here's a minimal server that uses this queue for background email sending, including a correct graceful shutdown:

package main

import (
    "context"
    "fmt"
    "log"
    "log/slog"
    "net/http"
    "os"
    "os/signal"
    "syscall"
    "time"

    "yourmodule/jobqueue"
)

func main() {
    ctx, cancel := context.WithCancel(context.Background())
    defer cancel()

    q := jobqueue.New(5, 100) // 5 workers, buffer of 100 jobs
    q.Start(ctx)

    mux := http.NewServeMux()
    mux.HandleFunc("/register", func(w http.ResponseWriter, r *http.Request) {
        email := r.FormValue("email")
        if email == "" {
            http.Error(w, "missing email", http.StatusBadRequest)
            return
        }

        enqueued := q.Enqueue(func(ctx context.Context) error {
            slog.Info("sending welcome email", "to", email)
            time.Sleep(50 * time.Millisecond)
            return nil
        })

        if !enqueued {
            http.Error(w, "server busy", http.StatusTooManyRequests)
            return
        }
        fmt.Fprintln(w, "registered")
    })

    srv := &http.Server{Addr: ":8080", Handler: mux}

    go func() {
        if err := srv.ListenAndServe(); err != http.ErrServerClosed {
            log.Fatal(err)
        }
    }()

    sig := make(chan os.Signal, 1)
    signal.Notify(sig, syscall.SIGTERM, syscall.SIGINT)
    <-sig

    shutdownCtx, shutdownCancel := context.WithTimeout(context.Background(), 30*time.Second)
    defer shutdownCancel()

    if err := srv.Shutdown(shutdownCtx); err != nil {
        log.Printf("server shutdown error: %v", err)
    }

    q.Stop() // wait for in-flight jobs
    slog.Info("shutdown complete")
}

The shutdown sequence matters: stop accepting new HTTP requests first, then drain the queue. If you reverse the order, new jobs can arrive after the channel is closed — that panics on a send to a closed channel.

Adding retry logic

Production jobs fail. The minimal version above logs the error and moves on. If you need retries, wrap the job at the call site rather than baking retry logic into the worker:

func withRetry(attempts int, delay time.Duration, job jobqueue.Job) jobqueue.Job {
    return func(ctx context.Context) error {
        var err error
        for i := 0; i < attempts; i++ {
            if err = job(ctx); err == nil {
                return nil
            }
            if i < attempts-1 {
                select {
                case <-time.After(delay * time.Duration(i+1)):
                case <-ctx.Done():
                    return ctx.Err()
                }
            }
        }
        return fmt.Errorf("job failed after %d attempts: %w", attempts, err)
    }
}

// Usage:
q.Enqueue(withRetry(3, 500*time.Millisecond, func(ctx context.Context) error {
    return sendEmail(ctx, "[e-mail]")
}))

The time.Duration(i+1) multiplier gives you linear backoff: 500ms, 1s, 1.5s. The ctx.Done() select inside the sleep ensures the job respects cancellation instead of waiting out the full delay on shutdown.

When this design breaks

This approach has real limits. Knowing them in advance saves you from a bad surprise in production.

Process restart loses all queued jobs. If you deploy mid-queue or the process crashes, everything in the channel is gone. If that's unacceptable, you need a persistent queue backed by a database or message broker.

No visibility into the queue. You can add atomic.Int64 counters for enqueued, completed, and failed counts and expose them via a /metrics endpoint, but you won't get job-level introspection without more code.

No priority lanes. A single channel is FIFO. If you need urgent jobs to preempt slow ones, you need multiple queues at different priority levels with a dispatcher that drains high-priority first.

Memory is bounded by your buffer size. Monitor your queue depth in production — a consistently full queue means either your workers are too slow or your traffic exceeds what this pattern can handle, and you need to scale out or switch to a distributed solution.

For teams running job queues inside security tooling or alerting pipelines, make sure your deployment configuration is solid too — we publish free security hardening checklists covering application server and container configurations that are easy to miss.

The takeaway

A channel-based job queue in Go handles a wide range of async workloads with zero operational overhead. The implementation above is around 60 lines, shuts down correctly, and is straightforward to test with a few goroutines and a timeout. Add a retry wrapper when you need resilience. Reach for Redis or a message broker only when you actually need persistence, distributed workers, or job introspection — not as the default starting point.

Start with the simplest thing that could work. In Go, that's often a buffered channel and a handful of goroutines.

---

I run AYI NEDJIMI Consultants, a cybersecurity consulting firm. We publish free security hardening checklists — PDF and Excel.