Building GoRelay: A Durable Task Queue for Go with Zero Infrastructure

Every Go developer has written this line:

go sendEmail(user)

It works perfectly... until it doesn't. Your server crashes, the task disappears forever, and your user never gets that welcome email.

I kept building the same solution: a task queue with retries, persistence, and monitoring. But every time, I had to set up Redis, configure workers, and write boilerplate code.

So I built GoRelay - a durable task queue that works out of the box.

What is GoRelay?

GoRelay turns any Go function into a crash-resistant, retryable background job.

The API is simple:

r.Register("email.send", SendEmail)
r.Enqueue("email.send", EmailPayload{To: "[e-mail]"})

What makes it different:

• Setup: go get and done (no Redis required)
• Default storage: SQLite (file-based, zero config)
• Dashboard: Built-in, no separate setup
• Retries: Automatic with exponential backoff
• Backends: SQLite, PostgreSQL, Redis - same code works everywhere

The Journey to v1.0

Building an open source library taught me more than I expected.

First, I started with the API first. I wrote example code showing how I WANTED it to work, then made it work that way.

Second, storage abstraction is critical. GoRelay supports SQLite, PostgreSQL, and Redis with the same interface. Start with SQLite on your laptop, deploy to PostgreSQL in production, scale to Redis for high throughput - no code changes needed.

Third, the dashboard changed everything. I almost skipped it. Big mistake. Being able to see tasks in real-time - their status, history, and errors - is what makes GoRelay trustworthy for production use.

How It Works Under the Hood

Lock-Free Ring Buffer

GoRelay uses a lock-free ring buffer for task passing between producers and consumers. This means no mutex contention and no goroutine blocking during normal operation.

Priority Queues

Tasks are processed in priority order: High, Normal, Low. This ensures critical payments never wait behind analytics jobs.

Automatic Retries

Failed tasks retry with exponential backoff. The delay increases with each attempt: 1s, 2s, 4s, 8s, up to 1 hour. Random jitter prevents thundering herd problems when services recover.

Multiple Storage Backends

Each backend is optimized for its strength:

• SQLite: Embedded, zero config, perfect for side projects and internal tools
• PostgreSQL: ACID compliant, supports multiple workers, ideal for production SaaS
• Redis: High throughput, great for existing Redis shops

Real-World Use Cases

GoRelay is being used for:

• Welcome emails after user signup
• PDF report generation
• SMS notifications
• Data processing pipelines
• Scheduled maintenance tasks

What I Learned Building Open Source

Documentation is code. I spent 40% of my time on README, examples, and godoc. Worth it. Users judge your library by its documentation first.

Tests build confidence. Writing tests caught bugs I didn't know I had. The race detector (go test -race) is your best friend.

Release early, release often. v1.0 doesn't need every feature. Ship something useful, then iterate.

What's Next for GoRelay

• v1.1: Webhooks and Outbox pattern for transactional consistency
• v1.2: Idempotency keys to prevent duplicate processing
• v2.0: Distributed tracing and Prometheus metrics

Try GoRelay Today

Install:

go get github.com/amitstephen-dev/gorelay@v1.0.0

Complete working example:

package main

import (
    "fmt"
    "github.com/amitstephen-dev/gorelay"
)

type EmailPayload struct {
    To   string
    Body string
}

func SendEmail(payload interface{}) error {
    p := payload.(*EmailPayload)
    fmt.Printf("Sending to: %s\n", p.To)
    return nil
}

func main() {
    r := gorelay.New()
    r.Register("email.send", SendEmail, &EmailPayload{})
    r.EnableDashboard(":8080")
    r.Start()

    r.Enqueue("email.send", &EmailPayload{
        To:   "[e-mail]",
        Body: "Welcome to GoRelay!",
    })

    select {}
}

Open http://localhost:8080 to see the dashboard with real-time task monitoring.

How Fast Is It?

Instead of giving you marketing numbers, I encourage you to benchmark GoRelay on your own hardware with your own workload patterns.

The architecture is designed for efficiency:

• Lock-free ring buffer for zero contention
• Zero-copy JSON for small payloads
• Batch writes to reduce database round trips

Actual throughput depends on your storage backend:

• SQLite: Great for development and moderate workloads
• PostgreSQL: Production-ready with strong consistency
• Redis: Highest throughput for high-volume workloads

Memory usage is typically under 50MB for most workloads.

Links

• GitHub: github.com/amitstephen-dev/gorelay
• Documentation: pkg.go.dev/github.com/amitstephen-dev/gorelay
• Report Issues: github.com/amitstephen-dev/gorelay/issues

Closing Thoughts

Building GoRelay taught me that simple tools can solve complex problems. You don't always need Kubernetes or microservices. Sometimes, a well-designed library with a SQLite backend is all you need.

The best part? You can start with zero infrastructure. No Redis. No Postgres. Just go get and go run.

If GoRelay saves you time or solves a problem, star the repo on GitHub - it helps other developers find it.

Happy coding!

Built with love for the Go community

Comments

[Amit Stephen]

hi