--
I'm Pires Aurélio , a Go Backend Engineer from Luanda, Angola. I build production systems in public.
For 2 years I built scrapers and REST APIs. I got tired of tutorials that only show "Hello World" with Gin. No jobs, no retries, no real problems.
So I built a real one: A Production-Ready Background Job Queue in Go.
My Public Work:
- Main Scraper Framework (Python AsyncIO): https://github.com/Aureliopires186/py-websites-scraper
- Go Production Examples: https://github.com/Aureliopires186/go-serpapi-examples
- Blog - REST API with Go (Gin): https://blogspotangola.blogspot.com/2026/09/how-to-create-simple-rest-api-with.html
- LinkedIn / Portfolio: https://www.linkedin.com/in/pires-aur%C3%A9lio-511330385
What We Are Building
A REST API where you POST a task, it goes to Postgres, and Go workers process it in background with retries, rate limiting and graceful shutdown. Like Sidekiq / Celery, but in pure Go.
Stack: Go + Gin + Postgres + Goroutines + Docker
Features:
- REST API with Clean Architecture
- Postgres as queue (no Redis needed)
- Worker Pool with 5 Goroutines
- Graceful Shutdown
- Retry with exponential backoff
- Architecture (The Key for Recruiters)
Client -> POST /jobs -> API (Gin) -> Postgres (jobs table) -> Worker Pool -> Process -> Update Status
Why Postgres and not Redis? For portfolio and for cost. One less service. And it proves you understand ACID, transactions and FOR UPDATE SKIP LOCKED.
- Database Schema - The Foundation
sql
CREATE TABLE jobs (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
payload JSONB NOT NULL,
status VARCHAR(20) DEFAULT 'pending',
attempts INT DEFAULT 0,
max_attempts INT DEFAULT 3,
created_at TIMESTAMPTZ DEFAULT NOW(),
next_retry_at TIMESTAMPTZ DEFAULT NOW()
);
CREATE INDEX idx_jobs_status_next_retry ON jobs(status, next_retry_at);
This index `idx_jobs_status_next_retry` is critical. It makes the worker query O(log n) even with 1 million jobs.
3. The Worker - The Heart of the System
This is what 90% of tutorials don't teach. This is what makes you Senior.
package worker
import (
"context"
"log"
"math"
"time"
)
func (w *Worker) Start(ctx context.Context) {
for i := 0; i < w.concurrency; i++ {
go func(workerID int) {
log.Printf("Worker %d started", workerID)
for {
select {
case <-ctx.Done():
log.Printf("Worker %d shutting down gracefully", workerID)
return
default:
job, err := w.repo.FetchPendingJob(ctx) // Uses FOR UPDATE SKIP LOCKED
if err!= nil {
time.Sleep(1 * time.Second)
continue
}
if job == nil {
time.Sleep(500 * time.Millisecond)
continue
}
w.processWithRetry(ctx, job)
}
}
}(i)
}
}
func (w *Worker) processWithRetry(ctx context.Context, job *Job) {
err := w.processor.Process(job)
if err!= nil {
job.Attempts++
if job.Attempts >= job.MaxAttempts {
w.repo.MarkFailed(ctx, job.ID)
log.Printf("Job %s failed permanently after %d attempts", job.ID, job.Attempts)
} else {
backoff := time.Duration(math.Pow(2, float64(job.Attempts))) * time.Second
w.repo.MarkForRetry(ctx, job.ID, time.Now().Add(backoff))
log.Printf("Job %s failed, retrying in %v", job.ID, backoff)
}
return
}
w.repo.MarkDone(ctx, job.ID)
}
*What recruiters see here:*
1. `FOR UPDATE SKIP LOCKED` - You know how to prevent double-processing
2. `context.Context` - You know Graceful Shutdown
3. Exponential Backoff - You think about production resilience
4. The Repository Query (The Secret Sauce)
func (r *JobRepo) FetchPendingJob(ctx context.Context) (*Job, error) {
query := `
SELECT id, payload, attempts, max_attempts
FROM jobs
WHERE status = 'pending' AND next_retry_at <= NOW()
ORDER BY created_at ASC
LIMIT 1
FOR UPDATE SKIP LOCKED`
// This SKIP LOCKED is GOLD. Two workers will never get the same job.
//...
}
5. REST API with Gin - Clean Handler
func (h *JobHandler) CreateJob(c *gin.Context) {
var req CreateJobRequest
if err := c.ShouldBindJSON(&req); err!= nil {
c.JSON(400, gin.H{"error": err.Error()})
return
}
job, err := h.service.Enqueue(c.Request.Context(), req.Payload)
if err!= nil {
c.JSON(500, gin.H{"error": "failed to enqueue job"})
return
}
c.JSON(201, job)
}
func (h *JobHandler) ListJobs(c *gin.Context) {
jobs, _ := h.service.List(c.Request.Context())
c.JSON(200, jobs)
}
No business logic in handler. That's Clean Architecture. Handler -> Service -> Repo.
6. Docker Compose - One Command Run
version: '3.8'
services:
api:
build:.
ports: ["8080:8080"]
environment:
- DATABASE_URL=postgres://postgres:postgres@db:5432/jobs?sslmode=disable
depends_on: [db]
db:
image: postgres:15
environment:
- POSTGRES_DB=jobs
- POSTGRES_USER=postgres
- POSTGRES_PASSWORD=postgres
ports: ["5432:5432"]
`docker-compose up` and the whole system is up.
7. What I Learned Building This in Luanda
1. *Concurrency is not parallelism:* Goroutines are cheap (2KB), but Postgres connections are not. I use a pgx pool with max 10 connections.
2. *Observability from day 1:* I added JobID in every log. In a real company, this would be Prometheus + Grafana.
3. *Failure is the default:* 80% of the code is handling retries, shutdown, and errors. 20% is the happy path.
Conclusion
Any junior can build a CRUD. A mid-level builds a system that doesn't break when 10,000 jobs arrive at once.
This project proves:
- You understand Go concurrency
- You understand databases beyond SELECT *
- You think about failure, not just success
Full code of this queue is coming to my GitHub next week. Follow for Part 2: Adding Rate Limiter, Metrics and Dead Letter Queue.
Built in public from Luanda, Angola 🇦🇴
If you are a recruiter looking for a Go backend engineer who ships production-ready systems, let's talk.
---
Examples using public technologies. All code is my own. Not affiliated with any provider.
Author : pires Aurélio
E-mail: aureliopires186@gmail.com
https://www.linkedin.com/in/pires-aur%C3%A9lio-511330385
https://github.com/Aureliopires186
https://coderlegion.com/user/Pires+Aurlio
https://blogspotangola.blogspot.com/2026/09/building-production-ready-google-jobs.html
Top comments (0)