Short answer: write the business change and its background-job intent in one PostgreSQL transaction, then let a separate Go worker claim work with a lease; treat delivery as at-least-once and make the effect idempotent with a durable key.
That choice starts with a constraint, not a queue brand. When an Express request returns 202, it has recorded a promise. A payment or ledger operation that runs twice is a reconciliation break, while one that vanishes is an audit failure. The API therefore needs a durable record of intent before it acknowledges the caller.
The queue is evidence.
Why enqueueing and publishing are different failure boundaries
Publishing directly from the request handler creates a dual write. The database commit can succeed while the publish call fails, or a message can be delivered before the database transaction rolls back. Both outcomes leave the ledger and the work stream disagreeing. A job row inserted in the same transaction as the business row removes that gap: either both records commit, or neither exists.
The idempotency key belongs to that durable intent. Store a request fingerprint beside it, and reject a reused key whose body differs. A unique constraint makes the decision atomic instead of relying on a race-prone read followed by an insert.
const enqueueSQL = `
INSERT INTO jobs (kind, idempotency_key, fingerprint, payload)
VALUES ($1, $2, $3, $4)
ON CONFLICT (kind, idempotency_key) DO NOTHING
RETURNING id`
func enqueue(ctx context.Context, tx *sql.Tx, kind, key, fingerprint string, payload []byte) (int64, error) {
var id int64
err := tx.QueryRowContext(ctx, enqueueSQL, kind, key, fingerprint, payload).Scan(&id)
if errors.Is(err, sql.ErrNoRows) {
return existingJob(ctx, tx, kind, key, fingerprint)
}
return id, err
}
The HTTP layer can return the existing job identifier for an identical retry. It should return a client error for a conflicting payload, because silently changing the meaning of a key makes later reconciliation impossible. In practice, that means the response path has three materially different outcomes: a new insert returns 202 with a fresh job id; a matching replay returns the same id without another side effect; and a mismatched replay returns 422 with no state change. Keeping those branches explicit matters during incident review, because a generic success response erases whether the API accepted new work or merely acknowledged a retry.
How should a Node.js Express API, Postgres queue, and worker handle retries?
The consumer must assume it can die after the side effect and before the completion update. Claim work with FOR UPDATE SKIP LOCKED, set claimed_until, and allow an expired lease to be claimed again. This is the same shape as a broker visibility timeout: a message is hidden for a window, then redelivered if the consumer does not finish. The window must exceed the slowest legitimate operation, with enough margin for database and network delay.
const claimSQL = `
UPDATE jobs
SET state = 'running', attempts = attempts + 1,
claimed_until = now() + interval '90 seconds'
WHERE id = (
SELECT id FROM jobs
WHERE (state = 'queued' AND run_after <= now())
OR (state = 'running' AND claimed_until < now())
ORDER BY run_after
LIMIT 1
FOR UPDATE SKIP LOCKED
)
RETURNING id, idempotency_key, payload, attempts`
func (w *Worker) runOnce(ctx context.Context) error {
job, err := w.claim(ctx)
if err != nil { return err }
receipt, err := w.effect.Send(ctx, job.Payload, job.IdempotencyKey)
if err != nil { return w.deferJob(ctx, job, backoff(job.Attempts)) }
_, err = w.db.ExecContext(ctx,
`UPDATE jobs SET state = 'done', receipt = $2, finished_at = now()
WHERE id = $1 AND state = 'running'`, job.ID, receipt)
return err
}
The downstream effect must receive the same key, and completion must record a provider receipt, not merely a local done flag. During a rate limit, defer the job; a 429 is a scheduling signal. A worker that treats an empty receipt as success can make queue depth look healthy while the ledger remains unsettled. I am not sure every retry library distinguishes that case, so the integration test should.
What does a durable queue need to prove during reconciliation?
Keep state transitions and receipts as audit evidence. PCI DSS requires audit-log history to be retained for at least twelve months, with the most recent three months immediately available for analysis; retention policy is therefore part of the queue design, not a cleanup afterthought. Record the request fingerprint, attempt count, lease expiry, and final receipt, and make a reconciliation query compare business intents with external receipts.
Priority is a policy, not a correctness mechanism. Broker priority only reorders messages that are waiting; if consumers keep up, it may have no visible effect. In a SQL claim query, ordering every row by priority can starve low-priority work, so reserve capacity or use aging when fairness matters.
Choosing the handoff and rolling it out are part of the same operational decision.
| Handoff | Durable record | Redelivery | Poor fit when |
|---|---|---|---|
| PostgreSQL job table | Same transaction as business data | Lease expiry | Sustained throughput outgrows one primary or many consumer groups need fan-out |
| Transactional outbox plus broker | Database intent, broker delivery | Visibility timeout | No team owns the relay and its monitoring |
| Direct publish from handler | Broker only | Broker policy | Every intent must reconcile with a ledger |
The catch is operational coupling: a queue table shares connection pools, locks, vacuum work, and failure domains with the API. Stick with a dedicated broker when the stream is very large, mostly stateless, and does not need row-level reconciliation. Keep the table when transactional evidence matters more than peak throughput.
Roll out in stages. Add the table and transactional enqueue path, run consumers in a shadow mode that records intended effects, then enable the external call. Kill a worker after the effect and before the update; the second attempt must produce the same receipt. Monitor oldest queued-job age, redeliveries per hour, and intents older than a day without a receipt. Those signals expose scheduling drift before a customer does.
Top comments (0)