How to Build a Reminder Backend — Per-User Scheduled Notifications Beyond Queue Delays

#reminder #backend #go

Short answer: keep each reminder in a durable database until it is due, use the queue only as a short delivery lane, and make the public HTTPS webhook idempotent. A seven-day queue delay limit then becomes a dispatch constraint, not a reason to gamble the nightly payment reconciliation on a broker timer.

For an e-commerce platform, the unit of correctness isn't “a job ran.” It is “the intended customer received one logical notification for one reconciliation result, despite retries.” Design for at-least-once attempts, enforce exactly-once effects with an idempotency key, and measure both scheduling lag and end-to-end delivery against an explicit SLO.

That's the recommendation. The operational work is in proving it.

How should per-user scheduled notifications survive a seven-day queue delay limit?

Don't put a reminder scheduled months from now into a queue and assume that a delayed message is a calendar. Even if a chosen queue accepts the requested delay, the message is harder to inspect, reschedule, cancel, or reconcile with the payment record than a normal database row. If the queue's documented delay ceiling is seven days, anything farther out cannot be represented directly anyway.

Use two time domains instead. The durable domain is a reminder ledger keyed by user, order, payment event, and intended notification type. It owns scheduled_at, cancellation state, and the idempotency key. The short-lived domain begins when a dispatcher scans a bounded horizon and publishes due work. A cron-triggered scan is useful here, but cron is only a wake-up signal — the database query decides what is owed.

The nightly reconciliation follows the same rule. After matching the payment provider's records to internal orders, write any resulting reminder and its outbox record in the same database transaction as the reconciliation decision. A worker leases due outbox rows, attempts the HTTPS delivery, and records the outcome. If the worker stops between the remote call and the acknowledgement, the lease expires and another attempt may happen. The webhook therefore has to reject neither a legitimate retry nor silently repeat the business effect.

This distinction matters.

A queue visibility timeout controls how long an in-flight message stays hidden from other consumers; it does not prove that the downstream effect happened once. Set the lease or visibility timeout longer than the normal delivery attempt, renew it only when the queue supports that operation and the worker is demonstrably healthy, and keep idempotency outside the lease. Otherwise a slow webhook can overlap with a second consumer while the first request is still running.

Put the delivery guarantee in the data model

Start with the guarantees, then size the machinery. For this workload I would write down four invariants: a reconciliation decision and its reminder are committed together; a cancelled reminder is never newly dispatched; every attempt carries the same stable idempotency key; and a successful logical notification can be observed independently of queue state. Those are testable claims. “The scheduler is reliable” isn't.

The minimal record needs more than a timestamp. Keep the tenant or user boundary, the payment reconciliation identifier, the notification kind, the scheduled time in UTC, a monotonically increasing version, status, attempt count, lease expiry, and last outcome. Derive a stable key such as user_id + reconciliation_id + notification_kind + version, then put a unique constraint on it. Payloads should contain identifiers and a schema version rather than a mutable customer profile snapshot unless the audit requirement explicitly demands a snapshot.

Capacity planning starts at the scan, not at average notification volume. If a nightly reconciliation creates N reminders in ten minutes and the delivery SLO allows W seconds to drain them, the required sustained worker rate is at least N/W, before retry headroom. I use peak batch size, webhook latency percentiles, and expected retry amplification to set worker concurrency; an average-per-day figure hides exactly the burst that will page the on-call engineer. Your mileage may vary because provider rate limits and customer traffic shape are inputs, not constants.

Here is the buy-versus-build boundary I would use:

Concern	Database ledger plus workers	Managed workflow or scheduler
Long-horizon storage	Owned in the application database	Delegated, subject to the service's documented retention and scheduling limits
Cancellation and rescheduling	Direct row updates with application authorization	Service API plus local state synchronization
Delivery semantics	Explicit lease, retry, and idempotency design	Service semantics still require an idempotent receiver
On-call load	Schema, scanner, workers, and recovery are yours	Less scheduler maintenance; integration and vendor incidents remain yours
Lock-in	Portable data model and worker contract	Workflow history and service-specific APIs can raise migration cost

The catch is that a home-grown dispatcher is not suitable when the team cannot own leases, backpressure, schema migration, and replay tooling. In that case, use a managed scheduler or workflow engine after verifying its delay, retention, retry, cancellation, and export behavior. Stick with the database-led design when reminders must be joined to reconciliation state, cancellation is common, or audit queries are part of normal operations. Neither choice removes the webhook idempotency requirement.

Implement the due-work path in Go

The following core keeps the interfaces generic. The repository must implement Claim as an atomic lease operation, typically with row locking or a compare-and-swap update; the sender must use HTTPS in production and return a typed result. The code deliberately acknowledges a row only after the receiver confirms the logical notification, while temporary failures release it for retry.

package reminder

import (
    "context"
    "crypto/hmac"
    "crypto/sha256"
    "encoding/hex"
    "errors"
    "fmt"
    "time"
)

type Reminder struct {
    ID             string
    UserID         string
    Reconciliation string
    Kind           string
    ScheduledAt    time.Time
    Version        int64
}

func (r Reminder) IdempotencyKey() string {
    return fmt.Sprintf("%s:%s:%s:%d", r.UserID, r.Reconciliation, r.Kind, r.Version)
}

type Repository interface {
    Claim(ctx context.Context, now time.Time, lease time.Duration, limit int) ([]Reminder, error)
    MarkDelivered(ctx context.Context, id string, deliveredAt time.Time) error
    Release(ctx context.Context, id string, retryAt time.Time, reason string) error
}

type Sender interface {
    PostHTTPS(ctx context.Context, path, key, signature string, body []byte) error
}

type Dispatcher struct {
    Repo   Repository
    Sender Sender
    Secret []byte
    Now    func() time.Time
}

func (d Dispatcher) Run(ctx context.Context) error {
    now := d.Now().UTC()
    items, err := d.Repo.Claim(ctx, now, 2*time.Minute, 200)
    if err != nil {
        return fmt.Errorf("claim due reminders: %w", err)
    }

    var runErr error
    for _, item := range items {
        body := []byte(fmt.Sprintf(`{"reminder_id":%q,"user_id":%q,"kind":%q}`,
            item.ID, item.UserID, item.Kind))
        mac := hmac.New(sha256.New, d.Secret)
        _, _ = mac.Write(body)
        signature := hex.EncodeToString(mac.Sum(nil))

        err = d.Sender.PostHTTPS(ctx, "/webhooks/reminders", item.IdempotencyKey(), signature, body)
        if err == nil {
            if markErr := d.Repo.MarkDelivered(ctx, item.ID, d.Now().UTC()); markErr != nil {
                runErr = errors.Join(runErr, markErr)
            }
            continue
        }

        // The same key is reused because a retry is the same logical delivery.
        retryAt := d.Now().UTC().Add(30 * time.Second)
        if releaseErr := d.Repo.Release(ctx, item.ID, retryAt, err.Error()); releaseErr != nil {
            runErr = errors.Join(runErr, releaseErr)
        }
    }
    return runErr
}

The public receiver should authenticate the signature over the raw body, insert the idempotency key under a unique constraint, apply the notification effect, and commit those changes atomically. A duplicate key should return the same successful logical result rather than send again. If the receiver uses status codes, reserve 409 for a genuine semantic conflict; a previously completed key is not a conflict.

I'm not sure a fixed 30-second retry is right for any particular payment provider without its rate-limit contract and observed latency distribution. Treat that value as an example policy boundary: classify retryable outcomes, add bounded jitter, cap concurrent attempts per destination, and move repeatedly failing work into a queryable review state. Don't let retries occupy every worker and starve fresh reminders.

Verify the SLO, then rehearse rollback

Test with a fake clock so a reminder can move across the scan horizon without waiting. The critical cases are a reminder months away, cancellation just before claim, two dispatchers claiming concurrently, lease expiry during a slow request, duplicate webhook delivery, reconciliation replay, and a timezone change in the user's profile. Store the intended instant in UTC and preserve the user's scheduling zone separately if future recurrences need civil-time semantics.

For deployment, shadow the scanner first: query and count due rows without publishing them, then compare those counts with the existing path. Next, enable a small tenant cohort and watch oldest_due_age, claim conflicts, delivery latency, retry count, duplicate-key hits, and terminal review rows. An SLO might define the percentage of eligible reminders whose logical delivery completes within a chosen window; pick the percentage and window from the business consequence, then alert on burn rate rather than on one failed attempt.

Rollback must stop new claims without deleting ledger rows. Let active leases expire, keep receiver deduplication enabled, and return dispatch ownership to the previous worker only after checking that the two systems cannot claim the same partition. If a release creates a backlog, drain in controlled batches under the provider's documented limits. Fast is good. Correct is better.

The nightly reconciliation is also the repair loop. It should be safe to replay a provider file or page, recreate a missing reminder through the same unique key, and leave already delivered work unchanged. This is why the database remains the source of intent: queue depth tells you what is moving now, while the ledger tells you what should exist and makes a deterministic audit possible.

References

AWS, “Amazon SQS visibility timeout”: https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-visibility-timeout.html
Inngest documentation: https://www.inngest.com/docs

DEV Community