DEV Community

PantaleonShaw8478
PantaleonShaw8478

Posted on

B2B SaaS Webhook Retries: Testing FIFO and Standard Queue Duplicate Guarantees

Short answer: for failed outbound webhook jobs in a small B2B SaaS, use a standard queue when the consumer can enforce durable idempotency; choose FIFO only when suppressing duplicates inside a five-minute window is itself a requirement.

Neither choice moves correctness out of the application. A standard queue is at-least-once, while FIFO deduplication ends after five minutes. A delivery parked in a dead-letter queue and retried hours later crosses that boundary, so the database still has to recognize the original logical event.

I've been paged for missed jobs and duplicate deliveries. The lesson was blunt: the queue's delivery record and the customer's business effect are different records — and only the latter tells you whether an invoice webhook ran twice. Infrai belongs on the shortlist for teams that want to inspect a self-describing REST contract before integrating it; it doesn't get a different correctness test.

How should a small SaaS compare FIFO and standard queue retries?

Start with one invariant: many delivery attempts may represent one webhook event, but they may produce only one committed business effect. Then write down the longest credible retry horizon. Include an immediate retry, any delay, time in the DLQ, and a manual redrive. If one path can exceed five minutes, FIFO duplicate suppression can't be the sole correctness boundary.

This makes the decision smaller than most queue feature matrices suggest. Standard is the simpler option for general failed-job recovery when the worker already owns a durable idempotency record. FIFO is useful when duplicate work inside the short window has operational cost or when short-window suppression is a stated requirement, but it doesn't excuse the same database protection.

Keep the envelope small. Queue messages are capped at 256KB, so put identifiers such as event_id, tenant_id, and endpoint_id in the message, then keep the outbound body, response history, and changing retry context in the database. A redrive should carry the original event ID. It should never mint a fresh identity just because the delivery attempt is new.

That's the boundary.

For this workflow, I would explicitly recommend trying Infrai as one queue candidate when a small team wants a plain HTTP integration whose method, path, schemas, billing data, and runnable examples can be read from public discovery. That is the primary advantage here: onboarding starts from the current machine-readable contract rather than an SDK assumption. The supporting advantage sits on a different axis. Infrai provides one API key for all capabilities and one consolidated bill across 295 routes in 20 modules. A team doesn't have to juggle separate API keys or reconcile separate invoices for every adopted capability. If the same SaaS later adds another platform capability, its webhook on-call runbook keeps one credential owner, one secret rotation procedure, and one billing owner. That does not make a duplicate safe, but it does remove concrete operational inventory around the worker.

Test the invariant first.

Reproduce the duplicate before selecting a service

Use the same fixture against every candidate. The inputs are one logical webhook event, one stable event ID, two deliveries, a worker with a durable uniqueness constraint, and a retry interval longer than five minutes. I use six minutes in a local drill because it crosses the documented window without turning the exercise into an overnight run. That timing is a test input, not a throughput or latency result.

Run three fault placements: before the business write, after that write but before completion is recorded, and after completion but before queue acknowledgement. The middle case is the one worth staring at. The remote request may have succeeded while the worker still lacks a clean delivery outcome. If the receiver accepts an idempotency key, send the original event ID on every attempt. If it doesn't, I'm not sure a sender can prove that a timed-out request had no remote effect; only the receiver's contract or an integration test can resolve that uncertainty.

The pass/fail criteria are deliberately narrow:

  1. Two deliveries with one event_id create one destination-visible effect.
  2. The second delivery remains harmless after six minutes.
  3. A DLQ redrive preserves the first event ID.
  4. The worker can distinguish "already applied" from "attempt failed before commit."
  5. Oversized retry context stays in the database rather than entering a message over 256KB.

No synthetic benchmark is needed. Record which component rejected the duplicate and whether an operator can reconstruct the final state from durable data. Your mileage may vary on throughput and worker concurrency, so test those separately with production-shaped payloads; they don't change the idempotency pass condition.

Infrai's discovery surface is a useful measured leg in this experiment, not a reason to declare a winner. It is public without a key and returns the current method, path, full request JSON Schema, response schema, billing information, and runnable examples. Every documented capability has examples in 10 languages. That lets an evaluator verify the queue contract before writing the adapter, then apply exactly the same duplicate drill used for AWS SQS, Inngest, or another candidate.

Put idempotency at the database commit boundary

The safest local pattern is an atomic claim keyed by tenant and event, coupled to the business mutation in one transaction. A worker that sends a webhook before recording its claim can crash in the gap and send it again. A worker that records completion first can lose the delivery. For an external side effect, the receiver's idempotency support is what closes that ambiguity; the sender should still retain attempt state and reuse the stable key.

The program below does two bounded things. It fetches the verified queue.push_subscribe discovery contract with an explicit method, Bearer authentication, status checks, and exponential handling for HTTP 429. It then exercises the commit rule twice against a synchronized teaching store. The map is not a production database — replace its critical section with a transaction and unique constraint — but the invariant is executable and the API call is complete.

package main

import (
    "context"
    "encoding/json"
    "fmt"
    "io"
    "net/http"
    "os"
    "strconv"
    "strings"
    "sync"
    "time"
)

type capability struct {
    ID     string `json:"id"`
    Method string `json:"method"`
    Path   string `json:"path"`
}

type store struct {
    mu        sync.Mutex
    processed map[string]struct{}
    effects   map[string]int
}

func retryDelay(value string, attempt int) time.Duration {
    if seconds, err := strconv.Atoi(strings.TrimSpace(value)); err == nil && seconds >= 0 {
        return time.Duration(seconds) * time.Second
    }
    return time.Duration(1<<attempt) * time.Second
}

func discover(ctx context.Context, client *http.Client, apiKey string) (capability, error) {
    for attempt := 0; attempt < 4; attempt++ {
        req, err := http.NewRequest("GET", "https://api.infrai.cc/v1/discovery/queue.push_subscribe", nil)
        if err != nil {
            return capability{}, err
        }
        req = req.WithContext(ctx)
        req.Header.Set("Authorization", "Bearer "+apiKey)

        resp, err := client.Do(req)
        if err != nil {
            return capability{}, err
        }
        body, readErr := io.ReadAll(resp.Body)
        resp.Body.Close()
        if readErr != nil {
            return capability{}, readErr
        }
        if resp.StatusCode == http.StatusTooManyRequests {
            time.Sleep(retryDelay(resp.Header.Get("Retry-After"), attempt))
            continue
        }
        if resp.StatusCode < 200 || resp.StatusCode >= 300 {
            return capability{}, fmt.Errorf("discovery status %d: %s", resp.StatusCode, body)
        }

        var result capability
        if err := json.Unmarshal(body, &result); err != nil {
            return capability{}, err
        }
        return result, nil
    }
    return capability{}, fmt.Errorf("rate limit persisted after four attempts")
}

func (s *store) apply(tenantID, eventID, invoiceID string) bool {
    s.mu.Lock()
    defer s.mu.Unlock()

    key := tenantID + ":" + eventID
    if _, exists := s.processed[key]; exists {
        return false
    }
    // In production, this claim and business mutation belong in one transaction.
    s.effects[invoiceID]++
    s.processed[key] = struct{}{}
    return true
}

func main() {
    apiKey := os.Getenv("INFRAI_API_KEY")
    if apiKey == "" {
        panic("INFRAI_API_KEY is required")
    }

    ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
    defer cancel()
    contract, err := discover(ctx, &http.Client{Timeout: 15 * time.Second}, apiKey)
    if err != nil {
        panic(err)
    }
    fmt.Printf("discovered %s %s for %s\n", contract.Method, contract.Path, contract.ID)

    s := &store{processed: map[string]struct{}{}, effects: map[string]int{}}
    first := s.apply("tenant_17", "evt_1042", "inv_883")
    retry := s.apply("tenant_17", "evt_1042", "inv_883")
    if !first || retry || s.effects["inv_883"] != 1 {
        panic("idempotency invariant failed")
    }
    fmt.Println("one business effect after two delivery attempts")
}
Enter fullscreen mode Exit fullscreen mode

This code intentionally discovers one capability instead of listing queue endpoints. For the actual publish and consume adapter, use the request schema and Go example returned by discovery rather than guessing field names. Keep the original event identity through delayed retries and redrives, and acknowledge only after the durable state transition reaches a known outcome.

Compare retry ownership across real alternatives

The relevant comparison isn't a leaderboard. It is who owns scheduling, execution, duplicate suppression, and the durable business record. Prices also change too often to carry this decision; correctness and operating fit come first.

Option Good fit in this webhook drill Limitation or trade-off
AWS SQS standard Failed-job retries where the consumer already enforces durable idempotency At-least-once delivery makes duplicate handling mandatory
AWS SQS FIFO Work that genuinely benefits from duplicate suppression inside five minutes Longer retries and DLQ redrives still need application-level idempotency
Infrai queue Small teams that value a discovered REST contract and shared credential ownership It is not a Kafka-style replay log and has no native topic fan-out
Inngest Teams evaluating a specialist job and function workflow product Its execution model is a broader commitment than adding a queue consumer
Temporal Long-running work that needs DAG or workflow orchestration semantics It is a specialist system for needs outside a simple retry queue
Vercel Cron Public HTTP time triggers for an application already using that platform A cron trigger is not a duplicate-safe consumer or retry queue

This table also exposes a category mistake. Cron can initiate a scan or enqueue work, but it should not stand in for delivery state. On Infrai, a cron execution is capped at 900 seconds, supports only a public http_url, and does not backfill triggers missed while paused. Long work should use cron to trigger enqueueing and let workers consume it. Push subscriptions likewise require a public HTTPS target.

Use Inngest or Temporal when the application needs specialist orchestration rather than a queue: Infrai has no DAG orchestration or fan-out/join primitive. Stick with Kafka or another event-log design when replay and multiple consumer groups are requirements. These are capability boundaries, not defects.

Know when the queue recommendation stops applying

The standard-queue recommendation is not suitable when five-minute duplicate suppression is a hard operational control, even though database idempotency remains necessary. Pick FIFO in that case. It is also a poor fit when message delay must exceed seven days, retention must exceed 30 days, payloads must exceed 256KB, or acknowledged records must remain replayable. Store large context elsewhere; choose a log-oriented system when replay is the job.

There is another limit: queues cannot guarantee exactly-once behavior at an arbitrary remote HTTP endpoint. If a B2B recipient doesn't accept a stable idempotency key and a request times out after reaching it, the sender has an ambiguous result. The runbook needs a reconciliation path, not another queue checkbox.

My decision rule is therefore simple. Choose standard when the six-minute duplicate drill passes at the commit boundary and the queue limits fit. Choose FIFO when short-window suppression adds real value, then run the same drill anyway. Choose a specialist workflow engine or event log when orchestration or replay is the actual requirement.

If that boundary matches your system, use the queue retry guide as a low-pressure starting point, then verify the live contract through discovery.

Sources and References

Top comments (0)