Resilient ERP & CRM API Integrations: Retries, DLQs, Replay

Every ERP, CRM or accounting integration works in the demo. It fails on a Tuesday afternoon when the partner API returns 429 for forty minutes, a webhook arrives twice, an order update lands before the order it updates, or someone renames a custom field in the CRM. The integration did not have a bug. It had no design for failure.

At Loop Origin a large share of our work is connecting custom Laravel and Node applications to ERPs, CRMs, accounting tools and courier platforms. This is the architecture I now use by default: a transactional outbox, rate-limited idempotent workers, a dead-letter table with replay, and a daily reconciliation that catches what everything else missed.

How do you build API integrations that survive outages and rate limits?

Direct answer: Never call a partner API inside the user's request. Write an outbox record in the same database transaction as your business change, deliver it with a queued worker that respects rate limits, retries with backoff and sends an idempotency key, park permanently failing messages in a dead-letter table you can replay, and reconcile both systems daily.

The failure modes you are designing for

Failure	What you see	Mitigation	Where it lives
Partner outage or timeout	5xx, connection resets, slow responses	Queue, timeout, exponential backoff with a deadline	Worker
Rate limiting	429 with or without Retry-After	Client-side limiter plus honouring Retry-After	Worker middleware
Duplicate delivery	Same webhook or job twice	Idempotency keys outbound, unique event IDs inbound	Database constraints
Out-of-order events	Update arrives before create	Per-entity ordering, or re-fetch current state	Worker and handler
Lost writes	Order saved, sync never sent	Transactional outbox	Same transaction as business data
Schema drift	Field renamed, enum value added	Versioned mappers, validation, alerting	Mapping layer
Silent divergence	Totals differ weeks later	Scheduled reconciliation	Reporting job

The architecture in one picture

[App request] --DB transaction--> orders + integration_outbox
                                         |
                       dispatcher (every few seconds)
                                         v
                   queue: erp-sync (rate limited, retries, backoff)
                        |                         |
                  success: mark sent       final failure: integration_dead_letters
                                                  |
                                     replay command / admin screen
[Partner webhooks] -> verify signature -> integration_inbox (unique event_id) -> handlers
[Nightly] reconciliation job compares both sides and opens tickets for differences

The rest of this article fills in each box.

The transactional outbox

The classic bug: save the order, commit, then call the ERP. If the process dies between the commit and the call, the ERP never hears about the order. Calling the ERP before committing is worse, because a later rollback leaves a phantom order in the ERP. The outbox writes the intent to sync in the same transaction as the business change.

Schema::create('integration_outbox', function (Blueprint $t) {
    $t->id();
    $t->string('target');                 // 'erp', 'crm', 'accounting'
    $t->string('event');                  // 'order.created'
    $t->string('aggregate_id');           // order id
    $t->uuid('idempotency_key')->unique();
    $t->json('payload');
    $t->string('status')->default('pending'); // pending|sent|dead
    $t->unsignedInteger('attempts')->default(0);
    $t->timestamp('available_at')->useCurrent();
    $t->timestamps();
    $t->index(['status', 'available_at']);
});

DB::transaction(function () use ($data) {
    $order = Order::create($data);
    IntegrationOutbox::create([
        'target' => 'erp', 'event' => 'order.created',
        'aggregate_id' => (string) $order->id,
        'idempotency_key' => (string) Str::uuid(),
        'payload' => OrderErpMapper::toPayload($order),
    ]);
});

A scheduled dispatcher picks pending rows and pushes jobs. Delivery is now at-least-once, which is why idempotency is not optional.

The worker: rate limits, backoff and a deadline

Laravel's queue gives you most of what you need out of the box. I covered Supervisor, Redis and failed-job basics in Running Laravel Queues in Production; here is the integration-specific part.

// AppServiceProvider::boot()
RateLimiter::for('erp-api', fn () => Limit::perMinute(50));

class PushOutboxToErp implements ShouldQueue
{
    use Queueable;

    public function __construct(public int $outboxId) {}

    public function middleware(): array
    {
        $row = IntegrationOutbox::find($this->outboxId);
        return [
            new RateLimited('erp-api'),
            (new WithoutOverlapping("erp:{$row?->aggregate_id}"))->releaseAfter(30),
        ];
    }

    public function backoff(): array { return [10, 60, 300, 900]; }
    public function retryUntil(): DateTime { return now()->addHours(6); }

    public function handle(ErpClient $erp): void
    {
        $row = IntegrationOutbox::findOrFail($this->outboxId);
        if ($row->status === 'sent') return;           // already delivered

        $res = $erp->post('/orders', $row->payload, [
            'Idempotency-Key' => $row->idempotency_key,
        ]);

        if ($res->status() === 429) {
            $this->release((int) ($res->header('Retry-After') ?: 60));
            return;
        }
        $res->throw();                                  // 5xx -> retry with backoff
        $row->update(['status' => 'sent', 'attempts' => $row->attempts + 1]);
    }

    public function failed(Throwable $e): void
    {
        IntegrationDeadLetter::fromOutbox($this->outboxId, $e);
    }
}

A few decisions in that code matter more than they look:

• The limit is set below the partner's documented limit, because other systems may share your API key.
• WithoutOverlapping per aggregate keeps two updates for the same order from racing, which handles most out-of-order problems.
• retryUntil instead of a fixed attempt count, so a long outage does not burn every retry in the first ten minutes.
• 4xx validation errors should not be retried. I map 400 and 422 straight to the dead-letter table, since retrying a bad payload only adds noise.

If the partner does not support idempotency keys, look up by your own reference first (for example, an external order number field) before creating. The strategy and factory patterns I use for payment and courier gateways work the same way for multiple ERP or CRM targets.

Inbound webhooks: verify, store, then process

Partners retry webhooks, sometimes many times. Treat every inbound event the same way:

1. Verify the signature (usually an HMAC of the raw body with a shared secret) before parsing.
2. Insert into an integration_inbox table with a unique constraint on the partner's event ID. A duplicate insert means you have already seen it; return 200.
3. Respond quickly, then process from the inbox with a queued job.
4. Re-fetch the current state of the entity from the partner API when ordering matters, instead of trusting the event payload.

The same idempotent-ledger thinking appears in my article on multi-vendor marketplace payments, where duplicate payment webhooks are the expensive version of this problem.

Dead letters and replay

A dead-letter table is only useful if someone looks at it and can fix and resend. Mine stores the outbox ID, target, last HTTP status, the error message and a truncated response body. An Artisan command replays after the cause is fixed:

php artisan integration:replay --target=erp --since="2026-10-27 09:00" --dry-run
php artisan integration:replay --target=erp --ids=812,813,840

Replay resets the outbox row to pending with the same idempotency key, so a message that actually reached the partner is not created twice. I alert when the dead-letter count for any target is above zero for more than an hour, not on every individual failure.

Choosing a sync strategy

Strategy	Latency	Load on partner API	Complexity	Use it when
Outbound push via outbox	Seconds to minutes	Proportional to changes	Medium	Your app is the source of truth for the record
Partner webhooks into inbox	Seconds	Low	Medium	Partner supports reliable, signed webhooks
Polling with an updated_since cursor	Minutes	Steady, predictable	Low	No webhooks, or webhooks are unreliable
Nightly full or windowed reconciliation	Hours	Burst at night	Low to medium	Always, as a safety net

Most production integrations I build combine two: push or webhooks for speed, plus cursor polling or reconciliation for correctness.

Field ownership and conflict rules

Two-way sync without rules ends in a loop of each system overwriting the other. Before writing code, I agree a field ownership table with the client: the CRM owns contact details, the ERP owns credit limits and stock, the web app owns orders until they are invoiced. Each mapper only writes the fields its side owns, and every write carries a source marker so an echo of your own change is ignored.

Implementation checklist

1. Never call a partner API inside a user request or a database transaction.
2. Write outbox rows in the same transaction as the business change.
3. Send an idempotency key or check by external reference before creating.
4. Rate-limit below the partner's documented limit and honour Retry-After.
5. Use exponential backoff with an overall deadline, and do not retry 4xx validation errors.
6. Serialize work per entity to avoid out-of-order updates.
7. Verify webhook signatures and deduplicate on event ID.
8. Store dead letters with enough context to fix and replay.
9. Version your field mappers and alert on validation failures.
10. Reconcile key totals daily and review differences.

Our Laravel development and custom web application teams apply this checklist on every integration, and we wrote about common local integration targets in API integration for Pakistani businesses.

Frequently asked questions

What is a dead-letter queue in API integrations?

It is a holding area for messages that failed after all retries. Instead of disappearing into logs, each failed message is stored with its error so a developer can fix the cause and replay it safely, usually with the original idempotency key.

Why use a transactional outbox instead of calling the API after saving?

Because the process can crash between the database commit and the API call, and the partner never hears about the change. Writing an outbox row in the same transaction guarantees the intent to sync is saved whenever the business data is.

How should I handle HTTP 429 responses?

Pause and retry after the time in the Retry-After header if present, otherwise back off exponentially. Also limit your own request rate below the partner's documented limit so you rarely hit 429 in the first place.

What if the partner API does not support idempotency keys?

Store your own reference in a field on the partner record, such as an external order number, and search for it before creating. Combined with per-entity serialization, this prevents most duplicates.

Webhooks or polling: which is more reliable?

Webhooks are faster but can be delayed, duplicated or dropped. Polling with an updated_since cursor is slower but predictable. For business-critical data I use webhooks for speed plus polling or reconciliation as the safety net.

How often should integrations be reconciled?

Daily is a sensible default for orders, invoices and stock. High-volume or financial flows may justify hourly checks of totals and counts, with a full comparison overnight.

---

Follow & Connect

• Website: Loop Origin
• LinkedIn: Loop Origin
• X: Muhammad Ali Mughal (@AliMughal887)
• GitHub: Loop Origin (@LoopOrigin)