DEV Community: Muhammad Masad Ashraf

How to Build a Shopify Webhook Replay System That Never Loses Events

Muhammad Masad Ashraf — Fri, 22 May 2026 15:41:32 +0000

Your Shopify integration fires perfectly in staging. Then production hits. Your server goes down for two minutes during a deploy. Shopify sends 40 webhook events. Your handler catches none of them.

That is real data loss. Orders not fulfilled. Inventory not updated. Customers not synced.

A webhook replay system is the fix. This guide walks you through building one that stores every incoming event, retries failures intelligently, and lets you replay any batch of events on demand.

Why Webhooks Fail in the First Place

Shopify retries failed webhooks up to 19 times over 48 hours. After that, you are on your own. Here is what causes failures:

Failure Cause	Description
Server downtime	Endpoint offline when Shopify fires
Timeout	Handler takes longer than 5 seconds
Code errors	Bug in your handler throws an exception
Rate limiting	System overloaded, rejects the request
DB unavailability	Database is down when handler tries to write
Deployment gaps	Deploy restarts server at the wrong moment

After 48 hours and 19 retries, Shopify flags your endpoint as failing. Eventually it stops sending events entirely.

A replay system gets you out of that hole.

The Core Architecture

A production webhook replay system has four stages:

Shopify
   |
   v
[Ingestion Endpoint]   <-- ACK immediately, store raw payload
   |
   v
[Event Store (DB)]     <-- durable record of every event
   |
   v
[Processing Queue]     <-- background worker picks up events
   |
   v
[Handler]              <-- idempotent processing logic
   |
   +-- Success --> mark 'succeeded'
   +-- Failure --> retry with backoff --> Dead Letter Queue

The key rule: never process inside the ingestion endpoint. Shopify waits only 5 seconds. Acknowledge fast, process separately.

Stage 1: The Ingestion Endpoint

Your ingestion endpoint does exactly two things:

Verify the HMAC signature
Write the raw payload to the event store and return 200 OK

app.post('/webhooks', express.raw({ type: 'application/json' }), async (req, res) => {
  const hmac = req.headers['x-shopify-hmac-sha256'];
  const webhookId = req.headers['x-shopify-webhook-id'];
  const topic = req.headers['x-shopify-topic'];

  // Verify HMAC first
  if (!verifyHmac(req.body, hmac)) {
    return res.status(401).send('Unauthorized');
  }

  // Store immediately, respond fast
  await db.query(`
    INSERT INTO webhook_events (shopify_id, topic, payload, hmac, status)
    VALUES ($1, $2, $3, $4, 'pending')
    ON CONFLICT (shopify_id) DO NOTHING
  `, [webhookId, topic, req.body.toString(), hmac]);

  res.status(200).send('OK');
});

The ON CONFLICT DO NOTHING handles Shopify's own retries. Same event, same ID, no duplicate row.

Stage 2: The Event Store Schema

CREATE TABLE webhook_events (
  id              UUID PRIMARY KEY DEFAULT gen_random_uuid(),
  shopify_id      VARCHAR(255) UNIQUE,
  topic           VARCHAR(100) NOT NULL,
  payload         JSONB NOT NULL,
  hmac            VARCHAR(255),
  status          VARCHAR(20) DEFAULT 'pending',
  retry_count     INT DEFAULT 0,
  last_error      TEXT,
  created_at      TIMESTAMPTZ DEFAULT NOW(),
  processed_at    TIMESTAMPTZ
);

CREATE INDEX idx_webhook_status ON webhook_events(status);
CREATE INDEX idx_webhook_topic  ON webhook_events(topic);
CREATE INDEX idx_webhook_created ON webhook_events(created_at);

Index on status, topic, and created_at. These are the three axes you will query for replay.

Storage options by scale:

Option	Best For
PostgreSQL	Most Shopify apps. Full SQL, JSONB indexing.
Redis Streams	High-throughput ingestion, consumer groups.
AWS SQS + DynamoDB	Managed infra, no ops overhead.
Kafka	Enterprise-scale event sourcing.

Stage 3: The Processing Worker

async function processWebhookEvents() {
  const events = await db.query(`
    SELECT * FROM webhook_events
    WHERE status = 'pending'
    ORDER BY created_at ASC
    LIMIT 50
    FOR UPDATE SKIP LOCKED
  `);

  for (const event of events.rows) {
    try {
      await handleEvent(event.topic, JSON.parse(event.payload));

      await db.query(`
        UPDATE webhook_events
        SET status = 'succeeded', processed_at = NOW()
        WHERE id = $1
      `, [event.id]);

    } catch (err) {
      const newRetryCount = event.retry_count + 1;
      const newStatus = newRetryCount >= 5 ? 'dead' : 'pending';

      await db.query(`
        UPDATE webhook_events
        SET status = $1, retry_count = $2, last_error = $3
        WHERE id = $4
      `, [newStatus, newRetryCount, err.message, event.id]);
    }
  }
}

FOR UPDATE SKIP LOCKED prevents multiple workers from grabbing the same event. Safe for concurrent processing.

Stage 4: Exponential Backoff

Do not retry immediately. Back off with jitter:

function getRetryDelay(retryCount) {
  const baseDelays = [30, 120, 600, 3600, 21600]; // seconds
  const base = (baseDelays[retryCount] || 21600) * 1000;
  const jitter = Math.random() * 1000;
  return base + jitter;
}

Retry	Delay
1	30 seconds
2	2 minutes
3	10 minutes
4	1 hour
5+	6 hours

Jitter prevents retry storms when many events fail simultaneously.

Stage 5: The Replay Trigger

This is the part that earns the "replay" in "replay system." When you fix a bug and need to reprocess failed events:

async function replayEvents({ topic, status = 'failed', from, to, batchSize = 100 }) {
  const events = await db.query(`
    SELECT * FROM webhook_events
    WHERE status = $1
      AND ($2::text IS NULL OR topic = $2)
      AND ($3::timestamptz IS NULL OR created_at >= $3)
      AND ($4::timestamptz IS NULL OR created_at <= $4)
    ORDER BY created_at ASC
    LIMIT $5
  `, [status, topic, from, to, batchSize]);

  for (const event of events.rows) {
    // Reset to pending so the worker picks it up
    await db.query(`
      UPDATE webhook_events
      SET status = 'pending', retry_count = 0, last_error = NULL
      WHERE id = $1
    `, [event.id]);
  }

  return events.rowCount;
}

// Usage: replay all failed orders/paid from the last 24 hours
await replayEvents({
  topic: 'orders/paid',
  status: 'failed',
  from: new Date(Date.now() - 86400000)
});

Idempotency: The Non-Negotiable Requirement

When you replay, your handler runs twice for the same payload. Without idempotency, you get duplicate orders, double charges, corrupted inventory.

Use the shopify_id (the X-Shopify-Webhook-Id) as your deduplication key:

async function handleOrderCreate(payload) {
  const shopifyOrderId = payload.id;

  // Check if already processed
  const existing = await db.query(
    'SELECT id FROM orders WHERE shopify_order_id = $1',
    [shopifyOrderId]
  );

  if (existing.rows.length > 0) {
    console.log(`Order ${shopifyOrderId} already processed. Skipping.`);
    return;
  }

  // Safe to process
  await db.query(
    'INSERT INTO orders (shopify_order_id, ...) VALUES ($1, ...)',
    [shopifyOrderId, ...]
  );
}

No idempotency = no safe replay. Build this first.

Dead Letter Queue

Events that hit max retries go to DLQ status. Monitor these aggressively:

// Query DLQ events
const dlqEvents = await db.query(`
  SELECT topic, COUNT(*) as count, MIN(created_at) as oldest
  FROM webhook_events
  WHERE status = 'dead'
  GROUP BY topic
  ORDER BY count DESC
`);

DLQ growth is not a metric to check weekly. Alert on it in real time. Every row is an event that never reached its destination.

When to Replay What

Scenario	Action
Server down for 2 hours	Automatic retry handles it
Bug deployed and fixed	Manual replay by time range
New integration added	Manual replay to backfill history
Downstream API was down	Automatic retry with backoff
Data migration	Manual replay by topic and date

Mistakes That Kill Replay Systems

Not storing the raw payload. Store the exact bytes Shopify sent. If you transform first, you cannot replay the original.

Processing in the ingestion endpoint. 5-second window. Any business logic risks timeout and a missed event.

No idempotency. Replay becomes dangerous instead of safe.

Unlimited retries. Cap at 5-7. Move the rest to DLQ.

Ignoring DLQ depth. Silent DLQ growth means silent data loss.

Tools to Accelerate This

Tool	Role
BullMQ (Node.js)	Queue + retry + backoff out of the box
Sidekiq (Ruby)	Background jobs with retry logic
AWS SQS + Lambda	Fully managed, serverless
Temporal	Durable workflows with built-in replay
pg-boss	PostgreSQL-backed job queue

Wrapping Up

The pattern is simple:

Ingest fast, store everything
Process asynchronously
Retry with backoff
Route terminal failures to DLQ
Replay on demand after fixes Every layer compounds reliability. Start with the event store. Add the worker. Add retry logic. Then add the replay trigger.

Your Shopify store generates events constantly. A replay system makes sure none of them disappear.

Originally published at kolachitech.com

Stop Losing Shopify Webhooks: A Retry Strategy That Survives Real Outages

Muhammad Masad Ashraf — Fri, 22 May 2026 00:40:32 +0000

Stop Losing Shopify Webhooks: A Retry Strategy That Survives Real Outages

Here is a scenario every Shopify developer eventually lives through.

It is the middle of a deploy. Your endpoint is down for maybe 90 minutes. Nothing dramatic. But while you were shipping, three orders came in, an inventory sync fired, and a customer updated their address.

Shopify tried to deliver those webhooks. It retried. Then it gave up.

Those events are gone. Not delayed. Gone.

I have watched this exact thing happen, and the worst part is how quiet it is. No error in your logs. No alert. Just a slow drift between what Shopify knows and what your system knows. You find out days later when a customer asks where their order went.

So let us talk about how to actually fix this, properly, with a retry strategy that survives more than a quick hiccup.

First, Know What Shopify Actually Does

You cannot build a good retry layer until you know what the platform gives you for free.

When Shopify sends a webhook, it waits 5 seconds for your endpoint to respond. Any 2xx status code is a success. Anything else, including a timeout, is a failure.

After a failure, Shopify retries. And here is the part people get wrong, because the internet is full of outdated info:

As of the September 2024 policy update, Shopify retries a failed webhook up to 8 times over a 4-hour window with exponential backoff.

The old "19 retries over 48 hours" number you will find in older blog posts is dead. If your reliability code was written before late 2024, your timing assumptions are probably wrong.

A quick reference:

Behavior	Detail
Timeout	5 seconds per attempt
Success	Any 2xx status
Retries	Up to 8 attempts
Window	4 hours total
Backoff	Exponential
Payload	Original payload from trigger time
The scary part	Persistent failures delete the subscription

That last row is the one that ruins weekends. If your endpoint fails enough, Shopify does not just drop events. It removes the webhook subscription. New events stop firing entirely until you re-register. Silent and total.

The Problem With 4 Hours

Exponential backoff over 8 attempts in 4 hours front-loads everything. Most of your retries happen in the first 30 minutes. By hour 2, roughly 5 of your 8 attempts are spent.

So Shopify's retry system is genuinely good at one thing: surviving short, transient blips. A momentary network drop. A one-off timeout.

It is bad at the thing that actually hurts you: real outages.

A deploy that runs long
A downstream API that rate-limits you for hours
A traffic spike during a sale that pushes responses past 5 seconds
A botched HMAC secret rotation that 401s every webhook

In all of those, the 4-hour window runs out before you recover. Shopify did its job. Your data is still lost.

The conclusion is simple: you need your own retry layer. Shopify's is the floor, not the ceiling.

Rule Zero: Acknowledge Fast, Process Later

Before any retry logic, fix the most common mistake in Shopify webhook code: running business logic inside the webhook endpoint.

If your handler hits a database, calls an external API, or does anything heavy, you are gambling against that 5-second timeout. Cross it once and Shopify marks a perfectly good delivery as failed.

Your endpoint should do almost nothing:

// The entire job of your webhook endpoint
app.post("/webhooks/orders-create", async (req, res) => {
  // 1. Verify the HMAC signature
  if (!verifyHmac(req)) return res.sendStatus(401);

  // 2. Push the raw payload onto a queue
  await queue.add("orders-create", {
    body: req.rawBody,
    eventId: req.headers["x-shopify-event-id"],
    triggeredAt: req.headers["x-shopify-triggered-at"],
  });

  // 3. Respond immediately
  res.sendStatus(200);
});

That is it. A background worker pulls from the queue and does the real work. This one change eliminates the majority of timeout failures, and it turns retries into a calm internal concern instead of a race against a clock.

Building the Retry Layer

Your retry logic lives in the worker, not the endpoint. When processing fails, the worker decides: retry, or give up?

Step 1: Classify the error

Not every failure deserves a retry. Retrying a permanent error just burns resources.

Error type	Examples	Action
Transient	Timeout, 503, deadlock, rate limit	Retry with backoff
Permanent	Invalid payload, missing field, validation error	Straight to the dead letter queue

Retrying a malformed payload 8 times will not magically make it valid. Categorize first, act second.

Step 2: Exponential backoff with jitter

For transient errors, retry with increasing delays. Each retry waits longer than the last.

Attempt	Delay
1	30 seconds
2	2 minutes
3	8 minutes
4	30 minutes
5	2 hours
6	6 hours
7	24 hours

Notice the window stretches across days, not Shopify's 4 hours. That is the whole point of building your own layer.

But pure exponential backoff has a trap. If 500 webhooks fail at the same moment, they all retry at the same moment, and your recovering service gets hammered flat again. This is the thundering herd.

Fix it with jitter: add a small random offset to each delay. Instead of retrying at exactly 8 minutes, retry somewhere between 7 and 9. It spreads the load and smooths recovery.

function nextDelay(attempt) {
  const base = Math.min(30 * 2 ** attempt, 86400); // cap at 24h
  const jitter = base * (Math.random() * 0.3); // up to 30% jitter
  return base + jitter;
}

Step 3: Cap retries, then use a dead letter queue

Retries cannot run forever. Cap them, usually somewhere between 5 and 10 attempts. When a webhook exhausts its retries, it does not vanish. It moves to a dead letter queue.

The DLQ stores the failed webhook with full context: original payload, error message, attempt count, timestamp. Now you can:

Inspect it and find the root cause
Reprocess it once the bug is fixed
Alert when the DLQ grows past a normal threshold

That alert is your early warning system. A spike in DLQ volume tells you something is wrong before any customer notices.

The Duplicate Problem Nobody Mentions

Retries create a brand new problem: the same webhook arriving twice.

Shopify delivers a webhook, your slow response times out, Shopify retries, but your worker already processed the first one. Now you have a duplicate. Without protection, that means double inventory adjustments, duplicate emails, double-charged orders.

The fix is idempotency. Every Shopify webhook carries an X-Shopify-Event-Id header, and that value stays identical across every retry of the same event.

Use it as a dedup key:

async function processWebhook(job) {
  const { eventId } = job.data;

  if (await alreadyProcessed(eventId)) {
    return; // safe no-op
  }

  await doTheRealWork(job.data);
  await markProcessed(eventId);
}

A retry strategy without deduplication is incomplete. Build this in from day one.

Watch Out for Stale Payloads

One more subtlety. When Shopify retries, it sends the original payload from when the event was triggered, not the current state.

If an order changed three times during the retry window, a late retry still carries the first version. Apply it blindly and you overwrite newer data with older data.

Always check the X-Shopify-Triggered-At header against your own records. If your data is already newer, skip the stale update or fetch fresh data from the Admin API instead.

The Layer Even Retries Cannot Save You From

Here is the uncomfortable truth: even a perfect retry layer cannot recover an event Shopify never sent, or one it dropped after its own retries failed.

For that, you need reconciliation. Periodically poll the Shopify Admin API and compare its data against yours. Hourly for high-value topics like orders/*. Daily for lower-stakes data.

Reconciliation is the final safety net. It catches events lost to extended outages, dropped events, and gaps from a removed subscription.

Combine the three layers and you lose almost nothing:

Layer	Catches
Shopify retries	Short transient blips
Your retry layer + DLQ	Longer outages, downstream failures
Reconciliation	Everything else, including dropped events

Do Not Forget Monitoring

A retry strategy fails silently without eyes on it. Track these:

Webhook failure rate
DLQ size and growth
Retry volume
Subscription status (run a daily check, alert on any missing topic)
Processing latency creeping toward 5 seconds

Shopify's Dev Dashboard has a delivery metrics report with response codes and retry counts per topic. Use it alongside your own monitoring, not instead of it.

TL;DR

Shopify gives you 8 retries over 4 hours. Real outages last longer than that. So:

Acknowledge fast. Return 200 in under 5 seconds, queue the payload.
Classify errors. Transient gets retried, permanent goes to the DLQ.
Exponential backoff with jitter. Retry over days, not hours.
Idempotency. Use X-Shopify-Event-Id to kill duplicates.
Dead letter queue. Nothing gets lost, everything is replayable.
Reconcile. Poll the Admin API to catch what slipped through.
Monitor. Surface problems before customers do.

Build these layers once and your integration stops losing data, even on its worst day.

If you want the full guide, you can read it here.

How does your team handle webhook reliability? I would genuinely like to hear what has worked for you in the comments.

Building a Dead Letter Queue for Shopify Webhooks (Production-Ready Guide)

Muhammad Masad Ashraf — Mon, 18 May 2026 16:02:08 +0000

Building a Dead Letter Queue for Shopify Webhooks

Your Shopify webhook just failed. Network timeout. Database deadlock. Third-party API down.

The webhook payload? Gone forever.

Unless you have a dead letter queue.

This guide walks through building a production-ready DLQ system for Shopify webhooks. I'll show you the architecture, code examples, and mistakes to avoid based on real production systems.

Why Shopify's Built-in Retry Isn't Enough

Shopify retries failed webhooks up to 19 times over 48 hours. After that, they stop delivering to your endpoint.

Attempt 1: Immediate
Attempt 2: 1 second later
Attempt 3: 2 seconds later
...
Attempt 19: 48 hours later
Then: 💀 Webhook deleted forever

For production systems handling payments, orders, and inventory, this isn't acceptable. You need a safety net.

Architecture Overview

Here's the high-level flow:

Shopify → Your Webhook Endpoint → Primary Queue → Worker → Success ✓
                    ↓                                    ↓
              200 OK (always)                        Failure ✗
                                                          ↓
                                                   Dead Letter Queue
                                                          ↓
                                                   Retry Logic

Critical principle: Always return 200 OK to Shopify immediately. Process webhooks asynchronously.

The Stack

I'll use this stack for examples (adapt to your needs):

Queue: Redis with Bull (Node.js) or AWS SQS
Database: PostgreSQL for DLQ storage
Runtime: Node.js with Express
Monitoring: Custom dashboard + alerts

Implementation: Step by Step

1. Webhook Reception (Never Block Here)

// webhook-receiver.js
const express = require('express');
const crypto = require('crypto');
const { primaryQueue } = require('./queues');

const app = express();

app.post('/webhooks/shopify/:topic', express.raw({ type: 'application/json' }), async (req, res) => {
  const topic = req.params.topic;
  const hmac = req.get('X-Shopify-Hmac-Sha256');
  const body = req.body;

  // Verify webhook signature
  if (!verifyWebhook(body, hmac)) {
    return res.status(401).send('Invalid signature');
  }

  // Respond to Shopify IMMEDIATELY
  res.status(200).send('OK');

  // Queue for async processing (never blocks)
  try {
    await primaryQueue.add(topic, {
      topic,
      shopDomain: req.get('X-Shopify-Shop-Domain'),
      webhookId: req.get('X-Shopify-Webhook-Id'),
      payload: JSON.parse(body.toString()),
      receivedAt: new Date().toISOString()
    }, {
      attempts: 6,
      backoff: {
        type: 'exponential',
        delay: 10000 // Start with 10 seconds
      }
    });
  } catch (error) {
    // If queuing fails, write directly to DLQ
    await saveToDLQ({
      topic,
      payload: JSON.parse(body.toString()),
      error: 'Failed to queue webhook',
      failureReason: error.message
    });
  }
});

function verifyWebhook(body, hmac) {
  const hash = crypto
    .createHmac('sha256', process.env.SHOPIFY_WEBHOOK_SECRET)
    .update(body)
    .digest('base64');
  return hash === hmac;
}

2. Worker with DLQ Fallback

// webhook-worker.js
const { primaryQueue, dlqQueue } = require('./queues');
const { processWebhook } = require('./processors');
const { saveToDLQ, incrementRetryCount } = require('./dlq-storage');

primaryQueue.process(async (job) => {
  const { topic, payload, webhookId, shopDomain } = job.data;

  try {
    // Your business logic here
    await processWebhook(topic, payload, shopDomain);

    // If we got here, processing succeeded
    return { success: true, webhookId };

  } catch (error) {
    // Categorize the error
    if (isTransientError(error)) {
      // Let Bull retry with exponential backoff
      throw error;
    } else {
      // Permanent error - move to DLQ immediately
      await saveToDLQ({
        webhookId,
        topic,
        shopDomain,
        payload,
        error: error.message,
        stackTrace: error.stack,
        retryCount: job.attemptsMade,
        firstFailedAt: job.timestamp,
        lastAttemptedAt: new Date().toISOString()
      });

      // Don't throw - we handled it
      return { success: false, movedToDLQ: true };
    }
  }
});

// When all retries exhausted, Bull calls this
primaryQueue.on('failed', async (job, error) => {
  if (job.attemptsMade >= job.opts.attempts) {
    await saveToDLQ({
      webhookId: job.data.webhookId,
      topic: job.data.topic,
      shopDomain: job.data.shopDomain,
      payload: job.data.payload,
      error: 'Max retries exceeded',
      stackTrace: error.stack,
      retryCount: job.attemptsMade,
      firstFailedAt: job.timestamp,
      lastAttemptedAt: new Date().toISOString()
    });
  }
});

function isTransientError(error) {
  // Network issues, rate limits, timeouts
  return error.code === 'ECONNREFUSED' ||
         error.code === 'ETIMEDOUT' ||
         error.statusCode === 429 ||
         error.statusCode === 503 ||
         error.message.includes('deadlock');
}

3. DLQ Storage Schema

-- dlq_webhooks table
CREATE TABLE dlq_webhooks (
  id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
  webhook_id VARCHAR(255) UNIQUE,
  topic VARCHAR(100) NOT NULL,
  shop_domain VARCHAR(255) NOT NULL,
  payload JSONB NOT NULL,
  error_message TEXT,
  stack_trace TEXT,
  retry_count INTEGER DEFAULT 0,
  first_failed_at TIMESTAMP NOT NULL,
  last_attempted_at TIMESTAMP NOT NULL,
  status VARCHAR(50) DEFAULT 'pending', -- pending, processing, resolved, abandoned
  resolved_at TIMESTAMP,
  notes TEXT,
  created_at TIMESTAMP DEFAULT NOW()
);

-- Indexes for querying
CREATE INDEX idx_dlq_status ON dlq_webhooks(status);
CREATE INDEX idx_dlq_topic ON dlq_webhooks(topic);
CREATE INDEX idx_dlq_shop ON dlq_webhooks(shop_domain);
CREATE INDEX idx_dlq_first_failed ON dlq_webhooks(first_failed_at);

4. DLQ Storage Functions

// dlq-storage.js
const { Pool } = require('pg');
const pool = new Pool();

async function saveToDLQ(webhookData) {
  const {
    webhookId,
    topic,
    shopDomain,
    payload,
    error,
    stackTrace,
    retryCount = 0,
    firstFailedAt,
    lastAttemptedAt
  } = webhookData;

  const query = `
    INSERT INTO dlq_webhooks (
      webhook_id, topic, shop_domain, payload,
      error_message, stack_trace, retry_count,
      first_failed_at, last_attempted_at
    ) VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9)
    ON CONFLICT (webhook_id) 
    DO UPDATE SET
      retry_count = dlq_webhooks.retry_count + 1,
      last_attempted_at = $9,
      error_message = $5,
      stack_trace = $6
    RETURNING id
  `;

  try {
    const result = await pool.query(query, [
      webhookId,
      topic,
      shopDomain,
      JSON.stringify(payload),
      error,
      stackTrace,
      retryCount,
      firstFailedAt || new Date(),
      lastAttemptedAt || new Date()
    ]);

    // Alert if this is a new DLQ entry
    if (result.rowCount > 0) {
      await alertDLQEntry(webhookId, topic, shopDomain);
    }

    return result.rows[0];
  } catch (err) {
    console.error('Failed to save to DLQ:', err);
    // This is serious - DLQ itself failed
    // Log to external service (Sentry, etc.)
  }
}

async function getDLQWebhooks(status = 'pending', limit = 100) {
  const query = `
    SELECT * FROM dlq_webhooks
    WHERE status = $1
    ORDER BY first_failed_at ASC
    LIMIT $2
  `;

  const result = await pool.query(query, [status, limit]);
  return result.rows;
}

async function markAsResolved(webhookId, notes = null) {
  const query = `
    UPDATE dlq_webhooks
    SET status = 'resolved',
        resolved_at = NOW(),
        notes = $2
    WHERE webhook_id = $1
  `;

  await pool.query(query, [webhookId, notes]);
}

module.exports = {
  saveToDLQ,
  getDLQWebhooks,
  markAsResolved
};

5. Automated DLQ Retry Worker

// dlq-retry-worker.js
const { getDLQWebhooks, markAsResolved, saveToDLQ } = require('./dlq-storage');
const { processWebhook } = require('./processors');

async function processDLQWebhooks() {
  const webhooks = await getDLQWebhooks('pending', 50);

  for (const webhook of webhooks) {
    try {
      // Update status to prevent concurrent processing
      await updateDLQStatus(webhook.webhook_id, 'processing');

      // Try to process again
      await processWebhook(
        webhook.topic,
        webhook.payload,
        webhook.shop_domain
      );

      // Success! Mark as resolved
      await markAsResolved(webhook.webhook_id, 'Auto-resolved by retry worker');

    } catch (error) {
      // Still failing
      const newRetryCount = webhook.retry_count + 1;

      if (newRetryCount >= 10) {
        // Give up after 10 DLQ retries
        await updateDLQStatus(webhook.webhook_id, 'abandoned');
        await alertPermanentFailure(webhook);
      } else {
        // Save back to DLQ with updated count
        await saveToDLQ({
          ...webhook,
          retryCount: newRetryCount,
          error: error.message,
          stackTrace: error.stack,
          lastAttemptedAt: new Date().toISOString()
        });
      }
    }
  }
}

// Run every 5 minutes
setInterval(processDLQWebhooks, 5 * 60 * 1000);

Monitoring Dashboard

// dlq-dashboard.js - Express route
app.get('/api/dlq/stats', async (req, res) => {
  const stats = await pool.query(`
    SELECT
      status,
      COUNT(*) as count,
      MIN(first_failed_at) as oldest,
      MAX(first_failed_at) as newest
    FROM dlq_webhooks
    GROUP BY status
  `);

  const byTopic = await pool.query(`
    SELECT
      topic,
      COUNT(*) as count
    FROM dlq_webhooks
    WHERE status = 'pending'
    GROUP BY topic
    ORDER BY count DESC
  `);

  res.json({
    stats: stats.rows,
    byTopic: byTopic.rows
  });
});

Alerting Setup

// alerting.js
const { sendSlackAlert } = require('./slack');

async function alertDLQEntry(webhookId, topic, shopDomain) {
  // Check DLQ depth
  const depth = await getDLQDepth();

  if (depth > 100) {
    await sendSlackAlert({
      channel: '#alerts-critical',
      text: `🚨 DLQ depth exceeds 100: ${depth} webhooks`,
      priority: 'critical'
    });
  }
}

async function alertPermanentFailure(webhook) {
  await sendSlackAlert({
    channel: '#alerts-high',
    text: `⚠️ Webhook permanently failed after 10 retries\nTopic: ${webhook.topic}\nShop: ${webhook.shop_domain}\nWebhook ID: ${webhook.webhook_id}`,
    priority: 'high'
  });
}

Testing Your DLQ

// test-dlq.js
describe('Dead Letter Queue', () => {
  it('should save webhook to DLQ on processing failure', async () => {
    const testWebhook = {
      webhookId: 'test-123',
      topic: 'orders/create',
      shopDomain: 'test-shop.myshopify.com',
      payload: { id: 123, total: 50.00 },
      error: 'Database connection failed',
      stackTrace: 'Error: ...',
      retryCount: 3
    };

    await saveToDLQ(testWebhook);

    const dlqWebhooks = await getDLQWebhooks('pending');
    expect(dlqWebhooks).toHaveLength(1);
    expect(dlqWebhooks[0].webhook_id).toBe('test-123');
  });

  it('should retry webhook from DLQ', async () => {
    // Test retry logic
  });

  it('should mark webhook as abandoned after max retries', async () => {
    // Test abandonment logic
  });
});

Production Considerations

Idempotency

Store processed webhook IDs to prevent duplicate processing:

async function processWebhook(topic, payload, shopDomain) {
  const webhookId = payload.id || payload.admin_graphql_api_id;

  // Check if already processed
  const alreadyProcessed = await redis.get(`processed:${webhookId}`);
  if (alreadyProcessed) {
    console.log(`Webhook ${webhookId} already processed, skipping`);
    return;
  }

  // Process webhook
  await yourBusinessLogic(payload);

  // Mark as processed (expire after 7 days)
  await redis.setex(`processed:${webhookId}`, 7 * 24 * 60 * 60, '1');
}

Exponential Backoff Configuration

const RETRY_CONFIG = {
  attempts: 6,
  backoff: {
    type: 'exponential',
    delay: 10000, // 10 seconds
    options: {
      maxDelay: 1800000, // 30 minutes max
    }
  }
};

// Results in retry schedule:
// 1. 10 seconds
// 2. 1 minute
// 3. 5 minutes
// 4. 15 minutes
// 5. 30 minutes
// 6. 30 minutes

Memory Management

For high-volume stores:

// Process DLQ in batches
async function processDLQBatch(batchSize = 50) {
  const webhooks = await getDLQWebhooks('pending', batchSize);

  // Process in parallel with concurrency limit
  await Promise.all(
    webhooks.map(webhook => 
      limiter.schedule(() => retryWebhook(webhook))
    )
  );
}

// Use p-limit for concurrency control
const pLimit = require('p-limit');
const limiter = pLimit(10); // Max 10 concurrent

Common Mistakes to Avoid

❌ Processing webhooks synchronously in the HTTP handler

// DON'T DO THIS
app.post('/webhook', async (req, res) => {
  await processWebhook(req.body); // Blocks!
  res.status(200).send('OK'); // Too late, Shopify timed out
});

❌ No maximum retry limit

// DON'T DO THIS
while (!success) {
  await retry(); // Infinite loop
}

❌ Not categorizing errors

// DON'T DO THIS
catch (error) {
  throw error; // Always retry, even for permanent failures
}

Monitoring Checklist

Monitor these metrics:

✅ DLQ depth (alert > 100)
✅ Age of oldest message (alert > 4 hours)
✅ Retry success rate
✅ Error patterns (group by error type)
✅ Processing latency (p50, p95, p99)
✅ Webhook topics in DLQ (which events fail most)

Resources

Full Implementation

The complete working implementation with all the code above is available on our blog:
👉 Dead Letter Queue Shopify Webhooks: Complete Guide

What's your approach to webhook reliability? Drop your setup in the comments below.

Building Shopify integrations? We help e-commerce brands build fault-tolerant systems. Check out Kolachi Tech for more technical deep-dives.

Scalable Shopify Integration Patterns Every Developer Should Know

Muhammad Masad Ashraf — Thu, 14 May 2026 22:38:17 +0000

Your Shopify integration works great. Until it does not.

You hit a traffic spike. Webhooks start failing. Your API quota drains in seconds. Orders process out of order. Data between your store and your warehouse drifts further apart by the minute.

This is not a bug. It is an architecture problem. And it has well-understood solutions.

This post covers the core scalable Shopify integration patterns I see used in high-volume production systems, with practical notes on when and how to apply each one.

The Classic Failure Stack

Before the patterns, here is what going wrong looks like in practice:

Shopify fires webhook
  -> Your handler receives it
  -> Handler calls ERP API (2s)
  -> Handler updates database (1s)
  -> Handler calls shipping provider (3s)
  -> Total: 6s
  -> Shopify timeout: 5s
  -> Shopify marks delivery as FAILED
  -> Shopify retries
  -> You process the same order twice
  -> Inventory is now wrong

Every step in that chain is fixable. Here is how.

Pattern 1: Queue-Based Webhook Processing

The rule: Never do real work inside a webhook handler.

POST /webhooks/orders/create
  -> Parse and validate HMAC     (fast)
  -> Push payload to queue       (fast)
  -> Return 200                  (done)

Queue Worker:
  -> Pull job from queue
  -> Process order
  -> Call ERP, update DB, notify warehouse

Tools that work well here: BullMQ (Node.js), Sidekiq (Ruby), Celery (Python), SQS (any language).

The handler should complete in under 500ms. Everything else belongs in a worker.

Pattern 2: Idempotency Keys

The rule: Processing the same event twice should produce the same result as processing it once.

async function processOrder(webhookPayload) {
  const idempotencyKey = `order-created-${webhookPayload.id}`;

  const alreadyProcessed = await redis.get(idempotencyKey);
  if (alreadyProcessed) {
    return; // skip, already handled
  }

  await doTheActualWork(webhookPayload);

  await redis.set(idempotencyKey, '1', 'EX', 86400); // expire after 24h
}

Without this, any retry from Shopify or your own queue creates duplicates. With it, retries are safe by default.

Pattern 3: Caching Shopify API Responses

The rule: Do not call Shopify on every request for data that rarely changes.

async function getProduct(productId) {
  const cacheKey = `shopify:product:${productId}`;

  const cached = await redis.get(cacheKey);
  if (cached) return JSON.parse(cached);

  const product = await shopify.product.get(productId);
  await redis.set(cacheKey, JSON.stringify(product), 'EX', 300); // 5 min TTL

  return product;
}

// Invalidate when Shopify tells you the data changed
app.post('/webhooks/products/update', async (req, res) => {
  const { id } = req.body;
  await redis.del(`shopify:product:${id}`);
  await queue.add('sync-product', { productId: id });
  res.sendStatus(200);
});

Cache product data, metafields, and store config. Do not cache inventory or order status.

Pattern 4: Event-Driven Architecture

Instead of services calling each other directly, they emit and consume events.

Shopify
  -> fires "order/created" webhook
    -> Order Service processes order
      -> emits "order.confirmed" event
        -> Inventory Service decrements stock
        -> Notification Service sends confirmation email
        -> Analytics Service logs the conversion

Each service is independent. Each can fail and recover without affecting the others. New integrations subscribe to existing events without touching existing code.

Message brokers that work well: RabbitMQ, Kafka, AWS EventBridge, Google Pub/Sub.

Pattern 5: Async Processing for Slow Operations

For anything that takes more than a second or two, go async.

POST /api/bulk-import
  -> Validate the request
  -> Create a job record with status: "pending"
  -> Push to queue
  -> Return { jobId: "abc123", status: "pending" }

GET /api/jobs/abc123
  -> Return { jobId: "abc123", status: "processing", progress: 42 }

// or use a webhook to notify when done

This pattern applies to: bulk product imports, inventory reconciliation, report generation, large sync jobs.

Pattern 6: Circuit Breakers for External Dependencies

When a downstream service is failing, stop hammering it. Use a circuit breaker.

const breaker = new CircuitBreaker(callShippingProvider, {
  timeout: 3000,
  errorThresholdPercentage: 50,
  resetTimeout: 30000
});

breaker.fallback(() => ({ status: 'queued', message: 'Will retry shortly' }));

const result = await breaker.fire(orderData);

Libraries: opossum (Node.js), resilience4j (Java), pybreaker (Python).

Circuit breakers prevent one failing dependency from cascading into a full system outage.

Pattern 7: Multi-Service by Domain

Split your integration by business domain, not by technical layer.

Instead of:
  integration-monolith/
    orders.js
    inventory.js
    shipping.js
    notifications.js

Do this:
  order-service/         <- owns order lifecycle
  inventory-service/     <- owns stock levels
  shipping-service/      <- owns carrier integrations
  notification-service/  <- owns all outbound messaging

Each service:

Has its own database
Scales independently
Deploys independently
Fails in isolation

When to Apply Each Pattern

Pattern	Start applying at...
Queue-based webhooks	Day 1, always
Idempotency keys	Day 1, always
Caching layer	When API errors appear
Async architecture	When handlers exceed 2s
Event-driven design	When services start coupling
Circuit breakers	When you have 3+ external dependencies
Multi-service split	When one part needs to scale differently

Quick Wins Checklist

[ ] Webhook handler returns 200 in under 500ms
[ ] Every job has an idempotency key check
[ ] Frequently read data is cached in Redis
[ ] Slow operations run in background workers
[ ] Retry logic exists for all external API calls
[ ] Dead letter queue captures persistently failed jobs
[ ] At least one circuit breaker on a critical dependency

Final Thought

Scalable Shopify integration patterns are not about adding complexity for its own sake. They are about removing fragility before it costs you customers, revenue, or a 3am incident.

Start with the queue and idempotency. Everything else can follow as the system demands it.

Full guide with infrastructure breakdowns: kolachitech.com

Drop questions or your own pattern war stories in the comments.

Multi-Service Architecture for Shopify Ecosystems

Muhammad Masad Ashraf — Wed, 13 May 2026 18:45:27 +0000

Introduction

Most Shopify stores start simple. One theme, a few apps, the Admin API, and a payment provider.

Then growth happens.

You add a custom fulfillment service. Then an ERP sync. Then a loyalty engine. Then a multi-warehouse inventory system. Before long, you are managing a dozen interconnected systems, all talking to Shopify in different ways, at different times, with different failure modes.

That is not a simple store anymore. That is a distributed ecosystem. And without a deliberate architecture, it becomes a maintenance nightmare.

Multi-service Shopify architecture is the discipline of designing that ecosystem intentionally. It gives each service a clear responsibility, a defined communication contract, and a path to scale independently without breaking everything else.

This guide walks you through the core concepts, patterns, and practical decisions you need to build a Shopify ecosystem that grows with your business.

What Is Multi-Service Architecture in the Shopify Context?

Multi-service architecture means splitting your Shopify ecosystem into distinct, independently deployable services. Each service owns a specific domain and communicates with other services through well-defined interfaces.

In a Shopify ecosystem, services typically map to business domains:

Service	Responsibility
Order Service	Receives and processes Shopify order events
Inventory Service	Syncs stock levels across warehouses and Shopify
Fulfillment Service	Manages pick, pack, and ship workflows
Notification Service	Sends email, SMS, and push notifications
Analytics Service	Aggregates and stores business metrics
Customer Service	Manages customer profiles and loyalty data
Pricing Service	Handles discount rules, pricing tiers, and promotions

Each service runs independently. A failure in the Notification Service should not take down order processing. A spike in analytics events should not slow down fulfillment.

That isolation is the primary value of multi-service Shopify architecture.

Monolith vs. Multi-Service: When to Make the Switch

Not every Shopify store needs a multi-service architecture. Starting with a monolith is often the right call.

Factor	Monolith	Multi-Service
Team size	1 to 3 developers	4 or more developers
Throughput	Low to moderate	High or unpredictable
Domain complexity	Simple	Multiple distinct domains
Deployment frequency	Infrequent	Independent per service
Failure tolerance	One service can fail and be acceptable	Individual failures must not cascade
Scaling needs	Scale the whole app	Scale individual services

The signal to move toward multi-service architecture is usually one of three things: your monolith is getting hard to deploy without breaking things, a single domain needs to scale faster than the rest, or different teams need to work independently without stepping on each other.

The Core Patterns of Shopify SOA

1. Domain-Driven Service Boundaries

Each service should own a single business domain. Do not split services by technical layer (database service, API service). Split them by what they do for the business.

Good boundaries follow natural seams in your data. The Order Service owns order records. The Inventory Service owns stock counts. Neither reaches into the other's database directly.

When services need shared data, they communicate through APIs or events. They never share a database table. This boundary enforcement is what makes services independently deployable.

2. The API Gateway Pattern

In a multi-service ecosystem, you do not expose every internal service directly to Shopify or external consumers. You use an API gateway as the single entry point.

The gateway handles:

Request routing to the correct upstream service
Authentication and rate limiting
Response aggregation from multiple services
Protocol translation (REST to gRPC, for example)

Shopify webhooks hit the gateway first. The gateway validates the HMAC signature, identifies the event type, and routes it to the correct internal service. No individual service needs to handle authentication logic.

Pairing the API gateway with Shopify webhooks creates a clean, centralized entry point for all Shopify-originated traffic.

3. Event-Driven Communication

Services in a Shopify ecosystem should communicate through events wherever possible. Events decouple services from each other. The Order Service does not need to know the Notification Service exists. It just fires an order.paid event. Any service that cares about paid orders subscribes to it.

This is the foundation of event-driven architecture for Shopify apps. It keeps services loosely coupled, independently deployable, and easy to extend.

Add a new service? Subscribe it to the relevant events. Remove a service? Other services continue without any changes.

4. Synchronous vs. Asynchronous Communication

Not all inter-service communication should use events. Some operations need an immediate response.

Use these as guidelines:

Pattern	When to Use	Example
Synchronous (REST/GraphQL)	Need an immediate response	Pricing service returning a discount value at checkout
Asynchronous (events/queues)	Fire-and-forget, no immediate response needed	Triggering a fulfillment job after order payment
Request-reply via queue	Async but need confirmation	Confirming inventory reservation

For async workflows, build on queue-based processing so events are durable, retryable, and independent of service uptime.

Service Orchestration vs. Choreography in Shopify

These are two distinct approaches to coordinating work across services. Both apply in Shopify ecosystems.

Orchestration

One central service (the orchestrator) coordinates the workflow. It calls services in sequence and handles the results.

Example: The Order Service orchestrates the post-purchase workflow. It calls the Inventory Service to reserve stock, calls the Fulfillment Service to create a shipment, then calls the Notification Service to send a confirmation.

Pros: Easy to trace and debug. The workflow logic lives in one place.

Cons: The orchestrator becomes a bottleneck and a single point of failure.

Choreography

Services react to events autonomously. No central coordinator. Each service does its job when it sees the right event.

Example: The Order Service publishes order.paid. The Inventory Service hears it and deducts stock. The Fulfillment Service hears it and creates a shipment. The Notification Service hears it and sends a confirmation. All in parallel, none aware of the others.

Pros: Highly decoupled and scalable.

Cons: Harder to trace. Business logic is spread across services.

Most mature Shopify ecosystems use both. Orchestration for complex workflows that need guaranteed sequencing. Choreography for high-volume, loosely coupled event processing.

For the async side of this, see our deep dive on async Shopify architectures.

Shopify as the Source of Truth

In a multi-service ecosystem, every service eventually needs to agree on what happened in Shopify.

Shopify is the source of truth for orders, products, customers, and inventory. Your internal services hold derived or cached copies of that data for their own processing needs.

This has two important implications:

First, when services conflict, Shopify wins. Your Analytics Service may have a different order count than Shopify for a few minutes during a sync. Shopify's count is correct.

Second, your services must be able to rebuild their local state from Shopify if something goes wrong. Design every service with a re-sync capability that pulls current state from the Shopify Admin API or Shopify GraphQL API on demand.

This also means your services need to handle distributed Shopify inventory sync gracefully, with eventual consistency baked in rather than bolted on after the fact.

Building Resilient Services in a Shopify Ecosystem

Each individual service must be built to fail gracefully. In a multi-service architecture, external dependencies will be unavailable sometimes. A service that crashes when a downstream call fails will create cascading failures.

Circuit Breakers

A circuit breaker monitors calls to a downstream service. If failures cross a threshold, it opens the circuit and stops sending requests for a set period. This prevents your service from waiting on a dead dependency.

Retries with Idempotency

Retries are essential. But retries without idempotency create duplicate operations. Every write operation in every service must be idempotent. Passing the same request twice must produce the same result.

For Shopify-specific patterns, our guide on idempotency strategies in Shopify systems covers the implementation in detail.

Timeouts

Every synchronous call between services must have a timeout. Never wait indefinitely for a response. Set aggressive timeouts and handle the timeout case explicitly.

Without timeouts, one slow service can exhaust the thread pool of every service that calls it.

Bulkhead Isolation

Partition your service's resources so one failing integration does not consume all available capacity. Run your Shopify webhook handlers on a separate thread pool from your ERP sync jobs. Isolate your database connection pools per domain.

For a complete treatment of these patterns in the Shopify context, read our guide on fault-tolerant Shopify integration.

Data Management Across Services

Data management is the hardest part of multi-service Shopify architecture.

One Database Per Service

Each service owns its own data store. No shared databases. This is not optional if you want true service independence.

The immediate objection is always: how do you join data across services? You do not join across service boundaries. You use APIs or events to compose data.

If the Order Service needs the customer's loyalty tier from the Customer Service, it makes an API call at runtime or subscribes to customer.tier_updated events to keep a local cache.

Caching for Cross-Service Data

Frequently read, rarely updated data from other services can be cached locally. The Pricing Service's discount rules do not change every second. The Fulfillment Service can cache them with a short TTL.

This reduces inter-service API calls and improves resilience. If the Pricing Service is temporarily unavailable, the Fulfillment Service can continue with cached rules.

Build your caching strategy on solid foundations. Our guide on Shopify caching layers explains the patterns that work at scale.

Sagas for Distributed Transactions

In a relational database, you use transactions to keep operations atomic. In a multi-service ecosystem, you cannot do a transaction across service boundaries.

You use sagas instead.

A saga is a sequence of local transactions, each publishing an event that triggers the next step. If any step fails, compensating transactions undo the previous steps.

Example: Processing a Shopify order across services.

Step	Action	Compensating Action
1	Reserve inventory in Inventory Service	Release reservation
2	Create shipment in Fulfillment Service	Cancel shipment
3	Charge payment via Payment Service	Refund payment
4	Send confirmation via Notification Service	Send cancellation

If Step 3 fails, the saga executes the compensating actions for Steps 1 and 2 automatically. No manual cleanup required.

Scaling Individual Services

One of the biggest advantages of multi-service architecture is independent scaling.

Your Inventory Service may need ten instances during a flash sale but only one during off-peak hours. Your Analytics Service may need to scale horizontally to handle event ingestion volume. Your Notification Service needs to handle bursts without blocking order processing.

Scale each service based on its own load profile. Do not scale everything together.

For Shopify-specific scaling challenges, our post on scaling Shopify apps to millions of requests covers the infrastructure decisions in detail.

Pair independent scaling with load balancing strategies to distribute traffic evenly across service instances and avoid hot spots.

Service Communication Contracts

Services talk to each other through APIs and events. Those interfaces are contracts. Breaking a contract breaks every service that depends on it.

Follow these rules for managing contracts:

Version your APIs. Never change an existing endpoint in a breaking way. Add a new version instead. Run both in parallel until all consumers migrate.

Schema your events. Define a schema for every event your service publishes. Consumers rely on that schema. Changing it without notice breaks them silently.

Publish your contracts. Use OpenAPI specs for REST APIs and JSON Schema or Avro for event payloads. Keep them in a shared repository all teams can access.

Test contracts explicitly. Use consumer-driven contract tests to verify that a service's API or event schema still satisfies every consumer's expectations before deploying a change.

This discipline prevents the integration failures that make multi-service architectures hard to maintain.

Observability in a Multi-Service Shopify Ecosystem

You cannot debug a distributed system the way you debug a monolith. A request that touches five services needs end-to-end tracing to diagnose failures.

Build these three pillars of observability into every service from day one:

Distributed tracing. Assign a trace ID to every Shopify event when it enters your system. Pass it through every service call so you can reconstruct the full path of any request across all services.

Structured logging. Every service logs structured JSON with the trace ID, service name, event type, and outcome. This makes logs searchable and correlatable across services.

Metrics. Each service exposes metrics on latency, error rate, throughput, and queue depth. Aggregate them in a central dashboard so you can see the health of the entire ecosystem at a glance.

The middleware layer is often where observability is most valuable. Our guide on resilient Shopify middleware covers how to instrument the integration layer effectively.

A Reference Architecture for Shopify Ecosystems

Here is a reference architecture that brings all these patterns together:

Shopify Platform
      |
      | (Webhooks + API calls)
      v
  API Gateway
  (Auth, routing, rate limiting)
      |
      +---> Order Service -------> Message Bus <------- Inventory Service
      |           |                     |                      |
      |           v                     |                      v
      |     Local DB (orders)           |              Local DB (inventory)
      |                                 |
      +---> Analytics Service <---------+
      |                                 |
      +---> Notification Service <------+
      |                                 |
      +---> Fulfillment Service <-------+
                  |
                  v
          Local DB (shipments)
                  |
                  v
         3PL / Warehouse API

Every service has its own database. All async communication flows through the message bus. Shopify enters the ecosystem through the API gateway only. Each service can be scaled, deployed, and restarted independently.

Summary

Multi-service Shopify architecture is not about adding complexity for its own sake.

It is about giving each part of your ecosystem the freedom to evolve, scale, and fail independently. When done well, it makes large Shopify ecosystems easier to maintain, easier to extend, and far more resilient under load.

Start with clear domain boundaries. Define communication contracts before writing code. Build resilience patterns into every service. Instrument everything from day one.

The investment upfront pays back every time you need to scale a single service, onboard a new team, or swap out an integration without touching the rest of the system.

Frequently Asked Questions

Q: What is multi-service architecture for Shopify?
It is a design approach where your Shopify ecosystem is split into independently deployable services, each owning a specific business domain like orders, inventory, or fulfillment, and communicating through APIs or events.

Q: When should I switch from a monolith to multi-service Shopify architecture?
When your codebase is hard to deploy without breaking things, when different domains need to scale independently, or when separate teams need to work without interfering with each other.

Q: What is the difference between service orchestration and choreography in Shopify?
Orchestration uses a central service to coordinate workflows in sequence. Choreography lets services react to events autonomously without a central coordinator. Most Shopify ecosystems use both depending on the workflow complexity.

Q: How do services share data without a shared database?
Through API calls at runtime, event-driven data propagation, or local caches of frequently read data from other services. Direct cross-service database queries are not permitted in multi-service architecture.

Q: How do I handle distributed transactions across Shopify services?
Use the saga pattern. Each step in the transaction publishes an event triggering the next step. Failed steps trigger compensating actions to undo previous steps automatically.

Q: Does Shopify support multi-service architecture natively?
Shopify provides webhooks, the Admin API, and the GraphQL API as integration points. Multi-service architecture is an application-level pattern you implement in your own infrastructure using those Shopify tools as the integration layer.

Published by KolachiTech — Shopify development specialists.

Handling Eventual Consistency in Shopify Integrations

Muhammad Masad Ashraf — Mon, 11 May 2026 19:28:11 +0000

If you have built a Shopify integration before, you have hit this wall.

Your warehouse system shows 50 units. Shopify shows 45. An order comes in for 10 units. Now you have a problem you did not plan for.

This is eventual consistency at work. And it is one of the most common sources of production bugs in Shopify-connected systems.

This post walks through why it happens and exactly how to handle it.

What Eventual Consistency Actually Means

Eventual consistency means all systems will reflect the same data state eventually, but not at the same instant.

It sits at the opposite end of the spectrum from strong consistency, where every read returns the latest write. Strong consistency requires locking and synchronous coordination. That works fine on a single database. It does not scale to distributed commerce infrastructure serving millions of merchants.

Shopify prioritizes availability and performance over strict consistency. That is the right call for a platform at its scale. But it shifts the responsibility of handling consistency onto you, the integration developer.

Why This Happens in Shopify Integrations

There are five common causes:

Asynchronous webhooks
Shopify webhooks are fire-and-forget. There is no guaranteed delivery order. Retries create duplicates. If your endpoint is down, you miss events entirely.

API rate limits
When your sync job hits rate limits, some records update later than others. You end up with a window where data is partially synced.

Multiple integration points
Your store likely connects to an ERP, CRM, warehouse system, and marketing platform. Each one processes at a different speed and fails in different ways.

Network partitions
Any service in your stack can fail mid-sync. When it recovers, it has to reconcile against a Shopify state that moved on without it.

Concurrent writes
Two systems write to the same resource at the same time. Without conflict detection, the last write wins, which may not be the correct one.

The Strategies That Actually Work

1. Make Every Handler Idempotent

This is the most important one. If you do nothing else, do this.

When Shopify retries a webhook, your handler will receive the same event twice. Without idempotency, you process the order twice, deduct inventory twice, or send the confirmation email twice.

The fix is simple. Store a deduplication key before processing:

const dedupKey = `${topic}:${payload.id}`;
const alreadyProcessed = await db.events.findOne({ key: dedupKey });

if (alreadyProcessed) return res.sendStatus(200);

await db.events.insert({ key: dedupKey });
processEvent(payload);

Use the Shopify resource ID combined with the event topic as your key. Check it before doing any work. Write it as part of the same transaction if your database supports it.

2. Never Process Webhooks Synchronously

Your webhook endpoint should do one thing: accept the payload and return 200.

Processing the event inside the HTTP handler means any slowness or failure in your downstream systems causes Shopify to mark your endpoint as failed and retry, creating more duplicates.

Instead, push to a queue immediately:

app.post('/webhooks/orders-create', async (req, res) => {
  await queue.push({
    topic: req.headers['x-shopify-topic'],
    payload: req.body
  });
  res.sendStatus(200);
});

Your worker picks it up, processes it at its own pace, and retries independently from your HTTP layer.

3. Use Exponential Backoff with Jitter on Retries

When a downstream system is unavailable, retrying immediately makes things worse. All your queued workers collide on the same recovering service.

Use exponential backoff:

const delay = Math.min(
  BASE_DELAY * Math.pow(2, attemptNumber) + Math.random() * 1000,
  MAX_DELAY
);

The jitter (random component) spreads retries across a time window so workers do not all hit the service at the same second.

Always set a dead-letter queue for events that exhaust all retries. You need to know what failed and why.

4. Define Source of Truth Per Data Domain

Most multi-system conflicts come from two services both believing they own the same data.

Be explicit about ownership:

Data Domain	Source of Truth
Product catalog	Shopify or PIM
Inventory levels	Warehouse Management System
Order status	Shopify
Customer profiles	CRM
Fulfillment status	3PL or OMS

Once ownership is defined, all other systems become consumers. They read from the source of truth. They do not write back to it. This eliminates an entire category of conflicts.

5. Add a Polling Fallback Alongside Webhooks

Webhooks miss events. It happens. Endpoints go down, Shopify retries expire, network conditions cause silent failures.

Pair every webhook subscription with a scheduled polling job. The job fetches records where updated_at is greater than the last successful sync timestamp:

query {
  orders(first: 50, query: "updated_at:>2026-05-12T10:00:00Z") {
    edges {
      node {
        id
        updatedAt
        fulfillmentStatus
      }
    }
  }
}

Shopify's GraphQL API makes this efficient with cursor-based pagination and filtered queries. This job is your safety net for everything the webhook layer missed.

6. Use Optimistic Concurrency Control for Writes

When two systems write to the same resource, you need conflict detection.

Optimistic concurrency control works like this:

Read the resource and note the current version (use updated_at from Shopify)
Make your changes locally
Before writing, assert the version has not changed
If it has changed, re-fetch and retry

This prevents silent overwrites where a stale write replaces a more recent update from another system.

7. Set Freshness SLAs Per Data Type

Not all data needs to be equally fresh. Setting explicit staleness windows lets you make smarter sync decisions:

Data Type	Acceptable Staleness	Sync Method
Inventory counts	30-60 seconds	Webhook + polling fallback
Order status	Near real-time	Webhook with queue
Product descriptions	5-15 minutes	Scheduled batch
Customer loyalty points	1-2 minutes	Event-driven
Pricing rules	Near real-time	Webhook + cache invalidation

Once you have these defined, you can size your infrastructure appropriately instead of treating everything as equally urgent.

Monitoring Consistency Issues

Visibility is as important as the fix. Build these in from day one:

Lag metrics — Measure the time between a Shopify event firing and your system completing the action. Alert when it exceeds your SLA.

DLQ depth alerts — A dead-letter queue with growing depth means something is consistently failing. Treat it like an on-call alert.

Reconciliation jobs — Run periodic jobs that compare your local state against Shopify's current state. Log every discrepancy. Patterns reveal systemic problems.

Correlation IDs — Attach a unique ID to every event on ingestion and propagate it across all services. This lets you trace a single order through your entire stack.

Multi-Store Complexity

If you run multiple Shopify stores, consistency challenges multiply.

The standard approach:

Use a centralized sync coordinator that receives events from all stores and fans them out to downstream systems
Apply a global inventory buffer to absorb sync lag during peak periods
Use two-phase confirmation for high-value actions: soft-hold the resource first, confirm after all systems acknowledge

Quick Reference

Pattern	Problem It Solves
Idempotent handlers	Duplicate webhook delivery
Queue-based ingestion	Synchronous processing failures
Exponential backoff + jitter	Retry storms after outages
Source-of-truth ownership	Conflicting cross-system writes
Polling fallback	Silently missed webhook events
Optimistic concurrency	Stale write overwrites recent update
DLQ monitoring	Silent failures going undetected

Wrapping Up

Eventual consistency is not something you fix. It is something you design for.

Start with idempotency. Move event processing off the HTTP layer. Define who owns what data. Build in polling as a fallback. Measure lag continuously.

These patterns will not eliminate inconsistency windows. But they will stop those windows from becoming customer-facing bugs or business-critical failures.

Full guide with architecture patterns for multi-store setups and fault tolerance:

Handling Eventual Consistency in Shopify Integrations

Explore proven strategies to manage Shopify sync delays, distributed consistency, and data conflicts across third-party systems.

kolachitech.com

What patterns have you found most effective for handling consistency in your Shopify integrations? Drop them in the comments.

Designing Resilient Shopify Middleware

Muhammad Masad Ashraf — Fri, 08 May 2026 21:42:34 +0000

Most Shopify middleware works fine in staging. It breaks in production, during a sale, at 11pm on a Friday.

After auditing dozens of integrations, the failures are almost always the same. Not exotic bugs. Simple architecture decisions made when the system was small and never revisited.

The Pattern That Breaks Everything

Webhook received
  → Direct API call to ERP
    → ERP times out (3s)
      → Shopify retries
        → Duplicate event processed
          → Inventory corrupted
            → Weekend on-call incident

The fix is not a try-catch block. It is replacing the pattern entirely.

Rule 1: Acknowledge the webhook in under 200ms. Process it asynchronously.
Rule 2: Write state and outbox entries in one transaction. Let a worker handle external calls.
Rule 3: Tag every event with an idempotency key before it touches anything.

Reference Architecture

[Shopify]   [ERP]   [WMS]   [Marketplaces]
    |        |       |           |
    v        v       v           v
   ----- API Gateway / Webhook Receiver -----
                     |
              [Message Broker]
                     |
   +-----------------+-----------------+
   |                 |                 |
[Transformer]   [Router]          [Audit Log]
   |                 |
   v                 v
[Domain Services: Orders, Inventory, Customers]
                     |
                     v
              [Outbound Workers]
                     |
                     v
   [Shopify Admin API]   [3rd Party APIs]

Every layer has one job.

Webhook receivers do not transform.
Transformers do not call APIs.
Workers do not own state.

Core Principles Before You Write Any Code

Principle	What It Means	Why It Matters
Loose coupling	Services talk via events, not direct calls	One slow service does not block others
Idempotency	Same event applied twice = same result	Safe to retry without double charges
Backpressure	Upstream slows when consumers lag	No queue explosions or OOM crashes
Observability	Every event is traceable end-to-end	You can actually debug incidents
Graceful degradation	Non-critical features fail soft	Core checkout never goes down

Building Blocks and Their Resilience Role

Component	Role	Resilience Pattern
API Gateway	Receive and validate inbound traffic	Rate limiting, schema validation
Webhook Receiver	Accept Shopify events	Fast 200 ACK, async processing
Message Broker	Decouple producers from consumers	Durable queues, partitioning
Transformer	Map between data formats	Pure functions, no side effects
Domain Services	Hold canonical state	Postgres with row-level locking
Outbound Workers	Push updates to external systems	Retries, circuit breakers
Audit Log	Record every event	Append-only, 30-90 day retention

Failure Modes You Must Design For

Failure Mode	Trigger	Mitigation
Shopify rate limits	Burst traffic, bulk updates	Token bucket, respect `X-Shopify-Shop-Api-Call-Limit`
Webhook retry storms	Shopify re-delivers for 48h	Idempotency keys + dedup table
Network timeouts	Flaky third-party API	Circuit breaker + exponential backoff
Bad payloads	Schema changes upstream	Strict validation + dead letter queue
Slow consumers	Heavy DB writes	Backpressure + queue partitioning
Region outage	Vendor failure	Multi-region failover + replay logs

The 5 Resilience Patterns That Actually Work

1. Circuit Breaker

Watches error rates on outbound calls. Stops calling a failing service when errors spike above threshold. Gives the dependency time to recover.

CLOSED → (error rate > 50% over 30s) → OPEN
OPEN   → (wait 60s)                  → HALF-OPEN
HALF-OPEN → (3 successful probes)    → CLOSED

2. Exponential Backoff with Jitter

Naive retries create thundering herds. Add jitter to spread the load.

function getBackoffDelay(attempt) {
  const base = Math.min(200 * Math.pow(2, attempt), 60000);
  const jitter = base * 0.2 * Math.random();
  return base + jitter;
}

// attempt 0 → ~200ms
// attempt 1 → ~400ms
// attempt 2 → ~800ms
// attempt 5 → ~6400ms (capped at 60s)

3. Bulkheads

Inventory updates run on a separate worker pool from order pushes. One clogged queue cannot block the other.

4. Timeouts on Everything

// Every outbound call needs a timeout. No exceptions.
const response = await fetch(url, {
  signal: AbortSignal.timeout(5000) // 5s for sync calls
});

5. Dead Letter Queue

After N failed retries, move the event to a DLQ.

Alert on growth
Replay after fixing root cause
Never let a bad event block the main pipeline

Idempotency: The Most Important Property

There is no exactly-once delivery.

There is only:

at-least-once delivery
plus idempotent consumers

async function processInventoryEvent(event) {
  const alreadyProcessed = await redis.get(`idem:${event.event_id}`);

  if (alreadyProcessed) return; // skip duplicate

  await db.transaction(async (trx) => {
    await trx('inventory')
      .update({ quantity: event.absolute })
      .where({
        sku: event.sku,
        location_id: event.location_id
      });

    await trx('outbox').insert({
      payload: event,
      status: 'pending'
    });
  });

  await redis.setex(
    `idem:${event.event_id}`,
    604800,
    '1'
  ); // 7-day TTL
}

The Outbox Pattern: One Rule That Eliminates Dual-Write Bugs

When you change internal state and need to notify an external system, never do both in the same step.

WITHOUT OUTBOX (dangerous)

1. UPDATE inventory SET quantity = 42
2. POST /shopify/inventory_levels
   (fails? state wrong. success? maybe duplicate)

WITH OUTBOX (safe)

1. BEGIN TRANSACTION
   UPDATE inventory SET quantity = 42
   INSERT INTO outbox (payload, status) VALUES (...)
2. COMMIT
3. Separate worker polls outbox
   → calls Shopify
   → marks row done

If Shopify is slow, the row waits.

If the worker crashes, it restarts and retries.

Internal state is always consistent.

Distributed Shopify Inventory Sync as a Real Example

Step	What Middleware Does
1	Receives inventory webhook from Shopify
2	Returns 200 immediately, publishes to broker
3	Transformer converts to canonical event schema
4	Inventory service applies delta with optimistic concurrency
5	Outbox records pending downstream push
6	Worker pushes to ERP, WMS, and marketplace channels
7	Reconciler validates full state every 5 minutes

Observability Baseline

You cannot operate what you cannot see.

Four pillars, from day one:

Pillar	Tool Options	Key Question Answered
Logs	Datadog, ELK, Loki	What happened to event X?
Metrics	Prometheus, CloudWatch	Is queue lag growing?
Traces	OpenTelemetry, Jaeger	Where did request Y spend time?
Alerts	PagerDuty, Opsgenie	What needs attention right now?

Metrics that catch the most incidents

Queue consumer lag (alert at > 30s)
Dead letter queue depth (alert at > 100 unprocessed)
Outbound API error rate (alert at > 2%)
Webhook 5xx rate (alert at > 0.5%)
DB connection pool saturation (alert at > 80%)

Race Condition Handling

Two webhooks for the same SKU arrive at the same millisecond.

Without protection, you lose one.

Option 1: Partition by key (preferred)

Hash SKU to a stream partition.

All events for one SKU process in order on one consumer.

Option 2: Optimistic concurrency

UPDATE inventory
SET quantity = 40,
    version = version + 1
WHERE sku = 'TSHIRT-RED-M'
  AND location_id = 'wh_LA'
  AND version = 7;

Option 3: Distributed locks (last resort)

Serialize via Redis or ZooKeeper.

Works, but adds latency and contention.

Common Mistakes That Cost Merchants Real Money

Calling Shopify or ERP APIs directly from webhook handlers
Retries with no backoff (thundering herd on every incident)
Infinite retries (burns rate limit quota silently)
Logs without trace IDs (cannot debug incidents after the fact)
No dead letter queue (one bad event blocks the whole pipeline)
Ignoring Shopify schema changes in API responses

Build vs Buy

Scenario	Recommendation
Simple ERP sync, low volume	Use a connector (Celigo, Pipe17)
Custom flows, mid volume	Hybrid: connector + custom services
Complex multichannel, high volume	Build a custom middleware platform
Regulated industry	Build with audit trail baked in

The Full Architecture Guide

This post covers the core patterns.

The full guide on our blog goes deeper with:

Complete component-by-component breakdown
Database schema design notes
Serverless vs container trade-offs
REST vs GraphQL decision framework
Caching layer design
Phased implementation roadmap

👉 Read it here:

Designing Resilient Shopify Middleware

Final Thought

Resilience is not a feature you bolt on later.

It is an architectural property.

The systems that survive Black Friday are usually the systems that were designed to fail safely from day one.

Distributed Shopify Inventory Sync: Architecture Guide for Scale

Muhammad Masad Ashraf — Thu, 07 May 2026 10:20:22 +0000

Keeping inventory accurate across Shopify, warehouses, and marketplaces sounds simple. At scale, it is one of the hardest engineering problems in ecommerce.

A single API call after each sale works fine at 200 orders a day. At 20,000 concurrent transactions, it collapses.

Here is what breaks first:

Overselling when two orders hit the same SKU simultaneously
Stale counts when a warehouse update takes minutes to reflect
Silent failures when a sync call times out with no retry
Duplicate decrements when a webhook fires twice

These are predictable failure modes of monolithic sync. A distributed architecture fixes all of them.

The Four Layers You Need

1. Event Producer Layer
Captures inventory change events from Shopify webhooks, WMS, POS, and marketplaces.

2. Message Queue Layer
Events land in a durable queue (Kafka, RabbitMQ, SQS). Nothing gets lost.

3. Microservices Processing Layer
Dedicated services consume events, apply business logic, push updates downstream.

4. State Store Layer
Redis holds the current inventory truth. Shopify is updated from here asynchronously.

Each layer scales independently. Each can fail without taking down the others.

Event-Driven Is the Only Foundation That Works

Stop polling. React to events.

Shopify fires these webhooks you need to capture:

inventory_levels/update
orders/create
orders/cancelled
refunds/create

Each webhook hits your receiver, gets acknowledged immediately, then lands on a queue for async processing.

Never process a webhook synchronously inside the HTTP response window. Timeouts will cause missed events and your inventory will drift.

Microservices Breakdown

Service	Job
Webhook Receiver	Validates HMAC, publishes to queue
Order Event Consumer	Reads order events, calculates deltas
Inventory Adjuster	Applies changes with optimistic locking
Shopify Sync Service	Pushes updates via GraphQL API
WMS Connector	Bidirectional warehouse sync
Notification Service	Low-stock alerts and reorder triggers

One service, one job. Deploy and scale them independently.

Solving Concurrency: Three Patterns

Two orders. One unit left. Both read stock as 1. Both decrement. Stock hits -1.

Here is how you stop it:

Optimistic Locking
Version numbers on every record. Assert version has not changed before writing. Retry on conflict. Best for low-contention SKUs.

Pessimistic Locking
Lock before reading. One writer at a time. Slower but safe. Use during flash sales.

Atomic Counters (Recommended)
Redis DECRBY is atomic. Use Redis as your inventory counter, sync to Shopify asynchronously. Fastest and most reliable for high volume.

Fault Tolerance Checklist

Dead Letter Queue on every message queue
Exponential backoff: 1s, 2s, 4s, 8s on API retries
Idempotency keys on every sync operation
Circuit breakers to stop hammering degraded services
Correlation IDs on every event for end-to-end tracing

If you skip any of these, you will debug silent inventory drift at 2am.

Caching Strategy

Do not hit the Shopify API on every inventory read.

Use write-through caching with Redis:

Every update writes to Redis first
Shopify sync happens asynchronously
Reads always hit Redis (fast)
Webhook fires? Invalidate the cache key immediately

TTL of 30 to 60 seconds works for most inventory read patterns.

Which Queue Should You Pick?

Queue	Best For
AWS SQS	Simplest to operate, great for most stores
Apache Kafka	High volume, ordered event streams
RabbitMQ	Complex routing between services

SQS is the right default. Move to Kafka only when you need strict event ordering at millions of events per day.

Metrics to Watch

Queue lag (messages behind)
Sync latency (event to Shopify update)
DLQ message count
Inventory mismatch rate
API rate limit hits

Set alerts on DLQ growth and queue lag. These are your earliest warning signals before inventory starts drifting.

Bottom Line

A distributed Shopify inventory sync system rests on four things:

Event-driven ingestion
Queue-based async processing
Atomic counters for concurrency
Idempotent operations throughout

Get these right and oversells, stale counts, and silent sync failures become engineering history rather than daily incidents.

Originally published on KolachiTech

How Race Conditions Silently Break Shopify Orders (And How to Fix Them)

Muhammad Masad Ashraf — Wed, 06 May 2026 15:14:22 +0000

Flash sales. Product drops. Viral moments.

They're what every Shopify merchant dreams of — and exactly when race conditions turn that dream into a support ticket nightmare.

Let me break down what's actually happening, why Shopify's built-in protections aren't enough, and the specific patterns that fix it.

What Is a Race Condition in Shopify?
A race condition happens when two or more processes read and write the same resource simultaneously, and the result depends on which one finishes first.

In order processing, the classic scenario:

Customer A checks stock → 1 unit available → proceeds to checkout
Customer B checks stock → 1 unit available → proceeds to checkout
Customer A completes purchase → inventory = 0
Customer B completes purchase → inventory = -1

Both reads happened before either write completed. That gap between read and write is where race conditions live.

Why Shopify's Defaults Aren't Enough

Shopify has built-in inventory tracking and "deny checkout when out of stock" settings. For basic stores, that's fine.

It breaks down the moment you add:

Third-party fulfillment apps that write inventory via API
Custom order processing logic in private apps
Webhooks firing to external systems that write back to Shopify
Multi-location inventory with allocation logic
Headless storefronts (Hydrogen or custom) with custom cart logic

Once any of these exist, you are responsible for managing concurrency in your own code.

Fix #1: Use Atomic GraphQL Mutations

The Shopify GraphQL API's inventoryAdjustQuantities mutation applies inventory changes atomically. Instead of the dangerous read-calculate-write pattern:

❌ Read stock: 5
Calculate: 5 - 1 = 4
Write: 4
(Another process can read 5 between your read and write)

You send a delta:

mutation {
  inventoryAdjustQuantities(input: {
    reason: "correction"
    name: "available"
    changes: [{
      inventoryItemId: "gid://shopify/InventoryItem/12345"
      locationId: "gid://shopify/Location/67890"
      delta: -1
    }]
  }) {
    inventoryAdjustmentGroup {
      createdAt
      changes {
        name
        delta
      }
    }
  }
}

Shopify applies the delta at the database level. No read-before-write gap.

Fix #2: Idempotent Webhook Handlers

Shopify retries webhook delivery up to 19 times if your endpoint doesn't return a 200 OK within 5 seconds.

Without idempotency, one real order can trigger:

1. Multiple fulfillment requests
2. Duplicate customer notifications
3. Repeated inventory deductions

Every Shopify webhook payload includes an X-Shopify-Webhook-Id header. Here's the pattern:

async function handleOrderWebhook(req, res) {
  const webhookId = req.headers['x-shopify-webhook-id'];

  // 1. Check if we've already processed this webhook
  const existing = await db.webhookLog.findOne({ webhookId });

  if (existing) {
    // Already processed — return 200 and stop here
    return res.status(200).send('Already processed');
  }

  // 2. Record this webhook BEFORE processing
  await db.webhookLog.create({ webhookId, processedAt: new Date() });

  // 3. Now process your order logic safely
  await processOrder(req.body);

  return res.status(200).send('OK');
}

This ensures each logical event is processed exactly once, regardless of how many times Shopify retries.

Fix #3: Distributed Locking with Redis

For flash sales or any scenario with tight inventory, pessimistic locking prevents two processes from modifying the same resource simultaneously.

Redis SET NX EX (set if not exists, with expiry) is the standard pattern:

const redis = require('ioredis');
const client = new redis();

async function acquireLock(resourceId, ttlSeconds = 10) {
  const lockKey = `lock:product:${resourceId}`;
  const lockValue = `process-${Date.now()}-${Math.random()}`;

  // NX = only set if not exists, EX = expire after ttl seconds
  const result = await client.set(lockKey, lockValue, 'NX', 'EX', ttlSeconds);

  if (result === 'OK') {
    return lockValue; // Lock acquired
  }
  return null; // Lock already held by another process
}

async function releaseLock(resourceId, lockValue) {
  const lockKey = `lock:product:${resourceId}`;
  const current = await client.get(lockKey);

  // Only release if we own the lock
  if (current === lockValue) {
    await client.del(lockKey);
  }
}

// Usage in your order processing logic
async function processOrderWithLock(productId, orderId) {
  const lock = await acquireLock(productId);

  if (!lock) {
    throw new Error('Could not acquire lock — try again');
  }

  try {
    // Safe to read and write inventory here
    await updateInventory(productId, -1);
    await createFulfillment(orderId);
  } finally {
    await releaseLock(productId, lock);
  }
}

Fix #4: Queue-Based Order Processing

The most robust solution: eliminate the race condition entirely by serializing writes through a queue.

Webhook Arrives
↓
Push to Queue (Redis / SQS / RabbitMQ)
↓
Worker picks next job
↓
Acquire lock on resource
↓
Process order logic
↓
Release lock → Acknowledge job

With a queue, you also get:

Rate limiting writes to the Shopify API (staying within API limits)
Safe retries on failures without duplicating successful operations
Independent scaling of workers vs API layer

Testing for Race Conditions (Before Production Finds Them)
Concurrent request simulation:
# k6 load test — 100 virtual users hitting checkout simultaneously k6 run --vus 100 --duration 10s checkout-test.js

Chaos test: Deliberately delay your webhook endpoint to force retries. Count how many times your processing logic runs per order.

Database audit: Log every inventory write with the associated order ID. Duplicate entries for the same order ID = confirmed race condition.

Negative inventory monitoring:

// Run this on a cron job
async function checkNegativeInventory() {
  const products = await shopify.product.list({ limit: 250 });
  const oversold = products.filter(p =>
    p.variants.some(v => v.inventory_quantity < 0)
  );

  if (oversold.length > 0) {
    await alertSlack(`⚠️ Negative inventory detected: ${oversold.map(p => p.title).join(', ')}`);
  }
}

Where to Start

Don't try to implement everything at once. Here's the priority order:

Idempotent webhook handlers — highest impact, lowest complexity
Atomic GraphQL mutations for inventory writes — swap out read-write patterns
Distributed locking — for any custom inventory logic
Queue-based processing — as order volume grows

Race conditions aren't a Shopify problem. They're a distributed systems problem that shows up in Shopify stores. The good news is the fixes are well-understood — they just have to be deliberately implemented.

Full guide with architecture patterns and a checklist: Race conditions Shopify Orders

Shopify Idempotency Strategies: A Developer's Survival Guide

Muhammad Masad Ashraf — Tue, 05 May 2026 20:11:23 +0000

You built a Shopify integration. It works great. Until it doesn't.

A customer's network drops right after the checkout POST fires. They refresh. The browser retries. Shopify receives two identical order creation requests. You now have a double order, a double charge, and a very confused customer.

This is an idempotency problem. And it is more common than most Shopify developers expect.

What Is Idempotency in a Shopify Context?
An operation is idempotent if running it once or a hundred times produces the same result. In Shopify integrations, this covers:

• API mutations (order creation, fulfillment, refunds)
• Webhook processing (order.paid, inventory.update)
• Background job retries
• Third-party sync operations

Strategy 1: Use Idempotency Keys on REST API Calls
Shopify's REST Admin API accepts an Idempotency-Key header on POST requests. Generate a UUID v4 per logical operation and send it with the request.

const { v4: uuidv4 } = require('uuid');  // Generate ONCE per business event, store it const idempotencyKey = uuidv4();  await fetch('https://your-store.myshopify.com/admin/api/2024-01/orders.json', {   method: 'POST',   headers: {     'Content-Type': 'application/json',     'X-Shopify-Access-Token': process.env.SHOPIFY_TOKEN,     'Idempotency-Key': idempotencyKey, // same key on every retry   },   body: JSON.stringify(orderPayload) });

Key rule: Generate the key once. Store it before the request. Reuse it on every retry.

Strategy 2: Deduplicate Webhooks with X-Shopify-Webhook-Id
Shopify does not guarantee exactly-once webhook delivery. Every delivery includes an X-Shopify-Webhook-Id header. Use it as your deduplication key.

Express.js webhook handler

app.post('/webhooks/orders/paid', async (req, res) => {   const webhookId = req.headers['x-shopify-webhook-id'];

Check Redis for this ID (48hr TTL)

const seen = await redis.get(`webhook:${webhookId}`);   if (seen) {     return res.status(200).json({ status: 'already_processed' });   }

Mark as seen BEFORE processing

await redis.setex(`webhook:${webhookId}`, 172800, '1');    // Now safely process   await processOrderPaid(req.body);   res.status(200).json({ status: 'ok' }); });

Why store BEFORE processing? If processing fails and you stored AFTER, the next retry will be treated as a duplicate and skipped. Store the key first, then process, and let your retry logic handle actual failures separately.

Strategy 3: Safe Retries with Exponential Backoff

async function retryWithBackoff(fn, key, maxRetries = 5) {   for (let attempt = 0; attempt < maxRetries; attempt++) {     try {       return await fn(key); // always pass same key     } catch (err) {       if (attempt === maxRetries - 1) throw err;       const baseDelay = Math.pow(2, attempt) * 1000;       const jitter = Math.random() * 400 - 200;       await sleep(baseDelay + jitter);     }   } }

The jitter prevents thundering herd problems when many clients retry simultaneously after an outage.

Strategy 4: Database-Level Unique Constraints
App-level checks can fail under race conditions. Two requests can both pass your "does this exist?" check before either writes. Add a database constraint as the hard backstop.

PostgreSQL: prevent duplicate order processing

CREATE UNIQUE INDEX idx_idempotency   ON processed_events (idempotency_key);  -- Catch the constraint violation in your app try {   await db.query(     'INSERT INTO processed_events (idempotency_key, result) VALUES ($1, $2)',     [key, JSON.stringify(result)]   ); } catch (err) {   if (err.code === '23505') { // unique violation     const existing = await db.query(       'SELECT result FROM processed_events WHERE idempotency_key = $1',       [key]     );     return JSON.parse(existing.rows[0].result); // return cached result   }   throw err; }

GraphQL: No Native Header, So Use Upserts
Shopify's GraphQL API does not support the Idempotency-Key header. Design mutations to be naturally idempotent instead.

Use productUpdate (upsert) instead of productCreate where possible

mutation UpdateProductInventory($id: ID!, $qty: Int!) {   inventoryAdjustQuantity(input: {     inventoryItemId: $id     availableDelta: $qty   }) {     inventoryLevel { available }     userErrors { field message }   } }

For operations with no upsert, check for existing records first and return the existing result if found.

Quick Reference: What to Use Where

REST mutations: Idempotency-Key header + store key before request
GraphQL mutations: Upserts + client-side dedup table
Webhooks: X-Shopify-Webhook-Id + Redis with 48hr TTL
Background jobs: Job ID as idempotency key + status tracking
Database: Unique constraint on idempotency_key as final guard

Further Reading
Shopify Webhooks Deep Dive | Fault-Tolerant Shopify Integration | Queue-Based Webhook Processing

What idempotency patterns do you use in your Shopify stack?
Read more

Queue-Based Shopify Webhook Processing: Why It Matters and How to Build It

Muhammad Masad Ashraf — Fri, 01 May 2026 23:39:00 +0000

Shopify webhooks are HTTP POST requests fired on store events: orders, inventory updates, checkouts, customers. By default, your endpoint handles them synchronously. That works until traffic spikes.

The async pattern:
Shopify fires webhook → Receiver validates HMAC, pushes to queue, returns 200 → Worker pulls job, runs business logic, marks complete

Your endpoint responds in milliseconds. Your worker takes as long as it needs.

The two things that actually trip people up in production:

Idempotency. Shopify retries on failure. If your worker processes a duplicate orders/paid, you might charge a customer twice. Fix: check the webhook ID in Redis before processing. Skip if seen. Simple.

Dead letter queues. Jobs that fail all retries need somewhere to go. Log them, alert on them, reprocess them manually. Without a DLQ, failed jobs vanish and you find out from a merchant complaint.

Stack recommendations: BullMQ + Redis for Node.js, Laravel Queues for PHP, Celery for Python, SQS + Lambda for AWS-native.

Full post with code, priority queue tiers by webhook topic, and a go-live checklist: https://kolachitech.com/queue-based-shopify-webhook-processing/