Async Event Processing Architectures for Shopify — A Developer's Guide

Synchronous processing kills Shopify scalability. Every blocked operation is a timeout waiting to happen.

This guide covers the async event processing patterns that production Shopify systems actually use. We'll go from webhook receiver basics all the way to saga orchestration and event sourcing.

---

Why Async Matters for Shopify

Shopify fires webhooks for orders, fulfillments, inventory changes, customer updates, and more. If you process these synchronously inside your endpoint, you will hit timeouts, lose data, and bring down your system during traffic spikes.

The fix is simple in concept: accept fast, process later.

Your webhook endpoint acknowledges receipt in milliseconds. A background worker handles the actual business logic. This keeps your store responsive and your integrations stable.

---

Pattern 1: The Webhook Receiver

Your endpoint has one job: validate the request, push to a queue, return HTTP 200 — all within 5 seconds.

import express from 'express';
import crypto from 'crypto';
import { OrderQueue } from './queues';

const app = express();
app.use(express.raw({ type: 'application/json' }));

app.post('/webhooks/orders/create', async (req, res) => {
  // 1. Validate HMAC signature
  const hmac = req.headers['x-shopify-hmac-sha256'];
  const hash = crypto
    .createHmac('sha256', process.env.SHOPIFY_SECRET)
    .update(req.body)
    .digest('base64');

  if (hash !== hmac) {
    return res.status(401).send('Unauthorized');
  }

  // 2. Parse and enqueue
  const order = JSON.parse(req.body);
  await OrderQueue.add('process', order, {
    jobId: `order_${order.id}`, // Deduplication key
    attempts: 5,
    backoff: { type: 'exponential', delay: 2000 }
  });

  // 3. Respond immediately
  res.status(200).send('OK');
});

Never run ERP calls, inventory syncs, or email sends inside this handler. Push to queue and return.

---

Pattern 2: Idempotent Queue Workers

Shopify retries webhooks for 48 hours. You will receive duplicates. Every worker must handle this gracefully.

import { Worker } from 'bullmq';
import { redis } from './redis';
import { db } from './db';

const worker = new Worker('orders', async (job) => {
  const order = job.data;

  // Guard against duplicate processing
  const alreadyProcessed = await db.processedEvents.findUnique({
    where: { shopifyOrderId: String(order.id) }
  });

  if (alreadyProcessed) {
    console.log(`Order ${order.id} already processed. Skipping.`);
    return { skipped: true };
  }

  // Run actual business logic
  await Promise.allSettled([
    syncInventory(order),
    notifyFulfillmentTeam(order),
    updateAnalytics(order),
    syncToERP(order)
  ]);

  // Mark as processed (idempotency record)
  await db.processedEvents.create({
    data: { shopifyOrderId: String(order.id), processedAt: new Date() }
  });

  return { success: true };
}, { connection: redis });

The processedEvents table is your source of truth for deduplication. Always check it first.

---

Pattern 3: Dead Letter Queue Handling

Not all failures are transient. Some events fail because of bad data, schema mismatches, or downstream service bugs. These need to be isolated instead of retrying forever.

import { Queue, Worker, QueueEvents } from 'bullmq';

const mainQueue = new Queue('orders', { connection: redis });
const dlq = new Queue('orders-dlq', { connection: redis });

const worker = new Worker('orders', processOrder, {
  connection: redis,
});

worker.on('failed', async (job, err) => {
  if (job.attemptsMade >= job.opts.attempts) {
    console.error(`Moving job ${job.id} to DLQ`, err.message);

    await dlq.add('failed', {
      originalJob: job.data,
      error: err.message,
      failedAt: new Date().toISOString(),
      attempts: job.attemptsMade
    });
  }
});

Set up alerts on DLQ size. Any event in there needs human review.

---

Pattern 4: Out-of-Order Event Handling

Shopify makes no delivery order guarantees. A fulfillments/create webhook can arrive before orders/create. Your system must handle this.

async function handleFulfillmentCreated(event) {
  const order = await db.orders.findUnique({
    where: { shopifyId: String(event.order_id) }
  });

  // Order not yet created locally — reschedule
  if (!order) {
    console.warn(`Order ${event.order_id} not found, rescheduling fulfillment event`);
    await fulfillmentQueue.add('retry', event, {
      delay: 10000, // Wait 10 seconds before retry
      attempts: 10
    });
    return;
  }

  // Safe to proceed
  await db.fulfillments.create({
    data: {
      orderId: order.id,
      shopifyFulfillmentId: String(event.id),
      status: event.status,
      trackingNumber: event.tracking_number
    }
  });
}

---

Pattern 5: Saga Pattern for Multi-Service Coordination

When an order needs to update inventory, charge payment, and trigger fulfillment, you need transactional guarantees across services. Sagas handle this with compensation logic.

async function createOrderSaga(order) {
  const completedSteps = [];

  try {
    // Step 1
    await reserveInventory(order);
    completedSteps.push('inventory');

    // Step 2
    await chargePayment(order);
    completedSteps.push('payment');

    // Step 3
    await createFulfillment(order);
    completedSteps.push('fulfillment');

    // Step 4
    await sendOrderConfirmation(order);

    return { success: true };

  } catch (error) {
    console.error(`Saga failed at step after: ${completedSteps.join(', ')}`);

    // Compensate completed steps in reverse
    if (completedSteps.includes('payment')) {
      await refundPayment(order);
    }
    if (completedSteps.includes('inventory')) {
      await releaseInventory(order);
    }

    throw error;
  }
}

Sagas trade atomicity for availability. Each step must also be idempotent.

---

Pattern 6: Circuit Breaker for External APIs

When your ERP, WMS, or third-party API goes down, you don't want to hammer it with retries. Circuit breakers stop the calls automatically and resume when the service recovers.

import CircuitBreaker from 'opossum';

const options = {
  timeout: 5000,       // Fail after 5 seconds
  errorThresholdPercentage: 50, // Open after 50% failures
  resetTimeout: 30000  // Try again after 30 seconds
};

const erpBreaker = new CircuitBreaker(syncOrderToERP, options);

erpBreaker.fallback(async (order) => {
  // Queue for later instead of failing hard
  await retryQueue.add('erp-sync', order, { delay: 60000 });
  return { queued: true };
});

erpBreaker.on('open', () => {
  console.error('ERP circuit breaker OPEN - service is down');
});

erpBreaker.on('close', () => {
  console.log('ERP circuit breaker CLOSED - service recovered');
});

// Use in your worker
async function processOrder(order) {
  await erpBreaker.fire(order);
}

---

Queue Infrastructure Reference

Tool	Best For	Persistence	Delay Jobs	Priority
BullMQ (Redis)	Node.js apps	Yes	Yes	Yes
RabbitMQ	Multi-language	Yes	Plugin	Yes
AWS SQS	Serverless	Yes	Yes	Via FIFO
Google Pub/Sub	GCP workloads	Yes	No	No
Kafka	High-throughput streams	Yes	No	Partition-based

For most Shopify app use cases, BullMQ with Redis gives you the best balance of features and simplicity.

---

Monitoring Checklist

Track these metrics or you're flying blind:

• Queue depth — growing consistently = you need more workers
• Job processing time (p95) — slowdowns show up here first
• Failure rate — spikes indicate downstream issues
• DLQ size — any growth needs immediate investigation
• Retry count — high values suggest a systemic problem

Log every state transition with structured fields:

logger.info('order.processing.started', {
  orderId: order.id,
  jobId: job.id,
  workerId: worker.id,
  timestamp: Date.now()
});

---

Key Takeaways

• Respond to webhooks in under 5 seconds — always push to queue first
• Make every handler idempotent — duplicates are guaranteed
• Use exponential backoff — don't hammer failing services
• Send to DLQ after max retries — don't retry poisoned messages forever
• Handle out-of-order delivery — reschedule, don't crash
• Use sagas for multi-step transactions — compensate on failure
• Monitor queue depth, latency, and DLQ size continuously

---

Originally published on Kolachi Tech. Kolachi Tech specializes in Shopify development, integrations, and custom app architecture.