DEV Community: Muhammad Masad Ashraf

Shopify GraphQL Pagination: How to Handle Large Datasets Without Slowing Down Your App

Muhammad Masad Ashraf — Fri, 12 Jun 2026 18:53:55 +0000

When you build Shopify apps or integrations, pagination becomes important very quickly.

A small test store may have a few products and orders.

A real merchant store can have thousands of products, variants, orders, customers, inventory items, metafields, and fulfillment records.

You cannot fetch all of that data in one Shopify GraphQL request.

You need pagination.

More importantly, you need pagination that performs well.

Poor Shopify GraphQL pagination can create slow syncs, API throttling, timeout errors, duplicate processing, and incomplete exports.

This post explains how Shopify GraphQL pagination works and how to handle large Shopify datasets in a practical way.

What Shopify GraphQL Pagination Solves

Pagination lets your app retrieve data in smaller chunks.

Instead of asking Shopify for 50,000 products at once, your app asks for 100 or 250 products per request.

Shopify returns the data and gives your app information about the next page.

This protects your app from huge responses and protects Shopify from heavy requests.

It also gives your integration more control over retries, progress tracking, and background processing.

Shopify Uses Cursor-Based Pagination

Shopify GraphQL uses cursor-based pagination.

That means you do not request data using page numbers.

You request the next page using a cursor from the previous response.

A basic product pagination query looks like this:

query GetProducts($cursor: String) {
  products(first: 100, after: $cursor) {
    nodes {
      id
      title
      handle
      updatedAt
    }
    pageInfo {
      hasNextPage
      endCursor
    }
  }
}

The first time you run this query, pass cursor as null.

Shopify returns the first 100 products and gives you an endCursor.

Use that endCursor as the after value in the next request.

Keep doing this until hasNextPage is false.

Why Cursors Work Better Than Page Numbers

Offset pagination usually works like this:

page=1
page=2
page=3

or:

offset=5000&limit=100

This approach becomes inefficient when datasets grow.

The system may need to skip many records before returning the next page.

It also becomes less reliable when data changes while your app is reading it.

Shopify data changes constantly.

Orders are created. Products are updated. Inventory changes. Customers register.

Cursor-based pagination handles this type of data better because it continues from a known position.

Page Size Is a Performance Decision

Shopify GraphQL connections can return up to 250 resources in one page.

That does not mean you should always use first: 250.

For simple queries, 250 may work well.

Example:

products(first: 250) {
  nodes {
    id
    title
  }
}

But if your query loads nested data, 250 can become too heavy.

Example:

products(first: 250) {
  nodes {
    id
    title
    variants(first: 50) {
      nodes {
        id
        sku
        inventoryQuantity
        metafields(first: 20) {
          nodes {
            key
            value
          }
        }
      }
    }
  }
}

This query may look convenient, but it can become expensive because it fetches products, variants, inventory, and metafields together.

For complex queries, use a smaller page size.

A practical starting point:

Query Type	Page Size
Basic products	100 to 250
Products with variants	50 to 100
Orders with line items	25 to 100
Metafield-heavy queries	25 to 100
Admin UI lists	20 to 50
Background sync jobs	100 to 250

The right number depends on query cost, response size, processing time, and failure rate.

Watch GraphQL Query Cost

GraphQL performance is not only about request count.

Shopify calculates query cost.

A simple query costs less.

A nested query costs more.

If you try to fetch everything in one request, you may hit throttling or timeout issues faster.

Instead of creating one huge query, split the workflow.

For example:

Fetch product IDs and basic product fields.
Save them locally.
Fetch variants in a separate job.
Fetch metafields only when needed.
Process everything through a queue.

This keeps your queries smaller and easier to retry.

A Reliable Pagination Loop

A production pagination loop should not be a basic while loop only.

It should track progress and recover safely.

A good flow looks like this:

1. Start with cursor = null
2. Request a page
3. Process records
4. Save endCursor
5. Check hasNextPage
6. Respect throttle status
7. Retry failed requests with backoff
8. Resume from saved cursor if the job stops

You should also make processing idempotent.

For example, if you sync products into your own database, use the Shopify product ID as a unique key and upsert the record.

Do not blindly insert records.

If your worker crashes and retries the same page, idempotent logic prevents duplicate data.

Use Filters Before Pagination

Do not paginate more data than you need.

If you only need recent orders, filter by date.

Example:

query GetRecentOrders($cursor: String) {
  orders(first: 100, after: $cursor, query: "created_at:>=2026-01-01") {
    nodes {
      id
      name
      createdAt
      displayFinancialStatus
    }
    pageInfo {
      hasNextPage
      endCursor
    }
  }
}

This reduces the number of pages your app needs to process.

Filters are especially useful for incremental syncs.

Instead of syncing all products every time, sync records updated after the last successful sync.

When to Use Bulk Operations

Cursor pagination is great for many use cases.

Use it for:

Admin tables
Small exports
Search results
Incremental syncs
Recent orders
Focused data reads

But for very large exports, use Shopify Bulk Operations.

Bulk Operations work better for:

Full product catalog exports
Historical order exports
Large customer exports
ERP syncs
Large metafield audits
Reporting pipelines

With Bulk Operations, Shopify processes the job asynchronously and gives you a file to download when it is ready.

This avoids thousands of paginated API calls.

Use Webhooks for Real-Time Updates

Pagination should not replace webhooks.

Use webhooks when you need real-time updates.

Use pagination for backfills, audits, and reconciliation.

A better architecture looks like this:

Webhook receives new order
Queue stores the event
Worker fetches missing details with GraphQL
Scheduled job reconciles recent records
Bulk Operation handles full historical export

This approach keeps your app fast and reduces unnecessary API calls.

It also helps when webhook delivery is delayed or when your system needs to verify data consistency.

Common Mistakes

Here are mistakes I often see in Shopify GraphQL pagination workflows:

Mistake	Problem
Always using `first: 250`	Heavy queries may slow down
Fetching too many nested fields	Query cost increases
Not saving cursors	Failed jobs restart from zero
Ignoring throttling	API calls start failing
No retry logic	Temporary errors break syncs
No idempotency	Duplicate records appear
Using pagination for huge exports	Bulk Operations would perform better
No logs or alerts	Failures stay hidden

Most pagination issues happen because developers treat the job like a simple data loop.

For production apps, treat it like a data pipeline.

Final Thoughts

Shopify GraphQL pagination is one of the most important patterns for scalable Shopify apps.

Cursor-based pagination helps your app fetch large datasets in manageable chunks.

But good performance depends on more than cursors.

You need lean queries, smart page sizes, filters, saved cursors, retries, queues, webhook support, and Bulk Operations for very large datasets.

For small and medium workflows, cursor pagination is usually enough.

For large exports, Bulk Operations are the better option.

For real-time updates, use webhooks.

The strongest Shopify integrations combine all three.

Originally published on KolachiTech: https://kolachitech.com/shopify-graphql-pagination/

Shopify Bulk Operations API at Scale: A Practical Developer Guide

Muhammad Masad Ashraf — Thu, 11 Jun 2026 17:55:51 +0000

Every Shopify developer hits this wall eventually.

You need to export 200,000 orders. Or sync a 500K-product catalog. Or run a price update across every variant you carry. Standard GraphQL queries collapse under that pressure. Rate limits fire. Timeouts pile up. Your integration breaks.

The Shopify Bulk Operations API exists to fix this. It processes millions of records asynchronously, returns output as a downloadable JSONL file, and sidesteps the throttling that kills traditional query approaches.

Here is how it works, and how to scale it properly.

What Is the Shopify Bulk Operations API?

It is a subset of the Shopify GraphQL Admin API built for large-scale data retrieval and mutation. Instead of paginating through thousands of API calls, you submit one GraphQL operation and Shopify runs it server-side in the background.

When the job completes, Shopify generates a JSONL file available at a signed URL. Each line in the file is one resource object.

Two operation types exist:

Operation Type	What It Does	Common Use Case
`bulkOperationRunQuery`	Exports data asynchronously	Orders export, catalog dump, customer list
`bulkOperationRunMutation`	Applies mutations to a large dataset	Price updates, tag writes, metafield updates

Both follow the same lifecycle: create, poll, download.

The Operation Lifecycle

Step 1: Submit

Send a bulkOperationRunQuery mutation. Shopify queues the job and returns an operation ID with status CREATED.

mutation {
  bulkOperationRunQuery(
    query: """
    {
      products {
        edges {
          node {
            id
            title
            variants {
              edges {
                node {
                  id
                  price
                  sku
                }
              }
            }
          }
        }
      }
    }
    """
  ) {
    bulkOperation {
      id
      status
    }
    userErrors {
      field
      message
    }
  }
}

Step 2: Poll for Status

Query currentBulkOperation at intervals until status reaches COMPLETED or FAILED.

query {
  currentBulkOperation {
    id
    status
    errorCode
    createdAt
    completedAt
    objectCount
    fileSize
    url
    partialDataUrl
  }
}

Poll every 3 to 10 seconds for small jobs. For large datasets, back off to 30 to 60 seconds.

Step 3: Download and Parse

When COMPLETED, the url field contains a signed link valid for 7 days. Parse the JSONL line by line. Parent-child relationships are encoded via __parentId on child nodes.

Status reference:

Status	Meaning	Action
CREATED	Queued, not yet running	Keep polling
RUNNING	Actively processing	Keep polling
COMPLETED	Done	Download file
FAILED	Error encountered	Check `errorCode`, retry
CANCELED	Manually or auto-canceled	Resubmit if needed
CANCELING	Cancel in progress	Wait, then resubmit

Why This Beats Standard Pagination at Scale

Standard GraphQL pagination uses cursor-based after arguments. Fine for 10,000 records. For 1 million, you make thousands of API calls and burn through your query cost budget fast.

Bulk Operations bypass per-request rate limits almost entirely. Shopify does the work server-side. You wait, then download one file.

Factor	Standard Pagination	Bulk Operations API
Rate Limit Exposure	High	Very low
Max Records	Limited by throttle	Millions
Processing Model	Synchronous	Asynchronous
Error Recovery	Per-request	Job-level
Data Format	JSON in response	JSONL file download
Best For	Up to ~50K records	50K to millions

Key Constraints You Must Know

One operation per store at a time

Only one bulk operation runs per store at any moment. Submitting a second while the first runs fails immediately. In a multi-tenant app, build per-store operation locks into your job queue.

No nested mutations

Bulk mutations require a flat JSONL input file. Each line maps to one mutation call. Nested mutations inside a single bulk operation are not supported.

Stream the output file

JSONL output can reach several gigabytes. Never load it fully into memory. Use line-by-line buffered reads.

Check `partialDataUrl` on failure

When a job fails or gets canceled, Shopify may still produce a partialDataUrl containing whatever completed before failure. Always check this field. Process the partial data, then retry only the remaining records.

Bulk Mutations: The Real Power Move

Bulk mutations handle operations like price updates, tag management, and metafield writes across millions of records.

The flow:

1. Stage an upload via stagedUploadsCreate to get a signed PUT URL.

2. Upload your JSONL input file. Each line is a JSON object with variables for one mutation call.

{"input": {"id": "gid://shopify/ProductVariant/123456789", "price": "29.99"}}

3. Submit the bulk mutation referencing the staged upload URL in bulkOperationRunMutation.

Output results include a __lineNumber field so you can map every success and failure back to your input file precisely.

Scaling Patterns for Production Systems

Pattern 1: Webhook-Triggered Completion

Subscribe to BULK_OPERATIONS_FINISH instead of polling. Shopify pushes completion status and the download URL to your endpoint. Your system stays idle until Shopify calls you.

Make your webhook handler idempotent. Shopify can fire the same completion event more than once.

Pattern 2: Job Orchestration Layer

For apps managing bulk operations across hundreds or thousands of merchant stores, build a layer that:

Queues bulk operation submissions per store
Tracks the active operation ID per store in a database
Handles completions via webhooks
Retries failed jobs with exponential backoff
Logs partialDataUrl before discarding failed results

Pattern 3: Chunked Mutation Strategy

Even though bulk mutations support millions of records, split very large jobs into chunks of 50K to 100K records per operation. Smaller jobs complete faster, fail cheaper, and produce lighter output files.

Track your last successfully completed chunk in a persistent state store. On failure, resume from the checkpoint rather than reprocessing everything.

Pattern 4: JSONL Processing Pipeline

Build a streaming parser that:

Reads the output file line by line
Reconstructs parent-child relationships via __parentId
Writes records to your database or downstream system
Tracks failed lines separately for targeted retries

Use worker processes separate from your web layer for this step. Monitor memory consumption carefully on multi-gigabyte files.

Error Codes and Retry Logic

Error Code	Cause	Action
`ACCESS_DENIED`	Missing API scope	Update OAuth scopes
`INTERNAL_SERVER_ERROR`	Shopify-side failure	Retry with exponential backoff
`TIMEOUT`	Query too complex or dataset too large	Simplify query, chunk inputs
`TOO_MANY_FILE_STORAGE_REQUESTS`	Too many staged uploads in flight	Throttle upload submissions

For timeouts: simplify the query first. Remove fields your downstream system does not use. A leaner query fixes most timeouts without reducing dataset scope.

Performance Benchmarks

Dataset Size	Typical Completion Time	Approximate File Size
10,000 products	30 to 90 seconds	5 to 15 MB
100,000 orders	3 to 8 minutes	100 to 300 MB
500,000 customers	15 to 40 minutes	500 MB to 2 GB
1M+ line items	30 to 90 minutes	2 to 10 GB

These are estimates, not guarantees. Always build for the upper end of each range. Never assume a fixed completion time in production logic.

App Architecture Implications

Bulk operations shift your bottleneck. You stop hammering the API with thousands of requests. Instead, you make a few API calls, then process a large local file. The bottleneck moves from network throughput to local compute and storage I/O.

Plan accordingly:

Use worker processes separate from your web layer
Write processed results to a fast intermediate store (Redis, PostgreSQL) before your final destination
Use streaming HTTP clients that do not buffer the full response body
Alert on jobs stuck in RUNNING state beyond expected thresholds

Separate real-time event pipelines from batch bulk operation pipelines entirely. Use event-driven patterns for real-time updates. Reserve the bulk pipeline for scheduled or triggered large-batch jobs.

Wrapping Up

The Shopify Bulk Operations API is the right tool any time your data volume pushes past what paginated GraphQL can handle cleanly. The lifecycle is simple. The constraints are manageable. The architecture patterns are proven.

Build it with webhook-triggered completion, idempotent processing, chunked mutations, and streaming JSONL parsing, and you get a system that handles enterprise-scale data without breaking under load.

Originally published on KolachiTech: https://kolachitech.com/bulk-operations-api-at-scale/

KolachiTech is a Shopify-focused development agency specializing in API architecture, ERP integrations, and scalable app development. If you need a production-grade bulk operations system built for your store or app, get in touch.

Shopify GraphQL Query Cost Optimization: Stop Burning API Points

Muhammad Masad Ashraf — Tue, 09 Jun 2026 16:27:46 +0000

Every Shopify app developer hits the same wall eventually.

You build a feature, push it to production, and watch your API calls start failing at scale. The culprit is almost always the same: inefficient GraphQL queries burning through your rate limit bucket faster than it refills.

Shopify's GraphQL Admin API uses cost-based throttling, not request-per-second counting. A single poorly structured query can cost 1,000 points. A well-optimized one may cost under 50.

This post covers how the system works, where developers waste points, and what to do about it.

How the Leaky Bucket Works

Shopify assigns a calculated cost to every query before executing it. That cost gets deducted from your bucket instantly.

Parameter	Value
Bucket size (standard)	1,000 points
Bucket size (Shopify Plus)	2,000 points
Refill rate	50 points/second
Max single query cost	1,000 points
Throttle error	`THROTTLED`

Every API response includes cost metadata under extensions.cost:

{
  "extensions": {
    "cost": {
      "requestedQueryCost": 412,
      "actualQueryCost": 412,
      "throttleStatus": {
        "maximumAvailable": 1000,
        "currentlyAvailable": 588,
        "restoreRate": 50
      }
    }
  }
}

Parse this on every single call. Most developers ignore it until production breaks.

How Cost Gets Calculated

Cost is driven by two things: the number of objects you request and the depth of connections.

The problem with nested connections

The first argument multiplies cost across every level of nesting. Here is what an expensive query looks like:

{
  products(first: 250) {
    edges {
      node {
        id
        title
        variants(first: 100) {
          edges {
            node {
              id
              price
              inventoryItem {
                id
              }
            }
          }
        }
      }
    }
  }
}

This query easily hits a calculated cost of 25,000+. Shopify rejects it before execution.

The optimized version

{
  products(first: 50) {
    edges {
      node {
        id
        title
        variants(first: 10) {
          edges {
            node {
              id
              price
            }
          }
        }
      }
    }
  }
}

Smaller page sizes, fewer fields, no 3-level nesting. Cost drops by 90%+.

7 Strategies to Reduce GraphQL Points

1. Request Only the Fields You Need

GraphQL gives you field-level control. Use it.

Never pull an entire product object when you only need id and title. Unused fields carry cost weight inside nested connections.

Audit every query in your codebase. Remove what your app does not consume.

2. Reduce Connection Page Sizes

first: 250 is the single most common reason for runaway costs.

Start with first: 50. Paginate across multiple requests. Three paginated calls at first: 50 are almost always cheaper than one call at first: 250 with deep nesting.

Build pagination into your data layer from day one.

3. Split Deeply Nested Queries

Instead of fetching products, variants, and inventory in one query, split them.

# Step 1: Fetch products
{
  products(first: 50) {
    edges {
      node {
        id
        title
      }
    }
  }
}

# Step 2: Fetch variants by product ID
{
  product(id: "gid://shopify/Product/123456789") {
    variants(first: 10) {
      edges {
        node {
          id
          price
          inventoryQuantity
        }
      }
    }
  }
}

Two focused queries consistently beat one deeply nested monster query.

4. Use Query Cost Extensions for Real-Time Monitoring

Log extensions.cost on every API call. Build an alerting rule that fires when actualQueryCost exceeds a threshold you define.

Here is a minimal Node.js wrapper pattern:

async function shopifyQuery(query, variables = {}) {
  const response = await fetch(SHOPIFY_GRAPHQL_ENDPOINT, {
    method: 'POST',
    headers: {
      'Content-Type': 'application/json',
      'X-Shopify-Access-Token': ACCESS_TOKEN,
    },
    body: JSON.stringify({ query, variables }),
  });

  const data = await response.json();

  // Always log cost data
  if (data.extensions?.cost) {
    const { actualQueryCost, throttleStatus } = data.extensions.cost;
    console.log(`Query cost: ${actualQueryCost} | Available: ${throttleStatus.currentlyAvailable}`);

    // Alert threshold
    if (actualQueryCost > 200) {
      console.warn(`High-cost query detected: ${actualQueryCost} points`);
    }
  }

  return data;
}

5. Build Leaky Bucket Awareness Into Your Client

Check throttleStatus.currentlyAvailable before firing a query. If available points are below your expected cost, wait.

async function throttleAwareQuery(query, estimatedCost = 100) {
  const status = await getCurrentBucketStatus();

  if (status.currentlyAvailable < estimatedCost) {
    const waitSeconds = Math.ceil(
      (estimatedCost - status.currentlyAvailable) / status.restoreRate
    );
    console.log(`Waiting ${waitSeconds}s for bucket to refill...`);
    await sleep(waitSeconds * 1000);
  }

  return shopifyQuery(query);
}

This prevents burst failures without requiring exponential backoff from the start.

6. Cache Infrequently-Changing Data

Not every query needs a fresh Shopify API hit. Product catalog data, collection structures, and metafield definitions rarely change.

Use Redis with a short TTL:

async function getCachedProducts(cacheKey, ttlSeconds = 300) {
  const cached = await redis.get(cacheKey);
  if (cached) return JSON.parse(cached);

  const fresh = await shopifyQuery(PRODUCTS_QUERY);
  await redis.setex(cacheKey, ttlSeconds, JSON.stringify(fresh));
  return fresh;
}

You reduce Shopify GraphQL query cost by not making the call at all when cached data is still valid.

7. Use Direct GID Lookups Over List Queries

List queries scan potentially hundreds of records. Direct lookups target exactly what you need.

# Expensive: scans all products
{
  products(first: 250) {
    edges {
      node { id title }
    }
  }
}

# Cheap: direct lookup by known GID
{
  product(id: "gid://shopify/Product/123456789") {
    id
    title
    variants(first: 5) {
      edges {
        node { id price }
      }
    }
  }
}

Store Shopify GIDs in your local database. Retrieve them with targeted queries instead of scanning lists.

Use Bulk Operations for Large Data Sets

Stop using paginated GraphQL queries for bulk data jobs. Shopify's Bulk Operations API handles this better in every way.

Bulk operations:

Run asynchronously in the background
Do not consume your standard rate limit bucket
Return results as a downloadable JSONL file

mutation {
  bulkOperationRunQuery(
    query: """
    {
      products {
        edges {
          node {
            id
            title
            variants {
              edges {
                node {
                  id
                  price
                }
              }
            }
          }
        }
      }
    }
    """
  ) {
    bulkOperation {
      id
      status
    }
    userErrors {
      field
      message
    }
  }
}

Use bulk operations for: full catalog exports, historical order analysis, large-scale metafield reads, and inventory reconciliation across locations.

Handling Throttle Errors

When you exhaust your bucket, Shopify returns HTTP 200 with an error body:

{
  "errors": [
    {
      "message": "Throttled",
      "extensions": {
        "code": "THROTTLED",
        "retryAfter": 4.5
      }
    }
  ]
}

Your retry handler:

async function queryWithRetry(query, maxRetries = 3) {
  for (let attempt = 0; attempt < maxRetries; attempt++) {
    const data = await shopifyQuery(query);

    const throttleError = data.errors?.find(
      (e) => e.extensions?.code === 'THROTTLED'
    );

    if (throttleError) {
      const waitMs = (throttleError.extensions.retryAfter + 0.5) * 1000;
      console.log(`Throttled. Retrying in ${waitMs}ms...`);
      await sleep(waitMs);
      continue;
    }

    return data;
  }

  throw new Error('Max retries exceeded after throttle errors');
}

Always add a small buffer (0.5s) to the retryAfter value. Network latency means hitting the retry exactly at the boundary often fails again.

Common Mistakes at a Glance

Mistake	Cost Impact	Fix
Fetching all fields by default	3x to 10x waste	Select only needed fields
`first: 250` everywhere	Exponential multiplier	Start with `first: 50`, paginate
Ignoring `extensions.cost`	Zero visibility	Log cost on every call
Nested connections 3+ levels	Cost explodes	Split into separate queries
No bucket awareness	Burst failures	Read `currentlyAvailable` before querying
Polling instead of webhooks	Continuous drain	Replace polling with event-driven webhooks

Pre-Ship Checklist

Before any new GraphQL query goes to production:

[ ] Only requesting fields the app actually uses
[ ] Connection page sizes set to minimum viable first value
[ ] Nested connections flattened or split where possible
[ ] extensions.cost logged and monitored
[ ] Throttle error handler reads retryAfter
[ ] Frequently-read data cached with TTL
[ ] Bulk Operations used for large data jobs
[ ] Known IDs queried directly by GID

Wrapping Up

Shopify GraphQL query cost is not a niche concern. It directly determines whether your app stays reliable at scale or becomes a 2am debugging session.

The fix is straightforward: read your cost data, reduce page sizes, split nested queries, cache where you can, and handle throttle errors properly. None of this is complex. It just has to be built intentionally.

For the full breakdown including the GraphQL vs REST comparison table and Hydrogen-specific patterns, read the original post on the KolachiTech blog.

Originally published at kolachitech.com

How to Verify Shopify Webhooks Correctly (And Stop Getting It Wrong)

Muhammad Masad Ashraf — Fri, 05 Jun 2026 19:35:27 +0000

Originally published on KolachiTech

I have reviewed a lot of Shopify app codebases. One thing shows up constantly: webhook verification that looks correct but silently fails. The developer gets frustrated, removes the check, ships without it, and ships a vulnerability instead.

This post covers how Shopify webhook verification actually works, the implementation mistakes that trip people up, and what a production-ready setup looks like. I will share working Node.js and Python code you can drop in today.

Why Your Webhook Endpoint Is a Target

Your Shopify webhook URL is just an HTTP endpoint. It does not require authentication to reach. Anyone who knows it can POST to it.

Without verification, an attacker can:

Fake order fulfillment events
Inject customer records
Trigger inventory changes
Flood your endpoint until it falls over

This matters more than you think. If your webhooks feed an ERP, a CRM, or a fulfillment warehouse, one forged payload corrupts data in every downstream system simultaneously.

Verification is not optional. It is the lock on the front door.

What Shopify Actually Does

Shopify uses HMAC-SHA256 to sign every webhook request it sends.

Here is the exact flow:

Shopify takes the raw bytes of your request body
It runs HMAC-SHA256 using your app's shared secret as the key
It Base64-encodes the result
It drops that value into the X-Shopify-Hmac-SHA256 request header
It sends the request to your endpoint

Your job is to repeat steps 1 through 3 on your side and compare what you compute with what Shopify sent.

If they match: authentic. If they differ: reject immediately, do not process.

The Node.js Implementation

const crypto = require('crypto');

function verifyShopifyWebhook(rawBody, hmacHeader, secret) {
  const generatedHmac = crypto
    .createHmac('sha256', secret)
    .update(rawBody, 'utf8')
    .digest('base64');

  const hmacBuffer = Buffer.from(hmacHeader, 'base64');
  const generatedBuffer = Buffer.from(generatedHmac, 'base64');

  // Length check first to avoid exceptions in timingSafeEqual
  if (hmacBuffer.length !== generatedBuffer.length) {
    return false;
  }

  // Constant-time comparison — never use ===
  return crypto.timingSafeEqual(hmacBuffer, generatedBuffer);
}

Express.js setup that actually works:

app.post(
  '/webhooks/shopify',
  express.raw({ type: 'application/json' }), // Buffer raw body BEFORE any parsing
  (req, res) => {
    const hmac = req.headers['x-shopify-hmac-sha256'];
    const secret = process.env.SHOPIFY_WEBHOOK_SECRET;

    if (!verifyShopifyWebhook(req.body, hmac, secret)) {
      return res.status(401).send('Unauthorized');
    }

    // Safe to parse now
    const payload = JSON.parse(req.body);

    // Acknowledge immediately, process async
    res.status(200).send('OK');
    processWebhookAsync(payload);
  }
);

The Python Implementation

import hmac
import hashlib
import base64

def verify_shopify_webhook(raw_body: bytes, hmac_header: str, secret: str) -> bool:
    digest = hmac.new(
        secret.encode('utf-8'),
        raw_body,
        digestmod=hashlib.sha256
    ).digest()
    computed_hmac = base64.b64encode(digest).decode('utf-8')

    # Constant-time comparison
    return hmac.compare_digest(computed_hmac, hmac_header)

Django/Flask usage:

@app.route('/webhooks/shopify', methods=['POST'])
def shopify_webhook():
    raw_body = request.get_data()  # Raw bytes, not request.json
    hmac_header = request.headers.get('X-Shopify-Hmac-SHA256', '')
    secret = os.environ.get('SHOPIFY_WEBHOOK_SECRET')

    if not verify_shopify_webhook(raw_body, hmac_header, secret):
        return 'Unauthorized', 401

    payload = request.json
    return 'OK', 200

The Mistakes That Break Verification (And Ship Vulnerabilities)

Mistake 1: Parsing before verifying

This is the most common one. Your body parsing middleware runs before your verification code and transforms the raw bytes. Your computed HMAC no longer matches what Shopify signed.

The developer sees 401 everywhere. They remove verification. The app ships insecure.

Fix: Use express.raw() on your webhook route specifically, not express.json().

Mistake 2: Using `===` for comparison

// WRONG — leaks timing information
if (generatedHmac === hmacHeader) { ... }

// RIGHT — constant-time
crypto.timingSafeEqual(hmacBuffer, generatedBuffer);

An attacker can measure microsecond differences in how long === takes to compare strings and deduce the correct signature byte by byte. This is a real attack class called a timing side-channel.

Mistake 3: No replay protection

HMAC proves the payload is authentic. It does not prove it is fresh.

A payload from yesterday is still cryptographically valid today. An attacker who intercepts a legitimate webhook can replay it as many times as they want.

Fix: Validate the timestamp in the payload. Reject anything older than 5 minutes. Store event IDs and reject duplicates.

function isReplayAttack(payload) {
  const eventTime = new Date(payload.created_at).getTime();
  const now = Date.now();
  const fiveMinutes = 5 * 60 * 1000;

  return (now - eventTime) > fiveMinutes;
}

Mistake 4: No shop domain validation

All stores using your app share the same signing secret. A valid webhook from store A is also cryptographically valid when replayed against your store B handler.

Fix: Always validate X-Shopify-Shop-Domain against your database of registered shops.

const shopDomain = req.headers['x-shopify-shop-domain'];

if (!registeredShops.includes(shopDomain)) {
  return res.status(403).send('Forbidden');
}

Mistake 5: Storing the secret in code

# Never do this
SHOPIFY_WEBHOOK_SECRET = "shpss_abc123"  # Committed to git

# Do this
process.env.SHOPIFY_WEBHOOK_SECRET  # Environment variable or secrets manager

Logs are often less locked down than your codebase. Never log the secret either.

The Production Checklist

Before shipping any webhook consumer:

Transport:

HTTPS only, no HTTP fallback
TLS 1.2 or higher

Verification:

Raw body buffered before any parsing middleware
HMAC computed with correct shared secret
Constant-time comparison used
Failed verification returns 401/403 immediately

Replay protection:

Timestamp validated, reject payloads older than 5 minutes
Event IDs deduplicated (Redis or DB)
Shop domain validated against allowed list

Secrets:

Secret in environment variable or secrets manager
Secret rotation procedure documented and tested
Verification failures logged (not the secret, the failure)

Reliability:

Handler returns 200 immediately, processes async
Dead letter queue configured for processing failures

Why Acknowledging Immediately Matters

Shopify expects a 2xx response within 5 seconds. If you do slow processing synchronously, you will hit timeouts, and Shopify will retry the webhook. Retries without idempotency controls mean double-processing.

The pattern is: verify, acknowledge with 200, hand off to a queue or async worker, process there.

res.status(200).send('OK');         // Shopify gets its answer
processWebhookAsync(payload);       // You do the actual work

Scaling Shopify Webhooks to Handle Millions of Events: A Practical Guide

Muhammad Masad Ashraf — Thu, 04 Jun 2026 10:08:28 +0000

Scaling Shopify Webhooks to Handle Millions of Events: A Practical Guide

If you've built a Shopify integration that gets any real traffic, you've probably hit this issue: your webhook endpoint becomes a bottleneck.

Let me walk you through the patterns that actually work at scale, with code examples you can use today.

The Problem: Synchronous Processing Doesn't Scale

Your first webhook implementation probably looks something like this:

app.post('/webhooks/orders/create', (req, res) => {
  // Validate webhook signature
  if (!validateWebhookSignature(req)) {
    return res.status(401).send('Unauthorized');
  }

  const order = req.body;

  // Do all the work synchronously
  updateInventory(order);
  createCustomerRecord(order.customer);
  sendConfirmationEmail(order);
  syncToERP(order);
  updateAnalytics(order);

  // Finally return response
  res.status(200).send('OK');
});

This works fine until:

One slow operation blocks everything. If sendConfirmationEmail() takes 5 seconds, every webhook takes 5 seconds.
You hit Shopify's timeout. Shopify waits 28 seconds max. If your endpoint doesn't respond by then, it retries. Now you have duplicates.
Resource exhaustion. Each concurrent webhook ties up a worker process. At 1,000 webhooks/second, you need 5,000+ workers.
Cascading failures. One slow external service (email, payment processor, ERP) slows down everything.

The solution? Decouple receiving from processing.

Solution 1: Queue-Based Processing

Instead of doing all the work in the webhook handler, push events to a message queue and process them asynchronously:

const AWS = require('aws-sdk');
const sqs = new AWS.SQS();

app.post('/webhooks/orders/create', async (req, res) => {
  // Validate signature
  if (!validateWebhookSignature(req)) {
    return res.status(401).send('Unauthorized');
  }

  try {
    // Just push to queue and return immediately
    await sqs.sendMessage({
      QueueUrl: process.env.QUEUE_URL,
      MessageBody: JSON.stringify({
        type: 'order.created',
        webhookId: req.headers['x-shopify-webhook-id'],
        data: req.body,
        timestamp: new Date().toISOString()
      })
    }).promise();

    // Return 200 OK to Shopify in milliseconds
    res.status(200).send('OK');
  } catch (error) {
    console.error('Queue push failed:', error);
    // Still return 200 to Shopify - we can retry later
    res.status(202).send('Accepted');
  }
});

Now your endpoint responds in milliseconds. Shopify is happy. Workers process the queue asynchronously:

// Worker process (separate from webhook handler)
const processQueue = async () => {
  while (true) {
    const messages = await sqs.receiveMessage({
      QueueUrl: process.env.QUEUE_URL,
      MaxNumberOfMessages: 10,
      WaitTimeSeconds: 20
    }).promise();

    if (!messages.Messages) continue;

    for (const message of messages.Messages) {
      try {
        const event = JSON.parse(message.Body);

        // Now do all the work asynchronously
        await processOrder(event.data);

        // Delete from queue after successful processing
        await sqs.deleteMessage({
          QueueUrl: process.env.QUEUE_URL,
          ReceiptHandle: message.ReceiptHandle
        }).promise();
      } catch (error) {
        console.error('Processing failed:', error);
        // Leave in queue - will retry automatically
      }
    }
  }
};

processQueue();

Benefits:

Webhook endpoint responds in <10ms
Process slow operations without timeout risk
Scale workers independently
Handle spikes gracefully

Trade-offs:

Slight delay in processing (seconds to minutes, not milliseconds)
Need infrastructure for queue system
Must handle duplicates (webhooks can retry)

Solution 2: Dead Letter Queues for Failure Handling

Even with queues, events fail. Your database goes down. An API times out. You need a safety net.

Enter Dead Letter Queues (DLQ). When processing fails repeatedly, move to DLQ for investigation:

const MAX_RETRIES = 3;

const processQueue = async () => {
  while (true) {
    const messages = await sqs.receiveMessage({
      QueueUrl: process.env.QUEUE_URL,
      MaxNumberOfMessages: 10
    }).promise();

    if (!messages.Messages) continue;

    for (const message of messages.Messages) {
      let retryCount = 0;

      try {
        const event = JSON.parse(message.Body);

        // Extract retry count from message attributes
        if (message.Attributes?.ApproximateReceiveCount) {
          retryCount = parseInt(message.Attributes.ApproximateReceiveCount);
        }

        // Attempt processing with exponential backoff
        try {
          await processOrder(event.data);
        } catch (error) {
          if (retryCount < MAX_RETRIES) {
            // Requeue with backoff
            const delay = Math.min(300, Math.pow(2, retryCount) * 10);
            await sqs.changeMessageVisibility({
              QueueUrl: process.env.QUEUE_URL,
              ReceiptHandle: message.ReceiptHandle,
              VisibilityTimeout: delay
            }).promise();

            console.log(`Retry ${retryCount}/${MAX_RETRIES} after ${delay}s`, error.message);
            continue;
          } else {
            // Move to DLQ
            throw new Error(`Failed after ${MAX_RETRIES} retries: ${error.message}`);
          }
        }

        // Success - delete from queue
        await sqs.deleteMessage({
          QueueUrl: process.env.QUEUE_URL,
          ReceiptHandle: message.ReceiptHandle
        }).promise();

      } catch (error) {
        // Send to Dead Letter Queue
        console.error('Moving to DLQ:', error);

        await sqs.sendMessage({
          QueueUrl: process.env.DLQ_URL,
          MessageBody: JSON.stringify({
            originalMessage: message.Body,
            error: error.message,
            failedAt: new Date().toISOString(),
            retryCount: retryCount
          })
        }).promise();

        // Remove from main queue
        await sqs.deleteMessage({
          QueueUrl: process.env.QUEUE_URL,
          ReceiptHandle: message.ReceiptHandle
        }).promise();
      }
    }
  }
};

Key patterns:

Exponential backoff prevents overwhelming failing services
DLQ captures unrecoverable failures
Log failure reasons for debugging
Implement manual replay when issue is fixed

Solution 3: Idempotency (The Silent Killer)

Here's what nobody tells you: Shopify can send the same webhook multiple times.

Network hiccup? Retry. Your endpoint slow? Timeout and retry. You need idempotency or you'll have duplicate orders.

Track webhook IDs in your database:

const processOrder = async (order) => {
  const webhookId = order.webhookId || order.id; // Shopify webhook ID

  // Check if already processed
  const existing = await db.webhooks.findOne({ webhookId });
  if (existing) {
    console.log(`Webhook ${webhookId} already processed, skipping`);
    return; // Already done
  }

  try {
    // Do the actual processing
    await updateInventory(order);
    await createCustomerRecord(order.customer);
    await sendConfirmationEmail(order);
    await syncToERP(order);

    // Record that we processed this webhook
    await db.webhooks.insertOne({
      webhookId,
      processedAt: new Date(),
      success: true,
      orderData: order
    });

  } catch (error) {
    // Still record the attempt for debugging
    await db.webhooks.insertOne({
      webhookId,
      processedAt: new Date(),
      success: false,
      error: error.message,
      orderData: order
    });
    throw error;
  }
};

This single pattern prevents:

Duplicate orders
Duplicate charges
Duplicate inventory adjustments
Duplicate customer records

Solution 4: Event-Driven Architecture

As you scale, processing one event type directly in your worker becomes complex:

// Anti-pattern: worker does everything
const processOrder = async (order) => {
  updateInventory(order);      // slow
  createCustomer(order);       // sometimes fails
  sendEmail(order);            // unreliable
  updateAnalytics(order);      // high volume
  syncERP(order);              // complex
  assignLoyaltyPoints(order);  // business logic
  // One failure fails everything
};

Better: emit events and let services consume independently:

const EventEmitter = require('events');
const orderEvents = new EventEmitter();

// Main webhook handler - just emit event
const processOrder = async (order) => {
  // Record webhook processed
  await recordWebhook(order.id);

  // Emit event - handlers consume independently
  orderEvents.emit('order.created', {
    id: order.id,
    customerId: order.customer.id,
    amount: order.total_price,
    timestamp: new Date()
  });
};

// Separate services consume the event
orderEvents.on('order.created', async (event) => {
  try {
    await updateInventory(event);
  } catch (error) {
    // Log and queue retry - doesn't affect other consumers
    console.error('Inventory update failed:', error);
  }
});

orderEvents.on('order.created', async (event) => {
  try {
    await sendConfirmationEmail(event);
  } catch (error) {
    // Log and queue retry - inventory still updated
    console.error('Email failed:', error);
  }
});

orderEvents.on('order.created', async (event) => {
  try {
    await syncERP(event);
  } catch (error) {
    console.error('ERP sync failed:', error);
  }
});

Benefits:

Each service scales independently
One failure doesn't cascade
Easy to add new consumers
Clear separation of concerns

At scale, use Kafka, RabbitMQ, or AWS SNS/SQS for event distribution.

The Monitoring You Need

You can't manage what you don't measure. Instrument your webhook system:

const prometheus = require('prom-client');

// Create metrics
const webhookCounter = new prometheus.Counter({
  name: 'webhooks_received_total',
  help: 'Total webhooks received',
  labelNames: ['type', 'status']
});

const processingDuration = new prometheus.Histogram({
  name: 'webhook_processing_duration_seconds',
  help: 'Time to process webhook',
  labelNames: ['type'],
  buckets: [0.1, 0.5, 1, 5, 10]
});

const queueDepth = new prometheus.Gauge({
  name: 'webhook_queue_depth',
  help: 'Number of pending webhooks'
});

// Use in your code
app.post('/webhooks/orders/create', async (req, res) => {
  const startTime = Date.now();

  try {
    // ... process webhook
    webhookCounter.inc({ type: 'order.create', status: 'queued' });
    res.status(200).send('OK');
  } catch (error) {
    webhookCounter.inc({ type: 'order.create', status: 'failed' });
    res.status(202).send('Retry');
  } finally {
    const duration = (Date.now() - startTime) / 1000;
    processingDuration.observe({ type: 'order.create' }, duration);
  }
});

// Monitor queue depth
setInterval(async () => {
  const depth = await sqs.getQueueAttributes({
    QueueUrl: process.env.QUEUE_URL,
    AttributeNames: ['ApproximateNumberOfMessages']
  }).promise();

  queueDepth.set(parseInt(depth.Attributes.ApproximateNumberOfMessages));
}, 30000);

Alert on:

Failure rate > 0.5%
Queue depth > 10,000
Processing latency p99 > 5s
DLQ messages appearing

Implementation Checklist

Building scalable webhook infrastructure requires getting multiple pieces right:

Foundation
[ ] Validate webhook signatures
[ ] Use message queue (SQS, RabbitMQ, Redis)
[ ] Implement idempotency
[ ] Add basic monitoring

Reliability
[ ] Implement dead letter queues
[ ] Add exponential backoff
[ ] Log all failures with context
[ ] Implement manual replay

Scalability
[ ] Add caching layers
[ ] Optimize database indexes
[ ] Horizontal scaling of workers
[ ] Monitor queue depth

Advanced
[ ] Event-driven architecture
[ ] Multiple queue consumers
[ ] Distributed tracing
[ ] Multi-region setup

Common Gotchas

Gotcha 1: Lambda timeout with SQS
Lambda has max 15-minute timeout. For long-running webhooks, use EC2/ECS workers or Fargate instead.

Gotcha 2: Forgetting webhook signature validation
Always validate. Otherwise anyone can send fake webhooks to your system.

Gotcha 3: Not idempotent processing
The second webhook always arrives. Your code must handle it.

Gotcha 4: Monitoring too late
Add monitoring before you hit scale issues. You need baseline metrics to recognize problems.

Gotcha 5: Forgetting Shopify rate limits
Shopify has API rate limits. If you process 1M webhooks, don't make 1M API calls back to Shopify immediately.

Resources

AWS SQS - Simple queue service
RabbitMQ - Open source message broker
Shopify Webhook Documentation
Distributed Tracing with Jaeger

Read the Complete Guide

For the complete technical guide with more code examples, architecture diagrams, and a detailed implementation checklist, check out the full article on Kolachi Tech.

The complete guide includes:

Advanced scaling patterns for multi-region deployments
Detailed monitoring and observability strategies
Complete 8-step implementation checklist
Architecture diagrams and design patterns
More real-world code examples

Have you scaled webhooks to production? What patterns saved you? Drop a comment below - I'd love to hear your war stories!

Shopify Webhook Observability: Complete Implementation Guide

Muhammad Masad Ashraf — Wed, 03 Jun 2026 22:12:51 +0000

Shopify Webhook Observability: Complete Implementation Guide

Webhooks are the nervous system of your Shopify integration. When they fail silently, data corrupts and you don't know why.

This guide covers production-ready webhook observability: structured logging, health metrics, error recovery, and debugging strategies.

The Problem

Webhooks are asynchronous. Your app receives events, processes them, and moves on. When something fails:

Shopify doesn't know your app crashed
You don't know a webhook failed
Data stays out of sync
Problems compound until customers complain

Without observability, you're flying blind.

What You'll Learn

Structured logging patterns for webhooks
Health metrics and dashboards
Error categorization and retry logic
Dead letter queue implementation
Debugging production incidents

Let's build visibility.

1. Structured Logging Foundation

All observability starts with logs.

Bad logs (unstructured text):

2024-06-04 14:23:45 Webhook received
2024-06-04 14:23:46 Processing
2024-06-04 14:23:47 Done

Unstructured logs are difficult to search, filter, and aggregate.

Good logs (JSON structured format):

{
  "timestamp": "2024-06-04T14:23:45Z",
  "webhook_id": "gid://shopify/WebhookEventBridge/12345",
  "shop_id": "gid://shopify/Shop/67890",
  "event_type": "orders/create",
  "status": "success",
  "duration_ms": 245,
  "attempt": 1,
  "signature_valid": true
}

Structured logs in JSON format enable filtering, aggregation, and analysis.

Entry Point Logging

Log immediately when webhook arrives. Here's a complete Express.js implementation:

const express = require('express');
const winston = require('winston');

const logger = winston.createLogger({
  format: winston.format.json(),
  transports: [
    new winston.transports.File({ filename: 'webhooks.log' })
  ]
});

app.post('/webhooks/shopify', (req, res) => {
  const webhookId = req.headers['x-shopify-webhook-id'];
  const shop = req.headers['x-shopify-shop-api-version'];
  const hmac = req.headers['x-shopify-hmac-sha256'];
  const topic = req.headers['x-shopify-topic'];

  logger.info('webhook_received', {
    webhook_id: webhookId,
    shop_id: shop,
    topic,
    timestamp: new Date().toISOString(),
    body_size: req.rawBody.length
  });

  res.status(202).send('Accepted');

  // Process async
  processWebhookAsync(req.body, webhookId, hmac);
});

Why this approach: Return 202 immediately. Process asynchronously. Shopify's timeout is 5 seconds; don't waste it.

Signature Verification Logging

Verify Shopify's HMAC signature before processing any webhook:

const crypto = require('crypto');

function verifySignature(rawBody, signature, secret) {
  const hash = crypto
    .createHmac('sha256', secret)
    .update(rawBody, 'utf8')
    .digest('base64');

  return hash === signature;
}

async function processWebhookAsync(body, webhookId, signature) {
  const apiSecret = process.env.SHOPIFY_API_SECRET;

  if (!verifySignature(JSON.stringify(body), signature, apiSecret)) {
    logger.error('webhook_signature_invalid', {
      webhook_id: webhookId,
      provided_signature: signature.substring(0, 16) + '...'
    });
    return;
  }

  logger.debug('webhook_signature_verified', {
    webhook_id: webhookId
  });

  // Continue processing
}

Never process unverified webhooks. Signature verification prevents unauthorized access.

Processing Step Logging

Log each stage of webhook processing with timing information. This helps identify where delays occur:

async function processWebhookAsync(body, webhookId, signature) {
  const startTime = Date.now();

  try {
    // Verify signature
    verifySignature(JSON.stringify(body), signature, apiSecret);

    // Parse webhook
    const webhook = parseWebhook(body);
    logger.debug('webhook_parsed', {
      webhook_id: webhookId,
      resource_id: webhook.id,
      resource_type: webhook.type
    });

    // Check duplicate
    const isDuplicate = await checkDuplicate(webhookId);
    if (isDuplicate) {
      logger.info('webhook_duplicate_detected', {
        webhook_id: webhookId,
        resource_id: webhook.id
      });
      return;
    }

    // Store to database
    await storeWebhook(webhook);
    logger.info('webhook_stored', {
      webhook_id: webhookId,
      resource_id: webhook.id
    });

    // Sync with systems
    await syncDownstream(webhook);
    logger.info('webhook_processed', {
      webhook_id: webhookId,
      duration_ms: Date.now() - startTime,
      resource_id: webhook.id
    });

  } catch (error) {
    logger.error('webhook_processing_failed', {
      webhook_id: webhookId,
      error: error.message,
      stack: error.stack,
      duration_ms: Date.now() - startTime
    });

    // Retry or dead letter queue
    await handleError(error, body, webhookId);
  }
}

External Service Logging

Track interactions with third-party APIs and downstream services. Log both successful calls and failures with timing information:

async function syncDownstream(webhook) {
  const endpoints = [
    { name: 'inventory-sync', url: 'https://inventory.internal/sync' },
    { name: 'analytics-pipeline', url: 'https://analytics.internal/events' }
  ];

  for (const endpoint of endpoints) {
    const startTime = Date.now();
    try {
      const response = await fetch(endpoint.url, {
        method: 'POST',
        body: JSON.stringify(webhook),
        timeout: 5000
      });

      const duration = Date.now() - startTime;

      if (!response.ok) {
        logger.warn('external_service_error', {
          webhook_id: webhook.id,
          service: endpoint.name,
          status: response.status,
          duration_ms: duration
        });
      } else {
        logger.debug('external_service_success', {
          webhook_id: webhook.id,
          service: endpoint.name,
          duration_ms: duration
        });
      }
    } catch (error) {
      logger.error('external_service_timeout', {
        webhook_id: webhook.id,
        service: endpoint.name,
        error: error.message,
        duration_ms: Date.now() - startTime
      });

      // Store for retry
      await storeFailedSync(webhook.id, endpoint.name);
    }
  }
}

2. Health Metrics and Dashboards

Track three key metrics:

Delivery Rate

Delivery Rate = (Webhooks Successfully Processed / Total Webhooks) x 100

Target: 99.9% or higher.

Processing Latency

async function calculateLatencyMetrics() {
  const logs = await queryLogs({
    status: 'success',
    timeRange: '24h'
  });

  const durations = logs
    .map(log => log.duration_ms)
    .sort((a, b) => a - b);

  return {
    median: percentile(durations, 50),
    p95: percentile(durations, 95),
    p99: percentile(durations, 99),
    max: Math.max(...durations)
  };
}

function percentile(arr, p) {
  const index = Math.ceil((p / 100) * arr.length) - 1;
  return arr[Math.max(0, index)];
}

Target: Median under 100ms, p95 under 500ms, p99 under 2s.

Error Rate by Type

async function getErrorMetrics() {
  const errorLogs = await queryLogs({
    status: 'error',
    timeRange: '24h'
  });

  const errorsByType = {};

  for (const log of errorLogs) {
    const type = log.error_type || 'unknown';
    errorsByType[type] = (errorsByType[type] || 0) + 1;
  }

  const total = errorLogs.length;

  return Object.entries(errorsByType).reduce((acc, [type, count]) => {
    acc[type] = {
      count,
      percentage: ((count / total) * 100).toFixed(2) + '%'
    };
    return acc;
  }, {});
}

Track:

Timeout errors (slow external services)
Database errors (connection pool exhausted)
Validation errors (bad payload)
Rate limit errors (too many requests)
Service unavailable (external APIs down)

Building a Dashboard

Use your logging platform's dashboard feature (Datadog, Splunk, CloudWatch, etc.). Here are example queries to calculate key metrics:

Query 1: Calculate delivery rate percentage over the last 24 hours

SELECT count(*) as total,
       sum(case when status = 'success' then 1 else 0 end) as successful,
       (sum(case when status = 'success' then 1 else 0 end) * 100.0 / count(*)) as delivery_rate
FROM webhooks
WHERE timestamp > now() - interval '24 hours'

Query 2: Calculate latency percentiles (50th, 95th, 99th) for successful webhooks

SELECT percentile_cont(0.5) WITHIN GROUP (ORDER BY duration_ms) as p50,
       percentile_cont(0.95) WITHIN GROUP (ORDER BY duration_ms) as p95,
       percentile_cont(0.99) WITHIN GROUP (ORDER BY duration_ms) as p99
FROM webhooks
WHERE timestamp > now() - interval '24 hours'
  AND status = 'success'

Query 3: Calculate error rate by error type

       count(*) as count,
       (count(*) * 100.0 / (SELECT count(*) FROM webhooks WHERE status = 'error')) as percentage
FROM webhooks
WHERE timestamp > now() - interval '24 hours'
  AND status = 'error'
GROUP BY error_type
ORDER BY count DESC

3. Error Handling and Retry Logic

Categorize errors. Different errors need different responses.

Error Classification

Different error types require different handling strategies. This function classifies errors so you can respond appropriately:

const ErrorType = {
  TRANSIENT_TIMEOUT: 'transient_timeout',
  TRANSIENT_NETWORK: 'transient_network',
  PERMANENT_INVALID_DATA: 'permanent_invalid_data',
  PERMANENT_VALIDATION: 'permanent_validation',
  SYSTEMIC_SERVICE_DOWN: 'systemic_service_down',
  RATE_LIMITED: 'rate_limited',
  DATABASE_ERROR: 'database_error'
};

function classifyError(error, statusCode) {
  if (error.code === 'ECONNREFUSED' || error.code === 'ENOTFOUND') {
    return ErrorType.TRANSIENT_NETWORK;
  }

  if (error.code === 'ETIMEDOUT') {
    return ErrorType.TRANSIENT_TIMEOUT;
  }

  if (statusCode === 429) {
    return ErrorType.RATE_LIMITED;
  }

  if (statusCode === 503 || statusCode === 502) {
    return ErrorType.SYSTEMIC_SERVICE_DOWN;
  }

  if (statusCode === 400 || statusCode === 422) {
    return ErrorType.PERMANENT_INVALID_DATA;
  }

  if (error.type === 'ValidationError') {
    return ErrorType.PERMANENT_VALIDATION;
  }

  if (error.message.includes('database')) {
    return ErrorType.DATABASE_ERROR;
  }

  return 'unknown';
}

Exponential Backoff Retry

Implement retry logic with exponential backoff and jitter to handle transient failures gracefully:

async function processWithRetry(webhook, maxAttempts = 3) {
  const delays = [1000, 2000, 4000]; // 1s, 2s, 4s

  for (let attempt = 1; attempt <= maxAttempts; attempt++) {
    try {
      await processWebhook(webhook);

      logger.info('webhook_success', {
        webhook_id: webhook.id,
        attempt
      });
      return;

    } catch (error) {
      const errorType = classifyError(error, error.statusCode);

      // Don't retry permanent errors
      if (errorType.startsWith('permanent')) {
        logger.error('webhook_permanent_failure', {
          webhook_id: webhook.id,
          error_type: errorType,
          error: error.message,
          attempt
        });
        throw error;
      }

      // Last attempt?
      if (attempt === maxAttempts) {
        logger.error('webhook_max_retries_exceeded', {
          webhook_id: webhook.id,
          error_type: errorType,
          attempts: maxAttempts
        });
        throw error;
      }

      // Retry with backoff + jitter
      const baseDelay = delays[attempt - 1] || delays[delays.length - 1];
      const jitter = Math.random() * 1000;
      const totalDelay = baseDelay + jitter;

      logger.warn('webhook_retry', {
        webhook_id: webhook.id,
        error_type: errorType,
        attempt,
        next_retry_ms: Math.round(totalDelay)
      });

      await new Promise(resolve => setTimeout(resolve, totalDelay));
    }
  }
}

Dead Letter Queue Pattern

Implement a dead letter queue to capture webhooks that fail after all retry attempts. This prevents data loss and enables async reprocessing:

// PostgreSQL schema
const schema = `
CREATE TABLE IF NOT EXISTS dead_letter_queue (
  id SERIAL PRIMARY KEY,
  webhook_id VARCHAR(255) UNIQUE NOT NULL,
  payload JSONB NOT NULL,
  error_type VARCHAR(255),
  error_message TEXT,
  attempts INT DEFAULT 1,
  first_failure_at TIMESTAMP DEFAULT NOW(),
  last_failure_at TIMESTAMP DEFAULT NOW(),
  status VARCHAR(50) DEFAULT 'pending',
  created_at TIMESTAMP DEFAULT NOW()
);

CREATE INDEX idx_dlq_status ON dead_letter_queue(status);
CREATE INDEX idx_dlq_first_failure ON dead_letter_queue(first_failure_at);
`;

// Store failed webhook
async function storeInDeadLetterQueue(webhook, error) {
  const errorType = classifyError(error, error.statusCode);

  try {
    await db.query(
      `INSERT INTO dead_letter_queue (webhook_id, payload, error_type, error_message)
       VALUES ($1, $2, $3, $4)
       ON CONFLICT (webhook_id) DO UPDATE SET
         attempts = attempts + 1,
         last_failure_at = NOW(),
         error_message = $4`,
      [webhook.id, JSON.stringify(webhook), errorType, error.message]
    );

    logger.warn('webhook_moved_to_dlq', {
      webhook_id: webhook.id,
      error_type: errorType
    });

  } catch (dbError) {
    logger.error('dlq_storage_failed', {
      webhook_id: webhook.id,
      error: dbError.message
    });
  }
}

// Reprocess dead letter queue
async function reprocessDeadLetterQueue() {
  const pendingWebhooks = await db.query(
    `SELECT * FROM dead_letter_queue
     WHERE status = 'pending'
     AND last_failure_at < NOW() - INTERVAL '1 hour'
     ORDER BY attempts ASC
     LIMIT 100`
  );

  for (const row of pendingWebhooks.rows) {
    try {
      await processWithRetry(row.payload);

      await db.query(
        `UPDATE dead_letter_queue SET status = 'processed' WHERE id = $1`,
        [row.id]
      );

      logger.info('dlq_webhook_reprocessed', {
        webhook_id: row.webhook_id,
        attempts: row.attempts
      });

    } catch (error) {
      logger.warn('dlq_reprocess_failed', {
        webhook_id: row.webhook_id,
        error: error.message
      });

      // Update last failure time, will retry again later
      await db.query(
        `UPDATE dead_letter_queue 
         SET last_failure_at = NOW()
         WHERE id = $1`,
        [row.id]
      );
    }
  }
}

// Run periodically (every 5 minutes)
setInterval(reprocessDeadLetterQueue, 5 * 60 * 1000);

4. Detecting Duplicates

Shopify retries webhooks when it doesn't receive a 200 response within 5 seconds. You must detect and handle duplicate webhook deliveries:

// Redis store for duplicate detection
const redis = require('redis');
const client = redis.createClient();

async function checkAndMarkProcessed(webhookId) {
  const key = `webhook:${webhookId}`;

  // Set with 48-hour expiration (Shopify retry window)
  const wasProcessed = await client.set(
    key,
    Date.now(),
    'EX',
    48 * 60 * 60,
    'NX' // Only set if key doesn't exist
  );

  if (!wasProcessed) {
    return true; // Already processed
  }

  return false; // First time seeing this webhook
}

// In your webhook handler
async function processWebhookAsync(body, webhookId, signature) {
  const isDuplicate = await checkAndMarkProcessed(webhookId);

  if (isDuplicate) {
    logger.info('webhook_duplicate_ignored', {
      webhook_id: webhookId,
      topic: body.topic
    });
    return;
  }

  // Process new webhook
  await processWithRetry(body);
}

Alternative (database-based duplicate detection using PostgreSQL unique constraint):

async function checkAndMarkProcessed(webhookId) {
  try {
    await db.query(
      `INSERT INTO processed_webhooks (webhook_id, processed_at)
       VALUES ($1, NOW())`,
      [webhookId]
    );
    return false; // First time
  } catch (error) {
    if (error.code === '23505') { // Unique constraint violation
      return true; // Already processed
    }
    throw error;
  }
}

5. Production Alerts

Set up automated alerts for critical metrics. These thresholds should trigger notifications to your team:

const alertThresholds = {
  deliveryRate: 0.99, // Alert if below 99%
  errorRate: 0.01, // Alert if above 1%
  p95Latency: 2000, // Alert if above 2s (ms)
  dlqSize: 100, // Alert if more than 100 pending
  noWebhooksIn: 3600 // Alert if no webhooks in 1 hour (seconds)
};

async function checkAlerts() {
  const metrics = await getMetrics();

  if (metrics.deliveryRate < alertThresholds.deliveryRate) {
    await sendAlert({
      severity: 'critical',
      title: 'Low webhook delivery rate',
      message: `Delivery rate: ${(metrics.deliveryRate * 100).toFixed(2)}%`,
      details: metrics
    });
  }

  if (metrics.p95Latency > alertThresholds.p95Latency) {
    await sendAlert({
      severity: 'warning',
      title: 'High webhook latency',
      message: `P95 latency: ${metrics.p95Latency}ms`,
      details: metrics
    });
  }

  if (metrics.dlqSize > alertThresholds.dlqSize) {
    await sendAlert({
      severity: 'warning',
      title: 'Dead letter queue growing',
      message: `${metrics.dlqSize} pending webhooks in DLQ`,
      details: metrics
    });
  }

  if (metrics.lastWebhookAge > alertThresholds.noWebhooksIn) {
    await sendAlert({
      severity: 'critical',
      title: 'No webhooks received',
      message: `Last webhook ${metrics.lastWebhookAge}s ago`,
      details: metrics
    });
  }
}

// Run every 5 minutes
setInterval(checkAlerts, 5 * 60 * 1000);

6. Testing Your Setup

Implement health check endpoints and test webhook functionality to verify your observability infrastructure is working:

// Health check endpoint
app.get('/health/webhooks', async (req, res) => {
  const metrics = await getMetrics();

  const isHealthy = 
    metrics.deliveryRate >= 0.99 &&
    metrics.errorRate <= 0.01 &&
    metrics.p95Latency <= 2000 &&
    metrics.dlqSize <= 100;

  res.status(isHealthy ? 200 : 503).json({
    status: isHealthy ? 'healthy' : 'degraded',
    metrics,
    timestamp: new Date().toISOString()
  });
});

// Test webhook endpoint
app.post('/webhooks/test', (req, res) => {
  const testWebhook = {
    id: 'test-' + Date.now(),
    topic: 'test/event',
    shop_id: 'test-shop',
    created_at: new Date().toISOString(),
    data: { test: true }
  };

  logger.info('test_webhook_received', testWebhook);

  res.status(200).json({ success: true });
});

Key Takeaways

Log structured data - JSON with context, not raw payloads
Track three metrics - Delivery rate, latency, error rate
Categorize errors - Different errors need different handling
Implement retries - Exponential backoff with jitter
Use dead letter queues - Prevent data loss
Detect duplicates - Shopify retries; you must handle it
Alert on thresholds - Know when things break
Test thoroughly - Load test at production scale

Resources

Ready to implement? Start with structured logging, then add metrics, then implement retries and DLQ. Each piece builds on the previous.

What webhook patterns have you encountered? Share in the comments!

Async Event Processing Architectures for Shopify — A Developer's Guide

Muhammad Masad Ashraf — Tue, 02 Jun 2026 23:17:46 +0000

Synchronous processing kills Shopify scalability. Every blocked operation is a timeout waiting to happen.

This guide covers the async event processing patterns that production Shopify systems actually use. We'll go from webhook receiver basics all the way to saga orchestration and event sourcing.

Why Async Matters for Shopify

Shopify fires webhooks for orders, fulfillments, inventory changes, customer updates, and more. If you process these synchronously inside your endpoint, you will hit timeouts, lose data, and bring down your system during traffic spikes.

The fix is simple in concept: accept fast, process later.

Your webhook endpoint acknowledges receipt in milliseconds. A background worker handles the actual business logic. This keeps your store responsive and your integrations stable.

Pattern 1: The Webhook Receiver

Your endpoint has one job: validate the request, push to a queue, return HTTP 200 — all within 5 seconds.

import express from 'express';
import crypto from 'crypto';
import { OrderQueue } from './queues';

const app = express();
app.use(express.raw({ type: 'application/json' }));

app.post('/webhooks/orders/create', async (req, res) => {
  // 1. Validate HMAC signature
  const hmac = req.headers['x-shopify-hmac-sha256'];
  const hash = crypto
    .createHmac('sha256', process.env.SHOPIFY_SECRET)
    .update(req.body)
    .digest('base64');

  if (hash !== hmac) {
    return res.status(401).send('Unauthorized');
  }

  // 2. Parse and enqueue
  const order = JSON.parse(req.body);
  await OrderQueue.add('process', order, {
    jobId: `order_${order.id}`, // Deduplication key
    attempts: 5,
    backoff: { type: 'exponential', delay: 2000 }
  });

  // 3. Respond immediately
  res.status(200).send('OK');
});

Never run ERP calls, inventory syncs, or email sends inside this handler. Push to queue and return.

Pattern 2: Idempotent Queue Workers

Shopify retries webhooks for 48 hours. You will receive duplicates. Every worker must handle this gracefully.

import { Worker } from 'bullmq';
import { redis } from './redis';
import { db } from './db';

const worker = new Worker('orders', async (job) => {
  const order = job.data;

  // Guard against duplicate processing
  const alreadyProcessed = await db.processedEvents.findUnique({
    where: { shopifyOrderId: String(order.id) }
  });

  if (alreadyProcessed) {
    console.log(`Order ${order.id} already processed. Skipping.`);
    return { skipped: true };
  }

  // Run actual business logic
  await Promise.allSettled([
    syncInventory(order),
    notifyFulfillmentTeam(order),
    updateAnalytics(order),
    syncToERP(order)
  ]);

  // Mark as processed (idempotency record)
  await db.processedEvents.create({
    data: { shopifyOrderId: String(order.id), processedAt: new Date() }
  });

  return { success: true };
}, { connection: redis });

The processedEvents table is your source of truth for deduplication. Always check it first.

Pattern 3: Dead Letter Queue Handling

Not all failures are transient. Some events fail because of bad data, schema mismatches, or downstream service bugs. These need to be isolated instead of retrying forever.

import { Queue, Worker, QueueEvents } from 'bullmq';

const mainQueue = new Queue('orders', { connection: redis });
const dlq = new Queue('orders-dlq', { connection: redis });

const worker = new Worker('orders', processOrder, {
  connection: redis,
});

worker.on('failed', async (job, err) => {
  if (job.attemptsMade >= job.opts.attempts) {
    console.error(`Moving job ${job.id} to DLQ`, err.message);

    await dlq.add('failed', {
      originalJob: job.data,
      error: err.message,
      failedAt: new Date().toISOString(),
      attempts: job.attemptsMade
    });
  }
});

Set up alerts on DLQ size. Any event in there needs human review.

Pattern 4: Out-of-Order Event Handling

Shopify makes no delivery order guarantees. A fulfillments/create webhook can arrive before orders/create. Your system must handle this.

async function handleFulfillmentCreated(event) {
  const order = await db.orders.findUnique({
    where: { shopifyId: String(event.order_id) }
  });

  // Order not yet created locally — reschedule
  if (!order) {
    console.warn(`Order ${event.order_id} not found, rescheduling fulfillment event`);
    await fulfillmentQueue.add('retry', event, {
      delay: 10000, // Wait 10 seconds before retry
      attempts: 10
    });
    return;
  }

  // Safe to proceed
  await db.fulfillments.create({
    data: {
      orderId: order.id,
      shopifyFulfillmentId: String(event.id),
      status: event.status,
      trackingNumber: event.tracking_number
    }
  });
}

Pattern 5: Saga Pattern for Multi-Service Coordination

When an order needs to update inventory, charge payment, and trigger fulfillment, you need transactional guarantees across services. Sagas handle this with compensation logic.

async function createOrderSaga(order) {
  const completedSteps = [];

  try {
    // Step 1
    await reserveInventory(order);
    completedSteps.push('inventory');

    // Step 2
    await chargePayment(order);
    completedSteps.push('payment');

    // Step 3
    await createFulfillment(order);
    completedSteps.push('fulfillment');

    // Step 4
    await sendOrderConfirmation(order);

    return { success: true };

  } catch (error) {
    console.error(`Saga failed at step after: ${completedSteps.join(', ')}`);

    // Compensate completed steps in reverse
    if (completedSteps.includes('payment')) {
      await refundPayment(order);
    }
    if (completedSteps.includes('inventory')) {
      await releaseInventory(order);
    }

    throw error;
  }
}

Sagas trade atomicity for availability. Each step must also be idempotent.

Pattern 6: Circuit Breaker for External APIs

When your ERP, WMS, or third-party API goes down, you don't want to hammer it with retries. Circuit breakers stop the calls automatically and resume when the service recovers.

import CircuitBreaker from 'opossum';

const options = {
  timeout: 5000,       // Fail after 5 seconds
  errorThresholdPercentage: 50, // Open after 50% failures
  resetTimeout: 30000  // Try again after 30 seconds
};

const erpBreaker = new CircuitBreaker(syncOrderToERP, options);

erpBreaker.fallback(async (order) => {
  // Queue for later instead of failing hard
  await retryQueue.add('erp-sync', order, { delay: 60000 });
  return { queued: true };
});

erpBreaker.on('open', () => {
  console.error('ERP circuit breaker OPEN - service is down');
});

erpBreaker.on('close', () => {
  console.log('ERP circuit breaker CLOSED - service recovered');
});

// Use in your worker
async function processOrder(order) {
  await erpBreaker.fire(order);
}

Queue Infrastructure Reference

Tool	Best For	Persistence	Delay Jobs	Priority
BullMQ (Redis)	Node.js apps	Yes	Yes	Yes
RabbitMQ	Multi-language	Yes	Plugin	Yes
AWS SQS	Serverless	Yes	Yes	Via FIFO
Google Pub/Sub	GCP workloads	Yes	No	No
Kafka	High-throughput streams	Yes	No	Partition-based

For most Shopify app use cases, BullMQ with Redis gives you the best balance of features and simplicity.

Monitoring Checklist

Track these metrics or you're flying blind:

Queue depth — growing consistently = you need more workers
Job processing time (p95) — slowdowns show up here first
Failure rate — spikes indicate downstream issues
DLQ size — any growth needs immediate investigation
Retry count — high values suggest a systemic problem

Log every state transition with structured fields:

logger.info('order.processing.started', {
  orderId: order.id,
  jobId: job.id,
  workerId: worker.id,
  timestamp: Date.now()
});

Key Takeaways

Respond to webhooks in under 5 seconds — always push to queue first
Make every handler idempotent — duplicates are guaranteed
Use exponential backoff — don't hammer failing services
Send to DLQ after max retries — don't retry poisoned messages forever
Handle out-of-order delivery — reschedule, don't crash
Use sagas for multi-step transactions — compensate on failure
Monitor queue depth, latency, and DLQ size continuously

Originally published on Kolachi Tech. Kolachi Tech specializes in Shopify development, integrations, and custom app architecture.

Shopify Webhooks Don't Arrive in Order - Here's How to Handle It

Muhammad Masad Ashraf — Fri, 29 May 2026 19:48:54 +0000

If you have ever built a Shopify integration and started seeing weird inventory counts, duplicate orders, or fulfillments firing before payment confirmation — welcome to the club.

The culprit is almost always the same: out-of-order webhook delivery.

Shopify does not guarantee the order in which webhook events arrive. That one sentence should be in every Shopify integration tutorial. It rarely is.

Let me break down why this happens and how to actually fix it.

Why Events Arrive Out of Order

Each webhook Shopify fires is an independent HTTP POST. No coordination. No sequence guarantee.

Here is what causes ordering to break:

Network latency — two requests take different paths, arrive in reverse
Retry logic — a failed event retried 30 minutes later arrives after newer events
Parallel workers — your consumer processes two events simultaneously; lighter one finishes first
FIFO queue trade-offs — standard queues (SQS, etc.) don't guarantee strict ordering

So you subscribe to orders/create and orders/updated. The update can genuinely arrive first. Your handler looks up the order, finds nothing, and either errors or skips. The order never enters your system.

The Damage This Causes

Here are the real scenarios I see teams hit:

1. Order Updated Before Created
Handler can't find the order. Skips silently. Order is now invisible to your system.

2. Fulfillment Before Payment
Your 3PL integration fires a pick request for an unpaid order. Policy violation at minimum. Financial exposure at worst.

3. Double Inventory Decrement
Flash sale. Rapid-fire inventory_levels/update events. Parallel consumers. Same decrement applies twice. Stock goes negative.

4. Cancellation Before Creation
orders/cancelled arrives. No record found. Discarded. Then orders/create arrives and creates an active order with no cancelled state.

The Fixes (Layered Approach)

No single fix solves this. You layer strategies based on your risk tolerance.

Fix 1: Timestamp Comparison

Before applying any event, compare its updated_at against your stored record:

if (incomingEvent.updated_at <= storedRecord.updated_at) {
  console.log('Stale event, discarding');
  return;
}
applyUpdate(incomingEvent);

Simple. Works well for orders/updated and inventory_levels/update.

Fix 2: Idempotent Handlers

Every handler must be safe to run twice. Use the X-Shopify-Webhook-Id header to track processed events:

const webhookId = req.headers['x-shopify-webhook-id'];

const alreadyProcessed = await db.webhookLog.findOne({ webhookId });
if (alreadyProcessed) return res.sendStatus(200); // acknowledge but skip

await db.webhookLog.create({ webhookId });
await processEvent(payload);

Always return 200 even when skipping. Shopify retries anything that isn't a 2xx.

Fix 3: State Machine for Orders

Model your order lifecycle explicitly. Reject invalid transitions:

const validTransitions = {
  null: ['created'],
  created: ['paid', 'cancelled'],
  paid: ['fulfilled', 'refunded', 'cancelled'],
  fulfilled: ['refunded'],
  cancelled: [],
};

function canTransition(currentState, newState) {
  return validTransitions[currentState]?.includes(newState) ?? false;
}

If orders/fulfilled arrives and your order is still in created state, that is an invalid transition. Route it to a dead letter queue, not the bin.

Fix 4: FIFO Queue Per Resource ID

Route all webhook events into a FIFO queue. Use the resource ID (order ID, product ID) as the partition key.

Shopify --> Webhook endpoint (ACK immediately)
        --> SQS FIFO Queue (MessageGroupId = order_id)
             --> Consumer (processes one event per order at a time)

This guarantees events for the same order process sequentially, not in parallel.

Fix 5: Fetch Before Acting

For critical operations, don't trust the payload alone. Fetch current state from the Shopify API before making changes:

async function handleFulfillment(payload) {
  const order = await shopify.order.get(payload.id);

  if (order.financial_status !== 'paid') {
    console.warn('Order not paid, skipping fulfillment');
    return;
  }

  await fulfillmentService.process(order);
}

Costs one extra API call. Eliminates an entire class of stale-state bugs.

Fix 6: Dead Letter Queue

Never silently discard events you can't process. Route them to a DLQ:

try {
  await processWebhook(event);
} catch (err) {
  if (err.type === 'UNPROCESSABLE') {
    await dlq.send(event);
  } else {
    throw err; // let the queue retry
  }
}

Review DLQ items daily. Most will be ordering artifacts you can replay manually.

Fix 7: Reconciliation Job

No ordering strategy is perfect. Run a scheduled job that compares your local state against Shopify's API:

// Runs every 30 minutes
async function reconcile() {
  const recentOrders = await shopify.order.list({ updated_at_min: thirtyMinutesAgo });

  for (const order of recentOrders) {
    const localOrder = await db.orders.findOne({ shopifyId: order.id });

    if (!localOrder || localOrder.status !== order.financial_status) {
      await syncOrder(order);
    }
  }
}

This is your safety net. It catches everything the real-time system misses.

The Full Architecture

Shopify
  --> Webhook Endpoint (ACK < 5s, no processing)
  --> FIFO Message Queue (partitioned by resource ID)
       --> Consumer Worker
            --> Dedup check (X-Shopify-Webhook-Id)
            --> Timestamp comparison (discard stale)
            --> State machine validation
            --> Fetch current state (for critical ops)
            --> Apply update
       --> Dead Letter Queue (unprocessable events)
  --> Reconciliation Job (every 30 min)

This handles duplicate events, out-of-order events, lost events, and stale overwrites.

Quick Reference

Problem	Fix
Update arrives before create	State machine + DLQ for invalid transitions
Duplicate events	Idempotency via X-Shopify-Webhook-Id
Stale overwrites	Timestamp comparison before applying
Parallel worker conflicts	FIFO queue per resource ID
Unresolvable ordering	Dead letter queue + manual review
Missed events	Reconciliation job

TL;DR

Shopify does not guarantee webhook event order
Network latency, retries, and parallel workers all cause out-of-order delivery
Layer your defenses: timestamps, idempotency, state machines, FIFO queues, DLQ, reconciliation
Always acknowledge webhooks immediately and process asynchronously
Never silently discard events you can't handle

Build for out-of-order delivery from day one and you will save yourself a lot of production debugging.

Have you hit webhook ordering bugs in production? Drop your war story in the comments.

Handling Duplicate Shopify Webhook Events (And Why You Must)

Muhammad Masad Ashraf — Mon, 25 May 2026 18:23:57 +0000

You built a Shopify integration. It works great in dev. Then in production,
a customer gets charged twice. Or an order ships twice. Or your inventory
goes negative overnight.

The culprit almost always? Duplicate webhook events.

Shopify guarantees at-least-once delivery, not exactly-once. Your endpoint
will receive the same event more than once. Here is how to handle it properly.

Why Duplicates Happen

Shopify retries a webhook if your server does not respond with 2xx within
5 seconds. It retries up to 19 times over 48 hours.

Duplicates hit you when:

Your server is slow to respond
A network timeout occurs mid-request
Your server restarts while processing
A queue consumer crashes and re-pulls the job

Step 1: Respond Immediately, Process Later

Never do heavy work inside the webhook handler. Respond fast, queue the work.

app.post('/webhooks/orders-paid', async (req, res) => {
  res.status(200).send('OK'); // Shopify gets its response immediately
  await queue.push({ topic: 'orders/paid', payload: req.body });
});

Step 2: Build a Dedup Key

Do NOT use X-Shopify-Webhook-Id as your dedup key. That header changes
on every retry attempt. Use the resource ID from the payload instead.

const dedupKey = `orders/paid:${payload.id}`;

This stays the same across all retries for the same event.

Step 3: Check Redis Before Processing

const alreadySeen = await redis.get(dedupKey);

if (alreadySeen) {
  console.log('Duplicate detected, skipping:', dedupKey);
  return;
}

await redis.setex(dedupKey, 86400, '1'); // TTL: 24 hours

Step 4: Add a Database Safety Net

Redis can go down. Your database should be the last line of defense.

CREATE TABLE processed_webhook_events (
  dedup_key VARCHAR(255) UNIQUE NOT NULL,
  processed_at TIMESTAMP DEFAULT NOW()
);

const result = await db.raw(`
  INSERT INTO processed_webhook_events (dedup_key)
  VALUES (?)
  ON CONFLICT (dedup_key) DO NOTHING
  RETURNING id
`, [dedupKey]);

if (result.rows.length === 0) return; // Already processed

The ON CONFLICT DO NOTHING is atomic. Even 10 concurrent requests for
the same event will only insert once.

Step 5: Make Your Handler Idempotent

Dedup catches most duplicates. Idempotent logic catches the rest.

For inventory, always set absolute values, never increment or decrement:

// BAD - breaks on duplicate
await db.inventory.decrement({ quantity: 5 });

// GOOD - safe to run multiple times
await db.inventory.update({ quantity: newAbsoluteValue });

High-Risk Events to Watch

Event	Risk	Fix
`orders/paid`	Double fulfillment	DB unique constraint
`inventory_levels/update`	Wrong stock count	Use absolute values
`refunds/create`	Double refund	Check refund ID first
`customers/create`	Duplicate accounts	Check email uniqueness

Quick Checklist Before You Ship

[ ] Webhook responds in under 5 seconds
[ ] Processing is async
[ ] Dedup key = topic + resource ID
[ ] Redis check at entry point
[ ] DB unique constraint as fallback
[ ] Inventory uses absolute values
[ ] Load tested with concurrent duplicate requests

That's the full pattern. Two layers of protection: Redis for speed,
database for correctness. Your handlers stay idempotent as a safety net.

Full guide with queue-level dedup (SQS + BullMQ) and monitoring setup
on our blog: kolachitech.com

How to Build a Shopify Webhook Replay System That Never Loses Events

Muhammad Masad Ashraf — Fri, 22 May 2026 15:41:32 +0000

Your Shopify integration fires perfectly in staging. Then production hits. Your server goes down for two minutes during a deploy. Shopify sends 40 webhook events. Your handler catches none of them.

That is real data loss. Orders not fulfilled. Inventory not updated. Customers not synced.

A webhook replay system is the fix. This guide walks you through building one that stores every incoming event, retries failures intelligently, and lets you replay any batch of events on demand.

Why Webhooks Fail in the First Place

Shopify retries failed webhooks up to 19 times over 48 hours. After that, you are on your own. Here is what causes failures:

Failure Cause	Description
Server downtime	Endpoint offline when Shopify fires
Timeout	Handler takes longer than 5 seconds
Code errors	Bug in your handler throws an exception
Rate limiting	System overloaded, rejects the request
DB unavailability	Database is down when handler tries to write
Deployment gaps	Deploy restarts server at the wrong moment

After 48 hours and 19 retries, Shopify flags your endpoint as failing. Eventually it stops sending events entirely.

A replay system gets you out of that hole.

The Core Architecture

A production webhook replay system has four stages:

Shopify
   |
   v
[Ingestion Endpoint]   <-- ACK immediately, store raw payload
   |
   v
[Event Store (DB)]     <-- durable record of every event
   |
   v
[Processing Queue]     <-- background worker picks up events
   |
   v
[Handler]              <-- idempotent processing logic
   |
   +-- Success --> mark 'succeeded'
   +-- Failure --> retry with backoff --> Dead Letter Queue

The key rule: never process inside the ingestion endpoint. Shopify waits only 5 seconds. Acknowledge fast, process separately.

Stage 1: The Ingestion Endpoint

Your ingestion endpoint does exactly two things:

Verify the HMAC signature
Write the raw payload to the event store and return 200 OK

app.post('/webhooks', express.raw({ type: 'application/json' }), async (req, res) => {
  const hmac = req.headers['x-shopify-hmac-sha256'];
  const webhookId = req.headers['x-shopify-webhook-id'];
  const topic = req.headers['x-shopify-topic'];

  // Verify HMAC first
  if (!verifyHmac(req.body, hmac)) {
    return res.status(401).send('Unauthorized');
  }

  // Store immediately, respond fast
  await db.query(`
    INSERT INTO webhook_events (shopify_id, topic, payload, hmac, status)
    VALUES ($1, $2, $3, $4, 'pending')
    ON CONFLICT (shopify_id) DO NOTHING
  `, [webhookId, topic, req.body.toString(), hmac]);

  res.status(200).send('OK');
});

The ON CONFLICT DO NOTHING handles Shopify's own retries. Same event, same ID, no duplicate row.

Stage 2: The Event Store Schema

CREATE TABLE webhook_events (
  id              UUID PRIMARY KEY DEFAULT gen_random_uuid(),
  shopify_id      VARCHAR(255) UNIQUE,
  topic           VARCHAR(100) NOT NULL,
  payload         JSONB NOT NULL,
  hmac            VARCHAR(255),
  status          VARCHAR(20) DEFAULT 'pending',
  retry_count     INT DEFAULT 0,
  last_error      TEXT,
  created_at      TIMESTAMPTZ DEFAULT NOW(),
  processed_at    TIMESTAMPTZ
);

CREATE INDEX idx_webhook_status ON webhook_events(status);
CREATE INDEX idx_webhook_topic  ON webhook_events(topic);
CREATE INDEX idx_webhook_created ON webhook_events(created_at);

Index on status, topic, and created_at. These are the three axes you will query for replay.

Storage options by scale:

Option	Best For
PostgreSQL	Most Shopify apps. Full SQL, JSONB indexing.
Redis Streams	High-throughput ingestion, consumer groups.
AWS SQS + DynamoDB	Managed infra, no ops overhead.
Kafka	Enterprise-scale event sourcing.

Stage 3: The Processing Worker

async function processWebhookEvents() {
  const events = await db.query(`
    SELECT * FROM webhook_events
    WHERE status = 'pending'
    ORDER BY created_at ASC
    LIMIT 50
    FOR UPDATE SKIP LOCKED
  `);

  for (const event of events.rows) {
    try {
      await handleEvent(event.topic, JSON.parse(event.payload));

      await db.query(`
        UPDATE webhook_events
        SET status = 'succeeded', processed_at = NOW()
        WHERE id = $1
      `, [event.id]);

    } catch (err) {
      const newRetryCount = event.retry_count + 1;
      const newStatus = newRetryCount >= 5 ? 'dead' : 'pending';

      await db.query(`
        UPDATE webhook_events
        SET status = $1, retry_count = $2, last_error = $3
        WHERE id = $4
      `, [newStatus, newRetryCount, err.message, event.id]);
    }
  }
}

FOR UPDATE SKIP LOCKED prevents multiple workers from grabbing the same event. Safe for concurrent processing.

Stage 4: Exponential Backoff

Do not retry immediately. Back off with jitter:

function getRetryDelay(retryCount) {
  const baseDelays = [30, 120, 600, 3600, 21600]; // seconds
  const base = (baseDelays[retryCount] || 21600) * 1000;
  const jitter = Math.random() * 1000;
  return base + jitter;
}

Retry	Delay
1	30 seconds
2	2 minutes
3	10 minutes
4	1 hour
5+	6 hours

Jitter prevents retry storms when many events fail simultaneously.

Stage 5: The Replay Trigger

This is the part that earns the "replay" in "replay system." When you fix a bug and need to reprocess failed events:

async function replayEvents({ topic, status = 'failed', from, to, batchSize = 100 }) {
  const events = await db.query(`
    SELECT * FROM webhook_events
    WHERE status = $1
      AND ($2::text IS NULL OR topic = $2)
      AND ($3::timestamptz IS NULL OR created_at >= $3)
      AND ($4::timestamptz IS NULL OR created_at <= $4)
    ORDER BY created_at ASC
    LIMIT $5
  `, [status, topic, from, to, batchSize]);

  for (const event of events.rows) {
    // Reset to pending so the worker picks it up
    await db.query(`
      UPDATE webhook_events
      SET status = 'pending', retry_count = 0, last_error = NULL
      WHERE id = $1
    `, [event.id]);
  }

  return events.rowCount;
}

// Usage: replay all failed orders/paid from the last 24 hours
await replayEvents({
  topic: 'orders/paid',
  status: 'failed',
  from: new Date(Date.now() - 86400000)
});

Idempotency: The Non-Negotiable Requirement

When you replay, your handler runs twice for the same payload. Without idempotency, you get duplicate orders, double charges, corrupted inventory.

Use the shopify_id (the X-Shopify-Webhook-Id) as your deduplication key:

async function handleOrderCreate(payload) {
  const shopifyOrderId = payload.id;

  // Check if already processed
  const existing = await db.query(
    'SELECT id FROM orders WHERE shopify_order_id = $1',
    [shopifyOrderId]
  );

  if (existing.rows.length > 0) {
    console.log(`Order ${shopifyOrderId} already processed. Skipping.`);
    return;
  }

  // Safe to process
  await db.query(
    'INSERT INTO orders (shopify_order_id, ...) VALUES ($1, ...)',
    [shopifyOrderId, ...]
  );
}

No idempotency = no safe replay. Build this first.

Dead Letter Queue

Events that hit max retries go to DLQ status. Monitor these aggressively:

// Query DLQ events
const dlqEvents = await db.query(`
  SELECT topic, COUNT(*) as count, MIN(created_at) as oldest
  FROM webhook_events
  WHERE status = 'dead'
  GROUP BY topic
  ORDER BY count DESC
`);

DLQ growth is not a metric to check weekly. Alert on it in real time. Every row is an event that never reached its destination.

When to Replay What

Scenario	Action
Server down for 2 hours	Automatic retry handles it
Bug deployed and fixed	Manual replay by time range
New integration added	Manual replay to backfill history
Downstream API was down	Automatic retry with backoff
Data migration	Manual replay by topic and date

Mistakes That Kill Replay Systems

Not storing the raw payload. Store the exact bytes Shopify sent. If you transform first, you cannot replay the original.

Processing in the ingestion endpoint. 5-second window. Any business logic risks timeout and a missed event.

No idempotency. Replay becomes dangerous instead of safe.

Unlimited retries. Cap at 5-7. Move the rest to DLQ.

Ignoring DLQ depth. Silent DLQ growth means silent data loss.

Tools to Accelerate This

Tool	Role
BullMQ (Node.js)	Queue + retry + backoff out of the box
Sidekiq (Ruby)	Background jobs with retry logic
AWS SQS + Lambda	Fully managed, serverless
Temporal	Durable workflows with built-in replay
pg-boss	PostgreSQL-backed job queue

Wrapping Up

The pattern is simple:

Ingest fast, store everything
Process asynchronously
Retry with backoff
Route terminal failures to DLQ
Replay on demand after fixes Every layer compounds reliability. Start with the event store. Add the worker. Add retry logic. Then add the replay trigger.

Your Shopify store generates events constantly. A replay system makes sure none of them disappear.

Originally published at kolachitech.com

Stop Losing Shopify Webhooks: A Retry Strategy That Survives Real Outages

Muhammad Masad Ashraf — Fri, 22 May 2026 00:40:32 +0000

Stop Losing Shopify Webhooks: A Retry Strategy That Survives Real Outages

Here is a scenario every Shopify developer eventually lives through.

It is the middle of a deploy. Your endpoint is down for maybe 90 minutes. Nothing dramatic. But while you were shipping, three orders came in, an inventory sync fired, and a customer updated their address.

Shopify tried to deliver those webhooks. It retried. Then it gave up.

Those events are gone. Not delayed. Gone.

I have watched this exact thing happen, and the worst part is how quiet it is. No error in your logs. No alert. Just a slow drift between what Shopify knows and what your system knows. You find out days later when a customer asks where their order went.

So let us talk about how to actually fix this, properly, with a retry strategy that survives more than a quick hiccup.

First, Know What Shopify Actually Does

You cannot build a good retry layer until you know what the platform gives you for free.

When Shopify sends a webhook, it waits 5 seconds for your endpoint to respond. Any 2xx status code is a success. Anything else, including a timeout, is a failure.

After a failure, Shopify retries. And here is the part people get wrong, because the internet is full of outdated info:

As of the September 2024 policy update, Shopify retries a failed webhook up to 8 times over a 4-hour window with exponential backoff.

The old "19 retries over 48 hours" number you will find in older blog posts is dead. If your reliability code was written before late 2024, your timing assumptions are probably wrong.

A quick reference:

Behavior	Detail
Timeout	5 seconds per attempt
Success	Any 2xx status
Retries	Up to 8 attempts
Window	4 hours total
Backoff	Exponential
Payload	Original payload from trigger time
The scary part	Persistent failures delete the subscription

That last row is the one that ruins weekends. If your endpoint fails enough, Shopify does not just drop events. It removes the webhook subscription. New events stop firing entirely until you re-register. Silent and total.

The Problem With 4 Hours

Exponential backoff over 8 attempts in 4 hours front-loads everything. Most of your retries happen in the first 30 minutes. By hour 2, roughly 5 of your 8 attempts are spent.

So Shopify's retry system is genuinely good at one thing: surviving short, transient blips. A momentary network drop. A one-off timeout.

It is bad at the thing that actually hurts you: real outages.

A deploy that runs long
A downstream API that rate-limits you for hours
A traffic spike during a sale that pushes responses past 5 seconds
A botched HMAC secret rotation that 401s every webhook

In all of those, the 4-hour window runs out before you recover. Shopify did its job. Your data is still lost.

The conclusion is simple: you need your own retry layer. Shopify's is the floor, not the ceiling.

Rule Zero: Acknowledge Fast, Process Later

Before any retry logic, fix the most common mistake in Shopify webhook code: running business logic inside the webhook endpoint.

If your handler hits a database, calls an external API, or does anything heavy, you are gambling against that 5-second timeout. Cross it once and Shopify marks a perfectly good delivery as failed.

Your endpoint should do almost nothing:

// The entire job of your webhook endpoint
app.post("/webhooks/orders-create", async (req, res) => {
  // 1. Verify the HMAC signature
  if (!verifyHmac(req)) return res.sendStatus(401);

  // 2. Push the raw payload onto a queue
  await queue.add("orders-create", {
    body: req.rawBody,
    eventId: req.headers["x-shopify-event-id"],
    triggeredAt: req.headers["x-shopify-triggered-at"],
  });

  // 3. Respond immediately
  res.sendStatus(200);
});

That is it. A background worker pulls from the queue and does the real work. This one change eliminates the majority of timeout failures, and it turns retries into a calm internal concern instead of a race against a clock.

Building the Retry Layer

Your retry logic lives in the worker, not the endpoint. When processing fails, the worker decides: retry, or give up?

Step 1: Classify the error

Not every failure deserves a retry. Retrying a permanent error just burns resources.

Error type	Examples	Action
Transient	Timeout, 503, deadlock, rate limit	Retry with backoff
Permanent	Invalid payload, missing field, validation error	Straight to the dead letter queue

Retrying a malformed payload 8 times will not magically make it valid. Categorize first, act second.

Step 2: Exponential backoff with jitter

For transient errors, retry with increasing delays. Each retry waits longer than the last.

Attempt	Delay
1	30 seconds
2	2 minutes
3	8 minutes
4	30 minutes
5	2 hours
6	6 hours
7	24 hours

Notice the window stretches across days, not Shopify's 4 hours. That is the whole point of building your own layer.

But pure exponential backoff has a trap. If 500 webhooks fail at the same moment, they all retry at the same moment, and your recovering service gets hammered flat again. This is the thundering herd.

Fix it with jitter: add a small random offset to each delay. Instead of retrying at exactly 8 minutes, retry somewhere between 7 and 9. It spreads the load and smooths recovery.

function nextDelay(attempt) {
  const base = Math.min(30 * 2 ** attempt, 86400); // cap at 24h
  const jitter = base * (Math.random() * 0.3); // up to 30% jitter
  return base + jitter;
}

Step 3: Cap retries, then use a dead letter queue

Retries cannot run forever. Cap them, usually somewhere between 5 and 10 attempts. When a webhook exhausts its retries, it does not vanish. It moves to a dead letter queue.

The DLQ stores the failed webhook with full context: original payload, error message, attempt count, timestamp. Now you can:

Inspect it and find the root cause
Reprocess it once the bug is fixed
Alert when the DLQ grows past a normal threshold

That alert is your early warning system. A spike in DLQ volume tells you something is wrong before any customer notices.

The Duplicate Problem Nobody Mentions

Retries create a brand new problem: the same webhook arriving twice.

Shopify delivers a webhook, your slow response times out, Shopify retries, but your worker already processed the first one. Now you have a duplicate. Without protection, that means double inventory adjustments, duplicate emails, double-charged orders.

The fix is idempotency. Every Shopify webhook carries an X-Shopify-Event-Id header, and that value stays identical across every retry of the same event.

Use it as a dedup key:

async function processWebhook(job) {
  const { eventId } = job.data;

  if (await alreadyProcessed(eventId)) {
    return; // safe no-op
  }

  await doTheRealWork(job.data);
  await markProcessed(eventId);
}

A retry strategy without deduplication is incomplete. Build this in from day one.

Watch Out for Stale Payloads

One more subtlety. When Shopify retries, it sends the original payload from when the event was triggered, not the current state.

If an order changed three times during the retry window, a late retry still carries the first version. Apply it blindly and you overwrite newer data with older data.

Always check the X-Shopify-Triggered-At header against your own records. If your data is already newer, skip the stale update or fetch fresh data from the Admin API instead.

The Layer Even Retries Cannot Save You From

Here is the uncomfortable truth: even a perfect retry layer cannot recover an event Shopify never sent, or one it dropped after its own retries failed.

For that, you need reconciliation. Periodically poll the Shopify Admin API and compare its data against yours. Hourly for high-value topics like orders/*. Daily for lower-stakes data.

Reconciliation is the final safety net. It catches events lost to extended outages, dropped events, and gaps from a removed subscription.

Combine the three layers and you lose almost nothing:

Layer	Catches
Shopify retries	Short transient blips
Your retry layer + DLQ	Longer outages, downstream failures
Reconciliation	Everything else, including dropped events

Do Not Forget Monitoring

A retry strategy fails silently without eyes on it. Track these:

Webhook failure rate
DLQ size and growth
Retry volume
Subscription status (run a daily check, alert on any missing topic)
Processing latency creeping toward 5 seconds

Shopify's Dev Dashboard has a delivery metrics report with response codes and retry counts per topic. Use it alongside your own monitoring, not instead of it.

TL;DR

Shopify gives you 8 retries over 4 hours. Real outages last longer than that. So:

Acknowledge fast. Return 200 in under 5 seconds, queue the payload.
Classify errors. Transient gets retried, permanent goes to the DLQ.
Exponential backoff with jitter. Retry over days, not hours.
Idempotency. Use X-Shopify-Event-Id to kill duplicates.
Dead letter queue. Nothing gets lost, everything is replayable.
Reconcile. Poll the Admin API to catch what slipped through.
Monitor. Surface problems before customers do.

Build these layers once and your integration stops losing data, even on its worst day.

If you want the full guide, you can read it here.

How does your team handle webhook reliability? I would genuinely like to hear what has worked for you in the comments.

Building a Dead Letter Queue for Shopify Webhooks (Production-Ready Guide)

Muhammad Masad Ashraf — Mon, 18 May 2026 16:02:08 +0000

Building a Dead Letter Queue for Shopify Webhooks

Your Shopify webhook just failed. Network timeout. Database deadlock. Third-party API down.

The webhook payload? Gone forever.

Unless you have a dead letter queue.

This guide walks through building a production-ready DLQ system for Shopify webhooks. I'll show you the architecture, code examples, and mistakes to avoid based on real production systems.

Why Shopify's Built-in Retry Isn't Enough

Shopify retries failed webhooks up to 19 times over 48 hours. After that, they stop delivering to your endpoint.

Attempt 1: Immediate
Attempt 2: 1 second later
Attempt 3: 2 seconds later
...
Attempt 19: 48 hours later
Then: 💀 Webhook deleted forever

For production systems handling payments, orders, and inventory, this isn't acceptable. You need a safety net.

Architecture Overview

Here's the high-level flow:

Shopify → Your Webhook Endpoint → Primary Queue → Worker → Success ✓
                    ↓                                    ↓
              200 OK (always)                        Failure ✗
                                                          ↓
                                                   Dead Letter Queue
                                                          ↓
                                                   Retry Logic

Critical principle: Always return 200 OK to Shopify immediately. Process webhooks asynchronously.

The Stack

I'll use this stack for examples (adapt to your needs):

Queue: Redis with Bull (Node.js) or AWS SQS
Database: PostgreSQL for DLQ storage
Runtime: Node.js with Express
Monitoring: Custom dashboard + alerts

Implementation: Step by Step

1. Webhook Reception (Never Block Here)

// webhook-receiver.js
const express = require('express');
const crypto = require('crypto');
const { primaryQueue } = require('./queues');

const app = express();

app.post('/webhooks/shopify/:topic', express.raw({ type: 'application/json' }), async (req, res) => {
  const topic = req.params.topic;
  const hmac = req.get('X-Shopify-Hmac-Sha256');
  const body = req.body;

  // Verify webhook signature
  if (!verifyWebhook(body, hmac)) {
    return res.status(401).send('Invalid signature');
  }

  // Respond to Shopify IMMEDIATELY
  res.status(200).send('OK');

  // Queue for async processing (never blocks)
  try {
    await primaryQueue.add(topic, {
      topic,
      shopDomain: req.get('X-Shopify-Shop-Domain'),
      webhookId: req.get('X-Shopify-Webhook-Id'),
      payload: JSON.parse(body.toString()),
      receivedAt: new Date().toISOString()
    }, {
      attempts: 6,
      backoff: {
        type: 'exponential',
        delay: 10000 // Start with 10 seconds
      }
    });
  } catch (error) {
    // If queuing fails, write directly to DLQ
    await saveToDLQ({
      topic,
      payload: JSON.parse(body.toString()),
      error: 'Failed to queue webhook',
      failureReason: error.message
    });
  }
});

function verifyWebhook(body, hmac) {
  const hash = crypto
    .createHmac('sha256', process.env.SHOPIFY_WEBHOOK_SECRET)
    .update(body)
    .digest('base64');
  return hash === hmac;
}

2. Worker with DLQ Fallback

// webhook-worker.js
const { primaryQueue, dlqQueue } = require('./queues');
const { processWebhook } = require('./processors');
const { saveToDLQ, incrementRetryCount } = require('./dlq-storage');

primaryQueue.process(async (job) => {
  const { topic, payload, webhookId, shopDomain } = job.data;

  try {
    // Your business logic here
    await processWebhook(topic, payload, shopDomain);

    // If we got here, processing succeeded
    return { success: true, webhookId };

  } catch (error) {
    // Categorize the error
    if (isTransientError(error)) {
      // Let Bull retry with exponential backoff
      throw error;
    } else {
      // Permanent error - move to DLQ immediately
      await saveToDLQ({
        webhookId,
        topic,
        shopDomain,
        payload,
        error: error.message,
        stackTrace: error.stack,
        retryCount: job.attemptsMade,
        firstFailedAt: job.timestamp,
        lastAttemptedAt: new Date().toISOString()
      });

      // Don't throw - we handled it
      return { success: false, movedToDLQ: true };
    }
  }
});

// When all retries exhausted, Bull calls this
primaryQueue.on('failed', async (job, error) => {
  if (job.attemptsMade >= job.opts.attempts) {
    await saveToDLQ({
      webhookId: job.data.webhookId,
      topic: job.data.topic,
      shopDomain: job.data.shopDomain,
      payload: job.data.payload,
      error: 'Max retries exceeded',
      stackTrace: error.stack,
      retryCount: job.attemptsMade,
      firstFailedAt: job.timestamp,
      lastAttemptedAt: new Date().toISOString()
    });
  }
});

function isTransientError(error) {
  // Network issues, rate limits, timeouts
  return error.code === 'ECONNREFUSED' ||
         error.code === 'ETIMEDOUT' ||
         error.statusCode === 429 ||
         error.statusCode === 503 ||
         error.message.includes('deadlock');
}

3. DLQ Storage Schema

-- dlq_webhooks table
CREATE TABLE dlq_webhooks (
  id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
  webhook_id VARCHAR(255) UNIQUE,
  topic VARCHAR(100) NOT NULL,
  shop_domain VARCHAR(255) NOT NULL,
  payload JSONB NOT NULL,
  error_message TEXT,
  stack_trace TEXT,
  retry_count INTEGER DEFAULT 0,
  first_failed_at TIMESTAMP NOT NULL,
  last_attempted_at TIMESTAMP NOT NULL,
  status VARCHAR(50) DEFAULT 'pending', -- pending, processing, resolved, abandoned
  resolved_at TIMESTAMP,
  notes TEXT,
  created_at TIMESTAMP DEFAULT NOW()
);

-- Indexes for querying
CREATE INDEX idx_dlq_status ON dlq_webhooks(status);
CREATE INDEX idx_dlq_topic ON dlq_webhooks(topic);
CREATE INDEX idx_dlq_shop ON dlq_webhooks(shop_domain);
CREATE INDEX idx_dlq_first_failed ON dlq_webhooks(first_failed_at);

4. DLQ Storage Functions

// dlq-storage.js
const { Pool } = require('pg');
const pool = new Pool();

async function saveToDLQ(webhookData) {
  const {
    webhookId,
    topic,
    shopDomain,
    payload,
    error,
    stackTrace,
    retryCount = 0,
    firstFailedAt,
    lastAttemptedAt
  } = webhookData;

  const query = `
    INSERT INTO dlq_webhooks (
      webhook_id, topic, shop_domain, payload,
      error_message, stack_trace, retry_count,
      first_failed_at, last_attempted_at
    ) VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9)
    ON CONFLICT (webhook_id) 
    DO UPDATE SET
      retry_count = dlq_webhooks.retry_count + 1,
      last_attempted_at = $9,
      error_message = $5,
      stack_trace = $6
    RETURNING id
  `;

  try {
    const result = await pool.query(query, [
      webhookId,
      topic,
      shopDomain,
      JSON.stringify(payload),
      error,
      stackTrace,
      retryCount,
      firstFailedAt || new Date(),
      lastAttemptedAt || new Date()
    ]);

    // Alert if this is a new DLQ entry
    if (result.rowCount > 0) {
      await alertDLQEntry(webhookId, topic, shopDomain);
    }

    return result.rows[0];
  } catch (err) {
    console.error('Failed to save to DLQ:', err);
    // This is serious - DLQ itself failed
    // Log to external service (Sentry, etc.)
  }
}

async function getDLQWebhooks(status = 'pending', limit = 100) {
  const query = `
    SELECT * FROM dlq_webhooks
    WHERE status = $1
    ORDER BY first_failed_at ASC
    LIMIT $2
  `;

  const result = await pool.query(query, [status, limit]);
  return result.rows;
}

async function markAsResolved(webhookId, notes = null) {
  const query = `
    UPDATE dlq_webhooks
    SET status = 'resolved',
        resolved_at = NOW(),
        notes = $2
    WHERE webhook_id = $1
  `;

  await pool.query(query, [webhookId, notes]);
}

module.exports = {
  saveToDLQ,
  getDLQWebhooks,
  markAsResolved
};

5. Automated DLQ Retry Worker

// dlq-retry-worker.js
const { getDLQWebhooks, markAsResolved, saveToDLQ } = require('./dlq-storage');
const { processWebhook } = require('./processors');

async function processDLQWebhooks() {
  const webhooks = await getDLQWebhooks('pending', 50);

  for (const webhook of webhooks) {
    try {
      // Update status to prevent concurrent processing
      await updateDLQStatus(webhook.webhook_id, 'processing');

      // Try to process again
      await processWebhook(
        webhook.topic,
        webhook.payload,
        webhook.shop_domain
      );

      // Success! Mark as resolved
      await markAsResolved(webhook.webhook_id, 'Auto-resolved by retry worker');

    } catch (error) {
      // Still failing
      const newRetryCount = webhook.retry_count + 1;

      if (newRetryCount >= 10) {
        // Give up after 10 DLQ retries
        await updateDLQStatus(webhook.webhook_id, 'abandoned');
        await alertPermanentFailure(webhook);
      } else {
        // Save back to DLQ with updated count
        await saveToDLQ({
          ...webhook,
          retryCount: newRetryCount,
          error: error.message,
          stackTrace: error.stack,
          lastAttemptedAt: new Date().toISOString()
        });
      }
    }
  }
}

// Run every 5 minutes
setInterval(processDLQWebhooks, 5 * 60 * 1000);

Monitoring Dashboard

// dlq-dashboard.js - Express route
app.get('/api/dlq/stats', async (req, res) => {
  const stats = await pool.query(`
    SELECT
      status,
      COUNT(*) as count,
      MIN(first_failed_at) as oldest,
      MAX(first_failed_at) as newest
    FROM dlq_webhooks
    GROUP BY status
  `);

  const byTopic = await pool.query(`
    SELECT
      topic,
      COUNT(*) as count
    FROM dlq_webhooks
    WHERE status = 'pending'
    GROUP BY topic
    ORDER BY count DESC
  `);

  res.json({
    stats: stats.rows,
    byTopic: byTopic.rows
  });
});

Alerting Setup

// alerting.js
const { sendSlackAlert } = require('./slack');

async function alertDLQEntry(webhookId, topic, shopDomain) {
  // Check DLQ depth
  const depth = await getDLQDepth();

  if (depth > 100) {
    await sendSlackAlert({
      channel: '#alerts-critical',
      text: `🚨 DLQ depth exceeds 100: ${depth} webhooks`,
      priority: 'critical'
    });
  }
}

async function alertPermanentFailure(webhook) {
  await sendSlackAlert({
    channel: '#alerts-high',
    text: `⚠️ Webhook permanently failed after 10 retries\nTopic: ${webhook.topic}\nShop: ${webhook.shop_domain}\nWebhook ID: ${webhook.webhook_id}`,
    priority: 'high'
  });
}

Testing Your DLQ

// test-dlq.js
describe('Dead Letter Queue', () => {
  it('should save webhook to DLQ on processing failure', async () => {
    const testWebhook = {
      webhookId: 'test-123',
      topic: 'orders/create',
      shopDomain: 'test-shop.myshopify.com',
      payload: { id: 123, total: 50.00 },
      error: 'Database connection failed',
      stackTrace: 'Error: ...',
      retryCount: 3
    };

    await saveToDLQ(testWebhook);

    const dlqWebhooks = await getDLQWebhooks('pending');
    expect(dlqWebhooks).toHaveLength(1);
    expect(dlqWebhooks[0].webhook_id).toBe('test-123');
  });

  it('should retry webhook from DLQ', async () => {
    // Test retry logic
  });

  it('should mark webhook as abandoned after max retries', async () => {
    // Test abandonment logic
  });
});

Production Considerations

Idempotency

Store processed webhook IDs to prevent duplicate processing:

async function processWebhook(topic, payload, shopDomain) {
  const webhookId = payload.id || payload.admin_graphql_api_id;

  // Check if already processed
  const alreadyProcessed = await redis.get(`processed:${webhookId}`);
  if (alreadyProcessed) {
    console.log(`Webhook ${webhookId} already processed, skipping`);
    return;
  }

  // Process webhook
  await yourBusinessLogic(payload);

  // Mark as processed (expire after 7 days)
  await redis.setex(`processed:${webhookId}`, 7 * 24 * 60 * 60, '1');
}

Exponential Backoff Configuration

const RETRY_CONFIG = {
  attempts: 6,
  backoff: {
    type: 'exponential',
    delay: 10000, // 10 seconds
    options: {
      maxDelay: 1800000, // 30 minutes max
    }
  }
};

// Results in retry schedule:
// 1. 10 seconds
// 2. 1 minute
// 3. 5 minutes
// 4. 15 minutes
// 5. 30 minutes
// 6. 30 minutes

Memory Management

For high-volume stores:

// Process DLQ in batches
async function processDLQBatch(batchSize = 50) {
  const webhooks = await getDLQWebhooks('pending', batchSize);

  // Process in parallel with concurrency limit
  await Promise.all(
    webhooks.map(webhook => 
      limiter.schedule(() => retryWebhook(webhook))
    )
  );
}

// Use p-limit for concurrency control
const pLimit = require('p-limit');
const limiter = pLimit(10); // Max 10 concurrent

Common Mistakes to Avoid

❌ Processing webhooks synchronously in the HTTP handler

// DON'T DO THIS
app.post('/webhook', async (req, res) => {
  await processWebhook(req.body); // Blocks!
  res.status(200).send('OK'); // Too late, Shopify timed out
});

❌ No maximum retry limit

// DON'T DO THIS
while (!success) {
  await retry(); // Infinite loop
}

❌ Not categorizing errors

// DON'T DO THIS
catch (error) {
  throw error; // Always retry, even for permanent failures
}

Monitoring Checklist

Monitor these metrics:

✅ DLQ depth (alert > 100)
✅ Age of oldest message (alert > 4 hours)
✅ Retry success rate
✅ Error patterns (group by error type)
✅ Processing latency (p50, p95, p99)
✅ Webhook topics in DLQ (which events fail most)

Resources

Full Implementation

The complete working implementation with all the code above is available on our blog:
👉 Dead Letter Queue Shopify Webhooks: Complete Guide

What's your approach to webhook reliability? Drop your setup in the comments below.

Building Shopify integrations? We help e-commerce brands build fault-tolerant systems. Check out Kolachi Tech for more technical deep-dives.

DEV Community: Muhammad Masad Ashraf

Shopify GraphQL Pagination: How to Handle Large Datasets Without Slowing Down Your App

What Shopify GraphQL Pagination Solves

Shopify Uses Cursor-Based Pagination

Why Cursors Work Better Than Page Numbers

Page Size Is a Performance Decision

Watch GraphQL Query Cost

A Reliable Pagination Loop

Use Filters Before Pagination

When to Use Bulk Operations

Use Webhooks for Real-Time Updates

Common Mistakes

Final Thoughts

Shopify Bulk Operations API at Scale: A Practical Developer Guide

What Is the Shopify Bulk Operations API?

The Operation Lifecycle

Step 1: Submit

Step 2: Poll for Status

Step 3: Download and Parse

Why This Beats Standard Pagination at Scale

Key Constraints You Must Know

One operation per store at a time

No nested mutations

Stream the output file

Check partialDataUrl on failure

Bulk Mutations: The Real Power Move

Scaling Patterns for Production Systems

Pattern 1: Webhook-Triggered Completion

Pattern 2: Job Orchestration Layer

Pattern 3: Chunked Mutation Strategy

Pattern 4: JSONL Processing Pipeline

Error Codes and Retry Logic

Performance Benchmarks

App Architecture Implications

Wrapping Up

Shopify GraphQL Query Cost Optimization: Stop Burning API Points

How the Leaky Bucket Works

How Cost Gets Calculated

The problem with nested connections

The optimized version

7 Strategies to Reduce GraphQL Points

1. Request Only the Fields You Need

2. Reduce Connection Page Sizes

3. Split Deeply Nested Queries

4. Use Query Cost Extensions for Real-Time Monitoring

5. Build Leaky Bucket Awareness Into Your Client

6. Cache Infrequently-Changing Data

7. Use Direct GID Lookups Over List Queries

Use Bulk Operations for Large Data Sets

Handling Throttle Errors

Common Mistakes at a Glance

Pre-Ship Checklist

Wrapping Up

How to Verify Shopify Webhooks Correctly (And Stop Getting It Wrong)

Why Your Webhook Endpoint Is a Target

What Shopify Actually Does

The Node.js Implementation

The Python Implementation

The Mistakes That Break Verification (And Ship Vulnerabilities)

Mistake 1: Parsing before verifying

Mistake 2: Using === for comparison

Mistake 3: No replay protection

Mistake 4: No shop domain validation

Mistake 5: Storing the secret in code

The Production Checklist

Why Acknowledging Immediately Matters

What to Read Next

Scaling Shopify Webhooks to Handle Millions of Events: A Practical Guide

Scaling Shopify Webhooks to Handle Millions of Events: A Practical Guide

The Problem: Synchronous Processing Doesn't Scale

Solution 1: Queue-Based Processing

Solution 2: Dead Letter Queues for Failure Handling

Solution 3: Idempotency (The Silent Killer)

Solution 4: Event-Driven Architecture

The Monitoring You Need

Implementation Checklist

Common Gotchas

Resources

Read the Complete Guide

Shopify Webhook Observability: Complete Implementation Guide

Check `partialDataUrl` on failure

Mistake 2: Using `===` for comparison