Raizan

Posted on Aug 17 • Originally published at chasebot.online

How To Implement Idempotent Webhook Payload Processing

#tutorial #programming #webhooks #backend

What You'll Need

n8n Cloud or self-hosted n8n
Hetzner VPS or Contabo VPS for hosting
DigitalOcean as alternative hosting

Understanding Idempotency in Webhook Processing
Designing an Atomic Idempotency Layer with Redis
Implementing Full Idempotent Webhook Processing in Node.js
Building Idempotent Workflows in n8n
Getting Started

Understanding Idempotency in Webhook Processing

Every developer running webhooks in production eventually faces duplicate events. Provider platforms like Stripe, Shopify, GitHub, and Twilio operate on an at-least-once delivery model. If your endpoint takes longer than a few seconds to return a 200 OK status code, or if a transient network packet drop occurs on the response path, the provider will retry sending the exact same payload.

Without explicit safeguards, duplicate inbound payloads lead to severe backend inconsistencies. You end up with double-charged credit cards, duplicate database records, corrupted inventory levels, or multiple transactional emails sent to customers.

Idempotency guarantees that an operation produces the exact same side-effects regardless of how many times it receives the same input request. When processing webhooks, an idempotent system identifies whether a specific payload has already been received, is currently being processed by another thread, or was successfully handled in the past.

When hosting microservices on a cloud provider or self-hosting on a high-performance instance like a Hetzner VPS, handling concurrency properly requires a centralized mechanism. You can read our detailed breakdown on selecting optimal compute infrastructure in our guide on How to Set Up a VPS for Automation (Hetzner vs Contabo vs Railway).

Webhook idempotency requires three primary steps:

Extracting or generating a unique deterministic key for the incoming payload.
Acquiring an atomic lock before executing downstream business logic.
Storing and returning the final execution result for subsequent duplicate incoming calls.

Before scaling out your webhook receivers, you should also protect your HTTP infrastructure against denial of service scenarios or sudden provider burst traffic. Learn how to restrict traffic floods by reading our guide on Protecting Self Hosted Endpoints with Rate Limiting.

💡 Fast-Track Your Project: Don't want to configure this yourself? I build custom n8n pipelines and bots. Message me with code SYS3-DEVTO.

Designing an Atomic Idempotency Layer with Redis

To prevent race conditions where two identical webhook payloads hit separate container instances simultaneously, local in-memory storage like Node.js Map objects will fail. You need a fast, centralized key-value store with atomic primitives. Redis is the standard choice for this architecture.

Our state machine uses three primary states for every payload key:

PROCESSING: Set atomically using Redis SET key value NX PX ttl when the payload first arrives. This acts as a distributed lock.
COMPLETED: Set after downstream business logic finishes successfully. It stores the cached HTTP response status and body.
FAILED: Optional state set if processing encounters a fatal error, allowing immediate retry on subsequent webhook attempts.

Here is how the atomic key creation logic works:

              Inbound Webhook Received
                         │
                         ▼
             Extract Idempotency Key
                         │
                         ▼
           Redis Query: GET idempotency:key
                         │
        ┌────────────────┴────────────────┐
        │                                 │
   Key Exists?                        Key Absent?
        │                                 │
   ┌────┴─────────────────────────┐       ▼
   │                              │  Atomic SET NX PX
   ▼                              ▼  (State: PROCESSING)
State == COMPLETED        State == PROCESSING     │
   │                              │               ▼
   ▼                              ▼      Execute Downstream
Return Cached             Return 409      Business Logic
Response (200 OK)         Conflict                │
                                                  ▼
                                         Update Redis Key
                                         (State: COMPLETED + Response)

By relying on SET NX (Set if Not Exists), Redis guarantees that exactly one thread can transition a key from non-existent to PROCESSING at any microsecond.

Implementing Full Idempotent Webhook Processing in Node.js

Below is a complete production-grade Express.js application implementing atomic idempotency checks using the ioredis client library. It handles payload signature hashing, lock acquisition, duplicate rejection, error recovery, and cached response returns.

import express from 'express';
import Redis from 'ioredis';
import crypto from 'crypto';

const app = express();
app.use(express.json());

const redis = new Redis({
  host: process.env.REDIS_HOST || '127.0.0.1',
  port: Number(process.env.REDIS_PORT) || 6379,
});

const LOCK_TTL_MS = 30000;
const RECORD_TTL_SEC = 86400;

function deriveIdempotencyKey(req) {
  const headerKey = req.headers['x-idempotency-key'] || req.headers['stripe-signature'];
  if (headerKey) {
    return `idempotency:${headerKey}`;
  }
  const payloadHash = crypto
    .createHash('sha256')
    .update(JSON.stringify(req.body))
    .digest('hex');
  return `idempotency:hash:${payloadHash}`;
}

async function simulateDatabaseWrite(data) {
  return new Promise((resolve) => {
    setTimeout(() => {
      resolve({
        transactionId: `txn_${Date.now()}`,
        status: 'SUCCESS',
        account: data.accountId,
        amount: data.amount,
      });
    }, 1500);
  });
}

app.post('/api/v1/webhooks/payment', async (req, res) => {
  const idempotencyKey = deriveIdempotencyKey(req);
  const lockAcquired = await redis.set(
    idempotencyKey,
    JSON.stringify({ status: 'PROCESSING', startedAt: Date.now() }),
    'PX',
    LOCK_TTL_MS,
    'NX'
  );

  if (!lockAcquired) {
    const rawData = await redis.get(idempotencyKey);
    if (!rawData) {
      return res.status(409).json({
        error: 'Concurrent request conflict. Please retry in a few seconds.',
      });
    }

    const existingRecord = JSON.parse(rawData);

    if (existingRecord.status === 'COMPLETED') {
      return res.status(existingRecord.responseStatus).json({
        ...existingRecord.responseBody,
        _idempotent_replayed: true,
      });
    }

    if (existingRecord.status === 'PROCESSING') {
      return res.status(409).json({
        error: 'Webhook event is currently being processed by another worker.',
        _idempotent_retryable: true,
      });
    }
  }

  try {
    const dbResult = await simulateDatabaseWrite(req.body);

    const successResponse = {
      message: 'Payment processed successfully',
      result: dbResult,
    };

    const completionPayload = {
      status: 'COMPLETED',
      responseStatus: 200,
      responseBody: successResponse,
      completedAt: Date.now(),
    };

    await redis.set(
      idempotencyKey,
      JSON.stringify(completionPayload),
      'EX',
      RECORD_TTL_SEC
    );

    return res.status(200).json(successResponse);
  } catch (err) {
    await redis.del(idempotencyKey);
    return res.status(500).json({
      error: 'Internal processing failure. Lock cleared for retry.',
      details: err.message,
    });
  }
});

const PORT = process.env.PORT || 3000;
app.listen(PORT, () => {
  console.log(`Webhook server listening on port ${PORT}`);
});

This implementation prevents double execution. If the same payload arrives five times concurrently, one process gets the lock, three get 409 Conflict errors while processing occurs, and subsequent requests after completion receive the exact cached success response.

Building Idempotent Workflows in n8n

Low-code workflow tools like n8n often suffer from duplicate executions when webhooks trigger workflow runs asynchronously. If you host n8n, managing your compute footprint becomes essential. Check out our detailed pricing breakdown in Windmill vs n8n vs Make workflow pricing 2026 to analyze execution costs across self-hosted and cloud platforms.

You can implement the exact same atomic Redis locking strategy directly inside an n8n workflow using custom JavaScript nodes and the standard Redis integration.

Here is the exact code snippet to plug into an n8n Code Node immediately after receiving a Webhook trigger to calculate the payload hash and evaluate lock state:

const crypto = require('crypto');

const rawPayload = $input.item.json.body || $input.item.json;
const headers = $input.item.json.headers || {};

let idempotencyKey = headers['x-idempotency-key'] || headers['x-delivery'];

if (!idempotencyKey) {
  const hash = crypto.createHash('sha256');
  hash.update(JSON.stringify(rawPayload));
  idempotencyKey = hash.digest('hex');
}

const redisKey = `n8n:idempotency:${idempotencyKey}`;

return [
  {
    json: {
      redisKey: redisKey,
      idempotencyKey: idempotencyKey,
      payload: rawPayload,
      processedAt: new Date().toISOString()
    }
  }
];

Following the Code Node, connect a Redis Node configured with the command SET using arguments:

Key: ={{ $json.redisKey }}
Value: PROCESSING
Options: NX enabled, EX set to 60 seconds

Use an n8n Switch Node to branch execution based on the Redis Node output:

If Redis returned OK: Route to your database nodes, API calls, and email notifications. At the end of the success path, insert a second Redis node setting the key to COMPLETED with a 24-hour expiration (EX 86400).
If Redis returned null: Route immediately to a Respond to Webhook Node returning HTTP Status 200 with response JSON: {"status": "ignored", "reason": "duplicate_event"}.

This pattern isolates your heavy automation paths, preventing external retry storms from degrading your server performance or blowing through API usage quotas.

Getting Started

To deploy production-ready idempotent endpoints and scalable automation pipelines, set up your infrastructure with these providers:

Spin up a cloud workflow engine using n8n Cloud or self-host for unlimited executions.
Deploy robust Redis and Node.js microservices on a high-speed Hetzner VPS or budget-friendly Contabo VPS.
Scale isolated cloud services using DigitalOcean Droplets.

Outsource Your Automation

Don't have time? I build production n8n workflows, WhatsApp bots, and fully automated YouTube Shorts pipelines. Hire me on Fiverr, mention SYS3-DEVTO for priority. Or DM at chasebot.online.

Originally published on Automation Insider.

DEV Community