kevin.s

Posted on Jul 16 • Edited on Jul 28

Build a Payment Automation Studio for Crypto Merchants

#webdev #api #crypto #backend

The payment succeeded.

The customer still has no access.

OxaPay reports the invoice as paid, but the SaaS plan remains inactive. The payment callback reached the merchant's server, yet the downstream account service timed out.

The payment system worked.

The automation around it failed.

This distinction matters because payment is rarely the final business outcome.

A confirmed payment may need to:

deliver a digital product
activate a SaaS plan
provision a hosting account
assign a software license
grant Telegram or Discord access
update an order
update a CRM
notify a finance team
create a support case
write a reporting record
call a merchant-owned API

A Payment Automation Studio turns verified payment events into controlled business actions.

It is not a webhook relay.

It is not a collection of if status === "paid" statements.

It is an execution system that can prove:

which event arrived
whether the event was authentic
which workflow version matched
which conditions passed
which actions were attempted
which actions completed
which actions failed
whether an action was retried
whether approval was required
whether the final business outcome was verified
whether a human must intervene

This article uses OxaPay as the payment infrastructure reference, but the automation architecture is provider-agnostic.

This article is part of 10 Crypto Payment Products Developers Can Build for Merchants.

The real automation model

The simplest payment automation looks like this:

Payment paid
-> Activate SaaS plan

That works in a demo.

A production workflow needs more stages:

Payment event
-> Verify authenticity
-> Preserve evidence
-> Normalize event
-> Validate state transition
-> Match published workflow
-> Evaluate conditions
-> Verify current provider state
-> Execute actions idempotently
-> Verify outcomes
-> Retry or escalate
-> Record final result

The reliable model is:

Verified event
-> Versioned policy
-> Controlled actions
-> Verified outcome

The difference between a webhook script and an automation product is everything around the action.

Do not promise exactly-once delivery

Payment callbacks, queues, and distributed workers normally behave with at-least-once delivery.

That means:

the same callback may arrive twice
the same queue job may run twice
a worker may crash after completing an external action
the response to an external API may be lost
a recovery job may rediscover an earlier payment

The system should not pretend these events happen exactly once.

Instead, design for effectively-once business execution:

At-least-once event delivery
+
Event deduplication
+
Business-transition deduplication
+
Action-level idempotency
+
Outcome reconciliation
=
Effectively-once merchant outcome

For example, a callback may be processed several times, but the customer must receive only one license.

Where the Automation Studio fits

A Crypto PaymentOps Service manages the broader operational payment lifecycle.

A Crypto Payment Reconciliation Tool detects disagreements between payment, order, fulfillment, and finance records.

A Crypto Payment Support Desk helps agents investigate and resolve incidents.

The Automation Studio has a narrower responsibility:

Convert verified payment events into repeatable business actions.

For example:

OxaPay payment becomes paid
-> Verify order
-> Activate subscription
-> Send confirmation
-> Update CRM
-> Notify merchant

Reconciliation may later detect that activation did not happen.

The Support Desk may help resolve the incident.

The Automation Studio owns the workflow execution itself.

It is not an n8n clone

A useful Payment Automation Studio does not need to compete with every general automation platform.

Its advantage is payment-specific control.

It understands:

payment states
track_id
merchant order_id
amount verification
confirmation policy
underpayments
expired sessions
refunds
fulfillment idempotency
payment recovery
paid-but-not-completed incidents

A generic automation tool can connect APIs.

A payment-specific product can understand whether a business action is safe.

Separate the control plane from the execution plane

The product has two major areas.

Control plane

The control plane lets merchants and operators:

create workflows
configure triggers
define conditions
select actions
connect credentials
validate definitions
run simulations
publish versions
pause workflows
review execution history
approve sensitive actions
replay failed executions

Execution plane

The execution plane:

receives events
validates signatures
stores evidence
matches published workflows
creates executions
claims action jobs
calls connectors
retries failures
verifies outcomes
opens operational cases

Do not let workflow-editing requests execute business actions directly.

Publishing and execution should remain separate.

Define workflows as data

A workflow should not exist only as hardcoded application logic.

Represent it as a versioned definition.

name: Deliver license after confirmed payment
version: 4

trigger:
  source: oxapay
  type: payment.status_changed
  transition: paid

conditions:
  all:
    - field: event.order_id
      operator: exists

    - field: event.status
      operator: equals
      value: paid

    - field: order.status
      operator: not_in
      value:
        - canceled
        - refunded

actions:
  - id: load_order
    type: internal.get_order
    input:
      order_id: "{{ event.order_id }}"

  - id: verify_payment
    type: oxapay.verify_payment
    depends_on:
      - load_order
    input:
      track_id: "{{ event.track_id }}"
      expected_order_id: "{{ order.external_order_id }}"
      expected_amount: "{{ order.total }}"

  - id: assign_license
    type: license.assign
    depends_on:
      - verify_payment
    input:
      product_id: "{{ order.product_id }}"
      order_id: "{{ order.id }}"
      customer_email: "{{ order.customer_email }}"

  - id: send_delivery
    type: email.send
    depends_on:
      - assign_license
    input:
      to: "{{ order.customer_email }}"
      template: license_delivery
      license_key: "{{ actions.assign_license.license_key }}"

  - id: notify_merchant
    type: telegram.send_message
    depends_on:
      - assign_license
    input:
      chat_id: "{{ merchant.telegram_chat_id }}"
      message: "Order {{ order.id }} was paid and delivered."

retry_policy:
  max_attempts: 5
  strategy: exponential

failure_policy:
  create_case: true
  notify_merchant: true

This definition contains:

Trigger
Conditions
Action graph
Dependencies
Retry policy
Failure policy

A merchant can understand the business flow.

A developer can validate, version, test, and audit it.

Use an explicit workflow lifecycle

A workflow should move through controlled states.

draft
-> validated
-> published
-> paused
-> retired

Draft

The merchant can edit the workflow.

It cannot process live events.

Validated

The workflow has passed structural and security checks.

Published

An immutable version can process new events.

Paused

The version remains available for investigation, but no new execution starts.

Retired

The workflow is no longer available for new events.

Existing executions still reference it.

Do not allow an editable draft to become production logic automatically.

Validate workflows before publishing

Validation should check:

trigger type is supported
all action IDs are unique
dependencies reference valid actions
the action graph has no cycles
required connector credentials exist
condition operators are permitted
template variables are valid
sensitive actions include approval policy
retry limits are within platform bounds
generic HTTP destinations satisfy security rules
the workflow has at least one terminal outcome

export function validateWorkflowDefinition(
  definition,
) {
  const errors = [];

  if (!definition.trigger?.type) {
    errors.push("Workflow trigger is required");
  }

  const actionIds = new Set();

  for (const action of definition.actions ?? []) {
    if (!action.id) {
      errors.push("Every action requires an id");
      continue;
    }

    if (actionIds.has(action.id)) {
      errors.push(
        `Duplicate action id: ${action.id}`,
      );
    }

    actionIds.add(action.id);
  }

  for (const action of definition.actions ?? []) {
    for (const dependency of action.depends_on ?? []) {
      if (!actionIds.has(dependency)) {
        errors.push(
          `Action ${action.id} depends on unknown action ${dependency}`,
        );
      }
    }
  }

  if (containsDependencyCycle(definition.actions ?? [])) {
    errors.push("Workflow contains a dependency cycle");
  }

  if (errors.length > 0) {
    throw new WorkflowValidationError(errors);
  }
}

The published version should be immutable.

Editing creates a new draft version.

Pin every execution to one version

Suppose a merchant edits a workflow after an incident.

The historical execution must still show the exact logic used at that time.

Workflow: Deliver License
Published version: 4

Execution: EX-1042
Workflow version: 3
Payment event: EVT-5508

The execution stays attached to version 3.

Without immutable versions, the dashboard may display today's workflow while explaining an event processed under yesterday's rules.

The production architecture

+----------------------+
| Merchant Checkout    |
+----------+-----------+
           |
           | Create payment
           v
+----------------------+
| OxaPay API           |
+----------+-----------+
           |
           | Signed callback
           v
+----------------------+
| Webhook Gateway      |
+----------+-----------+
           |
           | Validate and persist
           v
+----------------------+
| Event Inbox          |
+----------+-----------+
           |
           v
+----------------------+
| Transactional Outbox |
+----------+-----------+
           |
           v
+----------------------+
| Event Normalizer     |
+----------+-----------+
           |
           v
+----------------------+
| Workflow Matcher     |
+----------+-----------+
           |
           v
+----------------------+
| Execution Planner    |
+----------+-----------+
           |
           v
+----------------------+
| Action Queue         |
+----------+-----------+
           |
      +----+------------------+
      |                       |
      v                       v
Connector Workers       Approval Queue
      |                       |
      +-----------+-----------+
                  |
                  v
+----------------------+
| Outcome Verification |
+----------+-----------+
           |
           v
Retry / Replay / Review / Metrics

The webhook endpoint should not:

activate subscriptions
assign licenses
provision servers
send customer emails
update CRMs
initiate payouts

Its job is limited:

Identify the merchant endpoint.
Preserve the raw request body.
validate the HMAC signature.
Persist the event durably.
Create an outbox record.
Return the required response.

Slow and failure-prone work belongs in workers.

Use a tenant-specific webhook endpoint

A multi-merchant system needs the correct Merchant API Key before it can validate an OxaPay payment callback.

Do not trust an unverified payload to select that key.

Use a merchant-specific public endpoint identifier:

https://studio.example.com/webhooks/oxapay/{endpoint_id}

The endpoint ID:

identifies the merchant integration
is not the Merchant API Key
provides no dashboard access
can be rotated independently
should be difficult to guess

The backend uses it to load the encrypted Merchant API Key before validating the request.

Receive OxaPay callbacks safely

OxaPay payment callbacks include an HMAC header calculated with SHA-512 over the raw request body.

The Merchant API Key is used to validate payment callbacks.

import crypto from "node:crypto";
import express from "express";

const app = express();

app.post(
  "/webhooks/oxapay/:endpointId",
  express.raw({
    type: "application/json",
    limit: "256kb",
  }),
  async (req, res) => {
    const endpoint =
      await db.webhookEndpoint.findUnique({
        where: {
          publicId: req.params.endpointId,
        },
        include: {
          merchant: true,
        },
      });

    if (!endpoint || !endpoint.enabled) {
      return res.status(404).send("unknown endpoint");
    }

    const rawBody = req.body;
    const receivedHmac = req.get("HMAC");

    const merchantApiKey = await decryptSecret(
      endpoint.merchant
        .oxapayMerchantApiKeyEncrypted,
    );

    const expectedHmac = crypto
      .createHmac("sha512", merchantApiKey)
      .update(rawBody)
      .digest("hex");

    if (!safeEqualSha512(receivedHmac, expectedHmac)) {
      await recordRejectedWebhook({
        merchantId: endpoint.merchantId,
        endpointId: endpoint.id,
        reason: "invalid_hmac",
        receivedAt: new Date(),
      });

      return res.status(401).send("invalid signature");
    }

    let payload;

    try {
      payload = JSON.parse(
        rawBody.toString("utf8"),
      );
    } catch {
      return res.status(400).send("invalid json");
    }

    const payloadHash = crypto
      .createHash("sha256")
      .update(rawBody)
      .digest("hex");

    try {
      await persistEventAndOutbox({
        merchantId: endpoint.merchantId,
        endpointId: endpoint.id,
        provider: "oxapay",
        payloadHash,
        rawPayload: payload,
        receivedAt: new Date(),
      });

      return res.status(200).send("ok");
    } catch (error) {
      console.error(
        "Webhook persistence failed",
        error,
      );

      return res.status(500).send("failed");
    }
  },
);

function safeEqualSha512(received, expected) {
  const sha512Hex = /^[a-f0-9]{128}$/i;

  if (
    !received ||
    !expected ||
    !sha512Hex.test(received) ||
    !sha512Hex.test(expected)
  ) {
    return false;
  }

  return crypto.timingSafeEqual(
    Buffer.from(received, "hex"),
    Buffer.from(expected, "hex"),
  );
}

Do not parse the body with normal JSON middleware before HMAC verification.

The exact raw bytes must remain available.

Use a transactional outbox

Consider this sequence:

1. Store payment event
2. Publish queue message

The database write succeeds.

The queue becomes unavailable before step 2.

The event exists, but no workflow will process it.

Use one database transaction:

Insert event
+
Insert outbox record
+
Commit

A separate dispatcher publishes pending outbox records.

async function persistEventAndOutbox({
  merchantId,
  endpointId,
  provider,
  payloadHash,
  rawPayload,
  receivedAt,
}) {
  return db.$transaction(async (tx) => {
    const existing =
      await tx.paymentEvent.findUnique({
        where: {
          merchantId_payloadHash: {
            merchantId,
            payloadHash,
          },
        },
      });

    if (existing) {
      return existing;
    }

    const event = await tx.paymentEvent.create({
      data: {
        merchantId,
        endpointId,
        provider,
        source: "webhook",
        payloadHash,
        rawPayload,
        signatureValid: true,
        receivedAt,
      },
    });

    await tx.outboxJob.create({
      data: {
        merchantId,
        topic: "payment.event_received",
        payload: {
          paymentEventId: event.id,
        },
      },
    });

    return event;
  });
}

The queue dispatcher can retry without losing the original event.

Use a normalized event envelope

Do not let provider-specific payloads spread across workflow definitions.

Create an internal event envelope.

import crypto from "node:crypto";

export function normalizeOxaPayEvent({
  merchantId,
  paymentEventId,
  payload,
  source,
}) {
  return {
    eventId: crypto.randomUUID(),
    storedEventId: paymentEventId,
    merchantId,
    provider: "oxapay",
    type: "payment.status_changed",
    source,
    correlationId:
      String(payload.track_id ?? paymentEventId),
    causationId: paymentEventId,
    causationDepth: 0,
    occurredAt: payload.date
      ? new Date(payload.date)
      : new Date(),
    data: {
      trackId: String(payload.track_id ?? ""),
      orderId: payload.order_id
        ? String(payload.order_id)
        : null,
      status: String(payload.status ?? "")
        .trim()
        .toLowerCase(),
      amount: payload.amount ?? null,
      currency: payload.currency ?? null,
      network: payload.network ?? null,
      transactionHash:
        payload.tx_hash ??
        payload.transaction_hash ??
        null,
    },
    providerPayload: payload,
  };
}

Workflow conditions then evaluate:

event.data.status
event.data.orderId
event.data.amount
event.data.currency

instead of using provider-specific fields throughout the codebase.

Use correlation and causation IDs

Payment automation can create new events.

For example:

Payment paid
-> Activate subscription
-> Subscription activated event
-> Send onboarding email

Without causation tracking, one workflow may trigger another workflow that recreates the original event.

This can produce an automation loop.

Every internal event should record:

correlationId: groups events belonging to one business process
causationId: identifies the event or action that created this event
causationDepth: limits how far an event chain can continue

Example:

Payment event EVT-1
correlation_id = track_8182
causation_id = webhook_5508
depth = 0

Subscription event EVT-2
correlation_id = track_8182
causation_id = action_activate_subscription
depth = 1

Prevent workflow loops

A workflow should not trigger itself indefinitely.

Use several controls:

maximum causation depth
workflow execution deduplication
action-generated event metadata
trigger filters
suppression of identical transitions
merchant-configured loop rules

function assertEventChainAllowed(event) {
  const MAX_CAUSATION_DEPTH = 10;

  if (
    event.causationDepth >
    MAX_CAUSATION_DEPTH
  ) {
    throw new PermanentWorkflowError(
      "Maximum automation chain depth exceeded",
    );
  }
}

Also prevent one workflow version from processing the same business transition twice:

workflow_version_id
+
provider_track_id
+
normalized_transition

Separate three forms of deduplication

Exact event deduplication

The same raw callback arrives more than once.

merchant_id + payload_hash

Business-transition deduplication

Different payloads represent the same transition.

merchant_id
+ provider
+ track_id
+ transition_to_paid

Action idempotency

The same business transition attempts the same action again.

merchant_id
+ workflow_version_id
+ track_id
+ transition
+ action_id

Example:

m_42:wf_9_v4:track_8182:paid:assign_license

All three are necessary.

A raw-body hash alone cannot prevent two different paid payloads from assigning two licenses.

A practical data model

CREATE TABLE payment_events (
  id UUID PRIMARY KEY,
  merchant_id UUID NOT NULL,
  endpoint_id UUID,
  provider TEXT NOT NULL,
  source TEXT NOT NULL,
  payload_hash TEXT NOT NULL,
  provider_track_id TEXT,
  provider_order_id TEXT,
  provider_status TEXT,
  normalized_status TEXT,
  correlation_id TEXT,
  causation_id TEXT,
  raw_payload JSONB NOT NULL,
  signature_valid BOOLEAN NOT NULL,
  received_at TIMESTAMP NOT NULL DEFAULT NOW(),
  UNIQUE (merchant_id, payload_hash)
);

CREATE TABLE business_transitions (
  id UUID PRIMARY KEY,
  merchant_id UUID NOT NULL,
  provider TEXT NOT NULL,
  provider_track_id TEXT NOT NULL,
  transition_key TEXT NOT NULL,
  payment_event_id UUID NOT NULL REFERENCES payment_events(id),
  created_at TIMESTAMP NOT NULL DEFAULT NOW(),
  UNIQUE (
    merchant_id,
    provider,
    provider_track_id,
    transition_key
  )
);

CREATE TABLE workflows (
  id UUID PRIMARY KEY,
  merchant_id UUID NOT NULL,
  name TEXT NOT NULL,
  lifecycle_status TEXT NOT NULL DEFAULT 'draft',
  current_draft_version INTEGER,
  published_version INTEGER,
  created_at TIMESTAMP NOT NULL DEFAULT NOW(),
  updated_at TIMESTAMP NOT NULL DEFAULT NOW()
);

CREATE TABLE workflow_versions (
  id UUID PRIMARY KEY,
  workflow_id UUID NOT NULL REFERENCES workflows(id),
  version INTEGER NOT NULL,
  definition JSONB NOT NULL,
  definition_hash TEXT NOT NULL,
  lifecycle_status TEXT NOT NULL,
  created_by UUID,
  published_by UUID,
  created_at TIMESTAMP NOT NULL DEFAULT NOW(),
  published_at TIMESTAMP,
  UNIQUE (workflow_id, version)
);

CREATE TABLE workflow_executions (
  id UUID PRIMARY KEY,
  merchant_id UUID NOT NULL,
  workflow_version_id UUID NOT NULL REFERENCES workflow_versions(id),
  payment_event_id UUID NOT NULL REFERENCES payment_events(id),
  business_transition_id UUID REFERENCES business_transitions(id),
  correlation_id TEXT NOT NULL,
  causation_id TEXT,
  status TEXT NOT NULL,
  context JSONB NOT NULL,
  started_at TIMESTAMP,
  completed_at TIMESTAMP,
  created_at TIMESTAMP NOT NULL DEFAULT NOW(),
  UNIQUE (
    workflow_version_id,
    business_transition_id
  )
);

CREATE TABLE action_executions (
  id UUID PRIMARY KEY,
  workflow_execution_id UUID NOT NULL REFERENCES workflow_executions(id),
  action_id TEXT NOT NULL,
  action_type TEXT NOT NULL,
  idempotency_key TEXT NOT NULL UNIQUE,
  status TEXT NOT NULL,
  attempt_count INTEGER NOT NULL DEFAULT 0,
  lease_owner TEXT,
  lease_expires_at TIMESTAMP,
  request_data JSONB,
  result_data JSONB,
  outcome_verified BOOLEAN NOT NULL DEFAULT FALSE,
  last_error TEXT,
  next_attempt_at TIMESTAMP,
  started_at TIMESTAMP,
  completed_at TIMESTAMP,
  created_at TIMESTAMP NOT NULL DEFAULT NOW(),
  UNIQUE (workflow_execution_id, action_id)
);

CREATE TABLE connector_credentials (
  id UUID PRIMARY KEY,
  merchant_id UUID NOT NULL,
  connector_type TEXT NOT NULL,
  display_name TEXT NOT NULL,
  encrypted_secret JSONB NOT NULL,
  status TEXT NOT NULL DEFAULT 'active',
  last_verified_at TIMESTAMP,
  rotated_at TIMESTAMP,
  created_at TIMESTAMP NOT NULL DEFAULT NOW()
);

CREATE TABLE approval_requests (
  id UUID PRIMARY KEY,
  merchant_id UUID NOT NULL,
  workflow_execution_id UUID NOT NULL REFERENCES workflow_executions(id),
  action_execution_id UUID NOT NULL REFERENCES action_executions(id),
  approval_type TEXT NOT NULL,
  snapshot_hash TEXT NOT NULL,
  status TEXT NOT NULL DEFAULT 'pending',
  requested_at TIMESTAMP NOT NULL DEFAULT NOW(),
  decided_at TIMESTAMP,
  decided_by UUID,
  decision_reason TEXT
);

CREATE TABLE execution_cases (
  id UUID PRIMARY KEY,
  merchant_id UUID NOT NULL,
  workflow_execution_id UUID REFERENCES workflow_executions(id),
  action_execution_id UUID REFERENCES action_executions(id),
  case_type TEXT NOT NULL,
  severity TEXT NOT NULL,
  status TEXT NOT NULL DEFAULT 'open',
  summary TEXT NOT NULL,
  evidence JSONB,
  recommended_action TEXT,
  resolution_note TEXT,
  created_at TIMESTAMP NOT NULL DEFAULT NOW(),
  resolved_at TIMESTAMP
);

CREATE TABLE outbox_jobs (
  id UUID PRIMARY KEY,
  merchant_id UUID NOT NULL,
  topic TEXT NOT NULL,
  payload JSONB NOT NULL,
  status TEXT NOT NULL DEFAULT 'pending',
  attempt_count INTEGER NOT NULL DEFAULT 0,
  available_at TIMESTAMP NOT NULL DEFAULT NOW(),
  published_at TIMESTAMP,
  created_at TIMESTAMP NOT NULL DEFAULT NOW()
);

The schema separates:

Evidence
Business transition
Workflow definition
Workflow execution
Action execution
Credential
Approval
Operational case

That separation makes investigation and replay possible.

Prevent invalid payment-state regression

Distributed events may arrive late or out of order.

The local payment state should not move silently from a stronger state back to a weaker state.

For example:

paid -> paying

should normally be rejected.

However:

paid -> refunding -> refunded

may be valid.

const ALLOWED_TRANSITIONS = {
  new: new Set([
    "waiting",
    "paying",
    "paid",
    "underpaid",
    "expired",
  ]),

  waiting: new Set([
    "paying",
    "paid",
    "underpaid",
    "expired",
  ]),

  paying: new Set([
    "paid",
    "underpaid",
    "expired",
  ]),

  paid: new Set([
    "refunding",
    "refunded",
  ]),

  underpaid: new Set([
    "manual_accept",
    "paid",
    "expired",
  ]),

  manual_accept: new Set([
    "refunding",
    "refunded",
  ]),

  refunding: new Set([
    "refunded",
  ]),

  refunded: new Set(),
  expired: new Set(),
};

export function canTransition(
  currentStatus,
  nextStatus,
) {
  if (
    !currentStatus ||
    currentStatus === nextStatus
  ) {
    return true;
  }

  return (
    ALLOWED_TRANSITIONS[
      currentStatus
    ]?.has(nextStatus) ?? false
  );
}

Merchant policy may require additional transitions.

The important point is that late events must not silently reverse business state.

Use a restricted condition language

Do not execute arbitrary merchant JavaScript inside the workflow engine.

Allow a small set of operators:

equals
not_equals
exists
not_exists
greater_than
less_than
in
not_in
starts_with
contains

const CONDITION_OPERATORS = {
  equals: (actual, expected) =>
    actual === expected,

  not_equals: (actual, expected) =>
    actual !== expected,

  exists: (actual) =>
    actual !== null &&
    actual !== undefined,

  in: (actual, expected) =>
    Array.isArray(expected) &&
    expected.includes(actual),

  greater_than: (actual, expected) =>
    decimal(actual).greaterThan(
      decimal(expected),
    ),
};

Validate field paths and operators when the workflow is published.

Do not let merchants access arbitrary process memory, environment variables, or internal database fields.

Add dry runs before live execution

A merchant should be able to test a workflow without sending real emails, assigning real licenses, or changing a production account.

Dry run mode should:

load a sample event
evaluate triggers
evaluate conditions
render action inputs
validate credentials
show which actions would run
replace side effects with simulated results
report missing data
report policy violations

Example result:

Workflow: Deliver License
Version: Draft 5

Trigger matched: yes
Conditions passed: 3 of 3

Action load_order:
Would execute

Action verify_payment:
Would execute

Action assign_license:
Simulation only

Action send_delivery:
Simulation only

Warnings:
Customer email is missing

A dry run tests one event.

Historical simulation tests many.

Simulate against historical events

Before publishing a changed workflow, run it against previous payment events.

Example:

Simulation period: Last 30 days
Events tested: 1,842
Would match: 1,730
Would skip: 112
Would create permanent failure: 18
Would require approval: 4
Potential duplicate action keys: 0

Historical simulation helps detect:

overbroad conditions
missing fields
unintended matches
newly blocked actions
changes in action volume
possible automation loops

The simulation must not call production connectors.

Match only published workflow versions

The matcher should load published versions for the merchant and trigger type.

export async function findMatchingWorkflows({
  merchantId,
  event,
}) {
  const versions =
    await db.workflowVersion.findMany({
      where: {
        lifecycleStatus: "published",
        workflow: {
          merchantId,
          lifecycleStatus: "published",
        },
      },
      include: {
        workflow: true,
      },
    });

  return versions.filter((version) => {
    const definition = version.definition;

    return (
      triggerMatches(
        definition.trigger,
        event,
      ) &&
      conditionsPass(
        definition.conditions,
        {
          event,
        },
      )
    );
  });
}

For higher volume, index workflows by:

merchant_id
+ trigger_source
+ trigger_type
+ transition

Do not scan every workflow for every event.

Verify sensitive actions before execution

A valid webhook proves that OxaPay sent the callback.

Before an irreversible action, retrieve the latest Payment Information.

Examples of sensitive actions include:

assigning a software license
granting long-term access
provisioning paid infrastructure
increasing an internal customer balance
creating a partner earning
preparing a payout

async function getOxaPayPayment({
  merchantApiKey,
  trackId,
}) {
  const response = await fetch(
    `https://api.oxapay.com/v1/payment/${encodeURIComponent(trackId)}`,
    {
      method: "GET",
      headers: {
        "Content-Type": "application/json",
        merchant_api_key: merchantApiKey,
      },
    },
  );

  const payload = await response.json();

  if (!response.ok) {
    throw new TransientWorkflowError(
      payload?.error?.message ??
        `Payment lookup failed with ${response.status}`,
    );
  }

  return payload.data;
}

Then verify the business facts:

function assertPaymentCanFulfill({
  payment,
  order,
}) {
  if (
    String(payment.status)
      .toLowerCase() !== "paid"
  ) {
    throw new PermanentWorkflowError(
      "Payment is not in the paid state",
    );
  }

  if (
    String(payment.order_id) !==
    String(order.externalOrderId)
  ) {
    throw new PermanentWorkflowError(
      "Payment order_id does not match the order",
    );
  }

  if (
    decimal(payment.amount).lessThan(
      decimal(order.total),
    )
  ) {
    throw new PermanentWorkflowError(
      "Paid amount does not satisfy the order",
    );
  }
}

Use decimal arithmetic for financial comparisons.

Do not use JavaScript floating-point arithmetic for payment decisions.

Plan actions as a dependency graph

A workflow may include independent and dependent actions.

Verify payment
      |
      v
Assign license
   |       |
   v       v
Email   Notify merchant

send_delivery depends on assign_license.

notify_merchant may also depend on assign_license.

The planner should create an action execution only when all dependencies are complete.

function isActionReady({
  action,
  completedActionIds,
}) {
  return (
    action.depends_on ?? []
  ).every((dependencyId) =>
    completedActionIds.has(
      dependencyId,
    ),
  );
}

Do not force every workflow into one long sequential process when actions can safely run in parallel.

Claim actions with a worker lease

Several workers may attempt to execute the same action.

Use a database claim or lease.

export async function claimActionExecution({
  actionExecutionId,
  workerId,
}) {
  const now = new Date();
  const leaseExpiresAt = new Date(
    now.getTime() + 60_000,
  );

  return db.actionExecution.updateMany({
    where: {
      id: actionExecutionId,
      status: {
        in: [
          "queued",
          "retry_scheduled",
        ],
      },
      OR: [
        {
          leaseExpiresAt: null,
        },
        {
          leaseExpiresAt: {
            lt: now,
          },
        },
      ],
    },
    data: {
      status: "running",
      leaseOwner: workerId,
      leaseExpiresAt,
      startedAt: now,
      attemptCount: {
        increment: 1,
      },
    },
  });
}

Only the worker that successfully claims the action should execute it.

The idempotency key remains necessary because a worker may crash after completing an external action but before updating the database.

Execute each action idempotently

export async function executeActionOnce({
  execution,
  action,
  context,
}) {
  const idempotencyKey = [
    execution.merchantId,
    execution.workflowVersionId,
    context.event.data.trackId,
    context.event.data.status,
    action.id,
  ].join(":");

  const existing =
    await db.actionExecution.findUnique({
      where: {
        idempotencyKey,
      },
    });

  if (
    existing?.status === "completed"
  ) {
    return existing.resultData;
  }

  const record =
    existing ??
    (await db.actionExecution.create({
      data: {
        workflowExecutionId:
          execution.id,
        actionId: action.id,
        actionType: action.type,
        idempotencyKey,
        status: "running",
        attemptCount: 1,
        requestData: renderInput(
          action.input,
          context,
        ),
        startedAt: new Date(),
      },
    }));

  try {
    const result =
      await runConnectorAction({
        type: action.type,
        input: record.requestData,
        context,
        idempotencyKey,
      });

    await db.actionExecution.update({
      where: {
        id: record.id,
      },
      data: {
        status: "accepted",
        resultData:
          sanitizeConnectorResult(
            result,
          ),
      },
    });

    return result;
  } catch (error) {
    await recordActionFailure({
      actionExecutionId: record.id,
      error,
    });

    throw error;
  }
}

When the downstream API supports idempotency keys, pass the same key to it.

Your local database should not be the only protection against repeated external actions.

Distinguish request acceptance from outcome verification

An HTTP 200 response does not always prove that the business outcome happened.

For important connectors, use three states:

requested
accepted
verified

Examples:

SaaS activation returned 200
-> Query account
-> Confirm plan is active

Hosting provisioning returned accepted
-> Poll service state
-> Confirm server is active

License assignment returned success
-> Verify license belongs to the order

Telegram invite created
-> Confirm expected member joined

export async function verifyActionOutcome({
  actionExecution,
  connector,
}) {
  const verification =
    await connector.verifyOutcome({
      request:
        actionExecution.requestData,
      result:
        actionExecution.resultData,
    });

  if (!verification.verified) {
    throw new TransientWorkflowError(
      verification.reason ??
        "Business outcome is not verified",
    );
  }

  await db.actionExecution.update({
    where: {
      id: actionExecution.id,
    },
    data: {
      status: "completed",
      outcomeVerified: true,
      completedAt: new Date(),
    },
  });
}

A merchant-grade automation system verifies business outcomes, not only API responses.

Build connectors behind one interface

Start with a small connector set:

internal order API
HTTP request
email
Telegram
Discord
Google Sheets
license service
SaaS account service
support case creation

const connectorActions = {
  "email.send": async ({
    input,
    idempotencyKey,
  }) => {
    return emailProvider.send({
      to: input.to,
      subject: input.subject,
      html: input.html,
      idempotencyKey,
    });
  },

  "telegram.send_message": async ({
    input,
  }) => {
    return telegramClient.sendMessage({
      chatId: input.chatId,
      text: input.message,
    });
  },

  "license.assign": async ({
    input,
    idempotencyKey,
  }) => {
    return licenseService.assign({
      productId: input.productId,
      orderId: input.orderId,
      customerEmail:
        input.customerEmail,
      idempotencyKey,
    });
  },

  "http.request": async ({
    input,
    idempotencyKey,
  }) => {
    return controlledHttpClient.request({
      url: input.url,
      method: input.method ?? "POST",
      headers: {
        ...input.headers,
        "Idempotency-Key":
          idempotencyKey,
      },
      body: input.body,
    });
  },
};

Connectors should reference stored credential IDs.

Do not embed secrets inside workflow definitions.

Protect the generic HTTP connector

A generic HTTP action can become an SSRF and secret-exfiltration tool.

Restrict:

protocols other than HTTPS
loopback addresses
private IP ranges
cloud metadata endpoints
link-local addresses
DNS rebinding
excessive redirects
large response bodies
long timeouts
merchant-controlled authorization headers
secrets in URLs
unapproved destination domains

A safer first version can require merchants to allowlist destination domains.

function assertAllowedDestination(url) {
  const parsed = new URL(url);

  if (parsed.protocol !== "https:") {
    throw new PermanentWorkflowError(
      "Only HTTPS destinations are allowed",
    );
  }

  if (
    !destinationAllowlist.has(
      parsed.hostname,
    )
  ) {
    throw new PermanentWorkflowError(
      "Destination is not allowlisted",
    );
  }
}

Run connector workers in a restricted network environment when possible.

Distinguish transient and permanent failures

Not every error should be retried.

Transient failures

Examples:

HTTP 429
temporary network timeout
provider 502 or 503
email service unavailable
CRM rate limit
database interruption

Recommended action:

Retry with backoff

Permanent failures

Examples:

order does not exist
customer email is missing
payment is not paid
amount does not match
connector credential is invalid
access plan no longer exists
destination is blocked
workflow input cannot be rendered

Recommended action:

Stop retrying
Create operational case

Ambiguous failures

Examples:

external action may have completed, but the response was lost
timeout occurred after request transmission
downstream system accepted work but did not return an operation ID

Recommended action:

Reconcile the outcome
Do not blindly retry

class TransientWorkflowError extends Error {}
class PermanentWorkflowError extends Error {}
class AmbiguousWorkflowError extends Error {}

Ambiguous failures require a stricter policy because repeating the action may duplicate fulfillment.

Use controlled retry policies

A retry policy may use exponential backoff:

Attempt 1: immediately
Attempt 2: after 30 seconds
Attempt 3: after 2 minutes
Attempt 4: after 10 minutes
Attempt 5: after 30 minutes

After the final retry:

Action: failed
Workflow: needs_attention
Merchant: notified
Operational case: created

The dashboard should show:

attempt count
last error
next retry time
error classification
completed dependencies
whether manual retry is safe
whether outcome reconciliation is required

Do not restart the entire workflow when one action fails.

Resume from the failed action when the workflow semantics allow it.

Preserve partial success

Consider:

License assigned: yes
Order updated: yes
Delivery email sent: no
CRM updated: yes

The workflow is not completely successful.

It is also not completely failed.

Represent this as:

Workflow status: partially_completed
Failed action: send_delivery
Completed actions:
- assign_license
- update_order
- update_crm

Retry only the email.

Do not assign another license.

A single automation_status = failed field cannot represent this safely.

Add approval gates for sensitive actions

Some actions should not execute automatically.

Examples:

issue refund
create payout
change customer balance
revoke long-term access
disable merchant account
send high-value credit
call a sensitive internal administrative API

A workflow can pause before the action:

- id: create_refund
  type: oxapay.refund_review
  approval:
    required: true
    role: finance_admin
    expires_after_minutes: 1440
  input:
    track_id: "{{ event.data.track_id }}"

The approval request should freeze:

workflow version
rendered action input
destination
amount
currency
reason
prior action results

Calculate a snapshot hash.

If any important field changes, invalidate the approval.

Do not let someone approve one action and execute a modified version later.

Keep payout automation outside the first MVP

Payment-triggered payouts have a larger blast radius than email or fulfillment actions.

Payout automation requires:

separate Payout API Key
verified destinations
limits
approval policy
immutable ledger
payout reconciliation
ambiguous-dispatch handling
additional audit logs

The Crypto Revenue Split and Payout System covers that architecture.

The Automation Studio can create an approved payout request in another system.

It should not send money automatically unless the product has been designed specifically for that responsibility.

Support safe replay

Replay is useful when:

a connector was unavailable
credentials were corrected
a workflow bug was fixed
a missed event was recovered
an execution was interrupted
outcome verification failed

Replay must not mean:

Run the whole workflow again

A replay operation should:

Load the original event.
Load the selected workflow version.
Inspect completed action records.
Reuse completed idempotent results.
Reconcile ambiguous actions.
Retry only eligible actions.
create a replay audit record.

The operator may choose:

Replay with original workflow version

or:

Replay with current published version

These options have different consequences.

The choice must be visible in the execution history.

Recover missed events with Payment History

Webhooks provide the real-time path.

They should not be the only path.

Callbacks may be missed because of:

server downtime
deployment errors
invalid callback responses
queue failures
database outages
incorrect endpoint configuration

Use OxaPay Payment History for recovery.

A practical schedule:

Every 10 minutes:
- query an overlapping recent period
- upsert provider records by track_id
- compare provider and local status
- create recovery events
- enqueue workflows that never started

Every night:
- compare paid payments with completed workflows
- identify paid payments with incomplete outcomes
- generate unresolved-case report

A recovered event should use:

source = payment_history_backfill

Do not make it appear as if the original webhook arrived.

The source of evidence matters during incident investigation.

A complete first workflow

The strongest first use case is software-license delivery.

Customer creates order
-> Merchant creates OxaPay invoice
-> Customer pays
-> OxaPay sends payment update
-> Studio validates callback
-> Event is stored
-> Paid transition is deduplicated
-> Published workflow matches
-> Payment Information is refreshed
-> Order ID and amount are verified
-> License is assigned idempotently
-> License assignment is verified
-> Delivery email is sent
-> Merchant is notified
-> Execution is recorded

Failure paths:

Payment is not paid
-> Do not deliver

Order not found
-> Create review case

Amount mismatch
-> Stop workflow
-> Create high-severity case

License service unavailable
-> Retry

License request timed out ambiguously
-> Check existing license assignment
-> Do not blindly assign again

Email unavailable
-> Keep assigned license
-> Retry only email

Retries exhausted
-> Notify merchant
-> Create support case

This workflow contains the core reliability problems the product must solve.

Low-code implementation with n8n or Make

A full SaaS platform is not the only starting path.

OxaPay provides an official n8n integration with operations for:

invoice generation
white-label payments
static addresses
Payment Information
payment search
payment webhook triggers
payout operations
common account operations
swaps

A practical n8n workflow can look like:

OxaPay Trigger
-> Filter status = paid
-> Get Payment Information
-> Query merchant order API
-> Verify order and amount
-> Deliver product
-> Send Telegram notification
-> Store execution result

OxaPay is also available through Make with modules for payment generation, payment lookup, searches, webhook watchers, payouts, static addresses, and other operations.

Low-code tools are useful for:

validating merchant demand
building managed workflows
testing connector requirements
serving lower-volume merchants
learning which workflow templates repeat

They do not automatically provide:

action-level idempotency
historical workflow versions
tenant isolation
approval snapshots
loop prevention
outcome verification
recovery guarantees
merchant-specific replay controls

Low-code orchestration does not remove payment-engineering responsibilities.

The execution dashboard

The dashboard should answer:

Did the event arrive?

Was it authentic?

Which workflow matched?

Which version ran?

Which actions completed?

What failed?

What requires approval?

What requires manual review?

Was the final outcome verified?

Event inbox

Show:

provider
track_id
order ID
payment status
source
HMAC validity
correlation ID
received time
matched workflows

Workflow executions

Show:

workflow name
workflow version
trigger event
execution status
start and completion time
completed actions
failed action
pending approval
retry state
replay history

Action details

Show:

action type
rendered input
masked credential
masked destination
attempt history
response summary
outcome verification
idempotency key
manual retry eligibility

Needs attention

Show:

paid payment without completed fulfillment
exhausted retries
permanent connector errors
ambiguous outcomes
missing order mapping
invalid credentials
recovered events
approval requests
possible workflow loops

Operational health

Track:

webhook acceptance rate
invalid HMAC rate
event-to-execution latency
first-attempt action success rate
retry rate
ambiguous-outcome rate
median time from paid to verified outcome
paid payments with incomplete workflows
oldest unresolved execution
duplicate events safely ignored

The product should surface failures before the customer reports them.

The MVP

Do not build Zapier for every crypto merchant.

Choose one niche and three workflows.

For a digital product seller:

Workflow one

Paid invoice
-> Verify order
-> Deliver product
-> Notify merchant

Workflow two

Expired invoice
-> Update order
-> Send safe customer message
-> Notify support for high-value orders

Workflow three

Payment History recovery
-> Find paid payment
-> Detect incomplete delivery
-> Create operational case

Start with four connectors:

internal HTTP API
email
Telegram
Google Sheets or CSV export

The first product should include:

event inbox
workflow definitions
immutable published versions
dry run
payment-event ingestion
action execution
idempotency
retries
outcome verification
needs-attention queue
Payment History recovery
audit log

It does not need:

drag-and-drop workflow design
hundreds of connectors
AI-generated workflows
template marketplace
multi-provider routing
automated payouts
advanced business intelligence
arbitrary merchant code

Reliability is more valuable than connector count.

Productize the merchant outcome

Do not sell:

OxaPay webhook integration.

Sell:

Automatic digital delivery after confirmed crypto payment, with retries, logs, and recovery.

Or:

Crypto-paid SaaS activation with payment verification and paid-but-not-activated monitoring.

Or:

Payment-to-CRM automation for international service businesses.

The merchant buys a business outcome.

The API, queue, workflow engine, and connector code are implementation details.

A practical commercial model may combine:

implementation fee
reusable workflow template
monthly monitoring
connector maintenance
execution dashboard
incident support
custom workflow changes

The recurring value comes from maintaining a business-critical workflow, not merely writing the original integration.

Security boundaries

A Payment Automation Studio stores credentials and executes merchant-authorized actions.

Minimum safeguards include:

HTTPS-only callback endpoints
HMAC validation over raw request bodies
encrypted Merchant API Keys
encrypted connector credentials
tenant-isolated storage
secrets excluded from logs
masked execution inputs and outputs
restricted expression language
workflow publish permissions
approval roles
audit logs for edits, publishing, replay, and approval
connector credential rotation
outbound request restrictions
rate limits
body-size limits
separate staging and production credentials
dead-letter handling
backup and restoration testing
operational alerting

The execution system should fail closed.

When payment evidence or action outcome is uncertain, stop and review.

Production tests

Test at least:

Invalid HMAC changes no business state

Duplicate raw webhook creates one stored event

Different paid payloads create one paid transition

Paying does not trigger fulfillment

Paid triggers one workflow execution

Duplicate queue job creates no duplicate action

Two workers cannot claim the same action simultaneously

Duplicate action attempt assigns one license

Ambiguous license timeout is reconciled before retry

Workflow version remains attached to historical execution

Draft workflow processes no live events

Historical simulation performs no external action

Dependency cycle blocks workflow publishing

Generic HTTP connector rejects private network destinations

Permanent error creates no automatic retry

Transient error follows retry policy

Partial success preserves completed actions

Approval applies to an immutable action snapshot

Causation-depth limit prevents workflow loop

Replay does not rerun completed irreversible actions

Payment History recovers a missed paid event

Paid payment without verified outcome creates a case

These tests are part of the product.

They are not optional engineering polish.

Common mistakes

Executing actions inside the webhook request

Persist first.

Execute asynchronously.

Promising exactly-once delivery

Distributed systems retry.

Build effectively-once business execution through idempotency and reconciliation.

Deduplicating events but not transitions

Different payloads may represent the same paid transition.

Deduplicating transitions but not actions

One transition may still execute an irreversible action twice.

Editing published workflows

Create a new immutable version instead.

Publishing without simulation

Test workflows against historical events before sending them to production.

Retrying every timeout

Some timeouts have ambiguous outcomes.

Reconcile before repeating an irreversible action.

Replaying the whole workflow

Reuse completed action results and replay only eligible work.

Allowing arbitrary code in conditions

Use a restricted expression language.

Ignoring automation loops

Track causation and enforce a maximum chain depth.

Hiding partial success

Record each action independently.

Treating API acceptance as business completion

Verify the final outcome.

Automating payouts too early

Use dedicated ledger, approval, destination, and reconciliation controls.

What makes this a real product?

A webhook script says:

If payment is paid, send an email.

A Payment Automation Studio says:

Receive callback
-> validate origin
-> preserve evidence
-> normalize event
-> deduplicate business transition
-> select immutable workflow version
-> evaluate restricted conditions
-> plan action dependencies
-> verify payment state
-> claim actions safely
-> execute idempotently
-> verify business outcomes
-> retry temporary failures
-> reconcile ambiguous outcomes
-> pause for approvals
-> recover missed events
-> prevent automation loops
-> show the merchant exactly what happened

That is the difference between connecting an API and operating a merchant workflow.

Final takeaway

The business opportunity is not simply accepting crypto payments.

It is controlling what happens after those payments.

Merchants need payment events to activate products, update systems, notify teams, create records, and handle failures without operational chaos.

OxaPay provides the payment primitives:

invoice generation
unique track_id references
merchant order_id references
payment callbacks
HMAC verification
Payment Information
Payment History
white-label payments
static addresses
SDKs
n8n and Make integrations

The Automation Studio provides the execution layer:

verified event ingestion
workflow definitions
immutable versions
safe conditions
action dependencies
idempotency
retries
approvals
replay
recovery
loop prevention
outcome verification
operational visibility

Start with one workflow:

Confirmed payment
-> Verified order
-> Idempotent delivery
-> Verified outcome

Then add simulation, retries, recovery, approvals, connectors, and monitoring.

Do not begin by building a large automation marketplace.

Build one payment-driven workflow that a merchant can trust with a real customer order.

That is where an automation script becomes payment infrastructure.

Which workflow would you build first: digital delivery, SaaS activation, paid community access, or merchant reporting?

References

OxaPay payment infrastructure

OxaPay automation integrations

Top comments (0)

Some comments may only be visible to logged-in visitors. Sign in to view all comments.