Crypto Payment Reconciliation: The Layer Most Developers Forget

Most developers treat crypto payment integration as a webhook problem.

Create an invoice.

Wait for the callback.

Update the order.

Move on.

That works until the first real production mismatch appears.

The gateway says an invoice is paid, but your database still says pending.

The customer has a transaction hash, but the order was never updated.

A webhook arrived, but your server returned 500.

A payment was completed after the invoice expired.

Support sees an unpaid order, finance sees received funds, and engineering has to open logs to understand what happened.

That is the moment you discover the missing layer:

Webhooks tell your system what changed. Reconciliation proves your system still agrees with reality.

Crypto payment reconciliation is the process of comparing your local payment state with external payment truth: the gateway, the blockchain, settlement records, order records, and ledger entries.

It is not optional.

It is what prevents small asynchronous failures from becoming lost revenue, incorrect fulfillment, duplicate credits, support chaos, and accounting gaps.

This article explains how to design crypto payment reconciliation from a developer’s perspective: state drift, source-of-truth boundaries, webhook failure, invoice lookup, transaction matching, audit trails, anomaly detection, repair jobs, settlement reconciliation, and the operational dashboards that make crypto payment systems trustworthy in production.

I’ll use OxaPay as one implementation reference where useful because its API exposes common reconciliation primitives such as invoice generation, track_id, signed webhooks, payment statuses, and payment lookup by track_id. The architecture itself applies to most crypto payment gateways.

---

The Short Version

A webhook-driven payment system asks:

Did we receive the event?

A reconciled payment system asks:

Does every system agree on what happened?

That is a much stronger question.

A reliable crypto payment system must compare at least these records:

local order
local invoice
local payment events
gateway payment status
blockchain transaction data
merchant balance / settlement record
business fulfillment state

If those records disagree, the system needs to detect the mismatch, repair it safely when possible, or send it to manual review with enough context for support.

That is reconciliation.

---

Why Reconciliation Exists

Crypto payments are asynchronous.

The user pushes funds from a wallet or exchange.

The gateway monitors the blockchain.

The gateway sends callbacks to your server.

Your server updates local state.

Your order system triggers fulfillment.

Your finance system records revenue.

Your settlement system moves funds later.

That flow contains many boundaries.

wallet → blockchain
blockchain → gateway
gateway → webhook
webhook → database
database → order system
order system → fulfillment
payment balance → accounting

Every boundary can fail.

A webhook may be delayed.

Your server may be down.

A queue worker may crash.

A database transaction may roll back.

A customer may pay after expiration.

A duplicate event may be ignored incorrectly.

A provider status may update after your local status freezes.

A paid invoice may not trigger fulfillment.

A fulfilled order may not have a final payment record.

Reconciliation exists because no distributed payment system should assume that one event path is perfect.

---

The Most Common Production Mismatch

The most common mismatch looks like this:

Gateway: invoice paid
Merchant DB: payment pending
WooCommerce / Shopify / app order: unpaid
Customer: already sent funds
Support: confused

This can happen even when every system is “working.”

Maybe the webhook was sent, but your endpoint timed out.

Maybe your endpoint received the webhook, but a later database write failed.

Maybe your code updated the payment table but did not update the order table.

Maybe the provider retried the callback, but your idempotency logic incorrectly treated it as already processed.

Maybe the customer paid after expiration and your system ignored the event.

Without reconciliation, this mismatch stays hidden until the customer complains.

With reconciliation, the system finds it automatically.

---

Reconciliation Is Not Polling Instead of Webhooks

Some developers hear “reconciliation” and think:

Should I just poll the payment API instead of using webhooks?

No.

That is the wrong framing.

Webhooks and reconciliation solve different problems.

Layer	Purpose
Webhooks	Fast event delivery
Reconciliation	Consistency verification and repair

Webhooks are for responsiveness.

Reconciliation is for correctness.

A good system uses both.

Webhook path:
payment event happens
→ provider sends callback
→ merchant updates state quickly

Reconciliation path:
scheduled job checks unresolved or risky records
→ compares local state with provider state
→ repairs mismatch or flags review

Webhooks are the primary update mechanism.

Reconciliation is the safety net.

---

The Reconciliation Mindset

A payment reconciliation system should be designed around one principle:

Every important payment fact must be independently verifiable later.

That means you should be able to answer:

Which order was this payment for?
Which invoice did the gateway create?
Which provider reference identifies it?
Which webhook events arrived?
Which transaction hash was detected?
Which asset and network were used?
What amount was expected?
What amount was received?
What did the gateway report?
What did our database record?
What business action did we take?
Was fulfillment triggered?
Was settlement completed?
Did finance record it?

If you cannot answer these questions from stored data, reconciliation will become guesswork.

Guesswork does not scale.

---

The Systems That Need to Agree

Crypto payment reconciliation is not only “database vs provider.”

It often involves several systems.

1. Order system
2. Invoice/payment table
3. Webhook event table
4. Gateway payment API
5. Blockchain transaction data
6. Fulfillment system
7. Merchant balance / settlement records
8. Accounting or reporting layer

A small merchant may only have the first four.

A larger platform may have all eight.

The architecture should make each layer explicit.

---

A Practical Reconciliation Model

At minimum, store these local records:

orders
payments / invoices
payment_events
payment_transactions
settlements or balance movements

You do not need a huge system on day one, but you need enough structure to compare states.

A minimal payment table might look like this:

CREATE TABLE crypto_payments (
  id BIGSERIAL PRIMARY KEY,
  order_id TEXT NOT NULL,
  provider TEXT NOT NULL,
  provider_track_id TEXT NOT NULL UNIQUE,
  pricing_currency TEXT NOT NULL,
  expected_amount NUMERIC(18, 8) NOT NULL,
  received_amount NUMERIC(36, 18) NULL,
  selected_asset TEXT NULL,
  selected_network TEXT NULL,
  gateway_status TEXT NULL,
  payment_status TEXT NOT NULL,
  business_status TEXT NOT NULL,
  payment_url TEXT NULL,
  expires_at TIMESTAMP NULL,
  paid_at TIMESTAMP NULL,
  last_provider_check_at TIMESTAMP NULL,
  last_reconciled_at TIMESTAMP NULL,
  reconciliation_status TEXT NOT NULL DEFAULT 'not_checked',
  created_at TIMESTAMP NOT NULL DEFAULT NOW(),
  updated_at TIMESTAMP NOT NULL DEFAULT NOW()
);

A webhook event table:

CREATE TABLE crypto_payment_events (
  id BIGSERIAL PRIMARY KEY,
  provider TEXT NOT NULL,
  provider_track_id TEXT NOT NULL,
  event_hash TEXT NOT NULL UNIQUE,
  provider_status TEXT NULL,
  raw_payload JSONB NOT NULL,
  received_at TIMESTAMP NOT NULL DEFAULT NOW(),
  processed_at TIMESTAMP NULL,
  processing_status TEXT NOT NULL DEFAULT 'received',
  processing_error TEXT NULL
);

A transaction table:

CREATE TABLE crypto_payment_transactions (
  id BIGSERIAL PRIMARY KEY,
  payment_id BIGINT NOT NULL,
  tx_hash TEXT NOT NULL,
  asset TEXT NOT NULL,
  network TEXT NOT NULL,
  expected_amount NUMERIC(36, 18) NULL,
  received_amount NUMERIC(36, 18) NULL,
  confirmations INTEGER NULL,
  tx_status TEXT NOT NULL,
  detected_at TIMESTAMP NULL,
  confirmed_at TIMESTAMP NULL,
  UNIQUE (payment_id, tx_hash)
);

These tables give your system enough memory to understand what happened.

Without this structure, reconciliation has nothing reliable to compare.

---

Source of Truth Is Not One Thing

A common mistake is asking:

What is the source of truth?

In payment systems, the better question is:

Source of truth for what?

Different systems own different facts.

Fact	Likely Source of Truth
Customer ordered item	Merchant order system
Invoice was created	Payment provider + local invoice table
Provider payment status	Gateway API
Blockchain transaction exists	Blockchain / gateway monitor
Webhook was received	Local event table
Order was fulfilled	Merchant fulfillment system
Funds were settled	Gateway balance / settlement ledger
Revenue was recognized	Accounting system

Trying to force one system to be the source of truth for everything creates bad architecture.

Reconciliation works by comparing specialized truths.

---

Local State vs External Truth

Your database may say:

payment_status = waiting_for_payment

The gateway may say:

status = paid

That does not automatically mean your database is wrong.

It means your database is stale.

The reconciliation system needs to ask:

Can we safely move local status from waiting_for_payment to paid?
Was the payment already fulfilled?
Was the order cancelled?
Did the invoice expire?
Was the payment underpaid?
Is this a duplicate provider result?
Does this require manual review?

A reconciliation job should not blindly overwrite state.

It should apply safe transitions.

---

The Reconciliation Loop

A basic reconciliation loop looks like this:

1. Select payments that need checking
2. Fetch external payment truth
3. Normalize external status
4. Compare with local state
5. Decide whether to repair, ignore, or flag
6. Apply idempotent state transition
7. Record reconciliation result
8. Emit alerts or admin notes if needed

In code terms:

async function reconcileOpenPayments() {
  const payments = await findPaymentsNeedingReconciliation();

  for (const payment of payments) {
    try {
      const providerPayment = await fetchProviderPayment(payment.provider_track_id);

      const externalStatus = normalizeProviderStatus(providerPayment.data.status);
      const nextStatus = mapGatewayStatusToPaymentStatus(externalStatus);

      const decision = decideReconciliationAction({
        localPayment: payment,
        providerPayment: providerPayment.data,
        nextStatus
      });

      await applyReconciliationDecision(payment, decision, providerPayment.data);
    } catch (error) {
      await recordReconciliationFailure(payment, error);
    }
  }
}

The important part is not the language.

The important part is the decision layer.

Reconciliation should not be:

fetch provider status → overwrite local status

It should be:

fetch provider status → compare → validate transition → repair safely

---

Which Payments Should Be Reconciled?

You do not need to check every payment forever.

Start with records that are likely to drift.

Good candidates:

created
waiting_for_payment
payment_detected
confirming
underpaid
manual_review
expired_recently
paid_but_not_fulfilled
fulfilled_but_payment_not_final
webhook_failed
event_received_but_not_processed
settlement_pending

You can define selection rules like:

SELECT *
FROM crypto_payments
WHERE payment_status IN (
  'created',
  'waiting_for_payment',
  'payment_detected',
  'confirming',
  'underpaid',
  'manual_review'
)
AND created_at > NOW() - INTERVAL '7 days';

Add special checks for stale states:

SELECT *
FROM crypto_payments
WHERE payment_status = 'confirming'
AND updated_at < NOW() - INTERVAL '30 minutes';

And for suspicious business mismatches:

SELECT *
FROM crypto_payments
WHERE payment_status = 'paid'
AND business_status != 'fulfilled'
AND paid_at < NOW() - INTERVAL '10 minutes';

That last query is powerful.

It catches payments that succeeded but did not trigger the expected business action.

---

Reconciliation Frequency

Reconciliation does not need to run at the same frequency for every payment.

Use different schedules for different risk zones.

Payment Group	Suggested Frequency
newly created invoices	every 1–2 minutes
payment detected / confirming	every 1–5 minutes
underpaid / manual review	every 10–30 minutes
expired within last few hours	every 10–30 minutes
paid but not fulfilled	every few minutes
settlement pending	hourly or daily
historical audit	daily

The goal is not to overload the provider API.

The goal is to reduce the time that your system remains inconsistent.

Use backoff.

A payment that was created two minutes ago deserves more attention than a failed invoice from three weeks ago.

---

Avoid Infinite Reconciliation

A reconciliation system should not keep retrying forever with no strategy.

Store retry metadata.

ALTER TABLE crypto_payments
ADD COLUMN reconciliation_attempts INTEGER NOT NULL DEFAULT 0,
ADD COLUMN next_reconciliation_at TIMESTAMP NULL,
ADD COLUMN last_reconciliation_error TEXT NULL;

Then apply backoff:

attempt 1 → retry in 1 minute
attempt 2 → retry in 5 minutes
attempt 3 → retry in 15 minutes
attempt 4 → retry in 1 hour
attempt 5 → manual review

This prevents a broken payment from consuming resources forever.

It also makes stuck cases visible.

---

Mapping Gateway Statuses

Every provider has its own status vocabulary.

Your application should not scatter provider-specific strings throughout the codebase.

Normalize them.

Using OxaPay as an implementation reference, payment statuses may include values such as new, waiting, paying, paid, manual_accept, underpaid, refunding, refunded, and expired.

A simple mapper:

function mapGatewayStatusToPaymentStatus(status) {
  const normalized = String(status || "").toLowerCase();

  const map = {
    new: "created",
    waiting: "waiting_for_payment",
    paying: "payment_detected",
    paid: "paid",
    manual_accept: "paid_manual",
    underpaid: "underpaid",
    refunding: "refunding",
    refunded: "refunded",
    expired: "expired"
  };

  return map[normalized] || "manual_review";
}

This gives your system one internal language.

If you change providers later, most business logic remains stable.

---

Safe State Transitions

Reconciliation must respect valid state transitions.

Example:

const allowedTransitions = {
  created: [
    "waiting_for_payment",
    "payment_detected",
    "paid",
    "expired",
    "manual_review"
  ],
  waiting_for_payment: [
    "payment_detected",
    "paid",
    "underpaid",
    "expired",
    "manual_review"
  ],
  payment_detected: [
    "confirming",
    "paid",
    "underpaid",
    "manual_review"
  ],
  confirming: [
    "paid",
    "underpaid",
    "manual_review"
  ],
  underpaid: [
    "paid",
    "expired",
    "manual_review"
  ],
  expired: [
    "late_payment",
    "manual_review"
  ],
  late_payment: [
    "paid",
    "manual_review",
    "refunded"
  ],
  paid: [
    "refunding",
    "refunded"
  ],
  paid_manual: [
    "refunding",
    "refunded"
  ],
  refunding: [
    "refunded"
  ],
  refunded: [],
  manual_review: [
    "paid",
    "expired",
    "refunded"
  ]
};

function canTransition(from, to) {
  return from === to || allowedTransitions[from]?.includes(to);
}

Why allow expired → late_payment?

Because crypto payments can arrive after expiration.

The invoice may be expired from a checkout perspective, but money still moved.

A good reconciliation system makes that visible.

---

Decision Matrix

Reconciliation should produce decisions, not just updates.

Example:

Local State	Provider State	Business Context	Decision
waiting_for_payment	paid	order still active	repair to paid, trigger fulfillment once
waiting_for_payment	expired	no transaction	mark expired
expired	paid	late payment allowed	move to late_payment or paid based on policy
paid	paid	already fulfilled	no-op
paid	refunded	refund confirmed	update refund status
underpaid	paid	remaining amount completed	repair to paid
paid	underpaid	impossible regression	flag manual review
created	paid	webhook missed	repair to paid, record missed event
paid	provider not found	provider error	retry later, do not downgrade
fulfilled order	unpaid provider	serious mismatch	manual review

This table is where production maturity shows.

A reconciliation system should be careful with destructive changes.

For example, do not move a paid order back to unpaid just because one provider lookup fails.

---

Fetching Provider Payment Information

Most payment gateways expose some way to look up a payment by provider reference.

In OxaPay, the Payment Information endpoint lets developers retrieve payment details by track_id.

Example request:

curl -X GET https://api.oxapay.com/v1/payment/184747701 \
  -H "merchant_api_key: $OXAPAY_MERCHANT_API_KEY" \
  -H "Content-Type: application/json"

Example wrapper:

async function fetchOxaPayPayment(trackId) {
  const response = await fetch(`https://api.oxapay.com/v1/payment/${trackId}`, {
    method: "GET",
    headers: {
      "merchant_api_key": process.env.OXAPAY_MERCHANT_API_KEY,
      "Content-Type": "application/json"
    }
  });

  const result = await response.json();

  if (!response.ok || result.status !== 200) {
    throw new Error(`Payment lookup failed: ${JSON.stringify(result.error)}`);
  }

  return result.data;
}

In your own abstraction, do not let the rest of the system depend on one provider response shape.

Wrap it:

async function fetchProviderPayment(provider, trackId) {
  if (provider === "oxapay") {
    const data = await fetchOxaPayPayment(trackId);

    return {
      provider: "oxapay",
      trackId,
      gatewayStatus: data.status,
      expectedAmount: data.amount,
      receivedAmount: data.received_amount,
      asset: data.currency,
      network: data.network,
      txHash: data.tx_hash,
      raw: data
    };
  }

  throw new Error(`Unsupported provider: ${provider}`);
}

This makes reconciliation easier to maintain if your payment infrastructure changes later.

---

Repairing State Safely

When reconciliation finds a mismatch, repair should be idempotent.

Example:

async function applyReconciliationDecision(payment, decision, providerPayment) {
  await db.transaction(async (trx) => {
    await trx.reconciliation_runs.insert({
      payment_id: payment.id,
      previous_status: payment.payment_status,
      provider_status: providerPayment.gatewayStatus,
      decision: decision.type,
      reason: decision.reason,
      raw_provider_payload: providerPayment.raw
    });

    if (decision.type === "noop") {
      return;
    }

    if (decision.type === "manual_review") {
      await trx.crypto_payments.update({
        id: payment.id,
        reconciliation_status: "manual_review",
        last_reconciled_at: new Date()
      });

      return;
    }

    if (decision.type === "repair_status") {
      await trx.crypto_payments.update({
        id: payment.id,
        payment_status: decision.nextStatus,
        gateway_status: providerPayment.gatewayStatus,
        last_provider_check_at: new Date(),
        last_reconciled_at: new Date(),
        reconciliation_status: "repaired"
      });

      if (decision.nextStatus === "paid") {
        await fulfillOrderOnce(trx, payment.order_id, payment.id);
      }
    }
  });
}

The fulfillOrderOnce function matters.

Reconciliation may discover a missed paid status.

That does not mean it should blindly trigger fulfillment again.

Use a uniqueness rule.

CREATE TABLE order_fulfillments (
  id BIGSERIAL PRIMARY KEY,
  order_id TEXT NOT NULL UNIQUE,
  payment_id BIGINT NOT NULL,
  fulfilled_at TIMESTAMP NOT NULL DEFAULT NOW()
);

Then fulfillment becomes safe to call from both webhook processing and reconciliation.

---

Reconciliation Run Table

Store reconciliation runs.

Do not just update payment rows silently.

Example:

CREATE TABLE payment_reconciliation_runs (
  id BIGSERIAL PRIMARY KEY,
  payment_id BIGINT NOT NULL,
  provider TEXT NOT NULL,
  provider_track_id TEXT NOT NULL,
  local_status_before TEXT NOT NULL,
  provider_status TEXT NULL,
  decision TEXT NOT NULL,
  reason TEXT NULL,
  raw_provider_payload JSONB NULL,
  created_at TIMESTAMP NOT NULL DEFAULT NOW()
);

This table gives you an audit trail.

It answers:

When did we check this payment?
What did the provider say?
What did our system say before the check?
What did reconciliation decide?
Did we repair anything?
Why was it sent to manual review?

This is extremely useful for production support.

---

Webhook Events and Reconciliation Should Share the Same State Engine

Do not build two separate status systems:

webhook handler status logic
reconciliation job status logic

That creates divergence.

Both should call the same internal state engine.

webhook event
        |
        v
normalize provider status
        |
        v
apply payment transition
        |
        v
business side effects

reconciliation lookup
        |
        v
normalize provider status
        |
        v
apply payment transition
        |
        v
business side effects

The source is different.

The transition engine should be the same.

That makes behavior consistent whether the update came from a webhook or from reconciliation.

---

Reconciliation and Idempotency Work Together

Reconciliation will find the same paid payment more than once.

That is normal.

Your system should handle it.

A payment already marked paid should not trigger fulfillment again.

An already recorded transaction should not be duplicated.

An already refunded order should not be refunded twice.

Idempotency rules should exist at multiple levels:

Level	Idempotency Rule
Webhook event	unique event hash
Transaction	unique provider + tx hash
Payment state	safe transition guard
Fulfillment	unique order fulfillment record
Credit balance	unique ledger entry reference
Refund	unique refund reference
Reconciliation	no repeated side effects for same state

Idempotency is not one feature.

It is a pattern across the whole payment system.

---

Reconciling Transactions

Payment status is not the only thing to reconcile.

You may also need transaction reconciliation.

For each provider payment, compare transaction-level data:

tx_hash
asset
network
received_amount
confirmation count
detected_at
confirmed_at

If the provider reports a transaction that your local system does not have, insert it.

If your local transaction exists but confirmation status changed, update it.

Example:

async function reconcileTransactions(payment, providerPayment) {
  const transactions = providerPayment.transactions || [];

  for (const tx of transactions) {
    await db.crypto_payment_transactions.upsert({
      payment_id: payment.id,
      tx_hash: tx.hash,
      asset: tx.asset,
      network: tx.network,
      received_amount: tx.amount,
      confirmations: tx.confirmations,
      tx_status: tx.status,
      detected_at: tx.detected_at,
      confirmed_at: tx.confirmed_at
    }, {
      conflict: ["payment_id", "tx_hash"]
    });
  }
}

One invoice can have multiple transactions.

Your reconciliation model should allow that.

---

Reconciling Amounts

Amount reconciliation is subtle in crypto.

A merchant may price an invoice in fiat, while the customer pays in crypto.

You need to compare:

expected commercial amount
expected crypto amount
received crypto amount
received fiat-equivalent amount
fees
settlement amount

At minimum, store:

pricing_currency
expected_amount
selected_asset
selected_network
expected_crypto_amount
received_crypto_amount
received_fiat_value

Then detect cases:

Case	Meaning
received = expected	normal paid
received < expected	underpaid
received > expected	overpaid
received after expiration	late payment
received on wrong network	support/recovery issue
received in multiple txs	aggregate before deciding
provider says paid but local amount missing	repair local amount

Never rely only on a boolean paid.

Payment amount is part of the truth.

---

Reconciling Orders

A payment can be paid while the order is not fulfilled.

That is a mismatch.

A payment can be unpaid while the order is fulfilled.

That is a more serious mismatch.

Order reconciliation checks business effects.

Examples:

SELECT *
FROM crypto_payments p
JOIN orders o ON o.id = p.order_id
WHERE p.payment_status = 'paid'
AND o.fulfillment_status = 'not_fulfilled'
AND p.paid_at < NOW() - INTERVAL '10 minutes';

And the opposite:

SELECT *
FROM crypto_payments p
JOIN orders o ON o.id = p.order_id
WHERE p.payment_status NOT IN ('paid', 'paid_manual')
AND o.fulfillment_status = 'fulfilled';

The first case may require repair.

The second case requires investigation.

Reconciliation is not only about fixing pending payments.

It also catches dangerous business inconsistencies.

---

Reconciling Settlement

Payment completion and settlement are different.

A customer may pay an invoice, but funds may still need to be:

credited to merchant balance
converted
withdrawn
posted to ledger
matched to accounting records

For small integrations, settlement reconciliation may be manual.

For larger systems, it needs to be explicit.

Separate these fields:

payment_status
settlement_status
ledger_status

Example:

Status Type	Example
payment_status	paid
settlement_status	credited_to_balance
ledger_status	posted

A paid invoice that never appears in finance reporting is still a problem.

It may not affect the customer, but it affects the business.

---

Handling Late Payments

Late payments are a major reason reconciliation exists.

A customer may pay after invoice expiration because:

• they used an exchange withdrawal
• they waited too long
• the blockchain was congested
• the payment page expired before the transaction appeared
• they copied the address and paid later

The provider may later show:

expired invoice received funds

Your local system may show:

order cancelled

Reconciliation should detect this and move the payment into a visible state.

expired
→ late_payment
→ manual_review

Example decision:

function decideLatePayment(payment, providerPayment) {
  if (payment.payment_status === "expired" && providerPayment.gatewayStatus === "paid") {
    return {
      type: "manual_review",
      nextStatus: "late_payment",
      reason: "payment_received_after_expiration"
    };
  }

  return null;
}

Do not silently ignore late money movement.

Even if the order should not be fulfilled, finance and support need to know funds arrived.

---

Handling Underpayments

Underpaid invoices should not disappear into failure.

Reconciliation should keep checking if the provider allows additional payments or if the customer completes the remaining amount later.

Example:

Provider: underpaid
Local: waiting_for_payment
Decision: move to underpaid, notify customer/support

Later:

Provider: paid
Local: underpaid
Decision: repair to paid, trigger fulfillment once

This is why underpaid → paid should be a valid transition.

A strict state machine that treats underpaid as terminal creates unnecessary manual work.

---

Handling Overpayments

Overpayment reconciliation should record the excess.

Do not simply mark paid and discard the difference.

Possible handling:

mark invoice paid
record overpaid amount
create admin note
optionally create refund/credit task

Store:

expected_amount
received_amount
overpaid_amount
overpayment_policy

This matters for accounting and customer trust.

---

Handling Provider API Failures

A reconciliation job depends on provider API availability.

That API can fail.

Your job should not panic.

Classify provider errors:

Error Type	Handling
timeout	retry with backoff
429 rate limit	slow down
5xx provider error	retry later
401 auth error	alert immediately
payment not found	retry if recent, manual review if persistent
malformed response	alert engineering
network error	retry later

Do not downgrade payments because one lookup failed.

A provider error means:

external truth unavailable right now

It does not mean:

payment failed

---

Rate Limits and Batching

Reconciliation can create unnecessary API load if implemented carelessly.

Good practices:

• reconcile only unresolved or risky payments
• prioritize recent invoices
• use exponential backoff
• stop checking terminal states after a safe window
• batch where provider supports it
• cache recent lookup results briefly
• avoid multiple workers reconciling the same payment at once
• respect provider rate limits

Use locks.

Example:

ALTER TABLE crypto_payments
ADD COLUMN reconciliation_locked_until TIMESTAMP NULL;

Worker selection:

SELECT *
FROM crypto_payments
WHERE next_reconciliation_at <= NOW()
AND (
  reconciliation_locked_until IS NULL
  OR reconciliation_locked_until < NOW()
)
LIMIT 100;

Then lock selected rows before processing.

Without locks, multiple workers may reconcile the same payment and create duplicate work.

---

Manual Review Is a Feature

Manual review should not be treated as failure.

It is a valid state.

Some cases should not be repaired automatically:

provider says paid, order was already refunded
provider status regressed unexpectedly
payment arrived after order cancellation
wrong network suspected
large overpayment
provider data incomplete
transaction hash conflicts with another invoice
fulfilled order has unpaid payment status

Manual review should include context.

A good review record includes:

order_id
provider_track_id
local_status
provider_status
expected_amount
received_amount
asset
network
tx_hash
reason
recommended action
raw provider payload

Support should not see only:

manual_review

They should see why.

---

Alerting Rules

Not every mismatch needs a pager.

But some do.

Useful alert categories:

Alert	Severity
paid provider status but local pending for more than 10 minutes	medium
fulfilled order with unpaid payment status	high
webhook signature failures spike	high
provider API authentication failure	critical
reconciliation job not running	high
many expired invoices later paid	medium
underpayment rate increasing	medium
duplicate webhook rate unusually high	low/medium
settlement mismatch	high
manual review queue growing	medium

Metrics should support alerts.

Do not rely only on logs.

---

Reconciliation Metrics

Track these metrics:

open_payments_count
stale_pending_count
paid_not_fulfilled_count
fulfilled_not_paid_count
webhook_failed_count
reconciliation_run_count
reconciliation_repair_count
manual_review_count
provider_lookup_failure_count
average_time_to_payment
average_time_to_reconciliation_repair
underpayment_rate
late_payment_rate
settlement_mismatch_count

These metrics tell you whether your payment system is healthy.

For example:

• high stale_pending_count may mean callbacks are failing
• high paid_not_fulfilled_count may mean order transition logic is broken
• high manual_review_count may mean policies are unclear
• high provider_lookup_failure_count may mean API or auth issues
• high late_payment_rate may mean invoice lifetime is too short

Reconciliation is not just repair.

It is observability.

---

Admin and Support View

A reconciliation system should surface useful information to humans.

For each payment, show:

order_id
provider
track_id
payment_url
gateway_status
payment_status
business_status
expected_amount
received_amount
asset
network
tx_hash
invoice_created_at
expires_at
paid_at
last_webhook_at
last_provider_check_at
last_reconciled_at
reconciliation_status
manual_review_reason

Also show the timeline:

12:00 invoice created
12:03 webhook received: waiting
12:06 webhook received: paying
12:08 webhook failed during processing
12:12 reconciliation checked provider: paid
12:12 local state repaired: waiting_for_payment → paid
12:12 order fulfillment triggered

This kind of timeline reduces support time dramatically.

It also reduces pressure on engineering.

---

Example Implementation: OxaPay as a Reference

OxaPay is useful as an implementation reference because it exposes the core primitives a reconciliation system needs.

The Generate Invoice endpoint creates a payment request and returns a track_id and payment_url.

The Webhook documentation explains callback delivery, HMAC SHA-512 validation over the raw POST body, and the need to return HTTP 200 with ok after successful processing.

The Payment Status Table gives developers provider-side statuses that can be mapped into internal payment states.

The Payment Information endpoint lets your backend retrieve payment details by track_id, which is exactly what reconciliation needs.

The important point is not that reconciliation belongs to one provider.

The important point is that reliable crypto payment systems need these primitives:

provider reference
signed event delivery
status table
payment lookup
local event storage
idempotent state transitions
manual review path
reconciliation job

Whether you use OxaPay, another gateway, or your own infrastructure, those primitives still need to exist.

---

A Minimal Reconciliation Architecture

A practical architecture looks like this:

Webhook Receiver
  |
  | signed payment events
  v
Event Store
  |
  | raw events, hashes, processing status
  v
Payment State Engine
  |
  | normalize status, apply transitions
  v
Order / Business Layer
  |
  | fulfill, hold, refund, notify
  ^
  |
Reconciliation Worker
  |
  | provider lookup by track_id
  v
Provider Payment API

The key design principle:

Webhooks and reconciliation should both feed the same payment state engine.

Do not build separate logic paths.

---

Minimal Reconciliation Pseudocode

Here is the whole system in compact form:

async function runReconciliationBatch() {
  const payments = await selectPaymentsForReconciliation();

  for (const payment of payments) {
    await reconcileOnePayment(payment);
  }
}

async function reconcileOnePayment(payment) {
  const providerPayment = await fetchProviderPayment(
    payment.provider,
    payment.provider_track_id
  );

  const nextStatus = mapGatewayStatusToPaymentStatus(
    providerPayment.gatewayStatus
  );

  const decision = decideReconciliationAction({
    payment,
    providerPayment,
    nextStatus
  });

  await applyDecisionIdempotently({
    payment,
    providerPayment,
    decision
  });
}

function decideReconciliationAction({ payment, providerPayment, nextStatus }) {
  if (payment.payment_status === nextStatus) {
    return { type: "noop", reason: "already_consistent" };
  }

  if (!canTransition(payment.payment_status, nextStatus)) {
    return {
      type: "manual_review",
      reason: "invalid_state_transition",
      nextStatus
    };
  }

  if (payment.payment_status === "expired" && nextStatus === "paid") {
    return {
      type: "manual_review",
      reason: "late_payment_after_expiration",
      nextStatus: "late_payment"
    };
  }

  if (nextStatus === "paid") {
    return {
      type: "repair_and_fulfill_once",
      reason: "provider_paid_local_not_paid",
      nextStatus
    };
  }

  return {
    type: "repair_status",
    reason: "safe_state_repair",
    nextStatus
  };
}

This is not complete production code, but the shape is right.

It has:

selection
provider lookup
normalization
decision
safe transition
manual review
idempotent repair

That is the core of reconciliation.

---

Testing Reconciliation

Test reconciliation directly.

Do not only test webhooks.

Important test cases:

provider paid, local pending
provider paid, local expired
provider underpaid, local waiting
provider paid, local underpaid
provider expired, local waiting
provider refunded, local paid
provider lookup timeout
provider auth failure
provider not found
order fulfilled, payment not paid
payment paid, order not fulfilled
duplicate reconciliation run
two workers select same payment
manual review case
late payment after expiration

Each case should have an expected decision.

If you cannot define the expected decision, your production system will behave inconsistently.

---

Production Checklist

Before shipping crypto payment reconciliation, verify this:

Every invoice stores provider_track_id
Every webhook event is stored with a unique hash
Provider statuses are normalized in one place
Payment state transitions are guarded
Fulfillment is idempotent
Paid payments can be repaired if webhook fails
Expired invoices can detect late payments
Underpaid invoices can later become paid
Provider lookup failures do not downgrade local state
Manual review has reasons and context
Reconciliation runs are logged
Reconciliation has backoff and retry limits
Workers use locks or safe selection
Metrics track stale states and repairs
Support can search by order_id, track_id, tx_hash
Settlement state is separate from payment state
Webhooks and reconciliation use the same state engine

This checklist is the difference between:

We hope callbacks arrive.

and:

We can prove our payment state is correct.

---

Final Thought

Crypto payment reconciliation is easy to ignore because the first integration usually works without it.

The first invoice is created.

The first webhook arrives.

The first order is marked paid.

Everything looks done.

But production is not the first transaction.

Production is the thousandth transaction, the missed callback, the late payment, the underpaid invoice, the duplicate webhook, the customer with a transaction hash, the order stuck in pending, and the finance report that does not match the gateway balance.

That is where reconciliation matters.

Webhooks make crypto payment systems responsive.

Reconciliation makes them trustworthy.

If you are building a crypto payment integration, do not stop at:

Can we receive payment events?

Ask the harder question:

Can we prove later that our local payment state still matches what actually happened?

That question is where serious payment infrastructure begins.