DEV Community

Cover image for Sign it, store it, then send it: how a crypto payout engine avoids paying twice
Coldtill
Coldtill

Posted on

Sign it, store it, then send it: how a crypto payout engine avoids paying twice

Receiving crypto is forgiving. If your watcher crashes, the deposit is still on chain when it comes back, and you count it then. Sending is the opposite. A payout is the one place a payment gateway moves money on its own, and the usual reflex for a failed request, "just retry it", is how you end up refunding a customer twice.

I want to walk through how the payouts side of Coldtill deals with this, because most of the interesting decisions aren't about crypto at all. They're about what you do when you don't know whether something happened.

Some scope first, so nobody reads more into this than is there. In Coldtill a payout is money sent from a project's hot wallet: a refund, a supplier payment, a withdrawal to an exchange. Payouts work on the EVM networks and Solana. Bitcoin has no hot wallet in Coldtill, so there are no Bitcoin payouts, and Tron payouts aren't supported yet (the Tron hot wallet only pays for sweeps). Everything below is about EVM chains and Solana.

The four ways a payout goes wrong

Before any design, list the failures. These are the ones that actually cost money:

  1. The process dies between signing and broadcasting. On restart, did the transaction go out or not?
  2. The transaction is stuck. It was broadcast with a fee the network doesn't want right now, and it just sits there.
  3. A node says no, but another node said yes. With several RPC providers and failover, "rejected" from one of them doesn't mean the network never saw it.
  4. You genuinely can't tell. Something spent from the wallet, and you can't match it to the payout.

The common mistake in all four is the same: building a new transaction to "try again" while the old one might still land. Two different signed transactions for one payout means two payouts.

Layer one: make the request itself idempotent

The first duplicate usually comes from your own code, not the chain. A shop's refund job times out, retries, and asks the gateway to pay again.

Coldtill's payout endpoint takes an Idempotency-Key header. This is the example from the API docs:

curl -X POST https://pay.example.com/api/v1/withdrawal/merchant \
  -H "API-Key: $COLDTILL_API_KEY" -H 'Content-Type: application/json' \
  -H 'Idempotency-Key: refund-INV-1001' \
  -d '{"blockchainCode":"BASE","currencyCode":"USDC","amount":"12.50","toAddress":"0xAbC…"}'
Enter fullscreen mode Exit fullscreen mode

Retrying with the same key and the same parameters returns the original payout. Reusing the key with different parameters returns 409. That second rule matters more than it looks: if your code reuses a key for a different amount, you want a loud error, not a silent "here's the old payout".

Two practical habits on the caller's side:

  • Derive the key from something that already exists exactly once in your system, like the refund or invoice id. A random UUID generated inside the retry loop defeats the point.
  • Store the payout id the response gives you, and look the payout up by that id later instead of creating it again.

API payouts also need a full-access key and can only go to addresses in the project's address book, which can only be edited in the dashboard. That's a security control rather than an idempotency one, but it means a bug in your retry logic can, at worst, pay a known address twice, not a random one.

Layer two: a state machine with honest states

Here are the payout statuses from the docs, roughly in the order a payout moves through them:

Status Meaning
pending-approval Held by a limit or the auto-approve amount
pending-otp-verification Approval started, waiting for the emailed code
pending Approved, queued for signing
signing Signed and stored, being broadcast
initiated Broadcast, waiting for confirmations
sent Confirmed on chain
failed Definitely did not happen
rejected / cancelled Stopped by an admin, or cancelled before signing
needs-review Can't tell whether it landed

The word I'd underline is "definitely" in failed. In a lot of systems, failed means "an error happened somewhere". Here it means the transaction reverted, or was refused and is unknown to every node. Anything short of that certainty gets a different status. That's what makes it safe for your shop to treat payout.failed as "you may try again".

Layer three: sign, store, then broadcast

The payouts process claims a pending payout and reserves what the chain needs to make the transaction unique: a nonce on EVM, a recent blockhash on Solana. Then it signs the transaction and stores the signed bytes before broadcasting.

The order is the whole trick. In sketch form (this is the idea, not Coldtill's source):

claim payout P (status: pending)
reserve nonce N for the hot wallet         # EVM
tx = sign(P.to, P.amount, nonce=N, fees)
in one DB transaction:
    save tx bytes and tx hash against P
    set P.status = signing
broadcast(tx)                               # may crash here
set P.status = initiated
Enter fullscreen mode Exit fullscreen mode

If the process crashes after saving but before broadcasting, the restart finds a stored transaction and broadcasts the same bytes. Same bytes, same hash. If the first broadcast did get out, the second one is a harmless duplicate the network ignores. There's never a moment where the code has to choose between "maybe it went out" and "sign a fresh one".

Get the order wrong, broadcast first and save after, and a crash leaves you with a transaction on the network your database knows nothing about. The docs put it simply: a crash can never lead to a second, different transaction for the same payout.

Layer four: when it's stuck, replace it, don't add to it

On EVM chains, a transaction stuck with a low fee is re-signed with the same nonce, the same recipient, the same amount, and doubled fees. Because both versions share a nonce, only one of them can ever be mined. You've raised the bid on the same payout, not queued a second one.

One operational consequence: the bump costs native coin. If a payout sits in initiated, check that the hot wallet holds enough ETH, POL or BNB to pay the higher fee.

On Solana there's no nonce to reuse in the same way. A transaction carries a recent blockhash, and if that blockhash expires before the transaction lands, it can no longer be included. Re-signing means a new blockhash, which means a genuinely different transaction. So before doing that, Coldtill asks every node, with full history search, whether the old transaction ever landed. It re-signs only if none of them has seen it.

Asking one node isn't enough, because one node can be behind, rate-limiting you, or simply wrong. For ordinary reads Coldtill tries its preferred node first. For "this payout never landed", the decision where being wrong pays someone twice, every node gets a vote.

Layer five: don't believe the first "no"

Here's a failure that only shows up once you have more than one RPC provider. You broadcast to node A, it times out, you fail over to node B, and node B says "insufficient funds". Did the payout fail?

Maybe not. Node A may have accepted it, and the wallet now has less money because the payout is already in the mempool. So Coldtill doesn't fail a payout on an outright refusal. It waits two minutes and fails it only if the transaction is still unknown.

Layer six: when you don't know, say so

The last state is the one I like most, because it's the one most systems leave out.

needs-review means the hot wallet's nonce was used, but Coldtill can't match the transaction that used it to this payout, even after 30 minutes. The docs give two causes: someone used the hot wallet outside Coldtill, or your nodes are badly out of sync.

Coldtill won't guess. A payout wrongly marked failed invites a retry that pays twice, and a payout wrongly marked sent tells your customer money is on its way when it isn't. So a person decides:

  1. Open the hot wallet's address in a block explorer and look for a transfer of that amount to that address.
  2. In Payouts, click Resolve, choose "It was sent" or "It was not sent", add a note such as the explorer link, and confirm with two-factor.

The decision goes into the audit log and sends a final payout.sent or payout.failed webhook.

The Coldtill dashboard, with Payouts and Hot wallets in the sidebar next to the payments overview

What this means for the code on your side

If your shop or back office reacts to payout webhooks, the states map to fairly simple rules:

  • payout.sent: mark the refund done and store the transaction hash.
  • payout.failed: safe to create a new payout if you still want to pay. Use a new idempotency key, because it's a new payout.
  • payout.needs-review: do nothing automatic. The webhook docs say it directly: don't retry until an admin has checked the hot wallet's history and resolved it.
  • Everything else (pending-approval, pending, initiated): wait.

And one rule for whoever runs the server: don't spend from the hot wallet with other software. Coldtill expects to be the only thing using it, and a wallet shared with a script or a browser extension is exactly how you produce needs-review payouts in the first place. Keep the float small, about a week of gas plus expected payouts is the guidance in the docs, and top it up from your cold wallet.

The general lesson

None of this is specific to one gateway. If you're building anything that sends money, the pattern is:

  • make requests idempotent at the edge,
  • persist what you're about to do before you do it,
  • replace stuck work instead of duplicating it,
  • require more evidence for "it didn't happen" than for "it did",
  • and keep a state for "I don't know" that stops automation and asks a human.

The last one feels like admitting defeat when you design it. In practice it keeps the other four honest. The full payout flow, including limits and the two-step approval for large amounts, is in the payouts section of the Coldtill docs.

Top comments (0)