DEV Community: Obinna Victor

Idempotency Is Not Just UUIDs: Handling Ambiguous Payment Outcomes

Obinna Victor — Thu, 02 Jul 2026 10:35:12 +0000

Most developers learn idempotency as:

“Send an idempotency key so the same request does not happen twice.”

That is true.

But in financial systems, it is not enough.

Idempotency is not just about generating a UUID.

It is about making sure the same financial action cannot be executed twice, even when workers crash, providers timeout, webhooks arrive late, and clients retry.

This becomes even more important when the action touches money:

payment
transfer
refund
stablecoin transfer
blockchain transaction submission
AI-agent requested financial action

This is one of the core problems I’m thinking about while building Azums, a non-custodial durable financial execution layer for APIs and AI agents.

Azums launches with Paystack payments, Paystack transfers, Paystack refunds, Solana transactions, and Solana USDT transfers.

But before talking about Azums, let’s talk about the real problem.

The dangerous payment flow

Imagine this flow:

Client
→ Your API
→ Payment Provider
→ Provider processes request

Your system calls the provider.

Then the request times out.

What do you do?

Many systems do this:

timeout = failed
retry = try again

That is dangerous.

Because the provider may have already received and processed the request.

So your system may be in this state:

Your API thinks: failed
Provider state: maybe succeeded
Customer state: unknown

That is not failure.

That is an ambiguous outcome.

And ambiguous outcomes require a different design.

The wrong way to use idempotency

A common mistake is generating a new UUID for every retry.

Example:

const idempotencyKey = crypto.randomUUID();

await provider.transfer({
amount,
recipient,
idempotencyKey,
});

This looks fine until retry happens.

If the client or server creates a new key on retry, the provider may treat it as a new financial action.

That defeats the point of idempotency.

The key idea is:

Same financial action must map to the same idempotency identity.

For many systems, a better key is derived from stable business identity:

workspace_id + action_type + external_reference

Example:

workspace_123:paystack_transfer:order_789

or:

workspace_123:refund:payment_ref_456:refund_001

The key should represent the action, not the attempt.

Action vs attempt

This distinction matters.

A financial action is the thing the user or system wants:

Refund ₦10,000 for payment X
Transfer ₦50,000 to recipient Y
Submit signed Solana USDT transfer Z

An attempt is one try to execute that action.

One action may have multiple attempts.

But it should have only one final financial execution.

A simplified model:

financial_actions (
id,
workspace_id,
action_type,
idempotency_key,
payload_hash,
status,
provider_reference,
created_at,
updated_at
);

execution_attempts (
id,
financial_action_id,
attempt_number,
status,
started_at,
finished_at,
error_class,
provider_reference
);

The action is the root object.

Attempts are evidence.

Do not bill, receipt, or finalize based only on attempts.

Finalize based on the root financial action and external truth.

Payload fingerprinting

The same idempotency key should not allow different payloads.

Example:

First request:

{
"amount": 10000,
"recipient": "recipient_A"
}

Retry with same key but changed payload:

{
"amount": 50000,
"recipient": "recipient_B"
}

This should not execute.

The system should return conflict.

A common pattern:

idempotency_key + payload_hash

If the key exists and the payload hash matches, return the existing transaction.

If the key exists and the payload hash differs, reject with conflict.

Example behavior:

same key + same payload = return existing transaction
same key + different payload = 409 conflict
new key + valid payload = create new transaction

This prevents accidental or malicious mutation of a financial action.

Retries need execution boundaries

Retries are not always safe.

A worker can fail at different points:

before provider call
during provider call
after provider accepted request
after provider completed request
before local database update
before receipt creation
before callback delivery

Each point requires different handling.

If failure happens before provider submission, retry may be safe.

If failure happens after provider submission, retry may duplicate money movement unless the system first checks provider truth.

So the system needs an execution boundary:

Before provider submission:
safe to retry

After provider submission:
do not blindly retry
verify provider state first

This is where many systems break.

Webhooks are not enough

Webhooks are useful, but they are not enough.

A webhook can be:

delayed
duplicated
missed
delivered out of order
received before internal state is ready

A strong system treats webhook events as signals, not final truth.

The safer model:

Webhook = fast signal
Provider verification = direct check
Reconciliation = external truth comparison

For Paystack, that means verifying payment, transfer, or refund status against Paystack.

For Solana, that means checking signature, confirmation, finality, token movement, amount, recipient, and network.

For Solana USDT, that also means verifying the correct token mint and token account movement.

Receipt creation should happen after truth

A receipt should not just mean:

“Our worker finished.”

A receipt should mean:

“We have enough evidence to state the final outcome of this financial action.”

That receipt should include:

transaction id
action type
amount or asset
rail
provider or chain reference
final outcome
policy decision
approval reference if needed
reconciliation status
receipt hash
created timestamp

This is important because customers do not only need logs.

They need proof.

What about AI agents?

AI agents make this problem more urgent.

If an AI agent can request a refund or transfer, the system must answer:

Who is the agent?
What can it request?
What amount is allowed?
Does this require approval?
Was a one-use permission created?
Was the permission consumed?
Was execution submitted once?
Was the outcome verified?
Was a receipt created?

An AI agent should not directly call the payment provider.

It should request a financial action through a control layer.

That control layer should enforce policy, approval, idempotency, receipts, reconciliation, and exceptions.

This is one of the reasons I’m building Azums.

A better lifecycle

A safer financial execution lifecycle looks like this:

Request received
→ identity verified
→ idempotency checked
→ payload fingerprinted
→ policy evaluated
→ approval required if needed
→ one-use permission created if needed
→ execution queued
→ provider or chain execution submitted once
→ outcome tracked
→ receipt created
→ callback delivered
→ reconciliation checked
→ exception opened if reality disagrees

This is more work than simply calling a provider API.

But this is the work that makes financial execution trustworthy.

How Azums approaches this

Azums is being built as a durable financial execution layer for APIs and AI agents.

The first launch scope is narrow:

Paystack payments
Paystack transfers
Paystack refunds
Solana transactions
Solana USDT transfers

Azums is non-custodial.

It does not hold customer funds or private keys.

The goal is to control execution around financial actions:

identity
idempotency
policy
approval
one-use permissions
provider or chain execution
receipts
callbacks
reconciliation
exceptions
audit proof

The key belief is:

Financial systems should never silently fail, duplicate money movement, or pretend uncertainty is success.

Final thought

Idempotency is not just a UUID.

It is a correctness contract.

It says:

This financial action has one identity, one payload, one lifecycle, and one final truth.

When you combine idempotency with durable execution, receipts, reconciliation, and exceptions, you get a system that can survive the messy reality of financial infrastructure.

That is the kind of system I believe more teams will need as APIs and AI agents start touching real money.

I’m building Azums to explore that future.

Follow the build at getazums.com

One RPC Provider Is Not Blockchain Reliability

Obinna Victor — Sun, 28 Jun 2026 19:30:33 +0000

-One RPC Provider Is Not Blockchain Reliability

A lot of blockchain applications start with a very simple backend setup:

BACKEND_RPC_URL=https://some-rpc-provider.com

Then everything goes through that one provider.

balance checks
account reads
transaction lookups
latest block/slot checks
transaction simulation
transaction submission

At first, this feels fine.

The app works.
The backend can read chain state.
The frontend can show balances.
Transactions can be submitted.

But in production, one RPC URL can quietly become a hidden source of fragility.

Because one RPC provider is not the blockchain itself.

It is only one gateway into the blockchain.

If your entire backend depends on that one gateway, then your app is not only trusting the blockchain.

It is trusting one provider’s availability, freshness, latency, rate limits, supported methods, and view of the chain.

That is not reliability.

That is a single point of failure.

What RPC means in blockchain

RPC means Remote Procedure Call.

In general backend terms, RPC is a way for one program to ask another program or server to execute an operation and return a result.

In blockchain systems, an RPC endpoint lets your application talk to a blockchain node.

For example, your app may call:

getBalance
getAccountInfo
getTransaction
getLatestBlockhash
sendTransaction
simulateTransaction

Your app asks the RPC node a question or submits a transaction.

The RPC node responds or broadcasts the transaction to the network.

So a typical design looks like this:

Frontend / Backend
        ↓
RPC Provider
        ↓
Blockchain Network

This is normal.

The problem starts when this becomes your entire reliability model.

The naive architecture

A lot of apps are designed like this:

User
  ↓
Frontend
  ↓
Backend
  ↓
One RPC URL
  ↓
Blockchain

The backend trusts whatever the provider says.

If the provider responds, the backend assumes the response is enough.

If the provider times out, the backend treats it like failure.

If the provider cannot find a transaction, the backend assumes the transaction is missing.

If the provider is rate-limited, the app becomes degraded.

This works on the happy path.

It does not handle production reality well.

Failure mode 1: the RPC provider is slow

Imagine the backend needs a response within two seconds.

The provider responds after ten seconds.

Your backend times out.

A beginner system may treat that timeout as failure.

But timeout does not mean the blockchain failed.

It means:

My system did not receive an answer in time.

That is not the same thing as:

The operation failed on-chain.

This distinction matters a lot.

Especially when the request is related to transaction status or submission.

Failure mode 2: the provider is rate-limited

Now imagine your product starts getting more usage.

Many users are checking balances.
Workers are checking transaction status.
Background jobs are polling chain state.
The dashboard is refreshing operational data.

Then the RPC provider starts returning rate-limit errors.

If your backend only has one provider, your whole system becomes dependent on that provider’s quota.

A better system should be able to fail over, shed load, cache safe reads, or route intelligently.
Failure mode 3: the provider is stale

This is one of the most dangerous cases.

Suppose your backend checks whether a transaction landed.

Provider A says:

Transaction not found

But Provider B can already see it.

If your backend only trusts Provider A, it may mark the transaction as failed or unknown too early.

In blockchain systems, stale reads can create bad product behavior:

wrong user balances
wrong transaction status
incorrect failure messages
unnecessary retries
confused operators
broken reconciliation

One provider’s view is not always enough.

Failure mode 4: provider-specific errors

Sometimes one provider returns an error while another provider would have succeeded.

This can happen because of:

provider outage
regional latency
method support differences
rate limits
stale indexers
degraded nodes
provider-specific bugs
chain lag

So the problem is not simply:

Did the blockchain work?

The better question is:

Is this provider giving my backend a reliable view of the blockchain?

The dangerous part: knowing what actually happened

In blockchain systems, the dangerous part is not only sending a transaction.

The dangerous part is knowing what actually happened after the transaction was sent.

Questions your backend should be able to answer:

Did the transaction land?
Did it fail?
Is it still pending?
Did the provider timeout before returning the signature?
Did the provider return stale data?
Did another provider see the transaction?
Was the backend trusting incomplete information?
Is it safe to retry?
What evidence do we have?

If your system cannot answer those questions, your operators are blind.

And if operators are blind, users eventually feel it.

A better architecture: RPC reliability layer

A better architecture introduces an RPC reliability layer.

User
↓
Frontend
↓
Backend
↓
RPC Gateway / Reliability Layer
↓
Provider A
Provider B
Provider C
↓
Blockchain Network

The backend no longer directly trusts one provider.

Instead, it routes through infrastructure that understands provider failure.

This RPC layer can handle:

provider health checks
request timeouts
failover
provider scoring
method-aware routing
safe caching
request coalescing
cross-provider validation
operational response headers
status visibility

The difference is mindset.

The naive system says:

I trust this one RPC URL.

The better system says:

I route through infrastructure that understands RPC failure.

Provider failover

Provider failover means:

If Provider A fails, try Provider B.
If Provider B fails, try Provider C.

Simple idea, big impact.

One provider being down should not mean your whole blockchain app is down.

The system should track things like:

which provider is healthy
which provider is slow
which provider recently failed
which provider is behind
which provider is rate-limited
which provider has better success rate

Then it should route intelligently.

Timeout handling

Timeouts should be treated carefully.

A timeout is not proof of failure.

It is proof that the backend did not receive an answer in time.

For read requests, retries are usually safer.

For write requests like sendTransaction, retry behavior needs more care.

Why?

Because transaction submission and transaction status are not the same as reading account data.

Your system needs method awareness.

Method policy

Not all RPC methods should be treated the same.

Some methods are read-only.

Some submit transactions.

Some can be cached.

Some should never be cached.

Some are more important for correctness.

Examples:
getBalance -> read
getAccountInfo -> read
getTransaction -> status/evidence read
getLatestBlockhash -> time-sensitive read
simulateTransaction -> simulation
sendTransaction -> write/broadcast

A serious RPC gateway should have method policy.

That policy can answer:

Can this method be cached?
Should this method be validated?
Is this method consensus-critical?
Is retry safe?
Should this method use multiple providers?

This is how you move from random RPC calls to infrastructure design.

Request coalescing

Request coalescing is another useful pattern.

Imagine 100 requests ask for the same data at the same time.

The naive system sends 100 identical upstream calls to the RPC provider.

That increases:

load
cost
rate-limit risk
latency pressure

A better gateway can notice that the requests are identical.

It sends one upstream request and shares the result with the waiting callers.

100 identical local requests
↓
1 upstream RPC call
↓
shared result

That is request coalescing.

It helps the backend stay stable under traffic.

Caching, but carefully

Caching can help, but it must be method-aware.

The mistake is caching everything blindly.

If you cache the wrong data, your app may show stale state.

If you cache nothing, your app may hit rate limits faster and waste money.

So caching should depend on the method.

Questions to ask:

Is this method safe to cache?
How long should it be cached?
Is this data user-facing?
Is this data used for a financial or execution decision?
Is stale data dangerous here?

Caching is not just a performance feature.

In blockchain infrastructure, it is also a correctness decision.

Cross-provider validation

For some important reads, one provider may not be enough.

Example:

Provider A: transaction not found
Provider B: transaction found
Provider C: transaction found

What should the backend believe?

For every request, cross-provider validation may be too expensive.

But for important reads around execution state, settlement, wallet safety, or transaction status, it can protect your system from trusting one stale provider.

The point is not to call every provider all the time.

The point is to know when one signal is too weak.

Operator visibility

A good RPC reliability layer should not hide what happened.

It should expose operational truth.

When something goes wrong, operators should be able to answer:

Which provider did we use?
Did the provider timeout?
Did we retry another provider?
Was the response from cache?
Was the request coalesced?
Did we validate across providers?
Did providers disagree?
Was this method considered critical?
Did the backend make a decision from enough evidence?

This is why response metadata matters.

Infrastructure should not only return data.

It should explain how the data was obtained.

My RPC Gateway project

I explored this idea in my RPC Gateway project.

GitHub:

https://github.com/BlockForge-Dev/RPC-Gateway

The goal of the project is not just to forward JSON-RPC requests.

The goal is to make provider failure, latency variance, stale reads, and read disagreement visible.

The project explores:

multi-provider failover
adaptive hedging
predictive provider scoring
Solana method policy
request coalescing
method-aware caching
consensus validation for important reads
provider health tracking
response headers that explain selected provider, attempts, cache behavior, hedging, and validation

I also recorded a video explaining the idea here:

https://youtu.be/Mv7R9ISm0rA

Bigger lesson

Blockchain apps are not reliable just because they use a blockchain.

Your backend can still be fragile.

Your RPC provider can fail.
Your worker can crash.
Your webhook can arrive late.
Your provider can be stale.
Your transaction can be ambiguous.
Your UI can show the wrong state.

So serious blockchain infrastructure needs to design for failure from the beginning.

Not after users complain.

Not after funds are stuck.

Not after operators are confused.

From the beginning.

Final thought

One RPC provider is not blockchain reliability.

One RPC URL is just one view into the chain.

If that view is slow, stale, rate-limited, or down, your backend needs a better plan.

That better plan includes:

multiple providers
health checks
timeouts
failover
method policy
safe retries
request coalescing
caching with care
cross-provider validation for important reads
status visibility
receipts
reconciliation

This is the kind of backend and blockchain infrastructure I build.

I build systems around reliable transaction execution, RPC reliability, operator truth, receipts, and reconciliation.

GitHub:

https://github.com/BlockForge-Dev

Why I treat API timeouts as "unknown", not failures

Obinna Victor — Fri, 29 May 2026 12:23:31 +0000

Every payment gateway I've ever worked on had the same hidden bug.

A provider API times out. The code says "failure". So you retry. But the original request actually succeeded – the provider just took too long to respond. Now you've double‑charged the customer.

I built Azums, an open‑source payment gateway in Rust, specifically to stop this pattern.

__The fix: make ambiguity explicit.

Instead of pending → success/fail, I designed a state machine with five states:

pending (request sent, waiting)
succeeded (confirmed success)
failed (confirmed failure)
retryable (temporary error, safe to retry)
unknown (timeout or ambiguous response – needs investigation)

When a timeout happens, the system doesn't guess. It marks the transaction as unknown and stops. No blind retries. No double charges.

Why this matters beyond payments

This same pattern applies anywhere you talk to unreliable external systems:

Blockchain RPCs that timeout after the transaction was submitted
AI agent API calls that hang but may have executed
Messaging queues that lose acknowledgements

Treating ambiguity as a real state is the difference between a system that guesses and a system that you can trust.

My full implementation is on GitHub: BlockForge-Dev/Azums

What's your worst "timeout caused a disaster" story? Let me know in the comments.