DEV Community: Jayesh Pamnani

The backend mistake that causes duplicate payments and orders

Jayesh Pamnani — Tue, 19 May 2026 06:33:02 +0000

One of the most dangerous backend mistakes is assuming this:

“If the request failed, nothing happened.”

That assumption causes duplicate payments, duplicate orders, duplicate invoices, and a lot of production cleanup.

In real systems, a failed response does not always mean the operation failed.

Sometimes the operation succeeded, but the response never came back.

Where this usually happens

This problem appears a lot in flows like:

payment processing
order creation
invoice generation
webhook handling
third-party API integrations
background job retries

Example:

Your backend sends a payment request to a payment gateway.

The gateway charges the customer.

But before your server receives the response, something fails:

network timeout
server restart
gateway delay
proxy timeout
worker crash

Your system thinks:

“Payment failed. Let’s retry.”

But the payment may have already succeeded.

Now the customer gets charged twice.

The real problem

The issue is not the retry itself.

Retries are necessary.

The real problem is retrying without idempotency.

Without idempotency, every retry is treated like a new action.

So this:

Create order
Create order again
Create order again

becomes three different orders.

And this:

Charge customer
Retry charge
Retry charge again

can become multiple charges.

What idempotency means

Idempotency means the same operation can be repeated safely without creating duplicate results.

If the same request is sent twice, the system should return the same result instead of doing the action again.

For example:

Request ID: payment_123
Action: charge customer $100

If the request is received again with the same ID, the system should not charge again.

It should return the original result.

How to fix it

1. Use idempotency keys

Every critical operation should have a unique idempotency key.

Good examples:

order_982_checkout_attempt_1
payment_982_customer_45
invoice_982

Store this key before executing the operation.

If the same key appears again, return the existing result.

2. Store operation status

Do not only store final success or failure.

Track states like:

pending
processing
success
failed

This helps avoid duplicate execution when something is already in progress.

3. Make retries safe

Retries should be controlled.

Avoid blind retries.

A retry system should know:

what operation is being retried
whether it already succeeded
how many times it was retried
what error caused the retry
whether it is safe to retry

4. Handle webhooks carefully

Payment gateways often send webhooks after checkout.

But webhooks can arrive:

late
multiple times
out of order

Never assume a webhook is unique.

Always check the external transaction ID before creating or updating records.

5. Use database constraints

Application logic is not enough.

Add database-level protection where possible.

For example:

UNIQUE(transaction_id)
UNIQUE(idempotency_key)
UNIQUE(external_order_reference)

This gives you a final safety layer if application logic fails.

The mistake most teams make

They design for the happy path.

Customer clicks pay.
Payment succeeds.
Order is created.
Invoice is generated.

But production does not only run happy paths.

Production has:

timeouts
retries
duplicate callbacks
delayed webhooks
partial failures
users clicking twice
workers restarting mid-process

If your backend does not expect these cases, duplicates are only a matter of time.

A better approach

For critical flows, think like this:

Can this operation run twice safely?

If the answer is no, the system is fragile.

Payments, orders, invoices, and stock updates should never depend only on “the request probably ran once”.

They need protection at the design level.

How we handle this at BrainPack

At BrainPack, we design payment and order flows with idempotency from the beginning.

For critical operations, we use idempotency keys, transaction tracking, retry-safe processing, webhook validation, and database-level uniqueness rules.

The goal is simple:

A timeout should not become a duplicate payment.

A retry should not become a duplicate order.

And a webhook should not create business data twice.

The hidden problem with long-running API requests

Jayesh Pamnani — Mon, 11 May 2026 06:23:17 +0000

A request taking 2-3 seconds locally does not feel like a problem.

In production, it becomes one very quickly.

Most backend issues I’ve seen around APIs were not caused by bad logic.

They were caused by requests staying open for too long.

Why this becomes dangerous

Long-running requests hold resources.

Usually:

database connections
memory
worker threads
external API sessions

One slow request is manageable.

Hundreds of slow requests at the same time start creating bottlenecks across the entire system.

And the worst part is that it often happens gradually.

Everything works fine in staging.

Production traffic exposes the real problem.

Common causes

1. Too much business logic inside a single request

A request comes in and the API tries to:

validate data
generate reports
process images
call external APIs
update multiple systems
send emails

All before returning a response.

This is one of the biggest architectural mistakes in backend systems.

2. Waiting on third-party APIs

External services are unpredictable.

Even if your own system is optimized, a slow payment gateway or ERP API can keep your request hanging for several seconds.

Now multiply that by hundreds of concurrent users.

3. Database queries that grow over time

A query that works fine with 10,000 rows behaves very differently with 10 million.

This is why APIs suddenly become slow months after deployment.

The code did not change.

The data volume did.

4. File processing during requests

Uploading files is fine.

Processing them synchronously is where problems start.

PDF generation, image optimization, AI processing, video conversion - these should rarely happen inside the request lifecycle.

What long-running requests actually cause

People usually think:
“Worst case, the API is slow.”

The real impact is much worse.

You start seeing:

request queues
worker exhaustion
timeout errors
database connection starvation
memory spikes
cascading failures across services

One slow endpoint can affect unrelated parts of the system.

The fix is usually architectural

The solution is not increasing server size forever.

The real fix is separating:

immediate response
background execution

A better pattern

Instead of this:

Client → API → Heavy processing → Response

Do this:

Client → API → Queue job → Immediate response

Worker → Process asynchronously

The API should respond quickly.

Heavy operations should happen in workers, queues, or event-driven systems.

Another important fix: timeouts

A surprising number of systems have no proper timeout handling.

Every external request should have:

connection timeout
read timeout
retry strategy
failure handling

Otherwise your workers end up waiting forever.

The mindset shift

Fast APIs are not only about speed.

They are about system stability.

A backend that responds quickly under load is usually designed around:

short-lived requests
async processing
isolation between services
controlled retries

That architecture matters more than raw server power.

Most production performance problems are not caused by traffic alone.

They come from APIs trying to do too much before returning a response.

How we handle this at BrainPack

At BrainPack, we design backend systems with this in mind from the start.

Long-running operations are separated from the request lifecycle using queues, workers, event-driven flows, and execution layers that keep APIs responsive even under heavy operational load.

The goal is simple:

Keep the system reactive for users while heavy processing happens safely in the background.

The cron job mistake most backend developers make

Jayesh Pamnani — Tue, 05 May 2026 10:10:37 +0000

Most developers assume one thing:

“If the cron is scheduled, it will run.”

That assumption breaks in production.

Cron jobs don’t fail loudly.

They fail silently.

And when they fail, no one notices until something important is already broken.

The real problem

Cron jobs are usually treated as “set and forget”.

Add a schedule
Write the logic
Deploy

Done.

But in reality, cron jobs depend on multiple things:

server uptime
environment variables
database connections
external APIs
timezones

If any of these fail, your cron job either:

doesn’t run
crashes midway
or runs incorrectly

And most of the time, you won’t even know.

What this looks like in production

invoices not generated
emails not sent
reports not updated
cleanup scripts not executed

Everything looks fine on the surface.

Until a user reports it.

Or worse - business logic silently stops working for days.

The mistake

Treating cron jobs like background utilities instead of critical systems.

If your cron job handles anything important, it is part of your core system.

It should be treated the same way as an API endpoint.

How to fix it

1. Add proper logging

Not just print statements.

Log:

start time
end time
success/failure
error details

If you can’t trace execution, you don’t control it.

2. Track execution status

Store every run in a database or monitoring system.

Example:

last run time
status (success/failed)
duration

This helps you detect:

missed runs
long-running jobs
repeated failures

3. Add alerts

If a cron fails, you should know immediately.

Slack alerts
email alerts
monitoring tools

Waiting for users to report issues is not a strategy.

4. Handle retries properly

External APIs fail. Networks fail.

Your cron should:

retry with limits
handle partial failures
avoid duplicate execution (idempotency)

5. Make jobs idempotent

If your cron runs twice, nothing should break.

This is critical when:

retries happen
servers restart
jobs overlap

6. Set timeouts

Never allow a cron to hang forever.

define execution limits
kill stuck processes
log timeout failures

A better approach

Stop thinking in terms of cron only.

Think in terms of job systems.

queue-based workers
event-driven triggers
background processing frameworks

Cron should trigger work.

Not handle everything itself.

The shift

Cron jobs are not “background scripts”.

They are part of your production system.

If they fail, your business logic fails.

Treat them like first-class components.

Monitor them. Track them. Own them.

If you’ve worked on production systems, you’ve probably seen at least one silent cron failure.

Curious how others are handling this in their stack.

Structural limitations in SaaS products that block AI-first workflows

Jayesh Pamnani — Wed, 29 Apr 2026 18:50:42 +0000

Most SaaS products were not built for AI.

They were built for forms, dashboards, and predictable workflows.

You can add AI on top. Many do.

But once you try to run real workflows through it, the limitations show up quickly.

1. CRUD-based architecture

Most SaaS systems are built around CRUD.

Create. Read. Update. Delete.

Every flow is:

user inputs data
system stores it
user retrieves it

AI does not fit this model.

AI:

generates outputs
makes decisions
works with incomplete context

Trying to fit AI into CRUD flows creates friction everywhere.

2. Rigid data models

SaaS systems rely on fixed schemas.

defined fields
strict validation
predictable relationships

AI needs flexibility.

unstructured input
variable output
evolving context

When the data model is rigid, AI either:

gets restricted
or breaks the system assumptions

3. No decision layer

Typical SaaS architecture has:

UI
backend services
database

There is no layer designed for decision-making.

So AI ends up:

embedded inside services
scattered across endpoints

This creates:

inconsistent behavior
hard debugging
duplicated logic

Without a dedicated layer, AI becomes unmanageable.

4. Synchronous execution model

SaaS systems assume fast responses.

AI does not guarantee that.

responses take time
retries are needed
outputs may need validation

If everything is synchronous:

APIs slow down
users wait
systems timeout

AI-first workflows require async design.

Most SaaS systems don’t have that built in.

5. No tolerance for uncertainty

SaaS systems expect exact outputs.

AI produces probabilistic results.

That means:

output can vary
structure can change
confidence is not guaranteed

Without handling this:

wrong data gets stored
actions trigger incorrectly
trust in the system drops

AI needs validation and control layers.

Most SaaS products don’t have them.

6. No system-wide context

SaaS products operate in silos.

Each module:

has its own data
its own logic
its own workflows

AI needs cross-system context.

history
relationships
external data

Without that, AI becomes:

shallow
inaccurate
limited in value

What we changed

We stopped trying to fit AI into SaaS patterns.

Instead:

introduced a separate AI layer
moved workflows to async where needed
allowed flexible data handling
added validation for every AI output
built context across systems

AI is not a feature inside SaaS.

It is a layer that sits across the entire system.

Final thought

Most SaaS products don’t fail with AI because of models.

They fail because their architecture was never designed for it.

If the system is built for predictable flows, AI will always feel like an add-on.

And add-ons don’t scale.

What breaks when you try to plug AI into an existing backend

Jayesh Pamnani — Tue, 28 Apr 2026 07:07:32 +0000

Most teams think adding AI is simple.

Call an API. Send some data. Get a response.

In reality, the moment you try to plug AI into an existing backend, things start breaking.

Not because AI is hard.

Because your backend was never designed for it.

1. Your data is not usable

AI needs clean, structured, and consistent data.

Most backends don’t have that.

You’ll find:

missing fields
inconsistent formats
duplicated records
business logic spread across code

Your system works because humans understand the gaps.

AI doesn’t.

So before AI works, you end up fixing your data layer.

2. Your workflows are too rigid

Backends are built for deterministic flows.

AI is not deterministic.

Example:

Backend expects exact input
AI returns slightly different structure
Flow breaks

Or worse:

AI output needs validation
system has no place to handle it

You realize quickly that your workflows are too strict for AI.

3. No place for AI in the architecture

Most systems don’t have a layer for decision-making.

They have:

controllers
services
database

Where does AI sit?

If you plug it directly:

logic gets mixed
debugging becomes harder
behavior becomes unpredictable

Without a proper layer, AI turns into scattered API calls.

4. Latency becomes a problem

Your backend was designed for fast responses.

AI is slower.

Now:

APIs take longer
users wait
timeouts start happening

If you don’t redesign flows:

async handling
queues
background jobs

your system performance drops.

5. Cost is not controlled

Normal backend logic has predictable cost.

AI doesn’t.

more tokens → more cost
retries → double cost
bad prompts → wasted cost

Without control:

costs grow fast
no visibility on usage

Most systems are not built to track this.

6. No way to handle bad output

Traditional systems assume correct output.

AI can:

hallucinate
return partial data
format responses differently

If your system trusts the output:

wrong actions happen
data corruption starts

You need validation layers.

Most backends don’t have them.

What we changed

After hitting these issues, we stopped treating AI like a feature.

Instead:

cleaned and structured data first
added an AI layer between backend and logic
moved AI flows to async where needed
validated every AI response
tracked usage and cost

Final thought

AI doesn’t break your system.

It exposes what was already weak.

If your backend is not designed for flexibility, visibility, and control - AI will make that obvious very quickly.

Why integrations fail in production (not theory, what actually breaks)

Jayesh Pamnani — Mon, 27 Apr 2026 13:35:24 +0000

On paper, integrations look simple.

ERP connects to CRM. CRM connects to external tools. APIs handle the rest.

Everything looks clean.

Then you go live - and things slowly start breaking.

Not immediately. But enough to cause real problems.

Here’s what actually goes wrong.

1. Systems don’t agree on what data means

A “customer” is not the same everywhere.

In CRM - it’s a lead with activity
In ERP - it’s a billing entity
In other tools - it might just be an email

You map fields and think you're done.

Then you get:

duplicates
missing data
updates overriding each other

The issue is not mapping. It’s the data model.

2. No clear source of truth

Multiple systems updating the same fields is where things fall apart.

CRM updates phone number
ERP updates it later
Another tool syncs an older value back

Now you have conflicting data and no clarity.

If ownership is not defined, consistency is impossible.

3. APIs work, workflows don’t

APIs returning 200 does not mean your system works.

Example:

CRM creates a deal
ERP needs validated customer data
External tool triggers actions

If timing and sequence are not defined:

things happen too early
required data is missing
processes silently fail

APIs move data. Workflows define behavior.

4. No handling for failure

Production always fails at some point.

API timeouts
network issues
third-party downtime

If your system assumes everything works:

data gets lost
retries create duplicates
processes break halfway

You need:

retries
idempotency
proper logging

Otherwise, failures stay hidden until users notice.

5. Real-time everywhere (when it shouldn’t be)

Real-time sounds good. It’s often a mistake.

It creates:

race conditions
higher load
harder debugging

Not everything needs to be instant.

Some flows should be queued or batched.

6. No visibility across systems

When something breaks, no one knows where.

Each system has its own logs.

There is no single place to see:

what triggered
what failed
what is pending

Without visibility, debugging becomes guesswork.

What changed for us

We stopped treating integration as just connecting APIs.

Instead:

defined ownership of data
controlled workflows centrally
handled failure as a default case
made everything traceable

Final thought

Most integrations don’t fail because of bad code.

They fail because no one designed how systems should behave together.