DEV Community: Shashank Pandey

Idempotency: The Bug You Don't Notice Until Production

Shashank Pandey — Fri, 07 Aug 2026 09:36:43 +0000

I didn't learn about idempotency from a system-design interview.

I learned it while building one of my projects.

Everything seemed to be working exactly as expected.

A webhook arrived.

My backend processed it.

The database was updated.

The request completed successfully.

Then it happened again.

And again.

The same event was being processed more than once.

At first, I thought the webhook provider was doing something weird.

It wasn't.

My system simply wasn't designed for reality.

The Assumption That Broke Everything

When I first built the webhook flow, the mental model was pretty simple:

Webhook arrives
      ↓
Process webhook
      ↓
Update database
      ↓
Done

It feels reasonable.

An event happens → the provider sends it → my server processes it.

One event.

One request.

One operation.

Except distributed systems don't work that cleanly.

The actual world looks more like:

Webhook Provider
       |
       | POST /webhook
       ↓
    My API
       |
       ↓
   Processing
       |
       ↓
   Database

And somewhere along the way, things can fail.

The provider might send the request successfully.

My server might process it successfully.

But the response might never make it back.

Maybe the connection times out.

Maybe the server crashes immediately after processing.

Maybe there is a transient network failure.

From my application's perspective:

"Everything worked."

From the provider's perspective:

"Did they receive it?"

And when the provider doesn't receive the expected response, it may retry.

So now:

Event #123
    ↓
Request #1 → processed
    ↓
Response lost
    ↓
Request #2 → processed AGAIN

That's where the fun begins.

The Duplicate Event

Imagine my webhook receives this:

{
  "id": "evt_123",
  "type": "something.completed"
}

The first request arrives:

evt_123
   ↓
Process
   ↓
Database update
   ↓
Success

Then the provider retries:

evt_123
   ↓
Process
   ↓
Database update AGAIN
   ↓
Success

The second request isn't a new event.

It's another delivery of the same logical event.

That distinction is the key.

             ONE EVENT
                 |
       ┌─────────┼─────────┐
       ↓         ↓         ↓
   Delivery 1 Delivery 2 Delivery 3
       |         |         |
       └─────────┼─────────┘
                 ↓
        ONE BUSINESS EFFECT

The transport layer can deliver the event multiple times.

My application needs to make sure the business operation remains correct.

That's idempotency.

So, What Actually Is Idempotency?

In simple terms:

An operation is idempotent when repeating it produces the same final result as performing it once.

Mathematically:

f(f(x)) = f(x)

For webhooks, that means:

Receive event
    ↓
Process event
    ↓
Receive same event again
    ↓
Don't create another side effect

The important part isn't necessarily preventing the second request.

You usually can't control that.

The important part is preventing the duplicate effect.

That's a subtle but important distinction.

The First Solution I Thought Of

Once I understood what was happening, the obvious solution was:

"I'll just store the webhook ID."

Something like:

processed_webhooks

+----------------+
| event_id       |
+----------------+
| evt_123       |
| evt_456       |
| evt_789       |
+----------------+

Then:

Webhook arrives
      ↓
Extract event_id
      ↓
Have I processed this?
    /       \
  YES        NO
   |          |
   ↓          ↓
Ignore      Process
             ↓
       Store event_id

Conceptually, that's correct.

But there's another trap hiding inside it.

The Race Condition

Imagine two copies of the same webhook arrive almost simultaneously.

Request A                    Request B
    |                            |
    ↓                            ↓
Check event_id              Check event_id
    |                            |
    ↓                            ↓
"Not found"                 "Not found"
    |                            |
    ↓                            ↓
Process                     Process

Both requests checked before either one had recorded the event.

Now both execute the side effect.

evt_123
  ↓
Worker A → Process
  ↓
Worker B → Process

So this isn't enough:

if not already_processed(event_id):
    process_event()
    mark_as_processed(event_id)

There's a gap between:

check

and

insert

And concurrent requests can exploit that gap without doing anything malicious.

This is where the database becomes more than just storage.

Let the Database Enforce the Rule

Instead of trusting application logic alone, make the event ID unique.

For example:

CREATE TABLE webhook_events (
    event_id VARCHAR(255) PRIMARY KEY,
    processed_at TIMESTAMP NOT NULL
);

Now the database itself guarantees:

evt_123 → allowed
evt_123 → rejected
evt_123 → rejected

Even if multiple workers race to insert the same event ID, only one can win.

That changed how I thought about idempotency.

It's not simply:

if already_processed:
    return

It's a data consistency invariant:

The same logical event must not be able to create multiple copies of the same business effect.

And invariants are often much safer when your database enforces them.

Then I Found the Next Problem

But there's a deeper issue.

Suppose I do:

1. Insert event ID
2. Process event
3. Return success

What happens here?

Insert event ID
      ↓
SUCCESS
      ↓
Process business operation
      ↓
💥 Server crashes

The event is already marked as processed.

But the actual operation never finished.

The provider retries.

My application checks the event:

evt_123
   ↓
Already exists
   ↓
Skip

Now I've prevented duplication...

but I've also prevented recovery.

That's when idempotency stopped looking like a simple duplicate-check problem.

It became a failure-handling problem.

Idempotency Is About Failure

This is probably the biggest lesson I took away from the experience.

When everything works:

Webhook
   ↓
Process
   ↓
Database
   ↓
Success

Almost any implementation looks correct.

The interesting cases are:

What if the request arrives twice?

What if two copies arrive simultaneously?

What if the server crashes halfway through?

What if the database succeeds but the response fails?

What if the external API succeeds but my server crashes before recording it?

What if the worker retries?

These aren't weird edge cases.

They're normal failure modes in distributed systems.

And webhooks make them particularly obvious because retries are expected behavior.

The Mental Model I Use Now

I no longer think about a webhook as:

request → function → response

I think about it as:

event → state transition

The event has an identity.

The system has a state.

And processing the same event again shouldn't produce an invalid state.

For example:

RECEIVED
   ↓
PROCESSING
   ↓
COMPLETED

If something fails:

RECEIVED
   ↓
PROCESSING
   ↓
FAILED
   ↓
RETRY
   ↓
PROCESSING
   ↓
COMPLETED

Now the system has a way to reason about recovery instead of simply hoping the request succeeds.

And This Doesn't Stop at Webhooks

This is where the lesson became much bigger than the original bug.

The same problem exists with:

Webhooks
Message queues
Background workers
Scheduled jobs
Serverless functions
Payment processing
Event-driven architectures
Retry mechanisms
Distributed APIs

Anywhere something can be retried, you need to ask:

What happens if this operation runs twice?

Because eventually, it probably will.

The Real Meaning of "Exactly Once"

You'll sometimes hear people talk about exactly-once processing.

It sounds like the perfect solution:

Event → Process exactly once

But in real distributed systems, that's an extremely strong guarantee.

Your application may not be able to control how many times a message or webhook is delivered.

What it can control is whether repeated delivery produces repeated side effects.

So a much more useful design goal is:

At-least-once delivery
+
Idempotent processing
=
Correct business result

The event might arrive three times.

The system should still produce the result intended by one logical event.

That's a much more practical way to think about reliability.

Designing an Idempotent Webhook

If I were designing the webhook from scratch today, I'd start with these questions.

1. What uniquely identifies the event?

Prefer the provider's event ID when available.

evt_123

Not:

timestamp + user_id + event_type

unless you have a very specific reason to construct your own key.

2. Can the database enforce uniqueness?

Use something like:

event_id UNIQUE

or a primary key.

Don't rely solely on:

if not exists:

because concurrent workers exist.

3. What happens if processing fails halfway through?

You need a recovery strategy.

Possible approaches include:

transactions
processing states
retry queues
dead-letter queues
outbox patterns

The right choice depends on the system.

4. What about external side effects?

This one is easy to overlook.

Your database might be transactional.

The external API you're calling probably isn't part of that transaction.

If you call another service, ask:

Can I safely retry this request?

If that service supports idempotency keys, use them.

Now you have:

Webhook event ID
        ↓
Your idempotency key
        ↓
Downstream API idempotency key

The guarantee propagates through the system.

5. What happens when the worker crashes?

Assume it will.

Because eventually it will.

Design your processing state so that the system can distinguish between:

completed
failed
still processing
stuck
retryable

instead of treating everything as simply:

processed = true

What I Would Tell Myself Before Building It

If I could go back to the beginning of that project, I'd ask one question before writing the webhook handler:

"What happens if I receive this exact event twice?"

Then I'd ask:

"What happens if I receive it twice at exactly the same time?"

And finally:

"What happens if my server crashes halfway through?"

Those three questions expose a surprising amount of architectural weakness.

The Bug Wasn't Really the Bug

Looking back, the duplicate webhook wasn't the most interesting part.

The interesting part was realizing that my original mental model was wrong.

I was thinking:

Request
   ↓
Code
   ↓
Response

Production forced me to think:

Event
   ↓
Network
   ↓
Retries
   ↓
Concurrency
   ↓
Partial failure
   ↓
State
   ↓
Recovery

That's a very different way of thinking about backend systems.

And that's probably why idempotency is one of those concepts that feels trivial when you first hear the definition...

until you have to build it.

My Takeaway

Retries aren't bugs.

They're a normal part of distributed systems.

The mistake is designing your application as if retries won't happen.

A webhook provider can send the same event again.

A queue can deliver the same message again.

A worker can execute the same job again.

An API client can retry the same request again.

Your job isn't always to stop those things from happening.

Your job is to make sure repeating them doesn't corrupt the system.

That's idempotency.

And for me, it wasn't something I really understood because I read the definition.

I understood it because one of my projects broke my assumptions.

Sometimes production is a surprisingly effective teacher.

Designing a Cost-Effective Serverless Certificate Generation System on AWS

Shashank Pandey — Mon, 13 Jul 2026 10:56:56 +0000

How I used AWS Lambda and Amazon S3 to build a scalable certificate platform that minimizes storage costs while improving security.

Introduction

In my previous article, I shared why I decided to build a serverless certificate generation system for our student community. As our events grew, manually generating and distributing certificates became increasingly difficult because:

Gmail SMTP had daily sending limits and required one of us to keep a laptop running until all 200-300 certificate emails were delivered.
Google Apps Script required storing every generated certificate in Google Drive, leading to unnecessary long-term storage.
Traditional mailing APIs still required certificate generation to run on our own infrastructure, making the process compute-intensive and consuming our Cloudflare Workers quota.
Bulk email delivery through these approaches also increased the risk of emails being delayed or marked as spam.

These limitations made us rethink the entire workflow. Instead of trying to optimize the existing solution, we designed a serverless architecture on AWS that could generate certificates on demand, store them securely, and significantly reduce operational overhead.

The Challenge

Every event concludes with hundreds of participants expecting their certificates within minutes.

While generating a certificate isn't particularly difficult, doing it reliably for hundreds of users introduces several operational challenges.

The system needed to:

Scale automatically during peak demand.
Deliver certificates quickly.
Avoid long-running servers.
Reduce long-term storage.
Keep certificate files secure.
Be inexpensive enough for a student-led community.

Rather than permanently storing every generated PDF, I wanted a smarter approach that stored only what was necessary.

Architecture Overview

The platform uses a simple serverless architecture built entirely on managed AWS services.

Figure 1. High-level architecture of the serverless certificate generation workflow. AWS Lambda checks whether a certificate already exists in Amazon S3. If available, it generates a pre-signed URL for secure access. Otherwise, the certificate is regenerated, uploaded to Amazon S3, and served to the user. An Amazon S3 Lifecycle Rule automatically removes regenerated certificates after the configured retention period.

The idea is straightforward:

If a certificate already exists, serve it immediately.
If it doesn't, regenerate it from stored metadata.
Store it temporarily.
Automatically remove it later using Amazon S3 Lifecycle Rules.

Why AWS Lambda?

Certificate generation is not a continuous workload.

Immediately after an event, hundreds of participants may request certificates simultaneously, while on other days there may be no requests at all.

AWS Lambda was the ideal choice because it:

Eliminates server management.
Automatically scales with demand.
Charges only when code runs.
Integrates easily with other AWS services.

This means we never have to keep a server running just to generate certificates.

Why Amazon S3?

Certificates are simply generated PDF files.

Amazon S3 provides:

Highly durable object storage.
Excellent scalability.
Low storage costs.
Fine-grained access control.

Instead of exposing the bucket publicly, users receive temporary pre-signed URLs that allow secure downloads.

On-Demand Certificate Generation

One of the most important design decisions was treating certificates as derived artifacts instead of permanent data.

The participant information, event details, and certificate template remain the source of truth.

Whenever someone requests a certificate:

AWS Lambda checks whether the PDF already exists in Amazon S3.
If it exists, the system immediately returns a pre-signed URL.
If it doesn't exist, Lambda regenerates the certificate using the stored metadata.
The newly generated PDF is uploaded to Amazon S3.
A pre-signed download URL is returned to the user.

Because certificates can always be recreated, there is no need to keep every generated PDF forever.

Automatic Cleanup with Amazon S3 Lifecycle Rules

Keeping generated certificates forever didn't make much sense.

Most participants download their certificate only once or twice. Storing every generated file indefinitely would gradually increase storage usage while retaining documents that might never be accessed again.

To solve this, regenerated certificates are uploaded to Amazon S3 temporarily.

An Amazon S3 Lifecycle Rule automatically removes these files after a short retention period.

This provides several benefits:

Reduces unnecessary storage.
Keeps the bucket clean.
Limits the lifetime of generated certificate files.
Ensures certificates are recreated only when needed.

Since the system always retains the certificate metadata, deleting the generated PDF never results in data loss.

Security Considerations

Security was another important factor in the architecture.

The platform follows a few simple principles:

Amazon S3 buckets remain private.
Users never receive direct bucket access.
Downloads are provided using temporary pre-signed URLs.
IAM roles follow the principle of least privilege.
Generated certificate files are automatically removed after a limited period.

By reducing how long generated PDFs remain in storage, the platform also minimizes unnecessary exposure of sensitive documents.

Cost Optimization

As a student community, every cloud resource matters.

Several architectural decisions help keep operational costs low:

AWS Lambda runs only when certificates are requested.
Amazon S3 stores objects efficiently.
Lifecycle Rules automatically clean up generated files.
Certificates are regenerated only when required instead of occupying storage forever.

Rather than paying to retain thousands of PDFs indefinitely, the platform stores only what is necessary and recreates generated artifacts on demand.

Lessons Learned

Building this system reinforced several architectural principles.

First, not every generated file needs to be stored permanently. If it can be recreated reliably from source data, temporary storage is often the better choice.

Second, managed AWS services significantly reduce operational complexity. Instead of maintaining servers or writing cleanup scripts, services such as AWS Lambda and Amazon S3 handle much of the infrastructure automatically.

Finally, simple architectures are often the easiest to maintain. By focusing only on the services required for the workflow, the platform remains easy to understand, cost-effective, and scalable.

Conclusion

Migrating our certificate platform to a serverless architecture solved several challenges that we had experienced with previous approaches.

Instead of depending on laptops, SMTP servers, or long-running processes, the platform now generates certificates on demand using AWS Lambda, stores them securely in Amazon S3, and automatically cleans up generated files using Amazon S3 Lifecycle Rules.

The result is a solution that is scalable, secure, and significantly easier to operate.

Sometimes the best optimization isn't generating files faster—it's recognizing that you don't need to keep them forever.

Why I Built a Serverless Certificate Generation System

Shashank Pandey — Tue, 07 Jul 2026 10:25:35 +0000

Why I Built a Serverless Certificate Generation System

When I was the Group Leader of the AWS Student Builders Group at Lovely Professional University, one of the biggest challenges wasn't organizing events.

It was what happened after they ended.

Every event meant generating and distributing certificates to hundreds of attendees. Which sounds easy for now.

That's exactly what I thought. Turns out, it wasn't.

Especially when you're running a student community with little to no budget and infrastructure decisions directly affect what you can (and can't) afford.

The First Attempt

Like every developer, I started with the quickest solution.

Google Apps Script + Gmail.

It was easy to build, easy to maintain, and honestly, it worked pretty well...until our events started getting bigger.

Once we crossed the 200+ attendee mark, Gmail started becoming the bottleneck.

Sending that many emails in a short period meant running into sending limits, risking spam filters, and hoping every participant actually received their certificate.

What was supposed to be a one-click automation slowly became a process of checking failed deliveries, retrying emails, and occasionally answering, "Hey, I still haven't received my certificate."

Clearly, this wasn't going to scale.

The "Let's Put It on the Website" Phase

My next thought was simple.

"Why email certificates at all? Why not let attendees generate them themselves?"

So that's exactly what I did.

Participants could log into the event platform and generate their certificates whenever they wanted.

Problem solved...or so I thought.

The website was running on Cloudflare Workers, and generating PDF certificates turned out to be one of the most compute-heavy operations in the entire application.

The more certificates people generated, the more Workers quota disappeared.

At one point, the application felt like it was spending more time generating PDFs than actually serving the website.

I had fixed the email problem. Now I had a compute problem.

Taking a Step Back

Instead of looking for a bigger server or a more expensive plan, I asked myself a different question.

Why is the website generating certificates in the first place?

The website's job is to serve users.

Certificate generation is a completely separate task.

It doesn't have to happen inside the same request. It doesn't even have to happen on the same infrastructure.

That realization changed everything.

Instead of making the website do all the heavy lifting, it would simply say,

"Hey, someone requested a certificate."

And another system would take care of the rest.

That's when I decided to move towards an event-driven, serverless architecture.

How It Works Today

The platform already had a QR-code-based attendance system.

When attendees scanned the checkout QR code, their attendance was marked as complete.

Once that happened, they became eligible for a certificate.

A Generate Certificate button appeared on their dashboard, allowing them to generate the certificate up to three times for that event.

From the participant's perspective, that's all they ever see.

Attend the event.

Scan the QR code.

Click one button.

Download the certificate.

Simple.

Behind that single click, however, there's an event-driven workflow handling all the heavy processing without putting additional load on the main application.

Why I'm Happy With This Approach

The biggest lesson from this project wasn't learning a new AWS service.

It was learning that sometimes the solution isn't making your application stronger.

It's making your application responsible for less.

Once I stopped treating certificate generation as a website feature and started treating it as an independent workflow, everything else started falling into place.

The application became lighter. The workflow became more scalable.

And the operational cost became so low that our student community could comfortably afford it ourselves.

Literally around the cost of a few evening snacks.

What's Next?

This article was about why this system exists.

In the next one, I'll go full engineering mode.

I'll walk through the complete architecture, explain why I chose a serverless event-driven approach, why each AWS service earned its place in the system, and the trade-offs I had to make along the way.

Fine-Tuning Isn’t Enough Anymore | Amazon Nova Forge Changes the Game

Shashank Pandey — Wed, 11 Feb 2026 13:33:54 +0000

For the last two years, enterprise AI customization has revolved around three techniques:

Prompt engineering
Retrieval-Augmented Generation (RAG)
Supervised fine-tuning

They work. But they all share the same limitation:

They modify a model after its core intelligence is already formed. And that’s the real bottleneck.

The Problem with “Late-Stage” Customization
By the time you fine-tune a model, its:

Representations are already shaped
Internal reasoning patterns are already formed
Safety alignment is already baked in
Generalization boundaries are already defined

Fine-tuning becomes a surface-level adjustment.

Continued pre-training (CPT) on proprietary data goes deeper, but introduces another issue:

Catastrophic forgetting.
When you train only on domain-specific data, the model starts losing foundational capabilities:

Instruction following
General reasoning
Safety robustness

This is where Amazon Nova Forge fundamentally changes the game.

1) Starting From Early Checkpoints

Instead of customizing a fully trained model, Nova Forge allows organizations to start from:

Pre-training checkpoints
Mid-training checkpoints
Post-training checkpoints

This matters technically because representation learning is still malleable at earlier stages.

You’re not just adjusting weights for task behavior; you’re influencing how the model forms abstractions.

That’s a different class of customization.

2) Data Mixing as a First-Class Strategy

A major innovation is structured dataset blending.
Instead of training solely on proprietary corpora, Nova Forge blends:

Organization-specific data
Nova-curated general training datasets

Training runs on managed infrastructure through Amazon SageMaker and integrates into Amazon Bedrock for deployment.

This approach:

Preserves general intelligence
Reduces overfitting
Mitigates catastrophic forgetting
Maintains instruction-following capability

Technically, this resembles controlled continued pre-training with safety-aware balancing.

3) Reinforcement Learning in Your Own Environment
This is where it gets interesting.

Nova Forge enables reinforcement learning using:

Custom reward functions
Multi-turn rollouts
External orchestration systems
Domain-specific simulators

Instead of static supervised tuning, organizations can:

Reward accurate molecular structures
Penalize unsafe robotic behaviors
Optimize multi-step agent workflows

This moves enterprise AI closer to environment-aware, task-optimized frontier systems without training from scratch.

4) Why This Is Strategically Important

Nova Forge is not just a feature release.

It signals AWS moving beyond:

Hosting foundation models
Offering fine-tuning APIs

Toward:

Enabling organizations to co-develop frontier-level models without absorbing full pre-training costs.

That’s a big shift in the AI stack.

What This Means for Builders and DevRel
For engineers:

This reframes customization from: “Which prompt works best?” to “Where in the training lifecycle should I intervene?”

For DevRel and community leaders:
Understanding this shift matters.
Explaining:

Why catastrophic forgetting happens
Why early checkpoint intervention matters
Why RL environments change domain alignment

is the kind of depth that moves conversations beyond surface-level AI hype.

Enterprise AI is evolving from prompt engineering to model engineering.

And Nova Forge is a signal that customization is moving earlier deeper and closer to the foundation itself.