DEV Community: Aliasgar

The Difference Between Retry and Idempotency They're Not the Same

Aliasgar — Mon, 06 Jul 2026 01:56:57 +0000

Let me start with the confusion that prompted this post: in a recent mentoring session, a senior engineer described their payment service as "idempotent" because it had exponential backoff on retries. That's a category error and it's the kind of mistake that bites teams in production with real financial consequences.

Retry and idempotency are complementary, not synonymous.

Two Different Actors, Two Different Concerns

Retry logic lives in the client. It's the client's decision to resend a request after a failure or timeout.

Idempotency lives in the server. It's the server's guarantee that receiving the same request multiple times will have the same effect as receiving it once.

These are independent properties. Here's what each combination looks like:

Retries without server idempotency: every retry is a new operation. Network timeout on a payment? Retry charges the customer again. This is the double-charge bug.

Server idempotency without client retries: the server is prepared for duplicates, but the client gives up after one timeout. The payment succeeded, the UI says error. Support tickets follow.

Neither: timeouts result in unknown state, retries cause duplicates, correctness depends on luck.

Both, correctly implemented: the client retries safely, the server deduplicates, the user sees a correct result. This is the design you want.

What Good Retry Logic Looks Like

int attempt = 0;
while (attempt < MAX_ATTEMPTS) {
    try {
        return paymentClient.submit(request, idempotencyKey);
    } catch (TimeoutException | ServiceUnavailableException e) {
        long delay = BASE_DELAY_MS * (1L << attempt) + random.nextInt(JITTER_MS);
        Thread.sleep(delay);
        attempt++;
    }
}
throw new PaymentSubmissionFailedException("Max retries exceeded");

The critical detail: the same idempotency key on every retry attempt. If you're generating a new idempotency key on each retry, you've broken the connection between retry and idempotency. Each retry looks like a new request to the server.

What Good Server-Side Idempotency Looks Like

public PaymentResponse processPayment(String idempotencyKey, PaymentRequest request) {
    try (Lock lock = distributedLock.acquire(idempotencyKey)) {
        Optional<PaymentResponse> stored = idempotencyStore.get(idempotencyKey);
        if (stored.isPresent()) {
            return stored.get();
        }
        PaymentResponse response = paymentGateway.charge(request);
        idempotencyStore.save(idempotencyKey, response, TTL_24H);
        return response;
    }
}

Two non-obvious requirements here:

Distributed lock: without a lock, two concurrent requests with the same key can both pass the "not found" check and both execute the payment.

Durable storage: the idempotency store must survive restarts. A cache with eviction can lose a key making a stored payment look new on the next request.

The Race Condition Most Teams Miss

Without a distributed lock:

Request A arrives. Store check: not found.
Request B arrives (duplicate, concurrent). Store check: not found.
Request A processes payment. Stores result.
Request B processes payment again. Overwrites result.

Customer charged twice. The lock closes this window.

The Atomic Write Problem

What if the server processes the payment but crashes before storing the idempotency key? The client retries. The server has no record of the first request. It processes again.

The cleanest solution is the transactional outbox pattern: write the payment result and the idempotency record in the same database transaction, then publish events asynchronously. It's more complex but eliminates this failure mode entirely.

The Takeaway

Retry is client resilience. Idempotency is server correctness. You need both, explicitly connected via the idempotency key passed on every retry.

When reviewing payment API designs, I check three things: Is the client sending the same key on retries? Is the server storing results durably with a lock? Is the store-and-execute step atomic? If any are missing, the system has a correctness gap that will manifest as duplicate charges under load.

If you found this useful, I run 1:1 mentoring sessions for Java/backend engineers at topmate.io/aliasgar_kantawala

My Java interview guides and system design resources are at aliasgarmk.gumroad.com

Idempotency in Payment APIs — Not Optional, Not Later

Aliasgar — Thu, 02 Jul 2026 20:07:39 +0000

There's a version of this story that plays out at almost every fintech startup I've worked with: the team ships a payment API, it works in testing, it goes live. Then a user's connection drops mid-request. The mobile app retries. The backend processes it twice. Two charges hit the customer's account. And suddenly you're in an incident with your compliance team asking uncomfortable questions.

Idempotency is the only thing standing between you and that story. After 11 years building payment systems — including integrations with instant payment rails, card processors, and core banking platforms — I have a strong opinion: if your payment API doesn't implement idempotency from day one, you've already shipped a bug. You just haven't seen it detonate yet.

What Idempotency Actually Means

An idempotent operation produces the same result no matter how many times you apply it. In the context of a REST API: if the client sends the same request multiple times, the server executes it exactly once and returns the same response every time.

HTTP GET and DELETE are idempotent by definition. POST is not — unless you design it to be. Payment submission is POST. So you have to build idempotency yourself.

The standard mechanism: the client generates a unique idempotency key (a UUID) and includes it in the request header. The server stores the key alongside the resulting response. On subsequent requests with the same key, the server returns the stored response without reprocessing.

Why Networks Make This Non-Negotiable

Mobile clients, third-party integrators, internal services — they all operate over networks that can fail. A timeout doesn't tell the client whether the request was received. The connection might have dropped before the request arrived. Or after processing. Or mid-response.

The only safe behavior for a client facing a timeout on a payment request is to retry. Which means your server must be able to receive the same payment request twice and process it exactly once. Without idempotency, you're putting the burden of deduplication on the client — which is both incorrect and unenforceable.

At scale, this gets worse. Under load, timeouts become more frequent. Retries increase. Without idempotency, your duplication rate scales with your traffic.

Getting the Key Scope Right

Here's where teams commonly go wrong: scoping the idempotency key to the request body hash.

My recommendation: require the client to provide the idempotency key explicitly — a UUID they generate before the request. This makes deduplication intent explicit and removes ambiguity. Stripe does this. Braintree does this. There's a reason.

Pair the key with the user ID and operation type in your storage key. A key generated by user A should never accidentally deduplicate a request from user B with the same UUID.

Idempotency Is Not Caching

This distinction matters more than it sounds. Caching is about performance — a cache miss is acceptable. Idempotency is about correctness — a "miss" on your idempotency store could result in a duplicate charge.

Your idempotency store needs:

Durability: if your service restarts, you cannot lose stored idempotency records
Atomicity: storing the key and executing the payment must be atomic, or a concurrent duplicate can slip through
TTL policy: 24 hours is a common baseline

A Redis cache with default eviction policy fails at least two of these.

The Implementation Skeleton (Spring Boot)

@PostMapping("/payments")
public ResponseEntity<PaymentResponse> createPayment(
        @RequestHeader("Idempotency-Key") String idempotencyKey,
        @RequestBody PaymentRequest request) {

    Optional<PaymentResponse> cached = idempotencyStore.get(idempotencyKey);
    if (cached.isPresent()) {
        return ResponseEntity.ok(cached.get());
    }

    // Distributed lock on the key closes the race window
    try (Lock lock = distributedLock.acquire(idempotencyKey)) {
        // Re-check after acquiring lock
        cached = idempotencyStore.get(idempotencyKey);
        if (cached.isPresent()) return ResponseEntity.ok(cached.get());

        PaymentResponse response = paymentService.process(request);
        idempotencyStore.save(idempotencyKey, response, Duration.ofHours(24));
        return ResponseEntity.ok(response);
    }
}

The lock on the idempotency key is critical — it closes the race window between the initial check and the store. Without it, two concurrent requests with the same key can both pass the check and both execute.

The Takeaway

Idempotency is a correctness requirement for any API that mutates financial state. The clients calling your API will retry. Networks will drop packets. Your job is to make that safe.

Design it in from the start. It's an order of magnitude easier than retrofitting it after your first duplicate-charge incident.

Eventual Consistency Is Misunderstood by Most Engineers

Aliasgar — Mon, 29 Jun 2026 21:08:06 +0000

I've interviewed dozens of engineers at the senior level. When I ask them to explain eventual consistency, most recite the textbook answer: "data will converge to a consistent state — eventually." What they can't tell me is what that looks like when something goes wrong at 2 AM with real money on the line.

After 11 years building distributed systems I've come to believe that eventual consistency is the most confidently misunderstood concept in backend engineering.

What the Textbook Gets Right (and What It Skips)

The CAP theorem tells you a distributed system can guarantee at most two of three properties: Consistency, Availability, and Partition tolerance. Under network partitions — not exceptional events, they're normal — you choose between C and A.

Eventual consistency is the CP-to-AP trade. You relax the guarantee that every read reflects the most recent write, in exchange for continued availability. The system promises that, absent further writes, all replicas will converge.

What the textbook skips: convergence requires you to design it. It doesn't happen by default. If two nodes accept conflicting writes, someone has to decide which one wins — and that decision has to be domain-correct, not just technically convenient.

A Real Reconciliation Bug

Here's the scenario that taught me to respect this deeply: two services consumed the same payment event stream. Service A updated the payment status. Service B, running slightly behind, read the old status — decided the payment was still pending — and updated it back.

Both services behaved correctly in isolation. Both passed unit tests. The bug was emergent — a classic read-modify-write race under eventual consistency. The result was a payment that appeared confirmed in one view and pending in another. The reconciliation file we sent to Banco Central at end of day was wrong.

Debugging it required correlating log timestamps across three services and figuring out that a 200ms replication lag was enough to cause the race. That's not a code bug. That's an architectural assumption that never got written down.

"Eventually" Has No Default SLA

When someone says "the system is eventually consistent," that "eventually" could mean 50 milliseconds or 50 minutes depending on replication topology, network conditions, and write volume.

In OLTP systems under load, I've seen replication lag spike from under a second to several minutes. If your business logic assumes convergence within a heartbeat, you have an implicit SLA you've never declared — and it will fail in production at the worst possible moment.

The moment I started treating eventual consistency seriously was when I started asking: what is the maximum staleness this system can tolerate? If the answer is "I don't know," that's a design gap, not a performance tuning problem.

Conflict Resolution Is a Domain Problem, Not a Technical One

Most engineers reach for "last write wins" as their default conflict resolution strategy. It's simple, easy to implement, and almost always wrong in financial systems.

Consider two concurrent updates to a payment record: one marks it confirmed, the other marks it as flagged for review. Last write wins gives you a coin flip on which state survives. That's not a distributed systems problem — it's a correctness problem with business consequences.

The right approach is to model your domain state as a CRDT-compatible structure or to use explicit versioning and reject conflicting writes at the application layer.

When to Choose Strong Consistency

Strong consistency is expensive in latency. It requires coordination across replicas before returning a response.

But there are domains where the cost of a wrong read exceeds any latency penalty you'd pay for coordination. Financial ledgers are the clearest example. If reading an account balance returns a stale value, you may approve a transaction that should be rejected. The cost — chargebacks, regulatory exposure, fraud liability — dwarfs any latency improvement.

My rule: if a stale read can trigger a downstream action with irreversible consequences, use strong consistency and optimize later. You can always relax constraints. You can't always reverse a bad payment.

What Good Eventual Consistency Looks Like

When eventual consistency is the right choice, the design still requires explicit work:

Idempotent writes: every write operation must be safe to retry without side effects
Version vectors or timestamps: to detect conflicts when they occur
Compensating transactions: when eventual convergence reveals a conflict, you need a path to correct it
Monitoring on replication lag: treat lag as a business metric, not just an infrastructure metric

None of this is accidental. It's deliberate design that most systems skip because engineers assume convergence is automatic.

The Takeaway

Eventual consistency isn't a shortcut to availability — it's a contract you make with your users about the accuracy of their data. In payments and financial systems, that contract has teeth. Before you choose it, know your conflict resolution model, know your staleness tolerance, and know what happens when convergence fails.

The engineers who internalize this ship more reliable systems. The ones who don't end up explaining reconciliation mismatches to compliance teams on Friday afternoons.

How to Design an Event-Driven Analytics Pipeline — A System Design Deep Dive

Aliasgar — Thu, 25 Jun 2026 22:33:19 +0000

Interview prep series: real problems, real trade-offs, no hand-waving.

The Problem Worth Solving

Here's a scenario straight from a fintech startup I worked at.

We had field agents responsible for onboarding customers, managing investments, and handling deposits and withdrawals. Management wanted to shift from a fixed salary model to fixed + commission — where commission depended on each agent's monthly performance.

The immediate instinct was: just write a SQL query. And that's exactly what we did, initially. But it quickly fell apart:

The query joined across 7+ tables representing different workflow states
It had to be re-run every time someone asked "how is agent X doing?"
Any change to the pay structure meant rewriting the query
It didn't scale — every CXO-level ad hoc report became a fire drill

The underlying issue wasn't the SQL. It was that the system was storing events but not state — and retrieving current state required replaying all those events manually every single time.

This is the problem the design solves.

Defining the Requirements

Before jumping to architecture, get the requirements sharp.

Functional:

Track every meaningful user action across all front-ends (web app, customer app, agent app)
Generate scheduled reports (daily customer activity, monthly agent performance)
Support on-demand report generation via API
Archive old data cost-effectively

Non-Functional:

The analytics pipeline must not slow down the core application
Reports can tolerate some delay — this is not a real-time dashboard
The system should be pluggable — drop it into any service without re-architecting that service
Data older than 12 months should move to cheaper storage automatically

Notice what's not in the requirements: low latency reads, strong consistency, sub-second aggregations. This is an eventually consistent, batch-oriented analytics system. That shapes every decision downstream.

Estimation

Quick back-of-envelope to understand the data shape:

Assume ~50 agent actions per agent per day, ~200 agents → ~10,000 events/day
Each event ~2KB (enriched JSON with context) → ~20MB/day at raw staging
Monthly reports cover ~300,000 events → manageable for Lambda batch processing
After 12 months: ~7GB of raw event data → S3 archival is cost-effective at this scale

At this scale, you don't need a distributed streaming platform. Kafka is present here, but for a different reason — we'll get to that.

The Architecture: Three-Part System

The design splits into three loosely coupled parts. This is the plug-and-play property: each part can fail or be replaced without breaking the others.

Part 1 — Event Capture and Staging

Every front-end (web, mobile, agent) fires events on meaningful user actions. The back-end receives these events, enriches them based on event type (adds agent ID, customer context, timestamp, session metadata), and writes them to the existing RDBMS.

Why RDBMS at this stage? Because each event is:

Written once — no updates, no reads until the cron picks it up
Structured — enrichment at write time means the schema is predictable
Transactional — you want writes to succeed or fail atomically

A cron job runs at a configured interval, reads the unprocessed events, copies them to AWS S3, and on success marks them for deletion. A second pass deletes the marked records. This two-step (mark → delete) pattern is intentional — if the S3 write fails mid-batch, nothing gets deleted. You get at-least-once delivery to S3, which is fine for analytics.

Data lifecycle: After 12 months in S3, events are moved to local NAS (or cold storage like S3 Glacier). This keeps active storage costs low without destroying the audit trail.

[Web App / Mobile / Agent App]
        |
        ▼
[Back-end Service] → enriches event → [RDBMS Staging Table]
                                              |
                                        [Cron Job]
                                              |
                                        [AWS S3 Bucket]
                                              |
                              (after 12 months) → [NAS / Cold Storage]

Key interview insight: The cron-based approach is deliberately simple. In a higher-volume system you'd replace it with a CDC (Change Data Capture) mechanism using Debezium or similar — but at this scale, simplicity wins.

Part 2 — S3 Processing and Report Generation

Once data lands in S3, report generation becomes a data transformation problem. Python scripts and AWS Lambda read from S3, apply aggregations, and write results (CSV, JSON) back to a designated S3 output directory.

Lambda works well here because:

Reports are batch jobs — they run on a schedule, not continuously
Individual report jobs are stateless and bounded in time
Lambda scales to run multiple reports in parallel without managing servers

Generated reports are stored in S3 and a notification (email with a download link) is sent to the requester. The report is never pushed directly — only a link. This decouples generation latency from delivery.

Part 3 — The Controller (On-Demand API + Scheduling)

This is the operability layer — and it's often missing from system design answers.

The controller is a small Java module that:

Schedules when to move data from staging to S3 (configurable intervals)
Schedules when to trigger which reports (daily, weekly, monthly)
Exposes a REST API for on-demand report generation

The on-demand API is fire-and-forget. The client calls POST /reports/generate with parameters, gets back a 202 Accepted immediately. The request is pushed to a Kafka topic. A consumer picks it up, processes the report, writes to S3, and emails the link.

Why Kafka here, not a simple task queue or direct Lambda invocation?

Durability: Kafka is log-based — messages are persisted and replayable if the consumer crashes mid-processing
Backpressure: If report demand spikes, Kafka buffers the requests; consumers process at their own pace
Replayability: If a report generation bug is found, you can replay the topic to regenerate affected reports

[Client] → POST /reports/generate
              |
        [REST Controller] → 202 Accepted
              |
        [Kafka Topic: report-requests]
              |
        [Report Consumer]
              |
        [S3 Output] → [Email Notification]

Trade-Off Discussion

These are the questions an interviewer is likely to probe:

Why not stream events directly to S3 instead of staging in RDBMS first?

You could use Kinesis Firehose or Kafka directly. But at this scale, staging in RDBMS first keeps the architecture within the team's existing operational expertise. The cron approach is easy to monitor, debug, and replay if something goes wrong. Streaming adds operational overhead that isn't justified until volume demands it.

Why not use a dedicated analytics DB (Redshift, BigQuery) instead of S3 + Lambda?

For ad hoc SQL queries and dashboards, a warehouse would be better. But the requirement here is structured reports on a schedule, not exploratory analytics. S3 + Lambda is cheaper, has no idle cost, and fits the access pattern exactly.

What happens if the cron fails halfway through a batch?

The mark-then-delete pattern handles this. Undeleted records will be picked up in the next cron run and re-written to S3 (idempotent if keyed properly). At-least-once delivery is acceptable for analytics.

What if report volume grows 100x?

Replace the cron with CDC + Kafka Streams for near-real-time event flow. Move report generation from Lambda to a Spark or Flink job on EMR for large aggregations. The S3-centric design makes this migration possible without changing the data model.

What This Design Gets Right

Three things worth remembering from this design:

Decoupling at every boundary. The front-ends don't know about S3. The cron doesn't know about report formats. The controller doesn't know about Lambda. Each part can be replaced independently.

Matching storage to access pattern. RDBMS for write-once structured events. S3 for batch reads and cost-effective archival. Kafka for durable async processing. No single store is doing all three jobs.

Operability is part of the design. The controller module — configurable schedules, on-demand API, email notifications — is what turns a pipeline into a product. Systems that work but can't be operated in production are incomplete designs.

Interview Cheat Sheet

Question	Answer
Why stage in RDBMS before S3?	Write-once, structured, transactional — fits the access pattern; simpler than streaming at this scale
Why Kafka for on-demand reports?	Durable, replayable, handles backpressure — fire-and-forget API needs async processing guarantee
Why Lambda for report generation?	Stateless batch jobs, pay-per-use, no idle cost — matches the workload perfectly
How do you handle cron failure?	Mark-for-deletion before delete — unprocessed records are re-picked on next run
How do you scale this?	CDC + Kafka Streams replaces cron; Spark/Flink replaces Lambda at high volume
What's the data lifecycle?	Hot (RDBMS staging) → Warm (S3, 12 months) → Cold (NAS/Glacier)

Closing Thought

The best system design answers don't start with "let's use Kafka and microservices." They start with the problem, reason through the constraints, and arrive at the simplest architecture that meets the requirements — with a clear articulation of what you'd change if those constraints shifted.

This pipeline is deliberately simple. And that's the point.

Found this useful? I write about distributed systems, fintech engineering, and senior-level interview prep at aliasgarmk.gumroad.com. Drop a reaction if you'd like more designs in this series.

Why Daily Transaction Volume is the Wrong Number to Quote in System Design Interviews

Aliasgar — Mon, 22 Jun 2026 23:14:14 +0000

The Mistake Most Candidates Make
Picture this: You are in a system design interview. The interviewer asks you to design a ledger system or a payment settlement pipeline. Eager to show you can handle scale, you confidently state, "We'll design this to handle 100K transactions a day."

The interviewer nods. But that number just told them very little about whether you can actually engineer the system.

Leading with total daily volume is the most common trap candidates fall into. It sounds like a big, impressive number. But in reality, quoting a daily aggregate proves you are looking at the system through a marketing lens, not an engineering one.

What Actually Matters: The Shape of the Workload
Systems do not experience load as a perfectly smooth, distributed stream over 24 hours. They experience load in waves, spikes, and violent bursts. To design a resilient architecture, the raw daily number is useless. You need three specific dimensions:

Peak TPS (Transactions Per Second): The absolute highest throughput the system must sustain during peak traffic.

Workload Shape: How traffic distributes over time — business hours concentration, sudden marketing flashes, or scheduled batch windows.

Burst Duration: How long that peak sustained load lasts. Is it a 5-second spike or a 45-minute sustained mountain?

Let's do the math that most engineers skip in an interview.

If you take 100K transactions and spread them evenly across 24 hours, you get roughly 1 TPS — one transaction per second. A modest, single-core database instance running on a laptop can handle that in its sleep. You don't need a distributed system for 1 TPS; you barely need a framework.

But real-world systems don't work that way.

A Real-World Case Study: The Deceptive Average
I ran into this exact problem while building an end-of-day (EOD) reconciliation service for a major instant payment rail — one that processes tens of thousands of financial transactions daily and must verify every single one against an external banking ledger within a strict compliance window.

On paper, the system processed around 100K transactions a day. Reasonable. Manageable. Nothing alarming.

But the shape of the workload told a completely different story.

Daytime traffic trickled in slowly. Then, as merchants settled their books, payroll batches triggered, and consumers made last-minute transfers, load climbed sharply toward end of day. And at the reconciliation window itself — where the external bank flushed the final ledger data in a concentrated burst — everything had to be processed before the next business day opened.

When we actually measured the workload shape during that final window, the naive average and the engineering reality had almost nothing in common:

Naive average: ~1 TPS, spread across 24 hours

Actual peak: 22 TPS, concentrated in a 45-minute EOD window

That is a 22x gap between the number you'd quote in a planning meeting and the number your system actually has to survive.

Why This Changes Every Engineering Decision
If you design for 1 TPS average, you might reasonably ask: do we even need a message queue here? A simple cron job running a sequential loop would work.

But when you design for a 22 TPS burst with heavy downstream dependencies, the entire architectural conversation shifts.

Thread pool sizing — your reconciliation worker can no longer be a single-threaded job. You have to reason about concurrency, resource contention, and memory footprints before writing a single line of code.

Idempotency is non-negotiable — at burst load, network hiccups happen and retries are guaranteed. If your database upsert logic isn't strictly idempotent, you will be reconciling duplicates at 3 AM wondering why the ledgers don't balance.

Downstream capacity limits — the external banking APIs you are calling have their own strict rate limits. Your 22 TPS ambition is irrelevant if the counterparty throttles you at 5 TPS. This forces you to design token-bucket rate limiters and robust client-side queuing — and to invest in building an in-house downstream simulator that covers all failure paths, not just the happy path.

Pragmatic architecture over greenfield hype — when faced with a compressed burst window and complex recovery paths, a trendy event-driven microservices approach might introduce massive operational complexity. If the external bank operates on a secure, file-based exchange model, a robust file-chunking batch architecture might actually be the more defensible, resilient choice for Phase 1. It's boring, but it's correct.

The Questions an Interviewer Actually Wants You to Ask
Whenever an interviewer hands you a vague requirement like "Design a system that handles X hundred thousand requests a day," don't immediately start drawing boxes on the whiteboard.

Stop. And ask the single question that separates senior practitioners from theorists:

"What does the peak traffic look like, and over what specific window does it occur?"

From there, you want to understand whether specific business events — market close, midnight clearing, payroll batches — concentrate the load into a predictable window, and what the latency and rate limits of your downstream dependencies look like precisely during that window.

When you ask these questions, the interviewer knows you have actually operated systems at scale. You are no longer guessing; you are gathering the constraints required to build a real system.

Closing Thought
Daily transaction volume is a marketing number.

Peak TPS over a compressed burst window is an engineering number.

The next time you are in the hot seat, ignore the aggregate daily volume. Figure out the shape of the peak, calculate the true strain on the system, and design for that instead.

If this kind of thinking is what you want to sharpen before your next system design round, I work with senior engineers on exactly this — translating real production experience into interview-ready depth. You can find me at topmate.io/aliasgar_kantawala.