DEV Community: saurabh gupta

What Building a Fintech Ledger Taught Me About Idempotency

saurabh gupta — Sat, 19 Sep 2026 02:55:47 +0000

What Building a Fintech Ledger Taught Me About Idempotency

I built a double-entry payments ledger on AWS EKS — six FastAPI services, Terraform, an SNS/SQS async pipeline, the whole thing. Going in, I assumed the hard parts would be infrastructure: Fargate quirks, Terraform state, getting Prometheus to scrape anything.

Those were annoying. But they were findable — something crashes, you read logs, you fix it.

The genuinely hard part was a question that sounds too simple to be interesting:

What happens if the same "send money" request arrives twice?

That question turned out to have four different answers depending on where in the system you ask it, and getting it wrong in any one of those places would have silently moved someone's money twice.

The version that looks right and isn't

My first instinct was the obvious one: check whether it already happened.

existing = db.query("SELECT * FROM transactions WHERE idempotency_key = %s", key)
if existing:
    return existing
# ...otherwise process the transfer

This reads fine. It's also broken, and it's broken in the way that's hardest to catch: only under concurrency.

Between the SELECT and the INSERT there's a window. If two requests carrying the same idempotency key arrive close enough together — the original, plus a retry fired because the client's HTTP request timed out — both run that SELECT before either has committed anything. Both see nothing. Both proceed. Two debits.

And client retries aren't an edge case. They're the normal path: a mobile app on a flaky connection, a load balancer timing out, a user double-tapping "Send" because the spinner hasn't resolved. A payments system that can't survive a retry isn't handling a rare scenario badly — it's handling the common one badly.

The guarantee has to live in the database

The thing that took me embarrassingly long to internalize: no amount of application logic fixes this. Any "check, then act" sequence has a window unless something below it makes the two steps atomic.

So the actual mechanism is four words:

CREATE TABLE transactions (
    id               UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    idempotency_key  TEXT NOT NULL UNIQUE,
    ...
);

UNIQUE. That's it. That's the guarantee. Everything else in my idempotency story is an optimization sitting on top of it.

INSERT INTO transactions (idempotency_key, ...)
VALUES ($1, ...)
ON CONFLICT (idempotency_key) DO NOTHING
RETURNING id;

No row returned means someone else already claimed that key — go fetch their result and return it. A row returned means you own the work, and nobody can take it from you, because Postgres enforces that at the storage layer rather than trusting that your Python checked first.

Redis is a shortcut, not the promise

I do cache idempotency keys in ElastiCache to skip the database on hot retries. But two rules kept that cache from quietly becoming a liability:

It is never the source of truth. A cache miss, an evicted key, a Redis node dying — none of it changes correctness, because the UNIQUE constraint is still there underneath. Losing Redis costs latency, not money.
Keys are written only after the database transaction commits. Caching before commit means caching an outcome that might still roll back — a bug that would be nearly impossible to reproduce and catastrophic when it fired.

The test that actually proves it

I could have written "added a unique constraint, therefore idempotent" in the README and moved on. Plenty of projects do.

The problem: a sequential test — send the request, wait for the response, send it again — passes even against my broken check-then-insert version. There's no race, because the first INSERT has long since committed by the time the second SELECT runs. You get a green checkmark that proves nothing.

So the test fires both requests at once:

with ThreadPoolExecutor(max_workers=2) as pool:
    f1 = pool.submit(fire_transfer, same_idempotency_key)
    f2 = pool.submit(fire_transfer, same_idempotency_key)
    r1, r2 = f1.result(), f2.result()

assert r1.json()["id"] == r2.json()["id"]
entries = get_ledger_entries(r1.json()["id"])
assert len(entries) == 2   # one debit, one credit. never four.

Only the concurrent version touches the actual race window. Writing the sequential one first and feeling good about it would have been a false sense of security I'd have carried all the way to production.

The bug that taught me idempotency has a failure path too

This one I found by accident, weeks later, while adding Prometheus metrics — and it's the part I'd most want to talk through in an interview.

Failed transfers (insufficient funds, frozen account) were doing this:

with conn:                                  # commits on exit, rolls back on exception
    ...
    cur.execute("UPDATE transactions SET status = 'failed' WHERE id = %s", (txn_id,))
    raise HTTPException(422, "insufficient funds")   # <-- inside the block

Raising inside with conn: rolls back the transaction — including the "mark as failed" update I'd just written. So a failed transfer left zero trace in the database. The idempotency key was never durably claimed. Which means a retry of that request looked brand new, and got processed from scratch.

I'd built idempotency for the success path and completely forgotten the failure path. The fix was to stash the exception and raise it after the transaction block exits cleanly, so the failure status actually commits:

pending_error = None
with conn:
    ...
    if insufficient_funds:
        cur.execute("UPDATE transactions SET status = 'failed' ...")
        pending_error = HTTPException(422, "insufficient funds")
# block exited normally -> the 'failed' row is committed
if pending_error:
    raise pending_error

The lesson generalizes past this bug: "what happens if this arrives twice" has to be answered for every outcome, not just the happy one. A rejected request is still a request that can be retried.

Idempotency at the seam between two systems

Once transfers were safe, I hit the same problem one layer out. A completed transfer needs to trigger fraud scoring and a notification. The obvious implementation:

db.commit()                    # write the ledger entries
sns.publish(event)             # tell everyone about it

That's a dual write — two independent systems, two independent failure modes. If the commit succeeds and the publish fails, the money moved and nothing downstream ever hears about it. Reverse the order and you announce a transfer that then fails to commit. There is no ordering of those two lines that is safe, because they can't be one atomic operation.

The fix is the Outbox Pattern, and it's the design decision I'm happiest with in the whole project. The transfer writes an outbox_events row in the same database transaction as the ledger entries:

BEGIN;
  INSERT INTO ledger_entries ...;   -- debit + credit
  INSERT INTO outbox_events  ...;   -- "this happened"
COMMIT;

Now the event's existence is exactly as reliable as the money movement — same transaction, same atomicity, no gap. A separate poller is the only thing in the entire system that talks to SNS:

SELECT * FROM outbox_events WHERE published = false
ORDER BY created_at LIMIT 20
FOR UPDATE SKIP LOCKED;

SKIP LOCKED is what makes that poller safe to run as multiple replicas with zero coordination logic — each instance grabs rows nobody else currently holds. It runs as one replica today, but the scaling path needs no redesign.

And then the duplicates come back

Here's the part that ties it all together: SQS is at-least-once. A consumer will occasionally see the same message twice — a visibility timeout expiring mid-processing, a redrive, an ack that didn't land.

So the exact question from the very beginning shows up again, in a completely different place, for completely different reasons. "What if this arrives twice" isn't a property of your HTTP API. It's a property of every boundary in a distributed system, and each one needs its own answer.

Where the discipline paid off

The payoff wasn't during normal operation. It was during the failure testing.

When I deliberately drained both Fargate nodes running a service back-to-back, and when I exercised an RDS Multi-AZ failover, requests in flight at the wrong moment failed — connections dropped, queries timed out. In most systems that's genuinely alarming: did the transfer happen? Is it safe to retry? Could I double-charge someone?

Here the answer was boring, which is the highest compliment you can pay a failure mode. Retry with the same idempotency key. If it committed before the disruption, the retry finds the existing row and returns it. If it didn't, the retry processes it fresh. Exactly once, either way.

Same story with the poison-message test: I deliberately pushed a malformed event into the fraud queue and watched it fail, retry, fail again, and land in the dead-letter queue after real SQS retries — while valid messages behind it kept processing normally. No head-of-line blocking, nothing silently lost.

That's the real return on this work. Idempotency isn't just politeness toward flaky mobile clients. It's what converts "we had a database failover" from a forensic investigation into what state the system might be in into a thing that happened for ninety seconds and then stopped happening. You answer the duplicate question once, up front, instead of re-litigating it during every incident for the rest of the system's life.

The takeaway

If I compressed this whole project into one paragraph for someone building something similar:

Put the correctness guarantee in a database constraint, not in application logic. Write a test that genuinely races it, because a sequential test will lie to you. Answer the duplicate question for failures too, not just successes. And at every boundary between two systems, ask it again — because the answer doesn't carry over.

The Kubernetes manifests, the Terraform modules, the Grafana dashboards — that's all real work, and I learned a lot building it. But it's work you can course-correct on later. A ledger that silently double-debits someone under concurrent retries is not something you course-correct on later.

You get that right at the schema level, or you don't get it right at all.

FinLedger is open source: github.com/saurabhg4356/finledger — including the full design doc, the chaos-engineering runbooks, and a written list of the nine production bugs I hit and diagnosed along the way.

ShopSphere — A Cloud-Native E-Commerce Platform

saurabh gupta — Mon, 03 Aug 2026 12:11:15 +0000

8 Things That Broke When I Deployed Kubernetes on AWS EKS Fargate (And How I Fixed Them)

Building a production-shaped microservices platform from scratch — what the tutorials don't cover.

I spent 8 weeks building ShopSphere: a cloud-native e-commerce backend with 3 FastAPI microservices, deployed on Amazon EKS, monitored with Prometheus and Grafana, secured with GuardDuty and AWS Secrets Manager, and delivered by a GitHub Actions CI/CD pipeline.

The stack: Python 3.13 + FastAPI → Docker → ECR → EKS Fargate → RDS PostgreSQL → ALB → Terraform IaC → GitHub Actions.

Everything I've written below is real. I didn't learn it from a tutorial — I learned it because it broke at 11pm and I had to figure out why.

Why EKS Fargate (and the EC2 quota wall)

The original plan was a standard EKS cluster with managed EC2 node groups. That plan died immediately.

New AWS accounts start with an EC2 vCPU quota of 0 for several instance families. Both On-Demand and Spot were blocked. Requesting quota increases takes days and isn't guaranteed. The project couldn't wait.

Fargate is the alternative: AWS runs each pod on its own dedicated microVM. No EC2 Auto Scaling Groups, no node management, no quota to hit. You pay per pod-second of CPU and memory rather than per instance.

I migrated the cluster to Fargate. This was the right call. But Fargate has its own set of constraints that are scattered across AWS documentation, GitHub issues, and Stack Overflow threads — never in one place. Here's what I hit.

Problem 1: CoreDNS silently failing to schedule

Symptom: After applying the Fargate profile and deploying, service name resolution failed cluster-wide. The ALB controller couldn't start. External Secrets Operator couldn't start. Prometheus couldn't scrape anything. Everything that needed to reach some-service.namespace.svc.cluster.local just timed out.

Root cause: Fargate uses a mutating admission webhook to intercept pod creation and inject Fargate-specific configuration. For a pod to be scheduled on Fargate, the webhook needs to process it. For the webhook to process it, the pod needs the annotation eks.amazonaws.com/compute-type: fargate.

CoreDNS's default Kubernetes deployment doesn't have this annotation. So Fargate's webhook ignores CoreDNS pods, they never get scheduled, and they sit Pending indefinitely. Since CoreDNS is the cluster's DNS resolver, everything else fails.

Fix:
bash kubectl patch deployment coredns -n kube-system \ --type=json \ -p='[{"op":"add","path":"/spec/template/metadata/annotations/eks.amazonaws.com~1compute-type","value":"fargate"}]' kubectl rollout restart deployment/coredns -n kube-system

This is documented in AWS's EKS + Fargate guide but easy to miss when you're following a general EKS tutorial.

Problem 2: Prometheus, AlertManager, Grafana all refuse to start

Symptom: helm install monitoring prometheus-community/kube-prometheus-stack completes without error. But all pods are stuck with Pod not supported on Fargate: volumes not supported.

Root cause: kube-prometheus-stack's default Helm values request PersistentVolumeClaims backed by EBS storage for Prometheus (metrics storage), AlertManager (alert state), and Grafana (dashboard state). Fargate pods cannot mount EBS volumes. At all. It's not a configuration issue — it's a fundamental architectural constraint.

Fix: Switch to ephemeral in-pod storage for the entire monitoring stack:

`yaml

prometheus-values.yaml

prometheus:
prometheusSpec:
storageSpec: {} # no PVC — ephemeral storage
retention: 6h # short retention; this is dev, not production

alertmanager:
alertmanagerSpec:
storage: {} # no PVC

grafana:
persistence:
enabled: false # no PVC
sidecar:
dashboards:
enabled: true # load dashboards from ConfigMaps instead
`

The monitoring stack becomes stateless by design. Prometheus re-scrapes from pod startup on restart. Grafana dashboards live in ConfigMaps — code-defined, version-controlled, no state to lose.

Problem 3: External Secrets Operator webhook port collision

Symptom: ESO installs successfully. The SecretStore applies without error. But ExternalSecret objects never sync — they stay in a permanent pending state. ESO pods show TLS errors in their logs: x509: certificate is valid for [...], not for [fargate-node-ip].

Root cause: ESO's admission webhook runs on port 10250 by default. On Fargate, every pod runs in its own microVM that has its own kubelet — also on port 10250. When ESO registers its webhook with the Kubernetes API server, and the API server tries to call the webhook to validate ExternalSecret objects, it connects to what it thinks is the ESO webhook address but is actually the Fargate node's kubelet on that port. The TLS certificate ESO presents doesn't include the Fargate node's internal address in its SANs — hence the mismatch.

This affects several Kubernetes webhook-based operators on Fargate: cert-manager, ADOT, and ESO all have open issues for this.

Fix:
`yaml

In the ESO Helm values

webhook:
port: 9443 # anything other than 10250
`

Problem 4: Fargate profile updates stranding pods permanently Pending

Symptom: After adding a new namespace to the Fargate profile, some pods in existing namespaces get stuck Pending and never schedule, despite the profile update completing successfully.

Root cause: Fargate profiles are immutable — adding a namespace selector requires destroying and recreating the profile. AWS does this automatically during an update, but there's a brief window (a few seconds to a minute) where no Fargate profile is active. Any pod that gets created during this window goes through normal Kubernetes scheduling. The default scheduler tries to find an EC2 node — there are none. The pod sits Pending. When the new profile comes back, it only evaluates pods at creation time, not retroactively. Pods already stuck Pending with the wrong scheduler state never get reconsidered for Fargate.

Fix: After any Fargate profile update, find and delete all Pending pods so they get recreated and scheduled correctly:

`bash
kubectl get pods --all-namespaces | grep Pending

For each stuck pod:

kubectl delete pod -n
`

Automate this if you're doing frequent profile updates during setup.

Problem 5: IRSA roles referenced but not created

Symptom: Pods start successfully. Health checks pass. But any AWS API call — reading from Secrets Manager, listing ECR images — fails with AccessDenied. The pod appears healthy but is silently broken.

Root cause: I added IRSA annotations to the Kubernetes ServiceAccounts:

yaml annotations: eks.amazonaws.com/role-arn: "arn:aws:iam::123456789012:role/shopsphere-user-service-role"

But I hadn't actually created that IAM role in Terraform yet. The annotation references a role that doesn't exist. Kubernetes applies the ServiceAccount fine. The pod starts fine. The EKS pod identity webhook injects the AWS_ROLE_ARN and AWS_WEB_IDENTITY_TOKEN_FILE environment variables correctly. The AWS SDK tries to assume the role — and gets AccessDenied because the role doesn't exist.

This is completely invisible until you actually make an AWS API call from inside the pod.

Fix: For each service, add to Terraform:

hcl resource "aws_iam_role" "user_service" { name = "shopsphere-user-service-role" assume_role_policy = jsonencode({ Version = "2012-10-17" Statement = [{ Effect = "Allow" Principal = { Federated = "arn:aws:iam::${var.aws_account_id}:oidc-provider/${local.oidc_provider}" } Action = "sts:AssumeRoleWithWebIdentity" Condition = { StringEquals = { "${local.oidc_provider}:sub" = "system:serviceaccount:shopsphere:user-service-sa" } } }] }) }

Every Kubernetes ServiceAccount that needs AWS permissions needs a corresponding IAM role and OIDC trust policy in Terraform. No exceptions.

Problem 6: read-only root filesystem breaking container startup

Symptom: After adding readOnlyRootFilesystem: true and capabilities: drop: [ALL] to the pod security context, pods fail to start with permission errors.

Root cause: The original Dockerfile used a CMD pattern that:

chown-ed the application directory at container start
Used su or gosu to drop from root to the app user

Both require either write access to the filesystem (for chown) or Linux capabilities that we just dropped (for su/gosu). With readOnlyRootFilesystem: true and no capabilities, the container can't start.

Fix: Move all ownership-setting to Dockerfile build time:

`dockerfile

In the runtime stage, before switching to appuser:

RUN adduser --disabled-password --no-create-home appuser && \
chown -R appuser:appuser /app

USER appuser

CMD just starts the app — no chown, no su needed

CMD ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8001"]
`

For volumes that need write access at runtime (temp files, SQLite), use emptyDir volumes in the pod spec and configure fsGroup in the security context — Kubernetes handles the ownership:

`yaml
securityContext:
fsGroup: 1000 # Kubernetes chowns volume mounts to this GID at pod start
volumes:

name: tmp emptyDir: {} `

Problem 7: Diagnosing an account-level ELB restriction

Symptom: Creating a LoadBalancer-type Service fails. The AWS Load Balancer Controller logs show OperationNotPermitted. IAM permissions look correct. Quotas look fine. No relevant errors in CloudTrail except the failure itself.

Root cause: Some AWS accounts have account-level API restrictions applied that are distinct from IAM permissions and service quotas. These appear in the raw API response but not in the kubectl error summary. The pattern: OperationNotPermitted rather than AccessDenied or LimitExceeded.

This required opening an AWS Support case. It's not a configuration mistake. Attempting to work around it through configuration changes wastes time.

Workaround while the Support case resolves: kubectl port-forward for Grafana and AlertManager access. This is actually more secure — no public LoadBalancer for the monitoring stack.

Lesson: Read the full raw API error response, not just the kubectl summary. OperationNotPermitted and AccessDenied have different root causes and require different responses. Knowing which one you're looking at tells you whether to keep debugging configuration or open a Support case.

Problem 8: Fargate profile namespace selectors and webhook timing

Symptom: After applying a new Fargate profile that should cover a new namespace, pods in that namespace still don't schedule. They show 0/0 nodes available.

Root cause: A Fargate profile only schedules pods that match its namespace + label selectors, but the profile has to exist before the pod is created. If you apply the profile and then immediately apply the namespace and deployment in the same kubectl apply -f, there's a race — the pods may be created before the profile is fully active.

Fix: Add a sleep or confirmation step:

`bash

Wait for the Fargate profile to be ACTIVE before deploying

aws eks wait fargate-profile-active \
--cluster-name shopsphere-cluster \
--fargate-profile-name shopsphere-fp

Then deploy

kubectl apply -f k8s/base/
`

What the finished system looks like

After solving all of the above:

git push origin main → 4 GitHub Actions jobs (test → scan → approve → deploy) → new version live in EKS with zero manual steps
curl http://ALB_URL/health → 200 from all 3 services
Grafana shows p95 latency, request rate, error rate, and pod health in real time
AlertManager sent a real email when I deliberately triggered an error spike, and a resolution email when I fixed it
kubectl get externalsecret -n shopsphere → SecretSynced — DB credentials come from Secrets Manager, not a YAML file
kubectl get networkpolicies -n shopsphere → 5 policies, default-deny enforced
GuardDuty and CloudTrail monitoring the account

The project is at github.com/saurabhg4356/shopsphere with full source, architecture diagram, and setup instructions.

What I'd tell someone starting this today

Set up a AWS budget alert before your first terraform apply. The NAT Gateway is always running and costs money even when nothing is happening.
Read the Fargate-specific documentation, not just the EKS documentation. The two have meaningfully different constraints and the Fargate docs are more scattered.
When something fails, read the full raw API error. kubectl describe gives summaries. The raw API response — in CloudTrail, in pod events, in controller logs — tells you the actual error code, which often tells you exactly what category of problem you're dealing with.
Terraform modules from the start. I built a flat structure first and refactored it into modules. It's much easier to start modular than to extract modules from a flat structure later.
The hard problems aren't the code. FastAPI is simple. Docker is simple. Kubernetes YAML is tedious but learnable. The genuinely hard part is the system-level debugging — when six different components interact and only one of them is wrong, and that one has a misleading error message.
`