DEV Community: Nazmul Ahasan

Binance Hack DevSecOps Lessons: The Chain Did What It Was Told

Nazmul Ahasan — Wed, 04 Nov 2026 19:03:41 +0000

On October 29, 2026, Binance lost about $6.3 billion. Nobody broke the cryptography, and no consensus bug was involved. The blockchain processed signed transactions exactly as it was designed to. As far as the chain could tell, everything was in order.

That is the part I keep returning to. The chain did what it was told. Everything that failed happened off-chain: people, laptops, credentials, deploy pipelines, and the policy around signing. Those are platform and DevSecOps problems. Most of us run systems shaped like that, whether or not crypto is involved.

This post covers what is publicly known, why a valid signature tells you nothing about intent, and the controls I think platform teams should be building. It ends with a checklist.

1. What happened

Here is the public record so far. I've stuck to points confirmed by the sources linked below.

The size. Binance put the loss at about $6.3 billion, per Reuters. Chainalysis has a slightly higher estimate of about $6.4 billion in its latest findings.

How they got in. Binance's early finding pointed at one engineer's machine. Its November 3 report replaced that story with a broader one. As CoinDesk reports, the campaign was multi-stage and began in June. It used a trojanized tool download and a fake job interview. Two wallet developers' machines were compromised. The attackers then used stolen credentials to deploy a container that ran a malicious script. This was not one laptop and not one engineer's permissions. It was a patient campaign that moved through several layers.

The signatures were valid. The transfers carried valid signatures (CoinDesk, Chainalysis), and Binance says it has found no evidence that a private key leaked. Cointelegraph's coverage frames it the same way: the keys were misused, not broken.

The mechanism. The attacker relied on EIP-7702 delegation. A hot wallet labelled "Binance 14" received 50 authorizations that delegated it to a sweeper contract, according to CoinDesk's breakdown and Blockaid's analysis. EIP-7702 lets an externally owned account point at contract code. Once that delegation exists, the account behaves the way the delegated code says. One permission ended up controlling a lot of money.

The freezes. Stablecoin issuers moved fast. In early reporting, CNBC said Tether had frozen more than $1.5 billion and Circle more than $800 million. Since then, Blockaid and Binance have put the total frozen across issuers at about $3 billion.

Attribution. In an October 31 alert, the FBI attributed the attack to North Korea's TraderTraitor, also known as Lazarus. Chainalysis agrees with that attribution. See also coverage from AP and The Record. Fake job interviews and trojanized developer tools are standard tradecraft for that group, so the entry path fits.

2. Why valid signatures aren't valid intent

A signature proves one thing: whoever controls the key, or a system authorized to use it, approved these bytes. It doesn't tell you whether a human meant to approve them, whether the business would have approved them, or whether the code that requested the signature was the code you shipped.

Most signing setups assume that anything able to reach the signer is trustworthy. That assumption holds only while the path to the signer stays clean. In this incident the path ran through a developer endpoint, then stolen credentials, then a container deploy, then a script, then the signer. All of those links are off-chain, and every one of them is something a platform team owns.

I think of it like this. Cryptography authenticates. It doesn't authorize. Authorization lives in policy, and policy has to sit somewhere the attacker can't simply deploy around. If your only gate is "does this request come from a workload with access to the signer," the attacker's job becomes "get a workload with access to the signer." In practice that means stealing deploy credentials.

EIP-7702 makes the point sharper. A delegation authorization is a signed message that can hand ongoing control of an account to contract code. If your policy treats it like any other low-value message, you have left a door open that is worth far more than any single transfer.

3. Practical controls for platform teams

None of what follows is new. That's my point. The incident used familiar paths, so the defenses are familiar too. What matters is whether they are actually enforced at the layers that count.

Endpoint hardening and EDR for engineers

Engineers' machines are production-adjacent. They hold SSH keys, cloud sessions, kubeconfigs, and repository access. Treat them that way:

Run EDR on every engineering endpoint and actually staff the alert triage.
Block or sandbox unsigned binaries and tools from unknown sources. A "tool download" sent during a "job interview" should never run on a machine that holds production credentials.
Separate the device you browse and take calls on from the one you use for privileged access. Privileged access workstations, or at minimum isolated browser profiles and VMs, shrink the blast radius of a single click.
Watch for credential-store access and odd token use on developer machines.

Phishing-resistant MFA

TOTP codes and push approvals get phished and fatigued. Use FIDO2/WebAuthn hardware keys or platform passkeys for SSO, cloud consoles, CI/CD, and anything else that can deploy. Bind sessions to the device where you can.

Short-lived credentials

Stolen credentials only matter while they work. Swap long-lived keys for short-lived ones issued through OIDC federation, so CI gets cloud roles by workload identity, humans get just-in-time elevation, and tokens expire in minutes, not months. Static deploy credentials sitting on a developer laptop are an incident that hasn't been triggered yet.

A signing-service policy engine

This is the control I would put first for anyone holding keys. The signer should never be a dumb oracle. Put a policy engine in front of it that evaluates every request against limits and context before any key material is touched. It should run separately from the workloads that call it, with its own deploy path and its own approvers.

Pseudo policy:

signing_policy:
  default: deny

  rules:
    - name: hot-wallet-withdrawals
      match:
        wallet_tier: hot
        tx_type: transfer
      require:
        destination_in_allowlist: true
        amount_usd_max_per_tx: 250000
        velocity:
          window: 1h
          amount_usd_max: 2000000
          tx_count_max: 200
        caller_workload_identity_in: [withdrawal-service-prod]
        caller_image_digest_in: approved_release_digests
      on_violation: block_and_page

    - name: delegation-and-permission-changes
      match:
        tx_type: [eip7702_authorization, approve, set_owner, upgrade]
      require:
        human_approvals: 2          # distinct people, distinct devices
        approver_auth: webauthn
        delegate_target_in_allowlist: true
        change_ticket_linked: true
      on_violation: block_and_page

    - name: anomaly-circuit-breaker
      match:
        any: true
      when:
        - outflow_usd_15m > 3 * baseline_p99
        - new_destinations_15m > 20
      action: halt_hot_wallet_signing

Three ideas carry the weight. Default deny. Velocity limits that stop a drain partway even when each individual request looks fine. Treating permission and delegation changes, such as EIP-7702 authorizations, as high-risk operations that need multiple humans, never as routine traffic.

Least privilege and admission control for container deploys

In this incident, stolen credentials were enough to deploy a container running a malicious script. Admission control is where you make "can authenticate to the cluster" different from "can run arbitrary code next to the signer."

Here is a Kyverno policy that only admits signed images from your registry into the signing namespace:

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: signer-namespace-signed-images-only
spec:
  validationFailureAction: Enforce
  background: false
  rules:
    - name: verify-image-signature
      match:
        any:
          - resources:
              kinds: [Pod]
              namespaces: [wallet-signing]
      verifyImages:
        - imageReferences:
            - "registry.example.com/wallet/*"
          mutateDigest: true
          verifyDigest: true
          attestors:
            - entries:
                - keyless:
                    issuer: "https://token.actions.githubusercontent.com"
                    subject: "https://github.com/example-org/wallet-service/.github/workflows/release.yml@refs/heads/main"
    - name: block-other-registries
      match:
        any:
          - resources:
              kinds: [Pod]
              namespaces: [wallet-signing]
      validate:
        message: "Only images from registry.example.com/wallet are allowed."
        pattern:
          spec:
            containers:
              - image: "registry.example.com/wallet/*"

This ties what runs to an image built by the release pipeline from main. A human holding stolen kubeconfig can no longer push an ad-hoc image. Pair it with RBAC under which no human identity can create pods in that namespace directly, and with a ban on kubectl exec into signer workloads.

Blast-radius segmentation

Keep hot wallets small. Put signing infrastructure in its own cluster or account with its own identity provider boundaries. Don't let the network path from general workloads to the signer exist at all. Cap hot wallet balances so that the worst case is a bad day and not an existential one. Rebalancing from cold storage should be slow, deliberate, and reviewed by humans.

Detection on anomalous deploys and withdrawals

Your detections should cover both sides of the off-chain path:

Deploys: a new image digest in a sensitive namespace, a deploy from outside the pipeline, a deploy by a human identity, a deploy at an unusual hour, or a new service account token mounted somewhere unexpected.
Withdrawals: outflow above baseline, a spike in new destinations, any delegation or approval transaction, and patterns that look like a sweep.

Wire these to automatic circuit breakers, not only pages. A detection that waits for someone to wake up is too slow when money moves in minutes.

4. Checklist

[ ] EDR on every engineering endpoint, with alerts triaged by a named owner
[ ] Policy against running unvetted tools on machines holding production credentials
[ ] Privileged access separated from everyday browsing and calls
[ ] Phishing-resistant MFA (FIDO2/passkeys) for SSO, cloud, CI/CD, and cluster access
[ ] No long-lived deploy credentials; OIDC federation and just-in-time elevation
[ ] Signing service behind a default-deny policy engine
[ ] Per-transaction limits, destination allowlists, and velocity caps
[ ] Delegation and permission changes (including EIP-7702) require multi-party approval
[ ] Admission control enforcing signed images from the release pipeline in signer namespaces
[ ] No direct human pod creation or exec in signer namespaces
[ ] Signing infrastructure segmented in its own cluster or account
[ ] Hot wallet balances capped; cold-storage refills reviewed by humans
[ ] Detections for out-of-pipeline deploys and new digests in sensitive namespaces
[ ] Detections for withdrawal anomalies, with automatic signing halt
[ ] Incident runbook covering stablecoin issuer and partner freeze contacts
[ ] Hiring and interview processes covered in security awareness (fake recruiters are a known vector)

Closing thoughts

I don't read this incident as a story about blockchains failing. The chain was the one component that behaved exactly as specified. It's a story about trust that leaked from a developer's machine into credentials, from credentials into a deploy, and from a deploy into a signer that didn't ask enough questions. Those layers belong to platform engineering. If you own them, the job is to make sure that something being signed is never enough on its own to show that it was intended.

I wrote shorter takes on LinkedIn and X. More of my writing is on my site.

I lead platform engineering for fintech. See more of my work or how we can work together at Nazmul Ahasan.

Sources

Forward Deployed Engineer challenge: I built a company ERP portal in 2 days with AI

Nazmul Ahasan — Wed, 21 Oct 2026 19:09:43 +0000

Forward Deployed Engineer challenge: I built a company ERP portal in 2 days with AI

A company posted a Forward Deployed Engineer (FDE) job ad with a take-home brief: build an internal operations portal in three working days. I took the challenge and shipped it alone in two, with AI doing much of the typing. The company took the ad down and adopted the portal. Today about 700 employees use it.

I'm Nazmul Ahasan. I run Platform Engineering as a Senior Engineering Manager, and I write about this kind of work on my site, Nazmul Ahasan. To be clear up front: I didn't hold the FDE title. I took the FDE challenge. This post covers what I built, how I'd use AI on a build like this, what you must check by hand, and the part I think matters most: the approval engine.

What the Forward Deployed Engineer challenge asked for

The brief described an internal ops/ERP portal. I called it OpsFlow (the full write-up is in the OpsFlow case study). The scope I delivered:

Multi-step approvals, driven by config rather than hard-coded flows
Leave requests: employee → line manager → HR → automatic email → leave balance deducted
Salary certificates
Payroll
Advance salary

The stack was boring on purpose: Next.js, Prisma and PostgreSQL, running on AWS EC2. Boring is what you want when you have two days. Every tool in that list is one I could debug without a search engine, which matters when the AI gets something confidently wrong.

That's the real point of an FDE-style brief, I think. It isn't asking whether you can write code. It's asking whether you can sit with a messy business problem, pick a shape for it fast, and put something real in front of users. AI changes how fast you can write code. It doesn't change who is responsible for the result.

How I'd use AI on a build like this

I used AI coding assistants throughout. The way I think about them: a fast, tireless junior engineer with zero context about the business and a habit of sounding certain. That framing decides everything else.

Where AI earns its place:

Scaffolding. Next.js pages, forms, table views, Prisma model boilerplate, API route shells. High volume, low risk, and where most of two days would otherwise go.
Repetitive variations. Once one approval flow works, the next ones are mostly "same pattern, different fields and steps". AI is very good at following a pattern you've already set.
Rubber-ducking the design. Talking through the data model before writing it surfaces awkward cases early, like what happens when the approver is also the requester.

The prompting workflow I'd insist on:

Design first, by hand. Write the data model and the workflow config format yourself. Everything else depends on them, and they're small enough to think through properly.
Give the model the contract. Paste the schema and types into the prompt and ask for code that fits them, instead of asking for "an approval system" and getting a generic one.
Small, reviewable chunks. One route, one component, one function at a time. If you can't review a diff in a couple of minutes, the request was too big.
Never accept code you can't explain. If the output uses something you don't understand, learn it or rewrite it.

The trap with AI-assisted speed is that you can generate code faster than you can understand it. The fix isn't to slow the AI down. It's to keep the parts that need real thought small, and keep them in your own hands.

The parts you must check by hand

Some code is cheap to get wrong. A misaligned table is a bug report. A wrong payroll calculation breaks trust with hundreds of people. Split the codebase along that line. This is the checklist I'd hold any AI-assisted internal tool to:

Auth and RBAC. Every server action and API route that changes state must check who the user is and what role they hold, on the server. AI-generated code tends to put the check in the UI (hide the button) and forget the endpoint. Read every mutation path for that.

Payroll and salary math. Money logic gets read line by line: rounding, deductions, advance salary recovery, partial months. Check worked examples against what HR already calculates, not against what the model says is correct.

Migrations. Prisma makes schema changes easy, which is exactly why they need reading. Every generated migration gets a human look before it touches a real database, especially anything that drops or renames a column.

Approval edge cases. Double clicks. Two approvers acting at once. An approver acting on a step that has already moved on. A requester approving their own request. A rejected request being approved later. These are the bugs that make people stop trusting a system, and AI-generated happy-path code misses them.

Testing. The highest-value tests are on the state-transition function, because every workflow goes through it. After that, walk each flow end to end as each role.

The approval engine design

This is the part I'd defend in a design review. Leave, salary certificates and advance salary all have the same shape: someone asks, a sequence of people approve or reject, and when it's done, something happens. So I built one engine and made each flow a piece of config.

The model has four parts:

ApprovalWorkflow: a named workflow, such as "leave".
ApprovalStep: an ordered step in a workflow, with the role that's allowed to act on it.
ApprovalRequest: one instance, such as one employee's leave request, with its current step and status.
ApprovalAction: an append-only record of who did what, and when.

Prisma schema sketch

This is a sanitized sketch, not the production schema.

enum RequestStatus {
  PENDING
  APPROVED
  REJECTED
  CANCELLED
}

enum ActionType {
  APPROVE
  REJECT
}

model ApprovalWorkflow {
  id        String            @id @default(cuid())
  key       String            @unique // e.g. "leave"
  name      String
  steps     ApprovalStep[]
  requests  ApprovalRequest[]
  createdAt DateTime          @default(now())
}

model ApprovalStep {
  id         String           @id @default(cuid())
  workflowId String
  workflow   ApprovalWorkflow @relation(fields: [workflowId], references: [id])
  order      Int
  key        String           // e.g. "line_manager"
  role       String           // role allowed to act on this step

  @@unique([workflowId, order])
}

model ApprovalRequest {
  id          String           @id @default(cuid())
  workflowId  String
  workflow    ApprovalWorkflow @relation(fields: [workflowId], references: [id])
  requesterId String
  currentStep Int              @default(0)
  status      RequestStatus    @default(PENDING)
  payload     Json             // flow-specific data, e.g. leave dates
  version     Int              @default(0) // optimistic concurrency
  actions     ApprovalAction[]
  createdAt   DateTime         @default(now())
  updatedAt   DateTime         @updatedAt
}

model ApprovalAction {
  id             String          @id @default(cuid())
  requestId      String
  request        ApprovalRequest @relation(fields: [requestId], references: [id])
  stepOrder      Int
  actorId        String
  type           ActionType
  comment        String?
  idempotencyKey String          @unique
  createdAt      DateTime        @default(now())

  @@unique([requestId, stepOrder]) // one decision per step
}

Two constraints do a lot of quiet work here. idempotencyKey is unique, so a retried request can't record a second action. And @@unique([requestId, stepOrder]) means a step can only be decided once, whatever the application code does.

A config-driven workflow definition

The steps live in the database so they're inspectable, but the behaviour lives in typed config. Adding a workflow means adding a definition, not writing a new state machine.

type Role = "EMPLOYEE" | "LINE_MANAGER" | "HR" | "FINANCE";

interface StepDef {
  key: string;
  role: Role;
}

interface CompletionContext<P> {
  tx: TransactionClient; // Prisma transaction client
  requestId: string;
  requesterId: string;
  payload: P;
  enqueueEmail: (msg: { to: string; template: string; data: unknown }) => Promise<void>;
}

interface WorkflowDef<P> {
  key: string;
  name: string;
  steps: StepDef[];
  validate: (payload: unknown) => P;
  onComplete?: (ctx: CompletionContext<P>) => Promise<void>;
}

interface LeavePayload {
  leaveType: string;
  startDate: string;
  endDate: string;
  days: number;
}

export const leaveWorkflow: WorkflowDef<LeavePayload> = {
  key: "leave",
  name: "Leave request",
  steps: [
    { key: "line_manager", role: "LINE_MANAGER" },
    { key: "hr", role: "HR" },
  ],
  validate: (input) => parseLeavePayload(input), // throws on bad input
  async onComplete({ tx, requesterId, payload, enqueueEmail }) {
    await tx.leaveBalance.update({
      where: {
        employeeId_leaveType: { employeeId: requesterId, leaveType: payload.leaveType },
      },
      data: { remaining: { decrement: payload.days } },
    });
    await enqueueEmail({
      to: requesterId,
      template: "leave-approved",
      data: { startDate: payload.startDate, endDate: payload.endDate },
    });
  },
};

The employee is the implicit first step: creating the request is their action. After that it's line manager, then HR, then the completion hook sends the email and deducts the balance.

This is the way I'd design it: the balance deduction runs inside the same transaction as the final approval, so if the deduction fails, the approval doesn't happen. A request marked "approved" with an untouched balance is worse than a clear error. And the email should be queued rather than sent inline, because you can't roll back an email. That's the design that holds up.

The state-transition function

Every approve and reject goes through one function. Role check, step check, idempotency and concurrency all live here, inside a transaction.

import { Prisma, PrismaClient } from "@prisma/client";

const prisma = new PrismaClient();

type Decision = "APPROVE" | "REJECT";

export async function decide(params: {
  requestId: string;
  actor: { id: string; roles: Role[] };
  decision: Decision;
  idempotencyKey: string;
  comment?: string;
}) {
  const { requestId, actor, decision, idempotencyKey, comment } = params;

  return prisma.$transaction(
    async (tx) => {
      // 1. Idempotency: same key, same answer.
      const existing = await tx.approvalAction.findUnique({ where: { idempotencyKey } });
      if (existing) {
        return tx.approvalRequest.findUniqueOrThrow({ where: { id: requestId } });
      }

      const request = await tx.approvalRequest.findUniqueOrThrow({
        where: { id: requestId },
        include: { workflow: true },
      });
      if (request.status !== "PENDING") {
        throw new Error("Request is no longer pending");
      }

      const def = getWorkflowDef(request.workflow.key);
      const step = def.steps[request.currentStep];
      if (!step) throw new Error("No active step");

      // 2. Role and separation-of-duties checks, on the server.
      if (!actor.roles.includes(step.role)) {
        throw new Error("Not allowed to act on this step");
      }
      if (actor.id === request.requesterId) {
        throw new Error("Cannot approve your own request");
      }

      // 3. Record the decision. Unique constraints back this up.
      await tx.approvalAction.create({
        data: {
          requestId,
          stepOrder: request.currentStep,
          actorId: actor.id,
          type: decision,
          comment,
          idempotencyKey,
        },
      });

      const isLastStep = request.currentStep === def.steps.length - 1;
      const next =
        decision === "REJECT"
          ? { status: "REJECTED" as const }
          : isLastStep
            ? { status: "APPROVED" as const }
            : { currentStep: request.currentStep + 1 };

      // 4. Optimistic concurrency: only update the version we read.
      const updated = await tx.approvalRequest.updateMany({
        where: { id: requestId, version: request.version },
        data: { ...next, version: { increment: 1 } },
      });
      if (updated.count !== 1) {
        throw new Error("Request changed, please retry");
      }

      // 5. Side effects run in the same transaction.
      if (decision === "APPROVE" && isLastStep && def.onComplete) {
        await def.onComplete({
          tx,
          requestId,
          requesterId: request.requesterId,
          payload: def.validate(request.payload),
          enqueueEmail: (msg) => enqueueEmailInTx(tx, msg),
        });
      }

      return tx.approvalRequest.findUniqueOrThrow({ where: { id: requestId } });
    },
    { isolationLevel: Prisma.TransactionIsolationLevel.Serializable },
  );
}

A few choices worth calling out:

Belt and braces. The idempotency lookup handles the polite retry. The unique constraints handle the impolite race. The version check catches two approvers acting on the same step at once.
Separation of duties is explicit. "Can't approve your own request" is one line. It's also exactly the line AI-generated code tends to leave out.
Audit by construction. ApprovalAction is append-only, so "who approved this and when" is a query, not an investigation.
One place to test. Leave, salary certificates and advance salary all go through decide. Get it right once and every flow benefits.

Ideas for hardening it further down the line: delegation when an approver is on leave, escalation for stuck requests, and versioning workflow definitions so in-flight requests keep the rules they started with.

Deploying it

The portal runs on AWS EC2 with PostgreSQL behind it. For a short build, my bias is the simplest deploy that's repeatable, and here's what I'd insist on: one build command, migrations applied as an explicit step and not on app boot, a database backup before any migration, TLS in front, and a process manager that restarts the app if it dies. Platform engineering habits don't switch off because the deadline is short. If anything, they matter more.

What I took from it

AI made the two-day timeline possible. It didn't make the product trustworthy. The things that made people willing to use it, with the right people approving, correct balances and correct money, came from deciding which parts got careful human attention.

If you're looking at a Forward Deployed Engineer role, or any role where you build directly for the business, this is the skill I'd practise. Not prompting. Judgement about where AI output is good enough and where it has to be proven.

The company took the job ad down and adopted the portal. About 700 people use it. That's the outcome I care about more than the two days.

I lead platform engineering for fintech. Read the full case study here or see how we can work together at Nazmul Ahasan.

Next.js basePath 404 Nginx: Running Two Next.js Apps on One Domain

Nazmul Ahasan — Wed, 21 Oct 2026 19:09:39 +0000

Next.js basePath 404 Nginx: Running Two Next.js Apps on One Domain

If you've ever hit a Next.js basePath 404 behind Nginx, you already know the feeling: the HTML loads, the page looks half-alive, and every script and stylesheet comes back 404. I ran into that, and three other problems, while getting two Next.js apps to live under one domain behind a single Nginx server. This post covers all four, what was actually happening in each case, and the config I'd use now.

Some context. I'm Nazmul Ahasan, a Senior Engineering Manager in Platform Engineering. I write about this kind of thing on my site, Nazmul Ahasan. At FundedNext I picked up the "Help Me Choose My Plan" AI wizard when it was an MVP about 10% done and took it to about 90%. It's live in production now, and roughly $1M a month in plan purchases go through it. That's why I care about the plumbing. When that much checkout traffic goes through a page, a broken asset path is not a cosmetic bug.

The wizard is one of two Next.js apps behind one Nginx domain. Everything below uses generic examples: example.com, an app at /app-a, the wizard at /wizard, and the loopback address 127.0.0.1, and upstream ports 3000 and 3001. Those are placeholders, not our real setup.

The setup: two Next.js apps, one Nginx

The shape is simple:

https://example.com/app-a/... → Next.js app A on 127.0.0.1:3000 (example port)
https://example.com/wizard/... → the wizard on 127.0.0.1:3001 (example port)

Both apps are built and run on their own. Nginx owns the domain and routes by path prefix. On paper it's a ten-line config. In practice, every one of my four war stories comes from the gap between "Nginx routes by prefix" and "Next.js assumes it owns the root."

War story 1: the Next.js basePath 404 behind Nginx

What's actually going on

By default a Next.js app thinks it lives at /. Its HTML points at assets like /_next/static/chunks/main-abc123.js. Put two apps on one domain and both of them want /_next/static. Nginx can only send that path to one upstream, so the other app's assets 404. Or worse, they get answered by the wrong app.

basePath fixes this by making Next.js own its prefix completely. Pages, links, and static assets all move under it, so the wizard's assets become /wizard/_next/static/... and stop colliding with app A.

I hit a basePath 404 on the wizard. The symptom of this class of bug is consistent: the HTML loads, then requests under /_next/static fail, so the page shows up unstyled and dead. The usual culprit is one of three things. basePath isn't set, basePath doesn't match the Nginx location, or Nginx strips the prefix before the request reaches the app. The check I'd run first is the Network tab. Take the exact URL of a failing chunk and ask whether it carries the prefix the app was built with.

The next.config.js

// next.config.js for the wizard app (example values)
/** @type {import('next').NextConfig} */
const nextConfig = {
  // Every route and every /_next asset now lives under /wizard.
  basePath: '/wizard',

  // Leave assetPrefix alone unless static assets come from a different
  // origin (a CDN, say). basePath already prefixes /_next/static.
  // assetPrefix: 'https://cdn.example.com/wizard',
};

module.exports = nextConfig;

App A gets basePath: '/app-a' the same way.

Two things trip people up here. First, basePath is applied at build time. Change it and you have to rebuild; restarting the server isn't enough. Second, basePath and assetPrefix are different jobs. basePath moves the whole app. assetPrefix only changes where the browser fetches /_next/static from. I've seen people set both to /wizard and wonder why asset URLs come out wrong. If you serve everything from the same origin, basePath alone is enough.

Also, hardcoded paths don't get rewritten for you. next/link and next/image respect basePath. A raw <img src="/logo.png"> or fetch('/api/plans') does not. That's a quieter version of the same 404.

The Nginx side

This is where the classic bug lives:

# WRONG: the trailing slash on proxy_pass strips /wizard/ from the request
location /wizard/ {
    proxy_pass http://127.0.0.1:3001/;
}

When proxy_pass includes a URI (even just /), Nginx replaces the matched location prefix with it. A request for /wizard/_next/static/chunks/x.js reaches Next.js as /_next/static/chunks/x.js. But the app was built with basePath: '/wizard' and expects the prefix, so it returns 404. Leave the URI off and Nginx forwards the path untouched:

# Example config: example.com, ports 3000/3001 are placeholders
upstream app_a  { server 127.0.0.1:3000; }
upstream wizard { server 127.0.0.1:3001; }

server {
    listen 443 ssl http2;
    server_name example.com;

    proxy_http_version 1.1;
    proxy_set_header Host              $host;
    proxy_set_header X-Real-IP         $remote_addr;
    proxy_set_header X-Forwarded-For   $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;

    # --- Wizard (basePath: '/wizard') ---
    # Hashed build assets: safe to cache forever.
    location /wizard/_next/static/ {
        proxy_pass http://wizard;          # no URI: keep /wizard in the path
        proxy_hide_header Cache-Control;
        add_header Cache-Control "public, max-age=31536000, immutable" always;
    }

    location = /wizard { return 301 /wizard/; }

    location /wizard/ {
        proxy_pass http://wizard;
        proxy_hide_header Cache-Control;
        add_header Cache-Control "no-store" always;
    }

    # --- App A (basePath: '/app-a') ---
    location /app-a/_next/static/ {
        proxy_pass http://app_a;
        proxy_hide_header Cache-Control;
        add_header Cache-Control "public, max-age=31536000, immutable" always;
    }

    location = /app-a { return 301 /app-a/; }

    location /app-a/ {
        proxy_pass http://app_a;
        proxy_hide_header Cache-Control;
        add_header Cache-Control "no-store" always;
    }
}

The rule I follow now: the path Nginx forwards must still contain the basePath. If you really want Nginx to strip the prefix, then don't use basePath. Pick one place to own the prefix, not both.

One caveat on the /_next/static blocks. Next.js already sends immutable for those files in production, so the explicit header is belt and braces. I keep it so the intent is visible in Nginx.

War story 2: Next.js hydration not working in production

What a corrupt build looks like

This one is nastier than a 404 because the page looks fine. Server-rendered HTML arrives, the layout is right, the text is right. Then nothing responds. Buttons and inputs do nothing. The client never took over.

Hydration is React attaching event handlers and state to HTML the server already rendered. To do that, the browser needs the exact JavaScript chunks that build produced. If the HTML comes from one build and the chunks come from another, or the chunks on disk are incomplete, hydration fails. Common ways to get there:

A build interrupted mid-write, leaving .next partly stale.
An incremental build reusing a cached .next that doesn't match the current source or dependencies.
Rolling or multi-instance deploys where instances run different builds, so HTML from instance one asks for chunks that only instance two has.
Stale HTML cached somewhere pointing at chunks that no longer exist (more on that in story 3).

I hit this with a corrupt build. The usual culprit is a stale or partial .next directory, and the typical first symptom is a page that renders but ignores every click.

How to spot it

Open the console first. Look for ChunkLoadError, Loading chunk ... failed, 404s on /_next/static/chunks/*.js in the Network tab, or React hydration errors such as "Hydration failed because the initial UI does not match what was rendered on the server." A page that renders but has no interactivity and no errors is rare. Usually something is shouting.
Compare build IDs. Every Next.js build gets an ID, written to .next/BUILD_ID on the server. With the Pages Router you can read the one the HTML was rendered with from the page:

   // In the browser console (Pages Router)
   window.__NEXT_DATA__.buildId

Then on each server instance:

   cat .next/BUILD_ID

If they differ, or differ between instances behind the same upstream, you've found your mismatch. With the App Router there's no __NEXT_DATA__. Compare the hashed chunk URLs in the HTML source against what's actually in .next/static on the server.

Do a clean rebuild. Don't trust the cache when you're debugging this:

   rm -rf .next
   npm ci
   npm run build

If a clean build fixes it, the problem was the build artifact, not your code.

What fixes it, nearly every time, is the clean rebuild.

My opinion: make clean builds the default in CI, and make sure every instance serving a basePath runs the same BUILD_ID. Build caching speeds things up until it hands you a production incident that costs more than all the minutes it ever saved.

War story 3: HTML cached for a year

I hit this one too: HTML getting cached for a year. Here's how it usually happens. Someone writes one broad caching rule like "cache everything under this prefix for a year", because the static assets deserve it. But HTML also lives under that prefix. Now the browser, or any cache in between, holds a year-old HTML document. That document points at hashed chunk filenames from a build that's long gone.

Hashed assets and HTML have opposite caching needs:

/_next/static/* filenames contain a content hash. A new build means new filenames, so the old file never changes. max-age=31536000, immutable is correct.
HTML has a stable URL (/wizard/) but its contents change with every deploy. Cache it for a year and you pin users to an old build. That old build's chunks may 404 after a deploy, and you're back in story 2: a page that renders and doesn't hydrate.

I switched the HTML to no-store. no-cache is also valid. It lets caches store the response but forces revalidation every time. For a checkout-adjacent flow I'd rather pay for a fresh render than reason about revalidation behavior across every proxy in the path.

The usual culprit is a single Nginx location or CDN rule that applies a long max-age to a whole path prefix. The check I'd run first is curl -I on the HTML URL itself, not on an asset.

Next.js headers() vs Nginx

You might reach for headers() in next.config.js:

// next.config.js — fine for most custom headers
module.exports = {
  basePath: '/wizard',
  async headers() {
    return [
      {
        source: '/:path*',
        headers: [{ key: 'X-Frame-Options', value: 'DENY' }],
      },
    ];
  },
};

That works for headers like X-Frame-Options, but be careful with Cache-Control. Next.js docs warn that Cache-Control values set this way can be overwritten in production for pages and static assets. Next manages those itself. For dynamically rendered pages Next already sends private, no-cache, no-store, max-age=0, must-revalidate. For static or ISR pages it sends s-maxage values tied to revalidation. So for a hard guarantee I enforce Cache-Control at Nginx, as in the config above: proxy_hide_header Cache-Control to drop the upstream value, then add_header ... always so it also applies to error responses.

One Nginx gotcha worth knowing: add_header directives are inherited from the parent block only if the child block defines none of its own. Add a single add_header inside a location, and every server-level add_header (security headers included) silently disappears for that location. Repeat them, or put them in an include file.

Then verify, don't assume:

curl -sI https://example.com/wizard/ | grep -i cache-control
curl -sI https://example.com/wizard/_next/static/chunks/<some-chunk>.js | grep -i cache-control

War story 4: resetting wizard state with a key

A multi-step wizard holds a lot of local state: current step, answers, derived recommendations. At some point you need a real "start over". The instinct is to write a reset() that sets every piece of state back to its initial value. That works until someone adds a new useState and forgets to add it to reset(). Now "start over" leaves one stale field behind.

React already has a cleaner tool. A component's identity is its position in the tree plus its key. Change the key and React treats it as a different component. It unmounts the old instance, throwing away all its state, effects, and refs, and mounts a fresh one with initial state everywhere.

'use client';
import { useState } from 'react';

function PlanWizard({ onRestart }) {
  const [step, setStep] = useState(0);
  const [answers, setAnswers] = useState({});
  // ...every other piece of state, no manual reset needed

  return (
    <div>
      {/* steps... */}
      <button onClick={onRestart}>Start over</button>
    </div>
  );
}

export default function WizardPage() {
  const [runId, setRunId] = useState(0);

  // Bumping the key remounts PlanWizard with completely fresh state.
  return <PlanWizard key={runId} onRestart={() => setRunId((n) => n + 1)} />;
}

This resets everything inside PlanWizard, including state you'll add next year. The trade-offs are worth knowing. Effects run cleanup and fire again on remount, so any fetch on mount happens again. State that lives outside the component (a context above it, a global store, localStorage, URL query params) is not reset by a key change. You still have to clear that yourself.

That's how I reset the wizard's state: a new key, a clean remount. The usual trigger for this pattern is an explicit "start over", or a change of context that makes the previous answers meaningless. If you use it, check whether anything outside the component also holds wizard state, and clear it in the same handler.

What I'd tell myself before starting

Running two Next.js apps behind one Nginx domain is a reasonable setup. It just punishes vague ownership. The rules I work by now:

One owner per prefix. Either basePath in Next.js or prefix rewriting in Nginx, never both. Keep proxy_pass free of a URI when using basePath.
Builds are artifacts, not suggestions. Clean builds in CI, one BUILD_ID per deployed version, and check the console before you check the code.
Cache by content type, not by prefix. Hashed assets get immutable for a year. HTML gets no-store or no-cache. Enforce it where you can actually guarantee it, and verify with curl -I.
Let React do resets. A changed key is the most complete reset you can write, because you don't write it.

None of these are exotic. The trap is that each failure shows up somewhere other than its cause.

I lead platform engineering for fintech. Read the full case study here or see how we can work together at Nazmul Ahasan.

How I Migrate DigitalOcean Spaces to S3: 12 Million Objects, Two Hops, Zero Failures

Nazmul Ahasan — Tue, 20 Oct 2026 13:13:31 +0000

How I Migrate DigitalOcean Spaces to S3: 12 Million Objects, Two Hops, Zero Failures

Recently I had to migrate DigitalOcean Spaces to S3 for a fintech platform I run infra for. The source was a Space in an EU region and the destination was S3 in an APAC region: about 2.5 TiB across about 12 million objects. The final hop finished with every object succeeded and 0 failed.

That result is clean. The route to it wasn't. This is the version of the story with the mistakes left in, because the mistakes are where the useful lessons are.

I'm Nazmul Ahasan, and I lead Platform Engineering at NEXT Ventures. I write about AWS, cost and reliability work at nazmul-ahasan.pro.bd, and the business side of this move is in the object storage migration case study.

The shape of the problem

Look at the average object size before anything else: about 220 KB. This isn't a "move a few big files" job. It's a massive-small-file workload, which means the limit is requests per second, not bandwidth. Every object costs you a GET, a PUT, and possibly a HEAD or two. Multiply that by about twelve million.

The second fact is geography. EU to APAC is roughly 250–280 ms round trip. A copy tool running anywhere along that path pays that latency on every small object. You can throw concurrency at it, but you'll hit the source's rate limits long before you beat physics.

So I didn't do a direct copy. I did two hops:

Hop 1, same metro: rclone on an EC2 instance in the same EU metro, copying DO Spaces to an S3 staging bucket in <eu-region>. Low latency on both sides.
Hop 2, server-side: S3 Batch Operations copying staging to the production bucket in the APAC region. The bytes travel over the AWS backbone and never pass through a machine I manage.

If you take one thing from this post, take that one. Do the slow, chatty part of the transfer where latency is cheap, then let AWS move the bulk between its own regions.

Hop 1: rclone, DO Spaces to S3 in the EU

Config, and my first dumb mistake

Spaces keys come from the DigitalOcean control panel under API → Spaces Keys. The secret is shown exactly once. Copy it straight into your secret store.

The working config:

[dospaces]
type = s3
provider = DigitalOcean
endpoint = <do-eu-region>.digitaloceanspaces.com
region = <do-eu-region>
access_key_id = <SPACES_KEY>
secret_access_key = <SPACES_SECRET>

[s3eu]
type = s3
provider = AWS
env_auth = true
region = <eu-region>

My first version had the Space name baked into the endpoint host, my-space.<do-eu-region>.digitaloceanspaces.com, because that's the URL you see in the DO console. rclone lsd dospaces: came back empty. No error, just nothing. The endpoint should be the regional host, and the bucket goes in the path: dospaces:my-source-space. Five minutes lost, and a reminder that a silent empty result is worse than an error.

A smaller red herring: on the AWS side the instance role was scoped to specific buckets, so rclone lsd s3eu: returned a 403 on ListAllMyBuckets. That's harmless. You don't need to list every bucket in the account. Go straight to s3eu:my-staging-bucket-eu and move on. Don't widen the policy to make a cosmetic error go away.

Tuning: SlowDown, then crawling

My first serious run fired a lot of concurrency at Spaces and got 503 SlowDown back. That's DO telling you to back off. Two things were burning requests: raw parallelism, and rclone doing a HEAD on objects it didn't need to inspect.

Once the 503s stopped it was still crawling at about 150 objects per second. The cause was modtime comparison. To compare modification times, rclone HEADs the object on both sides. Two extra round trips per object for twelve million objects is a lot of waiting. For a one-shot migration into an empty bucket, size is a perfectly good comparison.

I raised the request rate step by step, --tpslimit 300 → 600 → 800, and watched for throttling at each step. This is the command I finished with:

rclone copy dospaces:my-source-space s3eu:my-staging-bucket-eu \
  --size-only \
  --transfers=96 --checkers=32 \
  --tpslimit=800 --tpslimit-burst=100 \
  --s3-no-head-object --s3-no-head \
  --s3-upload-concurrency=2 --s3-chunk-size=5M \
  --s3-no-check-bucket \
  --retries=10 --low-level-retries=20 \
  --stats=1m --stats-one-line \
  --log-file=rclone-hop1.log -v

Why the flags matter:

--size-only removes the modtime HEADs. It's the biggest single speedup.
--s3-no-head and --s3-no-head-object skip the HEAD before and after each upload. For small files, those HEADs are a big share of the total work.
--s3-upload-concurrency=2 and --s3-chunk-size=5M fit objects of about 220 KB. Big multipart settings do nothing here except use memory.
--s3-no-check-bucket stops rclone from checking or creating the bucket over and over.
--tpslimit=800 is the number that matters. At 800 TPS I saw zero throttling.

It sustained about 400 objects per second, peaking around 145 MiB/s. Hop 1 took about a day, tuning included.

The six hours I lost

This was my most expensive mistake, and it had nothing to do with clever flags.

The instance started as a c6i.xlarge with 8 GB of RAM. rclone builds directory listings in memory, and a full listing of 12 million objects holds about 7 GB. The first real run ran out of memory and froze the box. Nothing crashed loudly. The process just stopped making progress, and I didn't notice for about six hours.

I also tried --fast-list, which is supposed to cut list calls by fetching everything in one go. On this bucket it sat idle for around 30 minutes building the listing before it copied anything. On a box that's already short of memory, that's the wrong trade.

The fixes:

Resize to an r6i.4xlarge. Memory-optimised, because the listing is the workload.
Work per top-level folder instead of the whole bucket at once.
Monitor long runs. Tail --stats-one-line output, alert if the transferred count stops moving, watch memory. A watch loop on the log would have saved me most of a working day.

My opinion: any copy job expected to run longer than lunch needs a liveness check. "It's probably still going" doesn't count as monitoring.

Verifying hop 1

A full-bucket rclone check hit the same memory wall, so I checked per folder:

rclone check dospaces:my-source-space/Invoices s3eu:my-staging-bucket-eu/Invoices \
  --one-way --size-only \
  --missing-on-dst missing.txt --differ differ.txt

--one-way asks only one question: is everything at the source present at the destination? For a migration, that's the right question.

The gaps were real, and they were all explainable. The application was still live. New uploads under the current month's <YYYY/MM> prefix kept arriving after the bulk pass had listed them. Some files had been regenerated in place with a different size. Both kinds got targeted re-copies:

rclone copy dospaces:my-source-space s3eu:my-staging-bucket-eu \
  --files-from missing.txt --no-traverse --size-only

Feeding the list in with --files-from plus --no-traverse means rclone copies exactly those keys without listing the bucket again. For a sense of scale, the top-ups were a few thousand files in total, mostly certificates and invoices, plus a handful of agreements, photos and dispute files with size differences. This bucket holds user documents like invoices and certificates. "Mostly there" isn't acceptable for data like that.

Hop 2: S3 Batch Operations to migrate DigitalOcean Spaces to S3 in APAC

Once staging matched the source, the rest was meant to be the easy part. It was, eventually.

The IAM role

Batch Operations assumes a role that trusts batchoperations.s3.amazonaws.com:

{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Principal": { "Service": "batchoperations.s3.amazonaws.com" },
    "Action": "sts:AssumeRole"
  }]
}

Permissions, split by purpose:

{
  "Version": "2012-10-17",
  "Statement": [
    { "Sid": "SourceRead", "Effect": "Allow",
      "Action": ["s3:GetObject", "s3:GetObjectTagging", "s3:GetObjectVersion",
                 "s3:ListBucket", "s3:GetBucketLocation"],
      "Resource": ["arn:aws:s3:::my-staging-bucket-eu", "arn:aws:s3:::my-staging-bucket-eu/*"] },
    { "Sid": "DestWrite", "Effect": "Allow",
      "Action": ["s3:PutObject", "s3:PutObjectTagging", "s3:PutObjectAcl",
                 "s3:GetBucketLocation", "s3:ListBucket"],
      "Resource": ["arn:aws:s3:::my-prod-bucket-apac", "arn:aws:s3:::my-prod-bucket-apac/*"] },
    { "Sid": "ManifestAndReports", "Effect": "Allow",
      "Action": ["s3:GetObject", "s3:PutObject"],
      "Resource": ["arn:aws:s3:::my-prod-bucket-apac/batch-manifests/*",
                   "arn:aws:s3:::my-prod-bucket-apac/batch-reports/*"] }
  ]
}

Four failures before one success

I tried to be clever with the generated manifest first. The console and API can build the manifest for you from filters, so you don't write a CSV yourself. That's how the day went:

Instant failure from the EligibleForReplication filter. That filter only works on versioned buckets, and staging wasn't versioned. I removed it.
Missing PutInventoryConfiguration. The auto-generated manifest uses inventory machinery under the hood, and the role needs that permission.
301 PermanentRedirect, 100% failure. An auto-manifest job runs in the source region. Writing cross-region to APAC from an EU job redirected every task.
CrossRegionOperationProhibited. When I tried to fix the region, Batch Operations made it clear it can't generate a manifest across regions.

The lesson: if source and destination are in different regions, build the manifest yourself and run the job in the destination region.

Building the manifest by hand

s5cmd is far faster than the AWS CLI at listing millions of keys. I piped its output through sed to strip it down to keys, then through a short Python step that URL-encodes each key:

s5cmd ls 's3://my-staging-bucket-eu/*' \
  | sed -E 's/^.{19}[[:space:]]+[0-9]+[[:space:]]+//' \
  | python3 -c '
import sys, urllib.parse
for line in sys.stdin:
    k = line.rstrip("\n")
    if k:
        print("my-staging-bucket-eu," + urllib.parse.quote(k, safe=""))
' > manifest.csv

wc -l manifest.csv   # should match the object count
aws s3 cp manifest.csv s3://my-prod-bucket-apac/batch-manifests/manifest.csv
aws s3api head-object --bucket my-prod-bucket-apac \
  --key batch-manifests/manifest.csv --query ETag --output text

Use safe=''. Encode everything, slashes included, and Batch decodes it correctly. Keys with spaces, plus signs, or non-ASCII characters are where hand-built manifests usually break. The file came to about 1.23 GB with one line per object, which matched the object count exactly. It went into the destination-region bucket next to the job.

The job

aws s3control create-job --region "$DST_REGION" \
  --account-id "$ACCOUNT_ID" \
  --operation '{"S3PutObjectCopy":{
      "TargetResource":"arn:aws:s3:::my-prod-bucket-apac",
      "MetadataDirective":"COPY",
      "StorageClass":"STANDARD"}}' \
  --manifest '{"Spec":{"Format":"S3BatchOperations_CSV_20180820","Fields":["Bucket","Key"]},
     "Location":{"ObjectArn":"arn:aws:s3:::my-prod-bucket-apac/batch-manifests/manifest.csv",
                 "ETag":"<manifest-etag>"}}' \
  --report '{"Bucket":"arn:aws:s3:::my-prod-bucket-apac","Prefix":"batch-reports",
     "Format":"Report_CSV_20180820","Enabled":true,"ReportScope":"FailedTasksOnly"}' \
  --priority 10 --role-arn "$BATCH_ROLE_ARN" \
  --confirmation-required --client-request-token "$(uuidgen)"

MetadataDirective: COPY keeps content types and user metadata. FailedTasksOnly keeps the report small. If it's empty, that's your proof.

The job went through the lifecycle you'd hope for. Preparing read every manifest row (about 12 million) with 0 errors. It then sat Suspended, waiting for my confirmation. I read the summary, then released it:

aws s3control update-job-status --region "$DST_REGION" \
  --account-id "$ACCOUNT_ID" --job-id "$JOB_ID" --requested-job-status Ready

Active, then Complete: every object succeeded, 0 failed. Always keep --confirmation-required on. Validating the manifest before twelve million PUTs is cheap insurance.

The mistake I'm a bit embarrassed by

Before I settled on Batch, I briefly ran an rclone S3-to-S3 copy on the EC2 box and thought it was a backbone transfer. It isn't. rclone always streams the bytes through the machine it runs on. Server-side copy within a single provider is a narrow case, and it doesn't cover cross-region at this scale with this tooling. If you want the bytes to stay inside AWS, use Batch Operations or replication, not a relay instance.

What hop 2 cost

Hop 2's bill comes down to three things: the Batch Operations job and per-object fees, about 12 million PUT requests in the destination region, and roughly 2.5 TB of inter-region transfer. All three were small. DigitalOcean egress and the EC2 time for hop 1 come on top. Staging in the EU isn't free, but it's cheap next to an engineer babysitting a latency-bound copy for a week.

Cutover and cleanup

The manifest is a snapshot. Anything written to Spaces after you list it isn't in it. So the cutover went like this:

Switch application writes to the APAC bucket, or briefly pause uploads.
Run a final delta sync of the recent prefixes directly from DO to the APAC bucket. It's small enough that latency doesn't matter. Use --checksum, because this is the last chance to catch an in-place change:

   rclone copy dospaces:my-source-space/<YYYY/MM> s3apac:my-prod-bucket-apac/<YYYY/MM> --checksum -v

Clean up: terminate the EC2 instance, put a lifecycle expiration rule on the staging bucket, delete the manifest and report prefixes, and remove the Batch role.
Rotate every credential used during the migration. Spaces keys, any access keys, all of it. Migration credentials are long-lived by accident and broad by design, so don't leave them lying around.

What I'd do differently

Start on a memory-optimised instance and work per folder from minute one.
Monitor from the first run. Six hours of a frozen box is the cost I resent most.
Use --size-only and the no-HEAD flags immediately for small files into an empty destination.
Skip the generated manifest for cross-region jobs. Build the CSV and run in the destination region.
Remember that rclone is a relay, not a backbone.

Two hops, about a day and a half including mistakes, and zero failed objects.

I lead platform engineering for fintech. Read the full case study here or see how we can work together at Nazmul Ahasan.

We almost handed out AWS keys to every team. So I built an LLM gateway instead.

Nazmul Ahasan — Thu, 16 Jul 2026 17:47:00 +0000

A few months ago the ask landed on my desk that a lot of infra people are getting right now: "The XYZ (Non Tech) teams want to build on our LLMs. Can you give them access to Bedrock?"

Simple enough on paper. AWS Bedrock was already in our account, the models were enabled, IAM can hand out permissions. Ten minutes of work.

Except it isn't ten minutes of work, and if you treat it like it is, you'll regret it in about three weeks.

The two obvious ways to do this

There are basically two, and neither is good once you think past the first user.

Option one is to hand out IAM credentials, or a role, that can call Bedrock. Fine for one person. Do it a few times, though, and you've got AWS keys living in .env files, a CI variable here, someone's Postman collection there, maybe a notebook somewhere you'll never hear about. Rotating any of it means finding every place it landed. And IAM permissions tend to be broader than you meant them to be a policy that lets someone call Bedrock usually lets them reach a few other things too, because nobody scopes these tightly on day one and then day one becomes a year ago.

Option two is one shared key behind an internal endpoint, and everyone points at that. Quicker to stand up. But now every request looks the same, so you can't tell whose code is burning what. A single bad loop can drain the month's budget before lunch and you won't know which team shipped it. And you can't cut off one person without rotating the key for all of them.

Both fall apart the moment more than two or three people are involved. And the part that stung most: when finance eventually asks why the AI line jumped, you've got nothing to show them.

This is a solved problem, mostly

There's a whole category of tools for this now people call them AI gateways, or LLM gateways. LiteLLM, Portkey, Helicone, Cloudflare's AI Gateway, and a pile of others. They all sit between your apps and the model provider and give you back the two things you want: control over who can call what, and a record of what everyone spent. Same idea as an ordinary API gateway, pointed at tokens.

I spent a while looking at the hosted ones, and for most teams I'd tell you to pick one and move on. We didn't. The reason is unglamorous. We're a fintech company, and routing prompts and completions, some of which touch data I don't want leaving our network, through somebody else's SaaS just to get a usage graph was a trade I wasn't going to make. I wanted the whole thing inside our own perimeter, talking straight to Bedrock, with the logs in a database we own. So I built a small version.

One rule: nobody gets an AWS credential

The whole design comes down to one decision, and it's a blunt one. Nobody gets AWS credentials. Not developers, not services, nothing except the gateway itself.

What people get instead is a key the gateway invents a prefixed token, llmkey_9f3c… or whatever, that AWS has never heard of. You drop it in your app, point the client at the gateway's URL instead of the Bedrock endpoint, and you're done. The only place real AWS credentials exist is the gateway process, in environment variables, and they never get copied out.

Everything else is downstream of that. Once a request shows up carrying a key you handed out, you know who's calling. And once you know who's calling, you can start refusing things.

What you can lock down, per key

A few things, and they're set per key.

You can pin the models a key is allowed to touch. Most internal tools don't need the top-tier model a ticket summarizer is perfectly happy on something small and fast, and there's no reason to let that key reach for a model that costs several times more, whether by accident or because somebody got curious one afternoon.

You can hand each key a token budget and count against it on every call. Nothing fancy, just a number and a running total. When a key's creeping toward its limit I can see it on the dashboard before it's a problem, and when it hits the ceiling it stops. That one thing removes the runaway-script scenario, which is the one that keeps you up at night.

And keys carry names and owners. So when you're staring at usage you're looking at "the internal analytics job" or an actual person, not a random string you have to go trace back at 11pm.

The request path

Here's what a call does. It's not exciting:

Request comes in with a gateway key.
Look it up. Active? Allowed to use the model it's asking for? Any budget left?
If any of those is a no, reject it right there, before we've spent a cent on AWS.
If it all checks out, forward to Bedrock with the real credentials.
When the response comes back, pull the token counts out of it, write a row to the usage log, and bump the key's running total.

Step 5 is the reason all the reporting works later. Every call becomes a row: input tokens, output tokens, latency, model, cost. Every screen in the app is just a query over that one table.

Stripped down, the check in the middle is about this simple:

def handle_request(gateway_key, model_id, prompt):
    key = keys.lookup(gateway_key)

    if not key or not key.active:
        raise Unauthorized("Invalid or disabled key")

    if model_id not in key.allowed_models:
        raise Forbidden(f"Key not permitted to use {model_id}")

    if key.tokens_used >= key.token_limit:
        raise QuotaExceeded("Token budget exhausted")

    # nothing hits AWS until we get here
    response = bedrock.invoke_model(model_id, prompt)

    usage = response.usage  # input / output token counts
    cost = price(model_id, usage.input_tokens, usage.output_tokens)

    log_usage(key.id, model_id, usage, cost)
    keys.increment_tokens(key.id, usage.total_tokens)

    return response

Working out the cost

Cost is the part people assume will be fiddly. It isn't, as long as you don't bake the prices into the code. Bedrock charges a per-1K rate for input and a different per-1K rate for output, the rates differ by model, and they change over time. So they live in settings, and the cost function just reads them:

def price(model_id, input_tokens, output_tokens):
    rate = pricing[model_id]  # per-1K rates, editable in settings
    return (input_tokens  / 1000 * rate.input_price
          + output_tokens / 1000 * rate.output_price)

Because that runs and gets stored on every request, nobody has to reconstruct spending afterward. I can pull it apart by model, by key, over whatever window. The first time I did that and found one key sitting on most of the month's cost, it turned into a five-minute chat with the team that owned it instead of a week of guessing.

Dashboards, and who's allowed to touch them

On top of that log table is the part everyone opens first: usage over time, which models are getting hammered, cost per period, the biggest keys ranked by spend. If you've seen any observability dashboard it looks like that, except the axes are tokens and money.

One thing mattered more than I expected. The people who need to look at this aren't always the people who should be able to change it. Someone on finance, or a team lead, wants to watch the numbers. They've got no reason to be creating keys or editing the price table. So there are roles: full admins who can do everything, and read-only accounts that can look and nothing else. Small feature. Saved an awkward conversation more than once.

Stuff I got wrong

A few things I didn't see coming, or saw coming and ignored:

Streaming ruins the tidy version Stream a completion and there's no neat usage object waiting up front. You either count tokens as they arrive or grab the metadata Bedrock sends at the very end. I retrofitted this and it was more painful than just doing it right would have been.
A single lifetime budget per key seems fine, and isn't It holds up until someone legitimately needs more, at which point you're editing limits by hand every few days and wishing you'd built monthly resets or a top-up path at the start.
The log table grows faster than you'd guess One row per request is nothing right up until it's millions of rows. Have a rough plan for archiving or rolling up old data before you're doing it in a hurry.
Keep cost at full precision Per-request cost is a sliver of a cent. Round on the way in and your monthly totals come out wrong by an amount that's just small enough that someone will eventually notice and stop trusting the whole thing.

None of these are hard. They're the sort of thing you only really learn by getting them a little wrong first.

Was it worth building?

For us, yeah. Not because it's clever, it isn't, it's a reverse proxy with a database bolted to it. It's worth it because of what changed day to day. Someone wants Bedrock access, I generate a key, done, no IAM ticket. No real credential has left the gateway even once. A key misbehaves, I revoke that one key and everyone else carries on.

And when AI spend comes up, which it does constantly now, I open a page instead of a spreadsheet.

The last one is the part I'd push hardest on. Every company I talk to is having some flavor of the same conversation right now: who's using these models, on what, at what cost, and whether any of it is governed. You can answer that with a policy doc that's stale the week you write it, or you can let your infrastructure answer it by itself. I know which of those stays current.

If you've just caught the "get everyone onto our LLMs" request yourself, that's the one bit of advice I'd hand you. Don't pass out keys. Stand something small up in front of Bedrock first. It's a weekend of work and it saves you from a mess that's a real pain to clean up later.

I lead platform engineering for fintech. Read more on my site or see how we can work together at Nazmul Ahasan.

AWS Cloud Cost: Why Most Companies Overspend — and What to Do About It

Nazmul Ahasan — Thu, 05 Mar 2026 17:22:11 +0000

Companies move to AWS expecting three things: scalability, flexibility, and lower costs. The first two usually hold up. The third? That's where I've seen things go sideways — repeatedly.

I've spent a good chunk of my career working across cloud environments, helping teams untangle billing nightmares and architect systems that don't hemorrhage money. And the uncomfortable truth I keep running into is simple:

AWS isn't expensive. Running AWS without paying attention is.

Let me walk through what I actually see happening out there, and what I think more teams should be doing differently.

The promises vs. the reality

The pitch is always the same. Pay only for what you use. No upfront hardware investment. Scaling that just works.

All technically true. But there's a massive asterisk nobody talks about during the sales cycle — the cloud rewards discipline and absolutely punishes negligence. On-prem, your spending was capped by physical hardware. You bought a rack, you used a rack. With AWS, there's no ceiling, and that elasticity can work against you if nobody's watching.

I had a client a couple of years ago who migrated about 200 workloads into EC2 over six months. Nine months into AWS, their monthly cloud spend was about 65% higher than their old on-prem costs, and climbing. Not because AWS was more expensive — because they'd replicated every bad habit from their data center and added new ones on top.

Where the money actually goes

The lift-and-shift trap

This is the most common one. A team takes their on-prem VMs — say, a 16-vCPU server running an internal app — and stands up an identical EC2 instance. No benchmarking, no utilization checks, nothing. Just "make it the same size so nothing breaks."

The problem is that most of those servers were oversized to begin with. I've pulled CloudWatch metrics on freshly migrated instances and found utilization hovering around 12–18%. That's a machine doing almost nothing, billed around the clock. Multiply that across a fleet and you've got a serious problem.

Dev and staging environments that never sleep

This one drives me a little crazy because it's so fixable.

Production needs to run 24/7 — obviously. But I routinely find dev, staging, QA, and sandbox environments humming along at 2 AM on a Sunday. Nobody's using them. Nobody even remembers they're on. But they're racking up charges because no single person is responsible for shutting them down.

I worked with one mid-sized engineering org — maybe 8 teams — where non-production environments accounted for nearly 40% of their monthly compute bill. On infrastructure that was idle roughly 65% of the time.

The on-demand default

AWS gives you several ways to pay less: Savings Plans, Reserved Instances, Spot. But a surprising number of companies run everything on-demand because it's the path of least resistance. Nobody has to forecast. Nobody has to commit. Engineering doesn't have to talk to finance.

The tradeoff is you're paying full sticker price for workloads that have been running predictably for months or even years. I've seen teams save 30–40% just by committing to a one-year Savings Plan on compute they were clearly going to keep running anyway. It's not a risky bet — it's common sense that nobody gets around to acting on.

Storage: the quiet budget killer

Storage costs don't spike. They creep. That's what makes them dangerous.

S3 buckets with no lifecycle policies, so data just accumulates forever. EBS snapshots from instances that were terminated a year ago. Detached volumes sitting there, still on the meter. Log files nobody's looked at since 2022, stored in standard-tier S3 instead of Glacier or just deleted outright.

Individually, none of these line items look alarming. But I've audited accounts where storage waste was quietly eating a double-digit percentage of the monthly bill. It adds up faster than people expect.

Kubernetes cost sprawl

EKS is a whole separate conversation, honestly.

The pattern I see most often: developers set CPU and memory requests conservatively high because they'd rather have headroom than get paged at 3 AM. Fair enough — I get it. But the cluster autoscaler provisions nodes based on those requests, not actual usage. So you end up with a bunch of nodes running at 25% utilization, fully billed, because the resource requests told Kubernetes it needed all that capacity.

Tuning pod requests and limits, setting up HPA properly, right-sizing node groups — this stuff isn't glamorous, but it can cut EKS costs dramatically. I've seen 30–50% reductions from a focused two-week tuning effort.

Why this keeps happening

Here's the thing — it's almost never a skills problem. It's an incentives problem.

Engineers are measured on uptime and reliability, not cost. If my bonus depended on keeping an SLA, I'd overprovision too. That's rational behavior given the incentives. The issue is that nobody's creating a counterbalancing reason to optimize spend.

Finance teams, meanwhile, are used to predictable costs — hardware depreciation, annual license renewals. AWS billing is dynamic, granular, and frankly kind of confusing if you're not technical. So finance sees the bill, maybe flags it when it spikes, but doesn't have the context to ask the right questions.

What bridges that gap is a FinOps practice — finance, engineering, and leadership sharing ownership of cloud spend. But most companies I've worked with either don't have one or have a version that's really just one person generating Cost Explorer reports that nobody reads.

What actually works

No magic bullets here, but these consistently move the needle.

Bake cost into architecture decisions

Not as an afterthought. Before you deploy, estimate your utilization, pick the right instance family, define your scaling thresholds, and figure out your pricing model. If your architecture review only covers security and availability, you're leaving money on the table.

Enforce tagging — I mean it

If you can't attribute a cost to a team, a project, and an environment, you can't control it. I'm a big fan of making deployment pipelines reject resources that don't carry mandatory tags. Sounds heavy-handed, but it works. Once teams see their name next to a dollar amount, behavior changes fast.

A basic setup might look like:

# Example: enforcing tags in a CloudFormation resource
Resources:
  MyInstance:
    Type: AWS::EC2::Instance
    Properties:
      InstanceType: t3.medium
      Tags:
        - Key: Project
          Value: !Ref ProjectName
        - Key: Environment
          Value: !Ref EnvType
        - Key: Owner
          Value: !Ref TeamOwner
        - Key: CostCenter
          Value: !Ref CostCenterCode

Automate environment scheduling

Probably the single highest-ROI change most companies can make. A Lambda function that shuts down dev and staging at 7 PM and spins them back up at 7 AM — that's a 50% compute reduction on those environments right there.

# Simplified Lambda to stop instances by tag
import boto3

def lambda_handler(event, context):
    ec2 = boto3.client('ec2')

    response = ec2.describe_instances(
        Filters=[
            {'Name': 'tag:Environment', 'Values': ['dev', 'staging']},
            {'Name': 'instance-state-name', 'Values': ['running']}
        ]
    )

    instance_ids = []
    for reservation in response['Reservations']:
        for instance in reservation['Instances']:
            instance_ids.append(instance['InstanceId'])

    if instance_ids:
        ec2.stop_instances(InstanceIds=instance_ids)
        print(f"Stopped {len(instance_ids)} instances")

You can get fancier with CI/CD-triggered provisioning, but even a simple scheduler pays for itself immediately.

Stop defaulting to on-demand

Review your usage quarterly. Anything that's been running steadily for 3+ months is a candidate for Savings Plans or RIs. Spot is great for fault-tolerant batch jobs and test workloads. This isn't complicated — it just requires someone to actually own the decision.

Rightsize continuously

Not once. Not annually. Set a recurring calendar reminder — monthly or quarterly — to review EC2, RDS, and EKS utilization. Workloads change. Traffic patterns shift. An instance that was right-sized six months ago might be oversized now.

AWS Compute Optimizer can point you in the right direction, but someone still has to act on the recommendations. Tooling without follow-through is just dashboard decoration.

Tie cost to business outcomes

"Our cloud spend is huge" is not a useful data point on its own.

"Our cost per active user dropped 20% this quarter" or "cost per transaction is flat while traffic doubled" — that's something you can reason about, benchmark, and improve. When cost is framed as a business metric rather than an infrastructure line item, the conversations get a lot more productive.

Where does your team fall?

I think about cloud cost management as a rough progression:

Cloud chaos — No tagging, no budgets, no ownership. The bill is a surprise every month.

Reactive control — Someone set up budget alerts. There's a cleanup sprint every quarter when the CFO starts asking questions.

Proactive optimization — Automated scheduling, a real Savings Plans strategy, regular utilization reviews built into team rhythms.

Cost-aware culture — Teams discuss spend in sprint reviews. Architects weigh pricing alongside performance. Engineering and finance share a vocabulary.

Most companies I encounter are somewhere between the first two. And honestly, getting from chaos to proactive is where the biggest savings happen — usually without a massive organizational overhaul.

TL;DR

AWS is a phenomenal platform. But the flexibility that makes it powerful is the same flexibility that makes it expensive if you're not intentional.

The companies I've seen manage this well all share a few traits: they treat cost as a design constraint rather than an afterthought, they create accountability through tagging and visibility, and they build lightweight processes that keep optimization from being a heroic one-off effort.

None of this is rocket science. Most of it isn't even hard. It just requires someone to care enough to make it a priority — and to keep caring about it month after month.

What's your experience? Have you found specific tools or practices that helped rein in cloud spend? I'd love to hear what's worked (or spectacularly hasn't) for your team.

I lead platform engineering for fintech. Read the full case study here or see how we can work together at Nazmul Ahasan.