Four Failures I Kept Repeating on AWS Amplify — Full env Replacement, Green Build but 500 at Runtime, Silent Outages, Domain Migration

From April to July 2026, I ran several Next.js products on AWS Amplify Hosting (AWS's managed service for hosting and deploying web apps). This article sorts the "Amplify-specific failures" I kept repeating into four categories: env vars disappearing, builds passing but the app failing at runtime, failures progressing silently, and domain migrations tripping over stale CloudFront (AWS's CDN) allocations left behind. Because I hit each of these across multiple products, I'm writing down the reproduction conditions and how to prevent them.

Let me put the conclusions first (reading the full article takes about 10 minutes).

• For any API that touches env, confirm up front whether it appends or replaces. Amplify's update-app --environment-variables replaces the entire map with the one you pass. Intending to add one variable, I wiped the existing env entirely and took down production authentication. A fix isn't done until you rebuild after editing env.
• A passing build doesn't guarantee it works in production. Amplify SSR (server-side rendering) doesn't pass branch environment variables to the runtime Lambda, and NEXT_PUBLIC_* is baked in at build time. So the build passes, yet at runtime it returns 500 for missing env. Bake env into .env.production during preBuild, or route it through a layer that resolves env at runtime.
• Don't hide failures. If you hide ACCESS_DENIED behind HTTP 200 + sample data, the outage shows up in neither monitoring nor tests. 87–89% of the bill was build time, so what needed cutting was auto-build, not SSR. Don't write secrets into the build spec.
• Deleting a domain association is instant downtime, and you can't roll it back quickly. The faster you spin delete and recreate, the more failures pile up waiting on CloudFront to release. Waiting 75 minutes, trying just once, and not retrying on failure was the right answer.

Failure 1: update-app "replaces all" of your env

The first one was an incident where production OAuth authentication on a certain membership community site suddenly stopped. The cause: a past session had run the following command intending to add just one environment variable.

aws amplify update-app --app-id <APP_ID> \
  --environment-variables NEW_KEY=value

update-app --environment-variables replaces the entire existing environment-variable map with the map you pass. It does not append. As a result, previously configured values such as ONION_CLIENT_ID were blown away along with the whole map, the auth client ID turned into a nonexistent value, and production authentication was wiped out. What made it worse: after fixing the value in the Console, we did not rebuild. Amplify bakes environment variables into the build artifact at build time, so the build with the broken value baked in kept running in production — a state of "the value in the Console is correct, but production stays broken."

From this we set a rule: "Touch Amplify env manually from the Console; touching env via the CLI update-app is banned in principle," and encoded it into a skill (an automated check). Two lessons. For any env-manipulation API, always confirm whether it appends or replaces. And a fix isn't done until you rebuild after editing env.

Failure 2: The build passes, but it fails at runtime

The next category is nastier because it takes longer to discover. In Amplify SSR (server-side rendering), branch environment variables are not propagated to the runtime Lambda (AWS's serverless function runtime).

Concretely, it was a two-stage trap. When I put an admin console on Amplify, all of NextAuth's (a Next.js authentication library) API routes returned 500. On investigation: (1) NEXT_PUBLIC_* variables are statically inlined at next build time, so if unset at build time an empty string gets baked into the bundle. (2) On top of that, app-/branch-level environment variables enter the build container but are not forwarded to the Lambda runtime that runs SSR. The build sees the environment variables, so it passes normally, and only at runtime does it fail with "the value is missing." The /api/_debug-env I made for diagnostics itself also returned 404, because "Next.js API routes can't start with _."

The workaround is to generate and bake in .env.production during the preBuild phase, while the variables are still visible.

# amplify.yml (bake env into .env.production during preBuild)
frontend:
  phases:
    preBuild:
      commands:
        - |
          cat > .env.production <<EOF
          NEXT_PUBLIC_API_BASE=${NEXT_PUBLIC_API_BASE}
          NEXTAUTH_URL=${NEXTAUTH_URL}
          EOF
        - npm ci

The same trap recurred later when I introduced a type-safe env library (the kind where createEnv validates process.env at module-load time). Local and CI builds passed, but the production Lambda threw Invalid environment variables and returned 500. "The build passing" was, conversely, hiding the failure. The response was four stages: immediately revert the three PRs → record the incident → redesign with a resolveRuntimeEnv approach (a layer that safely resolves env at runtime) → re-apply gradually starting from staging. After that I made this a permanent rule as "build-once + runtime injection." In Amplify SSR, "the build passes" and "it works at runtime" are different worlds.

Failure 3: Failures progress silently

The third is the "it's broken but no alarm goes off" category. I'll present three.

The first. An outage where every list in the admin console went blank. The SSR compute role (computeRoleArn) was null and access to DynamoDB (AWS's managed NoSQL database) was entirely blocked, yet because the API was designed to silently return HTTP 200 + sample data on ACCESS_DENIED, the failure showed up in neither monitoring nor tests. Moreover, production E2E had never once gone through the path where the auth bypass was disabled, so "data fetching after authentication" was an entire hole in verification. From this we established a three-part set: fail-loud (don't hide errors behind a 200 — fail instead), staging promotion, and authenticated E2E.

The second. When I broke down one month's Amplify bill in Cost Explorer (AWS's cost-analysis tool), 87–89% of the cost was BuildDuration (build time), while the SSR runtime was only $3–7 a month. The original savings plan of "make SSR lighter" was off the mark; the correct move was "stop auto-build and deploy only the artifact." Silent cost leaks are the same hole.

The third. amplify.yml wrote SES_ACCESS_KEY_ID/SECRET in plaintext into .env.production at build time, and it got copied into .next/ and remained in both the deploy artifact and the build logs — a leak vector. Fortunately the attacker's action was a single CreateUser (blocked by AccessDenied) with no spike in daily cost and zero actual harm, but it left the lesson that writing secrets into the build spec turns the logs into a leak surface.

Failure 4: Domain migration, and CloudFront that won't release

The last is infrastructure migration. In the process of consolidating multiple Amplify apps into a single hub, I re-assigned domain associations many times. What I learned here is that deleting an Amplify domain association is instant downtime and can't be rolled back quickly.

In one canary migration, I followed the steps "delete the old association → immediately create it on the hub → switch DNS," but CloudFront's alias release lagged and the hub-side association went FAILED. Even reverting DNS to the old target didn't recover, because the old association was already deleted — about 6 minutes of downtime. It came back only after I waited for the alias to be released (about 6 minutes), then deleted and recreated the FAILED one.

Further, in another domain's migration, creation failed 5 times in a row. Each time I repeated delete → recreate, CloudFront distributions that held onto the alias and wouldn't disappear (left waiting for release — so-called zombies) piled up, making the next attempt FAIL immediately. The faster you spin delete and recreate, the more failures — a vicious cycle. The countermeasure was a reversal of thinking.

# Anti-churn procedure (pseudocode)
1. Point DNS at the new target
2. "Do nothing" for 75 minutes until every zombie's alias lock is released
3. Try creating the association "just once"
4. If it fails, treat it as a deep incident, don't retry, exit immediately (don't spin back into the spiral)

It was a rare case where "not retrying" was the right answer.

As backstory: early on, when putting a pnpm monorepo's Next.js SSR on Amplify, I have a record of firing off 14 fix commits over two days on nothing but "where to copy node_modules." After wandering through standalone output, changing the copy destination, and resolving symlinks, I arrived at "the new Amplify handles SSR natively, so standalone isn't needed."

Why build success and runtime success don't line up on Amplify

The root of failures 1–3 is the same: Amplify SSR's execution model. next build runs in a build container, where env is visible. The generated SSR runs as a separate Lambda, where branch env is not passed automatically. On top of that, NEXT_PUBLIC_* is baked into the code as a string at build time, becoming something different from the runtime process.env. In other words, the "moment env takes effect" is split across three places: static inlining (build time), baking into the artifact (build time), and the Lambda execution environment (runtime). If you don't know about this split and assume "I set env, so it's usable everywhere," the build is green but production fails. That's why editing env requires a rebuild, and values needed at runtime must be baked in during preBuild or resolved through a runtime-injection layer — all consequences of this execution model.

Transferable lessons

• For any API that touches env, always confirm "append or replace." Amplify's update-app --environment-variables replaces all. A fix isn't done until you rebuild after editing env.
• In Amplify SSR, "the build passes" and "it works at runtime" are different. Values needed at runtime should be baked into .env.production during preBuild or passed through a runtime-resolution layer. Before adding load-time validation, always exercise runtime in staging.
• Don't let failures go silent. Don't hide ACCESS_DENIED behind 200 + sample data — fail instead (fail-loud). Break down cost in Cost Explorer, and don't write secrets into the build spec.
• Deleting a domain association is instant downtime. Wait for the alias to be released before switching DNS, and if failures continue, switch to "wait and try just once" to stop retrying.