DEV Community

Cover image for My main branch wouldn't deploy: CloudFormation forensics on Amplify Gen 2
Ibukun Demehin
Ibukun Demehin

Posted on

My main branch wouldn't deploy: CloudFormation forensics on Amplify Gen 2

#aws #amplify #cloudformation #debugging

My dev branch deployed clean. The exact same commit on main — the merge that fast-forwarded dev into main, byte-for-byte identical backend — rolled back every single time. Amplify Hosting went red, CloudFormation logged ROLLBACK_COMPLETE, and the only explanation it offered was this:

Validation failed with 1 error(s). Call DescribeEvents to retrieve the
full list of issues with resource and property details, resolve each
error, then retry the operation.
Enter fullscreen mode Exit fullscreen mode

No resource name I could act on. No property. One error, unnamed. Here's how I found the one error — and it turned out to be a name two git branches had been quietly fighting over.

TL;DR — On Amplify Gen 2, dev and main are branches of the same Amplify app. If you publish an SSM parameter keyed only on the app id (/myapp/${AWS_APP_ID}/…), both branches generate the identical parameter name. Whichever deploys first owns it; the second branch's deploy dies with a bare Validation failed with 1 error(s) on exactly the nested stack that creates the parameter, then ROLLBACK_COMPLETE. Fix: put the environment in the path (/myapp/${AWS_APP_ID}/${env}/…).

(Part 4 of Building CannyCart, a voice-first shopping app I'm building in public. Self-contained CloudFormation debugging story — no earlier context needed.)

The setup: one app, two branches, a shared backend

For anyone landing here from the error text: AWS Amplify Gen 2 (CDK under the hood), deployed via ampx pipeline-deploy in Amplify Hosting. One Amplify app, two branches wired to it — dev and main. The backend is shared code: Cognito auth, an AppSync GraphQL API, and three DynamoDB-backed models — UserProfile, ShoppingList, ShoppingItem.

One detail matters more than any other, so here it is up front. A post-confirmation Lambda needs the UserProfile table name. You can't hand a Lambda a table name as a CDK token across stacks — auth depends on data, data would depend on auth, and the deploy fails with a circular dependency. The house pattern (and AWS's recommendation) is to publish the name to SSM Parameter Store from inside the table's own stack, and give the Lambda only the SSM path string:

const amplifyAppId = process.env.AWS_APP_ID ?? "local";
const ssmPrefix = `/cannycart/${amplifyAppId}`;          // ← the bug lives here

const userProfileParamName = `${ssmPrefix}/UserProfileTableName`;
new StringParameter(Stack.of(userProfileTable), "UserProfileTableNameParam", {
  parameterName: userProfileParamName,
  stringValue: userProfileTable.tableName,
});
Enter fullscreen mode Exit fullscreen mode

Read that ssmPrefix and hold onto it. It's keyed on the app id and nothing else.

dev worked. Then I merged to main.

dev had been deploying happily for days. I opened a PR, merged dev into main, and watched the first main build. It got a long way:

✔ Backend synthesized in 13.23 seconds
✔ Type checks completed in 15.43 seconds
✔ Built and published assets
Enter fullscreen mode Exit fullscreen mode

Then CloudFormation started creating resources for real. Auth stack: complete. The data stack came up and began creating the three model stacks in parallel. Two of them sailed through:

CREATE_COMPLETE | AWS::CloudFormation::Stack | ShoppingList.NestedStackResource
CREATE_COMPLETE | AWS::CloudFormation::Stack | ShoppingItem.NestedStackResource
Enter fullscreen mode Exit fullscreen mode

And the third:

CREATE_FAILED | AWS::CloudFormation::Stack | …UserProfileNestedStack…
  Validation failed with 1 error(s). Call DescribeEvents to retrieve the
  full list of issues…
Enter fullscreen mode Exit fullscreen mode

That failure rolled straight up the tree:

CREATE_FAILED | data.NestedStackResource | Embedded stack … was not
  successfully created: The following resource(s) failed to create:
  [amplifyDataUserProfileNestedStackUserProfileNestedStackResource…].
ROLLBACK_IN_PROGRESS | amplify-…-main-branch-… | Rollback requested by user.
Enter fullscreen mode Exit fullscreen mode

Four minutes of DELETE_IN_PROGRESS later — the whole backend, auth pool and all, torn back down — it ended where every attempt ended:

ROLLBACK_COMPLETE | AWS::CloudFormation::Stack | amplify-…-main-branch-…
Enter fullscreen mode Exit fullscreen mode

Same commit as dev. Green synth, green type-check. Dead on create.

Wrong theory: "it's a flaky deploy, just retry"

The rollback message literally says "Rollback requested by user," which reads like something transient — as if a retry would clear it. It won't, and that phrasing is a red herring: "user" is CloudFormation's own deployment role, and the rollback was requested because a resource failed, not because anything flaked. I re-ran the build. It failed at the identical stack with the identical bare validation error. Whatever this was, it was deterministic and it was specific to main.

The clue the log did give me: only UserProfile died

Here's the thing I almost scrolled past. Three model stacks were created in the same deploy, from the same schema, with the same auth rules. ShoppingList and ShoppingItem both succeeded. Only UserProfile failed.

If this were a broken schema or a bad credential or a region problem, all three would fail. One failing — and the same one every time — means UserProfile has something the other two don't. So what's different about UserProfile?

Back to that code. ShoppingList and ShoppingItem are plain models. UserProfile is the only stack that also creates an extra resource: the UserProfileTableNameParam SSM StringParameter. Of everything unique to that one stack, a named, must-be-globally-unique resource is the obvious suspect. The bare "Validation failed" was CloudFormation refusing to create a parameter — and the reason was one DescribeEvents call deeper than the build log prints.

The actual cause: two branches, one parameter name

The parameter name is /cannycart/${AWS_APP_ID}/UserProfileTableName. AWS_APP_ID is the Amplify app id — and dev and main are two branches of the same app. They resolve AWS_APP_ID to the identical value. So both branches try to create the exact same SSM parameter:

/cannycart/d216bdrgjoaojz/UserProfileTableName
Enter fullscreen mode Exit fullscreen mode

dev deployed first and created it. When main deployed and its UserProfile stack tried to create the same parameter name, SSM rejected it — the parameter already exists — and CloudFormation surfaced that rejection as the maddeningly generic Validation failed with 1 error(s). The final toolkit error names the culprit stack twice but still never prints the reason:

[CloudFormationDeploymentError] The CloudFormation deployment has failed.
  ROLLBACK_COMPLETE: Embedded stack …UserProfileNestedStack… was not
  successfully created: Validation failure detected. See the operation's
  FAILED event for details.
Enter fullscreen mode Exit fullscreen mode

The reason lived only in the FAILED event, behind DescribeEvents. The build log will point at the stack all day; it will not tell you the parameter was a duplicate.

The fix: put the environment in the path

An app-id-only namespace assumes one deploy per app. The moment a second branch of the same app deploys, the namespace collides. The fix is to make the path unique per environment — main → prod, other branches → the branch name, local sandbox → your username:

- const ssmPrefix = `/cannycart/${amplifyAppId}`;
+ // dev and main are branches of the SAME Amplify app, so an app-id-only
+ // path collides between them. env makes it unique per environment.
+ const ssmPrefix = `/cannycart/${amplifyAppId}/${env}`;
Enter fullscreen mode Exit fullscreen mode

Now dev publishes /cannycart/d216bdrgjoaojz/dev/UserProfileTableName and main publishes /cannycart/d216bdrgjoaojz/prod/UserProfileTableName. No collision. main deployed green on the next run.

What I took away

  • A nested-stack rollback hides its real reason one level down. Validation failed with 1 error(s) in the build log is a pointer, not an answer — the actual cause is behind DescribeEvents / the stack's FAILED event in the console. Go there first instead of theorizing.
  • When N identical things deploy and only one fails, debug the difference. Two model stacks succeeded and one didn't; the failing one was the only one creating a named global resource. That asymmetry was the whole case.
  • Namespace shared infrastructure by environment, not just by app. Any globally-unique name — SSM parameters, S3 buckets, IAM roles — keyed on AWS_APP_ID alone will collide the instant a second branch of that app deploys. Put the env in the key from day one.
  • "It works on dev" can mean "dev got there first." A resource that another environment already created will fail silently for the next one. Shared-app, multi-branch setups make this a standing trap.

Next up

The feature this whole backend exists for: talking to your shopping list. You say "milk, a couple of bananas, a dozen eggs," and on-device speech recognition plus a Claude Haiku extraction Lambda turn it into a categorised, structured list — with a review step so a bad parse never silently pollutes your data. That's Post 5.

What's the most useless "helpful" error message your cloud provider has handed you? Validation failed with 1 error(s) — with the one error omitted — is my current favourite.

Top comments (0)