DEV Community

Cover image for Resolving Control Tower StackSet Drift from Orphaned Accounts
Anwaar Hussain
Anwaar Hussain

Posted on • Edited on

Resolving Control Tower StackSet Drift from Orphaned Accounts

#aws #controltower #devops #cloudformation

I upgraded AWS Control Tower landing zone from version 3.3 to 4.0. The update failed the first time. StackSet instances showed INOPERABLE status with a Service-Linked Role conflict pointing to accounts that no longer existed in the organisation.

Turns out, accounts had been removed from AWS Organizations without first being un-enrolled from Control Tower. Their StackSet instances remained as orphans. Control Tower couldn't assume the execution role in deleted accounts. Every landing zone operation hit this wall.

Here's how I fixed it.

Why the Service-Linked Role conflict occurs

When Control Tower enrols an account, it creates a Service-Linked Role (AWSServiceRoleForAWSControlTower) and registers that account as a resource consumer of the role. If you close or remove the account from the organisation without un-enrolling it first, the role's resource reference still points to the deleted account. During a landing zone update, Control Tower tries to reconcile the StackSet across all registered accounts. It finds the orphaned reference, cannot reach the account, and fails with the INOPERABLE conflict.

The error

StackSet Id: AWSControlTowerBP-BASELINE-SERVICE-LINKED-ROLE Status: INOPERABLE ResourceLogicalId: ControlTowerServiceRole ResourceType: AWS::IAM::ServiceLinkedRole Reason: Resource of type 'AWS::IAM::ServiceLinkedRole' with identifier 'AWSServiceRoleForAWSControlTower' has a conflict. SLR [AWSServiceRoleForAWSControlTower] is in use by other resources referencing an orphaned account no longer in the organisation.

The fix

Step 1: Remove orphaned stack instances from the affected StackSet. Select "Retain Stacks" because the deleted account is unreachable.

aws cloudformation delete-stack-instances \
  --stack-set-name AWSControlTowerBP-BASELINE-SERVICE-LINKED-ROLE \
  --accounts <ORPHANED_ACCOUNT_ID> \
  --regions <GOVERNED_REGIONS> \
  --retain-stacks \
  --no-cli-pager
Enter fullscreen mode Exit fullscreen mode

Step 2: Check your other Control Tower baseline StackSets for the same orphaned accounts and repeat. Common ones include:

  • AWSControlTowerBP-BASELINE-CONFIG
  • AWSControlTowerBP-BASELINE-CLOUDWATCH
  • AWSControlTowerBP-BASELINE-ROLES
  • AWSControlTowerBP-BASELINE-SERVICE-ROLES
  • AWSControlTowerBP-VPC-ACCOUNT-FACTORY-V1

The exact StackSets in your environment may vary depending on your Control Tower version and configuration. Check all StackSets prefixed with AWSControlTowerBP- for orphaned instances.

Step 3: Navigate to Service Catalog > Provisioned products. If any orphaned accounts have a provisioned product in TAINTED or ERROR state, terminate it.

Step 4: Re-run the Control Tower landing zone update.

Step 5: Re-register your Organisational Units. This propagates updated guardrails and baselines to enrolled accounts under the new landing zone version.

Console path: AWS Control Tower > Organization > select the OU > Re-register OU.

Things to note

  • Only remove instances for orphaned accounts. Do not touch active, enrolled accounts.
  • "Retain Stacks" is critical. Without it, the delete operation fails because the StackSet cannot assume a role in deleted accounts.
  • This issue can occur during any Control Tower landing zone upgrade, not just 3.3 to 4.0.

Prevention

Always un-enrol accounts from Control Tower before closing them in AWS Organizations. AWS documentation states this explicitly: "You must unenroll the account before you close it."

References

Top comments (0)