DEV Community: Rafagross

AWS Organizations in Production: Structure, SCP Evaluation, and Landing Zone Gotchas

Rafagross — Sun, 27 Sep 2026 05:20:57 +0000

Most teams adopt AWS Organizations for consolidated billing and stop there. The real value is the control plane: a tree of accounts where guardrails attached at the top apply to everything below. That same tree is also where most "why is this denied?" tickets come from, because the evaluation rules are less intuitive than IAM alone.

This post covers the structure, how policies are evaluated, and the gotchas worth knowing before you run it in production.

The Account Is the Boundary

An AWS account is the hard boundary for resources, security, and billing. IAM manages identities inside an account. Organizations manages the accounts themselves.

That distinction drives the whole multi-account model: separate accounts give you blast-radius isolation, clean cost attribution, and a place to attach controls that no one inside the account can override.

Structure

An organization has:

One management account. It owns the organization and is the payer account. It should run org and billing tasks only. No workloads.
One root. The top of the tree. Policies attached here apply to every account.
Organizational units (OUs). Containers for accounts. OUs can nest up to five levels deep under the root.
Member accounts. Where workloads, logging, and security tooling live.
Policies. Attached at the root, an OU, or an individual account.

A common baseline layout:

OU	Accounts	Purpose
Security	Log Archive, Security Tooling (Audit)	Centralized logs, delegated admin for security services
Infrastructure	Network, Shared Services	Transit, DNS, shared tooling
Workloads	Prod, NonProd	Application accounts, grouped by lifecycle
Sandbox	Per-developer or per-team	Experimentation with looser controls
Suspended	Accounts pending closure	Deny-all quarantine

Tip: Design OUs around the controls and lifecycle you want to apply, not around the org chart. Teams get reorganized; "prod needs stricter guardrails than sandbox" does not change.

Enable All Features

Organizations has two feature sets. Consolidated-billing mode only merges invoices. All features adds the authorization policies (SCPs, RCPs), management policies (tag, backup, EC2), trusted access for service integrations, and delegated administrators. If you are building a landing zone, you need all features.

Policy Types That Matter Day to Day

Policy	What it does
SCP (service control policy)	Sets the maximum permissions for IAM users and roles in member accounts. Grants nothing on its own.
RCP (resource control policy)	Sets the maximum permissions on resources in member accounts (S3, KMS, Secrets Manager, SQS, and many more), regardless of who is calling. Grants nothing on its own.
Tag policy	Standardizes tag keys and values so cost allocation and ABAC stay consistent.
Backup policy	Deploys AWS Backup plans centrally across accounts and Regions.
EC2 policy (declarative)	Enforces EC2, VPC, and EBS baselines org-wide, such as IMDS defaults, AMI and EBS snapshot block public access, serial console access, and VPC Block Public Access. The configuration is maintained even as the service adds new APIs.

How Inheritance Actually Works

This is the part that trips people up.

A Deny is simple: attach it anywhere and it applies to everything below that point.

An Allow is not inherited the way people expect. For an action to be permitted in an account, an SCP must allow it at every level from the root down to the account: the root, each OU in the path, and the account itself. If any level is missing the Allow, the action is implicitly denied, even if an IAM policy in the account grants *:*.

Root            FullAWSAccess        -> allows *
  OU Workloads  AllowOnlyApprovedSvc -> allows ec2:*, s3:*, ... (no dynamodb:*)
    Prod acct   FullAWSAccess        -> allows *

Result in Prod: dynamodb:* is denied. The OU level never allowed it.

This is why the default FullAWSAccess SCP exists at every level. If you replace it with an allow-list at the OU, that allow-list becomes the ceiling for every account underneath.

Most common SCP incident: Someone detaches FullAWSAccess from an OU while testing an allow-list SCP, and every account under that OU loses access to services that weren't on the list. Prefer deny-list SCPs (explicit Deny statements on top of FullAWSAccess) unless you have a specific reason to allow-list.

How a Request Is Evaluated

For a principal in a member account, access requires an Allow from every applicable layer:

Identity-based policy on the user or role must allow the action.
SCPs in the account's path must allow it.
RCPs in the path must allow it for the target resource (if the service supports RCPs).
Permission boundaries, session policies, and resource-based policies still apply as usual.

An explicit Deny at any layer wins. SCPs and RCPs only ever reduce what is possible; they never grant access.

A practical consequence: when you debug an AccessDenied in a member account and the IAM policy looks correct, check the SCPs on every node in the account's path, not just the ones attached directly to the account.

Landing Zone Pattern

The goal is to separate the governance plane from workloads:

Management account: org and billing only. Tightly restricted human access.
Security OU: Log Archive receives org-wide CloudTrail and Config data. Security Tooling (Control Tower calls it Audit) is the delegated administrator for GuardDuty, Security Hub, and similar services.
Infrastructure OU: networking and shared services.
Workloads OU: prod and non-prod split so guardrails can differ.

AWS Control Tower orchestrates this on top of Organizations: Account Factory for provisioning, managed controls (preventive via SCPs/RCPs, detective via AWS Config, proactive via CloudFormation hooks), and drift detection. If you manage infrastructure as code, Account Factory for Terraform (AFT) provisions accounts through a Terraform pipeline.

Operating Lifecycle

Identity: IAM Identity Center for human access, cross-account roles for automation, least privilege throughout.
Provisioning: Account Factory, AFT, or the Organizations API. Never click-ops accounts in production.
Baseline: logging, Config, security services, and networking applied from day one, typically via StackSets or Control Tower.
Guardrails: preventive, detective, and proactive controls, plus drift detection.
Lifecycle: move accounts between OUs as their purpose changes. To retire one, move it to a Suspended OU with a deny-all SCP, then close it. A closed account can be reopened during the post-closure period (90 days).

Production Gotchas

SCPs and RCPs never restrict the management account. Anything you run there is outside your guardrails. That is the main reason to keep workloads out of it.
SCPs don't affect service-linked roles. AWS services acting through service-linked roles are not limited by your SCPs.
SCPs do apply to delegated administrator accounts. They are member accounts. Make sure your guardrails don't block the security services you delegated.
OUs nest at most five levels deep. Deep hierarchies are also harder to reason about during an incident. Keep it shallow.
Delegate security services to the Audit/Security Tooling account instead of administering them from the management account.
Limits: SCP documents max out at 10,240 characters, and each root, OU, or account can have up to 10 SCPs attached. Plan statement consolidation early.

Quick Inventory Commands

# Root ID and enabled policy types
aws organizations list-roots

# All accounts in the org
aws organizations list-accounts

# OUs under a parent (root or OU)
aws organizations list-organizational-units-for-parent --parent-id r-examplerootid

# SCPs attached to a specific target
aws organizations list-policies-for-target \
  --target-id 123456789012 \
  --filter SERVICE_CONTROL_POLICY

Run these from the management account or a delegated administrator.

Sources

EC2 Instance Unreachable via SSM Session Manager

Rafagross — Sun, 13 Sep 2026 03:12:04 +0000

Summary

This runbook resolves the situation where an EC2 instance is running and visible in the console but either does not appear in the Systems Manager Fleet Manager inventory, or returns a connection error when you attempt to start a Session Manager session.

When to Use This Runbook

Use this runbook when you observe any of the following:

Session Manager shows "Instance not connected" or "Unable to start session"
The instance does not appear in Systems Manager → Fleet Manager → Managed Nodes
ssm:StartSession returns TargetNotConnected
A previously working instance stopped responding to SSM after a restart, IAM change, or network modification

Step 1: Verify Instance State

Console path: EC2 → Instances → [Instance ID]

Confirm the instance is in running state and that both Status checks show 2/2 passed.

If status checks are failing, stop here and use the EC2 Status Check runbook. SSM is irrelevant if the instance itself is impaired.

Expected result: Instance state = running, 2/2 status checks passed.

Step 2: Check SSM Managed Node Status

Console path: Systems Manager → Fleet Manager → Managed nodes

Search for the instance ID. If it does not appear, or shows Connection Lost, the issue is one of: missing IAM role, stopped SSM agent, or blocked network path.

Expected result: Instance appears with status Online. If missing or Connection Lost, continue to Step 3.

Step 3: Verify IAM Instance Profile

Console path: EC2 → Instances → [Instance ID] → Security tab → IAM Role

The instance must have an IAM role attached with the AmazonSSMManagedInstanceCore managed policy (or equivalent custom policy granting the minimum SSM actions).

Minimum required IAM actions if using a custom policy:

ssm:UpdateInstanceInformation
ssm:ListInstanceAssociations
ssm:DescribeInstanceProperties
ssm:DescribeDocumentParameters
ssmmessages:CreateControlChannel
ssmmessages:CreateDataChannel
ssmmessages:OpenControlChannel
ssmmessages:OpenDataChannel
ec2messages:AcknowledgeMessage
ec2messages:DeleteMessage
ec2messages:FailMessage
ec2messages:GetEndpoint
ec2messages:GetMessages
ec2messages:SendReply

Note: IAM propagation delay:
IAM role changes applied to a running instance take effect on next SSM agent heartbeat, typically within 2-3 minutes. If you just updated the role, wait 5 minutes and re-check Fleet Manager before continuing.

Expected result: IAM role attached with AmazonSSMManagedInstanceCore or equivalent. If missing, attach the role and wait 5 minutes.

Step 4: Check SSM Agent on the Instance

Console path: EC2 → Instances → [Instance ID] → Actions → Monitor and troubleshoot → Get system log

Look for lines referencing amazon-ssm-agent. A healthy agent shows:

amazon-ssm-agent.service: active (running)

Errors to look for:

Failed to start Amazon SSM Agent — agent start failure, likely OS-level issue
Error connecting to endpoint — network path problem (proceed to Step 5)
No SSM lines at all — agent not installed or not running

If the agent is not running, you can use EC2 Run Command with AWS-RunShellScript to restart it — but only if the instance is already registered (partial connectivity). If it's fully unreachable, use EC2 Instance Connect or serial console (if available) to restart manually.

Expected result: SSM agent running. No connection errors.

Step 5: Diagnose the Network Path

SSM Session Manager requires outbound HTTPS (port 443) from the EC2 instance to three regional endpoints:

Endpoint	Purpose
`ssm.{region}.amazonaws.com`	Agent registration
`ssmmessages.{region}.amazonaws.com`	Session data channel
`ec2messages.{region}.amazonaws.com`	EC2 message delivery

For private instances (no NAT Gateway, VPC endpoint required):

Console path: VPC → Endpoints

Verify that all three interface endpoints exist, are associated with the correct VPC, and have status available. Check that the endpoint security group allows inbound HTTPS from the instance's security group.

Console path: VPC → Security Groups → [Endpoint SG]

Inbound rule required:

Type: HTTPS (443)
Source: [Instance security group ID]

For instances with NAT Gateway:

Console path: VPC → Route Tables → [Instance subnet's route table]

Confirm a 0.0.0.0/0 route pointing to the NAT Gateway exists and the NAT Gateway is in available state.

Note:
If the instance has a public IP and is in a public subnet with an IGW route, outbound port 443 directly to AWS endpoints is sufficient. No VPC endpoints needed in that case.

Expected result: Either VPC endpoints present and available, or valid NAT/IGW route exists. Instance SG allows outbound 443.

Step 6: Check Instance Security Group Outbound Rules

Console path: EC2 → Instances → [Instance ID] → Security tab → Security groups → [SG ID] → Outbound rules

Confirm outbound HTTPS is allowed. The default AWS SG allows all outbound traffic. If rules have been tightened:

Type: HTTPS (443)
Destination: 0.0.0.0/0

OR (preferred, more restrictive):

Type: HTTPS (443)
Destination: [VPC endpoint prefix list or specific endpoint IPs]

Expected result: Outbound 443 allowed from instance security group.

Step 7: Force SSM Agent Re-Registration (Last Resort)

If all above checks pass but the instance still doesn't appear in Fleet Manager, the agent's registration may be stale (common after AMI snapshots or instance cloning).

Use EC2 Run Command with AWS-RunShellScript:

# Amazon Linux 2 / AL2023
sudo systemctl stop amazon-ssm-agent
sudo rm -rf /var/lib/amazon/ssm/registration
sudo systemctl start amazon-ssm-agent
sudo systemctl status amazon-ssm-agent

Wait 2 minutes, then re-check Fleet Manager.

Validation Checks

Check	How to Verify	Expected Result
Instance in Fleet Manager	SSM → Fleet Manager → search instance ID	Status: Online
Session starts	SSM → Session Manager → Start session → select instance	Terminal opens
Agent heartbeat	CloudWatch Logs → /aws/ssm/amazon-ssm-agent	Recent heartbeat log entries

Rollback

No destructive changes are made by this runbook. If you attached a new IAM role that you want to remove:

Console path: EC2 → Instances → [Instance ID] → Actions → Security → Modify IAM role → No role

Escalation

Condition	Next Action
All steps pass but instance still unreachable	Open AWS Support case — provide instance ID, region, VPC endpoint IDs, and CloudWatch agent logs
Agent crashes immediately on start	Check OS disk space (`df -h`), memory, and `/var/log/amazon/ssm/amazon-ssm-agent.log` for errors
Instance in private subnet, no VPC endpoints, no NAT	Network path is broken by design — work with network team to add VPC endpoints or NAT