Solved: Found this gem in Production. Have you ever seen an SCP written like this?

🚀 Executive Summary

TL;DR: Poorly structured AWS Service Control Policies (SCPs) in production can lead to security vulnerabilities and operational friction. This article provides strategies to identify and refactor problematic SCPs using a deny-first, single-responsibility approach, and advocates for proactive validation with AWS Config and management via Infrastructure as Code (IaC) like Terraform or CloudFormation.

🎯 Key Takeaways

• Problematic SCPs exhibit unintelligible logic, unexpected behavior, maintenance nightmares, auditing headaches, lack of documentation, and can be overly broad or specific.
• SCP evaluation prioritizes explicit ‘Deny’ statements over ‘Allow’ statements, and multiple applicable SCPs are logically combined, requiring an action to be allowed by all and denied by none.
• Robust SCP refactoring adheres to three principles: Deny-First/Least Privilege, Single Responsibility Principle for each SCP, and Clear Naming/Documentation.
• Proactive validation using AWS Config and Conformance Packs can enforce SCP best practices like attachment status, naming conventions, and size limits.
• Managing SCPs with Infrastructure as Code (IaC) tools like Terraform or CloudFormation enables version control, peer review, automated deployment, modularity, and testing, preventing ‘gems’ from appearing.

This post dissects poorly structured AWS Service Control Policies (SCPs) found in production environments, offering practical strategies to identify, refactor, and manage them effectively for enhanced security and operational clarity.

The Production “Gem”: Unraveling Cryptic AWS SCPs

As DevOps engineers, we often encounter historical artifacts in production environments. Sometimes these are elegant, well-maintained systems; other times, they are a labyrinth of uncommented code, manual configurations, or, in the case of cloud, convoluted policy documents. The Reddit thread title “Found this gem in Production. Have you ever seen an SCP written like this?” perfectly encapsulates the moment of dread and bewilderment when you stumble upon an AWS Service Control Policy (SCP) that seems designed to confuse rather than control.

SCPs are powerful, foundational guardrails in an AWS Organization, enforcing permissions across all member accounts. A poorly written SCP can lead to unexpected service disruptions, security vulnerabilities, or simply create an unmanageable mess. This article will dive into identifying problematic SCPs, explaining why they are detrimental, and providing actionable solutions to bring order, clarity, and security back to your AWS Organization.

Symptoms: Identifying a Problematic SCP

A “gem” SCP isn’t just ugly; it actively causes operational friction and security concerns. Here are common symptoms indicating you have one (or many) in your environment:

• Unintelligible Logic: The policy’s intent is unclear due to overly complex conditions, extensive use of NotAction or NotResource without clear purpose, or multiple conflicting Effect statements within a single policy.
• Unexpected Behavior: Services are unexpectedly blocked or, conversely, highly privileged actions are inadvertently allowed, despite what the policy seems to imply.
• Maintenance Nightmares: Modifying the SCP requires extensive testing and guesswork, often leading to unintended side effects. Engineers avoid touching it.
• Auditing Headaches: It’s nearly impossible to confirm compliance or understand the blast radius of a policy change.
• Lack of Documentation: Absence of comments, descriptions, or a clear naming convention makes the SCP a black box.
• Overly Broad or Specific: Policies that are either too permissive (e.g., "Action": "*", "Resource": "*" with many complex denies) or so specific they become brittle and hard to maintain.

Consider this hypothetical “gem” SCP, a classic example of what you might find:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AllowAllButThenDenySome",
      "Effect": "Allow",
      "Action": "*",
      "Resource": "*"
    },
    {
      "Sid": "RestrictSpecificServicesAndRegionsUnlessAdmin",
      "Effect": "Deny",
      "NotAction": [
        "s3:GetObject",
        "ec2:Describe*",
        "organizations:ListAccounts"
      ],
      "Resource": [
        "arn:aws:s3:::my-prod-data/*",
        "arn:aws:ec2:*:*:instance/*"
      ],
      "Condition": {
        "StringNotLike": {
          "aws:PrincipalArn": [
            "arn:aws:iam::*:role/AdminRole",
            "arn:aws:iam::*:user/DevOpsTeamUser"
          ]
        },
        "StringNotEquals": {
          "aws:RequestedRegion": [
            "us-east-1",
            "us-west-2"
          ]
        }
      }
    },
    {
      "Sid": "AnotherDenyForIAMAndOrgs",
      "Effect": "Deny",
      "Action": [
        "iam:*Delete*",
        "organizations:*Destroy*",
        "account:DeleteAccount"
      ],
      "Resource": "*"
    }
  ]
}

This SCP is problematic for several reasons:

• Conflicting Intent: It starts with a broad "Allow": "*" but then attempts to deny specific actions. While SCP evaluation prioritizes explicit denies, starting with an overly permissive statement can obscure the true intent and make it harder to reason about.
• Overuse of NotAction: The NotAction element in the second statement is difficult to read and maintain. It’s often clearer to explicitly list what is denied rather than what isn’t.
• Complex Conditions: The combination of StringNotLike and StringNotEquals for both Principal ARN and Region makes the policy hard to parse quickly.
• Monolithic Design: It tries to achieve multiple distinct goals (general restrictions, region restrictions, admin exemptions, specific forbidden actions) within a single policy.

Understanding SCP Evaluation: The Foundation of Sanity

Before diving into solutions, a quick refresher on SCP evaluation is crucial:

• Default Deny: By default, if no SCP explicitly allows an action, it’s implicitly denied.
• Explicit Deny Overrides: An explicit "Deny" statement in any applicable SCP will always override an explicit "Allow" statement, whether in another SCP or an IAM policy.
• Multiple SCPs: If multiple SCPs apply to an account (e.g., one attached to the Root and another to an OU), they are logically combined. An action must be explicitly allowed by *all* applicable SCPs and *not* explicitly denied by *any* applicable SCP (and also allowed by IAM policies).

The best practice for SCPs is generally a “deny-first” approach, where you explicitly deny specific dangerous actions or resources and implicitly allow everything else (to be controlled by IAM policies). The “gem” SCP above, by starting with a broad allow, violates this clarity principle.

Solution 1: Refactoring for Clarity and Maintainability

The first step to taming “gem” SCPs is refactoring them into a clean, predictable structure. This involves breaking down monolithic policies, adopting clear naming conventions, and adhering to security best practices.

Three Principles of Robust SCP Refactoring

1. Deny-First, Least Privilege: Design SCPs to explicitly deny actions that are never allowed, regardless of IAM policies. Avoid broad Allow statements in SCPs.
2. Single Responsibility Principle: Each SCP should ideally address a single, logical control objective (e.g., restrict regions, deny root user actions, forbid specific services).
3. Clear Naming and Documentation: Use descriptive names, SIDs, and potentially an external README or Wiki to explain the purpose of each SCP.

Refactoring Example: From “Gem” to Guardrail

Let’s take parts of our “gem” SCP and refactor them into cleaner, purpose-driven policies.

Original “Gem” Statement (for reference):

{
  "Sid": "RestrictSpecificServicesAndRegionsUnlessAdmin",
  "Effect": "Deny",
  "NotAction": [
    "s3:GetObject",
    "ec2:Describe*",
    "organizations:ListAccounts"
  ],
  "Resource": [
    "arn:aws:s3:::my-prod-data/*",
    "arn:aws:ec2:*:*:instance/*"
  ],
  "Condition": {
    "StringNotLike": {
      "aws:PrincipalArn": [
        "arn:aws:iam::*:role/AdminRole",
        "arn:aws:iam::*:user/DevOpsTeamUser"
      ]
    },
    "StringNotEquals": {
      "aws:RequestedRegion": [
        "us-east-1",
        "us-west-2"
      ]
    }
  }
}

Refactored Policy 1: Deny Operations Outside Approved Regions (Deny-Regions.json)

This policy focuses solely on restricting operations to specific AWS regions, which is a common and critical guardrail.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "DenyOperationsOutsideApprovedRegions",
      "Effect": "Deny",
      "NotAction": [
        "s3:GetObject",
        "s3:PutObject",
        "s3:ListBucket",
        "ec2:DescribeRegions",
        "cloudfront:*",
        "iam:*",
        "organizations:*",
        "support:*",
        "account:*"
      ],
      "Resource": "*",
      "Condition": {
        "StringNotEquals": {
          "aws:RequestedRegion": [
            "us-east-1",
            "us-west-2"
          ]
        }
      }
    }
  ]
}

• Why it’s better: Clear intent (region restriction). The NotAction list is still there, but it specifically allows actions that must work globally or cross-region (IAM, Organizations, S3 global objects, CloudFront, etc.) which is a common pattern for region restriction policies.
• Consideration: Carefully curated NotAction is acceptable for region restrictions to avoid breaking global services.

Refactored Policy 2: Deny Dangerous IAM/Organization Actions (Deny-ForbiddenAdminActions.json)

This policy specifically prevents highly destructive actions related to identity and access management or AWS Organizations.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "DenyIAMAccountAndOrgDestructiveActions",
      "Effect": "Deny",
      "Action": [
        "iam:*Delete*",
        "iam:*DetachPolicy",
        "iam:*RemoveRoleFromInstanceProfile",
        "organizations:*Delete*",
        "organizations:*LeaveOrganization",
        "account:DeleteAccount",
        "account:CloseAccount"
      ],
      "Resource": "*"
    }
  ]
}

• Why it’s better: Focused on a single security objective. Uses direct Action statements for clarity.

Refactored Policy 3: Deny Root User Operations (Deny-RootUserAccess.json)

This policy ensures the root user cannot perform any actions, enforcing the principle of least privilege for the root account.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "DenyAllRootUserOperations",
      "Effect": "Deny",
      "Action": "*",
      "Resource": "*",
      "Condition": {
        "StringLike": {
          "aws:PrincipalArn": [
            "arn:aws:iam::*:root"
          ]
        }
      }
    }
  ]
}

• Why it’s better: Simple, clear, and addresses a critical security control.

By breaking down the complex “gem” into these manageable, single-purpose SCPs, your organization gains significant clarity, reduces the risk of unintended consequences, and makes auditing far easier.

Solution 2: Proactive Validation with AWS Config and Conformance Packs

Refactoring existing SCPs is reactive. To prevent new “gems” from appearing and ensure ongoing compliance, implement proactive validation. AWS Config, especially with Conformance Packs, is an excellent tool for this.

Leveraging AWS Config for SCP Governance

AWS Config continuously monitors and records your AWS resource configurations, allowing you to assess, audit, and evaluate the configurations against desired baselines. While AWS Config doesn’t directly analyze the *logic* within an SCP to tell you if it’s “well-written,” it can check for metadata, attachment status, and other properties.

Custom Config Rules for SCPs

You can create custom Config rules to enforce certain best practices for your SCPs. For instance:

• SCP Attachment: Ensure specific SCPs are attached to particular OUs or the Root.
• SCP Naming Conventions: Flag SCPs that don’t adhere to your organizational naming scheme (e.g., must start with “Deny-“).
• SCP Size Limits: Monitor SCPs that exceed a certain size (SCPs have a 5120-character limit), which often indicates complexity.

Here’s an example of a custom Config rule (using an AWS Lambda function as the source) that checks if all SCPs have a specific tag (e.g., Owner) and don’t exceed a certain size:

// Lambda function (Node.js) for a custom Config rule
const AWS = require('aws-sdk');
const organizations = new AWS.Organizations();

exports.handler = async (event, context) => {
    const invokingEvent = JSON.parse(event.invokingEvent);
    const ruleParameters = JSON.parse(event.ruleParameters || '{}');

    const expectedTagKey = ruleParameters.ExpectedTagKey || 'Owner';
    const maxSize = parseInt(ruleParameters.MaxSizeKB || '4', 10) * 1024; // Default 4KB

    let compliance = 'COMPLIANT';
    let annotation = '';

    try {
        const { Policies } = await organizations.listPolicies({ Filter: 'SERVICE_CONTROL_POLICY' }).promise();

        for (const policy of Policies) {
            const { Content } = await organizations.describePolicy({ PolicyId: policy.Id }).promise();
            const policyContent = JSON.stringify(JSON.parse(Content.Content)); // Re-parse to normalize whitespace

            if (policyContent.length > maxSize) {
                compliance = 'NON_COMPLIANT';
                annotation += `Policy '${policy.Name}' (ID: ${policy.Id}) exceeds max size of ${maxSize / 1024}KB (${policyContent.length / 1024}KB). `;
            }

            const tagsResponse = await organizations.listTagsForResource({ ResourceId: policy.Id }).promise();
            const hasOwnerTag = tagsResponse.Tags.some(tag => tag.Key === expectedTagKey);

            if (!hasOwnerTag) {
                compliance = 'NON_COMPLIANT';
                annotation += `Policy '${policy.Name}' (ID: ${policy.Id}) is missing required tag '${expectedTagKey}'. `;
            }
        }
    } catch (err) {
        console.error("Error evaluating policies:", err);
        return {
            complianceType: 'NOT_APPLICABLE',
            annotation: `Error during evaluation: ${err.message}`
        };
    }

    return {
        complianceType: compliance,
        annotation: annotation || 'All SCPs are compliant with tagging and size requirements.'
    };
};

To deploy this, you would create a Lambda function, then create an AWS Config custom rule that triggers this Lambda periodically or on SCP changes.

AWS Conformance Packs

Conformance Packs allow you to easily deploy a collection of AWS Config rules and remediation actions as a single entity. You can define a Conformance Pack that includes custom rules for SCPs, along with other security best practices, and deploy it across your entire AWS Organization. This provides continuous, automated monitoring and reporting on your SCP compliance.

Deploying a Conformance Pack can be done via the AWS Management Console, AWS CLI, or IaC tools (like CloudFormation or Terraform).

# Example AWS CLI command to deploy a Conformance Pack
aws configservice put-conformance-pack \
    --conformance-pack-name "OrganizationalSCPGovernance" \
    --template-s3-uri "s3://your-config-bucket/conformance-pack-templates/scp-governance-pack.yaml" \
    --delivery-s3-bucket "your-config-delivery-bucket"

The scp-governance-pack.yaml would define the custom Config rules and potentially managed rules relevant to your SCPs.

Solution 3: Managing SCPs with Infrastructure as Code (IaC)

The most effective long-term solution for managing SCPs, preventing “gems,” and ensuring consistency is to treat them as Infrastructure as Code. This brings all the benefits of software development to your policy management:

• Version Control: Track changes, roll back to previous versions, and understand who changed what and why.
• Peer Review: All SCP changes go through a review process, catching errors and improving clarity.
• Automated Deployment: Deploy SCPs consistently and reliably across OUs and accounts.
• Modularity: Define SCPs in separate files for better organization and reusability.
• Testing: Implement automated tests (e.g., policy linting, simulated API calls) before deployment.

IaC Tools for SCP Management: CloudFormation vs. Terraform

Both AWS CloudFormation and HashiCorp Terraform are excellent choices for managing SCPs as code. They offer similar capabilities but have different ecosystems and syntax.

Feature	AWS CloudFormation	HashiCorp Terraform
Vendor Lock-in	AWS-specific.	Cloud-agnostic, but AWS provider is specific.
Syntax	YAML or JSON.	HashiCorp Configuration Language (HCL).
State Management	Managed by AWS (event history, stack drift).	Local or remote state (e.g., S3, Terraform Cloud). Crucial for tracking infra.
Modularity	Nesting stacks, custom resources, macros.	Modules (local and remote), workspaces.
Deployment Model	Stack-based, changes are applied transactionally.	Graph-based, calculates dependency tree, applies changes.
Resource Type	AWS::Organizations::Policy, AWS::Organizations::PolicyAttachment	aws_organizations_policy, aws_organizations_policy_attachment

Example: Managing SCPs with Terraform

Terraform is a popular choice for IaC due to its declarative nature and extensive provider ecosystem. Here’s how you might define and attach the “Deny Root User Operations” SCP using Terraform:

# main.tf
resource "aws_organizations_policy" "deny_root_access" {
  name        = "Deny-RootUserOperations"
  description = "Deny all actions for the root user in member accounts."
  content = jsonencode({
    Version = "2012-10-17",
    Statement = [
      {
        "Sid": "DenyAllRootUserOperations",
        "Effect": "Deny",
        "Action": "*",
        "Resource": "*",
        "Condition": {
          "StringLike": {
            "aws:PrincipalArn": [
              "arn:aws:iam::*:root"
            ]
          }
        }
      }
    ]
  })
  type        = "SERVICE_CONTROL_POLICY"
  tags = {
    Environment = "Production"
    ManagedBy   = "Terraform"
  }
}

resource "aws_organizations_policy_attachment" "deny_root_access_attach_to_ou" {
  policy_id = aws_organizations_policy.deny_root_access.id
  target_id = "ou-abcd-efgh" # Replace with your Organizational Unit ID
}

resource "aws_organizations_policy_attachment" "deny_root_access_attach_to_account" {
  policy_id = aws_organizations_policy.deny_root_access.id
  target_id = "123456789012" # Replace with a specific Account ID if needed
}

With this Terraform configuration:

• The SCP definition is in a human-readable and version-controllable file.
• The jsonencode function correctly formats the policy content.
• Attachments to OUs or individual accounts are explicitly managed.
• Tags are applied for better governance and identification.

You would then use terraform plan to review changes and terraform apply to deploy them.

Example: Managing SCPs with AWS CloudFormation

For those invested in the CloudFormation ecosystem, you can achieve the same results.

# scp-root-deny.yaml
AWSTemplateFormatVersion: '2010-09-09'
Description: CloudFormation template for Deny-RootUserOperations SCP

Resources:
  DenyRootUserPolicy:
    Type: AWS::Organizations::Policy
    Properties:
      Content: |
        {
          "Version": "2012-10-17",
          "Statement": [
            {
              "Sid": "DenyAllRootUserOperations",
              "Effect": "Deny",
              "Action": "*",
              "Resource": "*",
              "Condition": {
                "StringLike": {
                  "aws:PrincipalArn": [
                    "arn:aws:iam::*:root"
                  ]
                }
              }
            }
          ]
        }
      Description: Deny all actions for the root user in member accounts.
      Name: Deny-RootUserOperations
      Type: SERVICE_CONTROL_POLICY
      Tags:
        - Key: Environment
          Value: Production
        - Key: ManagedBy
          Value: CloudFormation

  AttachDenyRootUserPolicyToOU:
    Type: AWS::Organizations::PolicyAttachment
    Properties:
      PolicyId: !Ref DenyRootUserPolicy
      TargetId: "ou-abcd-efgh" # Replace with your Organizational Unit ID

  AttachDenyRootUserPolicyToAccount:
    Type: AWS::Organizations::PolicyAttachment
    Properties:
      PolicyId: !Ref DenyRootUserPolicy
      TargetId: "123456789012" # Replace with a specific Account ID if needed

This CloudFormation template achieves the same outcome, defining the SCP and its attachments within a YAML structure. Deploying this would be done via aws cloudformation deploy or the AWS console.

Conclusion: From “Gem” to Gold Standard

Discovering a convoluted SCP in production is a rite of passage for many DevOps engineers. While initially daunting, these “gems” present a valuable opportunity to solidify your organization’s cloud governance and security posture.

By systematically applying the solutions discussed – refactoring for clarity, leveraging AWS Config for proactive validation, and embracing Infrastructure as Code for management – you can transform these problematic policies into reliable, understandable, and securely enforced guardrails. The goal is to move beyond mere compliance to a state of robust, transparent, and easily auditable security, ensuring that future “gems” are only found in museums, not your AWS Organization.

---

👉 Read the original article on TechResolve.blog