DEV Community: ICHINO Kazuaki

Idea for Managing Policies Across Nearly 1,000 AWS Organizations — A Reseller's Perspective

ICHINO Kazuaki — Wed, 29 Apr 2026 03:09:19 +0000

"A Note from the Author"

I work in the Technical Support division of an AWS reseller operating under the AWS Solution Provider Program in Japan. This post is written from that perspective — managing hundreds of AWS Organizations on behalf of our customers and wrestling with the policy sprawl that comes with it.

Japan's IT industry has traditionally relied heavily on outsourcing to external vendors for building and managing cloud infrastructure, rather than developing in-house capabilities. Decision-making can be slow when multiple organizations are involved, and the AWS reseller model itself is a product of this outsourcing culture. Many Japanese companies purchase AWS through resellers rather than directly, which means resellers like us end up provisioning and managing a large number of AWS accounts and Organizations on their behalf.
As a reseller, we face a unique challenge: we need to enforce restrictions across all of these Organizations to protect both ourselves and our customers, but the exact restrictions vary depending on each customer's configuration. This leads to a proliferation of policy variants that are mostly the same but differ in small, critical ways — and keeping them all in sync is a real headache.
This article is based on a presentation I gave at Security-JAWS #40, a community event focused on AWS security in Japan, held on February 12, 2026.
My English writing skills are limited, so I used GenAI (Kiro CLI) to help translate this article from the original Japanese. I hope it reads well — any awkwardness is on me, not the AI.

Hello, everyone. I'm Ichino (@kazzpapa3), a Technical Support Engineer at an AWS partner company. My favorite AWS services are the AWS CLI and AWS CloudTrail. My least favorite (as a support engineer) is AWS Billing — the billing logic is just too convoluted.

Today I want to talk about a challenge we're facing: how to properly manage IAM and SCP policies across nearly 1,000 AWS Organizations.

Background

I introduced myself as a Technical Support Engineer, but I also work in the department that manages the specifications for our resold AWS accounts.

Here's the situation:

We have multiple account provisioning configurations, and there are many small differences between them.
Most of the policy content is shared, but those small differences cause policies to proliferate.
When the shared parts need updating, we have to make the same change in multiple places.

This is the problem I want to address.

Oh, and personally — I think JSON came too early for humanity. More on that later.

Why the Differences Exist

Many people assume that reseller-provided AWS accounts can't use AWS Organizations or AWS Control Tower. That's actually not the case with our company — we do allow customers to use both.

However, as a reseller, there are certain operations we can't permit. We make the management account available to customers but restrict what they can do with it. The exact restrictions depend on the state of each Organization:

Support case access: Some Organizations deny all support:* actions (customers must go through the reseller for support), while others allow customers to file support cases directly.
Root user management: In some Organizations, the reseller manages root user credentials for member accounts; in others, the customer manages them. This changes the IAM conditions in the policy.
Organization-level operations: Leaving the Organization is always denied, but other operations may vary.

These differences exist in both IAM policies and SCPs.

The Scale of the Problem

Nearly 1,000 management accounts doesn't mean 1,000 unique policies — but it does mean several distinct patterns have emerged. When a fundamental AWS update requires a policy change (think re:Invent season with its flood of announcements), we need to update the relevant section across every variant. That review process is painful.

Visually, think of each policy as a series of blocks — one per Sid or Condition. Most blocks are shared (shown in blue), but a few differ (shown in other colors). The challenge is keeping the blue blocks in sync across all variants.

The Building-Block Idea

Since this is AWS, why not think in terms of building blocks? What if we could break policies into modular parts and compose them as needed — like LEGO bricks?

Instead of maintaining each policy as a standalone document, we'd maintain individual components at the smallest reasonable granularity and assemble them at build time.

A conceptual view of the building block components.
Blue blocks represent shared elements, while red and yellow blocks represent the parts that differ between the two policies.

An illustration of how the smaller, modularized blocks are assembled to form the diagram above.
Some elements use different parts, and we also envision embedding different parts into shared components to build larger, coarser-grained modules.

Should We Use IaC?

We considered AWS CDK. You could model PolicyStatement objects as reusable classes or functions, manage shared parts in JSON, and add variant-specific pieces in code.

But there were concerns:

Deploying to ~1,000 accounts means creating ~1,000 stacks and running them all. That's a lot of CloudFormation overhead.
The learning curve for CDK felt disproportionate to the problem we're solving.
We wanted to explore whether a simpler, non-IaC approach could work first.

To be clear: we're not against IaC. We're just exploring alternatives that might better fit our deployment model and team skills.

The Approach: YAML Modules + `yq`

With some help from generative AI for brainstorming, we arrived at a lightweight approach: write policy components in YAML and use the yq command-line tool to assemble them.

Writing Policies in YAML

Each component is a separate YAML file. For example, even the Version declaration gets its own file:

VersionDeclaration.yaml

# Extracted just the IAM JSON policy Version element.
# Needs review if the Version specification is ever revised.
# https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_policies_elements_version.html
Version: "2012-10-17"
Statement:

Here's a module that denies support:* (used when customers must go through the reseller for support):

BasicRestrictionAsResaler.yaml

  - Sid: BasicRestrictionAsResaler
    Effect: Deny
    Action:
      - support:*
      # Summary: Deny all AWS Support actions
      # Relaxable?: No — customers must not file support cases directly

      - supportplans:*
      # Summary: Deny all support plan modifications
      # Relaxable?: No — support plan must not be changed

      - tax:*
      # Summary: Deny all tax setting modifications
      # Relaxable?: No — tax settings must remain configured for Japan

    # Deny on all resources
    Resource: '*'

And here's the variant that allows support:* (for customers who are permitted to file cases directly):

BasicRestrictionAsResalerForResold.yaml

  - Sid: BasicRestrictionAsResalerForResold
    Effect: Deny
    Action:
      - supportplans:*
      # Summary: Deny all support plan modifications
      # Relaxable?: No — support plan must not be changed

      - tax:*
      # Summary: Deny all tax setting modifications
      # Relaxable?: No — tax settings must remain configured for Japan

    # Deny on all resources
    Resource: '*'

Notice the only difference: the first variant includes support:* in the deny list; the second omits it.

Assembling with `yq`

To build a policy that denies support access:

yq eval-all '
  select(fileIndex == 0) |
  .Statement = [
    (load("IAMPolicyRestrictionForCustomer.yaml") | .[0]),
    (load("BasicRestrictionAsResaler.yaml") | .[0]),        # This line differs
    (load("RestrictionToAWSOrganizationsAsResaler.yaml") | .[0])
  ]
' VersionDeclaration.yaml > foo.yaml

To build a policy that allows support access, just swap one module:

yq eval-all '
  select(fileIndex == 0) |
  .Statement = [
    (load("IAMPolicyRestrictionForCustomer.yaml") | .[0]),
    (load("BasicRestrictionAsResalerForResold.yaml") | .[0]),  # This line differs
    (load("RestrictionToAWSOrganizationsAsResaler.yaml") | .[0])
  ]
' VersionDeclaration.yaml > bar.yaml

Here's what yq is doing:

eval-all — operate across multiple files.
select(fileIndex == 0) — use the first file (VersionDeclaration.yaml) as the base.
.Statement = [...] — populate the Statement array by loading each module file and extracting its first element.
Output the assembled YAML.

Converting to JSON

Once assembled, convert to JSON for use as an actual IAM policy:

yq -o=json foo.yaml > foo.json

The result is a standard IAM policy document:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "BasicRestrictionAsResaler",
      "Effect": "Deny",
      "Action": [
        "support:*",
        "supportplans:*",
        "tax:*"
      ],
      "Resource": "*"
    }
  ]
}

(Other statements omitted for brevity.)

Validating the Output

We can validate the generated policy using IAM Access Analyzer:

aws accessanalyzer validate-policy \
  --policy-type IDENTITY_POLICY \
  --policy-document file://foo.json
{
    "findings": []
}

If findings comes back empty, the policy document is syntactically valid. Note that this doesn't verify whether the policy does what you intend — that's a separate review.

What We Learned

The modular approach feels promising. Here's what stood out:

Single source of truth for shared components. When a shared module needs updating, you change it once. Every policy that includes it picks up the change at build time.
YAML enables inline documentation. This was an unexpected but significant benefit. JSON doesn't support comments, so our existing policy documents had no way to explain why a particular action was denied. With YAML, we can annotate each action with its rationale — "Know Why" documentation becomes a natural part of the policy source. (Further proof that JSON came too early for humanity.)
Low barrier to entry. yq is a single binary with a straightforward syntax. No SDK, no runtime, no framework to learn.

JSONC as an alternative

After presenting an earlier version of this talk at JAWS-UG DE&I, someone pointed out that JSONC (JSON with Comments) exists as a specification. That's a fair point — but YAML still wins for us because of the modular composition workflow with yq.

The Remaining Challenge: Deployment

The policy composition problem feels solved, but deployment is still an open question.

Currently, we use bulk-update scripts that overwrite policies across all accounts. This works because:

The accounts are under our management.
Policy names and role names are stable and guaranteed to exist.

We're considering whether CloudFormation StackSets might be a better deployment mechanism, but there's a migration challenge: the existing IAM resources weren't deployed via CloudFormation, so importing them into StackSets without conflicts requires careful planning.

The policy assembly step (YAML modules + yq) is independent of the deployment mechanism. Whether we stick with scripts, move to StackSets, or even use CDK for deployment, the modular source-of-truth approach works either way.

A likely next step is integrating this into a GitHub Actions pipeline: commit a module change → CI assembles all affected policies → validates them with Access Analyzer → produces deployment-ready JSON artifacts.

Closing Thoughts

We chose not to use IaC for this particular problem, but that's not a rejection of IaC in general. Our reasoning:

The resources are under our direct control, with stable naming conventions.
A "nonexistent" state is essentially impossible in our environment.
A simple bulk-update script is more predictable and faster than rolling out ~1,000 CloudFormation stacks.

That said, I'd love to hear if you have a better approach. This is very much a work in progress, and I'm open to suggestions.

Security is job zero.

Thank you for reading.

References

Rethinking What You Need to Do When Your Access Keys Are Compromised

ICHINO Kazuaki — Wed, 29 Apr 2026 01:15:39 +0000

"A Note from the Author"

I work in the Technical Support division of an AWS reseller operating under the AWS Solution Provider Program in Japan. This post is written from that perspective — sitting between our customers and AWS, handling incidents day-to-day.

Japan's IT industry has traditionally relied heavily on outsourcing to external vendors for building and managing cloud infrastructure, rather than developing in-house capabilities. This means that when a security incident like an access key leak occurs, the response often involves coordination across multiple organizations, which can slow down decision-making at a critical moment. Some of the observations in this post reflect that reality.
Also, my English writing skills are limited, so I used GenAI (Kiro CLI) to help translate this article from the original Japanese. I hope it reads well — any awkwardness is on me, not the AI.

Hello, everyone.

I'm Ichino from the Technical Support team, and I'm a big fan of the AWS CLI.

Unintended access key leaks have been around for a long time, and they show no signs of stopping — whether it's accidental commits to public repositories on GitHub or exposure through server-side vulnerabilities.

Today, I want to walk through what you need to do when an access key leak happens and what kind of response is required.

The Typical Notification

When unauthorized use of an access key is suspected, AWS Support notifies the account holder by opening a support case with a message like the one below.¹

Note that AWS does not publicly disclose exactly what behaviors trigger detection (that's internal information), nor can we give a definitive answer on how quickly notifications are sent.

That said, as a colleague of mine has written, my personal impression is that detection tends to be fast when sts:GetCallerIdentity — commonly used as the first move in an attack — is called from an unusual location.
(This is just my gut feeling and not backed by any official data.)

Japanese article / 攻撃の初動としても利用される GetCallerIdentity とは何か

Example Subject Line

[CASE 123456789012345] [Action Required] Unexpected Activity Detected on your AWS Account 123456789012

Example Body

We detected potentially unexpected activity in your AWS account. This activity is related to your AWS access key(s) belonging to User(s) on the account. Please review the detailed list of access key(s) and User(s) in the existing Support Case within your AWS Console.

Refer to the user guide [1] for detailed instructions.

As a security best practice, we recommend that you enable multi-factor authentication (MFA) [2].

Step 1: If your application uses the exposed access key, you need to replace the key. To replace the key, first create a second key (at that point, both keys will be active). Then, modify your application to use the new key.

Next, disable (do not delete) the exposed key by clicking on the "Make inactive" option in the console. If there are any problems with your application, you can reactivate the exposed key. When your application is fully functional using the new key, please delete the exposed access key(s).

To delete IAM user keys, go to [3].
To delete Root user keys, go to [4].

Please note, only rotating and deleting the exposed key may not be sufficient to protect your account, continue to Step 2.

Step 2: Check your CloudTrail log for unwanted activity.

Check your account for any unwanted activity, such as creation of unapproved IAM users and/or associated passwords (login profile), access keys, policies, roles or temporary security credentials by checking your CloudTrail log, and immediately delete them.

To delete IAM users, go to [5].
To delete policies, go to [6].
To delete roles, go to [7].

Please note, deleting IAM users may impact production workloads and should be done carefully.

Step 3: Review your AWS account for any unwanted usage.
Check your account for any unwanted usage, such as EC2 instances, Lambda functions, or EC2 Spot bids by logging into your AWS Management Console and reviewing each service page. You can also do this by checking the "Bills" page in the Billing console [8].

Please note, unwanted usage can occur in any region and your console only displays one region at a time. To switch regions, use the drop-down menu in the top-right corner of the console.

Step 4: Please work with your TAM/Account Manager and respond to the existing Support Case or create a new one [9] to confirm completion of steps 1-3 and apply for a billing adjustment, if applicable. We will work with you to evaluate billing adjustments after the account is secured.

We will provide a list of potentially unexpected resources related to this event through the associated Support Case within the next 2 hour(s). Please review the resources listed for each of your services, and take action to stop, delete, or terminate any that are unwanted. The deeplinks in the attached file should take you to the exact resource page, or you can manually navigate to the resource to investigate. In addition to the resources detailed in the Support Case, it's important that you review all your services and credentials to identify and address any other unwanted usage, as described.

If you need help securing your account, let us know through the Support Case where you can request a phone call or chat session for immediate assistance. Alternatively, if you believe that your account is secured and there is no unwanted access or usage, please contact us immediately via the Support Case to confirm this in writing.

Thank you for your immediate attention to this matter.

[1] https://aws.amazon.com/premiumsupport/knowledge-center/potential-account-compromise/
[2] https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_mfa_enable.html
[3] https://console.aws.amazon.com/iam/home#users
[4] https://console.aws.amazon.com/iam/home#security_credential
[5] https://console.aws.amazon.com/iamv2/home#/users
[6] https://console.aws.amazon.com/iam/home#/policies
[7] https://console.aws.amazon.com/iam/home#/roles
[8] https://console.aws.amazon.com/billing/home#/bill
[9] https://console.aws.amazon.com/support/home?#/

Breaking Down the Required Actions

Step 1

The notification says:

Step 1: If your application uses the exposed access key, you need to replace the key. To replace the key, first create a second key (at that point both keys will be active). Then, modify your application to use the new key.

Next, disable (do not delete) the exposed key by clicking on the "Make inactive" option in the console. If there are any problems with your application, you can reactivate the exposed key. When your application is fully functional using the new key, please delete the exposed access key(s).

The wording can be a bit confusing about whether you should or shouldn't delete the key, but here's the gist:

Ultimately, the compromised access key must be deleted.
- However, your organization may be legitimately using that key in applications or systems, so deleting it immediately could cause a business outage.
Create a new access key and update any applications or systems that were using the compromised one.
- Make sure your systems are configured to use the new key.
Once that's done, deactivate (not delete) the compromised key.
- By deactivating instead of deleting, you leave yourself a rollback path in case something breaks.
- Verify that your applications and systems work correctly with the new key.
After confirming there's no business impact, delete the compromised access key.

The key was originally issued for a purpose — it's being used in some application or system. The point is: keep your legitimate workloads running while discarding the key that fell into the wrong hands.

Step 2

The notification says:

Step 2: Check your CloudTrail log for unwanted activity.

Check your account for any unwanted activity, such as creation of unapproved IAM users and/or associated passwords (login profile), access keys, policies, roles or temporary security credentials by checking your CloudTrail log, and immediately delete them.

This step asks you to investigate whether any unauthorized changes or new resources were created, primarily focusing on IAM.

Here, you'll need to use AWS CloudTrail to investigate, keeping these important points in mind:

By default, CloudTrail records 90 days of management events as "Event history."
Event history is recorded per AWS Region, so you need to explicitly select the region you want to examine.²
AWS has a concept called "global service events," and IAM falls into this category.
Most global service events are logged as having occurred in the US East (N. Virginia) region.

In other words, to review CloudTrail event history for IAM resources, you need to check the US East (N. Virginia) region. ³

The compromised access key details are typically provided in a follow-up message within the support case opened by AWS, in a format like this:

Access Key: AKIAIOSFODNN7EXAMPLE
IAMUser: foouser
Event Name: GetCallerIdentity
Event Time: January 1, 2026 00:00:00 (UTC+00:00)
IP: 203.0.113.39
IP Country/Region: US

Use the timestamp, IAM user name, and access key ID from this information as the primary filters when reviewing CloudTrail event history in the US East (N. Virginia) region.

In recent incidents, attackers commonly delete the login profile (i.e., the console password) of existing IAM users to delay discovery. Rather than continuing to use the leaked key directly, they often create new IAM users with fresh access keys, building out additional IAM resources to carry out their activities. Modifications to existing IAM roles and policies are also frequently observed.

You need to verify all of these actions and confirm that no IAM resources were created, modified, or deleted without the account owner's intent.

In short, review all actions performed using the compromised access key across IAM resources. If new IAM resources were created, follow the chain — examine the activity performed by each newly created resource, then check what those resources created in turn. Repeat this process to uncover every unauthorized IAM resource.

Investigation perspective for IAM resources
You need to follow the chain: actions taken with the leaked key AND actions taken with any newly created keys.

For a complete list of IAM actions, refer to the official documentation below. Focus your review on non-Read operations — particularly Create and Update actions:

Actions, resources, and condition keys for AWS Identity and Access Management (IAM)

AWS CloudTrail Event history page in the Management Console
IAM event history must be checked in the US East (N. Virginia) region.

For detailed instructions on using CloudTrail, see the official documentation:

Viewing recent management events with the console

Step 3

The notification says:

Step 3: Review your AWS account for any unwanted usage. Check your account for any unwanted usage, such as EC2 instances, Lambda functions, or EC2 Spot bids by logging into your AWS Management Console and reviewing each service page. You can also do this by checking the "Bills" page in the Billing console.

Please note, unwanted usage can occur in any region and your console only displays one region at a time.

Unlike Step 2, this step requires you to check a broad range of AWS services — not just IAM — for any unauthorized resource creation, and to delete them as needed.

What Should You Look For?

This is a genuinely difficult question. In a nutshell, it depends on the IAM policies attached to the compromised access key's IAM user.

For example, Amazon SES SMTP users are internally issued as IAM users with access keys (though the notification subject and handling differ when these are abused). If created following the recommended procedure, the key is issued with only the ses:SendRawEmail permission via the AmazonSesSendingAccess inline policy at the time of writing. ⁴ In that case, the attacker can only send emails via Amazon SES. While mass spam distribution is still a serious problem — it degrades your SES sender reputation — it wouldn't lead to a proliferation of resources across many AWS services.

On the other hand, if the compromised key had *FullAccess, PowerUserAccess, or AdministratorAccess attached, the scope of what an attacker can do expands dramatically.

Therefore, you need to consider what the compromised and any newly created access keys are authorized to do. Based on that, you must check all enabled regions for unauthorized resource creation by the malicious actor.

Investigation perspective for Step 3
Here too, you need to check actions from both the leaked key and any newly created keys, across all enabled regions.

Manually reviewing event history across all regions with various filter conditions through the console is extremely tedious.

If you have a CloudTrail trail configured with events stored in an S3 bucket, querying with Athena is very useful. CloudTrail Lake (though new onboarding has been announced as ending) is also helpful. However, if these services weren't enabled when the leak occurred, you'll only have the default 90 days of management event history.

Reviewing default management events through the Management Console is very difficult, so I recommend using the AWS CLI to extract the data and convert it to CSV or another format for easier analysis.

I've put together a sample shell script that makes it easier to extract event history. It works in any environment with aws cli and jq installed — feel free to use it as a reference:

GitHub Repository : research-via-cloudtrail

Step 4

Step 4: Please work with your TAM/Account Manager and respond to the existing Support Case or create a new one to confirm completion of steps 1-3 and apply for a billing adjustment, if applicable.

After completing the verification and remediation described in Steps 1–3, you can apply for a billing adjustment if needed.

If you're using our billing agency service (as is common with AWS resellers in Japan), you cannot reply to or create support cases directly. You'll need to go through us — but you can request a review for billing adjustments related to AWS charges incurred by unauthorized resource creation.

However, neither AWS nor we can guarantee that charges will be reduced. Every case is individually reviewed by the relevant AWS team, taking into account the full circumstances — what happened, the background, and what remediation was performed. The review criteria and process are not disclosed; you simply receive an approval or denial. That said, if the same type of incident has occurred before and a credit was previously granted, a second occurrence is almost always denied.

For high-value unauthorized charges, the review may take considerable time. Even in cases where a credit is ultimately approved, you may need to pay the charges upfront first.

Do not assume that AWS will bail you out if something goes wrong. Think of it as: there's a possibility of relief, but it's not guaranteed. The priority should always be preventing these incidents from happening in the first place.

Closing Thoughts

Working in the Technical Support division — positioned between our customers and AWS — we see a lot of access key compromise incidents.

When an access key is compromised, the notification generally follows the flow we've walked through here. By working through each step methodically, you can address the situation.

However, attacker techniques have become increasingly sophisticated and automated in recent years. If a compromised access key has broad permissions for resource creation, a wide variety of resources can be spun up automatically, racking up charges in the hundreds of thousands of yen (thousands of dollars) in no time.

Leak vectors include accidental commits to public repositories like GitHub, exposed .env files, and other forms of unintended disclosure. Proper key management is critical.

For use cases that traditionally required access keys — such as calling AWS APIs from outside AWS — alternatives like the aws login command and IAM Roles Anywhere are now available. With these options on the table, it's worth revisiting whether your current use of access keys is still necessary.

That said, there are still scenarios where access keys are the only option for external access. In those cases, access keys aren't inherently evil. But following best practices — regular rotation, avoiding overly broad permissions — remains essential.

I usually close these posts with "I hope this helps someone," but this time: I hope you never need this post.

Until next time.

Cases opened by AWS are sometimes referred to as "outbound cases." ↩
CloudTrail Event history provides a viewable, searchable, downloadable, and immutable record of the past 90 days of CloudTrail management events in an AWS Region. (Quoted from the AWS CloudTrail documentation) ↩
For most services, events are recorded in the Region where the action occurred. For global services such as AWS Identity and Access Management (IAM), AWS STS, and Amazon CloudFront, events are delivered to trails that include global services. Most global service events are logged as occurring in the US East (N. Virginia) Region, though some are logged in other regions such as US East (Ohio) or US West (Oregon). (Quoted from the AWS CloudTrail documentation) ↩
https://docs.aws.amazon.com/ses/latest/dg/smtp-credentials.html ↩

DEV Community: ICHINO Kazuaki

Idea for Managing Policies Across Nearly 1,000 AWS Organizations — A Reseller's Perspective

Background

Why the Differences Exist

The Scale of the Problem

The Building-Block Idea

Should We Use IaC?

The Approach: YAML Modules + yq

Writing Policies in YAML

Assembling with yq

Converting to JSON

Validating the Output

What We Learned

JSONC as an alternative

The Remaining Challenge: Deployment

Closing Thoughts

References

Rethinking What You Need to Do When Your Access Keys Are Compromised

The Typical Notification

Example Subject Line

Example Body

Breaking Down the Required Actions

Step 1

Step 2

Step 3

What Should You Look For?

Step 4

Closing Thoughts

The Approach: YAML Modules + `yq`

Assembling with `yq`