CloudTrail S3 Glacier archival - It's a trap...

Mitchell Edom — Wed, 06 May 2020 12:11:36 +0000

I was recently asked to implement a lifecycle policy to archive central S3 stored CloudTrail logs to Glacier, which on face value seemed like a reasonable idea...

The environment is reasonably large and the volume of data generated by CloudTrail is not insignificant, existing stats;

>6TB S3 CloudTrail usage
>345,000,000 Files in CloudTrail bucket
>30,000,000 new CloudTrail files delivered monthly

Here are the current approximate costs based on above usage;

~6TB S3 Standard - $156 / month
~30,000,000 put requests - $165 / month
~$321 / month

Not a huge expense but not insignificant... I started looking into pricing for Glacier;

$0.005 per GB
$0.06 per 1,000 Lifecycle Transition requests
$0.025 per GB for Storage of 8KB file metadata per object...
$0.005 per GB for Storage of 32KB index and related metadata per object...

So if I assume that the initial life cycle transition archives 80% of the files, the following new monthly costs would apply;

1.2TB S3 Standard - $32 / month
5TB Glacier storage - $25 / month
2.1TB File meta data (8 * 276,000,000) * 0.025 = $52 "S3 storage overhead"
8.4TB Index meta data (32 * 276,000,000) * 0.005 = $42 "Glacier storage overhead"
30,000,000 transitions per month - (30,000,000 / 1000) * 0.06 = $1,800
~$1951 / month

Plus the sweetener... an upfront cost of $16,560 for initially transitioned files ((276,000,000 / 1000) * 0.06)

This obviously does not make financial sense and would have been a huge mistake to implement a net increase of ~$1630 plus the upfront shocker of $16560, needless to say we did not pursue the option to use Glacier archival for CloudTrail log files. There are other possible creative ways that could be implemented how ever those costs would likely exceed those of the raw S3 storage cost over time.

Disclaimer AWS do not recommend Glacier for archival of small files for exactly these cost implications, though it is something that one could easily fall victim to if implemented in a rush. That said hopefully this post might save someone a potentially major headache.

References

https://aws.amazon.com/s3/pricing/

https://docs.aws.amazon.com/AmazonS3/latest/dev/lifecycle-transition-general-considerations.html

Simplify AWS IAM Role Trust Policies in Organizations

Mitchell Edom — Fri, 20 Mar 2020 04:58:32 +0000

Background

As a company scales out the number of AWS accounts used for different workloads, they may require IAM roles which are able to be assumed by any other account within the organization to perform some action, if you are trusting accounts by adding each account principal to the trust policy you may soon find your self hitting the 2048 character limit.

Solution

Use conditions

When making use of conditional filters and the inherent trust that comes from being within an organization, you can dramatically reduce the size and complexity of policies.

From listing all explicit account IDs

  OrgWideRole:
    Type: AWS::IAM::Role
    Properties:
      RoleName: OrgWideRole
      AssumeRolePolicyDocument:
        Version: '2012-10-17'
        Statement:
          - Effect: Allow
            Principal:
              AWS:
                - '012345678901'
                - '123456789012'
                - '234567890123'
                - '345678901234'
                - '456789012345'
                - '567890123456'
                - '*150 or so more account IDs*'
            Action: sts:AssumeRole

To trusting the organization ID

  OrgWideRole:
    Type: AWS::IAM::Role
    Properties:
      RoleName: OrgWideRole
      AssumeRolePolicyDocument:
        Version: '2012-10-17'
        Statement:
          - Effect: Allow
            Principal:
              AWS:
                - '*'
            Action: sts:AssumeRole
            Condition:
              StringEquals:
                aws:PrincipalOrgID : o-12345abcd

You can change the behavior to limit to accounts associated with a specific OU by using conditional key aws:PrincipalOrgPaths

e.g.

  Condition:
    ForAnyValue:StringEquals:
      aws:PrincipalOrgPaths:
      - o-12345abcd/r-eg123/ou-example123/

Further extension of this is possible by using StringLike operator to trust all children of the OU

e.g.

  Condition:
    ForAnyValue:StringLike:
      aws:PrincipalOrgPaths: 
        - o-12345abcd/r-eg123/ou-example123*

There are more permutations of this possible to fit your needs, though implementing trust via organization membership or OU membership has significantly reduced our trust policy sizes while increasing scalability of the platform.

Useful References

AWS Global Condition Context Keys
IAM and STS Limits