DEV Community: Krzysztof Slowinski

1 million Amazon S3 buckets with AWS CDK

Krzysztof Slowinski — Tue, 11 Feb 2025 08:51:41 +0000

With this recent announcement from AWS it is now possible to create up to 1 million Amazon S3 buckets in a single AWS account. Let's look at how it can be achieved using AWS CDK infrastructure as code tool.

Disclaimer: per every additional Amazon S3 bucket above 2,000, there is currently a $0.02 monthly fee (details under Security & buckets), which will result in a $19,960 monthly fee if 1 million Amazon S3 buckets are created in a single AWS account.

Motivation

The amount of data created yearly is accelerating at a rapid pace, with the prediction of exponential growth between 2025 and 2035. This is reflected by the revenue generated by the storage market worldwide. Data storage systems need to adapt to that scale. Cloud computing infrastructure with scalability at its core is an alternative to consider when designing systems for the future.

In a scenario where storage needs are growing, logically grouping the data according to its type or sources could be an additional requirement. Implementing this requirement in AWS can be achieved by keeping each category in a distinct Amazon S3 bucket. An example is a data lake platform, where the architects have decided to place each dataset in a separate Amazon S3 bucket for cost and governance reasons. Thousands of Amazon S3 buckets could be created for large organisations that generate many datasets daily.

Solution

For solutions of this scale, examining organisational best practices is essential, as highlighted in this article. A dedicated platform engineering team could be assigned to concentrate on architecting and implementing complex solutions like the one presented here.

Initial Architecture

The platform engineering team might start by placing all Amazon S3 buckets in top-level AWS CDK stacks. It's a good start. However, the team quickly realises there is an issue with this approach. AWS CDK uses AWS CloudFormation for the deployment. AWS CDK stacks correspond to the CloudFormation stacks. The team discovers a current limit of 500 resources that can be placed in a single AWS CloudFormation stack (details under Resources). This is a hard limit, and it can't be increased by a request to AWS. Aiming to create 1 million resources will result in placing them into 2,000 AWS CloudFormation stacks, each composed of 500 resources with the KodlotS3Stack example below.

It turns out that the 2,000 AWS CloudFormation stacks that the team needs to create with this solution is exactly the limit on the number of stacks that can be created in a single AWS account (details under Stack). In this case, it is a soft limit, and it can be increased by a request to AWS. Still, if the AWS account is to be used for purposes other than storage, creating 2,000 top-level stacks can make finding other stacks difficult. The team starts to explore what alternative options they have to refine the initial architecture.

Refinement 1 - Using Nested Stacks

With the concept of AWS CloudFormation nested stacks (and the corresponding AWS CDK nested stacks), the top-level stack can contain up to 500 nested stacks in it. Each AWS CloudFormation nested stack can contain up to 500 resources, as presented in the KodlotS3NestedStack example below.

With the introduction of the KodlotS3NestedStack definition, the original KodlotS3Stack can now be defined as presented below.

With this refined architecture, one top-level AWS CDK stack contains 500 AWS CDK nested stacks. Each AWS CDK nested stack containing 500 resources results in 250,000 Amazon S3 buckets that can be created in a single top-level stack. The team can solve the original goal by creating four top-level stacks. The result is much simpler management, as the AWS CloudFormation console offers filtering of the nested stacks, which at this point are an implementation detail not interesting in the daily operations of the solution.

However, platform engineers encounter an error when attempting to deploy this solution. The next section describes the resolution step that the team must take.

Refinement 2 - Deployable State

With the top-level stack hierarchy introduced, AWS CloudFormation attempts to create 250,000 AWS resources in parallel. The team finds out that this is 100 times over the hard limit of the maximum number of AWS CloudFormation resources a nested stack can create, update, or delete per operation (details under Nested stacks). To avoid crossing this limit and maintain the current stack structure, platform engineers figured out that the nested stacks need to depend on each other, as presented on the diagram below by the arrows between each nested stack. The dependency is from the AWS CDK stack that needs to depend on another AWS CDK stack and can be added using stack.addDependency(stack) stack API.

With the dependencies indicated between the AWS CDK nested stacks, each nested stack is created and updated sequentially, starting with the first one. Only after the first one is done processing, the second one proceeds, and so on, until the last one. This means 500 resources are created, updated, or deleted at most at any given time, staying within the 2,500 hard limit. The team can now successfully deploy the solution.

Refinement 3 - Production Readiness

The platform engineering team can take the solution one step further to prepare it for real-world scenarios where Amazon S3 buckets will be removed and created regularly. From the original example, imagine a dataset that is no longer needed. In such a case, the corresponding Amazon S3 bucket can be deleted. However, if such a deletion occurs, it creates an empty slot in the AWS CDK stack hierarchy, and it would be optimal to use that slot the next time a new Amazon S3 bucket needs to be created.

To achieve the ability to dynamically discover empty slots in the AWS CDK stack hierarchy, Amazon DynamoDB can be utilised to create a lookup table containing the information on where each Amazon S3 bucket is currently deployed.

Nested Stack	Slot	Bucket Name
KodlotS3NestedStack1	1	KodlotBucket1
KodlotS3NestedStack1	2	KodlotBucket2
KodlotS3NestedStack1	3	KodlotBucket3
...	...	...
KodlotS3NestedStack1	500	KodlotBucket500
...	...	...
KodlotS3NestedStack500	1	KodlotBucket249501
...	...	...
KodlotS3NestedStack500	500	KodlotBucket250000

Assuming that KodlotBucket3 gets decommissioned, the lookup table is going to reflect that by having the item corresponding to composite key Nested Stack=KodlotS3NestedStack1, Slot=3 removed. The team needs to add a script executing after the deployment that updates the lookup table with the current state of the deployed resources. They also adjust the AWS CDK code to use the lookup table to allocate the resources. It makes the infrastructure code more complex and adds a network lookup as a part of the AWS CDK stack synthesis. Without the network connection and valid AWS access keys, platform engineers can't synthesise the stacks with this solution. This makes local development harder, and as the team matures, they start exploring how to address it.

Refinement 4 - Removing the AWS Account Lookup

To overcome the last obstacle of not requiring the connection to the AWS account in the infrastructure code, platform engineers substitute the Amazon DynamoDB lookup table with the AWS SSM parameters. This is because AWS CDK can handle the AWS SSM parameter lookup without the connection to the AWS account.

The team can create a lookup mechanism based on AWS SSM parameters with the following AWS SSM parameters using its hierarchical structure:

kodlot/nestedstack1/slot1/bucketname = KodlotBucket1
kodlot/nestedstack1/slot2/bucketname = KodlotBucket2
kodlot/nestedstack1/slot3/bucketname = KodlotBucket3
...
kodlot/nestedstack1/slot500/bucketname = KodlotBucket500
...
kodlot/nestedstack500/slot1/bucketname = KodlotBucket249501
...
kodlot/nestedstack500/slot500/bucketname = KodlotBucket250000

The platform engineering team discovers that there is a limit to this approach as well. AWS SSM has two types of parameters: standard and advanced. Currently, there can be 10,000 standard and 100,000 advanced parameters created per AWS account and AWS Region (details here). When creating 1 million Amazon S3 buckets, assigning an AWS SSM parameter to each Amazon S3 bucket is impossible. Since AWS SSM parameters allow sizes of 4KB for standard and 8KB for advanced parameters, the solution needs to bundle, e.g. each nested stack slots together into a single parameter, making it harder to manage. This path is for those platform engineers who want to explore this possibility, especially if the requirement is to create more than 100,000 Amazon S3 buckets crossing the limit of the number of AWS SSM parameters. It is worth it, as it will bring valuable lessons for the team.

Conclusions

Engineering scalable infrastructure as code solutions entails various challenges that can be seen as opportunities for learning and considering when architecting similar solutions. Platform engineering teams will face challenges described here and similar ones as they support the continuously growing needs. Through this experience, mature solutions are developed, and the platform engineering teams' problem-solving ability is developed. The creative mindset is key for exploring available options, combined with knowledge of the infrastructure as code tools and cloud services that can support creating the desired outcome. The right investments in ongoing learning for the teams are crucial, topped with the clear support of exploration and treating failures as a feedback loop for the future.

AWS CDK Validation in Python

Krzysztof Slowinski — Mon, 20 Jan 2025 08:08:43 +0000

Validation is a part of the AWS CDK app lifecycle described in the official documentation under the CDK app synthesis section:

All constructs that have implemented the validate method can validate themselves to ensure that they're in a state that will correctly deploy. You will get notified of any validation failures that happen during this phase. Generally, we recommend performing validation as soon as possible (usually as soon as you get some input) and throwing exceptions as early as possible. Performing validation early improves reliability as stack traces will be more accurate and ensures that your code can continue to execute safely.

While there are good articles describing how to handle AWS CDK validation in TypeScript linked below, I couldn't find a corresponding example in Python. After creating a few validations in Python, the following examples complement the ones in the articles.

The following examples we developed using AWS CDK v2.176.0 and Python 3.13.1.

Example Validation

The implementation of the validation is done using the validate method that returns a list of error messages. If the list is empty, there are no errors, and multiple entries correspond to the different validation checks that failed for this validation. The validation logic is encapsulated in a class annotated with @jsii.implements(IValidation), indicating that it is the correct type to be used for the validation. In this particular case, there is a check that the Amazon S3 bucket passed to the validation uses a custom AWS KMS key, giving more control over the encryption process compared to the default encryption used by Amazon S3.

@jsii.implements(IValidation)
class KodlotS3KmsValidator:
    """Checking that the S3 bucket is encrypted with a custom KMS key."""

    def __init__(self, s3_bucket: s3.Bucket):
        self.s3_bucket = s3_bucket

    def validate(self) -> List[str]:
        error_messages: List[str] = []
        if self.s3_bucket.encryption_key is None:
            error_messages.append(f"Bucket must be encrypted with a custom KMS key")
        return error_messages

AWS CDK Construct Example

This is a basic example of an AWS CDK construct adding a single validation using the add_validation method of the construct node in the scope tree. There was a mistake made while implementing, where the custom AWS KMS key was created but not passed to the encryption_key parameter of the Amazon S3 Bucket construct.

class KodlotS3Bucket(Construct):
    """S3 bucket with validations."""

    def __init__(self, scope: Construct, construct_id: str, bucket_name: str) -> None:
        super().__init__(scope, construct_id)

        s3_kms_key = kms.Key(self, f"KodlotS3KmsKey{bucket_name.title()}")
        self.s3_bucket = s3.Bucket(
            self,
            f"KodlotS3{bucket_name.title()}",
            bucket_name=bucket_name,
            encryption_key=None,
        )

        self.node.add_validation(KodlotS3KmsValidator(self.s3_bucket))

This type of mistake could easily be overlooked, and that's where the validation comes in help. Running CDK synthesis or deployment with this example results in the following error:

RuntimeError: Error: Validation failed with the following errors:
  [KodlotS3Stack/KodlotBucket1] Bucket must be encrypted with a custom KMS key

Multiple Validations

There can be multiple validators testing different aspects of our configurations, and they can be reused for different constructs if we find such common patterns. In this example, a validation check if the Amazon S3 bucket was not accidentally configured for static website hosting is introduced, which, in this scenario, would be a mistake.

@jsii.implements(IValidation)
class KodlotS3NotWebsiteValidator:
    """Checking that the S3 bucket is NOT configured for static website hosting."""

    def __init__(self, s3_bucket: s3.Bucket):
        self.s3_bucket = s3_bucket

    def validate(self) -> List[str]:
        error_messages: List[str] = []
        if self.s3_bucket.is_website:
            error_messages.append(f"Bucket must not be used for static website hosting")
        return error_messagesAWS CDK Construct Example Updated

AWS CDK Construct Example Updated

To use the additional validator checking that no website hosting is configured for the Amazon S3 bucket, the example CDK construct is updated, as shown below. This time, the original bug of not passing the custom AWS KMS key is kept. And now, additionally, there is a misconfiguration of passing website_index_document to the Amazon S3 Bucket construct, indicating that it will be used for static website hosting.

class KodlotS3Bucket(Construct):
    """S3 bucket with validations."""

    def __init__(self, scope: Construct, construct_id: str, bucket_name: str) -> None:
        super().__init__(scope, construct_id)

        s3_kms_key = kms.Key(self, f"KodlotS3KmsKey{bucket_name.title()}")
        self.s3_bucket = s3.Bucket(
            self,
            f"KodlotS3{bucket_name.title()}",
            bucket_name=bucket_name,
            encryption_key=None,
            website_index_document="index.html",
        )

        self.node.add_validation(KodlotS3KmsValidator(self.s3_bucket))
        self.node.add_validation(KodlotS3NotWebsiteValidator(self.s3_bucket))

This time, there are two different issues with our Amazon S3 bucket, and as we can see below, both are reported by the validation mechanism. Running CDK synthesis or deployment with this example results in the following errors:

RuntimeError: Error: Validation failed with the following errors:
  [KodlotS3Stack/KodlotBucket1] Bucket must be encrypted with a custom KMS key
  [KodlotS3Stack/KodlotBucket1] Bucket must not be used for static website hosting

This is very powerful, as we can collect the different configuration issues across our cloud infrastructure and address them all without needing to rerun and fix them one by one.

AWS CDK Stack Validation

It is worth remembering that validations are not reserved for AWS CDK constructs and can also be added at an AWS CDK stack level. Here, we can f.e. check if a given stack does not contain unwanted resources. If we have a requirement that our stack should not contain any AWS Lambda resources, then we can model that requirement using the validation presented below. Using the node construct's children, it is possible to check if any resource is an instance of AWS CDK Function using Python's built-in isinstance function.

@jsii.implements(IValidation)
class KodlotNoLambdaInStackValidator:
    """Checking that NO Lambda function was defined in a given stack."""

    def __init__(self, stack: cdk.Stack):
        self.stack = stack

    def validate(self) -> List[str]:
        error_messages: List[str] = []
        for construct in self.stack.node.children:
            if isinstance(construct, _lambda.Function):
                error_messages.append(f"Stack must not contain any Lambda functions")
        return error_messages

AWS CDK Stack Example

To add the validation for the stack, we use the same add_validation method as for the constructs. Since KodlotS3Stack was identified as a stack that must not contain any AWS Lambda functions, it is a good place to test the example stack validation. If a mistake is made during development, such as the one presented below, where the developer defines the AWS Lambda function resource, the validation will take effect.

class KodlotS3Stack(cdk.Stack):
    """Stack containing S3 buckets."""

    def __init__(self, scope: Construct, construct_id: str, **kwargs) -> None:
        super().__init__(scope, construct_id, **kwargs)

        KodlotS3Bucket(self, "KodlotBucket1", bucket_name="kodlot1")

        inline_code = """
def handler(event, context):
    return 200
    """
        _lambda.Function(
            self,
            id="MyLambda",
            runtime=_lambda.Runtime.PYTHON_3_13,
            handler="index.handler",
            code=_lambda.Code.from_inline(inline_code),
        )

        self.node.add_validation(KodlotNoLambdaInStackValidator(self))

Running CDK synthesis or deployment with this example results in the following error:

RuntimeError: Error: Validation failed with the following errors:
  [KodlotS3Stack] Stack must not contain any Lambda functions

Built-in validations

It is also worth checking if AWS CDK does already provide checks for some aspects of our cloud infrastructure. In the context of Amazon S3, as it turns out, if we configure the bucket name of k1, resulting in an illegal name length, there will be the following build-in error:

RuntimeError: Invalid S3 bucket name (value: k1)
Bucket name must be at least 3 and no more than 63 characters

Similarly, if we configure the bucket name of k_1, resulting in an illegal underscore character present in it, there will be the following build-in error:

RuntimeError: Invalid S3 bucket name (value: k_1)
Bucket name must only contain lowercase characters and the symbols, period (.) and dash (-) (offset: 3)

Conclusions

AWS CDK validation is a helpful mechanism for developers to use in conjunction with unit and integration tests to ensure the overall quality of the cloud computing infrastructure. Having the ability to capture common misconfigurations and define standards will result in reusable validations that can be shared across the different components. Validations can be added on different levels of AWS CDK definitions, both for AWS CDK constructs and AWS CDK stacks. It is good to see that AWS CDK is capturing some common validations already, as they are foundational to the various cloud services and resources and should always be checked, thus reducing the chance of facing deployment errors in production.