DEV Community: Dickson

Granting cross-account access in AWS

Dickson — Mon, 31 Mar 2025 13:24:48 +0000

In a modern enterprise architecture, cloud resources are typically managed by individual teams and organized across multiple AWS accounts. However, certain resources are deliberately designated as shared components, such as a centralized API gateway integrated with diverse platforms, or a unified storage solution accessed by various applications. Therefore, it is imperative to carefully design a secure approach to granting such access.

Scenario

Consider that during execution, a Lambda function in one AWS account (Account A with account ID 111111111111) needs to access a S3 bucket located in another AWS account (Account B with account ID 222222222222).

Note: While the example here focuses on accessing a S3 bucket, the concepts discussed later can be applied to all AWS services.

To understand the concept of cross-account access, it is essential to first examine how access control operates within a single AWS account. An IAM user is an entity that interacts with different AWS resources. Permissions can be assigned individually to each user, ensuring the restriction to only the operations or resources necessary for completing specific tasks.

At this stage, one might naturally contemplate setting up an IAM user in Account B as a service account to enable the Lambda function to programmatically access the S3 bucket using its security credentials. However, while technically feasible, this approach relies on long-term access keys, which poses an account security risk and contravenes the security best practices.

In light of this, IAM roles eliminate standard long-term credentials and provide temporary security credentials when being used. An IAM role is similar to an IAM user in that it is also governed by permission policies that define the authorized actions to perform on AWS. Instead of being exclusively associated with one user, a role is intended to be assumable by anyone who needs it.

Building upon the prior architectural blueprint, slight adjustments can be made to incorporate the utilization of an IAM role. Instead of accessing the S3 bucket as an IAM user in Account B, the Lambda function now assumes a role in Account B and accesses the S3 bucket during the role session.

Creating IAM Role in the Destination Account

The IAM role in the destination account should include the permissions required for executing operations on designated resources. Determining the precise permissions can be approached as though the resources are being accessed locally within the same account. There is no need to account for cross-account usage at the moment. In the mentioned scenario, AmazonS3ReadOnlyAccess serves as an appropriate initial permission policy. It may be advisable to further restrict the resources to one or a few specific buckets of interest. The following shows an example permission policy that allows all read operations on a bucket in Account B.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:Get*",
                "s3:List*",
                "s3:Describe*",
                "s3-object-lambda:Get*",
                "s3-object-lambda:List*"
            ],
            "Resource": "*"
        }
    ]
}

Note: The permissions can be tested by utilizing an IAM user within the same account to assume the role.

Once the necessary permissions are properly configured, the trust policy can be configured to allow entities from other AWS accounts to assume this role. The following trust policy allows any entities in Account A to assume this role in Account B.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "sts:AssumeRole",
            "Principal": {
                "AWS": "111111111111"
            }
        }
    ]
}

Configuring Permissions in the Source Account

The destination account now permits entities in the source account to assume a role within it. However, it is still necessary to grant an AssumeRole permission to these entities in the source account so that they can successfully initiate an AssumeRole request. In the context of the above scenario, each Lambda function is assigned an underlying execution role, which provides the necessary permissions for the function to access AWS services and resources. Hence, the following policy should be appended into the Lambda execution role for it to assume the role in Account B.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "sts:AssumeRole",
            "Resource": "*"
        }
    ]
}

Testing for Cross-Account Access

With all configurations completed, the Lambda function code in Account A can now be updated as specified, followed by testing to verify its access to the S3 bucket in Account B.

import boto3

def lambda_handler(event, context):
    # Configure assume role session
    assume_role = boto3.client('sts').assume_role(
        RoleArn='arn:aws:iam::222222222222:role/s3-role',
        RoleSessionName=context.function_name
    )
    session = boto3.session.Session(
        aws_access_key_id=assume_role['Credentials']['AccessKeyId'],
        aws_secret_access_key=assume_role['Credentials']['SecretAccessKey'],
        aws_session_token=assume_role['Credentials']['SessionToken']
    )

    # List all objects in a S3 bucket in another account
    objects = session.client('s3').list_objects(
        Bucket='bucket-in-222222222222',
    )['Contents']
    print(objects)

As indicated in the function logs, the objects variable contains all objects within the bucket bucket-in-222222222222, signifying a successful configuration for cross-account access.

Restricting IAM Policies (Recommended)

As observed, the permissions are either applied universally to all resources or granted to all principals within the AWS account, resulting in a scope that significantly exceeds the actual requirements. In alignment with the principle of least privilege, identities should only be permitted to perform the smallest set of actions necessary to fulfill a specific task.

The following outlines some suggested enhancements based on the example scenario, assuming that the Lambda function is the only entity in Account A requiring access to the bucket bucket-in-222222222222 in Account B.

Restrict the scope of the S3 permissions assigned to the IAM role in Account B to the bucket bucket-in-222222222222.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:Get*",
                "s3:List*",
                "s3:Describe*",
                "s3-object-lambda:Get*",
                "s3-object-lambda:List*"
            ],
            "Resource": "arn:aws:s3:::bucket-in-222222222222"
        }
    ]
}

Restrict the principal scope of the trust policy on the IAM role in Account B to the Lambda execution role in Account A.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "sts:AssumeRole",
            "Principal": {
                "AWS": "arn:aws:iam::111111111111:role/service-role/lambda-execution-role"
            }
        }
    ]
}

Restrict the scope of the AssumeRole permission assigned to the Lambda execution role in Account A to the IAM role in Account B.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "sts:AssumeRole",
            "Resource": "arn:aws:iam::222222222222:role/s3-role"
        }
    ]
}

Depending on your specific use case, you may choose to adopt all or only some of the suggested changes.

This article outlines one method for enabling cross-account access through role assumption. Administrators of the destination account (i.e. Account B) are fully responsible for determining the precise permissions required for a task and configuring them in a role. In contrast, administrators of the source account (Account A) only need to assign the AssumeRole permission to the IAM users or groups involved in cross-account operations. This approach enhances architectural agility, as any changes in permission requirements can be addressed by simply updating the relevant policies in the destination account.

Alternative approaches, such as using resource-based policies, can also address the same issue. It is recommended to evaluate and compare these methods to determine which best aligns with your specific environment and requirements.

References

Establishing a secure connection to the AWS Elastic Beanstalk application

Dickson — Fri, 28 Feb 2025 08:50:51 +0000

Although the Django site can now be accessed publicly with a custom domain name, one may notice a "Not Secure" warning as shown below. This issue stems from the Hypertext Transfer Protocol (HTTP) used between the browser and the application for exchanging information, also known as the client-server communication.

In HTTP, the messages transferred are plaintext, so unauthorized parties can intercept any data sent over the network, which poses a serious security risk especially if users need to submit sensitive data like credit card details. On the contrary, Hypertext Transfer Protocol Secure (HTTPS) combines HTTP requests and responses with SSL and TLS technology. HTTPS websites share an SSL/TLS certificate which contains cryptographic information, so that the data exchanged are encrypted.

AWS Certificate Manager (ACM) helps provision, manage and deploy public and private SSL/TLS certificates. ACM supports a wide range of AWS services including Elastic Beanstalk. ACM also simplifies security management by automating the renewal of expiring certificates.

Requesting a SSL/TLS certificate

In the ACM console, click Request.
Keep the default certificate type and click Next.
Enter the fully qualified domain name, keep other values default, and click Request.

Note: If you intend to use one single certificate for the entire domain, you may enter the root domain (i.e. elasticbeanstalkapp.com) and all subdomains (i.e. *.elasticbeanstalkapp.com) as the fully qualified domain names.
Note down the certificate ARN and click Create records in Route 53.
Keep the default selection and click Create records.

The certificate status should be changed to "Issued" by refreshing the page after a minute or two.

Configuring a secure listener

Create a configuration file named alb-secure-listener.config in the .ebextensions directory with the following contents, and set the value of SSLCertificateArns to the ARN noted in the previous step.

option_settings:
  aws:elbv2:listener:443:
    ListenerEnabled: 'true'
    Protocol: HTTPS
    SSLCertificateArns: arn:aws:acm:us-east-1:123456789012:certificate/add3a42c-fe19-42ec-9fa5-8bd00b9082e0

Re-deploy the Django site.

eb deploy

After the environment update completes, navigating to the Route 53 domain name with HTTPS (i.e. https://www.elasticbeanstalkapp.com) gets the Django site with a secure connection.

Note: The above configuration terminates secure connections at the load balancer and uses HTTP on the backend. If you are required to secure all network connections, follow Configuring end-to-end encryption in a load-balanced Elastic Beanstalk environment to update the configuration of the load balancer.

Having provided a secure connection does not guarantee all users will navigate to the Django site through HTTPS, and hence there is still a potential security loophole when the site is accessed with HTTP. An effective solution would be to set up a redirection rule, ensuring the site is only served under HTTPS.

Redirecting HTTP traffic to HTTPS traffic

Create a configuration file named alb-http-to-https-redirection.config in the .ebextensions directory with the following contents.

Resources:
  AWSEBV2LoadBalancerListener:
    Type: AWS::ElasticLoadBalancingV2::Listener
    Properties:
      LoadBalancerArn:
        Ref: AWSEBV2LoadBalancer
      Port: 80
      Protocol: HTTP
      DefaultActions:
        - Type: redirect
          RedirectConfig:
            Host: "#{host}"
            Path: "/#{path}"
            Port: "443"
            Protocol: "HTTPS"
            Query: "#{query}"
            StatusCode: "HTTP_301"

Re-deploy the Django site.

eb deploy

After the environment update completes, navigating to the Route 53 domain name with HTTP (i.e. http://www.elasticbeanstalkapp.com) results in a redirection to HTTPS (i.e. https://www.elasticbeanstalkapp.com).

It is worth to note the Django site is still accessible by navigating to the Elastic Beanstalk domain name (i.e. https://djangoproj-dev.eba-ttkddb9r.us-east-1.elasticbeanstalk.com). Depending on your requirements, you may want to deny such public access by removing the corresponding domain name from the ALLOWED_HOSTS list in the file named settings.py in the ebdjango directory, or to only allow conditional access by setting up additional listener rules in the load balancer.

References

Creating a custom domain name for the AWS Elastic Beanstalk application

Dickson — Mon, 01 Apr 2024 03:38:47 +0000

Amazon Route 53 is a highly available and scalable Domain Name System (DNS) service. Besides traffic management, Route 53 provides various features such as domain name registration, DNS routing policies and health checks.

Registering a domain

In the Route 53 console, select Registered domains in the navigation pane and click Register domains.
Enter a desired domain name and click Search. From the list of the available domains, choose one and click Select.
Click Proceed to checkout.
Select the duration and/or (un)check the Auto-renew option, and click Next.
Enter the contact information and click Next.
Review the information, accept the terms and conditions, and click Submit.

Note: The registration can take up to 15 minutes. You may receive an email to verify your email address for the domain name, or the domain name will be suspended. If you receive an email about failing to register the domain name, create an Account and billing support case with service Domains and category Registration issue. The issue should be resolved in 2-3 days.

Creating records in Route 53 Hosted Zone

Select Hosted zones in the navigation pane and click the hosted zone created in the previous step.
Click Create record.
Enter www as the subdomain, toggle on the switch for Alias, select Alias to Elastic Beanstalk environment, select the region of the environment, select the environment accordingly, keep other values default, and click Create records.

Updating the Django settings

Append the Route 53 domain name to the ALLOWED_HOSTS list in the file named settings.py in the ebdjango directory, and save the file.

ALLOWED_HOSTS = [
    "djangoproj-dev.eba-ttkddb9r.us-east-1.elasticbeanstalk.com",
    "www.elasticbeanstalkapp.com",
]

Re-deploy the Django site.

eb deploy

After the environment update completes, the Django site is updated. At this stage, navigating to either the Elastic Beanstalk domain name (i.e. http://djangoproj-dev.eba-ttkddb9r.us-east-1.elasticbeanstalk.com) or the Route 53 domain name (i.e. http://www.elasticbeanstalkapp.com) gets the same Django site.

References

Deploying a Django site on AWS Elastic Beanstalk

Dickson — Mon, 01 Apr 2024 03:37:46 +0000

While a Django site can be set up locally with ease, making it accessible everywhere takes a substantial amount of effort. Ranging from infrastructure management to network configurations, these steps are complex, error-prone and time-consuming for developers and businesses, and hence the showstoppers to web application deployment.

AWS Elastic Beanstalk is a managed service for deploying and scaling web applications and services. Apart from capacity provisioning and load balancing, Elastic Beanstalk also provides built-in auto-scaling, monitoring and logging capabilities, enhancing the application reliability and availability.

Prerequisites

Git
Python 3
pip
virtualenv

Configuring the Django project for Elastic Beanstalk

Change into the outer djangoproj directory, if needed.

cd djangoproj
Get the required packages for the project.

pip freeze > requirements.txt
Create a directory named .ebextensions.

mkdir .ebextensions
Create a configuration file named django.config in the .ebextensions directory with the following contents.
```
option_settings:
  aws:elasticbeanstalk:container:python:
    WSGIPath: djangoproj.wsgi:application
```
Note: The above WSGIPath value refers to the application callable in the wsgi.py of the djangoproj project.

The following shows an expected directory structure after the configuration. You may have more directories if you have created or installed other applications.

djangoproj/

    .ebextensions/

        django.config

    djangoproj/

        init.py

        settings.py

        urls.py

        asgi.py

        wsgi.py

    db.sqlite3

    manage.py

    requirements.txt

Installing the EB CLI

Note: It is recommended to complete the installation on a separate terminal so that the changes would not affect the code for the Django project.

Clone the setup scripts.

git clone https://github.com/aws/aws-elastic-beanstalk-cli-setup.git
Install/Upgrade the EB CLI.

Unix: python ./aws-elastic-beanstalk-cli-setup/scripts/ebcli_installer.py
Windows: python .\aws-elastic-beanstalk-cli-setup\scripts\ebcli_installer.py

If the output contains an instruction to add the EB CLI (and Python) executable files to the shell's $PATH variable, follow it and run the command. You also need to restart the opened terminal(s) before running any eb commands there.

Optionally, you can clean up the cloned setup scripts.

Configuring the EB CLI

Run eb init and follow the prompt instructions.

Enter the number that corresponds to the desired region.

Select a default region
1) us-east-1 : US East (N. Virginia)
2) us-west-1 : US West (N. California)
3) us-west-2 : US West (Oregon)
4) eu-west-1 : EU (Ireland)
5) eu-central-1 : EU (Frankfurt)
...
(default is 3):

Enter the access key and secret key.

Note: The credentials are used by the EB CLI for managing Elastic Beanstalk applications. If you do not have the keys, follow Manage access keys to create one.
```
You have not yet set up your credentials or your credentials are incorrect.
You must provide your credentials.
(aws-access-id): 
(aws-secret-key): 
```

Enter an Elastic Beanstalk application name.

Enter Application Name
(default is "djangoproj"):

Enter Y to confirm Python as the platform.

It appears you are using Python. Is this correct?
(Y/n):

Enter 1 to select Python 3.11 as the platform branch.

Select a platform branch.
1) Python 3.11 running on 64bit Amazon Linux 2023
2) Python 3.9 running on 64bit Amazon Linux 2023
3) Python 3.8 running on 64bit Amazon Linux 2
4) Python 3.7 running on 64bit Amazon Linux 2 (Deprecated)
(default is 1):

Ignore the message related to CodeCommit setup.
```
Cannot setup CodeCommit because there is no Source Control setup, continuing with initialization
```
Note: Alternatively, the EB CLI provides integration with Git. For details, please refer to Using the EB CLI with Git.
Enter Y to set up SSH connection to any instances in the Elastic Beanstalk environment.
```
Do you want to set up SSH for your instances?
(Y/n): 
```
Note: This connection allows you to inspect logs or to check other metrics on the instances (to be created in the next step) for easier troubleshooting.
Enter a SSH key pair name and enter a passphrase.
```
Type a keypair name.
(Default is aws-eb): 
```
Note: The EB CLI registers this key pair with the instances and stores the private key in the local folder named .ssh in the user directory.

In the Elastic Beanstalk console, you should see the newly created application is listed on the Applications page.

Deploying the Django site

Create an Elastic Beanstalk environment named djangoproj-dev.

eb create djangoproj-dev

Note: The load-balanced environment is created under the application created in the previous step. The Django site will also be deployed to the this environment. The entire creation takes several minutes.

Follow the next steps only after the environment creation completes.
Get the domain name of the environment.

eb status
```
Environment details for: djangoproj-dev
  Application name: djangoproj
  ...
  CNAME: djangoproj-dev.eba-ttkddb9r.us-east-1.elasticbeanstalk.com
  ...
```
Note: The domain name refers to the above CNAME value.

The domain name can be also obtained from the Environments page.
Append the domain name to the ALLOWED_HOSTS list in the file named settings.py in the ebdjango directory, and save the file.
```
ALLOWED_HOSTS = ["djangoproj-dev.eba-ttkddb9r.us-east-1.elasticbeanstalk.com"]
```
Re-deploy the Django site.

eb deploy

After the environment update completes, the Django site is deployed successfully with Elastic Beanstalk. You will see the following screen after navigating to the domain name above. Alternatively, this can be achieved by running eb open.

Configuring the health check (Optional)

You may notice the Health value is Red from the terminal or Severe from the Elastic Beanstalk console. While this issue should not cause any impacts to your application, this section provides a simple fix.

Create a file named middleware.py in the inner djangoproj directory with the following contents.

from django.http import HttpResponse

class HealthCheckMiddleware:
    def __init__(self, get_response):
        self.get_response = get_response

    def __call__(self, request):
        if request.path == "/_health":
            return HttpResponse("OK")
        return self.get_response(request)

Note: The health check path is set to the root (i.e. "/_health").

Prepend the health check middleware to the MIDDLEWARE list in the file named settings.py in the ebdjango directory, and save the file.

MIDDLEWARE = [
    "djangoproj.middleware.HealthCheckMiddleware",
    ...
    "django.middleware.common.CommonMiddleware",
    ...
]

Create a configuration file named healthcheck.config in the .ebextensions directory with the following contents.

option_settings:
  AWSEBV2LoadBalancerTargetGroup.aws:elasticbeanstalk:environment:process:default:
    HealthCheckPath: /_health

Re-deploy the Django site.

eb deploy

References

Creating a Django application

Dickson — Mon, 01 Apr 2024 03:37:15 +0000

Django is a high-level Python web framework that simplifies the process of web development. It abstracts away some common repetitive tasks such as user authentication, content administration and site maps, allowing developers to focus on application-specific logic. Django also addresses the well-known security concerns, and ensures good performance even under heavy loads.

The benefits brought by Django make it a popular choice for web development across various domains, ranging from web applications and portals, to APIs and microservices. This article will guide through the creation of a basic Django project and application.

Prerequisites

Python 3
pip

Setting up the environment

Create a new virtual environment named djangoenv.

python -m venv ~/.virtualenvs/djangoenv

Note: It is a good idea to keep all the virtual environments in a single folder. In this guide, it is in .virtualenvs/ in the home directory.
Activate the virtual environment.

Unix: source ~/.virtualenvs/djangoenv/bin/activate
Windows: %HOMEPATH%\.virtualenvs\djangoenv\Scripts\activate.bat

Note: To deactivate the virtual environment, simply run deactivate. Remember to reactivate the environment whenever you need to install new packages or start the server.
Install Django.

python -m pip install Django

Creating a Django project

django-admin startproject djangoproj

This command generates a Django project named djangoproj with the following directory structure.

djangoproj/
    djangoproj/
        __init__.py
        settings.py
        urls.py
        asgi.py
        wsgi.py
    db.sqlite3
    manage.py

Starting the development server

Change into the outer djangoproj directory.

cd djangoproj

Run the Django site locally.

python manage.py runserver

The console should show an output similar to the following. To stop the development server, press Ctrl+C.

Watching for file changes with StatReloader
Performing system checks...

System check identified no issues (0 silenced).

You have 18 unapplied migration(s). Your project may not work properly until you apply the migrations for app(s): admin, auth, contenttypes, sessions.
Run 'python manage.py migrate' to apply them.

March 5, 2024 - 00:00:00
Django version 5.0.3, using settings 'djangoproj.settings'
Starting development server at http://127.0.0.1:8000/
Quit the server with CONTROL-C.

Creating a Django application

Note: A project is a collection of configurations and apps for a particular website. An application is responsible for delivering a service, such as a blog system or a database of public records. A project can contain multiple applications; an application can be in multiple projects.

The project is now set up, and it is ready to create new applications by running python manage.py startapp <app_name>.

Follow the tutorial attached in the references to create and customize the views and models that build up the application. Alternatively, check out and install some third-party applications, such as the Django REST framework for building web APIs.

References

AWS SSM Automation for Encrypting RDS Instances

Dickson — Sun, 22 Jan 2023 18:11:55 +0000

Amazon Relational Database Service (Amazon RDS) is a managed database solution providing various engine options such as MySQL, MariaDB and PostgreSQL. With the growing concern on data protection, Amazon RDS supports encryption so that data stored at rest in the underlying storage is encrypted using the keys managed in AWS Key Management Service (KMS).

While an RDS instance can be encrypted at creation, the encryption cannot be enabled after the instance is created. The workaround is to create another encrypted RDS instance, but the process involves a few manual actions which may be error-prone. In this article, we will go through an equivalent automation that leverages AWS Systems Manager (SSM). This creates an encrypted copy of an unencrypted RDS instance.

Note: There are some DB instance classes that do not support Amazon RDS encryption. For details, please refer to https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Overview.Encryption.html#Overview.Encryption.Availability.

Encrypting a RDS instance

Create an unencrypted snapshot of the unencrypted RDS instance.
Create an encrypted copy of the unencrypted snapshot.
Restore an encrypted RDS instance from the encrypted snapshot.

Writing a custom SSM Automation document

A SSM document comprises of some top-level data elements, such as schemaVersion, parameters and mainSteps.

`schemaVersion`

Specifically, Automation documents must use schema version 0.3.

`parameters`

DBInstanceResourceId or DBInstanceIdentifier: Resource ID or identifier of the DB instance to be encrypted
DBSnapshotIdentifier: Identifier for the unencrypted DB snapshot
EncryptedDBSnapshotIdentifier: Identifier for the encrypted DB snapshot
KmsKeyId: ID, ARN or Alias of the AWS KMS Customer Master Key (CMK)
InstanceTags: Tags to be added to the DB instance
SnapshotTags: Tags to be added to the DB snapshot
AutomationAssumeRole: ARN of the role that allows Automation to perform the actions

`mainSteps`

Get the identifier of the unencrypted RDS instance.

This step exists to cater for the integration with AWS Config mentioned in the upcoming Advanced usage section.

import boto3

def handler(event, context):
    resource_id = event["DBInstanceResourceId"]
    instance_id = event["DBInstanceIdentifier"]
    if instance_id == "":
        rds_client = boto3.client("rds")
        instances = rds_client.describe_db_instances(Filters=[{
            "Name": "dbi-resource-id",
            "Values": [resource_id]
        }]).get("DBInstances", [{}])
        instance_id = instances[0].get("DBInstanceIdentifier", "")
    return {"instance_id": instance_id}

Create an unencrypted snapshot of the unencrypted RDS instance.

import json
import re
from datetime import datetime
import boto3

DB_INSTANCE_ID = "DBInstanceIdentifier"
INSTANCE_TAGS = "InstanceTags"
SNAPSHOT_ARN = "DBSnapshotArn"
SNAPSHOT_ID = "DBSnapshotIdentifier"
SNAPSHOT_TAGS = "SnapshotTags"
PH_DATE = "date"
PH_DATETIME = "datetime"
PH_EXECUTION_ID = "execution-id"
PH_INSTANCE_ID = "db-instance-id"
PH_SNAPSHOT_ID = "db-snapshot-id"
PH_TIME = "time"
PH_TAG_VAL_STR = "{{{}}}"
TAG_SHORTCUT_EXPR = "Key=(.+),\s*Value=(.*)"

def parse_tags(tags_str):
    if re.match("({};?)+".format(TAG_SHORTCUT_EXPR), tags_str):
        matches = [re.match(TAG_SHORTCUT_EXPR, t.strip()) for t in tags_str.split(";")]
        return [{"Key": m.group(1), "Value": m.group(2) if m.lastindex > 1 else ""} for m in matches]
    else:
        return json.loads(tags_str)

def build_tags(tag_str, context, tag_vars=None):
    if tag_str == "":
        return []
    placeholders = tag_data(ctx=context, tag_vars=tag_vars)
    tags = parse_tags(tag_str)
    for tag in tags:
        value = tag.get("Value")
        for p in placeholders:
            value = value.replace(PH_TAG_VAL_STR.format(p), str(placeholders[p]))
        tag["Value"] = value
    return tags

def template_string(s, context, str_vars=None):
    result = s
    data = tag_data(ctx=context, tag_vars=str_vars)
    for p in data:
        result = result.replace(PH_TAG_VAL_STR.format(p), str(data[p]))
    return result

def tag_data(ctx, tag_vars):
    def clean(s):
        return s.replace(":", "").replace("-", "").replace("T", "")
    dt = datetime.now().replace(microsecond=0)
    data = {
        PH_DATETIME: clean(dt.isoformat()),
        PH_DATE: clean(dt.date().isoformat()),
        PH_TIME: clean(dt.time().isoformat()),
        PH_EXECUTION_ID: ctx.get("automation:EXECUTION_ID")
    }
    if tag_vars is not None:
        for t in tag_vars:
            data[t] = tag_vars[t]
    return data

def handler(event, context):
    client = boto3.client("rds")
    inst_id = event[DB_INSTANCE_ID]
    snapshot_str = event.get(SNAPSHOT_ID, "").strip()
    if snapshot_str == "":
        snapshot_str = "{db-instance-id}-{datetime}"
    tag_vars = {
        PH_INSTANCE_ID: inst_id,
        SNAPSHOT_ID: ""
    }
    snapshot_id = template_string(snapshot_str, context, tag_vars)
    args = {
        DB_INSTANCE_ID: inst_id,
        SNAPSHOT_ID: snapshot_id
    }
    response = client.create_db_snapshot(**args)
    snapshot_arn = response["DBSnapshot"]["DBSnapshotArn"]

    snapshot_tag_str = event.get(SNAPSHOT_TAGS, "")
    if len(snapshot_tag_str) > 0:
        snapshot_tags = build_tags(snapshot_tag_str, context, tag_vars)
        if len(snapshot_tags) > 0:
            client.add_tags_to_resource(ResourceName=snapshot_arn, Tags=snapshot_tags)

    instance_tag_str = event.get(INSTANCE_TAGS, "")
    if len(instance_tag_str) > 0:
        tag_vars[PH_SNAPSHOT_ID] = snapshot_id
        instance_tags = build_tags(instance_tag_str, context, tag_vars)
        if len(instance_tags) > 0:
            db_arn = ":".join(snapshot_arn.split(":")[0:5]) + ":db:" + inst_id
            client.add_tags_to_resource(ResourceName=db_arn, Tags=instance_tags)
    return {"snapshot_id" : snapshot_id}

Verify that the unencrypted snapshot created in the previous step exists.

This step is necessary because it takes time for the snapshot to become ready for use.

import boto3
import time

rds_client = boto3.client("rds")

def handler(event, context):
    snapshot_id = event["DBSnapshotIdentifier"]
    while True:
        try:
            snapshots = rds_client.describe_db_snapshots(DBSnapshotIdentifier=snapshot_id).get("DBSnapshots", [{}])
            if snapshots[0].get("Status", "") == "available":
                return
            time.sleep(20)
        except Exception as e:
            print(e)
            time.sleep(20)
            pass

Copy the unencrypted snapshot to an encrypted snapshot.

import boto3
from datetime import datetime

def handler(event, context):
    SOURCE_SNAPSHOT_ID = event["DBSnapshotIdentifier"]
    DEST_SNAPSHOT_ID = event["EncryptedDBSnapshotIdentifier"]
    if event["EncryptedDBSnapshotIdentifier"] == "":
        DEST_SNAPSHOT_ID = event["DBSnapshotIdentifier"] + "-encrypted"
    kmskey_id = event["KmsKeyId"]
    if event["KmsKeyId"] == "":
        kmskey_id = "alias/aws/rds"
    client = boto3.client("rds")
    response = client.copy_db_snapshot(
        SourceDBSnapshotIdentifier=SOURCE_SNAPSHOT_ID,
        TargetDBSnapshotIdentifier=DEST_SNAPSHOT_ID,
        KmsKeyId=kmskey_id,
        CopyTags=True,
    )
    snapshot_id = response["DBSnapshot"]["DBSnapshotIdentifier"]
    return {"snapshot_id" : snapshot_id}

Verify that the encrypted snapshot created in the previous step exists.

Similar to step 3, this step is necessary because it takes time for the snapshot to become ready for use.

import boto3
import time

rds_client = boto3.client("rds")

def handler(event, context):
    snapshot_id = event["EncryptedDBSnapshotIdentifier"]
    while True:
        try:
            snapshots = rds_client.describe_db_snapshots(DBSnapshotIdentifier = snapshot_id).get("DBSnapshots", [{}])
            if snapshots[0].get("Status", "") == "available" and snapshots[0].get("Encrypted", False) == True:
                return
            time.sleep(20)
        except Exception as e:
            print(e)
            time.sleep(20)
            pass

Delete the unencrypted snapshot.

This step is optional during the encryption, but doing so helps remove redundant resource and hence saves the cost.

import boto3
import time

rds_client = boto3.client("rds")

def handler(event, context):
    snapshot_id = event["DBSnapshotIdentifier"]
    wait_period = 5
    retries = 5
    while True:
        try:
            rds_client.delete_db_snapshot(DBSnapshotIdentifier=snapshot_id)
            return True
        except Exception as ex:
            # As the list of snapshot is eventually consistent old snapshots might appear in listed snapshots
            if getattr(ex, "response", {}).get("Error", {}).get("Code", "") == "InvalidSnapshot.NotFound":
                return False
            # Throttling might occur when deleting snapshots too fast
            if "throttling" in ex.message.lower():
                retries -= 1
                if retries == 0:
                    raise ex
                time.sleep(wait_period)
                wait_period = min(wait_period + 10 , 30)
                continue
            raise ex

Rename the unencrypted RDS instance.

This step can be achieved by executing the ModifyDBInstance API. The purpose is to allow the encrypted RDS instance to be created with the same identifier as the old unencrypted RDS instance, so that the connection(s) from other clients can be kept unchanged.

Verify that the unencrypted RDS instance is renamed in the previous step.

This step is necessary because it takes time for the DB instance to reflect changes.

import boto3
import time

rds_client = boto3.client("rds")

def handler(event, context):
    instance_id = event["RenamedDBInstanceIdentifier"]
    while True:
        try:
            instances = rds_client.describe_db_instances(DBInstanceIdentifier=instance_id).get("DBInstances", [{}])
            if instances[0].get("DBInstanceStatus", "") == "available":
                return
            time.sleep(20)
        except Exception as e:
            print(e)
            time.sleep(20)
            pass

Restore an encrypted RDS instance from the encrypted snapshot.

This step can be achieved by executing the RestoreDBInstanceFromDBSnapshot API.

Verify that the encrypted RDS instance created in the previous step exists.

This step is necessary because it takes time to create the RDS instance.

import boto3
import time

rds_client = boto3.client("rds")

def handler(event, context):
    instance_id = event["EncryptedDBInstanceIdentifier"]
    while True:
        try:
            instances = rds_client.describe_db_instances(DBInstanceIdentifier = instance_id).get("DBInstances", [{}])
            if instances[0].get("StorageEncrypted", False) == True:
                return
            time.sleep(20)
        except Exception as e:
            print(e)
            time.sleep(20)
            pass

The complete SSM document be found in https://github.com/tiksangng/aws-community-resources/blob/main/config-remediation/rds-encryption/EncryptRDSInstance.yaml. For higher readability and future maintenance, some descriptions are supplemented in every parameter and intermediate step.

Creating a custom SSM Automation document

In the AWS Systems Manager console, select Documents in the navigation pane, click Create document and choose Automation.
For Name, enter EncryptRDSInstance. Switch to the Editor tab, click Edit and replace the content with the above document. Click Create automation.

Executing the SSM Automation

To test for the automation, an unencrypted RDS instance named rds-without-encryption has been prepared in advance.

On the Documents page of the AWS Systems Manager console, switch to the Owned by me tab and click EncryptRDSInstance.
Click Execute automation.
In the Input parameters section, specify the DBInstanceIdentifier and AutomationAssumeRole. Optionally, you can specify other parameters as well. Click Execute.

The execution will be completed after a few minutes. The Overall status should show Success.

From the RDS Management Console, you can also see the original unencrypted DB instance rds-without-encryption is now renamed to rds-without-encryption-unencrypted, and there is a new encrypted DB instance created with the same name rds-without-encryption as the original unencrypted DB instance.

Advanced usage

The above SSM document can serve as the remediation method for the AWS Config rule rds-storage-encrypted. The entire solution can be deployed with an AWS CloudFormation template, which can be found in https://github.com/tiksangng/aws-community-resources/blob/main/config-remediation/rds-encryption/main.yaml.

Note: The remediation is not set to automatic since the deletion of the original unencrypted RDS instance is not included in the automation.

For a step-by-step walkthrough in deploying AWS Config rule and remediation method, please refer to the following article.

AWS Config Auto Remediation for Configuring S3 Lifecycle Rule

Dickson for AWS Community Builders ・ Jan 22 '23

#beginners #discuss #productivity

References

AWS Config Auto Remediation for Configuring S3 Lifecycle Rule

Dickson — Sun, 22 Jan 2023 04:01:03 +0000

Amazon Simple Storage Service (Amazon S3) is a popular storage solution offering high availability, scalability and durability. Enterprises around the world have put petabytes of data on S3 for various use cases, ranging from big data analysis to critical data backup. With the increasing number of objects stored in S3 buckets, storage cost becomes a major concern.

To ensure the objects are stored cost-effectively, S3 lifecycle configuration can be configured in every bucket so that all objects are automatically transitioned, archived or expired after a certain time. In this article, we will go through a solution that leverages AWS Config and AWS Systems Manager (SSM). This guarantees all S3 buckets to have at least one active lifecycle rule.

Note: Lifecycle configuration cannot be set on buckets with MFA delete enabled.

The Config rule evaluates the lifecycle rule configuration of all S3 buckets. If a bucket is found non-compliant, AWS Config applies the remediation using a SSM Automation document.

Creating an AWS Config rule

In the AWS Config console, select Rules in the navigation pane and click Add rule. If you cannot find the button, set up the AWS Config first.
Among the AWS managed rules, choose the s3-lifecycle-policy-check rule and click Next.

Note: There is another similar rule called s3-version-lifecycle-policy-check which checks on versioned S3 buckets.
Leave all the values as default and click Next. Optionally, you can specify the parameters such as targetTransitionDays and targetTransitionStorageClass.
Review all the configurations and click Add rule.

The rule is now created and shows up on the console. If there is any noncompliant resource (i.e. S3 bucket with no lifecycle policy configured), you should see the count on the compliance column.

Creating a custom SSM Automation document

Since there is no existing automation for configuring S3 lifecycle rule, we have to write our own document. The automation will execute both the PutBucketLifecycleConfiguration and GetBucketLifecycleConfiguration APIs.

description: |
  ### Document Name - ConfigureS3BucketLifecycleRule

  ## What does this document do?
  This document is used to create or modify the lifecycle rule configuration for an Amazon S3 bucket.

  ## Input Parameters
  * BucketName: (Required) Name of the S3 bucket (not the ARN).
  * TransitionDays: (Optional) Number of days after creation when objects are transitioned to the specified storage class.
    * Default: 90
  * TransitionStorageClass: (Optional) Storage class to which the object is transitioned.
    * Default: "INTELLIGENT_TIERING"
  * NoncurrentTransitionDays: (Optional) Number of days after becoming noncurrent when objects are transitioned to the specified storage class.
    * Default: 30
  * NoncurrentTransitionStorageClass: (Optional) Storage class to which the noncurrent object is transitioned.
    * Default: "INTELLIGENT_TIERING"
  * AutomationAssumeRole: (Required) ARN of the role that allows Automation to perform the actions.

  ## Output Parameters
  * GetBucketLifecycleConfiguration.Output - JSON formatted response from the GetBucketLifecycleConfiguration API call
schemaVersion: "0.3"
assumeRole: "{{ AutomationAssumeRole }}"
outputs:
  - GetBucketLifecycleConfiguration.Output
parameters:
  BucketName:
    type: String
    description: (Required) Name of the S3 bucket (not the ARN).
    allowedPattern: (?=^.{3,63}$)(?!^(\d+\.)+\d+$)(^(([a-z0-9]|[a-z0-9][a-z0-9\-]*[a-z0-9])\.)*([a-z0-9]|[a-z0-9][a-z0-9\-]*[a-z0-9])$)
  TransitionDays:
    type: Integer
    description: (Optional) Number of days after creation when objects are transitioned to the specified storage class.
    default: 90
  TransitionStorageClass:
    type: String
    description: (Optional) Storage class to which the object is transitioned.
    default: INTELLIGENT_TIERING
    allowedValues:
      - STANDARD_IA
      - INTELLIGENT_TIERING
      - ONEZONE_IA
      - GLACIER
      - GLACIER_IR
      - DEEP_ARCHIVE
  NoncurrentTransitionDays:
    type: Integer
    description: (Optional) Number of days after becoming noncurrent when objects are transitioned to the specified storage class.
    default: 30
  NoncurrentTransitionStorageClass:
    type: String
    description: (Optional) Storage class to which the noncurrent object is transitioned.
    default: INTELLIGENT_TIERING
    allowedValues:
      - STANDARD_IA
      - INTELLIGENT_TIERING
      - ONEZONE_IA
      - GLACIER
      - GLACIER_IR
      - DEEP_ARCHIVE
  AutomationAssumeRole:
    type: String
    description: (Required) ARN of the role that allows Automation to perform the actions.
    allowedPattern: ^arn:(aws[a-zA-Z-]*)?:iam::\d{12}:role/[\w+=,.@-]+
mainSteps:
  - name: PutBucketLifecycleConfiguration
    action: "aws:executeAwsApi"
    description: |
      ## PutBucketLifecycleConfiguration
      Creates or modifies the lifecycle configuration for a S3 Bucket.
    isEnd: false
    inputs:
      Service: s3
      Api: PutBucketLifecycleConfiguration
      Bucket: "{{BucketName}}"
      LifecycleConfiguration:
        Rules:
          - Filter:
              Prefix: ""
            ID: "Default Lifecycle Rule"
            Status: Enabled
            Transitions:
              - Days: "{{ TransitionDays }}"
                StorageClass: "{{ TransitionStorageClass }}"
            NoncurrentVersionTransitions:
              - NoncurrentDays: "{{ NoncurrentTransitionDays }}"
                StorageClass: "{{ NoncurrentTransitionStorageClass }}"
    isCritical: true
    maxAttempts: 2
    timeoutSeconds: 600
  - name: GetBucketLifecycleConfiguration
    action: "aws:executeScript"
    description: |
      ## GetBucketLifecycleConfiguration
      Retrieves the S3 lifecycle configuration for a S3 Bucket.
      ## Outputs
      * Output: JSON formatted response from the GetBucketLifecycleConfiguration API call.
    timeoutSeconds: 600
    isCritical: true
    isEnd: true
    inputs:
      Runtime: python3.6
      Handler: validate_s3_bucket_lifecycle_configuration
      InputPayload:
        Bucket: "{{BucketName}}"
        TransitionDays: "{{ TransitionDays }}"
        TransitionStorageClass: "{{ TransitionStorageClass }}"
        NoncurrentTransitionDays: "{{ NoncurrentTransitionDays }}"
        NoncurrentTransitionStorageClass: "{{ NoncurrentTransitionStorageClass }}"
      Script: |-
        import boto3

        def validate_s3_bucket_lifecycle_configuration(event, context):
            s3_client = boto3.client("s3")
            bucket = event["Bucket"]
            transition_days = event["TransitionDays"]
            transition_storage_class = event["TransitionStorageClass"]
            noncurrent_transition_days = event["NoncurrentTransitionDays"]
            noncurrent_transition_storage_class = event["NoncurrentTransitionStorageClass"]

            output = s3_client.get_bucket_lifecycle_configuration(Bucket=bucket)
            updated_rules = output["Rules"]

            if any(
                any(updated_transition["Days"] == transition_days
                    and updated_transition["StorageClass"] == transition_storage_class
                    for updated_transition in updated_rule["Transitions"])
                and any(updated_noncurrent_transition["NoncurrentDays"] == noncurrent_transition_days
                        and updated_noncurrent_transition["StorageClass"] == noncurrent_transition_storage_class
                        for updated_noncurrent_transition in updated_rule["NoncurrentVersionTransitions"])
                and updated_rule["Status"] == "Enabled"
                for updated_rule in updated_rules
            ):
                return {
                    "output":
                    {
                        "message": "Bucket lifecycle configuration successfully set.",
                        "configuration": updated_rules
                    }
                }
            else:
                info = "CONFIGURATION VALUES DO NOT MATCH WITH PARAMETERS PROVIDED VALUES TransitionDays: {}, TransitionStorageClass: {}, NoncurrentTransitionDays: {}, NoncurrentTransitionStorageClass: {}".format(
                    transition_days,
                    transition_storage_class,
                    noncurrent_transition_days,
                    noncurrent_transition_storage_class
                )
                raise Exception(info)
    outputs:
      - Name: Output
        Selector: $.Payload.output
        Type: StringMap

The same document can also be found in https://github.com/tiksangng/aws-community-resources/blob/main/config-remediation/s3-lifecycle-rule/ConfigureS3BucketLifecycleRule.yaml.

In the AWS Systems Manager console, select Documents in the navigation pane, click Create document and choose Automation.
For Name, enter ConfigureS3BucketLifecycleRule. Switch to the Editor tab, click Edit and replace the content with the above document. Click Create automation.

Creating an IAM role for the SSM automation

In the IAM console, select Roles in the navigation pane and click Create role.
For Use case, choose Systems Manager. Click Next.
Choose AmazonSSMAutomationRole and click Next.
For Role name, enter s3-configure-bucket-lifecycle-rule. Click Create role.
Click View role.
Click Add permissions and choose Create inline policy.

Switch to the JSON tab, replace the content with the below document. Click Review policy.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "s3:PutLifecycleConfiguration",
                "s3:GetLifecycleConfiguration"
            ],
            "Resource": "*",
            "Effect": "Allow"
        }
    ]
}

For Name, enter s3-configure-bucket-lifecycle-rule. Click Create policy.

Setting up an automatic remediation for the AWS Config rule

On the Rules page of the AWS Config console, choose the s3-lifecycle-policy-check rule and click View details.
Click Actions and choose Manage remediation.
For Select remediation method, choose Automatic remediation. For Remediation action details, choose ConfigureS3BucketLifecycleRule. For Resource ID parameter, choose BucketName. For AutomationAssumeRole, enter arn:aws:iam::<<aws-account-id>>:role/s3-configure-bucket-lifecycle-rule. Leave other values as default and click Save changes. Optionally, you can specify the parameters such as TransitionDays and TransitionStorageClass.

The remediation is now configured. If there was any noncompliant resource before, it will become compliant after some time. In the future, all new S3 buckets will also have the lifecycle rule configured upon creation.

Advanced usage

The entire solution can be deployed with an AWS CloudFormation template, which can be found in https://github.com/tiksangng/aws-community-resources/blob/main/config-remediation/s3-lifecycle-rule/main.yaml.

For enterprise use, this template can be deployed into multiple accounts easily using an AWS CloudFormation StackSet.

Note: If the solution is deployed in multiple regions, it is advised to separate the IAM role creation from the template as IAM resources are global constructs.