DEV Community: Andrii Shykhov

Static Website on S3 with CloudFront and Route 53

Andrii Shykhov — Thu, 13 Mar 2025 21:02:08 +0000

Introduction

This project demonstrates how AWS services like S3, CloudFront, and Route 53 can be used to host a static website. CloudFormation automates the infrastructure provisioning. The example site has a simple structure with index.html and error.html files.

Cost Estimation

The cost of this solution depends on usage, but approximate monthly expenses include:

S3 Storage: $0.023/GB.
CloudFront Data Transfer: $0.085/GB for North America and Europe.
Route 53 Domain Registration: ~$12 per year (varies by domain TLD).
Route 53 Hosted Zone: $0.50 per month for the first 25 hosted zones.
CloudFront Requests: $0.0075 per 10,000 HTTP/HTTPS requests.

For a small static site with low traffic, the cost can be as low as a few dollars per month.

Advantages of CloudFront and Route 53 Over Direct S3 Access

Better Performance: CloudFront caches content in edge locations globally, reducing latency for users.
Security: S3 bucket access can be restricted to CloudFront, preventing direct public access.
Custom Domain: Route 53 allows seamless domain management instead of using an S3 bucket URL.
HTTPS Support: CloudFront provides free SSL/TLS certificates via ACM, securing website traffic.
Reduced Costs: CloudFront reduces data transfer costs compared to direct S3 access.
Error Handling: Custom error pages can be displayed instead of raw S3 errors.

Project structure with infrastructure schema and site example

Here is configuration in infrastructure/root.yaml CloudFormation template:

    AWSTemplateFormatVersion: '2010-09-09'
    Description: CFN template for S3 Static Website with CloudFront and Route53

    Parameters:
      HostedZoneName:
        Type: String
        Default: ''
      HostedZoneId:
        Type: String
        Default: ''
      AcmCertificateArn:
        Type: String
        Default: ''

    Resources:
      StaticWebsiteBucket:
        Type: 'AWS::S3::Bucket'
        Properties:
          BucketName: !Sub 'static-website-${AWS::AccountId}'
          WebsiteConfiguration:
            IndexDocument: index.html
            ErrorDocument: error.html

      BucketPolicy:
        Type: 'AWS::S3::BucketPolicy'
        Properties:
          Bucket: !Ref StaticWebsiteBucket
          PolicyDocument:
            Statement:
              - Action: 's3:GetObject'
                Effect: Allow
                Resource: !Sub 'arn:${AWS::Partition}:s3:::${StaticWebsiteBucket}/*'
                Principal:
                  Service: 'cloudfront.amazonaws.com'

      OriginAccessControl:
        Type: 'AWS::CloudFront::OriginAccessControl'
        Properties:
          OriginAccessControlConfig:
            Name: 'OAC-StaticWebsite'
            OriginAccessControlOriginType: 's3'
            SigningBehavior: 'always'
            SigningProtocol: 'sigv4'

      CloudFrontDistribution:
        Type: 'AWS::CloudFront::Distribution'
        Properties:
          DistributionConfig:
            Enabled: true
            DefaultRootObject: index.html
            Aliases:
              - !Ref HostedZoneName
            Origins:
              - Id: S3Origin
                DomainName: !GetAtt StaticWebsiteBucket.RegionalDomainName
                OriginAccessControlId: !Ref OriginAccessControl
                S3OriginConfig:
                  OriginAccessIdentity: ''
            DefaultCacheBehavior:
              TargetOriginId: S3Origin
              ViewerProtocolPolicy: 'redirect-to-https'
              AllowedMethods: ['GET', 'HEAD']
              CachedMethods: ['GET', 'HEAD']
              ForwardedValues:
                QueryString: false
                Cookies:
                  Forward: none
            CustomErrorResponses:
              - ErrorCode: 403
                ResponsePagePath: '/error.html'
                ResponseCode: 200
              - ErrorCode: 404
                ResponsePagePath: '/error.html'
                ResponseCode: 200
            ViewerCertificate:
              AcmCertificateArn: !Ref AcmCertificateArn
              SslSupportMethod: 'sni-only'
            PriceClass: PriceClass_100

      Route53RecordSet:
        Type: 'AWS::Route53::RecordSet'
        Properties:
          HostedZoneId: !Ref HostedZoneId
          Name: !Ref HostedZoneName
          Type: 'A'
          AliasTarget:
            DNSName: !GetAtt CloudFrontDistribution.DomainName
            HostedZoneId: 'Z2FDTNDATAQYW2' # the hosted zone ID applicable only for CloudFront

    Outputs:
      CloudFrontURL:
        Description: 'CloudFront Distribution URL'
        Value: !Sub 'https://${CloudFrontDistribution.DomainName}'

Infrastructure schema

Site example

Site error handling

Prerequisites

Ensure the following prerequisites are in place:

An AWS account with sufficient permissions to create and manage resources.
The AWS CLI installed on the local machine.
A registered domain in Route 53.

Deployment

1.Clone the repository.

    git clone git@gitlab.com:Andr1500/s3_static_website.git

2.Create ACM certificate.

    aws acm request-certificate \
        --domain-name 'yourdomain.com' \
        --validation-method DNS \
        --idempotency-token 'cert_request' --region us-east-1

3.Fill in all necessary Parameters in infrastructure/root.yaml CloudFormation template, and create the CloudFormation stack.

    aws acm list-certificates --region us-east-1

    aws route53 list-hosted-zones-by-name --dns-name 'yourdomain.com' --query 'HostedZones[0].Id' --output text

    aws cloudformation create-stack \
        --stack-name static-website \
        --template-body file://infrastructure/root.yaml \
        --capabilities CAPABILITY_NAMED_IAM \
        --disable-rollback

4.Retrieve outputs of the CloudFormation stack.

    aws cloudformation describe-stacks \
        --stack-name static-website \
        --query "Stacks[0].Outputs" --output json

5.Upload static website files.

    export BUCKET_NAME=<Bucket name>

    aws s3 cp site_files s3://$BUCKET_NAME/ --recursive

6.Update files to the S3 bucket and invalidate CloudFormation cache.

    aws s3 sync site_files s3://$BUCKET_NAME

    aws cloudfront list-distributions --query "DistributionList.Items[*].[Id,DomainName]"

    aws cloudfront create-invalidation --distribution-id YOUR_DISTRIBUTION_ID --paths "/*"

7.Cleanup.

    aws s3 rm s3://$BUCKET_NAME --recursive

    aws cloudformation delete-stack --stack-name static-website

    aws acm delete-certificate --certificate-arn <certificate_arn>

Conclusion

By leveraging AWS services like S3, CloudFront, and Route 53, a static website can be deployed and accessed via a custom domain. This solution provides a scalable, cost-effective, and secure static site hosting infrastructure without the complexity of traditional web servers.

If you found this post helpful and interesting, please click the reaction button below to show your support. Feel free to use and share this post. 🙂

CloudFormation policy compliance monitoring: leveraging CloudTrail and Athena

Andrii Shykhov — Thu, 13 Mar 2025 20:53:08 +0000

Introduction:

This post details an implementation that monitors who or what changed AWS resources created by CloudFormation. By leveraging AWS Config, CloudTrail, Athena, and Lambda, changes can be tracked, logs can be analysed, and compliance reporting can be automated. The collected data is stored in Amazon S3, making it accessible for audits and compliance verification.

About the Project:

This post builds upon my earlier article on monitoring CloudFormation stack drift using AWS Config Rules. In this enhanced version, the monitoring capabilities are extended by:

Tracking user activities affecting CloudFormation resources.
Logging change details to Amazon S3 via CloudTrail.
Processing and querying logs using AWS Athena.
Automating remediation through AWS Systems Manager and Lambda.

Core Components:
AWS Config Rule: Monitors stacks for drift using CLOUDFORMATION_STACK_DRIFT_DETECTION_CHECK.
Systems Manager Automation Runbook: Invokes a Lambda function for compliance checks.
Remediation Action: Executes the InvokeLambdaFromConfig automation document.
Amazon S3 Bucket: Stores logs from CloudTrail and Athena query results.
Athena Table: Organises and queries raw log data.
CloudTrail Trail: Captures AWS API activity logs.
Lambda Function: Extracts CloudFormation resource names and queries Athena for recent changes.
Infrastructure Scema:

Configuration in infrastructure/monitoring_stack_cloudtrail.yaml CloudFormation template:

    AWSTemplateFormatVersion: '2010-09-09'
    Description: CloudTrail setup for monitoring CFN stack modifications

    Parameters:
      AthenaDatabaseName:
        Type: String
        Description: Athena database name for running queries
        Default: 'cloudtrail_logs'
      StackNameToMonitor:
        Type: String
        Description: CloudFormation stack name to monitor
        Default: 'base-infrastructure'
      MaximumExecutionFrequency:
        Type: String
        Description: The maximum frequency with which drift in CloudFormation stacks need to be evaluated
        Default: 'One_Hour'

    Resources:
    #################################
    # CloudTrail and Athena
    #################################
      CloudTrailLogsBucket:
        Type: AWS::S3::Bucket
        Properties:
          BucketName: !Sub "aws-cloudtrail-logs-${AWS::AccountId}"
          VersioningConfiguration:
            Status: Enabled
          LifecycleConfiguration:
            Rules:
              - Id: ExpireLogs
                Status: Enabled
                ExpirationInDays: 365

      CloudTrailLogsBucketPolicy:
        Type: AWS::S3::BucketPolicy
        Properties:
          Bucket: !Ref CloudTrailLogsBucket
          PolicyDocument:
            Version: "2012-10-17"
            Statement:
              - Sid: "AWSCloudTrailAclCheck"
                Effect: Allow
                Principal:
                  Service: cloudtrail.amazonaws.com
                Action: s3:GetBucketAcl
                Resource: !Sub "arn:${AWS::Partition}:s3:::aws-cloudtrail-logs-${AWS::AccountId}"
                Condition:
                  StringEquals:
                    AWS:SourceArn: !Sub "arn:${AWS::Partition}:cloudtrail:${AWS::Region}:${AWS::AccountId}:trail/monitoring-cfn-policy-compliance"
              - Sid: "AWSCloudTrailWrite"
                Effect: Allow
                Principal:
                  Service: cloudtrail.amazonaws.com
                Action: s3:PutObject
                Resource: !Sub "arn:${AWS::Partition}:s3:::aws-cloudtrail-logs-${AWS::AccountId}/AWSLogs/${AWS::AccountId}/*"
                Condition:
                  StringEquals:
                    AWS:SourceArn: !Sub "arn:${AWS::Partition}:cloudtrail:${AWS::Region}:${AWS::AccountId}:trail/monitoring-cfn-policy-compliance"
                    s3:x-amz-acl: "bucket-owner-full-control"
              - Sid: "AthenaQueryResultPutObject"
                Effect: Allow
                Principal:
                  Service: athena.amazonaws.com
                Action: s3:PutObject
                Resource: !Sub "arn:${AWS::Partition}:s3:::aws-cloudtrail-logs-${AWS::AccountId}/athena-results/*"
                Condition:
                  StringEquals:
                    aws:SourceArn: !Sub "arn:${AWS::Partition}:athena:${AWS::Region}:${AWS::AccountId}:workgroup/primary"

      CloudTrail:
        Type: AWS::CloudTrail::Trail
        Properties:
          TrailName: monitoring-cfn-policy-compliance
          S3BucketName: !Ref CloudTrailLogsBucket
          IncludeGlobalServiceEvents: true
          IsMultiRegionTrail: true
          EnableLogFileValidation: false
          IsOrganizationTrail: false
          IsLogging: true

      AthenaDatabase:
        Type: AWS::Glue::Database
        Properties:
          CatalogId: !Ref AWS::AccountId
          DatabaseInput:
            Name: !Ref AthenaDatabaseName

      AthenaTable:
        Type: AWS::Glue::Table
        Properties:
          CatalogId: !Ref AWS::AccountId
          DatabaseName: !Ref AthenaDatabase
          TableInput:
            Name: !Sub "aws_cloudtrail_logs_${AWS::AccountId}"
            TableType: EXTERNAL_TABLE
            Parameters:
              classification: cloudtrail
            StorageDescriptor:
              Columns:
                - Name: eventVersion
                  Type: string
                - Name: userIdentity
                  Type: struct<type:string,principalId:string,arn:string,accountId:string,invokedBy:string,accessKeyId:string,userName:string,sessionContext:struct<attributes:struct<mfaAuthenticated:string,creationDate:string>,sessionIssuer:struct<type:string,principalId:string,arn:string,accountId:string,username:string>,ec2RoleDelivery:string,webIdFederationData:struct<federatedProvider:string,attributes:map<string,string>>>>
                - Name: eventTime
                  Type: string
                - Name: eventSource
                  Type: string
                - Name: eventName
                  Type: string
                - Name: awsRegion
                  Type: string
                - Name: sourceIpAddress
                  Type: string
                - Name: userAgent
                  Type: string
                - Name: errorCode
                  Type: string
                - Name: errorMessage
                  Type: string
                - Name: requestParameters
                  Type: string
                - Name: responseElements
                  Type: string
                - Name: additionalEventData
                  Type: string
                - Name: requestId
                  Type: string
                - Name: eventId
                  Type: string
                - Name: resources
                  Type: array<struct<arn:string,accountId:string,type:string>>
                - Name: eventType
                  Type: string
                - Name: apiVersion
                  Type: string
                - Name: readOnly
                  Type: string
                - Name: recipientAccountId
                  Type: string
                - Name: serviceEventDetails
                  Type: string
                - Name: sharedEventID
                  Type: string
                - Name: vpcEndpointId
                  Type: string
                - Name: tlsDetails
                  Type: struct<tlsVersion:string,cipherSuite:string,clientProvidedHostHeader:string>
              Location: !Sub "s3://aws-cloudtrail-logs-${AWS::AccountId}/AWSLogs/${AWS::AccountId}/CloudTrail/"
              InputFormat: com.amazon.emr.cloudtrail.CloudTrailInputFormat
              OutputFormat: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
              SerdeInfo:
                SerializationLibrary: org.apache.hive.hcatalog.data.JsonSerDe

    #################################
    # Lambda function
    #################################
      LambdaExecutionRole:
        Type: AWS::IAM::Role
        Properties:
          RoleName: LambdaAthenaQueryExecutionRole
          AssumeRolePolicyDocument:
            Version: '2012-10-17'
            Statement:
              - Effect: Allow
                Principal:
                  Service: 
                    - lambda.amazonaws.com
                    - athena.amazonaws.com
                Action:
                  - sts:AssumeRole
          Policies:
            - PolicyName: CloudFormationDescribe
              PolicyDocument:
                Version: "2012-10-17"
                Statement:
                  - Effect: Allow
                    Action:
                      - cloudformation:DescribeStackResources
                    Resource: "arn:aws:cloudformation:*"
            - PolicyName: AthenaQueryPolicy
              PolicyDocument:
                Version: "2012-10-17"
                Statement:
                  - Effect: Allow
                    Action:
                    - athena:StartQueryExecution
                    - athena:GetQueryExecution
                    - athena:GetQueryResults
                    - athena:GetWorkGroup
                    - athena:GetDataCatalog
                    - athena:GetTableMetadata
                    - glue:GetDatabase
                    - glue:GetTable
                    - glue:GetPartitions
                    Resource: "*"
                  - Effect: Allow
                    Action:
                    - s3:PutObject
                    - s3:GetObject
                    - s3:ListBucket
                    - s3:GetBucketLocation
                    - s3:PutObjectAcl
                    Resource: 
                      - !Sub "arn:${AWS::Partition}:s3:::${CloudTrailLogsBucket}"
                      - !Sub "arn:${AWS::Partition}:s3:::${CloudTrailLogsBucket}/*"
                  - Effect: Allow
                    Action:
                      - lambda:AddPermission
                    Resource: "*"
                  - Effect: Allow
                    Action:
                      - logs:CreateLogGroup
                      - logs:CreateLogStream
                      - logs:PutLogEvents
                    Resource: "*"

      CheckCloudTrailLogsLambda:
        Type: AWS::Lambda::Function
        Properties:
          FunctionName: CheckCloudTrailLogsLambda
          Runtime: nodejs22.x
          Handler: index.handler
          Role: !GetAtt LambdaExecutionRole.Arn
          Timeout: 120
          MemorySize: 256
          Environment:
            Variables:
              STACKS_TO_MONITOR: !Ref StackNameToMonitor
              ATHENA_DATABASE: !Ref AthenaDatabase
              ATHENA_TABLE: !Ref AthenaTable
              S3_OUTPUT_BUCKET: !Ref CloudTrailLogsBucket
          Code:
            ZipFile: |
              const { AthenaClient, StartQueryExecutionCommand } = require("@aws-sdk/client-athena");
              const { CloudFormationClient, DescribeStackResourcesCommand } = require("@aws-sdk/client-cloudformation");

              const athena = new AthenaClient({});
              const cloudformation = new CloudFormationClient({});

              exports.handler = async (event) => {
                  console.log("Event received:", JSON.stringify(event, null, 2));

                  const stacks = process.env.STACKS_TO_MONITOR.split(",");
                  const tableName = process.env.ATHENA_TABLE;
                  const databaseName = process.env.ATHENA_DATABASE;
                  const s3Bucket = process.env.S3_OUTPUT_BUCKET;

                  let resourceNames = [];

                  // Extract resource names from the CloudFormation stacks
                  for (const stack of stacks) {
                      const stackResources = await cloudformation.send(
                          new DescribeStackResourcesCommand({ StackName: stack })
                      );

                      stackResources.StackResources.forEach(resource => {
                          if (resource.PhysicalResourceId) {
                              resourceNames.push(resource.PhysicalResourceId);
                          }
                      });
                  }

                  // Construct Athena query
                  let whereClause = resourceNames.map(name => `resource.arn LIKE '%${name}%'`).join(" OR ");
                  let queryString = `
                      SELECT 
                          userIdentity.userName AS username,
                          eventName AS action,
                          eventTime AS timestamp,
                          resource.arn AS resource_arn,
                          sourceIPAddress AS request_source,
                          userAgent AS user_agent
                      FROM ${tableName}
                      CROSS JOIN UNNEST(resources) AS t(resource)
                      WHERE (${whereClause})
                      AND eventName IS NOT NULL
                      AND userIdentity.userName IS NOT NULL
                      AND from_iso8601_timestamp(eventTime) >= current_timestamp - INTERVAL '1' HOUR
                      ORDER BY from_iso8601_timestamp(eventTime) DESC;
                  `;

                  // Run the Athena query
                  const params = {
                      QueryString: queryString,
                      QueryExecutionContext: { Database: databaseName },
                      ResultConfiguration: { OutputLocation: `s3://${s3Bucket}/athena-results/` }
                  };

                  try {
                      const command = new StartQueryExecutionCommand(params);
                      const queryExecution = await athena.send(command);
                      console.log("Query started:", queryExecution.QueryExecutionId);
                      return { status: "Query started successfully", queryExecutionId: queryExecution.QueryExecutionId };
                  } catch (error) {
                      console.error("Error running query:", error);
                      throw error;
                  }
              };

      LambdaPermissionForConfig:
        Type: AWS::Lambda::Permission
        Properties:
          FunctionName: !Ref CheckCloudTrailLogsLambda
          Action: lambda:InvokeFunction
          Principal: config.amazonaws.com

    #################################
    # Config Rule
    #################################
      IamRoleForConfig2:
        Type: AWS::IAM::Role
        Properties:
          RoleName: CfnDriftDetectionForCloudTrail
          Description: IAM role for AWS Config to access CloudFormation drift detection
          AssumeRolePolicyDocument:
            Version: '2012-10-17'
            Statement:
              - Effect: Allow
                Principal:
                  Service: config.amazonaws.com
                Action:
                  - sts:AssumeRole
          ManagedPolicyArns:
            - arn:aws:iam::aws:policy/ReadOnlyAccess
          Policies:
            - PolicyName: CloudFormationDriftDetectionpolicy
              PolicyDocument:
                Version: "2012-10-17"
                Statement:
                  - Effect: Allow
                    Action:
                      - cloudformation:DetectStackResourceDrift
                      - cloudformation:DetectStackDrift
                      - cloudformation:DescribeStacks
                      - cloudformation:DescribeStackResources
                      - cloudformation:BatchDescribeTypeConfigurations
                      - cloudformation:DescribeStackResourceDrifts
                      - cloudformation:DescribeStackDriftDetectionStatus
                    Resource: !Sub "arn:${AWS::Partition}:cloudformation:${AWS::Region}:${AWS::AccountId}:*"

      ConfigRuleCheckCloudTralLogs:
        DependsOn:
        - LambdaPermissionForConfig
        Type: AWS::Config::ConfigRule
        Properties:
          ConfigRuleName: ConfigRuleCheckCloudTrailLogs
          Description: AWS Config rule to detect drift in CFN stacks and check CloudTrail logs
          Scope:
            TagKey: stack-name
            TagValue: !Ref StackNameToMonitor
          Source:
            Owner: AWS
            SourceIdentifier: CLOUDFORMATION_STACK_DRIFT_DETECTION_CHECK
          MaximumExecutionFrequency: !Ref MaximumExecutionFrequency
          InputParameters:
            cloudformationRoleArn: !GetAtt IamRoleForConfig2.Arn

      IamRoleForRemediation:
        Type: AWS::IAM::Role
        Properties:
          RoleName: AwsConfigRemediationActionInvokeLambda
          Description: IAM role for AWS Config remediation action to invoke Lambda function
          AssumeRolePolicyDocument:
            Version: '2012-10-17'
            Statement:
              - Effect: Allow
                Principal:
                  Service:
                    - config.amazonaws.com
                    - ssm.amazonaws.com
                Action:
                  - sts:AssumeRole
          Policies:
            - PolicyName: InvokeLambdaPolicy
              PolicyDocument:
                Version: '2012-10-17'
                Statement:
                  - Effect: Allow
                    Action:
                      - lambda:InvokeFunction
                    Resource: !GetAtt CheckCloudTrailLogsLambda.Arn

      SsmDocumentInvokeLambda:
        Type: AWS::SSM::Document
        Properties:
          DocumentType: Automation
          Name: InvokeLambdaFromConfig
          Content:
            schemaVersion: "0.3"
            description: "SSM Automation document to invoke a Lambda function"
            parameters:
              AutomationAssumeRole:
                type: String
                description: (Optional) The ARN of the role that allows Automation to perform the actions
                default: !GetAtt IamRoleForRemediation.Arn
            mainSteps:
              - name: InvokeLambda
                action: aws:invokeLambdaFunction
                inputs:
                  FunctionName: !Ref CheckCloudTrailLogsLambda
                  Payload: '{}'
                  InvocationType: Event
                  LogType: None
                maxAttempts: 2
                timeoutSeconds: 30
                onFailure: Abort
                isCritical: true
            assumeRole: !GetAtt IamRoleForRemediation.Arn

      RemediationActionInvokeLambda:
        Type: AWS::Config::RemediationConfiguration
        Properties:
          ConfigRuleName: !Ref ConfigRuleCheckCloudTralLogs
          TargetType: SSM_DOCUMENT
          TargetId: !Ref SsmDocumentInvokeLambda
          Automatic: true
          MaximumAutomaticAttempts: 2
          RetryAttemptSeconds: 30
          Parameters:
            AutomationAssumeRole:
              StaticValue:
                Values:
                  - !GetAtt IamRoleForRemediation.Arn

Prerequisites:

Ensure the following prerequisites are in place:

An AWS account with sufficient permissions to create and manage resources.
The AWS CLI installed on the local machine.
CloudFormation infrastructure deployed from my previous post (if applicable).

Deployment:

Deploy the CloudFormation Stack.

    aws cloudformation create-stack \
        --stack-name monitoring-policy-compliance \
        --template-body file://infrastructure/monitoring_stack_cloudtrail.yaml \
        --capabilities CAPABILITY_NAMED_IAM \
        --disable-rollback

2.Test the resources deployed with the stack. Change value of the resource from the base-infrastructure stack and evaluate the drift detection rule to verify functionality.

    aws ssm put-parameter --name "ConnectionToken" --value "secret_token_value_2" --type "String" --overwrite

    aws configservice start-config-rules-evaluation --config-rule-names ConfigRuleCheckCloudTrailLogs

3.After drift detection runs, review Athena query results stored in S3 under /athena-results in .csv file.

    aws s3 ls s3://<your-bucket-name>/athena-results/ --recursive

    aws s3 cp s3://<your-bucket-name>/<report_name>.csv ./

Here is an example of logs from this file:

)

4.Cleanup Resources. After testing, stop the CloudTrail Trail logging, delete all data from the S3 bucket, and delete the CloudFormation stack.

    aws cloudtrail stop-logging --name monitor-cfn-policy-compliance

    aws s3 rm s3://<your-bucket-name> --recursive

    aws cloudformation delete-stack --stack-name monitoring-policy-compliance

Conclusion:

Implementing this solution provides visibility into changes affecting CloudFormation-managed resources. This enhances security, compliance tracking, and audit readiness. The ability to log and query user actions simplifies responses to compliance requests from clients, regulators, or security teams.

If you found this post helpful and interesting, please click the reaction button below to show your support. Feel free to use and share this post. 🙂

CloudFormation Drift Detection and Notification with AWS Config Remediation Action

Andrii Shykhov — Thu, 13 Mar 2025 20:45:14 +0000

Introduction:

CloudFormation stack drift occurs when resources created by a CloudFormation stack are manually changed or deleted. This post explains the process of automatically detecting stack drift using an AWS Config Rule, triggering a Remediation Action with the AWS-PublishSNSNotification Systems Manager automation runbook, and sending notifications via an SNS topic for efficient monitoring and response.

About the project:

This solution deploys an automated system for monitoring and alerting on CloudFormation stack drift. The core components of the project are:

AWS Config Rule: Monitors stacks for drift using the CLOUDFORMATION_STACK_DRIFT_DETECTION_CHECK rule.
Remediation Action: Utilizes the AWS-PublishSNSNotification Systems Manager automation runbook to trigger notifications as a remediation step.
SNS Notifications: Alerts are sent to a preconfigured endpoint upon drift detection.
CloudFormation Templates: Streamline the deployment and configuration of resources.

Config Rule and Remediation Action configurations from infrastructure/monitoring_stack.yaml template:

Parameters:
  StackNameToMonitor:
    Type: String
    Description: Name of the CloudFormation stack to monitor for drift detection
    Default: 'base-infrastructure'
  ConfigRuleName:
    Type: String
    Description: Provide the name for the Config rule
    Default: 'CloudFormationDriftDetection'

Resources:
  IamRoleForConfig:
    Type: AWS::IAM::Role
    Properties:
      RoleName: CloudFormationDriftDetectionRole
      Description: IAM role for AWS Config to access CloudFormation drift detection
      AssumeRolePolicyDocument:
        Version: '2012-10-17'
        Statement:
          - Effect: Allow
            Principal:
              Service: config.amazonaws.com
            Action:
              - sts:AssumeRole
      ManagedPolicyArns:
        - arn:aws:iam::aws:policy/ReadOnlyAccess
      Policies:
        - PolicyName: CloudFormationDriftDetectionpolicy
          PolicyDocument:
            Version: "2012-10-17"
            Statement:
              - Effect: Allow
                Action:
                  - cloudformation:DetectStackResourceDrift
                  - cloudformation:DetectStackDrift
                  - cloudformation:DescribeStacks
                  - cloudformation:DescribeStackResources
                  - cloudformation:BatchDescribeTypeConfigurations
                  - cloudformation:DescribeStackResourceDrifts
                  - cloudformation:DescribeStackDriftDetectionStatus
                Resource: !Sub "arn:${AWS::Partition}:cloudformation:${AWS::Region}:${AWS::AccountId}:*"

  ConfigRuleToDetectCfnDrift:
    Type: AWS::Config::ConfigRule
    Properties:
      ConfigRuleName: !Ref ConfigRuleName
      Description: AWS Config rule to detect drift in CloudFormation stacks
      Scope:
        TagKey: stack-name
        TagValue: !Ref StackNameToMonitor
      Source:
        Owner: AWS
        SourceIdentifier: CLOUDFORMATION_STACK_DRIFT_DETECTION_CHECK
      MaximumExecutionFrequency: !Ref MaximumExecutionFrequency
      InputParameters:
        cloudformationRoleArn: !GetAtt IamRoleForConfig.Arn

  IamRoleForRemediation:
    Type: AWS::IAM::Role
    Properties:
      RoleName: AwsConfigRemediationAction
      Description: IAM role for AWS Config remediation action to publish SNS notifications
      AssumeRolePolicyDocument:
        Version: '2012-10-17'
        Statement:
          - Effect: Allow
            Principal:
              Service:
                - config.amazonaws.com
                - ssm.amazonaws.com
            Action:
              - sts:AssumeRole
      Policies:
        - PolicyName: SNSPublishPolicy
          PolicyDocument:
            Version: '2012-10-17'
            Statement:
              - Effect: Allow
                Action:
                  - sns:Publish
                Resource: !Ref SnsTopicToNotifyCfnDriftAlarm

  RemediationActionForConfigRule:
    Type: AWS::Config::RemediationConfiguration
    Properties:
      ConfigRuleName: !Ref ConfigRuleName
      TargetType: SSM_DOCUMENT
      TargetId: AWS-PublishSNSNotification
      Automatic: true
      MaximumAutomaticAttempts: 2
      RetryAttemptSeconds: 30
      Parameters:
        AutomationAssumeRole:
          StaticValue:
            Values:
              - !GetAtt IamRoleForRemediation.Arn
        Message:
          StaticValue:
            Values:
              - !Sub |
                "*** CloudFormation Stack drift detected! ***"
                "Account: ${AWS::AccountId}"
                "CloudFormation Stack: ${StackNameToMonitor}"
        TopicArn:
          StaticValue:
            Values:
              - !Ref SnsTopicToNotifyCfnDriftAlarm

Prerequisites:

Ensure the following prerequisites are in place:

An AWS account with sufficient permissions to create and manage resources.
The AWS CLI installed on the local machine.

Deployment:

1.Clone the repository.

    git clone https://gitlab.com/Andr1500/cfn-drift-detection.git

2.Deploy the base infrastructure. Use the infrastructure/infrastructure_stack.yaml template to set up the foundational resources.

    aws cloudformation create-stack \
        --stack-name base-infrastructure \
        --template-body file://infrastructure/infrastructure_stack.yaml \
        --capabilities CAPABILITY_NAMED_IAM \
        --tags Key=stack-name,Value=base-infrastructure

3.Verify AWS Config setup. Check if AWS Config is enabled, and create necessary resources if missing.

    aws configservice describe-configuration-recorders

    aws configservice describe-delivery-channels

    aws iam get-role --role-name AWSServiceRoleForConfig

    aws iam create-service-linked-role --aws-service-name config.amazonaws.com

    aws configservice put-configuration-recorder \
        --configuration-recorder name=default,roleARN=arn:aws:iam::${AWS::AccountId}:role/aws-service-role/config.amazonaws.com/AWSServiceRoleForConfig

    aws configservice put-delivery-channel \
        --delivery-channel-name default \
        --s3-bucket-name <YourBucketName>

4.Deploy the monitoring stack using the infrastructure/monitoring_stack.yaml template. It is important to grant necessary access for the Config Rule, otherwise, you will have an error for the monitored stack resources scope:AWS CloudFormation failed to detect drift. The default compliance state has been set to NON_COMPLIANT. Re-evaluate the rule and try again. If the problem persists contact AWS CloudFormation support.

    aws cloudformation create-stack \
        --stack-name monitoring-infrastructure \
        --template-body file://infrastructure/monitoring_stack.yaml \
        --capabilities CAPABILITY_NAMED_IAM \
        --disable-rollback

5.Test the monitoring stack. Change value of the resource from the base infrastructure stack and evaluate the drift detection rule to verify functionality.

    aws ssm put-parameter --name "ConnectionToken" --value "secret_token_value_2" --type "String" --overwrite

    aws configservice start-config-rules-evaluation --config-rule-names CloudFormationDriftDetection

6.Clean up resources. After testing, clean up resources by deleting the CloudFormation stacks.

    aws cloudformation delete-stack --stack-name infrastructure_stack.yaml

    aws cloudformation delete-stack --stack-name monitoring_stack.yaml

Conclusion:

By implementing remediation actions with the AWS-PublishSNSNotification Systems Manager automation runbook, you can automate notifications with minimal setup. While this approach is efficient, it’s essential to consider the 256-character limit for the Message field, which may restrict detailed notifications.

For scenarios requiring advanced workflows or richer notifications, consider augmenting the solution with EventBridge rules or Lambda functions. With this system in place, you can ensure timely responses to drift events and maintain infrastructure compliance effectively.

If you found this post helpful and interesting, please click the reaction button to show your support. Feel free to use and share this post.

OpenTelemetry Tracing for File Downloads from an S3 Bucket via API Gateway and Lambda Functions

Andrii Shykhov — Thu, 13 Mar 2025 20:38:17 +0000

Introduction:

This post demonstrates how to build a serverless solution using AWS services like API Gateway, Lambda functions, and S3 buckets to enable file downloads using presigned URLs. By integrating OpenTelemetry tracing, we can gain deep insights into each request’s journey. The focus is on creating a secure and efficient method for downloading files from an S3 bucket while tracing the interactions from API Gateway (REST API) through Lambda to S3.

About the project:

In this project, we build a serverless architecture that allows users to securely download files from an S3 bucket via an API Gateway endpoint. API Gateway is secured using a Lambda Authorizer to ensure only authorized requests are processed. The architecture also incorporates OpenTelemetry tracing, providing detailed visibility into each request’s lifecycle — from API Gateway to Lambda functions and interactions with the S3 bucket. All resources (except the parameter in the Parameter Store) were created with CloudFormation.

For the Lambda functions to work properly, the project uses the AWS Distro for OpenTelemetry (ADOT) Lambda layer, which enables tracing. You can find more information about setting up this layer here.

The core components of the project are:
API Gateway: Serves as the entry point for API requests, secured with a Lambda Authorizer.
Lambda Functions: Handle the business logic, including:

Authenticating requests via the Lambda Authorizer.
Generating presigned URLs for downloading files from the S3 bucket.
Handling error cases, such as incorrect API Gateway resources or invalid file names. S3 Bucket: Stores the files for download. OpenTelemetry Tracing: Tracks and logs the flow of requests for debugging and performance optimization.

The Main Lambda function and IAM role configuration in infrastructure/root.yaml CloudFormation template:

    Parameters:
      S3BucketName:
        Type: String
        Default: 'store-files-20241010'
      StageName:
        Type: String
        Default: 'dev'
      OtelLambdaLayerArn:
        Type: String
        Default: 'arn:aws:lambda:eu-central-1:901920570463:layer:aws-otel-python-amd64-ver-1-25-0:1'
    Resources:
      MainLambdaFunction:
        DependsOn:
          - S3Bucket
        Type: AWS::Lambda::Function
        Properties:
          FunctionName: MainLambdaFunction
          Description: Makes requests to the S3 bucket
          Runtime: python3.12
          Handler: index.lambda_handler
          Role: !GetAtt MainLambdaExecutionRole.Arn
          Timeout: 30
          MemorySize: 512
          Environment:
            Variables:
              S3_BUCKET_NAME: !Ref S3BucketName
              OTEL_PROPAGATORS: xray
              AWS_LAMBDA_EXEC_WRAPPER: /opt/otel-instrument
          TracingConfig:
            Mode: Active
          Layers:
            - !Ref OtelLambdaLayerArn
          Code:
            ZipFile: |
              import json
              import boto3
              import os
              import logging
              from botocore.exceptions import ClientError
              from opentelemetry import trace

              # Initialize logging
              logger = logging.getLogger()
              logger.setLevel(logging.INFO)

              # OpenTelemetry Tracing
              tracer = trace.get_tracer(__name__)

              # Initialize the S3 client
              s3_client = boto3.client('s3')

              def lambda_handler(event, context):
                  # Start tracing for the request
                  with tracer.start_as_current_span("main-handler-span") as span:
                      # Log the entire event received
                      logger.info('Received event: %s', json.dumps(event))

                      # Extract and log routeKey, path, and method
                      route_key = event.get("routeKey")
                      path = event.get("path")
                      http_method = event.get("httpMethod")

                      logger.info('Route Key: %s', route_key)
                      logger.info('HTTP Method: %s', http_method)
                      logger.info('Path: %s', path)

                      # Add tracing attributes
                      span.set_attribute("routeKey", route_key)
                      span.set_attribute("http.method", http_method)
                      span.set_attribute("http.path", path)

                      # Handle routes based on HTTP method and path
                      if route_key == "GET /list" or (http_method == "GET" and path == "/list"):
                          logger.info("Handling /list request")
                          return list_objects(span)
                      elif route_key == "GET /download" or (http_method == "GET" and path == "/download"):
                          logger.info("Handling /download request")
                          return generate_presigned_url(event, span)
                      else:
                          # Return a specific error message for invalid routes
                          logger.error("Invalid route: %s", route_key)
                          return {
                              'statusCode': 404,
                              'body': json.dumps({"error": "Resource name is incorrect."})
                          }

              def list_objects(span):
                  bucket_name = os.environ.get('S3_BUCKET_NAME')
                  try:
                      logger.info("Listing objects in bucket: %s", bucket_name)
                      response = s3_client.list_objects_v2(Bucket=bucket_name)
                      if 'Contents' not in response:
                          logger.info("No objects found in bucket.")
                          return {
                              'statusCode': 200,
                              'body': json.dumps({"objects": []})
                          }
                      object_keys = [{'File': obj['Key']} for obj in response['Contents']]
                      logger.info("Found %d objects in bucket.", len(object_keys))
                      span.set_attribute("s3.object_count", len(object_keys))
                      return {
                          'statusCode': 200,
                          'body': json.dumps({"objects": object_keys})
                      }
                  except ClientError as e:
                      logger.error("Error listing objects in the bucket: %s", e)
                      return {
                          'statusCode': 500,
                          'body': json.dumps({"error": "Error listing objects in the bucket."})
                      }

              def generate_presigned_url(event, span):
                  bucket_name = os.environ.get('S3_BUCKET_NAME')
                  query_string_params = event.get("queryStringParameters")
                  if not query_string_params:
                      logger.error("Query string parameters are missing or request is incorrect.")
                      return {
                          'statusCode': 400,
                          'body': json.dumps({"error": "Request is incorrect. Please provide valid query parameters."})
                      }
                  object_key = query_string_params.get("objectKey")
                  logger.info("Bucket name: %s", bucket_name)
                  logger.info("Requested object key: %s", object_key)
                  if not object_key:
                      logger.error("objectKey is missing in queryStringParameters.")
                      return {
                          'statusCode': 400,
                          'body': json.dumps({"error": "objectKey is required."})
                      }
                  # Check if the object exists
                  try:
                      logger.info("Checking if object %s exists in bucket %s", object_key, bucket_name)
                      s3_client.head_object(Bucket=bucket_name, Key=object_key)
                  except ClientError as e:
                      if e.response['Error']['Code'] == '404':
                          logger.error("Object %s does not exist in bucket %s", object_key, bucket_name)
                          return {
                              'statusCode': 404,
                              'body': json.dumps({"error": "Object name is incorrect or object doesn't exist."})
                          }
                      else:
                          logger.error("Error checking object existence in bucket: %s", e)
                          return {
                              'statusCode': 500,
                              'body': json.dumps({"error": "Error checking object existence."})
                          }
                  # Generate presigned URL if the object exists
                  try:
                      logger.info("Generating presigned URL for object: %s", object_key)
                      presigned_url = s3_client.generate_presigned_url(
                          'get_object',
                          Params={'Bucket': bucket_name, 'Key': object_key},
                          ExpiresIn=900
                      )
                      logger.info("Generated presigned URL: %s", presigned_url)
                      span.set_attribute("s3.object_key", object_key)
                      return {
                          'statusCode': 200,
                          'body': json.dumps({"url": presigned_url})
                      }
                  except ClientError as e:
                      logger.error("Error generating presigned URL for object %s: %s", object_key, e)
                      return {
                          'statusCode': 500,
                          'body': json.dumps({"error": "Error generating presigned URL."})
                      }

      MainLambdaExecutionRole:
        Type: AWS::IAM::Role
        Properties:
          RoleName: MainLambdaExecutionRole
          AssumeRolePolicyDocument:
            Version: '2012-10-17'
            Statement:
              - Effect: Allow
                Principal:
                  Service:
                    - lambda.amazonaws.com
                Action:
                  - sts:AssumeRole
          ManagedPolicyArns:
            - arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole
          Policies:
            - PolicyName: S3BucketAccessPolicy
              PolicyDocument:
                Version: '2012-10-17'
                Statement:
                  - Effect: Allow
                    Action:
                      - s3:ListBucket
                      - s3:GetObject
                      - s3:HeadObject
                    Resource:
                      - !Sub arn:${AWS::Partition}:s3:::${S3BucketName}/*
                      - !Sub arn:${AWS::Partition}:s3:::${S3BucketName}
            - PolicyName: XRayAccessPolicy
              PolicyDocument:
                Version: '2012-10-17'
                Statement:
                  - Effect: Allow
                    Action:
                      - xray:PutTraceSegments
                      - xray:PutTelemetryRecords
                    Resource: "*"

Prerequisites:

Ensure the following prerequisites are in place:

An AWS account with sufficient permissions to create and manage resources.
The AWS CLI installed on the local machine.

Deployment:

1.Clone the repository.

    git clone https://gitlab.com/Andr1500/api-gateway-dowload-from-s3.git

2.Create an authorization token in the AWS Systems Manager Parameter Store.

    aws ssm put-parameter --name "AuthorizationLambdaToken" --value "token_value_secret" --type "SecureString"

3.Fill in all necessary Parameters in infrastructure/root.yaml CloudFormation template, and create the CloudFormation stack.

    aws cloudformation create-stack \
        --stack-name api-gw-dowload-from-s3 \
        --template-body file://infrastructure/root.yaml \
        --capabilities CAPABILITY_NAMED_IAM --disable-rollback

4.Retrieve the Invoke URLs of the Stage.

    aws cloudformation describe-stacks --stack-name api-gw-dowload-from-s3 --query "Stacks[0].Outputs"

5.Copy some files to the S3 bucket for testing.

    aws s3 cp images/apigw_lambda_s3.png  s3://s3_bucket_name/apigw_lambda_s3.png

6.Test API and download the file with CURL.
IMPORTANT! After the infrastructure creation, it can take some time for the bucket name to propagate across AWS regions. During this time “Temporary Redirect” response might be received for requests to the object with presigned URL. In my case, even though the region was specified in the origin domain name, it was not working, only 1 day later downloading with the presigned URL worked correctly.
To view the X-Ray trace map and segments timeline, navigate to the AWS Console: CloudWatch -> X-Ray traces -> Traces. In the Traces section, a list of trace IDs will be displayed. Clicking on a trace ID will reveal the Trace map, Segments timeline, and associated Logs.

    export APIGW_TOKEN='token_value_secret'

    curl -X GET -H "Authorization: Bearer $APIGW_TOKEN" "https://api_id.execute-api.eu-central-1.amazonaws.com/dev/list"

    curl -X GET -H "Authorization: Bearer $APIGW_TOKEN" "https://api_id.execute-api.eu-central-1.amazonaws.com/download?objectKey=apigw_lambda_s3.png"

    curl -O "https://s3_bucket_name.s3.amazonaws.com/apigw_lambda_s3.png?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=ASIASAQWERTY123456%2Feu-central-1%2Fs3%2Faws4_request&X-Amz-Date=20241003T125740Z&X-Amz-Expires=600&X-Amz-SignedHeaders=host&X-Amz-Security-Token=token_value&X-Amz-Signature=signature_value"

7.Clean Up Resources. After testing, clean up resources by deleting the token from the Parameter Store, and the CloudFormation stack.

    aws ssm delete-parameter --name "AuthorizationLambdaToken"

    aws cloudformation delete-stack --stack-name api-gw-dowload-from-s3

Conclusion:

By leveraging AWS services like API Gateway, Lambda, and S3, along with the power of OpenTelemetry tracing, this project provides a robust solution for tracing file downloads. Using CloudFormation to manage the infrastructure as code ensures easy deployment and repeatability.

If you found this post helpful and interesting, please click the button to show your support. Feel free to use and share this post. You can also support me with a virtual coffee 🙂

Implementing Tracing with AWS X-Ray service and AWS X-Ray SDK for REST API Gateway and Lambda functions

Andrii Shykhov — Thu, 13 Mar 2025 20:29:53 +0000

Introduction:

This post details the implementation of tracing with AWS X-Ray and the AWS X-Ray SDK within a serverless AWS infrastructure. The focus is on integrating tracing across various AWS services, including API Gateway (REST API), Lambda functions, Systems Manager Parameter Store, and the Amazon Bedrock model.

About the Project:

Based on my earlier article that covers interacting with Amazon Bedrock models using API Gateway and Lambda functions, this post extends the existing infrastructure by adding configurations for tracing with AWS X-Ray. Initially, the setup utilized an HTTP API, which lacks native X-Ray support; therefore, a REST API was adopted to enable tracing. Additional information on API Gateway tracing with X-Ray is available in the official documentation. The AWS X-Ray SDK documentation can be found here.

The Main Lambda function and IAM role configuration in infrastructure/tracing-rest-api.yaml CloudFormation template:

    Parameters:
      BedrockModelId:
        Type: String
        Default: 'amazon.titan-text-express-v1'
      LambdaLayerVersionArn:
        Type: String
        Default: 'arn:aws:lambda:<region>:<account_id>:layer:aws-xray-sdk-layer:<version_number>'

    Resources:   
      MainLambdaFunction:
        Type: AWS::Lambda::Function
        Properties:
          FunctionName: MainLambdaFunction
          Description: Make requests to Bedrock models
          Runtime: python3.12
          Handler: index.lambda_handler
          Role: !GetAtt MainLambdaExecutionRole.Arn
          Timeout: 30
          MemorySize: 512
          TracingConfig:
            Mode: Active
          Layers:
            - !Ref LambdaLayerVersionArn
          Environment:
            Variables:
              BEDROCK_MODEL_ID: !Ref BedrockModelId
          Code:
            ZipFile: |
              import json
              import boto3
              import os
              import logging
              from botocore.exceptions import ClientError
              from aws_xray_sdk.core import patch_all, xray_recorder

              # Initialize logging
              logger = logging.getLogger()
              logger.setLevel(logging.INFO)

              # Initialize the X-Ray SDK
              patch_all()

              # Initialize the Bedrock Runtime client
              bedrock_runtime = boto3.client('bedrock-runtime')

              def lambda_handler(event, context):
                  try:
                      # Retrieve the model ID from environment variables
                      model_id = os.environ['BEDROCK_MODEL_ID']

                      # Validate the input
                      input_text = event.get("queryStringParameters", {}).get("inputText")
                      if not input_text:
                          logger.error('Input text is missing in the request')
                          raise ValueError("Input text is required in the request query parameters.")

                      # Prepare the payload for invoking the Bedrock model
                      payload = json.dumps({
                          "inputText": input_text,
                          "textGenerationConfig": {
                              "maxTokenCount": 8192,
                              "stopSequences": [],
                              "temperature": 0,
                              "topP": 1
                          }
                      })

                      logger.info('Payload for Bedrock model: %s', payload)

                      # Create a subsegment for the Bedrock model invocation
                      with xray_recorder.in_subsegment('Bedrock InvokeModel') as subsegment:
                          # Invoke the Bedrock model
                          response = bedrock_runtime.invoke_model(
                              modelId=model_id,
                              contentType="application/json",
                              accept="application/json",
                              body=payload
                          )

                          logger.info('Response from Bedrock model: %s', response)

                          # Check if the 'body' exists in the response and handle it correctly
                          if 'body' not in response or not response['body']:
                              logger.error('Response body is empty')
                              raise ValueError("Response body is empty.")

                          # Read and process the response
                          response_body = json.loads(response['body'].read().decode('utf-8'))
                          logger.info('Processed response body: %s', response_body)

                          return {
                              'statusCode': 200,
                              'body': json.dumps(response_body)
                          }

                  except ClientError as e:
                      logger.error('ClientError: %s', e)
                      return {
                          'statusCode': 500,
                          'body': json.dumps({"error": "Error interacting with the Bedrock API"})
                      }
                  except ValueError as e:
                      logger.error('ValueError: %s', e)
                      return {
                          'statusCode': 400,
                          'body': json.dumps({"error": str(e)})
                      }
                  except Exception as e:
                      logger.error('Exception: %s', e)
                      return {
                          'statusCode': 500,
                          'body': json.dumps({"error": "Internal Server Error"})
                      }

      MainLambdaExecutionRole:
        Type: AWS::IAM::Role
        Properties:
          RoleName: MainLambdaExecutionRole
          AssumeRolePolicyDocument:
            Version: '2012-10-17'
            Statement:
              - Effect: Allow
                Principal:
                  Service:
                    - lambda.amazonaws.com
                Action:
                  - sts:AssumeRole
          ManagedPolicyArns:
            - arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole
          Policies:
            - PolicyName: BedrockAccessPolicy
              PolicyDocument:
                Version: '2012-10-17'
                Statement:
                  - Effect: Allow
                    Action:
                      - bedrock:InvokeModel
                      - bedrock:ListFoundationModels
                    Resource: '*'
            - PolicyName: XRayAccessPolicy
              PolicyDocument:
                Version: '2012-10-17'
                Statement:
                  - Effect: Allow
                    Action:
                      - xray:PutTelemetryRecords
                      - xray:PutAnnotation
                      - xray:PutTraceSegments
                    Resource: '*'

Prerequisites:

Ensure the following prerequisites are in place:

An AWS account with sufficient permissions to create and manage resources.
The AWS CLI installed on the local machine.

Deployment:

Before Starting: If the infrastructure from my earlier post has not been set up, follow steps 1 – 4 from this article before proceeding with the deployment steps below.

1.Create a Lambda layer that includes the AWS X-Ray SDK. Use the layer’s version ARN in the tracing-rest-api.yaml CloudFormation template.

    aws lambda publish-layer-version --layer-name aws-xray-sdk-layer --zip-file fileb://infrastructure/aws-xray-sdk-layer-layer/aws-xray-sdk-layer-layer.zip --compatible-runtimes python3.12

2.Update the existing CloudFormation stack to enable X-Ray tracing and transition from an HTTP API to a REST API.

    aws cloudformation update-stack \
        --stack-name apigw-lambda-bedrock \
        --template-body file://infrastructure/tracing-rest-api.yaml \
        --capabilities CAPABILITY_NAMED_IAM \
        --disable-rollback

3.Retrieve the Invoke URL for the API Gateway Stage using the retrieve_invoke_url_rest_api.sh script. Test the API using CURL.

    ./scripts/retrieve_invoke_url_rest_api.sh

    export APIGW_TOKEN='token_value'

    curl -s -X GET -H "Authorization: Bearer $APIGW_TOKEN" "https://api_id.execute-api.eu-central-1.amazonaws.com/dev/invoke?inputText=your_question" | jq -r '.results[0].outputText'

4.To view the X-Ray trace map and segments timeline, navigate to the AWS Console: CloudWatch -> X-Ray traces -> Traces. In the Traces section, a list of trace IDs will be displayed. Clicking on a trace ID will reveal the Trace map, Segments timeline, and associated Logs.


![X-Ray Trace Map](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/voy3jootl0gy0ufj080m.png)

![X-Ray Segments Timeline](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ev6b0d86fjdt7kv12ptg.png)

5.Clean Up Resources. After testing, clean up resources by deleting the Lambda layer version, the token from the Parameter Store, and the CloudFormation stack.

aws lambda delete-layer-version --layer-name aws-xray-sdk-layer --version-number <number>

aws ssm delete-parameter --name "AuthorizationLambdaToken"

aws cloudformation delete-stack --stack-name apigw-lambda-bedrock

Conclusion:

The integration of AWS X-Ray into a serverless infrastructure provides deep insights into the performance of APIs and Lambda functions, as well as their interactions with other AWS services such as Systems Manager and Bedrock. This enhanced visibility facilitates effective debugging, performance optimization, and comprehensive system monitoring, contributing to the smooth and efficient operation of applications.

If you found this post helpful and interesting, please click the button to show your support. Feel free to use and share this post. You can also support me with a virtual coffee 🙂

AWS Lambda URL Invocations with IAM Authentication and Throttling Limits

Andrii Shykhov — Thu, 13 Mar 2025 20:17:08 +0000

Introduction:

This blog post details how to securely invoke AWS Lambda functions using Lambda URLs with AWS IAM authentication and throttling limits to add more security to the Lambda function. It covers setting up the necessary infrastructure using AWS CloudFormation, generating AWS Signature Version 4 for authenticated requests, and testing the setup using both the command line and testing tool like Postman.

About the project:

Use Case Scenario: Invoking Lambda Functions from Different Environments. This setup allows Lambda functions to be invoked from various environments, including non-AWS infrastructure, such as local environments, on-premises servers.

For unauthorized or throttled requests that don’t invoke the Lambda function, no charges will be incurred. Here are the key points:

Throttling and Concurrency: Setting ReservedConcurrentExecutions will limit the number of concurrent executions. If the limit is reached, additional requests will be throttled, and no charges will be incurred for these throttled requests.
Authorization Failures: Requests that fail due to authorization errors (e.g., invalid IAM credentials) will not result in Lambda function invocation. These requests are handled only by the IAM service, and no charges will be incurred for these unauthorized requests.
Additional Security: AWS Shield Standard is automatically available for Lambda functions. More information can be found here.
Monitoring: Monitoring of the Lambda function throttling can be helpful by creating a CloudWatch alarm based on the “Throttles” metric.

References and useful links:
Authenticating requests documentation: AWS S3 — Authenticating Requests.
Examples of creating AWS SigV4 signatures in different programming languages: AWS SigV4 Signing Examples.
Note: Function URLs are not supported in some regions. More information can be found here.

The generate_sigv4.sh script:

    #!/bin/sh

    # Request parameters
    METHOD="POST"
    SERVICE="lambda"
    REQUEST_PAYLOAD='{"inputText": "Request to Lambda."}'  # simple payload

    # Environment variables
    AWS_ACCESS_KEY=$AWS_ACCESS_KEY_ID
    AWS_SECRET_KEY=$AWS_SECRET_ACCESS_KEY
    REGION=$AWS_REGION
    HOST=$LAMBDA_FUNCTION_HOST
    ENDPOINT="https://${HOST}/"

    # Check if environment variables are set
    if [ -z "$AWS_ACCESS_KEY" ] || [ -z "$AWS_SECRET_KEY" ] || [ -z "$HOST" ]; then
      echo "Error: AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, and LAMBDA_FUNCTION_HOST must be set as environment variables."
      exit 1
    fi

    # Get the current date and time
    AMZ_DATE=$(date -u +"%Y%m%dT%H%M%SZ")
    DATE_STAMP=$(date -u +"%Y%m%d")

    # Create a canonical request
    CANONICAL_URI="/"
    CANONICAL_QUERYSTRING=""
    CANONICAL_HEADERS="content-type:application/json\nhost:${HOST}\nx-amz-date:${AMZ_DATE}\n"
    SIGNED_HEADERS="content-type;host;x-amz-date"
    PAYLOAD_HASH=$(printf "$REQUEST_PAYLOAD" | openssl dgst -sha256 | sed 's/^.* //')
    CANONICAL_REQUEST="${METHOD}\n${CANONICAL_URI}\n${CANONICAL_QUERYSTRING}\n${CANONICAL_HEADERS}\n${SIGNED_HEADERS}\n${PAYLOAD_HASH}"

    # Create a string to sign
    ALGORITHM="AWS4-HMAC-SHA256"
    CREDENTIAL_SCOPE="${DATE_STAMP}/${REGION}/${SERVICE}/aws4_request"
    STRING_TO_SIGN="${ALGORITHM}\n${AMZ_DATE}\n${CREDENTIAL_SCOPE}\n$(printf "$CANONICAL_REQUEST" | openssl dgst -sha256 | sed 's/^.* //')"

    # Create the signing key
    kSecret=$(printf "AWS4${AWS_SECRET_KEY}" | xxd -p -c 256)
    kDate=$(printf "${DATE_STAMP}" | openssl dgst -sha256 -mac HMAC -macopt hexkey:"$kSecret" | sed 's/^.* //')
    kRegion=$(printf "${REGION}" | openssl dgst -sha256 -mac HMAC -macopt hexkey:"$kDate" | sed 's/^.* //')
    kService=$(printf "${SERVICE}" | openssl dgst -sha256 -mac HMAC -macopt hexkey:"$kRegion" | sed 's/^.* //')
    kSigning=$(printf "aws4_request" | openssl dgst -sha256 -mac HMAC -macopt hexkey:"$kService" | sed 's/^.* //')

    # Create the signature
    SIGNATURE=$(printf "$STRING_TO_SIGN" | openssl dgst -sha256 -mac HMAC -macopt hexkey:"$kSigning" | sed 's/^.* //')

    # Create the authorization header
    AUTHORIZATION_HEADER="${ALGORITHM} Credential=${AWS_ACCESS_KEY}/${CREDENTIAL_SCOPE}, SignedHeaders=${SIGNED_HEADERS}, Signature=${SIGNATURE}"

    # Output the necessary information for test tool like Postman
    echo  "Endpoint: ${ENDPOINT}\n"
    echo  "x-amz-date: ${AMZ_DATE}\n"
    echo  "Authorization: ${AUTHORIZATION_HEADER}\n"
    echo  "Payload: ${REQUEST_PAYLOAD}\n"

    # Output the curl command
    echo  "curl -X ${METHOD} \"${ENDPOINT}\" -H \"Content-Type: application/json\" -H \"x-amz-date: ${AMZ_DATE}\" -H \"Authorization: ${AUTHORIZATION_HEADER}\" -d '${REQUEST_PAYLOAD}'\n"

Prerequisites:

Before starting, ensure the following requirements are met:

An AWS account with permissions to create resources.
AWS CLI https://aws.amazon.com/cli/ installed on local machine.
OpenSSL installed on the test environment.

Deployment:

1.Clone the repository.

    git clone https://gitlab.com/Andr1500/lambda-url-with-iam-auth.git

2.Check and increase the Lambda function Concurrent Executions quota (if necessary).

    aws service-quotas get-service-quota \
        --service-code lambda \
        --quota-code L-B99A9384

    aws service-quotas request-service-quota-increase \
        --service-code lambda \
        --quota-code L-B99A9384 \
        --desired-value 100

3.Fill in all necessary parameters in the infrastructure/root.yaml CloudFormation template and create the CloudFormation stack.

    aws cloudformation create-stack \
        --stack-name invoke-lambda-url-iam-auth \
        --template-body file://infrastructure/root.yaml \
        --capabilities CAPABILITY_NAMED_IAM \
        --disable-rollback

4.Retrieve outputs for setting up environment variables.

    aws cloudformation describe-stacks \
        --stack-name invoke-lambda-url-iam-auth \
        --query "Stacks[0].Outputs" --output json

5.Set up region, Lambda function host, Access, and Secret keys for the created IAM user as environment variables in the test environment.

    export AWS_ACCESS_KEY_ID=<Access Key>
    export AWS_SECRET_ACCESS_KEY=<Secret Access Key>
    export AWS_REGION=<Region>
    export LAMBDA_FUNCTION_HOST=<function-url.lambda-url-region.amazonaws.com>

6.Generate AWS Signature v4 and POST request using the generate_sigv4.sh script.

    ./generate_sigv4.sh

    curl -X POST "https://.<url-id>.<region>.on.aws/" -H "Content-Type: application/json" -H "x-amz-date: 20240619T150022Z" -H "Authorization: AWS4-HMAC-SHA256 Credential=<ACCESS-KEY>/20240619/<region>/lambda/aws4_request, SignedHeaders=content-type;host;x-amz-date, Signature=<signarure-content>" -d '{"inputText": "Request to Lambda."}'

Tests from command line and Postman:

7.Delete the CloudFormation stack.

    aws cloudformation delete-stack --stack-name invoke-lambda-url-iam-auth

Conclusion:

This project demonstrates how to securely invoke AWS Lambda functions using Lambda URLs with IAM authentication. Additional security can be added by throttling limits and monitoring the Lambda function with CloudWatch alarms.

If you found this post helpful and interesting, please click the button to show your support. Feel free to use and share this post. 🙂

Interacting with Amazon Bedrock Model through API Gateway and Lambda Functions

Andrii Shykhov — Thu, 13 Mar 2025 20:00:27 +0000

Introduction:

In this post, we will explore how to interact with an Amazon Bedrock model through a secured API Gateway and Lambda function. The API Gateway is secured using a Lambda Authorizer, ensuring that only authorized requests can access the Bedrock model. This setup provides a scalable and secure way to integrate machine learning models into applications.

About the Project:

In this project, we have set up an Amazon Bedrock model, an API Gateway, and two Lambda functions: the “Authorizer” Lambda function, which acts as an access gatekeeper, and the “Main” Lambda function, which sends requests to the Bedrock model. We also use the Systems Manager Parameter Store for storing the authorization token securely.

Currently, AWS does not support AWS CLI access to the Amazon Bedrock service directly, and granting access to models must be done via the AWS Console. The token in the Parameter Store was created using AWS CLI because CloudFormation does not yet support the creation of parameters with the SecureString type. For more information, see the AWS CloudFormation SSM Parameter documentation.

All other resources — Lambda functions and API Gateway — were created using CloudFormation, ensuring that the infrastructure is managed as code and can be easily deployed and maintained.

This project is based on my other project about serverless architecture using API Gateway, Lambda Authorizer, and Secrets Manager. In current version, we store tokens in Parameter Store instead of Secrets Manager for lower cost and added the necessary integration between the Main Lambda function and the Bedrock model.

The Main Lambda function and IAM role configuration in infrastructure/root.yaml CloudFormation template:

    Parameters:
      BedrockModelId:
        Type: String
        Default: ''

    Resources:  
      MainLambdaFunction:
          Type: AWS::Lambda::Function
          Properties:
            FunctionName: MainLambdaFunction
            Description: Make requests to Bedrock models
            Runtime: python3.12
            Handler: index.lambda_handler
            Role: !GetAtt MainLambdaExecutionRole.Arn
            Timeout: 30
            MemorySize: 512
            Environment:
              Variables:
                BEDROCK_MODEL_ID: !Ref BedrockModelId
            Code:
              ZipFile: |
                import json
                import boto3
                from botocore.exceptions import ClientError

                bedrock_runtime = boto3.client('bedrock-runtime')

                def lambda_handler(event, context):
                    try:
                        model_id = os.environ['BEDROCK_MODEL_ID']

                        # Validate the input
                        input_text = event.get("queryStringParameters", {}).get("inputText")
                        if not input_text:
                            raise ValueError("Input text is required in the request query parameters.")

                        # Prepare the payload for invoking the Bedrock model
                        payload = json.dumps({
                            "inputText": input_text,
                            "textGenerationConfig": {
                                "maxTokenCount": 8192,
                                "stopSequences": [],
                                "temperature": 0,
                                "topP": 1
                            }
                        })

                        # Invoke the Bedrock model
                        response = bedrock_runtime.invoke_model(
                            modelId=model_id,
                            contentType="application/json",
                            accept="application/json",
                            body=payload
                        )

                        # Check if the 'body' exists in the response and handle it correctly
                        if 'body' not in response or not response['body']:
                            raise ValueError("Response body is empty.")

                        response_body = json.loads(response['body'].read().decode('utf-8'))

                        return {
                            'statusCode': 200,
                            'body': json.dumps(response_body)
                        }

                    except ClientError as e:
                        return {
                            'statusCode': 500,
                            'body': json.dumps({"error": "Error interacting with the Bedrock API"})
                        }
                    except ValueError as e:
                        return {
                            'statusCode': 400,
                            'body': json.dumps({"error": str(e)})
                        }
                    except Exception as e:
                        return {
                            'statusCode': 500,
                            'body': json.dumps({"error": "Internal Server Error"})
                        }

        MainLambdaExecutionRole:
          Type: AWS::IAM::Role
          Properties:
            RoleName: MainLambdaExecutionRole
            AssumeRolePolicyDocument:
              Version: '2012-10-17'
              Statement:
                - Effect: Allow
                  Principal:
                    Service:
                      - lambda.amazonaws.com
                  Action:
                    - sts:AssumeRole
            ManagedPolicyArns:
              - arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole
            Policies:
              - PolicyName: BedrockAccessPolicy
                PolicyDocument:
                  Version: '2012-10-17'
                  Statement:
                    - Effect: Allow
                      Action:
                        - bedrock:InvokeModel
                        - bedrock:ListFoundationModels
                      Resource: '*'

Prerequisites:

Before you start, make sure the following requirements are met:

An AWS account with permissions to create resources.
AWS CLI installed on your local machine.

Deployment:

1.Clone the repository.

    git clone https://gitlab.com/Andr1500/api-gateway-lambda-bedrock.git

2.Set up Amazon Bedrock model.

Go to the Amazon Bedrock service in the AWS Console.
Navigate to Get Started -> Request model access -> Modify model access -> Choose the appropriate model available in your region (e.g., Titan Text G1 — Express) -> Next -> Submit.
Wait a few minutes and refresh the page to see “Access granted”.

Navigate to Overview -> choose Provider of the available model -> choose the model -> see the API request of the model. For different models the API request configuration can be different.

3.Create authorization token in AWS Systems Manager Parameter Store.

    aws ssm put-parameter --name "AuthorizationLambdaToken" --value "token_value_secret" --type "SecureString"

4.Fill in all necessary Parameters in infrastructure/root.yaml CloudFormation template, and scripts/retrieve_invoke_url.sh script, and create the CloudFormation stack.

    aws cloudformation create-stack \
        --stack-name apigw-lambda-bedrock \
        --template-body file://infrastructure/root.yaml \
        --capabilities CAPABILITY_NAMED_IAM \
        --disable-rollback

5.Retrieve the Invoke URL of the Stage using the scripts/retrieve_invoke_url.sh script.

6.Test the Bedrock model, Main Lambda function, and API request from CLI. It is necessary to have a workaround with saving content response to some file because, according to the documentation for aws bedrock-runtime invoke-model and aws lambda invoke, the outfile option is required.

7.Delete token from SM Parameter Store, and the CloudFormation stack.

    aws ssm delete-parameter --name "AuthorizationLambdaToken"

    aws cloudformation delete-stack --stack-name apigw-lambda-bedrock

Conclusion:

By leveraging API Gateway, Lambda functions, and Amazon Bedrock models, we can create a scalable and efficient solution for deploying machine learning models in a serverless environment. With the addition of a Lambda Authorizer, this solution is more secure, protecting against unauthorized access.

If you found this post helpful and interesting, please click the button to show your support. Feel free to use and share this post.

Managing CloudFormation nested stacks with AWS CodePipeline: A GitOps approach

Andrii Shykhov — Mon, 10 Feb 2025 09:22:00 +0000

Introduction:

In this blog post, we delve into creating CloudFormation nested stacks and automating their deployment using AWS CodePipeline. This discussion builds on the GitOps deployment strategy previously explored, where we detailed modeling GitOps environments and promoting releases across them using an “environment-per-folder” approach. While my earlier blog post (link here) focused on updating standalone CloudFormation stacks with the CloudFormation Git Sync feature, here we expand our scope to include nested stacks, offering a more structured solution for managing complex infrastructure.

About the Project:

This project is structured into two main folders: codepipeline and infrastructure. The codepipeline folder contains the codepipeline_pipeline.yaml template, which establishes all required configurations for orchestrating the deployment of nested stacks.
Key resources set up by this template include:
CodeStar Connection: Facilitates integration with remote Git repository.
IAM Roles and Policies: Three roles are defined to securely manage resources during the pipeline operations — creating resources via CloudFormation, executing CodeBuild projects, and running CodePipeline workflows.
S3 Bucket and S3 Bucket Policy: Provides storage for nested stack templates and CodePipeline artifacts.
CodeBuild Project: Executes cfn-lint to ensure the nested stack templates adhere to best practices before deployment.
The infrastructure folder has environment-specific folders containing simple CloudFormation templates for S3 bucket creation and associated policies. These serve as examples to illustrate the deployment process within a GitOps framework.

Pipeline Overview:
The CodePipeline setup includes several stages crucial for streamlined operations:
Source: Monitors the Git repository for changes and triggers the pipeline.
CFN-Lint: Validates the CloudFormation templates against common errors and best practices.
Copy-to-S3: Processes and stores files from the Source stage to the S3 bucket.
Deploy-CFN-stacks: Handles the creation and updates of CloudFormation nested stacks for each environment.

The project structure:

    ├── codepipeline
    │   └── codepipeline_pipeline.yaml
    └── infrastructure
        ├── development
        │   ├── root.yaml
        │   ├── s3_bucket.yaml
        │   └── s3_bucket_policy.yaml
        ├── production
        │   ├── root.yaml
        │   ├── s3_bucket.yaml
        │   └── s3_bucket_policy.yaml
        └── staging
            ├── root.yaml
            ├── s3_bucket.yaml
            └── s3_bucket_policy.yaml

The CodePipeline pipeline configuration from codepipeline_pipeline.yaml:

    CreateCfnStackFromRepo:
      Type: 'AWS::CodePipeline::Pipeline'
      Properties:
        Name: !Ref CodePipelineName
        RoleArn: !GetAtt CodePipelineRole.Arn
        ArtifactStore:
          Type: S3
          Location: !Ref S3BucketName
        Stages:
          - Name: Source
            Actions:
              - Name: Source
                ActionTypeId:
                  Category: Source
                  Owner: AWS
                  Provider: CodeStarSourceConnection
                  Version: '1'
                RunOrder: 1
                Configuration:
                  BranchName: !Ref BranchName
                  ConnectionArn: !GetAtt GitLabConnection.ConnectionArn
                  DetectChanges: 'true'
                  FullRepositoryId: !Ref FullRepositoryId
                  OutputArtifactFormat: CODE_ZIP
                OutputArtifacts:
                  - Name: SourceArtifact
                Namespace: SourceVariables
          - Name: CFN-Lint
            Actions:
              - Name: Run-CFN-Lint
                ActionTypeId:
                  Category: Build
                  Owner: AWS
                  Provider: CodeBuild
                  Version: '1'
                Configuration:
                  ProjectName: !Ref CfnlintCodeBuildProject
                InputArtifacts:
                  - Name: SourceArtifact
                OutputArtifacts:
                  - Name: CflintArtifact
                RunOrder: 1
          - Name: Copy-to-S3
            Actions:
              - Name: Copy-to-S3
                ActionTypeId:
                  Category: Deploy
                  Owner: AWS
                  Provider: S3
                  Version: '1'
                RunOrder: 1
                Configuration:
                  BucketName: !Ref S3BucketName
                  Extract: 'true'
                InputArtifacts:
                  - Name: SourceArtifact
          - Name: Deploy-CFN-stacks
            Actions:
              - Name: DeployDevelopmentStack
                ActionTypeId:
                  Category: Deploy
                  Owner: AWS
                  Provider: CloudFormation
                  Version: '1'
                Configuration:
                  ActionMode: CREATE_UPDATE
                  Capabilities: 'CAPABILITY_NAMED_IAM,CAPABILITY_AUTO_EXPAND'
                  StackName: !Sub '${CodePipelineName}-development'
                  TemplatePath: SourceArtifact::infrastructure/development/root.yaml
                  RoleArn: !GetAtt CloudFormationExecutionRole.Arn
                  ParameterOverrides: |
                    {
                      "Environment": "development"
                    }
                InputArtifacts:
                  - Name: SourceArtifact
                RunOrder: 1
              - Name: DeployStagingStack
                ActionTypeId:
                  Category: Deploy
                  Owner: AWS
                  Provider: CloudFormation
                  Version: '1'
                Configuration:
                  ActionMode: CREATE_UPDATE
                  Capabilities: 'CAPABILITY_NAMED_IAM,CAPABILITY_AUTO_EXPAND'
                  StackName: !Sub '${CodePipelineName}-staging'
                  TemplatePath: SourceArtifact::infrastructure/staging/root.yaml
                  RoleArn: !GetAtt CloudFormationExecutionRole.Arn
                  ParameterOverrides: |
                    {
                      "Environment": "staging"
                    }
                InputArtifacts:
                  - Name: SourceArtifact
                RunOrder: 1
              - Name: DeployProductionStack
                ActionTypeId:
                  Category: Deploy
                  Owner: AWS
                  Provider: CloudFormation
                  Version: '1'
                Configuration:
                  ActionMode: CREATE_UPDATE
                  Capabilities: 'CAPABILITY_NAMED_IAM,CAPABILITY_AUTO_EXPAND'
                  StackName: !Sub '${CodePipelineName}-production'
                  TemplatePath: SourceArtifact::infrastructure/production/root.yaml
                  RoleArn: !GetAtt CloudFormationExecutionRole.Arn
                  ParameterOverrides: |
                    {
                      "Environment": "production"
                    }
                InputArtifacts:
                  - Name: SourceArtifact
                RunOrder: 1

CodePipeline pipeline stages schema:

Prerequisites:

Before you start, make sure the following requirements are met:

An AWS account with permissions to create resources.
AWS CLI installed on your local machine.
Account in any Git provider for remote Git repository.

Deployment:

Clone the repository to get started.

    git clone https://gitlab.com/Andr1500/cloudformation_stack_from_codepipeline.git

Set up the required AWS and GitLab integration:

a. Deploy a CloudFormation stack with necessary services and roles:

    aws cloudformation create-stack \
        --stack-name codepipeline-pipeline-cfn \
        --template-body file://codepipeline/codepipeline_pipeline.yaml \
        --capabilities CAPABILITY_NAMED_IAM --disable-rollback

b. Authorise the CodeStar connection via the AWS Console under CodePipeline settings to link to your Git repository.

Create CloudFormation nested stacks. Update the templates in the infrastructure folders as needed, then commit and push these changes. The pipeline will handle creating the stacks automatically.
Update nested stacks. Always use Git to make changes to your templates and push them to apply. Avoid using the AWS Management Console for updates. To move updates from one environment (like development) to another (like staging), copy the files, commit, and push:

    cp -rf infrastructure/development/* infrastructure/staging/

Delete nested Stacks. If you no longer need the nested stacks, they can be deleted either through the AWS Management Console, using the AWS CLI, or by configuring the CodePipeline to perform the deletion. To enable deletion via the pipeline, modify the Deploy-CFN-stacks stage in your pipeline configuration by changing the ActionMode from CREATE_UPDATE to DELETE_ONLY. This adjustment will instruct the pipeline to remove the stacks rather than update or create new ones.

Conclusion:

AWS CodePipeline is a powerful tool that can be customized for complex deployment tasks like managing CloudFormation nested stacks. For enhanced oversight, consider setting up a Notification rule linked to an SNS topic to monitor pipeline activities. You can find the setup for this in my other project here.

If you found this post helpful and interesting, please click the reaction button below to show your support for the author. Feel free to use and share this post! 🙂

GitOps deployment strategy with CloudFormation’s Git Sync feature

Andrii Shykhov — Mon, 10 Feb 2025 09:14:00 +0000

Introduction:

In this blog post, we have the process of modeling GitOps environments and promoting releases between them using AWS CloudFormation’s Git Sync feature. Here we use the “environment-per-folder” approach, more information on how to model GitOps environments and promote releases between them is in this article. More information about the CloudFormation Git Sync concept, prerequisites, and walkthrough is here.

About the project:

In this project we have 2 folders. The folder git_connection has CloudFormation template git_connection.yaml which creates all necessary configurations for the creation of CloudFormation stacks with the Git Sync option. With this stack, we create resources: CodeStar connection for connection to our Gitlab account, two IAM roles — first to update the stack from the Git repository and second to use for all operations performed on the stack, SSM parameter for S3 bucket prefix name. The folder infrastructure has environment folders. In every folder, we have a deployment file and a template file. In the template file, we have a simple configuration for the creation of an S3 bucket with an S3 Bucket policy. We assume that we have the same template file for all environments and specify parameters in our deployment files.

The project structure:

    ├── git_connection
    │   └── git_connection.yaml
    └── infrastructure
        ├── development
        │   ├── deployment_parameters.yaml
        │   └── s3bucket.yaml
        ├── production
        │   ├── deployment_parameters.yaml
        │   └── s3bucket.yaml
        └── staging
            ├── deployment_parameters.yaml
            └── s3bucket.yaml

The CloudFormation git_connection.yaml template:

    AWSTemplateFormatVersion: '2010-09-09'
    Description: 'Connect Gitbab repository with AWS'

    Parameters:
      ConnectionName:
        Type: String
        Default: 'Gitlab-to-CloudFormation'
      S3BucketPrefixName:
        Type: String
        Default: 'cf-app-files'

    Resources:
      GitLabConnection:
        Type: 'AWS::CodeStarConnections::Connection'
        Properties:
          ConnectionName: !Ref ConnectionName
          ProviderType: 'GitLab'

      CloudFormationS3AccessRole:
        Type: AWS::IAM::Role
        Properties:
          RoleName: CloudFormationS3AccessRole
          AssumeRolePolicyDocument:
            Version: '2012-10-17'
            Statement:
              - Effect: Allow
                Principal:
                  Service: 
                    - cloudformation.amazonaws.com
                Action: 
                  - sts:AssumeRole
          Policies:
            - PolicyName: CloudFormationS3ManagementPolicy
              PolicyDocument:
                Version: '2012-10-17'
                Statement:
                  - Effect: Allow
                    Action:
                      - s3:CreateBucket
                      - s3:DeleteBucket
                      - s3:PutBucketPolicy
                      - s3:GetBucketPolicy
                      - s3:ListBucket
                      - s3:PutBucketTagging
                      - s3:PutBucketPolicy
                      - s3:DeleteBucketPolicy
                    Resource: '*'
                  - Effect: Allow
                    Action:
                      - ssm:GetParameters
                    Resource: '*'
                  - Effect: Allow
                    Action:
                      - cloudformation:*
                    Resource: '*'

      GitSyncRole:
        Type: AWS::IAM::Role
        Properties:
          RoleName: CFNGitSyncRole
          AssumeRolePolicyDocument:
            Version: 2012-10-17
            Statement:
              - Sid: CfnGitSyncTrustPolicy
                Effect: Allow
                Principal:
                  Service: cloudformation.sync.codeconnections.amazonaws.com
                Action: sts:AssumeRole
          Policies:
            - PolicyName: GitSyncPermissions
              PolicyDocument:
                Version: 2012-10-17
                Statement:
                  - Sid: SyncToCloudFormation
                    Effect: Allow
                    Action:
                      - cloudformation:CreateChangeSet
                      - cloudformation:DeleteChangeSet
                      - cloudformation:DescribeChangeSet
                      - cloudformation:DescribeStackEvents
                      - cloudformation:DescribeStacks
                      - cloudformation:ExecuteChangeSet
                      - cloudformation:GetTemplate
                      - cloudformation:ListChangeSets
                      - cloudformation:ListStacks
                      - cloudformation:ValidateTemplate
                    Resource: '*'
                  - Sid: PolicyForManagedRules
                    Effect: Allow
                    Action:
                      - events:PutRule
                      - events:PutTargets
                    Resource: '*'
                    Condition:
                      StringEquals:
                        events:ManagedBy: cloudformation.sync.codeconnections.amazonaws.com
                  - Sid: PolicyForDescribingRule
                    Effect: Allow
                    Action: events:DescribeRule
                    Resource: '*'

      SsmS3BucketPrefixName:
        Type: AWS::SSM::Parameter
        Properties:
          Name: S3BucketPrefixName
          Type: String
          Value: !Ref S3BucketPrefixName
          Description: "Prefix of the S3 bucket name  for git sync cf stacks"

The CloudFormation s3bucket.yaml template:

    AWSTemplateFormatVersion: '2010-09-09'
    Description: 'S3 bucket with a dynamically created name and apply a policy for access from EC2 instances'

    Parameters:
      S3BucketPrefixName:
        Type: AWS::SSM::Parameter::Value<String>
        Default: ''
      Environment:
        Description: List of available environments
        Type: String
        Default: dev
        AllowedValues:
          - dev
          - stag
          - prod
        ConstraintDescription: Use valid environment [dev, stag, prod]

    Mappings:
      EnvironmentLabel:
        dev:
          label: development
        stag:
          label: staging
        prod:
          label: production

    Resources:
      S3BucketFiles:
        Type: AWS::S3::Bucket
        Properties:
          BucketName:
            !Sub
              - '${S3BucketPrefixName}-${UsedEnvironmentLabel}'
              - UsedEnvironmentLabel: !FindInMap [ EnvironmentLabel, !Ref Environment, label ]
          Tags:
            - Key: Name
              Value:
                !Sub
                  - '${S3BucketPrefixName}-${UsedEnvironmentLabel}'
                  - UsedEnvironmentLabel: !FindInMap [ EnvironmentLabel, !Ref Environment, label ]
            - Key: Environment
              Value: !FindInMap [ EnvironmentLabel, !Ref Environment, label ]

      S3BucketPolicy:
        Type: AWS::S3::BucketPolicy
        Properties:
          Bucket: !Ref S3BucketFiles
          PolicyDocument:
            Version: '2012-10-17'
            Statement:
              - Effect: Allow
                Principal:
                  Service: 
                    - ec2.amazonaws.com
                Action:
                  - s3:GetObject
                Resource: 
                  !Sub
                    - 'arn:${AWS::Partition}:s3:::${S3BucketPrefixName}-${UsedEnvironmentLabel}/*'
                    - UsedEnvironmentLabel: !FindInMap [ EnvironmentLabel, !Ref Environment, label ]

The CloudFormation development/deployment_parameters.yaml template:

    template-file-path: './infrastructure/development/s3bucket.yaml'

    parameters:
      S3BucketPrefixName: 'S3BucketPrefixName'
      Environment: 'dev'

    tags:
      InfraDeploymentProcess: 'cf stack with git sync option for dev env'

Prerequisites:

Before you start, make sure the following requirements are met:

An AWS account with permissions to create resources.
AWS CLI installed on your local machine.
Account in any Git provider for remote Git repository.

Deployment:

Clone the repository

    git clone https://gitlab.com/Andr1500/cloudformation_sync_from_git.git

Creation of necessary configuration for interconnection between Gitlab and AWS account.

a. Create CloudFormation stack with CodeStar connection and necessary IAM roles:

    aws cloudformation create-stack \
        --stack-name cf-git-sync-config \
        --template-body file://git_connection/git_connection.yaml \
        --capabilities CAPABILITY_NAMED_IAM

b. Make necessary changes, commit it, and push it to your remote repository.

c. Update CodeStar pending connection. Open the AWS Console, next go: CodePipeline -> Settings -> Connections -> choose the created connection in pending status -> Update pending connection -> depends on your provider make authorisation, and grant necessary access to the repository.

PreBuild step (optional) before CloudFormation Git Sync stack creation.

Sometimes, if you already earlier linked the repository with the AWS account by CodeStar connection after deleting it and recreating again, you will have the issue during the process of creating the CloudFormation stack: the repository will be already linked to the “old” connection. In this case, we should “unlink” the repository with the AWS CLI command by deleting “old” repository link and “link” again:

    aws codestar-connections list-repository-links

    aws codestar-connections delete-repository-link --repository-link-id 1234567-76a3-4f20-858f-qwerty12345

Creation of CloudFormation stacks for different environments.

Open the AWS Console, next go: CloudFormation -> Create stack (button) -> With new resources -> Choose options “Template is ready” and “Sync from Git” -> Next -> Provide stack name (for example, “cf-git-deployment”), choose option “I am providing my own file in my repository”, choose option “Link a Git repository” (if you create it for the first time), choose the repository provider, connection, repository, branch, provide the deployment file path, choose the IAM role for CloudFormation to update the stack from the Git repository -> Next -> choose the IAM role for CloudFormation to use for all operations performed on the stack, choose the stack failure options -> Next -> Review and create -> Submit.

Repeat a similar procedure for creating CloudFormation stacks for other environments, in this case, we need to specify different CF stack names and different deployment file paths.

Update CloudFormation stacks and promote changes between the environments.

IMPORTANT ! All changes (Creation of new resources, updating existing resources) with the created CloudFormation stacks should be done with the Git repository, not with the CloudFormation “Update” button.

a. Make necessary changes in the CF template file for dev env, commit changes, and push it. Check if the changes applied in the CloudFormation stack are related to the development environment.

b. Make a copy of the template file from one environment to another (for example, from dev env to stag env), commit changes, and push it. Check if the changes applied in the CloudFormation stack are related to a staging environment.

    cp infrastructure/development/s3bucket.yaml infrastructure/staging/s3bucket.yaml

Conclusion:

imho, for now, the CloudFormation Git Sync feature has some advantages and disadvantages. As one of the advantages — we have only one source of code and we can simply manage changes in our infrastructure. As a disadvantage — we can’t create nested stack with this feature because in AWS::CloudFormation::Stack resource TemplateURL parameter supports only the S3 bucket URL. Also, we can’t integrate the created Webhook into our Gitlab CI/CD pipeline because the Webhook’s secret token is hidden and we don’t have any option for using existed Webhook in our pipeline. We can’t create CloudFormation Git Sync stack with the AWS CLI command create-stack because, similar to nested stack, — template-url supports only the S3 bucket URL.

If you found this post helpful and interesting, please click the clap button below to show your support. Feel free to use and share it.

Monitoring EC2 instances deployed with Blue/Green deployment

Andrii Shykhov — Mon, 03 Feb 2025 22:42:38 +0000

Introduction:

In this post, we have a configuration for monitoring EC2 instances deployed with the Blue/Green deployment strategy. This configuration consists resources:
Lambda function with necessary access which does:

get the instance IDs based on the instance names,
get the metric's value for the instance from the AWS/EC2 CloudWatch namespace,
enhance the metric’s data with additional information,
put changed metric's data to custom ws-deployment namespace;

EventBridge schedule rule which runs the lambda function every 5 minutes;
CloudWatch alarms which monitor the EC2 instances based on the metrics from the custom ws-deployment namespace.

The reason for this configuration is that for the AWS/EC2 namespace as metric dimension we have only InstanceId, not InstanceName, more information about CloudWatch metrics is here :

    aws cloudwatch list-metrics --namespace AWS/EC2 --metric-name CPUUtilization   
    {
        "Metrics": [
            {
                "Namespace": "AWS/EC2",
                "MetricName": "CPUUtilization",
                "Dimensions": [
                    {
                        "Name": "InstanceId",
                        "Value": "i-1234567890abcde"
                    }
                ]
            },

This post is the third part series of posts about Blue/Green deployment on AWS EC2 instances with the Systems Manager Automation runbook, the first part is here, and the second part is here.

About the project:

All infrastructure is created with CloudFormation template infrastructure/ec2_monitoring.yaml and has independent deployment from the ec2-bluegreen-deployment stack. In the Systems Manager Automation runbook configuration, we have only one EC2 instance for creation, but the lambda function can work with many instances, for this, we only need to specify instance names as a comma-separated list of the “InstanceNames” parameter.

ec2_monitoring.yaml template:

    AWSTemplateFormatVersion: '2010-09-09'
    Description: 'CloudWatch metrics and alarms for monitoring the deployed EC2 instances'

    Parameters:
      TransformMetricsLfName:
        Type: String
        Default: 'TransformEc2Metrics'
      CustomNamespace:
        Type: String
        Default: 'ws-deployment'
      InstanceNames:
        Type: String
        Description: 'Comma-separated list of the instance names'
        Default: 'ws-instance'
      MetricNames:
        Type: String
        Description: 'Comma-separated list of the metric names'
        Default: 'CPUUtilization,StatusCheckFailed_Instance,StatusCheckFailed_System'
      MetricUnits:
        Type: String
        Description: 'Comma-separated list of the metric units'
        Default: 'Percent,Count,Count'
      SnsTopicName:
        Type: String
        Default: 'blue-green-deployment-notifications'

    Resources:
    #####################################
    #  Lambda Function configuration
    #####################################
      TransformEc2MetricsRole:
        Type: AWS::IAM::Role
        Properties:
          RoleName: TransformEc2MetricsRole
          AssumeRolePolicyDocument:
            Version: '2012-10-17'
            Statement:
              - Effect: Allow
                Principal:
                  Service: lambda.amazonaws.com
                Action: sts:AssumeRole
          ManagedPolicyArns:
            - 'arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole'
          Policies:
            - PolicyName: TransformEc2Metricspolicy
              PolicyDocument:
                Version: '2012-10-17'
                Statement:
                  - Effect: Allow
                    Action:
                      - ec2:DescribeInstances
                      - cloudwatch:GetMetricStatistics
                      - cloudwatch:PutMetricData
                      - cloudwatch:GetMetricData
                    Resource: '*'

      LambdaLogsGroup:
        Type: AWS::Logs::LogGroup
        DeletionPolicy: Delete
        UpdateReplacePolicy: Retain
        Properties:
          LogGroupName: !Sub '/aws/lambda/${TransformMetricsLfName}'
          RetentionInDays: '7'

      TransformCustomMetrics:
        Type: AWS::Lambda::Function
        Properties:
          FunctionName: !Ref TransformMetricsLfName
          Description: 'transforming and processing metrics from AWS/EC2 namespace'
          Runtime: python3.12
          Handler: index.lambda_handler
          Timeout: 30
          Role: !GetAtt TransformEc2MetricsRole.Arn
          LoggingConfig:
            LogGroup: !Sub '/aws/lambda/${TransformMetricsLfName}'
          Environment:
            Variables:
              instance_names: !Ref InstanceNames
              metric_names: !Ref MetricNames
              metric_units: !Ref MetricUnits
              custom_namespace: !Ref CustomNamespace
          Code:
            ZipFile: |
              import boto3
              import os
              from datetime import datetime, timedelta

              def lambda_handler(event, context):
                  try:
                      instance_names = [name.strip() for name in os.environ['instance_names'].split(',')]
                      metric_names =  [metric.strip() for metric in os.environ['metric_names'].split(',')]
                      metric_units =  [unit.strip() for unit in os.environ['metric_units'].split(',')]
                      custom_namespace = os.environ['custom_namespace']

                      # Initialize EC2 and CloudWatch clients
                      ec2_client = boto3.client('ec2')
                      cloudwatch_client = boto3.client('cloudwatch')

                      # Get instance IDs based on the instance names
                      instance_ids, instance_names_result = get_instance_ids(instance_names, ec2_client)
                      if not instance_ids:
                          print("[INFO] No instances found for transforming metrics.")
                          return

                      # Get metrics for each instance
                      for instance_id, instance_name in zip(instance_ids, instance_names_result):
                          metrics = get_instance_metrics(instance_id, metric_names, metric_units, instance_name, cloudwatch_client)

                          # Put formatted metrics to custom CloudWatch namespace
                          put_custom_metrics(metrics, custom_namespace, cloudwatch_client)

                  except Exception as e:
                      print(f"Error with proceeding metrics transformation: {str(e)}")

              def get_instance_ids(instance_names, ec2_client):
                  instance_ids = []
                  instance_names_result = []

                  for instance_name in instance_names:
                      response = ec2_client.describe_instances(
                          Filters=[
                              {'Name': 'tag:Name', 'Values': [instance_name]},
                              {'Name': 'instance-state-name', 'Values': ['running']}
                          ]
                      )

                      # Extract instance IDs from the response
                      ids = [instance['InstanceId'] for reservation in response['Reservations'] for instance in reservation['Instances']]

                      # append instance IDs
                      if ids:
                          instance_ids.extend(ids)
                          instance_names_result.append(instance_name)

                  return instance_ids, instance_names_result

              def get_instance_metrics(instance_id, metric_names, metric_units, instance_name, cloudwatch_client):
                  end_time = datetime.utcnow()
                  start_time = end_time - timedelta(minutes=10)

                  metrics_dict = {}

                  for metric_name, unit in zip(metric_names, metric_units):
                      # take necessary metric values
                      id_for_query = metric_name.lower()

                      query = {
                          "Id": id_for_query,
                          "MetricStat": {
                              "Metric": {
                                  "Namespace": "AWS/EC2",
                                  "MetricName": metric_name,
                                  "Dimensions": [
                                      {"Name": "InstanceId", "Value": instance_id}
                                  ]
                              },
                              "Period": 300,
                              "Stat": "Average",
                              "Unit": unit
                          },
                          "ReturnData": True
                      }

                      response = cloudwatch_client.get_metric_data(
                          MetricDataQueries=[query],
                          StartTime=start_time,
                          EndTime=end_time
                      )

                      # Extract data from the response
                      metric_data_results = response.get('MetricDataResults', [])

                      if not metric_data_results:
                          print(f"No data available for metric: {metric_name} related to {instance_name}")
                          continue

                      values = metric_data_results[0].get('Values', [])

                      if not values:
                          print(f"No values available for metric: {metric_name} related to {instance_name}")
                          continue

                      # Get the latest value
                      latest_value = values[-1]

                      # Combine metric name, value, unit, instance name into a dictionary
                      metrics_dict[id_for_query] = {
                          'MetricName': metric_name,
                          'Value': latest_value,
                          'Unit': unit,
                          'InstanceId': instance_id,
                          'InstanceName': instance_name
                      }
                  return metrics_dict

              def put_custom_metrics(metrics_dict, custom_namespace, cloudwatch_client):
                  for metric_id, metric_info in metrics_dict.items():
                      metric_name = metric_info['MetricName']
                      value = metric_info['Value']
                      dimensions = [
                          {'Name': 'InstanceName', 'Value': metric_info['InstanceName']}
                      ]
                      unit = metric_info['Unit']
                      instance_name = metric_info['InstanceName']

                      response = cloudwatch_client.put_metric_data(
                          Namespace=custom_namespace,
                          MetricData=[
                              {
                                  'MetricName': metric_name,
                                  'Dimensions': dimensions,
                                  'Value': value,
                                  'Unit': unit
                              }
                          ]
                      )

                      # Print information about the success or failure process
                      if response['ResponseMetadata']['HTTPStatusCode'] == 200:
                          print(f"Successfully put metric data for {metric_name} in {custom_namespace} related to {instance_name}")
                      else:
                          print(f"Failed to put metric data for {metric_name} in {custom_namespace}. Response: {response}")

      LambdaInvokePermission:
        Type: AWS::Lambda::Permission
        Properties:
          Action: lambda:InvokeFunction
          FunctionName: !Ref TransformCustomMetrics
          Principal: events.amazonaws.com
          SourceArn: !GetAtt ScheduleRule.Arn

      ScheduleRule:
        Type: AWS::Events::Rule
        Properties:
          Name: TransformCustomMetricsScheduleRule
          ScheduleExpression: 'rate(5 minutes)'
          Targets:
            - Arn: !GetAtt TransformCustomMetrics.Arn
              Id: TransformCustomMetricsTarget

    #####################################
    #  CloudWatch Alarms
    #####################################
      CPUAlarm: 
        Type: AWS::CloudWatch::Alarm
        Properties:
          AlarmName: 
            !Sub 
              - '${InstanceName} - High CPU Usage'
              - InstanceName: !Select [0, !Split [",", !Ref InstanceNames]]
          AlarmDescription: 'High CPU Usage'
          AlarmActions:
          - !Sub 'arn:${AWS::Partition}:sns:${AWS::Region}:${AWS::AccountId}:${SnsTopicName}'
          OKActions:
          - !Sub 'arn:${AWS::Partition}:sns:${AWS::Region}:${AWS::AccountId}:${SnsTopicName}'
          MetricName: !Select [0, !Split [",", !Ref MetricNames]]
          Unit: !Select [0, !Split [",", !Ref MetricUnits]]
          Namespace: !Ref CustomNamespace
          Statistic: Average
          Period: 300
          EvaluationPeriods: 3
          Threshold: 90
          ComparisonOperator: GreaterThanOrEqualToThreshold
          Dimensions:
          - Name: InstanceName
            Value: !Select [0, !Split [",", !Ref InstanceNames]]

      SystemStatusAlarm:
        Type: AWS::CloudWatch::Alarm
        Properties:
          AlarmName: 
            !Sub 
              - '${InstanceName} - System Status Check Failed'
              - InstanceName: !Select [0, !Split [",", !Ref InstanceNames]]
          AlarmDescription: 'System Status Check Failed'
          Namespace: !Ref CustomNamespace
          MetricName: !Select [1, !Split [",", !Ref MetricNames]]
          Unit: !Select [1, !Split [",", !Ref MetricUnits]]
          Statistic: Minimum
          Period: 300
          EvaluationPeriods: 1
          ComparisonOperator: GreaterThanThreshold
          Threshold: 0
          AlarmActions:
          - !Sub 'arn:${AWS::Partition}:sns:${AWS::Region}:${AWS::AccountId}:${SnsTopicName}'
          OKActions:
          - !Sub 'arn:${AWS::Partition}:sns:${AWS::Region}:${AWS::AccountId}:${SnsTopicName}'
          Dimensions:
          - Name: InstanceName
            Value: !Select [0, !Split [",", !Ref InstanceNames]]

      InstanceStatusAlarm:
        Type: AWS::CloudWatch::Alarm
        Properties:
          AlarmName: 
            !Sub 
              - '${InstanceName} - Instance Status Check Failed'
              - InstanceName: !Select [0, !Split [",", !Ref InstanceNames]]
          AlarmDescription: 'Instance Status Check Failed'
          Namespace: !Ref CustomNamespace
          MetricName: !Select [2, !Split [",", !Ref MetricNames]]
          Unit: !Select [2, !Split [",", !Ref MetricUnits]]
          Statistic: Minimum
          Period: 300
          EvaluationPeriods: 1
          ComparisonOperator: GreaterThanThreshold
          Threshold: 0
          AlarmActions:
          - !Sub 'arn:${AWS::Partition}:sns:${AWS::Region}:${AWS::AccountId}:${SnsTopicName}'
          OKActions:
          - !Sub 'arn:${AWS::Partition}:sns:${AWS::Region}:${AWS::AccountId}:${SnsTopicName}'
          Dimensions:
          - Name: InstanceName
            Value: !Select [0, !Split [",", !Ref InstanceNames]]

Infrastructure schema and list of metrics from the ws-deployment namespace:

Deployment:

clone the repository (if you don’t have already cloned it).

    git clone https://gitlab.com/Andr1500/ssm_runbook_bluegreen.git

Put “dummy” metrics data with AWS CLI into the custom namespace for each metric. It is necessary because without this data CloudWatch alarms were not created correctly.

    aws cloudwatch put-metric-data \
        --namespace "ws-deployment" \
        --metric-name "CPUUtilization" \
        --dimensions "InstanceName=ws-instance" \
        --value 70 --unit Percent

Create CloudFormation stack.

    aws cloudformation create-stack \
        --stack-name ws-ec2-monitoring \
        --template-body file://ec2_monitoring.yaml \
        --capabilities CAPABILITY_NAMED_IAM --disable-rollback

Conclusion:

In this post, we showed how we can monitor EC2 instances deployed with the Blue/Green deployment strategy. To be sure that the lambda function works correctly we can add the configuration of the “InsufficientDataActions” parameter in the CloudWatch alarms for sending notifications in case of changing the CloudWatch alarm state to “Insufficient data”. If you need to have more specific CloudWatch metrics from the EC2 instances — here is my post about Monitoring Disk Space (as example) with CloudWatch agent.

If you found this post helpful and interesting, please click the reaction button below to show your support for the author. Feel free to use and share this post!

CodePipeline pipeline for automation Blue/Green deployment with SM Automation runbook

Andrii Shykhov — Mon, 03 Feb 2025 22:35:38 +0000

Introduction:

In this post, we have a configuration for Systems Manager automation execution on GitLab webhook event with CodePipeline pipeline.
CodePipeline pipeline consists of stages: Source, Copy-to-s3, Invoke-lambda.
Source: retrieves code changes when a webhook event is sent from GitLab;
Copy-to-s3: files from the Source stage are unzipped before saving and save files to the S3 bucket;
Invoke-lambda: invoke a Lambda function to perform execution of the SM Automation runbook.
AWS CodeStar Gitlab Connections service is not available in all regions, more information here .
Monitoring of the pipeline is done with the notification rule which sends error notifications to the SNS topic.

This post is the second part of my series of posts about Blue/Green deployment on AWS EC2 instances with the Systems Manager Automation runbook, the first part is here, and the third part is here.

About the project:

All infrastructure is created with CloudFormation template infrastructure/codepipeline_pipeline.yaml and has independent deployment from the ec2-bluegreen-deployment stack.

codepipeline_pipeline.yaml template:

    AWSTemplateFormatVersion: '2010-09-09'
    Description: 'Codepipeline: take source code, store to S3, execute SSM runbook'

    Parameters:
      SsmDocumentName:
        Type: String
        Default: 'Ec2BlueGreenDeployment'
      ConnectionName:
        Type: String
        Default: 'gitlab-to-s3'
      S3BucketName:
        Type: String
        Default: ''
      BranchName:
        Type: String
        Default: ''
      FullRepositoryId:
        Type: String
        Default: ''
      CodePipelineName:
        Type: String
        Default: 'execute-blue-green-runbook'
      TopicName:
        Type: String
        Default: 'blue-green-deployment-status'

    Resources:
      GitLabConnection:
        Type: 'AWS::CodeStarConnections::Connection'
        Properties:
          ConnectionName: !Ref ConnectionName
          ProviderType: 'GitLab'

      ExecuteSsmRunbookRole:
        Type: 'AWS::IAM::Role'
        Properties:
          RoleName: ExecuteSsmRunbookRole
          AssumeRolePolicyDocument:
            Version: '2012-10-17'
            Statement:
              - Effect: Allow
                Principal:
                  Service: lambda.amazonaws.com
                Action: sts:AssumeRole
          ManagedPolicyArns:
            - 'arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole'
          Policies:
            - PolicyName: ExecuteSsmRunbook
              PolicyDocument:
                Version: '2012-10-17'
                Statement:
                  - Effect: Allow
                    Action:
                      - ssm:StartAutomationExecution
                    Resource:
                      - !Sub 'arn:${AWS::Partition}:ssm:${AWS::Region}:${AWS::AccountId}:*'
            - PolicyName: ReturnMessageToCodepipeline
              PolicyDocument:
                Version: '2012-10-17'
                Statement:
                  - Effect: Allow
                    Action:
                      - codepipeline:PutJobFailureResult
                      - codepipeline:PutJobSuccessResult
                    Resource: '*'


      ExecuteSsmRunbook:
        Type: 'AWS::Lambda::Function'
        Properties:
          FunctionName: ExecuteSsmRunbook
          Handler: index.lambda_handler
          Role: !GetAtt ExecuteSsmRunbookRole.Arn
          Environment:
            Variables:
              SsmDocumentName: !Ref SsmDocumentName
          Runtime: python3.12
          Timeout: 10
          Code:
            ZipFile: |
              import boto3
              import os
              from botocore.exceptions import ClientError

              def lambda_handler(event, context):
                  try:
                      aws_region = os.environ.get('AWS_REGION')
                      SsmDocumentName = os.environ['SsmDocumentName']
                      codepipeline = boto3.client('codepipeline', region_name=aws_region)
                      ssm = boto3.client('ssm')

                      # Retrieve the Job ID from the Lambda action
                      job_id = event.get('CodePipeline.job', {}).get('id')

                      # Notify CodePipeline of a successful job
                      def put_job_success(message):
                          params = {
                              'jobId': job_id,
                              'executionDetails': {
                                  'summary': str(message),
                                  'percentComplete': 100,
                                  'externalExecutionId': context.aws_request_id
                              }
                          }
                          codepipeline.put_job_success_result(**params)
                          print(message)

                      # Notify CodePipeline of a failed job
                      def put_job_failure(message):
                          params = {
                              'jobId': job_id,
                              'failureDetails': {
                                  'message': str(message),
                                  'type': 'JobFailed',
                                  'externalExecutionId': context.aws_request_id
                              }
                          }
                          codepipeline.put_job_failure_result(**params)
                          print(message)

                      # Start the SSM Automation runbook execution
                      response = ssm.start_automation_execution(DocumentName=SsmDocumentName)
                      execution_id = response['AutomationExecutionId']

                      put_job_success(f'Started SSM Automation execution with ID: {execution_id}')

                  except Exception as e:
                      # Log any unexpected errors and fail the job
                      error_message = f'Error: {str(e)}'
                      print(error_message)
                      put_job_failure(error_message)
                      raise e

      SSMAutomationTriggerPermissions:
        Type: AWS::Lambda::Permission
        Properties:
          Action: 'lambda:InvokeFunction'
          FunctionName: !GetAtt ExecuteSsmRunbook.Arn
          Principal: 'codepipeline.amazonaws.com'

      CodePipelineRole:
        Type: 'AWS::IAM::Role'
        Properties:
          RoleName: 'CodePipelineRole'
          AssumeRolePolicyDocument:
            Version: '2012-10-17'
            Statement:
              - Effect: 'Allow'
                Principal:
                  Service: 'codepipeline.amazonaws.com'
                Action: 'sts:AssumeRole'
          Policies:
            - PolicyName: 'CodePipelineFullAccess'
              PolicyDocument:
                Version: '2012-10-17'
                Statement:
                  - Effect: 'Allow'
                    Action:
                      - 'codepipeline:*'
                    Resource: !Sub 'arn:${AWS::Partition}:codepipeline:${AWS::Region}:${AWS::AccountId}:*'
            - PolicyName: 'S3FullAccess'
              PolicyDocument:
                Version: '2012-10-17'
                Statement:
                  - Effect: 'Allow'
                    Action:
                      - s3:PutObject
                      - s3:GetObject
                      - s3:ListBucket
                    Resource: !Sub 'arn:${AWS::Partition}:s3:::*'
            - PolicyName: 'LambdaInvokeAccess'
              PolicyDocument:
                Version: '2012-10-17'
                Statement:
                  - Effect: 'Allow'
                    Action:
                      - lambda:InvokeFunction
                    Resource: !Sub 'arn:${AWS::Partition}:lambda:${AWS::Region}:${AWS::AccountId}:function:*'
            - PolicyName: 'CodeStarSourceConnectionAccess'
              PolicyDocument:
                Version: '2012-10-17'
                Statement:
                  - Effect: 'Allow'
                    Action:
                      - codestar-connections:UseConnection
                    Resource: !Sub 'arn:${AWS::Partition}:codestar-connections:${AWS::Region}:${AWS::AccountId}:connection/*'
            - PolicyName: 'SendNotificationsBySns'
              PolicyDocument:
                Version: '2012-10-17'
                Statement:
                  - Effect: 'Allow'
                    Action:
                      - sns:Publish
                    Resource:  '*'

      CodePipelineTriggerRunbook:
        Type: AWS::CodePipeline::Pipeline
        Properties:
          Name: !Ref CodePipelineName
          RoleArn: !GetAtt CodePipelineRole.Arn
          ArtifactStore:
            Type: S3
            Location: !Ref S3BucketName
          Stages:
            - Name: Source
              Actions:
                - Name: Source
                  ActionTypeId:
                    Category: Source
                    Owner: AWS
                    Provider: CodeStarSourceConnection
                    Version: '1'
                  RunOrder: 1
                  Configuration:
                    BranchName: !Ref BranchName
                    ConnectionArn: !Ref 'GitLabConnection'
                    DetectChanges: 'true'
                    FullRepositoryId: !Ref FullRepositoryId
                    OutputArtifactFormat: CODE_ZIP
                  OutputArtifacts:
                    - Name: SourceArtifact
                  Namespace: SourceVariables
            - Name: Copy-to-s3
              Actions:
                - Name: copy-to-s3
                  ActionTypeId:
                    Category: Deploy
                    Owner: AWS
                    Provider: S3
                    Version: '1'
                  RunOrder: 1
                  Configuration:
                    BucketName: !Ref S3BucketName
                    Extract: 'true'
                  InputArtifacts:
                    - Name: SourceArtifact
            - Name: Invoke-lambda
              Actions:
                - Name: invoke-lambda
                  ActionTypeId:
                    Category: Invoke
                    Owner: AWS
                    Provider: Lambda
                    Version: '1'
                  RunOrder: 1
                  Configuration:
                    FunctionName: !Ref ExecuteSsmRunbook
                  OutputArtifacts:
                    - Name: Message
                  InputArtifacts:
                    - Name: SourceArtifact
          PipelineType: 'V2'

      CodePipelineNotificationRule:
        Type: 'AWS::CodeStarNotifications::NotificationRule'
        Properties:
          DetailType: 'BASIC'
          EventTypeIds:
            - 'codepipeline-pipeline-pipeline-execution-failed'
            - 'codepipeline-pipeline-stage-execution-failed'
            - 'codepipeline-pipeline-action-execution-failed'
          Name: SsmPipelineNotificationRule
          Resource: !Sub 'arn:${AWS::Partition}:codepipeline:${AWS::Region}:${AWS::AccountId}:${CodePipelineName}'
          Targets:
            - TargetAddress: !Sub 'arn:${AWS::Partition}:sns:${AWS::Region}:${AWS::AccountId}:${TopicName}'
              TargetType: 'SNS'

Deployment and infrastructure schemas:

Prerequisites:

Before you start, make sure the following requirements are met:

An AWS account with permissions to create resources.
AWS CLI installed on your local machine.

Deployment:

Clone the repository (if you don’t have already cloned it), navigate to the cloned repository, and push the repository to your remote repository.

    git clone https://gitlab.com/Andr1500/ssm_runbook_bluegreen.git

Fill all necessary Parameters in the infrastructure/codepipeline_pipeline.yaml file, go to the infrastructure directory, and create CloudFormation stack.

    aws cloudformation create-stack \
        --stack-name codepipeline-pipeline\
        --template-body file://codepipeline_pipeline.yaml \
        --capabilities CAPABILITY_NAMED_IAM --disable-rollback

Update CodeStar pending connection. Open the AWS Console, next go: CodePipeline -> Settings -> Connections -> choose the created connection in pending status -> Update pending connection -> depends on your provider make authorisation, and grant necessary access to the repository. More information about CodeStar connections here .
For deployment of a new version of the application — make changes in application/index.html, commit changes, and push it to your remote repository. The deployment process will be started automatically and in case of any issues with the deployment, both the CodePipeline pipeline and runbook, you will receive an email with issues details.

Conclusion:

In this post, we showed how the CodePipeline pipeline can automate Blue/Green deployment with Application Load Balancer’s weighted target group feature realised with the Systems Manager Automation runbook.

If you found this post helpful and interesting, please click the reaction button below to show your support for the author. Feel free to use and share this post!

Blue/Green deployment on AWS EC2 instances with Systems Manager Automation runbook

Andrii Shykhov — Mon, 03 Feb 2025 22:29:56 +0000

Introduction:

Blue/Green deployment strategy in this post is based on this AWS post about fine-tuning Blue/Green deployments on Application Load Balancer. But why is this solution instead of using CodeDeploy?

In our case we need to have immediate swap traffic from the “Green” environment to “Blue”, CodeDeploy, for now, doesn’t have this option for the EC2/On-premises compute platform, even with the option “Reroute traffic immediately” we have a response in the same time from “Green” and “Blue” EC2 instances for some time during deployment processes.

Another issue is that CodeDeploy deployment on the EC2/On-premises compute platform is based on the instances tags, after the deployment process we need to have a solution for managing tags on the “Blue” instance to be ready for the next deployment.

Also, there is an issue with the creation necessary configuration of the CodeDeploy deployment group with CloudFormation (CloudFormation is used for the creation of infrastructure in AWS).

The Systems Manager Automation runbook can be more flexible in deployment configuration related to our requirements.

This post is the first part of my series of posts about Blue/Green deployment on AWS EC2 instances with the Systems Manager Automation runbook, the second part is here and the third part is here.

About the project:

All infrastructure in AWS (except S3 bucket, Route 53 domain, and SSL/TLS certificate) for this project is created with CloudFormation and can be found in this repository. S3 bucket is necessary for storing nested stack templates and application files. Route 53 domain and SSL/TLS certificate are necessary for secure WEB access to the EC2 instances through the Application Load Balancer. Amazon SNS topic is necessary for sending notifications about the status of the deployment.

The main steps of the Systems Manager Automation runbook:

a) checking if we have already any execution related to the runbook, if yes — skip the deployment;

b) creating a “Green” EC2 instance with all necessary configuration, waiting a few minutes to make sure everything is configured correctly;

c) making a reboot of the instance and checking the status of the instance after reboot;

d) making deployment of the “Green” EC2 instance: registering the instance to the necessary Target Group, making swap weights of the Target Groups in the listener rule configuration, terminating “Blue” EC2 instance;

e) sending notification about status of the deployment.

Blue/Green deployment step configuration from the Systems Manager runbook:

    - name: BlueGreenDeployment
      action: aws:executeScript
      maxAttempts: 2
      timeoutSeconds: 300
      isCritical: true
      onFailure: step:SendNotification
      inputs:
        Runtime: python3.8
        Handler: BlueGreenDeployment
        InputPayload:
          Region: "{{ global:REGION }}"
          ListenerArn: "{{ ListenerArn }}"
          InstanceIds: "{{ LaunchInstance.CreatedInstanceId }}"
        Script: |
          import boto3
          import time
          from botocore.exceptions import ClientError

          def BlueGreenDeployment(event, context):
              try:
                  elbv2 = boto3.client('elbv2', region_name=event['Region'])
                  ec2_client = boto3.client('ec2', region_name=event['Region'])

                  # Register instance with the target group
                  chosen_target_group, target_group_arn_1, target_group_arn_2, instances_tg1, instances_tg2, weight_tg1, weight_tg2 = get_listener_details(elbv2, event['ListenerArn'])

                  if chosen_target_group is None:
                      raise Exception("[ERROR] We don't have necessary target group.")

                  elbv2.register_targets(
                      TargetGroupArn=chosen_target_group,
                      Targets=[
                          {'Id': instance_id}
                          for instance_id in event['InstanceIds']
                      ],
                  )
                  print(f"[INFO] Instance {event['InstanceIds']} registered with the target group {chosen_target_group}")

                  # Wait for instances to be healthy
                  wait_for_instances_to_be_healthy(elbv2, chosen_target_group, event['InstanceIds'], max_wait_time=300)

                  if instances_tg1 or instances_tg2:
                      swap_weights(elbv2, event['ListenerArn'], target_group_arn_1, target_group_arn_2, weight_tg1, weight_tg2)

                      time.sleep(15)

                      # Terminate "Blue" instance
                      if instances_tg1:
                          for instance_id in instances_tg1:
                              if instance_id not in instances_tg2 and weight_tg1 == 100:
                                  ec2_client.terminate_instances(InstanceIds=[instance_id])
                                  print(f"Instance {instance_id}  terminated from Target Group 1")
                      if instances_tg2:
                          for instance_id in instances_tg2:
                              if instance_id not in instances_tg1 and weight_tg2 == 100:
                                  ec2_client.terminate_instances(InstanceIds=[instance_id])
                                  print(f"Instance {instance_id}  terminated from Target Group 2")
                      print("[INFO] Traffic swap and EC2 instance termination completed successfully.")
                  else:
                      print("[INFO] Traffic swap and EC2 instance termination were skipped")
              except ClientError as e:
                  raise Exception("[ERROR]", e)

          # get ARNs of target groups and weights of this groups in Listener rule
          def get_listener_details(elb_client, listener_arn):
              # Get the current state of the listener and its rules
              listener_description = elb_client.describe_rules(ListenerArn=listener_arn)

              # Get the Target Group ARNs and weights from the listener rule
              target_group_arn_1 = listener_description['Rules'][0]['Actions'][0]['ForwardConfig']['TargetGroups'][0]['TargetGroupArn']
              target_group_arn_2 = listener_description['Rules'][0]['Actions'][0]['ForwardConfig']['TargetGroups'][1]['TargetGroupArn']
              weight_tg1 = listener_description['Rules'][0]['Actions'][0]['ForwardConfig']['TargetGroups'][0]['Weight']
              weight_tg2 = listener_description['Rules'][0]['Actions'][0]['ForwardConfig']['TargetGroups'][1]['Weight']

              # Get instance IDs from target groups
              instances_tg1 = get_instance_ids(elb_client, target_group_arn_1)
              instances_tg2 = get_instance_ids(elb_client, target_group_arn_2)

              # Check if instances are registered to the target groups
              if not instances_tg1 and not instances_tg2:
                  chosen_target_group = target_group_arn_2 if weight_tg2 == 100 else target_group_arn_1
              elif not instances_tg1:
                  chosen_target_group = target_group_arn_1
              elif not instances_tg2:
                  chosen_target_group = target_group_arn_2
              else:
                  chosen_target_group = None
              return chosen_target_group, target_group_arn_1, target_group_arn_2, instances_tg1, instances_tg2, weight_tg1, weight_tg2

          # Take Instances IDs registered to the Target Groups
          def get_instance_ids(elb_client, target_group_arn):
            response = elb_client.describe_target_health(TargetGroupArn=target_group_arn)
            instance_ids = [target['Target']['Id'] for target in response['TargetHealthDescriptions'] if target['TargetHealth']['State'] == 'healthy']
            return instance_ids

          def wait_for_instances_to_be_healthy(elbv2_client, target_group_arn, instance_ids, max_wait_time=300, polling_interval=10):
              start_time = time.time()

              while time.time() - start_time < max_wait_time:
                  # Describe the health of the instances in the target group
                  health_response = elbv2_client.describe_target_health(TargetGroupArn=target_group_arn)

                  # Check if all specified instances are healthy
                  healthy_instance_ids = {health['Target']['Id'] for health in health_response['TargetHealthDescriptions'] if health['TargetHealth']['State'] == 'healthy'}
                  if set(instance_ids) == healthy_instance_ids:
                      print(f"[INFO] All instances {instance_ids} are healthy in {target_group_arn}")
                      return

                  # Wait before the next check
                  time.sleep(polling_interval)

              # If the loop exits, raise an exception indicating that instances are not healthy
              raise Exception(f"[ERROR] Instances {instance_ids} in target group {target_group_arn} did not become healthy within the specified time.")

          def swap_weights(elb_client, listener_arn, target_group_arn_1, target_group_arn_2, weight_tg1, weight_tg2):
              # Swap the weights for the listener rule
              elb_client.modify_listener(
                  ListenerArn=listener_arn,
                  DefaultActions=[
                      {
                          "Type": "forward",
                          "ForwardConfig": {
                              "TargetGroups": [
                                  {
                                      "TargetGroupArn": target_group_arn_1,
                                      "Weight": weight_tg2
                                  },
                                  {
                                      "TargetGroupArn": target_group_arn_2,
                                      "Weight": weight_tg1
                                  }
                              ]
                          }
                      }
                  ]
              )
              print("Weight swap completed successfully")
      nextStep: SendNotification

Deployment and infrastructure schemas:

Prerequisites:

Before you start, make sure the following requirements are met:

An AWS account with permissions to create resources.
AWS CLI installed on your local machine.

Deployment:

Clone the repository.

    git clone https://gitlab.com/Andr1500/ssm_runbook_bluegreen.git

Create an S3 bucket with a unique name for nested stack templates and application files. Here are requirements to the bucket naming rules.

    on Linux:
    date=$(date +%Y%m%d%H%M%S)

    on Windows PowerShell:
    $date = Get-Date -Format "yyyyMMddHHmmss"

    aws s3api create-bucket --bucket cloudformation-app-files-${date} --region YOUR_REGION \
     --create-bucket-configuration LocationConstraint=YOUR_REGION

Add policy to the S3 bucket for access from the EC2 instance.

    aws s3api put-bucket-policy --bucket cloudformation-app-files-${date} \
    --policy '{"Version":"2012–10–17","Statement":[{"Effect":"Allow","Principal":{"Service":"ec2.amazonaws.com"},"Action":"s3:GetObject","Resource":"arn:aws:s3:::'"cloudformation-app-files-${date}"'/*"}]}'

Fill in all necessary Parameters in the infrastructure/root.yaml file and send all nested stack files to the S3 bucket.

    aws s3 cp infrastructure s3://cloudformation-app-files-${date}/infrastructure  --recursive

Go to the infrastructure directory and create CloudFormation stack.

    aws cloudformation create-stack \
        --stack-name ec2-bluegreen-deployment \
        --template-body file://root.yaml \
        --capabilities CAPABILITY_NAMED_IAM \
        --parameters ParameterKey=UserData,ParameterValue="$(base64 -i user_data.txt)" \
        --disable-rollback

Open your mailbox and confirm your subscription to the SNS topic. Access to the deployed EC2 instance is possible through the Systems Manager. Go to AWS console -> AWS Systems Manager -> Fleet Manager -> choose the created EC2 instance -> Node actions -> Connect -> Start terminal session. Here you can check if everything was created and configured correctly during the deployment process.
For manual deployment. Send all application files to the S3 bucket.

    aws s3 cp application s3://cloudformation-app-files-${date}/application  --recursive

Start Systems Manager Automation runbook execution, if everything is ok — you receive an email with the information “Deployment Status: Success” and will have WEB access to the deployed EC2 instance through the WEB browser, in case of any failure — you receive an email with information “Deployment Status: Failed” and details about the failed step. For deployment of a new version of the application — make changes in infrastructure/index.html, send changes to the S3 bucket, and start the Systems Manager Automation runbook again.

    aws ssm start-automation-execution --document-name "Ec2BlueGreenDeployment"

Deployment test. In tests/test_deployment.sh you can find a simple script for making a test of availability and response from “Green” and “Blue” EC2 instances.
Deletion all files from the S3 bucket and deletion of the S3 bucket.

    aws s3 rm s3://cloudformation-app-files-${date} --recursive
    aws s3 rb s3://cloudformation-app-files-${date} --force

Conclusion:

In this post, we showed how to perform Blue/Green deployment with Application Load Balancer’s weighted target group feature realised with Systems Manager Automation runbook.

If you found this post helpful and interesting, please click the reaction button below to show your support for the author. Feel free to use and share this post!