DEV Community: EduardoCusihuaman

Containers, Containers, Containers!

EduardoCusihuaman — Mon, 19 Aug 2024 15:15:19 +0000

Yep, you already know how to build your container. But now you want to get that container running on AWS with as little fuss as possible, right? No problem. ⚡

In this quick guide, I'll show you how to deploy your container to AWS Fargate using CloudFormation. It’s fast, easy, and doesn’t require you to wade through endless steps. Just the essentials to get your container online.

Before We Begin

We’re using CloudFormation here because it’s perfect for quickly spinning up your infrastructure with minimal hassle. Sure, there are other tools like Terraform or CDK, but for a straightforward ECS setup, CloudFormation is your go-to.

We’re keeping it simple: we’ll create a VPC, a couple of public subnets, an ECS cluster, and a load balancer to get your containers up and running. No need for extra complexity—just the basics to get your containerized app deployed fast. And yes, we’ve got IAM roles to keep everything secure. Let’s dive in!

Architecture Boring Stuff
LET'S KICK IT OFF!
- Step-01: Deploy CloudFormation Stack for ECS Fargate
- Step-02: Create AWS IAM User for Pipeline
- Step-03: Add Secrets to GitHub Secrets
- Step-04: Github Actions
- Step-05: Cleaning Up
Pricing
Conclusion

This table of contents should make it easy to navigate through the guide.

Architecture Boring Stuff

User Interaction

User Request: A user sends an HTTP request to the Application Load Balancer (ALB) (ALBDNSName).
ALB Routing: The ALB routes the request to the ECS service, which runs your containerized application.
ECS Task Handling: The ECS tasks, running within the Fargate cluster, handle the request, with containers managed by ECS.

ECS and Load Balancer

ECS Fargate Cluster (ECSFargateCluster): Manages and scales the containers based on defined parameters.
Application Load Balancer (ALB): Distributes incoming traffic to the appropriate ECS tasks, ensuring availability and responsiveness.
Target Group (ALBTargetGroup): Monitors the health of your ECS tasks and directs traffic only to healthy instances.

IAM Roles

IAMRoleForECS: Grants permissions for ECS tasks to pull images from ECR, send logs to CloudWatch, and perform other necessary actions.
ECRCleanupLambdaRole: Allows the Lambda function to delete images from the ECR repository during the cleanup process.

Pipeline User for GitHub Actions

The GitHub Actions workflow uses an IAM user (PipelineUser) to deploy updates to the ECS service. The workflow includes steps for:

Checking out the code.
Building the container image.
Pushing the image to ECR.
Configuring AWS credentials.
Updating the ECS task definition with the new image.
Triggering a new deployment of the ECS service.

Flow

User -> Application Load Balancer -> ECS Service -> ECS Task (Container)
ECS Tasks -> Fargate Cluster (for running containers)
Pipeline User -> GitHub Actions -> ECR and ECS (for deployment)

cloudformation-template.yaml

AWSTemplateFormatVersion: "2010-09-09"
Description: AWS ECS Infrastructure setup using a new VPC

Parameters:
    VpcCidr:
        Description: "CIDR block for the VPC"
        Type: String
        Default: "30.0.0.0/16"

    PublicSubnet1Cidr:
        Description: "CIDR block for the public subnet 1"
        Type: String
        Default: "30.0.1.0/24"

    PublicSubnet2Cidr:
        Description: "CIDR block for the public subnet 2"
        Type: String
        Default: "30.0.2.0/24"

    ECRRepositoryName:
        Description: The name of the ECR repository
        Type: String
        Default: "application-repository"

    ClusterName:
        Description: The name of the ECS Fargate Cluster
        Type: String
        Default: "EcsFargateCluster"

    ServiceName:
        Description: The name of the ECS Service
        Type: String
        Default: "EcsService"

    TaskFamilyName:
        Description: The family name of the ECS Task Definition
        Type: String
        Default: "TaskDefinition"

    ContainerName:
        Description: The name of the container in the ECS Task Definition
        Type: String
        Default: "application-container"

Resources:
    # VPC and Networking resources
    VPC:
        Type: AWS::EC2::VPC
        Properties:
            CidrBlock: !Ref VpcCidr
            EnableDnsSupport: true
            EnableDnsHostnames: true
            Tags:
                - Key: Name
                  Value: vpc

    # Internet Gateway
    InternetGateway:
        Type: AWS::EC2::InternetGateway
        Properties:
            Tags:
                - Key: Name
                  Value: igw

    AttachGateway:
        Type: AWS::EC2::VPCGatewayAttachment
        Properties:
            VpcId: !Ref VPC
            InternetGatewayId: !Ref InternetGateway

    # Public Subnets
    PublicSubnet1:
        Type: AWS::EC2::Subnet
        Properties:
            VpcId: !Ref VPC
            CidrBlock: !Ref PublicSubnet1Cidr
            AvailabilityZone: !Select
                - 0
                - Fn::GetAZs: !Ref "AWS::Region"
            MapPublicIpOnLaunch: true
            Tags:
                - Key: Name
                  Value: public-subnet-1

    PublicSubnet2:
        Type: AWS::EC2::Subnet
        Properties:
            VpcId: !Ref VPC
            CidrBlock: !Ref PublicSubnet2Cidr
            AvailabilityZone: !Select
                - 1
                - Fn::GetAZs: !Ref "AWS::Region"
            MapPublicIpOnLaunch: true
            Tags:
                - Key: Name
                  Value: public-subnet-2

    # Route Tables and Routes
    PublicRouteTable:
        Type: AWS::EC2::RouteTable
        Properties:
            VpcId: !Ref VPC
            Tags:
                - Key: Name
                  Value: public-rt

    PublicRoute:
        Type: AWS::EC2::Route
        Properties:
            RouteTableId: !Ref PublicRouteTable
            DestinationCidrBlock: 0.0.0.0/0
            GatewayId: !Ref InternetGateway

    PublicSubnet1RouteTableAssociation:
        Type: AWS::EC2::SubnetRouteTableAssociation
        Properties:
            SubnetId: !Ref PublicSubnet1
            RouteTableId: !Ref PublicRouteTable

    PublicSubnet2RouteTableAssociation:
        Type: AWS::EC2::SubnetRouteTableAssociation
        Properties:
            SubnetId: !Ref PublicSubnet2
            RouteTableId: !Ref PublicRouteTable

    # Security Group
    SecurityGroup:
        Type: AWS::EC2::SecurityGroup
        Properties:
            GroupDescription: "Default security group for the VPC"
            VpcId: !Ref VPC
            SecurityGroupIngress:
                - IpProtocol: tcp
                  FromPort: 80
                  ToPort: 80
                  CidrIp: 0.0.0.0/0
                - IpProtocol: tcp
                  FromPort: 443
                  ToPort: 443
                  CidrIp: 0.0.0.0/0
                - IpProtocol: tcp
                  FromPort: 22
                  ToPort: 22
                  CidrIp: 0.0.0.0/0
            SecurityGroupEgress:
                - IpProtocol: "-1"
                  CidrIp: 0.0.0.0/0
            Tags:
                - Key: Name
                  Value: default-sg

    # ECS Fargate Cluster
    ECSFargateCluster:
        Type: AWS::ECS::Cluster
        Properties:
            ClusterName: !Ref ClusterName

    # ECR Repository
    ECRRepository:
        Type: AWS::ECR::Repository
        Properties:
            RepositoryName: !Ref ECRRepositoryName

    # Custom Resource Hook for ECR Cleanup
    ECRCleanupHook:
        Type: "Custom::ECRCleanup"
        Properties:
            ServiceToken: !GetAtt ECRCleanupLambdaFunction.Arn
            RepositoryName: !Ref ECRRepositoryName
        DependsOn: ECRRepository

    # ECS Task Definition
    ECSTaskDefinition:
        Type: AWS::ECS::TaskDefinition
        Properties:
            Family: !Ref TaskFamilyName
            NetworkMode: awsvpc
            RequiresCompatibilities:
                - FARGATE
            Cpu: "512"
            Memory: "1024"
            ExecutionRoleArn: !Ref IAMRoleForECS
            TaskRoleArn: !Ref IAMRoleForECS
            ContainerDefinitions:
                - Name: !Ref ContainerName
                  Image: "nginx:latest"
                  PortMappings:
                      - ContainerPort: 80
                  Environment:
                      - Name: ENVIRONMENT
                        Value: "production"
                  LogConfiguration:
                      LogDriver: awslogs
                      Options:
                          awslogs-group: !Ref CloudWatchLogGroup
                          awslogs-region: !Ref "AWS::Region"
                          awslogs-stream-prefix: ecs

    # ECS Service
    ECSService:
        Type: AWS::ECS::Service
        DependsOn: ALB
        Properties:
            Cluster: !Ref ECSFargateCluster
            ServiceName: !Ref ServiceName
            DesiredCount: 2
            LaunchType: FARGATE
            TaskDefinition: !Ref ECSTaskDefinition
            NetworkConfiguration:
                AwsvpcConfiguration:
                    AssignPublicIp: ENABLED
                    SecurityGroups:
                        - !Ref SecurityGroup
                    Subnets:
                        - !Ref PublicSubnet1
                        - !Ref PublicSubnet2
            LoadBalancers:
                - ContainerName: !Ref ContainerName
                  ContainerPort: 80
                  TargetGroupArn: !Ref ALBTargetGroup

    # ALB
    ALB:
        Type: AWS::ElasticLoadBalancingV2::LoadBalancer
        Properties:
            Name: "ALB"
            Subnets:
                - !Ref PublicSubnet1
                - !Ref PublicSubnet2
            SecurityGroups:
                - !Ref SecurityGroup
            LoadBalancerAttributes:
                - Key: idle_timeout.timeout_seconds
                  Value: "60"

    ALBListener:
        Type: AWS::ElasticLoadBalancingV2::Listener
        Properties:
            LoadBalancerArn: !Ref ALB
            Port: 80
            Protocol: HTTP
            DefaultActions:
                - Type: forward
                  TargetGroupArn: !Ref ALBTargetGroup

    ALBTargetGroup:
        Type: AWS::ElasticLoadBalancingV2::TargetGroup
        Properties:
            VpcId: !Ref VPC
            Protocol: HTTP
            Port: 80
            TargetType: ip
            HealthCheckEnabled: true
            HealthCheckPath: "/" # Set to a valid endpoint in your application
            HealthCheckIntervalSeconds: 30
            HealthCheckTimeoutSeconds: 5
            HealthyThresholdCount: 2
            UnhealthyThresholdCount: 3
            Matcher:
                HttpCode: "200"
            Name: "TargetGroup"

    # IAM Role for ECS Task
    IAMRoleForECS:
        Type: AWS::IAM::Role
        Properties:
            AssumeRolePolicyDocument:
                Version: "2012-10-17"
                Statement:
                    - Effect: Allow
                      Principal:
                          Service:
                              - ecs-tasks.amazonaws.com
                      Action:
                          - sts:AssumeRole
            Policies:
                - PolicyName: "ECRPullPolicy"
                  PolicyDocument:
                      Version: "2012-10-17"
                      Statement:
                          - Effect: Allow
                            Action:
                                - ecr:GetAuthorizationToken
                                - ecr:GetDownloadUrlForLayer
                                - ecr:BatchGetImage
                                - ecr:BatchCheckLayerAvailability
                            Resource: "*"
                - PolicyName: "CloudWatchLogsPolicy"
                  PolicyDocument:
                      Version: "2012-10-17"
                      Statement:
                          - Effect: Allow
                            Action:
                                - logs:CreateLogStream
                                - logs:PutLogEvents
                                - logs:CreateLogGroup
                            Resource: "*"

    # CloudWatch Log Group
    CloudWatchLogGroup:
        Type: AWS::Logs::LogGroup
        Properties:
            LogGroupName: "/ecs/logs"
            RetentionInDays: 7

    # Lambda Function to Delete ECR Images
    ECRCleanupLambdaRole:
        Type: AWS::IAM::Role
        Properties:
            AssumeRolePolicyDocument:
                Version: "2012-10-17"
                Statement:
                    - Effect: Allow
                      Principal:
                          Service: lambda.amazonaws.com
                      Action: sts:AssumeRole
            Policies:
                - PolicyName: LambdaCloudWatchLogsPolicy
                  PolicyDocument:
                      Version: "2012-10-17"
                      Statement:
                          - Effect: Allow
                            Action:
                                - logs:CreateLogGroup
                                - logs:CreateLogStream
                                - logs:PutLogEvents
                            Resource:
                                - arn:aws:logs:*:*:*
                - PolicyName: ECRDeletePolicy
                  PolicyDocument:
                      Version: "2012-10-17"
                      Statement:
                          - Effect: Allow
                            Action:
                                - ecr:BatchDeleteImage
                                - ecr:ListImages
                            Resource: !Sub "arn:aws:ecr:${AWS::Region}:${AWS::AccountId}:repository/${ECRRepositoryName}"

    ECRCleanupLambdaFunction:
        Type: AWS::Lambda::Function
        Properties:
            FunctionName: ecr-cleanup-lambda
            Handler: index.handler
            Runtime: python3.9
            Role: !GetAtt ECRCleanupLambdaRole.Arn
            Timeout: 60
            Code:
                ZipFile: |
                    import boto3
                    def handler(event, context):
                        if event['RequestType'] == 'Delete':
                            ecr = boto3.client('ecr')
                            images_to_delete = ecr.list_images(repositoryName=event['repository_name'])['imageIds']
                            if images_to_delete:
                                ecr.batch_delete_image(
                                    repositoryName=event['repository_name'],
                                    imageIds=images_to_delete
                                )
                        return 'Cleanup complete'

Outputs:
    VpcId:
        Value: !Ref VPC
        Description: The VPC Id where the ECS Cluster is deployed.
    ECSFargateCluster:
        Value: !Ref ECSFargateCluster
        Description: The ECS Fargate Cluster created by the template.
    ALBDNSName:
        Value: !GetAtt ALB.DNSName
        Description: The DNS name of the Application Load Balancer.
    ALBEndpoint:
        Value: !Sub "http://${ALB.DNSName}"
        Description: The HTTP endpoint of the Application Load Balancer.

The same template without creating a new vpc:
cloudformation-template-with-existing-vpc.yaml

Note: It's important to edit the task's port if needed. By default, this is set up with a placeholder using nginx listening on port 80 and the health check is set to /. However, you can change these settings to fit your requirements.

LET'S KICK IT OFF!

Alright, buckle up! If you've got an AWS account and the AWS CLI locked and loaded, we're ready to roll. (Pro tip: jq is handy, but hey, we won't judge if you skip it.)

Step-01: Deploy CloudFormation Stack for ECS Fargate

export CLUSTER_NAME=EcsFargateCluster
export SERVICE_NAME=EcsService
export TASK_FAMILY_NAME=TaskDefinition
export REPOSITORY_NAME=application-repository
export CONTAINER_NAME=application-container
export STACK_NAME=aws-ecs-fargate-stack

aws cloudformation deploy \
    --stack-name $STACK_NAME \
    --template-file ecs-fargate-cloudformation.yaml \
    --parameter-overrides \
        ClusterName=$CLUSTER_NAME \
        ServiceName=$SERVICE_NAME \
        TaskFamilyName=$TASK_FAMILY_NAME \
        ECRRepositoryName=$REPOSITORY_NAME \
        ContainerName=$CONTAINER_NAME \
    --capabilities CAPABILITY_NAMED_IAM \
    --region us-east-1

Note: The stack uses predefined default names for easy identification in the pipeline, u can override it:

ClusterName: EcsFargateCluster
ServiceName: EcsService
TaskFamilyName: TaskDefinition
ECRRepositoryName: application-repository
ContainerName: application-container

Step-02: Create AWS IAM User for Pipeline

In this step, we'll create a Pipeline User that GitHub Actions will use. This user will have the necessary policies attached to push container images to ECR and update our ECS service. Here's how to do it:

# Set user name
export USER_NAME="codepipeline"

# Create IAM user
aws iam create-user --user-name $USER_NAME

# Attach necessary policies
# ⚠️ THIS IS NOT PRODUCTION READY - USE A LEAST PRIVILEGE ROLE INSTEAD
aws iam attach-user-policy --user-name $USER_NAME --policy-arn arn:aws:iam::aws:policy/AmazonECS_FullAccess
aws iam attach-user-policy --user-name $USER_NAME --policy-arn arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryFullAccess

# Create access keys for the user
ACCESS_KEYS=$(aws iam create-access-key --user-name $USER_NAME)

# Extract AccessKeyId and SecretAccessKey
ACCESS_KEY_ID=$(echo $ACCESS_KEYS | jq -r '.AccessKey.AccessKeyId')
SECRET_ACCESS_KEY=$(echo $ACCESS_KEYS | jq -r '.AccessKey.SecretAccessKey')

# ⚠️ Output the Access Key details
echo "Access Key ID: $ACCESS_KEY_ID"
echo "Secret Access Key: $SECRET_ACCESS_KEY"

Step-03: Add Secrets to GitHub Secrets

After creating the IAM user and generating the access keys, follow these steps to add these credentials to your GitHub
repository secrets for use in GitHub Actions:

Navigate to Your GitHub Repository:
- Go to the main page of your repository on GitHub.
Access Repository Settings:
- Click on the Settings tab at the top of the repository page.
Navigate to Secrets:
- In the left sidebar, click on Secrets and variables > Actions.
Add New Repository Secrets:
- Click New repository secret and add the following:

AWS_ACCESS_KEY_ID: Enter your AccessKeyId value.
AWS_SECRET_ACCESS_KEY: Enter your SecretAccessKey value.

By doing this, you'll securely store the necessary credentials for your GitHub Actions workflow.

Step-04: Github Actions

Time to automate the deployment with GitHub Actions! Here's how you can set up your pipeline:

Copy the following YAML content into .github/workflows/deploy.yml:

name: Build, Push to ECR, and Deploy to ECS Fargate

on:
  push:
    branches:
      - main

jobs:
  build-and-deploy:
    runs-on: ubuntu-latest

    steps:
      # Step 1: Checkout the latest code from the main branch
      - name: Checkout code
        uses: actions/checkout@v3

      # Step 2: Configure AWS credentials for the GitHub Actions runner
      - name: Configure AWS credentials
        uses: aws-actions/configure-aws-credentials@v1
        with:
          aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
          aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
          aws-region: us-east-1 # Change to your region

      # Step 3: Login to Amazon ECR to push Docker images
      - name: Login to Amazon ECR
        id: login-ecr
        uses: aws-actions/amazon-ecr-login@v1

      # Step 4: Build the Docker image, tag it, and push it to Amazon ECR
      - name: Build, tag, and push image to Amazon ECR
        id: build-image
        env:
          ECR_REGISTRY: ${{ steps.login-ecr.outputs.registry }}
          ECR_REPOSITORY: application-repository # Change this to your ECR repository name
          IMAGE_TAG: ${{ github.sha }}
        run: |
          docker build -t $ECR_REGISTRY/$ECR_REPOSITORY:$IMAGE_TAG .
          docker push $ECR_REGISTRY/$ECR_REPOSITORY:$IMAGE_TAG
          echo "image=$ECR_REGISTRY/$ECR_REPOSITORY:$IMAGE_TAG" >> $GITHUB_OUTPUT

      # Step 5: Download the current ECS task definition to update it with the new image
      - name: Download current task definition
        run: |
          aws ecs describe-task-definition --task-definition TaskDefinition \ # Change this to your Task Family Name
          --query taskDefinition > ecs-task-definition.json

      # Step 6: Replace the image in the ECS task definition with the newly built image
      - name: Fill in the new image ID in the Amazon ECS task definition
        id: task-def
        uses: aws-actions/amazon-ecs-render-task-definition@v1
        with:
          task-definition: ecs-task-definition.json
          container-name: application-container # Change this to your container name
          image: ${{ steps.build-image.outputs.image }}

      # Step 7: Deploy the updated task definition to the ECS Fargate service
      - name: Deploy updated task definition to ECS Fargate
        uses: aws-actions/amazon-ecs-deploy-task-definition@v2
        with:
          task-definition: ${{ steps.task-def.outputs.task-definition }}
          service: EcsService # Change this to your ECS service name
          cluster: EcsFargateCluster # Change this to your ECS Fargate cluster name
          wait-for-service-stability: true

Important: The container image in the ECR repository is updated with each push to the main branch. The service name EcsService, cluster name EcsFargateCluster, and other values are predefined in your CloudFormation stack, ensuring smooth integration with your AWS infrastructure.

Now, commit and push your changes to trigger the pipeline:

git commit -am “feat(ci): deployment”
git push origin main

Once pushed, the pipeline will automatically build, push your Docker image to ECR, and update your ECS Fargate service with the latest code. 🚀

Cleaning Up

To delete the CloudFormation stack, use the following command:

# Set the stack name
export STACK_NAME="aws-ecs-fargate-stack"

# Delete the CloudFormation stack
aws cloudformation delete-stack --stack-name $STACK_NAME

# Delete the IAM user
aws iam delete-user --user-name $USER_NAME

BLAZINGGG ENJOYY 🎉🔥

You can see your pipeline triggered, the Docker image built, pushed to ECR, and your ECS Fargate service updated. This workflow will automatically deploy your changes to the Fargate cluster every time you push to the main branch. 🚀

To check if everything's working, grab the ALB DNS Name from the stack output or the AWS console and make an HTTP request to your application. 🌐

~/edu curl http://your-alb-dns-name/ # Replace with your actual ALB DNS name

Pricing

Here’s an estimated cost breakdown for the AWS ECS infrastructure setup you provided, considering that you have a single ECS Fargate task running:

1. VPC and Subnets

VPC: No additional cost.
Subnets: No direct cost for subnets.

2. Internet Gateway

Cost: $0.045 per hour.

3. ECS Fargate

vCPU (0.5 vCPU): $0.04048 per vCPU-hour.
Memory (1 GB): $0.004445 per GB-hour.

For a single task running 24/7:

vCPU Cost (0.5 vCPU): 0.5 * 24 * 30 * $0.04048 = $14.58 per month.
Memory Cost (1 GB): 1 * 24 * 30 * $0.004445 = $3.20 per month.

4. Application Load Balancer (ALB)

ALB: $0.0225 per hour.
Data Processed by ALB: $0.008 per GB.

For 24/7 operation:

ALB Cost: 24 * 30 * $0.0225 = $16.20 per month.

5. ECR Storage

Cost: $0.10 per GB per month.

6. CloudWatch Logs

Storage: $0.03 per GB.
Ingested Logs: $0.50 per GB.

Total Estimated Monthly Cost

Adding up the individual components, here’s an estimated monthly cost:

ECS Fargate: ~$17.78
ALB: ~$16.20
Internet Gateway: ~$32.40 (if used)
ECR Storage: Variable depending on image size.
CloudWatch Logs: Variable depending on log volume.

Estimated Total: ~$66.38 per month (excluding data transfer and storage variations).

Alternative Pricing without Internet Gateway:
If you use the default AWS network configuration and don’t need an Internet Gateway:

Estimated Total: ~$34.58 per month (excluding Internet Gateway cost).

Conclusion

While serverless and Fargate offer great convenience, costs can escalate quickly. For instance, running a 0.5 vCPU and 1GB memory task costs around $18/month just for compute. Additional services like the Application Load Balancer (ALB) and CloudWatch Logs further increase expenses. If you're working on smaller projects or development environments, consider skipping features like the Internet Gateway or ALB to save costs. Always tailor your setup to your specific needs to optimize cloud spending.

And there you have it! A blazing fast, simple way to deploy your containers on AWS ECS Fargate. Enjoy!

Happy deploying :)

How on earth do you deploy AWS Lambdas?

EduardoCusihuaman — Mon, 05 Aug 2024 16:07:32 +0000

Hey there, if you've landed here, you're probably a developer interested in deploying an AWS Lambda as quickly and simply as possible. ⚡

You’ve come to the right place! Let me share some of my thoughts on Lambda and serverless. Many tutorials out there involve a lot of manual steps and can be hard to maintain or even set up if you just want to try out AWS services as quickly as possible.

Before Everything

This example uses CloudFormation because, although it isn't the best IaC tool, it's the quickest for a one-click deployment POC. Even though Terraform isn’t ideal for this, CDK or SAM may be easy to maintain.

The resources involved are minimal: a bucket where we'll store our code as a zip file, a helper Lambda to create an empty zip as a placeholder and then clean the bucket when we delete it, and our actual Lambda with a public endpoint URL. There is no API Gateway here—too much complexity for a simple POC. Of course, there are IAM roles for each Lambda.

Content

Architecture Boring Stuff
LET’S START!
- Step-01: 🚀 Deploy CloudFormation Stack for Code
- Step-02: 🛠️ Create AWS IAM User for Pipeline
- Step-03: 🔒 Add Secrets to GitHub Secrets
- Step-04: 📦 Dependencies
- Step-05: 🤖 Pipeline Magic
- Cleaning Up

Architecture Boring Stuff

User Interaction

A user sends an HTTP request to the Lambda function URL (LambdaFunctionUrl).
The LambdaFunction handles the HTTP request, assuming the LambdaExecutionRole IAM role for the necessary permissions.
It generates a URL and interacts with an S3 bucket (codebucket), which stores the .NET code.

S3 and Helper Lambda

The S3 bucket (codebucket) stores the .NET code.
A helper Lambda function (HelperS3ObjectsFunction) manages S3 objects within this bucket, such as creating and deleting objects.
The helper Lambda function assumes the HelperS3ObjectsWriteRole IAM role to manage these S3 operations.

IAM Roles

LambdaExecutionRole: Provides the necessary permissions for the primary Lambda function.
HelperS3ObjectsWriteRole: Provides permissions for the helper Lambda function to manage S3 objects.

Pipeline User for GitHub Actions
The GitHub Actions workflow uses an IAM user (PipelineUser) to deploy updates to the Lambda function and upload the packaged code to the S3 bucket. The workflow includes steps for:

Checking out the code
Setting up .NET
Installing dependencies
Building the project
Zipping the Lambda package
Configuring AWS credentials
Uploading the package to S3
Updating the Lambda function code

Flow

User -> Lambda Function URL -> Lambda Function -> S3 Bucket
Helper Lambda -> S3 Bucket (for management tasks)
Pipeline User -> GitHub Actions -> S3 Bucket and Lambda Function (for deployment)

cloudformation-template.yaml

AWSTemplateFormatVersion: '2010-09-09'
Description: Code Bucket with CodeLambda and Helper

Parameters:
  S3BucketName:
    Description: Name for the code bucket
    Type: String
    Default: blazing-lambda-code-bucket
    AllowedPattern: "^[a-z0-9-]{3,63}$"
    ConstraintDescription: "Bucket name must be between 3 and 63 characters long and contain only lowercase letters, numbers, and hyphens."
  LambdaExecutionRoleName:
    Description: "The name of the IAM role for the Lambda function"
    Type: String
    Default: "LambdaExecutionRole"

Resources:
  codebucket:
    Type: AWS::S3::Bucket
    Properties:
      BucketName: !Ref S3BucketName

  HelperS3ObjectsWriteRole:
    Type: AWS::IAM::Role
    Properties:
      Path: /
      AssumeRolePolicyDocument:
        Version: 2012-10-17
        Statement:
          - Effect: Allow
            Principal:
              Service: lambda.amazonaws.com
            Action: sts:AssumeRole
      Policies:
        - PolicyName: S3Access
          PolicyDocument:
            Version: 2012-10-17
            Statement:
              - Sid: AllowLogging
                Effect: Allow
                Action:
                  - "logs:CreateLogGroup"
                  - "logs:CreateLogStream"
                  - "logs:PutLogEvents"
                Resource: "*"
              - Sid: S3BucketAccess
                Effect: Allow
                Action:
                  - "s3:ListBucket"
                  - "s3:GetObject"
                  - "s3:DeleteObject"
                  - "s3:DeleteObjectVersion"
                  - "s3:PutObject"
                Resource:
                  - !Sub "arn:aws:s3:::${S3BucketName}"
                  - !Sub "arn:aws:s3:::${S3BucketName}/*"

  helpers3objectshook:
    Type: "Custom::S3Objects"
    Properties:
      ServiceToken: !GetAtt HelperS3ObjectsFunction.Arn
      Bucket: !Ref S3BucketName
    DependsOn: codebucket

  HelperS3ObjectsFunction:
    Type: AWS::Lambda::Function
    Properties:
      Description: Delete objects from bucket and create my-lambda.zip
      Handler: index.handler
      Runtime: python3.9
      Role: !GetAtt HelperS3ObjectsWriteRole.Arn
      Timeout: 120
      Code:
        ZipFile: |
          import os
          import json
          import cfnresponse
          import boto3
          import logging
          from zipfile import ZipFile

          from botocore.exceptions import ClientError
          client = boto3.client('s3')
          logger = logging.getLogger()
          logger.setLevel(logging.INFO)

          def handler(event, context):
            logger.info("Received event: %s" % json.dumps(event))
            bucket = event['ResourceProperties']['Bucket']

            result = cfnresponse.SUCCESS

            try:
              if event['RequestType'] == 'Delete':
                result = delete_objects(bucket)
              elif event['RequestType'] == 'Create':
                result = create_zip_and_upload(bucket)
            except ClientError as e:
              logger.error('Error: %s', e)
              result = cfnresponse.FAILED

            cfnresponse.send(event, context, result, {})

          def delete_objects(bucket):
            paginator = client.get_paginator('list_objects_v2')
            page_iterator = paginator.paginate(Bucket=bucket)
            objects = [{'Key': x['Key']} for page in page_iterator for x in page['Contents']]
            client.delete_objects(Bucket=bucket, Delete={'Objects': objects})
            return cfnresponse.SUCCESS

          def create_zip_and_upload(bucket):
            zip_file_path='/tmp/my-lambda.zip'
            dummy_file_path='/tmp/dummy.txt'
            with open(dummy_file_path,'w') as dummy_file:
              dummy_file.write("This is a placeholder text.")
            with ZipFile(zip_file_path,'w') as zipf:
              zipf.write(dummy_file_path,arcname='dummy.txt')
            boto3.client('s3').upload_file(zip_file_path,bucket,'my-lambda.zip')

            client.upload_file(zip_file_path, bucket, 'my-lambda.zip')
            return cfnresponse.SUCCESS

  LambdaExecutionRole:
    Type: 'AWS::IAM::Role'
    Properties:
      RoleName: !Ref LambdaExecutionRoleName
      AssumeRolePolicyDocument:
        Version: '2012-10-17'
        Statement:
          - Effect: Allow
            Principal:
              Service: lambda.amazonaws.com
            Action: sts:AssumeRole
      Policies:
        - PolicyName: LambdaS3AccessPolicy
          PolicyDocument:
            Version: '2012-10-17'
            Statement:
              - Effect: Allow
                Action:
                  - s3:GetObject
                  - s3:ListBucket
                Resource:
                  - !Sub "arn:aws:s3:::${S3BucketName}"
                  - !Sub "arn:aws:s3:::${S3BucketName}/*"
        - PolicyName: LambdaCloudWatchLogsPolicy
          PolicyDocument:
            Version: '2012-10-17'
            Statement:
              - Effect: Allow
                Action:
                  - logs:CreateLogGroup
                  - logs:CreateLogStream
                  - logs:PutLogEvents
                Resource:
                  - arn:aws:logs:*:*:*

  CodeLambdaFunction:
    Type: 'AWS::Lambda::Function'
    Properties:
      FunctionName: 'aws-lambda-code'
      Handler: 'ChauMundo::ChauMundo.LambdaEntryPoint::FunctionHandlerAsync'
      Runtime: 'dotnet8'
      Role: !GetAtt LambdaExecutionRole.Arn
      Code:
        S3Bucket: !Ref S3BucketName
        S3Key: 'my-lambda.zip'
      Environment:
        Variables:
          ASPNETCORE_ENVIRONMENT: 'Production'
    DependsOn: HelperS3ObjectsFunction

  CodeLambdaFunctionUrl:
    Type: 'AWS::Lambda::Url'
    Properties:
      AuthType: NONE
      TargetFunctionArn: !GetAtt CodeLambdaFunction.Arn

  CodeLambdaInvokePermission:
    Type: 'AWS::Lambda::Permission'
    Properties:
      FunctionName: !Ref CodeLambdaFunction
      Action: 'lambda:InvokeFunctionUrl'
      Principal: '*'
      FunctionUrlAuthType: 'NONE'

Outputs:
  LambdaFunctionUrl:
    Description: "The URL endpoint of the Lambda function"
    Value: !GetAtt CodeLambdaFunctionUrl.FunctionUrl

LET’S START!

Let's get going if you already have an AWS account and the AWS CLI installed (jq is also necessary but not mandatory).

Step-01: 🚀 Deploy CloudFormation Stack for Code

export S3_BUCKET_NAME=blazing-lambda-code-bucket
export STACK_NAME=aws-serverless-stack

aws cloudformation deploy \
    --stack-name $STACK_NAME \
    --template-file cloudformation-template.yaml \
    --parameter-overrides S3BucketName=$S3_BUCKET_NAME \
    --capabilities CAPABILITY_NAMED_IAM \
    --region us-east-1

Note: The Lambda has a predefined name for easy identification in the pipeline: aws-lambda-code. The Lambda's zip file is called my-lambda.zip—important as it serves as a placeholder for deploying our empty Lambda.

Step-02: 🛠️ Create AWS IAM User for Pipeline

In this step, we'll create a Pipeline User that GitHub Actions will use. This user will have the necessary policies attached to copy files to our bucket and update our Lambda function.

# Set user name
export USER_NAME="codepipeline"

# Create IAM user
aws iam create-user --user-name $USER_NAME

# Attach necessary policies
# ⚠️ THIS IS NOT PRODUCTION READY - USE A LEAST PRIVILEGE ROLE INSTEAD
aws iam attach-user-policy --user-name $USER_NAME --policy-arn arn:aws:iam::aws:policy/AmazonS3FullAccess
aws iam attach-user-policy --user-name $USER_NAME --policy-arn arn:aws:iam::aws:policy/AWSLambda_FullAccess

# Create access keys for the user
ACCESS_KEYS=$(aws iam create-access-key --user-name $USER_NAME)

{
    "AccessKey": {
        "UserName": "codepipeline",
        "AccessKeyId": "AKI***NWJVT",
        "Status": "Active",
        "SecretAccessKey": "ZVAf261Bu***JEaTbiIkQckj",
        "CreateDate": "2024-08-05T13:41:48+00:00"
    }
}

# Extract AccessKeyId and SecretAccessKey
ACCESS_KEY_ID=$(echo $ACCESS_KEYS | jq -r '.AccessKey.AccessKeyId')
SECRET_ACCESS_KEY=$(echo $ACCESS_KEYS | jq -r '.AccessKey.SecretAccessKey')

# ⚠️ Output the Access Key details
echo "Access Key ID: $ACCESS_KEY_ID"
echo "Secret Access Key: $SECRET_ACCESS_KEY"

Step-03: 🔒 Add Secrets to GitHub Secrets

After creating the IAM user and generating the access keys, follow these steps to add these credentials to your GitHub repository secrets for use in GitHub Actions:

Navigate to Secrets in Your GitHub Repository:
- Go to the main page of your repository on GitHub.
- Click on the Settings tab at the top of the repository page.
- In the left sidebar, click on Secrets and variables > Actions.
Add New Repository Secret:
- Click the New repository secret button.
- AWS_ACCESS_KEY_ID:
  - Name: AWS_ACCESS_KEY_ID
  - Value: Enter the AccessKeyId value you obtained.
  - Click Add secret.
- AWS_SECRET_ACCESS_KEY:
  - Name: AWS_SECRET_ACCESS_KEY
  - Value: Enter the SecretAccessKey value you obtained.
  - Click Add secret.
- S3_BUCKET_NAME:
  - Name: S3_BUCKET_NAME
  - Value: Enter your S3 bucket name (e.g., blazing-lambda-code-bucket).
  - Click Add secret.

Following these steps, you'll securely add the credentials to your GitHub repository for use in your GitHub Actions workflow.

Step-04: 📦 Dependencies

Some compiled languages, like in this C# example, need some dependencies:

dotnet add package Amazon.Lambda.AspNetCoreServer
dotnet add package Amazon.Lambda.AspNetCoreServer.Hosting
dotnet add package Amazon.Lambda.Serialization.SystemTextJson

And a small entry point:

using Amazon.Lambda.APIGatewayEvents;
using Amazon.Lambda.AspNetCoreServer;
using Amazon.Lambda.Core;

namespace HelloWorld
{
   public class LambdaEntryPoint : APIGatewayHttpApiV2ProxyFunction
   {
       protected override void Init(IWebHostBuilder builder)
       {
           builder.UseStartup<Startup>();
       }
   }
}

public class Startup
{
   public Startup(IConfiguration configuration)
   {
       Configuration = configuration;
   }

   public IConfiguration Configuration { get; }

   public void ConfigureServices(IServiceCollection services)
   {
       services.AddEndpointsApiExplorer();
       services.AddSwaggerGen();
   }

   public void Configure(IApplicationBuilder app, IWebHostEnvironment env)
   {
       app.UseHttpsRedirection();
       app.UseRouting();
       app.UseEndpoints(endpoints =>
       {
           endpoints.MapGet("/hello", () =>
           {
               return Results.Ok("hi");
           });
       });
   }
}

Docs: AWS Lambda C#

For Java, it's similar:

import org.springframework.cloud.function.adapter.aws.SpringBootRequestHandler;

public class HelloHandler extends SpringBootRequestHandler<String, String> {}

Docs: Spring Cloud Function AWS Lambda

Step-05: 🤖 Pipeline Magic

Copy the pipeline into .github/workflows/deploy.yml:

name: Deploy to AWS Lambda

on:
  push:
    branches:
      - 'main'
      - 'master'

jobs:
  deploy:
    runs-on: ubuntu-latest

    steps:
      - name: Checkout code
        uses: actions/checkout@v4

      - name: Setup .NET
        uses: actions/setup-dotnet@v4
        with:
          dotnet-version: '8.0.x'

      - name: Install dependencies
        run: dotnet restore ./HelloWorld/HelloWorld.csproj

      - name: Build project
        run: dotnet publish ./HelloWorld/HelloWorld.csproj -c Release -o ./publish

      - name: Zip Lambda package
        run: |
          cd publish
          zip -r ../my-lambda.zip .

      - name: Configure AWS credentials
        uses: aws-actions/configure-aws-credentials@v4
        with:
          aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
          aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
          aws-region: us-east-1

      - name: Upload to S3
        run: aws s3 cp my-lambda.zip s3://${{ secrets.S3_BUCKET_NAME }}/my-lambda.zip

      - name: Update Lambda function code
        run: |
          aws lambda update-function-code --function-name aws-lambda-code --s3-bucket ${{ secrets.S3_BUCKET_NAME }} --s3-key my-lambda.zip

Important: The my-lambda.zip was named to overwrite the previous zip with our actual code. aws-lambda-code is predefined in the CloudFormation stack.

git commit -am “feat(ci): deployment”
git push origin main

Cleaning Up

To delete the CloudFormation stack, use the following command:

# Set the stack name
export STACK_NAME="aws-serverless-stack"

# Delete the CloudFormation stack
aws cloudformation delete-stack --stack-name $STACK_NAME

# Delete the IAM user
aws iam delete-user --user-name $USER_NAME

BLAZINGGG ENJOYY 🎉🔥

You can see your pipeline triggered, the code zipped, uploaded to S3, and the Lambda updated. This workflow will trigger on each commit to the main branch. 🚀

We take the Lambda URL from the stack output or directly from the Lambda interface to make an HTTP request. 🌐

~/edu curl https://6u7hqooiek7rasomnytusyjp2i0lecee.lambda-url.us-east-1.on.aws/hello/
hi

The best part? You can go to S3 and, if you want to try out DynamoDB, go to the LambdaExecutionRole and add more permissions. You can do it in IAM or redeploy the CloudFormation stack if you're brave enough.

C# example code with IaC

And there you have it! A blazing fast, simple way to deploy AWS Lambdas. Enjoy!

Happy deploying :)

CICD Static Website to S3 with Github Actions

EduardoCusihuaman — Tue, 23 Feb 2021 16:27:27 +0000

Hi!!
A few days ago, I worked with a developer on a simple landing page with zero complications. He needed a simple way to visualize his changes, obviously in our current AWS infrastructure.

The best way to achieve this was with Github Actions + S3 Static websites, and I'll tell you how I did it.

Steps

Provision an S3 Bucket and IAM User with Cloudformation.
Add the AWS secrets from our IAM user to the Github repository.
Deploy an app to an S3 Bucket through Github actions.

1. Resources provisioning 🏗️

First, we must deploy our resources by AWS Cloudformation. In the following cf template (main.yml), it's defined:

S3 Bucket
Bucket Policy
IAM User
IAM Policy inline

If you have already installed AWS CLI:

aws cloudformation create-stack --stack-name your-stack-name --capabilities CAPABILITY_IAM --template-body file://main.yml --parameters ParameterKey=BucketName,ParameterValue=your-unique-bucket-name

Otherwise:

For this need to login into the AWS Console -> Cloudformation -> Stack -> Create Sack -> Upload a template file.

It will ask you for confirmation that this cloudformation template will create IAM resources, in this case, a user.

2. Secrets 🤫

Now, we go for the IAM User: https://console.aws.amazon.com/iam/

IAM -> Users -> your-new-iam-user -> Security Credentials
Access Key -> Create access key

Download .csv file

We need to set AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY in the GitHub repository's secrets section required to upload files to our S3 bucket.

S3_BUCKET: The name of the destination bucket
S3_BUCKET_REGION: The destination bucket region

Settings -> Secrets -> New repository secret

3. Github Actions 🚀

I highly recommend using this continuous deployment only for stage environments. For that reason, we'll create a QA branch.

git checkout -b QA

Creating branch from Github UI

Next, we'll create our deploy.yml file in the path: .github/workflows

mkdir -p .github/workflows && touch .github/workflows/deploy.yml

If you want to run it on a specific branch, change branches

Github action: https://github.com/Reggionick/s3-deploy.

Our app is already deployed by Github Actions!

Done! 🙌

Check your S3 WebsiteURL from the last Cloudformation output:

http://your_bucket_name.s3-website-your_region.amazonaws.com/

For HTTPS: Configuring a static website using a custom domain registered with Route 53

Bonus 🎁

If you work with react, the following pipeline can help you:

react-router 404 errors: https://via.studio/journal/hosting-a-reactjs-app-with-routing-on-aws-s3
cors-cloudformation example: https://git.io/JtQZK

If you have any comments or improvements for this post, I would be glad to receive them.

Thank you for your time 👋👨‍💻.

DEV Community: EduardoCusihuaman

Containers, Containers, Containers!

Before We Begin

Table of Contents

Architecture Boring Stuff

User Interaction

ECS and Load Balancer

IAM Roles

Pipeline User for GitHub Actions

Flow

LET'S KICK IT OFF!

Step-01: Deploy CloudFormation Stack for ECS Fargate

Step-02: Create AWS IAM User for Pipeline

Step-03: Add Secrets to GitHub Secrets

Step-04: Github Actions

Cleaning Up

BLAZINGGG ENJOYY 🎉🔥

Pricing

1. VPC and Subnets

2. Internet Gateway

3. ECS Fargate

4. Application Load Balancer (ALB)

5. ECR Storage

6. CloudWatch Logs

Total Estimated Monthly Cost

Conclusion

How on earth do you deploy AWS Lambdas?

Before Everything

Content

Architecture Boring Stuff

LET’S START!

Step-01: 🚀 Deploy CloudFormation Stack for Code

Step-02: 🛠️ Create AWS IAM User for Pipeline

Step-03: 🔒 Add Secrets to GitHub Secrets

Step-04: 📦 Dependencies

Step-05: 🤖 Pipeline Magic

Cleaning Up

BLAZINGGG ENJOYY 🎉🔥

CICD Static Website to S3 with Github Actions

Steps

1. Resources provisioning 🏗️

2. Secrets 🤫

3. Github Actions 🚀

Done! 🙌

Bonus 🎁