AWS community day Workshop: Building Your First DevOps Blue/Green Pipeline with ECS

In this workshop, I will guide you step by step how to build a Blue-Green deployment pipeline with AWS pipeline and deploys on AWS ECS with EC2. This deployment strategy helps teams to first test an app while routing traffic to a non-prod route, when confirmed that the app functions as expected, then the traffic is routed to prod. In case of any failure, a rollback is initiated for the previously working version hence avoiding downtime.

Architecture

App Flow

The Infrastructure consists of an Application Load Balancer which exposes the app on Port 80, routes traffic to the autoscaling group managed by ECS, a fleet of EC2 instances running in two AZs for High availability. Each of these instances have ECS agent installed and the necessary software for running our Docker containers. These interact with Aurora PostgreSQL database in RDS for data storage. Aurora PostgreSQL database is configured to replicate data in another availability zone for a resilient database configuration. When there is a spike in traffic, ECS will launch other containers in order to accommodate the load.

Deployment Flow

Provided the developers do changes and want to update the deployed app, our CI/CD pipeline is configured to automate the deployment. With code stored in GitLab, once there is a push, AWS CodePipeline is triggered via a connection app setup and authenticated to Gitlab, Codebuild builds, containerize and push the image to ECR, next Codedeploy deploys the new app, first will deploy it with port 8080 for tests, if the team is happy with their changes, they can approve for traffic to be switch to the newly deployed container and terminate the former. The former containers can be configured to still run for some time while the newly deployed is tested, if everything is fine, then the last version of the deployed app can be terminated.

Build it step by step

**Note**: _If you face issues putting it up and working, go to the optional stage and deploy with Cloudformation_

Ensure you have a code editor like VScode, you have a Gitlab account, an AWS account will appropriate permissions(full access to RDS, ECS, ECR, IAM), configure awscli on you terminal with access keys

Clone the Code

On your local computer terminal(i use wsl with ubuntu os) run

git clone https://gitlab.com/ndzenyuy/tripmgmt.git
cd tripmgmt
rm -rf .git
git init
code .

It will clone the repository that has the application code to your local machine, enter the folder and remove the .git file so as to remove the url for the repository from whence the code was pulled, therefore making it possible to use the code and push to your own repository.

On the browser, go to GitLab and create an account if you don't have one, in it, create a repository named "tripmgmt", you can make the project public and don't create it with a READme.md file. Now copy the repository http url and go back to your terminal and run the following:

git remote add origin <your repo url>
git add .
git commit -m "Initial commit"
git push --set-upstream origin main

These will push the code to the newly created repository.

Setup AWS Resources

Create required AWS IAM Roles

• Task Execution Role: Grants the Amazon ECS container and Fargate agents permission to make AWS API calls on your behalf.

To create the ecsTaskExecutionRole IAM role

1. Open the IAM console
2. In the navigation pane, choose Roles, Create role.
3. In the Trusted entity type, select AWS Service
4. Under Use case, select Elastic Container Service and Elastic Container Service Task, then choose Next: Permissions.
5. In the Attach permissions policy section, search for AmazonECSTaskExecutionRolePolicy, select the policy, and then choose Next: Review.
6. For Role Name, type ecsTaskExecutionRole and choose Create role.
7. Please save value of the Role ARN in the open text file to use in the later section of the workshop.

• ECS Container Instance Role: This IAM role is required for the EC2 launch type. The Amazon ECS container agent makes calls to the Amazon ECS API on your behalf. Container instances that run the agent require an IAM policy and role for the service to know that the agent belongs to you.

To create the ecsInstanceRole IAM role

1. In the navigation pane, choose Roles, Create role.
2. In the Trusted entity type, select AWS Service
3. Under Use case, select EC2, then choose Next: Permissions.
4. In the Attach permissions policy section, search for AmazonEC2ContainerServiceforEC2Role, select the policy, and then choose Next: Review.
5. For Role Name, type ecsInstanceRole and choose Create role.

• ECS CodeDeploy Role: Before you can use the CodeDeploy blue/green deployment type with Amazon ECS, the CodeDeploy service needs permissions to update your Amazon ECS service on your behalf. These permissions are provided by the CodeDeploy IAM role.

To create the ecsCodeDeployRole IAM role

1. In the navigation pane, choose Roles, Create role.
2. In the Trusted entity type, select AWS Service
3. Under Use case, select CodeDeploy and CodeDeploy - ECS, then choose Next: Permissions.
4. In the Attach permissions policy section, AWSCodeDeployRoleForECS policy should come as selected, choose Next: Review.
5. For Role Name, type ecsCodeDeployRole and choose Create role

Create Container Repository

1. Open the Amazon ECR console.
2. In the navigation pane, under Private Registry choose Repositories.
3. On the Repositories page, choose Create repository.
4. For Repository name, enter a name for your repository like devops/tripmgmtdemo
5. For Tag immutability, keep it Mutable for this workshop. Repositories configured with immutable tags will prevent image tags from being overwritten. For more information, see Image Tag Mutability.
6. For Encryption settings, select AWS Key Management Service (KMS). Use an AWS managed key. To read more about encrypting at rest, see ECR Encryption at Rest.
7. For Scan on push, keep it Disabled for this workshop. Repositories configured to scan on push will start an image scan whenever an image is pushed, otherwise image scans need to be started manually. For more information, see Image Scanning.
8. Choose Create repository.
9. Please save value of the Repository URL in the open text file to use in the later section of the workshop.

Setup Aurora PostgreSQL RDS Database

In this section, we will setup Aurora PostgreSQL RDS with Multi-AZ configuration.

1. Go to the AWS Management Console and open the Amazon RDS console.
2. In the navigation pane, choose Databases.
3. Choose Create database.
4. In Choose a database creation method, choose Standard Create.
5. In Engine options, choose Aurora (PostgreSQL Compatible).
6. In Templates, choose Dev/Test template.
   1. Clear Auto generate a password check box.
   2. Enter Master password value as SuperSecretPGSqlPwd##2006 and enter the same password in Confirm password.
7. In the DB cluster identifier field, give Aurora DB cluster name, tripmgmtdb-cluster
8. To enter your master password, do the following in Credential Settings
9. For Instance configuration, choose db.t4g.medium under Burstable classes (includes t classes)
10. For Availability & durability, choose Create an Aurora Replica/Reader node in a different AZ (recommended for scaled availability)
11. Keep defaults for Connectivity, Babelfish settings, Database authentication, Monitoring sections.
12. For Additional configuration, enter tripmgmt under Initial database name.
13. Choose Create database.
14. For Databases, choose the name of the new Aurora DB cluster. On the RDS console, the details for new DB cluster appear. The DB cluster and its DB instance have a status of creating until the DB cluster is ready to use. When the state changes to available for both, you can connect to the DB cluster. Depending on the DB instance class and the amount of storage, it can take up to 20 minutes before the new DB cluster is available.
15. On the Connectivity & security tab, note the port and the endpoint of the writer DB instance. Please save value of the endpoint and port of the cluster in the open text file to use in the later section of the workshop. Endpoint URL should be of format tripmgmtdb-cluster.cluster-UNIQUEID.AWSREGION.rds.amazonaws.com
16. Click on Writer Node and on the Connectivity & security tab, please save value of the Security Group Id under Security section (i.e. sg-xxxxxxxx) in the open text file to use in the later section of the workshop.

• Create Application Load Balancer and Target Groups

We will create an Amazon EC2 application load balancer. This will be the public endpoint to access Trip Management Monolith Application. The load balancer must use a VPC with two public subnets in different Availability Zones.
First create Target groups for your load balancer

1. Sign in to the AWS Management Console and open the Amazon EC2 console
2. In the navigation pane, choose Target groups and select Create target group.
3. Under Basic configuration, select IP addresses target type.
4. In Name, enter a target group name alb-tg-tripmgmtdemo-1
5. In Protocol choose HTTP. In Port, enter 80.
6. Keep rest of the settings as defaults.
7. Select Next: Register Targets.
8. Select Create target group.
9. Repeat above steps to create another target group for Protocol HTTP and Port 8080, name it alb-tg-tripmgmtdemo-2

To create an Amazon EC2 application load balancer

1. Sign in to the AWS Management Console and open the Amazon EC2 console
2. In the navigation pane, choose Load Balancers, choose Create Load Balancer.
3. Choose Application Load Balancer, and then choose Create.
4. In Name, enter the name of your load balancer tripmgmtdemo-alb
5. In Scheme, choose internet-facing.
6. In IP address type, choose ipv4.
7. Under Network mapping, in VPC, choose the default VPC, and then choose any two availability zones under Mappings. Please save value of the Availability Zone / Subnet ids in the open text file to use in the later section of the workshop.
8. Under Security groups, choose Create new security group
   1. Give it the name ALBSecurityGroup in the Security group name field.
   2. Give it the description in the Description field.
   3. Add Inbound Rule, Allow 80 port (HTTP) inbound traffic from My IP
   4. Add Inbound Rule, Allow 8080 port (CustomTCPPort) inbound traffic from My IP
   5. Select Create security group
   6. Please save value of the newly created Security Group Id (i.e. sg-xxxxxxxx..) in the open text file to use in the later section of the workshop.
   7. Come back to Create Application Load Balancer tab, refresh Security groups and select newly created Security group, removing any existing pre-selected Security groups.
9. Under Listeners and routing, configure two listener ports for your load balancer:
   1. Under Protocol, choose HTTP, and Under Port, enter 80.
   2. Under Default action, select alb-tg-tripmgmtdemo-1 Target Group.
   3. Choose Add listener.
   4. Under Protocol, choose HTTP, and Under Port, enter 8080.
   5. Under Default action, select alb-tg-tripmgmtdemo-2 Target Group.
10. Choose Create load balancer.
11. Go to Load Balancers and click on the newly created load balancer. From the Description tab, please save value of the DNS name of the Load Balancer in the open text file to use in the later section of the workshop.

Setup ECS Cluster

Create ECS Cluster

Amazon ECS creates an Amazon EC2 Auto Scaling launch template and Auto Scaling group on your behalf as part of the AWS CloudFormation stack.

1. Open the Amazon ECS console.
2. In the navigation pane, choose Clusters.
3. On the Clusters page, choose Create Cluster.
4. Under Cluster configuration, for Cluster name, enter ecs-cluster-tripmgmtdemo
5. Under Infrastructure, keep AWS Fargate (serverless) selected and also select Amazon EC2 instances. Next, configure the Auto Scaling group which acts as the capacity provider.

Select Create new group, and then provide the following details about the group:

• For Operating system/Architecture, choose Amazon Linux 2.
• For EC2 instance type, choose t3.large. Here, t3.large selected as required for this workshop to complete in given time.

• For Capacity, enter 2 for the minimum number and 4 for the maximum number of instances to launch in the Auto Scaling group.

• Under Network settings for Amazon EC2 instances, choose default VPC and same two subnets, which were selected at that time of creating Application Load Balancer in previous section.

• Choose Create

Create Security Group for EC2 Container Instance

1. Open the Amazon EC2 console.
2. On the navigation pane, under Network & Security, choose Security Groups
3. On the next page, choose Create security group.
4. For Security group name, enter ECS-ALB-SecurityGroup for the security group name and give meaningful description under Description.
5. Choose default VPC
6. Add Inbound Rule, Allow 80 port (HTTP) inbound traffic from Application Loadbalancer Security Group you have created in earlier section (i.e. sg-xxxxx).
7. Add Inbound Rule, Allow 8080 port (CustomTCPPort) inbound traffic from Application Loadbalancer Security Group you have created in earlier section (i.e. sg-xxxxx).
8. Select Create security group
9. Please save value of the newly created Security Group Id (i.e. sg-xxxxxxxx..) in the open text file to use in the later section of the workshop.

Create ECS Task

1. First create Amazon CloudWatch Log Group, where taskdef is configured to send all logs.

   1. Go to CloudWatch Console
   2. Select Log Groups,
   3. Click Create log group button,
   4. Log Group Name: enter tripmgmt-demo-ecstask-loggrp and Retention setting select 5 days
   5. Choose Create

2. In order to create an ECS Task, we will need to create a ECS Task Definition file. Go to you cli, make sure it is pointing to the folder tripmgmt and delete the file taskdef.json by running the command
   

rm taskdef.json

1. Copy below given environment variables in plain text-file and change respective environment variable values as per the workshop resources created by you in the previous sections. After modifying values of respective environment variables, execute these export commands on the commandline.

export LOG_GROUP=tripmgmt-demo-ecstask-loggrp
export DB_USERNAME=postgres
export DB_PASSWORD=SuperSecretPGSqlPwd##2006
export AWS_DEFAULT_REGION=<<AWS-REGION>>

export TASK_EXECUTION_ROLE_ARN="arn:aws:iam::<<AccountID>>:role/ecsTaskExecutionRole"
export AURORA_PGSQL_RDS_URL="tripmgmtdb-cluster.cluster-<<UNIQUEID>>.<<AWS-REGION>>.rds.amazonaws.com"
export ECR_LATEST_IMAGE_URL="<<AccountID>>.dkr.ecr.<<AWS-REGION>>.amazonaws.com/devops/tripmgmtdemo:latest"

• LOG_GROUP : Verify name as you have created in Step No 1 in this section, only change if you have used different log group name.
• DB_USERNAME: Database username set while creating Aurora PostgreSQL DB Cluster.
• DB_PASSWORD: Database password set while creating Aurora PostgreSQL DB Cluster.
• TASK_EXECUTION_ROLE_ARN: Task Execution Role created in Create IAM Roles section.
• AURORA_PGSQL_RDS_URL: Aurora PostgreSQL RDS Cluster URL as created in Create Aurora PostgreSQL DB section.
• ECR_LATEST_IMAGE_URL: Elastic Container Registry Image URL from Create Container Repository section.
• Create the task definition from the command line by running

envsubst < "taskdef_src.json" > "taskdef.json"
aws ecs register-task-definition --cli-input-json file://taskdef.json

1. Please save the value of the TaskDefinition ARN from output in the open text file to use in the later section of the workshop. It is at the very top of the output and should look something like arn:aws:ecs:<>:<>:task-definition/task-tripmgmt:1.

2. Commit taskdef.json in repository.
   

git add taskdef.json
git commit -m "Updated Taskdef"

1. You can also verify the Task Definition in the Amazon ECS console , left navigation bar under Amazon ECS, Task Definitions.

Prepare AppSpec File
AWS CodeDeploy requires AppSpec YAML-formatted file to manage deployment. In the project folder on the terminal, run the code below and edit the value of the taskdefinition arn to reflect what you had in step 4 above(similar to: arn:aws:ecs:<>:<>:task-definition/task-tripmgmt:1.)

nano appspec.yaml

1. Commit appspec.yaml in repository

git add appspec.yaml
git commit -m "appspec file with Taskdef url"
git push

Optional: Deploy the infrastructure with Cloudformation (Recommended)

1. Login to the console and search for cloudformation
2. Create a new stack
3. Prerequisite - Prepare template: Choose existing stack
4. Specify template: Upload a template file
5. Upload a template file: Choose file -> search on your computer, in the tripmgmt/cloudformation-infra folder/subfolder and choose tripmgmt.yml

On specify Stack details

Stack name: tripmgmt
KeyPairName: <select a keypair in you account or create on if absent>
PublicSubnet1Id: <select a public subnet>
PublicSubnet2Id: <select a public subnet>
VpcId: <select the default vpc id>
click next

Scroll down and acknowledge,

and click next scroll and click submit

This will create the infrastructure needed to run our Pipeline.

DevOps Pipeline with Blue/Green Deployment

The Engineering team commit changes to code base and then push those changes to a git repository. It triggers AWS CodePipeline action. AWS CodeBuild will compile source code and build & store container image in an Elastic Container Registry. AWS CodeDeploy will initiate Blue/Green deployment of ECS Task(s) in an ECS Cluster through ECS Service.

Setup CodeBuild with GitLab Integration

To setup AWS CodeBuild, first we need to create/modify buildspec.yml file to describe build steps. After that, we need to configure the CodeBuild Project linking with the GitLab Repository.
On the cli, run the code below and edit the values of the following parameters to suite those you have:

• AWS_ACCOUNT_ID
• AWS_DEFAULT_REGION
• REPOSITORY_URI

nano buildspec.yml

After editing, run ctrl + x, when prompted to save changes press Y then press enter to confirm.

Setup CodeBuild Project

1. Open the CodeBuild console.
2. On the Build projects page, choose Create build project.

1. In Project configuration: Enter a name for this build project tripmgmt-demo-build. Build project names must be unique across each AWS account. You can also include an optional description of the build project to help other users understand what this project is used for.

1. If required, Select Build badge to make your project's build status visible and embeddable.
2. In Source: For Source provider, choose GitLab.
3. You will then get an error stating that you have not connected to GitLab.
4. Press the Manage account credentials link.
5. Select GitLab as your credential type.
6. In Connection, press create a new GitLab connection.
7. Give the connection the name tripmgmt-connection.
8. Press the Connect to GitLab button and authenticate.
9. Authorize AWS Connector for GitLab.
10. After being redirected back to the AWS Console, press Connect.
11. You should now be able to save and add your new GitLab connection.
12. From Repository, choose the repository you have created for this project demo.

1. For Source version, type main.
2. In Environment: choose Managed Image with Operating system as Ubuntu,
3. For Runtime(s) choose Standard and Image choose the one with Standard:5, having Image Version as Always use the latest image for this runtime version

1. For Service role choose New service role and give it a meaningful name codebuild-tripmgmt-demo-build-service-role
2. Open Aditional configuration in the same section. Check mark Privileged action.
3. For Buildspec, select that you want to Use a buildspec file, add the buildspec.yml file location and name - buildspec.yml

1. For Logs, check mark CloudWatch logs and give Group name as codebuild-logs, Stream name as tripmgmtdemo-build.
2. Click Create build project.

1. Now edit newly created service role codebuild-tripmgmt-demo-build-service-role to allow accessing ECR repository through AmazonEC2ContainerRegistryPowerUser managed policy.

   1. Open codebuild-tripmgmt-demo-build-service-role role from IAM console .
   2. Choose Attach policies.
   3. To narrow the available policies to attach, for Filter, type AmazonEC2ContainerRegistryPowerUser
   4. Check the box to the left of the AWS managed policy and choose Attach policy and Update.

2. Select the build project and Start build to test GitLab and CodeBuild integration.

3. Post successful build, you can verify new docker image in Amazon ECR console.

Deploy on Amazon ECS with EC2

As per the architectural diagram, Engineering team commit changes to code base and then push those changes to a Git repository. It triggers AWS CodePipeline action, AWS CodeBuild will compile source code and build & store container image in an Elastic Container Registry. AWS CodeDeploy will initiate Blue/Green deployment of ECS Task(s) in an ECS Cluster through ECS Service. Upon successful deployment of ECS Task(s), users can access Web Portal through DNS URL pointing to ALB endpoint. ALB endpoint serves client requests from ECS Cluster after automatically adjusting capacity to maintain steady state.

Create CodeDeploy Application

To create a CodeDeploy application

1. Open the CodeDeploy console and choose Create application.
2. In Application name, enter the name you want to use Deploy-Tripmgmt-Demo-App
3. In Compute platform, choose Amazon ECS.
4. Choose Create application.

To create a CodeDeploy deployment group

1. On your application page's Deployment groups tab, choose Create deployment group.
2. In Deployment group name, enter a name that describes the deployment group deploygrp-tripmgmt-demo
3. In Service role, choose a service role ecsCodeDeployRole, which was created in earlier section Create required AWS IAM Roles and grants CodeDeploy access to Amazon ECS.
4. In Environment configuration, choose your Amazon ECS cluster name and service name.
5. From Load balancers, choose the name of the load balancer that serves traffic to your Amazon ECS service.
6. From Production listener port, choose the port and protocol for the listener that serves production traffic to your Amazon ECS service. (Production listener port : 80 and Test listener port : 8080)
7. From Target group 1 name and Target group 2 name, choose the target groups used to route traffic during your deployment. Make sure that these are the target groups you created for your load balancer.

8. Under Deployment settings,

   • Choose Specify when to reroute traffic and choose 0 days, 0 hours, and 5 minutes. This will reroute the traffic to successfully deployed updated tasks after 5 Minutes. This is the time required by you to verify the successfully deployed updated tasks and decide whether to rollback or continue reroute the traffic.
   • Under Deployment configuration, choose CodeDeployDefault:ECSAllAtOnce. It will configure CodeDeploy to shift all traffic to the updated Amazon ECS container at once.
   • Under Original revision termination, and choose 0 days, 0 hours, and 0 minutes. It will configure CodeDeploy to terminate the original tasks immediately after rerouting the traffic. You can choose to keep the original tasks for a desired duration as required. After termination of original tasks, you cannot rollback manually or automatically.

9. Choose Create deployment group.

Create CodePipeline

In this section, we will create a DevOps pipeline with the following actions:

• A source stage with a GitLab action
• A build stage with a CodeBuild action
• A deployment stage with an Amazon ECS deploy action with Blue/Green Deployment.

1. Sign in to the AWS Management Console and open the CodePipeline console
2. On the Welcome page, Getting started page, or the Pipelines page, choose Create pipeline.
3. In Step 1: Choose creation settings, pick Build custom pipeline
4. In Step 2: Choose pipeline settings, in Pipeline name, enter codepipeline-tripmgmt-demo.
5. In Service role, Choose New service role to allow CodePipeline to create a new service role in IAM. Enter name as codepipeline-tripmgmt-demo-service-role

1. In Artifact store, Choose Default location to use the default artifact store, such as the Amazon S3 artifact bucket designated as the default, for your pipeline in the region you have selected for your pipeline.
2. Choose Next.
3. In Step 2: Add source stage, in Source provider, choose GitLab. In Connection select the connection you created earlier. Following this, select the repository and default branch you chose earlier. The rest can be kept at default.
4. Choose Next.
5. In Step 3: Add build stage, in Build provider, choose Other build providers and then AWS CodeBuild. In Project name choose the name of the build project. The rest can be kept default.
6. Choose Next.
7. Choose Skip test stage at the bottom of the page.
8. In Step 4: Add deploy stage, in Deploy provider choose Amazon ECS (Blue/Green)

1. In AWS CodeDeploy application name, choose CodeDeploy Application name. In AWS CodeDeploy deployment group, choose CodeDeploy Application's Deployment Group.
2. In Amazon ECS task definition, choose BuildArtifact and enter taskdef.json.
3. In AWS CodeDeploy AppSpec file, choose BuildArtifact and enter appspec.yaml.
4. Choose Next.

1. Review the information, and then choose Create pipeline. Immediately, stop the Pipeline Execution with Stop and Abandon pop-up action.
   Modify Deploy Stage

   1. Open (or remain in) the CodePipeline console & select the Pipeline name, for example codepipeline-tripmgmt-demo.
   2. Click Edit, and scroll down at bottom and "Edit Stage" , click on Edit icon of Deploy Action Group.
   3. In Input artifacts, choose SourceArtifact and remove BuildArtifact
   4. Modify Amazon ECS task definition, choose SourceArtifact and keep taskdef.json.
   5. Modify AWS CodeDeploy AppSpec file, choose SourceArtifact and keep appspec.yaml.
   6. Choose Done
   7. Scroll up and press Save.

DevOps Pipeline in Action

1. Go to Trip Management Monolith Application code base, preferrably home page and make a visible modification which you can identify after releasing a change. Look at the below screenshot, having (Tripment - New Change) at the end at line#7. File path: src/main/webapp/app/home/home.component.html

1. Commit applied changes and push changes to the GitLab repository. This will trigger the pipeline:

On the browser where you pasted the DNS of the load balancer, check the app that is deployed, it should still be the one without the changes

After the deployment is complete, before traffic is rerouted, check the browser with the 8080 port, you should see the web page with the changes:

Code deploy still deploying the new container on ECS, traffic is still flowing to the initial deployment. Once the deployment stage is complete, it will forward test traffic to the same alb link but on a different port:8080

The pipeline will wait 5 minutes for the newly deployed app to be tested(as configured in code pipeline). Once the time elapses, the pipeline will automatically route traffic

After tests, the engineer can now reroute traffic by clicking on reroute traffic, after they are sure their changes are as expected:

The traffic will instantly be rerouted to the new tasks, and will keep the former tasks should incase there is need for rollback after live traffic is routed to it, now both links for test and prod will display the same app, while waiting for a termination of the former:

Traffic activity:

During the deployment phase, out of expected 2 task, ECS runs 4 tasks, this will ensure that the newly deployed app is stable before the termination of these can be authorized.

Clean-up

On the console, go to Cloudformation stacks and delete tripmgmt stack, or if you created the resources on the console, then delete them in the order they were created. Go to Developer tools and delete the pipeline, in codebuild, delete the build job.