Today's tech world, there's a clear truth that your organization needs to be agile as well as your workloads need to run smooth. Especially in containers area, there are plenty of methods to deploy your containerized workloads to your environment.
In this post, I'd like to mention powering AWS Fargate - a serverless compute that run containers without needing to manage your infrastructure, with Infrastructure as Code AWS CloudFormation - to provision your fargate workloads, including load balancing and rolling-update deployment features.
I assume that you already have that in use or familiar with tech stack that I mentioned below.
Imagine the following case:
- You have created your ECS cluster with Fargate option,
- You already have provisioned internet-facing ELB,
- You already have VPC, subnets and security group for your application.
But in the continuation, you need to:
- Provision your AWS Fargate workloads,
- Attach security group,
- Expose as a ECS service,
- Associate with ELB, create your ELB listener routing rule and more.
All these steps become unmanageable and tedious after number of your application, environment (dev,test,staging,prod) and workload grows.
To make this agile and automated, we'll leverage AWS CloudFormation, combining with GitLab CI/CD.
Step 1 - Building our CloudFormation template
Using AWS CloudFormation to provision and update our resources in AWS environment helps us in a way to centralize and track our each change.
We need to define each resource definitions to our CloudFormation template. This is the core component we'll work on it.
deploy-fargate.yaml
AWSTemplateFormatVersion: 2010-09-09
Description: An example CloudFormation template for Fargate.
Parameters:
VPC:
Type: String
Default: <VPC_ID_HERE>
SubnetPublicA:
Type: String
Default: <PUBLIC_SUBNET_A>
SubnetPublicB:
Type: String
Default: <PUBLIC_SUBNET_B>
SubnetPublicC:
Type: String
Default: <PUBLIC_SUBNET_C>
Image:
Type: String
Default: <ACCOUNT_ID>.dkr.ecr.eu-central-1.amazonaws.com/nginx:latest
ClusterName:
Type: String
Description: ECS_CLUSTER_NAME here
Default: <ECS_CLUSTER_NAME>
ServiceName:
Type: String
Description: ECS_SERVICE_NAME here
Default: "API_NAME-prod-svc"
TaskDefinitionName:
Type: String
Description: Task Definition Name
Default: "API_NAME-prod-fargate"
ContainerPort:
Type: Number
Default: 3000
ContainerSecurityGroup:
Type: String
Description: api-container-sec-rules
Default: <SECURITY_GROUP_ID>
ELBListenerArn:
Type: String
Default: <ELB_LISTENER_ARN>
Resources:
TaskDefinition:
Type: AWS::ECS::TaskDefinition
Properties:
# Name of the task definition.
Family: !Ref TaskDefinitionName
NetworkMode: awsvpc
RequiresCompatibilities:
- FARGATE
# 1024 (1 vCPU) - Available memory values: 2GB, 3GB, 4GB, 5GB, 6GB, 7GB, 8GB
Cpu: 1024
# 2GB, 3GB, 4GB, 5GB, 6GB, 7GB, 8GB - Available cpu values: 1024 (1 vCPU)
Memory: 3GB
# "The ARN of the task execution role that containers in this task can assume. All containers in this task are granted the permissions that are specified in this role."
ExecutionRoleArn: !GetAtt ExecutionRole.Arn
# "The (ARN) of an IAM role that grants containers in the task permission to call AWS APIs on your behalf."
TaskRoleArn: !Ref TaskRole
ContainerDefinitions:
- Name: API_NAME
Image: !Ref Image
Cpu: 0
Essential: true
PortMappings:
- ContainerPort: !Ref ContainerPort
Protocol: tcp
LogConfiguration:
LogDriver: awslogs
Options:
awslogs-region: !Ref AWS::Region
awslogs-group: !Ref LogGroup
awslogs-stream-prefix: ecs
LogGroup:
Type: AWS::Logs::LogGroup
Properties:
LogGroupName: !Join ['', [/ecs/, !Ref TaskDefinitionName]]
RetentionInDays: 14
# A role needed by ECS
ExecutionRole:
Type: AWS::IAM::Role
Properties:
RoleName: !Join ['', [!Ref ServiceName, "ECSExecutionRole"]]
AssumeRolePolicyDocument:
Statement:
- Effect: Allow
Principal:
Service: ecs-tasks.amazonaws.com
Action: 'sts:AssumeRole'
ManagedPolicyArns:
- 'arn:aws:iam::aws:policy/service-role/AmazonECSTaskExecutionRolePolicy'
-'arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly'
# A role for the containers
TaskRole:
Type: AWS::IAM::Role
Properties:
RoleName: !Join ['', [!Ref ServiceName, "ECSTaskRole"]]
AssumeRolePolicyDocument:
Statement:
- Effect: Allow
Principal:
Service: ecs-tasks.amazonaws.com
Action: 'sts:AssumeRole'
ManagedPolicyArns:
- 'arn:aws:iam::aws:policy/service-role/AmazonECSTaskExecutionRolePolicy'
- 'arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly'
Service:
Type: AWS::ECS::Service
DependsOn:
- LoadBalancerListenerRule
Properties:
ServiceName: !Ref ServiceName
Cluster: !Ref ClusterName
TaskDefinition: !Ref TaskDefinition
DeploymentConfiguration:
MinimumHealthyPercent: 100
MaximumPercent: 200
DesiredCount: 1
HealthCheckGracePeriodSeconds: 120
CapacityProviderStrategy:
- CapacityProvider: FARGATE_SPOT
Base: 0
Weight: 1
NetworkConfiguration:
AwsvpcConfiguration:
AssignPublicIp: ENABLED
Subnets:
- !Ref SubnetPublicA
- !Ref SubnetPublicB
- !Ref SubnetPublicC
SecurityGroups:
- !Ref ContainerSecurityGroup
LoadBalancers:
- ContainerName: API_NAME
ContainerPort: !Ref ContainerPort
TargetGroupArn: !Ref TargetGroup
TargetGroup:
Type: AWS::ElasticLoadBalancingV2::TargetGroup
Properties:
HealthCheckIntervalSeconds: 30
HealthCheckPath: /API_NAME/health
HealthCheckTimeoutSeconds: 5
UnhealthyThresholdCount: 2
HealthyThresholdCount: 3
TargetType: ip
Name: !Ref ServiceName
Port: !Ref ContainerPort
Protocol: HTTP
TargetGroupAttributes:
- Key: deregistration_delay.timeout_seconds
Value: 30 #default 300 seconds
VpcId: !Ref VPC
LambdaDescribeELBListenerPriority:
Type: 'Custom::LambdaDescribeELBListenerPriority'
Properties:
ServiceToken: 'arn:aws:lambda:eu-central-1:<ACCOUNT_ID>:function:DescribeELBListener'
LoadBalancerListenerRule:
Type: AWS::ElasticLoadBalancingV2::ListenerRule
#DependsOn: GetListenerRulesLambdaFunction
Properties:
Actions:
- Type: forward
TargetGroupArn: !Ref TargetGroup
Conditions:
- Field: host-header
HostHeaderConfig:
Values:
- "api.example.com"
- Field: path-pattern
PathPatternConfig:
Values:
- "/API_NAME*"
ListenerArn: !Ref ELBListenerArn
Priority: !GetAtt LambdaDescribeELBListenerPriority.NextPriorityValue
Outputs:
NextPriorityValue:
Value: !GetAtt LambdaDescribeELBListenerPriority.NextPriorityValue
In this template, a few sections need to be mentioned to clarify the case we are dealing with.
In the Parameters section, provide some constants that we already created before such as VPC Id, Subnets, ECS Cluster Name, Security Group, ELB Listener etc.
You can also create these from scratch but in my case they all have created before.As a Capacity Provider, FARGATE_SPOT is used, but you can change it to FARGATE as your needs or you can benefit combining both.
As a placeholder, API_NAME is used. Replace API_NAME with application name that needed to be run on AWS Fargate.
We did not declare auto scale actions for Fargate service. Desired count set to 1. Planning to mention auto scale policy for Fargate in the next post.
Declared Custom Resource LambdaDescribeELBListenerPriority which describes the ELB Listener and finds next available priority number to create listener routing rule.
That custom resource is a bit headache, I expect CloudFormation to handle automatically finding next available ELB Listener priority number and put my rule to there. But it does not. It expects you to provide priority number. In a development lifecycle and CI/CD perspective, it's impossible to know which priority number is free and set it before running CloudFormation template. Therefore, we write a simple lambda function that describes ELB Listener, takes max priority number and adds +1 to create available priority number.
I provide the related lambda function below.
DescribeELBListener
import boto3
import json
import urllib3
http = urllib3.PoolManager()
SUCCESS = "SUCCESS"
FAILED = "FAILED"
def lambda_handler(event, context):
elbv2 = boto3.client('elbv2')
# Get all listener rules for the provided ARN
response = elbv2.describe_rules(ListenerArn='<ELB_LISTENER_ARN>')
# Filter out rules with non-numeric priorities
filtered_rules = [rule for rule in response['Rules'] if str(rule['Priority']).isdigit()]
# Find the maximum priority among the remaining rules
max_priority = max(filtered_rules, key=lambda x: x['Priority'])['Priority']
# Prepare the CloudFormation (CF) stack event response payload
responseValue = int(max_priority) + 1
responseData = {'NextPriorityValue': responseValue}
send(event, context, SUCCESS, responseData)
def send(event, context, responseStatus, responseData, physicalResourceId=None, noEcho=False, reason=None):
responseUrl = event['ResponseURL']
responseBody = {
'Status': responseStatus,
'Reason': reason or f'See the details in CloudWatch Log Stream: {context.log_stream_name}',
'PhysicalResourceId': physicalResourceId or context.log_stream_name,
'StackId': event['StackId'],
'RequestId': event['RequestId'],
'LogicalResourceId': event['LogicalResourceId'],
'NoEcho': noEcho,
'Data': responseData
}
json_responseBody = json.dumps(responseBody, default=str)
headers = {
'content-type': '',
'content-length': str(len(json_responseBody))
}
try:
response = http.request('PUT', responseUrl, headers=headers, body=json_responseBody)
print("Status code:", response.status)
except Exception as e:
print("send(..) failed executing http.request(..):", e)
Step 2 - Prepare CI/CD Pipeline - GitLab
Now we need to prepare CI/CD side to run our CloudFormation template.
Never use your own credentials to access AWS and run CloudFormation template from your local environment.
Here, we use centralized private GitLab to run CloudFormation stack and provided AWS access by following least-privileged permission policy to access resources.
Pay attention to "image": ".dkr.ecr.eu-central-1.amazonaws.com/nginx:latest" section. Latest tag will be replaced with our respectful container image tag during CI/CD job.
Building GitLab CI/CD Pipeline
To automate and run our IaC template, we leverage version control system, so each our iteration is trackable and easy to apply changes.
- Below fully ready .gitlab-ci.yml file that includes build & IaC deployment stages.
variables:
API: nginx
REGISTRY: "<your_aws_account_number_here>.dkr.ecr.eu-central-1.amazonaws.com"
ECS_CLUSTER_NAME: "YOUR_ECS_CLUSTER_NAME"
ECS_SERVICE_NAME: "${API}-prod-svc"
ECS_TASK_FAMILY: "${API}-prod-fargate"
CF_STACK_NAME: '${API}-cf-template-${CI_PIPELINE_IID}'
stages:
- build
- deploy
- update
before_script:
- echo "Build Name:" "$CI_JOB_NAME"
- echo "Branch:" "$CI_COMMIT_REF_NAME"
- echo "Build Stage:" "$CI_JOB_STAGE"
build:
stage: build
script:
- $(aws ecr get-login --no-include-email --region eu-central-1)
- VER=$(cat ${PWD}/package.json | jq --raw-output '.version')
- echo $VER
- docker build -t ${API} .
- docker tag ${API} ${REGISTRY}/${API}:${VER}-${CI_ENVIRONMENT_NAME}-${CI_PIPELINE_IID}
- docker push ${REGISTRY}/${API}:${VER}-${CI_ENVIRONMENT_NAME}-${CI_PIPELINE_IID}
environment:
name: prod
deploy_cloudformation:
stage: deploy
when: manual
image:
name: amazon/aws-cli:latest
entrypoint: ['']
rules:
- if: $API != "null" && $CI_COMMIT_BRANCH == "master"
script:
- echo "Deploying your IaC CloudFormation..."
- yum install jq -y
- jq -Version
- VER=$(cat ${PWD}/package.json | jq --raw-output '.version')
- echo $VER
- echo "API Name ----> ${API} <----"
- echo "ECS FARGATE Cluster is = ${ECS_CLUSTER_NAME}"
- sed -i 's/API_NAME/'"${API}"'/g' deploy-fargate.yaml #replace API_NAME placeholder with the container that we want to run on AWS Fargate.
- cat deploy-fargate.yaml
- |
aws cloudformation create-stack \
--stack-name $CF_STACK_NAME \
--template-body file://deploy-fargate.yaml \
--capabilities CAPABILITY_NAMED_IAM \
--parameters \
ParameterKey=Image,ParameterValue=${REGISTRY}/${API}:${VER}-${CI_ENVIRONMENT_NAME}-${CI_PIPELINE_IID}
- echo "Visit https://api.example.com to see changes"
needs:
- job: build
optional: true
tags:
- gitlab-dind-runner
environment:
name: prod
url: https://api.example.com
With this, in every CI/CD pipeline we run, build our container image, tag with respectful CI/CD pipeline id, then inject it to CloudFormation template to run it on ECS Fargate.
Updating ECS service - rolling update
In the final step, we need to update ECS service with our updated task revision. To do this, we need to run cloudformation update-stack instead of create-stack. Below pipeline will trigger a rolling update for ECS service.
update_cloudformation:
stage: update
when: manual
image:
name: amazon/aws-cli:latest
entrypoint: ['']
rules:
- if: $API != "null" && $CI_COMMIT_BRANCH == "master"
script:
- echo "Deploying your IaC CloudFormation..."
- yum install jq -y
- jq -Version
- VER=$(cat ${PWD}/package.json | jq --raw-output '.version')
- echo $VER
- echo "API Name ----> ${API} <----"
- echo "ECS FARGATE Cluster is = ${ECS_CLUSTER_NAME}"
- sed -i 's/API_NAME/'"${API}"'/g' deploy-fargate.yaml #replace API_NAME placeholder with the container that we want to run on AWS Fargate.
- cat deploy-fargate.yaml
- |
aws cloudformation update-stack \
--stack-name $CF_STACK_NAME \
--template-body file://deploy-fargate.yaml \
--capabilities CAPABILITY_NAMED_IAM \
--parameters \
ParameterKey=Image,ParameterValue=${REGISTRY}/${API}:${VER}-${CI_ENVIRONMENT_NAME}-${CI_PIPELINE_IID}
- echo "Visit https://api.example.com to see changes"
needs:
- job: build
optional: true
tags:
- gitlab-dind-runner
environment:
name: prod
url: https://api.example.com
That's it! Now you are able to deploy your containerized application with zero downtime, and in an automated way.
In this post, I wanted to mention how to automate deployment process of your container to Amazon ECS Fargate using AWS CloudFormation & GitLab CI/CD.
References:
https://docs.gitlab.com/runner/
https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/AWS_ECS.html
Top comments (0)