Powering AWS Fargate with IaC - AWS CloudFormation

Today's tech world, there's a clear truth that your organization needs to be agile as well as your workloads need to run smooth. Especially in containers area, there are plenty of methods to deploy your containerized workloads to your environment.

In this post, I'd like to mention powering AWS Fargate - a serverless compute that run containers without needing to manage your infrastructure, with Infrastructure as Code AWS CloudFormation - to provision your fargate workloads, including load balancing and rolling-update deployment features.

I assume that you already have that in use or familiar with tech stack that I mentioned below.

• Amazon ECS Cluster - AWS Fargate
• AWS CloudFormation
• GitLab CI/CD

Imagine the following case:

• You have created your ECS cluster with Fargate option,
• You already have provisioned internet-facing ELB,
• You already have VPC, subnets and security group for your application.

But in the continuation, you need to:

• Provision your AWS Fargate workloads,
• Attach security group,
• Expose as a ECS service,
• Associate with ELB, create your ELB listener routing rule and more.

All these steps become unmanageable and tedious after number of your application, environment (dev,test,staging,prod) and workload grows.

To make this agile and automated, we'll leverage AWS CloudFormation, combining with GitLab CI/CD.

Step 1 - Building our CloudFormation template

Using AWS CloudFormation to provision and update our resources in AWS environment helps us in a way to centralize and track our each change.

We need to define each resource definitions to our CloudFormation template. This is the core component we'll work on it.

deploy-fargate.yaml

AWSTemplateFormatVersion: 2010-09-09
Description: An example CloudFormation template for Fargate.
Parameters:
  VPC:
    Type: String
    Default: <VPC_ID_HERE>
  SubnetPublicA:
    Type: String
    Default: <PUBLIC_SUBNET_A>
  SubnetPublicB:
    Type: String
    Default: <PUBLIC_SUBNET_B>
  SubnetPublicC:
    Type: String
    Default: <PUBLIC_SUBNET_C>
  Image:
    Type: String
    Default: <ACCOUNT_ID>.dkr.ecr.eu-central-1.amazonaws.com/nginx:latest
  ClusterName:
    Type: String
    Description: ECS_CLUSTER_NAME here
    Default: <ECS_CLUSTER_NAME>   
  ServiceName:
    Type: String
    Description: ECS_SERVICE_NAME here
    Default: "API_NAME-prod-svc"
  TaskDefinitionName: 
    Type: String
    Description: Task Definition Name
    Default: "API_NAME-prod-fargate"
  ContainerPort:
    Type: Number
    Default: 3000
  ContainerSecurityGroup:
    Type: String
    Description: api-container-sec-rules    
    Default: <SECURITY_GROUP_ID>
  ELBListenerArn:
    Type: String
    Default: <ELB_LISTENER_ARN>


Resources:
  TaskDefinition:
    Type: AWS::ECS::TaskDefinition
    Properties:
      # Name of the task definition.
      Family: !Ref TaskDefinitionName
      NetworkMode: awsvpc
      RequiresCompatibilities:
        - FARGATE
      # 1024 (1 vCPU) - Available memory values: 2GB, 3GB, 4GB, 5GB, 6GB, 7GB, 8GB
      Cpu: 1024
      # 2GB, 3GB, 4GB, 5GB, 6GB, 7GB, 8GB - Available cpu values: 1024 (1 vCPU)
      Memory: 3GB
      # "The ARN of the task execution role that containers in this task can assume. All containers in this task are granted the permissions that are specified in this role."
      ExecutionRoleArn: !GetAtt ExecutionRole.Arn
      # "The (ARN) of an IAM role that grants containers in the task permission to call AWS APIs on your behalf."
      TaskRoleArn: !Ref TaskRole
      ContainerDefinitions:
        - Name: API_NAME
          Image: !Ref Image
          Cpu: 0
          Essential: true
          PortMappings:
            - ContainerPort: !Ref ContainerPort
              Protocol: tcp
          LogConfiguration:
            LogDriver: awslogs
            Options:
              awslogs-region: !Ref AWS::Region
              awslogs-group: !Ref LogGroup
              awslogs-stream-prefix: ecs              

  LogGroup:
    Type: AWS::Logs::LogGroup   
    Properties:
      LogGroupName: !Join  ['', [/ecs/, !Ref TaskDefinitionName]]
      RetentionInDays: 14

  # A role needed by ECS
  ExecutionRole:
    Type: AWS::IAM::Role
    Properties:
      RoleName: !Join ['', [!Ref ServiceName, "ECSExecutionRole"]]
      AssumeRolePolicyDocument:
        Statement:
          - Effect: Allow
            Principal:
              Service: ecs-tasks.amazonaws.com
            Action: 'sts:AssumeRole'
      ManagedPolicyArns:
        - 'arn:aws:iam::aws:policy/service-role/AmazonECSTaskExecutionRolePolicy'
        -'arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly'

  # A role for the containers
  TaskRole:
    Type: AWS::IAM::Role
    Properties:
      RoleName: !Join ['', [!Ref ServiceName, "ECSTaskRole"]]
      AssumeRolePolicyDocument:
        Statement:
          - Effect: Allow
            Principal:
              Service: ecs-tasks.amazonaws.com
            Action: 'sts:AssumeRole'
      ManagedPolicyArns:
        - 'arn:aws:iam::aws:policy/service-role/AmazonECSTaskExecutionRolePolicy'
        - 'arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly'


  Service:
    Type: AWS::ECS::Service
    DependsOn:
      - LoadBalancerListenerRule    
    Properties: 
      ServiceName: !Ref ServiceName
      Cluster: !Ref ClusterName
      TaskDefinition: !Ref TaskDefinition
      DeploymentConfiguration:
        MinimumHealthyPercent: 100
        MaximumPercent: 200
      DesiredCount: 1
      HealthCheckGracePeriodSeconds: 120
      CapacityProviderStrategy:
        - CapacityProvider: FARGATE_SPOT
          Base: 0
          Weight: 1      
      NetworkConfiguration: 
        AwsvpcConfiguration:
          AssignPublicIp: ENABLED
          Subnets:
            - !Ref SubnetPublicA
            - !Ref SubnetPublicB
            - !Ref SubnetPublicC
          SecurityGroups:
            - !Ref ContainerSecurityGroup
      LoadBalancers:
        - ContainerName: API_NAME
          ContainerPort: !Ref ContainerPort
          TargetGroupArn: !Ref TargetGroup

  TargetGroup:
    Type: AWS::ElasticLoadBalancingV2::TargetGroup
    Properties:
      HealthCheckIntervalSeconds: 30
      HealthCheckPath: /API_NAME/health
      HealthCheckTimeoutSeconds: 5
      UnhealthyThresholdCount: 2
      HealthyThresholdCount: 3
      TargetType: ip
      Name: !Ref ServiceName
      Port: !Ref ContainerPort
      Protocol: HTTP
      TargetGroupAttributes:
        - Key: deregistration_delay.timeout_seconds
          Value: 30  #default 300 seconds
      VpcId: !Ref VPC

  LambdaDescribeELBListenerPriority:
    Type: 'Custom::LambdaDescribeELBListenerPriority'
    Properties:
      ServiceToken: 'arn:aws:lambda:eu-central-1:<ACCOUNT_ID>:function:DescribeELBListener'

  LoadBalancerListenerRule:
    Type: AWS::ElasticLoadBalancingV2::ListenerRule
    #DependsOn: GetListenerRulesLambdaFunction
    Properties:
      Actions:
        - Type: forward
          TargetGroupArn: !Ref TargetGroup
      Conditions:
        - Field: host-header
          HostHeaderConfig:
            Values:
              - "api.example.com"
        - Field: path-pattern
          PathPatternConfig:
            Values:
              - "/API_NAME*"
      ListenerArn: !Ref ELBListenerArn
      Priority: !GetAtt LambdaDescribeELBListenerPriority.NextPriorityValue

Outputs:
  NextPriorityValue:
    Value: !GetAtt LambdaDescribeELBListenerPriority.NextPriorityValue      

In this template, a few sections need to be mentioned to clarify the case we are dealing with.

• In the Parameters section, provide some constants that we already created before such as VPC Id, Subnets, ECS Cluster Name, Security Group, ELB Listener etc.
  You can also create these from scratch but in my case they all have created before.

• As a Capacity Provider, FARGATE_SPOT is used, but you can change it to FARGATE as your needs or you can benefit combining both.

• As a placeholder, API_NAME is used. Replace API_NAME with application name that needed to be run on AWS Fargate.

• We did not declare auto scale actions for Fargate service. Desired count set to 1. Planning to mention auto scale policy for Fargate in the next post.

• Declared Custom Resource LambdaDescribeELBListenerPriority which describes the ELB Listener and finds next available priority number to create listener routing rule.

That custom resource is a bit headache, I expect CloudFormation to handle automatically finding next available ELB Listener priority number and put my rule to there. But it does not. It expects you to provide priority number. In a development lifecycle and CI/CD perspective, it's impossible to know which priority number is free and set it before running CloudFormation template. Therefore, we write a simple lambda function that describes ELB Listener, takes max priority number and adds +1 to create available priority number.

I provide the related lambda function below.

DescribeELBListener

import boto3
import json
import urllib3

http = urllib3.PoolManager()
SUCCESS = "SUCCESS"
FAILED = "FAILED"

def lambda_handler(event, context):
    elbv2 = boto3.client('elbv2')

    # Get all listener rules for the provided ARN
    response = elbv2.describe_rules(ListenerArn='<ELB_LISTENER_ARN>')

    # Filter out rules with non-numeric priorities
    filtered_rules = [rule for rule in response['Rules'] if str(rule['Priority']).isdigit()]

    # Find the maximum priority among the remaining rules
    max_priority = max(filtered_rules, key=lambda x: x['Priority'])['Priority']

    # Prepare the CloudFormation (CF) stack event response payload    
    responseValue = int(max_priority) + 1
    responseData = {'NextPriorityValue': responseValue}

    send(event, context, SUCCESS, responseData)

def send(event, context, responseStatus, responseData, physicalResourceId=None, noEcho=False, reason=None):
    responseUrl = event['ResponseURL']

    responseBody = {
        'Status': responseStatus,
        'Reason': reason or f'See the details in CloudWatch Log Stream: {context.log_stream_name}',
        'PhysicalResourceId': physicalResourceId or context.log_stream_name,
        'StackId': event['StackId'],
        'RequestId': event['RequestId'],
        'LogicalResourceId': event['LogicalResourceId'],
        'NoEcho': noEcho,
        'Data': responseData
    }

    json_responseBody = json.dumps(responseBody, default=str)

    headers = {
        'content-type': '',
        'content-length': str(len(json_responseBody))
    }
    try:
        response = http.request('PUT', responseUrl, headers=headers, body=json_responseBody)
        print("Status code:", response.status)

    except Exception as e:
        print("send(..) failed executing http.request(..):", e)

Step 2 - Prepare CI/CD Pipeline - GitLab

Now we need to prepare CI/CD side to run our CloudFormation template.

• Never use your own credentials to access AWS and run CloudFormation template from your local environment.

• Here, we use centralized private GitLab to run CloudFormation stack and provided AWS access by following least-privileged permission policy to access resources.

• Pay attention to "image": ".dkr.ecr.eu-central-1.amazonaws.com/nginx:latest" section. Latest tag will be replaced with our respectful container image tag during CI/CD job.

Building GitLab CI/CD Pipeline

To automate and run our IaC template, we leverage version control system, so each our iteration is trackable and easy to apply changes.

• Below fully ready .gitlab-ci.yml file that includes build & IaC deployment stages.

variables:
  API: nginx
  REGISTRY: "<your_aws_account_number_here>.dkr.ecr.eu-central-1.amazonaws.com"
  ECS_CLUSTER_NAME: "YOUR_ECS_CLUSTER_NAME"
  ECS_SERVICE_NAME: "${API}-prod-svc"
  ECS_TASK_FAMILY: "${API}-prod-fargate"
  CF_STACK_NAME: '${API}-cf-template-${CI_PIPELINE_IID}'  

stages:
  - build
  - deploy
  - update

before_script:
  - echo "Build Name:" "$CI_JOB_NAME"
  - echo "Branch:" "$CI_COMMIT_REF_NAME"
  - echo "Build Stage:" "$CI_JOB_STAGE"


build:
  stage: build
  script:
    - $(aws ecr get-login --no-include-email --region eu-central-1)
    - VER=$(cat ${PWD}/package.json | jq --raw-output '.version')
    - echo $VER    
    - docker build -t ${API} .
    - docker tag ${API} ${REGISTRY}/${API}:${VER}-${CI_ENVIRONMENT_NAME}-${CI_PIPELINE_IID}
    - docker push ${REGISTRY}/${API}:${VER}-${CI_ENVIRONMENT_NAME}-${CI_PIPELINE_IID}
  environment:
    name: prod


deploy_cloudformation:
  stage: deploy
  when: manual
  image:
    name: amazon/aws-cli:latest
    entrypoint: ['']
  rules:
    - if: $API != "null" && $CI_COMMIT_BRANCH == "master"
  script:
    - echo "Deploying your IaC CloudFormation..."
    - yum install jq -y
    - jq -Version
    - VER=$(cat ${PWD}/package.json | jq --raw-output '.version')
    - echo $VER
    - echo "API Name ----> ${API} <----"
    - echo "ECS FARGATE Cluster is = ${ECS_CLUSTER_NAME}"
    - sed -i 's/API_NAME/'"${API}"'/g' deploy-fargate.yaml #replace API_NAME placeholder with the container that we want to run on AWS Fargate.
    - cat deploy-fargate.yaml
    - |
      aws cloudformation create-stack \
        --stack-name $CF_STACK_NAME \
        --template-body file://deploy-fargate.yaml \
        --capabilities CAPABILITY_NAMED_IAM \
        --parameters \
      ParameterKey=Image,ParameterValue=${REGISTRY}/${API}:${VER}-${CI_ENVIRONMENT_NAME}-${CI_PIPELINE_IID}
    - echo "Visit https://api.example.com to see changes"
  needs:
    - job: build
      optional: true
  tags:
    - gitlab-dind-runner
  environment:
    name: prod
    url: https://api.example.com

With this, in every CI/CD pipeline we run, build our container image, tag with respectful CI/CD pipeline id, then inject it to CloudFormation template to run it on ECS Fargate.

Updating ECS service - rolling update

In the final step, we need to update ECS service with our updated task revision. To do this, we need to run cloudformation update-stack instead of create-stack. Below pipeline will trigger a rolling update for ECS service.

update_cloudformation:
  stage: update
  when: manual
  image:
    name: amazon/aws-cli:latest
    entrypoint: ['']
  rules:
    - if: $API != "null" && $CI_COMMIT_BRANCH == "master"
  script:
    - echo "Deploying your IaC CloudFormation..."
    - yum install jq -y
    - jq -Version
    - VER=$(cat ${PWD}/package.json | jq --raw-output '.version')
    - echo $VER
    - echo "API Name ----> ${API} <----"
    - echo "ECS FARGATE Cluster is = ${ECS_CLUSTER_NAME}"
    - sed -i 's/API_NAME/'"${API}"'/g' deploy-fargate.yaml #replace API_NAME placeholder with the container that we want to run on AWS Fargate.
    - cat deploy-fargate.yaml
    - |
      aws cloudformation update-stack \
        --stack-name $CF_STACK_NAME \
        --template-body file://deploy-fargate.yaml \
        --capabilities CAPABILITY_NAMED_IAM \
        --parameters \
      ParameterKey=Image,ParameterValue=${REGISTRY}/${API}:${VER}-${CI_ENVIRONMENT_NAME}-${CI_PIPELINE_IID}
    - echo "Visit https://api.example.com to see changes"
  needs:
    - job: build
      optional: true
  tags:
    - gitlab-dind-runner
  environment:
    name: prod
    url: https://api.example.com

That's it! Now you are able to deploy your containerized application with zero downtime, and in an automated way.

In this post, I wanted to mention how to automate deployment process of your container to Amazon ECS Fargate using AWS CloudFormation & GitLab CI/CD.

References:
https://docs.gitlab.com/runner/
https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/AWS_ECS.html