Containers are becoming the standard way of deploying software. Every cloud vendor now offers one or multiple ways to run containers on their platform. Most of our clients uses AWS to host their SaaS solution. As part of a new development for one of our clients we have decided to move away from Elastic Beanstalk and embrace containers. Amazon Elastic Container Service is an orchestration service that supports Docker containers and is generally available for over a year. Given our small development team it seemed like the best choice since it takes away most of the cluster management headaches. In this post I will describe how we deploy a container to ECS using CloudFormation.
ECS Cluster definition
At Bright Inventions we often use CloudFormation for infrastructure configuration since it allows us to version and track changes easily. The first piece of infrastructure we need is an ECS cluster. A cluster is a logical group of tasks/containers running inside ECS. In particular, since we will be using EC2 Launch Type, a cluster can also be though of as a group of EC2 instances with an ECS Agent installed.
"ECSMainCluster": {
"Type": "AWS::ECS::Cluster",
"Properties": {
"ClusterName": "app-stack-main"
}
},
"ECSAutoScalingGroup": {
"Type": "AWS::AutoScaling::AutoScalingGroup",
"Properties": {
"VPCZoneIdentifier": [
{ "Ref": "PrivateASubnet" },
{ "Ref": "PrivateBSubnet" }
],
"LaunchConfigurationName": {
"Ref": "ContainerHostInstances"
},
"MinSize": "1",
"MaxSize": "6",
"DesiredCapacity": "1",
"Tags": [
{
"Key": "Name",
"Value": "app-stack-ecs",
"PropagateAtLaunch": true
}
]
},
"CreationPolicy": {
"ResourceSignal": {
"Timeout": "PT5M"
}
},
"UpdatePolicy": {
"AutoScalingReplacingUpdate": {
"WillReplace": "true"
}
}
}
As you can see above, the ECSMainCluster is mostly a declaration. What follows is an auto scaling group that will launch and manage EC2 instances. The VPCZoneIdentifier lists 2 VPC subnets created in separate availability zones. This is vital for availability as it causes the EC2 instances to run on physically separate hardware. For brevity, I've omitted their configuration from this post. However, if you are interested in this topic head over to this post. The specified LaunchConfigurationName named ContainerHostInstances details how the EC2 instance should look like.
"ContainerHostInstances": {
"Type": "AWS::AutoScaling::LaunchConfiguration",
"Properties": {
"ImageId": "ami-880d64f1",
"SecurityGroups": [
{ "Ref": "ECSSecurityGroup" }
],
"InstanceType": "t2.medium",
"IamInstanceProfile": { "Ref": "ECSHostEC2InstanceProfile" },
"KeyName": "private-key-pair",
"UserData": {
"Fn::Base64": {
"Fn::Join": [
"",
[
"#!/bin/bash -xe\n",
"echo ECS_CLUSTER=",
{
"Ref": "ECSMainCluster"
},
" >> /etc/ecs/ecs.config\n",
"yum install -y aws-cfn-bootstrap\n",
"/opt/aws/bin/cfn-signal -e $? ",
" --stack ",
{
"Ref": "AWS::StackName"
},
" --resource ECSAutoScalingGroup ",
" --region ",
{
"Ref": "AWS::Region"
},
"\n"
]
]
}
}
}
},
"ECSHostEC2InstanceProfile": {
"Type": "AWS::IAM::InstanceProfile",
"Properties": {
"Path": "/",
"Roles": [
{
"Ref": "ECSHostEC2Role"
}
]
}
},
The first important property is the ImageId which uses Amazon ECS-optimized Linux AMI ID. Next we have a security group that adds rules for incoming traffic on application ports. Next we have IamInstanceProfile which references an instance profile ECSHostEC2InstanceProfile that in turn assumes a role policy required by the ECS Agent to deploy and configure containers.
Inside UserData we define a shell script that informs ECS Agent about the cluster it is running in. I will omit ECSHostEC2Role definition since it is well described in the documentation.
With the above we are now ready to deploy an ECS Cluster through CloudFormation template. However, a cluster without containers is pretty meaningless.
ECS Service and Task definition
In AWS lingo an ECS Service describes a minimal configuration required to deploy and run a Task Definition. A Task Definition in turn describes how to configure and run a set of containers that form a single logical component.
"EmailSenderService": {
"Type": "AWS::ECS::Service",
"Properties": {
"Cluster": { "Ref": "ECSMainCluster" },
"DesiredCount": 2,
"DeploymentConfiguration": { "MinimumHealthyPercent": 50 },
"Role": { "Ref": "ECSServiceRole" },
"TaskDefinition": { "Ref": "EmailSenderTask" }
}
},
"EmailSenderTask": {
"Type": "AWS::ECS::TaskDefinition",
"Properties": {
"Family": "app-stack-email-sender",
"ContainerDefinitions": [{
"Name": "app-stack-email-sender",
"Essential": "true",
"Image": { "Ref": "EmailSenderTaskDockerImage" },
"LogConfiguration": {
"LogDriver": "awslogs",
"Options": {
"awslogs-group": { "Ref": "EmailSenderLogsGroup" },
"awslogs-region": { "Ref": "AWS::Region" },
"awslogs-stream-prefix": "email-sender",
"awslogs-datetime-format": "%Y-%m-%d %H:%M:%S.%L"
}
},
"PortMappings": [{ "ContainerPort": 8080 }],
"Environment": [{
"Name": "DEPLOY_ENV",
"Value": { "Ref": "DeployEnv" }
}]
}]
}
},
"EmailSenderLogsGroup": {
"Type": "AWS::Logs::LogGroup",
"Properties": {
"LogGroupName": "app-stack-email-sender",
"RetentionInDays": 14
}
}
The EmailSenderService is pretty straightforward to understand. The EmailSenderTask defines a single container. The app-stack-email-sender task definition states that the Image is a reference to a parameter passed in to the CloudFormation template when creating or updating the stack. Its value must be a name of a Docker image that can be pulled by the ECS Agent. The repository can either be public or private. When hosting your own, private Docker image repository you need to make sure the ECS Agent has the correct credentials configured. Thankfully there is Elastic Container Registry which offers private repositories that are automatically configured when using ECS as long as ECSHostEC2Role policy allows ECR related actions.
Next we have the LogConfiguration that pushes containers logs to the EmailSenderLogsGroup CloudWatch Log Group so that we can can inspect them through AWS Console. The PortMappings lists ports exposed by a running container. Note that we have not defined the host port and it will get assigned automatically. This is important when running multiple instances of the same container. I'll describe it in a bit more detail in the next post. Last but not least the Environment section lists environment variables passed to the container instances on startup. Here we are referencing a DeployEnv stack parameter that allows us to inform the application running inside the container about the current deployment environment e.g. staging vs production.
As you can see above it takes couple of steps to use CloudFormation to deploy a container to ECS. It is true that it requires more configuration than Elastic Beanstalk. However, it allows for better utilization of EC2 instances and an uniform approach to deployment and configuration regardless of the application technology used inside the container. Moreover it is more future proof as with few adjustments it should be possible to switch to Fargate Launch Mode. Using this mode releases us from the burden of EC2 ECS cluster management tasks. Deploying more services and tasks will require separate CloudFormation resource definitions. However, with the help of cloudform it easy keep the CloudFormation template DRY.
Originally published at brightinventions.pl
By Piotr Mionskowski, Software Engineer @ Bright Inventions
Email Stackoverflow Personal blog
Top comments (6)
Interesting article, I shall have to give this a try. Have you looked into Terraform from Hashicorp? It's an easy way to define infrastructure for docker containers much like you've done but it can be used across different platforms. I looked how it can work out what you've got and what you want and only make changes that it needs to to make the target match your definition, bringing them in sync.
Terraform is a more sane way to drive cloudformation essentially (when you run it on aws). Cloudformation has gotten slightly better over the years with yaml support but it is still quite messy to deal with.
I find cloudformation gets rather unwieldly and it is rather unforgiving if you have a tiny mistake in your thousands of lines of templates. Add to that the lengthy edit, deploy, wait for it, oh shit that didn't work either cycles and you are looking at hell. I seem to get stuck with doing relatively simple changes quite often because of this. I sort of dread having to change anything because I know it is going to suck up hours of my time when I sit down and do it.
For this reason, I try to minimize the scope of what cloudformation does. Anti patterns here are having lots of software provisioning driven by cloudformation, running it every time you need to deploy a docker container, copy pasting large bits of yaml between stacks (this is where terraform provides better tooling), excessive polling happening either in aws or in your custom templates waiting for stuff to come up, etc.
You might like cloudform.
I'm glad you found the article interesting.
I did look at Terraform a while back but I remember that I wasn't sure if:
The truth is that we use mostly AWS for hosting solutions we build for our clients so we're it's the most familiar platform to us.
The cloudform aim is to use fully-fledged programming language to build the template. There are Transforms too but I find it somewhat easier to use cloudform to reduce the size of infrastructure configuration file.
When we looked into a solution, we didn't particularly want to be tied to amazon, although the implementation you make in terraform is quite specific. From memory, it seemed easier to use and kept everything in sync well so that's our reasons for choosing it.
I haven't checked recently, but terraform seemed to be right up to date with the latest AWS features, so there weren't any causes for concern there for us, but that might have changed (hopefully not!).
We do something similar. One crucial difference, we bake our own amis using packer and ansible. Our amis have everything preinstalled and configured ready to run. This includes docker, aws-cfn-bootstrap, misc beats components (filebeat, auditbeat, metricbeat) that are configured to start logging to our centralized logging cluster (elastic cloud) and a few other things we need.
Pre baking our amis, means we don't lose time at deploy time reinstalling the same bits of software over and over again. All our configuration is injected via the environment or amazon's parameter store (for secure stuff).
I wouldn't call this ideal; at this point I'm looking forward to moving towards a Kubernetes based setup. Eks sounds interesting but I might also end up using kops.