DEV Community: Gregory Ledray

KISS with Docker Compose

Gregory Ledray — Fri, 26 Jul 2024 17:48:51 +0000

KISS - Keep It Simple Stupid - is a design mantra we live by in software engineering. But too often we fail to apply it to AWS infrastructure. I struggle with this because I want to do the right thing. The “right thing” is to follow AWS guidance. Guidance which sells you on replacing open source software with AWS native versions. Yet the more AWS services you use, the harder it is to manage all aspects of those services - necessitating yet more services. It becomes necessary to undergo extensive training to properly understand these services and measure risk, and the complexity of important tasks like disaster recovery balloons.

KISS with Docker Compose takes a different approach. It takes ~~Bikini Bottom and pushes it somewhere else~~ a Docker Compose app from your development machine and copies it into an EC2 instance with ports 80 and 443 open (this, and everything else, is configurable). It then pulls your Docker images and starts Docker-Compose.

The result? Stupid simple infrastructure which works the same way in the cloud as it does on your development machine.

I’ve created an AWS CDK package and published it to Github to help you try this approach: https://github.com/Gregory-Ledray/kiss-docker-compose-on-aws and give a tutorial on how it works below under “Try It Out”.

Evaluating KISS Docker Compose

Evaluation needs to be split into three parts:

Why Bother?
Can Docker Compose meet your requirements on an infinitely powerful and reliable machine?
Is an EC2 Instance sufficient to handle the Docker Compose app’s needs?

Why Bother?

Cheap: All code runs on one small EC2. Compare this to running one small EC2 for your code + a small RDS instance for your database + a NAT gateway. It’s also far less complex and labor intensive to set up and maintain.
Simple: It runs the same way on your machine as it runs in the cloud.
Fast: Works by default.

Can Docker Compose Meet Your Requirements?

Docker supports running Compose on a single server for production, so probably yes. It is production ready with support for health checks, restarting, secrets, environment configuration, networking, GPUs, etc. If the EC2 instance is infinitely powerful and reliable then Docker Compose is probably enough for you. But therein lies the rub.

Is an EC2 Instance Sufficient?

Therein lies the issues, and some good reasons to not use this approach:

Is the 99.99% uptime of an EC2 instance sufficient?
Is your application going to exceed the RAM of the EC2 instance? We are able to provide swap space to help, but that only goes so far.
Is your application going to exceed the file storage of the EC2 instance? By default, we only launch the EC2 instance with 8GB of storage, although this is fully configurable.
Is your application going to run out of CPU? This configuration doesn’t allow for auto scaling.
Disaster recovery requires backups. Doing this safely means stopping the instance, creating a backup of the file storage, and then restarting the instance, which means downtime.
Deployments are simple but cause downtime. Whenever you restart the instance it re-pulls all the Docker Images used by Docker Compose, so to deploy new images you first push those images to the repository and then restart the VM.

When thinking about drawbacks though, we need to consider how solvable these issues are and discount those which are insolvable:

No single instance of ECS or Fargate will have more than 99.99% uptime because they are all built on EC2, so to get more than 99.99% uptime you’ll need to run several instances. If your application containers are stateful, that requires code changes.
If your app exceeds RAM limits + Swap in ECS you’ll also see a node failure. Fargate doesn’t even have swap space. If this is a concern, then (1) seems relevant again.
Exceeding storage is a risk with every approach, but in some recommended architectures some or all of the risk is offloaded to another AWS service like RDS. If you expect to see large growth in storage it’s better to run at least your DB on RDS.
Running out of CPU is a mostly mitigated risk when you use ECS or Fargate.
Creating backups of databases is relatively easy and safe with RDS.
Services like ECS and Fargate deploy changes without downtime.

The Verdict

KISS Docker Compose works for production systems which tolerate 2-3 minutes of downtime during code deployments and system backups. You must also size your EC2 instance so it can handle all traffic spikes.

If you have a fairly predictable or small workload, conduct load testing, and tolerate downtime, then it’s a great low-cost way to run a full stack application on AWS.

Try KISS Docker Compose

First, follow “Getting Started with the AWS CDK” to be able to create CDK applications. Then:

mkdir kissdc
cd kissdc
cdk init app --language typescript
npm i aws-cdk-lib
npm i kiss-docker-compose
curl -O https://raw.githubusercontent.com/Gregory-Ledray/kiss-docker-compose-on-aws/main/test/docker-compose.yml

Now open lib/kissdc-stack.ts and copy-paste:

import * as cdk from 'aws-cdk-lib';
import { Construct } from 'constructs';
import * as fs from 'fs';
import { KissDockerCompose } from 'kiss-docker-compose'

export class KissdcStack extends cdk.Stack {
    constructor(scope: Construct, id: string, props?: cdk.StackProps) {
        super(scope, id, props);

        const dockerComposeFileAsString = fs.readFileSync('./docker-compose.yml', 'utf8');

        const kissDockerCompose = new KissDockerCompose(this, 'kiss-docker-compose', { dockerComposeFileAsString });

        // Exporting the value so you can find it easily
        new cdk.CfnOutput(this, 'Kiss-Docker-Compose-public-ip', {
            value: kissDockerCompose.ec2Instance?.instancePublicDnsName ?? '',
            exportName: 'Kiss-Docker-Compose-public-ip',
        });
    }
}

Deploy:

npx cdk deploy

After deployment finishes, you will see the export Kiss-Docker-Compose-public-ip, as below.

 ✅  KissdcStack

✨  Deployment time: 268.26s

Outputs:
KissdcStack.KissDockerComposepublicip = ec2-98-80-9-243.compute-1.amazonaws.com

To verify the deployment worked, curl that URL and verify you get a response from NGINX. For example:

curl ec2-98-80-9-243.compute-1.amazonaws.com

Finally, destroy the created infrastructure so you don’t rack up AWS charges:

npx cdk destroy

Evolving AWS Infrastructure

This is Part 1 of a series on Evolving AWS Infrastructure. Follow me on dev.to to help find the next posts. This particular post is a follow up to https://dev.to/gregoryledray/apply-kiss-to-infrastructure-3j6d which outlined this approach but didn’t provide the code needed to make it easy to use.

Apply KISS to Infrastructure

Gregory Ledray — Thu, 14 Oct 2021 22:50:55 +0000

I took a system which used to have ~120 resources, deleted half of it, and replaced those resources with one VM. The new system isn’t just simpler - it performs better than the old one.

Oftentimes when I read infrastructure guides for deploying a workload on AWS the guide is walking me through how to use and coordinate a half dozen AWS services to do something simple, like run an API. Don’t do that. Do what I did. Apply KISS (Keep It Simple Stupid) to your infrastructure to meet your infrastructure needs.

The Service: IntelligentRx.com

https://intelligentrx.com gives people discount coupons for prescriptions. It is written in Vue.js and .NET and runs on AWS.

Infrastructure Goals

It must run the code, preferably the same way it works on localhost
It must have a high uptime
It must tell me when it is not working
It should be easy to set up
It should be easy to maintain
It should be inexpensive
It should be secure

When I started, I inherited a complex CloudFormation infrastructure which I expanded horizontally as I added more services. A few weeks ago the site’s CloudFormation infrastructure looked like this:

Today, it looks like this:

It now contains half of the resources it used to contain.

Let’s compare these two setups based on the original goals.

Runs my code: Both. Runs my code the same way I run on localhost? Only the new setup.
High uptime: The new setup may have a higher uptime O.O
Uptime notifications: Both.
Setup cost: The original setup was 1,200 lines of infrastructure-as-code yml. The second setup is 680 lines of infrastructure-as-code yml. The original setup contained about 120 Resources managed by CloudFormation. The second setup contains 66 Resources. Not only is the new system easier to set up because it has fewer moving parts, but the parts which are used are less complex.
Maintenance: Fewer parts => less maintenance and less knowledge needed for maintenance. The deployment time has gone from ~15 minutes to ~5 minutes for each deployment. Naturally, this shaves 10 minutes off my dev cycle every time I make an infrastructure change.
Cost: Fewer parts => cheaper.
Security: Fewer parts => fewer opportunities to mess up configuration & less attack surface => more secure.

The second setup is the clear winner.

What Changed?

The old setup deployed into ECS (Elastic Container Service [runs container images on EC2 for you]). The new setup builds a new container image, restarts the website which is hosted on a t2.large EC2 VM, and then the VM automatically pulls the new image and runs it with Docker Compose.

Yep, that’s right: I took a system which used to have ~70 CloudFormation resources between ECS and the networking resources and Redis and put it all on one chonky VM. Right about now some of you are thinking I’m crazy. Let’s look at the most obvious questions / potential problems with this setup.

What About Scalability?

My biggest fear with this change was that the website would not be able to handle a sudden influx of requests. ECS is more complex, but it also scales. With a single VM on EC2 “scaling” involves manually changing the VM size. That’s not practical if the site has a sudden burst of traffic.

To test if the new system can scale, I used https://loader.io/ to try to figure out how much of a load that one VM can handle. What happens if it receives 100 requests per second? 400 requests per second? As it turns out, this one chonky VM can handle 100 requests per second, every second, for 30 seconds without a sweat:

It can handle 400 requests per second too, although the average response time jumps to 1.1 seconds per request:

Regardless, being able to service 400 requests per second is more than enough for this website.

But What About Other Stress Tests?

Other stress tests can cause the website to fail, but not for want of trying: the parts which fail are 3rd party, not this VM or the AWS infrastructure.

So You Traded Away Reliability and Scalability For Simplicity?

Yes. With EC2, the website goes down for 2 - 3 minutes whenever I deploy a change. If the underlying VM fails, the VM is automatically restarted and the website comes up shortly thereafter.

[Edit] When you restart the VM, isn't there downtime?

Yes. As commenters pointed out, this does cause prod downtime while the VM restarts, pulls the new image, etc (1 - 3 minutes in my experience). I compensate in my frontend code with (pseudocode):

if (prodAPICallFailed) {
  fetchWithRetries('staging.example.com/api/endpoint');
}

This only works if your frontend is hosted somewhere else, like in CloudFront. Readers of this post, please recognize that this approach has some asterisks. There are good performance, observability, and reliability reasons for why I still have 600 lines of yml in my CloudFormation template. For example, I use CloudFront to distribute the frontend's code, stored in S3, via a CDN. However, these other features and components have been easy to maintain and deploy quickly, unlike the dozens of components required to set up load balancing to auto-scaling ECS, which this post's approach replaces.

Does This Really Save Time?

YES! Ways I have saved time:

I went from 15 minutes to test infrastructure changes down to 5 minutes because deployments are faster.
By using Docker Compose, I know that code which works on my machine works the same way on this VM, giving me a lot of peace of mind. I also gained the ability to make and test some configuration changes locally.
I’m resolving infrastructure bugs faster because the key part of the infrastructure - where I run the code - is now using widely used open source technologies like Docker and Systemd. Googling for answers to my problems has become trivial instead of a headache.

How It Works

When I make a code change and push, the CI/CD pipeline kicks off:

The continuous deployment pipeline creates a new docker image or three and pushes it / them to ECR.
CloudFormation is updated (CloudFormation still holds most of the infrastructure)
The VM which runs the code is restarted.
When booting up, the VM runs a systemd daemon.
The daemon’s pre-start script pulls the updated container image into the VM.
The daemon’s command runs docker-compose up
As the application generates logs, they are sent to CloudWatch.

These are the steps you can take to develop such a system yourself:

Create or refactor an app to use Docker Compose.
Test with docker-compose up on localhost.
In your Continuous Deployment stage of your CI/CD pipeline, build a Docker image and push it into an ECR repository called “dev” or “prod” or whatever you call your environment.
In your deployment script, run these commands or something similar AFTER you have pushed the new image:

INSTANCE_ID=$(aws ec2 describe-instances --filters Name=tag:Name,Values=host-$CURRENT_ENVIRONMENT --region us-east-1 --output text --query 'Reservations[*].Instances[*].InstanceId')

echo $INSTANCE_ID

aws ec2 reboot-instances --instance-ids $INSTANCE_ID

In AWS, create a VM.
Install Docker on the VM.
Set up logging so that your Docker Compose logs flow to CloudWatch.
Create a Systemd file which runs Docker Compose at startup.
Edit that Systemd file to include ExecStartPre=/home/ec2-user/docker-compose-setup.sh
Create /home/ec2-user/docker-compose-setup.sh with these contents so that when your VM restarts & the systemd file is run, the first thing it does is pull an up to date copy of your Docker image:

#!/bin/sh
/usr/local/bin/aws ecr get-login-password --region us-east-1 | /usr/bin/docker login --username AWS --password-stdin [your-aws-account-number].dkr.ecr.us-east-1.amazonaws.com
# Note: The production image (AMI) uses the following line:
# /usr/bin/docker pull [your-aws-account-number].dkr.ecr.us-east-1.amazonaws.com/prod:latest
/usr/bin/docker pull [your-aws-account-number].dkr.ecr.us-east-1.amazonaws.com/dev:latest
/usr/local/bin/docker-compose down

chmod +x /home/ec2-user/docker-compose-setup.sh
Restart the VM.
Your application should now be running on localhost. You can test if the website is publicly accessible via its ephemeral IP address. You can also attach an elastic IP address and then point your website at this IP address.
Create an image of this VM. If you are using CloudFormation or another service, reference this AMI (Amazon Machine Image) and you can integrate this VM into your CloudFormation / other service template.

Recap

I cut deployment times, made debugging easier, and did it all in just a couple billable hours. When I started, I didn’t know anything about systemd or EC2 or Docker Compose, but now that I’ve experienced these tools I’m definitely going to use them again. I’d encourage you to give it a try too! Perhaps you will see the same time savings I am currently enjoying.

Developers Aren't Essential because they Write Code

Gregory Ledray — Fri, 16 Apr 2021 17:30:26 +0000

They are essential because they know things and think logically. You can build a website without a developer using Squarespace and you can do it for free with WordPress. You can build a form based app with ServiceNow and you can do it for free with (shameless plug for my free tool) Polyapp. You can build video games in Unity. There are similar no-code tools which let you build apps for Android and iOS.

A lot of people will jump at this and say, "but without a developer, you can't calculate X Y Z!" or "those tools aren't flexible enough to replace the app I'm building!" Obviously someone needs to code everything, but in large developer ecosystems someone has probably already coded something just like what you're writing and your job is reduced to configuring that thing. Most of the code I used to write was configuring components someone else wrote on the front end and transforming data from the database into the UI and back again on the back end. Even those things are just configuration. HTML is configuring components defined by the HTML standard. Typing out a data model in C# and setting up Entity Framework is just a way to configure the .NET framework. You could write a UI with all of the different choices for HTML and Entity Framework model choices as a form, and have someone select from the list of choices and all of that work would be poof - gone. Incidentally, this is sort of how Polyapp works.

Most application coding doesn't need to be done by someone with knowledge of how to set up Visual Studio or the intricacies of planning and developing an object oriented data model. Most development work could be done by someone who has a problem sitting down, finding a template for a solution to their problem, and then going through many, many, many options describing the solution and adjusting them to fit their needs. Doing things themselves would save them a ton of time and heartache when the thing they paid for doesn't turn out the way they wanted. But that's not what happens.

It's not what happens because small fractions of every applications require a developer, and developers aren't educated to use a hybrid model with some no-code development and some code. They believe the only options are 100% coded by me or 0% coded by me.

It's not what happens because most people have no clue that low-code or no-code tools exist, and even if they do they don't understand concepts like foreign key columns and one-to-many references.

It's not what happens because creating apps requires dozens of steps and most people aren't good at logically thinking through each step.

It's not what happens because developers know things and think logically.