DEV Community: Robert Reiz

Deploy Rails static assets to CloudFront CDN - during Docker build time

Robert Reiz — Sun, 10 Jan 2021 10:56:57 +0000

Ruby on Rails is a great Framework to build modern web applications. By default, all the static assets like CSS, JavaScript, and images, are served directly from the Ruby server. That works fine but doesn't offer the best performance. A ruby server like Puma or Unicorn is not optimized to serve static assets. A better choice would be to server the static assets from an Nginx instance. And even better than Nginx would be to serve the static assets from a CDN (Content Delivery Network). CloudFront is the CDN from Amazon. If you are using AWS anyway, that's your goto CDN.

Assumptions

This article assumes:

Basic knowledge about Ruby, Git, GitHub Actions, Docker and the Rails framework
The application uses a GitHub Action for deployment
The application uses AWS ECS Fargate for running Docker containers
The application uses AWS S3 and AWS CloudFront for statis assets
The AWS infrastructure is already setup an not a topic of this article.

S3 and CloudFront

A CloudFront CDN bucket is always linked to an S3 bucket. The content of the S3 bucket is then mirrored on the CloudFront bucket. That means, during deployment, we need to upload our static assets to the right S3 bucket, which is linked to our CloudFront bucket.
If you want to learn how to correctly setup CloudFront & S3, read this article or the official AWS docs.

Rails configuration

In your Rails application under config/environments/production.rb you can configure an asset host like this:

# Enable serving of images, CSS, and JS from an asset server.
  config.action_controller.asset_host = ENV['RAILS_ASSET_HOST']

In the example above the asset host is pulled from the ENV variable RAILS_ASSET_HOST, which is set during deployment. If you deploy your Rails application to ECS Fargate, you will have somewhere an ECS task-definition.json for your application. In that task-definition.json, in the environment section, you would set the RAILS_ASSET_HOST ENV variable, like this here:

{ 
  "name" : "RAILS_ASSET_HOST", 
  "value": "https://d32v8iqllp6n8e.cloudfront.net/"
}

The ENV variable is pointing directly to your CloudFront CDN URL.

GitHub Action

GitHub Actions are a great way to trigger tests, builds, and deployments. Your GitHub Action configuration might look like this one in .github/workflows/aws-test-deploy.yml:

on:
  push:
    branches: 
    - aws/test

name: Deploy to AWS Test Cluster

jobs:
  deploy:
    name: Deploy
    runs-on: ubuntu-latest

    steps:
    - name: Checkout
      uses: actions/checkout@v2

    - name: Configure AWS credentials
      uses: aws-actions/configure-aws-credentials@v1
      with:
        aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
        aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
        aws-region: eu-central-1

    - name: Login to Amazon ECR
      id: login-ecr
      uses: aws-actions/amazon-ecr-login@v1

The above configuration tells GitHub to trigger the Action on each git push to the aws/test branch. The Action will be performed on the latest Ubuntu instance. The current source code will be checked out from the Git repository into the Ubuntu instance. Furthermore, the aws-actions module will be configured with the AWS credentials we stored in the GitHub secret store of the Git repository.

The next part of the config file contains the important part:

- name: Build, tag, and push image to Amazon ECR
  id: build-image
  env:
    ECR_REGISTRY: ${{ steps.login-ecr.outputs.registry }}
    ECR_REPOSITORY: ve/web-test
    IMAGE_TAG: ${{ github.sha }}
    TEST_AWS_CONFIG: ${{ secrets.TEST_AWS_CONFIG }}
    TEST_AWS_CREDENTIALS: ${{ secrets.TEST_AWS_CREDENTIALS }}
    TEST_RAILS_MASTER_KEY: ${{ secrets.TEST_RAILS_MASTER_KEY }}
  run: |
    mkdir .aws
    echo "$TEST_AWS_CONFIG" > .aws/config
    echo "$TEST_AWS_CREDENTIALS" > .aws/credentials
    echo "$TEST_RAILS_MASTER_KEY" > config/master.key
    echo "test-ve-web-assets" > s3_bucket.txt
    docker build -t $ECR_REGISTRY/$ECR_REPOSITORY:$IMAGE_TAG .
    docker push $ECR_REGISTRY/$ECR_REPOSITORY:$IMAGE_TAG
    echo "::set-output name=image::$ECR_REGISTRY/$ECR_REPOSITORY:$IMAGE_TAG"

Here we set a bunch of ENV variables for the ECR Docker Registry on AWS and the image tag name, which will be equal to the latest commit SHA of the current branch.

Then we set the ENV variable TEST_AWS_CONFIG to the value of the GitHub secret ${{ secrets.TEST_AWS_CONFIG }}, which contains the regular AWS config file content for the current runtime. On your localhost, you find that file under ~/.aws/config. Usually, that looks like this:

[default]
region = eu-central-1
output = json

Then we set the ENV variable TEST_AWS_CREDENTIALS to the value of the GitHub secret ${{ secrets.TEST_AWS_CREDENTIALS }}, which contains the AWS credentials for the current runtime. On your localhost, you find that file under ~/.aws/credentials. Usually, that looks like this:

[default]
aws_access_key_id = ABCDEF123456789
aws_secret_access_key = abcdefghijklmno/123456789

The TEST_AWS_CREDENTIALS variable has to contain AWS credentials that have the permission to upload files to our corresponding S3 bucket.

In the run section we pipe the content of TEST_AWS_CONFIG into the file .aws/config in the current work directory. And we pipe the content of TEST_AWS_CREDENTIALS into .aws/credentials. And we pipe the name of the S3 bucket ("test-ve-web-assets") into the file s3_bucket.txt.
Now all the necessary credentials for the S3 upload are in the current working directory. Now we can start to build our Docker image with docker build -t $ECR_REGISTRY/$ECR_REPOSITORY:$IMAGE_TAG .

Dockerfile

Our Dockerfile describes a so-called multi-stage build. Multi-stage builds are a great way to clean up Docker layers that contain sensitive information, like for example AWS credentials.

Our Dockerfile starts like this:

FROM versioneye/base-web:1.2.0 AS builderAssets

WORKDIR /usr/src/app_build

COPY .aws/config /root/.aws/config
COPY .aws/credentials /root/.aws/credentials
COPY . .

As base image we start with versioneye/base-web:1.2.0, which is a preconfigure Alpine Docker image with some preinstalled Ruby & Node dependencies. It's based on the ruby:2.7.1-alpine Docker image.

We copy our .aws/config to /root/.aws/config and our .aws/credentials to /root/.aws/credentials, because the AWS CLI looks for that files at that place by default.

We copy all files from the current git branch to the working directory in the Docker image at /usr/src/app_build. Now we have all files in place inside the Docker image.

As next step we need to install the AWS CLI:

# Install AWS CLI
RUN apk add python3; \
    apk add curl; \
    mkdir /usr/src/pip; \
    (cd /usr/src/pip && curl -O https://bootstrap.pypa.io/get-pip.py); \
    (cd /usr/src/pip && python3 get-pip.py --user); \
    /root/.local/bin/pip install awscli --upgrade --user;

Now the AWS CLI is installed and the AWS credentials are at the right place. With the next step we will:

delete unnecessary files from the current working dir.
install NPM dependencies
install Gem dependencies
precompile the static Rails assets
upload the static Rails assets to our S3 bucket

# Compile assets and upload to S3
RUN rm -Rf .bundle; \
    rm -Rf .aws; \
    rm -Rf .git; \
    rm bconfig; \
    yarn install; \
    bundle config set without 'development test'; \
    bundle install; \
    NO_DB=true rails assets:precompile; \
    /root/.local/bin/aws s3 sync ./public/ s3://`cat s3_bucket.txt`/ --acl public-read; \
    /root/.local/bin/aws s3 sync ./public/assets s3://`cat s3_bucket.txt`/assets --acl public-read; \
    /root/.local/bin/aws s3 sync ./public/assets/font-awesome s3://`cat s3_bucket.txt`/assets/font-awesome --acl public-read; \
    /root/.local/bin/aws s3 sync ./public/packs s3://`cat s3_bucket.txt`/packs --acl public-read; \
    /root/.local/bin/aws s3 sync ./public/packs/js s3://`cat s3_bucket.txt`/packs/js --acl public-read;

In the last 5 lines we are simply using the AWS CLI to sync files from inside the Docker image with the S3 bucket which is defined in the s3_bucket.txt file. The AWS CLI which currently runs on Alpine Linux doesn't support recursive uploads, that's why we need to do the sync command for each directory separately.

Now the static Rails assets are uploaded to AWS S3/CloudFront. But the AWS credentials are still stored in the Docker layers. If we publish that Docker image to a public Docker registry, somebody could fish out the AWS credentials from the Docker layer and compromise our application. That's why we are using Docker multi-stage builds to prevent that from happening.
The next part of the Dockerfile looks like this:

FROM versioneye/base-web:1.2.0 as builderDeps

COPY --from=builderAssets /usr/src/app_build /usr/src/app

WORKDIR /usr/src/app

RUN yarn install --production=true; \
    bundle config set without 'development test'; \
    bundle install;

EXPOSE 8080

CMD bundle exec puma -C config/puma.rb

The above lines are starting pretty much a completely new Docker build. We start again with our Docker base image and we copy all files from the previous build from /usr/src/app_build to our current build into /usr/src/app. We install again the dependencies for Node.JS and Ruby, we expose Port 8080 and we set the run command with CMD.

The Docker image we get out there does NOT include any AWS credentials and also no AWS CLI! It contains only the application code and the corresponding dependencies.

Summary

We are using a Docker multi-stage build to upload static files to S3 and to leave no traces behind. The first stage we use to install the AWS CLI, AWS secret credentials and to perform the actual upload of the static assets to S3/CloudFront.

The 2nd stage we use to install the application dependencies and to configure the Port and the CMD command. The Docker image we get after the 2nd stage does NOT include any AWS secrets and no AWS CLI.

Let me know what you think about this strategy. You find it useful? Any improvements?

Nginx forward proxy - as Docker image

Robert Reiz — Fri, 08 Jan 2021 10:26:05 +0000

Nginx is a very fast HTTP and reverse proxy server. Usually, Nginx is used to serve and cache static assets or as proxy or load balancer for incoming traffic to application servers. But it can be used as forward proxy as well.

Assume you have a network where you want to control outgoing traffic. You either want to:

Deny all outgoing calls by default and only allow HTTP(S) calls to whitelisted URLs.
Allow all outgoing calls by default and only block HTTP(S) calls to blacklisted URLs.

The Docker daemon can be configured that way that it routes all traffic through a proxy. This proxy can be an Nginx which is configured as forward proxy.

I have a build a Docker image which contains Nginx preconfigured as forward proxy. Check out the full notes and Docker at this GitHub repo.

Redirects with AWS S3

Robert Reiz — Sat, 05 Dec 2020 11:13:10 +0000

Assume you have a web application deployed on AWS ECS Fargate, with an ALB (Application Load Balancer) in front of it. The ALB is doing the SSL termination as well. Your domain mydomain.com is mapped to the ALB and everything works fine!

But now you want to redirect the traffic from www.mydomain.com to your main domain mydomain.com. Of course you could create another ALB for that, but that would increase your monthly AWS bill. A much better and cheaper solution is to use S3 for that.

First, let's create a new S3 bucket for that purpose. If you work with Terraform, you just need to customize this code snippet:

resource "aws_s3_bucket" "www_redirect" {
  bucket = var.s3_www
  acl    = "public-read"

  website {
    redirect_all_requests_to = "https://mydomain.com"
  }

  tags = {
    Name        = var.s3_www
    Environment = var.environment
  }
}

If you don't use Terraform, simply read this article to find out how to do it manually.

That code snippet creates a new empty S3 bucket with public read permissions. The S3 bucket is empty. It doesn't contains any files/objects. It is configured that way that it redirects all requests immediately to https://mydomain.com.

Now we only need to map the subdomain www to that S3 bucket. In your DNS Zone file you have to create an A record with the value of the S3 bucket address. If your domain is managed by AWS Route 53, you navigate to the desired zone file and click "Create record". Then you should see something like this:

Simple routing is fine for us. In the next form you have to define the subdomain "www" and choose the Endpoint to which it should be mapped. Here we have to choose "Alias to S3 website endpoint".

Next, you have to choose your region and the corresponding S3 bucket, which we just created before.

And that's it! Now the subdomain "www" is mapped to the S3 bucket, which will redirect every request to "https://mydomain.com".

How to speed up your daily Docker builds

Robert Reiz — Thu, 14 May 2020 16:38:38 +0000

Nowadays, there is no way around Docker. It's a great technology that ensures that your software works on production the same as on your laptop.

If you have a medium size Node.JS or Ruby on Rails project, with a hand full of dependencies, this command can take a couple of minutes:

docker build -t orga/project:1.0.0 .

Especially if you have dependencies with native extensions like libxml or sassc, the build will take a long time.

The build time can be reduced dramatically if you are using a Docker base image which includes already the majority of the needed dependencies. That's why I have, in most of my projects, a directory called docker_base, which contains a Dockerfile and package manager files for my base image. For a typical Ruby on Rails project the Dockerfile would look like this:

FROM ruby:2.5-alpine

ENV RAILS_ENV=production

WORKDIR /usr/src/app_base

COPY docker_base/Gemfile .
COPY package.json .

RUN apk update; \
    apk add build-base; \
    apk add libxml2-dev; \
    apk add libxslt-dev; \
    apk add ruby-nokogiri; \
    apk add yarn; \
    apk add tzdata; \
    yarn install --production=true; \
    bundle config build.nokogiri --use-system-libraries; \
    bundle install --without development test;

The first COPY command adds a Gemfile with the most important dependencies. These Gemfile does not include private (closed source) dependencies. That way the Docker base image can be later hosted on a public Docker Hub repository. The second COPY command adds the package.json file for the frontend dependencies.

The RUN command is doing all the hard work which we want to avoid on our daily builds. It's installing all the native system libraries which are required for the Ruby and Node dependencies. In the last 3 lines we finally install all the Node and Ruby dependencies.

Let's build the base image like this:

docker build -t versioneye/base-web:1.0.0 .

Now the regular Dockerfile in the root directory of the project can inherit from this base image. The first line of that Dockerfile would look like this:

FROM versioneye/base-web:1.0.0

The dependencies in your Gemfile and package.json might change every day. That's why it's important that you run the install steps again in your main Dockerfile! That way new dependencies will be installed. However, the build time shrinks dramatically because the base image contains already the majority of the dependencies. In my case I could reduce the build time from 5 minutes to under 1 minute!

What do you think about this method? Do you have another trick to reduce the build time of your daily Docker build?