DEV Community: Rangle.io

Running Jenkins and Persisting state locally using Docker

Rangle.io — Tue, 10 Dec 2019 17:44:55 +0000

Jenkins is one of the most popular Continuous Integration and Delivery Servers today, so it's only natural that you're probably interested in learning more about it. When starting out, you'll need to first run it on your local machine. However, the problem with that is the Jenkins configuration files will then live directly on your machine. A better solution is to run it as a Docker container, here are some of the reasons why:

All of your Jenkins configuration files live inside the container rather than the host machine. Knowing that all the files you need are inside the container, you can eliminate the issue of accidentally mixing your files with Jenkins configuration files.
Docker instances are easier to manage if you are interested in running Jenkins on multiple platforms
You can easily create and destroy the Jenkins server and remove all the Jenkins data

Another benefit of using containers is persisting the state of your Jenkins server using Docker volumes. Why do this?

You get to keep all your projects and configurations even after restarting your computer (local machine)
You don't need to run the whole Jenkins setup again
You can remove your container instance and still able to recover the state of your Jenkins server

With that in mind, I'll show you how you can start configuring Jenkins and persisting state on your local machine. Let's get started!

Jenkins setup

Before we get started, you'll need to install Docker on your machine. If you're not sure how you can refer to this official documentation.

After installing Docker, download the latest stable Jenkins image by running:

docker image pull jenkins/jenkins:lts

You should see something like this:

Persisting Jenkins Data

You can create a volume by running the command below:

docker volume create [YOUR VOLUME]

Volumes are used to make sure that you don't lose your Jenkins data. If you are using the -v flag on container creation ( docker container run), feel free to skip this step since Docker will automatically create the volume for you.

Run the container by attaching the volume and assigning the targeted port. In this example, we'll also run it in detached mode. Here is the command to run your Docker container:

docker container run -d \ -p [YOUR PORT]:8080 \ -v [YOUR VOLUME]:/var/jenkins_home \ --name jenkins-local \ jenkins/jenkins:lts

If you were wondering what the arguments stand for, here is what each means:

-d: detached mode
-v: attach volume
-p: assign port target
-name: name of the container

And now, we're ready to take a look at an example of how you could run this command:

docker container run -d -p 8082:8080 \ -v jenkinsvol1:/var/jenkins_home \ --name jenkins-local \ jenkins/jenkins:lts

After running the command, you should be able to see the code to be used for the next step on your setup.

In the command we ran, /var/jenkins_home is the path to where the Jenkins state is stored on our container instance. The most important argument we pass when it comes to data persistence in this example is the -v **[YOUR VOLUME]**:/var/jenkins_home. This argument is what helps Docker link the volume to the file inside the container. To learn more about Docker volumes, you can check out the official documentation. If you're not familiar with Docker, you can start with this helpful docker blog series, Learning Docker - The Command Line Interface. You'll notice that I am using port 8082 instead of the default 8080. The reason, other than demonstrating that you can use other ports, is that port 8080 is used by some web frameworks.

If you run docker ps, you should see your docker container running

After confirming that your container is running, go to localhost:[YOUR PORT] ( localhost:8082 on my example) on your browser and you should see this page:

As a part of the Jenkins setup, we need to view the password inside the container instance. In order to do this, we need to use the CONTAINER ID (or the name) and run docker exec.

Here is the full command:

docker container exec \ [CONTAINER ID or NAME] \ sh -c "cat /var/jenkins_home/secrets/initialAdminPassword"

After running the command, you should see the code. Copy the code and paste it on the webpage to unlock Jenkins. After unlocking, click on Install suggested plugins on the Customize Jenkins page.

Wait until the installation is complete and then you can proceed in creating your first admin user.

After creating the admin user, setup the Instance configuration. Since you are only using Jenkins locally, leave the URL to your localhost URL. Click on Save and Finish to start using Jenkins.

Confirming Jenkins state is persisted

Once you get to the Jenkins home page, create a new job.

Name the job Test and set it to be a Freestyle project.

Leave everything as default and add a new Shell Execution under Build. Add this as the command:

echo "Working"

Save the job and click on Build Now to start running the job.

Click on the Build Status (blue ball) under Build History (left sidebar) to view the console output. You should see that our command ran with no problems.

Recovering Jenkins

After confirming that the job runs, we want to make sure that we can recover our Jenkins configuration if needed. In Jenkins, all of the configuration is stored in /var/jenkins_home/ by default. Remember that we are using docker volume to store information about our Jenkins instance.

In order to check if the persistence works correctly, let's destroy our current docker container and see if we can log in as an admin and view our job build history.

First, run docker container kill [CONTAINER ID] to stop the instance. After doing so, run docker container rm [CONTAINER ID] to completely remove the container instance.

Visit localhost:[YOUR PORT] ( localhost:8082 on my example) to confirm that the Jenkins instance is not running anymore. You should see something like this:

Run the same command we used to create the container instance in order to recover the Jenkins instance.

docker container run -d -p 8082:8080 \ -v jenkinsvol1:/var/jenkins_home \ --name jenkinslocal \ jenkins/jenkins:lts

You should get a new CONTAINER ID after running the command. Visit localhost:[YOUR PORT] ( localhost:8082 on my example). You should see the login page. Login with the admin credentials you set during Jenkins initialization.

After logging in, you should see the job you created and view the console output.

In summary

I hope you can now see why Docker is the best way to start learning or at least playing around with Jenkins. You can easily run a Jenkins instance as a Docker container and persist your Jenkins server state using Docker Volumes. In case you need to restart or recover your Jenkins instance, all of the state is stored inside the Docker Volume. If you want to read more about Jenkins, Docker and DevOps, check out our other blogs, here.

Docker for Frontend Devs: Custom Docker Images for Development

Rangle.io — Fri, 06 Dec 2019 21:16:04 +0000

By: Benjamin Martin

Let's take a moment to consider what is important for local development. For me, I want to make sure all my developers are using the same dependencies, and I don't want to worry about what versions they have installed. No more "but it works on my machine" excuses. At the same time, I want to make sure we retain the conveniences of HMR (Hot Module Replacement) so that developers don't need to constantly refresh the application to see their changes reflected. We don't want to lose fast feedback.

In this article, we'll look at how we can setup Docker for a boilerplate VueJS app with custom Dockerfiles from which our images and containers will be built and how we gain efficiencies from these.

In case you missed the first part in this series, check here to learn more about the command line interface that Docker ships with. We need to use the commands from that article in this section. If you are already familiar with Docker CLI, please continue to follow along.

Prerequisite: Create our project

This is of course a Docker article, so please ensure you have Docker installed. You can follow the official install instructions for Docker here. Since I'm using Vue, I've used the VueCLI to spin up a quick workspace with vue create docker-demo.

The configuration I selected (seen below) will be relevant to do E2E testing and unit testing which will become part of our CI/CD pipeline.

Once everything is installed, cd into our new project folder, open an IDE and let's dig in.

Custom Docker Image for Development

If you've played with Docker but not built your own image, you probably know we specify an image when we execute our docker run command. Those images are pulled from Docker Hub or some other remote repository (if that image is not found locally). In our case though, we want to build a custom image.

In the root of our project, create a file named Dockerfile.dev. This will be our development image. In that file, copy the following code into it.

# Base Image
FROM node:9.11.1

ENV NODE_ENV=development
ENV PORT=8080

WORKDIR /usr/src/app
COPY package*.json /usr/src/app/
RUN cd /usr/src/app && CI=true npm install

EXPOSE 8080
CMD ["npm", "run", "serve"]

Ok... but what does all this do? Let's dig into it.

Dockerfile Commands and Keywords

FROM specifies the preexisting image on which to build our custom image. Since we are running a node application, we've chosen one of their official Docker images.

FROM node:9.11.1 means our application image will start with the node v 9.11.1 image

ENV sets environment variables

ENV PORT=8080 sets the environment variable PORT for later use

ENV NODE_ENV=development sets the environment variable NODE_ENV for use within our app

WORKDIR sets the working directory within the container

WORKDIR /usr/src/app defines /usr/src/app/ as our working directory within the docker image

COPY copies new files, directories or remote files into the container/image

COPY package*.json /usr/src/app/ copies our package.json and package-lock.json into our working directory

RUN executes a command in a new layer on top of the current image and commits it. When you run the build, you will see a hash representing each layer of our final image

RUN cd /usr/src/app/ && CI=true npm install changes the working directory to where the package.json is and installs all our dependencies to this folder within the image. This makes it so that the image holds frozen copies of the dependencies. Our Docker image, not our host machine, is responsible for our dependencies

EXPOSE allows us to access a port on the container from our host machine

EXPOSE 8080 matches the port on which our app is running, inside the container and allows us to access our app from our host machine

CMD provides the default initialization command to run when our container is created, like a startup script

CMD ["npm", "run", "serve"] sets this as our default command when we start our container. This is not run when building the image, it only defines what command should be run when the container starts.

I know you're anxious to get this running, but hold your horses. Let's look closer at our Dockerfile.dev and understand why we did what we did.

Dockerfile Structure Recommendations

So, Where's my app?

Right. We didn't use the COPY command to copy our full workspace. Had we done so, we'd need to run docker build and docker run for every code change. We don't want to do this over and over for development. We can be more efficient

Caching Dependencies

We are taking advantage of how Docker layers the images. As Docker builds our image, you'll see a hash for each layer as it is completed. What's more is that Docker also caches these layers. If Docker can see that nothing has changed on that layer from a previous build (and previous layers are also identical) then Docker will use a cached version of that layer, saving you and your developers precious time! When a layer changes, any cached layers on top of it are invalidated and will be rebuilt.

Therefore, if there is no change to our package.json or the package-lock.json then our entire image is cacheable and doesn't need to be rebuilt!

Priority

This is also why you want to have other Dockerfile commands that change less frequently near the top of our file. As soon as one layer of our cache is invalidated, for example, if you change ENV PORT=8080 to another port, that cached layer and every cached layer after it is invalidated and Docker will have to rebuild those layers.

Building the Custom Docker Image

Now, build the image with this command: docker build --tag docker_demo:latest --file Dockerfile.dev .

Using --tag in the docker build command allows us to easily reference this image from our docker run command

The . at the end of the docker build command references the context where our custom Dockerfile can be found. So, this command should be run from the root of our project directory

You can run it with docker run docker_demo:latest, but unfortunately, we have more work to do to get it working quickly and easily from the command line.

Running our Container: Quality of Life Improvements

We're going to be executing our docker run command daily, if not more frequently. However, if we simply execute the docker run docker_demo:latest command, Docker will create a new container each time. Docker won't stop the old container unless you do so explicitly. This is very useful in many cases, but since we've hardcoded the host port, we'll run into port collisions on our host machine.

In order for us to easily stop and remove our old containers, we should name them so we can easily refer to them later. Additionally, I want the running container to be removed if I cancel the running process.

docker run --rm -it\
--name docker_demo_container\
docker_demo:latest

What was added?

We added a --name field to the end of our run command. This allows us to reference the container without looking up the hash. Now, we can easily stop our container by name.

We also added the --rm and -it flags to our docker run command. The --rm flag tells Docker to remove the container if and when it is stopped. The -it flag keeps the terminal live and interactive once the container is started.

Mounting Host Directories

Let's go back to our docker run command and let's find a way to mount our workspace directory to a folder within our container. We can do this by adding a mount point to our container in the docker run command. This will tell Docker that we want to create an active link between our host machine's folder (src) and the Docker container folder (dst). Our new command should look like this:

docker run --rm -it\
--name docker_demo_container\
--mount type=bind,src=`pwd`,dst=/usr/src/app\
docker_demo:latest

But this could conflict with our host machine's node_modules folder since we're mounting our entire pwd to our app's location in the image (in case one of our developers accidentally runs npm install on their host machine). So, let's add a volume to ensure we preserve the node_modules that exists within our container.

docker run --rm -it\
--name docker_demo_container\
--mount type=bind,src=`pwd`,dst=/usr/src/app\
--volume /usr/src/app/node_modules\
docker_demo:latest

Accessing Ports Inside the Container

If you tried the above command (and you're running a VueJS app), you should see:

 App running at:
  - Local:   http://localhost:8080/

  It seems you are running Vue CLI inside a container.
  Access the dev server via http://localhost:<your container's external mapped port>/

Docker is giving you a hint that we need to expose a port from our container and publish it on our host machine. We do this by adding the --publish flag to our run command. (We already have the EXPOSE command in our Dockerfile.dev)

--publish <host-port>:<container-port> tells Docker that traffic to the host machine (i.e. via localhost) on port <host-port> should be directed towards the container at the <container-port> that you define.

`docker run` in One Command

Let's take a look at our final run command:

docker run --rm -it\
--name docker_demo_container\
--publish 4200:8080\
--mount type=bind,src=`pwd`,dst=/usr/src/app\
--volume /usr/src/app/node_modules\
docker_demo:latest

Running the above command will finally allow us to access our app via http://localhost:4200.

Testing it out

Let's build a fresh copy and run it. If you try changing one of our file's templates, you'll see everything is still functioning as it should be.

But speaking of testing, what about unit tests? Well, once our container is running, we can open a new terminal and docker exec a command to run in our container.

docker exec -it docker_demo_container npm run test:unit

The above command will create an interactive terminal connection with our container docker_demo_container and execute the command npm run test:unit in it, allowing us to run unit tests for our app.

In Closing

We now have a way to build our development images and run them locally while maintaining the conveniences of Hot Module Replacement to keep our development workflow efficient. Our developers don't need to worry about dependencies on their host machine colliding with those in the image. No more "but it works on my machine" excuses. And, we also have a command we can easily run to execute our unit tests.

If you find anything I missed or want to chat more about Docker, please reach out to me!

Connecting Stripe events to the AWS EventBridge

Rangle.io — Wed, 20 Nov 2019 15:28:39 +0000

Stripe is a great platform for running an online business, especially on account of the developer-centric API that makes it easy to collect payments, set up subscriptions and more.

Many of these APIs result in Stripe generating one or more events that you can subscribe to via a webhook. These events could represent a successful payment, or perhaps a new subscriber to your SaaS platform. In total, there are well over 100 different types of events that you can subscribe to, so how can we build a loosely-coupled cloud-native event-driven architecture to handle these events? How can we make this serverless?

It's easy, we'll use AWS EventBridge.

Why AWS EventBridge?

For those of you just learning about AWS EventBridge, it is a serverless event bus supporting event-driven architectures, released July 2019. An event bus is a central location where business events can be published and routing rules can be configured to send those events to downstream functions or services that operate on them. By using an event bus as an intermediate layer, this decouples services from needing direct knowledge of how to communicate with each other, which allows development teams to build and operate independently.

One reason, among many, why AWS EventBridge should be considered is the cost. With AWS EventBridge, you have a fully managed cloud-native event bus, requiring no effort to set up and no servers to pay for - and it costs a very reasonable $1 per million events. Latency is typically about half a second. By going with a fully managed solution, you're saving your team from spending time on 'undifferentiated heavy lifting' associated with infrastructure, giving them more time to focus on delivering customer value.

The solution

We'll show here how we can set up AWS EventBridge so that it can begin ingesting events from Stripe. As part of that ingestion, we'll check that the event is truly from Stripe by verifying the signature on the event. Let's get started.

Using an an open-source project we created called stripe-eventbridge, you can quickly set up the plumbing to connect Stripe events to AWS EventBridge via a Lambda function that will then validate the authenticity of incoming events (so that downstream functions won't have to). This allows for the events that are published on the event bridge to be routed to various downstream functions through simple routing rules which we show at the end of the post.

With this deployed, you can set up many downstream Lambda functions to handle the wide array of events generated by Stripe and configure the routing to these event handlers with AWS EventBridge based on the Stripe event type. All of this without having to worry about event signatures, since events are only placed on the EventBridge if they pass validation.

The stripe-eventbridge project creates an endpoint that you configure in the Stripe Dashboard as the destination for the webhook events. When an event arrives, the endpoint will invoke a Lambda that we provide which has these responsibilities:

to validate that the event has a valid signature (i.e. that it was generated by Stripe)
to place validated events (only) on the AWS EventBridge
to notify an SNS topic if an event fails to validate (e.g. due to an invalid signature)

Solution overview

stripe-eventbridge uses the Serverless Framework to generate the infrastructure in the AWS section of the diagram above. The Serverless Framework allows us to automatically create and deploy the CloudFormation stacks needed to realize the above configuration. The stacks can be deployed via sls deploy and rolled back via sls remove using the command line client.

Serverless components in stripe-eventbridge

Let's walk through what stripe-eventbridge creates:

A Lambda - this function validates incoming events, and if they are from Stripe (i.e. signed correctly) then they are relayed to the AWS EventBridge where downstream services can read from them.
An API Gateway endpoint - this is the webhook endpoint that the Lambda above is listening to for new events.
An AWS SecretManager secret - this secret is created with an empty value, and you populate this with the signing key assigned to your webhook endpoint in the Stripe Dashboard
An SNS topic - if the webhook fails validation or processing, then a notification is sent to a topic called stripe-webhook-event-failed-to-validate. You can configure subscriptions to this topic, so that you are emailed (for example) whenever an event fails processing.

If you'd like to get started, you can visit the GitHub project here:

https://github.com/rangle/stripe-eventbridge

Routing by event type

Now that you have events being published to the EventBridge, you can configure routing based on the event type using Rule patterns like this. The detail-type values are exactly the values of the underlying Stripe events - you can see a full list here: https://stripe.com/docs/api/events/types

{

  "detail-type": [

    "payment_intent.succeeded"

  ],

  "source": [

    "Stripe"

  ]

}

Alternately, if you are creating your event handlers using the Serverless Framework, then you can declare the same routing in your infrastructure code (this is really good practice) in your serverless.yml as follows:


    handler: handler.myLambdaFunction

    events:

      - eventBridge:

          pattern:

            source:

              - Stripe

            detail-type:

              - payment_intent.succeeded

And that's all!

I hope you have found this valuable. Stay tuned, we'll be exploring more with Serverless in upcoming posts.

Running Jenkins and Persisting state locally using Docker

Rangle.io — Fri, 15 Nov 2019 20:42:15 +0000

All of your Jenkins configuration files live inside the container rather than the host machine. Knowing that all the files you need are inside the container, you can eliminate the issue of accidentally mixing your files with Jenkins configuration files.
Docker instances are easier to manage if you are interested in running Jenkins on multiple platforms
You can easily create and destroy the Jenkins server and remove all the Jenkins data

Another benefit of using containers is persisting the state of your Jenkins server using Docker volumes. Why do this?

You get to keep all your projects and configurations even after restarting your computer (local machine)
You don't need to run the whole Jenkins setup again
You can remove your container instance and still able to recover the state of your Jenkins server

With that in mind, I'll show you how you can start configuring Jenkins and persisting state on your local machine. Let's get started!

Jenkins setup

Before we get started, you'll need to install Docker on your machine. If you're not sure how you can refer to this official documentation.

After installing Docker, download the latest stable Jenkins image by running:

You should see something like this:

Persisting Jenkins Data

You can create a volume by running the command below:
docker volume create [YOUR VOLUME]

Volumes are used to make sure that you don't lose your Jenkins data. If you are using the -v flag on container creation (docker container run), feel free to skip this step since Docker will automatically create the volume for you.

Run the container by attaching the volume and assigning the targeted port. In this example, we'll also run it in detached mode. Here is the command to run your Docker container:
docker container run -d \ -p [YOUR PORT]:8080 \ -v [YOUR VOLUME]:/var/jenkins_home \ --name jenkins-local \ jenkins/jenkins:lts

If you were wondering what the arguments stand for, here is what each means:

-d: detached mode
-v: attach volume
-p: assign port target
---name: name of the container

And now, we're ready to take a look at an example of how you could run this command:
docker container run -d -p 8082:8080 \ -v jenkinsvol1:/var/jenkins_home \ --name jenkins-local \ jenkins/jenkins:lts

After running the command, you should be able to see the code to be used for the next step on your setup.

If you run docker ps, you should see your docker container running

After confirming that your container is running, go to localhost:[YOUR PORT] (localhost:8082 on my example) on your browser and you should see this page:

As a part of the Jenkins setup, we need to view the password inside the container instance. In order to do this, we need to use the CONTAINER ID (or the name) and run docker exec.

Here is the full command:
docker container exec \ [CONTAINER ID or NAME] \ sh -c "cat /var/jenkins_home/secrets/initialAdminPassword"

Wait until the installation is complete and then you can proceed in creating your first admin user.

After creating the admin user, setup the Instance configuration. Since you are only using Jenkins locally, leave the URL to your localhost URL. Click on Save and Finish to start using Jenkins.

Confirming Jenkins state is persisted

Once you get to the Jenkins home page, create a new job.

Name the job Test and set it to be a Freestyle project.

Leave everything as default and add a new Shell Execution under Build. Add this as the command:
echo "Working"

Save the job and click on Build Now to start running the job.

Click on the Build Status (blue ball) under Build History (left sidebar) to view the console output. You should see that our command ran with no problems.

Recovering Jenkins

In order to check if the persistence works correctly, let's destroy our current docker container and see if we can log in as an admin and view our job build history.

First, run docker container kill [CONTAINER ID] to stop the instance. After doing so, run docker container rm [CONTAINER ID] to completely remove the container instance.

Visit localhost:[YOUR PORT] (localhost:8082 on my example) to confirm that the Jenkins instance is not running anymore. You should see something like this:

Run the same command we used to create the container instance in order to recover the Jenkins instance.

docker container run -d -p 8082:8080 \ -v jenkinsvol1:/var/jenkins_home \ --name jenkinslocal \ jenkins/jenkins:lts

You should get a new CONTAINER ID after running the command. Visit localhost:[YOUR PORT] (localhost:8082 on my example). You should see the login page. Login with the admin credentials you set during Jenkins initialization.

After logging in, you should see the job you created and view the console output.

In summary

CD with Docker and Feature Branch Testing

Rangle.io — Fri, 25 Oct 2019 17:28:06 +0000

By: Ben Martin

If you've been following along with my Docker series (you can find my latest article about Continuous Integration (CI) here) then you must be pretty happy to have your CI pipeline solving all the world's problems. Your developers are pretty content, but we know there's more we could do. And, I mean, isn't developer happiness the real reason you're reading a DevOps article?

In this article, I'll outline how you can take the CI pipeline one step further to address the Continuous Deployment (CD) aspect of CICD. More specifically, we'll address how one would configure feature branch specific deployments of our app in order to quickly and easily manually test their features, even on mobile devices, before merging their feature branch PR to the develop branch.

Prerequisite Knowledge

If you haven't been following along with my previous articles, to get the most out of this article, you should first have a Dockerized application. If you also have a CircleCI pipeline configured, that's a huge head start! We will be building off that. In our hypothetical situation, we're using an AWS EC2 instance from the free tier in AWS. If you're new to AWS, this could easily be replaced by other technologies. One would only need a server to deploy to. In the past, I used a $5 Digital Ocean droplet to accomplish the same effect, but it's up to you to determine where you want to deploy.

Here's what you'll need:

An AWS Account with an EC2 instance configured to expose ports 22, 80 and 443 and an SSH key for CircleCI to use
Source code of Dockerized App hosted on GitHub
Docker Hub (or some other Docker image repository you can publish to)
CircleCI connected to your app's GitHub repo and your Docker Hub repo
Domain name pointing to your server
Wildcard TLS Certificate for the above domain name

Note: Details for most of the above, aside from anything deployment related, can be found in my previous article.

The Goal: Fast Feedback

As a huge proponent of Agile development methodologies, fast feedback is critical for success, and the higher fidelity the feedback, the better! Unit tests and end-to-end tests all have their place. But, having human eyes review UI changes running in a production-like environment in the browser helps us identify bugs before they become an issue that could impact other teams.

Why should I bother?

In my case, I was working on a project recently where I was developing an Angular application to be consumed through a mobile device. The app required use of the mobile device's camera and accelerometer for the features I was developing. Thus, I wanted to be able to quickly test these features with my actual phone while I was still working out the implementation details.

Sounds simple enough, I just needed to deploy my code somewhere I could access from my device. But this process needed to be automated.

I decided to leverage my existing CI pipeline and extend it to do Continuous Deployments (CD). My CI pipeline was already configured to run my unit tests and end-to-end tests, but I wanted to also deploy my code to a server that I could access from my phone for manual testing.

Here are some key considerations for extending this CI/CD pipeline.

Feature Branch Environments

If you've worked on an enterprise webapp, you may be familiar with terms like dev, int(egration) and staging. Typically, these are environments where different states of our application reside. Each of these environments are meant to host various states of your app, whether that be the branch actively under development, your next release or the last release that you're still supporting with bug fixes.

What often gets lost in the mix is our feature branches. When our developers pick up a ticket, they may want to actively test their code on a suite of devices to ensure cross device and cross browser compatibility. Plus, having this requirement for our developers acts as an additional safeguard for preventing bugs from getting into the develop branch of our code.

If you have ever asked your boss for branch specific environments, you may well have been scoffed at claiming "there's no room in the budget". Well, what if I told you that through the power of Docker and the free tier on AWS (or an existing server you may have running), you can have branch-specific environments beyond dev, int and staging.

Docker To The Rescue!

Docker is the perfect tool for solving this issue for several reasons. In order to see why, consider what would be difficult about running multiple copies of our app on the same environment. Well, even if we imagine a static site there's a lot to cover off. When we deploy our app, what folder do we deploy it to? How do we handle port collisions? Asset loading with relative URLs? How do we notify nginx of the new route/site? There's a lot of configuration and mess that goes into doing it directly on the host machine that we can delegate to Docker to handle for us.

In the end, we'll have a long lived reverse nginx proxy running in a container on our host machine. This container will listen to Docker for other containers we run. Using environment variables in the docker run command, we will be able to tell this nginx proxy to update the configuration to point dynamic subdomains to our feature branch specific containers.

Docker with nginx

Most of the heavy lifting is done by the nginx proxy container. I found an excellent dockerized nginx proxy on GitHub to aid in this process. What is useful about this Docker image is that once you have it running, it will listen to your other docker commands that are run on the host machine, in particular your docker run command. In particular, it will look for environment variables attached to your docker run command to interpret how to update the internal configuration and point it to your app container that is being started.

This reverse proxy image should be running in a container on your server. This container's role is to listen to other Docker commands that you run and automatically update the nginx configuration within this reverse proxy container.

Passing Inputs to your Docker Run

Once you have the nginx proxy running in a Docker container, you can leverage your CI pipeline (like CircleCI) to get the environment variables needed to pass to your server in the docker run command. Just make sure that CircleCI is authorized to access your server via SSH. A secure way to do this is using SSH keys. Generate a key and provide the private key to CircleCI.

Then, add the public key to the ~/.ssh/authorized_keys of your server. From there, make a bash script to manage your branch-specific containers.

CircleCI exposes a number of environment variables to your pipeline that you can use in your jobs to pass to your bash script.

In my case, I used CircleCI to run scp to copy a deployment script to the testing server. Once the bash file is there, it would be executed with the arguments to tell it what sub-domain to use for this container. I used a sanitized version of the branch name. For example, feature/ABC-42-my-feature becomes feature_abc-42-my-feature.myexampledomain.com That subdomain would also be the alias for the container, allowing my script to stop a running container with outdated code when new commits are made to that particular branch.

Here's an example of the run command from my deployment script that provides the environment variables (VIRTUAL_HOST, VIRTUAL_PROTO and VIRTUAL_PORT) expected by the reverse nginx proxy container. The nginx container is listening to the Docker process on your host machine. When it sees a docker run command, it is also looking for those three environment variables. If it sees them in the command, it will update the nginx configuration to point the virtual host to the container that is being run. This is the command that will be run on the host machine to deploy. These environment variables are defined in the CI pipeline.

docker run --expose 443 -e VIRTUAL_HOST=${URL_SUBDOMAIN}.myexampledomain.com -e VIRTUAL_PROTO=https -e VIRTUAL_PORT=443 -d --rm=true --name ${CONTAINER_NAME} ${DOCKER_IMAGE}

The environment variables I am using here are set inside my bash script to ensure I pull the correct image, apply the right name to the container and assign the URL, port and protocol. Putting it together, we can add a "job" to our CircleCI configuration aliased as deploy. This deploy job would look something like this.

scp deploy-${IMAGE_NAME}.sh ${PROD_SERVER_USER}@${PROD_SERVER_HOST}:/root/
ssh -o StrictHostKeyChecking=no ${PROD_SERVER_USER}@${PROD_SERVER_HOST} "/bin/bash /root/deploy-${IMAGE_NAME}.sh $IMAGE_NAME:$TAG ${URL_SUBDOMAIN}"

Note: PROD_SERVER_USER and PROD_SERVER_HOST are set in the .circleci/config.yml file

Note: $IMAGE_NAME:$TAG and ${URL_SUBDOMAIN} are passed as arguments to the deploy script and are used in our docker run command above.

In the bash file, it is important to make sure there isn't already a container running with the same name. It should first be stopped and removed.

Note: In my case, because I was using HTTPS in order to gain access to the mobile device's camera, I needed to setup TLS with a wildcard domain so that I could add subdomains on the fly and still have TLS support.

Results

Now, anytime this repository receives updates to one of the feature branches, that code (assuming it passes all the steps defined in our CICD pipeline prior to deployment) will be deployed to our testing server, on a custom subdomain.

In my project, I took the branch name, sanitized it by removing any special characters, and used that as my subdomain. So, when I push commits to my branch called feature/mobile-view, I could see my changes by visiting this subdomain: https://feature_mobile-view.myexampledomain.com.

The value here being that in order for your developers to see code they're working on in a production-like environment, they only need to push their feature branch to the remote repository. Even if their code is incomplete, they can quickly prototype and test features that would be otherwise difficult to do so in a local environment, like mobile device features, etc.

Further Improvements

We got what we wanted, but that doesn't mean it's perfect. There are many ways we could continue to polish this solution. Let's look at a few.

HTTPS

Although we didn't dig into it here, it is convenient to add a wildcard TLS certificate to the server you are running. You can also look at https://github.com/JrCs/docker-letsencrypt-nginx-proxy-companion to see how people have combined a separate container to automatically generate the certificates for the Docker containers being spun up. Similar to the image we used for the proxy, this one would also listen to your docker run commands for environment variables to automate this process.

In the project that inspired this setup, I needed HTTPS support to access the device's camera and accelerometer, so, given that this was a fun little side project, manually uploading a single wildcard certificate was sufficient for me.

Another alternative is to put this solution behind an application load balancer (ALB) through AWS. Do pay attention to which services are compatible with the free tier of new AWS accounts. Otherwise be warned that there will be costs involved in fleshing out the solution further.

nginx Restarting

In case the nginx proxy container ever crashed, you'd want to define what your restart policy is. A simple addition, is to add --restart unless-stopped to the docker run command for your nginx reverse proxy. But you also need to think about if the server restarts. There are many tools that one could use to manage starting containers on boot. Even just systemd to turn it into a service would meet these requirements.

Deployment Script

This was something that resided in the code repository so that CircleCI would have access to it when it pulled the code from the repo. Alternatively, this could also reside on the deployment server instead. There will be pros and cons regardless. Discuss with your team which approach works best for you.

AWS EC2 Configuration

There is a lot that could be said here from security to resource management and configuration automation. Some things you want to remember is always start with the fewest permissions and add more as needed. If you're using EC2, leverage the user script for your AWS instance to ensure your server is fully configured and has all the services running to ensure the nginx-proxy stays online. Write cron scripts to automatically remove containers that have been online for longer than "X" days. If there is a new commit to a particular branch, that container would be refreshed. Old containers would likely represent feature branches that have been merged.

Speaking of which...

GitHub Webhooks

Configuring webhooks to ping your server when a branch gets merged or deleted would be another way to ensure you don't have too many inactive containers on your server.

Conclusion

Hopefully this gives you an idea of what needs to be considered when updating your CICD pipeline to support feature branch specific environments and how easy it can be. Just remember the goal: fast feedback.

I am confident that if you implement a solution similar to that described above, your developers will let you know how much they love it, ranting and raving about how great it is. Just be careful. When they eventually move to a new project, they might just realize how much you've spoiled them ;)

DEV Community: Rangle.io

Running Jenkins and Persisting state locally using Docker

Jenkins setup

Persisting Jenkins Data

Confirming Jenkins state is persisted

Recovering Jenkins

In summary

Docker for Frontend Devs: Custom Docker Images for Development

Prerequisite: Create our project

Custom Docker Image for Development

Dockerfile Commands and Keywords

Dockerfile Structure Recommendations

Caching Dependencies

Priority

Building the Custom Docker Image

Running our Container: Quality of Life Improvements

What was added?

Mounting Host Directories

Accessing Ports Inside the Container

docker run in One Command

Testing it out

In Closing

Connecting Stripe events to the AWS EventBridge

Why AWS EventBridge?

The solution

Serverless components in stripe-eventbridge

Routing by event type

And that's all!

Running Jenkins and Persisting state locally using Docker

Jenkins setup

Persisting Jenkins Data

Confirming Jenkins state is persisted

Recovering Jenkins

In summary

CD with Docker and Feature Branch Testing

Prerequisite Knowledge

The Goal: Fast Feedback

Why should I bother?

Feature Branch Environments

Docker To The Rescue!

Docker with nginx

Passing Inputs to your Docker Run

Results

Further Improvements

HTTPS

nginx Restarting

Deployment Script

AWS EC2 Configuration

GitHub Webhooks

Conclusion

`docker run` in One Command