DEV Community: Cadu Ribeiro

Running Neovim with Devcontainers

Cadu Ribeiro — Thu, 16 May 2024 15:00:00 +0000

In this post, I will show you how you can use Neovim with DevContainers to simplify the development environment setup. DevContainers is an open specificationthat that allows containers to be used as a complete development environment with all tools necessary for the development lifecycle. To read more about DevContainers, check my previous post here or the official documentation here.

One thing that most people don’t know is that DevContainers is not only for VSCode. It is a specification that can be used with any editor or IDE (although the integration with VSCode has the best support).

For example, the video below shows DevContainers being used with RubyMine:

Your browser does not support the video tag.

Let’s see how we can use Neovim with DevContainers.

The example project

We need Docker installed on our machine to run DevContainers. If you don’t have Docker installed, you can install it by following the instructions here.

We will create a new Ruby on Rails project for this example and use the new rails-new tool to generate the project. This tool is a new way to generate Rails applications even when you don’t have Ruby installed on your machine, as it uses Docker to run the generator (useful for people who don’t want to install Ruby on their machines and rely only on Docker). It is still in the experimental phase, but it is already usable. See here for installation instructions.

I’ve installed the rails-new tool on my machine using Cargo (the Rust package manager). If you don’t have Cargo installed, you can install it by following the instructions here.

cargo install --git https://github.com/rails/rails-new

Now, let’s create a new Rails project using the rails-new tool:

rails-new nvim-devcontainer-post -- --main -d postgresql

This command will create a new Rails project named nvim-devcontainer-post with the --main flag, which will use Rails’ main branch instead of the regular releases version, because the Rails on the main branch already generates a project with DevContainers configured by default (This will be available on Rails 8). The -d postgresql flag will configure the project to use PostgreSQL as the database.

After running this command, you will have a new Rails project with DevContainers configured. You can check that the .devcontainer folder was created with the necessary files for DevContainers.

➜ nvim-devcontainer-post git:(main) ✗ ls .devcontainer
Dockerfile compose.yaml devcontainer.json

Our project is ready for us to start using Neovim with DevContainers. You can even open the project in VSCode and see that the DevContainers is working:

Your browser does not support the video tag.

Let’s destroy these containers so we can install more tools:

docker compose -f .devcontainer/compose.yaml down

Installing Neovim in the DevContainer

The way DevContainers work is by running the editor or IDE server inside the container, this way the editor or IDE can access all the tools installed in the container like Ruby, the language server (LSP), linting and formatting tools, etc. This is how VSCode and RubyMine work with DevContainers, they have their server running inside the container and communicate with the editor running on the host machine via remote editing. This is why we need to install Neovim in the DevContainer to use it with DevContainers.

If you read my previous post about DevContainers, you know that we can install additional tools in the DevContainer by adding them to the Dockerfile or even using DevContainer’s Features to install them. Both ways are valid, but I prefer to install them via Features.

I’ve created a repository with Features to install Neovim and Tmux in the DevContainer. You can check the repository here.

So, let’s install Neovim in our DevContainer using the Features. First, we need to edit the .devcontainer/devcontainer.json file and add the following content to the features key:

  "features": {
    "ghcr.io/devcontainers/features/github-cli:1": {},
    "ghcr.io/rails/devcontainer/features/activestorage": {},
    "ghcr.io/rails/devcontainer/features/postgres-client": {},
    "ghcr.io/duduribeiro/devcontainer-features/neovim:1": { "version": "nightly" },
  },

We’ve added the neovim Feature to the features key, the other three Features are the default Features that Rails added to the project when we created the project. I’ve used the nightly version of Neovim, because my Neovim configuration uses some features that are only available in the nightly version.

Starting the DevContainer

How can we start the DevContainer without needing to open the project in VSCode? We can use the devcontainers-cli to start the DevContainer from the command line without needing to open the project in VSCode.

You can install the devcontainers-cli by running the following command: (check here for more information on the installation process)

npm install -g @devcontainers/cli

The devcontainers-cli is a CLI tool that allows you to control DevContainers from the command line. It is still missing some features (like stopping the DevContainer), but you can already use it to start and execute commands in the DevContainer.

Let’s build and start our DevContainer using the devcontainers-cli:

devcontainer build --workspace-folder .

....

devcontainer up --workspace-folder .

This command will build the DevContainer and start it. After running this command you will receive the message with the outcome status:

[+] Running 4/4
 ✔ Container nvim_devcontainer_post-redis-1 Started 0.0s
 ✔ Container nvim_devcontainer_post-postgres-1 Started 0.0s
 ✔ Container nvim_devcontainer_post-selenium-1 Started 0.0s
 ✔ Container nvim_devcontainer_post-rails-app-1 Started 0.0s
{"outcome":"success", ...}

This means that the DevContainer was started successfully. Now we can execute commands in the DevContainer using the devcontainer exec command, like this:

devcontainer exec --workspace-folder . ls

Dockerfile Gemfile Gemfile.lock README.md Rakefile  app bin config config.ru db lib log public storage test tmp vendor

we see that the ls command was executed in the DevContainer and we received the output with the files in the root of the project.

Different from VSCode or RubyMine that have a client running on the host machine that communicates with the server running in the container, Neovim will run inside the container and we access it via the terminal. This may can change in the future if they implement a remote editing feature that allows us to run the server in the container and the client on the host machine, but for now, we need to run Neovim using the devcontainer-cli.

devcontainer exec --workspace-folder . nvim

Neovim is now running inside the DevContainer and we can use it to edit files in the project:

But this is not using any configuration or plugins because we are running Neovim in the DevContainer and we don’t have any configuration files or plugins installed there. You can install your Neovim configuration in the DevContainer manually via terminal but everytime that you need to rebuild the container you will need to install it again. A solution for this is to copy your Neovim configuration from your host machine to the DevContainer.

I have my Neovim configurations in my machine at ~/.config/nvim, so I can copy it to the DevContainer during the start process. Let’s see how we can do this.

Before, let’s stop our containers:

docker compose -f .devcontainer/compose.yaml down

Copying Neovim configuration to the DevContainer

To copy the Neovim configuration from the host machine to the DevContainer, we can specify mount points during the devcontainer up command. This is how I do:

devcontainer up --mount "type=bind,source=$HOME/.config/nvim,target=/home/vscode/.config/nvim" --workspace-folder .

DevContainers has a way to specify mount points on the .devcontainer/devcontainer.json file (see the json reference and look for mounts) but it didn’t work for me, so I use the --mount flag in the devcontainer up command.

This command will mount the ~/.config/nvim folder from the host machine to the /home/vscode/.config/nvim folder in the DevContainer. Now we can run Neovim with our configuration:

devcontainer exec --workspace-folder . nvim

And this is how Neovim is running with my configuration in the DevContainer, with all my plugins and settings and even running the LSP:

Your browser does not support the video tag.

This is how I use Neovim with DevContainers and I hope this post helps you with this config too. DevContainers is a great tool to simplify the development environment setup and I think (and hope) that most editors and IDEs will support it in the future.

Using Devcontainers to set up your development environment

Cadu Ribeiro — Thu, 28 Sep 2023 15:00:00 +0000

One common problem in software development is setting up the project’s development environment. Have you ever joined a project, opened the README.md, and found a README HELL, filled with endless instructions on how to configure the project? And halfway through the instructions, you encounter an error while running a command because you’re not on the same operating system version as the person who wrote the README, or because the documentation is outdated.

Quoting Vladimir Dementyev in his amazing talk Terraforming Legacy Rails applications:

Developers should be able to run project with the least possible effort

for example, cloning the code and running a few scripts for server setup and start.

GitHub published a blog post in 2015 where they demonstrated how they did this at the time. They state:

Having a bootstraping experience for projects reduces friction and encourages contribution.

Reduces friction: This is very interesting when we think about onboarding a new person to the project. Do you remember the past times when a newcomer would take days to get the project up and running on their computer?

And in this post, GitHub shows how they solved this in 2015: A set of scripts, with a standard naming convention across all projects, that handle dependency installation and updates, project setup, test execution, and environment configuration. They refer to this set of scripts as ‘Scripts to Rule Them All.’


https://www.youtube.com/watch?v=HgOha2D5kt8&themeRefresh=1

This model greatly helped new hires at the company configure and start projects within half a day. In another post, they mention that in the vast majority of cases, everything worked without issues and that when something didn't work, there was a Slack channel called #friction where others debugged and helped resolve system issues. Event with nvironment setup scripts, issues persisted because the company scripts were based on macOS, while individuals might use different operating systems like Linux or even a more recent macOS version not yet supported by the scripts. These errors continued to create friction when starting a development environment for the project.

How can we further reduce friction? Enter DevContainers.

What are DevContainers?

Development Containers (or DevContainers) is an open specification that allows containers to be used as a complete development environment, enabling us to run our applications, dependencies like databases and messaging services, and other tools necessary for the development lifecycle. DevContainers can be run locally or in a remote environment (including services like GitHub Codespaces and GitPod).

In this text, I won’t enter into the basics of containers/docker. If you’d like to learn and know more, please visit https://www.docker.com/ and https://cloud.google.com/learn/what-are-containers.

The DevContainers specification states that your project have a folder called .devcontainer with a devcontainer.json file. To read the complete specification, visit this link. In summary, the file contains the image (or Dockerfile) to be used, the container’s forwarded ports, specific product customizations (e.g., installing extensions by default in VSCode), and more.

One of the inclusions in the specification is the DevContainer Features, which are independent and shareable units containing installation or configuration code for the container. These features are installed on top of the image used in the DevContainer. The idea behind features is to easily add more tools and libraries to the DevContainer. The following example demonstrates how to install the GitHub CLI, for instance:

// .devcontainer/devcontainer.json
{
  "name": "MyApp DevContainer",
  //...
  "features": {
    "ghcr.io/devcontainers/features/github-cli:1": {
        "version": "latest"
    }  
  }
}

A list of existing Features can be found here.

Advantages and disadvantages of using DevContainers

One of the evident advantages we observe in using DevContainers is reducing friction in project setup. Having this reproducible development environment ensures that everyone on the team is using the same environment, making project setup easier. This will make new team members onboarding much faster and easier. It’s common that we don’t see problem with slow onboarding because we assume it will only happen once, which is not always true (maybe you got a brandly new computer?). Another advantage is that no matter which operating system you are using, the development environment will work, and you don’t need separate instructions in the docs (no more “if you are on Linux, do this” in your README).

But of course, this approach may have some disadvantages. Possible speed reduction (especially on macOS). We know that Docker on macOS runs on top of a virtual machine, which impacts performance a bit, as discussed in this article. There are some alternatives that promise to improve Docker’s speed on macOS, such as Colima and OrbStack. My friend Felipe Vaz is using Colima and said it is good. I am currently testing OrbStack based on a recommendation from Rob Zolkos.

Also, stuff that works well, such as integration tests with browsers (e.g., using Selenium), can be a trickier to set up and get working perfectly.

As with everything in software development, it’s a tradeoff, and you should consider the pros and cons to see if it’s worth it in your case. For me, the advantages outweigh the disadvantages, and it’s worth using. It may not be the case for you.

Production Containers vs. DevContainers

If I already have containers and a Dockerfile for my production app running, why not use them instead of a separate container for development?

When we talk about reproducible environments with DevContainers, this question often arises, and it’s a good question. One of the 12factors suggests having parity between development and production, which leads us to try using the same production container image in development. However, note that this advice emphasizes making environments as similar as POSSIBLE, not EXACTLY THE SAME.”

Keep development, staging, and production as similar AS POSSIBLE

This is a subtle distinction but makes a difference when we deploy our development containers. One of the points that the “Dev/prod parity” factor addresses is:

Backing services, such as the app’s database, queueing system, or cache, is one area where dev/prod parity is important. Many languages offer libraries which simplify access to the backing service, including adapters to different types of services….

Developers sometimes find great appeal in using a lightweight backing service in their local environments, while a more serious and robust backing service will be used in production. For example, using SQLite locally and PostgreSQL in production; or local process memory for caching in development and Memcached in production.

The twelve-factor developer resists the urge to use different backing services between development and production

And that’s the idea that Dev/prod parity brings. Try to make your development environment as similar as POSSIBLE to production. If you use PostgreSQL in production, don’t use SQLite as your development database because there are differences between them that can cause compatibility issues in your application, and you’ll only notice them in production. Note that the primary focus of Dev/prod parity is on using different backing services, not telling you to use the SAME IMAGE used in production for development.

Production containers have different requirements from development containers!

When we think about deploying our application in containers, we have concern such as:

Attempting to minimize the size of the final container image as much as possible
Having as few dependencies as possible
Exposing the minimum number of open ports
Reducing the application’s memory consumption

Furthermore, the images we use as a base for our production containers are based on very small and highly suitable images for a production environment (such as debian-slim or alpine), but they are not as suitable for a development environment. (You’ll want to run fzf in your DevContainer, but it doesn’t make sense to have it in your production image, for example.)

In the development environment, we want a complete system (such as ubuntu or debian) with various utilities and auxiliary tools to assist the daily work of project contributors (e.g., installing fzf for searching, vim for quick file editing, a more comprehensive shell with multiple auto-completions) and we can leave more ports open to facilitate application debugging.

On the Overview page of DevContainers, in the Development vs production section, you can find the following passage:

While deployment and development containers may resemble one another, you may not want to include tools in a deployment image that you use during development.”


https://containers.dev/overview#Development-vs-production

In the image, we can see that a DevContainer for a development environment (inner loop) can include various things that are not necessary for production. In fact, we can consider that in the Outer loop (CI), this DevContainer can be used with even fewer items included.

Considering these points, it might make sense to have two container definitions, one for production and one for development. To achieve this, I like to have two Dockerfile definitions. One at the root of the project to define the production image and another inside .devcontainer for the development environment using DevContainers.

So, the project structure looks like this:

% tree -a | grep Dockerfile -C 1
├── .devcontainer
│   ├── Dockerfile
--
├── Dockerfile

Practical Example: A Ruby on Rails Application

For this practical example, I’ll be using a Ruby on Rails application. It’s also necessary to have prior knowledge of containers, Docker, and Docker Compose.

This GitHub repository contains the source code used in this post and it’s broken down into tags containing the progress. The tag 0-initial contains the base application used from here on.

This application was generated using Rails 7.1. Rails 7.1 introduced a feature that generates a Dockerfile by default when you create a new application. However, this Dockerfile is optimized for production, as stated in the release note:

It’s important to note that these files are not meant for development purposes.

As we’ve seen earlier, the purpose of a Dockerfile for the development environment is different from one for production. So, let’s create our Dockerfile that will be used in our DevContainer.

The Dockerfile for the DevContainer.

Let’s begin by creating our Dockerfile that will be used in our DevContainer. Inside the .devcontainer folder, I will create my Dockerfile:

# .devcontainer/Dockerfile

ARG DEBIAN_FRONTEND=noninteractive
ARG VARIANT=bullseye

FROM mcr.microsoft.com/vscode/devcontainers/base:${VARIANT}

RUN apt-get update && \
    apt-get -y install --no-install-recommends \
    build-essential gnupg2 tar git zsh libssl-dev zlib1g-dev libyaml-dev curl libreadline-dev \
    postgresql-client libpq-dev \
    imagemagick libjpeg-dev libpng-dev libtiff-dev libwebp-dev libvips \
    tzdata \
    tmux \
    vim

# Install rbenv and ruby
USER vscode

ARG RUBY_VERSION="3.2.2"
RUN git clone https://github.com/rbenv/rbenv.git /home/vscode/.rbenv \
    && echo '[-f "/home/vscode/.rbenv/bin/rbenv"] && eval "$(rbenv init - bash)" # rbenv' >> /home/vscode/.zshrc \
    && echo '[-f "/home/vscode/.rbenv/bin/rbenv"] && eval "$(rbenv init - bash)" # rbenv' >> /home/vscode/.bashrc \
    && echo 'export PATH="/home/vscode/.rbenv/bin:$PATH"' >> /home/vscode/.zshrc \
    && echo 'export PATH="/home/vscode/.rbenv/bin:$PATH"' >> /home/vscode/.bashrc \
    && mkdir -p /home/vscode/.rbenv/versions \
    && mkdir -p /home/vscode/.rbenv/plugins \
    && git clone https://github.com/rbenv/ruby-build.git /home/vscode/.rbenv/plugins/ruby-build

ENV PATH "/home/vscode/.rbenv/bin/:HOME/.rbenv/shims/:$PATH"

RUN rbenv install $RUBY_VERSION && \
    rbenv global $RUBY_VERSION && \
    rbenv versions

COPY .devcontainer/welcome.txt /usr/local/etc/vscode-dev-containers/first-run-notice.txt

I won’t go into much detail on how it works but will instead explain the reasons behind some decisions made. To understand more about how a Dockerfile works, the official documentation is the best reference.

The base image used.

In the first instructions, we define which base image we will use:

# .devcontainer/Dockerfile

ARG VARIANT=bullseye

FROM mcr.microsoft.com/vscode/devcontainers/base:${VARIANT}

The FROM instruction specifies to Docker which image to use as the base for ours. And here, we’ve already made the first decision. Which one will we use?

We can use any base image for ours (including using debian directly). So, could we use the ruby:3 image directly? Yes.

FROM ruby:3

is a valid Dockerfile to use as a DevContainer, and in fact, some open-source projects (e.g., Forem) use it. However, Microsoft’s mcr.microsoft.com/vscode/devcontainers/ images are specifically prepared for use in DevContainers, adding various development tools. Here, for example, you can see one of the scripts executed in the mcr.microsoft.com/vscode/devcontainers/ images.

For this reason, I prefer to use the mcr.microsoft.com/vscode/devcontainers images. It’s just a personal preference and doesn’t mean that using other images as DevContainers is wrong. You can use any Docker image as a base.

And why use `mcr.microsoft.com/vscode/devcontainers/base` instead of `mcr.microsoft.com/vscode/devcontainers/ruby`?

This is another decision made based on preference. I recently used mcr.microsoft.com/vscode/devcontainers/base in my projects and manually installed Ruby due to a small “issue” (which might be specific to me 😀) I found in mcr.microsoft.com/vscode/devcontainers/ruby. The mcr.microsoft.com/vscode/devcontainers/ruby image installs two version managers: rbenv and rvm, which can cause some issues. One issue I noticed is when your project has a .ruby-version file (as in our example, generated by Rails). The .ruby-version file tells the version managers which Ruby version to install. However, what I noticed is that when you don’t have this file, the mcr.microsoft.com/vscode/devcontainers/ruby image installs Ruby using rbenv, and when you do have this file in the project, it installs Ruby using rvm.

And what’s the problem with installing it using rvm? Technically, there’s no issue, but you may run into some cases like I did. I was using the VSCode extension for ruby-lsp, and the extension’s version manager detection was set to auto (https://github.com/Shopify/vscode-ruby-lsp#ruby-version-managers){:target=”\_blank”}. The extension tried to detect the installed version using rbenv, but my DevContainer was using rvm. Check out these two issues, https://github.com/devcontainers/images/issues/572 and https://github.com/microsoft/vscode-dev-containers/issues/704, for more details.

Also, it’s not necessary to have a version manager in a container. When you need to use a different Ruby version, you can install it directly in the container, avoiding the installation of multiple versions together.

Because of this difference, I preferred to use the base image and manually install Ruby with the following instructions:

# Install rbenv and ruby
USER vscode

ARG RUBY_VERSION="3.2.2"
RUN git clone https://github.com/rbenv/rbenv.git /home/vscode/.rbenv \
    && echo '[-f "/home/vscode/.rbenv/bin/rbenv"] && eval "$(rbenv init - bash)" # rbenv' >> /home/vscode/.zshrc \
    && echo '[-f "/home/vscode/.rbenv/bin/rbenv"] && eval "$(rbenv init - bash)" # rbenv' >> /home/vscode/.bashrc \
    && echo 'export PATH="/home/vscode/.rbenv/bin:$PATH"' >> /home/vscode/.zshrc \
    && echo 'export PATH="/home/vscode/.rbenv/bin:$PATH"' >> /home/vscode/.bashrc \
    && mkdir -p /home/vscode/.rbenv/versions \
    && mkdir -p /home/vscode/.rbenv/plugins \
    && git clone https://github.com/rbenv/ruby-build.git /home/vscode/.rbenv/plugins/ruby-build

ENV PATH "/home/vscode/.rbenv/bin/:/home/vscode/.rbenv/.rbenv/shims/:$PATH"

RUN rbenv install $RUBY_VERSION && \
    rbenv global $RUBY_VERSION && \
    rbenv versions

The last instruction in the Dockerfile simply copies the file .devcontainer/welcome.txt to the location /usr/local/etc/vscode-dev-containers/first-run-notice.txt in the container. This file sets up a message that will be displayed when we open the terminal in VSCode.

Let’s create our .devcontainer/welcome.txt then.

👋 Welcome to "DemoApp"!

🛠️ Your environment is fully setup with all the required software.

Our Dockerfile is ready. We can build the image to see if everything is okay by running docker build in the root of the project:

% docker build -f .devcontainer/Dockerfile .

[+] Building 0.1s (10/10) FINISHED docker:orbstack
 => [internal] load build definition from Dockerfile 0.0s
 => => transferring dockerfile: 1.47kB 0.0s
 => [internal] load .dockerignore 0.0s
 => => transferring context: 766B 0.0s
 => [internal] load metadata for mcr.microsoft.com/vscode/devcontainers/base:bullseye 0.0s
 => [internal] load build context 0.0s
 => => transferring context: 265B 0.0s
 => [1/5] FROM mcr.microsoft.com/vscode/devcontainers/base:bullseye 0.0s
 => CACHED [2/5] RUN apt-get update && apt-get -y install --no-install-recommends build-essential gnupg2 tar git zsh libssl-dev zlib1g-dev libyaml-dev curl libreadline-dev postgresql-client libpq 0.0s
 => CACHED [3/5] RUN git clone https://github.com/rbenv/rbenv.git /home/vscode/.rbenv && echo '[-f "/home/vscode/.rbenv/bin/rbenv"] && eval "$(rbenv init - bash)" # rbenv' >> /home/vscode/.zshrc & 0.0s
 => CACHED [4/5] RUN rbenv install 3.2.2 && rbenv global 3.2.2 && rbenv versions 0.0s
 => [5/5] COPY .devcontainer/welcome.txt /usr/local/etc/vscode-dev-containers/first-run-notice.txt 0.0s
 => exporting to image 0.0s
 => => exporting layers 0.0s
 => => writing image sha256:1ca67988b183ade50ca4f4a76e24d7cf76de0f7e7a12b0ea1d516fc25a67b501

The output indicates that our image is okay, and now we can proceed with building our devcontainer.json.

The source code up to this point: Link

Specifying our development container in the `devcontainer.json`.

As we saw at the beginning, the .devcontainer/devcontainer.json file contains the necessary configurations for our DevContainer so that tools and services that support the devcontainer specification can start up and connect to the DevContainer. With this specification, we will be able to make VSCode set up our environment when it detects the project.

// .devcontainer/devcontainer.json

// For format details, see https://aka.ms/devcontainer.json. For config options, see the
// README at: https://github.com/devcontainers/templates/tree/main/src/ruby-rails-postgres
{
    "name": "DemoApp DevContainer",
    "build": {
        "dockerfile": "Dockerfile",
        "context": ".."
    },
    "workspaceFolder": "/workspaces/${localWorkspaceFolderBasename}",
    "remoteEnv": {
        "GIT_EDITOR": "code --wait"
    }
}

The complete specification and documentation for devcontainer.json can be found here, but to summarize, here we define that our DevContainer is named “DemoApp DevContainer” and specify that it will use our Dockerfile created in the previous step. If we open our project with VSCode, we will receive a message indicating that it has detected a DevContainer specification and suggests opening the project inside the container:

In this popup, we can click on Reopen in container, or in the command palette, you can search for the command directly.

Demo time!

In the demo, we see that when we use the “Reopen in container” option, VSCode opens the project inside the container. Extensions run within the container and not on the local operating system. This allows extensions (e.g., LSP) to run their clients within the container itself. Read here for more information on the DevContainers architecture. The image below illustrates this:


https://code.visualstudio.com/docs/devcontainers/containers

We also see that the tools are already installed in our container. We have ruby and even vim inside the container. And this is the difference from a production container. Here, we have everything we need for a complete development environment.

Source code up to this point: Link

Project Dependencies

Inside our DevContainer, if we try to run the bin/setup command of our project (which installs dependencies and sets up the database), we will encounter some errors.

The first error we receive:

bin/setup:8:in `system': No such file or directory - bun (Errno::ENOENT)
        from bin/setup:8:in `system!'
        from bin/setup:21:in `block in <main>'
        from /home/vscode/.rbenv/versions/3.2.2/lib/ruby/3.2.0/fileutils.rb:244:in `chdir'
        from /home/vscode/.rbenv/versions/3.2.2/lib/ruby/3.2.0/fileutils.rb:244:in `cd'
        from bin/setup:11:in `<main>'

Our Rails project was generated using bun as the JavaScript package manager and bundler. (Rails now supports Bun thanks to the great work of Jason Meller in this PR). However, our DevContainer does not have bun installed. We can fix this in two ways:

Modify our Dockerfile to install bun
Use the DevContainer Features ghcr.io/shyim/devcontainers-features/bun

We will proceed with the second option. It’s a simple option for installing tools when they are available. We modify our devcontainer.json to include the following instruction:

    "features": {
        "ghcr.io/shyim/devcontainers-features/bun:0": {}
    },

Our devcontainer.json looks like this now:

// For format details, see https://aka.ms/devcontainer.json. For config options, see the
// README at: https://github.com/devcontainers/templates/tree/main/src/ruby-rails-postgres
{
    "name": "DemoApp DevContainer",
    "build": {
        "dockerfile": "Dockerfile",
        "context": ".."
    },
    "features": {
        "ghcr.io/shyim/devcontainers-features/bun:0": {}
    },
    "workspaceFolder": "/workspace",
    "remoteEnv": {
        "GIT_EDITOR": "code --wait"
    }
}

VSCode notices that we have modified the specification of our DevContainer and suggests rebuilding it:

After finishing the DevContainer build, if we run bun -v, we will see that it is now installed:

vscode ➜ /workspace (main) $ bun -v
1.0.3

Running bin/setup again, and now the failure occurs when preparing the database.

PG::ConnectionBad: could not connect to server: No such file or directory
        Is the server running locally and accepting
        connections on Unix domain socket "/var/run/postgresql/.s.PGSQL.5432"?

Our project is configured to use PostgreSQL, and this error occurs because we don’t have it running in our container. Let’s run our database using Docker Compose, which allows us to define a development environment with multiple containers. Instead of adding PostgreSQL to our Dockerfile, we’ll add an additional container to our environment via Compose.

Let’s create our .devcontainer/docker-compose.yml file. Refer to the documentation for more information on Docker Compose specification.

version: '3'

services:
  app:
    build:
      context: ..
      dockerfile: .devcontainer/Dockerfile

    volumes:
      - ..:/workspace:cached
      - $HOME/.ssh/:/home/vscode/.ssh/
    depends_on:
      - postgres
    environment:
      - DATABASE_URL=postgres://postgres:postgres@postgres:5432

    # Overrides default command so things don't shut down after the process ends.
    command: sleep infinity

    # Runs app on the same network as the database container, allows "forwardPorts" in devcontainer.json function.
    network_mode: service:postgres

  postgres:
    image: postgres:15-alpine
    restart: unless-stopped
    volumes:
      - postgres-data:/var/lib/postgresql/data
    environment:
      POSTGRES_USER: postgres
      POSTGRES_DB: postgres
      POSTGRES_PASSWORD: postgres
    healthcheck:
      test: pg_isready -U postgres -h 127.0.0.1
      interval: 5s

volumes:
  postgres-data:

In summary, in our Compose file, we define two services: the app, which will be our devcontainer and is built based on our Dockerfile, and the postgres, which will use the official postgres image. We add an environment variable DATABASE_URL with the value postgres://postgres:postgres@postgres:5432 to inform our project about this database endpoint. In our devcontainer, we override the default command with command: sleep infinity to prevent the container from exiting when the main process finishes.

We also modify the .devcontainer/devcontainer.json file to use Docker Compose instead of just our Dockerfile. We remove the build directive and add the dockerComposeFile and service sections. Here’s the resulting file:

// .devcontainer/devcontainer.json

// For format details, see https://aka.ms/devcontainer.json. For config options, see the
// README at: https://github.com/devcontainers/templates/tree/main/src/ruby-rails-postgres
{
    "name": "DemoApp DevContainer",
    "dockerComposeFile": "docker-compose.yml",
    "service": "app",
    "features": {
        "ghcr.io/shyim/devcontainers-features/bun:0": {}
    },
    "workspaceFolder": "/workspace",
    "remoteEnv": {
        "GIT_EDITOR": "code --wait"
    }
}

After rebuilding the DevContainer and accessing it, we run bin/setup again, and now the database is created, and our setup is completed successfully.

Project code up to this point: link

Running the project

With the project set up, we can start the project with the command bin/dev.

The application will start inside the container, and VSCode will give us the option to open the browser. When we access at http://localhost:3000 it show the page with success:

Everything is set up, and our project is running, achieving our goal. We have a development environment that anyone who clones the project can run in just a few minutes.

Some improvements to our DevContainer

Let’s add a few more things to our DevContainer to improve our experience.

The first change we’ll make is to add an onCreateCommand. This directive tells the DevContainer what command to run when it’s created. Cloud DevContainer services (like GitHub Codespaces) also use this command for caching and prebuilding the container to reduce setup time. In our onCreateCommand script, we’ll update system gems (like Bundler) and run bin/setup. This way, every time a DevContainer is created, we won’t need to run bin/setup and can directly start the server when we initiate the container.

// .devcontainer/devcontainer.json

 "onCreateCommand": ".devcontainer/onCreateCommand.sh"


#!/usr/bin/env bash

# .devcontainer/onCreateCommand.sh

echo "Updating RubyGems..."
gem update --system -N

echo "Setup.."
bin/setup

echo "Seeding database..."
bin/rails db:seed

echo "Done!"

Another improvement we can make is to install extensions and add VSCode settings by default. This way, everyone who starts a DevContainer will have the project’s standard extensions installed. In the example below, we have some extensions like ruby-lsp and sqltools for connecting to the database. We also adjust the sqltools settings to include the database connection information.

// .devcontainer/devcontainer.json

    "customizations": {
        "vscode": {
            // Set *default* container specific settings.json values on container create.
            "settings": {
                "sqltools.connections": [
                    {
                        "name": "Development Database",
                        "driver": "PostgreSQL",
                        "previewLimit": 50,
                        "server": "postgres",
                        "port": 5432,
                        "database": "devcontainer_rails_demo_development",
                        "username": "postgres",
                        "password": "postgres"
                    },
                    {
                        "name": "Test Database",
                        "driver": "PostgreSQL",
                        "previewLimit": 50,
                        "server": "postgres",
                        "port": 5432,
                        "database": "devcontainer_rails_demo_test",
                        "username": "postgres",
                        "password": "postgres"
                    }
                ],
                "editor.formatOnSave": true
            },
            "extensions": [
                "Shopify.ruby-lsp",
                "manuelpuyol.erb-linter",
                "GitHub.github-vscode-theme",
                "eamodio.gitlens",
                "aki77.rails-db-schema",
                "bung87.rails",
                "mtxr.sqltools-driver-pg",
                "mtxr.sqltools",
                "testdouble.vscode-standard-ruby"
            ],
            "rubyLsp.enableExperimentalFeatures": true
        }
    },

We can also fix the ports that are accessible from the container. When we start the server, VSCode identifies that port 3000 is open and performs an auto-forward. However, we can pre-establish this in the devcontainer.json and even open the database port:

    // .devcontainer/devcontainer.json

    // Use 'forwardPorts' to make a list of ports inside the container available locally.
    "forwardPorts": [
        3000,
        5432,
    ],
    "portsAttributes": {
        "3000": {
            "label": "web",
            "onAutoForward": "notify",
            "requireLocalPort": true
        },
        "5432": {
            "label": "postgres"
        }      
    }

With these final improvements, our DevContainer is now ready for use. The final version up and running:

And with that, we’ve achieved the goal: using containers to reduce friction in setting up your development environment. The final source code of the project is available at https://github.com/duduribeiro/devcontainer-rails-demo/. You can clone it and set up the environment in just a few minutes to try it out for yourself.

Bonus:

Now that you have your development environment setup with DevContainers, you can also easily use services like GitHub Codespaces

Even this blog post was written inside a DevContainer. I needed to install Ruby 2.6.8 and some dependencies failed to intall on my machine. So I decided to run this on a DevContainer to have a easy way to run the project every time I want to write a new post.

References:

Thanks ☕️

Creating a Docker image with a preloaded database

Cadu Ribeiro — Fri, 29 Jan 2021 15:00:00 +0000

Imagine that we have the following Postgresql database dump:

--
-- PostgreSQL database dump
--

-- Dumped from database version 11.5
-- Dumped by pg_dump version 11.3

SET statement_timeout = 0;
SET lock_timeout = 0;
SET idle_in_transaction_session_timeout = 0;
SET client_encoding = 'UTF8';
SET standard_conforming_strings = on;
SELECT pg_catalog.set_config('search_path', '', false);
SET check_function_bodies = false;
SET xmloption = content;
SET client_min_messages = warning;
SET row_security = off;

--
-- Name: my_db; Type: DATABASE; Schema: -; Owner: postgres
--

CREATE DATABASE my_db WITH TEMPLATE = template0 ENCODING = 'UTF8' LC_COLLATE = 'en_US.utf8' LC_CTYPE = 'en_US.utf8';

ALTER DATABASE my_db OWNER TO postgres;

\connect my_db

SET statement_timeout = 0;
SET lock_timeout = 0;
SET idle_in_transaction_session_timeout = 0;
SET client_encoding = 'UTF8';
SET standard_conforming_strings = on;
SELECT pg_catalog.set_config('search_path', '', false);
SET check_function_bodies = false;
SET xmloption = content;
SET client_min_messages = warning;
SET row_security = off;

SET default_tablespace = '';

SET default_with_oids = false;

--
-- Name: clients; Type: TABLE; Schema: public; Owner: postgres
--

CREATE TABLE public.clients (
    id integer NOT NULL,
    name character varying(150) NOT NULL
);

ALTER TABLE public.clients OWNER TO postgres;

--
-- Name: clients_id_seq; Type: SEQUENCE; Schema: public; Owner: postgres
--

CREATE SEQUENCE public.clients_id_seq
    AS integer
    START WITH 1
    INCREMENT BY 1
    NO MINVALUE
    NO MAXVALUE
    CACHE 1;

ALTER TABLE public.clients_id_seq OWNER TO postgres;

--
-- Name: clients_id_seq; Type: SEQUENCE OWNED BY; Schema: public; Owner: postgres
--

ALTER SEQUENCE public.clients_id_seq OWNED BY public.clients.id;

--
-- Name: clients id; Type: DEFAULT; Schema: public; Owner: postgres
--

ALTER TABLE ONLY public.clients ALTER COLUMN id SET DEFAULT nextval('public.clients_id_seq'::regclass);

--
-- Data for Name: clients; Type: TABLE DATA; Schema: public; Owner: postgres
--

COPY public.clients (id, name) FROM stdin;
1   Client 1
2   Client 2
\.

--
-- Name: clients_id_seq; Type: SEQUENCE SET; Schema: public; Owner: postgres
--

SELECT pg_catalog.setval('public.clients_id_seq', 2, true);

--
-- Name: clients clients_pkey; Type: CONSTRAINT; Schema: public; Owner: postgres
--

ALTER TABLE ONLY public.clients
    ADD CONSTRAINT clients_pkey PRIMARY KEY (id);

--
-- PostgreSQL database dump complete
--

It is a simple database with a Clients table and 2 records.

If we want to start a Postgresql Docker container with this dump loaded to share with our team, we can add this SQL file into the /docker-entrypoint-initdb.d/ folder inside the container, like explained into the Postgresql Image docs from DockerHub.

Initialization scripts If you would like to do additional initialization in an image derived from this one, add one or more *.sql, *.sql.gz, or *.sh scripts under /docker-entrypoint-initdb.d (creating the directory if necessary). After the entrypoint calls initdb to create the default postgres user and database, it will run any *.sql files, run any executable *.sh scripts, and source any non-executable *.sh scripts found in that directory to do further initialization before starting the service.

The following Dockerfile uses postgres:11-alpine as base image and copies test_dump.sql file to the entrypoint folder.

FROM postgres:11-alpine

COPY test_dump.sql /docker-entrypoint-initdb.d/

If we build this image

$ docker image build . -t preloaded_db:latest

and start a container with the generated image

$ docker container run -d --rm -p 5432:5432 -e POSTGRES_PASSWORD=postgres -e POSTGRES_USER=postgres --name test_preloaded_db preloaded_db:latest

we can see in our database that the database was created. (password is postgres)

$ psql -h localhost -U postgres
postgres=# \c my_db
psql (11.3, server 11.5)
You are now connected to database “my_db” as user “postgres”.
my_db=# SELECT * FROM clients;
 id | name
 — — + — — — — —
 1 | Client 1
 2 | Client 2
(2 rows)

Awesome. Now we have a docker image that has our database loaded. But if we check the log of this container

$ docker container logs test_preloaded_db

we can see CREATE DATABASE and CREATE TABLE commands.

/usr/local/bin/docker-entrypoint.sh: running /docker-entrypoint-initdb.d/test_dump.sql
SET
SET
SET
SET
SET
 set_config
------------

(1 row)

SET
SET
SET
SET
CREATE DATABASE
ALTER DATABASE
...

This tell us that the dump is being processed every time we create the container. If we destroy this container and create a new one, the dump will be processed again. This works fine but if we have a big database with a big dump file, the startup process of the container will be slow because it can take some time to process the whole dump. We can fix it by keeping the database preloaded in the image.

Before we moving on, let’s destroy the container we created

$ docker container rm -f test_preloaded_db

Preloading the database in the image

To preload the database in the image, we need to tell our Dockerfile to execute the same entrypoint of the original PostgreSQL image so it can execute the dump in the build step. Let’s use Multi-Stage build to divide our build in two steps. The first one will execute the entrypoint with the dump file and the second one will copy the data folder to the resulting image.

# dump build stage
FROM postgres:11-alpine as dumper

COPY test_dump.sql /docker-entrypoint-initdb.d/

RUN ["sed", "-i", "s/exec \"$@\"/echo \"skipping...\"/", "/usr/local/bin/docker-entrypoint.sh"]

ENV POSTGRES_USER=postgres
ENV POSTGRES_PASSWORD=postgres
ENV PGDATA=/data

RUN ["/usr/local/bin/docker-entrypoint.sh", "postgres"]

# final build stage
FROM postgres:11-alpine

COPY --from=dumper /data $PGDATA

In the first step, we have the following instructions:

FROM postgres:11-alpine as dumper We define the base image our step will use. postgres with the 11-alpine tag in this case.
COPY test_dump.sql /docker-entrypoint-initdb.d/ Copy the test_dump.sql file to the /docker-entrypoint-initdb.d/ folder.
RUN [“sed”, “-i”, “s/exec "$@"/echo "skipping…"/”, “/usr/local/bin/docker-entrypoint.sh”] We need to execute this sed command in order to remove the exec "$@" content that exists in the docker-entrypoint.sh file so it will not start the PostgreSQL daemon (we don’t need it on this step).
ENV POSTGRES_USER=postgres; ENV POSTGRES_PASSWORD=postgres; ENV PGDATA=/data Sets environment variables to define user and password and tell PostgreSQL to use /data as data folder, so we can copy it in the next step
RUN [“/usr/local/bin/docker-entrypoint.sh”, “postgres”] Execute the entrypoint itself. It will execute the dump and load the data into /data folder. Since we executed the sed command to remove the $@ content it will not run the PostgreSQL daemon

The second step contains only this instruction:

COPY — from=dumper /data $PGDATA This will copy all files from /data folder from the dumper step into the $PGDATA from this current step, making our data preloaded when we start the container (without needing to run the dump every time we create a new container).

If we build this Dockerfile

$ docker image build . -t preloaded_db:new

We can see in the output the dump being processed and after everything is finished, the image is built.

and we can start the container with this new image

$ docker container run -d --rm -p 5432:5432 -e POSTGRES_PASSWORD=postgres -e POSTGRES_USER=postgres --name test_preloaded_db preloaded_db:latest

and our database is loaded

$ psql -h localhost -U postgres
psql (11.3, server 11.5)
Type “help” for help.
postgres=# \c my_db
psql (11.3, server 11.5)
You are now connected to database “my_db” as user “postgres”.
my_db=# SELECT * FROM clients;
 id | name
 — — + — — — — —
 1 | Client 1
 2 | Client 2
(2 rows)

But if we check the logs now, the dump is not being processed every time we create the container

$ docker container logs test_preloaded_db
2019–09–16 01:42:22.458 UTC [1] LOG: listening on IPv4 address “0.0.0.0”, port 5432
2019–09–16 01:42:22.458 UTC [1] LOG: listening on IPv6 address “::”, port 5432
2019–09–16 01:42:22.460 UTC [1] LOG: listening on Unix socket “/var/run/postgresql/.s.PGSQL.5432”
2019–09–16 01:42:22.470 UTC [18] LOG: database system was shut down at 2019–09–16 01:41:02 UTC
2019–09–16 01:42:22.473 UTC [1] LOG: database system is ready to accept connections

We can see that only the PostgreSQL startup is being done. No dump is being executed because it was executed in the build image step.

Creating a Makefile to make the process easier

I like to create a Makefile to make easier the process of making a database dump and creating an image. This Makefile will contain commands to create the dump the database, create an image and tag it by date allowing me to have daily dumps on my registry to download.

default: all

.PHONY: default all fetch_dump

date := `date '+%Y-%m-%d'`
TARGET_IMAGE ?= my_app

all: check_vars fetch_dump generate_image push_to_registry clean finished

check_vars:
    @test -n "$(DB_ENDPOINT)" || (echo "You need to set DB_ENDPOINT environment variable" >&2 && exit 1)
    @test -n "$(DB_NAME)" || (echo "You need to set DB_NAME environment variable" >&2 && exit 1)
    @test -n "$(DESTINATION_REPOSITORY)" || (echo "You need to set DESTINATION_REPOSITORY environment variable" >&2 && exit 1)

fetch_dump: DB_USER ?= postgres
fetch_dump:
    @echo ""
    @echo "====== Fetching remote dump ======"
    @PGPASSWORD="$(DB_PASSWORD)" pg_dump -h $(DB_ENDPOINT) -d $(DB_NAME) -U $(DB_USER) > dump.sql

generate_image:
generate_image:
    @docker build . -t $(TARGET_IMAGE):latest -t $(DESTINATION_REPOSITORY)/$(TARGET_IMAGE):latest -t $(DESTINATION_REPOSITORY)/$(TARGET_IMAGE):$(date)

push_to_registry:
    @echo ""
    @echo "====== Pushing image to repository ======"
    @docker push $(DESTINATION_REPOSITORY)/$(TARGET_IMAGE)

clean:
    @echo ""
    @echo "====== Cleaning used files ======"
    @rm -f dump.sql

finished:
    @echo ""
    @echo "Finished with success. Pushed image to $(DESTINATION_REPOSITORY)/$(TARGET_IMAGE)"

And I can execute the following command to generate my image with a new dump

$ make DB_ENDPOINT=127.0.0.1 DB_USER=postgres DB_PASSWORD=postgres DB_NAME=my_db TARGET_IMAGE=myapp-data DESTINATION_REPOSITORY=gcr.io/my_project

This command usually is integrated in a Cron job in some server to be executed daily. With this I can have on my image registry dumps from each day.

Another interesting thing to do is to add some SQL script to obfuscate users data. This article can be helpful if you want to achive this

Thanks ☕️

Reduce your Docker images (an example with Ruby)

Cadu Ribeiro — Tue, 26 Feb 2019 15:14:06 +0000

A big problem that we face when deploying Docker into production is the image size. Large images take longer to download, consume much of your cloud network traffic quota, cost more money to be stored on the repository and don’t bring any good value.

In most situations, when we create a Docker image, we add steps and dependencies that sometimes we don’t need in the final image that will run in production.

I will use the following application as an example:

https://github.com/opensanca/opensanca_jobs

This is the Dockerfile that generates our image

FROM ruby:2.5.0-alpine
LABEL maintainer="[contato@opensanca.com.br](mailto:contato@opensanca.com.br)"
ARG rails\_env="development"
ARG build\_without=""
ENV SECRET\_KEY\_BASE=dumb
RUN apk update \
&& apk add \
openssl \
tar \
build-base \
tzdata \
postgresql-dev \
postgresql-client \
nodejs \
&& wget [https://yarnpkg.com/latest.tar.gz](https://yarnpkg.com/latest.tar.gz) \
&& mkdir -p /opt/yarn \
&& tar -xf latest.tar.gz -C /opt/yarn --strip 1 \
&& mkdir -p /var/app
ENV PATH="$PATH:/opt/yarn/bin" BUNDLE\_PATH="/gems" BUNDLE\_JOBS=2 RAILS\_ENV=${rails\_env} BUNDLE\_WITHOUT=${bundle\_without}
COPY . /var/app
WORKDIR /var/app
RUN bundle install && yarn && bundle exec rake assets:precompile
CMD rails s -b 0.0.0.0

And the command used to build it:

docker build -t openjobs:latest --build-arg build\_without="development test" --build-arg rails\_env="production" .

This build generated an image with almost 1GB!!! 😱.

This image has some unnecessary stuff, like node but yarn (we only need them to precompile the assets but not to execute the application itself).

Multi-Stage build

Docker introduced the concept of Multi-Stage build in version 17.05. This build technic allows us to split our Dockerfile into several statements FROM. Each statement can use a different base image and you can copy artifacts from one stage to another, without bringing stuff that you don’t want in the final image. Our final image will only contain the build wrote in the last stage.

Now we have a Dockerfile divided into two stages. Pre-build and Final-Build.

# pre-build stage
FROM ruby:2.5.0-alpine AS pre-builder
ARG rails\_env="development"
ARG build\_without=""
ENV SECRET\_KEY\_BASE=dumb
RUN apk add --update --no-cache \
openssl \
tar \
build-base \
tzdata \
postgresql-dev \
postgresql-client \
nodejs \
&& wget [https://yarnpkg.com/latest.tar.gz](https://yarnpkg.com/latest.tar.gz) \
&& mkdir -p /opt/yarn \
&& tar -xf latest.tar.gz -C /opt/yarn --strip 1 \
&& mkdir -p /var/app
ENV PATH="$PATH:/opt/yarn/bin" BUNDLE\_PATH="/gems" BUNDLE\_JOBS=2 RAILS\_ENV=${rails\_env} BUNDLE\_WITHOUT=${bundle\_without}
COPY . /var/app
WORKDIR /var/app
RUN bundle install && yarn && bundle exec rake assets:precompile

# final build stage
FROM ruby:2.5.0-alpine
LABEL maintainer="[contato@opensanca.com.br](mailto:contato@opensanca.com.br)"
RUN apk add --update --no-cache \
openssl \
tzdata \
postgresql-dev \
postgresql-client
COPY --from=pre-builder /gems/ /gems/
COPY --from=pre-builder /var/app /var/app
ENV RAILS\_LOG\_TO\_STDOUT true
WORKDIR /var/app
EXPOSE 3000
CMD rails s -b 0.0.0.0

In the pre-build stage we install node and yarn, all dependencies and precompile the assets. In the final stage, we use an alpine image (which is very small) with ruby, we install only the necessary dependencies to run the application and we then copy the libraries and assets generated in the build-stage with the following command:

COPY --from=pre-builder /gems/ /gems/
COPY --from=pre-builder /var/app /var/app

Doing the build with this Dockerfile, we have now a 562MB image.

We have already reduced almost half the image size, but can we reduce it further?? 🤔

Yes. We can do some actions to reduce more this image.

Removing unnecessary files

We can delete files that are not necessary from the image, like cache and temporary files used by the installed libraries. We can add a .dockerignore file, telling the build what not to send to the image.

# build stage
FROM ruby:2.5.0-alpine AS pre-builder
ARG rails\_env="development"
ARG build\_without=""
ENV SECRET\_KEY\_BASE=dumb
RUN apk add --update --no-cache \
openssl \
tar \
build-base \
tzdata \
postgresql-dev \
postgresql-client \
nodejs \
&& wget [https://yarnpkg.com/latest.tar.gz](https://yarnpkg.com/latest.tar.gz) \
&& mkdir -p /opt/yarn \
&& tar -xf latest.tar.gz -C /opt/yarn --strip 1 \
&& mkdir -p /var/app
ENV PATH="$PATH:/opt/yarn/bin" BUNDLE\_PATH="/gems" BUNDLE\_JOBS=4 RAILS\_ENV=${rails\_env} BUNDLE\_WITHOUT=${bundle\_without}
COPY . /var/app
WORKDIR /var/app
RUN bundle install && yarn && bundle exec rake assets:precompile \
&& rm -rf /gems/cache/\*.gem \
&& find /gems/gems/ -name "\*.c" -delete \
&& find /gems/gems/ -name "\*.o" -delete

# final stage
FROM ruby:2.5.0-alpine
LABEL maintainer="[contato@opensanca.com.br](mailto:contato@opensanca.com.br)"
RUN apk add --update --no-cache \
openssl \
tzdata \
postgresql-dev \
postgresql-client
COPY --from=pre-builder /gems/ /gems/
COPY --from=pre-builder /var/app /var/app
ENV RAILS\_LOG\_TO\_STDOUT true
WORKDIR /var/app
EXPOSE 3000
CMD rails s -b 0.0.0.0

In this new Dockerfile, we added this part that removes caches and temporary C files used to build the libraries:

&& rm -rf /gems/cache/\*.gem \
&& find /gems/gems/ -name "\*.c" -delete \
&& find /gems/gems/ -name "\*.o" -delete

We also included our .dockerignore to tell the build process the files that we don’t want in the image:

.env\*
.git
.gitignore
.codeclimate.yml
.dockerignore
.gitlab-ci.yml
.hound.yml
.travis.yml
LICENSE.md
README.md
docker-compose.\*
Dockerfile
log/\*
node\_modules/\*
public/assets/\*
storage/\*
public/packs/\*
public/packs-test/\*
tmp/\*

With these two steps, now our image has 272MB.

We can reduce it even more. For production, we don’t need test folders, npm raw folder (they are already included on the asset pipeline), no precompiled assets and caches.

To remove this files, we can include a strategy of passing an argument to build (we will call it: to_remove)

...
ARG to\_remove
...
RUN bundle install && yarn && bundle exec rake assets:precompile \
&& rm -rf /usr/local/bundle/cache/\*.gem \
 && find /usr/local/bundle/gems/ -name "\*.c" -delete \
 && find /usr/local/bundle/gems/ -name "\*.o" -delete \
 && rm -rf $to\_remove # Here we remove all files that we passed as an argument to the build.
...

In this argument, we will pass all the files that we don’t want in production:

docker build -t openjobs:reduced --build-arg build\_without="development test" --build-arg rails\_env="production" . --build-arg to\_remove="spec node\_modules app/assets vendor/assets lib/assets tmp/cache"

Notice the — build-arg to_remove=”spec node_modules app/assets vendor/assets lib/assets tmp/cache”. These are the folders that we want to remove from our build process. We don’t need them to run in production.

Removing these files, now we have an image with 164MB, almost 6 times smaller than the original one.

If you still don’t believe me and want to see it, this is the PR that generates this reduction: https://github.com/opensanca/opensanca_jobs/pull/164

Cheers 🍻

An Introduction into Spacemacs from a Vim user

Cadu Ribeiro — Fri, 23 Nov 2018 12:10:41 +0000

I’m a Vim user. I use it in my workflow all the time, but I enjoy trying other editors sometimes and I will share my experience playing with Spacemacs.

What is Spacemacs?

Spacemacs is a starter kit for Emacs that makes it easier for anyone to use it. It has the Evil plugin by default. Evil is a layer for Emacs that emulates Vim. I always try vim mode in all other editors and they all are not so good, Evil is one of the best Vim emulators that I’ve already seen. It makes it easier for me to use Emacs, since it emulates Vim commands that I already know.

Spacemacs has easy to remember keybindings. Everything starts with the SPC key (it is like the leader key from Vim) and all belongs to some group. For example: SPC b for all buffers commands and SPC p for all project commands. If you don’t remember the command, press SPC SPC and it brings an autocomplete buffer to search for commands.

How do I experiment another editor?

It’s hard for me to learn all the stuffs from a tool in a day or two. So when I want to learn more about an editor, I force myself to experiment it for at least one week, trying to do everything on the new editor without opening others. To use an editor I need to learn how to search files, how to navigate through files and how split the window, only this.

Installation

First, you should install Emacs.

You can use Homebrew to install emacs on mac:

brew install emacs --use-git-head --cocoa --srgb

To install on Linux, you can follow this documentation.

Now you can install the Spacemacs distribution:

git clone https://github.com/syl20bnr/spacemacs ~/.emacs.d

Now open Emacs..

Spacemacs Startup Screen

The Basics

Spacemacs has the concept oflayers . The idea is to have a collection of packages with some configurations. The documentation says:

Layer: A collected unit of configuration that can be enabled (or disabled) in Spacemacs. A layer typically brings together one or more packages, as well as the glue configuration code required to make them play well with each other and Spacemacs in general.

Once you open Emacs, enter SPC f e d, this will take you to Spacemacs configuration file (~/.spacemacs). Here you can add layers inside dotspacemacs-configuration-layers. For example:

dotspacemacs-configuration-layers
 '(
 elixir
 (ruby :variables ruby-version-manager 'rvm)
 ruby-on-rails
 )

This adds layers for working with Elixir, Ruby and Ruby on Rails

Navigating through a file

To navigate on the file, I use the Vim commands.

gg =\> goes to the top of the file
G =\> goes to the bottom
h =\> left
j =\> down
k =\> up
l =\> right

(to learn more about navigation, read here)

Loading your project

After loaded, you can use SPC a d (or File -> Open Directory) to open your project.

It will load this buffer and you can find your project file. Hit Enter to load it

Spacemacs includes Projectile, an Emacs library for handling projects. So, after you added your project the first time with SPC a d , you can use SPC p p to Switch between projects

Terminology

I like to use this image to understand Emacs terminology

https://emacs.stackexchange.com/questions/13583/whats-the-difference-between-a-buffer-a-file-a-window-and-a-frame

Usually in most applications we call the running instance as window, but on Emacs we call it the Frame . A frame can contain multiple Windows and inside each window we have a Buffer loaded (on the image above, the buffer loaded is indicated by the yellow circle). A buffer can be the content of a loaded file or the output of some command execution. Every time you open a file, it changes the active buffer to this file, but the buffer with the past file opened is still alive. You can use SPC b b to see all buffers and switch them

You can easily switch between the last used buffer with SPC TAB

Searching for a file

The first thing that I learn in a new editor is: How can I search for a file?

In Spacemacs we can do a fuzzy file search pressing SPC p f

Fuzzy file search

Another way is to search for Classes and methods using Tags. SPC p G will generate a TAGS file for your project. After generated, you can search through tags with SPC p g

Searching by Class/Method Name using Tags

You can also open a tree explorer using SPC p t

SPC p t will open the tree explorer

Splitting window

Another feature that I use a lot is splitting the window. You can use the Emacs keybinding to do it:

SPC w - =\> Split window into two windows, one above the other.
SPC w / =\> Split window into two windows, side by side.

Or using the vim commands:

:sp =\> Split window into two windows, one above the other.
:vsp =\> Split window into two windows, side by side.

Splitting the window

After splitting, you will notice that a number is defined in the window

To jump into a window, you can press SPC NUMBER to go to that window (for example, SPC 3 take me to the right window on the image)

Switching between windows

You can kill a window with SPC w d or :q .

Sometimes when working on more than one window in the same frame (for example, having a class and a test file opened together), I like to maximize one of them. You can use SPC w m to maximize and use it again to return it.

My Workflow working with it

Usually, my workflow with Vim involves Tmux too. I typically have 3 Tmux panes open. One for the editor (Vim), another for the terminal (to run tests, ssh, etc), and one for the server (and the logs).

My workflow with Vim and tmux

I also like to use vim-test to run my tests inside Vim

Emacs for another side, have the idea that you can do everything on the Emacs itself. You can start your server on it, open a terminal, etc, without having the needing to have multiple terminals for each thing (although I still have they open when working on Emacs 😄)

My next examples use the rails layer on Spacemacs.

I can run bundle install (to install my ruby dependencies) inside it with SPC m b i

I can run the rails server inside it and check the logs:

and also run the tests directly from Emacs:

Follow a simple mindmap that I created for these basic commands:

And this is all the basics that I need to use it and that I want to share.

Cheers 🍻

Easy deploy your Docker applications to AWS using ECS and Fargate

Cadu Ribeiro — Wed, 31 Jan 2018 17:03:58 +0000

In this post, I will try to demonstrate how you can deploy your Docker application into AWS using ECS and Fargate.

As an example, I will deploy this app to ECS. The source can be found here.

I will use Terraform to spin the infrastructure so I can easily track everything that I create as a code. If you want to learn the basics of Terraform, please read my post about it.

ECS

What is ECS?

The Elastic Container Service (ECS) is an AWS Service that handles the Docker containers orchestration in your EC2 cluster. It is an alternative for Kubernetes, Docker Swarm, and others.

ECS Terminology

To start understanding what ECS is, we need to understand its terms and definitions that differs from the Docker world.

Cluster: It is a group of EC2 instances hosting containers.
Task definition: It is the specification of how ECS should run your app. Here you define which image to use, port mapping, memory, environments variables, etc.
Service: Services launches and maintains tasks running inside the cluster. A Service will auto-recover any stopped tasks keeping the number of tasks running as you specified.

Fargate

Fargate is a technology that allows running containers in ECS without needing to manage the EC2 servers for cluster. You only deploy your Docker applications and set the scaling rules for it. Fargate is an execution method from ECS.

Show me the code

The full example is on Github.

The project structure

Our Terraform project is composed of the following structure:

├── modules

│ └── code_pipeline

│ └── ecs

│ └── networking

│ └── rds

├── pipeline.tf

├── production.tf

├── production_key.pub

├── terraform.tfvars

└── variables.tf

Modules are where we will store the code that handles the creation of a group of resources. It can be reused by all environments (Production, Staging, QA, etc.) without needing to duplicate a lot of code.
production.tf is the file that defines the environment itself. It calls the modules passing variables to it.
pipeline.tf Since the pipeline can be a global resource without needing to isolate per environment. This file will handle the creation of this pipeline using the code_pipeline module.

First part, the networking

The branch with this part can be found here.

The first thing that we need to create is the VPC with 2 subnets (1 public and 1 private) in each Availability Zone. Each Availability Zone is a geographically isolated region. Keeping our resources in more than one zone is the first thing to achieve high availability. If one physical zone fails for some reason, your application can answer from the others.

Our networking

Keeping the cluster on the private subnet protects your infrastructure from external access. The private subnet is allowed only to be accessed from resources inside the public network (In our case, will be the Load Balancer only).

This is the code to create this structure (it is practically the same from my introduction post of Terraform):

https://medium.com/media/3000fa3a6cfa0ccdb9678d3e4660424d/href

The above code creates the VPC, 4 subnets (2 public and 2 private) in each Availability zone. It also creates a NAT to allow the private network access the internet.

The Database

The branch with this part can be found here.

We will create a RDS database. It will be located on the private subnet. Allowing only the public subnet to access it.

https://medium.com/media/59d7deafef35b800131d6d8061ad1e82/href

With this code, we create the RDS resource with values received from the variables. We also create the security group that should be used by resources that want to connect to the database (in our case, the ECS cluster).

Ok. Now we have the database. Let’s finally create our ECS to deploy our app \o/.

Take Three: The ECS

The branch with this part can be found here.

We are approaching the final steps. Now, it is the part that we define the ECS resources needed for our app.

The ECR repository

The first thing is to create the repository to store our built images.

https://medium.com/media/356acc348a4487a6434c6e60919f7c62/href

The ECS cluster

Next, we need our ECS cluster. Even using Fargate (that doesn’t need any EC2), we need to define a cluster for the application.

https://medium.com/media/007965a6291dc6a49086c14471cdd92f/href

The tasks definitions

Now, we will define 2 task definitions.

Web: Contains the definition of the web app itself.
Db Migrate: This task will only run the command to migrate our database and will die. Since it is a single run task, we don’t need a service for it. https://medium.com/media/cbba359b380baf0de6398ebbf29d66c0/href

The tasks definitions are configured in a JSON file and rendered as a template in Terraform.

This is the task definition of the web app:

https://medium.com/media/cd914e0e8af12d8d338da2acd81e85d2/href

In the file above, we are defining the task to ECS. We pass the created ECR image repository as variable to it. We also configure other variables so ECS can start our Rails app.

The definition of the DB migration task is almost the same. We only change the command that will be executed.

The load balancers

Before creating the Services, we need to create the load balancers. They will be on the public subnet and will forward the requests to the ECS service.

https://medium.com/media/3033baa23cf2903207443201d7762619/href

In the file above we define that our target group will use HTTP on port 80. We also create a security group to allow access into the port 80 from the internet. After, we create the Application Load Balancer and the listener. To use Fargate, you should use an Application Load Balancer instead an Elastic Load Balancer.

Finally, the ECS service

Now we will create the service. To use Fargate, we need to specify the lauch_type as Fargate.

https://medium.com/media/9a7598bcb9d94c58ca13ca1be88a7d69/href

Auto-scaling

Fargate allows us to auto-scale our app easily. We only need to create the metrics in CloudWatch and trigger to scale it up or down.

https://medium.com/media/0163c1beacb8ade34e9f09209dd80916/href

We create 2 auto scaling policies. One to scale up and other to scale down the desired count of running tasks from our ECS service.

After, we create a CloudWatch metric based on the CPU. If the CPU usage is greater than 85% from 2 periods, we trigger the alarm_action that calls the scale-up policy. If it returns to the Ok state, it will trigger the scale-down policy.

The Pipeline to deploy our app

Our infrastructure to run our Docker app is ready. But it is still boring to deploy it to ECS. We need to manually push our image to the repository and update the task definition with the new image and update the new task definition. We can run it through Terraform, but it could be better if we have a way to push our code to Github in the master branch and it deploys automatically for us.

Entering, CodePipeline and CodeBuild.

CodePipeline is a Continuous Integration and Continuous Delivery service hosted by AWS.

CodeBuild is a managed build service that can execute tests and generate packages for us (in our case, a Docker image).

With it, we can create pipelines to delivery our code to ECS. The flow will be:

You push the code to master’s branch
CodePipeline gets the code in the Source stage and calls the Build stage (CodeBuild).
Build stage process our Dockerfile building and pushing the Image to ECR and triggers the Deploy stage
Deploy stage updates our ECS with the new image

Let’s define our Pipeline with Terraform:

https://medium.com/media/68275343cb24ec50baf32ce6cc1002e3/href

In the above code, we create a CodeBuild project, using the following buildspec (build specifications file):

https://medium.com/media/40583c74d4308338360de35bb99a1921/href

We defined some phases in the above file.

pre_build: Upgrade aws-cli, set some environment variables: REPOSITORY_URL with the ECR repository and IMAGE_TAG with the CodeBuild source version. The ECR repository is passed as a variable by Terraform.
build: Build the Dockerfile from the repository tagging it as LATEST in the repository URL.
post_build: Push the image to the repository. Creates a file named imagedefinitions.json with the following content: ‘[{“name”:”web”,”imageUri”:REPOSITORY_URL”}]’ This file is used by CodePipeline to upgrade your ECS cluster in the Deployment stage.
artifacts: Get the file created in the last phase and uses as the artifact.

After, we create a CodePipeline resource with 3 stages:

Source: Gets the repository from Github (change it by your repository information) and pass it to the next stage.
Build: Calls the CodeBuild project that we created in the step before.
Production: Gets the artifact from Build stage (imagedefinitions.json) and deploy to ECS.

Let’s see they working together?

Running all together

The code with the full example is here.

Clone it. Also, since we use Github as the CodePipeline source provider, you need to generate a token to access the repositories. Read here to generate yours.

After generating your token, export it as an environment variable.

$ export GITHUB\_TOKEN=YOUR\_TOKEN

Now, we need to import the modules and the provider library.

$ terraform init

Now, let the magic begin!

$ terraform apply

it will display that Terraform will create some resources, and if you want to continue

Type yes.

Terraform will start create our infraestructure.

Seriously, get a coffee until it finishes.

AWESOME!. Our infrastructure is ready!!. If you enter in your CodePipeline at AWS Dashboard, you can see that it also triggered the first build:

Wait until all the Stages are green.

Get your Load Balancer DNS and check the deployed application:

$ terraform output alb\_dns\_name

It is working \o/

Finally, the app is running. Almost magic!

If you have any questions about it, contact me. This was just an introduction post about ECS with Fargate using Terraform.

Cheers 🍻

Evaluate your ruby code directly from VIM

Cadu Ribeiro — Mon, 07 Aug 2017 21:41:45 +0000

When I am writing code, usually I want to evaluate some piece of code. I used to do the following actions:

Copy and paste my code to IRB (or run my ruby script file directly from the terminal).
When using tmux, send my code directly from vim to tmux with the vim-tmux-runner plugin.

The first option needs an extra work of copying and pasting content. I prefer the second option, but sometimes I forgot to attach the VTR pane and get errors.

Now I’m using https://github.com/JoshCheek/seeing_is_believing along with https://github.com/t9md/vim-ruby-xmpfilter plugin

I set my .vimrc with the following content (I use Plug to manage my dependencies):

Plug 't9md/vim-ruby-xmpfilter'

" Enable seeing-is-believing mappings only for Ruby
let g:xmpfilter\_cmd = "seeing\_is\_believing"

autocmd FileType ruby nmap \<buffer\> \<F4\> \<Plug\>(seeing\_is\_believing-mark)
autocmd FileType ruby xmap \<buffer\> \<F4\> \<Plug\>(seeing\_is\_believing-mark)
autocmd FileType ruby imap \<buffer\> \<F4\> \<Plug\>(seeing\_is\_believing-mark)

autocmd FileType ruby nmap \<buffer\> \<F6\> \<Plug\>(seeing\_is\_believing-clean)
autocmd FileType ruby xmap \<buffer\> \<F6\> \<Plug\>(seeing\_is\_believing-clean)
autocmd FileType ruby imap \<buffer\> \<F6\> \<Plug\>(seeing\_is\_believing-clean)

autocmd FileType ruby nmap \<buffer\> \<F5\> \<Plug\>(seeing\_is\_believing-run)
autocmd FileType ruby xmap \<buffer\> \<F5\> \<Plug\>(seeing\_is\_believing-run)
autocmd FileType ruby imap \<buffer\> \<F5\> \<Plug\>(seeing\_is\_believing-run)

Now I can visual select my code, use F4 to mark and that line will be evaluated, press F5 and get the result of that code. After, I can clean all marks with F6 .

Cheers,

🍻

Creating review apps per pull requests

Cadu Ribeiro — Tue, 04 Jul 2017 11:44:00 +0000

In this post I will show a simple example about how to create apps for each pull request (or, creating your own Gitlab/Heroku Review App).

Let’s imagine the following scenario of a development team:

Developer creates a new feature branch.
Developer pushes the branch
Developer opens a pull request so other developers can check his code and test the feature
CI runs the tests and makes the branch green if it passes

Now the developer should send the feature to someone else to test it. The feature should not be merged into master so it can be tested. They should be tested isolated because one feature can interfere with another and master should only have deployable code. With this in mind, we will create an environment for each pull request using Docker.

I will use Jenkins as CI because I want to use an open source tool to demonstrate this, but you can use any CI tool that you prefer.

This post will assume that you already have Jenkins installed. I will use this opensource Rails application as an example. Fork this repo into your account and clone it in your computer.

git clone [https://github.com/yourgithubusername/opensanca\_jobs.git](https://github.com/duduribeiro/openjobs_experiment.git)

You can get the full code repo that I used on https://github.com/duduribeiro/openjobs_jenkins_test

Configuring Docker

The first step that we need is to configuring our app with docker.

A Dockerfile is a file that has instructions to build an image that contains our application. You can learn more about Docker in their documentation.

Create a file named Dockerfile in the app folder:

This is our instructions to build an image. The first command FROM specifies to docker that we will use the image ruby:2.4.1 as our base image.

After, the first RUN command installs all dependencies the app needs: yarn, imagemagick and node (to precompile the assets. A fancy solution is to use a different container only with node and sprockets to precompile the assets). The second RUN command, creates a folder /var/app that will be responsible to store the application. The COPY command moves the current folder to the container in the /var/app folder. The next RUN command installs all dependencies from Rails and yarn. The CMD specify the command that the container should execute when it runs the image. You can learn more about Dockerfile here.

With this file we can build our image.

docker build -t myimage .

With this command, we are building our Dockerfile and generating an image with the tag myimage .

Running docker image ls we can check our image.

Now we can start our application:

docker run -d -p 3000:3000 myimage

The -d option tells to docker that this container will run in background and -p 3000:3000 will connect the local port 3000 with the exposed 3000 port from the container.

We can navigate now to http://localhost:3000.

And we receive this error. This is because our app requires a database connection. We need to run a new container with the database and link both containers so they can communicate with each other. Each container should have only one purpose, so, you should not run more than 1 service in a single container (ie: a container with the application and the database).

Instead of manually run each container, we will use docker compose , a tool to help us to run multi containers applications.

Use docker ps and docker kill to destroy your application container/

Get the id of the container

Kill the container

Create a docker-compose.yml file:

In this file, we are creating 2 services. One for the database using the postgres image, and another one with our web application using our Dockerfile image.

In the environment item on web service, I’m setting the DATABASE_URL environment variable. If this environment variable is set, rails will use it replacing the loaded configurations from config/database.yml. Read more about this here. DATABASE_URL follows this pattern for postgres:

postgres://user:password@host/database_name.Since our database user does not have password, we leave empty after the : . We don’t fill the database_name because we want the configured one in config/database.yml (one database name per environment).

Create the databases and run the migrations:

docker-compose run --rm web rake db:create db:migrate

In this command, we will run a container with the web service and execute the command rake db:create db:migrate . The --rm option is to remove the container after the execution.

Run the tests:

docker-compose run --rm -e RAILS\_ENV=test web rake db:drop db:create db:migrate

docker-compose run --rm web rspec

Success

Now we can start the application:

docker-compose up

Access http://localhost:3000 again and now our app is working.

Configuring our pipeline

We will use Jenkins pipeline as code to configure our pipeline. Read more about it here. We need to have docker, jq and Jenkins installed on CI server.

We will use the following Jenkins plugins: Blue Ocean, Blue Ocean Pipeline Editor, GitHub Pipeline for Blue Ocean

Create a file named Jenkinsfile in the project root with the following content:

We have 4 stages on this pipeline:

Build : It will build our Dockerfile and generate an image from this build tagged with openjobs with version named latest . It will use docker-compose to build, install dependencies, create and migrate the database created with compose .
Tests : It will run 2 parallel steps, one to run unit tests, and another to run feature tests.
Deploy to staging : If it is the master branch, it will deploy the app to staging.domain . We will cover this in the next steps.
Create feature environment : If it isn’t the master branch, it will deploy the app to branchname.domain . We will cover this in the next steps.

Let’s create our pipeline on Jenkins. Push your code, go to your Jenkins, and access the blue ocean interface and click in the New Pipeline button:

In the next screen, select Github, choose you account and find the repository. Click in Create Pipeline

And our build is passing 🎉

The red icon on the build is because this step will not run, since the branch built was master .

Now we need to create our environments after the build and dynamic route a domain to a specific container.

Entering, Traefik

Traefik is a tool that will help us make the dynamic routing and act as a load balancer. Example: If I access http://mybranch.mydomain.com , I want to access the container containing the app with mybranch that should be started by Jenkins.

Creating dynamic environments

The following steps should be executed on CI server.

We will use Docker Swarm. It is very helpful to create a docker cluster. I will use only one server to demonstrate, but you are able also to create a cluster.

Initialize swarm cluster:

docker swarm init --advertise-addr=10.10.0.5

This will initialize our server as the swarm master. It will generate a command that you can use to join the cluster in other servers.

Create a network that we can use with traefik and our containers:

docker network create --driver=overlay --attachable traefik-net

Initialize traefik:

docker service create \
--name traefik \
--constraint 'node.role==manager' \
--publish 80:80 \
--publish 8081:8080 \
--mount type=bind,source=/var/run/docker.sock,target=/var/run/docker.sock \
--network traefik-net \
traefik \
--docker \
--docker.swarmmode \
--docker.domain=apps.carlosribeiro.me \
--docker.watch \
--logLevel=DEBUG \
--web

This will initialize traefik. You should treat this initialization on some startup script on your server.

If we access our server in 8081 port, we can access traefik dashboard.

The docker.domain specify that we will access the apps through appname.apps.carlosribeiro.me . To allow this, I created two alias on my DNS server pointing to the CI server’s IP:

apps.carlosribeiro.me
*.apps.carlosribeiro.me

Editing Jenkinsfile to create the environments

Lets add a method on Jenkinsfile that will create a environment when the build is triggered.

This method will accept an argument to inform the name of the environment. This name will be used to prefix the services name. It will stop all docker-compose containers that are still running. After, it will remove if exist, services with the same name (This is to recreate the environment when we push again to the branch). After, we create 3 services. One for Postgres, another for Redis, and the app itself using the imaged that we built on build step. The last command, runs a temporary docker container to create and migrate the database and precompile the assets (you can fetch your previously dump from production too).

Look that in the app service, we specify some environment variables.

REDIS_URL, DATABASE_URL : Endpoint to connect in both services using the previously created services hosts.
RAILS_ENV: We set it to production . It will enforce that our app should behave like a production environment.
RAILS_SERVE_STATIC_FILES : Since we will not have a nginx in front of the app server, we need to set this to tell rails to service static files for us.

— label ‘traefik.port’ will inform traefik what is the container’s exposed port. In our case, is the 3000.

— name inform the service name that traefik will identify as the prefix on domain.

Now, lets call this method on our pipeline’s stages.

If it is master branch, we will deploy into an app called staging . If not, it will call the app with the same name of branch.

Look the final Jenkinsfile:

Push master and let’s wait Jenkins create our staging for us.

Our build passes…

… and our app is deployed on staging:

Now let’s open a pull request to change this button color.

git checkout -b 01-change-button-color

Edit app/assets/stylesheets/application.scss

.btn-register-vacancy {
  background-color: #2769ff;
  width: 300px;
  height: 35px;
  border-radius: 5px;
}

Push this branch, and open a pull request.

Github will tell us that the build is pending.

…

Job is passing

and Github is informed:

And now we have our dynamic environment up and running

I can send this URL to a QA or a PO review it before merging. If something is wrong, it can be validated before it is on master.

That’s all

Cheers 🍻

Creating your cloud servers with Terraform

Cadu Ribeiro — Tue, 04 Apr 2017 14:56:49 +0000

Terraforming Mars

Sometimes when you handle a lot of servers in the cloud, it is pretty easy to get lost on your infrastructure. “Where is that freaking server that I can’t find?”, or even “Why is this instance for?”.

In this post, I will introduce one tool that I found myself liking a lot: To have a bootstrap of an Infrastructure as Code flow in my applications. Terraform. Infrastructure as Code (IaC) allows us to have a repository with code that describes our infrastructure. This way, we can avoid reminding how rebuild the entire infrastructure for an application. It will be on the code and it can be versioned and tested. And if something goes wrong, we can revert it. It is very useful having a continuous integration of the infrastructure. The whole team knows how it was built and all the pieces. You can apply the same flow that you use in you app code to your infrastructure, i.e: Someone makes a change, open a Pull Request, someone reviews it and after it is approved, you merge and your CI tools apply the changes in your environment.

Why Terraform? What is the difference between Chef, Puppet or Ansible?

Chef, Puppet and Ansible are IaC tools too, but they focus on configuring operating system and applications. They are called Configuration Management Tools and they also can build infrastructure on the cloud with a help of plugins, but usually it is hard to configure and sometimes it is limited. With Terraform you can build from the services to the networking part. You can use Terraform to create the infrastructure and a configuration management tool to configure the applications. Terraform can’t replace your configuration management tool, but it’s made to work together with it.

A practical example

In order to follow this article, you’ll need an AWS account.

Let’s create the servers for our web application. Take a look at this diagram of the resources that we’ll build.

We will create 2 subnets (one for public access and another private). We have Elastic Load Balancer in the public subnet to handle the traffic to our web servers. Our web servers will be on the private subnet and it will only be accessible through the Load Balancer. This mean that we won’t have direct access to make connections (for example, SSH) on the server. In order to access via SSH an instance on a private subnet, you’ll need a bastion host and connect to the web server through it. Thus, we will create the bastion host on the public subnet.

Before getting your hands dirty, you need to install Terraform. Follow the instructions of this link to install it in your machine.

Directory structure

Let’s create a folder to handle our infrastructure code.

mkdir ~/terraform

I like to follow this pattern when working with Terraform:

├── modules
│ ├── networking
│ └── web
├── production
└── staging

Let’s create this folder structure

cd ~/terraform
mkdir -p modules/{networking,web} production staging

Our modules folder contains all the shared code to create the pieces of the infrastructure (web servers, app servers, databases, vpc, etc). Each folder inside the modules folder is related to a specific module.

Next, I have folders for my environments (staging, production, qa, development, etc). Each of this folder contains code to use our shared modules and create a different architecture for each environment (This is my personal approach using Terraform, but feel free to work on a different way).

Our first module: Networking

Let’s create our networking module. This will be responsible for creating the networking pieces of our infrastructure, like VPC, subnets, routing table, NAT server and the bastion instance.

Before we get deep in the code, I wanna explain how terraform works:

Terraform will provide us with some commands. Some of them are:

plan : Displays all the changes that Terraform makes on our infrastructure

apply : Executes all the changes to the infrastructure

destroy : Destroys everything that was created with Terraform

When you run Terraform inside a directory, it loads ALL .tf files from the directory and execute them (will not load on subfolders). Terraform will first create a graph of the resources to apply only in the final phase, so you don’t need to specify the resources in any specific order. The graph will determine the relations between the resources and ensure that Terraform creates they in the right order.

Continuing on our networking module

Enter in the networking module

cd modules/networking

Let’s create our first tf file. The one that specifies all variables needed for our module.

touch variables.tf

Insert the following content on the variables.tf file:

These are all variables that our networking module needs in order to create all resources. We need the CIDR for the VPC and the subnets, the AWS region that we will use, the key name and the environment that we are building.

This is the way that you specify a variable in Terraform

variable "variable\_name" {
  description = "The description of the variable"
  default = "A default value if this isn't set
}

Ok. Now we have our variables.tf file to specify the interface of our module. Let’s create the file that will create networking stuffs for our module. Create the main.tf file in the networking folder. (you can specify any name that you want. Remember, all tf files will be loaded).

touch main.tf

Insert the following content on the main.tf file:

Here we are creating all the networking part of our infrastructure based on the diagram that we saw. A VPC, both subnets (public and private), the Internet Gateway to the public subnet, the NAT server for the private subnet, the bastion host and all security group for the VPC, allowing inbound and outbound inside the VPC, and the security group for the bastion host, allowing the SSH on the Port 22. You can check this link from AWS for more details.

This is how we create a resource on Terraform.

resource "resource" "name" {
  attribute = "value"
}

The resource is the name of the resource that we want to build. Each cloud provider has different resources. For example, these resources from AWS. We can also concatenate values. Remember that we created the variables file? We use them here with the var.variable_name . Like in this part of the code, which we use the key_name variable that we specified in the variables file:

resource "aws\_instance" "bastion" {
  ...
  key\_name = "${var.key\_name}"
  ...
}

This is how we created the instance and called its bastion. You can also get property of this resource in other parts of the code. Example:

resource "someresource" "somename" {
  attribute = "${aws\_instance.bastion.id}"
}

We use the same idea of var concatenation. But we specify ${resource_type.resource_name.property} .

Our networking module is almost ready. We need to output some variables after the module build the resources, so we can use it in other parts of the code. Terraform has the output command, allowing us to expose variables.

Create the output.tf file inside the networking folder.

touch output.tf

Insert the following content on the output.tf file:

This is how we output a variable from our module:

output "variable\_name" {
  value = "variable value"
}

This will allow us to get these variables outside the module.

Using our module to build the networking from our environment

Now that our networking module is ready, we can use it to build our networking from our environment (i.e, staging).

This is how our terraform folder looks now:

├── modules
│ ├── networking
│ │ ├── main.tf
│ │ ├── output.tf
│ │ └── variables.tf
│ └── web
├── production
└── staging

Go into staging folder.

cd ~/terraform/staging

First, create the public key for the staging servers

ssh-keygen -t rsa -C "staging\_key" -f ./staging\_key

Let’s create our main file. In this file, we will specify information of the AWS provider.

touch \_main.tf

(We use _ at the beginning of the name because since terraform loads all files alphabetically, we need this to be loaded first, since it will create the keypair)

You can specify things like Access and secret key in some ways:

Specify it directly in the provider (not recommended)

provider "aws" {
  region = "us-west-1"
  access\_key = "myaccesskey"
  secret\_key = "mysecretkey"
}

Using the AWS_ACCESS_KEY and AWS_SECRET_KEY environment variables

$ export AWS\_ACCESS\_KEY\_ID="myaccesskey"
$ export AWS\_SECRET\_ACCESS\_KEY="mysecretkey"

terraform plan

The second option is recommended because you don’t need to expose your secrets on the file. Bonus point : If you have the AWS cli you don’t need to export these variables. Only run the aws configure command and terraform will use the variables that you set on it.

Insert the following content on the _main.tf file:

We will only specify the region to the provider. Both access and secret key, we will rely on the AWS cli .

Now create the networking.tf file. It will use our module to create the resources.

touch networking.tf

Insert the following content on the networking.tf file:

This is how we use modules on Terraform.

module "name" {
  source = "location\_path"
  attribute = "value"
}

The module attributes are all variables that we specified before in the variables.tf file from our networking module. Look that we are passing more variables to our module attributes. We need our environment to require these variables too. Create the variables.tf file for our staging environment.

touch variables.tf

Add the following content to the variables.tf file:

This file follows the same pattern of the module’s variables file. These are all variables that we need to build our staging networking piece.

Ok… I think that we are ready to go. Your terraform’s folder structure should be like this:

├── modules
│ ├── networking
│ │ ├── main.tf
│ │ ├── output.tf
│ │ └── variables.tf
│ └── web
├── production
└── staging
    ├── \_main.tf
    ├── networking.tf
    └── variables.tf

Run this command on staging folder: (Terraform’s commands should be run on the environments folder).

cd ~/terraform/staging
terraform get
terraform plan

*the terraform get command only syncs all modules.

After executing the terraform plan command, it will ask you a lot of informations (our variables). Answer they:

It will output a lot of things. Resources that will be created. You can analyze it to check if everything is ok. terraform plan will output only the planned change of the infrastructure.

Now, let’s apply these modifications.

terraform apply

But it is asking for all that information again. To avoid this, you can follow 2 ways to automatically inject variable’s value to Terraform

Using a terraform.tfvars file and remember to ignore this file in your VCS.
Specifying variables in Environment variables. TF_VAR_environment=staging terraform apply (this can be useful when you run through some CI tool)

We will follow the first way. So, create a terraform.tfvars file

touch terraform.tfvars

Insert the following content on terraform.tfvars :

Now you can run the apply command without being asked for variable’s value.

terraform apply

You should receive the following message:

Congratulations..

This will generate 2 files on the staging folder: terraform.tfstate and terraform.tfstate.backup

Terraform controls all their resources on this terraform.tfstate file. You should NEVER delete this file. If you do, terraform will think that it needs to create new resources and will lose tracking with the others that it has been already created.

And how can I make my team in sync with the state?

You have 2 ways to keep the team with the remote in sync. You can commit this .tfstate file to your VCS repository, or use Terraform Remote State. If you rely on terraform’s remote state, I really recommend you to use some wrapper tool for Terraform like Terragrunt. It can handle the remote state locking and initializing it for you.

Creating the web servers

This is our folder structure until now.

├── modules
│ ├── networking
│ │ ├── main.tf
│ │ ├── output.tf
│ │ └── variables.tf
│ └── web
├── production
└── staging
    ├── \_main.tf
    ├── networking.tf
    ├── staging\_key
    ├── staging\_key.pub
    ├── terraform.tfstate
    ├── terraform.tfstate.backup
    ├── terraform.tfvars
    └── variables.tf

Go into our web module

cd ~/terrform/modules/web

Following the same flow we used to create the networking module, let’s create our variables file to specify the interface to our module:

touch variables.tf

Insert the following content on variables.tf :

Create the main.tf file to handle the creation of the resources.

touch main.tf

Insert the following content on main.tf :

This file is practically the same from our networking’s main.tf, we are only creating different resources.

Some differences that you can note:

In this example we are using the count attribute. It specifies to Terraform, to create N times this resource. If we pass 5 on the value, it will create 5 instances.

resource "aws\_instance" "web" {
  count = "${var.web\_instance\_count}"
  ...
}

and you can create dynamic names with the counting number using the count.index property, for example:

tags {
  Name = "web-server-${count.index + 1}"
}

We are passing a file to the user_data property. In order to execute some code on the instance initialization, we need to pass the user_data attribute to our instance and we are specifying a file.

user\_data = "${file("${path.module}/files/user\_data.sh")}"

This will load the content of the user_data.sh file and pass to the attribute.

But we haven’t created this file yet, let’s do it. On the web module folder, create the files folder and the user_data.sh file.

mkdir files
touch files/user\_data.sh

Insert the following content on the files/user_data.sh :

This will install the nginx when the instance is created.

Now, let’s create the output.tf file to get the load balancer’s DNS after the execution.

touch output.tf

Insert the following content on the output.tf :

Our web module is done.

This is how our terraform structure is now:

├── modules
│ ├── networking
│ │ ├── main.tf
│ │ ├── output.tf
│ │ └── variables.tf
│ └── web
│ ├── files
│ │ └── user\_data.sh
│ ├── main.tf
│ ├── output.tf
│ └── variables.tf
├── production
└── staging
    ├── \_main.tf
    ├── networking.tf
    ├── staging\_key
    ├── staging\_key.pub
    ├── terraform.tfstate
    ├── terraform.tfstate.backup
    ├── terraform.tfvars
    └── variables.tf

Using our web module

Let’s back to our staging folder

cd ~/terraform
cd staging

Now, we will use our recent created web module. Create a web.tf file

touch web.tf

Insert the following content on the web.tf :

This is pretty much the same way we used in our networking module. We are using almost the same variables that we already specified (except for web_instance_count . And some variables, we pass the output from our networking module.

"${module.networking.public\_subnet\_id}"

This way we get the public_subnet_id output created on the networking module.

Let’s add the web_instance_count variable to our variables.tf file and terraform.tfvars . This variable represents the number of web instances that we will be created.

Your variables.tf from staging folder should be like this:

And your terraform.tfvars should be like this:

Let’s create an output.tf for our staging environment. With this, we can get the ELB hostname from our web module.

touch output.tf

Add the following content to output.tf :

This is the final directory structure that we have:

├── modules
│ ├── networking
│ │ ├── main.tf
│ │ ├── output.tf
│ │ └── variables.tf
│ └── web
│ ├── files
│ │ └── user\_data.sh
│ ├── main.tf
│ ├── output.tf
│ └── variables.tf
├── production
└── staging
    ├── \_main.tf
    ├── networking.tf
    ├── output.tf
    ├── staging\_key
    ├── staging\_key.pub
    ├── terraform.tfstate
    ├── terraform.tfstate.backup
    ├── terraform.tfvars
    ├── variables.tf
    └── web.tf

You can now run terraform plan to check which resources will be created. (before, run terraform get to update the modules)

terraform get
terraform plan

Done. Terraform will create 5 new resources from our web module.

Let’s apply it.

terraform apply

Yay.. our instances was created. Let’s get our Load Balancer DNS and try to open it on a Browser:

[terraform](http://%24(terraform) output elb\_hostname

This command will return the DNS from our Load Balancer. Open it on a browser:

It’s working!!! With this, you finished the creation of your infrastructure.

Final Notes:

Your web instances don’t have public ips:

In order to SSH then, you need to use the bastion host created on the networking module.

Get the bastion host public IP,

and SSH on it with the -A flag to enable agent forwarding:

chmod 400 staging\_key.pub

ssh-add -K staging\_key

ssh -A ubuntu@52.53.227.241

Now, inside the Bastion host, you can connect into your web server private IP:

ssh ubuntu@10.0.2.113

Now, destroy everything

terraform destroy

You can get the full code of the example here.

That’s all.

Cheers,

🍻

Connecting on RDS Server that is not publicly accessible

Cadu Ribeiro — Tue, 10 Jan 2017 23:23:39 +0000

Let’s imagine the following scenario:

You have web servers on a public subnet that you can connect and your RDS instance is hosted on a private subnet. This way, your database instance is not publicly accessible through the internet and you can’t connect your local client with it.

It’s not possible to do a:

mysql -u user -p -h RDS\_HOST

To establish a connection with the database, you’ll need to use your public EC2 instances to act as a bridge to the RDS. Let’s make a SSH Tunnel.

ssh -i /path/to/keypair.pem -NL 9000:RDS\_ENDPOINT:3306 ec2-user@EC2\_HOST -v

-i /path/to/keypair.pem : The -i option will inform the ssh which key will be used to connect. If you already added your key with ssh-add, this is not necessary. -NL — N will not open a session with the server. It will set up the tunnel. L will set up the port forwarding.
-NL : N will not open a session with the server. It will set up the tunnel. L will set up the port forwarding.
9000:RDS_ENDPOINT:3306 : The -L option will make the port forwarding based on this argument. The first number 9000 is the local port that you want to use to connect with the remote host. RDS_ENDPOINT is the RDS host of your database instance. 3306 is the port of the remote host that you want to access (3306 is the MySQL’s default port).
ec2-user@EC2_HOST : How ssh your public EC2 instance.
-v : Is optional. With this you will print the ssh log on your terminal.

With this you can now connect to your private RDS instance using your local client.

mysql -h 127.0.0.1 -P9000 -u RDS\_USER -p

If your EC2 instance is on a private subnet too, you will need to set up a bastion host to make the bridge possible. Bastion host is an instance that will be placed on a public subnet and will be accessible using SSH. You will use the same SSH tunnel, only changing the host used to point the bastion host.

Cheers 🍻

DEV Community: Cadu Ribeiro

Running Neovim with Devcontainers

The example project

Installing Neovim in the DevContainer

Starting the DevContainer

Copying Neovim configuration to the DevContainer

Using Devcontainers to set up your development environment

What are DevContainers?

Advantages and disadvantages of using DevContainers

Production Containers vs. DevContainers

Practical Example: A Ruby on Rails Application

The Dockerfile for the DevContainer.

The base image used.

And why use mcr.microsoft.com/vscode/devcontainers/base instead of mcr.microsoft.com/vscode/devcontainers/ruby?

Specifying our development container in the devcontainer.json.

Project Dependencies

Running the project

Some improvements to our DevContainer

Creating a Docker image with a preloaded database

Preloading the database in the image

Creating a Makefile to make the process easier

Reduce your Docker images (an example with Ruby)

Multi-Stage build

Removing unnecessary files

An Introduction into Spacemacs from a Vim user

What is Spacemacs?

How do I experiment another editor?

Installation

The Basics

Navigating through a file

Loading your project

Terminology

Searching for a file

Splitting window

My Workflow working with it

Easy deploy your Docker applications to AWS using ECS and Fargate

ECS

ECS Terminology

Fargate

The project structure

First part, the networking

Take Three: The ECS

The ECR repository

The ECS cluster

The tasks definitions

The load balancers

Finally, the ECS service

Auto-scaling

The Pipeline to deploy our app

Running all together

Evaluate your ruby code directly from VIM

Creating review apps per pull requests

Configuring Docker

Configuring our pipeline

Creating dynamic environments

Editing Jenkinsfile to create the environments

Creating your cloud servers with Terraform

Why Terraform? What is the difference between Chef, Puppet or Ansible?

A practical example

Directory structure

Our first module: Networking

Continuing on our networking module

Using our module to build the networking from our environment

Creating the web servers

Using our web module

Final Notes:

Connecting on RDS Server that is not publicly accessible

And why use `mcr.microsoft.com/vscode/devcontainers/base` instead of `mcr.microsoft.com/vscode/devcontainers/ruby`?

Specifying our development container in the `devcontainer.json`.