DEV Community: Rui Trigo

Dockerfile Optimization for Fast Builds and Light Images

Rui Trigo — Fri, 05 Feb 2021 16:59:30 +0000

Docker builds images automatically by reading the instructions from a Dockerfile -- a text file that contains all commands, in order, needed to build a given image.

The explanation above was extracted from Docker’s official docs and summarizes what a Dockerfile is for. Dockerfiles are important to work with because they are our blueprint, our record of layers added to a Docker base image.

We will learn how to take advantage of BuildKit features, a set of enhancements introduced on Docker v18.09. Integrating BuildKit will give us better performance, storage management, and security.

Objectives

decrease build time;
reduce image size;
gain maintainability;
gain reproducibility;
understand multi-stage Dockerfiles;
understand BuildKit features.

Pre-requisites

knowledge of Docker concepts
Docker installed (currently using v19.03)
a Java app (for this post I used a sample Jenkins Maven app)

Let's get to it!

Simple Dockerfile example

Below is an example of an unoptimized Dockerfile containing a Java app. This example was taken from this DockerCon conference talk. We will walk through several optimizations as we go.

FROM debian
COPY . /app
RUN apt-get update
RUN apt-get -y install openjdk-11-jdk ssh emacs
CMD [“java”, “-jar”, “/app/target/my-app-1.0-SNAPSHOT.jar”]

Here, we may ask ourselves: how long does it take to build at this stage? To answer it, let's create this Dockerfile on our local development computer and tell Docker to build the image.

# enter your Java app folder
cd simple-java-maven-app-master
# create a Dockerfile
vim Dockerfile
# write content, save and exit
docker pull debian:latest # pull the source image
time docker build --no-cache -t docker-class . # overwrite previous layers
# notice the build time

0,21s user 0,23s system 0% cpu 1:55,17 total

Here’s our answer: our build takes 1m55s at this point.

But what if we just enable BuildKit with no additional changes? Does it make a difference?

Enabling BuildKit

BuildKit can be enabled with two methods:

Setting the DOCKER_BUILDKIT=1 environment variable when invoking the Docker build command, such as:

time DOCKER_BUILDKIT=1 docker build --no-cache -t docker-class .

Enabling Docker BuildKit by default, setting the daemon configuration in the /etc/docker/daemon.json feature to true, and restarting the daemon:

{ "features": { "buildkit": true } }

BuildKit Initial Impact

DOCKER_BUILDKIT=1 docker build --no-cache -t docker-class .

0,54s user 0,93s system 1% cpu 1:43,00 total

On the same hardware, the build took ~12 seconds less than before. This means the build got ~10,43% faster with almost no effort.

But now let’s look at some extra steps we can take to improve our results even further.

Order from least to most frequently changing

Because order matters for caching, we'll move the COPY command closer to the end of the Dockerfile.

FROM debian
RUN apt-get update
RUN apt-get -y install openjdk-11-jdk ssh emacs
RUN COPY . /app
CMD [“java”, “-jar”, “/app/target/my-app-1.0-SNAPSHOT.jar”]

Avoid "COPY ."

Opt for more specific COPY arguments to limit cache busts. Only copy what’s needed.

FROM debian
RUN apt-get update
RUN apt-get -y install openjdk-11-jdk ssh vim
COPY target/my-app-1.0-SNAPSHOT.jar /app
CMD [“java”, “-jar”, “/app/my-app-1.0-SNAPSHOT.jar”]

Couple apt-get update & install

This prevents using an outdated package cache. Cache them together or do not cache them at all.

FROM debian
RUN apt-get update && \
    apt-get -y install openjdk-11-jdk ssh vim
COPY target/my-app-1.0-SNAPSHOT.jar /app
CMD [“java”, “-jar”, “/app/my-app-1.0-SNAPSHOT.jar”]

Remove unnecessary dependencies

Don’t install debugging and editing tools—you can install them later when you feel you need them.

FROM debian
RUN apt-get update && \
    apt-get -y install --no-install-recommends \
    openjdk-11-jdk
COPY target/my-app-1.0-SNAPSHOT.jar /app
CMD [“java”, “-jar”, “/app/my-app-1.0-SNAPSHOT.jar”]

Remove package manager cache

Your image does not need this cache data. Take the chance to free some space.

FROM debian
RUN apt-get update && \
    apt-get -y install --no-install-recommends \
    openjdk-11-jdk && \
    rm -rf /var/lib/apt/lists/*
COPY target/my-app-1.0-SNAPSHOT.jar /app
CMD [“java”, “-jar”, “/app/my-app-1.0-SNAPSHOT.jar”]

Use official images where possible

There are some good reasons to use official images, such as reducing the time spent on maintenance and reducing the size, as well as having an image that is pre-configured for container use.

FROM openjdk
COPY target/my-app-1.0-SNAPSHOT.jar /app
CMD [“java”, “-jar”, “/app/my-app-1.0-SNAPSHOT.jar”]

Use specific tags

Don’t use latest as it’s a rolling tag. That’s asking for unpredictable problems.

FROM openjdk:8
COPY target/my-app-1.0-SNAPSHOT.jar /app
CMD [“java”, “-jar”, “/app/my-app-1.0-SNAPSHOT.jar”]

Look for minimal flavors

You can reduce the base image size. Pick the lightest one that suits your purpose. Below is a short openjdk images list.

Repository	Tag	Size
openjdk	8	634MB
openjdk	8-jre	443MB
openjdk	8-jre-slim	204MB
openjdk	8-jre-alpine	83MB

Build from a source in a consistent environment

Maybe you do not need the whole JDK. If you intended to use JDK for Maven, you can use a Maven Docker image as a base for your build.

FROM maven:3.6-jdk-8-alpine
WORKDIR /app
COPY pom.xml .
COPY src ./src
RUN mvn -e -B package
CMD [“java”, “-jar”, “/app/my-app-1.0-SNAPSHOT.jar”]

Fetch dependencies in a separate step

A Dockerfile command to fetch dependencies can be cached. Caching this step will speed up our builds.

FROM maven:3.6-jdk-8-alpine
WORKDIR /app
COPY pom.xml .
RUN mvn -e -B dependency:resolve
COPY src ./src
RUN mvn -e -B package
CMD [“java”, “-jar”, “/app/my-app-1.0-SNAPSHOT.jar”]

Multi-stage builds: remove build dependencies

Why use multi-stage builds?

separate the build from the runtime environment
DRY
different details on dev, test, lint specific environments
delinearizing dependencies (concurrency)
having platform-specific stages

FROM maven:3.6-jdk-8-alpine AS builder
WORKDIR /app
COPY pom.xml .
RUN mvn -e -B dependency:resolve
COPY src ./src
RUN mvn -e -B package

FROM openjdk:8-jre-alpine
COPY --from=builder /app/target/my-app-1.0-SNAPSHOT.jar /
CMD [“java”, “-jar”, “/my-app-1.0-SNAPSHOT.jar”]

Checkpoint

If you build our application at this point,

time DOCKER_BUILDKIT=1 docker build --no-cache -t docker-class .

0,41s user 0,54s system 2% cpu 35,656 total

you'll notice our application takes ~35.66 seconds to build. It's a pleasant improvement. From now on, we will focus on the features for more possible scenarios.

Multi-stage builds: different image flavors

The Dockerfile below shows a different stage for a Debian and an Alpine based image.

FROM maven:3.6-jdk-8-alpine AS builder
…
FROM openjdk:8-jre-jessie AS release-jessie
COPY --from=builder /app/target/my-app-1.0-SNAPSHOT.jar /
CMD [“java”, “-jar”, “/my-app-1.0-SNAPSHOT.jar”]

FROM openjdk:8-jre-alpine AS release-alpine
COPY --from=builder /app/target/my-app-1.0-SNAPSHOT.jar /
CMD [“java”, “-jar”, “/my-app-1.0-SNAPSHOT.jar”]

To build a specific image on a stage, we can use the --target argument:

time docker build --no-cache --target release-jessie .

Different image flavors (DRY / global ARG)

ARG flavor=alpine
FROM maven:3.6-jdk-8-alpine AS builder
…
FROM openjdk:8-jre-$flavor AS release
COPY --from=builder /app/target/my-app-1.0-SNAPSHOT.jar /
CMD [“java”, “-jar”, “/my-app-1.0-SNAPSHOT.jar”]

The ARG command can control the image to be built. In the example above, we wrote alpine as the default flavor, but we can pass --build-arg flavor=<flavor> on the docker build command.

time docker build --no-cache --target release --build-arg flavor=jessie .

Concurrency

Concurrency is important when building Docker images as it takes the most advantage of available CPU threads. In a linear Dockerfile, all stages are executed in sequence. With multi-stage builds, we can have smaller dependency stages be ready for the main stage to use them.

BuildKit even brings another performance bonus. If stages are not used later in the build, they are directly skipped instead of processed and discarded when they finish. This means that in the stage graph representation, unneeded stages are not even considered.

Below is an example Dockerfile where a website's assets are built in an assets stage:

FROM maven:3.6-jdk-8-alpine AS builder
…
FROM tiborvass/whalesay AS assets
RUN whalesay “Hello DockerCon!” > out/assets.html

FROM openjdk:8-jre-alpine AS release
COPY --from=builder /app/my-app-1.0-SNAPSHOT.jar /
COPY --from=assets /out /assets
CMD [“java”, “-jar”, “/my-app-1.0-SNAPSHOT.jar”]

And here is another Dockerfile where C and C++ libraries are separately compiled and take part in the builder stage later on.

FROM maven:3.6-jdk-8-alpine AS builder-base
…

FROM gcc:8-alpine AS builder-someClib
…
RUN git clone … ./configure --prefix=/out && make && make install

FROM g++:8-alpine AS builder-some CPPlib
…
RUN git clone … && cmake …

FROM builder-base AS builder
COPY --from=builder-someClib /out /
COPY --from=builder-someCpplib /out /

BuildKit Application Cache

BuildKit has a special feature regarding package managers cache. Here are some examples of cache folders typical locations:

Package manager	Path
apt	/var/lib/apt/lists
go	~/.cache/go-build
go-modules	$GOPATH/pkg/mod
npm	~/.npm
pip	~/.cache/pip

We can compare this Dockerfile with the one presented in the section Build from the source in a consistent environment. This earlier Dockerfile didn't have special cache handling. We can do that with a type of mount called cache: --mount=type=cache.

FROM maven:3.6-jdk-8-alpine AS builder
WORKDIR /app
RUN --mount=target=. --mount=type=cache,target /root/.m2 \
    && mvn package -DoutputDirectory=/

FROM openjdk:8-jre-alpine
COPY --from=builder /app/target/my-app-1.0-SNAPSHOT.jar /
CMD [“java”, “-jar”, “/my-app-1.0-SNAPSHOT.jar”]

BuildKit Secret Volumes

To mix in some security features of BuildKit, let's see how secret type mounts are used and some cases they are meant for. The first scenario shows an example where we need to hide a secrets file, like ~/.aws/credentials.

FROM <baseimage>
RUN …
RUN --mount=type=secret,id=aws,target=/root/.aws/credentials,required \
./fetch-assets-from-s3.sh
RUN ./build-scripts.sh

To build this Dockerfile, pass the --secret argument like this:

docker build --secret id=aws,src=~/.aws/credentials

The second scenario is a method to avoid commands like COPY ./keys/private.pem /root .ssh/private.pem, as we don't want our SSH keys to be stored on the Docker image after they are no longer needed. BuildKit has an ssh mount type to cover that:

FROM alpine
RUN apk add --no-cache openssh-client
RUN mkdir -p -m 0700 ~/.ssh && ssh-keyscan github.com >> ~/.ssh/known_hosts
ARG REPO_REF=19ba7bcd9976ef8a9bd086187df19ba7bcd997f2
RUN --mount=type=ssh,required git clone git@github.com:org/repo /work && cd /work && git checkout -b $REPO_REF

To build this Dockerfile, you need to load your private SSH key into your ssh-agent and add --ssh=default, with default representing the SSH private key location.

eval $(ssh-agent)
ssh-add ~/.ssh/id_rsa # this is the SSH key default location
docker build --ssh=default .

Conclusion

This concludes our demo on using Docker BuildKit to optimize your Dockerfiles and consequentially speed up your images’ build time.

These speed gains result in much-needed savings in time and computational power, which should not be neglected.

Like Charles Duhigg wrote on The Power of Habit: "small victories are the consistent application of a small advantage". You will definitely reap the benefits if you build good practices and habits.

The 6 Aspects You Must Secure On Your MongoDB Instances

Rui Trigo — Mon, 30 Nov 2020 10:57:50 +0000

After going through the adventure of deploying a high-availability MongoDB cluster on Docker and sharing it publicly, I decided to complement that tutorial with some security concerns and tips.

In this post, you'll learn a few details about MongoDB deployment vulnerabilities and security mechanisms. And more importantly, how to actually protect your data with these features.

Objectives

understand database aspects of security.
find ways to implement authentication, authorization, and accounting (AAA).
learn how to enable MongoDB security features.

Pre-requisites

Any running MongoDB instance on which you have full access will do. Standalone or replica set, containerized or not. We will also mention some details on MongoDB Docker instances, but we’ll keep Docker-specific security tips for another post.

List of Quick Wins

Accessing data in a database has several stages. We will take a look at these stages and find ways to harden them, to get a cumulative security effect at the end. Each of these stages will, most of the time, have the ability to block the next one (e.g. you need to have network access to get to the authentication part).

1. Network access

MongoDB’s default port is 27017 (TCP). Choosing a different port to operate might confuse some hackers, but it is still a minor security action because of port scanning, so you won't get that much out of it.

Assuming we choose the default port for our service, we will open that port on the database server's firewall. We do not wish to expose the traffic from this port to the internet. There are two approaches to solve that and both can be used simultaneously. One is limiting your traffic to your trusted servers through firewall configuration.

There’s a MongoDB feature you can use for this: IP Binding. You pass the --bind_ip argument on the MongoDB launch command to enable it. Let's say your app1 server needs to access the MongoDB server for data. To limit traffic for that specific server, you start your server as:

mongod --bind_ip localhost,app1

If you are using Docker, you can avoid this risk by using a Docker network between your database and your client application.

You can add another layer of network security by creating a dedicated network segment for databases, in which you apply an ACL (access list) in the router and/or switch configuration.

2. System access

The second A in AAA means authorization. We know privileged shell access is needed during database installation. When concluding the installation, locking system root user access is part of the drill.

Data analysts need to read database data and applications also need to read and (almost always) write data as well. As this can be addressed with database authentication (more on this on 4. Authorization), make sure to restrict root and other shell access to people who can't do their jobs without it. Only allow it for database and system administrators.

Furthermore, running MongoDB processes with a dedicated operating system user account is a good practice. Ensure that this account has permission to access data but no unnecessary permissions.

3. Authentication

Authentication is the first A in AAA. Authentication-wise, MongoDB supports 4 mechanisms:

SCRAM (default)
x.509 certificate authentication
LDAP proxy authentication
Kerberos authentication

If you are using MongoDB Enterprise Server, then you can benefit from LDAP and Kerberos support. Integrating your company identity and access management tool will make AAA 3rd A (Accounting) implementation easier, as every user will have a dedicated account associated with his records.

MongoDB has its own SCRAM implementations: SCRAM_SHA1 for versions below 4.0 and SCRAM_SHA256 for 4.0 and above. You can think of SHA-256 as the successor of SHA-1, so pick the latter if available on your database version.

Replica sets keyfiles also use the SCRAM authentication mechanism where these keyfiles contain the shared password between the replica set members. Another internal authentication mechanism supported in replica sets is x.509. You can read more on replica sets and how to generate keyfiles in our previous blog post.

To be able to use the x.509 certificates authentication mechanism, there are some requirements regarding certificate attributes. To enable x.509 authentication, add --tlsMode, --tlsCertificateKeyFile and --tlsCAFile (in case the certificate has a certificate authority). To perform remote connections to the database, specify the --bind_ip.

mongod --tlsMode requireTLS --tlsCertificateKeyFile <path to TLS/SSL certificate and key PEM file> --tlsCAFile <path to root CA PEM file> --bind_ip <hostnames>

To generate these certificates, you can use the openssl library on Linux or the equivalent on other operating systems.

openssl x509 -in <pathToClientPEM> -inform PEM -subject -nameopt RFC2253

The command returns the subject string as well as the certificate:

subject= CN=myName,OU=myOrgUnit,O=myOrg,L=myLocality,ST=myState,C=myCountry
-----BEGIN CERTIFICATE-----
# ...
-----END CERTIFICATE-----

Next, add a user on the $external database using the obtained subject string like in the example below:

db.getSiblingDB("$external").runCommand(
  {
    createUser: "CN=myName,OU=myOrgUnit,O=myOrg,L=myLocality,ST=myState,C=myCountry",
    roles: [
         { role: "readWrite", db: "test" },
         { role: "userAdminAnyDatabase", db: "admin" }
    ],
    writeConcern: { w: "majority" , wtimeout: 5000 }
  }
)

Finally, connect to the database with the arguments for TLS, certificates location, CA file location, authentication database, and the authentication mechanism.

mongo --tls --tlsCertificateKeyFile <path to client PEM file> --tlsCAFile <path to root CA PEM file>  --authenticationDatabase '$external' --authenticationMechanism MONGODB-X509

You have now successfully connected to your database using the x.509 authentication mechanism.

4. Authorization

For non-testing environments (like production) it is clearly not recommended to have Access Control disabled, as this grants all privileges to any successful access to the database. To enable authentication, follow the procedure below.

# start MongoDB without access control
mongod
# connect to the instance
mongo

// create the user administrator
use admin
db.createUser(
  {
    user: "myUserAdmin",
    pwd: passwordPrompt(), // or cleartext password
    roles: [ { role: "userAdminAnyDatabase", db: "admin" }, "readWriteAnyDatabase" ]
  }
)
// shutdown mongod instance
db.adminCommand( { shutdown: 1 } )

# start MongoDB with access control
mongo --auth

If you're using MongoDB on Docker, you can create an administrator through MONGO_INITDB_ROOT_USERNAME and MONGO_INITDB_ROOT_PASSWORD environment variables (-e argument). Like so:

docker run -d -e MONGO_INITDB_ROOT_USERNAME=<username> -e MONGO_INITDB_ROOT_PASSWORD=<password> mongo:4.4

Do not neglect human usability convenience. Make sure all passwords are strong, fit your company's password policy, and are stored securely.

MongoDB has a set of built-in roles and allows us to create new ones. Use roles to help when giving privileges while applying the principle of least privilege on user accounts and avoid user account abuse.

5. Encrypted connections

Let's now see how to configure encrypted connections to protect you from sniffing attacks.

If you think about internet browsers, you notice how they keep pressing for users to navigate on sites that support HTTP over TLS, also known as HTTPS. That enforcement exists for a reason: sensitive data protection, both for the client and the server. TLS is therefore protecting this sensitive data during the client-server communication, bidirectionally.

We have explained how to use TLS certificates on 4. Authentication and now we will see how to encrypt our communications between the database server and a client app through TLS configuration on the application’s MongoDB driver.

First, to configure the MongoDB server to require our TLS certificate, add the --tlsMode and --tlsCertificateKeyFile arguments:

mongod --tlsMode requireTLS --tlsCertificateKeyFile <pem>

To test the connection to mongo shell, type in:

mongo --tls --host <hostname.example.com> --tlsCertificateKeyFile <certificate_key_location>

Then, add TLS options to the database connection on your application code. Here is a snippet of a NodeJS application using MongoDB’s official driver package. You can find more of these encryption options on the driver documentation.

const MongoClient = require('mongodb').MongoClient;
const fs = require('fs');

// Read the certificate authority
const ca = [fs.readFileSync(__dirname + "/ssl/ca.pem")];

const client = new MongoClient('mongodb://localhost:27017?ssl=true', {
  sslValidate:true,
  sslCA:ca
});

// Connect validating the returned certificates from the server
client.connect(function(err) {
  client.close();
});

6. Encryption at rest

MongoDB Enterprise Server comes with an Encryption at Rest feature. Through a master and database keys system, this allows us to store our data in an encrypted state by configuring the field as encrypted on rest. You can learn more about the supported standards and enciphering/deciphering keys on the MongoDB documentation.

On the other side, if you will stick with the MongoDB Community, on v4.2 MongoDB started supporting Client-Side Field Level Encryption. Here’s how it works: you generate the necessary keys and load them in your database driver (e.g. NodeJS MongoDB driver). Then, you will be able to encrypt your data before storing it in the database and decrypt it for your application to read it.

Below, you can find a JavaScript code snippet showing data encryption and decryption happening on MongoDB’s NodeJS driver with the help of the npm package mongodb-client-encryption.

const unencryptedClient = new MongoClient(URL, { useUnifiedTopology: true });
  try {
    await unencryptedClient.connect();
    const clientEncryption = new ClientEncryption(unencryptedClient, { kmsProviders, keyVaultNamespace });

    async function encryptMyData(value) {
    const keyId = await clientEncryption.createDataKey('local');
    console.log("keyId", keyId);
    return clientEncryption.encrypt(value, { keyId, algorithm: 'AEAD_AES_256_CBC_HMAC_SHA_512-Deterministic' });
    }

    async function decryptMyValue(value) {
    return clientEncryption.decrypt(value);
    }

    const data2 = await encryptMyData("sensitive_data");
    const mKey = key + 1;
    const collection = unencryptedClient.db("test").collection('coll');
    await collection.insertOne({ name: data2, key: mKey });
    const a = await collection.findOne({ key: mKey });
    console.log("encrypted:", a.name);
    const decrypteddata = await decryptMyValue(a.name);
    console.log("decrypted:", decrypteddata);

  } finally {
    await unencryptedClient.close();
  }

Conclusion

While this post attempts to cover some of the most important quick wins you can achieve to secure your MongoDB instances, there is much more to MongoDB security.

Upgrading database and driver versions frequently, connecting a monitoring tool, and keeping track of database access and configuration are also good ideas to increase security.

Nevertheless, even if the system was theoretically entirely secured, it is always prone to human mistakes. Make sure the people working with you are conscious of the importance of keeping data secured - properly securing a system is always contingent on all users taking security seriously.

Security is everyone's job. Like in tandem kayaks, it only makes sense if everyone is paddling together in the same direction, with all efforts contributing to the same purpose.

Lastly, although this post has focused on database security, it’s also advisable that you protect the JavaScript source code of your web and mobile apps. See our tutorials on protecting React, Angular, Vue, React Native, Ionic, and NativeScript.

Connecting Sequelize To a PostgreSQL Cluster

Rui Trigo — Tue, 08 Sep 2020 11:44:41 +0000

Prologue

In a previous post, I showed how to automate a PostgreSQL fault-tolerant cluster with Vagrant and Ansible.

This kind of setup makes our database cluster resilient to server failure and keeps the data available with no need for human interaction. But what about the apps using this database? Are they fault-tolerant too?

ORMs like Sequelize have read replication features, which allows you to define your primary and standby nodes in the database connection. But what happens if your primary node, which is responsible for write operations, is offline and your app needs to continue saving data on your database?

One way to solve this is by adding an extra layer to the system - a load balancing layer - using PostgreSQL third-party tools like pgbouncer or Pgpool-II or even a properly configured HAproxy instance. Besides the complexity brought by this method, you could also be introducing an undesired single point of failure.

Another way is using a floating IP address/virtual IP address to assign to the current primary database node, so the application knows which node it must connect to when performing write operations even if another node takes up the primary role.

We will be using Digital Ocean for server creation and floating IP assignment, but the strategy also works with other cloud providers who support floating IP.

Objectives

connecting a NodeJS application with Sequelize to a PostgreSQL cluster in order to write to the primary node and read from standby nodes;
create and assign a Digital Ocean Floating IP (aka FLIP) to our current primary database node;
make repmgr interact with Digital Ocean CLI to reassign FLIP to new primary node on promotions;
keep this switchover transparent to the NodeJS application, so the whole system works without human help.

Pre-requisites

a Digital Ocean account and API token (create an account using my referral to get free credits)
a PostgreSQL cluster with repmgr on Digital Ocean (you can grab the Ansible playbook in this tutorial to configure it or just use a cluster with streaming replication and simulate failure + manual promotion);
NodeJS and npm installed (I'm using NodeJS v12 with npm v6);
a PostgreSQL user with password authentication which accepts remote connections from your application host (I'll be using postgres:123456).

Set up your cluster

Create your droplets

Create 3 droplets, preferably with the Ubuntu 20.04 operating system:

pg1 (primary)
pg2 (standby)
pg3 (witness)

To make configurations run smoother, add your public SSH key when creating the droplets. You can also use the key pair I provided on GitHub for testing purposes.

If you'd like to only use 2 droplets, you can ignore the third node as it will be a PostgreSQL witness.

Note: If you use an SSH private key which is shared publicly on the internet, your cluster can get hacked.

Assign a floating IP to your primary node

Create a floating IP address and assign it to your primary node (pg1).

Configure PostgreSQL with repmgr

As previously stated, you can use the Ansible playbook from the last post to speed up the configuration. Download it from GitHub and insert your gateway and droplets IPv4 addresses on group_vars/all.yaml:

client_ip: "<your_gateway_public_ipv4>"
node1_ip: "<droplet_pg1_ipv4>"
node2_ip: "<droplet_pg2_ipv4>"
node3_ip: "<droplet_pg3_ipv4>"
pg_version: "12"

Note: I am assuming you will run your app locally on your computer and it will connect to your droplets through your network gateway

If you don't know your current public gateway address, you can run:

curl ifconfig.io -4

Create an Ansible inventory file and add the playbook host_vars for each host. I named mine digitalocean:

[all]
pg1 ansible_host=<droplet_pg1_ipv4> connection_host="<droplet_pg1_ipv4>" node_id=1 role="primary"
pg2 ansible_host=<droplet_pg2_ipv4> connection_host="<droplet_pg2_ipv4>" node_id=2 role="standby"
pg3 ansible_host=<droplet_pg3_ipv4> connection_host="<droplet_pg3_ipv4>" node_id=3 role="witness"

Add the droplets to the list of SSH known hosts:

ssh root@<droplet_pg1_ipv4> exit
ssh root@<droplet_pg2_ipv4> exit
ssh root@<droplet_pg3_ipv4> exit

Now, run the playbook with:

ansible-playbook playbook.yaml -i digitalocean -e "ansible_ssh_user=root"

-i argument tells Ansible to run on the hosts we specified
-e "ansible_ssh_user=root” passes an environment variable to make Ansible connect as the root user.

NodeJS application

Let's write a simple app that manipulates a countries table. Keep in mind pluralization in Sequelize for JavaScript objects and default database table names. Set it up with:

mkdir sequelize-postgresql-cluster
cd sequelize-postgresql-cluster
npm init -y
npm install pg sequelize

Now, edit the index.js with the following:

const { Sequelize } = require('sequelize');

const primary_ipv4 = '<droplet_pg1_ipv4>'
const standby_ipv4 = '<droplet_pg2_ipv4>'

// new Sequelize(database, username, password)
const sequelize = new Sequelize('postgres', 'postgres', '123456', {
  dialect: 'postgres',
  port: 5432,
  replication: {
    read: [
      { host: standby_ipv4 },
      { host: primary_ipv4 }
      // witness node has no data, only metadata
    ],
    write: { host: primary_ipv4 }
  },
  pool: {
    max: 10,
    idle: 30000
  },
})

// connect to DB
async function connect() {
  console.log('Checking database connection...');
  try {
    await sequelize.authenticate();
    console.log('Connection has been established successfully.');
  } catch (error) {
    console.error('Unable to connect to the database:', error);
    process.exit(1);
  }
}

The code above created a Sequelize connection object named sequelize and configured our servers’ addresses in it. The connect function tests the connection to the database. Make sure your app can connect to it correctly before proceeding.

// model
const Country = sequelize.define('Country', {
  country_id: {
    type: Sequelize.INTEGER, autoIncrement: true, primaryKey: true
  },
  name: Sequelize.STRING,
  is_eu_member: Sequelize.BOOLEAN
},
{
  timestamps: false
});

async function create_table() {
  await sequelize.sync({force: true});
  console.log("create table countries")
};

// insert country
async function insertCountry() {
  const pt = await Country.create({ name: "Portugal", is_eu_member: true });
  console.log("pt created - country_id: ", pt.country_id);
}

// select all countries
async function findAllCountries() {
  const countries = await Country.findAll();
  console.log("All countries:", JSON.stringify(countries, null, 2));
}

async function run() {
  await create_table()
  await insertCountry()
  await findAllCountries()
  await sequelize.close();
}

run()

Country is our Sequelize model, a JavaScript object which represents the database table.
create_table(), insertCountry() and findAllCountries() functions are self-explanatory. They will be called through the run() function.

Run your app with:

node index.js

This will create the countries table on the PostgreSQL database, insert a row in it, and read table data. Because of streaming replication, this data will automatically be replicated into the standby node.

(Optional) Current status primary failure test

If you perform this step, you'll need to revert the PostgreSQL promotion and go back to the cluster’s initial state. There are instructions for this in the mentioned tutorial.

Turn off your pg1 droplet (this can be done through Digital Ocean’s interface). Due to repmgrd configuration, the standby node (pg2) promotes itself to the primary role, so your database cluster keeps working. This promotion will make your app still able to read data, but not write. Proceed by reverting the cluster to the previous status, with pg1 being the primary node.

Use a floating IP

Add the floating IP address to your app database connection object

To take advantage of floating IP, insert it into a variable and edit the write object of the sequelize object.

// insert this line
const floating_ipv4 = 'your_floating_ip_goes_here'
(...)
// change primary_ipv4 to floating_ipv4
write: { host: floating_ipv4 }

Digital Ocean CLI configuration

As we will configure pg2 node to interact with Digital Ocean and reassign the floating IP to its IPv4 address, we must configure doctl in this server. Access pg2 and do as following:

# as superuser
curl -sL https://github.com/digitalocean/doctl/releases/download/v1.46.0/doctl-1.46.0-linux-amd64.tar.gz | tar -xzv
sudo mv ~/doctl /usr/local/bin
# as postgres
doctl auth init
# insert Digital Ocean API token

Note: If using in production, secure the API token variable in Digital Ocean’s CLI configuration script and be careful with reassigning script permissions.

Place the script below on /var/lib/postgresql/promote-standby.sh with execution privileges. It promotes the standby node to primary, validates doctl project configuration and reassigns the floating IP to pg2.

#!/usr/bin/env bash
# assign digital ocean floating ip address to postgres cluster promoted standby node
# this script is expected to run automatically on a standby node during its automated promotion

# promote PostgreSQL standby to primary
repmgr standby promote -f /etc/repmgr.conf

PROJECT_EXISTS=$(doctl projects list | wc -l)

if [ 2 -gt $PROJECT_EXISTS ]; then
  echo "doctl CLI is not properly configured. Exiting."
  exit 1
fi

CURRENT_NODE_ASSIGNED_NAME=$(doctl compute floating-ip list | awk '{print $4}' | tail -n 1) # pg1
STANDBY_NODE_NAME=$(doctl compute droplet list | grep "pg2" | awk '{print $2}') # pg2
STANDBY_NODE_ID=$(doctl compute droplet list | grep "pg2" | awk '{print $1}') # <do droplet resource id>
FLOATING_IP_ADDRESS=$(doctl compute floating-ip list | awk '{print $1}' | tail -n 1) # <do flip ipv4>

echo "$FLOATING_IP_ADDRESS is currently assigned to $CURRENT_NODE_ASSIGNED_NAME. Reassigning to $STANDBY_NODE_NAME."

# remote address change
doctl compute floating-ip-action assign $FLOATING_IP_ADDRESS $STANDBY_NODE_ID

Add the script to the repmgr promote command

Now edit pg2 repmgr.conf file to invoke our promote-standby.sh script on promotion time.

promote_command = '/var/lib/postgresql/promote-standby.sh'

Run service postgresql restart && repmgrd to apply changes.

Final status primary failure test

Unlike before, when you turn off pg1, pg2 not only promotes itself but also takes over the floating IP, which the app is currently using to perform write operations. As pg2 was already in the sequelize variable’s read array, it is now capable and the sole responsible for data reads and writes. Wait a minute for the promotion to happen and test the app again:

node index.js

Conclusion

Picture yourself in a boat on a river (yes, it's a Beatles reference). If both your oars break loose and only one can be fixed on the spot, the boat motion will become defective and it will be hard to continue the trip.

In our specific case, before having a floating IP, your app would recover data read capability through database fault-tolerance behavior - but it wouldn't be able to perform writes in this condition. Now that your app follows the database's new primary node on automatic promotions, you can heal the cluster and revert it to the initial state in planned conditions and with no rush, as app features are safeguarded.

You can find the source code in this post on GitHub.

How To Automate PostgreSQL and repmgr on Vagrant

Rui Trigo — Tue, 28 Jul 2020 12:33:05 +0000

I often get asked if it's possible to build a resilient system with PostgreSQL.

Considering that resilience should feature cluster high-availability, fault tolerance, and self-healing, it's not an easy answer. But there is a lot to be said about this.

As of today, we can't achieve that level of resilience with the same ease as MongoDB built-in features. But let's see what we can in fact do with the help of repmgr and some other tooling.

At the end of this exercise, we will have achieved some things that come in handy, such as:

a few Ansible roles that can be reused for production
a Vagrantfile for single-command cluster deployment
a development environment that’s more realistic; being close to production state is good to foresee "production-exclusive issues"

Objectives

build a local development environment PostgreSQL cluster with fault tolerance capabilities;
develop configuration management code to reuse in production.

Pre-requisites

Install Vagrant, VirtualBox and Ansible

sudo apt install vagrant
sudo apt install virtualbox && sudo apt install virtualbox-dkms
sudo apt install ansible

Note: An alternative to installing Ansible on your host machine would be using the ansible-local Vagrant provider, which needs Ansible installed on the generated virtual machine instead.

Configuration

1. Write a Vagrantfile

You can use vagrant init to generate the file or simply create it and insert our first blocks.

Vagrant.configure("2") do |config|
  (1..3).each do |n|
    config.vm.define "node#{n}" do |define|
      define.ssh.insert_key = false
      define.vm.box = "ubuntu/bionic64"
      define.vm.hostname = "node#{n}"
      define.vm.network :private_network, ip: "172.16.1.1#{n}"

      define.vm.provider :virtualbox do |v|
        v.cpus = 2
        v.memory = 1024
        v.name = "node#{n}"
      end
    end
  end
end

Let's go block by block:

the 1st block is where we set up the Vagrant version;
on the 2nd block, we iterate the following code so we reuse it to generate 3 equal VMs;
OS, hostname and network settings are set in the 3rd block;
the 4th block contains VirtualBox specific settings.

You can create the servers with:

# create all 3 VMs
vagrant up
# or create only a specific VM
vagrant up node1

2. Add a provisioner

Just by doing the first step alone, we can already launch 3 working virtual machines. A little exciting, but the best is yet to come.

Launching virtual machines is a nice feature of Vagrant, but we want these servers to have PostgreSQL and repmgr configured, so we will use configuration management software to help us. This is the moment Ansible walks in to amaze us.

Vagrant supports several providers, two of them being Ansible and Ansible Local. The difference between them is where Ansible runs, or in other words, where it must be installed. By Vagrant terms, the Ansible provider runs on a host machine (your computer) and the Ansible Local provider runs on guest machines (virtual machines). As we already installed Ansible in the prerequisites section, we'll go with the first option.

Let's add a block for this provisioner in our Vagrantfile.

Vagrant.configure("2") do |config|
  (1..3).each do |n|
    config.vm.define "node#{n}" do |define|
      define.ssh.insert_key = false
      define.vm.box = "ubuntu/bionic64"
      define.vm.hostname = "node#{n}"
      define.vm.network :private_network, ip: "172.16.1.1#{n}"

      define.vm.provider :virtualbox do |v|
        v.cpus = 2
        v.memory = 1024
        v.name = "node#{n}"
      end

      if n == 3
        define.vm.provision :ansible do |ansible|
          ansible.limit = "all"
          ansible.playbook = "provisioning/playbook.yaml"

          ansible.host_vars = {
            "node1" => {:connection_host => "172.16.1.11",
                        :node_id => 1,
                        :role => "primary" },

            "node2" => {:connection_host => "172.16.1.12",
                        :node_id => 2,
                        :role => "standby" },

            "node3" => {:connection_host => "172.16.1.13",
                        :node_id => 3,
                        :role => "witness" }
          }
        end
      end

    end
  end
end

Ansible allows us to configure several servers simultaneously. To take advantage of this feature on Vagrant, we add ansible.limit = "all" and must wait until all 3 VMs are up. Vagrant knows they are all created because of the condition if n == 3, which makes Ansible only run after Vagrant iterated 3 times.

ansible.playbook is the configuration entry point and ansible.host_vars contains the Ansible host variables to be used on the tasks and templates we are about to create.

3. Create an organized Ansible folder structure

If you're already familiar with Ansible, there's little to learn in this section. For those who aren't, it doesn't get too complicated.

First, we have a folder for all Ansible files, named provisioning. Inside this folder, we have our aforementioned entry point playbook.yaml, a group_vars folder for Ansible group variables, and a roles folder.

We could have all Ansible tasks within playbook.yaml, but role folder structure helps with organization. You can read the Ansible documentation to learn the best practices. Below, you will find the folder structure for this tutorial.

project_root
| provisioning
|  |  group_vars
|  |  |  all.yaml
|  |  roles
|  |  |  postgres_12
|  |  |  registration
|  |  |  repmgr
|  |  |  ssh
|  |  playbook.yaml
|  Vagrantfile

4. Ansible roles

4.1 PostgreSQL role

To configure repmgr on PostgreSQL, we need to edit two well-known PostgreSQL configuration files: postgresql.conf and pg_hba.conf. We will then write our tasks to apply the configurations on tasks/main.yaml. I named the PostgreSQL role folder as postgres_12 but you can easily use another version if you want to.

postgres_12
|  tasks
|  |  main.yaml
|  templates
|  |  pg_hba.conf.j2
|  |  pg_hba.conf.j2

You can reuse the default file which comes with PostgreSQL installation and add the following lines to whitelist repmgr database sessions from your trusted VMs. Create an Ansible template file (Jinja2 format) like so:

# default configuration (...)

# repmgr
local   replication   repmgr                              trust
host    replication   repmgr      127.0.0.1/32            trust
host    replication   repmgr      {{ node1_ip }}/32       trust
host    replication   repmgr      {{ node2_ip }}/32       trust
host    replication   repmgr      {{ node3_ip }}/32       trust

local   repmgr        repmgr                              trust
host    repmgr        repmgr      127.0.0.1/32            trust
host    repmgr        repmgr      {{ node1_ip }}/32       trust
host    repmgr        repmgr      {{ node2_ip }}/32       trust
host    repmgr        repmgr      {{ node3_ip }}/32       trust

In the same fashion as pg_hba.conf, you can reuse the postgresql.conf default file and add a few more replication related settings to the bottom of the file:

# default configuration (...)

# repmgr
listen_addresses = '*'
shared_preload_libraries = 'repmgr'
wal_level = replica
max_wal_senders = 5
wal_keep_segments = 64
max_replication_slots = 5
hot_standby = on
wal_log_hints = on

The tasks below will install PostgreSQL and apply our configurations. Their names are self-explanatory.

- name: Add PostgreSQL apt key
  apt_key:
    url: https://www.postgresql.org/media/keys/ACCC4CF8.asc

- name: Add PostgreSQL repository
  apt_repository:
    # ansible_distribution_release = xenial, bionic, focal
    repo: deb http://apt.postgresql.org/pub/repos/apt/ {{ ansible_distribution_release }}-pgdg main

- name: Install PostgreSQL 12
  apt:
    name: postgresql-12
    update_cache: yes

- name: Copy database configuration
  template:
    src: full_postgresql.conf.j2
    dest: /etc/postgresql/12/main/postgresql.conf
    group: postgres
    mode: '0644'
    owner: postgres

- name: Copy user access configuration
  template:
    src: pg_hba.conf.j2
    dest: /etc/postgresql/12/main/pg_hba.conf
    group: postgres
    mode: '0640'
    owner: postgres

4.2 SSH server configuration

ssh
|  files
|  |  keys
|  |   |  id_rsa
|  |   |  id_rsa.pub
|  tasks
|  |  main.yaml

Generate a key pair to use throughout our virtual machines to allow access to them. If you don't know how to do it, this link can help. Just make sure the keys file paths match the paths in the next step.

The tasks below will install the OpenSSH server and apply our configurations. Their names are self-explanatory.

- name: Install OpenSSH
  apt:
    name: openssh-server
    update_cache: yes
    state: present

- name: Create postgres SSH directory
  file:
    mode: '0755'
    owner: postgres
    group: postgres
    path: /var/lib/postgresql/.ssh/
    state: directory

- name: Copy SSH private key
  copy:
    src: "keys/id_rsa"
    dest: /var/lib/postgresql/.ssh/id_rsa
    owner: postgres
    group: postgres
    mode: '0600'

- name: Copy SSH public key
  copy:
    src: "keys/id_rsa.pub"
    dest: /var/lib/postgresql/.ssh/id_rsa.pub
    owner: postgres
    group: postgres
    mode: '0644'

- name: Add key to authorized keys file
  authorized_key:
    user: postgres
    state: present
    key: "{{ lookup('file', 'keys/id_rsa.pub') }}"

- name: Restart SSH service
  service:
    name: sshd
    enabled: yes
    state: restarted

4.3 repmgr installation

repmgr
|  tasks
|  |  main.yaml
|  templates
|  |  repmgr.conf.j2

We configure settings like promote command, follow command, timeouts and retry count on failure scenarios inside repmgr.conf. We will copy this file to its default directory /etc to avoid passing the -f argument on the repmgr command all the time.

The tasks below will install repmgr and apply our configurations. Their names are self-explanatory.

- name: Download repmgr repository installer
  get_url:
    dest: /tmp/repmgr-installer.sh
    mode: 0700
    url: https://dl.2ndquadrant.com/default/release/get/deb

- name: Execute repmgr repository installer
  shell: /tmp/repmgr-installer.sh

- name: Install repmgr for PostgreSQL {{ pg_version }}
  apt:
    name: postgresql-{{ pg_version }}-repmgr
    update_cache: yes

- name: Setup repmgr user and database
  become_user: postgres
  ignore_errors: yes
  shell: |
    createuser --replication --createdb --createrole --superuser repmgr &&
    psql -c 'ALTER USER repmgr SET search_path TO repmgr_test, "$user", public;' &&
    createdb repmgr --owner=repmgr

- name: Copy repmgr configuration
  template:
    src: repmgr.conf.j2
    dest: /etc/repmgr.conf

- name: Restart PostgreSQL
  systemd:
    name: postgresql
    enabled: yes
    state: restarted

4.4 repmgr node registration

Finally, we reach the moment where fault tolerance is established.

registration
|  tasks
|  |  main.yaml

node_id = {{ node_id }}
node_name = 'node{{ node_id }}'
conninfo = 'host={{ connection_host }} user=repmgr dbname=repmgr'
data_directory = '/var/lib/postgresql/{{ pg_version }}/main'
use_replication_slots = yes
reconnect_attempts = 5
reconnect_interval = 1
failover = automatic
pg_bindir = '/usr/lib/postgresql/{{ pg_version }}/bin'
promote_command = 'repmgr standby promote -f /etc/repmgr.conf'
follow_command = 'repmgr standby follow -f /etc/repmgr.conf'
log_level = INFO
log_file = '/var/log/postgresql/repmgr.log'

This role was built according to the repmgr documentation and it might be the most complex role, as it needs to:

run some commands as root and others as Postgres;
stop services between reconfigurations;
have different tasks for primary, standby, and support witness role configuration (in case you want node3 to also be a standby node, just assign role: standby in Vagrantfile ansible.host_vars)

- name: Register primary node
  become_user: postgres
  shell: repmgr primary register
  ignore_errors: yes
  when: role == "primary"

- name: Stop PostgreSQL
  systemd:
    name: postgresql
    state: stopped
  when: role == "standby"

- name: Clean up PostgreSQL data directory
  become_user: postgres
  file:
    path: /var/lib/postgresql/{{ pg_version }}/main
    force: yes
    state: absent
  when: role == "standby"

- name: Clone primary node data
  become_user: postgres
  shell: repmgr -h {{ node1_ip }} -U repmgr -d repmgr standby clone
  ignore_errors: yes
  when: role == "standby"

- name: Start PostgreSQL
  systemd:
    name: postgresql
    state: started
  when: role == "standby"

- name: Register {{ role }} node
  become_user: postgres
  shell: repmgr {{ role }} register -F
  ignore_errors: yes
  when: role != "primary"

- name: Start repmgrd
  become_user: postgres
  shell: repmgrd
  ignore_errors: yes

5. Set group variables

Create a file group_vars/all.yaml to set your VMs IP addresses and the PostgreSQL version you would like to use. Like host_vars set on Vagrantfile, these variables will be placed in the templates placeholders.

client_ip: "172.16.1.1"
node1_ip: "172.16.1.11"
node2_ip: "172.16.1.12"
node3_ip: "172.16.1.13"
pg_version: "12"

6. Put all pieces together with a playbook

The only thing missing is the playbook itself. Create a file named playbook.yaml and invoke the roles we have been developing. gather_facts is an Ansible property to fetch operative system data like distribution (ansible_distribution_release) among other useful variables. You can also read these variables with the Ansible setup module.

- hosts: all
  gather_facts: yes
  become: yes
  roles:
    - postgres_12
    - ssh
    - repmgr
    - registration

7. Start cluster

It's finished. You can now start your cluster with vagrant up and then perform your connections and failover tests.

Testing cluster failover

Now that our cluster is up and configured, you can start by shutting down your standby node:

# save standby state and shut it down ungracefully
vagrant suspend node2

You will see that the cluster is operating normally. Bring the standby node back and it will stay that way.

# bring standby back online after suspension
vagrant resume node1

How about taking down the primary node?

# save primary state and shut it down ungracefully
vagrant suspend node1

At this point, as repmgrd is enabled, the standby node will retry connecting to the primary node the configured number of times (reconnect_attempts = 5) and, if it obtains no response, will promote itself to primary and take over write operations on the PostgreSQL cluster. Success!

To join the cluster again, the old primary node will have to lose its current data, clone the new primary data, and register as a new standby.

vagrant resume node1
vagrant ssh node1
service postgresql stop
rm -r /var/lib/postgresql/12/main
repmgr -h 172.16.1.12 -U -d repmgr standby clone
service postgresql start
repmgr standby register -F
repmgrd
repmgr service status

This last command shows us that the cluster is working properly, but with inverted roles.

postgres@node1:~$ repmgr service status
 ID | Name  | Role  | Status    | Upstream | repmgrd | PID   | Paused? | Upstream last seen
----+-------+---------+-----------+----------+---------+-------+---------+--------------------
 1  | node1 | standby |   running | node2   | running | 22490 | no      | n/a                
 2  | node2 | primary | * running |         | running | 22548 | no      | 0 second(s) ago   
 3  | node3 | witness | * running | node2   | running | 22535 | no      | 0 second(s) ago

Nothing wrong with this, but let's make these nodes switch their roles.

# ssh in and out just to add host key to known_hosts file
ssh <current_primary_ip_address> -o StrictHostKeyChecking=no
exit
# trigger switchover on current standby
repmgr standby switchover --siblings-follow

And we're back to the initial state.

postgres@node1:~$ repmgr service status
 ID | Name  | Role  | Status    | Upstream | repmgrd | PID   | Paused? | Upstream last seen
----+-------+---------+-----------+----------+---------+-------+---------+--------------------
 1  | node1 | primary | * running |         | running | 22490 | no      | n/a                
 2  | node2 | standby |   running | node1   | running | 22548 | no      | 0 second(s) ago   
 3  | node3 | witness | * running | node1   | running | 22535 | no      | 0 second(s) ago

Conclusion

We managed to build a fault-tolerant PostgreSQL cluster using Vagrant and Ansible.

High availability is a big challenge. Much like life’s own matters, we are only prepared for the biggest challenges when we fit that challenges’ conditions.

Production environment unique problems are natural and tough to guess. Bridging the gap between development and production is a way to prevent deployment/production issues. We can make some efforts toward that objective, and that is precisely what we just achieved with this high availability database setup.

You can find the source code of this tutorial here.

How To Achieve Mongo Replication on Docker

Rui Trigo — Thu, 21 May 2020 16:29:27 +0000

In the previous post, we showed how we used MongoDB replication to solve several problems we were facing.

Replication got to be a part of a bigger migration which brought stability, fault-tolerance, and performance to our systems. In this post, we will dive into the practical preparation of that migration.

Motivation

I noticed the lack of tutorials of setting up Mongo replication on Docker containers and wanted to fill this gap along with some tests to see how a Mongo cluster behaves on specific scenarios.

Objectives

To improve our production database and solve the identified limitations, our most clear objectives at this point were:

Upgrading Mongo v3.4 and v3.6 instances to v4.2 (all community edition);
Evolving Mongo data backup strategy from mongodump/mongorestore on a mirror server to Mongo Replication (active working backup server);
Merging Mongo Docker containers into a single container and Mongo Docker volumes into a single volume.

Step-by-step

1. Prepare applications for Mongo connection string change

When our applications were developed, there was no need to pass the Mongo connection URI through a variable, as most of the time Mongo was deployed as a microservice in the same stack as the application containers. With the centralization of Mongo databases, this change was introduced in the application code to update the variable on our CI/CD software whenever we need.

2. Generate and deploy keyfiles

MongoDB’s official documentation has step-by-step instructions on how to setup Keyfile authentication on a Mongo Cluster. Using keyfile authentication enforces Transport Encryption over SSL.

openssl rand -base64 756 > <path-to-keyfile>
chmod 400 <path-to-keyfile>

The keyfile is passed through a keyfile argument on the mongod command, as shown in the next step.

User authentication and role management is out of the scope of this post, but if you are going to use it, configure it before proceeding beyond this step.

3. Deploy existing containers with the `replSet` argument

mongod --keyfile /keyfile --replSet=rs-myapp

4. Define ports

Typically, in this step, you simply choose a server network port to serve your MongoDB. Mongo’s default port is 27017, but since in our case we had 4 apps in our production environment, we defined 4 host ports. You should always choose a network port per Mongo Docker container and stick with them.

27001 for app 1
27002 for app 2
27003 for app 3
27004 for app 4

At step 10, after having replication working, we'll only use and expose one port.

5. Assemble a cluster composed of 3 servers in different datacenters and regions

Preferably, set up 3 servers on different datacenters, or different regions if possible. This will allow for inter-regional availability. Aside from latency changes, your system will survive datacenter blackouts and disasters.

Why 3? It is the minimum number for a worthy Mongo cluster.

1 node: can't have high availability by itself;
2 nodes: no automatic failover — when one of them fails, the other one can't elect itself as primary alone;
3 nodes: minimum worth number — when one of them fails, the other two vote for the next primary node;
4 nodes: has the same benefits as 3 nodes plus one extra copy of data (pricier);
5 nodes: can withstand 2 nodes failure at the same time (even pricier).

There are Mongo clusters with arbiters, but that is out of the scope of this post.

6. Define your replica set members’ priorities

Adjust your priorities to your cluster size, hardware, location, or other useful criteria.

In our case, we went for:

appserver: 10 // temporarily primary
node1: 3 // designated primary
node2: 2 // designated first secondary being promoted
node3: 1 // designated second secondary being promoted

We set the node which currently had the data with priority: 10, since it had to be the primary in the sync phase, while the rest of the cluster is not ready. This allowed continuing serving database queries while data was being replicated.

7. Deploy Mongo containers scaling to N* on a Mongo cluster

(*N being the number of Mongo cluster nodes).

Use an orchestrator to deploy 4 Mongo containers in the environment, scaling to 3.

4 is the number of different Mongo instances;
3 is the number of Mongo cluster nodes.

In our case, this meant having 12 containers in the environment temporarily.

Remember to deploy them as replica set members, as shown in step 3.

8. Replication time!

This is the moment when we start watching database users and collection data getting synced. You can enter the mongo shell of a Mongo container (preferably primary) to check the replication progress. These two commands will show you the status, priority and other useful info:

rs.status()
# and
rs.conf()

When all members reach the secondary state, you can start testing. Stop the primary node to witness secondary promotion. This process is almost instantaneous.

You can stop the primary member by issuing the following command:

docker stop <mongo_docker_container_name_or_d>

When you bring it back online, the cluster will give back the primary role to the member with the highest priority. This process takes a few seconds, as it is not critical.

docker start <mongo_docker_container_name_or_id>

9. Extract Mongo containers from application servers

If everything is working at this point, you can stop the Mongo instance on which we previously set priority: 10 (stop command in the prior step) and remove that member from the replica set passing its hostname as parameter.

Repeat this step for every Mongo container you had on step 4.

10. Migrate backups and change which server they read the data from

As mentioned in the previous post, one handy feature of MongoDB replication is having a secondary member asking for data to mongodump from another secondary member.

Previously, we had the application + database server performing mongodump of its data. As we moved the data to the cluster, we also moved the automated backup tools to a secondary member, to take advantage of said feature.

11. Merge data from 4 Mongo Docker containers into one database

If you only had 1 Mongo Docker container at the start, skip to step 12.

Besides having simplicity telling us to do this before step 1, we decided to act cautiously and keep apps and databases working in a way as close as they were before until we mastered Mongo replication in our environment.

At this stage, we chose to import data from all Mongo databases to a single Mongo database — the one which contained the most data. When working with MongoDB, remember this line from the official docs:

In MongoDB, databases hold collections of documents.

That means we can take advantage of mongodump --db <dbname> and mongorestore --db <dbname> to merge Mongo data into the same instance (this goes for non-Docker as well).

12. Monitor cluster nodes and backups

When you have merged your databases into the same instance, you will shut down other instances, right? Then, you will only need to monitor the application and perform backups of that same instance. Don't forget to monitor the new cluster hardware. Even with automatic fault-tolerance, it is not recommended to leave our systems short. As a hint, there is a dedicated role for that called clusterMonitor.

Conclusion

Sharing this story about our database migration will hopefully help the community — especially those not taking full benefits from MongoDB already — to start seeing MongoDB in a more mature and reliable way.

Even though this is not a regular MongoDB replication "how-to" tutorial, this story shows important details about MongoDB’s internal features, our struggle to not leave any details behind, and, again, the benefits of such technology. That's what I believe technology is for — helping humans with their needs.

In case you're also interested in learning more about application security, we recommend reading our free data sheet on JavaScript Security Threats, which provides an overview of the most relevant attacks to JavaScript apps and how to prevent them.

How We Achieved MongoDB Replication on Docker

Rui Trigo — Tue, 21 Apr 2020 13:27:46 +0000

Prologue

Picture your database server. Now imagine it somehow breaks. Despair comes up and disturbs the reaction.

Maybe you lost data. Maybe you had too much downtime. Maybe you lost work hours. Maybe you lost precious time and money. High Availability is easily a nice-to-have but in times like these, you value it more.

MongoDB comes with clustering features that provide more storage capacity (sharding) and more reliability (replication). This article will focus on MongoDB replication on Docker containers.

Motivation

We felt the need to improve our production database and data backup strategy as we identified it was giving the servers a hard time performance-wise and the disaster recovery process was very hard in most procedures.

So, we started to design a migration plan to solve this. We also took the chance to update the Mongo version in use to benefit from new features and security improvements.

Before

Old Production Environment

Production servers

server_1: application services containers + 2 mongo containers
server_2: application services containers + 1 mongo containers
server_3: application services containers + 1 mongo containers

Mirror servers

mirror_server_1: application services containers + 2 mongo containers (updated once a day)
mirror_server_2: application services containers + 1 mongo containers (updated once a day)
(services and data on server3 were not in mirror environment)

The Mongo service was kept using a mongodump/mongorestore strategy.

mongodump/mongorestore Strategy

The first part of the mongodump/mongorestore strategy is composed of Cron jobs that dump the data in the Mongo database to a different Mongo instance with the mongodump utility.

mongodump is a utility for creating a binary export of the contents of a database. mongodump can export data from either mongod or mongos instances; i.e. can export data from standalone, replica set, and sharded cluster deployments.

mongodump --host=mongodb1.example.net --port=3017 --username=user --password="pass" --out=/opt/backup/mongodump-2013-10-24

The command above outputs a Mongo data dump file named mongodump-2013-10-24 on /opt/backup directory, from the connection to mongodb1.example.com.

The second part of this strategy is restoring the second database by the data in the mongodump with mongorestore utility.

The mongorestore program loads data from either a binary database dump created by mongodump or the standard input (starting in version 3.0.0) into a mongod or mongos instance.

mongorestore --host=mongodb1.example.net --port=3017 --username=user  --authenticationDatabase=admin /opt/backup/mongodump-2013-10-24

The command above writes the data from /opt/backup/mongodump-2013-10-24 file to the Mongo instance on the mongodb1.example.com connection.

WARNING: The process of mongorestore utility is NOT incremental. Restoring a database will delete all data prior to writing the mongodump data.

Problems and Limitations

Late backup data: Since mongodump ran daily at a specified time, the data from that time until the moment of the database switch would be lost.
Unavailability: The mongodump and mongorestore utilities took several hours to complete in the biggest databases. During the DB restore, nothing could be done as the Mongo data can't be used until mongorestore is finished. The DB will only be available when this is completed. Also, switching from a production environment to a mirror environment was a manual process which took some time.
High disk usage: Restoring a whole database (or several DBs simultaneously) would take up disks inodes, as well as a toll of usage in your disks.
Scalability limitations: Using a Mongo Docker instance for each database, even distributed by different servers, brought the need of setting up an instance, different network addresses and ports, and new backup containers (mongo-tools). A Mongo cluster would fit the needs for our applications and make database administration way simpler.
Reserved memory: By default, each Mongo container will try to cache all available memory until 60%. Since we previously had 1 Mongo container on two application servers and 2 containers in the same application server, all of them had at least 60% busy (in use + cached). Whenever there is more than one Mongo container, they will dispute all available memory to reach 60% each. (2 -> 120%, 3 -> 180%, 4 -> 240%, etc.). For these reasons, it is very important to set adequate container memory limits.
Amount of Docker volumes: MongoDB data, dumps, and metadata were scattered through several Docker volumes, mapped to different filesystem folders. Merging these databases would allow centralizing this data.
Security and features: Upgrading to Mongo 4 would solve security issues and bring more features to improve DB performance and replication, like non-blocking secondary reads, transactions and flow control.

Objectives

To improve our production database and solve the identified limitations, our most clear objectives at this point were:

Upgrading Mongo v3.4 and v3.6 instances to v4.2 (all community edition);
Changing Mongo data backup strategy from mongodump/mongorestore to Mongo Replication;
Merging Mongo containers into a single container and Mongo Docker volumes into a single volume.

And to get to these objectives, we defined the following plan:

Plan topics

Prepare applications for Mongo connection string change;
Assemble a cluster composed of 3 servers in different datacenters and regions;
Generate and deploy keyfiles on the filesystems;
Redeploy existing Mongo Docker containers with replSet argument;
Define network ports;
Deploy new 4 Mongo containers scaling to 3 (4 x 3 = 12) on a Mongo cluster;
Add new Mongo instances to the replica set to sync from old Mongo containers;
Stop Mongo containers from application servers and remove them from the replica set;
Migrate backups and change which server they read the data;
Merge data from 4 Mongo containers into one database;
Unify backups.

We will publish a second part of this tutorial soon, where we will go through each of these topics.

Results

Some of the achieved results were:

Fault-tolerance: Automatic and instant primary database switch.
Data redundancy: Instantaneously synced redundant data.
Inter-regional availability: Location disaster safeguarding.
Cluster hierarchy: Mongo replication allows nodes priority configuration, which allows the user to order nodes by hardware power, location, or other useful criteria.
Read operations balance: Read operations can be balanced through secondary nodes, like dashboards queries and mongodumps. Applications can also be configured (through Mongo connection URI) to perform read operations from secondary nodes, which increases database read capacity.
Performance: Now that memory used and cached is right for the system needs, Mongo databases are hosted in dedicated servers, its version got bumped and the cluster can balance read operations, performance improvements exceeded the expectations.

New Production Environment

Production application servers should connect to the Mongo Production Cluster using replica set;
The Mirror application server should connect to the Mongo Production Cluster and keep storing the most recent mongodumps;
The Mongo Cluster secondary node should mongodump the cluster data to the Mirror environment, asking for it to another secondary node.

Conclusion

This post is more than about MongoDB replication on Docker. It is about a victory in stopping the infrastructure growth going in the wrong direction and having things done the way we thought they should be.

Much like a tree growing on a vase, we should plant it in a garden, where it can grow freely. Now, we will watch that tree scale without adding a new vase every time it needs to grow and not be afraid of breaking them. That's what high availability clusters are all about — building an abstract module for the application layer which can scale and keep being used the same way.

The whole process was done with intervals between major steps, to allow checking if the new strategy was working for us. We are very glad to have all the problems in the before section solved.

Achieving this means that we are now prepared to scale easily and sleep well knowing that MongoDB has (at least) database fault-tolerance and recovers by itself instantaneously — which lowers the odds of disaster scenarios.

Stay tuned for part 2, where we’ll explore the whole technical setup.

Meanwhile, you may like our post about Row vs. Columnar Databases or our full-stack tutorial on Creating a Public File Sharing Service with Vue.js and Node.js.

Another resource that you may find useful is our free data sheet on JavaScript Security Threats, which provides an overview of the most relevant attacks to JavaScript apps and how to prevent them.

DEV Community: Rui Trigo

Dockerfile Optimization for Fast Builds and Light Images

Objectives

Pre-requisites

Simple Dockerfile example

Enabling BuildKit

BuildKit Initial Impact

Order from least to most frequently changing

Avoid "COPY ."

Couple apt-get update & install

Remove unnecessary dependencies

Remove package manager cache

Use official images where possible

Use specific tags

Look for minimal flavors

Build from a source in a consistent environment

Fetch dependencies in a separate step

Multi-stage builds: remove build dependencies

Checkpoint

Multi-stage builds: different image flavors

Different image flavors (DRY / global ARG)

Concurrency

BuildKit Application Cache

BuildKit Secret Volumes

Conclusion

The 6 Aspects You Must Secure On Your MongoDB Instances

Objectives

Pre-requisites

List of Quick Wins

1. Network access

2. System access

3. Authentication

4. Authorization

5. Encrypted connections

6. Encryption at rest

Conclusion

Connecting Sequelize To a PostgreSQL Cluster

Prologue

Objectives

Pre-requisites

Set up your cluster

Create your droplets

Assign a floating IP to your primary node

Configure PostgreSQL with repmgr

NodeJS application

(Optional) Current status primary failure test

Use a floating IP

Add the floating IP address to your app database connection object

Digital Ocean CLI configuration

Add the script to the repmgr promote command

Final status primary failure test

Conclusion

How To Automate PostgreSQL and repmgr on Vagrant

Objectives

Pre-requisites

Configuration

1. Write a Vagrantfile

2. Add a provisioner

3. Create an organized Ansible folder structure

4. Ansible roles

4.1 PostgreSQL role

4.2 SSH server configuration

4.3 repmgr installation

4.4 repmgr node registration

5. Set group variables

6. Put all pieces together with a playbook

7. Start cluster

Testing cluster failover

Conclusion

How To Achieve Mongo Replication on Docker

Motivation

Objectives

Step-by-step

1. Prepare applications for Mongo connection string change

2. Generate and deploy keyfiles

3. Deploy existing containers with the replSet argument

4. Define ports

5. Assemble a cluster composed of 3 servers in different datacenters and regions

6. Define your replica set members’ priorities

7. Deploy Mongo containers scaling to N* on a Mongo cluster

3. Deploy existing containers with the `replSet` argument