DEV Community: Fabio Oliveira Costa

Building a temporary database for testing and development

Fabio Oliveira Costa — Sun, 20 Jun 2021 16:32:42 +0000

TL;DR

Add the initial data to the volume, and run docker-compose with the renew anon volumes argument (-V)

Introduction

So you have built your system and now is time to test it, or you want to do a demo and everything needs to be set up on a given state. Unfortunately, you were testing some changes and now you need to clean up the database, or even worse perform a series of actions just to get it in the desired state.

Luckily there are a few actions you can take to make your life easier.

These next tips will assume you have docker and docker compose installed as these will be the backbone of the solution.

Step 1 - An empty database with docker

Let's say you just want a runnable database to start developing, perhaps you are playing with some ORM¹ solution, an empty database can be quite handy.

The following docker command can do the trick for postgres:

docker run -e POSTGRES_USER=my_user -e POSTGRES_PASSWORD=my_password -e POSTGRES_DB=some_db postgres:13

A one-liner and we have an accessible database to our heart's content.

Step 2 - A databse witth initial data and docker

A database without data is nice but a database already set up is even nicer. For that we can use volumes and an initial data for the database dump. [The initialization scripts section] of the postgres image tells us that all we need to do is copy some dump to the folder /docker-entrypoint-initdb.d/.

This would be done with this argument:

--mount type=bind,source=/some/path/my_dump.sql,target=/docker-entrypoint-initdb.d/dump.sql

So our full command would look like

docker run -e POSTGRES_USER=my_user -e POSTGRES_PASSWORD=my_password -e POSTGRES_DB=some_db  --mount type=bind,source=/some/path/my_dump.sql,target=/docker-entrypoint-initdb.d/dump.sql  postgres:13

Step 3 - Tidying it up with docker-compose

Docker is useful but docker-compose makes things more manageable.

With 2 tricks we can have our database running temporarily without much hassle.

Step 3.1 Convert the docker to docker-compose yaml

Let's just add the arguments of our command line to a compose file.

# docker-compose.yml
version: "3"
services:
  db:
    image: postgres:13
    environment:
      POSTGRES_PASSWORD: "my_password"
      POSTGRES_USER: "my_user"
      POSTGRES_DB: "some_db"
    volumes:
      -  ./some/path/my_dump.sql:/docker-entrypoint-initdb.d/my_dump.sql

This is very nice, run it with the following command, and your database will be there same as before.

docker-compose up

Step 3.2 Making sure our database remains the same

If we ran the previous command and delete a record we would be losing that record next time. This is sometimes not desirable for testing, demos, or even general development. I find oy very calming to have the same environment every time.

To achieve that we need to renew the volumes. Running it with the argument -V , from compose docs ,this states the following:

-V, --renew-anon-volumes Recreate anonymous volumes instead of retrieving data from the previous containers.

This makes our command to start the database looks like this:

docker-compose up -V db

Now if you delete some initial data and execute the command again the state will be brought right back to how it was.

Hope you folks find it helpful.

Object Relational Mapping ↩

Introduction to container based development part 4/4 - Compose

Fabio Oliveira Costa — Thu, 26 Mar 2020 20:15:34 +0000

This is the fourth part of our container based development series. Check the other posts at the end of this one.

This post is based on the container development article created by me.

Running containers manually is useful but it quickly gets out of hand when you need to manage complex architectures, to make things easy we can use docker-compose. It has a nice and centralized way to create multiple containers at the same time.

Let's port our old example with docker-compose and see how it looks.The official documentation is full of more info and all the files for this step are on the projects repository on the firstDockerCompose branch.

Let's create on the same folder that our files are a folder to use as a volume. It is customary to keep all docker-compose related files under the same root, this folder will be called "ourLocalVolume". Then we need to create a compose file. This compose file is defining a single service called runner, building it from a local dockerFile and creating a volume called ourLocalVolume mounted on the container /app/ourApp/data.

version: '3'
services:
  runner:
    build: .
    volumes:
      - ./ourLocalVolume:/app/ourApp/data

To run our solution we need to do the following command on the terminal:

docker-compose up

If we make a change to our file we need to rebuild it with:

docker-compose build

Or we need to run and build with:

docker-compose up --build

Rebuilding every time is not very productive. Our code is also very simple we are not doing anything special with the node image we could easily be more productive if our code changes on the host machine would be transferred to the container. The way to achieve that is by using a volume with our code and our data , actually after all this changes we don't even need to make our own custom Dockerfile we can do everything using docker-compose. We are going to tidy our space a little so see the changes the secondDockerCompose branch.

Our compose volume changed and now we are going to use the node image directly:

version: '3'
services:
  runner:
    image: node:13-alpine3.10
    volumes:
      - ./src:/app
    working_dir: /app
    command: node ./nodeCounter.js

Since our code now lives on a volume along with our data we can update our code without needing to rebuild an image. Also since we are essentially only using a node image we do not need to create a custom image every time.

Getting dangerous, multiple services!

Ok compose for a single service is nice but not as nice as managing an actual architecture.

We are going to maintain this counter but it will count to up to 20 and then reset and never stop.Also we are going to make a web frontend with php that is going to show the counter as it was at the moment somebody entered the page. Our file architecture will change again so check the
thirdDockerCompose branch.

We are going to change our node to run from a fixed place that is going to be our volume data

const fs = require('fs')
const path = require('path')

const counterFile = path.resolve('/data/counter.txt')

const COUNTER_LIMIT = 20
const INTERVAL = 1000
let counterData = 0
if (fs.existsSync(counterFile)) {
  const fileData = fs.readFileSync(counterFile)
  counterData = parseInt(fileData, 10)
}

const stopCounter = counterData + COUNTER_LIMIT

const recursiveCounter = () => {
  if (counterData === stopCounter) {
    counterData = 0
  }
  console.log(counterData)
  counterData++
  fs.writeFileSync(counterFile, counterData)
  setTimeout(recursiveCounter, INTERVAL)
}

recursiveCounter()

And this will be our PHP code:

<?php
    $counter = file_get_contents('/data/counter.txt');
?>
<html lang="en">
<head>
  <meta charset="UTF-8">
  <meta name="viewport" content="width=device-width, initial-scale=1.0">
  <meta http-equiv="X-UA-Compatible" content="ie=edge">
  <title>Counter <?=$counter ?></title>
</head>
<body>
  <h1>Welcome to the counter view</h1>
  <p>At this moment the counter is <?=$counter ?></p>
  <p>Reload for even more counter fun!</p>
</body>
</html>

Our docker-compose needs to change so it will have 2 services! Also we will link the volumes so the PHP can read what node is writing.

version: '3'
services:
  runner:
    image: node:13-alpine3.10
    volumes:
      - ./src/counter:/app
      - ./src/data:/data
    working_dir: /app
    command: node ./nodeCounter.js
  web:
    image: php:7.2-apache
    volumes:
      - ./src/web:/var/www/html/
      - ./src/data:/data
    ports:
      - 8081:80

If we run docker-compose up we should be able to see our page on http://localhost:8081. Reload and you will see changes.
Also Note that our system now never stop to exit we need to send a SIGTERM by typing "ctrl+c"

Let's take a moment to appreciate what is happening:

We created 2 services based on 2 different images.
We created 4 volumes being 2 of them shared between different containers.
We create a port forwarding from a container to our host machine.
We "installed" a php with apache and node with only 16 lines of code!

Now that you were introduced to containers see how they can help you be a more productive and happier dev. Also there is one bonus part on the PDF article when we add a Redis database to our Node, PHP project just for the laughs.

Have fun, be happy, be healthy be kind.

Introduction to container based development - Part 3/4 - Volumes

Fabio Oliveira Costa — Thu, 26 Mar 2020 19:50:47 +0000

This is the third part of our container based development series

This post is based on the container development article created by me.

On the previous articles we did a program that count from 0 to 19 but could not do anything besides that. A program that can not persist is not so useful so let's modify our program a little so it will count the current file content and then count to that value +20. For that we need to create a volume on the container, this will make a specific folder on the container persist when mounted. All the files for this step are on the projects repository on the secondContainer branch.

First we need to add a proper volume on our docker file.

Dockefile

FROM node:13-alpine3.10
RUN mkdir -p /app/ourApp/data
COPY ./nodeCounter.js  /app/ourApp/
WORKDIR  /app/ourApp/
VOLUME /app/ourApp/data/
CMD node ./nodeCounter.js

Then we need to make sure our program don't stop at 20 but stops at start + 20

nodeCounter.js

const fs = require('fs')
const path = require('path')

const counterFile = path.resolve(__dirname, './data/counter.txt')

const COUNTER_LIMIT = 20
const INTERVAL = 1000
let counterData = 0
if (fs.existsSync(counterFile)) {
  const fileData = fs.readFileSync(counterFile)
  counterData = parseInt(fileData, 10)
}

const stopCounter = counterData + COUNTER_LIMIT

const recursiveCounter = () => {
  if (counterData === stopCounter) {
    process.exit(0)
  }
  console.log(counterData)
  counterData++
  fs.writeFileSync(counterFile, counterData)
  setTimeout(recursiveCounter, INTERVAL)
}

recursiveCounter()

Then we need to build our image with the same name and a new tag.

docker build -t container_dev:second  .

We should see an output with the text:

"Successfully tagged container_dev:second"

But out work is not over we need to actually create a volume on our computer (the host) so our container can read and write persistent data.

docker volume create our-named-volume

But where physically our volume is? For that we can inspect the volume and check it's exact path.

docker volume inspect our-named-volume

That would give all the volume data like the one below, the mountpoint is where docker created our volume.

[
    {
        "CreatedAt": "2020-03-20T10:44:48+01:00",
        "Driver": "local",
        "Labels": {},
        "Mountpoint": "/var/lib/docker/volumes/our-named-volume/_data",
        "Name": "our-named-volume",
        "Options": {},
        "Scope": "local"
    }
]

Now to run our app with one new argument , -v to link the volume

docker run -v our-named-volume:/app/ourApp container_dev:second

Run it twice and you will notice we start where we stopped. If you open 2 shells and do it a little bit after the first one start you should see they are sharing and modifying the same file (There is a race condition here but we don't mind for now).

We have a way to persist state, actually we even have a way to share the same set of files between multiple containers!

I would like to call attention about the importance of mounting, if we run without the "-v" command we will have the same behavior as the first time, our state would not persist.

So let's see what we learned

For a volume to be accessible in the container we need to create it at some location inside the container.
We need to link a volume with the -v when running the docker run argument.
Multiple containers can have access to the same volume.

Next part: Containers and installing PHP alongside node in 16 lines of code!

Introduction to container based development - Part 2/4

Fabio Oliveira Costa — Thu, 26 Mar 2020 19:25:22 +0000

This is the second part of our container based development series. This post is based on the container development article created by me.

Creating our image - Pressing our own CD.

We are going create a node program its behavior is the following.

try to find a file and read from it;
output the content of the file;
save the content +1;
repeat every 1 second until 19.
if no content is found it will create the file and save with 0;

The code is on the repository on the firstContainer branch.

nodeCounter.js

const fs = require('fs')
const path = require('path')

const counterFile = path.resolve(__dirname, './counter.txt')

const COUNTER_LIMIT = 20
const INTERVAL = 1000
let counterData = 0
if (fs.existsSync(counterFile)) {
  const fileData = fs.readFileSync(counterFile)
  counterData = parseInt(fileData, 10)
}

const recursiveCounter = () => {
  if (counterData === COUNTER_LIMIT) {
    process.exit(0)
  }
  console.log(counterData)
  counterData++
  fs.writeFileSync(counterFile, counterData)
  setTimeout(recursiveCounter, INTERVAL)
}

recursiveCounter()

With this code on some folder we also need to configure our container, to make how our CD is going to be built.

Dockerfile (No extension)

FROM node:13-alpine3.10
RUN mkdir -p /app/ourApp/
COPY ./nodeCounter.js  /app/ourApp/
WORKDIR  /app/ourApp/
CMD node ./nodeCounter.js

The dockerfile above have some commands

FROM - We are using a Node image that uses the alpien linux operating system.
RUN - On this image we are creating a folder.
COPY - We are copying our node program to the folder.
WORKDIR - We change the working dir so all commands will run starting from this new location.
CMD - We say which command should run as soon as you play the CD.

To make our cd we execute the following on the terminal, it says to build a image called container_dev with the tag first using the dockerfile on the same folder I am executing the command.

docker build -t container_dev:first  .

The output will be something like:

Sending build context to Docker daemon  70.66kB
Step 1/5 : FROM node:13-alpine3.10
.......
Successfully built 50c9fae3a235
Successfully tagged container_dev:first

If we want to see our images we can type

docker images

To see our container data like Ids, names , tags and so on.

REPOSITORY                              TAG                 IMAGE ID
container_dev                           first               50c9fae3a235       
node                                    13-alpine3.10

Now that our image has a name and a tag we can run it:

docker run container_dev:first

And you should see a count from 0 to 19 and our image stopping. Congratulations you just created your first image!
This image is saving data on a file, run it again if we persisted data it should stop immediately since we are already on the 20 but now it will start again from 0, run it twice with a little delay each count will not interfere with each other. That is part of the security of containers they are isolated.

Our next topic: Volumes!

Introduction to container based development - Part 1/4

Fabio Oliveira Costa — Thu, 26 Mar 2020 19:19:53 +0000

This post is based on the container development article wrote by me. We are going to review some basics about containers and hopefully you are going to start loving and using them as much as me.

Requirements

Have docker and docker-compose installed.

Introduction

Containers borrow the name from shipping containers a mean of transportation that revolutionized the way we ship goods. Be it 100 tons of tuna, a whole house or thousands of unrelated Amazon packages they can be easily transported from anywhere to anywhere. Software containers are the same, the handlers of the containers don't care about what is inside they know how to handle containers and that is what make them flexible.

On the software world developing with containers allow us to eliminate the "it works on my machine", run multiple versions of the same software, create complex architectures on a local machine, make micros services with hundreds of different languages and many more nice things.

The concepts

Image

Think of an image as a read only data, like a music “CD”. You could play the PHP CD, perhaps you are feeling a little javazzy and play the java CD or you want to do your own thing so you remix a CD and put the data that you need. The CD does not have a state it is what it is and that particular CD will always have the same contents.
A image is built with layers on top of a filesystem. Usually you create an image using another image and create it with a series of instructions, if docker already has these instructions on cache it will use that cache instead of building from scratch. Still on our CD metaphor the CD is done with each track separated, you have the drums, the guitar, the bass and the vocals each on a different layer, if you had a karaoke version of the CD the studio one would be the karaoke with the vocals track on top of it.

Images are listed on a repository like docker hub and have a name and a tag.

Container

A container is a running image and it has state but does not have persistence. Think about a CD and somebody manipulating it like a DJ, it is doing changes to the current state of the CD but by the time the execution is done the CD is back on the case and nothing changed on it. The execution of the CD was the container and it can change as much as we need but it's changes will not change the image itself.

Volumes

A volume is where we have persistent state, this is where our changes are not lost, this is how containers read and write persistent data. For example think on an application that counts sales, it would be a pretty lousy application if we lost all sale data as soon as the application was shutdown, for that we persist data on volumes. Most database images will have a volume. Still on the musical analogy our volume is our live recording, we are working on it and it will change, even if we have some fixed tracks we are still changing the data on this one.

	State	Persistence
Image	No	No
Container	Yes	No
Volume	No	Yes

Getting started

Running the first image - Playing the CD

First we need our image on our system so we pull it from the repository. Execute the code below on you terminal.

docker pull docker/whalesay:latest

The output should say something like:

"Status: Downloaded newer image for docker/whalesay:latest"

The image is on our system now we need to run it (Play the cd):

docker run docker/whalesay cowsay "Docker is awesome"

And you should see the following:

< Docker is awesome >
 ------------------- 
    \
     \
      \     
                    ##        .            
              ## ## ##       ==            
           ## ## ## ##      ===            
       /""""""""""""""""___/ ===        
  ~~~ {~~ ~~~~ ~~~ ~~~~ ~~ ~ /  ===- ~~~   
       \______ o          __/            
        \    \        __/             
          \____\______/

So what happened? For a brief moment we run our image, that generated a container that printed a whale on the screen and then the container stopped.

Congratulations you just used your first image and created your first container!

Acknowledgement

Photo by Kaique Rocha from Pexels