DEV Community: Yanuar Arifin

Use Golang Migrate on Docker Compose

Yanuar Arifin — Mon, 27 May 2024 09:01:13 +0000

Previously I setup docker compose for golang application and PostgreSQL. It can run the application and can connect to PostgreSQL. While doing basic CRUD, I found that I need a tool to migrate my database structure and seed the database for easy onboarding and development. Thus I want to use golang-migrate to do that.

What Is Golang Migrate

Contrary to the name, it is not a migration tool specifically created for golang development, although we can use it as a golang package in your application. This tool is a CLI tool that can be installed on Windows, Mac, and Linux. As a CLI tool, it means that no matter what language you use to code, your migration can be managed using this golang-migrate.

Here is the documentation how to install it.

Golang migrate commands

What I use is this for now.

Create migration script:

migrate create -ext sql -dir migrations -seq create_users_table

migrate golang-migrate command
create create new migration script, both up and down
-ext sql use sql extension
-dir migrations the target directory where the migration files is generated
-seq make the file name sequential, increment from 1. if not it would be current datetime
create_users_table migration files name

which will create 2 files in migrations dir

    ├── Dockerfile
    ├── Dockerfile.multistage
    ├── README.md
    ├── cmd
    │   └── main.go
    ├── controllers
    ├── docker-compose.yml
    ├── go-market-warehouse-api
    ├── go.mod
    ├── go.sum
    ├── main
    ├── middlewares
    ├── migrations
+++ │   ├── 000001_create_users_table.down.sql <- this
+++ │   └── 000001_create_users_table.up.sql   <- and this
    ├── models
    └── repositories

000001_create_users_table.up.sql contains the DDL for the changes that I want to make, in this case creating users table

CREATE EXTENSION IF NOT EXISTS "uuid-ossp";
CREATE TABLE users (
    id UUID PRIMARY KEY DEFAULT uuid_generate_v4(),
    name VARCHAR NOT NULL,
    email VARCHAR(255) NOT NULL UNIQUE
);

and 000001_create_users_table.down.sql contains the DDL to undo the above code.

DROP TABLE IF EXISTS "users";

And Apply the migration once the DDL is supplied in the files:

migrate -path /migrations/ -database "postgres://username:password@host:port/db_name?sslmode=disable" up

Here's the database tables before we execute the migration:

And here it is after running migration

schema_migrations is used by golang migrate to track migrations
users table is the table that being generated from my script above

How To Use Golang Migrate in Docker Compose

Golang migrate has its own image in docker hub. The way to use this, is to create a new service for this golang migrate, then run the migration that targetted to PostgreSQL service. By creating it's own service and execute it, the migration is another service that is not added to the main application service, make the application smaller.

Here's the new golang-migrate service:

service:
  # ... go-market-warehouse-api & local-pg-16 services
  migrate:
    image: migrate/migrate
    depends_on:
      local-pg-16:
        condition: service_healthy
    networks:
      - my-local-net 
    volumes:
      - ./migrations/:/migrations
    command: ["-path", "/migrations/", "-database", "postgres://${PGUSER}:${PGPASSWORD}@local-pg-16:5432/postgres?sslmode=disable", "up"]

Full code in this commit

image: migrate/migrate base this service to the image of golang-migrate

depends_on: local-pg-16: condition: service_healthy I rearrange the application startup order to PostgreSQL then golang-migrate then go-market-warehouse-api because after we startup the database, I want the database to be updated using latest DDL then the application can connect to database.

networks use already declared network, which is my-local-net to connect to PostgreSQL service

volumes here I specify which directory contains the migration scripts, in my case, it is the migrations directory, the LEFT one ./migrations/.

commands is the arguments the migrate service use to run the migration

The commands is the bash command to apply the migration(s) like the one above without using docker.

migrate -path /migrations/ -database "postgres://${PGUSER}:${PGPASSWORD}@local-pg-16:5432/postgres?sslmode=disable" up

This is the docker log with migration executed

migrate-1 is the service name
1/u create_users_table 1 migration upped, with name of create_users_table

If no new migration found, there won't be message on the log.

That is it for this article, hope you enjoy it.

Setting Up Docker Compose for Golang Application

Yanuar Arifin — Fri, 24 May 2024 06:56:14 +0000

This post will explain how I create Dockerfile and docker-compose.yml for golang development. The full code can be seen here

This code for Dockerfile

FROM golang:1.22 

WORKDIR /app

COPY go.mod go.sum ./

RUN go mod download

COPY . ./

RUN CGO_ENABLED=0 GOOS=linux go build -o /go-market-warehouse-api ./cmd/main.go

EXPOSE 8000

CMD [ "/go-market-warehouse-api" ]

FROM golang:1.22 is basing the image to current newest golang version available

WORKDIR /app tell docker to create and make this /app directory as the current directory the docker is in

COPY go.mod go.sum ./ will copy go.mod and go.sum from project directory to /app directory inside docker

RUN go mod download will download the dependency listed in go.mod that we just copied

COPY . ./ will copy content in current directory (the project root dir ). Basically copying the project to docker

RUN CGO_ENABLED=0 GOOS=linux go build -o /go-market-warehouse-api ./cmd/main.go is building go binary to from cmd/main.go and give output to root dir / and use go-market-warehouse-api as the binary name, same as go-market-warehouse-api.exe in windows

EXPOSE 8000 tell the image it builds to open port 8000 to outside the image

CMD [ "/go-market-warehouse-api" ] will run the binary that we create

This setup Dockerfile will create an image that is around 1 Gb. Big but it contains necessary tools to build golang app. Suitable for development.

building docker-compose.yml

Create dedicated volume and network

The purpose for this docker-compose.yml is to run the golang application and a database in my application stack. My database choice is PostgreSQL.

Before we create the file, first I will create new docker network for the application and database to communicate. And a volume to persist data for the database, so when we restart the database, the data that we previously insert still exist.

Create volume:

docker volume create pg-16

This will create a volume called pg-16. You can check it using docker volume list

Create network:

docker network create -d bridge my-local-net

This will create a new bridge network called my-local-net. Check with docker network list

The bridge part is a network that can be accessed inside the docker

Application Service

Create the application service docker-compose.yml file:

services:
  go-market-warehouse-api:
    # This will be uncommented when db service is introduced
    # depends_on:
    #  local-pg-16:
    #    condition: service_healthy

    # always restart when app crashes
    restart: always
    build:
      context: .
    image: go-market-warehouse-api:v1.0
    container_name: go-market-warehouse-api
    hostname: go-market-warehouse-api
    networks:
      - my-local-net
    ports:
      - 80:8080
    environment:
      - PGUSER=${PGUSER:-totoro}
      - PGPASSWORD=${PGPASSWORD:?database password not set}
      - PGHOST=${PGHOST:-db}
      - PGPORT=${PGPORT:-5432}
      - PGDATABASE=${PGDATABASE:-mydb}
    deploy:
      restart_policy:
        condition: on-failure

services this mark the list of available services when we run docker compose. In this case, the services that is available only 1, called go-market-warehouse-api(the one under services:)

build and context: .: will build from current directory where docker-compose.yml is located(indicated by .)

image which image it will use to build. when not available locally, it will search from docker registry

container_name the name of the container to be built

hostname how can this service be called by other service in the network. Docker container can refer to other services location(ip) using this hostname

networks: the network this service attached to

ports: exposed ports by this service

environment : setting environment variable

deploy and its child specs : it will deploy the service and restart it when failure happens

Database Service

This is the description of the database service:

  local-pg-16:
    image: postgres:16.2
    container_name: local-pg-16
    hostname: local-pg-16
    networks:
      - my-local-net
    ports:
      - 5432:5432
      - 8080:8080
    volumes:
      - tes-pg:/var/lib/postgresql/data
    environment:
      - POSTGRES_PASSWORD=${PGPASSWORD}
    # Make sure postgres is ready to accept connection as the indicator that 
    # Postgres is ready
    # ref https://www.postgresql.org/docs/current/app-pg-isready.html
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U ${PGUSER}"]
      # a more complete command
      # test: ["CMD-SHELL", "pg_isready -U ${PGUSER} -d ${PGDATABASE} -h 127.0.0.1"]
      interval: 2s
      timeout: 10s
      retries: 5
volumes:
  tes-pg:
    external: true

networks:
  # this will always create new network
  # my-local-net:
  #   driver: bridge
  my-local-net:
    name: mynet
    external: true

image: postgres:16.2 : I want to use the current latest postgres, which is 16.2. Make sure you specify the major version(16 in this case), as the default is the latest, and when the version change, it almost always has incompatibility

container_name: the name of the container to be built

hostname: address for this service in the docker network

networks: specify which network this service attached to, so when you want to access this service, you need to attach to this network and use hostname to connect to the database. If the network is not specified, docker compose will create new network and attach all services described in the yml file to it, so all services can communicate with each other. I use predefined network because it will be easier when other container outside the one described in this docker-compose.yml to connect to the services here

ports : mapping exposed port from container to host(the computer)

volume: persist the data created by postgre to disk. So when we restart the container, it can resume using data from volume instead of creating data. Other container can use this data too. Beware though when you use different version of postgres, it could result in corrupt data(I did it with different mysql version before lol)

environment: this will setup the postgres password when we want to enter the postgres. You could setup username ,password, and database name, the default is postgres or if the username is specified and database name is not, it would default to username. I read the documentation here on POSTGRES_DB section

healthcheck : is how I defined healthcheck that I will use in the application service(the condition on depends_on key in go-market-warehouse-api service). The depends_on is to tell the go-market-warehouse-api that it should start when the the database service is deemed to be healthy, which I define as the database ready to accept TCP connection. Why it is necessary? if I don't specify the depends_on the application will try to connect to db when db is not fully instantiated or ready or both, so the application would throw error. If I only specify depends_on: local-pg-16, docker would start creating the application service after the database service is not yet ready to accept TCP connection. The are use cases when you need to seed data to db before the db is ready to accept connection too. Hence I use the healthcheck function

volumes: I specify volume here to point it at the predefined volume that I execute before I create the docker-compose.yml. If external: true is not specified, docker-compose will create new volume that has the name of <project-name>_tes-pg, same as network

network: specify which network to be used. my-local-net here is the name, name: mynet is the network name on the docker, and external: true is same as the volume, it tells docker to use predefined network that exists on the docker(external of this docker-compose.yml)

Here's the full docker-compose.yml:

services:
  go-market-warehouse-api:
    depends_on:
      local-pg-16:
        condition: service_healthy
    # always restart when app crashes
    restart: always
    build:
      context: .
    image: go-market-warehouse-api:v1.0
    container_name: go-market-warehouse-api
    hostname: go-market-warehouse-api
    networks:
      - my-local-net
    ports:
      - 80:8080
    environment:
      - PGUSER=${PGUSER:-totoro}
      - PGPASSWORD=${PGPASSWORD:?database password not set}
      - PGHOST=${PGHOST:-db}
      - PGPORT=${PGPORT:-5432}
      - PGDATABASE=${PGDATABASE:-mydb}
    deploy:
      restart_policy:
        condition: on-failure
  local-pg-16:
    image: postgres:16.2
    container_name: local-pg-16
    hostname: local-pg-16
    networks:
      - my-local-net
    ports:
      - 5432:5432
      - 8080:8080
    volumes:
      - tes-pg:/var/lib/postgresql/data
    environment:
      - POSTGRES_PASSWORD=${PGPASSWORD}
    # Make sure postgres is ready to accept connection as the indicator that 
    # Postgres is ready
    # ref https://www.postgresql.org/docs/current/app-pg-isready.html
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U ${PGUSER}"]
      # a more complete command
      # test: ["CMD-SHELL", "pg_isready -U ${PGUSER} -d ${PGDATABASE} -h 127.0.0.1"]
      interval: 2s
      timeout: 10s
      retries: 5
volumes:
  tes-pg:
    external: true

networks:
  # this will always create new network
  # my-local-net:
  #   driver: bridge
  my-local-net:
    name: mynet
    external: true

On `healthcheck` and `depends_on`

The depends_on is to tell the go-market-warehouse-api that it should start when the the database service is deemed to be healthy, which I define as the database ready to accept TCP connection. Why it is necessary? If I only specify depends_on: local-pg-16, docker would start creating the application service after the database service is not yet ready to accept TCP connection. Try by modify depends_on: local-pg-16 then run the docker compose, if you see the log, sometimes the app throws error could not connect to database and database still printing out logs when application log started even though you have specify depends_on. Hence I use depends_on: condition: service_healthy to specify when the database is deemed to be ready is when the condition: service_healthy is fulfilled.

what is the criteria for service_healthy? it use the definition of healthcheck in local-pg-16.

Full `docker-compose.yml`

Here it is


services:
  go-market-warehouse-api:
    depends_on:
      local-pg-16:
        condition: service_healthy
    # always restart when app crashes
    restart: always
    build:
      context: .
    image: go-market-warehouse-api:v1.0
    container_name: go-market-warehouse-api
    hostname: go-market-warehouse-api
    networks:
      - my-local-net
    ports:
      - 80:8080
    environment:
      - PGUSER=${PGUSER:-totoro}
      - PGPASSWORD=${PGPASSWORD:?database password not set}
      - PGHOST=${PGHOST:-db}
      - PGPORT=${PGPORT:-5432}
      - PGDATABASE=${PGDATABASE:-mydb}
    deploy:
      restart_policy:
        condition: on-failure
  local-pg-16:
    image: postgres:16.2
    container_name: local-pg-16
    hostname: local-pg-16
    networks:
      - my-local-net
    ports:
      - 5432:5432
      - 8080:8080
    volumes:
      - tes-pg:/var/lib/postgresql/data
    environment:
      - POSTGRES_PASSWORD=${PGPASSWORD}
    # Make sure postgres is ready to accept connection as the indicator that 
    # Postgres is ready
    # ref https://www.postgresql.org/docs/current/app-pg-isready.html
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U ${PGUSER}"]
      # a more complete command
      # test: ["CMD-SHELL", "pg_isready -U ${PGUSER} -d ${PGDATABASE} -h 127.0.0.1"]
      interval: 2s
      timeout: 10s
      retries: 5
volumes:
  tes-pg:
    external: true

networks:
  # this will create new network based on project directory name
  # my-local-net:
  #   driver: bridge
  my-local-net: # kind of alias for this docker-compose.yml
    name: mynet # mynet is the name when you run `docker network ls`
    external: true

You can try the project by clone this repo’s commit. Read the description on how to run the project.

Note on network and volume

In many tutorial, network and volume is declared WITHOUT external like this:

volumes:
  tes-pg:
networks:
  my-local-net:

This will create a new network with default name <project-name>_tes-pg and <project-name>_my-local-net

When external: true, no additional network and volume like the one above:

These screenshots use lazygit btw.

That’s all for this article, hope you learn something new.