DEV Community: Koen Verburg

Set up a Multi-Stage Docker Build for Go Applications

Koen Verburg — Sun, 22 Aug 2021 11:13:14 +0000

We are going to create a simple multi-stage Docker build setup for Go applications. It might sound daunting at first, but it's actually really easy.

A multi-stage Docker build serves a few purposes. First, the final image size is smaller than traditional images, second, it's more secure.

Using dockers layering, we can abstract the building processes into multiple stages. And only copy the output that we need for the next stage. For example, the binary that gets created we're running go build -o main.

The goal here is to end up with a docker image that only contains and runs the final binary. What that final binary is, does not really matter in this context. But in this article, I'm going to use a simple script that outputs text to the console.

Requirements

We are going to need the following tooling, so make sure you installed it on your machine.

Go 1.x or higher
Docker
VSCode or any other editor

Let's write some code

Let's create the folder structure so that we have a good base to build from.
Make a directory called src and cd into that.

Now, let's initialize our go package with the following command go mod init hello-world. This will create a file called go.mod.

With that part done, let's write a simple Hello World in Go.
Create a file called main.go in our src directory and add the following code.

package main

import "fmt"

func main() {
  fmt.Println("Hello World!")
}

If you run this using, go run main.go the output would be Hello World!.
Great, with that part out of the way, we can start with diving into Docker!

Golang base image

Change directory (cd ..) to the project root, here we are going to create a dockerfile. Paste in the following code and then I will explain what we are doing.

FROM golang:1.16-alpine

# Create a workspace for the app
WORKDIR /app

# Download necessary Go modules
COPY src/go.mod .
RUN go mod download

# Copy over the source files
COPY src/*.go ./

# Build
RUN go build -o /main

ENTRYPOINT ["/main"]

Here we are going to do a few things, on the FROM line we are going to specify which base image we are going to use.
Setting a WORKDIR is like setting a special folder from which we are going to work.
Then we are to copy over our go.mod and download our packages if needed. And then we are going to copy over our source code.

After this we are ready to build our go application, we do that using, RUN go build -o /main.
Now that we have created a single binary, we can set that as our entrypoint when we run our docker image.

To build our image, we will use docker build . -t hello-world-golang1.16, this will build the image.
After the build has finished. We can run our image as a container using the command docker run hello-world-golang1.16 to test if this works.
And you should get an output like Hello World!.

Let's also check what the size is of our image. Using the command, docker images we can see the size.

REPOSITORY               TAG       IMAGE ID       CREATED              SIZE
hello-world-golang1.16   latest    68b776701cbe   About a minute ago   304MB

Wow, that is a heavy image! Coming in at 304 MB. Let's try to cut off some fat here.

During my research on using multi-stage Docker builds. I saw that the Docker documentation used a Debian base image. Because of that, I wanted to also include it here as well.

Let's replace our dockerfile that we created earlier with the following. Note here the two comments, Builder and Runner. The Builder part is for building the go application and nothing more. The same can be said for the Runner part because that's sole purpose is running our application.

The builder has golang:1.6-alpine as base image because it needs to have the Go binary in its path. Because Go compiles to a single binary, running it does not need the presence of Go. So we can use a base image from Debian for our final image that will run our application.

#
# Builder
#

FROM golang:1.16-alpine AS builder

# Create a workspace for the app
WORKDIR /app

# Download necessary Go modules
COPY src/go.mod .
RUN go mod download

# Copy over the source files
COPY src/*.go ./

# Build
RUN go build -o /main

#
# Runner
#

FROM gcr.io/distroless/base-debian10 AS runner

WORKDIR /

# Copy from builder the final binary
COPY --from=builder /main /main

USER nonroot:nonroot

ENTRYPOINT ["/main"]

I'm changing the build command, so we don't overwrite the image we created earlier. This way we can see what our changes are doing to the final docker image.

Build our new image using docker build . -t hello-world-debian. Let's see if this changed the behavior of the container docker run hello-world-debian. And you should get the same output as before: Hello World!.

Let's check size using docker images.

REPOSITORY               TAG       IMAGE ID       CREATED              SIZE
hello-world-debian       latest    7ad8a5965a06   About a minute ago   21.1MB
hello-world-golang1.16   latest    68b776701cbe   About a minute ago   304MB

Look at that, 21.1 MB, that is a significant reduction in size!

Awesome, but can we push it even more? Yes, we can!

Using the smallest docker base image that there is.
Which is scratch. This is as bare as it gets. In the runner part of the docker file, change out the FROMstatement to scratch like below.

#
# Runner
#

# FROM gcr.io/distroless/base-debian10 AS runner
FROM scratch AS runner

Build our new image using docker build . -t hello-world-scratch.
Let's see if this changed the behavior of the container docker run hello-world-scratch. And you should get the same output as before: Hello World!.

Let's check the size once more, using docker images.

REPOSITORY               TAG       IMAGE ID       CREATED              SIZE
hello-world-scratch      latest    eb41c9777973   58 seconds ago       1.94MB
hello-world-debian       latest    7ad8a5965a06   About a minute ago   21.1MB
hello-world-golang1.16   latest    68b776701cbe   About a minute ago   304MB

Would you look at that, 1.94 MB! That is perfect!

Wrapping it up

I hope I have inspired you to use multi-stage Docker builds for your next project. This is still a simple two-stage Docker build, you can even use more stages if you need or when your project asks for it.

Here is a few useful links and also the source code, so you can try it out for yourself.

Publish an Article To DEV.to With GO

Koen Verburg — Sun, 16 May 2021 20:46:09 +0000

We are going to create a program that allows us to post draft articles from Markdown to DEV.to with ease.

Requirements

We are going to need the following tooling so make sure it is installed on your machine.

Go 1.x or higher
VS Code or any other editor
Any terminal
A DEV.to account
A DEV.to API key

Getting Started

1. Scaffolding the project

First, let's set up the project structure so have a base to work on.

mkdir devAutomation && cd devAutomation

touch .env
touch main.go

go mod init devAutomation

echo "# Hello Dev.To
Lorem ipsum dolor sit amet, consectetur adipiscing elit." >> example.md

Here we are doing a few things, on the first line, we are creating a directory for our project and changing directory into that folder.

Secondly, we are creating a few files. The .env file is for the environment variables like the API key which we are getting in step 2. And the main.go is the starting point of our program.

Then we need to initialize the module using go mod init and lastly we need an example Markdown file to post to DEV.to.

2. Getting the API key

Let's get the API key go to this page and scroll down to the section named "DEV API Keys". Create a new API key and name accordingly.

Let's save the API key in the .env file we created in step 1.

DEV_TO_API_KEY='your-api-key'

3. Reading Markdown

Reading the Markdown file is quite easy. We are going to read the example markdown using ioutil.ReadFile and using fmt.Println we are for now printing the contents to the terminal.

func main() {
    file, err := ioutil.ReadFile("./example.md")
    if err != nil {
        fmt.Println(err)
    }

    fmt.Println(string(file))
}

You should have an output looking like the following:

# Hello Dev.To
Lorem ipsum dolor sit amet, consectetur adipiscing elit.

4. Creating the API Request

The API call is the hardest part of this tutorial. First, we need some types. These are needed to ensure the data we are posting to the API will get through correctly.

Using these types we can also see that what we can send over to the API.

type DevTo struct {
    Article Article `json:"article"`
}

type Article struct {
    Title        string   `json:"title"`
    Published    bool     `json:"published"`
    Series       string   `json:"series"`
    MainImage    string   `json:"main_image"`
    Canonical    string   `json:"canonical_url"`
    Description  string   `json:"description"`
    Tags         []string `json:"tags"`
    BodyMarkdown string   `json:"body_markdown"`
}

Let's create a new function called publish and create the request logic in there.
For the sake of this demo, I will split the example markdown file on a new line so we have a separate title and body for our article.

For more advanced articles please take a look at the Frontmatter options that DEV.to supports.

In this part, we will create the responseBody and add our data.
Then we create the payload and added it to the request.

After that, we are doing a few if checks to make sure we get the appropriate log message printed to the terminal when the API call fails, succeeds, or returns with another status code.

func request(body string) {
    url := "https://dev.to/api/articles"
    content := strings.Split(body, "\n")
    responseBody := DevTo{
        Article: Article{
            Published:    false, // this will ensure it's a draft
            Title:        content[0],
            BodyMarkdown: content[1],
        },
    }

    payload, _ := json.Marshal(responseBody)
    request, err := http.NewRequest("POST", url, bytes.NewBuffer(payload))

    request.Header.Add("api-key", apikey)
    request.Header.Add("Content-Type", "application/json")

    client := &http.Client{}
    response, err := client.Do(request)

    if err != nil {
        log.Fatalln(err)
    }
    defer response.Body.Close()

    if response.StatusCode == 200 || response.StatusCode == 201 {
        log.Println("Created new article on Dev.To")
    }

    if response.StatusCode != 200 && response.StatusCode != 201 {
        responseMessage, _ := ioutil.ReadAll(response.Body)
        log.Fatalln(string(responseMessage))
    }
}

5. Loading in the API key

We also need to set the API to authorize the post request. For this, we will need the godotenv module to read the .env file.

go get github.com/joho/godotenv

Here we have a utility function to get the API key. On the first line, we are reading the .env file and if there is an error we will print that. Otherwise, it's correctly read and we print a message that it's correctly loaded in.

func env(key string) string {
    err := godotenv.Load("./.env")
    if err != nil {
        log.Fatalf("Error loading .env file")
    }

    log.Println("Loaded the .env file")

    return os.Getenv(key)
}

Revisited the request function we created in step 4, and add the following line.
If you have kept a close eye you would have spotted that apikey wasn't declared.

// ... other code
func request(body string) {
    apikey := env("DEV_TO_API_KEY") // <-- this line
    url := "https://dev.to/api/articles"
    content := strings.Split(body, "\n")

    // ...rest of the function
}

6. Connecting all the pieces

Now go back to the main function we created in step 3. And add the following line.

func main() {
    file, err := ioutil.ReadFile("./example.md")
    if err != nil {
        fmt.Println(err)
    }

    request(string(file)) // <-- this line
}

This will bootstrap all the functions we created and pass along the contents of the markdown file.

If you run this in the terminal you should see the following output

18:51:36 Loaded the .env file
18:51:37 Created new article on Dev.To

Open up your DEV.to dashboard, and you should see a new entry be listed there.

Wrapping it up.

Congrats, you now have a program to upload your Markdown-based articles to Dev.to using Golang.

Here Is My List of Technologies I'm Learning in 2021

Koen Verburg — Tue, 23 Mar 2021 21:13:00 +0000

This year I want to set a list of topics that I want to learn for this year.
At the end of the year, I will reflect on how much of the lists I have achieved.

1. Functional Programming

A simple gist of Functional Programming is all about functions for immutable variables. In contrast to Object-Oriented languages, which are more about a fixed set of functions and modifying or adding new variables.

A year ago I had my first go at it when I worked with Elixir but now that I don't use it anymore, it has faded a bit. And because Functional Programming has always intrigued me. I want to pick it up again.

In this year I will experiment with the concept of Functional Programming, what it is and what makes it functional? And how do Functional Programming Languages like Elixir and Haskell use these practices?

I will go back to Elixir to learn Functional Programming, but also I want to use these practices in other languages like TypeScript, C#, and Go. Because I use those on a daily basis.

2. Golang

Golang is created by Google and it's getting more and more adoption by large companies and Open-Source Software Projects. Think about Docker and Kubernetes which are quite popular for what they are, but they are also written in Go.

A few months ago Github Released their Profile Readme's and a Hypetrain started, Everyone was creating and polishing their Profile Readme's.

I looked around and saw there were a few people that create an auto-generating readme based on a cron schedule. I wanted that as well and started to think about which language to use for this. I could write it Python or NodeJS but I wanted something completely new, So I settled for Go.

Since then I have written small CLI programs in Go. But I'm still scratching the surface of the language but during this year I will write most of my APIs, Web Scrappers, and CLI tools in Go.

3. Docker

Docker is a set of tools for developing, running, and shipping your applications.

It allows for a simple Command Line Interface. And it separates your application runtime from the infrastructure it is running on.

I'm familiar with Docker and have run my personal website on Docker in production for a long time. Because I also want to educate other people on Docker I have to dive into it a bit deeper. Besides that, I also want to learn about networking, routing, and load balancing.

During this year I'm going to write Go programs and run those in Docker. What those programs will be, I don't know yet. Maybe a Slackbot or a simple website using Go.

4. Terraform

Terraform is an open-source Infrastructure As Code tool that allows for managing the cloud by defining it as readable configuration files.

What I like about Terraform is that you just specify your cloud provider and you are ready to create and document your Infrastructure as code. With the documentation nearby you can scaffold your Infrastructure pretty fast.

That this time I have used Terraform for only one project at work and personal prototypes. But the exposure I got was enough for me to know that this is a great tool for me to use.

During this year I want to set up more projects in the cloud with Terraform, think about VM's, AKS resources, and full DTAP-Streets with traditional App Services in Azure.

5. Kubernetes

Kubernetes is a container orchestration system to manage, scale and deploy your applications.

Kubernetes is still a new topic for me and I'm still scratching the surface on what I can do with it. But I think this is the future of managing your apps in the cloud.

I want to make sure that every project I deliver will be reliable and can handle any load the visitors can throw at it during Black Friday or other campaigns.

I'm currently running a k3s cluster on a set of Raspberry Pis at home. I'm documenting my progress in this repository. (small disclaimer: I'm merely documenting it for myself do not expect documentation grade reading materials at this point in time)

During this year I will setup k8s from scratch on VM's and later I want to look at what AKS (Azure Kubernetes Service) brings to the table.

That's the list, thank you for reading! What are you learning this year?

Be sure to follow me on Medium, Instagram, and Twitter where you can follow along with the process.

Links of the topics I talked about