DEV Community: Isa

Golang: A simple dockerized app to demonstrate the use of Handle, Handler, HandleFunc and HandlerFunc.

Isa — Wed, 08 Sep 2021 11:57:01 +0000

When trying to understand how to build a web application in Go, it took me a while to understand the logic behind Handle, Handler, HandleFunc and HandlerFunc. I had a hard time grasping the logic, so I decided to write an article to make sure I understood everything correctly (which I hadn't until the middle of the writing process) and to help out others who may also be struggling.

To demonstrate the concepts of Handle, Handler, HandleFunc and HandlerFunc, we will write a very simple Go code inside of a container to serve a couple of web pages. But before checking the code, we need to have at least a general understanding of what are handlers in the context of the net/http package in Go.

What are handlers?

On the net/http package, handlers are used to handle HTTP requests. This package has capabilities of initiating an HTTP server and the handlers' role is to handle the HTTP requests by receiving them and then responding back. So if the server receives a request to access the path "/products", a handler will handle that request by responding back with the code corresponding to that path. Technically speaking, a Handler is an interface that has a method called ServeHTTP which has two parameters: a ResponseWriter type (an interface) and a pointer to a Request type (a struct). So any object that has this ServeHTTP method with that signature is a handler.

Let's create our first page using Handle

Now let's jump to our code! We will build a page with the path "/handlepage":

package main

import (
  "fmt"
  "net/http"
)

type HandlePage struct {}

func (h HandlePage) ServeHTTP(w http.ResponseWriter, r *http.Request) {
  w.Header().Set("Content-Type", "text/html")
  fmt.Fprint(w, "<h1>Welcome to the handle page!</h1>")
}

func main() {
  NewHandlePage := new(HandlePage)

  http.Handle("/handlepage", NewHandlePage)
  http.ListenAndServe(":3000", nil)
}

If you run the code above, and visit the page localhost:3000/handlepage, you'll see the message "Welcome to the handle page!". But what does this code mean? Let's take a look at the main steps.

Understanding the code

First, we import the necessary packages to serve the page. Then we create a struct of type HandlePage. Next, we add the ServeHTTP method to the HandlePage struct (remember that any object that contains a ServeHTTP method with a ResponseWriter type and a pointer to a Request type as its parameters is a handler!). So by adding the ServeHTTP method to the HandlePage struct we have transformed it into a handler.

Let's look at what's inside of the ServeHTTP method.

  w.Header().Set("Content-Type", "text/html")
  fmt.Fprint(w, "<h1>Welcome to the handle page!</h1>")

The goal of the code is to write a simple html text to the page. So we set the content type as text/html and then we use fmt.Fprint to insert the html text we want to write.

Next, we go to the main function. We start by initializing an instance of HandlePage and assign it to newHandlePage. Then we call the Handle method and pass two arguments: a string ("/handlepage") and a handler (newHandlePage). Let's take a closer look to what this handle method does.

This is the definition we find on the official Go documentation:

func Handle(pattern string, handler Handler)
// Handle registers the handler for the given pattern in the DefaultServeMux.

So the Handle function accepts two arguments: a String type and a Handler type. But what is this Handler type? We've mentioned it briefly earlier, but let's take a closer look and understand how we can create a handler from a struct and then use it inside a Handle function. According to the documentation, the Handler type is an interface that responds to an HTTP request:

type Handler interface {
  ServeHTTP(ResponseWriter, *Request)
}

That Handler interface has the ServeHTTP behavior, which is the same behavior that we added to the HandlePage struct at the beginning of our code, so we can say that the HandlePage type implements the ServeHTTP method. When a type has all the methods defined by an interface, then we can use that type to implement that interface. What does that mean in our case? It means that we can use a HandlePage type as the Handler type in the second argument of the Handle function. So it's like we could rewrite the Handle function like this:

func Handle(pattern string, handler HandlePage)

We can only make that replacement because our HandlePage type implements all the methods found on the Handler type (which is only the ServeHTTP method). A side note here: interfaces can be a little confusing, so if you're not familiar with it, my suggestion is that you take a look at this article from Digital Ocean, and then at this one from Jon Calhoun.

So just to recap, why did we add the ServeHTTP method to our HandlePage struct? So that we can use the HandlePage type as an implementation of the Handler interface.

What about the Handle function? The Handle function's job is to register the handler for the given pattern on the DefaultServeMux (more about this ahead).

Finally, ListenAndServe is executed using port 3000. ListenAndServe starts an HTTP server, and when the second parameter is nil, the DefaultServeMux is used. DefaultServeMux is an instance of ServeMux that is used as the default multiplexer. A multiplexer's job is to redirect a request to a handler.

One interesting thing to notice is that the type of the second parameter of ListenAndServe is a Handler type. Hence, if a multiplexer (DefaultServeMux) is being used in the second parameter, it means it must have a ServeHTTP method to enable it to act as a handler. So the DefaultServeMux object is nothing but a handler that redirects requests made to an URL to the registered handler.

Now let's try to get the big picture of what happens when http.Handle("/handlepage", NewHandlePage) and http.ListenAndServe(":3000", nil) are executed. When http.Handle("/handlepage", NewHandlePage)is executed, the NewHandlePage handler for the "/handlepage" path is registered in the DefaultServeMux. When http.ListenAndServe(":3000", nil) is executed, the HTTP server is started and the DefaultServeMux can then redirect requests according to what was registered on it (so far we only registered the relationship between the "/handlepage" path and the NewHandlePage handler). When the "/handlepage" path is visited, DefaultServeMux redirects the request to the NewHandlePage handler and the ServeHTTP method is called, executing the code inside of it. So in our case, when we visit http://localhost:3000/handlepage, the code inside the ServeHTTP method,

w.Header().Set("Content-Type", "text/html")
fmt.Fprint(w, "<h1>Welcome to the handle page!</h1>")

is executed and we see the "Welcome to the handle page!" written on the page.

Ok, so now we have a general understanding of what handle and handlers do and how they relate to each other: the first is a function that uses the second, an interface (or an object that implements that interface), as one of its parameters. So since a Handle uses a Handler as one of its parameters, can we infer that HandleFunc uses a HandlerFunc as one of its parameters too? Well, not necessarily. So what is the difference between a HandleFunc and a Handle and between a Handler and a HandlerFunc?

Let's continue with our example to have a better understanding and make a brief recap on what we've done so far to create our single page on "/handlepage" using only Handle and Handler:
1- Created a HandlePage struct;
2- Added the ServeHTTP method to the HandlePage struct;
3- Created an instance of HandlePage and name it NewHandlePage;
4- Executed http.Handle having NewHandlePage as its handler.

Now let's assume we want to create several pages. If we follow the logic above, we would need to create a struct for every page and then add the ServeHTTP method to each. HandleFunc and HandlerFunc can help us simplify this flow. But how? Let's see!

The second page: using HandleFunc

We will now create a different page, with a path "/handlefuncpage". Let's add to the previous code all the steps necessary to create and serve this new page, but now using HandleFunc instead of Handle. Here's how our new code looks like:

package main

import (
  "fmt"
  "net/http"
)

type HandlePage struct {}

func (h HandlePage) ServeHTTP(w http.ResponseWriter, r *http.Request) {
  w.Header().Set("Content-Type", "text/html")
  fmt.Fprint(w, "<h1>Welcome to the handle page!</h1>")
}

func HandleFuncPage(w http.ResponseWriter, r *http.Request) {
  w.Header().Set("Content-Type", "text/html")
  fmt.Fprint(w, "<h1>Welcome to the handlefunc page!</h1>")
}

func main() {
  NewHandlePage := new(HandlePage)

  http.Handle("/handlepage", NewHandlePage)
  http.HandleFunc("/handlefuncpage", HandleFuncPage)
  http.ListenAndServe(":3000", nil)
}

So which steps did we have to take to create the new page on "/handlefuncpage" path?
1- Created a HandleFuncPage function;
2- Executed http.HandleFunc having the HandleFuncPage function as its second parameter.

When comparing the steps made to create the page on "/handlepage" and the ones made to create the new page on "/handlefuncpage", we will see that we can replace the first's three steps by the second's first step: instead of i) creating a struct, ii) adding the ServeHTTP method to it and iii) creating an instance of the struct, all we had to do was to create a function having a ResponseWriter type and a pointer of an Request type as its parameters. But how is this simplification possible? Let's compare HandleFunc to Handle and understand the differences.

func Handle(pattern string, handler Handler)
func HandleFunc(pattern string, handler func(ResponseWriter, *Request))

When comparing Handle and HandleFunc, we see that the second parameter, the handler, is different: whereas in Handle it is a Handler type, in HandleFunc it is a function with a ResponseWriter type and a pointer of Request type as parameters (yes, the same signature of the ServeHTTP method). So this means that to use HandleFunc, we need to have a function with the proper parameters to make it work like a handler. Once this is done, HandleFunc, when executed, will convert this function into a handler and register it on the DefaultServeMux.

Right, but where does HandlerFunc enter after all? It's nowhere to be seen in our code! To unfold this mystery, we first need to understand what a HandlerFunc is.

What is `HandlerFunc` and where is it?

A HandlerFunc is a function type that can convert any function with the right signature into a handler. But what is the right signature the function must have? Yes, the same signature of the ServeHTTP method: a ResponseWriter type and a pointer to a Request type. So when we have a function with that signature, we can use HandlerFunc to convert this function into a handler, allowing us to use it as the second parameter of a Handle function, for example.

One thing to notice is that HandlerFunc converts the function to a HandlerFunc type, and not a Handler type. So how can we use this HandlerFunc type as a handler? Because a HandlerFunc type implements the ServeHTTP method, the same method that the Handler type implements. Remember the interface logic we mentioned earlier: when a type has all the methods defined by an interface, then we can use that type to implement that interface.

Let's check the ServeHTTP method added to a HandlerFunc type and see if there's anything interesting:

// The HandlerFunc type is an adapter to allow the use of
// ordinary functions as HTTP handlers. If f is a function
// with the appropriate signature, HandlerFunc(f) is a
// Handler that calls f.
type HandlerFunc func(ResponseWriter, *Request)

// ServeHTTP calls f(w, r).
func (f HandlerFunc) ServeHTTP(w ResponseWriter, r *Request) {
  f(w, r)

When we convert a function to a handler using HandlerFunc, a ServeHTTP method is added to it. Calling ServeHTTP on this new handler will simply call the underlying function (f(w, r)). Think of when we added the ServeHTTP method to the HandlePage struct: calling ServeHTTP on this struct simply executes the code we wrote inside of it. The ServeHTTP method inside of a HandlerFunc type was built to have the same effect: execute the underlying function and hence the code inside of it.

Being able to convert any function with the appropriate signature into a handler is quite handy 😃, but going back to our mystery, we don't see this HandlerFunc being used anywhere. When we call HandleFunc in our code using our HandleFuncPage function as its second parameter, everything works as expected, as if the function is already a handler. How can this be?

To understand what is happening behind the hood, we must check the source code of HandleFunc:

func (mux *ServeMux) HandleFunc(pattern string, handler func(ResponseWriter, *Request)) {
  if handler == nil {
    panic("http: nil handler")
  }
  mux.Handle(pattern, HandlerFunc(handler))
}


func HandleFunc(pattern string, handler func(ResponseWriter, *Request)) {
  DefaultServeMux.HandleFunc(pattern, handler)
}

Ah! Now we can see HandlerFunc in action! Notice in the last line of the first block that HandlerFunc is being used to convert the function called handler into an actual HandlerFunc type (HandlerFunc(handler)). It is very important to understand that whenever you see something like var nf http.Handler = http.HandlerFunc(notFound), it means that you are converting a function into a handler type (in this case, we are converting the function notFound into a handler). Some people who are not familiar with how HandlerFunc works might think that it is a function call having notFound as an argument and that it might return a different thing.

Going back to our code, that's why when we called http.HandleFunc("/handlefuncpage", HandleFuncPage) everything worked: Handlefunc converted the HandleFuncPage function into a handler under the hood.

Right, so we now understand what a HandlerFunc does and how it is used by HandleFunc to convert functions into handlers. In our case, we didn't have to use HandlerFunc explicitly, but there might be cases where we would want to convert our functions into handlers and then use them elsewhere. For example, if for some reason we wanted to keep using Handle instead of HandleFunc, we could have converted our function HandleFuncPage using HandlerFunc and then used Handler to register it on DefaultServeMux, like this:

//Convert HandleFuncPage function into a handler:
hfp := http.HandlerFunc(HandleFuncPage)
//Use the converted function as a handler using Handle:
http.handle("/handlefuncpage", hfp)

Remember that the Handle function doesn't convert functions to handlers, so that's why we need to convert the function to a handler and then pass it inside Handle. In fact, HandleFunc was a shortcut created to avoid having to explicitly convert functions into handlers.

The end

And that's it! I hope you enjoyed this article and that your understanding of all those handy stuff is now a little bit better.

You can find the code here. You can open it in a devcontainer if you are using VSCode.

Thanks!

Using GitHub Actions to build test workflows on a Rails on Docker app (+ Postgres and Selenium) leveraging Docker layer caching

Isa — Tue, 25 May 2021 18:46:03 +0000

Ever since I've heard about Github Actions (from now on mentioned as GHA), I've been wanting to try it out in one of my projects. Also, I've been eager to write tests, as in the code bootcamp I took there wasn't an opportunity to implement them during the final project. So given those two topics I wanted to learn more about, I decided to couple them together and experiment creating a GitHub Actions workflow for tests. Although having come across tutorials/articles about setting up GHA workflows for Rails apps, I didn't find any specific tutorials explaining how to set up a GHA workflow to test a Rails on Docker app. Hence, I thought if I wrote an article explaining my approach I could help out someone.

I'm assuming that those reading this article are already somehow familiar with GHA, Docker, and Rails, so I will focus on explaining the workflow setups.

In the simple Rails app used to build the workflow, there are three services that we need to run the tests: the app itself, the database (Postgres), and a tool that allows us to perform the system tests by simulating a browser (Selenium). Hence, we need to build three containers, one for each of those services.

I decided to try out two approaches for the workflow: one that uses GHA services containers and another one that builds those services based on Dockerfiles that we will define. In the end, my favorite approach ended up being the first one, as it is easier to set up and a little bit faster. Let's take a look at those approaches in detail.

The Services Containers Approach
Our goal in setting up this workflow is to test the app every time there is a push action performed against our repository. There are only a few tests in the app (one model test and a couple of system tests) as the main purpose of this tutorial is to show a basic setup to implement a GHA workflow to run tests on every push (and not the testing methodology itself). We are using Rail's default test framework, Minitest.
We will start by showing up the test-services.yml file, placed inside the .github/workflows folder.

Let's walk through each step of this configuration.

First, we define which action will trigger this workflow: in our case, the push action will do it. Then we set up some env variables related to the test environment and specify which kind of runner we want to use (Ubuntu).

name: Test

on: [push]

jobs:
  test:
    env:
      RAILS_ENV: test
      NODE_ENV: test
    runs-on: ubuntu-latest # runner
}

Next, we enter the services' part (which should be placed before the steps part). We must specify all the services we need to run our tests: in this case, Postgres and Selenium with the Chrome browser. By doing this, GHA will create one container for each of the services and a specific network so that the containers can communicate with each other. The network part is important for this setup because later we will need to connect to it to execute our tests. You can read more about GHA Services Containers and how to set them up here.

    services:
      database:
        image: postgres
        env:
          POSTGRES_PASSWORD: postgres
        ports:
          - 5432:5432
        options: >-
          --health-cmd pg_isready
          --health-interval 10s
          --health-timeout 5s
          --health-retries 5
      chrome:
        image: selenium/standalone-chrome-debug
        ports:
          - 4444:4444
          - 5900:5900
        volumes:
          - /dev/shm:/dev/shm
}

Now we go to the "steps" part of the configuration. The first step, named "Output services network", will store, under the variable services_network, the name of the network created for the services' containers. Later, on the step named "Run tests", we will access this variable and use it to connect our main container to the services container network. In the following step, "Checkout code", we use a GHA action called checkout that "checks-out your repository under $GITHUB_WORKSPACE, so your workflow can access it."

    steps:

    - name: Output services network
      id: network
      run: |
        echo ::set-output name=services_network::${{ job.container.network }}
        echo ${{ job.container.network }}

    - name: Checkout code
      uses: actions/checkout@v2

The next step relies on a Docker action that installs Docker Buildx in the runner used to run our workflow. Buildx is a CLI plugin that allows the use of extra features of the BuildKit builder toolkit. The main reason we're installing Buildx is to take advantage of Buildkit's ability to store Docker layers as artifacts, which will later let us speed up our workflow by caching the layers of our Docker builds. The install: true option will make Buildx the default builder in Docker. In the following step, "Prepare tags", we are, well, preparing the tags for the images we'll build/use.

    - name: Set up Docker Buildx
      uses: docker/setup-buildx-action@v1
      with:
        version: latest
        install: true

    - name: Prepare Tags
      id: tags
      run: |
        TAG=$(echo $GITHUB_SHA | head -c7)
        IMAGE="dev/test"
        echo ::set-output name=tagged_image::${IMAGE}:${TAG}
        echo ::set-output name=tag::${TAG}

Next, we use a Github Action that allows caching dependencies for a job. Basically, it lets us specify a path that will be used to cache and restore the dependencies. You can specify any folder you want, as long as it's empty or containing only Buildx data. In the following step, "Build Code", we will finally build our main image, which contains the app we want to test. We use Docker's build-push action, which, as the name implies, lets us build and push Docker images on GHA runners. In this step there are some important configurations, so let's go over each one of them.

We start with the push: false, meaning we are not interested in pushing this image anywhere as we just want it to be tested (based on the official documentation, this option is false by default, hence we could have it removed from the configuration and the result would be the same).

Next comes file: .ci-services/Dockerfile-services.ci, which tells the builder which Dockerfile to use for the build. The load: true option will load the build result to Docker images. If we don't specify this option, the execution of the "Run tests" step will fail because it will not find the image built on this step to execute the tests.

Next comes the two configurations that will allow the caching and caching retrieval for our builds: cache-from and cache-to. On cache-from we are specifying the cache type (in our case is local, but it could also be on a registry), and its path (src=/tmp/.buildx-main-cache). This path should seem familiar, as we set it up on the previous step "Cache main image layers". So here we are telling the builder to use the folder /tmp/.buildx-main-cache, specified in the previous step, to retrieve the cache.

As for the cache-to option, it will write the new cache generated by the build on another folder called /tmp/.buildx-main-cache-new. We could have used the same folder as in the cache-from option, but as this build-push action still doesn't offer a way to clear up the cache, we have to cache to a different location, remove the /tmp/.buildx-main-cache folder and rename the /tmp/.buildx-main-cache-new path to /tmp/.buildx-main-cache (this removal and renaming commands are executed in the last step, "Move cache"). This way, we can prevent caching from becoming huge.

Although we are using a regular Dockerfile in this tutorial (without multi-stage building), it is worth mentioning an important configuration that impacts multi-stage buildings: the mode=max option inside the cache-to option. The mode=max option allows the caching of all layers generated by a multi-stage Dockerfile, and not just the final layer. If you are using a multi-stage Dockerfile and you need to access intermediate layers, you have to set mode=max on the cache-to entry.

The last option is the tag, which will receive as input the tagged_image variable set on the "Prepare Tags" step.

    - name: Cache main image layers
      uses: actions/cache@v2
      with:
        path: /tmp/.buildx-main-cache
        key: ${{ runner.os }}-buildx-main-${{ github.sha }}
        restore-keys: |
          ${{ runner.os }}-buildx-main-

    - name: Build code
      uses: docker/build-push-action@v2
      with:
        push: false
        file: .ci-services/Dockerfile-services.ci
        load: true
        cache-from: type=local,src=/tmp/.buildx-main-cache
        cache-to: type=local,mode=max,dest=/tmp/.buildx-main-cache-new
        tags: ${{ steps.tags.outputs.tagged_image }}

Finally, we have everything set to run our tests! Now we can run our docker-compose file with some env variables to build a container from the image created on the "Build code" step and execute our test service. The TEST_IMAGE_TAG will store the name of the image we built on the "Build code" step and the SERVICES_NETWORK will store the name of the network created for the services' container. We also specify which docker-compose file will be used (.ci-services/docker-compose.test.services.yml) and the service we want to run (test).

    - name: Run Tests
      run: |
        TEST_IMAGE_TAG=${{ steps.tags.outputs.tagged_image }} SERVICES_NETWORK=${{ steps.network.outputs.services_network }} docker-compose -f .ci-services/docker-compose.test.services.yml run test

Now let's take a look at the docker-compose file we are using to understand how those env variables are used.

It's straightforward to see that we are using the TEST_IMAGE_TAG env variable to tell docker-compose which image to use to build the test service. Inside the test service, we are also specifying two env variables: HUB_URL and PARALLEL_WORKERS. The first is a configuration related to Selenium and it is used on the application_system_test_case.rb file to tell Rails where to find the Chrome executable that will be used on system tests. Rails will then know that it should find and execute the Chrome browser located in the chrome container (defined on our services step), and not locally.

The second env variable, PARALLEL_WORKERS, just states that there should be only one "worker" to run the tests, which means the tests will run sequentially instead of simultaneously. Obviously, this isn't the most efficient setup, but creating more "workers" to run tests would mean launching multiple chrome containers, which would make our workflow a little bit more complicated. This article should help if you want to know more about how to scale the chrome service to run tests in parallel.

Going back to the docker-compose file, after setting env variables, the test service will execute a command to run Rails tests (rails test && rails test:system). At last we can see how the SERVICES_NETWORK env variable is used: it tells the test service container to connect to a specific network, the one created for the services' containers at the beginning of our workflow. Being connected to this network allows the test service container to access the database and the chrome service containers to run the tests.

The Dockerfile Containers Approach
I won't get into a lot of details about this approach as it has a lot of similarities with the previous approach. Same as before, the goal here is to test the app every time there is a push action performed against our repository. The main difference is that instead of using GHA service containers, we will be using Docker's build-push action to build those containers. So for this approach, we have the following test.yml placed on the .github/workflows folder:

Given that we are now building our two service containers from the scratch instead of using GHA service containers, the test.yml file becomes a little bit more extensive. The same steps we only had to perform for our main service in the previous approach now have to be performed for the postgres and chrome services as well: prepare the caching, build the images and move the cache.

Besides, we also need to have two additional Dockerfiles: one for the postgres service and one for the chrome service. The docker-compose file is also a little bit different: it now has to use env variables to refer to the images built for the postgres and chrome services to execute the tests. Here's how it looks:

Given that docker-compose is building all the services together and automatically creating a network for all of them, we don't need to worry about connecting to a specific network to make things work as we did in the first approach.

Wrapping up
As mentioned before, I prefer using the first approach (using GHA service containers) as is is more simple and a bit faster to execute. Anyway, I thought it would be worth mentioning the second approach (build the service containers with the build-push action) because it works and it was the first approach that I could implement.

You can find the repository with those two workflows here. It has a .devcontainer folder, so you can use Vscode and open the code inside a container.

I'm far from being an expert on GHA, Rails, or Docker, so please let me know if you think I can improve this workflow and/or if I wrote something wrong. 😃

I'd also like to mention this excellent article that helped me to set up my workflow and understand how to leverage Docker layer caching on GHA.

Thank you for reading this!