DEV Community: Julien Andrieux

Go tools & GitLab — how to do Continuous Integration like a boss

Julien Andrieux — Tue, 07 Nov 2017 11:48:08 +0000

This was originally posted on Medium

Go tools & GitLab — how to do Continuous Integration like a boss

At Pantomath, we use GitLab for all our development work. The purpose of this paper is not to present GitLab and all its features, but to introduce how we use these tools to ease our lives.

So what is it all about? To automate everything that is related to your development project, and let you focus on your code. We’ll cover the lint, unit tests, data race, memory sanitizer, code coverage, and build.

All the code shown in this post is available at https://gitlab.com/pantomath-io/demo-tools. So feel free to get the repository, and use the tags to navigate in it. The repository should be placed in the src folder of your $GOPATH:

$ go get -v -d gitlab.com/pantomath-io/demo-tools
$ cd $GOPATH/src/gitlab.com/pantomath-io/demo-tools

Go tools

Luckily, Go comes with a lot of useful tools, to build, test and check your code. In fact, it’s all there. We’ll just add extra tools to glue them together. But before we go there, we need to take them one by one, and see what they do.

Package list

Your go project is a collection of packages, as described in the official doc. Most of the following tools will be fed with these packages, and thus the first command we need is a way to list the packages. Hopefully, go covers our back with the list subcommand (read the fine manual and this excellent post from Dave Cheney):

$ go list ./..

Note that we want to avoid applying our tools on external ressources, and restrict it to our code. So we need to get rid of the vendor directories:

$ go list ./... | grep -v /vendor/

Lint

This is the very first tool we use on the code: the linter. Its role is to make sure that the code respects the code style. This may sounds like an optional tool, or at least a “nice-to-have but it really helps to keep consistent style over your project.

This linter is not part of go per se, so you need to grab it and install it by hand (see official doc).

The usage is fairly simple: you just run it on the packages of your code (you can also point the .go files):

$ golint -set_exit_status $(go list ./... | grep -v /vendor/)

Note the -set_exit_status option. By default, golint only prints the style issues, and returns (with a 0 return code), so the CI never considers something went wrong. If you specify the -set_exit_status, the return code from golint will be different from 0 if any style issue is encountered.

Unit test

These are the most common tests you can run on your code. For each .go file, we need to have an associated _test.go file holding the unit tests. You can run the tests for all the packages with the following command:

$ go test -short $(go list ./... | grep -v /vendor/)

Data race

This is usually a hard subject to cover, but the go tool has it by default (but only available on linux/amd64, freebsd/amd64, darwin/amd64 and windows/amd64). For more information about data race, see this article. Meanwhile, here is how to run it:

$ go test -race -short $(go list ./... | grep -v /vendor/)

Memory sanitizer

Clang has a nice detector for uninitialized reads called MemorySanitizer. The go test tool is kind enough to interact with this Clang module (as soon as you are on linux/amd64 host and using a recent version of Clang/LLVM (>=3.8.0). This command is how to run it:

$ go test -msan -short $(go list ./... | grep -v /vendor/)

Code coverage

This is also a must have to evaluate the health of your code, and see what the part of code is under unit tests and what part is not. Rob Pike wrote a full post on that very subject.

To calculate the code coverage ratio, we need to run the following script:

$ PKG_LIST=$(go list ./... | grep -v /vendor/)
$ for package in ${PKG_LIST}; do
    go test -covermode=count -coverprofile "cover/${package##*/}.cov" "$package" ;
done
$ tail -q -n +2 cover/*.cov >> cover/coverage.cov
$ go tool cover -func=cover/coverage.cov

If we want to get the coverage report in HTML format, we need to add the following command:

$ go tool cover -html=cover/coverage.cov -o coverage.html

Build

Last but not least, once the code has been fully tested, we might want to compile it to make sur we can build a working binary.

$ go build -i -v gitlab.com/pantomath-io/demo-tools

Makefile

git tag: init-makefile

Photo by Matt Artz on Unsplash

Now we have all the tools that we may use in the context of Continuous Integration, we can wrap them all in a Makefile, and have a consistent way to call them.

The purpose of this doc is not to present make, but you can refer to official documentation to learn more about it.

PROJECT_NAME := "demo-tools"
PKG := "gitlab.com/pantomath-io/$(PROJECT_NAME)"
PKG_LIST := $(shell go list ${PKG}/... | grep -v /vendor/)
GO_FILES := $(shell find . -name '*.go' | grep -v /vendor/ | grep -v _test.go)

.PHONY: all dep build clean test coverage coverhtml lint

all: build

lint: ## Lint the files
  @golint -set_exit_status ${PKG_LIST}

test: ## Run unittests
  @go test -short ${PKG_LIST}

race: dep ## Run data race detector
  @go test -race -short ${PKG_LIST}

msan: dep ## Run memory sanitizer
  @go test -msan -short ${PKG_LIST}

coverage: ## Generate global code coverage report
  ./tools/coverage.sh;

coverhtml: ## Generate global code coverage report in HTML
  ./tools/coverage.sh html;

dep: ## Get the dependencies
  @go get -v -d ./...

build: dep ## Build the binary file
  @go build -i -v $(PKG)

clean: ## Remove previous build
  @rm -f $(PROJECT_NAME)

help: ## Display this help screen
  @grep -h -E '^[a-zA-Z_-]+:.*?## .*$$' $(MAKEFILE_LIST) | awk 'BEGIN {FS = ":.*?## "}; {printf "\033[36m%-30s\033[0m %s\n", $$1, $$2}'

What do we have, now? One target for any tool previously presented, and 3 more targets for:

installation of dependencies (dep);
housekeeping of the project (clean);
some nice and shiny help (help).

Note that we also had to create a script for the code coverage work. This is because implementing loops over files in a Makefile is a pain. So the work is done in a bash script, and the Makefile only triggers this script.

You can try the Makefile with the following commands:

$ make help
$ make lint
$ make coverage

Continuous Integration

git tag: init-ci

Photo by Max Panamá on Unsplash

Now the tools are in place, and we can run various tests on our code, we’d like to automate these, on your repository. Luckily, GitLab offers CI pipelines just for this. And the setup for this is pretty straightforward: all you create is a .gitlab-ci.yml file at the root of the repository.

The full documentation on this Yaml file presents all the options, but you can start with this .gitlab-ci.yml:

image: golang:1.9

cache:
  paths:
    - /apt-cache
    - /go/src/github.com
    - /go/src/golang.org
    - /go/src/google.golang.org
    - /go/src/gopkg.in

stages:
  - test
  - build

before_script:
  - mkdir -p /go/src/gitlab.com/pantomath-io /go/src/_/builds
  - cp -r $CI_PROJECT_DIR /go/src/gitlab.com/pantomath-io/pantomath
  - ln -s /go/src/gitlab.com/pantomath-io /go/src/_/builds/pantomath-io
  - make dep

unit_tests:
  stage: test
  script:
    - make test

race_detector:
  stage: test
  script:
    - make race

memory_sanitizer:
  stage: test
  script:
    - make msan

code_coverage:
  stage: test
  script:
    - make coverage

code_coverage_report:
  stage: test
  script:
    - make coverhtml
  only:
  - master

lint_code:
  stage: test
  script:
    - make lint

build:
  stage: build
  script:
    - make

If you break down the file, here are some explanations on its content:

The first thing is to choose what Docker image will be used to run the CI. Head to the Docker Hub to choose the right image for your project.
Then, you specify some folders of this image to be cached. The goal here is to avoid downloading the same content several times. Once a job is completed, the listed paths will be archived, and next job will use the same archive.
You define the different stages that will group your jobs. In our case, we have two stages (to be processed in that order): test and build. We could have other stages, such as deploy.
The before_script section defines the commands to run in the Docker container right before the job is actually done. In our context, the commands just copy or link the repository deployed in the $GOPATH, and install dependencies.
Then come the actual jobs, using the Makefile targets. Note the special case for code_coverage_report where execution is restricted to the master branch (we don’t want to update the code coverage report from feature branches for instance).

As we commit/push the .gitlab-ci.yml file in the repository, the CI is automatically triggered. And the pipeline fails. How come?

The lint_code job fails because it can’t find the golint binary:

$ make lint
make: golint: Command not found
Makefile:11: recipe for target 'lint' failed
make: *** [lint] Error 127

So, update your Makefile to install golint as part of the dep target.

The memory_sanitizer job fails because gcc complains:

$ make msan
# runtime/cgo
gcc: error: unrecognized argument to -fsanitize= option: 'memory'
Makefile:20: recipe for target 'msan' failed
make: *** [msan] Error 2

But remember we need to use Clang/LLVM >=3.8.0 to enjoy the -msan option in go test command.

We have 2 options here:

either we setup Clang in the job (using before_script);
or we use a Docker image with Clang installed by default.

The first option is nice, but that implies to have this setup done for every single job. This is going to be so long, we should do it once and for all. So we prefer the second option, which is a good way to play with GitLab Registry.

git tag: use-own-docker

We need to create a Dockerfile for the container (as usual: read the official documentation for more options about it):

# Base image: 
FROM golang:1.9
MAINTAINER Julien Andrieux <julien@pantomath.io>

# Install golint
ENV GOPATH /go
ENV PATH ${GOPATH}/bin:$PATH
RUN go get -u github.com/golang/lint/golint

# Add apt key for LLVM repository
RUN wget -O - 
 | apt-key add -

# Add LLVM apt repository
RUN echo "deb 
 llvm-toolchain-stretch-5.0 main" | tee -a /etc/apt/sources.list

# Install clang from LLVM repository
RUN apt-get update && apt-get install -y --no-install-recommends \
    clang-5.0 \
    && apt-get clean \
    && rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*

# Set Clang as default CC
ENV set_clang /etc/profile.d/set-clang-cc.sh
RUN echo "export CC=clang-5.0" | tee -a ${set_clang} && chmod a+x ${set_clang}

The container built out of this Dockerfile will be based on golang:1.9 image (the one referenced in the .gitlab-ci.yml file).

While we’re at it, we install golint in the container, so we have it available. Then we follow official way of installing Clang 5.0 from LLVM repository.

Now we have the Dockerfile in place, we need to build the container image and make it available for GitLab:

$ docker login registry.gitlab.com
$ docker build -t registry.gitlab.com/pantomath-io/demo-tools .
$ docker push registry.gitlab.com/pantomath-io/demo-tools

The first command connects you to the GitLab Registry. Then you build the container image described in the Dockerfile. And finally, you push it to the GitLab Registry.

Take a look at the Registry for your repository, you’ll see your image, ready to be used. And to have the CI using your image, you just need to update the .gitlab-ci.yml file:

image: golang:1.9

becomes

image: registry.gitlab.com/pantomath-io/demo-tools:latest

One last detail: you need to tell the CI to use the proper compiler (i.e. the CC environment variable), so we add the variable initialization in the .gitlab-ci.yml file:

export CC=clang-5.0

Once the modification are done, next commit will trigger the pipeline, which now works:

https://gitlab.com/pantomath-io/demo-tools/pipelines/13497136

Badges

git tag: init-badges

Photo by Jakob Owens on Unsplash

Now the tools are in place, every commit will launch a test suite, and you probably want to show it, and that’s legitimate :) The best way to do so is to use badges, and the best place for it is the README file.

Edit it and add the 4 following badges:

Build Status: the status of the last pipeline on the master branch:

[![Build Status](https://gitlab.com/pantomath-io/demo-tools/badges/master/build.svg)](https://gitlab.com/pantomath-io/demo-tools/commits/master)

Coverage Report: the percentage of code covered by tests
Go Report Card:
License:

The coverage report needs a special configuration. You need to tell GitLab how to get that information, considering that there is a job in the CI that displays it when it runs.
There is a configuration to provide GitLab with a regexp, used in any job’ output. If the regexp matches, GitLab consider the match to be the code coverage result.

So head to Settings > CI/CD in your repository, scroll down to the Test coverage parsing setting in the General pipelines settings section, and use the following regexp:

total:\s+\(statements\)\s+(\d+.\d+\%)

You’re all set! Head to the overview of your repository, and look at your README:

Conclusion

What’s next? Probably more tests in you CI. You can also look at the CD (Continuous Deployment) to automate the deployment of your builds. The documentation can be done using GoDoc. Note that you generate a coverage report with the code_coverage_report, but don’t use it in the CI. You can make the job copy the HTML file to a web server, using scp (see this documentation on how to use SSH keys).

Many thanks to Charles Francoise who co-wrote this paper and https://gitlab.com/pantomath-io/demo-tools.

We are currently working on Pantomath. Pantomath is a modern, open-source monitoring solution, built for performance, that bridges the gaps across all levels of your company. The well-being of your infrastructure is everyone’s business. Keep up with the project

How we use gRPC to build a client/server system in Go

Julien Andrieux — Tue, 10 Oct 2017 09:50:59 +0000

This was originally posted on Medium

How we use gRPC to build a client/server system in Go

Photo by Igor Ovsyannykov on Unsplash

This post is a technical presentation on how we use gRPC (and Protobuf) to build a robust client/server system.

I won’t go into the details of why we chose gRPC as the main communication protocol between our client and server, a lot of great posts already cover the subject (like this one, this one or this one).
But just to get the big picture: we are building a client-server system, using Go, that needs to be fast, reliable, and that scales (thus we chose gRPC). We wanted the communication between the client and the server to be as small as possible, as secure as possible and fully compliant with both ends (thus we chose Protobuf).

We also wanted to expose another interface on the server side, just in case we add a client that has no compatible gRPC implementation: a traditional REST interface. And we wanted that at (almost) no cost.

Context

We will start building a very simple client/server system, in Go, that will just exchange dummy messages. Once both communicate and understand each other, we’ll add extra features, such as TLS support, authentication, and a REST API.

The rest of this articles assumes you understand basic Go programming. It also assumes that you have protobuf package installed, and the protoc command available (once again, many posts cover that topic, and there’s the official documentation).

You will also need to install Go dependencies, such as the go implementation for protobuf, and grpc-gateway.

All the code shown in this post is available at https://gitlab.com/pantomath-io/demo-grpc. So feel free to get the repository, and use the tags to navigate in it. The repository should be placed in the src folder of your $GOPATH:

$ go get -v -d gitlab.com/pantomath-io/demo-grpc
$ cd $GOPATH/src/gitlab.com/pantomath-io/demo-grpc

Defining the protocol

git tag: init-protobuf-definition

Photo by Mark Rasmuson on Unsplash

First of all, you need to define the protocol, i.e. to define what can be said between client and server, and how. This is where Protobuf comes into play. It allows you to define two things: Services and Messages. A service is a collection of actions the server can perform at the client’s request, a message is the content of this request. To simplify, you can say that service defines actions, while message defines objects.

Write the following in api/api.proto:

syntax = "proto3";
package api;

message PingMessage {
  string greeting = 1;
}

service Ping {
  rpc SayHello(PingMessage) returns (PingMessage) {}
}

So, you defined 2 things: a service called Ping that exposes a function called SayHello with an incoming PingMessage and returns a PingMessage ; and a message called PingMessage that consists in a single field called greeting which is a string.
You also specified that you are using the proto3 syntax, as opposed to proto2(see documentation).

This file is not usable like this: it needs to get compiled. Compiling the proto file means generating code for your chosen language, that your application will actually call.
In a shell, cd to the root directory of your project, and run the following command:

$ protoc -I api/ \
    -I${GOPATH}/src \
    --go_out=plugins=grpc:api \
    api/api.proto

This command generates the file api/api.pb.go, a Go source file that implements the gRPC code your application will use. You can look at it, but you shouldn’t change it (as it will be overwritten every time you run protoc).

You also need to define the function called by the service Ping, so create a file named api/handler.go:

package api

import (
  "log"

  "golang.org/x/net/context"
)

// Server represents the gRPC server
type Server struct {
}

// SayHello generates response to a Ping request
func (s *Server) SayHello(ctx context.Context, in *PingMessage) (*PingMessage, error) {
  log.Printf("Receive message %s", in.Greeting)
  return &PingMessage{Greeting: "bar"}, nil
}

the Server struct is just an abstraction of the server. It allows to “attach some resources to your server, making them available during the RPC calls;
the SayHello function is the one defined in the Protobuf file, as the rpc call for the Ping service. If you don’t define it, you won’t be able to create the gRPC server;
SayHello takes a PingMessage as parameter, and returns a PingMessage. The PingMessage struct is defined in the api.pb.go file auto-generated from the api.proto definition. The function also has a Context parameter (see further presentation in the official blog post). You’ll see later what use you can do of the Context. On the other side, it also returns an error, in case something bad happens.

Creating the simplest server

git tag: init-server

Photo by Nathan Dumlao on Unsplash

Now you have a protocol in place, you can create a simple server that implements the service and understands the message. Take your favorite editor and create the file server/main.go:

package main

import (
  "fmt"
  "log"
  "net"

  "gitlab.com/pantomath-io/demo-grpc/api"
  "google.golang.org/grpc"
)

// main start a gRPC server and waits for connection
func main() {
  // create a listener on TCP port 7777
  lis, err := net.Listen("tcp", fmt.Sprintf(":%d", 7777))
  if err != nil {
    log.Fatalf("failed to listen: %v", err)
  }

  // create a server instance
  s := api.Server{}

  // create a gRPC server object
  grpcServer := grpc.NewServer()

  // attach the Ping service to the server
  api.RegisterPingServer(grpcServer, &s)

  // start the server
  if err := grpcServer.Serve(lis); err != nil {
    log.Fatalf("failed to serve: %s", err)
  }
}

Let me break down to code to make it clearer:

note that you import the api package, so that the Protobuf service handlers and the Server struct are available;
the main function starts by creating a TCP listener on the port you want to bind your gRPC server to;
then the rest is pretty straight forward: you create an instance of your Server, create an instance of a gRPC server, register the service, and start the gRPC server.

You can compile your code to get a server binary:

$ go build -i -v -o bin/server gitlab.com/pantomath-io/demo-grpc/server

Creating the simplest client

git tag: init-client

Photo by Clem Onojeghuo on Unsplash

The client also imports the api package, so that the message and the service are available. So create the file client/main.go:

package main

import (
  "log"

  "gitlab.com/pantomath-io/demo-grpc/api"
  "golang.org/x/net/context"
  "google.golang.org/grpc"
)

func main() {
  var conn *grpc.ClientConn

  conn, err := grpc.Dial(":7777", grpc.WithInsecure())
  if err != nil {
    log.Fatalf("did not connect: %s", err)
  }
  defer conn.Close()

  c := api.NewPingClient(conn)

  response, err := c.SayHello(context.Background(), &api.PingMessage{Greeting: "foo"})
  if err != nil {
    log.Fatalf("Error when calling SayHello: %s", err)
  }
  log.Printf("Response from server: %s", response.Greeting)
}

Once again, the break down is pretty straight forward:

the main function instantiates a client connection, on the TCP port the server is bound to;
note the defer call to properly close the connection when the function returns;
the c variable is a client for the the Ping service, that calls the SayHello function, passing a PingMessage to it.

You can compile your code to get a client binary:

$ go build -i -v -o bin/client gitlab.com/pantomath-io/demo-grpc/client

Make them talk

You’ve just built a client and a server, so fire them in two terminals to test them:

$ bin/server
2006/01/02 15:04:05 Receive message foo

$ bin/client
2006/01/02 15:04:05 Response from server: bar

Tool to ease your life

git tag: init-makefile

Now the API, the client and the server are working, you may prefer to have a Makefile to compile the code, clean your folder, manage dependencies, etc.

So create this Makefile at the root of the project folder. Explaining this file is beyond the scope of this post, and it mostly uses compile command you already spawned previously.

To use the Makefile , try calling the following:

$ make help
api                            Auto-generate grpc go sources
build_client                   Build the binary file for client
build_server                   Build the binary file for server
clean                          Remove previous builds
dep                            Get the dependencies
help                           Display this help screen

Secure the communication

git tag: add-ssl

Photo by Nathaniel Tetteh on Unsplash

The client and the servers talk to each other, over HTTP/2 (transport layer on gRPC). The messages are binary data(thanks to Protobuf), but the communication is in plaintext. Fortunately, gRPC has SSL/TLS integration, that can be used to authenticate the server (from the client’s perspective), and to encrypt message exchanges.

You don’t need to change anything to the protocol: it remains the same. The changes take place in the gRPC object creation, on both client and server side. Note that if you change only one side, the connection won’t work.

Before you change anything in the code, you need to create a self-signed SSL certificate. The purpose of this post is not to explain how to do that, but the official OpenSSL documentation (genrsa, req, x509) can answer your question about it (DigitalOcean also has a nice and complete tutorial about it). Meanwhile, you can just use the files provided in the cert folder. The following commands have been used to generate the files:

$ openssl genrsa -out cert/server.key 2048
$ openssl req -new -x509 -sha256 -key cert/server.key -out cert/server.crt -days 3650
$ openssl req -new -sha256 -key cert/server.key -out cert/server.csr
$ openssl x509 -req -sha256 -in cert/server.csr -signkey cert/server.key -out cert/server.crt -days 3650

You can proceed and update the server definition to use the certificate and the key:

package main

import (
  "fmt"
  "log"
  "net"

  "gitlab.com/pantomath-io/demo-grpc/api"
  "google.golang.org/grpc"
  "google.golang.org/grpc/credentials"
)

// main starts a gRPC server and waits for connection
func main() {
  // create a listener on TCP port 7777
  lis, err := net.Listen("tcp", fmt.Sprintf("%s:%d", "localhost", 7777))
  if err != nil {
    log.Fatalf("failed to listen: %v", err)
  }

  // create a server instance
  s := api.Server{}

  // Create the TLS credentials
  creds, err := credentials.NewServerTLSFromFile("cert/server.crt", "cert/server.key")
  if err != nil {
    log.Fatalf("could not load TLS keys: %s", err)
  }

  // Create an array of gRPC options with the credentials
  opts := []grpc.ServerOption{grpc.Creds(creds)}

  // create a gRPC server object
  grpcServer := grpc.NewServer(opts...)

  // attach the Ping service to the server
  api.RegisterPingServer(grpcServer, &s)

  // start the server
  if err := grpcServer.Serve(lis); err != nil {
    log.Fatalf("failed to serve: %s", err)
  }
}

So what changed?

you created a credentials object (called creds) from your certificate and key files;
you created a grpc.ServerOption array and placed your credentials object in it;
when creating the grpc server, you provided the constructor with you array of grpc.ServerOption;
you must have noticed that you need to precisely specify the IP you bind your server to, so that the IP matches the FQDN used in the certificate.

Note that grpc.NewServer() is a variadic function, so you can pass it any number of trailing arguments. You created an array of options so that we can add other options later on.

If you compile your server now, and use the client you already have, the connection won’t work, and both sides will throw an error.

the server report the client is not handshaking with TLS:

2006/01/02 15:04:05 grpc: Server.Serve failed to complete security handshake from "localhost:64018": tls: first record does not look like a TLS handshake

the client has its connection closed before it can do anything:

2006/01/02 15:04:05 transport: http2Client.notifyError got notified that the client transport was broken read tcp localhost:64018->127.0.0.1:7777: read: connection reset by peer.
2006/01/02 15:04:05 Error when calling SayHello: rpc error: code = Internal desc = transport is closing

You need to use the exact same certificate file on the client side. So edit the client/main.go file:

package main

import (
  "log"

  "gitlab.com/pantomath-io/demo-grpc/api"
  "golang.org/x/net/context"
  "google.golang.org/grpc"
  "google.golang.org/grpc/credentials"
)

func main() {
  var conn *grpc.ClientConn

// Create the client TLS credentials
  creds, err := credentials.NewClientTLSFromFile("cert/server.crt", "")
  if err != nil {
    log.Fatalf("could not load tls cert: %s", err)
  }

  // Initiate a connection with the server
  conn, err = grpc.Dial("localhost:7777", grpc.WithTransportCredentials(creds))
  if err != nil {
    log.Fatalf("did not connect: %s", err)
  }
  defer conn.Close()

  c := api.NewPingClient(conn)

  response, err := c.SayHello(context.Background(), &api.PingMessage{Greeting: "foo"})
  if err != nil {
    log.Fatalf("error when calling SayHello: %s", err)
  }
  log.Printf("Response from server: %s", response.Greeting)
}

The changes on the client side are pretty much the same as on the server:

you created a credentials object with the certificate file. Note that the client do not use the certificate key, the key is private to the server;
you added an option to the grpc.Dial() function, using your credentials object. Note that the grpc.Dial() function is also a variadic function, so it accepts any number of options;
same server note applies for the client: you need to use the same FQDN to connect to the server as the one used in the certificate, or the transport authentication handshake will fail.

Both sides use credentials, so they should be able to talk just as before, but in an encrypted way. You can compile the code:

$ make

And run both sides in separate terminals:

$ bin/server
2006/01/02 15:04:05 Receive message foo

$ bin/client
2006/01/02 15:04:05 Response from server: bar

Identify the client

git tag: add-auth

Photo by chuttersnap on Unsplash

Another interesting feature of the gRPC server is the ability to intercept a request from the client. The client can inject information on the transport layer. You can use that feature to identify your client, because the SSL implementation authenticates the server (via the certificate), but not the client (all your clients are using the same certificate).

So you’ll update the client side to inject metadata on every call (like a login and password), and the server side to check these credentials for every incoming call.

On the client side, you just need to specify a DialOption on your grpc.Dial() call. But that DialOptionhas some constraints. Edit your client/main.go file:

package main

import (
  "log"

  "gitlab.com/pantomath-io/demo-grpc/api"
  "golang.org/x/net/context"
  "google.golang.org/grpc"
  "google.golang.org/grpc/credentials"
)

// Authentication holds the login/password
type Authentication struct {
  Login    string
  Password string
}

// GetRequestMetadata gets the current request metadata
func (a *Authentication) GetRequestMetadata(context.Context, ...string) (map[string]string, error) {
  return map[string]string{
    "login":    a.Login,
    "password": a.Password,
  }, nil
}

// RequireTransportSecurity indicates whether the credentials requires transport security
func (a *Authentication) RequireTransportSecurity() bool {
  return true
}

func main() {
  var conn *grpc.ClientConn

// Create the client TLS credentials
  creds, err := credentials.NewClientTLSFromFile("cert/server.crt", "")
  if err != nil {
    log.Fatalf("could not load tls cert: %s", err)
 }

  // Setup the login/pass
  auth := Authentication{
    Login:    "john",
    Password: "doe",
  }

  // Initiate a connection with the server
  conn, err = grpc.Dial("localhost:7777", grpc.WithTransportCredentials(creds), grpc.WithPerRPCCredentials(&auth))
  if err != nil {
    log.Fatalf("did not connect: %s", err)
   }
   defer conn.Close()

  c := api.NewPingClient(conn)

  response, err := c.SayHello(context.Background(), &api.PingMessage{Greeting: "foo"})
  if err != nil {
    log.Fatalf("error when calling SayHello: %s", err)
  }
  log.Printf("Response from server: %s", response.Greeting)
}

You define a struct to hold the collection on fields you want to inject in your rcp calls. In our case, just a login and password, but you can imagine any fields you want;
The auth variable holds the values you’ll be using;
You use grpc.WithPerRPCCredentials() function to create a DialOption object to the grpc.Dial() function;
Note that the grpc.WithPerRPCCredentials() function takes an interface as parameter, so your Authentication structure should comply to that interface. From the documentation, you know you should implement 2 methods on your structure: GetRequestMetadata and RequireTransportSecurity.
So you define GetRequestMetadata function that just returns a map of your Authentication structure;
And finally, you define RequireTransportSecurity function, that tells your grpc client if it should inject metadata at the transport level. In our current case, it always returns true, but you could have it return the value of a configuration boolean, for instance.

The client is up to push extra data during its calls to the server, but the server does not care, right now. So you need to tell him to check these metadata. Open server/main.go and update it:

package main

import (
  "fmt"
  "log"
  "net"
  "strings"

  "golang.org/x/net/context"

  "gitlab.com/pantomath-io/demo-grpc/api"
  "google.golang.org/grpc"
  "google.golang.org/grpc/credentials"
  "google.golang.org/grpc/metadata"
)

// private type for Context keys
type contextKey int

const (
  clientIDKey contextKey = iota
)

// authenticateAgent check the client credentials
func authenticateClient(ctx context.Context, s *api.Server) (string, error) {
  if md, ok := metadata.FromIncomingContext(ctx); ok {
    clientLogin := strings.Join(md["login"], "")
    clientPassword := strings.Join(md["password"], "")

    if clientLogin != "john" {
      return "", fmt.Errorf("unknown user %s", clientLogin)
    }
    if clientPassword != "doe" {
      return "", fmt.Errorf("bad password %s", clientPassword)
    }

    log.Printf("authenticated client: %s", clientLogin)
    return "42", nil
  }

  return "", fmt.Errorf("missing credentials")
}

// unaryInterceptor calls authenticateClient with current context
func unaryInterceptor(ctx context.Context, req interface{}, info *grpc.UnaryServerInfo, handler grpc.UnaryHandler) (interface{}, error) {
  s, ok := info.Server.(*api.Server)
  if !ok {
    return nil, fmt.Errorf("unable to cast server")
  }
  clientID, err := authenticateClient(ctx, s)
  if err != nil {
    return nil, err
  }

  ctx = context.WithValue(ctx, clientIDKey, clientID)
  return handler(ctx, req)
}

// main start a gRPC server and waits for connection
func main() {
  // create a listener on TCP port 7777
  lis, err := net.Listen("tcp", fmt.Sprintf("%s:%d", "localhost", 7777))
  if err != nil {
    log.Fatalf("failed to listen: %v", err)
  }

  // create a server instance
  s := api.Server{}

  // Create the TLS credentials
  creds, err := credentials.NewServerTLSFromFile("cert/server.crt", "cert/server.key")
  if err != nil {
    log.Fatalf("could not load TLS keys: %s", err)
  }

  // Create an array of gRPC options with the credentials
  opts := []grpc.ServerOption{grpc.Creds(creds),
    grpc.UnaryInterceptor(unaryInterceptor)}

  // create a gRPC server object
  grpcServer := grpc.NewServer(opts...)

  // attach the Ping service to the server
  api.RegisterPingServer(grpcServer, &s)

  // start the server
  if err := grpcServer.Serve(lis); err != nil {
    log.Fatalf("failed to serve: %s", err)
  }
}

Once again, let me break down this for you:

you add a new grpc.ServerOption to the array you created before (see why it’s an array, now?): grpc.UnaryInterceptor. And you pass a reference to a function to that function, so it knows who to call. The rest of the main code does not change;
you have to define the unaryInterceptor function, considering it gets a bunch of parameters:

a context.Context object, containing your data, and that will exist during all the lifetime of the request;
an interface{} which is the inbound parameter of the RPC call;
a UnaryServerInfo struct which contains a bunch of information about the call (such as the Server abstraction object, and the method call by the client);
a UnaryHandler struct which is the handler invoked by UnaryServerInterceptor to complete the normal execution of a unary RPC (i.e. an handler to what happens when the UnaryInterceptor returns).

the unaryInterceptor function makes sure the grpc.UnaryServerInfo has the right server abstraction, and call the authentication function, authenticateClient;
you define the authenticateClient function with your authentication logic — very very simple in this example. Note that it receives the context.Context as parameter, and extract the metadata from it. It checks the user, and returns its ID (in the form of a string, with a hypothetical error.
if the unaryInterceptor gets no error from the authenticateClient function, it pushes the clientID in the context.Context object, so that the rest of the execution chain can use it (remember the handler gets the context.Context object as parameter?);
Note that you created your type and const to reference the clientID in the context.Context map. This is just an handy way to avoid naming conflict and to allow constant reference.

You can compile the code:

$ make

And run both sides in separate terminals:

$ bin/server
2006/01/02 15:04:05 authenticated client: john
2006/01/02 15:04:05 Receive message foo

$ bin/client
2006/01/02 15:04:05 Response from server: bar

Obviously, your authentication logic will probably be smarter, comparing credentials against a database. The easy part of it is: your authentication function gets your abstraction of a Server, and this structure can hold your database handler.

Open to REST

git tag: add-rest

Photo by Rio Hodges on Unsplash

One last thing: you have a pretty neat server, client and protocol; serialized, encrypted and authenticated. But there is a important limit: your client needs to be gRPC compliant, that is be in the list of supported platforms. To avoid that limit, we can open the server to a REST gateway, allowing REST clients to perform requests it. Luckily, there is a gRPC protoc plugin to generate a reverse-proxy server which translates a RESTful JSON API into gRPC. We can use a few line of pure Go code to serve that reverse-proxy.

So edit your api/api.proto file to add some extra information:

syntax = "proto3";
package api;

import "google/api/annotations.proto";

message PingMessage {
  string greeting = 1;
}

service Ping {
  rpc SayHello(PingMessage) returns (PingMessage) {
    option (google.api.http) = {
      post: "/1/ping"
      body: "*"
    };
  }
}

The annotations.proto import allows protoc to understand the option set later in the file. And the option defines that method and the path to the endpoint.

Update the Makefile to add a target for this new Protobuf compilation:

SERVER_OUT := "bin/server"
CLIENT_OUT := "bin/client"
API_OUT := "api/api.pb.go"
API_REST_OUT := "api/api.pb.gw.go"
PKG := "gitlab.com/pantomath-io/demo-grpc"
SERVER_PKG_BUILD := "${PKG}/server"
CLIENT_PKG_BUILD := "${PKG}/client"
PKG_LIST := $(shell go list ${PKG}/... | grep -v /vendor/)

.PHONY: all api server client

all: server client

api/api.pb.go: api/api.proto

 -I api/ \
    -I${GOPATH}/src \
    -I${GOPATH}/src/github.com/grpc-ecosystem/grpc-gateway/third_party/googleapis \
    --go_out=plugins=grpc:api \
    api/api.proto

api/api.pb.gw.go: api/api.proto

 -I api/ \
    -I${GOPATH}/src \
    -I${GOPATH}/src/github.com/grpc-ecosystem/grpc-gateway/third_party/googleapis \
    --grpc-gateway_out=logtostderr=true:api \
    api/api.proto

api: api/api.pb.go api/api.pb.gw.go ## Auto-generate grpc go sources

dep: ## Get the dependencies

 get -v -d ./...

server: dep api ## Build the binary file for server

 build -i -v -o $(SERVER_OUT) $(SERVER_PKG_BUILD)

client: dep api ## Build the binary file for client

 build -i -v -o $(CLIENT_OUT) $(CLIENT_PKG_BUILD)

clean: ## Remove previous builds

 $(SERVER_OUT) $(CLIENT_OUT) $(API_OUT) $(API_REST_OUT)

help: ## Display this help screen

 -E '^[a-zA-Z_-]+:.*?## .*$$' $(MAKEFILE_LIST) | sort | awk 'BEGIN {FS = ":.*?## "}; {printf "\033[36m%-30s\033[0m %s\n", $$1, $$2}'

Generate the Go code for the gateway (the file api/api.pb.gw.go will be generated — just as api/api.pb.go, don’t edit it, it will be updated by compilation):

$ make api

The change on the server side is more important. The grpc.Serve() function is a blocking function, that returns only on error (or can get killed by a signal). As we need to start another server (the REST interface), we need this call to be non-blocking. Fortunately, we have goroutines just for that. And there is a trick on the authentication. As the REST gateway is just a reverse-proxy, it acts as a client from the gRPC perspective. Thus it needs to use a WithPerRPCCredentials option when dialing the server.

Head to your server/main.go file:

package main

import (
  "fmt"
  "log"
  "net"
  "net/http"
  "strings"

  "github.com/grpc-ecosystem/grpc-gateway/runtime"
  "golang.org/x/net/context"
  "gitlab.com/pantomath-io/demo-grpc/api"
  "google.golang.org/grpc"
  "google.golang.org/grpc/credentials"
  "google.golang.org/grpc/metadata"
)

// private type for Context keys
type contextKey int

const (
  clientIDKey contextKey = iota
)

func credMatcher(headerName string) (mdName string, ok bool) {
  if headerName == "Login" || headerName == "Password" {
    return headerName, true
  }
  return "", false
}

// authenticateAgent check the client credentials
func authenticateClient(ctx context.Context, s *api.Server) (string, error) {
  if md, ok := metadata.FromIncomingContext(ctx); ok {
    clientLogin := strings.Join(md["login"], "")
    clientPassword := strings.Join(md["password"], "")

    if clientLogin != "john" {
      return "", fmt.Errorf("unknown user %s", clientLogin)
    }
    if clientPassword != "doe" {
      return "", fmt.Errorf("bad password %s", clientPassword)
    }

    log.Printf("authenticated client: %s", clientLogin)
    return "42", nil
  }
  return "", fmt.Errorf("missing credentials")
}

// unaryInterceptor call authenticateClient with current context
func unaryInterceptor(ctx context.Context, req interface{}, info *grpc.UnaryServerInfo, handler grpc.UnaryHandler) (interface{}, error) {
  s, ok := info.Server.(*api.Server)
  if !ok {
    return nil, fmt.Errorf("unable to cast server")
  }
  clientID, err := authenticateClient(ctx, s)
  if err != nil {
    return nil, err
  }

  ctx = context.WithValue(ctx, clientIDKey, clientID)
  return handler(ctx, req)
}

func startGRPCServer(address, certFile, keyFile string) error {
  // create a listener on TCP port
  lis, err := net.Listen("tcp", address)
  if err != nil {
    return fmt.Errorf("failed to listen: %v", err)
  }

  // create a server instance
  s := api.Server{}

  // Create the TLS credentials
  creds, err := credentials.NewServerTLSFromFile(certFile, keyFile)
  if err != nil {
    return fmt.Errorf("could not load TLS keys: %s", err)
  }

  // Create an array of gRPC options with the credentials
  opts := []grpc.ServerOption{grpc.Creds(creds),
    grpc.UnaryInterceptor(unaryInterceptor)}

  // create a gRPC server object
  grpcServer := grpc.NewServer(opts...)

  // attach the Ping service to the server
  api.RegisterPingServer(grpcServer, &s)

  // start the server
  log.Printf("starting HTTP/2 gRPC server on %s", address)
  if err := grpcServer.Serve(lis); err != nil {
    return fmt.Errorf("failed to serve: %s", err)
  }

  return nil
}

func startRESTServer(address, grpcAddress, certFile string) error {
  ctx := context.Background()
  ctx, cancel := context.WithCancel(ctx)
  defer cancel()

  mux := runtime.NewServeMux(runtime.WithIncomingHeaderMatcher(credMatcher))

  creds, err := credentials.NewClientTLSFromFile(certFile, "")
  if err != nil {
    return fmt.Errorf("could not load TLS certificate: %s", err)
  }

  // Setup the client gRPC options
  opts := []grpc.DialOption{grpc.WithTransportCredentials(creds)}

  // Register ping
  err = api.RegisterPingHandlerFromEndpoint(ctx, mux, grpcAddress, opts)
  if err != nil {
    return fmt.Errorf("could not register service Ping: %s", err)
  }

  log.Printf("starting HTTP/1.1 REST server on %s", address)
  http.ListenAndServe(address, mux)

  return nil
}

// main start a gRPC server and waits for connection
func main() {
  grpcAddress := fmt.Sprintf("%s:%d", "localhost", 7777)
  restAddress := fmt.Sprintf("%s:%d", "localhost", 7778)
  certFile := "cert/server.crt"
  keyFile := "cert/server.key"

  // fire the gRPC server in a goroutine
  go func() {
    err := startGRPCServer(grpcAddress, certFile, keyFile)
    if err != nil {
      log.Fatalf("failed to start gRPC server: %s", err)
    }
  }()

  // fire the REST server in a goroutine
  go func() {
    err := startRESTServer(restAddress, grpcAddress, certFile)
    if err != nil {
      log.Fatalf("failed to start gRPC server: %s", err)
    }
  }()

  // infinite loop
  log.Printf("Entering infinite loop")
  select {}
}

So what happened?

you moved all the code for the gRPC server creation in a goroutine with a dedicated function (startGRPCServer), so it does not block the main;
you create a new goroutine with a dedicated function (startRESTServer) where you create an HTTP/1.1 server;
in startRESTServer where you create the REST gateway, you start by getting the context.Context background object (i.e. the root of the context tree). Then, you create a request multiplexer object, mux, with an option: runtime.WithIncomingHeaderMatcher. This option takes a function reference as parameter, credMatch, and is called for every HTTP header from the incoming request. The function evaluates whether or not the HTTP header should be passed to the gRPC context;
you defined the credMatch function to match the credentials header, allowing them to be metadata in the gRPC context. This is how you have your authentication working, because the reverse-proxy uses the HTTP headers it receives when it connects to the gRPC server;
you also create a credentials.NewClientTLSFromFile, to be used as a grpc.DialOption, just like you did in the client side;
you register your api endpoint, i.e. you make the link between your multiplexer, you gRPC server, using the context and the gRPC options;
and finally, you start an HTTP/1.1 server, and wait for incoming connections;
aside to your goroutine, you use a blocking select call, so that your program does not end right away.

Now build the whole project, so you can test the REST interface:

$ make

And run both sides in separate terminals:

$ bin/server
2006/01/02 15:04:05 Entering infinite loop
2006/01/02 15:04:05 starting HTTP/1.1 REST server on localhost:7778
2006/01/02 15:04:05 starting HTTP/2 gRPC server on localhost:7777
2006/01/02 15:04:05 authenticated client: john
2006/01/02 15:04:05 Receive message foo

$ curl -H "login:john" -H "password:doe" -X POST -d '{"greeting":"foo"}' '
{"greeting":"bar"}

One last swag…

git tag: add-swagger

Photo by Lorenzo Castagnone on Unsplash

The REST gateway is cool, but it would be even cooler to generate documentation from it, right?

You can do that for free, using a protoc plugin to generate a swagger json file:

protoc -I api/ \
  -I${GOPATH}/src \
  -I${GOPATH}/src/github.com/grpc-ecosystem/grpc-gateway/third_party/googleapis \
  --swagger_out=logtostderr=true:api \
  api/api.proto

This will generate a api/api.swagger.json file. As all generated code from Protobuf compilation, you should not edit it, but you can use it, and it can update it when you change your definition file.

You can add the compilation command in the Makefile.

Conclusion

You have a fully functional gRPC client and server, with SSL encryption & authentication, client identification, and a REST gateway (with its swagger file). Where to go from here?

You can push a little on the REST gateway, to make it HTTPS instead of HTTP. You can obviously add more complex data structure on your Protobuf, alongside with more service. You can benefit from HTTP/2 features, such as streaming, either from client to server, or from server to client, or bidirectional (but that’s only for gRPC, not for the REST, based on HTTP/1.1).

Many thanks to Charles Francoise who co-wrote this paper and https://gitlab.com/pantomath-io/demo-grpc.

Hi, I'm Julien Andrieux

Julien Andrieux — Sun, 23 Jul 2017 22:24:15 +0000

I have been coding for x years.

You can find me on Twitter as @chilladx

I live in Paris.

I work for Pantomath

I mostly program in these languages: Python & Go.

I am currently learning more about Go in general.

Nice to meet you.