In the previous lecture, we have learned how to use gRPC interceptors to authenticate users.
However, the API that we used to login user was insecure, which means the username and password were being sent in plaintext, and can be read by anyone who listens to the communication between the client and server.
So today we will learn how to secure the gRPC connection using TLS.
If you haven’t read my post about SSL/TLS. I highly recommend you to read it first to have a deep understanding about TLS before continue.
Here's the link to the full gRPC course playlist on Youtube
Github repository: pcbook-go and pcbook-java
Gitlab repository: pcbook-go and pcbook-java
Types of gRPC connections
There are 3 types of gRPC connections:
- The first one is insecure connection, which we’ve been using since the beginning of this course. In this connection, all data transfered between client and server is not encrypted. So please don’t use it in production!
- The second type is connection secured by server-side TLS. In this case, all the data is encrypted, but only the server needs to provide its TLS certificate to the client. You can use this type of connection if the server doesn’t care which client is calling its API.
- The third and strongest type is connection secured by mutual TLS. We use it when the server also needs to verify who’s calling its services. So in this case, both client and server must provide their TLS certificates to the other.
In this article, we will learn to implement both server-side and mutual TLS in Golang. So let’s get started!
Generate TLS certificates
First we write the cert/gen.sh script to generate TLS certificates.
rm *.pem
# 1. Generate CA's private key and self-signed certificate
openssl req -x509 -newkey rsa:4096 -days 365 -nodes -keyout ca-key.pem -out ca-cert.pem -subj "/C=FR/ST=Occitanie/L=Toulouse/O=Tech School/OU=Education/CN=*.techschool.guru/emailAddress=techschool.guru@gmail.com"
echo "CA's self-signed certificate"
openssl x509 -in ca-cert.pem -noout -text
# 2. Generate web server's private key and certificate signing request (CSR)
openssl req -newkey rsa:4096 -nodes -keyout server-key.pem -out server-req.pem -subj "/C=FR/ST=Ile de France/L=Paris/O=PC Book/OU=Computer/CN=*.pcbook.com/emailAddress=pcbook@gmail.com"
# 3. Use CA's private key to sign web server's CSR and get back the signed certificate
openssl x509 -req -in server-req.pem -days 60 -CA ca-cert.pem -CAkey ca-key.pem -CAcreateserial -out server-cert.pem -extfile server-ext.cnf
echo "Server's signed certificate"
openssl x509 -in server-cert.pem -noout -text
I encourage you to read my post about how to create and sign TLS certificate to understand how this script works.
Basically this script contains 3 parts:
- First, generate CA’s private key and its self-signed certificate.
- Second, create web server’s private key and CSR.
- And third, use CA’s private key to sign the web server’s CSR and get back its certificate.
The generated files that we care about in this video are:
- The CA’s certificate,
- The CA’s private key,
- The server’s certificate,
- And the server’s private key.
I’m gonna add a new command to the Makefile to run the certificate generation script. We just cd to the cert folder, run gen.sh, then get out of that folder.
...
cert:
cd cert; ./gen.sh; cd ..
.PHONY: gen clean server client test cert
Now let’s try it in the terminal.
make cert
All files are regenerated successfully.
Next step, I will show you how to secure our gRPC connection with server-side TLS.
Implement server-side TLS
Let’s open cmd/server/main.go file. I will add a function to load TLS credentials. It will returns a TranportCredentials object or an error.
func loadTLSCredentials() (credentials.TransportCredentials, error) {
// Load server's certificate and private key
serverCert, err := tls.LoadX509KeyPair("cert/server-cert.pem", "cert/server-key.pem")
if err != nil {
return nil, err
}
// Create the credentials and return it
config := &tls.Config{
Certificates: []tls.Certificate{serverCert},
ClientAuth: tls.NoClientCert,
}
return credentials.NewTLS(config), nil
}
For sever side TLS, we need to load server’s certificate and private key. So we use tls.LoadX509KeyPair() function to load the server-cert.pem and server-key.pem files from the cert folder. If there’s an error, just return it. Else, we will create the transport credentials from them.
We make a tls.Config object with the server certificate, and we set the ClientAuth field to NoClientCert since we’re just using server-side TLS. Finally we call credentials.NewTLS() with that config and return it to the caller.
In the main() function, we call loadTLSCredentials() to get the TLS credential object. If an error occurs, we just write a fatal log. Otherwise, we add the TLS credential to the gRPC server by using the grpc.Creds() option.
func main() {
...
tlsCredentials, err := loadTLSCredentials()
if err != nil {
log.Fatal("cannot load TLS credentials: ", err)
}
grpcServer := grpc.NewServer(
grpc.Creds(tlsCredentials),
grpc.UnaryInterceptor(interceptor.Unary()),
grpc.StreamInterceptor(interceptor.Stream()),
)
...
}
And that’s it for the server. Let’s run it in the terminal.
make server
The server is started. Now run the client:
make client
It failed because we haven’t enabled TLS on the client side yet. So let’s do that!
Similar to what we did on the server, I also add a function to load TLS credentials from files. But this time, we only need to load the certificate of the CA who signed the server’s certificate.
The reason is, client needs to verify the authenticity of the certificate it gets from the server to make sure that it’s the right server it wants to talk to.
func loadTLSCredentials() (credentials.TransportCredentials, error) {
// Load certificate of the CA who signed server's certificate
pemServerCA, err := ioutil.ReadFile("cert/ca-cert.pem")
if err != nil {
return nil, err
}
certPool := x509.NewCertPool()
if !certPool.AppendCertsFromPEM(pemServerCA) {
return nil, fmt.Errorf("failed to add server CA's certificate")
}
// Create the credentials and return it
config := &tls.Config{
RootCAs: certPool,
}
return credentials.NewTLS(config), nil
}
So here we load the ca-cert.pem file, then create a new x509 cert pool, and append the CA’s pem to that pool. Finally we create the credentials and return it. Note that we only need to set the RootCAs field, which contains the trusted CA’s certificate.
Now in the main() function, there are 2 connections which are still insecure. We will need to replace them with the secure TLS.
Let’s call loadTLSCredentials() to get the credentials object. Then change the grpc.WithInsecure() call to grpc.WithTransportCredentials(), and pass in the TLS credentials object that we’ve created.
func main() {
...
tlsCredentials, err := loadTLSCredentials()
if err != nil {
log.Fatal("cannot load TLS credentials: ", err)
}
cc1, err := grpc.Dial(*serverAddress, grpc.WithTransportCredentials(tlsCredentials))
if err != nil {
log.Fatal("cannot dial server: ", err)
}
...
cc2, err := grpc.Dial(
*serverAddress,
grpc.WithTransportCredentials(tlsCredentials),
grpc.WithUnaryInterceptor(interceptor.Unary()),
grpc.WithStreamInterceptor(interceptor.Stream()),
)
if err != nil {
log.Fatal("cannot dial server: ", err)
}
...
}
And we’re done. Let’s try it out!
This time the requests are successfully sent to the server. Perfect!
Subject Alternative Name (SAN)
There’s 1 thing I want to show you here. Remember that when we develop on localhost, It’s important to add the IP:0.0.0.0 as an Subject Alternative Name (SAN) extension to the certificate.
subjectAltName=DNS:*.pcbook.com,DNS:*.pcbook.org,IP:0.0.0.0
Let’s see what will happen if we remove this from the server-ext.cnf config file.
subjectAltName=DNS:*.pcbook.com,DNS:*.pcbook.org
Then regenerate the certificates, restart the server, and run the client again.
make cert
make server
make client
As you can see, there’s an error saying that the TLS handshake failed because it cannot validate the certificate for 0.0.0.0, since the SAN doesn’t contain this IP address.
On production, it will be OK because we use domain names instead.
Alright, so now you know how to enable server-side TLS for your gRPC connection. Let’s learn how to enable mutual TLS!
Implement mutual TLS
At the moment, the server has already shared its certificate with the client. For mutual TLS, the client also has to share its certificate with the server. So now let’s update this cert/gen.sh script to create and sign a certificate for the client.
...
# 4. Generate client's private key and certificate signing request (CSR)
openssl req -newkey rsa:4096 -nodes -keyout client-key.pem -out client-req.pem -subj "/C=FR/ST=Alsace/L=Strasbourg/O=PC Client/OU=Computer/CN=*.pcclient.com/emailAddress=pcclient@gmail.com"
# 5. Use CA's private key to sign client's CSR and get back the signed certificate
openssl x509 -req -in client-req.pem -days 60 -CA ca-cert.pem -CAkey ca-key.pem -CAcreateserial -out client-cert.pem -extfile client-ext.cnf
echo "Client's signed certificate"
openssl x509 -in client-cert.pem -noout -text
Let’s say for this tutorial, we use the same CA to sign both server and client’s certificates. In the real world, we might have multiple clients with different certificates signed by different CAs.
Now let’s regenearte the certificates.
make cert
OK the client’s certificate and private key are ready. To enable mutual TLS, on the server side cmd/server/main.go, we should change the ClientAuth field from tls.NoClientCert to tls.RequireAndVerifyClientCert.
func loadTLSCredentials() (credentials.TransportCredentials, error) {
...
// Create the credentials and return it
config := &tls.Config{
Certificates: []tls.Certificate{serverCert},
ClientAuth: tls.RequireAndVerifyClientCert,
}
return credentials.NewTLS(config), nil
}
We also need to provide a list of certificates of the trusted CA who signs our clients’ certificates.
In our case, we only have 1 single CA that signs both server’s and client’s certificate so we can simply copy the codes that we’ve written on the client side to load CA’s certificate and create a new certificate pool.
Then just update the variable names and error messages a bit to reflect the facts that this should be the CA who signs client’s certificate.
func loadTLSCredentials() (credentials.TransportCredentials, error) {
// Load certificate of the CA who signed client's certificate
pemClientCA, err := ioutil.ReadFile("cert/ca-cert.pem")
if err != nil {
return nil, err
}
certPool := x509.NewCertPool()
if !certPool.AppendCertsFromPEM(pemClientCA) {
return nil, fmt.Errorf("failed to add client CA's certificate")
}
// Load server's certificate and private key
serverCert, err := tls.LoadX509KeyPair("cert/server-cert.pem", "cert/server-key.pem")
if err != nil {
return nil, err
}
// Create the credentials and return it
config := &tls.Config{
Certificates: []tls.Certificate{serverCert},
ClientAuth: tls.RequireAndVerifyClientCert,
ClientCAs: certPool,
}
return credentials.NewTLS(config), nil
}
And we’re done with the server. Let’s run it in the terminal.
make server
Now if we connect the current client to this new server, it will fail because the server now also requires client to send its certificate.
Let’s go to the client code cmd/client/main.go to fix this. I will just copy the code to load certificate on the server side and change the file names to client-cert.pem and client-key.pem.
Then we have to add the client certificate to the TLS config by setting the Certificates field, just like what we did on the server side.
func loadTLSCredentials() (credentials.TransportCredentials, error) {
// Load certificate of the CA who signed server's certificate
pemServerCA, err := ioutil.ReadFile("cert/ca-cert.pem")
if err != nil {
return nil, err
}
certPool := x509.NewCertPool()
if !certPool.AppendCertsFromPEM(pemServerCA) {
return nil, fmt.Errorf("failed to add server CA's certificate")
}
// Load client's certificate and private key
clientCert, err := tls.LoadX509KeyPair("cert/client-cert.pem", "cert/client-key.pem")
if err != nil {
return nil, err
}
// Create the credentials and return it
config := &tls.Config{
Certificates: []tls.Certificate{clientCert},
RootCAs: certPool,
}
return credentials.NewTLS(config), nil
}
OK, now if we re-run the client, all the requests will be successful.
Awesome!
Private key encryption
One last thing before we finish. As you know, the current client’s and server’s private key that we are using are not encrypted. It’s because we use the -nodes option when generating them (in the cert/gen.sh file).
openssl req -newkey rsa:4096 -nodes -keyout server-key.pem -out server-req.pem -subj "/C=FR/ST=Ile de France/L=Paris/O=PC Book/OU=Computer/CN=*.pcbook.com/emailAddress=pcbook@gmail.com"
If we remove this -nodes option and run make cert, we will be asked to provide a passphrase to encrypt the server’s private key:
And the generated private key of the server is now encrypted. If we try to start the server with this key, it will return an error: cannot load TLS credentials. That’s because the key is encrypted.
We can add more codes to decrypt the key with the passphrase, but I think in the end, we still have to protect the passphrase by keeping it somewhere safe. So we can always store our unencrypted private key in that place as well.
For example, if you use amazon web service, you can store your private key or any other secrets in encrypted format with aws secret manager. Or you can use HashiCorp’s Vault for the same purpose.
And that’s everything I wanted to share with you in this article. I hope you find it useful. Thanks a lot for reading, and see you guys in the next one!
If you like the article, please subscribe to our Youtube channel and follow us on Twitter for more tutorials in the future.
If you want to join me on my current amazing team at Voodoo, check out our job openings here. Remote or onsite in Paris/Amsterdam/London/Berlin/Barcelona with visa sponsorship.
Top comments (7)
there is an error i n go1.18
cannot create auth interceptor: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: x509: certificate signed by unknown authority (possibly because of \"x509: cannot verify signature: insecure algorithm SHA1-RSA (temporarily override with GODEBUG=x509sha1=1)\" while trying to verify candidate authority certificate \"*.techschool.guru\")"
we can fix it temporarily by "GODEBUG=x509sha1=1 go run cmd/client/main.go -address 0.0.0.0:8080"
It's not a go's issue, you can generate cert with sha256 instead: techglimpse.com/sha256-hash-certif...
When I run this script "cert/gen.sh" I get this error
Hi Team,
I tried to call grpc connection with tls as you have provided steps.but getting server certificate "nil" while certificate has generated using this link (dev.to/techschoolguru/how-to-creat...)
certificate path is also correct.Kindly guide
Great article!
Maybe a last one with envoy to handle gRPC with frontend app.
I don't understand how to generate certs between frontend and backend app maybe it's just between envoy and backend!
It'll be a nice article :)
This article was an absolute LIFE SAVER! I struggled to get this working for days before coming across this. Thank you!!
Thank's,
you saved my time