Encrypt Status Communication Messages in Consul and Nomad

Consul, the service management and service discovery tool, and Nomad, the job scheduler, are central tools in my infrastructure at home project. Both tools run as daemons on all nodes. They exchange data with the Gossip protocol, and API endpoints via HTTP and RPC. There is one last requirement to fulfill: Securing these communications by encrypting all traffic. The particular Infrastrucure Security requirements are:

• INFSEC01: Secure Consul internal communication
• INFSEC02: Secure Nomad internal communication

This article shows you the essential steps to secure Nomad and Consul. There are excellent documentation on the Consul homepage and Nomad homepage, my approach compresses the rich information from these pages for you.

This article originally appeared at my blog.

Consul: Encrypt the Gossip Protocol

The Gossip protocol is used to constantly exchange information about your cluster. Enabling encryption is a two-step process: Generate a secret, and then set the nodes to use encrypted communication.

1. On any node in your cluster, run the command consul keygen and note down the password
2. On all nodes, add the following configuration options, then restart consul

  {
    encrypt = "SECRET"
    encrypt_verify_incoming = false
    encrypt_verify_outgoing = false
  }

1. On all nodes, set encrypt_verify_outgoing = true, then restart consul

  {
    encrypt = "SECRET"
    encrypt_verify_incoming = false
    encrypt_verify_outgoing = true
  } 

1. On all nodes, set encrypt_verify_incoming = true, then restart consul

  {
    encrypt = "SECRET"
    encrypt_verify_incoming = true
    encrypt_verify_outgoing = true
  }  

Now all consul traffic is encrypted.

Consul: TLS Encryption

To enable TLS encryption, the following steps are necessary:

1. Generate root certificate
2. Generate server and client certificates
3. Enable TLS encryption on the servers
4. Enable TLS encryption on the clients

Let’s detail these steps.

Generate Certificates

TLS certificates all start with a valid root certificate, then intermediate certificates, and finally the certificate that is used by your servers/clients. You can create your own root certificate with tools like OpenSSL, or you can use Nomads built-in commands. We will do the later.

First, we generate the root certificate as follows:

>> consul tls ca create
==> Saved consul-agent-ca.pem
==> Saved consul-agent-ca-key.pem

Second, we generate the server certificates. This certificate will be signed with the key Saved consul-agent-ca-key.pem. Be careful to set the correct name of your data center.

>>consul tls cert create -server -dc=infra

==> Using consul-agent-ca.pem and consul-agent-ca-key.pem
==> Saved infra-server-consul-0.pem
==> Saved infra-server-consul-0-key.pem

Enable TLS Encryption

Similar to the Gossip encryption, we need to add new configuration options and restart the server/clients one at a time. Perform the following steps.

1. Server: Copy the certificates and add the following configuration options.

   {
     "verify_incoming": false,
     "verify_outgoing": false,
     "verify_server_hostname": false,
     "ca_file": "consul-agent-ca.pem",
     "cert_file": "dc1-server-consul-0.pem",
     "key_file": "dc1-server-consul-0-key.pem"
   }

1. Restart the server nodes - be sure that they are running before continuing.
2. Clients: Copy the certificates, add the following configuration options.

   {
     "verify_incoming": true,
     "verify_outgoing": true,
     "verify_server_hostname": true,
     "ca_file": "consul-agent-ca.pem",
     "cert_file": "dc1-server-consul-0.pem",
     "key_file": "dc1-server-consul-0-key.pem"
   }

1. Restart the client nodes

2. Server: set the following configuration options
   

   {
     "verify_incoming": true,
     "verify_outgoing": true,
     "verify_server_hostname": true
   }
   

3. Restart the server nodes

Congratulations! All traffic within your consul cluster is now encrypted.

Nomad: Encrypt the Gossip Protocol

Encrypting the Gossip protocol in Nomad is simpler because you only need to modify the servers. The steps are similar: Create a secret, change the config file, then restart the servers.

1. Create a secret with nomad operator keygen
2. Add this secret to the server’s configuration files, and restart all servers

server {
  encrypt = SECRET
}

Nomad: TLS Encryption

The encryption process in Nomad is very similar to Consul: You create certificates, configure and restart the server, and then configure and restart the clients.

Generate Certificates

The Nomad documentation suggests to use the tool cfssl. I could not get it working, so I reused the Consul commands.

First, generate the root certificates. Be sure to set the correct domain

consul tls ca create -domain=nomad -name-constraint
==> Saved nomad-agent-ca.pem
==> Saved nomad-agent-ca-key.pem

Second, generate certificates for the server. Be sure to set the correct domain. Also, unless configured otherwise, the nomad agents will talk to the servers by using the datacenter global - add this as a flag to the command.

>> consul tls cert create -server -domain nomad -dc=global

==> Saved global-server-nomad-1.pem
==> Saved global-server-nomad-1-key.pem

Third, generate certificates for the clients.

>> consul tls cert create -client -domain nomad -dc=global

==> Using nomad-agent-ca.pem and nomad-agent-ca-key.pem
==> Saved global-client-nomad-1.pem
==> Saved global-client-nomad-1-key.pem

Enable TLS Encryption

These steps are similar to configuring TLS for Nomad.

1. Server: Copy the certificated, then add the following configuration options:

tls {
  http = true
  rpc  = true

  ca_file = "certs/nomad-agent-ca.pem"
  cert_file = "certs/global-server-nomad-0.pem"
  key_file = "certs/global-server-nomad-0-key.pem"

  verify_server_hostname = true
  verify_https_client    = false
 }

1. Restart the server
2. Clients: Copy the certificates, then add the following configuration options:

  tls {
    http = true
    rpc  = true

    ca_file = "certs/nomad-agent-ca.pem"
    cert_file = "certs/global-client-nomad-0.pem"
    key_file = "certs/global-client-nomad-0-key.pem"

    verify_server_hostname = true
    verify_https_client    = true
  }

1. Restart all clients

Congratulations, all Nomad traffic is now encrypted.

One more thing: If you want to access the Nomad HTTP UI, and have verify_https_client = true, you need to add the client certificate to your browser. If you are using a proxy server, you also need to add the Hostname with which you access Consul as a Subject Alternative Name (SAN) when creating the client certificates.

Conclusion

This article showed how to enable complete encryption for all client/server communication in Nomad and Consul. Gossip encryption protects the status updates between the nodes. It is enabled by setting a secret key and by adding configuration options to use encrypted incoming/outgoing messages. TLS encryption secures all HTTP and RPC endpoints of the clients/servers. You first create a root certificate and the client/server certificates. Then you add configuration options to all nodes, starting with the servers, continuing with the clients. After each change you do a rolling update of all nodes. With the final update, all traffic is encrypted.

You can verify the encryption by starting an agent from the command line. Take a look at the last line in the following log.

/usr/local/bin/consul agent -config-dir=/etc/consul/ -bind=192.168.2.201 -client=192.168.2.201
==> Starting Consul agent...
           Version: 'v1.7.1'
           Node ID: '6f859439-c48f-f3af-5282-26b9f9dccad2'
         Node name: 'raspi-3-1'
        Datacenter: 'infra' (Segment: '')
            Server: false (Bootstrap: false)
       Client Addr: [192.168.2.201] (HTTP: 8500, HTTPS: -1, gRPC: -1, DNS: 8600)
      Cluster Addr: 192.168.2.201 (LAN: 8301, WAN: 8302)
           Encrypt: Gossip: true, TLS-Outgoing: true, TLS-Incoming: true, Auto-Encrypt-TLS: true