Had an itch I've been meaning to scratch for a while. I build my Puppet environment using Terraform, which makes it nice and easy to tear things down and rebuild them. That is great, but it does leave me with an issue when it comes to the console SSL certificates.
Puppet will generate self-signed certs for the console, which work fine, but it was always a niggle that the certs couldn't be automagically coaxed into being valid.
Since moving over to kubernetes for my home lab, I've come to expect managed SSL certificates for any public facing services, without me having to do anything.
Finally set some time aside to look at the options and thought I'd publish the details of the journey as well as where I ended up.
Step 1 - Let's Encrypt
Obviously I wasn't going to reinvent the wheel. If I wanted to manage and use Let's Encrypt on a Puppet Enterprise server, I'd be using the Let's Encrypt module. The module makes it very simple to get the relevant packages installed and configured.
class { 'letsencrypt':
config => {
email => 'certs@albatrossflavour.com',
},
config_dir => '/etc/letsencrypt',
}
One of the beauties of this module is that it also sets up a cron
job to renew the generated certs, so you don't need to keep an eye on it.
Classify your server with that and you'll end up with certbot
and it's dependencies. Great start and feeling confident about the future!
Then we need to generate a certificates:
letsencrypt::certonly { 'puppet.gcp.albatrossflavour.com':
domains => ['puppet.gcp.albatrossflavour.com'],
manage_cron => true,
plugin => 'webroot',
webroot_paths => ['/var/www],
}
Annnnnnnd that's where things start to go south.
When you request a cert, the most simple method of validating you are who you say you are is to have a web server on the host respond to a query sent to port 80
. Sure we say, easy, couldn't take much to do!
By default, the Puppet Enterprise nginx
config does a 301
redirect for http
requests to https
. This means any queries from Let's Encrypt to validate the requests will end up as https
requests and the cert request will fail.
Step 2 - nginx
Let me introduce you to puppet_enterprise::profile::console::proxy::http_redirect::enable_http_redirect
. This parameter controls if that redirect is in place. So step .... 6(?) was to use hiera
to disable the http
redirect:
❯ cat data/role/role::pe::master.yaml
puppet_enterprise::profile::console::proxy::http_redirect::enable_http_redirect: false
Once we run this through a puppet
run, the redirect gets removed! Score!
Only problem is, without the redirect, the nginx
server doesn't listen on port 80
. OK, we can fix that easily. We could use the pe_nginx::directive
type, but I found it to be a bit of an overkill for what I needed. Instead I opted for a simple template:
file { '/etc/puppetlabs/nginx/conf.d/certs.conf':
ensure => file,
owner => 'root',
group => 'root',
mode => '0644',
content => template("${module_name}/cert_vhost.conf.erb"),
notify => Exec['pe_nginx'],
}
and the template is just:
server {
listen 80;
server_name <%= @fqdn %>;
index index.html;
location /.well-known {
root /var/www;
}
location / {
return 301 https://$server_name$request_uri;
}
}
(Sorry, still using erb
, I will at some point rewrite everthing in epp
)
The template keeps the 301
redirect in place for anything other than a request to /.well-known
, which is where Let's Encrypt looks for the validation info.
Run this through and the nginx
vhost gets created. However the letsencrypt::certonly
call still fails on the first run. The notify
to the pe_nginx
service, which is done when we create the cert_vhost.conf
, doesn't happen in the order we need. The Let's Encrypt module is trying to get a response before we've setup the vhost. Now this would work on subsequent runs, but Golden Rule #1 is to make sure, whenever possible, that you get a clean puppet run in one pass. Plus this was a challenge.
Before I worked on fixing that, I wanted to make sure I could make the rest of it work. Let's sum up where we are:
- I've got the Let's Encrypt client installed and configured
- I've got a new
nginx
vhost running that allows the Let's Encrypt web validation queries through and redirects any otherhttp
traffic. - After a couple of puppet runs, I've got valid SSL certs in
/etc/letsencrypt
Step 3 - The console certs
It'd been a while since I played with the Puppet console SSL certs, so I checked in with the source of truth, which outlines the steps we need to go through to use custom SSL certs with the console:
- Retrieve the custom certificate and private key.
- Move the certificate to
/etc/puppetlabs/puppet/ssl/certs/console-cert.pem
, replacing any existing file namedconsole-cert.pem
. - Move the private key to
/etc/puppetlabs/puppet/ssl/private_keys/console-cert.pem
, replacing any existing file namedconsole-cert.pem
.
So we can just create a file
resource that takes the Let's Encrypt cert/key and places them into the console SSL directory structure.
file { '/etc/puppetlabs/puppet/ssl/certs/console-cert.pem':
ensure => file,
owner => 'pe-puppet',
group => 'pe-puppet',
links => 'follow',
mode => '0640',
source => "/etc/letsencrypt/live/${facts['puppet_server']}/cert.pem",
backup => '.puppet_bak',
notify => Service['pe-nginx'],
subscribe => Letsencrypt::Certonly[$facts['puppet_server']],
}
file { '/etc/puppetlabs/puppet/ssl/private_keys/console-cert.pem':
ensure => file,
owner => 'pe-puppet',
group => 'pe-puppet',
links => 'follow',
mode => '0644',
source => "/etc/letsencrypt/live/${facts['puppet_server']}/privkey.pem",
backup => '.puppet_bak',
notify => Service['pe-nginx'],
subscribe => Letsencrypt::Certonly[$facts['puppet_server']],
}
Sure enough, when I try this on the Puppet server, life is good and we have a console with valid SSL certs.
Time to crack open a bottle of red and tick an item off my to do list.
Damn it, Golden Rule #2 raises it's head. It's not finished until you know it works fine from scratch... WITHOUT breaking Golden Rule #1.
Step 4 - Rebuild
Nuked the Puppet server and rebuilt it. Lots of failures on the first run (we kinda expected that), however they didn't go away. The nginx
service never gets restarted as the various dependencies can be resolved, but not in the way we need.
This is the fun (?) of declarative configuration management. I can only manage things once, which includes only being able to notify
the pe-nginx
service once. Then the compiler will figure out when the service is restarted, based on all of the dependencies in the catalog.
I played around with a lot of options, some waaaaay hackier than I wanted to go with. Plus this was becoming a fun exercise.
All I needed to be able to do was to inject a restart of the pe-nginx
service twice in a single run.
To get there, I bent a few of the future Golden Rules and came up with:
exec { 'restart_nginx':
command => '/bin/systemctl restart pe-nginx',
refreshonly => true,
}
This exec
resource will allow me to do a restart of pe-nginx
outside of, and before, the notify => Service['pe-nginx']
parameters.
Yes, it's a hack, but it's a hack on the side of the angels. Not only will it solve the issue of the process not working at all, but it will also allow the certs to be generated, and installed, in a single run.
Step 5 - Quick Robin, to the pdk
mobile
I did a fair bit of testing and found the end result to be far more reliable and useful than I thought it would be.
I first created a fully parameterised profile to manage the certs. Once I got that working, it was a no-brainer to create a module to share the love.
That's how I ended up with the pe_console_letsencrypt module. You can also just check out the code.
I've done a fair bit of testing, but I'm certainly not saying it's bulletproof.
Step 6 - Profit?
I'm still scratching my head to think of another way I could get the certs generated and in place in a single run without the exec
hack. That being said, I've had hacks a lot worse go live in the past!
Note
While researching the links for this post, I came across a blog less frequently updated than mine which has a post from 2017 showing a similar way of achieving the same thing, but without the hack to make it work in a single run.
Top comments (0)