Nginx, Let’s Encrypt and taking my own server down 😢

Photo by Simon Fitall on Unsplash

If you pay attention to Google and it’s indexing rules, you’ve probably heard that you need an SSL certificate on your site. I’ve known this for a while and my personal site at petetasker.com wasn’t a high priority situation.

Welp, I was bored one night and decided it was about time to get one of those fancy-pants free Let’s Encrypt certs installed on my old Linode server.

There are tons of resources on the internet that outline how to get an SSL certificate installed on your site, so I’m not going to go over that portion. What I will go over is how blindly following them can take down your server…

You see, most tutorials want you to use Certbot to install your certificate and configure Nginx.

certbot --nginx -d petetasker.com

Seems harmless enough, no? That little --nginx flag, if you didn’t read the fine print, will modify your virtual host config. Yeah. There’s an option when setting up the certificate to redirect all HTTP traffic to HTTPS, and obviously I said ‘sure’.

And queue the redirect loop. Site monitor email deluge, Twitter ‘the site is DOWN’ DM’s…

Everything is going swimmingly!

I managed to solve this issue by removing the following block in the /etc/nginx/sites-available/petetasker.com virtual host declaration:

if ($host = petetasker.com) {
    return 301 https://$host$request_uri;
} # managed by Certbot

And adding a simpler redirect block, just to be reallllly clear, at the top of the file:

server{
        listen 80;
        server_name petetasker.com;
        return 301 https://$server_name$request_uri;
}

So what’s the lesson here kids? The lesson is that if you’re using a tool that will ‘automagically’ do something for you, and it sounds too good to be true, it probably is.

If you’re wondering what the final virtual host block looks like:

server{
        listen 80;
        server_name petetasker.com;
        return 301 https://$server_name$request_uri;
}
server {

        listen 80 default_server;
        #listen [::]:80 default_server ipv6only=on;
        server_name petetasker.com www.petetasker.com;

        server_name petetasker.com www.petetasker.com;
        root /usr/share/nginx/sites/petetasker.com;
        index index.php index.html;

        location / {
                try_files $uri $uri/ /index.php?$args;
        }

        location ~ \.php$ {
                try_files $uri =404;
                fastcgi_pass unix:/var/run/php/php7.0-fpm.sock;
                fastcgi_index index.php;
                fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
                include fastcgi_params;
        }

    listen [::]:443 ssl ipv6only=on; # managed by Certbot
    listen 443 ssl; # managed by Certbot
    ssl_certificate /etc/letsencrypt/live/petetasker.com-0001/fullchain.pem; # managed by Certbot
    ssl_certificate_key /etc/letsencrypt/live/petetasker.com-0001/privkey.pem; # managed by Certbot
    include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
    ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot

}

The post Nginx, Let’s Encrypt and taking my own server down 😢 appeared first on 🔥 Database Critical 🔥.

Comments

[Jonathan Boudreau]

Seems to me like the lesson here is to not test in prod.

[Peter Tasker]

For a site that I care about, sure. For my own site and server it was a worthwhile experiment.

[Jonathan Boudreau]

I'd still spin up a test server temporarily (either locally or VPS).

[Remco van Bree]

A former co-worker of mine suggested this setup:

Add the following location to your server block:

location /.well-known/acme-challenge {
        proxy_pass http://[::1]:9999;
}

and then run certbot like this:

certbot certonly --standalone --http-01-port 9999 --noninteractive --agree-tos --email [e-mail] -d www.yoursite.com --post-hook "service nginx reload"

The renew job in my crontab looks like this:

14  4,16  * * * sudo certbot renew --standalone --http-01-port 9999 --renew-hook "service nginx reload" >> /var/log/le-renew.log

Of course that means that you actually have to add the key paths to your nginx config too, but once you have this up and running you will have a nice and stable config.

[Peter Tasker]

Neat! Yeah I think running the --standalone flag is the way to go. I gotta check the docs on certbot as I didn't realize you could specify a custom verification location directive.

[Jason C. McDonald]

When I set up my Let's Encrypt certificates, I hand-wrote my .conf files for Apache2, mainly because of this little personal quirk: I don't trust automagical things. I get mocked fairly frequently for my tendency to do things manually, especially with IT stuff, but my end results are usually cleaner and more stable than mainstream, I find.

[Peter Tasker]

Yeah I find 'magic' stuff tends to be overly verbose or just unnecessary. At worst it'll explode things - as I discovered.

[KhoPhi]

Automagics are a nightmare (especially when used unknowingly). I try to read the fine prints, dodging all automagics and actually manually doing my setups the first time.

I've experienced the --nginx flag does in the early days. Since I had all my nginx conf done already, the way I wanted, I just needed to do my ssl integration once, and for all.

So yeah, nice reminder.

[Peter Tasker]

Yep! The point of this post was mostly to share that point! Watch out for that --nginx flag.

[dallgoot]

nginx configuration is not that complicated ONCE you take the time to read through the docs e.g. the second "server" is listening on 80 when that port is supposedly meant to be redirected to 443, looks like a duplicate.

One good practice in nginx: make a separate file for "options" and include it, it helps keep visualizing quickly what the settings are.
Put the PHP config in a fastcgi_php.conf and you can have same settings for multiple sites even in the same instance.

[Peter Tasker]

Yep as you can probably tell I'm still learning about nginx config. I'm noticing that including files is the way to go. Plan is to start setting up some Fast CGI caching next so we'll see how that goes!

[Rishabh Gupta]

Also, backing up your nginx config is pretty important, maybe keeping it in private version control so anytime you do something wrong you can quickly revert it.

[Dinesh Rathee]

LetsEncrypt have revoked around 3 million certs last night due to a bug that they found. Are you impacted by this, Check out ?

DevTo
[+] dev.to/dineshrathee12/letsencrypt-...

GitHub
[+] github.com/dineshrathee12/Let-s-En...

LetsEncryptCommunity
[+] community.letsencrypt.org/t/letsen...

[Pascal Maniraho]

Hi Peter,
Thanks for the write-up, I found it better to generate certificates only, and symlink to them.
That guaranteed stability of my existing configuration.

I detailed steps here: How to install Let's Encrypt SSL Certificate on Ubuntu and Nginx Server, just in case anyone needs a help with this.

Thanks.