tl;dr Cloudflare’s “Always use https” breaks letsencrypt renewal
Sometimes you’re browsing through the settings of an app and you come across something that seems like such a good idea, and you just flip the switch. And then 3 months later… you find out that it has real consequences.
Let me tell you the story of how I spent my afternoon. It was sunny outside, so I was doing the responsible thing and staying inside so my moles don’t get all cancer-like. I finally had a spec of free time to deal with a nasty email I had gotten a few days back… from Let’s Encrypt.
It was a notice that a certificate was expiring. I loathe those mails. Why have I gotten so many to form such a strong opinion? Well, probably because I spent too long using IIS as a webserver before certbot worked on Windows. Even with my machete of a hack script, I still had to invoke the damn thing manually.
Certbot is a godsend — like your own personal Moses to part the Red Sea for you, and hold your hand as you walk its dampened path to the promised land. Being but a mere mortal, I do not pretend to understand the mystic magic that is at its core, I merely wave it in the general direction of a webserver and hope for the best. So when things go wrong, I do what every pale-skinned, be-speckled Egyptian slave would do — I ask the almighty Google wtf the error message means.
Now this usually works. Google is a kind and merciful God, with great bounties of knowledge that no one man could ever fully grasp. But today he was being a pestilent child, taunting me with repugnant tales of people solving very similar, but not quite the same problem. As if my sailing troubles could be solved by self-help guides to flying aircraft (but they’re both forms of transport!)
In my desperation I began fiddling with the nginx config & started wading through curl responses to try convince myself I was doing something that points me, even remotely, in the right direction.
301 http to https
But LE doesn’t listen to https
So why doesn’t it pick up the http
What else could be doing the redirect??!1
Cloudflare
Cloudflare is that rich uncle that you can always count on to bum a smoke. But he likes the sauce, so when you go out on his yacht, he’ll take you literally when you say you wanna “dance the hempen jig” and start tying the rope, rather than blazing that fattie that’s hanging out his shirt pocket.
So anyway, there it was in the curl headers. Plain as Red Sea mud. Cloudflare doing what you told it to. Let’s just uncheck that little switch, and…
Ba-bam! Woosh, whip! And suddenly that metaphorical storm outside that’s really inside (your head) abates and your little boat is once more on its merry way. Kinda anticlimactic. Maybe we should celebr — no. We just burned 3 hours for recklessly flicking a toggle switch. Them’s 3 hours we shan’t be gettin’ back. No need to celebrate undoing stupidity. A 12th cup of coffee is an acceptable consolation prize.
After time to reflect and just enough coffee to look like a meth addict, I came to the realization that we, as developers, live in a quantum state of information. The answer is either 42 seconds of Googling away, or it is unreachable. No one can know before you Google the error message, and doing so collapses the wave down on top of you and leaves you sputtering coffee all over your keyboard.
Our systems are now so complex that we need a whole team to take the Helm of the SS Kubernetes and forge a path to the buried treasure. Where once you floated in your dinghy greedily grabbing bottled messages from forums in the vain hope that they had the answers, you now sit on your magnificent vessel, surrounded by comrades, doing the same because it’s too much effort to take your headphones off and Slack is down again.
Maybe one day we’ll traverse the tricky Strait of Business Model and arrive triumphant on the new issue day to be rewarded with our cut of the booty. Maybe on that day, we’ll be able to return, broken and alone, to the solitude of our dinghy where we’ll finally find some time for that side project we started.
Oldest comments (1)
So Let's Encrypt works with the ACME protocol for validation, right? You can choose other protocols than http (or https) for validation. I personally like DNS. One cool little trick, ACME will follow CNAMES linked together until it finds the validation code.
This means you can host a "validation zone" for all of your LE certs in one place. I have DNS spread across NS1, CloudFlare, some Route53 & DYN and my own personal servers. It made more sense to host my own validation zone and write ansible hooks to update them when needed. The http challenge always seemed hoakey to me.
Example included. Perform a dig TXT on this host. The challenge points to a permanent CNAME which lands inside my aval (acme validation) zone. I only update that zone. Gxize.net zone lives in Cloudflare while the real devx.gxize.net lives in Route53. I just need ansible CF hooks to update TXT records in the aval zone when needed.
I made a simple convention... this is the rb2011 host under the devx subzone, separated my a dash. Also, neat caveat. The two zones don't have to be related. I could have a validation zone in any old domain/zone I wish. Register a domain just for validations...
Now, use DNS for your renewals and for security's sake, turn https back on. :D