Originally posted on brnrdo.hashnode.com.
Why I am writing this post
Hi. I'm a data scientist and I am currently developing a platform for MLOps using Docker Compose. This is to say, now that I am playing the role of an engineer, there are a lot of concepts that I am learning fast, and it can feel overwhelming and even scary. If you're new to deploying applications with CloudFlare and configuring a reverse-proxy with Traefik, I wrote this guide for you.
The goal of this post is also to show you how you can get that sweet green lock for SSL certificates by having Traefik fetch Let's Encrypt (LE) certificates for you and how to place your application behind a CloudFlare proxy. For sake of preview, here are some pain points that I've faced as a starting engineer.
- Cloudflare as your DNS server.
- CloudFlare flexible, full and strict modes.
- Watch out for https redirects in Traefik.
- Exposing your server in CloudFlare: Development mode and temporarily disabling CloudFlare to bypass its proxy.
- Traefik configuration to fetch Let's Encrypt.
This post is not supposed a complete tutorial to Docker Compose, Traefik, CloudFlare and Let's Encrypt - there is already a lot of resources out there for that purpose. Rather, it is almost a personal log of the obstacles I encountered that I hope helps other people overcome them.
What is CloudFlare, Traefik and Let's Encrypt?
CloudFlare (CF) is mainly a DNS server with extra features - these extra features are attributed to CloudFlare's (reverse-)proxy functions, which you can enable and disable whenever you want.
For example, you set your DNS records to point your domain and subdomains to the IP of the server where your application is running. When you set these records to 'proxied' (i.e. orange cloud) you benefit from the proxy functions that CloudFlare carries out, such as concealing the real IP address of your server and DDoS protection.
Traefik is a reverse-proxy that sits at the edge of your application on the server. This reverse-proxy is Wait, what? Another "proxy"?! Well yes, but this Traefik is configured in your docker compose application and has no overlap in functionality with CloudFlare. Traefik will intercept requests to a given route, say a-route.your-domain.com and match with any existing rules that you have set to a service running in Compose.
Let's Encrypt (LE) is a Certificate Authority (CA) that signs and ensures that your certificates are genuine to encrypt the connection between the clients and your server. The best part is that by using LE, you are taking advantage of the ACME protocol that provides you with autorenewals of your certificates. You could use a self-signed certificate, but there are disadvantages to that. You can use Let's Encrypt from Traefik to minimize set up effort.
Setting up Traefik
Setting up proxy entrypoints
It's best to test things locally as much as one can before deploying it to a live server. Your setup should have two entrypoints, which I have chosen to name web and websecure for the sake of having distinct names than http and https, so that we know which fields are arbitrary names given by us.
services:
traefik:
command:
- '--entrypoints.web.address=:80'
- '--entrypoints.websecure.address=:443'
The entrypoints are part of the dynamic configuration of Traefik. This setup is less than ideal as it allows for unecrypted http connections. In Traefik, a redirection to https can be done in the following way.
services:
traefik:
command:
- '--entrypoints.web.address=:80'
- '--entrypoints.web.http.redirections.entryPoint.to=websecure'
- '--entrypoints.web.http.redirections.entryPoint.scheme=https'
- '--entrypoints.websecure.address=:443'
One of the options available from CloudFlare is none other that HTTPS redirection, so the headers are already rewritten at CloudFlare's proxy. The redirection configured in Traefik (~'origin server' as per CloudFlare's terminology) acts as a failsafe should you disable CF's proxy.
Setting up Let's Encrypt (from Traefik)
This step is entirely optional if you're just developing on your machine. TLS can be enabled without LE, in which case, Traefik issues its own certificates.
Of course, what is desirable in production is to have CA certificates. Under the hood, Traefik uses lego, an LE CLI client, to connect to LE servers and fetch certificates. You only need to supply Traefik with an email and some options and Traefik will handle the rest for you. Here's an example of a setup.
services:
traefik:
command:
- '--certificatesresolvers.le.acme.email=${ACME_EMAIL}'
- '--certificatesresolvers.le.acme.storage=/letsencrypt/acme.json'
# TLS challenge serves the certificates back to the CA in order to renew them
- '--certificatesresolvers.le.acme.tlschallenge=true'
# Optionally use the staging server to prevent exhausting rate limits
- '--certificatesresolvers.le.acme.caserver=https://acme-staging-v02.api.letsencrypt.org/directory'
The bare minimum is comprised of the first two lines: the email and the TLS challenge. This challenge is the simplest one to setup, as the only thing to do is to enable a boolean flag. However, taking into account CloudFlare, CF does not work with the TLS challenge, and either the DNS challenge or the HTTP challenge must be configured in order to be able to have the edge proxy enabled.
As a note, the default method used for ACME authentication by the Let's Encrypt client utilizes the DVSNI method. This will fail for a domain which has Cloudflare enabled as we terminate SSL (TLS) at our edge and the ACME server will never see the certificate the client presents at the origin. Using alternate ACME validation methods, such as DNS or HTTP will complete successfully when Cloudflare is enabled.
Note that challenges are mutually exclusive per certificate resolver, and each router can only use one certificate resolver at a time. Effectively, a router can only use one type of ACME challenge at a time. The snippets below are illustrative of choosing just one ACME challenge for the le certificate provider that is being configured.
Setting up Let's Encrypt with HTTP challenge
services:
traefik:
command:
- '--certificatesresolvers.le.acme.httpchallenge=true'
- '--certificatesresolvers.le.acme.httpchallenge.entrypoint=web'
Setting up Let's Encrypt with DNS challenge
services:
traefik:
command:
- '--certificatesresolvers.le.acme.dnschallenge=true'
- '--certificatesresolvers.le.acme.dnschallenge.provider=cloudflare'
environment:
# you may choose to use secrets instead of environment variables like this
- CF_API_EMAIL=${CF_API_EMAIL}
- CF_API_KEY=${CF_API_KEY}
In this example, the cloudflare provider is being used because that's where the DNS records are set up - i.e. the nameservers of the domain are pointing to CloudFlare. If you are using another DNS server, then you must set the environment variables specific to your provider.
Enable the use of Let's Encrypt in a router
Refer to the section Using the certificate resolver, this blog post will be more clear than me on how to use Traefik :)
Setting up CloudFlare
DNS records and subdomains
You only need one A record for the root of the platform you're developing. For any subdomain, you can set CNAMEs that point to that A record. The A record should point at the IP where Traefik is being served. Traefik will handle routing to the correct machines itself i.e. load balancing - should you have your application running in Swarm mode.
For example, DNS records on CloudFlare could look like this:
As a general rule, you only need to set A records (@ and www) that point to the real IP of your server. If the services you defined in Traefik follow the template a-service.yourdomain.com, then you only need to set CNAME records with the Name a-service pointing to the Content yourdomain.com.
Note the orange clouds: they indicate that that requests to the specificied (sub)domain are being proxied by CloudFlare's edge proxy. They can be disabled one by one, but that is not of interest to what I write in this post.
CloudFlare options
In general, and from my experience, the default configurations work well. I only call your attention to the CF's encryption modes (which I mention in the Troubleshooting section) and the HSTS option, that could be problematic for your case.
Tips
A very useful tip is to enable CF "Development Mode" so that web requests bypass any cache that CF may have created and allows you to check for changes without interference of a cache.
If you have set up LE only with the TLS challenge: use the option "Pause CloudFlare on Site" before doing docker-compose up in your server. This will allow Let's Encrypt to find the real IP of your server without CF hiding it. Related with this is the use of a DNS client (or dnschecker.org) to check where the nameservers are pointing your names to. Using dog (a CLI DNS client), you should get a record pointing to the real IP of your live server:
$ dog some-domain.com
A some-domain.com. 38s xx.yyy.xx.yyy
Once the TLS challenge is complete and the certificates are issued, you should be able to check that
- the certificates are stored in the
acme.jsonfile - in the browser window, at the page of the service being served at the specified route rule, you should click on the lock icon in the address bar, open "certificates" and verify that they are in fact issued by Let's Encrypt. This means you set up Traefik + LE correctly!
Once everything is working as it should, you can click on the option "Enable CloudFlare on site" and let CF hide your IP again. The output of the DNS check should be something like this
$ dog auth.some-domain.com
A auth.some-domain.com. 5m00s 104.21.28.234
A auth.some-domain.com. 5m00s 172.67.147.200
Those IP's belong to CF proxy and the output confirms that CF is enabled again.
Some troubleshooting
The list below describes issues that I have experienced with CloudFlare and Traefik. It may not be exactly your case, but they are noted down in case you face them.
Browser throws
ERR_TOO_MANY_REDIRECTSwhen accessing a service: If you deployed using CloudFlare's DNS servers and set CF to 'Flexible mode' then you will incurr in this error. This is because you have https from the client to the CF proxy, and forced http between the CF proxy and the server (forced by CF). As Traefik was configured with an http-to-https redirection itself, this will cause an infinite loop of redirects. In order to solve this issue, you need to set CF to Full encryption mode, and even without LE (using Traefik's self signed certificate) your website/application is now accessible.-
Browser throws
ERR_SSL_VERSION_OR_CIPHER_MISMATCHwhen accessing a service: I ran into this error because the "edge certificates" (the certificates sitting at CloudFlare's proxy) did not have the same coverage for the routes I was setting up with Traefik.- I was setting up routes with the structure
service.platform.your-domain.com. This is called a 4th level subdomain. - The "edge" certificates provided by CloudFlare (the "edge" being CF's proxy) only provide SSL up to 3rd level subdomains, specifically, to
your-domain.comand*.your-domain.com. - Thus the LE certificates fetch by Traefik did not match the names "made available" by CloudFlare, and the error above was produced.
- The only solution is to upgrade to a Enterprise subscription to CloudFlare 💸, as this allows you to issue edge certificates for any level of subdomains you want.
- I was setting up routes with the structure
Closing words
I hope that you have found at least one solution to your problems if you're working with this particular set of tools (Docker Compose, Traefik, CloudFlare and Let's Encrypt). I may update this post as I add more functionalities to the platform I am developing.
Top comments (5)
Thx for sharing ~~
I also wrote an article "Deploying Traefik Proxy with Cloudflare Origin CA Certificate on k0s"
I came here today after struggling this week and this looks like everything I needed. Thank you so so much. I especially appreciate your breakdown into the "why" things are happening the way they are.
I've been running this configuration with the Cloudflare without paying for the Advanced Certificate Manager for months now. I followed TechnoTim: docs.technotim.live/posts/traefik-... and have not hit the SSL issue.
The domain I use is *.local.example.com
Thank you for sharing this! I originally followed the guide from TechnoTim on YouTube where I ran into the issue of ERR_SSL_VERSION_OR_CIPHER_MISMATCH
After tinkering around (finding out via the comment section that it can be done on a 4th level subdomain on the free plan), I made it work for some Docker containers. Since I am using Pihole, I tried setting local routes, which didn't seem to work too well, so I did it on my hosts' file. However, there's also a way to make it via CloudFlare Access by resolving internal hostnames via their gateway.
As for the Edge certificate limitation of the Free plan, there have been some changes recently with Wildcard proxy becoming available for everyone
Particularly at the end, it is mentioned: If you want to proxy a wildcard DNS record on a deeper level like *.mycoolwebpage.xyz
you can go through their Advanced Certificate Manager, which would only cost you $10 a month and not the enterprise cost...
I have been working on making this work for a while, maybe you have an opinion on which route would make better sense, as I run into a couple of issues with some of my docker containers. Such as the cipher mismatch and then I also have an HTTPS-only image, that ends up getting a Bad Gateway.
you made my day.
very detailed and explained the core issue clearly. really thanks.