IOT: Traefik Reverse Proxy for HTTPS Connection to Docker Containers

In your IOT home network, several applications are provided as Docker containers. Typically, containers expose a port on its host. To access the applications, you type in the IP address of the host and its exposed ports. This is ok if you have just one or two applications, but soon it will be hard to remember all the ports, and you are still using unencrypted HTTP which is not ideal when you use a WiFi connection to access the applications.

Both problems can be solved with a reverse proxy, an application that listens for incoming HTTP requests and forwards them to other applications. In this article, you will learn how to setup Traefik that enables to access Docker containers with custom URLs and HTTPs encryption. In essence, you will learn how to start Traefik alongside your other Docker containers, and then see make other Docker containers accessible by configuring them with container labels. Traefik uses these labels to auto-configure itself and then exposes the containers just as required.

The technical context of this article is Raspberry Pi OS 2022-09-22 and Traefik v2.8.0. All instructions should work with newer OS and library versions as well.

This article originally appeared at my blog admantium.com.

Traefik at a Glance

Traefik considers itself not only as a reverse proxy but as a universal edge router that can be deployed before several other platforms or architectures. Traefik calls these providers, and the list of supported providers is impressive: Docker, Consul, Nomad, Kubernetes and many more. Any application that should be exposed with Traefik needs to be defined with three configuration items:

• Routers: Each incoming request is processed by routers to determine if it should be forwarded. Routers are defined with a flexible rule set, including the host, the headers, the HTTP method, the path or the queries. You configure one or more rules that an incoming request needs to match, and then determine which middleware or service the request is forwarded too.
• Middleware: Optional configurations that modify the original request before it gets send to a service. Typically, you use this to manipulate the request headers, e.g. adding or dropping arbitrary headers, or you change the request path, e.g adding a prefix or using regular expressions. Other features include rate limiting or even adding basic auth.
• Service: Specify how to reach the applications themselves, and including configuration aspects such as TLS termination, sessions, or even load-balancing requests to multiple instances of the same application.
• Certificates: Enable TLS encryption for incoming requests by providing certificates. When using Traefik for publicly available hosts, you can use any SSL provider, or the free service Lets Encrypt. When using the Traefik in a local network, you need to create self-signed certificates.

These concepts enable a very flexible configuration that covers all parts of request processing. To further facilitate using Traefik in environments in which services get created and deleted dynamically, Traefik distinguishes its own configuration into static and dynamic. The static configuration considers immutable aspects of how Traefik itself should operate, such as its IP address, ports, whether to provide a dashboard, and the supported providers. The dynamic configuration specifies how a provider discovers new services and how to configure them so that they are exposed with Traefik. For example, when using Docker, Traefik needs to connect to the Docker daemon to get notifications about new or deleted containers, and then it invokes behavior configured for this particular event.

Overall, Traefik is an impressive application that serves all purposes of automatic and dynamic service discovery and configuration. Now, let’s see how to apply it for exposing Docker services.

Traefik Integration & Configuration for Exposing Docker Containers

To get Traefik running, following small steps are required:

• Add the Traefik container to the docker-compose.yml file
• Provide the static and dynamic configuration files
• Create the certificates that Traefik uses for encrypted traffic
• Start the Traefik container
• Configure and restart other Docker containers that should be exposed by Traefik
• Define DNS entries on all computers that should reach the containers

Step 1: Traefik Docker Configuration

Note: The following configuration files is based on the Github repository traefik-v2-https-ssl-localhost.

To get the container started, use the following template and replace the version string with the latest Traefik release. Also, use the same network name in which all other containers are running.

version: '3'

services:
  reverse-proxy:
    image: traefik:v2.8.8
    container_name: traefik
    restart: unless-stopped
    security_opt:
      - no-new-privileges:true
    ports:
      - 80:80
      - 443:443
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - ./volumes/traefik/config/traefik.yml:/etc/traefik/traefik.yml:ro
      - ./volumes/traefik/config/dynamic.yml:/etc/traefik/dynamic.yml:ro
      - ./volumes/traefik/certs:/etc/certs:ro
    networks:
      - iotstack_default

Step 2: Traefik Static and Dynamic Configuration

Note: The following configuration fails are based on the Github repository traefik-v2-https-ssl-localhost.

The basic content of the static file, which needs to be named traefik.yml, includes the following aspects:

• api: Enable the dashboard and allow plain HTTP traffic
• providers: Define the docker provider with its socket endpoint and whether to watch for changes. Additionally, define a file provider to inject the dynamic configuration.
• log: Configure which events should be logged and how they are formatted.
• entryPoints: Configure http and https ports to access the Traefik dashboard. You should only configure http during the initial setup of Traefik, and once things run, remove it.

Here is the complete file:

global:
  sendAnonymousUsage: false

api:
  dashboard: true
  insecure: false

providers:
  docker:
    endpoint: "unix:///var/run/docker.sock"
    watch: true
    exposedByDefault: false

  file:
    filename: /etc/traefik/dynamic.yml
    watch: true

log:
  level: INFO
  format: common

entryPoints:
  http:
    address: ":80"
  https:
    address: ":443"

In the file dynamic.yml, following aspects are defined:

• The traefik router specifies the rule how to reach the Traefik dashboard. Typically, a specific host name should be sufficient. In a local network, be sure to use the domain name local or localhost, or else your browser might refuse to load the page.
• The tls certificates for the domains that Traefik should respond to, specified as a list of pairs of certificate files and their key files.

Here is the complete configuration:

http:
  routers:
    traefik:
      rule: "Host(`traefik.docker.local`)"
      service: "api@internal"
      tls:
        domains:
          - main: "docker.local"
            sans:
              - "*.docker.local"
tls:
  certificates:
    - certFile: "/etc/certs/local-cert.pem"
      keyFile: "/etc/certs/local-key.pem"

Step 3: Certificate Creation

Certificate creation is a process involving many steps: you need to create a root certificate authority, then use this to create private key file, a signing request, and then sign the request using the private key file to obtain the certificate. Additionally, you need to install the root authority in your OS and browser keystore.

To get this done, you can use your operating systems cli tools such as openssl and follow these steps, or use a support tool such as mkcert. I used mkcert because it automizes all of these steps for you.

On the computer from which you want to access your docker containers, execute the following commands. Substitute the architecture with that of you OS, for example, for OsX it would be darwin-amd64 or darwin-arm64.

curl -JLO "https://dl.filippo.io/mkcert/latest?for=linux/amd64"
chmod +x mkcert-v*-linux-amd64
sudo cp mkcert-v*-linux-amd64 /usr/local/bin/mkcert

mkcert --install
mkcert docker.local *.docker.local

Then, copy the certificates to the specified location ./volumes/traefik/certs so that it can be used by Traefik.

Step 4: Start the Traefik Container

The Traefik containers' configuration is complete. Start it with docker-compose start -d traefik; docker logs -f traefik and study the log output to identify any configuration problems.

time="2023-01-15T18:05:59Z" level=info msg="Configuration loaded from file: /etc/traefik/traefik.yml"
time="2023-01-15T18:05:59Z" level=info msg="Traefik version 2.8.8 built on 2022-09-30T12:20:13Z"
time="2023-01-15T18:05:59Z" level=info msg="\nStats collection is disabled.\nHelp us improve Traefik by turning this feature on :)\nMore details on: https://doc.traefik.io/traefik/contributing/data-collection/\n"
time="2023-01-15T18:05:59Z" level=warning msg="Traefik Pilot is deprecated and will be removed soon. Please check our Blog for migration instructions later this year."
time="2023-01-15T18:05:59Z" level=info msg="Starting provider aggregator aggregator.ProviderAggregator"
time="2023-01-15T18:05:59Z" level=info msg="Starting provider *file.Provider"
time="2023-01-15T18:05:59Z" level=info msg="Starting provider *traefik.Provider"
time="2023-01-15T18:05:59Z" level=info msg="Starting provider *acme.ChallengeTLSALPN"
time="2023-01-15T18:05:59Z" level=info msg="Starting provider *docker.Provider"

You should see no errors with the default configuration, but use this command later on as well.

Step 5: Configure Docker Containers

When Traefik runs successfully, you now need to add your containers step by step. To discover the containers dynamically, Traefik watches changes in the Docker daemon. If a newly started container has certain labels, then Traefik will use and add it as a new service.

Lets discuss an example to see what is required - here is my definition for the home-assistant container:

home_assistant:
  container_name: home_assistant
  # ...
  ports:
  - "8123:8123"
  labels:
  - "traefik.enable=true"
  - "traefik.docker.network=iotstack_default"
  - "traefik.http.routers.home-assistant.rule=Host(`home-assistant.nexus.local`)"
  - "traefik.http.routers.home-assistant.tls=true"
  - "traefik.http.services.home-assistant.loadbalancer.server.port=8123"

In essence, you need the following labels:

• traefik.enable: Allow this container to be exposed by Traefik
• traefik.docker.network: The docker network name in which the container runs. The best way to get this information is to run docker network ls to see the list of all defined containers, and then to run docker network inspect $NETWORK_NAME to see which containers belong to it.
• traefik.http.routers.NAME.rule: This line defines the router rules that incoming requests need to match. Typically it should be enough to just use a specific hostname.
• traefik.http.routers.NAME.tls: Whether to configure TLS for this service.
• traefik.http.services.NAME.loadbalancer.server.port: Defines the port how to reach the service. You need to specify it whenever the container exposes multiple ports and when its HTTPS Traffic should not be sent to port 443.

Once you have declared these labels for a service in your docker-compose.yml file, run the following command to re-create the container and then Traefik:

docker-compose up --force-recreate --build -d home_assistant; docker logs -f home_assistant
docker-compose up --force-recreate --build -d traefik; docker logs -f traefik

When the log messages do not show any error, head over to the dashboard, and see the configured service:

Repeat these steps for each container, and always check if you can reach it.

Step 6: Define DNS Entries

Finally, you need to define local DNS entries to reach the services. You can either do this on your computer, for example for OsX and Linux you would add new entries to the file /etc/hosts, like this:

home-assistant.docker.local 192.168.4.11

Or you configure your local DHCP server to forward all requests with the specific domain to the IP address of the server that hosts the docker containers.

Troubleshooting

Access Docker Containers that run on Host Network

If you have a container that runs as network: host, Traefik can pick this container up to. When you run a container like this, the Docker daemon puts them in a special network, the docker0 interface with IP address

Add the following declaration to the Traefik container:

traefik:
  image: traefik:v2.8.8
  # ...
  extra_hosts:
  - host.docker.internal:172.17.0.1

The target container definition uses the very same labels as before, but you do not specify the traefik.docker.network.

Enable Reverse Proxying in Docker Services

Not all containers will run out of the box with a reverse proxy configuration.

In my case, one of them is Home Assistant. Looking into the log files, I could see the following error message:

home_assistant    | 2023-01-15 11:07:03 ERROR (MainThread) [homeassistant.components.http.forwarded] A request from a reverse proxy was received from 192.168.4.200, but your HTTP integration is not set-up for reverse proxies

In Home Assistant, reverse proxying needs to be explicitly allowed by adding the following configuration option to the Home Assistant configuration file:

http:
  use_x_forwarded_for: true
  trusted_proxies:
    - 172.31.0.0/22
    - 192.168.4.0/22

Similarly, if you have trouble with other applications, check the log file and then search the internet how to enable the application to work with a reverse proxy server.

Conclusion

Traefik is an edge router with a very flexible configuration that can be used to expose services hosted in Docker, Consul, Nomad, and Kubernetes. By combining routers, middleware and service configurations, incoming requests are matched on host or path, optionally headers added, and paths changed, and finally forwarded to the services. Traefik enables HTTPS encryption and works with official or self-signed certificates. In this article, you learned how to use Traefik for accessing local docker containers via a hostname and HTTPs. The article showed all required steps: a) add a Traefik service to your docker-compose.yml file, b) provide the static and dynamic configuration, c) add certificates, d) start the Traefik container and watch its log output to detect configuration errors, e) configure individual docker containers to be accessible from Traefik, and f) define DNS entries to reach the containers.

Comments

[Marcelo Facio Palin]

Congratulations on the explanation. Just with the intention of helping beginners, it's worth mentioning that if they are using Zsh, the command mkcert docker.local .docker.local should be mkcert 'docker.local' '.docker.local'. It's just a small detail, but it can help in following the article without losing focus. Thank you.