DEV Community: karvounis

Introducing a Terraform provider for Forem

karvounis — Thu, 14 Apr 2022 06:44:36 +0000

I am glad to announce that I have just released a Terraform provider for Forem! This provider is my small contibution to this amazing open-source project. You can find its source code on Github and the provider's page on Terraform registry.

TL;DR

Manage your Forem resources using code! Forem as Code (FaC) following the Infrastructure as Code (IaC) concept.

Need

Forem is a great software project and the platform that powers dev.to. When I published my first article in dev.to, I found it very easy to navigate through the browser interface of Forem, create, write, and edit the article.

However, while I was writing the second article, I realised that there are a lot of similarities between them and that certain things can be automated. I also realised that the more articles you publish, the harder their maintenance is going to get. Using just the browser does not scale well and there is certainly room for automation.

I decided to use Terraform to automate the management of Forem resources. Quoted from their website:

Terraform is an open-source infrastructure as code software tool that provides a consistent CLI workflow to manage hundreds of cloud services. Terraform codifies cloud APIs into declarative configuration files.

If there is an API, you can create a Terraform provider for it!

API

Forem has an API that you can use in order to create and update articles, listings and a few other resources. Instead of using the browser, you can send HTTP requests to the api endpoint of the Forem installation and read, create or update the resources you need.

Example

You can either visit the linkg https://dev.to/karvounis/basic-traefik-configuration-tutorial-593m from your browser or send the following curl request:

curl https://dev.to/api/articles/karvounis/basic-traefik-configuration-tutorial-593m

How to use it

The provider requires two arguments:

api_key (String) API key to be able to communicate with the FOREM API. Can be specified with the FOREM_API_KEY environment variable.
host (String) Host of the FOREM API. You can specify the dev.to or any other Forem installation. Can be specified with the FOREM_HOST environment variable. Default: https://dev.to/api.

In order to generate an API key, go to Settings -> Account -> DEV Community API Keys, give it a proper description and press the Generate API Key button.

Forem Articles through Terraform

You can create a new Forem article using the examples shown here. In this example, both example_file and example_full articles use the same tags that are defined in the locals block.

Imagine having way more articles maintained by Terraform and you need to remove a specific tag from all of them. With this provider, you can just remove that tag from the list, re-run the plan and Terraform will be smart enough to understand the difference and remove this tag from all the articles. By doing that, we saved ourselves a lot of clicks!

Currently supported

Currently, the lastest Terraform provider version is 1.0.1 and supports the following Forem resources and actions:

Resources	Actions
Articles	Create, Update, Read, Import
Listings	Create, Update, Read, Import

Data Sources	Actions
Articles	Get By ID, Get By username and slug
Listings	Get By ID
Users	Get By ID, Get By username

Conclusion

I really hope that this provider is going to help all people using Forem. In fact, this very article has been generated using this provider! Don’t take my word for it, check the Terraform article resource and the actual Markdown file! 😉

Any contributions to the provider are welcome and I would appreciate any feedback in the comments section! Cheers!

Useful links

Advanced Traefik configuration tutorial - TLS, dashboard, ping, metrics, authentication and more

karvounis — Thu, 24 Mar 2022 18:46:20 +0000

Introduction

This tutorial is the second part of the Traefik series. The first part can be found here.

In the previous tutorial, the basic Traefik concepts were explained and we showed a simple Traefik configuration running in standalone Docker. In this tutorial, we are going to cover some advanced concepts such as TLS, authentication and chain middlewares, the Traefik dashboard, Traefik metrics for Prometheus, and healthchecks.

The codebase for this tutorial can be found here. All docker-compose files that appear in the Traefik tutorials can be found here.

Prerequisites

Docker
Docker compose

All docker compose files have been tested with Docker 20.10.12 and docker-compose 1.24.0.

Create the required Docker networks



docker network create traefik_public
docker network create socket_proxy

TL;DR

Advanced concepts

TLS

The full docker-compose file for this section can be found here.

Traefik can be configured to accept incoming HTTPS connections in order to terminate the SSL connections (meaning that it will send decrypted data to the services). It can be configured to use an ACME provider (like Let's Encrypt) for automatic certificate generation. However, we are not going to cover this as there is already a plethora of very informative material on the subject online. In this tutorial, we are going to create our own CA and Traefik certificates and configure Traefik to use them.

Certificates

We need to create our own TLS certificates in order to have encryption in transit and properly secure the communication from and to Traefik. I have already generated some certificates which can be found here.

However, you can create your own certificates by running the following commands:



mkdir -p certs/{ca,traefik}
# Create CA certificates
openssl genrsa -out certs/ca/rootCA.key 4096
openssl req -x509 -new \
    -nodes \
    -sha256 \
    -days 3650 \
    -key certs/ca/rootCA.key \
    -subj "/C=GR/L=Athens/O=Karvounis Tutorials, Inc./CN=Karvounis Root CA/OU=CA department" \
    -out certs/ca/rootCA.pem
# Create Traefik wildcard certificates
openssl genrsa -out certs/traefik/traefik.key 4096
openssl req -new \
    -key certs/traefik/traefik.key \
    -subj "/C=GR/L=Athens/O=Karvounis Tutorials, Inc./CN=*.karvounis.tutorial/OU=Dev.to" \
    -out certs/traefik/traefik.csr
openssl x509 -req \
    -sha256 \
    -days 365 \
    -CA certs/ca/rootCA.pem \
    -CAkey certs/ca/rootCA.key \
    -CAcreateserial \
    -in certs/traefik/traefik.csr \
    -out certs/traefik/traefik.crt

The above commands will create the necessary certificates under the following directory structure:



$ tree certs/
certs/
├── ca
│   ├── rootCA.key
│   ├── rootCA.pem
│   └── rootCA.srl
└── traefik
    ├── traefik.crt
    ├── traefik.csr
    └── traefik.key

The generated traefik certificate is a wildcard certificate for *.karvounis.tutorial and will cover all the use cases for the tutorials. From now on, we are going to use these certificates in every docker-compose file.

Service configuration



traefik:
    image: traefik:v2.6
    command:
    # Entrypoints configuration
    - --entrypoints.web.address=:80
    ## Create a new entrypoint called `websecure` that is going to be used for TLS
    - --entrypoints.websecure.address=:443
    ## Forces redirection of incoming requests from `web` to `websecure` entrypoint
    ## https://doc.traefik.io/traefik/routing/entrypoints/#redirection
    - --entrypoints.web.http.redirections.entryPoint.to=websecure
    # Docker provider configuration
    - --providers.docker=true
    - --providers.docker.exposedbydefault=false
    - --providers.docker.endpoint=tcp://socket_proxy:2375
    - --providers.docker.network=traefik_public
    # File provider configuration
    - --providers.file.directory=/traefik/config/my_dynamic_conf
    # Logging configuration
    - --log.level=info
    - --log.format=json
    ports:
    - 80:80
    - 443:443
    volumes:
    - ./certs/traefik:/traefik/config/certs:ro
    - ./config.yml:/traefik/config/my_dynamic_conf/conf.yml:ro
    networks:
    - traefik_public
    - socket_proxy
    restart: unless-stopped
    depends_on:
    - socket_proxy

whoami:
    image: traefik/whoami:v1.7.1
    labels:
      - traefik.enable=true
      - traefik.http.routers.whoami_route.entrypoints=websecure
      - traefik.http.routers.whoami_route.rule=Host(`whoami.karvounis.tutorial`)
      - traefik.http.routers.whoami_route.service=whoami_service
      - traefik.http.routers.whoami_route.tls=true
      - traefik.http.services.whoami_service.loadbalancer.server.port=80
    networks:
      - traefik_public

Entrypoints configuration

--entrypoints.websecure.address=:443

Defines an entrypoint called websecure that will listen on port 443 of the Traefik container. This entrypoint is going to be used for all the TLS connections.

--entrypoints.web.http.redirections.entryPoint.to=websecure

It enables permanent redirecting of all incoming requests from the web entrypoint to the websecure entrypoint. That means that even if someone tries to send an HTTP request, that request will be redirected to HTTPS.

File provider configuration

TLS certification configuration is part of the dynamic configuration of Traefik. Unfortunately, we cannot use the Docker provider in order to dynamically configure tls certificates using labels. We have to use the File provider instead.

--providers.file.directory=/traefik/config/my_dynamic_conf

Points to the directory where Traefik can load the dynamic configuration from. In our case, we are going to mount a config.yml file that contains the paths to our certificates.

Volumes



volumes:
    - ./certs/traefik:/traefik/config/certs:ro
    - ./config.yml:/traefik/config/my_dynamic_conf/conf.yml:ro

Mounts the local ./certs/traefik folder and its contents (the Traefik certificates) to the /traefik/config/certs inside the container. Local ./config.yml file, that contains the dynamic configuration, will be mounted inside the /traefik/config/my_dynamic_conf/ directory, which is the directory that Traefik looks for its dynamic configuration from.

The path of Traefik's public key and private key in the container are /traefik/config/certs/traefik.crt and /traefik/config/certs/traefik.key respectively.

Config

The following config file, which can be found here, is used to define the paths in the container for the certificate and the key.



tls:
  certificates:
    - certFile: /traefik/config/certs/traefik.crt
      keyFile: /traefik/config/certs/traefik.key

whoami service labels

We are going to enable TLS for the whoami_route. This can be achieved by changing the value of the traefik.http.routers.whoami_route.entrypoints to websecure (the HTTPS entrypoint) and setting the traefik.http.routers.whoami_route.tls label to true.



labels:
    - traefik.enable=true
    - traefik.http.routers.whoami_route.entrypoints=websecure
    - traefik.http.routers.whoami_route.rule=Host(`whoami.karvounis.tutorial`)
    - traefik.http.routers.whoami_route.service=whoami_service
    - traefik.http.routers.whoami_route.tls=true
    - traefik.http.services.whoami_service.loadbalancer.server.port=80

Deployment

Deploy the containers by executing the following command:



docker-compose -f docker-compose.tls.yml up -d

Requests

First, we are going to send a request to http://whoami.karvounis.tutorial.



$ curl -H "Host: whoami.karvounis.tutorial" \
    http://localhost
# OR curl http://whoami.karvounis.tutorial
Moved Permanently

The response is Moved Permanently due to the redirection that we configured with this command: --entrypoints.web.http.redirections.entryPoint.to=websecure. Every request to the HTTP web entrypoint will be automatically redirected to the HTTPS websecure entrypoint!

Let's try to directly hit the HTTPS entrypoint:



$ curl --cacert ./certs/ca/rootCA.pem \
    https://whoami.karvounis.tutorial
Hostname: 5f1a8f92cd2b
IP: 127.0.0.1
IP: 172.19.0.2
RemoteAddr: 172.19.0.3:53910
GET / HTTP/1.1
Host: whoami.karvounis.tutorial
User-Agent: curl/7.68.0
Accept: */*
Accept-Encoding: gzip
X-Forwarded-For: 172.20.0.1
X-Forwarded-Host: whoami.karvounis.tutorial
X-Forwarded-Port: 443
X-Forwarded-Proto: https
X-Forwarded-Server: 40e915108da5
X-Real-Ip: 172.20.0.1

Success! Sending a request to the HTTPS entrypoint and specifying the public key of the CA, returned the expected response.

Ping

The full docker-compose file for this section can be found here.

Traefik provides a ping endpoint that, when enabled, can be used to check the health of the Traefik instance.

Configuration changes

--ping=true

Enables the /ping healthcheck URL. However, we are not going to expose it using a router. Instead, we are going to use the URL to check the health of the Docker container by leveraging docker-compose's healthcheck option.

healthcheck configuration

Every 10 seconds, Docker is going to execute the command traefik healthcheck --ping to establish the health of each Traefik instance (docs). If the command is unsuccessful for 3 consecutive times, Docker will mark the container as unhealthy and will restart it.



healthcheck:
    # Run traefik healthcheck command
    # https://doc.traefik.io/traefik/operations/cli/#healthcheck
    test: ["CMD", "traefik", "healthcheck", "--ping"]
    interval: 10s
    timeout: 5s
    retries: 3
    start_period: 5s

Deployment

Deploy the containers by executing the following command:



docker-compose -f docker-compose.ping.yml up -d

Check the status of the traefik service by executing:



docker-compose -f docker-compose.ping.yml ps traefik

After a few seconds, traefik service's status will change from starting to healthy.

Dashboard

The full docker-compose file for this section can be found here.

Traefik offers a dashboard where you can view all the active routers, services and middlewares. In this section, we are going to find out how to enable the dashboard and how to configure the routers to be able to access it.

Traefik service configuration

This is the first time we are going to add Docker labels to the traefik service. They are going to define a new router, called dashboard, which will only be accessible through TLS.

In order to enable the dashboard and the api, you have to add the --api.dashboard=true to the command configuration option of the traefik service.

Service labels

The Docker labels below define a new router called dashboard. This router uses a Host Based rule as well as two PathPrefix rules to be able to match all the necessary requests. This router is only accessible through the websecure entrypoint.



labels:
    - traefik.enable=true
    - traefik.http.routers.dashboard.rule=Host(`traefik.karvounis.tutorial`) && (PathPrefix(`/api`) || PathPrefix(`/dashboard`))
    - traefik.http.routers.dashboard.tls=true
    - traefik.http.routers.dashboard.entrypoints=websecure
    - traefik.http.routers.dashboard.service=api@internal

Deployment

Deploy the containers by executing the following command:



docker-compose -f docker-compose.dashboard.yml up -d

UI

You can access the dashboard by visiting https://traefik.karvounis.tutorial/dashboard/ and the api at https://traefik.karvounis.tutorial/api/rawdata.

Tip: Do not forget the trailing slash / in /dashboard/!

Authentication

The full docker-compose file for this section can be found here.

After exposing the dashboard in the previous section, it is clear that we need to secure it and restrict access only to authenticated users.
Traefik offers the following HTTP Authentication middlewares:

In this section, we are going to use the BasicAuth middleware to secure the dashboard router and the DigestAuth to secure the whoami router.

Configuration changes

traefik service:



labels:
    - traefik.enable=true
    - traefik.http.routers.dashboard.rule=Host(`traefik.karvounis.tutorial`) && (PathPrefix(`/api`) || PathPrefix(`/dashboard`))
    - traefik.http.routers.dashboard.tls=true
    - traefik.http.routers.dashboard.entrypoints=websecure
    - traefik.http.routers.dashboard.service=api@internal
    # Middlewares
    - traefik.http.routers.dashboard.middlewares=dashboard_auth
    ## Creates 2 authentication middlewares
    ### `dashboard_auth` is a BasicAuth middleware and is going to be used by the `dashboard` router.
    ### dashboard:tutorial
    - traefik.http.middlewares.dashboard_auth.basicauth.users=dashboard:$$2y$$05$$T/WVjQVqBc24NLUNI/xuVu0V2B.RPY50k2.CCH5JHGInb3EUeaDcO
    ### `auth` is a DigestAuth middleware and is going to be used by the `whoami_route` router.
    ### whoami:tutorial
    - traefik.http.middlewares.digest_auth.digestauth.users=whoami:traefik:f4ba293a96d5dcf51eb2f03b5931dd96

whoami service:



    labels:
      - traefik.enable=true
      - traefik.http.routers.whoami_route.entrypoints=websecure
      - traefik.http.routers.whoami_route.rule=Host(`whoami.karvounis.tutorial`)
      - traefik.http.routers.whoami_route.service=whoami_service
      - traefik.http.routers.whoami_route.tls=true
      # `whoami_route` uses the `digest_auth` middleware defined in the `traefik` service
      - traefik.http.routers.whoami_route.middlewares=digest_auth
      - traefik.http.services.whoami_service.loadbalancer.server.port=80

traefik.http.middlewares.dashboard_auth.basicauth.users

This label creates a new BasicAuth middleware called dashboard_auth. It contains the user with the credentials dashboard:tutorial and can contain an array of authorized users. If you have a great number of users, you can also add their credentials to a file, mount that file to the container and specify the usersFile option to point to that file.

You can generate the passwords with the following ways:



# Using the httpd:2.4-alpine image which is 58.2MB
$ docker run --rm httpd:2.4-alpine htpasswd -nbB dashboard tutorial | sed -e s/\\$/\\$\\$/g
# OR with `xmartlabs/htpasswd` docker image which is 9MB
$ docker run --rm -ti xmartlabs/htpasswd dashboard tutorial | sed -e s/\\$/\\$\\$/g
# OR without docker
$ htpasswd -nbB dashboard tutorial | sed -e s/\\$/\\$\\$/g

dashboard:$$2y$$05$$T/WVjQVqBc24NLUNI/xuVu0V2B.RPY50k2.CCH5JHGInb3EUeaDcO

Tip: when used in docker-compose.yml, all dollar signs in the hash need to be doubled for escaping!

traefik.http.middlewares.digest_auth.digestauth.users

This label creates a new DigestAuth middleware called digest_auth. It contains the user with the credentials whoami:tutorial and can contain an array of authorized users. The usersFile option is available here as well.

You can generate the digest credentials with the following commands:



$ print whoami:traefik:$(printf whoami:traefik:tutorial | md5sum | awk '{print $1}')
# OR with htdigest `htdigest [-c] passwordfile realm username` and type the password
$ htdigest -c /tmp/pwd_file traefik whoami && cat /tmp/pwd_file

whoami:traefik:f4ba293a96d5dcf51eb2f03b5931dd96

traefik.http.routers.dashboard.middlewares=dashboard_auth

Instructs the dashboard router to use the dashboard_auth as authentication middleware.

traefik.http.routers.whoami_route.middlewares=digest_auth

Instructs the whoami_route router to use the digest_auth as authentication middleware.

Tip: You can use middlewares defined in other services!

Deployment

Deploy the containers by executing the following command:



docker-compose -f docker-compose.auth.yml up -d

Requests

If you try to access the https://traefik.karvounis.tutorial/api/rawdata URL like before, you are going to get a 401 response status code.



$ curl --cacert ./certs/ca/rootCA.pem \
    https://traefik.karvounis.tutorial/api/rawdata
401 Unauthorized

In order to access the Traefik api using curl, you have to specify the basic auth user credentials.



$ curl --cacert ./certs/ca/rootCA.pem \
    -u dashboard:tutorial \
    https://traefik.karvounis.tutorial/api/version
{"Version":"2.6.0","Codename":"rocamadour","startDate":"2022-02-23T17:42:57.897252485Z","pilotEnabled":true}

If you want to access the whoami service, you need to specify the digest credentials of the whoami authorized user. Otherwise, Traefik is going to respond with a 401 as above.



$ curl --cacert ./certs/ca/rootCA.pem \
    --digest -u whoami:tutorial \
    https://whoami.karvounis.tutorial
Hostname: 637aadaf37b3
IP: 127.0.0.1
IP: 172.19.0.2
RemoteAddr: 172.19.0.3:40984
GET / HTTP/1.1
Host: whoami.karvounis.tutorial
User-Agent: curl/7.68.0
Accept: */*
Accept-Encoding: gzip
Authorization: Digest username="whoami", realm="traefik", nonce="2ZZJLL5tPk8bEDU8", uri="/", cnonce="YzUxNzMyZWIyODNjN2VlNDIyMDkyMmY3Nzc3YjVkNDE=", nc=00000001, qop=auth, response="518ade9a297beb2f5174e2368e8cf561", opaque="pfm5w5vBpLYhEv7A", algorithm="MD5"
X-Forwarded-For: 172.20.0.1
X-Forwarded-Host: whoami.karvounis.tutorial
X-Forwarded-Port: 443
X-Forwarded-Proto: https
X-Forwarded-Server: 5966f55786bb
X-Real-Ip: 172.20.0.1

Chain middleware

The full docker-compose file for this section can be found here.

In this section, we are going to create a new Chain middleware and instruct whoami_route to use it. This middleware consists of two middlewares:

A new HTTP RateLimit middleware called simple_ratelimit. This middleware is going to limit the maximum amount of allowed requests to the whoami_service service in a particular time period.
digest_auth which we have already seen in the Authentication section.

The chain middleware is first going to pass the request through the simple_ratelimit and then through the digest_auth middleware. If the request manages to successfully pass both, then it will reach the whoami_service. Traefik's documentation does not specify a hard limit for the amount of middlewares that you can chain.

Configuration changes

We are going to create the new simple_ratelimit and secured_chain middlewares by adding the following labels to the traefik service:



labels:
    - traefik.http.middlewares.secured_chain.chain.middlewares=simple_ratelimit,digest_auth
    ## The `simple_ratelimit` middleware allows
    ## an average of 5 requests per 5 seconds
    ## and a burst of 2 requests.
    - traefik.http.middlewares.simple_ratelimit.ratelimit.average=5
    - traefik.http.middlewares.simple_ratelimit.ratelimit.period=5s
    - traefik.http.middlewares.simple_ratelimit.ratelimit.burst=2

Based on the above numbers, the maximum allowed request rate is r=average/period=5/5s=1 request/second. If we exceed that rate, the requests are going to be automatically rejected with a 429 HTTP status code.

We also need to instruct whoami_route to use the secured_chain middleware:



labels:
    # Use the `secured_chain` chain middleware
    - traefik.http.routers.whoami_route.middlewares=secured_chain

Deployment

Deploy the containers by executing the following command:



docker-compose -f docker-compose.chain.yml up -d

Requests

Sending a simple curl request to https://whoami.karvounis.tutorial will work as before. Nothing out of the ordinary yet.



curl --cacert ./certs/ca/rootCA.pem \
    --digest -u whoami:tutorial \
    https://whoami.karvounis.tutorial

The following command is going to send a curl request to https://whoami.karvounis.tutorial every 1.5 seconds. All 5 requests are going to succeed because the rate is slower than the maximum allowed request rate.



SLEEP_TIMER=1.5s
for i in {1..5};
do
    echo "\nRequest number: $i"
    curl --cacert ./certs/ca/rootCA.pem \
        --digest -u whoami:tutorial \
        https://whoami.karvounis.tutorial
    sleep "${SLEEP_TIMER}"
done

On the other hand, the following command is going to send a curl request to https://whoami.karvounis.tutorial every 0.5 seconds. This request rate of 2 requests/sec is faster that the maximum allowed rate! The very first request will succeed but all the subsequent ones will be rejected with a 429 Too Many Requests HTTP status code.



SLEEP_TIMER=0.5s
for i in {1..5};
do
    echo "\nRequest number: $i"
    curl --cacert ./certs/ca/rootCA.pem \
        --digest -u whoami:tutorial \
        https://whoami.karvounis.tutorial
    sleep "${SLEEP_TIMER}"
done

Metrics

The full docker-compose file for this section can be found here.

Currently, Traefik supports 4 metrics backends:

Datadog
InfluxDB
Prometheus
StatsD

In this section, we are going to expose Traefik's metrics for Prometheus.

Configuration changes

First, we are going to enable the prometheus backend and disable the default internal router in order to allow one to create a custom router to the prometheus@internal service.



command:
    # Prometheus metrics
    ## Enable prometheus metrics
    - --metrics.prometheus=true
    ## Create a manual router instead of the default one.
    - --metrics.prometheus.manualrouting=true
    - --metrics.prometheus.addrouterslabels=true
    ...

The custom metrics router exposes the metrics through https://traefik.karvounis.tutorial/metrics and uses the dashboard_auth BasicAuth middleware for authentication.



labels:

    # metrics router configuration

    - traefik.http.routers.metrics.rule=Host(traefik.karvounis.tutorial) && PathPrefix(/metrics)

    - traefik.http.routers.metrics.tls=true

    - traefik.http.routers.metrics.entrypoints=websecure

    - traefik.http.routers.metrics.service=prometheus@internal

    - traefik.http.routers.metrics.middlewares=dashboard_auth

    ...

Deployment

Deploy the containers by executing the following command:



docker-compose -f docker-compose.metrics.yml up -d

Requests

Send the following request to https://traefik.karvounis.tutorial/metrics. The response contains all the Prometheus metrics for the particular Traefik instance.



curl --cacert ./certs/ca/rootCA.pem </span>

    -u dashboard:tutorial </span>

    https://traefik.karvounis.tutorial/metrics

Final notes

I hope you enjoyed this advanced tutorial on Traefik. Even more interesting tutorials are on the way! Please, let me know your thoughts in the comments section below! Cheers

You can find me on LinkedIn and Github.

Basic Traefik configuration tutorial

karvounis — Sat, 12 Feb 2022 12:20:39 +0000

Introduction

This tutorial aims to introduce Traefik, explain its basic components and show a basic configuration in Docker. I ❤️ Traefik and I hope you are going to love it too! We will go through the following:

Cover basic Traefik v2.X concepts. In this tutorial, we are going to use Traefik v2.6.0.
Setup and configure Traefik using the Docker provider in Standalone Engine mode.
Provide a docker-compose file to achieve the above.
Deploy a simple whoami service that Traefik is going to forward requests to.
- Scale whoami service to show how Traefik loadbalances the requests.

The codebase for this tutorial can be found here. All docker-compose files that appear in the Traefik tutorials can be found here.

What is Traefik

Traefik is an open-source reverse proxy (or Edge Router) and a loadbalancer written in Go. Its greatest strength is the dynamic generation of routes to your deployed services without any manual intervention.

Why Traefik

In modern architecture, a number of different services are deployed and each one of them has X amount of instances running at the same time. When you receive a request for a particular service, you must find a way to route this request to a healthy instance of this service and provide the response. What is more, the number of instances of a particular service can frequently change based on various factors such as traffic load and cpu and memory usage. You must find a way to dynamically update the routes to your services, route requests only to healthy instances and discard the unhealthy ones.

Traefik solves the above problem by dynamically updating the available routes to each service and their respective instances, thus making service discovery easy. Just imagine how difficult, error-prone and irritating task would be to manually update the routes whenever a service gets created/destroyed/scaled up or down, which can happen all too often during the day.

Prerequisites

Docker
Docker compose

All docker compose files have been tested with Docker 20.10.12 and docker-compose 1.24.0.

Basic concepts

Providers

Providers are the infrastructure components that Traefik is using for configuration discovery. Under the bonnet, Traefik queries the providers' APIs and based on the information it receives, it dynamically updates the routes.

There are a number of supported providers like Docker, ECS, Kubernetes, Consul, Rancher etc. In this tutorial, we are going to use the Docker provider and use container labels for routing configuration.

Entrypoints

Entrypoints listen for incoming traffic on specified ports and for a specific protocol (TCP or UDP). The most popular entrypoints are the ones that listen on ports 80 and 443. Entrypoints are part of the static configuration of Traefik, which means that you have to define them using a file (YAML or TOML) or CLI arguments.

Routers

Routers analyse the incoming requests and based on a set of rules make sure that the requests end up on the appropriate services. They may also use middlewares before forwarding the request.

Middlewares

Middlewares can be attached to routers and can be used to analyse/enhance/change/reject the requests before they reach the services. If wanted, they can be chained together and some common use cases are authentication, redirection, path modification etc.

Services

Services configure the way to forward the requests to your actual services. They configure things like load balancing (round-robin only as of version 2.6), health checks, sticky sessions etc.

Setup and configure Traefik

Preparation

Pull the necessary images



docker pull traefik:v2.6
docker pull tecnativa/docker-socket-proxy:latest
docker pull traefik/whoami:v1.7.1

Create the Docker networks that we are going to use



docker network create traefik_public
docker network create socket_proxy

traefik_public is going to be used by every service that needs to be exposed by Traefik.
socket_proxy is just going to be used by traefik and socket_proxy services to communicate and isolated from the services that need exposure through Traefik.

Configuration explanation

Our setup consists of three Docker services:

traefik: Listens to host port 80 and forwards the requests to any appropriate routes.
whoami: Defines a router and a service using Docker labels. The goal is to serve requests to this service while sitting behing the traefik reverse proxy.
socket_proxy: A security-enhanced proxy for the Docker Socket. It is only going to be used by Traefik.

We are now going to explain the configuration bit by bit. Tip: Skip to the full docker-compose file here.

Traefik service



  traefik:
    image: traefik:v2.6
    command:
      # Entrypoints configuration
      - --entrypoints.web.address=:80
      # Docker provider configuration
      - --providers.docker=true
      # Makes sure that services have to explicitly direct Traefik to expose them
      - --providers.docker.exposedbydefault=false
      # Use the secure docker socket proxy
      - --providers.docker.endpoint=tcp://socket_proxy:2375
      # Default docker network to use for connections to all containers
      - --providers.docker.network=traefik_public
      # Logging levels are DEBUG, PANIC, FATAL, ERROR, WARN, and INFO.
      - --log.level=info
    ports:
      - 80:80

Entrypoints configuration

--entrypoints.web.address=:80

Defines an entrypoint called web that will listen on port 80 of the Traefik container.
By specifying the ports using the short syntax HOST_PORT:CONTAINER_PORT



ports:
  - 80:80

you are publishing the port 80 inside the container to the host's port 80. That way, Traefik is going to receive the requests heading to localhost:80 or http://localhost.

Docker provider configuration

--providers.docker=true

Enables the docker provider.

--providers.docker.exposedbydefault=false

Only services that have the Docker label traefik.enable=true will be discovered and added to the routing configuration.

--providers.docker.endpoint=tcp://socket_proxy:2375

Instead of connecting directly to unix:///var/run/docker.sock we are going to use the socket_proxy container to be able to query the Docker endpoint as a security precaution.

--providers.docker.network=traefik_public

All exposed services through Traefik are going to use by default the traefik_public Docker network.

Whoami labels

The Docker service whoami is going to be exposed and only reachable through Traefik.



  whoami:
    image: traefik/whoami:v1.7.1
    labels:
      # Explicitly instruct Traefik to expose this service
      - traefik.enable=true
      # Router configuration
      ## Listen to the `web` entrypoint
      - traefik.http.routers.whoami_route.entrypoints=web
      ## Rule based on the Host of the request
      - traefik.http.routers.whoami_route.rule=Host(`whoami.karvounis.tutorial`)
      - traefik.http.routers.whoami_route.service=whoami_service
      # Service configuration
      ## 80 is the port that the whoami container is listening to
      - traefik.http.services.whoami_service.loadbalancer.server.port=80

traefik.enable=true

Explicitly instructs Traefik to add it to the routing configuration.

traefik.http.routers.whoami_route.entrypoints=web

The whoami_route route accepts requests only from the web entrypoint (port 80).

traefik.http.routers.whoami_route.rule=Host(`whoami.karvounis.tutorial`)

This label adds the Host rule. If a request's domain (host header value) is whoami.karvounis.tutorial then this router becomes active and forwards the request to the service.

traefik.http.routers.whoami_route.service=whoami_service

Service to use if the request matches the criteria of the whoami_route.

traefik.http.services.whoami_service.loadbalancer.server.port=80

The whoami_service service is going to send the request to the port 80 of the container (the value 80 is the default value for this setting). Useful when the default container port that a service is listening to is not 80 (i.e. Portainer is listening to 9000 so in that case you would configure that service like this traefik.http.services.portainer_service.loadbalancer.server.port=9000).

Complete configuration



version: "3.7"

services:
  traefik:
    image: traefik:v2.6
    command:
      # Entrypoints configuration
      - --entrypoints.web.address=:80
      # Docker provider configuration
      - --providers.docker=true
      # Makes sure that services have to explicitly direct Traefik to expose them
      - --providers.docker.exposedbydefault=false
      # Use the secure docker socket proxy
      - --providers.docker.endpoint=tcp://socket_proxy:2375
      # Default docker network to use for connections to all containers
      - --providers.docker.network=traefik_public
      # Logging levels are DEBUG, PANIC, FATAL, ERROR, WARN, and INFO.
      - --log.level=info
    ports:
      - 80:80
    networks:
      - traefik_public
      - socket_proxy
    restart: unless-stopped
    depends_on:
      - socket_proxy

  # https://github.com/traefik/whoami
  whoami:
    image: traefik/whoami:v1.7.1
    labels:
      # Explicitly instruct Traefik to expose this service
      - traefik.enable=true
      # Router configuration
      ## Listen to the `web` entrypoint
      - traefik.http.routers.whoami_route.entrypoints=web
      ## Rule based on the Host of the request
      - traefik.http.routers.whoami_route.rule=Host(`whoami.karvounis.tutorial`)
      - traefik.http.routers.whoami_route.service=whoami_service
      # Service configuration
      ## 80 is the port that the whoami container is listening to
      - traefik.http.services.whoami_service.loadbalancer.server.port=80
    networks:
      - traefik_public

  # https://github.com/Tecnativa/docker-socket-proxy
  # Security-enhanced proxy for the Docker Socket
  socket_proxy:
    image: tecnativa/docker-socket-proxy:latest
    restart: unless-stopped
    environment:
      NETWORKS: 1
      SERVICES: 1
      CONTAINERS: 1
      TASKS: 1
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
    networks:
      - socket_proxy

networks:
  traefik_public:
    external: true
  socket_proxy:
    external: true

Deploy the containers

Deploy the containers by executing the following command:



docker-compose up -d

Send a request to Traefik with the Host header set to whoami.karvounis.tutorial. This request matches the Host rule of the whoami_route router and will be forwarded to the whoami_service service. The response's Hostname field is the ID of the whoami Docker container.



$ curl -H "Host: whoami.karvounis.tutorial" http://localhost/

Hostname: ed1b87c345ad
IP: 127.0.0.1
IP: 172.19.0.2
RemoteAddr: 172.19.0.3:49146
GET / HTTP/1.1
Host: whoami.karvounis.tutorial
User-Agent: curl/7.68.0
Accept: */*
Accept-Encoding: gzip
X-Forwarded-For: 172.20.0.1
X-Forwarded-Host: whoami.karvounis.tutorial
X-Forwarded-Port: 80
X-Forwarded-Proto: http
X-Forwarded-Server: f71bbf328ed6
X-Real-Ip: 172.20.0.1

By sending the following request, Traefik will forward the request to whoami_service and will also forward the desired path /api to the service. The response from the whoami_service is the same response as above but in JSON.



$ curl -H "Host: whoami.karvounis.tutorial" http://localhost/api

{"hostname":"ed1b87c345ad","ip":["127.0.0.1","172.19.0.2"],"headers":{"Accept":["*/*"],"Accept-Encoding":["gzip"],"User-Agent":["curl/7.68.0"],"X-Forwarded-For":["172.20.0.1"],"X-Forwarded-Host":["whoami.karvounis.tutorial"],"X-Forwarded-Port":["80"],"X-Forwarded-Proto":["http"],"X-Forwarded-Server":["f71bbf328ed6"],"X-Real-Ip":["172.20.0.1"]},"url":"/api","host":"whoami.karvounis.tutorial","method":"GET"}

Try by yourself the following request



curl -H "Host: whoami.karvounis.tutorial" "http://localhost/data?size=1&unit=KB"

Alternatively, add the line 127.0.0.1 whoami.karvounis.tutorial to your /etc/hosts file and visit the http://whoami.karvounis.tutorial/ URL from your browser.

Scale whoami service to 3 containers



docker-compose up --scale whoami=3 -d

The above command will create 3 containers for the whoami Docker service instead of just 1. Send the following request multiple times and see Traefik load balance the requests between the 3 containers (Hostname keeps changing on every request).



curl -H "Host: whoami.karvounis.tutorial" http://localhost/

Isn't that wonderful and automagical?

Final notes

Congratulations! We successfully configured Traefik to run in Docker, listen for requests on port 80 and forward requests to the whoami service! However, this is just a pretty basic Traefik configuration and the beginning of your Traefik journey. We are going to cover some more advanced topics in the next tutorial! Until next time!

You can find me on LinkedIn and Github.