DEV Community: Andrew Minkin

Testing Kubernetes with Kuttl

Andrew Minkin — Tue, 20 Dec 2022 08:04:48 +0000

Automated testing is the only way to be sure that your code works. Enabling automated testing can be hard and we say a lot of tools to write automated tests in the industry since the beginning. Some veterans in the industry may remember Selenium, Cucumber frameworks that help automate testing in the browser. However, testing in Kubernetes can be hard.

In Percona we deal with Kubernetes and have different operators to automate the management of databases. It requires testing. A lot of testing. We have different frameworks to help us with it

Codecept.js to write UI tests for PMM. Also, we use a playwright for some cases.
We have tools to help us with API testing as well as automating some routines by running bash commands during the test step.

However, those frameworks are not applicable to test Kubernetes workloads as well as Kubernetes operators. I've been working in the PMM integrations team for six months and saw different approaches to automate testing for PMM/DBaaS. We have a Go test library with wrappers around kubectl and codecept.js for end-to-end tests for the User Interface.

What challenges do we have?

Well, to be sure that a database cluster creation works we need to automate the following steps

Installation of operators to Kubernetes cluster
Test integration with version service to respect compatibility matrix.
Create a database cluster and wait once it'll be available
Do some assertions against Kubernetes as well as UI.

The main pain point here is that we need to wait up to 10-15 minutes for each step and we can't have different test cases to cover as many cases as we can. Yet we can achieve some performance benefits by paralleling workloads, still, it requires learning Javascript and testing framework to work with it. We had some architectural changes recently and moved from our custom gRPC API to create and manage database clusters to an operator that runs on top of other operators and converts K8s' Custom Resource from generic format to operator specific. We had a couple of options for this new project and after research, we chose kuttl as a framework for integration/e2e testing.

What is KUTTL anyway and why should I care?

KUTTL is the KUbernetes Test TooL. It's written in Go and provides a declarative way to test Kubernetes operators using Kubernetes primitives. It's easy to start kuttling. Let's take a deeper look. I'll use dbaas-operator as an example. DBaaS-operator is an operator that has a simple and generic Custom Resource Definition available to create Percona Server MongoDB or Percona XtraDB Cluster instances in Kubernetes. It uses underlying operators as a dependency. We have the following structure

e2e-tests
├── kind.yml
├── kuttl-eks.yml
├── kuttl.yml
└── tests
    └── pxc
        ├── 00-assert.yaml
        ├── 00-deploy-operators.yaml
        ├── 01-assert.yaml
        ├── 01-deploy-pxc.yaml
        ├── 02-assert.yaml
        ├── 02-upgrade-pxc.yaml
        ├── 03-assert.yaml
        ├── 03-restart-pxc.yaml
        ├── 04-delete-cluster.yaml
        ├── 05-assert.yaml
        ├── 05-create-cluster.yaml
        ├── 06-assert.yaml
        ├── 06-scale-up-pxc.yaml
        ├── 07-assert.yaml
        ├── 07-scale-down-pxc.yaml
        └── 08-delete-cluster.yaml

2 directories, 19 files

Let's discuss these YAML files more

kind.yml contains settings to run Kind
kuttl.yml has all the required settings for the Kuttl framework and kuttl-eks.yml has some EKS-specific configurations
tests folder has test steps and assertions

Kind and KUTTL settings

Let's discuss Kind and kuttl settings and I'll start with KUTTL first

apiVersion: kuttl.dev/v1beta1
kind: TestSuite
kindConfig: e2e-tests/kind.yml  # Path to Kind config that will be used to create Kind clusters
crdDir: config/crd/bases        # Path to a directory that contains CRD files. Kuttl will apply them before running tests
artifactsDir: /tmp/             # Path to a directory to store artifacts such as logs and other information
testDirs:
- e2e-tests/tests               # Path to directories that have test steps

Kind config is quite easy

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
- role: worker
- role: worker
- role: worker
containerdConfigPatches:
- |-
  [plugins."io.containerd.grpc.v1.cri".registry.mirrors."localhost:5000"]
    endpoint = ["http://kind-registry:5000"]

The aforementioned config will use a local registry and will create 3 k8s worker nodes controlled by the control plane

Writing tests

At the first glance, kuttling can be easy because it uses Kubernetes primitives as a test step and assertion but I had a couple of problems testing my operator. Let's take a look at a couple of examples. Since dbaas-operator depends on the PXC operator we need to prepare our environment for testing. Let's write the first test that installs the PXC operator and ensures that it was installed.

cat e2e-tests/tests/pxc/00-deploy-pxc-operator.yml

apiVersion: kuttl.dev/v1beta1
kind: TestStep
timeout: 10 # Timeout for the test step
commands:
  - command: kubectl apply -f https://raw.githubusercontent.com/percona/percona-xtradb-cluster-operator/v${PXC_OPERATOR_VERSION}/deploy/bundle.yaml -n "${NAMESPACE}"

KUTTL test steps easily extensible with commands. One can run even scripts as a prerequisite for a test case. PXC operator installs CRDs and creates a deployment and here's an example of assertion.

cat e2e-tests/tests/pxc/00-assert.yml

apiVersion: kuttl.dev/v1beta1
kind: TestAssert
timeout: 120  # Timeout waiting for the state
---
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: perconaxtradbclusters.pxc.percona.com
spec:
  group: pxc.percona.com
  names:
    kind: PerconaXtraDBCluster
    listKind: PerconaXtraDBClusterList
    plural: perconaxtradbclusters
    shortNames:
    - pxc
    - pxcs
    singular: perconaxtradbcluster
  scope: Namespaced
---
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: databaseclusters.dbaas.percona.com
spec:
  group: dbaas.percona.com
  names:
    kind: DatabaseCluster
    listKind: DatabaseClusterList
    plural: databaseclusters
    shortNames:
    - db
    singular: databasecluster
  scope: Namespaced
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: percona-xtradb-cluster-operator
status:
  availableReplicas: 1
  observedGeneration: 1
  readyReplicas: 1
  replicas: 1
  updatedReplicas: 1

Our first test is ready and one needs to run kubectl kuttl test --config ./e2e-tests/kuttl.yml to run kuttl.

More advanced tests

We need to run our operator first to be able to work with resources and test it. KUTTL recomends configure it via TestSuite by the following example

apiVersion: kuttl.dev/v1beta1
kind: TestSuite
...
commands:
  - command: ./bin/manager
...

However, since dbaas-operator depends on underlying operators it needs to work correctly even if they are not present in a Kubernetes cluster. It has the following logic in the controller

// SetupWithManager sets up the controller with the Manager.
func (r *DatabaseReconciler) SetupWithManager(mgr ctrl.Manager) error {
    fmt.Println(os.Getenv("WATCH_NAMESPACE"))
    unstructuredResource := &unstructured.Unstructured{}
    unstructuredResource.SetGroupVersionKind(schema.GroupVersionKind{
        Group:   "apiextensions.k8s.io",
        Kind:    "CustomResourceDefinition",
        Version: "v1",
    })
    controller := ctrl.NewControllerManagedBy(mgr).
        For(&dbaasv1.DatabaseCluster{})
    err := r.Get(context.Background(), types.NamespacedName{Name: pxcCRDName}, unstructuredResource)
    if err == nil {
        if err := r.addPXCToScheme(r.Scheme); err == nil {
            controller.Owns(&pxcv1.PerconaXtraDBCluster{})
        }
    }
    err = r.Get(context.Background(), types.NamespacedName{Name: psmdbCRDName}, unstructuredResource)
    if err == nil {
        if err := r.addPSMDBToScheme(r.Scheme); err == nil {
            controller.Owns(&psmdbv1.PerconaServerMongoDB{})
        }
    }
    return controller.Complete(r)
}

The controller.Owns sets up a controller to watch specified resources and once they were changed it'll run a reconciliation loop to sync changes. Also, it checks that operator is present in the cluster by checking that deployment and CRDs are available. It means that to make the operator work correctly in tests we need to choose from the following options

Restart operator once the upstream operator was installed by sending HUP signal
Run the operator only after the underlying operator is present in a cluster

Hence, I moved the command as the next step before creating a cluster. You can see it below

cat e2e-tests/tests/pxc/01-deploy-pxc.yml

apiVersion: kuttl.dev/v1beta1
kind: TestStep
timeout: 10
commands:
  - script: WATCH_NAMESPACE=$NAMESPACE ../../../bin/manager
    background: true
---
apiVersion: dbaas.percona.com/v1
kind: DatabaseCluster
metadata:
  name: test-cluster
spec:
  databaseType: pxc
  databaseImage: percona/percona-xtradb-cluster:8.0.23-14.1
  databaseConfig: |
    [mysqld]
    wsrep_provider_options="debug=1;gcache.size=1G"
  secretsName: pxc-sample-secrets
  clusterSize: 1
  loadBalancer:
    type: haproxy
    exposeType: ClusterIP
    size: 1
    image: percona/percona-xtradb-cluster-operator:1.11.0-haproxy
  dbInstance:
    cpu: "1"
    memory: 1G
    diskSize: 15G

Note: command supports only simple commands and does not fully support env variables. It supports only $NAMESPACE, $PATH and $HOME. However, script solves the problem of setting WATCH_NAMESPACE environment variable.

In nutshell, the test step above does two things:

Runs the operator
Creates a database cluster

The assertion checks that kubernetes cluster has the DatabaseCluster object with ready status as well as PXC cluster with the same status.


apiVersion: kuttl.dev/v1beta1
kind: TestAssert
timeout: 600
---
apiVersion: dbaas.percona.com/v1
kind: DatabaseCluster
metadata:
  name: test-cluster
spec:
  databaseType: pxc
  databaseImage: percona/percona-xtradb-cluster:8.0.23-14.1
  databaseConfig: |
    [mysqld]
    wsrep_provider_options="debug=1;gcache.size=1G"
  secretsName: pxc-sample-secrets
  clusterSize: 1
  loadBalancer:
    type: haproxy
    exposeType: ClusterIP
    size: 1
    image: percona/percona-xtradb-cluster-operator:1.11.0-haproxy
  dbInstance:
    cpu: "1"
    memory: 1G
    diskSize: 15G
---
apiVersion: pxc.percona.com/v1
kind: PerconaXtraDBCluster
metadata:
  name: test-cluster
spec:
  allowUnsafeConfigurations: true
  crVersion: 1.11.0
  haproxy:
    enabled: true
    image: percona/percona-xtradb-cluster-operator:1.11.0-haproxy
    serviceType: ClusterIP
    size: 1
  pxc:
    configuration: |
      [mysqld]
      wsrep_provider_options="debug=1;gcache.size=1G"
    expose: {}
    image: percona/percona-xtradb-cluster:8.0.23-14.1
    livenessProbes: {}
    readinessProbes: {}
    resources:
      requests:
        cpu: "1"
        memory: 1G
    serviceType: ClusterIP
    sidecarResources: {}
    size: 1
    volumeSpec:
      persistentVolumeClaim:
        resources:
          requests:
            storage: 15G
  secretsName: pxc-sample-secrets
  updateStrategy: SmartUpdate
  upgradeOptions:
    apply: 8.0-recommended
    schedule: 0 4 * * *
status:
  ready: 2
  size: 2
  state: ready

Caveats and notes

I had problems running tests in Kind. They were flaky because PXC operator can't expose metrics and had problems with the liveness probe. I haven't figured out how to fix it but as a workaround, I use minikube to run tests

    minikube start --nodes=4 --cpus=2 --memory=4g --apiserver-names host.docker.internal --kubernetes-version=v1.23.6
    minikube kubectl -- config view --flatten --minify > ~/.kube/test-minikube
    KUBECONFIG=~/.kube/test-minikube kubectl kuttl test --config ./e2e-tests/kuttl.yml

Further steps

There's always room for improvement and I have these steps in mind

Use docker images and OLM bundles as a way to run the operator for tests. This will be the best way to simulate a production-like environment.
Add more advanced tests for database clusters such as running queries, trying loading data as well as capacity testing. It's easily achievable with kuttl

Leadership Lessons from Video Games

Andrew Minkin — Tue, 02 Aug 2022 12:21:08 +0000

I love playing games. My favorites are the World of Warcraft and the God Of War series. At the beginning of my career, it was just a way to relax and keep a work-life balance. After a while, I noticed wisdom inside them.

I became a tech team leader in 2014. As it usually happens, I had no idea what to do in the new position. Everything was new to me. The worst thing about my new position was that I had no plan to educate myself. Thus, I started to read nonfiction books about management, and I watched a lot of management talks from other professionals in this field on YouTube.

When I read a lot of books about management, philosophy, and psychology, I realized that you can learn how to lead from every situation in your life. For example, you can learn it in a gym from your coach, from movies, and even from video games.

Focus on your goal

Goals give you a destination that you need to reach. They help you to track your progress, discover problems, and give you a clear vision. Goals can help you in every area of your life, your job, and your personal life alike.

Once you have set goals for your project every team member should keep the focus on them. Focusing can help you to activate all abilities in your brain, such as learning, awarenesses, problem-solving, and decision making. Your team will solve problems faster when each team member focuses on one thing for a certain period. Sometimes people can forget about project goals, and it’s a leader’s responsibility to remind them of the key tasks that they should be focused on.

Once I was a leader in an enterprise project with a slow-release cycle. One main issue was that team didn’t build it from scratch by themselves. The project was new to us, and we made many necessary changes before deployment. Unfortunately, the codebase had a lack of unit tests, and we weren’t sure that our changes wouldn’t break anything. We decided to take a business trip and work on-site in order to decrease the feedback time of our actions and to react faster if anything went wrong.

We had a plan for deployment and discovered almost all the risks. We planned our deployment for the upcoming weekend, and this was the ideal time to deploy it because we had no users online.

One of my team members was worried about deployment and shared his concerns with us, let’s call him Mike.

Mike told us that we could break everything, and we would be unable to roll back the changes. Also, we didn’t have the necessary information in the log files to fix it. This situation was not likely to happen, but I thought that we could fix it if it occurred.

I used Kratos’ quote from the God Of War 4 game

Do not concern yourself with what might be. Focus on what is, and always remain vigilant. Kratos (God of War PS4)

I had played this game before our trip and this quote was quite funny to me. I had no idea that I would use it at work. A colleague of mine said ‘okay’ and started to prepare the deployment.

Deployment went smoothly without any problems. After a few days of monitoring and fixing small bugs, we determined that deployment had finished successfully.
One evening, later on, Mike came to me and said, “Thank you. Your words were useful for me”. After a small chat, I understood that a simple phrase could make another person confident. Also, he felt supported by his team.

A leader should keep focused on a team’s goals and dispel a team’s fears, doubts, and uncertainty. A leader needs to be aware of two other areas when leading a team: decision-making and responsibility.

Decision-making and taking responsibility

There are a lot of great leaders in the world. Most of them changed our lives and our attitude on some topics. One of them is Nelson Mandela. He is a remarkable example of a great leader. He was a South African revolutionary who ended the apartheid regime in his own country. In addition, he was elected as President of South Africa in a fully representative democratic election. He took responsibility for the anti-apartheid campaign and it was one of his goals in his life.

— A man chooses; a slave obeys. Bioshock

Mandela could not choose to end apartheid, but he decided to end it. He was a leader who cared and disobeyed the regime as others did.

As a leader of a team, you will always make decisions, and when you make decisions you need to have as many options as you can to choose the right decision. For example, you have a project with a big backlog and tight deadlines, and you realize that you will never finish all the tasks in the backlog by the deadline. You have many options on how to deal with this situation such as talking with your customer to cut features, focusing on the most significant features, or doing something else. However, it’s possible if you carry out the customer’s orders your team will be overloaded, and this can lead to the emotional burnout of your team members.

A great example of decision-making and taking responsibility is Jocko Willink’s story he gave in his TED Talk about extreme ownership. He told a great story about taking responsibility for the consequences of the entire situation. In his story, the actions were during the war in Iraq. There was a fog of war, and his battalion opened fire. In the end, a friendly Iraqi soldier was shot, and one of his soldiers was wounded.

The commanding officer said they needed to shut down all operations, and Jocko needed to prepare a debrief. “Who is responsible for what had happened?” was the critical question in the debrief. Jocko realized that as a senior man on the battlefield, he was responsible for everything that happened. Also, he shares lessons learned after the debriefing. When he took responsibility by taking ownership of the problem, the commanding officer started to trust him even more, and so did the team, and they realized that he would never shirk responsibility.

— Even the good leaders make poor decisions. It is the best leaders who take responsibility for them. Kratos (God of War PS4)

Jocko is an example of a good leader. He gave me a clear understanding of taking ownership.

When I started to own all project problems and shared this point of view within a team, everything became more accessible. It means that we are all responsible for the outcome of a project. When a team owns a project, they think differently. For instance, when a team is responsible for everything in a project, they care more about quality, about incidents concerning production, and they strive to have a project without problems, while before that, they can look the other way and disregard some problems. The team acts more proactively, and I even had suggestions from the team to talk to customers to gain time to decrease technical debt.

— We all make choices, but in the end, our choices make us. Andrew Ryan (BioShock)

The project's success depends on the manager, and all choices the manager makes give a result that can lead to success and failure. Let’s talk about how we can deal with failures.

Retrospect

Taking responsibility for the project’s success as a leader means that you’ll fail sometimes. People deal with failures differently. Some of them take them as lessons, while others take them seriously.

There are a lot of stories about the failures of successful people you can find on the Internet. Michael Jordan missed 9000 shots in his career. 12 major publishers rejected the first book of Harry Potter. One of Stephen King’s most successful books, Carrie, was rejected by 30 publishers. As you can see, most of them didn’t give up and continued achieving their goal.

How to deal with a possible failure? One way to deal with it is to face your fear and accept that you’ll fail anyway. There is a chance that it won’t happen, but most of the possible failures become real. When you accept that you will fail, you can change your interpretation of it. You can act differently, and you can see that failure is not as bad as you imagined; and when it’s not bad as you imagined, you can be patient when you deal with consequences.

You need to accept your failure and learn from it. Failure is a necessity for learning and you need to correct your behavior, find all errors you made, and find a solution on how to not repeat them. That’s how you’ll become better.

Don’t be sorry, be better. Kratos (God of War PS4)

Dealing with black boxes

Atreus: We will fight? Why?
Kratos: Because you are afraid of it. (God of War PS4)

Let’s imagine the situation when you need to take a project from another team. The project runs in production and has a lot of users. It was implemented poorly and had no tests and no documentation at all. The last team disappeared. You have no one to ask questions, and you need to implement features for the system. Also, you need to gather a team for the project.

Once you’ve onboarded new team members to the project, newbies will be afraid to change anything because they will suffer from a lack of documentation and tests in the project. Even if they get a working copy of a project on their machines, they will be afraid to deploy their changes to production because they are not sure they wouldn’t break anything.

As a leader, you should help your team to solve problems. You need to facilitate discussions, test, and deliver value. You need to have working sessions, discover the risks of changes, and reduce fear in the team by taking responsibility. For instance, you made a change, but your teammate is afraid to deploy it. He feels very uncomfortable in this situation. As a manager, you can push him to deploy it by taking responsibility for his action. You can say that you’ll inform customers, and we can roll it back if anything goes wrong.

Conclusion

The leader needs to constantly improve his skills. Also, he needs to focus on his and the project’s goals, make decisions and take responsibility for them, learn from his failures, and open black boxes. As a leader, you can find leadership lessons almost everywhere in your life. You can find them in the gym, in movies, and even in video games. You can find a lot of wisdom in video games’ stories that apply to real life.

Secure microservices with Kong and Ory

Andrew Minkin — Wed, 13 Apr 2022 13:25:16 +0000

Microservice architecture is nowadays almost a standard for backend development. An API gateway is an excellent way to connect a group of microservices to a single API accessible to users. API gateways are available from cloud providers such as AWS/Azure/Google Cloud Platform and Cloudflare. Kong is a scalable API gateway built on open source and as such can be an excellent alternative if you don't want to have your system locked in to a particular vendor.

This tutorial shows an example using Kong API gateway,
Ory Kratos, and Ory Oathkeeper.
The illustration below shows you the final architecture we are going to build in this guid

The full source code for this tutorial is available on
github

What we will use

Kong gateway can be an excellent solution for an ingress load balancer and API gateway if you do not want vendor lock-in of any cloud API Gateways in your application. Kong uses OpenResty and Lua. OpenResty extends Nginx with Lua scripting to use Nginx's event model for non-blocking I/O with HTTP clients and remote backends like PostgreSQL, Memcached, and Redis. OpenResty is not an Nginx fork, and Kong is not an Openresty fork. Kong uses OpenResty to enable API gateway features.
Oathkeeper acts like an identity and access proxy for our microservices. It allows to proxy only authenticated requests to our microservices, and so we don't need to implement a middleware to check authentication. It can also transform requests, for example convert session auth into JWT for a back-end service.
Kratos is the authentication provider; it handles all first-party authentication flows: username/password, forgot password, MFA/2FA, and more. It also provides OIDC/social login capabilities for example "Login with GitHub".

Building simple microservices

Let's say we have two microservices: hello and world. They are pretty simple and serve only to test our API gateway, but you can switch them out for more complex components.

The "World" microservice exposes a /world API endpoint and returns a simple JSON message:

package main

import (
    "encoding/json"
    "log"
    "net/http"
)

type Response struct {
    Message string `json:"message"`
}

func helloJSON(w http.ResponseWriter, r *http.Request) {
    response := Response{Message: "World microservice"}
    w.Header().Set("Content-type", "application/json")
    w.WriteHeader(http.StatusOK)
    json.NewEncoder(w).Encode(response)
}
func main() {
    http.HandleFunc("/world", helloJSON)
    log.Fatal(http.ListenAndServe(":8090", nil))

}

The "Hello" microservice exposes a /hello API endpoint and returns a simple JSON message:

package main

import (
    "encoding/json"
    "log"
    "net/http"
)

type Response struct {
    Message string `json:"message"`
}

func helloJSON(w http.ResponseWriter, r *http.Request) {
    response := Response{Message: "Hello microservice"}
    w.Header().Set("Content-type", "application/json")
    w.WriteHeader(http.StatusOK)
    json.NewEncoder(w).Encode(response)
}
func main() {
    http.HandleFunc("/hello", helloJSON)
    log.Fatal(http.ListenAndServe(":8090", nil))

}

We now want to secure the access to these microservices and let only authenticated users access these endpoints.

Okay. Let's start hacking, shall we?

Ory Kratos setup

Follow the Quickstart guide to set up Ory Kratos. In this tutorial you only need a docker-compose file with the following configuration:

  postgres-kratos:
    image: postgres:9.6
    ports:
      - "5432:5432"
    environment:
      - POSTGRES_USER=kratos
      - POSTGRES_PASSWORD=secret
      - POSTGRES_DB=kratos
    networks:
      - intranet

  kratos-migrate:
    image: oryd/kratos:v0.8.0-alpha.3
    links:
      - postgres-kratos:postgres-kratos
    environment:
      - DSN=postgres://kratos:secret@postgres-kratos:5432/kratos?sslmode=disable&max_conns=20&max_idle_conns=4
    networks:
      - intranet
    volumes:
      - type: bind
        source: ./kratos
        target: /etc/config/kratos
    command: -c /etc/config/kratos/kratos.yml migrate sql -e --yes

  kratos:
    image: oryd/kratos:v0.8.0-alpha.3
    links:
      - postgres-kratos:postgres-kratos
    environment:
      - DSN=postgres://kratos:secret@postgres-kratos:5432/kratos?sslmode=disable&max_conns=20&max_idle_conns=4
    ports:
      - '4433:4433'
      - '4434:4434'
    volumes:
      - type: bind
        source: ./kratos
        target: /etc/config/kratos
    networks:
      - intranet
    command: serve -c /etc/config/kratos/kratos.yml --dev --watch-courier

  kratos-selfservice-ui-node:
    image: oryd/kratos-selfservice-ui-node:v0.8.0-alpha.3
    environment:
      - KRATOS_PUBLIC_URL=http://kratos:4433/
      - KRATOS_BROWSER_URL=http://127.0.0.1:4433/
    networks:
      - intranet
    ports:
      - "4455:3000"
    restart: on-failure


  mailslurper:
    image: oryd/mailslurper:latest-smtps
    ports:
      - '4436:4436'
      - '4437:4437'
    networks:
      - intranet

Some notes on the network architecture:

HTTP :4433 and :4434 are the public and admin API's of Ory Kratos.
HTTP :4436 for Mailslurper - a mock Email server. You can get an activation link by accessing http://127.0.0.1:4436.
HTTP :4455 for the UI interface that allows one to start sign-up/login/recovery flows.

After running docker-compose up you can open http://127.0.0.1:4455/welcome to test your configuration.

Configuring Ory Oathkeeper

Now we can start configuring our gateways for this example. Kong is the entry point for the network traffic. Ory Oathkeeper would be accessible from the internal network only in this case. Let's review our architecture diagram from
before:

Oathkeeper checks sessions and proxies traffic to our microservice while Kong provides ingress load balancing. We can even set up Round-Robin DNS to have a more robust configuration for our service. Here is how we configure the access rules for Ory Oathkeeper:

-
  id: "api:hello-protected"
  upstream:
    preserve_host: true
    url: "http://hello:8090"
  match:
    url: "http://oathkeeper:4455/hello"
    methods:
      - GET
  authenticators:
    -
      handler: cookie_session
  mutators:
    - handler: noop
  authorizer:
    handler: allow
  errors:
    - handler: redirect
      config:
        to: http://127.0.0.1:4455/login
-
  id: "api:world-protected"
  upstream:
    preserve_host: true
    url: "http://world:8090"
  match:
    url: "http://oathkeeper:4455/world"
    methods:
      - GET
  authenticators:
    -
      handler: cookie_session
  mutators:
    - handler: noop
  authorizer:
    handler: allow
  errors:
    - handler: redirect
      config:
        to: http://127.0.0.1:4455/login

The Ory Oathkeeper configuration:

log:
  level: debug
  format: json

serve:
  proxy:
    cors:
      enabled: true
      allowed_origins:
        - "*"
      allowed_methods:
        - POST
        - GET
        - PUT
        - PATCH
        - DELETE
      allowed_headers:
        - Authorization
        - Content-Type
      exposed_headers:
        - Content-Type
      allow_credentials: true
      debug: true

errors:
  fallback:
    - json

  handlers:
    redirect:
      enabled: true
      config:
        to: http://127.0.0.1:4455/login
        when:
          -
            error:
              - unauthorized
              - forbidden
            request:
              header:
                accept:
                  - text/html
    json:
      enabled: true
      config:
        verbose: true

access_rules:
  matching_strategy: glob
  repositories:
    - file:///etc/config/oathkeeper/access-rules.yml

authenticators:
  anonymous:
    enabled: true
    config:
      subject: guest

  cookie_session:
    enabled: true
    config:
      check_session_url: http://kratos:4433/sessions/whoami
      preserve_path: true
      extra_from: "@this"
      subject_from: "identity.id"
      only:
        - ory_kratos_session

  noop:
    enabled: true

authorizers:
  allow:
    enabled: true

mutators:
  noop:
    enabled: true

Ory Oathkeeper now looks up a valid session in the request cookies, and proxies only authenticated requests. It redirects to login UI if there's no ory_kratos_session cookie available.

Adding Kong

Now all that is needed is to configure Kong:

services:
  kong-migrations:
    image: "kong:latest"
    command: kong migrations bootstrap
    depends_on:
      - db
    environment:
      <<: *kong-env
    networks:
      - intranet
    restart: on-failure

  kong:
    platform: linux/arm64
    image: "kong:latest"
    environment:
      <<: *kong-env
      KONG_ADMIN_ACCESS_LOG: /dev/stdout
      KONG_ADMIN_ERROR_LOG: /dev/stderr
      KONG_PROXY_LISTEN: "${KONG_PROXY_LISTEN:-0.0.0.0:8000}"
      KONG_ADMIN_LISTEN: "${KONG_ADMIN_LISTEN:-0.0.0.0:8001}"
      KONG_PROXY_ACCESS_LOG: /dev/stdout
      KONG_PROXY_ERROR_LOG: /dev/stderr
      KONG_PREFIX: ${KONG_PREFIX:-/var/run/kong}
      KONG_DECLARATIVE_CONFIG: "/opt/kong/kong.yaml"
    networks:
      - intranet
    ports:
      # The following two environment variables default to an insecure value (0.0.0.0)
      # according to the CIS Security test.
      - "${KONG_INBOUND_PROXY_LISTEN:-0.0.0.0}:8000:8000/tcp"
      - "${KONG_INBOUND_SSL_PROXY_LISTEN:-0.0.0.0}:8443:8443/tcp"
      - "127.0.0.1:8001:8001/tcp"
      - "127.0.0.1:8444:8444/tcp"
    healthcheck:
      test: ["CMD", "kong", "health"]
      interval: 10s
      timeout: 10s
      retries: 10
    restart: on-failure:5
    read_only: true
    volumes:
      - kong_prefix_vol:${KONG_PREFIX:-/var/run/kong}
      - kong_tmp_vol:/tmp
      - ./config:/opt/kong
    security_opt:
      - no-new-privileges

  db:
    image: postgres:9.6
    environment:
      POSTGRES_DB: kong
      POSTGRES_USER: kong
      POSTGRES_PASSWORD: kong
    healthcheck:
      test: ["CMD", "pg_isready", "-U", "kong"]
      interval: 30s
      timeout: 30s
      retries: 3
    restart: on-failure
    networks:
      - intranet

  hello:

The docker-compose creates three containers

db container with PostgreSQL database to store the configuration of services/routes for our API gateway.
kong-migrate to run migrations against the database.
kong container that exposes 8000 port for proxying traffic and 8001 port with admin API.

As last step, we need to create a service for Kong and configure routes.

#!/bin/bash

# Creates an secure-api service
# and proxies network traffic to oathkeeper
curl -i -X POST \
  --url http://localhost:8001/services/ \
  --data 'name=secure-api' \
  --data 'url=http://oathkeeper:4455'

# Creates routes for secure-api service
curl -i -X POST \
  --url http://localhost:8001/services/secure-api/routes \
  --data 'paths[]=/'\

Testing

You can open http://127.0.0.1:8000/hello or zttp://127.0.0.1:8000/world in your browser and there are two possible scenarios:

You receive {"message": "Hello microservice"} (or "World microservice").
The browser redirects you to http://127.0.0.1:4455/login.

Further steps

Configure id_token mutator to have the identity accessible as JWT token for your microservices.
Configure the password policy to better suit your use case.
Add two-factor authentication.
Consider using authentication based on subrequest result instead of having an additional reverse proxy inside your network
Kong auth request can be an excellent plugin to use oathkeeper as a decision API for Kong