DEV Community: Sander Rodenhuis

Delegating control with an advanced multi-tenant setup in Kubernetes

Sander Rodenhuis — Tue, 01 Mar 2022 12:16:35 +0000

Introduction

Kubernetes is still a relatively new technology but is being adopted at high speed. But be aware that Kubernetes is only a general-purpose cluster operating system kernel and requires additional applications and configuration to safely run and manage your containerized business applications. Also, Kubernetes is not the holy grail for everything. If you're just running a couple of containers, then Kubernetes might not be the ideal go-to technology because of the risk of underutilization. This makes sharing a K8s cluster with multiple teams an interesting option.

But allowing multiple tenants on a shared cluster comes with challenges. For instance, how do you make sure the tenants can work independently of each other without interfering with one another? Now, this would not be that hard to implement. But it becomes a little more complicated if you would like these tenants to also be able to share generic platform applications and allow one tenant more control compared to another. Delegating control within the whole stack requires a lot of integration and custom engineering.

In this article, I’ll explain how delegation of control can be easily implemented using the Otomi open source project.

What is delegation of control?

From a management perspective, the best definition of delegation is when an administrator gives a user or group of users the responsibility and authority to complete specific tasks. In IT, delegated administration or delegation of control is about the decentralization of role-based access control. In Kubernetes, this model scales poorly because access control can only be done on the API level using Kubernetes RBAC. Allowing one tenant more access then another tenant can result in a very complex configuration. And RBAC on the application level is implemented per application.

As a result, operation teams become burdened with lots of tasks. These - mainly not automated - tasks can incur high latency times or result in poor security practices.

Suppose you would like to:

Enforce specific resource quota to team A and offer team B the ability to adjust the configured resource quota
Allow team B to be able to configure network policies while the network policies of team A can only be implemented by the admin
Allow access to a team (tenant) on the platform based on an LDAP group membership and allow team B to change the group mapping, while the group mapping of team A can only be changed by the admin
Allow access to shared Kubernetes apps like HashiCorp Vault and/or Harbor per team/tenant

How much time would you think it takes to support this?

What is Otomi?

Otomi is a Kubernetes applications configuration & automation platform for Kubernetes and can be installed in one run on a Kubernetes cluster in Azure, AWS, or GCP. Otomi consists of a suite of pre-configured and integrated Kubernetes apps, combined with self-service and automation. The teams feature in Otomi offers an advanced multi-tenancy setup, where teams (tenants) get access to a web UI with self-service tasks and shared applications. Administrators can create teams and delegate control (configure what a team is allowed to do and access).

How Otomi supports delegation

Otomi can run in multi-tenant mode, allowing the creation of tenants (called Teams in Otomi). The foundation of a team is a Kubernetes namespace, combined with a default RBAC policy. Access to a team is controlled based on group membership. You can use Keycloak as an IdP or configure Keycloak to act as an identity broker using an external IdP (like Azure AD). A user who is a member of the group mapped to the team will automatically get access to a project in Harbor, a shared space in Vault, the logs of all pods running in the team namespace, and much more.

Teams have access to self-service tasks to add pre-deployed K8s and Knative services to the service mesh (based on Istio), configure public exposure, create K8s Jobs and Cronjobs, and configure ingress/egress network policies. An admin can decide which self-service tasks are available for a team.

An admin can delegate control based on the following self-service flags:

Services

Configure ingress: allow the use of the self-service feature to publicly expose services
NetworkPolicy: allow the use of the self-service feature to configure ingress/egress network policies

Team

Alerts: grant the team permission to configure Alerts for the team
OIDC: grant the team permission to configure OIDC for the team
Resource Quota: grant the team permission to configure Resource Quota for the team
DownloadKubeConfig: grant the team permission to download the KubeConfig file
Network policy: grant the team permission to enable/disable network policies for the team

Delegation in action

Sign in to the web UI (Otomi console) and create a new team. Provide a name for the team, specify Resource quota, enable network policies and select the self-service flags to only allow service/ingress.

Now sign out as the administrator and sign in to the console with a user who is only part of the team-demo group. Go to the Settings of the team (in the left pane) and notice that you can not change any settings. The only allowed self-service task is to create services and configure public exposure for a service.

Now create a new Service. Go to the Service section in the left pane and then click new Service. Notice that you can create a new service, and you can configure public exposure, but you can not modify the default network policies (by default all access other than public access is not allowed and the service is not allowed to access any external resources).

Wrapping up

Providing self-service to execute repeatable tasks on Kubernetes and also controlling who can execute these tasks using a simple web UI is something that is not offered by any product in the market today.

If you would like to have more control over what users of your Kubernetes clusters can and can not do, while at the same time offering them a standardized way of working, automation, self-service, and role-based access to shared applications in an advanced multi-tenant setup, then go and try out Otomi. Go to the GitHub project here to get started or visit otomi.io for more information.

Make developers self serving when using Kubernetes

Sander Rodenhuis — Thu, 11 Nov 2021 13:19:24 +0000

There's a lot of talk lately about developer self service for Kubernetes. It looks like every vendor in the Kubernetes space now claims they offer some kind of developer self service.

Some might disagree, but the overall opinion is that Kubernetes is not really developer friendly. Kubernetes has over 50 object types, complex RBAC and to get anything deployed, you'll need to write YAML manifest. But you also need to learn how to handle horizontal scaling, networking, secrets, ingress, and a hole lot more. The question is, why would a developer care about all this stuff. Developers don’t care about all the Kubernetes internals, they just like to get an HTTP(S) endpoint of their running application.

Operations and SRE on the other hand have a hard time guarantee application security and availability at all stages in the development lifecycle when using Kubernetes. Kubernetes is only the orchestrator and building a custom platform on top of Kubernetes (the DIY way) can be a daunting endeavor. Think of all the add-ons and applications required for observability, policy enforcement, network policies, tracing, vulnerability management, SSO, certificates, secrets management, and GitOps. Building some kind of developer self service features and automatic team onboarding would require even more time and a lot of companies would not even have the resources and money to do so.

So what if there was an easy and open source solution for this? Would you then try it out?

Introducing Otomi

Otomi is an open source project (see the GitHub project here) and is a single deployable package to offer a complete platform experience on top of (any) Kubernetes cluster. After installing Otomi on Kubernetes, you can create an account, sign in, create a team and add members to the team. Team members will get access to a project in Harbor (to push images to), all container logs of the team, a shared space in Vault, and self service features to publicly expose services or even deploy images using user friendly forms. By default Otomi is configured in permissive security mode, which means security policies are enabled, but non-blocking. This enables developers to learn Kubernetes security as they go.

Get started

To get started with Otomi, just spin up a Kubernetes cluster (or use the Quickstart) and deploy the chart.

Frist create a values.yaml file with the following values:

cluster:
  owner: myself
  k8sVersion: '1.20'
  name: my-cluster
  provider: # use azure|aws|google

Then deploy the chart:

helm repo add otomi https://otomi.io/otomi-core
helm repo update
helm install -f values.yaml otomi otomi/otomi

The installer job will now install Otomi on your cluster. You can follow the progress of the installer by looking at the logs of the installer job:

kubectl logs jobs/otomi -n default -f

When the installer has finished (which can take around 20 to 30 minutes), copy the URL and the generated password from the bottom of the logs.

Now the first thing you need to do is create a new user in Keycloak and add the user to the otomi-admin group. Go to https://keycloak.your-ip.nip.io and sign in with the user admin and the generated password provided in the logs. Check here for complete instructions on how to create users in Keycloak.

Now you can sign in to the Otomi console. Go to https://otomi.your-ip.nip.io and sign in with your newly created user.
As you would have noticed, the browser says the connection to this site is not secure. Because we did not use DNS with LetsEncrypt and also did not provide our own CA, Otomi has automatically generated a CA for you. But no worries, you can add the generated CA to your keychain. In the left pane of the console, click on Download CA and then add the CA to your KeyChain:

sudo security add-trusted-cert -d -r trustRoot -k /Library/Keychains/System.keychain ~/Downloads/ca.crt

To start using Otomi, you’ll first need to activate Drone.
To enable Drone, open the Drone app (using the shortcut in Otomi Console), and sign in with OpenID Connect using the newly created user. In Drone you’ll see the otomi/values repository created by Otomi. Now click on Activate, then click Activate Repository, and then click on Save. Now you’re ready to use Otomi. On otomi.io you can find the full post-installation steps.

Recap

Otomi offers a full platform experience of top of Kubernetes. It integrates and pre-configures a complete suite of open-source projects like Harbor, Knative, Istio, Prometheus, Keycloak, Gatekeeper, and many more. By installing Otomi on Kubernetes, you can use all of these projects out-of-the-box with sane defaults. So instead of installing, configuring, and integrating all of these projects yourself, the only thing you’ll need to do is install Otomi.

Otomi is ideal to use as a (Kubernetes-based) Developer Platform. By default Otomi runs in permissive mode, meaning all actions contrary to the security policies are only logged. Teams can directly access all the security logs and improve the security posture. This offers a learn-as-you-go experience.

Running Otomi without dependencies, now makes it very easy to get started with Otomi and explore all of its features.

And if you run into any issues, please report them here.