DEV Community: Ajit Vedpathak

Kubernetes Cluster Over-Provisioning: Proactive App Scaling

Ajit Vedpathak — Mon, 02 Nov 2020 19:05:46 +0000

Kubernetes Cluster Over-Provisioning: Proactive App Scaling

Scalability is a desirable attribute of a system or process. Poor scalability can result in poor system performance. Kubernetes is very much flexible, dynamic but when it comes to scaling, we encountered few challenges in application pod scaling which ended up having an adverse impact on application performance. In this blog, we will walk through the steps to overprovision a k8s cluster for scaling and failover.

Need of a cluster overprovisioning

Let's say we have a Kubernetes cluster running with X number of worker-nodes. All nodes in the cluster are running at their full capacity, meaning there is no space left on any of the nodes for incoming pods. Everything till now is working fine, applications running on the Kubernetes cluster are able to respond in time(We are scaling applications on the basis of Memory/CPU usage) but suddenly load/traffic on the application increases. As the load on the application increases, Memory/CPU consumption of an application also increases and Kubernetes starts scaling the application horizontally by adding new pods in the k8s cluster when the metric considered for scaling crosses a threshold value. But, all the newly created pods will go in a pending state (as all our worker nodes are running at full capacity).

The Horizontal Pod Autoscaler creates additional pod when the need for them arises. But what happens when all the nodes in the cluster are at full capacity and can’t run any more pods?
The Cluster Autoscaler takes care of automatically provisioning additional nodes when it notices a pod that can’t be scheduled to existing nodes because of the lack of resources on those nodes. A new node will be provisioned if, after a new pod is created and the Scheduler can’t schedule it to any of the existing nodes. The Cluster Autoscaler looks for such pods and signals the cloud provider (eg. AWS) to spin up an additional node and the problem lies in here. To provision, a new node in a cluster cloud provider may take some time (a minute or more) before the created nodes appear in Kubernetes cluster. It almost entirely depends on the cloud provider (We are using AWS) and the speed of node provisioning. It may take some time till the new pods can be scheduled.

Any pod that is in a pending state will have to wait till the time node gets added to the cluster. This is not an ideal situation for the applications which need quick scaling as the amount of request/load increases, as any delays in the scaling may hinder the application performance.

To overcome the problem, we thought of implementing the below solutions

Fix the number of extra nodes in the cluster
Make use of pod-priority-preemption

Fix the number of extra nodes in the cluster

One of the ideas was to add a fixed number of extra nodes in the cluster that will always be available, waiting, and ready to accept new pods. In this way, our apps would always be able to scale up without having to wait for AWS to create a new EC2 instance to join the Kubernetes cluster. The apps could always scale up without getting in a pending state. That was exactly what we wanted.
But, this was a temporary solution that was not cost-effective and efficient. Also overprovisioning was not dynamic, meaning the number of fixed extra nodes would not change as the cluster grows or shrinks and we started facing the same issue again.

Make use of pod-priority-preemption

The second solution was difficult to implement as it was difficult to decide the pod priority of different applications as we were having multiple applications that are sensitive to such scaling delays. Any delay in the scaling of an application may hinder the application performance and will have an adverse impact on application performance.

As we were in search of excellence, we came across a tool that would make use of cluster autoscaler to overscale the cluster.

Introduction to Horizontal Cluster Proportional Autoscaler

Horizontal cluster proportional autoscaler image watches over the number of schedulable nodes and cores of the cluster and resizes the number of replicas for the required resource.

If we want to configure dynamic overprovisioning of a cluster (e.g. 20% of resources in the cluster) then we need to use Horizontal Cluster Proportional Autoscaler.

It gives a simple control loop that watches the cluster size and scales the target controller. To overscale the cluster, we will create pods that will occupy space in the cluster and will do nothing. We can name these pods as placeholder pods (We will be using paused pod replica-controller as the target controller). Having these pods occupying extra space in the cluster will allow us to have some extra resources already created and ready to be used at any time.

But how we are going to use this extra space? This is where Kubernetes will help us with pod-priority-preemption.The placeholder pods(paused pods) will have the lowest priority assigned to them than the actual workload. Since these pods have a lower priority than regular pods, Kubernetes will compare the priority of all pods and evict those with lower priority as soon as resources become scarce. The placeholder pods then go into a pending state on which cluster-autoscaler reacts by adding new nodes in the cluster.

Cluster Proportional Autoscaler increases the number of replicas of placeholder pods when cluster grows and decreases the number of replicas if cluster shrinks.

Scaling with over-provisioning: what happens under the hood

Load hits the cluster
Kubernetes starts scaling application pods horizontally by adding new pods.
Kube-scheduler tries to place newly created application pods but finds insufficient resources.
placeholder-pods (pause pods in our case) gets evicted as it has low priority and application pods get placed.
Application pods gets placed and scaling happens immediately without any delay.
placeholder-pods go in pending state and cannot be scheduled due to insufficient resources.
Cluster autoscaler watches the pending pods and will scale the cluster by adding new nodes in the cluster
Kube-scheduler waits, for instance, to be provisioned, boot, join the cluster, and become ready.
Kube-scheduler notices there is a new node in the cluster where pods can be placed and will schedule placeholder-pods on such nodes.

Implementation

Note:- Change pod priority cut off in Cluster Autoscaler to -10 so pause pods are considered during scale down and scale-up. Set flag **expendable-pods-priority-cut-off to -10**.

This helm chart helps you to deploy the placeholder-pods (which will occupy extra space in the cluster) and cluster proportional autoscaler deployment (for dynamic overprovisioning).

List of Kubernetes resources that gets created with helm chart.

Placeholder-pod (paused-pod) deployment
Cluster proportional autoscaler deployment
Serviceaccount
Clusterrole
Clusterrolebinding
Configmap
Priority class for the paused pods with a priority value of -1

The helm chart uses the following configuration to overscale the cluster.
1 replica of placeholder-pod per node (with 2 core CPU and 2 Gi of memory request). For example, if the Kubernetes cluster is running with 15 nodes then 15 replicas of placeholder-pod will be there in the cluster.

This link will help you know different Control patterns and ConfigMap formats

Conclusion

Cluster-Overprovisioning is needed when we have applications running in the cluster which are sensitive to scaling delays and don't want to wait for new nodes to be created and join the cluster, for scaling.

With Cluster-Overprovisioning implemented, it takes only a few seconds to scale the application horizontally, maintaining the performance of the application as per increased usage or traffic or load.

References

User access control in k8s - X.509 Client Certificate approach

Ajit Vedpathak — Sun, 19 Jul 2020 15:07:34 +0000

One area of a Kubernetes that is critical to production deployments is security. Ensuring the control of who has access to your Information System and what users have access to is the objective of an Identity and Access management system. It is one of the fundamental processes in Security Management and it should be thoroughly taken care of.

This blog will take a practical look at authentication and authorization of users external to Kubernetes with Role-Based Access Control (RBAC).

Pre-requisites:

High-level understanding of Kubernetes concepts. Sample reference article.
You have a Kubernetes cluster running.
You have the kubectl command-line (kubectl CLI) installed.
OpenSSL installed

Why we need RBAC?

RBAC policies are crucial for the management of a k8s cluster, as with RBAC we can specify which types of actions are allowed depending on the user and the role in the organization. Examples include:

Secure your cluster by granting privileged operations (Example- accessing secrets) only to admin users.
User authentication in your cluster.
Limit resource creation (Example- pods, deployments) to specific namespaces.
Isolate resources access within your organization (for example, between departments).

To manage RBAC in Kubernetes, we need the following elements:

Roles and ClusterRoles: Both consist of rules. The difference between a Role and a ClusterRole in the scope: in a Role, the rules are applicable to a single namespace, whereas a ClusterRole is cluster-wide.Both Roles and ClusterRoles are mapped as API Resources inside our cluster.
RoleBindings and ClusterRoleBindings: Bind subjects to roles (i.e. the operations a given user can perform). As for Roles and ClusterRoles, the difference is the scope: a RoleBinding effective inside a namespace, whereas a ClusterRoleBinding is cluster-wide.
Subjects: The set of users and processes that want to access the Kubernetes API.
Resources: The set of Kubernetes API Objects available in the cluster. ( Pods, Deployments, Services, etc. )
Verbs: The set of operations that can be executed to the resources above (examples: get, watch, create, delete, etc.)
Users: These are meant for humans or processes living outside the cluster. In Kubernetes, there is no API call to add/create the users(for example you can not say kubectl create user )
service accounts: These are namespaced and associated with the pods via secret. These are managed by Kubernetes. By default service account have no access permissions.

With these elements in mind, We want to connect subjects, API resources, and operations. In other words, we want to specify, for a given user, which operations can be executed over a set of resources.

How to use RBAC?

There are multiple Authentication strategies followed by Kubernetes. We will be using X.509 Client Certificate for authentication purpose.

When we use X.509 Client Certificate Authentication strategy Kubernetes it will first create the certificate authority and it is a cluster-wide authority that is issuing certificates.

First step k8s will create a certificate authority and it will generate a certificate and that is going be a crucial component of authentication.
Once you have a CA Cert then you can basically create a certificate external to k8s and send it to k8s saying here is a new user that is going to be associated with the new certificate please sign it and approve it.
Then Kubernetes will take that and will add it to its own internal store and then it will authorize and approve.
Then Kubernetes generates a signed certificate and gives it back, which is going to be used by the user/admin who is going to access the k8s cluster.
The user will use a combination of the CA cert as certificate along with the pre-approved certificate associated with the user.

We will create an external user called Bob and the group engineering that he belongs to with the roles which grant him namespace level write access and a cluster-level read access.

Authentication

Steps: (Before we start please make sure all of the above pre-requisites are satisfied)

Create a namespace `engineering`

$ kubectl create namespace engineering
namespace/engineering created

Run a below command to get namespace

$ kubectl get ns

Create Private Key Using OpenSSL

After creating a namespace we will create a private key using OpenSSL. Before that create a new directory called cred and cd into that directory. To do that run below command

$ mkdir cred && cd cred
$ openssl genrsa -out Bob.key 2048

Extract certificate signing request from the private key created

Now we have created a private key with name Bob.key we will extract the CSR (certificate signing request) from this private key for that run below command.

$ openssl req -new -key Bob.key -out Bob.csr

You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields, there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [AU]:.
State or Province Name (full name) [Some-State]:.
Locality Name (eg, city) []:.
Organization Name (eg, company) [Internet Widgits Pty Ltd]:engineering
Organizational Unit Name (eg, section) []:.
Common Name (e.g. server FQDN or YOUR name) []:Bob
Email Address []:.

Please enter the following 'extra' attributes
to be sent with your certificate request
A challenge password []:.
An optional company name []:.

while creating a CSR you will be asked to enter certificate subject values. (Enter a value for Common Name and Organization Name. For all other fields enter the value as ' . ' and it will be treated as blank value)

The common name we are using here is Bob and the organization name engineering and this very critical to Kubernetes because when we are actually extracting the CSR file from the key we are mentioning that the name of the user is Bob and the group that he belongs to is engineering.

This is the only common identifier between the outside user and Kubernetes because when Kubernetes wants to find out more about Bob it actually looks at the common name in the certificate and then figures out the exact identity.

Convert the CSR file to base64

Now we have two files one is the private key that we have generated and the CSR file that we got out of it
This CSR file is basically the certificate request. We have to send this to Kubernetes and ask Kubernetes to register it and then as a cluster administrator will approve that request.

Before that, we have to convert the CSR file to base64 as base64 string is how Kubernetes will understand. For that run below command.

$ cat Bob.csr | base64 | tr -d '\n'

Copy the output of the above command.

Create CertificateSigningRequest

Now create a certificateSigningRequest.yaml and paste the copied output string of above command next to request: under spec:

CertificateSigningRequest
kind: CertificateSigningRequest
apiVersion: certificates.k8s.io/v1beta1
metadata:
  name: Bob
spec:
  groups:
  - system:authenticated
  request: <paste base64 string from above step>
  usages:
  - digital signature
  - key encipherment
  - server auh

Send certificate signing request to Kubernetes

$ kubectl create -f certificateSigningRequest.yaml

Get the status of a CSR created

$ kubectl get csr
NAME      AGE   REQUESTOR            CONDITION
Bob   7s    docker-for-desktop   Pending

As we can see the condition is pending because the Kubernetes administrator has to approve the signing request. To do that run the below command

Approve CertificateSigningRequest

#kubectl certificate approve <name of the certificateSigningRequest >

$ kubectl certificate approve Bob
certificatesigningrequest.certificates.k8s.io/Bob approved

Get the status of a CSR

$ kubectl get csr
NAME      AGE     REQUESTOR            CONDITION
Bob   3m21s   docker-for-desktop   Approved,Issued

Now we have sent the CSR and Kubernetes admin approved it. Kubernetes will turn the CSR request into a base64 encoded token and store it internally. Now we need to retrieve that token, decode the token and create a CRT file from that token which will be the final certificate which we are going to use when Bob gets authenticated.

Create a CRT file from a token

$ kubectl get csr Bob -o jsonpath='{.status.certificate}' | base64 --decode > Bob.crt

Now that we have two files a) Bob.key b) Bob.crt combination of both these files will help Bob to play with the Kubernetes cluster.

Configure The details in a config file (kubeconfig file)

We have to set the credentials in the kubeconfig file to get the Kube cluster access for that run a below command

$ kubectl config set-credentials Bob --client-certificate=Bob.crt --client-key=Bob.key
User "Bob" set.

Test if Bob can access the resources

So now to find out our Bob is ready to manage and access the resources in namespace engineering

$ kubectl auth can-i list pods -n engineering

if you run the above command you will get the output as yes because you are running a command as cluster-admin so to check is the Bob is able to list the pods inside lab4 namespace run below command

$ kubectl auth can-i list pods -n engineering --as Bob

The output of the above command will be no. This is because the Bob has been just authenticated by Kubernetes but it is not authorized to perform any action.

Authorization

Once an API request is authenticated, the next step is to determine whether the operation is allowed or not. This is done in the second stage of the access control pipeline.

Create a pod in namespace engineering

$ kubectl run nginx --image=nginx -n engineering

Test if Bob can access pod inside engineering namespace

To See if Bob can access the pod inside engineering namespace run a below command.

$ kubectl get pods -n engineering --as Bob
Error from server (Forbidden): pods is forbidden: User "Bob" cannot list resource "pods" in API group "" in the namespace "engineering"

As we can see the Bob is not able to access the pods as he is not authorized to even list the pods inside engineering namespace for that we have to create a role which will give them access to resources mentioned in the rule and bind that role to a user i.e. Bob.

Create a role for engineering namespace

So how does the role lookalike? Create a role.yaml file and paste the below content in the file and run a command to create a role

kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  namespace: engineering
  name: reader
rules:
- apiGroups: [""]
  resources: ["pods","services","nodes"]
  verbs: ["get","watch","list"]

$ kubectl create -f role.yaml

Get role

$ kubectl get roles -n engineering

We just have created a role that is not associated with any user and it grants access to resources pods, services, and nodes to watch and list operations inside engineering namespace.

Create role-binding

Now we have to associate this created role to a user we have created. To do so create a rolebinding.yaml paste the below text and run a command to create role-binding.
RoleBinding associates an existing role with Bob.

kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: read-access
  namespace: engineering # namespace where we have created the role and user
subjects:
- kind: User
  name: Bob # user name we have created
  apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: Role
  name: reader #name of the role we have created
  apiGroup: rbac.authorization.k8s.io

$ kubectl create -f rolebinding.yaml

Get rolebindings

$ kubectl get rolebindings -n engineering

Test if Bob can list pod inside engineering namespace

Now try to run the below command and see if Bob have access to get the pods inside engineering namespaces

$ kubectl get pods -n engineering --as Bob
NAME   READY   STATUS    RESTARTS   AGE
engineering   1/1     Running   0          6s

As we can see now Bob has access to get the pods from the namespace. Now If a Bob wants to know the nodes in the cluster what he does is he will run the command as $ kubectl get nodes --as Bob but he Bob will not be able to access the nodes as nodes are the cluster level resource and the object role is limited to namespace only. Current permission which is very restrictive prohibits Bob from accessing anything at cluster-level. The Bob still pretty much aligned to namespace engineering and he can list any resource that is specific to the namespace.

$ kubectl get nodes --as Bob
Error from server (Forbidden): nodes is forbidden: User "Bob" cannot list resource "nodes" in API group "" at the cluster scope

Create cluster-level premissions

Now we want to give Bob slightly better access permissions i.e. cluster-level permissions for that we have to create a new role called ClusterRole which will give access to cluster-level resources. To do so create a file

clusterRole.yaml and paste the below content in the file and run the command to create a cluster role.

Get ClusterRole

kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  # namespace is removed as cluster level role is not namespaced
  name: cluster-node-reader
rules:
- apiGroups: [""]
  resources: ["nodes"]
  verbs: ["get","watch","list"]

$ kubectl create -f clusterRole.yaml

Create role-binding

Now to bind this role to a user create clusterbinding.yaml and paste the below content in the file and run the command to create a cluster binding.

kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: cluster-binding 
subjects:
- kind: User
  name: Bob
  apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: ClusterRole
  name: cluster-node-reader
  apiGroup: rbac.authorization.k8s.io

$ kubectl create -f clusterbinding.yaml

Test if Bob can cluster-level resource

To see is Bob can now access the cluster level resource run a below command

$ kubectl get nodes --as Bob
NAME STATUS ROLES AGE VERSION
docker-desktop Ready master 90d v1.14.6

DEV Community: Ajit Vedpathak

Kubernetes Cluster Over-Provisioning: Proactive App Scaling

Kubernetes Cluster Over-Provisioning: Proactive App Scaling

Need of a cluster overprovisioning

To overcome the problem, we thought of implementing the below solutions

Fix the number of extra nodes in the cluster

Make use of pod-priority-preemption

Introduction to Horizontal Cluster Proportional Autoscaler

Scaling with over-provisioning: what happens under the hood

Implementation

Conclusion

References

User access control in k8s - X.509 Client Certificate approach

Pre-requisites:

Why we need RBAC?

To manage RBAC in Kubernetes, we need the following elements:

How to use RBAC?

Authentication

Steps: (Before we start please make sure all of the above pre-requisites are satisfied)

Create a namespace engineering

Run a below command to get namespace

Create Private Key Using OpenSSL

Extract certificate signing request from the private key created

Convert the CSR file to base64

Create CertificateSigningRequest

Send certificate signing request to Kubernetes

Get the status of a CSR created

Approve CertificateSigningRequest

Get the status of a CSR

Create a CRT file from a token

Configure The details in a config file (kubeconfig file)

Test if Bob can access the resources

Authorization

Create a pod in namespace engineering

Test if Bob can access pod inside engineering namespace

Create a role for engineering namespace

Get role

Create role-binding

Get rolebindings

Test if Bob can list pod inside engineering namespace

Create cluster-level premissions

Get ClusterRole

Create role-binding

Test if Bob can cluster-level resource

Create a namespace `engineering`