Managing Kubernetes Resources with Resource Quotas

Resource quota

When several users or teams share a cluster with a fixed number of nodes, there is a concern that one team could use more than its fair share of resources.

Resource quotas are a tool for administrators to address this concern.

A resource quota, defined by a ResourceQuota object, provides constraints that limit aggregate resource consumption per namespace. It can limit the quantity of objects that can be created in a namespace by type, as well as the total amount of compute resources that may be consumed by resources in that namespace.

Resource quotas work like this:

Different teams work in different namespaces. This can be enforced with RBAC.

The administrator creates one ResourceQuota for each namespace.

Users create resources (pods, services, etc.) in the namespace, and the quota system tracks usage to ensure it does not exceed hard resource limits defined in a ResourceQuota.

If creating or updating a resource violates a quota constraint, the request will fail with HTTP status code 403 FORBIDDEN with a message explaining the constraint that would have been violated.

If quota is enabled in a namespace for compute resources like cpu and memory, users must specify requests or limits for those values; otherwise, the quota system may reject pod creation. Hint: Use the LimitRanger admission controller to force defaults for pods that make no compute resource requirements.

Explanation of resource_quota.yaml file

This is a YAML file containing two Kubernetes objects.

apiVersion: v1
kind: Namespace
metadata:
  name: mynamespace

---
apiVersion: v1
kind: List
items:
  - apiVersion: v1
    kind: ResourceQuota
    metadata:
      name: user-compute-quota
      namespace: mynamespace
    spec:
      hard:
        requests.cpu: "1"
        requests.memory: 1Gi
        limits.cpu: "2"
        limits.memory: 2Gi
  - apiVersion: v1
    kind: ResourceQuota
    metadata:
      name: user-object-quota
      namespace: mynamespace
    spec:
      hard:
        configmaps: "10"
        persistentvolumeclaims: "4"
        replicationcontrollers: "20"
        secrets: "10"
        services: "10"
        services.loadbalancers: "2"

The first object is of kind "Namespace" and has a metadata field with the name "mynamespace". This is creating a namespace named "mynamespace" in Kubernetes.

The second object is of kind "List" and contains a list of two "ResourceQuota" objects. The first "ResourceQuota" object has a metadata field with the name "user-compute-quota" and namespace set to "mynamespace". The "spec" field defines resource usage limits for the namespace, such as CPU and memory requests and limits. Specifically, this quota allows for a maximum of 1 CPU and 1Gi of memory for requests and a maximum of 2 CPUs and 2Gi of memory for limits.

The second "ResourceQuota" object has a metadata field with the name "user-object-quota" and namespace set to "mynamespace". The "spec" field defines limits on the number of objects of different types that can be created in the namespace. Specifically, this quota allows for a maximum of 10 configmaps, 4 persistentvolumeclaims, 20 replicationcontrollers, 10 secrets, 10 services, and 2 load balancer services.

Steps to create ‘Resource Quota’ objects and observe their working

1. Create resource quota objects:

kubectl apply -f resource_quota.yaml

1. Verify created resource quota objects:

 kubectl get quota -n mynamespace 

NAME                 AGE   REQUEST                                                                                                                                   LIMIT
user-compute-quota   79s   requests.cpu: 0/1, requests.memory: 0/1Gi                                                                                                 limits.cpu: 0/2, limits.memory: 0/2Gi
user-object-quota    44s   configmaps: 1/10, persistentvolumeclaims: 0/4, replicationcontrollers: 0/20, secrets: 0/10, services: 0/10, services.loadbalancers: 0/2

1. Create deployment (dep_without_quota.yaml)

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  namespace: mynamespace
  labels:
    app: nginx
spec:
  replicas: 3
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
        - name: k8s-demo
          image: nginx
          ports:
            - name: nginx-port
              containerPort: 8080

kubectl create -f dep_without_quota.yaml

1. Verify deployment

kubectl get deploy -n mynamespace

NAME               READY   UP-TO-DATE   AVAILABLE   AGE
nginx-deployment   0/3     0            0           31s

nothing in ready state.

1. Explore reason

kubectl get rs -n mynamespace 

NAME                          DESIRED   CURRENT   READY   AGE
nginx-deployment-64ff9dbcdf   3         0         0       2m51s


kubectl describe rs nginx-deployment-64ff9dbcdf

...
Events:
  Type     Reason        Age                 From                   Message
  ----     ------        ----                ----                   -------
  Warning  FailedCreate  3m9s                replicaset-controller  Error creating: pods "nginx-deployment-64ff9dbcdf-wkr8s" is forbidden: failed quota: user-compute-quota: must specify limits.cpu for: k8s-demo; limits.memory for: k8s-demo; requests.cpu for: k8s-demo; requests.memory for: k8s-demo
  Warning  FailedCreate  3m9s                replicaset-controller  Error creating: pods "nginx-deployment-64ff9dbcdf-mhn6c" is forbidden: failed quota: user-compute-quota: must specify limits.cpu for: k8s-demo; limits.memory for: k8s-demo; requests.cpu for: k8s-demo; requests.memory for: k8s-demo
  Warning  FailedCreate  3m9s                replicaset-controller  Error creating: pods "nginx-deployment-64ff9dbcdf-77n7m" is forbidden: failed quota: user-compute-quota: must specify limits.cpu for: k8s-demo; limits.memory for: k8s-demo; requests.cpu for: k8s-demo; requests.memory for: k8s-demo
  Warning  FailedCreate  3m9s                replicaset-controller  Error creating: pods "nginx-deployment-64ff9dbcdf-t5m7q" is forbidden: failed quota: user-compute-quota: must specify limits.cpu for: k8s-demo; limits.memory for: k8s-demo; requests.cpu for: k8s-demo; requests.memory for: k8s-demo
  Warning  FailedCreate  3m9s                replicaset-controller  Error creating: pods "nginx-deployment-64ff9dbcdf-j6974" is forbidden: failed quota: user-compute-quota: must specify limits.cpu for: k8s-demo; limits.memory for: k8s-demo; requests.cpu for: k8s-demo; requests.memory for: k8s-demo
  Warning  FailedCreate  3m9s                replicaset-controller  Error creating: pods "nginx-deployment-64ff9dbcdf-ldtx8" is forbidden: failed quota: user-compute-quota: must specify limits.cpu for: k8s-demo; limits.memory for: k8s-demo; requests.cpu for: k8s-demo; requests.memory for: k8s-demo
  Warning  FailedCreate  3m9s                replicaset-controller  Error creating: pods "nginx-deployment-64ff9dbcdf-jzfqg" is forbidden: failed quota: user-compute-quota: must specify limits.cpu for: k8s-demo; limits.memory for: k8s-demo; requests.cpu for: k8s-demo; requests.memory for: k8s-demo
  Warning  FailedCreate  3m8s                replicaset-controller  Error creating: pods "nginx-deployment-64ff9dbcdf-xrp62" is forbidden: failed quota: user-compute-quota: must specify limits.cpu for: k8s-demo; limits.memory for: k8s-demo; requests.cpu for: k8s-demo; requests.memory for: k8s-demo
  Warning  FailedCreate  3m8s                replicaset-controller  Error creating: pods "nginx-deployment-64ff9dbcdf-s29ht" is forbidden: failed quota: user-compute-quota: must specify limits.cpu for: k8s-demo; limits.memory for: k8s-demo; requests.cpu for: k8s-demo; requests.memory for: k8s-demo
  Warning  FailedCreate  25s (x7 over 3m6s)  replicaset-controller  (combined from similar events): Error creating: pods "nginx-deployment-64ff9dbcdf-9gxvc" is forbidden: failed quota: user-compute-quota: must specify limits.cpu for: k8s-demo; limits.memory for: k8s-demo; requests.cpu for: k8s-demo; requests.memory for: k8s-demo

since quota is not specified in the deployment file, it fails to provision the pods.

1. Create deployment (dep_with_quota.yaml)

apiVersion: apps/v1
kind: Deployment
metadata:
  name: helloworld-deployment
  namespace: mynamespace
  labels:
    app: nginx-ssl
spec:
  replicas: 3
  selector:
    matchLabels:
      app: nginx-ssl
  template:
    metadata:
      labels:
        app: nginx-ssl
    spec:
      containers:
        - name: k8s-demo
          image: nginx:1.16
          ports:
            - name: nginxssl-port
              containerPort: 8081
          resources:
            requests:
              cpu: 200m
              memory: 0.5Gi
            limits:
              cpu: 400m
              memory: 1Gi

kubectl create -f dep_with_quota.yaml

1. Verify deployment

kubectl get rs -n mynamespace 

NAME                               DESIRED   CURRENT   READY   AGE
helloworld-deployment-7986cbf64d   3         2         2       45s
nginx-deployment-64ff9dbcdf        3         0         0       8m52s

only 2 out of 3 are in ready state.

1. Explore reason

kubectl get rs -n mynamespace 

NAME                               DESIRED   CURRENT   READY   AGE
helloworld-deployment-7986cbf64d   3         2         2       45s
nginx-deployment-64ff9dbcdf        3         0         0       8m52s

kubectl describe rs helloworld-deployment-7986cbf64d -n mynamespace

Events:
  Type     Reason            Age                From                   Message
  ----     ------            ----               ----                   -------
  Normal   SuccessfulCreate  59s                replicaset-controller  Created pod: helloworld-deployment-7986cbf64d-k46hw
  Warning  FailedCreate      59s                replicaset-controller  Error creating: pods "helloworld-deployment-7986cbf64d-cxjqx" is forbidden: exceeded quota: user-compute-quota, requested: limits.memory=1Gi,requests.memory=512Mi, used: limits.memory=2Gi,requests.memory=1Gi, limited: limits.memory=2Gi,requests.memory=1Gi
  Normal   SuccessfulCreate  59s                replicaset-controller  Created pod: helloworld-deployment-7986cbf64d-r68vf
  Warning  FailedCreate      59s                replicaset-controller  Error creating: pods "helloworld-deployment-7986cbf64d-sx2vj" is forbidden: exceeded quota: user-compute-quota, requested: limits.memory=1Gi,requests.memory=512Mi, used: limits.memory=2Gi,requests.memory=1Gi, limited: limits.memory=2Gi,requests.memory=1Gi
  Warning  FailedCreate      59s                replicaset-controller  Error creating: pods "helloworld-deployment-7986cbf64d-zx6pq" is forbidden: exceeded quota: user-compute-quota, requested: limits.memory=1Gi,requests.memory=512Mi, used: limits.memory=2Gi,requests.memory=1Gi, limited: limits.memory=2Gi,requests.memory=1Gi
  Warning  FailedCreate      59s                replicaset-controller  Error creating: pods "helloworld-deployment-7986cbf64d-khchm" is forbidden: exceeded quota: user-compute-quota, requested: limits.memory=1Gi,requests.memory=512Mi, used: limits.memory=2Gi,requests.memory=1Gi, limited: limits.memory=2Gi,requests.memory=1Gi
  Warning  FailedCreate      59s                replicaset-controller  Error creating: pods "helloworld-deployment-7986cbf64d-z8j4s" is forbidden: exceeded quota: user-compute-quota, requested: limits.memory=1Gi,requests.memory=512Mi, used: limits.memory=2Gi,requests.memory=1Gi, limited: limits.memory=2Gi,requests.memory=1Gi
  Warning  FailedCreate      59s                replicaset-controller  Error creating: pods "helloworld-deployment-7986cbf64d-7rqj4" is forbidden: exceeded quota: user-compute-quota, requested: limits.memory=1Gi,requests.memory=512Mi, used: limits.memory=2Gi,requests.memory=1Gi, limited: limits.memory=2Gi,requests.memory=1Gi
  Warning  FailedCreate      59s                replicaset-controller  Error creating: pods "helloworld-deployment-7986cbf64d-sb8ml" is forbidden: exceeded quota: user-compute-quota, requested: limits.memory=1Gi,requests.memory=512Mi, used: limits.memory=2Gi,requests.memory=1Gi, limited: limits.memory=2Gi,requests.memory=1Gi
  Warning  FailedCreate      59s                replicaset-controller  Error creating: pods "helloworld-deployment-7986cbf64d-pxsdv" is forbidden: exceeded quota: user-compute-quota, requested: limits.memory=1Gi,requests.memory=512Mi, used: limits.memory=2Gi,requests.memory=1Gi, limited: limits.memory=2Gi,requests.memory=1Gi
  Warning  FailedCreate      58s                replicaset-controller  Error creating: pods "helloworld-deployment-7986cbf64d-jdnfp" is forbidden: exceeded quota: user-compute-quota, requested: limits.memory=1Gi,requests.memory=512Mi, used: limits.memory=2Gi,requests.memory=1Gi, limited: limits.memory=2Gi,requests.memory=1Gi
  Warning  FailedCreate      39s (x8 over 57s)  replicaset-controller  (combined from similar events): Error creating: pods "helloworld-deployment-7986cbf64d-pjms2" is forbidden: exceeded quota: user-compute-quota, requested: limits.memory=1Gi,requests.memory=512Mi, used: limits.memory=2Gi,requests.memory=1Gi, limited: limits.memory=2Gi,requests.memory=1Gi

failed to provision 3rd pod as it exceeded quota

1. How much quota exhausted:

kubectl describe quota user-compute-quota -n mynamespace 

Name:            user-compute-quota
Namespace:       mynamespace
Resource         Used  Hard
--------         ----  ----
limits.cpu       800m  2
limits.memory    2Gi   2Gi
requests.cpu     400m  1
requests.memory  1Gi   1Gi

And they consumed resources as specified in resources specs of the helloworld deployment.

1. Create a limit ranger object:

apiVersion: v1
kind: LimitRange
metadata:
  name: limits-quota
  namespace: mynamespace
spec:
  limits:
    - default:
        cpu: 200m
        memory: 512Mi
      defaultRequest:
        cpu: 100m
        memory: 256Mi
      type: Container

kubectl create -f limit_range.yaml

1. Now retry dep_without_quota.yaml

kubectl create -f dep_without_quota.yaml

1. Verify again dep_without_quota replicas/pods:

kubectl get deploy -n mynamespace 

NAME               READY   UP-TO-DATE   AVAILABLE   AGE
nginx-deployment   3/3     3            3           13s

All 3 are in ready states

1. Finally check the consumption of quota

kubectl describe quota user-compute-quota -n mynamespace 

Name:            user-compute-quota
Namespace:       mynamespace
Resource         Used  Hard
--------         ----  ----
limits.cpu       600m    2
limits.memory    1536Mi  2Gi
requests.cpu     300m    1
requests.memory  768Mi   1Gi

All 3 pods got scheduled using default limits specified in limit_range.yaml This is because the deployment ‘nginx-deployment’ does not have resources spec specified hence the pods created out of it use default specs from LimitRange object.

Folks, if you like my content, would you consider following me on linked in at: https://www.linkedin.com/in/hitesh-pattanayak-52290b160/