Kubectl Top command:-Secrets behind scenes

Metrics Server is typically installed as an add-on in Kubernetes clusters, including in Minikube & other K8's clusters. It is not installed by default in most Kubernetes clusters, but it can be easily added. It is an optional add-on that you install to provide real-time resource utilization data for nodes and pods.

It is a lightweight service designed to work with the Kubernetes Metrics API to provide metrics like CPU and memory usage for horizontal pod autoscaling (HPA), the kubectl top command, and more.

If the Metrics Server is not installed, commands like kubectl top or the Horizontal Pod Autoscaler won’t have the necessary data for CPU and memory metrics.

In a Minikube Cluster:

Minikube is a local Kubernetes cluster, and similar to full Kubernetes clusters, the Metrics Server is also not enabled by default.

You can enable it in Minikube using a specific add-on command.
Installing Metrics Server in Minikube:

kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml

This will install the Metrics Server into your cluster.

1. Verify Metrics Server Deployment

After installing, check if the Metrics Server is running properly:

kubectl get pods -n kube-system

You should see a metrics-server pod running. For example:

NAME                                 READY   STATUS    RESTARTS   AGE
metrics-server-86cbb8457f-zkhrn       1/1     Running   0          1m

Now just execute below commands to listed all pods , nodes in default , respective namespace .

kubectl top pods 
kubectl top nodes 
kubectl top pods -n <namespace>
kubectl top pods --all-namespaces

Now deep dive a little bit .
View Resource Usage for a Specific Pod:-

kubectl top pod <pod-name> -n <namespace>

View Container Resource Usage within a Pod

kubectl top pod <pod-name> -n <namespace> --containers

Sort by Resource Usage (CPU or Memory):-

kubectl top pods -n <namespace> --sort-by=cpu
kubectl top pods -n <namespace> --sort-by=memory

Now let's walkthrough a real world troubleshooting scenario on where post installing Metrics server its not able to execute kubectl top commands and getting error as Metrics API not available .

Workaround Step by Step :-
I assume that Metrics server adds-on already installed ,so not going through the steps as those has been already mentioned above.

Verify Metrics Server Deployment

kubectl get pods -n kube-system

You should see a metrics-server pod running. For example:

metrics-server-86cbb8457f-zkhrn       1/1     Running   0          1m

Check Logs for Errors

kubectl logs -n kube-system <metrics-server-pod-name>

Now comes the most important part :-
Ensure Proper Configuration
The Metrics Server requires proper API access and SSL certificates to function correctly. Some common configuration issues might involve:

TLS or certificate issues: If you're using self-signed certificates, ensure the certificates are set up correctly.

API Server flags: The Kubernetes API server must allow the Metrics Server to access metrics. Ensure the --kubelet-insecure-tls flag is set if you are in a development environment:

Wait for Metrics to Populate
After starting the Metrics Server, it may take a few minutes for it to collect and expose metrics from the nodes and pods. Try running kubectl top nodes again after a couple of minutes

kubectl top nodes

** Check Kubelet Configuration**

The Kubelet (running on each node) must expose its metrics to the Metrics Server. Ensure that the --authentication-token-webhook and --authorization-mode=Webhook flags are enabled on your Kubelet configuration, allowing it to authenticate API requests.

Edit the Metrics Server deployment:

kubectl edit deployment metrics-server -n kube-system

Add the --kubelet-insecure-tls flag under the args section in the container definition

spec:
  containers:
  - name: metrics-server
    image: k8s.gcr.io/metrics-server/metrics-server:v0.6.2
    args:
      - --cert-dir=/tmp
      - --secure-port=4443
      - --kubelet-preferred-address-types=InternalIP,Hostname,ExternalIP
      - --kubelet-insecure-tls

Save and exit. Kubernetes will automatically update and restart the Metrics Server pod.

** Restart Metrics Server**
After applying any of the above changes, you should restart the Metrics Server to ensure it picks up the new configuration:

kubectl rollout restart deployment metrics-server -n kube-system

Check Logs and Verify
After restarting, check the Metrics Server logs to ensure the issue is resolved:

kubectl logs -n kube-system <metrics-server-pod-name>

Now run the top command again .

kubectl top nodes
kubectl top pods 

Conclusion:
The Metrics Server is an essential tool in Kubernetes, enabling real-time resource monitoring and driving features like the Horizontal Pod Autoscaler (HPA). While it focuses on lightweight and short-term metrics, it is a key component for ensuring efficient resource management and auto-scaling within a cluster.