Monitor and Visualize Nginx Ingress Controller Metrics on Amazon EKS with Prometheus & Grafana

In today’s digital landscape, it is very necessary to figure out trends in our data and act accordingly to ensure high availability of our application. Monitoring helps us to stay on top of the game and get insights on our product environment. An Ingress Controller acts as a bridge between Kubernetes Service and the external world and can be considered as a specialized load balancer for Kubernetes.

Pre-Requisites

1. A Kubernetes Cluster (Create an EKS Cluster from scratch)
2. Helm V3 (Helm Install Docs)

Install Kube-Prometheus-Stack using Helm

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts 
helm repo update 
helm install --create-namespace --namespace monitoring \
my-k8s-prom-stack prometheus-community/kube-prometheus-stack

Once you have the stack installed you should have Prometheus, Grafana and AlertManager installed onto your cluster.

Install Nginx Ingress Controller

helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx 
helm repo update 
helm install ingress-nginx ingress-nginx/ingress-nginx \
--namespace ingress-nginx --create-namespace \
--set controller.metrics.enabled=true \
--set controller.metrics.serviceMonitor.enabled=true \
--set controller.metrics.serviceMonitor.additionalLabels.release="my-k8s-prom-stack" \
--version=4.5.2 

Along with installing the ingress controller we are setting the controller metrics as true which will populate an additional metrics service and creating a service monitor for that service which will feed data onto Prometheus, the additional Label would be as per the selector you have set on the Prometheus Operator, follow the later part for more details on this.

You can check the Service Monitor selector on your Prometheus Operator using:

kubectl get prometheus --namespace monitoring prometheus-svc-name –oyaml (lookout for the section below on the manifest)

Once Nginx controller is installed you can verify the services that are spawned using:

kubectl get svc --namespace ingress-nginx

Monitor and Visualize Nginx Controller Metrics

To access the Prometheus and Grafana you can set ingress objects
Sample Ingress Manifest:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:    
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/proxy-body-size: 5m  
  name: prometheus-ingress
  namespace: monitoring
spec:
  rules:
  - host: prom.mydomain.com
    http:
      paths:
      - backend:
          service:
            name: my-k8s-prom-stack-kube-pro-prometheus
            port:
              number: 9090
        path: /
        pathType: ImplementationSpecific

Generate some load on the Ingress Controller by hitting the Ingress Endpoints.

Access Prometheus and Grafana to query and visualize the controller metrics:

Prom Query to check the number of 200 requests on ingress:
nginx_ingress_controller_requests{status=~'2..'}

Import Nginx Controller Dashboard on Grafana and visualize the metrics (Dashboard ID: 9614)

Thank you for reading this, you can follow me for more such content, or reach out to me on Linkedin.