Accessing Grafana via SSH Tunneling

Here is a quick guide "how-to" for configuring Prometheus and Grafana on Kubernetes Cluster that is behind firewalls and accessing the Grafana UI from a remote machine (your laptop at home) using SSH tunneling.

Setup Helm 3

You can use Helm 2 as well or skip this step if you already have helm installed.

Step 1: Install Helm 3

curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3
chmod 700 get_helm.sh
./get_helm.sh

Step 2: Add Stable Charts

helm repo add stable https://kubernetes-charts.storage.googleapis.com/

Setup Prometheus and Grafana using Prometheus Operator

This is the easiest way to setup Prometheus and Grafana, and have the Grafana configured to use Prometheus as a data source.

Step 3: Install Prometheus Operator

In my case, the Kubernetes cluster is behind a firewall. I am configuring the Granfana to be accessible via NodePort, as I need to access the Grafana UI using ssh tunnel.

kubectl create namespace prometheus-operator
helm install prometheus-operator stable/prometheus-operator -n prometheus-operator --set prometheusOperator.createCustomResource=false,grafana.service.type=NodePort

Step 4: Verify

kubectl get pods -n prometheus-operator

The above commands should show that all promtheus operator, prometheus, node exporter and grafana pods are running.

NAME                                                     READY   STATUS    RESTARTS   AGE
alertmanager-prometheus-operator-alertmanager-0          2/2     Running   0          30m
prometheus-operator-grafana-cf6954699-5rcgl              2/2     Running   0          30m
prometheus-operator-kube-state-metrics-5fdcd78bc-sckjv   1/1     Running   0          30m
prometheus-operator-operator-5dd8f8f568-52qk8            2/2     Running   0          30m
prometheus-operator-prometheus-node-exporter-p8pm8       1/1     Running   0          30m
prometheus-operator-prometheus-node-exporter-trlhp       1/1     Running   0          30m
prometheus-operator-prometheus-node-exporter-wsm4n       1/1     Running   0          30m
prometheus-prometheus-operator-prometheus-0              3/3     Running   1          30m

kubectl get svc -n prometheus-operator

Note that Grafana alone is running on NodePort

NAME                                           TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                      AGE
alertmanager-operated                          ClusterIP   None            <none>        9093/TCP,9094/TCP,9094/UDP   31m
prometheus-operated                            ClusterIP   None            <none>        9090/TCP                     30m
prometheus-operator-alertmanager               ClusterIP   10.102.104.48   <none>        9093/TCP                     31m
prometheus-operator-grafana                    NodePort    10.96.160.172   <none>        80:31409/TCP                 31m
prometheus-operator-kube-state-metrics         ClusterIP   10.105.92.154   <none>        8080/TCP                     31m
prometheus-operator-operator                   ClusterIP   10.99.15.245    <none>        8080/TCP,443/TCP             31m
prometheus-operator-prometheus                 ClusterIP   10.109.75.138   <none>        9090/TCP                     31m
prometheus-operator-prometheus-node-exporter   ClusterIP   10.98.128.115   <none>        9100/TCP                     31m

Configure SSH Tunnesl to access Grafana UI

You can skip this step, if you direct access to the Kubernetes Worker node IP from your machines.

Windows using PuTTY

• Get the Kubernetes Worker Node IP and the Grafana Node Port.
• Get the SSH server using which, Kubernetes Worker Node IP is accessible. Say this is Landing IP.
• Configure the PuTTY as follows:
  • Create a new Session with Landing IP, Landing Port
  • Create a Connection -> SSH -> Tunnels
  • Source Port = Grafana NodePort
  • Destination = Kubernetes Worker Node IP:Grafana Node Port
  • Open the PuTTY session. Enter SSH user name and passowrd for the Landing IP.
• Now you can access Grafana UI at the following URL. Default login and password ( admin/prom-operator )

  http://localhost:<Grafana-Node-Port>/

Linux using SSH

• Get the Kubernetes Worker Node IP and the Grafana Node Port.
• Get the SSH server using which, Kubernetes Worker Node IP is accessible. Say this is Landing IP.
• Open SSH tunnel using the following command.

  ssh -NL <Grafana-Node-Port>:<k8s-worker-node-IP>:<Grafana-Node-Port> <landing-machine-user>@<landing-machine-ip> -p <landing-machine-ssh-port>

• Now you can access Grafana UI at http://localhost:<Grafana-Node-Port>/. Default login and password ( admin/prom-operator )

Verify Granafa Dashboard

• Login to Granfa UI
• Click on Settings -> Data Source. You must see a Default Prometheus data source for http://prometheus-operator-prometheus:9090/
• Click on Dashboards -> Manage Dashboards. You must see a list of dashboards. Click on any of them like: kubernetes-compute-resources-cluster
• You must see some colors like:

References:

• https://helm.sh/docs/intro/install/
• https://github.com/helm/charts/issues/19452
• https://www.linode.com/docs/networking/ssh/using-ssh-on-windows/#ssh-tunneling-port-forwarding
• https://github.com/helm/charts/tree/master/stable/prometheus-operator
• https://github.com/helm/charts/tree/master/stable/grafana#configuration