Kubernetes - Monitor custom application metrics using Prometheus and Istio.

Kubernetes pod level metrics are captured by out of the box combination of Istio, Prometheus and Grafana components. In order to get your application metrics (e.g.: jvm metrics, custom counter metrics etc) onto Grafana, we can configure the existing setup in such a way that Istio will capture the metrics from pod and applications and merge them before sending to prometheus.

In order to do this, we need to have the application expose its metrics on some endpoint (eg: springboot provides out of the box facility to expose jvm/app metrics on actuator/prometheus api. )

Once this is setup, we can configure the deployment yaml to add annotations to the pod which help istio to scrape the metrics from the application, merge them with pod metrics and send to prometheus.

annotations:
prometheus.io/path: "/actuator/prometheus"
prometheus.io/port: "80"
prometheus.io/scheme: "http"
prometheus.io/scrape: "true"

With this setup, the application metrics will be scraped from the pod and be available along with other metrics on Grafana dashboard.