In this blog get-prometheus-metrics-from-a-express-js-app I exposed the metrics from an Express.js app to Prometheus. Now I am going to used these metrics in Grafana.
Install Grafana
Create a new project
oc new-project grafana
We can use the OperatorHub in Red Hat Openshift to install the Grafana-operator. I have logged in as kubeadmin user,and navigate to the OperatorHub and search for Grafana
Select the Grafana tile and continue on the install screen select install button
In the next screen select the grafana namespace and click install again
The operator should complete installing once finished go to the view operator button
We need to create a CR(custom resource) for Grafana to create a Grafana instance in the Grafana tile click on the create instance link
This will bring you to a form to populate the Grafana CR I just add a name and click create button
Thats it the Grafana instance should start, there is also a route created in the Grafana namespace.
Connecting to prometheus
Open the Grafana route in the browser
oc project grafana
oc get routes
NAME HOST/PORT PATH SERVICES PORT TERMINATION WILDCARD
grafana-route grafana-route-grafana.apps-crc.testing grafana-service 3000 edge None
# the HOST/PORT is the route so http://grafana-route-grafana.apps-crc.testing should open the grafana console
We will need to login to Grafana to be able to do anything. You can get the credentials for grafana in the Grafana CR
oc get grafana grafana -o yaml | grep admin
f:admin_password: {}
f:admin_user: {}
admin_password: secret
admin_user: root
# You can edit the secret in the Grafana CR also to change it from the default.
Select the Sign In on the bottom of the screen
And used the admin credentials in the login screen
We can now connect Grafana to Prometheus by adding a data source. Go to the now available gear icon, and select Data Source and select Prometheus as show below
In the Data Source form in the HTML section add url for the Prometheus service. The service url is in the following format.
# service-name.service-namespace.svc:port
http://prometheus.default.svc:9090
You can now see prometheus metrics in grafana.
Add some useful metrics to Grafana
You can play around with the Grafana UI to get familiar with it creating dashboards and panels or read the Grafana docs. Mainly its about adding the Prometheus expressions and pointing at the right data source.
http_request_duration_seconds_bucket
One of the metrics we get from the Express.js app is http_request_duration_seconds_bucket. When we use this metric Grafana prompts us to use Histogram_quantile with this metric.
# Prometheus Expression aggregates the 95th percentile
histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket[5m])) by (le))
As you can see I'm not seeing anything on the graph. This is because I am not generating any traffic. I wrote a small script to hit the endpoints
while true;
do
curl http://example-app-default.apps-crc.testing/
curl http://example-app-default.apps-crc.testing/hello
curl -X POST -H "Content-Type: application/json" -d '{"name": "test", "email": "test@example.com"}' http://example-app-default.apps-crc.testing/bye
done
After a few minutes I am seeing the metric show up
Up
I use this metric to determine if the container pods are up and running. As there should be 3 pods we sum the up's and divide by 3 to get a single metric , and add it to Grafana as a gauge panel
# Prometheus expression
sum(up)/3
Average request duration Express-prometheus
The Average request duration can be got by the following expression see prometheus docs for more info.
# Prometheus expression
rate(http_request_duration_seconds_sum[5m])/ rate(http_request_duration_seconds_count[5m])
Memory Metrics
There are a lot of memory metrics exposed by the Express.js app.
You can use any of these metrics for a panel.
In my example will use the Memory Heap Used vs Heap Total.
# Prometheus expressions
nodejs_heap_size_used_bytes
nodejs_heap_size_total_bytes
Adding the two metric to the panel
CPU Metrics
With CPU again there are a few metrics exposed from the Express.js app. Again when we add the CPU metric Grafana prompts us to use these metrics with rate
## Prometheus expressions
rate(process_cpu_seconds_total[5m])
rate(process_cpu_system_seconds_total[5m])
rate(process_cpu_user_seconds_total[5m])
Top comments (0)