DEV Community

Aleksi Waldén for Polar Squad

Posted on • Updated on

Prometheus Observability Platform: Application metrics

When creating our own applications, we need to use a metrics library to generate the metrics and then inside our application functions increment said metrics. With Go for example, we can use the Prometheus library. The metrics will then be exposed to the /metrics endpoint. If our application is inside a Kubernetes cluster with a prometheus-operator, we can use a ServiceMonitor to scrape its metrics. If we don’t have such a possibility, we can instead set up an application to send metrics straight to our long-term storage solution. For VictoriaMetrics, we can use the github.com/VictoriaMetrics/metrics library to send the metrics to VictoriaMetrics. Remember to add authentication logic into the section pushing the metrics to the long-term storage, if necessary.

Demo

This example assumes that you have completed the following steps as the components from those are needed:

Let's set up a hello-world Golang application in our cluster, and use ServiceMonitor to send its metrics to Prometheus.

First, we need to update our kube-prometheus-stack Helm deployment to pick up ServiceMonitor resources with a certain label attached. We need to pass the following value to our Helm chart:

prometheus:
  prometheusSpec:
    serviceMonitorSelector:
      matchExpressions:
      - key: app
        operator: Exists
Enter fullscreen mode Exit fullscreen mode

I have converted that into json so that it can be passed to Helm:

helm upgrade kube-prometheus-stack prometheus-community/kube-prometheus-stack --namespace prometheus --reuse-values --set-json 'prometheus.prometheusSpec.serviceMonitorSelector={"matchExpressions":[{"key":"app","operator":"Exists"}]}'
Enter fullscreen mode Exit fullscreen mode

This will update our kube-prometheus-stack to pick up ServiceMonitor resources from any namespace, as long as they have an app label attached.

Next, we are going to create a namespace for our hello-world application which is a simple Golang application exposing metrics via the Prometheus module. We will borrow this already-made application, which has logic defined to increment a metric called hello_processed_total each time the page is loaded.

To create a namespace and a pod, we use the following commands:

kubectl create namespace hello-world
Enter fullscreen mode Exit fullscreen mode
kubectl run hello-world --namespace=hello-world --image='okteto/hello-world:golang-metrics' --labels app=hello-world
Enter fullscreen mode Exit fullscreen mode

Next, we need to create a service for the new pod:

cat <<'EOF' | kubectl create -f -
apiVersion: v1
kind: Service
metadata:
  labels:
    app: hello-world
  name: hello-world
  namespace: hello-world
spec:
  ports:
  - name: http
    port: 8080
  selector:
    app: hello-world
  type: ClusterIP
EOF
Enter fullscreen mode Exit fullscreen mode

Now we can test that our application is working by port-forwarding it. We can also check what the hello_processed_total metric looks like:

kubectl port-forward -n hello-world services/hello-world 9090:8080
Enter fullscreen mode Exit fullscreen mode

Now navigate to http://localhost:9090 and http://localhost:9090/metrics. You should see a metric called hello_processed_total with a number attached. Each reload of the page will increment this number.

Metrics

Next, we need to set up a ServiceMonitor to send these metrics to Prometheus:

cat <<'EOF' | kubectl create -f -
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: hello-world
  namespace: hello-world
  labels:
    app: hello-world
spec:
  selector:
    matchLabels:
      app: hello-world
  endpoints:
    - port: http
EOF
Enter fullscreen mode Exit fullscreen mode

This ServiceMonitor will target services matching the label selector (app=hello-world) and will scrape the port called “http”.

Now, if we port-forward our Prometheus service, we should see a new service in the service discovery section:

kubectl port-forward -n prometheus services/kube-prometheus-stack-prometheus 9090:9090
Enter fullscreen mode Exit fullscreen mode

Navigate to http://localhost:9090/service-discovery and you should see that there is a new service discovered with the name serviceMonitor/hello-world/hello-world/0 and it should show 1/1 active targets.

We can now query the hello_processed_total metric:

Metrics in Prometheus

We have now achieved sending metrics from our custom app running in its own namespace into Prometheus.

Next part: Prometheus Observability Platform: Grafana

Top comments (0)