When creating our own applications, we need to use a metrics library to generate the metrics and then inside our application functions increment said metrics. With Go for example, we can use the Prometheus library. The metrics will then be exposed to the /metrics endpoint. If our application is inside a Kubernetes cluster with a prometheus-operator, we can use a ServiceMonitor to scrape its metrics. If we don’t have such a possibility, we can instead set up an application to send metrics straight to our long-term storage solution. For VictoriaMetrics, we can use the github.com/VictoriaMetrics/metrics library to send the metrics to VictoriaMetrics. Remember to add authentication logic into the section pushing the metrics to the long-term storage, if necessary.
Demo
This example assumes that you have completed the following steps as the components from those are needed:
Let's set up a hello-world Golang application in our cluster, and use ServiceMonitor to send its metrics to Prometheus.
First, we need to update our kube-prometheus-stack Helm deployment to pick up ServiceMonitor resources with a certain label attached. We need to pass the following value to our Helm chart:
prometheus:
prometheusSpec:
serviceMonitorSelector:
matchExpressions:
- key: app
operator: Exists
I have converted that into json so that it can be passed to Helm:
helm upgrade kube-prometheus-stack prometheus-community/kube-prometheus-stack --namespace prometheus --reuse-values --set-json 'prometheus.prometheusSpec.serviceMonitorSelector={"matchExpressions":[{"key":"app","operator":"Exists"}]}'
This will update our kube-prometheus-stack to pick up ServiceMonitor resources from any namespace, as long as they have an app label attached.
Next, we are going to create a namespace for our hello-world application which is a simple Golang application exposing metrics via the Prometheus module. We will borrow this already-made application, which has logic defined to increment a metric called hello_processed_total
each time the page is loaded.
To create a namespace and a pod, we use the following commands:
kubectl create namespace hello-world
kubectl run hello-world --namespace=hello-world --image='okteto/hello-world:golang-metrics' --labels app=hello-world
Next, we need to create a service for the new pod:
cat <<'EOF' | kubectl create -f -
apiVersion: v1
kind: Service
metadata:
labels:
app: hello-world
name: hello-world
namespace: hello-world
spec:
ports:
- name: http
port: 8080
selector:
app: hello-world
type: ClusterIP
EOF
Now we can test that our application is working by port-forwarding it. We can also check what the hello_processed_total
metric looks like:
kubectl port-forward -n hello-world services/hello-world 9090:8080
Now navigate to http://localhost:9090 and http://localhost:9090/metrics. You should see a metric called hello_processed_total
with a number attached. Each reload of the page will increment this number.
Next, we need to set up a ServiceMonitor to send these metrics to Prometheus:
cat <<'EOF' | kubectl create -f -
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: hello-world
namespace: hello-world
labels:
app: hello-world
spec:
selector:
matchLabels:
app: hello-world
endpoints:
- port: http
EOF
This ServiceMonitor will target services matching the label selector (app=hello-world
) and will scrape the port called “http”.
Now, if we port-forward our Prometheus service, we should see a new service in the service discovery section:
kubectl port-forward -n prometheus services/kube-prometheus-stack-prometheus 9090:9090
Navigate to http://localhost:9090/service-discovery and you should see that there is a new service discovered with the name serviceMonitor/hello-world/hello-world/0
and it should show 1/1 active targets.
We can now query the hello_processed_total metric
:
We have now achieved sending metrics from our custom app running in its own namespace into Prometheus.
Next part: Prometheus Observability Platform: Grafana
Top comments (0)