DEV Community: Julien Acroute

Beyond the Buzz: Embracing the Magic of eBPF in Kubernetes

Julien Acroute — Wed, 10 Apr 2024 14:02:32 +0000

Beyond the Buzz: Embracing the Magic of eBPF in Kubernetes

In a time where the buzz around Artificial Intelligence (AI) seems to overshadow everything else, this year's KubeCon Europe offered a refreshing perspective. While AI continues to be a hot topic, some in the Kubernetes community are starting to feel a bit tired of it. With all the hype and uncertainty surrounding AI, another hero has emerged: eBPF (Extended Berkeley Packet Filter).

AI: A Distant Shining Horizon

AI has certainly added some excitement to discussions about cloud-native technologies, from automating cluster troubleshooting to hosting AI on Kubernetes. But not everyone is fully on board. While AI has proved to be of great assistance - e.g. suggesting, fixing and reviewing code written by humans- some folks worry that too much focus on AI might distract from more practical, ready-to-implement advancements. The feeling is clear: while AI offers lots of possibilities, the horizon is a bit uncertain and overly hyped.

eBPF: Here and Now in Kubernetes Innovation

In contrast, eBPF is all about practical innovation. It's a technology that delivers real results, right here, right now. We need solutions for network security, observability, and performance today, and that's where eBPF shines. Unlike the abstract promises of AI, eBPF offers concrete tools and methods to improve Kubernetes environments immediately. For example, when it comes to network security, eBPF-powered tools like Cilium can do the job without needing the complexity of AI.
This shift towards valuing what's immediately useful over what's exciting but distant was noticeable at KubeCon. As we dive deeper into what eBPF can do, it becomes clear why this technology has captured the attention of the Kubernetes community.

The Way Forward with eBPF

By embracing eBPF, the Kubernetes community isn't just adopting new tools; it's championing a philosophy of practical, tangible progress. As we explore the latest eBPF innovations – from better security to revolutionary observability tools – we see the benefits eBPF brings to Kubernetes. It's a journey grounded in reality, offering not just a vision of the future, but a roadmap to get there.

Redefining Efficiency: eBPF's Radical Return to Linux's Roots

eBPF represents a refreshing departure from the conventional approach of stacking layer upon layer in pursuit of functionality, often resulting in bloated, resource-intensive systems, even for simple tasks like rendering a webpage or routing network packets between nodes or clusters. With eBPF, we're venturing back into the depths of the Linux system, where innovation meets efficiency. Here, we witness a paradigm shift—a departure from the status quo. The results speak for themselves: a nearly negligible overhead and remarkable responsiveness. In fact, eBPF brings us so close to real-time processing that it's revolutionizing how we think about performance in cloud-native environments. We're not just optimizing; we're redefining what's possible, and eBPF is leading the charge.

eBPF talks from KubeCon 2024

Cilium: Connecting, Observing, and Securing Service Mesh and Beyond with eBPF
Speakers: Liz Rice, Christine Kim, Nico Meisenzahl, Vlad Ungureanu
https://youtu.be/wq1TxZw1AaY?si=JTyhE333QfsGht0T
Dealing with eBPF’s Observability Data Deluge
Speaker: Anna Kapuścińska
https://youtu.be/yWB8n_e4N14?si=OyMJEKzbxS5zxA5P
Unlock Energy Consumption in the Cloud with eBPF
Speaker: Leonard Pahlke
https://youtu.be/lW9pZoKRJVs?si=rX5CQMaFuZBm8bBT
Fast and Efficient Log Processing with Wasm and eBPF
Speaker: Michael Yuan
https://youtu.be/4u7nUpZxr3g?si=pkcpoEwOeDaH5HcE
No 'Soup' for You! Enforcing Network Policies for Host Processes via eBPF
Speaker: Vinay Kulkarni
https://youtu.be/AWAf3H4Qwq8?si=qVqQfWb3J_905BCJ
eBPF’s Abilities and Limitations: The Truth
Speakers: Liz Rice & John Fastabend
https://youtu.be/tClsqnZMN6I?si=TyMFTMk4Q45K6T2v

Use Tetragon to Limit Network Usage for a set of Binary

Julien Acroute — Thu, 03 Aug 2023 07:39:02 +0000

A matter of trust

Many interesting software are coming from the community, many are distributed through the package manager of the operating system. But for the others, you can download them from Github release pages, use snap or homebrew to cite a few. But this last installation method bypasses the security team that tries to improve the security of your operating system. By doing so, you are implicitly trusting the author he is not distributing malware or implementing backdoors. How many tools did you install by hand? Do you really trust all of them? Confidence is very important, yet it would be nice to limit capabilities for a set of binary that you don't fully trust. In this blog post, we will use Tetragon to forbid network usage for tools that don't need to.

Goal

We will separate tools installed locally into two families. On one side, tools that need network access in a specific directory. Another directory for tools that don't need internet access.

For example the following tools use network sockets:

while these do not:

We will move the last tools in a specific directory ~/bin-no-network/ and use Tetragon to inject a policy in the kernel as a eBPF program to kill any binary located in ~/bin-no-network/ trying to open a network socket.

Tetragon installation

Tetragon is "Kubernetes-aware" but it can also be used outside Kubernetes on a regular workstation. You can deploy Tetragon as a container:

$ docker run --name tetragon --rm -d                 \
    --pid=host --cgroupns=host --privileged          \
    -v /sys/kernel/btf/vmlinux:/var/lib/tetragon/btf \
    quay.io/cilium/tetragon:v0.10.0

Because Tetragon needs to inject code in the kernel, we need to bypass most of the docker isolation mechanism. We only use the packaging feature of docker to avoid installation of system libraries and the binary. So you will have to trust ;-) this tetragon binary because it will run like a process running as root on your workstation. Note the mount point: /sys/kernel/btf/vmlinux, this is a kind of bridge to eBPF kernel features.

eBPF Principle

When using eBPF, applications are always composed of two parts, one deployed in the Kernel as an eBPF program and another running as a regular program in "user space". The user space program (Tetragon) will inject a small eBPF program in the kernel to intercept some system calls. This means that every application that uses this system call will trigger this eBPF program that can observe or modify the system call result. A specific data structure is then created in the kernel: a ring buffer. This is used by the eBPF program to store some interesting data. Finally, the user space program Tetragon, which has also read-only access to this data structure, will be able to retrieve information gathered by the eBPF program.
One good point with this architecture is that evaluation of rules is done within the kernel and does not require communication with the user space program. The only drawback is that information observed by the eBPF program can be overridden by new incoming information if the user space part does not read information fast enough. This is how ring buffers are designed.

Writing a Policy

Our goal is to forbid network usage coming from binaries in a specific folder.
For this we will need a CLI to interact with tetragon. We can use the tetra binary in the docker container:

$ alias tetra="docker exec -ti tetragon tetra"
$ tetra version
server version: v0.10.0
cli version: v0.10.0

Even if we are not using Kubernetes, we still need to write some yaml as Tetragon only understands TracingPolicy objects. A TracingPolicy is a kubernetes custom resource to install hooks in the kernel and actions.

Let's start with a TracingPolicy from the Tetragon documentation:

apiVersion: cilium.io/v1alpha1
kind: TracingPolicy
metadata:
  name: "connect"
spec:
  kprobes:
  - call: "tcp_connect"
    syscall: false
    args:
    - index: 0
      type: "sock"
  - call: "tcp_close"
    syscall: false
    args:
    - index: 0
      type: "sock"
  - call: "tcp_sendmsg"
    syscall: false
    args:
    - index: 0
      type: "sock"
    - index: 2
      type: int

This Tracing policy will just observe network events:

socket creation (tcp_connect)
traffic in the socket (tcp_sendmsg)
socket close (tcp_close)

Now we need to add:

an action when this kind of event is detected
a filter to only apply this action if this is generated from a binary located in our specific directory ~/bin-no-network/.

We need to add a selectors section in the TracingPolicy and use the matchBinaries selector to apply this policy only for binary in the ~/bin-no-network/ folder:

    selectors:
    - matchBinaries:
      - operator: "In"
        values:
          - "/home/jacroute/bin-no-network/curl"
          - "/home/jacroute/bin-no-network/jq"

Unfortunately, only 'In' and 'NotIn' operator are implemented for the matchBinaries selector. We cannot use the 'Prefix' operator like this:

    selectors:
    - matchBinaries:
      - operator: "Prefix"
        values:
          - "/home/jacroute/bin-no-network/"

So the TracingPolicy should look like:

apiVersion: cilium.io/v1alpha1
kind: TracingPolicy
metadata:
  name: "connect"
spec:
  kprobes:
  - call: "tcp_connect"
    syscall: false
    args:
    - index: 0
      type: "sock"
    selectors: &selector
    - matchBinaries:
      - operator: "In"
        values:
          - "/home/jacroute/bin-no-network/curl"
          - "/home/jacroute/bin-no-network/jq"
      matchActions:
      - action: Sigkill
  - call: "tcp_close"
    syscall: false
    args:
    - index: 0
      type: "sock"
    selectors: *selector
  - call: "tcp_sendmsg"
    syscall: false
    args:
    - index: 0
      type: "sock"
    - index: 2
      type: int
    selectors: *selector

Deploy the TracingPolicy

We first need to transfer the file to the container. Suppose that you stored the TracingPolicy in bin-no-network.yaml file, you can transfer the policy using the docker cp command:

$ docker cp bin-no-network.yaml tetragon:/tmp/bin-no-network.yaml

Then we can use the tetra cli to deploy this policy:

tetra tracingpolicy add /tmp/bin-no-network.yaml

Testing the TracingPolicy

Now we can test if the policy is blocking network access for the two binaries listed in the policy.

$ mkdir ~/bin-no-network/
$ cp /usr/bin/curl ~/bin-no-network/
$ cp /usr/bin/jq ~/bin-no-network/
$ ~/bin-no-network/curl google.fr
[1] 122677 killed ~/bin-no-network/curl google.fr
$ echo "{}" | ~/bin-no-network/jq
{}

curl tried to open a socket but the process was killed. jq does not try this kind of system call and was not killed.

Automatically populating the Policy

We can build a shell script to maintain the list of binaries synchronized with the content of the ~/bin-no-network/ directory. Using find we can list binaries in this folder:

$ find ~/bin-no-network/ -executable -type f
/home/jacroute/bin-no-network/curl
/home/jacroute/bin-no-network/jq

Then with awk with can add surrounding spaces and quotes needed to integrate the yaml:

find ~/bin-no-network/ -executable -type f | awk '{ print "          - \""$0"\""}'
          - "/home/jacroute/bin-no-network/curl"
          - "/home/jacroute/bin-no-network/jq"

Finally, integrate this in the yaml and inject the policy in the kernel with the following script:

#!/bin/bash

docker exec tetragon tetra sensors rm connect

policy=$(mktemp -p /dev/shm)

cat << EOF > $policy
apiVersion: cilium.io/v1alpha1
kind: TracingPolicy
metadata:
  name: "connect"
spec:
  kprobes:
  - call: "tcp_connect"
    syscall: false
    args:
    - index: 0
      type: "sock"
    selectors: &selector
    - matchBinaries:
      - operator: "In"
        values:
$(find ~/bin-no-network/ -executable -type f | awk '{ print "          - \""$0"\""}')
      matchActions:
      - action: Sigkill
  - call: "tcp_close"
    syscall: false
    args:
    - index: 0
      type: "sock"
    selectors: *selector
  - call: "tcp_sendmsg"
    syscall: false
    args:
    - index: 0
      type: "sock"
    - index: 2
      type: int
    selectors: *selector
EOF

docker cp $policy tetragon:/tmp/policy.yaml
docker exec tetragon tetra tracingpolicy add /tmp/policy.yaml
rm $policy

Building trust in your relationship with your workstation XD

After a first date with Tetragon during the Kubernetes Community Days during 2023, I was seduced by the way it's implemented: most of the time new features are implemented on top of others creating a complex multi layered cathedral. Perfect for a wedding, but I prefer simplicity.

Using eBPF to enforce "security" seems to be very efficient from the performance point of view. Bypassing Tetragon security by denial of service attacks is unlikely. For example, the sigkill action triggered by the Tetragon’s eBPF program is done synchronously, in the kernel and not at user space, which makes it hard to bypass. In other words, the security mechanism is fully and autonomously implemented at kernel level.

This mechanism proves to be effective in blocking network activity and building trust in our beloved workstation even when running potentially malicious binaries.

Using Tetragon in a Kubernetes context is the next step. The plan is to observe the behavior of an application in a development environment (files read/write, command executed) and generate a profile. Then, promote this profile from development to production, and enforce an allow-only behavior using Tetragon.

Using ArgoCD Pull Request Generator to review application modifications

Julien Acroute — Tue, 11 Apr 2023 14:29:12 +0000

As a developer, when modifications are pushed to a feature branch, you and your team want to test this new feature. If you have the chance to work with a stateless application, you can deploy another instance of the application with modifications from the feature branch.

An interesting feature of ArgoCD is the Pull Request Generator. It's a generator for ApplicationSet. An ApplicationSet is a template of ArgoCD Application associated with a generator. Generator can be a directory: an application will be created for every sub-folder. There is also the Cluster generator that deploy the same Application but in every cluster managed by ArgoCD.

The syntax for the Pull Request generator is quite simple:

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: myapps
spec:
  generators:
  - pullRequest:
      github:
        owner: camptocamp
        repo: myrepository
        tokenRef:
          secretName: github-token
          key: token
        labels:
        - deploy
  template:
    ...

Before testing this feature what is expected ?

A new ArgoCD Application is created when a pull request is created with a "deploy" label
In the template of the Application we can use metadata of the pull request: ID, title, description, labels, source and target branch name, commit ref, …
A comment is added to the PR when the Application is deployed and Synced with the available URLs

Now we will see how to implement this ;-)

Workflow before Implementation

Source Code Repository

Let's start with a simple application repository: git@github.com:camptocamp/frontend.git
Let's consider that this repository has a github action that builds a container image when a pull request is opened. The image is tagged with the short commit hash and the concatenation of the branch name and the short commit hash. For example, if the feature branch is name update_lib with the last commit: 4a9b29e, the following tags will be generated:

4a9b29e
update_lib-4a9b29e

Deployment Repository

There is another git repository: git@github.com:camptocamp/argocd-project-foo-apps.git to describe what needs to be deployed in each kubernetes cluster (dev, int, …). In this repository, we have one folder per environment:

apps
├── dev
│   ├── backend
│   └── frontend
├── int
│   ├── backend
│   └── frontend
└── prod
    ├── backend
    └── frontend

We are using an ApplicationSet to deploy every component defined in this git repository using the directory generator. For example, for "dev" env:

---
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: foo-dev
  namespace: argocd
spec:
  generators:
  - git:
      directories:
      - path: apps/dev/*
      repoURL: git@github.com:camptocamp/argocd-project-foo-apps.git
      revision: main
  template:
    metadata:
      name: foo-dev-{{path.basename}} # sub folder name
    spec:
      source:
        path: '{{path}}' # full path
        repoURL: git@github.com:camptocamp/argocd-project-foo-apps.git
        targetRevision: main
      project: default
      destination:
        namespace: foo-dev
        server: https://kubernetes.default.svc

This ApplicationSet will create one ArgoCD Application per folder in apps/dev/. Imagine that we have the following structure in git :

apps
└── dev
    ├── backend
    │   ├── Chart.yaml
    │   └── values.yaml
    ├── database
    │   ├── Chart.yaml
    │   └── values.yaml
    └── frontend
        ├── Chart.yaml
        └── values.yaml

This will generate 3 Argocd Applications :

foo-dev-database
foo-dev-backend
foo-dev-frontend

Review App Workflow

When a pull request is opened for a specific application, for example the frontend, we want to deploy a new instance of this specific component. We will monitor pull requests on the frontend repository.
So we want to create an additional ApplicationSet to deploy the frontend if a pull request is created with the label deploy in the frontend git repository.

Implementation

Create a token to access GitHub API

First step is to create a token to access the frontend repository.

Then we need to deploy this token as a kubernetes secret in the cluster :

export GITHUB_TOKEN=ghp_XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
kubectl create secret generic github-token \
  --from-literal=token=$GITHUB_TOKEN \
  -o yaml --dry-run=client > github-token-secret.yaml

If you follow the GitOps principles to deploy manifests in the cluster, you can commit this file.

Create a webhook

For a better user experience, we can setup a webhook that will notify ArgoCD when something changes on Github.
This webhook needs to be defined in the source code repository, where pull requests are created.
Go to the repository "Settings" and then "Webhooks". Click on the "Add webhook" button.

The Payload URL is the URL to access ArgoCD with the path /api/webhook.
The Content Type should be set to application/json.
It's a good idea to set a "Secret", just use a random string, we will use this random string in the next step.
Regarding events, we need to individually selects events and choose "Pull requests" events.

Deploy the new ApplicationSet

This new ApplicationSet will use the "Pull request" generator, this "generator" will monitor pull requests on the source code repository. Just create and commit a file with the following ApplicationSet manifest and deploy this new manifest.

---
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: foo-dev-review-frontend
  namespace: argocd
spec:
  generators:
  - pullRequest:
      github:
        owner: camptocamp
        repo: frontend # This is the application source code repo
        tokenRef:
          secretName: github-token
          key: token
        labels:
        - deploy # label on PR that trigger review app
  template:
    metadata:
      name: 'foo-dev-frontend-{{branch}}-{{number}}'
    spec:
      source:
        repoURL: git@github.com:camptocamp/argocd-project-foo-apps.git
        targetRevision: main
        path: apps/dev/frontend
        helm:
          parameters:
          - name: "image.tag"
            value: "{{branch}}-{{head_sha}}" # override of the image tag
          - name: "ingress.prefix"
            value: "{{branch}}" # add a prefix to the URL 
      project: default
      destination:
        server: https://kubernetes.default.svc
        namespace: foo-dev

This ApplicationSet will deploy a new instance of the frontend for each pull request with the deploy label. The image tag will be overridden with the branch name and the short commit hash and a unique prefix will be added to the URL to avoid conflicts with other instances. Finally, the name of the release is also unique as it's the same as the Application name: foo-dev-frontend-{{branch}}-{{number}}, this should also avoid conflicts on object names.

Configure ArgoCD to use secret for webhook

The last step is to protect the ArgoCD webhook with a password. The ArgoCD Helm chart allows setting a secret for the github webhook endpoint : configs.secret.githubSecret.

Testing

Now it's time to test. For this step, you just need to make a modification in the source repository and create a pull request for this. Please wait until the container image is built. Then by adding the deploy label a new ArgoCD app should be created :-)

Conclusion

This new ArgoCD feature is very interesting. Maybe it can help to have feedback on the pull requests to see the status of the review app. It can also be interesting to have the list of URLs available. ArgoCD already shows this information in the webUI. So maybe just a link to the ArgocD application is enough.

Camptocamp will participate in the KubeCon at Amsterdam from 18 to 21 april. Maybe we can meet there and discuss development workflows that really help developers.

Integrate an Application with Prometheus Operator and Package with a Helm Chart

Julien Acroute — Mon, 19 Jul 2021 08:59:28 +0000

In the previous posts, we saw:

how to implement metrics in applications
how to run the monitoring stack locally
how to test and debug metrics generated by a simple Python Flask application

In this post we will:

use Kubernetes Custom Resources to integrate our application with the Prometheus Operator
define some alerts based on the metrics generated by the application
deploy a custom dashboard in Grafana
package everything in a Helm chart, including a Grafana dashboard

Connect Prometheus to your Application

Prometheus will retrieve metrics from Pods with a /metrics HTTP endpoint. If the Prometheus Operator is deployed in your Kubernetes Cluster, the discovery of the Pods is done by deploying one of the following custom Kubernetes objects:

Using ServiceMonitor

When a ServiceMonitor object is deployed, Prometheus will create one target per address defined in the Endpoints object linked to the Service. This means every Pod is in a ready status used by the Service.

For example, if you have the following Service and Deployment in your cluster:



apiVersion: v1
kind: Service
metadata:
  name: webapp
  labels:
    component: backend
    instance: app
    name: containers-my-app
  namespace: my-app
spec:
  type: ClusterIP
  ports:
  - name: http
    port: 80
    protocol: TCP
    targetPort: webapp
  selector:
    component: backend
    instance: app
    name: containers-my-app



apiVersion: apps/v1
kind: Deployment
metadata:
  name: webapp
  labels:
    component: backend
    instance: app
    name: containers-my-app
  namespace: my-app
spec:
  selector:
    matchLabels:
      component: backend
      instance: app
      name: containers-my-app
  template:
    metadata:
      labels:
        component: backend
        instance: app
        name: containers-my-app
    spec:
      containers:
      - name: app
        image: ghcr.io/camptocamp/course_docker_backend:python
        ports:
        - containerPort: 8080
          name: webapp

As defined in the Deployment’s spec.template.metadata.labels field, Pods will have the following labels:

component: backend
instance: app
name: containers-my-app

The Service has a selector that matches labels of the Pod. Therefore the Service will load balance traffic to Pods deployed by the Deployment.

ServiceMonitor objects also use a selector to discover which Services need to be monitored. Prometheus will scrape metrics from every Pods behind selected Services.

For example, to retrieve metrics from our Pods, we can deploy the following ServiceMonitor object:



apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: webapp
  labels:
    component: backend
    instance: app
    name: containers-my-app
    release: prom
  namespace: my-app
spec:
  namespaceSelector:
    matchNames:
    - my-app
  selector:
    matchLabels:
      component: backend
      instance: app
      name: containers-my-app
  endpoints:
  - port: http

Prometheus Operator will search for Services:

in the my-app namespace,
with the following labels:
- component: backend
- instance: app
- name: containers-my-app
with a port named: http

It will then use the Service selector to find Pods. As a result, one target per Pod will be created in the Prometheus configuration.

Using PodMonitor

PodMonitor objects use a selector to find Pods directly. No Service needs to be deployed.

For our Pods, we can use the following PodMonitor:



apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
  name: webapp
  labels:
    acomponent: backend
    instance: app
    name: containers-my-app
    release: prom
  namespace: my-app
spec:
  namespaceSelector:
    matchNames:
    - my-app
  selector:
    matchLabels:
      component: backend
      instance: app
      name: containers-my-app
  podMetricsEndpoints:
  - port: webapp

The Prometheus operator will search for Pods:

in the my-app namespace,
with the following labels:
- component: backend
- instance: app
- name: containers-my-app
with a port named: webapp

For each Pod, a new target will be added to the Prometheus configuration.

Configure Alerts

Why use Alerts?

Gathering and storing metrics is very useful to investigate when something goes wrong. But there are often some modifications of one or a combination of metrics before a service becomes completely unusable.

A common example of this is remaining free disk space. Fixing hard thresholds with arbitrary values on disk space is usually inefficient (you might actually end up with 95%, 100% and 101% thresholds). What needs to be monitored is actually the estimated time left until the disk is full, which can be obtained by running a time regression on the metric.

Some other examples:

For an online store, having no purchases during the afternoon is unusual, maybe there is an issue that blocks users in the payment process.
If the ratio of HTTP responses with code 500 suddenly increases, investigation is needed.

This is the purpose of alerts: when such events are detected, the system notifies the right person, allowing you to keep your eyes off dashboards.

After investigating and finding the root cause, you should always ask yourself if you can build an alert to detect such a case. There is also the possibility of increasing the observability if some metrics are missing.

How to define Alerts?

The Prometheus Operator allows the definition of alerts with a custom Kubernetes object: PrometheusRule

In this custom resource, you can define multiple alerts:



apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: webapp
  namespace: my-app
  labels:
    component: backend
    instance: app
    name: containers-my-app
    app: kube-prometheus-stack
    release: prom
spec:
  groups:
    - name: app
      rules:
        - alert: NoProductViewSince1h
          expr: (view_total - view_total offset 1h) < 1
          for: 5m
          labels:
            severity: Critical

In the example above, only one alert is defined.

Find below what needs to be defined for usual cases for each alert:

alert: the alert name
expr: a PromQL expression that triggers an alert if it returns something. This is why most of the time a threshold is used. With the offset function we compute the views from the past hour, if the result is above the threshold 1, then an alert is created and pushed to AlertManager.
Optional labels: a set of labels, usually used for alert severity
Optional for: delays triggering the alert. The PromQL expression must return some sample during the duration of the field for, before an alert is triggered.

Troubleshooting

There are many selectors involved in this process:

selector on ServiceMonitor to find Services
selector on Service to find Pods
selector on PodMonitor to find Pods

There are also selectors to discover ServiceMonitor, PodMonitor, and PrometheusRule objects. Those selectors are defined in the Prometheus object using the following fields:

Discovery of ServiceMonitor:
- spec.serviceMonitorSelector
- spec.serviceMonitorNamespaceSelector
Discovery of PodMonitor:
- spec.podMonitorSelector
- spec.podMonitorNamespaceSelector
Dicovery of PrometheusRule:
- spec.ruleSelector
- spec.ruleNamespaceSelector

Check Selectors

If the target is not discovered by Prometheus:

Check that your ServiceMonitor or PodMonitor is deployed in a Namespace that matches the namespace selector in the Prometheus object.
Check that labels on your ServiceMonitor or PodMonitor match the selector in the Prometheus object.
Check that the selector on your ServiceMonitor or PodMonitor matches labels defined in the Service or Pod.
Check that the Service or Pod are deployed in the Namespace selected by the namespace selector defined in the ServiceMonitor or PodMonitor.

In order to check a label selector, you can use the -l option of kubectl. For example, to check the following selector in a ServiceMonitor:



  selector:
    matchLabels:
      component: backend
      instance: app
      name: containers-my-app

run the following command:



kubectl -l component=backend,instance=app,name=containers-my-app get service

Check Port Name or Number

Check that the port number or name matches a port defined in a Service or a Pod.

ServiceMonitor references either an incoming port defined in the Service or a Pod port:

ServiceMonitor.spec.endpoints.port references the name of a Service port: Service.spec.ports.name
ServiceMonitor.spec.endpoints.targetPort references a Pod port: Pod.spec.containers.ports.containerPort or Pod.spec.containers.ports.name

PodMonitor references port defined on Pod:

PodMonitor.spec.podMetricsEndpoints.port reference the name of a Pod port: Pod.spec.containers.ports.name

Note that Prometheus will only use Pods with a Ready state.

Create Grafana Dashboards as ConfigMaps

Grafana includes an auto discovery mechanism for dashboards. Any ConfigMap with a label grafana_dashboard=1 is loaded into Grafana.

The following ConfigMap will create a minimal dashboard in Grafana. Note that this ConfigMap needs to be deployed in the same Namespace as Grafana.



kind: ConfigMap
apiVersion: v1
metadata:
  name: my-dashboard
  labels:
    grafana_dashboard: "1"
data:
  dashboard.json: |
    { "title": "Product Views",
      "time": { "from": "now-6h", "to": "now" },
      "editable": false,
      "panels": [ {
          "gridPos": { "h": 9, "w": 12, "x": 0, "y": 0 },
          "id": 2,
          "targets": [ {
              "exemplar": true,
              "expr": "rate(view_total[2m])",
              "interval": "",
              "legendFormat": "{{product}}",
              "refId": "A"
            } ],
          "title": "Product View",
          "type": "timeseries"
        }
      ]
    }

Package Monitoring Objects with Applications using Helm Charts

Including monitoring objects in Application Helm Charts is a good way to maintain the observability layer of an application. The monitoring components can be versioned with application packaging. Also the deployment of monitoring can follow the same workflow as the application.

I will not explain how to package an application, but I’ll demonstrate how to include the following elements:

Dashboard: ConfigMap with dashboard code
Alerts: PrometheusRule with alerts definition
Metrics endpoint: PodMonitor or ServiceMonitor

In the chart’s values.yaml, add a new section for monitoring:



monitoring:
  alerts: true
  dashboard: true

The monitoring.alerts values controls the deployment of the PrometheusRule object. The deployment of the ConfigMap for the dashboard is controlled by monitoring.dashboard.

For the PodMonitor or ServiceMonitor objects, we can check if the Prometheus Operator is installed using the .Capabilities.APIVersions.Has function:



{{- if .Capabilities.APIVersions.Has "servicemonitor.monitoring.coreos.com/v1" }}
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
...
{{- end }}

Additionally, for alerts and dashboards, we can check the "values" set on the Helm release:



{{- if .Capabilities.APIVersions.Has "prometheusrule.monitoring.coreos.com/v1" }}
{{- if .Values.monitoring.alerts }}
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
...
{{- end }}
{{- end }}

A common workflow to maintain a dashboard is to:

edit dashboards in the Grafana Web UI
copy the JSON model from Web UI
paste the JSON to a file in the Helm Chart
commit and push modifications

⚠ If the JSON representation of the dashboard is stored in the ConfigMap code, you will have to indent the content properly:



kind: ConfigMap
apiVersion: v1
metadata:
  name: my-dashboard
  labels:
    grafana_dashboard: "1"
data:
  dashboard.json: |
    { "title": "Product Views",
      "time": { "from": "now-6h", "to": "now" },
      "editable": false,
      "panels": [ {
          "gridPos": { "h": 9, "w": 12, "x": 0, "y": 0 },
          "id": 2,
          "targets": [ {
              "exemplar": true,
              "expr": "rate(view_total[2m])",
              "interval": "",
              "legendFormat": "{{product}}",
              "refId": "A"
            } ],
          "title": "Product View",
          "type": "timeseries"
        }
      ]
    }

It is generally easier to store the dashboard code in a dedicated file and then load the contents in the ConfigMap with some Helm templating functions:



{{- if .Capabilities.APIVersions.Has "prometheusrule.monitoring.coreos.com/v1" }}

{{- if .Values.monitoring.dashboard }}

apiVersion: v1

kind: ConfigMap

…

data:


  dashboard.json: |


    {{ .Files.Get "dashboard.json" | trim | nindent 4 }}

{{- end }}

{{- end }}

Conclusion

Deploying monitoring stacks —Prometheus, Grafana, AlertManager, ElasticSearch, Loki, …— provides much more observability in a project, but at the cost of consuming more resources. Developers not using these features is a waste of resources. The project also has poor observability because only system metrics and maybe http metrics are retrieved. Adding application specific metrics and even business metrics allows you to build beautiful dashboards with colors and graphs that are very fun to report during boring review meetings.

Testing Application Monitoring Locally with a Docker Composition

Julien Acroute — Wed, 14 Apr 2021 13:37:35 +0000

In the previous post, we saw how to implement metrics in a simple Python Flask application. In this post we will see how to:

start a local monitoring stack with a Docker composition
configure Prometheus to scrape metrics from our application
build a Grafana dashboard
commit the Dashboard definition so other developers can also use it

Before launching an application in a Kubernetes cluster, we need to make sure that it generates the right metrics. This will be done using a Prometheus stack running in a Docker composition.

This test Prometheus stack will be made of:

Prometheus: Core component that will:
- scrape application metrics
- store metrics in a time series database
- expose metrics for use with Grafana
Grafana: Visualization of Prometheus metrics

To be able to run this Docker composition, you will need to install Docker and install docker-compose.

Start Prometheus

Let's write a docker-compose.yml file that will start a Docker container for Prometheus:



version: '3'
services:
  prometheus:
    image: prom/prometheus:v2.26.0
    ports:
      - "9090:9090"

Be sure to use only spaces for indentation in the YAML file, not tabs.

Start the Docker composition with:



docker-compose up -d

This composition starts Prometheus with the default configuration and allows access to the web interface on http://localhost:9090/.
You will be redirected to the "Graph" page where you can query metrics using the PromQL format.

In the Status → Targets menu, you can find all exporters from which Prometheus retrieves metrics.

In the default setup, there is only one defined target, which points to the metrics endpoint of Prometheus itself.

Now it's time to configure Prometheus to retrieve metrics from our application!

Prometheus is running in a container but needs to access the application that runs on the host. We will start the application with:



flask run --host 0.0.0.0

With the new --host option, our application is running on all network interfaces, including the interface used by containers. From a container, you can connect to the host with the special IP address 172.17.0.1 (the bridge address for the 172.17.0.0 network, linked to the host). So let's create a prometheus.yml file that contains the Prometheus configuration:



scrape_configs:
  - job_name: view_buy
    metrics_path: /metrics
    static_configs:
      - targets:
        - 172.17.0.1:5000

line 2: define the name of the Prometheus exporter; here we use the name of the application
line 5: IP address and port of the application

The prometheus.yml file must be in the same folder as docker-compose.yaml.

We edit the docker-compose.yaml as follow to inject the Prometheus configuration in the container:



version: '3'
services:
  prometheus:
    image: prom/prometheus:v2.26.0
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
    ports:
      - "9090:9090"

Let’s restart the Docker composition with:



docker-compose up -d

You should now be able to see a target that points to the application running on the host on the target list page.

This target has a static label with the name of the application set in the prometheus.yml file: job="view_buy".

We can now display all metrics from this target on the Graph page. In order to query the time series for our label, you just need to surround the label and value with curly braces: {job="view_buy"}.

It's time to generate some traffic!

Open 3 terminals and run the following commands:

watch -n 1 "curl localhost:5000/view/product1"
watch -n 2 "curl localhost:5000/view/product2"
watch -n 3 "curl localhost:5000/buy/product1"

On the Graph page of the Prometheus Web UI, you can query the view_total metric using the following PromQL expression: view_total. A visual representation of the metrics is available in the Graph tab, just under the "Execute" button.

Start Grafana

Add a new service to the Docker composition by editing the docker-compose.yml file:



version: '3'
services:
  prometheus:
    image: prom/prometheus:v2.26.0
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
    ports:
      - "9090:9090"
  grafana:
    image: grafana/grafana:7.5.2
    volumes:
      - ./datasource.yaml:/etc/grafana/provisioning/datasources/datasource.yaml
    ports:
      - "3000:3000"

We need to create the datasource.yaml file with the following content:



apiVersion: 1

datasources:
  - name: Prometheus
    type: prometheus
    access: proxy
    isDefault: true
    url: http://prometheus:9090/

This file configures the connection between Grafana and Prometheus. The hostname of the Prometheus container is prometheus because this is the name of the service in the Docker composition. Restart the Docker composition with:



docker-compose up -d

You can now access the Grafana Web UI using the default login/password admin/admin.

Skip the password change for now:

In the Explore view, you can use the following expression to visualize the number of views for each product: view_total.

The view_total metric is a Counter, a metric that always increases. As suggested by Grafana, it is better to display a temporal derivative of the value, which can be achieved using the rate() function.

Now we have more meaningful information about the number of views of products. The unit is view per second. For example, "product2" has 0.5 view per second, which makes sense as it corresponds to one view each 2 seconds.

Finally, in the legend, you can find the labels of each time series, one per product.

Let's create a dashboard now.

Creating a Grafana Dashboard

In the Grafana web UI, you can create a new dashboard. Choose "Add an empty panel". Then type the following expression: rate(view_total[5m]) and validate with 'Shift+Enter'.

To improve the legend, you can define the "Legend Format" and extract label values with the expression {{<label name>}}. For our application, we will use {{product}}.

The legend looks better as we now only have the name of the product. It's also possible to add a prefix and a suffix, using e.g.: Product {{product}} views.

Now you can click on the "Apply" button to save modifications of the dashboard. You can then add a new panel for purchases. Finally, click on the save 💾 button.

Persisting Metrics and Dashboards

If you stop or restart the current Docker composition, you will lose all the metrics retrieved by Prometheus and the Dashboard you just built. To persist data, with this local setup, we will use Docker volumes.

We will first create a Docker volume to persist Prometheus metrics. The time series database is stored in /prometheus/ inside the container so we will mount the volume on this path:



version: '3'
services:
  prometheus:
    image: prom/prometheus:v2.26.0
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
      - prometheus:/prometheus
    ports:
      - "9090:9090"
  grafana:
    image: grafana/grafana:7.5.2
    volumes:
      - ./datasource.yaml:/etc/grafana/provisioning/datasources/datasource.yaml
    ports:
      - "3000:3000"
volumes:
  prometheus:

line 7: we add a volume to persist the content of /prometheus folder
last 2 lines: a top level volumes key is used to define volumes used in the Docker composition

Finally, you can apply changes with docker-compose.yml. Metrics are now persisted across restarts of the Docker composition. Note that mounting the volume resets all metrics previously retrieved.

In order to persist the Grafana dashboard, we will create a file that contains the definition of the dashboard:

Switch back to the Grafana web ui,
Open your dashboard
Go to settings ⚙ and then "JSON Model"
Copy the JSON definition of the dashboard
Paste the copied content into a new dashboard.json file

The last step is to mount this dashboard in the Grafana container.

Create a dashboards.yaml file with the following contents:



apiVersion: 1

providers:
  - name: 'provisionned dashboards'
    orgId: 1
    folder: ''
    folderUid: ''
    type: file
    disableDeletion: false
    editable: true
    updateIntervalSeconds: 10
    allowUiUpdates: false
    options:
      path: /var/lib/grafana/dashboards

This file tells Grafana to load every dashboard defined in the /var/lib/grafana/dashboards. Finally, modify the Docker composition to include dashboards.yaml and dashboard.json in the Grafana container:



version: '3'
services:
  prometheus:
    image: prom/prometheus:v2.26.0
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
      - prometheus:/prometheus
    ports:
      - "9090:9090"
  grafana:
    image: grafana/grafana:7.5.2
    volumes:
      - grafana:/var/lib/grafana
      - ./datasource.yaml:/etc/grafana/provisioning/datasources/datasource.yaml
      - ./dashboards.yaml:/etc/grafana/provisioning/dashboards/dashboards.yaml
      - ./dashboard.json:/var/lib/grafana/dashboards/dashboard.json
    ports:
      - "3000:3000"

volumes:
    prometheus:
    grafana:

Update the Docker composition with docker-compose up -d. Your dashboard should be automatically loaded into Grafana.

Dashboard Modification Workflow

If you modify the dashboard, remember to copy the new JSON Model to the dashboard.json file and restart Grafana with docker-compose restart grafana. Now, your modifications are persisted as code in the dashboard.json and you can commit the dashboard definition within your project repository.

If someone wants to try your project, they just need to run the application and the monitoring stack with docker-compose up -d. They can then access your dashboard, see the evolution of metrics and create or modify graphs.

In this post, we were able to start a monitoring stack to retrieve metrics from our test application and build a dashboard to visualize the evolution of product views and purchases. We created several files that can be committed in Git and used later by other developers.

Conclusion

You are now ready to launch a monitoring stack aside your application, whichever language you’re using. You can also integrate the application as a new service in the Docker composition to ease test and demo.
This Docker composition should help you get your application monitoring code ready for production, so you can now deploy it on a Kubernetes cluster, and avoid the usual pitfalls of a new Kubernetes deployment.

In the next post, we will deploy the application and its monitoring in a Kubernetes cluster and see how to:

integrate the application monitoring with the Prometheus Operator
create alerts based on application metrics
package everything in a Helm Chart

Implement Prometheus Metrics in a Flask Application

Julien Acroute — Thu, 01 Apr 2021 09:41:54 +0000

In the previous post, we introduced the principles of Prometheus metrics series, labels, and saw which kind of metrics were useful to observe applications.

In this post, we will implement some metrics in a Flask application with 2 endpoints:

/view/<product>: Display the product information. This page simply displays the product name.
/buy/<product>: Purchase the product. Actually, this endpoint just displays a message.

We will start with the following implementation for our Flask application in a app.py file:

from flask import Flask

app = Flask(__name__)

@app.route('/view/<id>')
def view_product(id):
    return "View %s" % id

@app.route('/buy/<id>')
def buy_product(id):
    return "Buy %s" % id

First install Flask with pip install Flask or pip3 install Flask and then run application with flask run.

The purpose is to generate the following metrics to monitor product views and purchases:

# HELP view_total Product view
# TYPE view_total counter
view_total{product="product1"} 8
view_total{product="product2"} 2
# HELP buy_total Product buy
# TYPE buy_total counter
buy_total{product="product1"} 3

How to Generate Metrics

In order to generate metrics, we need HTTP endpoints that output text-based metrics. The default path is /metrics, but it can be modified. This can be implemented with a standard "route":

1 @app.route('/metrics')
2 def metrics():
3     metrics = ""
4     for id in view_metric:
5         metrics += 'view_total{product="%s"} %s\n'
6                        % (id, view_metric[id])
7     for id in buy_metric:
8         metrics += 'buy_total{product="%s"} %s\n'
9                        % (id, buy_metric[id])
10    return metrics

In this example, the view_metric and buy_metric variables contain a mapping between the product name and the count of views or purchases.

line 1: We create a new HTTP endpoint with the path /metrics; this endpoint will be used by Prometheus.
line 3: We initialize the result as an empty string
lines 4 to 6: For each product, we generate a line with:
- metric name: view
- label: product=<product name>
- value: the count of views; this value is fetched from view_metric
lines 7 to 9: Same for product purchases

Maintain the Metrics Variable up-to-date

In this approach, view_metric and buy_metric need to be updated elsewhere in the code when the product is viewed or bought. So the first thing to implement is a variable or object that holds metric values within application:

1 view_metric = {}
2
3 @app.route('/view/<id>')
4 def view_product(id):
5     # Update metric
6     if product not in view_metric:
7         view_metric[id] = 1
8     else:
9         view_metric[id] += 1
10   return "View %s" % id

line 1: a global variable that holds the total number of views for each product
lines 7 and 9: the global variable is updated

The code for the metrics() function is really simple, it does not compute anything, but simply retrieves values from an existing variable that is kept up-to-date. So the complexity is in the view_product() function. Each time a product is "viewed", a piece of code is run to maintain the view_metric counter up-to-date.

On-demand Metrics

Sometimes, the cost of maintaining these kinds of variables is higher than computing the values on-the-fly when the metrics() function is called. With the default configuration, Prometheus queries the /metrics endpoint once every 30 seconds. So if the variable needs to be updated several times during this interval, it might be a good idea to compute metrics on demand:

1 @app.route('/metrics')
2 def metrics():
3     metrics = ""
4     res = query('SELECT product, count(*) FROM product_view'\
5                 'GROUP BY product')
6     for line in res:
7         metrics += 'view_total{product="%s"} %s\n'
8                     % (line[0], line[1])
9     return metrics

With this approach, we don't need to modify other parts of the code. Instead, we query the database to retrieve this information.
Note that, in this example, the metrics concerning the purchases are removed to improve readability.

Using the Prometheus Client Library

There is probably a library for your language that will take care of producing Prometheus text format.

This will provide Counter and Gauge objects to implements your metrics:

1  from prometheus_client import Counter
2
3  view_metric = Counter('view', 'Product view')
4
5  @app.route('/view/<id>')
6  def view_product(id):
7      # Update metric
8      view_metric.inc()
9      return "View %s" % id

line 1: Loading the Prometheus client library
line 3: Creation of Counter metrics with name and description
line 8: Here the code is far more simple as we only need to call the inc() method of Counter object

Note that a _total suffix will be added to the metric name because it's a Counter and another metric with a _created suffix will contain the timestamp of the creation of the counter.

Adding Labels

So far, we haven’t handled labels. Let's see how we can add them now:

1 from prometheus_client import Counter
2 
3 view_metric = Counter('view', 'Product view', ['product'])
4
5 @app.route('/view/<id>')
6 def view_product(id):
7     # Update metric
8     view_metric.labels(product=id).inc()
9     return "View %s" % id

line 3: an additional parameter defines the allowed labels for the view metric
line 8: a call to labels() allows to set label values and thus select the time series that will be incremented

Finally, in the metrics() function, we just need to retrieve all the metrics in the Prometheus text format using the generate_latest() function:

1 from prometheus_client import generate_latest
2
3 @app.route('/metrics')
4 def metrics():
5     return generate_latest()

Here is a full example:

from flask import Flask
from prometheus_client import Counter, generate_latest

app = Flask(__name__)
view_metric = Counter('view', 'Product view', ['product'])
buy_metric = Counter('buy', 'Product buy', ['product'])

@app.route('/view/<id>')
def view_product(id):
    view_metric.labels(product=id).inc()
    return "View %s" % id

@app.route('/buy/<id>')
def buy_product(id):
    buy_metric.labels(product=id).inc()
    return "Buy %s" % id

@app.route('/metrics')
def metrics():
    return generate_latest()

Using Python Decorator

The python library also has some nice decorators.

For example, you can track the time spent in a function by using the @<metric>.time() decorator:

import time
import random
from prometheus_client import Summary

duration = Summary('duration_compute_seconds', 'Time spent in the compute() function')

@duration.time()
def compute():
    time.sleep(random.uniform(0, 10))

With a Counter, you can keep track of exceptions thrown in particular functions:

import time
import random
from prometheus_client import Counter

exception = Counter('compute_exception', 'Exception thrown in compute() function')

@exception.count_exceptions()
def compute():
    if random.uniform(0, 10) > 7:
        raise Exception("Random error")

On-demand Metrics with the Prometheus Library

The previously seen on-demand pattern can be implemented with the Prometheus Library by setting a callback function when defining the metric:

stock_metric = Counter('stock', 'Stock count')
stock_metric.set_function(compute_stock)
def compute_stock():
    res = query('SELECT count(*) FROM product_stock')
    for line in res:
        return line[0]

You can of course mix metrics managed with inc() and set() with other metrics using a callback function.

In the next blog post, we will see:

how to visualize the application metrics using a Docker composition that includes Prometheus and Grafana
how to leverage the new metrics in order to monitor the application in a Kubernetes cluster.

Intro to Prometheus for Developers

Julien Acroute — Fri, 26 Mar 2021 14:28:37 +0000

There are many tutorials to install Prometheus in a Kubernetes cluster. Some of them explain how to install an exporter for mainstream apps. See for example the following video:

Step by Step tutorial to monitor third-party apps using Prometheus (MongoDB example 🍃)

TechWorld with Nana ・ Sep 25 '20

#prometheus #devops #kubernetes #tutorial

But what about monitoring your own application? You may already have metrics for:

Cluster nodes with Node Exporter
Workload in the cluster: Pod metrics, aggregation by namespace
Third party apps: Load balancer, Database, Proxy

The core component, the application itself, should also be monitored. Some metrics can be retrieved from the middleware by using a dedicated exporter:

HTTP requests: total requests, duration, sessions
Memory usage: Garbage collector, cache

We still only have technical metrics about compute resources and their usage. This kind of metrics can be retrieved from any web project, as there always are HTTP requests, DB connection pool, memory, and CPU usage.

The most important indicators in your application are the ones that differ from other apps. For example, a web store app might have the following elements:

cart: total active cart count, total items in active carts
item: number of items available, number of items with no stock
customer: number of accounts, number of accounts with a connection in the last 3 months

You can build metrics within your app to be able to see the evolution of the cart, item, or customer accounts. Each application is unique, so only the people using or contributing to the application know which metric can be created to improve the level of observability of the application.

This series of blog posts will focus on adding high level metrics to existing applications. We will:

Modify an application to generate metrics in the Prometheus format
Connect Prometheus to the application
Create Grafana dashboard and configure alerts
Package monitoring stuff with the whole application using a Helm chart

External Exporter or Integrated in Application?

It's always better to include code that will generate metrics within the application. But sometimes you either:

cannot modify the application code
or the application already exposes some metrics, though not in the Prometheus format.

In this case, you may have to create a separate application that will compute metrics based on information retrieved from the main application, and act as an adapter.

What needs to be generated?

You need to generate one or more metrics. For each metric, the exporter needs to output:

a description
the name of the metrics
an optional list of labels
a decimal value

Metric Labels

Labels are used to create multiple metrics with the same name which refer to different elements. Labels add dimensions to metrics. With labels, you can have multiple values for the same timestamp.

For example, a metric regarding disk usage can look like this:

# HELP disk_usage_percent Usage of disk in percent (0-100)
# TYPE disk_usage_percent gauge
disk_usage_percent 63.7

If there is more than one partition, you can use labels to identify partitions, but keep the same metrics name:

# HELP disk_usage_percent Usage of disk in percent (0-100)
# TYPE disk_usage_percent gauge
disk_usage_percent{partition="/"} 63.7
disk_usage_percent{partition="/var"} 37.2
disk_usage_percent{partition="/tmp"} 12.5

For HTTP request metrics, you can use labels for:

the HTTP method: method="POST"
the path: path="/assets/script.js"
the browser type: browser="Firefox"

For a store, you can build a sales-related metric and use labels for:

the item category: category="Tools", category="Garden and Outdoor"
the brand: brand="Makita", brand="Weber"

You can also create a metric for the stock and use the same labels. This way, when displaying metrics, you will be able to correlate sales with stock of a specific category using labels.

Labels can be viewed as additional dimensions for a metric.

Every time series is uniquely identified by its metric name and optional
key-value pairs called labels.

Prometheus Text Format

Let's talk about the Prometheus format. It's a text based format:

# HELP http_requests_total The total number of HTTP requests.
# TYPE http_requests_total counter
http_requests_total{method="get",code="200"} 1234027
http_requests_total{method="post",code="200"} 1027
http_requests_total{method="post",code="400"} 3

Lines starting with # define metric metadata:

# HELP <metric name> <description>: Human readable description
# TYPE <metric name> [counter|gauge|histogram|summary]: Metric type

Most metrics use counter or gauge:

counter metrics are always incremental, cumulative, e.g.: error count, total requests
gauge metrics can increase and decrease, e.g.: connected customers, disk usage

The curly brackets contain a comma-separated collection of labels:

method="get"
code="200"

Finally, separated by a space, is the value.

So for our webstore, we can generate the following metrics:

# HELP disk_usage_percent Usage of disk in percent (0-100)
# TYPE disk_usage_percent gauge
disk_usage_percent{partition="/"} 63.4
disk_usage_percent{partition="/var"} 37.6
disk_usage_percent{partition="/tmp"} 12.3

# HELP http_requests_total The total number of HTTP requests.
# TYPE http_requests_total counter
http_requests_total{method="GET", code="200"} 1234027
http_requests_total{method="POST", code="200"} 1027
http_requests_total{method="POST", code="400"} 3

# HELP sales_total The total number of sales.
# TYPE sales_total counter
sales_total{category="Tools", brand="Makita"} 163376
sales_total{category="Tools", brand="Bosh"} 298425
sales_total{category="Garden and Outdoor", brand="Weber"} 18346
sales_total{category="Garden and Outdoor", brand="Hyundai"} 163376

# HELP stock The number of stock items.
# TYPE stock gauge
stock{category="Tools", brand="Makita"} 234
stock{category="Tools", brand="Bosh"} 456
stock{category="Garden and Outdoor", brand="Weber"} 27
stock{category="Garden and Outdoor", brand="Hyundai"} 45

Note that in this example, we mix metrics from different layers:

System metrics: disk usage
Middleware metrics: HTTP requests
Application metrics: stocks and sales

For system metrics, it is recommended to use a dedicated exporter like
Node Exporter. For middleware, you should be able to find an exporter for your needs: Django, RabbitMQ, Redis, Java JMX, PostgreSQL.

In the next blog post, we will start adding metrics to an existing application.