DEV Community

Arseny Zinchenko
Arseny Zinchenko

Posted on • Originally published at rtfm.co.ua on

Kubernetes: monitoring processes with process-exporter

#devops #kubernetes #monitoring

We are debugging one issue with memory usage in Kubernetes Pods, and decided to look at the memory and number of processes on the nodes.

The problem is that a Kubernetes Pod with Livekit usually consumes about 2 gigabytes of memory, but sometimes there are spikes of up to 10–11 gigabytes, which causes the Pod to crash:

What we want to determine is whether it is one process that starts to “eat” so much memory, or whether many processes are simply being created in the container?

The simplest option here is to use Prometheus Process Exporter, which runs as a DaemonSet, creates its own container on each WorkerNode, and collects statistics from /proc for all or selected processes on EC2.

There is a good (and working) Helm chart kir4h/process-exporter, let’s take it.

Contents

  • Starting Process Exporter
  • Creating Name Groups
  • Name Group Template variables and information about each process
  • The result of our debate

Starting Process Exporter

Add the repository, install:

$ helm repo add kir4h https://kir4h.github.io/charts
$ helm install my-process-exporter kir4h/process-exporter
Enter fullscreen mode Exit fullscreen mode

Or, in our case, we install it via Helm dependency and add the chart to Chart.yaml of our monitoring stack chart:

...
- name: process-exporter
  version: ~1.0
  repository: https://kir4h.github.io/charts
  condition: process-exporter.enabled
Enter fullscreen mode Exit fullscreen mode

Add values for it:

...
process-exporter:
  enabled: true
  tolerations:
  - effect: NoSchedule
    operator: Exists
  - key: CriticalAddonsOnly
    operator: Exists
    effect: NoSchedule
  - key: CriticalAddonsOnly
Enter fullscreen mode Exit fullscreen mode

Deploy and check DaemonSet:

$ kk get ds
NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
atlas-victoriametrics-process-exporter 9 9 9 9 9 <none> 76m
...
Enter fullscreen mode Exit fullscreen mode

And check ServiceMonitor:

$ kk get serviceMonitor | grep process
atlas-victoriametrics-process-exporter 3d3h
Enter fullscreen mode Exit fullscreen mode

For VictoriaMetrics, VMServiceScrape is automatically created:

$ kk get VMServiceScrape | grep process
atlas-victoriametrics-process-exporter 3d3h operational
Enter fullscreen mode Exit fullscreen mode

We check whether there are metrics, for example, for namedprocess_namegroup_memory_bytes:

Creating Name Groups

We now have data on all processes — we don’t need it.

Specifically, in our case, we are interested in statistics on our Backend API processes, Python processes.

We have three main ones: the Backend API itself, Celery Workers, and Livekit itself, and each service runs in its own Pods from separate Deployments.

Find processes in the submissions and see how they are launched.

Backend API:

root@backend-api-deployment-5695989cb5-rjhv9:/app# ps aux
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 1 0.0 0.2 40348 34712 ? Ss 07:59 0:02 /usr/local/bin/python /usr/local/bin/gunicorn challenge_backend.run_api:app [...]
root 7 1.2 2.5 2075368 414564 ? Sl 07:59 1:32 /usr/local/bin/python /usr/local/bin/gunicorn challenge_backend.run_api:app [...]
root 8 1.1 2.6 1999384 422228 ? Sl 07:59 1:23 /usr/local/bin/python /usr/local/bin/gunicorn challenge_backend.run_api:app [...]
root 9 1.2 2.6 2002492 429192 ? Sl 07:59 1:30 /usr/local/bin/python /usr/local/bin/gunicorn challenge_backend.run_api:app [...]
...
Enter fullscreen mode Exit fullscreen mode

Celery workers:

root@backend-celery-workers-deployment-5bc64557c8-zbq2j:/app# ps aux
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 1 0.2 1.4 544832 236720 ? Ss 07:27 0:24 /usr/local/bin/python /usr/local/bin/celery -A celery_app.app worker [...]
...
Enter fullscreen mode Exit fullscreen mode

Ta Livekit:

root@backend-livekit-agent-deployment-7d9bf86564-qgjzb:/app# ps aux
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 1 0.4 1.8 2112944 294772 ? Ssl 07:06 0:46 python -m cortex.livekit_agent.main start
root 24 0.0 0.0 15788 12860 ? S 07:06 0:00 /usr/local/bin/python -c from multiprocessing.resource_tracker import main;main (34)
root 25 0.0 0.6 342976 102852 ? S 07:06 0:02 /usr/local/bin/python -c from multiprocessing.forkserver import main [...]
...
Enter fullscreen mode Exit fullscreen mode

Add the configuration for process-exporter — describe nameMatchers:

...
process-exporter:
  enabled: true
  tolerations:
    operator: Exists
    effect: NoSchedule
  - key: CriticalAddonsOnly
  config:
    # metrics will be broken down by thread name as well as group name
    threads: true
    # any process that otherwise isn't part of its own group becomes part of the first group found (if any) when walking the process tree upwards
    children: true
    # means that on each scrape the process names are re-evaluated
    recheck: false
    # remove_empty_groups drop empty groups if no processes found
    remove_empty_groups: true
    nameMatchers: 
      # gunicorn (python + uvicorn workers)
      - name: "gunicorn"
        exe:
          - /usr/local/bin/python
        cmdline:
          - ".*gunicorn.*"

      # celery worker
      - name: "celery-worker"
        exe:
          - /usr/local/bin/python
        cmdline:
          - ".*celery.*worker.*"

      # livekit agent
      - name: "livekit-agent"
        exe:
          - python
          - /usr/local/bin/python
        cmdline:
          - ".*cortex.livekit_agent.main.*"

      # livekit multiprocessing helpers
      - name: "livekit-multiproc"
        exe:
          - /usr/local/bin/python
        cmdline:
          - ".*multiprocessing.*"
Enter fullscreen mode Exit fullscreen mode

Here, the exe is a list of executables (there can be several), and in cmdline are the arguments with which the process is launched.

That is, for Livekit, we have exe  —  “/usr/local/bin/python", and cmdline - "-c from multiprocessing.resource_tracker [...] " or "-c from multiprocessing.forkserver [...]".

Let’s deploy, and now there are only three groups left:

But there are nuances.

First, statistics are collected from each node across the entire group of processes.

That is, if we do the following:

sum(namedprocess_namegroup_memory_bytes{memtype="resident", groupname="celery-worker"}) by (groupname, instance, pod)
Enter fullscreen mode Exit fullscreen mode

This will give us the sum of all RSSs of all Celery workers on the node where the corresponding process-exporter Pod is running:

The second problem is that Process Exporter does not have a label named WorkerNode from which metrics are collected.

Therefore, we can only search manually here — by Pod IP (label instance) we can find its Node:

$ kk get pod -o wide | grep 10.0.45.166
atlas-victoriametrics-process-exporter-4zdzl 1/1 Running 0 6m51s 10.0.45.166 ip-10-0-40-195.ec2.internal <none> <none>
Enter fullscreen mode Exit fullscreen mode

And then see what kind of holes there are on this node:

$ kk describe node ip-10-0-40-195.ec2.internal | grep celery
  dev-backend-api-ns backend-celery-workers-deployment-5bc64557c8-hqhz4 200m (5%) 0 (0%) 1500Mi (10%) 0 (0%) 3h28m
  dev-backend-api-ns backend-celery-workers-long-running-deployment-57d7cb9984-nlfs4 200m (5%) 0 (0%) 1500Mi (10%) 0 (0%) 3h12m
  prod-backend-api-ns backend-celery-workers-deployment-5597dfd875-m7c2n 500m (12%) 0 (0%) 1500Mi (10%) 0 (0%) 99m
  staging-backend-api-ns backend-celery-workers-long-running-deployment-5bb44795b7-pcmj2 200m (5%) 0 (0%) 1500Mi (10%) 0 (0%) 103m
Enter fullscreen mode Exit fullscreen mode

And now let’s take a look at the processes and their RSS:

[root@ip-10-0-40-195 ec2-user]# ps -eo rss,cmd | grep celery
232888 /usr/local/bin/python /usr/local/bin/celery -A celery_app.app worker --loglevel=info -Q default
241656 /usr/local/bin/python /usr/local/bin/celery -A celery_app.app worker --loglevel=info -Q default
...
239232 /usr/local/bin/python /usr/local/bin/celery -A celery_app.app worker --loglevel=info -Q default
252240 /usr/local/bin/python /usr/local/bin/celery -A celery_app.app worker --loglevel=info -Q default
 2416 grep --color=auto celery
Enter fullscreen mode Exit fullscreen mode

On the graph, we have 4,604,280,832 bytes here:

Let’s calculate it ourselves:

[root@ip-10-0-40-195 ec2-user]# ps -eo rss,cmd | grep celery | grep -v grep | awk '{sum += $1} END {print sum*1024 " bytes"}'
4608430080 bytes
Enter fullscreen mode Exit fullscreen mode

Returning to the issue of there being no information on each process: we can obtain an average value for each one, because we have the metric namedprocess_namegroup_num_procs:

Let’s check again on the node itself:

[root@ip-10-0-40-195 ec2-user]# ps aux | grep celery | grep -v grep | wc -l
20
Enter fullscreen mode Exit fullscreen mode

And we can make such a request:

sum(namedprocess_namegroup_memory_bytes{memtype="resident", groupname="celery-worker", instance="10.0.45.166:9256"}) by (groupname, instance, pod)
/
sum(namedprocess_namegroup_num_procs{groupname="celery-worker", instance="10.0.45.166:9256"}) by (groupname, instance, pod)
Enter fullscreen mode Exit fullscreen mode

Result ~230 MB:

As we saw in the ps -eo rss,cmd output.

Name Group Template variables and information about each process

Or, if we really want to see statistics for each process, we can use dynamic names for groupname with {{.PID}} - then a separate group will be formed for each process, see Using a config file: group name:

...
    nameMatchers: 
      # gunicorn (python + uvicorn workers)
      - name: "gunicorn-{{.Comm}}-{{.PID}}"
        exe:
          - python
          - /usr/bin/python
          - /usr/local/bin/python
        cmdline:
          - ".*gunicorn.*"

      # celery worker
      - name: "celery-worker-{{.Comm}}-{{.PID}}"
        exe:
          - python
          - /usr/bin/python
          - /usr/local/bin/python
        cmdline:
          - ".*celery.*worker.*"

      # livekit agent
      - name: "livekit-agent-{{.Comm}}-{{.PID}}"
        exe:
          - python
          - /usr/bin/python
          - /usr/local/bin/python
        cmdline:
          - ".*livekit_agent.*"

      # livekit multiprocessing helpers
      - name: "livekit-multiproc-{{.Comm}}-{{.PID}}"
        exe:
          - python
          - /usr/bin/python
          - /usr/local/bin/python
        cmdline:
          - ".*multiprocessing.*"
Enter fullscreen mode Exit fullscreen mode

As a result, we have the following groups:

But this option is only OK if you need to debug something and disable it, because it will lead to a High cardinality issue.

The result of our debate

Actually, what we needed to find out was whether memory was “leaking” in a single process, or whether multiple processes were simply being created in a single Pod.

To do this, Grafana created a graph with the following query:

sum(namedprocess_namegroup_memory_bytes{memtype="resident", groupname=~"livekit-multiproc-.*"}) by (groupname, instance)
Enter fullscreen mode Exit fullscreen mode

We added graphs with Livekit metrics  —  lk_agents_active_job_count and lk_agents_child_process_count, and separately - a graph from VictoriaLogs, where we display the number of API requests for each user by the field token_email:

namespace: "prod-backend-api-ns" "GET /cortex/livekit-token" | unpack_json fields (token_email) | stats by (token_email) count()
Enter fullscreen mode Exit fullscreen mode

And as a result, we have the following picture:

Here we see that the same user starts making a bunch of requests to connect to Livekit, which creates a bunch of processes in Livekit Pod (a new Livekit Job for each request), and as a result, the total amount of memory in the Pod goes through the roof, because 40 processes at ~380 MB each is ~15 gigabytes of memory.

However, in each specific process, memory is maintained at a level of 300–400 megabytes.

It remains to figure out why the processes are crashing, but that’s a task for the developers.

Originally published at RTFM: Linux, DevOps, and system administration.

Top comments (0)