Taming cAdvisor's high CPU usage

When running multiple docker containers, be it on a single server, a Kubernetes cluster or in Docker Swarm, it is very important to be able to monitor their resource usage.

That's where cAdvisor comes in. It provides useful metrics to build Monitoring Dashboards using Grafana and Prometheus:

But there is one big problem: cAdvisor's default settings.

The consequence of running the default configuration is very high CPU usage. Especially noticeable in low-powered devices such as the Raspberry Pi's from the cluster in the graph above.

cAdvisor uses more CPU than the containers it is monitoring! 😭

The fix

Thankfully, it's quite simple. The parameters that make the biggest impact are:

• housekeeping_interval
• docker_only
• disable_metrics

A detailed explanation of each can be found here.

Docker-compose Swarm example

Below I'm defining a housekeeping_interval of 30 seconds, setting docker_only to true, and disabling metrics I'm not interested in, using disable_metrics.

NOTE: Beware that I'm using an ARM version of cAdvisor, change the image/tag accordingly.

version: '3.4'

services:
  cadvisor:
    image: zcube/cadvisor:latest
    ports:
      - published: 9102
        target: 9102
        mode: host
    command:
      - "--port=9102"
      - "--housekeeping_interval=30s"
      - "--docker_only=true"
      - "--disable_metrics=percpu,sched,tcp,udp,disk,diskIO,accelerator,hugetlb,referenced_memory,cpu_topology,resctrl"
    volumes:
      - /var/lib/docker/:/var/lib/docker:ro
      - /dev/disk/:/dev/disk:ro
      - /sys:/sys:ro
      - /var/run:/var/run:ro
      - /:/rootfs:ro
      - /sys/fs/cgroup:/cgroup:ro
      - /etc/machine-id:/etc/machine-id:ro
      - /etc/localtime:/etc/localtime:ro
    deploy:
      mode: global
      update_config:
        order: stop-first
      resources:
        reservations:
          memory: 80M
    healthcheck:
      test: wget --quiet --tries=1 --spider http://localhost:9102/healthz || exit 1
      interval: 15s
      timeout: 15s
      retries: 5
      start_period: 30s

Did it help?

Check this out:

I'm sure you can guess at what time the new settings kicked in 😏. From 0.68 vCPU to 0.08 vCPU!
Not bad at all.