DEV Community

Cover image for Taming cAdvisor's high CPU usage
Pablo Romeo for Cloud(x);

Posted on

Taming cAdvisor's high CPU usage

When running multiple docker containers, be it on a single server, a Kubernetes cluster or in Docker Swarm, it is very important to be able to monitor their resource usage.

That's where cAdvisor comes in. It provides useful metrics to build Monitoring Dashboards using Grafana and Prometheus:
Grafana Dashboard showing CPU and Memory metrics

But there is one big problem: cAdvisor's default settings.

The consequence of running the default configuration is very high CPU usage. Especially noticeable in low-powered devices such as the Raspberry Pi's from the cluster in the graph above.

cAdvisor uses more CPU than the containers it is monitoring! 😭

The fix

Thankfully, it's quite simple. The parameters that make the biggest impact are:

  • housekeeping_interval
  • docker_only
  • disable_metrics

A detailed explanation of each can be found here.

Docker-compose Swarm example

Below I'm defining a housekeeping_interval of 30 seconds, setting docker_only to true, and disabling metrics I'm not interested in, using disable_metrics.

NOTE: Beware that I'm using an ARM version of cAdvisor, change the image/tag accordingly.

version: '3.4'

    image: zcube/cadvisor:latest
      - published: 9102
        target: 9102
        mode: host
      - "--port=9102"
      - "--housekeeping_interval=30s"
      - "--docker_only=true"
      - "--disable_metrics=percpu,sched,tcp,udp,disk,diskIO,accelerator,hugetlb,referenced_memory,cpu_topology,resctrl"
      - /var/lib/docker/:/var/lib/docker:ro
      - /dev/disk/:/dev/disk:ro
      - /sys:/sys:ro
      - /var/run:/var/run:ro
      - /:/rootfs:ro
      - /sys/fs/cgroup:/cgroup:ro
      - /etc/machine-id:/etc/machine-id:ro
      - /etc/localtime:/etc/localtime:ro
      mode: global
        order: stop-first
          memory: 80M
      test: wget --quiet --tries=1 --spider http://localhost:9102/healthz || exit 1
      interval: 15s
      timeout: 15s
      retries: 5
      start_period: 30s
Enter fullscreen mode Exit fullscreen mode

Did it help?

Check this out:
Lower CPU usage after changing settings

I'm sure you can guess at what time the new settings kicked in 😏. From 0.68 vCPU to 0.08 vCPU!
Not bad at all.

Top comments (0)