DEV Community

John
John

Posted on

Netdata Alerts in Week One: Which to Keep, Tune or Silence, and How to Route Them to Slack, Discord and ntfy

#netdata #alerting #sre #devops

Keep Netdata's stock alerts for disk space, memory pressure, OOM kills and failed systemd units exactly as they ship. In the first week, raise the thresholds or lengthen the delays on the CPU, network error and TCP reset alerts, and silence the ones for hardware or services you don't run. Send CRITICAL alerts to ntfy so they reach a phone, and send WARNING alerts to a Slack or Discord channel that someone reads once a day. That leaves a small team with a handful of real alerts a week, not dozens.

TL;DR

  • Solo on-call founder with two VPS nodes, e.g. a SaaS API plus its database box: route CRITICAL to ntfy and WARNING to Discord, because one phone and one channel is all two people can realistically watch.
  • Team that already lives in Slack, e.g. a two-person agency using Slack with clients: use a private Slack channel for WARNING and ntfy for CRITICAL, because paging through the same app you chat in gets muted within days.
  • Docker-heavy single host, e.g. twelve containers on one mini PC: tune the per-container cgroup CPU and memory alerts first, because they produce most of the first-week noise on busy hosts.
  • Parent-child streaming setup, e.g. one parent collecting from four children: run health checks and notifications on the parent only, so one config decides what reaches you and each alert isn't sent several times.
  • Production database owner, e.g. a single PostgreSQL primary: keep the connection, disk and replication alerts at stock or tighter, because those failures cost data, not just uptime.
  • Staging or hobby node, e.g. a preview environment rebuilt weekly: set to: silent for most alerts on it and keep the dashboard, because alerts on a box you routinely destroy teach you to ignore alerts everywhere.

The central tradeoff is simple: every alert you silence is an outage you might hear about from a customer, and every alert you keep but ignore trains you to miss the one that matters.

Table of contents

Why do Netdata's stock alerts fire so often in the first week?

Netdata ships with hundreds of alert definitions. They are written to be useful on any Linux machine, so they aren't tuned to yours. A new install turns on almost all of them at once, before you've seen what normal looks like on your hosts. Most first-week noise has a few predictable causes.

  • Auto-discovery attaches alerts to everything it finds: every collector that detects a service, disk, interface or container brings its own alerts. You never explicitly asked for any of them, so it isn't obvious where a notification comes from.
  • Templates multiply across instances: one definition that watches network interfaces runs separately against eth0, docker0 and every veth pair Docker creates. A host with twenty containers can raise the same warning many times over for traffic that is perfectly normal.
  • Thresholds assume a generic server: the stock 10min_cpu_usage alert warns when average CPU over ten minutes crosses 85% and goes critical above 95%. A small VPS that runs a nightly build or backup crosses that line on schedule, every night.
  • Internet-facing noise looks like failure: the TCP reset and dropped-packet alerts react to port scanners and bots hitting a public IP. On a box exposed to the internet, that is background activity, not a sign of trouble.
  • Short-lived containers churn state: containers that start, stop and get rebuilt during deploys make their per-container alerts appear, change state and get removed. Each transition can produce a notification.

None of this means the defaults are wrong. It means the first week is when you find out which alerts match how your hosts actually behave, and that is exactly what the next few sections sort through.

How Netdata's health configuration is structured on disk

Before you change any alert, you need to know which files Netdata reads and which files a package upgrade overwrites. The layout differs slightly between install types. On a native package install, stock files live under /usr/lib/netdata/conf.d/ and your changes go under /etc/netdata/. A static install from the kickstart script puts both trees under /opt/netdata/. In the official Docker image, only /etc/netdata is worth mounting as a volume.

  • Stock alert definitions: /usr/lib/netdata/conf.d/health.d/ holds one .conf file per area, for example cpu.conf, ram.conf, disks.conf and tcp_resets.conf. Upgrades replace these files, so any edit you make there disappears.
  • Your overrides: /etc/netdata/health.d/ is where your versions go. A file here with the same name as a stock file replaces that stock file completely. Netdata does not merge the two line by line, so copy the whole file before you edit it.
  • The edit-config helper: run sudo ./edit-config health.d/cpu.conf from inside /etc/netdata. It copies the stock file into place and opens it, so you never start from a blank file by accident.
  • Alarm versus template: each definition starts with either alarm:, which is tied to one specific chart, or template:, which applies to every chart of a context such as every disk or every container. Most stock noise comes from templates, because a single definition runs against every instance.
  • Notification settings: /etc/netdata/health_alarm_notify.conf holds webhooks, recipients and roles, apart from the alert logic.
  • The global switch: the [health] section of netdata.conf turns the health engine on or off for the whole node.

After editing, run netdatacli reload-health to apply the change without restarting the agent.

Which stock alerts should you keep exactly as shipped?

Keep an alert as shipped when the condition it catches ends in data loss, a crash, or a service that stays down until someone steps in. A small team can't afford to learn about these from a customer, and the stock thresholds already leave room before real damage. You can check the exact names on your nodes in the dashboard's Alerts tab or with curl localhost:19999/api/v1/alarms?all.

  • disk_space_usage and disk_inode_usage: a full filesystem breaks databases, log writers and Docker image pulls all at once. Running out of inodes produces the same "no space left" errors while df -h still shows free space, which is exactly why a separate alert for it is worth having.
  • oom_kill: this fires when the kernel has already killed a process to free memory. It is not a prediction, so every notification means something on the host just died.
  • ram_available: this looks at memory the system can actually reclaim, not raw usage. That makes it a better early warning than ram_in_use on hosts where the page cache fills RAM by design.
  • systemd_service_unit_failed_state: a unit that has crashed and hit its restart limit won't recover on its own. On a host without containers, this is often the only sign a background worker has stopped.
  • docker_container_unhealthy: this goes off only for containers you gave a HEALTHCHECK, so it is quiet by default and precise when it does fire.
  • postgres_total_connection_utilization: running out of connections looks like an application outage even though the database itself is healthy.

If one of these turns out noisy, look at the host before you blame the threshold. On a disk that sits at 88% for a week, the fix is cleanup or more storage, not a quieter alert.

Which alerts are worth tuning rather than deleting?

Tune an alert when the condition it tracks does matter, but the stock threshold or time window doesn't fit how your hosts behave. Delete an alert only when the condition never matters to you. The difference is simple: a CPU pegged at 100% for two hours is a real problem, while a CPU at 90% for twelve minutes during a nightly build is not. The fix is to change when the alert fires, not whether it exists.

Alert Why it is noisy at stock settings What to change
10min_cpu_usage Scheduled builds, backups and image pulls push the average above the line every day Widen the lookup window to 30 minutes, or add delay: up 15m so short spikes never notify
cgroup_10min_cpu_usage Fires for each container, so one busy deploy produces several warnings Keep CRITICAL, and match only the containers that serve users with a chart labels filter
cgroup_ram_in_use Containers with tight memory limits sit near them by design, especially JVM and Node services Raise the WARNING level for those services, and keep oom_kill as the hard signal
10min_disk_backlog Backup and docker system prune runs queue I/O for minutes at a time Lengthen the up delay so it fires only when the backlog lasts longer than your longest backup
1m_ipv4_tcp_resets_sent Port scans against a public IP send resets all day Keep it as a low-priority notice, never a page, and raise the threshold once you know the baseline
inbound_packets_dropped_ratio Docker veth interfaces drop packets during container churn Limit it to physical interfaces with a chart labels match on the device name

Before you change anything, write down the value each alert showed when it fired during the week. Set the new threshold from those notes, not from a guess.

Which alerts can a small team silence outright?

Silence an alert when no value it could reach would make you change anything. There are two ways to do it, and the difference matters. Setting to: silent in the definition keeps the alert running, so its state still shows on the dashboard, but nothing gets sent. Disabling it by name in the [health] section of netdata.conf stops it from being evaluated at all. For a small team, to: silent is usually the better choice, because you can still look back at the history after an incident.

  • 1m_received_traffic_overflow and 1m_sent_traffic_overflow: these compare traffic to the interface's link speed. Virtual NICs on cloud VPS plans often report a speed that has nothing to do with your real bandwidth cap, so the percentage is meaningless.
  • load_average_1: one-minute load on a small VPS jumps with every cron job. 10min_cpu_usage, once tuned, already covers sustained saturation, so this adds noise without adding a signal.
  • system_clock_sync_state: on some virtual machines and inside certain container setups, the host or hypervisor manages time. The agent can report "unsynchronised" indefinitely while the clock is actually correct.
  • used_swap: if you set up swap or zram on purpose so a 2 GB box can absorb spikes, using swap is the plan, not a failure. oom_kill still catches the case where that plan runs out.
  • Web log redirect and bad-request ratios: on a public site, crawlers and vulnerability scanners generate 3xx and 4xx responses constantly. Your application's error tracking is a better place to catch real client errors.
  • Every alert on disposable staging nodes: match them with a host label and silence the lot, keeping only disk_space_usage, so a full disk doesn't block the next deploy.

Keep a short comment above each change explaining why you silenced it. Six months from now, that line is the only record anyone will have.

How do you change thresholds, delays and hysteresis without losing them on upgrade?

Copying a whole stock file into /etc/netdata/health.d/ works, but it freezes that file in time. When a later release fixes a stock definition in cpu.conf, your copy keeps the old version forever. A cleaner pattern is to leave stock files untouched, write your own definitions under new names, and turn off only the stock alerts you replaced.

  • Put overrides in one file: create something like /etc/netdata/health.d/local-overrides.conf and give each definition a new name, for example team_10min_cpu_usage. The name never matches a stock file, so upgrades can't shadow it or be shadowed by it.
  • Disable only the originals: in netdata.conf, set enabled alarms = !10min_cpu_usage !cgroup_ram_in_use * under [health]. The trailing * keeps every other stock alert active and upgradeable.
  • Widen the window with lookup: lookup: average -30m unaligned of user,system averages user and system CPU over 30 minutes, so a ten-minute spike can't push it over the line.
  • Add hysteresis inside warn: warn: $this > (($status >= $WARNING) ? (75) : (90)) raises at 90% but clears only below 75%. That stops an alert flapping around a single value.
  • Hold notifications with delay: delay: up 10m down 15m multiplier 1.5 max 1h waits ten minutes before telling you, waits fifteen before sending the all-clear, and stretches both delays when an alert keeps flipping, up to one hour.
  • Version the directory: keep /etc/netdata in a git repository. After each upgrade, diff the stock definitions you replaced against your own versions, so upstream fixes don't pass unnoticed.

Check a change by watching the renamed alert appear in api/v1/alarms?all before you trust it.

Anomaly detection versus fixed thresholds for a two-person rotation

Netdata trains small machine learning models for each metric, right on the agent, and marks every one-second sample as anomalous or normal. The result shows up as an anomaly rate. You can write an alert on it with a lookup that uses the anomaly-bit option, for example the average anomaly rate across a chart over the last 10 minutes. The question is whether that signal should ever wake someone up.

Aspect Fixed thresholds Anomaly rate alerts
What triggers it A value crosses a number you picked, such as disk above 90% A metric behaves unlike its own recent history
First-week behaviour Noisy until tuned, but the noise is predictable Unreliable while models are still training on only a few days of data
Catches well Hard limits: full disks, exhausted connections, OOM kills Unusual patterns with no obvious limit, such as a quiet API suddenly doing steady writes
Blind spot Anything you didn't think to set a threshold for Slow drift, because a disk filling 1% a day just becomes the new normal
Readable at 3 a.m. Yes: "disk at 94%" tells you what to do Rarely: "anomaly rate 40% on 12 charts" tells you to go and look
Resource cost Negligible Extra CPU on every node, which is why many parent-child setups turn off [ml] on children and let the parent train

For two people sharing on-call, page only on fixed thresholds. Send anomaly rate alerts, if you enable them, to the daily notice channel. Use them as a prompt to open the dashboard, not as proof something is broken. If an anomaly alert keeps pointing at a real problem, turn that into a fixed-threshold alert you can act on.

How do you send Netdata alerts to Slack, Discord and ntfy?

The agent sends notifications through alarm-notify.sh, and every setting it reads lives in health_alarm_notify.conf. Open it with sudo ./edit-config health_alarm_notify.conf from /etc/netdata. It is a long file, but each service needs only a switch, a destination and a default recipient.

  • Slack: create an incoming webhook for your workspace, then set SEND_SLACK="YES", paste the URL into SLACK_WEBHOOK_URL, and set DEFAULT_RECIPIENT_SLACK to a channel such as #ops-alerts. Each Slack webhook is tied to a single channel, so plan on one webhook for each channel you want to post to.
  • Discord: in the channel settings, create a webhook under Integrations. Set SEND_DISCORD="YES", put the URL in DISCORD_WEBHOOK_URL, and set DEFAULT_RECIPIENT_DISCORD to the channel name. Discord is often the free option for a small team that doesn't pay for Slack.
  • ntfy: set SEND_NTFY="YES" and put the full topic URL in DEFAULT_RECIPIENT_NTFY, for example https://ntfy.sh/ followed by your topic. On the public ntfy.sh server, anyone who knows a topic name can subscribe to it, so choose a long random name or run your own ntfy server and set NTFY_ACCESS_TOKEN.
  • Email as a fallback: SEND_EMAIL depends on a working sendmail on the host. Most small VPS setups don't have one configured, so turn it off rather than letting messages fail without anyone noticing.
  • Test before trusting: switch to the netdata user with sudo su -s /bin/bash netdata, then run /usr/libexec/netdata/plugins.d/alarm-notify.sh test. It sends a WARNING, a CRITICAL and a CLEAR to every destination you enabled.

If the test works from a terminal but real alerts never arrive, check that the netdata user can reach the internet through your firewall.

Splitting pages from notices with roles and severities

Every alert definition has a to: line naming a role, and most stock alerts use to: sysadmin. In health_alarm_notify.conf, each role maps to recipients for each service. Put those two ideas together and you get a two-tier setup: a short list of alerts that can wake a person up, and everything else collected for a daily look.

  • Role mappings: role_recipients_slack[sysadmin]="#ops-alerts" sends every sysadmin alert to that channel. Roles you don't map fall back to the DEFAULT_RECIPIENT_ value for each service, so a role you forgot to map still reaches someone.
  • The |critical modifier: add it to a recipient, as in role_recipients_ntfy[sysadmin]="https://ntfy.sh/<topic>|critical", and that destination receives only CRITICAL alerts and the CLEAR that follows. WARNING alerts still reach Slack or Discord, but they never buzz a phone.
  • A dedicated pager role: in your override file, set to: pager on the handful of alerts that justify waking someone, such as disk_space_usage, oom_kill and database connection exhaustion. Map pager to ntfy only, and a threshold edit elsewhere can never quietly add to the page list.
  • Keep a written record in chat: map pager to Slack or Discord as well as ntfy. The phone push gets attention, and the channel keeps a timestamped history both of you can scroll back through the next morning.
  • Per-person routing: with two people, give each their own ntfy topic and alternate which one sits in role_recipients_ntfy[pager] week by week. That is a simple on-call rotation without a paging SaaS.
  • Database and web roles: stock database alerts use roles like dba. Map them explicitly, or they fall through to your defaults and page you with the wrong priority.

Run alarm-notify.sh test "pager" after every mapping change.

Top comments (1)

Collapse
 
raknaos profile image
Raknaos •

The tune-versus-delete line is the part people skip. I have watched a box lose 10min_cpu_usage entirely because of a nightly job, then ship a genuine CPU runaway into silence two weeks later — widening the window or adding delay: up 15m keeps the alert for the case that actually matters.

Versioning the whole health directory is what makes tuning safe rather than brave, because the post-upgrade diff is the only moment you find out a stock definition you overrode has been fixed upstream. One thing I would like your read on: on the anomaly rate side, did you ever land on a threshold that survives a deploy window, or does the training period keep absorbing the spike and calling it normal?