Putting servers and routers in one dashboard makes life easier, but monitoring is only half finished if you still have to open the page every day to discover a problem. Alerts are the obvious next step: high CPU, a nearly full disk, a missing machine.
Writing a threshold rule is easy. Making the notification useful is harder. If every brief spike produces mail and every continuing fault repeats every few seconds, important messages become easy to ignore. Conversely, no email does not necessarily mean everything is healthy.
This fifth Fleetd article describes v0.3.0. After deploying the agent on Linux and OpenWrt, we now follow reports into alert records, notifications and resolution.
What does an alert actually mean?
Fleetd checks offline reporting, CPU, memory, disk space, inodes, failed services, abnormal containers and WireGuard handshake age. Connection-address changes are separate, one-off events.
These signals have different meanings. Disk pressure may soon block writes; high CPU may be a planned build; an old WireGuard handshake may reflect an idle peer. Crossing a threshold does not, by itself, prove a business outage.
A useful alert first identifies the machine, the object and the condition. Diagnosis and response depend on the machine's role. Fleetd provides concrete checks, not automatic root-cause analysis, and it does not restart services simply because CPU is high.
Separate spikes from sustained load
CPU and memory default to 90%, with a five-minute sustained window:
alerts:
offline: 40s
cpu_percent: 90
mem_percent: 90
sustain: 5m
disk_percent: 90
inode_percent: 90
wg_handshake: 5m
These are the v0.3.0 defaults, not universal recommendations. They are server-wide settings; there are no independent thresholds per machine in this configuration.
Consider a short task that drives CPU to 98% and then finishes, versus a machine whose load stays high. These are illustrative cases, not measured incidents. A sustained window filters the first and draws attention to the second, where you might investigate a queue, an abnormal process or expected heavy work.
Fleetd takes the minimum CPU and memory values among samples in the recent window. If the minimum exceeds the threshold, all received samples in that set exceed it. This is neither an average nor a count of isolated violations. A value exactly equal to 90% does not exceed 90%.
It also requires the oldest sample to be sufficiently old, allowing a thirty-second coverage tolerance. The five-minute default therefore does not imply waiting exactly five full minutes. The implementation does not require a minimum sample count or verify that every sampling gap is small. It evaluates received samples with a coverage allowance, not uninterrupted observation.
Memory usage is calculated from total memory minus MemAvailable, divided by total memory. It does not simply treat all Linux cache as unavailable, and a memory alert is not a diagnosis of an application leak.
Offline means reports stopped arriving
Agents normally report basic metrics every ten seconds; the server evaluates alerts every five seconds. With the default threshold, about forty seconds without a report makes a machine eligible for an offline alert at a subsequent evaluation.
The server uses its own receipt timestamp rather than the agent's clock for this check. Still, the cause could be a stopped agent, power loss, a router outage, DNS failure or an unreachable Fleetd server path. The alert alone cannot distinguish them.
The threshold tolerates several missed reports rather than reacting to one missing sample. Evaluation scheduling, database work and mail delivery also affect notification time: this is not a promise of an email exactly forty seconds after disconnection.
Two safeguards reduce false alarms. An approved machine that has never reported is not treated as an outage. After server startup, new offline alerts wait through a grace period of offline + 15s, about fifty-five seconds with defaults, allowing agents to reconnect.
That means enrollment still requires checking a real first report: silence from a newly approved machine does not prove successful setup. And if the Fleetd server itself stops, it cannot evaluate or send alerts. Monitoring the monitoring service requires an independent external check.
Losing contact must not resolve an existing fault
Suppose a machine has an active disk alert and then loses its connection. There is no new disk information. Resolving the disk alert merely because no current violation was observed would give a misleading recovery message.
Fleetd preserves active alerts on offline machines. It neither opens nor resolves other state alerts from their stale data, and adds an offline alert once allowed by the grace period.
When reports resume, evaluation continues from the available data. The offline alert may resolve while the disk alert remains. Restored communication is not restored disk capacity.
Slow collectors have a different timescale
Disk, service, container and WireGuard data are normally collected every six basic reporting rounds, roughly once a minute. Evaluating every five seconds cannot turn that into five-second real-time observation.
| Check | Current trigger |
|---|---|
| Disk space | Mount usage exceeds its threshold |
| Inodes | Used inode percentage exceeds its threshold |
| Services | A failed systemd unit is reported |
| Containers | Restarting, unhealthy status, or nonzero exit |
| WireGuard | A previously handshaking peer exceeds the handshake-age limit |
Disk and inode alerts do not use the CPU five-minute window. They evaluate the latest stored slow report. Inodes matter separately because a filesystem can have bytes available but no capacity to create more files.
Containers exiting successfully with code 0 are ignored: a finished one-off task is not necessarily a fault. Nonzero exits, unhealthy status and the restarting state trigger alerts. This is not restart-frequency analysis; the evaluator does not count restarts per minute. A running container also does not prove its business endpoint works.
WireGuard checks only peers with a previous handshake, skipping peers that have never handshaken. It compares the agent's report time with the handshake timestamp rather than mixing in the server's clock. An old handshake is a clue, not a complete connectivity test. Understand traffic and keepalive behavior before enabling the rule for intermittently active peers. wg_handshake: 0 disables it globally; there are no per-peer exclusions here.
Resolution means the rule no longer matches
Each evaluation calculates the desired set of alerts and compares it with unresolved records in the database. New entries open, absent entries resolve and continuing entries remain. Keys distinguish machines and monitored objects, such as individual mounts or containers.
For an online machine, a rule that no longer matches can resolve the existing alert. With CPU and memory, one sample at or below the threshold breaks the “all samples above” condition. There is no separate lower recovery threshold or “healthy for another two minutes” requirement. A sustained opening condition does not imply a sustained recovery condition.
Slow data has another boundary. A basic report without new slow data preserves the previous slow report. But if a new slow report contains an empty list because a collector tool disappeared or failed, an old object-specific alert can disappear too.
For example, a previously failed service may no longer be listed after its collection command fails. The implementation does not fully distinguish a successful healthy check from an unavailable check for each collector. A resolved record therefore is not independent proof of business recovery. As the previous article explained, failed-service collection uses systemd commands, not a complete procd health check on OpenWrt.
Deduplication does not guarantee delivery
Fleetd does not create a fresh record or send a fresh opening notification every five seconds for the same continuing alert. When the rule clears, it generates a resolution change. Opening and resolution changes from one evaluation are combined into one email.
The aggregation boundary is that evaluation, not a multi-minute window. There is no recurring reminder schedule for an unresolved fault. One-off events such as address changes have their own notification path and are not all merged into periodic evaluation mail.
Alert state is written before email is sent asynchronously. If sending fails, the server logs an error; there is no durable delivery queue or automatic retry. A record in the dashboard does not prove an email arrived. Continued silence might mean the same fault remains open, and the next evaluation will not automatically resend a failed opening notification.
Deduplication reduces noise but does not replace reliable delivery, reminders, acknowledgment or escalation. Those remain possible improvements.
Email is an output channel, not the only record
SMTP sends changes to configured recipients. Without a host or recipients, notifications are logged and alert state remains visible in the dashboard. language: zh selects Chinese server alerts and email; the web interface language is a separate browser-side choice.
Metrics are retained for seven days and resolved alerts for ninety days, with periodic cleanup. An old alert can therefore remain after its detailed metric history is gone. Longer-term investigations require an explicit retention plan.
A changed connection address is a one-off event when an existing address changes, not an ongoing fault that waits for the old IP to return. It may simply reflect a different network exit.
Validate the complete notification chain
For a small fleet, first make opening, recording, notification and resolution understandable, then tune thresholds. These are suggested acceptance scenarios, not faults already induced in production.
| Scenario | What to observe |
|---|---|
| Brief CPU spike | One high sample should not create a sustained alert |
| Sustained high test samples | Coverage requirements produce one opening, not repeated openings |
| Reports stop during an active fault | Preserve the existing fault and add offline when eligible |
| Reports resume but the fault remains | Resolve offline while retaining the original fault |
| SMTP unavailable | Dashboard record and delivery error log; no assumed retry |
| Collector unavailable | Check whether missing data affects resolution |
Use controlled data in an isolated environment for thresholds and time windows. There is no need to fill a production root filesystem or disconnect a router carrying your remote access.
Existing tests cover parts of slow-alert classification, including disks, normal and abnormal container exits, and WireGuard handshake conditions. An end-to-end test also covers a report producing a disk alert. They do not establish complete coverage of all transitions and mail delivery.
Fleetd currently aims for explainable alerts in a small self-hosted environment: what triggered them, why they are not repeated, when they close and where notification can fail. Explicit collector availability and data freshness, reliable mail delivery and per-machine rules are useful next steps.
The next article returns to agent maintenance: signed update verification, version checks and the boundaries of failure recovery.
- Fleetd repository
- Alert documentation
- v0.3.0 evaluation and transitions
- Report storage and memory calculation
- Email delivery
- Example configuration
Originally published at ay3s.com.
Top comments (0)