Testing a Zabbix Agent on Disposable VMs: 10 Silent Failures Found for a Few Cents

An agent install script that prints "Installation completed successfully" proves very little. The agent can be running, connected to the server, and reporting wrong values or nothing at all on some checks, without a single error message.

That is what we found while testing the Logiserv agent install script on brand-new virtual machines. Here is the method, what it cost, and the ten problems we found. Most of them affect any Zabbix installation, with or without Logiserv.

The method: VMs you delete instead of stopping

To test a first install, you need a clean machine every time. A reused VM keeps the agent, its configuration and its permissions: you are testing an upgrade, not an installation. Hence three rules:

• Delete rather than stop. A stopped instance no longer costs compute, but its disk is still billed. And a deleted VM starts clean next time.
• Self-destruction. Each VM is launched with shutdown -h +120 in its user data and the shutdown behavior set to Terminate: if forgotten, it disappears on its own after two hours.
• No open ports. Commands go through AWS Systems Manager (Run Command): no SSH, no keys to manage. The IAM account used can only launch small instances, only with a Purpose=agent-test tag, and can only delete those.

Total cost for the day, five VMs included: a few cents.

1. Test a distribution family, not an architecture

The first instinct is to test "one ARM Linux and one x86 Linux". But the script never distinguishes architecture: the Zabbix repository provides the right package. What it does split on is the family: Debian/Ubuntu (apt, repository built with $ID$VERSION_ID), RHEL and derivatives (dnf, el9 repository, SELinux), Amazon Linux, SUSE.

The chosen matrix covers both axes with two VMs: Ubuntu 24.04 on ARM and Rocky Linux 9 on x86. Debian 12 came later, and it revealed its own problems (point 7).

2. The smallest VMs have their own failures

Two false problems you need to learn to recognize:

• 512 MB of memory is not enough for dnf on Rocky Linux: the repository index download is killed for lack of memory. A 1 GB instance fixes it.
• CPU credits. A T3 instance in standard mode starts with no credits and drops to 10% of a core once they run out. After a few SELinux recompilations, our counting commands took 13 seconds instead of 50 milliseconds, and the agent killed them after 3 seconds: "Timeout while executing a shell script". It wasn't the script, it was the VM.

3. Under SELinux, wrong values with no error at all

Rocky, AlmaLinux and RHEL enable SELinux in enforcing mode. The policy shipped for the agent (zabbix_agent_t) prevents it from reading /var/log/secure, even with an ACL, and from running ip, ss, systemctl or rpm.

The trap: an item system.run["ss -tan state established | wc -l"] does not go into error. ss is blocked, wc -l counts zero lines, and Zabbix receives 0. The dashboard shows "0 established connections", with nothing red anywhere.

The only way to see it is to look for denials:

sudo ausearch -m AVC -ts recent -c zabbix_agentd

4. systemd denies too, and elsewhere

An item checked the firewall state with systemctl is-active firewalld. It returned: Failed to retrieve unit state: Access denied. Yet there was no denial in ausearch -m AVC.

The denial did not come from the kernel but from systemd itself, which consults SELinux before answering (init_t:system status). It is logged under a different type:

sudo ausearch -m USER_AVC -ts recent | grep zabbix_agent_t

5. Denials hidden by the base policy

Even with the agent's domain set to permissive, some denials did not show up. The SELinux policy contains dontaudit rules that block without logging. To see them during a diagnosis:

sudo semodule -DB   # disable dontaudit
# reproduce, then ausearch
sudo semodule -B    # restore them

That is how the denials for ss on its netlink socket and for rpm on its database appeared. The end result is a small SELinux module that grants only those reads and executions, and leaves SELinux in enforcing.

6. An ACL on /etc/shadow vanishes at the first password change

To detect a change to /etc/shadow, the template computed a checksum of the file. On RHEL there is no shadow group, and the file is mode 0000. An ACL seems to work... until the first passwd or useradd: those tools rewrite the file (new inode) without copying the ACLs.

The solution is simpler and safer: track the modification time (vfs.file.time[/etc/shadow,modify]), which only needs a stat. The agent no longer needs to read password hashes, on any distribution.

7. Debian 12 in the cloud: no logs, no cron

Debian 12 cloud images install neither rsyslog nor cron. /var/log contains no auth.log, no syslog, no kern.log: everything goes through journald.

Consequence: the checks for failed SSH logins, sudo commands or kernel errors read nothing. And an item vfs.file.cksum[/etc/crontab] fails, since the file does not exist.

A testing detail along the way: check for the presence of the daemon (command -v rsyslogd), not of the files. After rsyslog is removed, an old auth.log stays in place, frozen, and makes you think everything is fine.

8. A single failing item is enough to break a calculated score

Security scores are calculated items that add up around ten signals. If just one of those items is "not supported", Zabbix stops calculating anything: "Cannot evaluate function: item ... is not supported".

A file that is absent on one distribution (/var/log/syslog on RHEL, /etc/crontab on Debian 12) is therefore enough to make the whole score disappear. Two fixes:

• read logs with an expression that covers both families: logrt["/var/log/^(syslog|messages)$",...];
• for a legitimately absent file, add the preprocessing step "Check for not supported value", limited to the No such file error, which replaces the failure with 0. Permission errors remain visible.

9. Switching from log[] to logrt[] is not enough for agents already running

After replacing log["/var/log/syslog",...] with logrt["/var/log/^(syslog|messages)$",...] in the template, two Ubuntu servers kept failing: "Cannot obtain information for file "/var/log/^(syslog|messages)$"". Yet the agent could read /var/log/syslog without any problem.

It was looking for a file with literally that name. The item had kept the same ID, and the already-running agent had updated the key while keeping the old log mode. New agents did not have the problem.

Fix without restarting the agents: disable the items, wait a minute for the agent to forget them, then re-enable them.

10. The agent log doesn't tell the whole story

Last lesson, and the most general: the agent log is not enough to verify an installation.

• Preprocessing is applied server-side: the agent may log an error that the server then turns into a valid value.
• Some failures appear only server-side, such as a logrt with no matching file.
• Automatically discovered items (disks, network cards) only receive template changes at the next discovery run, once an hour here.

The reliable check goes through the Zabbix API: item.get with filter: {state: 1} lists all failing items, across the whole fleet, in a single request.

What we take away

• An install test is judged server-side, item by item, not by the script's final message.
• SELinux doesn't break loudly: it reports zeros. Look for AVC and USER_AVC, with dontaudit disabled.
• Disposable VMs are the right cost scale: one more distribution to test costs a cent and five minutes.

All these fixes are built into the Logiserv install script and its templates: a dedicated SELinux module, rsyslog installed when no syslog daemon is present, /etc/shadow tracked by its modification time. For a manual install, the guides for Rocky Linux, Debian and Amazon Linux detail every command.

Originally published in French on logiserv.app.