DEV Community: logiserv

Testing a Zabbix Agent on Disposable VMs: 10 Silent Failures Found for a Few Cents

logiserv — Sun, 25 Oct 2026 11:35:48 +0000

An agent install script that prints "Installation completed successfully" proves very little. The agent can be running, connected to the server, and reporting wrong values or nothing at all on some checks, without a single error message.

That is what we found while testing the Logiserv agent install script on brand-new virtual machines. Here is the method, what it cost, and the ten problems we found. Most of them affect any Zabbix installation, with or without Logiserv.

The method: VMs you delete instead of stopping

To test a first install, you need a clean machine every time. A reused VM keeps the agent, its configuration and its permissions: you are testing an upgrade, not an installation. Hence three rules:

Delete rather than stop. A stopped instance no longer costs compute, but its disk is still billed. And a deleted VM starts clean next time.
Self-destruction. Each VM is launched with shutdown -h +120 in its user data and the shutdown behavior set to Terminate: if forgotten, it disappears on its own after two hours.
No open ports. Commands go through AWS Systems Manager (Run Command): no SSH, no keys to manage. The IAM account used can only launch small instances, only with a Purpose=agent-test tag, and can only delete those.

Total cost for the day, five VMs included: a few cents.

1. Test a distribution family, not an architecture

The first instinct is to test "one ARM Linux and one x86 Linux". But the script never distinguishes architecture: the Zabbix repository provides the right package. What it does split on is the family: Debian/Ubuntu (apt, repository built with $ID$VERSION_ID), RHEL and derivatives (dnf, el9 repository, SELinux), Amazon Linux, SUSE.

The chosen matrix covers both axes with two VMs: Ubuntu 24.04 on ARM and Rocky Linux 9 on x86. Debian 12 came later, and it revealed its own problems (point 7).

2. The smallest VMs have their own failures

Two false problems you need to learn to recognize:

512 MB of memory is not enough for dnf on Rocky Linux: the repository index download is killed for lack of memory. A 1 GB instance fixes it.
CPU credits. A T3 instance in standard mode starts with no credits and drops to 10% of a core once they run out. After a few SELinux recompilations, our counting commands took 13 seconds instead of 50 milliseconds, and the agent killed them after 3 seconds: "Timeout while executing a shell script". It wasn't the script, it was the VM.

3. Under SELinux, wrong values with no error at all

Rocky, AlmaLinux and RHEL enable SELinux in enforcing mode. The policy shipped for the agent (zabbix_agent_t) prevents it from reading /var/log/secure, even with an ACL, and from running ip, ss, systemctl or rpm.

The trap: an item system.run["ss -tan state established | wc -l"] does not go into error. ss is blocked, wc -l counts zero lines, and Zabbix receives 0. The dashboard shows "0 established connections", with nothing red anywhere.

The only way to see it is to look for denials:

sudo ausearch -m AVC -ts recent -c zabbix_agentd

4. systemd denies too, and elsewhere

An item checked the firewall state with systemctl is-active firewalld. It returned: Failed to retrieve unit state: Access denied. Yet there was no denial in ausearch -m AVC.

The denial did not come from the kernel but from systemd itself, which consults SELinux before answering (init_t:system status). It is logged under a different type:

sudo ausearch -m USER_AVC -ts recent | grep zabbix_agent_t

5. Denials hidden by the base policy

Even with the agent's domain set to permissive, some denials did not show up. The SELinux policy contains dontaudit rules that block without logging. To see them during a diagnosis:

sudo semodule -DB   # disable dontaudit
# reproduce, then ausearch
sudo semodule -B    # restore them

That is how the denials for ss on its netlink socket and for rpm on its database appeared. The end result is a small SELinux module that grants only those reads and executions, and leaves SELinux in enforcing.

6. An ACL on /etc/shadow vanishes at the first password change

To detect a change to /etc/shadow, the template computed a checksum of the file. On RHEL there is no shadow group, and the file is mode 0000. An ACL seems to work... until the first passwd or useradd: those tools rewrite the file (new inode) without copying the ACLs.

The solution is simpler and safer: track the modification time (vfs.file.time[/etc/shadow,modify]), which only needs a stat. The agent no longer needs to read password hashes, on any distribution.

7. Debian 12 in the cloud: no logs, no cron

Debian 12 cloud images install neither rsyslog nor cron. /var/log contains no auth.log, no syslog, no kern.log: everything goes through journald.

Consequence: the checks for failed SSH logins, sudo commands or kernel errors read nothing. And an item vfs.file.cksum[/etc/crontab] fails, since the file does not exist.

A testing detail along the way: check for the presence of the daemon (command -v rsyslogd), not of the files. After rsyslog is removed, an old auth.log stays in place, frozen, and makes you think everything is fine.

8. A single failing item is enough to break a calculated score

Security scores are calculated items that add up around ten signals. If just one of those items is "not supported", Zabbix stops calculating anything: "Cannot evaluate function: item ... is not supported".

A file that is absent on one distribution (/var/log/syslog on RHEL, /etc/crontab on Debian 12) is therefore enough to make the whole score disappear. Two fixes:

read logs with an expression that covers both families: logrt["/var/log/^(syslog|messages)$",...];
for a legitimately absent file, add the preprocessing step "Check for not supported value", limited to the No such file error, which replaces the failure with 0. Permission errors remain visible.

9. Switching from log[] to logrt[] is not enough for agents already running

After replacing log["/var/log/syslog",...] with logrt["/var/log/^(syslog|messages)$",...] in the template, two Ubuntu servers kept failing: "Cannot obtain information for file "/var/log/^(syslog|messages)$"". Yet the agent could read /var/log/syslog without any problem.

It was looking for a file with literally that name. The item had kept the same ID, and the already-running agent had updated the key while keeping the old log mode. New agents did not have the problem.

Fix without restarting the agents: disable the items, wait a minute for the agent to forget them, then re-enable them.

10. The agent log doesn't tell the whole story

Last lesson, and the most general: the agent log is not enough to verify an installation.

Preprocessing is applied server-side: the agent may log an error that the server then turns into a valid value.
Some failures appear only server-side, such as a logrt with no matching file.
Automatically discovered items (disks, network cards) only receive template changes at the next discovery run, once an hour here.

The reliable check goes through the Zabbix API: item.get with filter: {state: 1} lists all failing items, across the whole fleet, in a single request.

What we take away

An install test is judged server-side, item by item, not by the script's final message.
SELinux doesn't break loudly: it reports zeros. Look for AVC and USER_AVC, with dontaudit disabled.
Disposable VMs are the right cost scale: one more distribution to test costs a cent and five minutes.

All these fixes are built into the Logiserv install script and its templates: a dedicated SELinux module, rsyslog installed when no syslog daemon is present, /etc/shadow tracked by its modification time. For a manual install, the guides for Rocky Linux, Debian and Amazon Linux detail every command.

Originally published in French on logiserv.app.

Install Zabbix Agent 2 on Ubuntu 24.04 and 22.04 in 10 Minutes

logiserv — Sat, 24 Oct 2026 07:51:21 +0000

To monitor an Ubuntu server with Zabbix, you need to install an agent: a lightweight service that collects system metrics (CPU load, memory, filesystem usage, running services, logs) and sends them to the monitoring server. See the full definition of the Zabbix agent.

This guide covers the complete manual installation on Ubuntu Server 20.04 LTS, 22.04 LTS and 24.04 LTS. The procedure works with any Zabbix server.

The easy way: with Logiserv

Logiserv is built on this same Zabbix engine and automates everything. From the Agent page of the dashboard, a command is generated for your fleet:

curl -sSL "https://www.logiserv.app/api/agent-script/linux?token=..." | sudo bash

The script detects the distribution, adds the repository, installs the package, gets a unique PSK key for the system from the server, writes the configuration, creates the host on the server side with the right templates, adjusts the permissions needed for security metrics and starts the service.

Allow 1 to 2 minutes per system with Logiserv. The detailed manual installation below takes 6 steps and about 10 minutes per server, not counting host creation on the Zabbix server side. See the detailed comparison.

Agent 1 or Agent 2?

Two packages coexist in the official repository:

zabbix-agent: the legacy agent written in C, configuration file zabbix_agentd.conf
zabbix-agent2: the current version written in Go, configuration file zabbix_agent2.conf

Agent 2 is recommended for any new installation: built-in plugins (PostgreSQL, MySQL, Docker, Redis…), a single persistent connection to the server, lower memory footprint. This guide uses Agent 2; the commands for Agent 1 are identical if you replace agent2 with agent everywhere.

Note: The package provided by the Ubuntu repositories (universe) is often much older than the current version. Use the official Zabbix repository described below to get an up-to-date version aligned with your server.

Prerequisites

sudo access on the system
the IP address or DNS name of your Zabbix server
outbound network access to port 10051/TCP on the server

Step 1: add the official Zabbix repository

The zabbix-release package name includes the Ubuntu version. For Ubuntu 24.04:

wget https://repo.zabbix.com/zabbix/7.4/release/ubuntu/pool/main/z/zabbix-release/zabbix-release_latest+ubuntu24.04_all.deb
sudo dpkg -i zabbix-release_latest+ubuntu24.04_all.deb
sudo apt update

For Ubuntu 22.04, replace ubuntu24.04 with ubuntu22.04; for 20.04, with ubuntu20.04.

Automatic version, whatever the release:

. /etc/os-release
wget -qO /tmp/zabbix-release.deb \
  "https://repo.zabbix.com/zabbix/7.4/release/ubuntu/pool/main/z/zabbix-release/zabbix-release_latest+ubuntu${VERSION_ID}_all.deb"
sudo dpkg -i /tmp/zabbix-release.deb
sudo apt update

Step 2: install the agent

sudo apt install -y zabbix-agent2

The package creates the zabbix system user, installs the systemd unit and places the configuration file in /etc/zabbix/.

To also install the optional plugins (PostgreSQL, MongoDB, MSSQL):

sudo apt install -y zabbix-agent2-plugin-*

Step 3: configure the agent

Edit /etc/zabbix/zabbix_agent2.conf:

sudo nano /etc/zabbix/zabbix_agent2.conf

The essential directives:

# Server allowed to query this agent (passive mode)
Server=192.168.1.10

# Server to which the agent pushes its metrics (active mode)
ServerActive=192.168.1.10

# Must match EXACTLY the host name defined on the Zabbix server side
Hostname=srv-web-01

LogFile=/var/log/zabbix/zabbix_agent2.log
LogFileSize=10

The Hostname pitfall

The Hostname value must be strictly identical to the "Host name" field of the host created in the Zabbix interface. A case difference is enough for active checks to never report, with no explicit error on the server side.

To automatically use the system's name:

HostnameItem=system.hostname

Active or passive mode

Passive (Server=): the server queries the agent, which must be reachable on port 10050. Problematic behind NAT.
Active (ServerActive=): the agent pushes its data. No inbound port to open, preferable for remote or cloud systems.

Both modes can coexist.

Tip: Don't feel like going through the next steps? The PSK key, copying it to the server, creating the host and starting the service are where mistakes happen most. With Logiserv, a single command does them all, with a unique key per system. Try it for free, no credit card.

Step 4: encrypt the connection with a PSK key

Without encryption, metrics travel in clear text. Generate a 32-byte key:

sudo mkdir -p /etc/zabbix
openssl rand -hex 32 | sudo tee /etc/zabbix/zabbix_agent2.psk > /dev/null
sudo chown zabbix:zabbix /etc/zabbix/zabbix_agent2.psk
sudo chmod 600 /etc/zabbix/zabbix_agent2.psk

Complete the configuration:

TLSConnect=psk
TLSAccept=psk
TLSPSKIdentity=PSK-srv-web-01
TLSPSKFile=/etc/zabbix/zabbix_agent2.psk

Copy the same key and identity into the Encryption tab of the host on the server side. Use a distinct identity per system: a compromised shared key exposes the whole fleet.

Warning: openssl rand -hex 32 appends a newline. Zabbix generally tolerates it, but in case of a SSL alert number 40 error, remove it: sudo truncate -s 64 /etc/zabbix/zabbix_agent2.psk

Step 5: open the port in the firewall

Note: With Logiserv, this step is unnecessary. The install command generated from the Agent page configures the agent in active mode only: it connects itself outbound to the Logiserv server on port 10051/TCP and listens on no port. No inbound port to open, no firewall rule to create.

Only in passive mode. With UFW:

sudo ufw allow from 192.168.1.10 to any port 10050 proto tcp

Restricting to the Zabbix server's address avoids exposing the agent to the whole network. In active mode, no inbound rule is needed.

Step 6: start and verify

sudo systemctl enable --now zabbix-agent2
sudo systemctl status zabbix-agent2

Test a metric locally, without going through the server:

zabbix_agent2 -t system.cpu.load[all,avg1]
zabbix_agent2 -t vfs.fs.size[/,pfree]
zabbix_agent2 -t agent.version

A response like system.cpu.load[all,avg1] [d|0.080000] confirms collection works.

Check the log:

sudo tail -f /var/log/zabbix/zabbix_agent2.log

Common error messages

Message	Likely cause
`no active checks on server`	`Hostname` doesn't match any host on the server side
`SSL alert number 40`	PSK key or identity mismatch between agent and server
`connection refused`	Service stopped, or port 10050 filtered
`Permission denied` on a log	The `zabbix` user isn't in the `adm` group

On that last point, some security metrics require access to system logs:

sudo usermod -aG adm zabbix
sudo systemctl restart zabbix-agent2

Monitoring multiple Ubuntu servers

Repeated on twenty systems, this procedure becomes costly: repository, package, configuration, unique PSK key, host creation on the server side, key copying. Allow ten minutes per server, with as many opportunities for drift.

The classic approaches: Ansible or an equivalent configuration tool (robust, but playbooks and a secrets vault to maintain), cloud-init at provisioning, or Zabbix auto-registration with actions and host metadata configured on the server side.

At scale

On a much larger fleet, a commenter on r/zabbix (u/fabomajstor, shared with permission) runs it end to end in about 30 seconds per server:

Provisioning: Red Hat Satellite deploys the host, then Ansible installs the agent, creates the zabbix user and its sudoers rules, configures the local firewall and writes the agent config.
Passive checks: on a closed network, the server and proxies do the polling, so scheduling and load stay central. Server= is restricted to the authorized server and proxy IPs.
One PSK per host: generated during provisioning and stored encrypted on the host. No recovery: if a key is in doubt, the host config is destroyed and rebuilt with a new one, so there is no key escrow to protect.
Registration: the Zabbix API attaches templates, host groups and inventory automatically.
Hardening: DenyKey=system.run[*]. If none of your templates use system.run, that line alone closes the remote command path. AllowKey is only needed for a few explicit exceptions.

Discussion on r/zabbix: https://www.reddit.com/r/zabbix/comments/1xebg9x/zabbix_agent_2_on_ubuntu_24042204_the_6_steps_and/

The same installation with Logiserv

Logiserv is built on this same Zabbix engine and automates everything. From the Agent page of the dashboard, a command is generated for your fleet:

curl -sSL "https://www.logiserv.app/api/agent-script/linux?token=..." | sudo bash

Manual installation vs. Logiserv

	Manual installation (this guide)	With Logiserv
Steps	6 steps, about fifteen commands	1 copy-pasted command
Time per server	About 10 minutes, excluding host creation	1 to 2 minutes, everything included
Repository	Package to pick according to the Ubuntu version	Detected automatically
Configuration file	Edited by hand	Written by the script
PSK key	To generate, secure, then copy to the server side	Unique, generated and registered automatically
Host on the Zabbix server	To create in the interface, templates to link	Created automatically in the fleet's group
Firewall	UFW rule in passive mode	No inbound port
Zabbix server	To host, secure, back up, update	Hosted and maintained by Logiserv
Dashboards and alerts	To build	Included, ready to use

Recognized distributions: Ubuntu, Debian, Raspberry Pi OS, RHEL, Rocky, AlmaLinux, Amazon Linux, SLES. The distribution is detected automatically via /etc/os-release.
Encryption: unique PSK per system. 32-byte key generated at enrollment, never shared between two hosts.
Inbound port required: none. Active mode: only outbound traffic to the server is needed.
Log permissions: configured automatically. The zabbix user is added to the adm and shadow groups for security checks.

How the Agent page works is detailed in the Logiserv documentation.

Going further

Install Zabbix Agent 2: the per-system guide: overview and other OSes
Install Zabbix Agent on Debian: the differences from Ubuntu
Install Zabbix Agent on Rocky Linux and AlmaLinux: the RHEL family
Why monitor a Linux server: what monitoring detects before the incident
Monitoring a server's disk space: the most frequent and most avoidable outage
Windows and Linux server monitoring: the guide: the complete approach

Originally published in French on logiserv.app.