DEV Community

Cover image for Zabbix Triggers, Dashboards, Graphs, Notifications, and Alerting.
Morodolu Oluwafikunayomi
Morodolu Oluwafikunayomi

Posted on

Zabbix Triggers, Dashboards, Graphs, Notifications, and Alerting.

Part 3 will focus on turning Zabbix from a monitoring dashboard into an active alerting system.
So far, we have successfully connected our monitored server to Zabbix. The Zabbix server can now collect metrics from our agent, such as CPU usage, memory utilization, disk space, and system information.

However, monitoring data alone is not enough.

Imagine a production server running out of disk space. Zabbix can collect the disk usage percentage every minute, but unless someone checks the dashboard manually, nobody knows there is a problem.

This is where triggers, alerts, and notifications come in.
A complete monitoring workflow looks like this:
Metric Collection
|
|
v
Zabbix Items
|
|
v
Triggers
|
|
v
Problem Detection
|
|
v
Notifications
|
|
v
Administrator Action

Understanding Zabbix Items

Before creating triggers, we need to understand items.

An item is a metric that Zabbix collects from a monitored host.

Examples:

Item ------> Purpose
CPU utilization------> Monitor processor usage
Memory utilization-----> Monitor RAM usage
Disk space-----> Monitor storage capacity
Network traffic -----> Monitor bandwidth
System uptime------> Monitor availability
example:

Host:
rhcsa

Item:
CPU utilization

Value:
35%
Enter fullscreen mode Exit fullscreen mode

Creating a Trigger

A trigger defines when a normal metric becomes a problem.

Think of it like this:

Item:

CPU usage = 95%

Trigger:

If CPU usage is above 90% for 5 minutes,
create a problem.

The trigger is the logic that converts data into an event.
Enter fullscreen mode Exit fullscreen mode

*Create a CPU Trigger
*
Go to:

Monitoring
    |
  Host
   |
specific Hosts(rhcsa)
   |
 Triggers
   |
   Create trigger
Enter fullscreen mode Exit fullscreen mode

Fill in:

Name:

High CPU Usage on {HOST.NAME} in this case ours is {rhcsa}

Severity:

Warning

Expression:

Click:

Add

Select:

Item:
CPU utilization

Condition:

Last value > 20

Example expression:

last(/rhcsa/system.cpu.util)>90

Click:

Add

Now Zabbix will monitor CPU usage.


last(/rhcsa/system.cpu.util,#20)>20 isnt the same last(/rhcsa/system.cpu.util)>20 do take note as the latter
Take the most recent value of system.cpu.util and check if it is greater than 20.

on the rhcsa where the agent is we would trigger a high cpu usage using:

yes > /dev/null
Enter fullscreen mode Exit fullscreen mode

or in the zabbix server :
zabbix_get -s 192.168.120.130(ip of the agent) -k system.cpu.util
43.189230

Creating Dashboards

Dashboards provide a visual overview of your infrastructure.
zabbix 7.0 already provides a nice visual dashboard all we need to do is edit as we see fit, such as editing and showing us which host/server/system is using the most cpu and adding more host to view
and change the parameters we see as well

I had edited what i wish to see which is all the host/system/server, their cpu usage, memory not in use and how many user are logged in, feel free to play around as long it not a live or a production server. :D

Configuring Notifications

A trigger detects a problem.

A notification tells someone about it.

The flow:

Trigger
|
|
v
Action
|
|
v
Media Type
|
|
v
Email / Slack / Teams / SMS

in order to achieve this we must create a user or users that will be assigned to receive such triggers:
Go to:
alert-->media


Select your user.

`Go to:

Media

Add:

Type:

Email

Example:

admin@example.com`

Configure Zabbix Email Alert Notifications (Complete Step-by-Step Guide)

After creating triggers in Zabbix, the next step is to configure alert actions so that Zabbix can automatically notify administrators when a problem occurs.

In this guide, we will configure Zabbix to send an email notification when CPU usage becomes high.

The process involves:

  1. Creating the trigger
  2. Creating an email media type
  3. Configuring Gmail SMTP
  4. Creating a Google App Password
  5. Assigning the media to a Zabbix user
  6. Creating an alert action
  7. Testing the notification

Step 1: Create an Email Media Type

First, we need to tell Zabbix how it should send emails.

Navigate to:

Alerts
   |
Media types
Enter fullscreen mode Exit fullscreen mode

Click:

Create media type
Enter fullscreen mode Exit fullscreen mode

Configure the following:

Name 

trigger or problem
Enter fullscreen mode Exit fullscreen mode

Type 

Email
Enter fullscreen mode Exit fullscreen mode

Email provider

Select:

Generic SMTP
Enter fullscreen mode Exit fullscreen mode

Configure the SMTP settings:

SMTP server:
smtp.gmail.com

SMTP server port:
465
Enter fullscreen mode Exit fullscreen mode

For Gmail, use SSL/TLS:

Connection security:
SSL/TLS
Enter fullscreen mode Exit fullscreen mode

Do not use port 587 with STARTTLS if your network blocks it.

Step 2: Configure Gmail Authentication

At first, we tried sending email without authentication, but Gmail rejected the request.

The error received was:

530 Authentication required
Enter fullscreen mode Exit fullscreen mode

Gmail does not allow applications to send email using only the email address.

You must use a Google App Password.

Step 3: Create a Google App Password

Open your Google Account:

https://myaccount.google.com
Enter fullscreen mode Exit fullscreen mode

Go to:

Security
Enter fullscreen mode Exit fullscreen mode

Enable:

2-Step Verification
Enter fullscreen mode Exit fullscreen mode

After enabling it, go back to:

Security
Enter fullscreen mode Exit fullscreen mode

Find:

App passwords
Enter fullscreen mode Exit fullscreen mode

Create a new app password.

Select:

Other (Custom name)
Enter fullscreen mode Exit fullscreen mode

Enter:

Zabbix
Enter fullscreen mode Exit fullscreen mode

Google will generate a 16-character password.

Example:

xxxx xxxx xxxx xxxx
Enter fullscreen mode Exit fullscreen mode

Copy this password.

Important:

Do not use your normal Gmail password.

The Google App Password is the password Zabbix uses to authenticate with Gmail SMTP.

Step 4: Complete the Zabbix Email Configuration

Return to:

Alerts
   |
Media types
   |
trigger or problem
Enter fullscreen mode Exit fullscreen mode

Configure:

Email 

morodoluoluwafikunayomo@gmail.com
Enter fullscreen mode Exit fullscreen mode

Authentication

Change:

None
Enter fullscreen mode Exit fullscreen mode

to:

Username and password
Enter fullscreen mode Exit fullscreen mode

Enter:

Username:

morodoluoluwafikunayomo@gmail.com
Enter fullscreen mode Exit fullscreen mode

Password:

Google App Password generated earlier
Enter fullscreen mode Exit fullscreen mode

Step 5: Add Message Templates

Without a message template, Zabbix will create the alert but fail to send it.

The error will look like:

No message defined for media type
Enter fullscreen mode Exit fullscreen mode

Go to:

Alerts
   |
Media types
   |
trigger or problem
   |
Message templates
Enter fullscreen mode Exit fullscreen mode

Click:

Add
Enter fullscreen mode Exit fullscreen mode

Create a Problem template.

Message type:

Problem
Enter fullscreen mode Exit fullscreen mode

Template:

Problem started at {EVENT.TIME} on {EVENT.DATE}

Problem name:
{EVENT.NAME}

Host:
{HOST.NAME}

Severity:
{EVENT.SEVERITY}

Operational data:
{EVENT.OPDATA}

Event ID:
{EVENT.ID}
Enter fullscreen mode Exit fullscreen mode

Save.

Step 6: Assign the Media Type to the Admin User

Creating a media type is not enough.

Zabbix users need to have a notification method assigned.

If this step is skipped, you will get:

No media defined for user
Enter fullscreen mode Exit fullscreen mode

Go to:

Users
   |
Admin
Enter fullscreen mode Exit fullscreen mode

Open the:

Media
Enter fullscreen mode Exit fullscreen mode

tab.

Click:

Add
Enter fullscreen mode Exit fullscreen mode

Configure:

Type:

trigger or problem
Enter fullscreen mode Exit fullscreen mode

Send to:

morodoluoluwafikunayomo@gmail.com
Enter fullscreen mode Exit fullscreen mode

When active:

1-7,00:00-24:00
Enter fullscreen mode Exit fullscreen mode

Select severities:

Warning
Average
High
Disaster
Enter fullscreen mode Exit fullscreen mode

Save the user.

Step 7: Create the Alert Action

Now we tell Zabbix what to do when a problem occurs.

Navigate to:

Alerts
   |
Actions
   |
Trigger actions
Enter fullscreen mode Exit fullscreen mode

Click:

Create action
Enter fullscreen mode Exit fullscreen mode

Action Name

Enter:

Send CPU Alerts
Enter fullscreen mode Exit fullscreen mode

Conditions

Click:

Add condition
Enter fullscreen mode Exit fullscreen mode

Select:

Trigger severity
Enter fullscreen mode Exit fullscreen mode

Set:

is greater than or equals
Enter fullscreen mode Exit fullscreen mode

Choose:

High
Enter fullscreen mode Exit fullscreen mode

The condition becomes:

Trigger severity >= High
Enter fullscreen mode Exit fullscreen mode

This means only High and Disaster problems will send notifications.

Operations

Go to:

Operations
Enter fullscreen mode Exit fullscreen mode

Click:

Add
Enter fullscreen mode Exit fullscreen mode

Select:

Send message
Enter fullscreen mode Exit fullscreen mode

Choose:

Send to users
Enter fullscreen mode Exit fullscreen mode

Select:

Admin (Zabbix Administrator)
Enter fullscreen mode Exit fullscreen mode

Select media type:

trigger or problem
Enter fullscreen mode Exit fullscreen mode

Save the action.

Make sure the action status is:

Enabled
Enter fullscreen mode Exit fullscreen mode

Step 8: Test the Alert

Generate a CPU problem on the monitored host:

Install stress if needed:

sudo apt install stress
Enter fullscreen mode Exit fullscreen mode

Run:

stress --cpu 4 --timeout 300
Enter fullscreen mode Exit fullscreen mode

Wait for the trigger to fire.

Check:

Alerts
   |
Action log
Enter fullscreen mode Exit fullscreen mode

A successful notification should show:

Status:
Sent
Enter fullscreen mode Exit fullscreen mode

The email should arrive in your Gmail inbox.

Common Problems and Fixes

Problem 1: Host availability is red

Error:

Get value from agent failed
Enter fullscreen mode Exit fullscreen mode

Check:

Agent configuration
Network connectivity
Port 10050
Enter fullscreen mode Exit fullscreen mode

Problem 2: No media defined for user

Cause:

The media type exists but is not attached to the user.

Fix:

Users
 |
Admin
 |
Media
 |
Add
Enter fullscreen mode Exit fullscreen mode

Problem 3: No message defined for media type

Cause:

The media type has no message template.

Fix:

Add:

Problem
Enter fullscreen mode Exit fullscreen mode

message template.

Problem 4: SMTP timeout

Example:

Failed to connect to smtp.gmail.com port 587
Enter fullscreen mode Exit fullscreen mode

Cause:

Network blocking SMTP port 587.

Fix:

Use:

Port:
465

Security:
SSL/TLS
Enter fullscreen mode Exit fullscreen mode

Problem 5: Gmail authentication error

Example:

530 Authentication required
Enter fullscreen mode Exit fullscreen mode

Cause:

Using Gmail password instead of App Password.

Fix:

Create a Google App Password and use it in Zabbix.

After completing these steps, Zabbix can automatically detect problems and send email notifications without requiring manual checking.

Top comments (0)