Passive monitoring, in a .net core worker service using Nagios NRDP protocol

Nagios for those not aware is an infrastructure monitoring system. In Nagios you can have two types of checks

• Active (run a scheduled script to figure out the problem)
• Passive (have the service report back to nagios in a scheduled manner)

When you have a big infrastructure you can use Nagios to monitor things like disc spaces/ temperatures/ dbs etc, but you can also use it to monitor applications. Especially if you have a microservice architecture and you have to monitor hundreds of different services it is convenient to have checks and alerts.

So for example

1. if you want to monitor the health of an API, you could write an active check that performs certain requests and compare the results with an expected outcome
2. if you want to monitor a consumer service, you could write an active check that sends a message and if possible examine its impact. But what if there is no visible impact?
3. if you have a worker service...its a closed system and monitoring from the outside system is nearly impossible.

And thus for point 3 and sometimes for point 2 we need to write a passive check. Which is a check within the worker/consumer services.

Nagios supports two different ways to do passive checks

• Nsca (Nagios Service Check Adapter)
• Nrdp (Nagios Remote Data Processor)

Nsca needs the messages to be encrypted, where Nrdp is a simple http request with either JSON or XML as a payload (and a secret key).

Here is a basic implementation of Nrdp using JSON.

The model that needs to be posted looks like:

  public class NrdpJsonMessage
    {
        [JsonProperty("checkresults")]
        public List<JsonRecord> Records { get; set; }
    }

    public class JsonRecord
    {
        [JsonProperty("checkresult")]
        public JsonCheckResult CheckResult { get; set; }
        [JsonProperty("hostname")]
        public string HostName { get; set; }
        [JsonProperty("servicename")]
        public string ServiceName { get; set; }
        [JsonProperty("state")]
        public string State { get; set; }
        [JsonProperty("output")]
        public string Output { get; set; }
    }

    public class JsonCheckResult
    {
        [JsonProperty("type")]
        public string Type { get; set; }
        [JsonProperty("checktype")]
        public string CheckType { get; set; }
    }

The class responsible for sending the passive check to nagios

public class Monitor
    {
        private readonly Configuration _configuration =
        new Configuration();

        public Monitor()
        {
        }

        public async Task<bool> SendPassiveCheck(NagiosWarningLevel level,
            string hostName, string serviceName, string message)
        {
            KeyValuePair<string, string> monitorObject;

            var nrdpMessage =
                new NrdpJsonMessage()
                {
                    Records = new List<JsonRecord>()
                    {
                        new JsonRecord()
                        {
                            CheckResult = new JsonCheckResult()
                            {
                                Type = "service",
                                CheckType = "1"
                            },
                            HostName = hostName,
                            ServiceName = serviceName,
                            State = Convert.ToString((int) level),
                            Output = message + " | perfdata=1;"
                        }
                    }
                };

            monitorObject = new KeyValuePair<string, string>("JSONDATA",
                JsonHelper.Serialize(nrdpMessage));

            var postDataObject = new FormUrlEncodedContent(new KeyValuePair<string, string>[]
            {
                new KeyValuePair<string, string>("cmd", "submitcheck"),
                new KeyValuePair<string, string>("token", _configuration.ProcessorPassword),
                monitorObject,
            });

            var data = await postDataObject.ReadAsStringAsync();

            var response = await _someTypeOfHttpRequestClass.Post(
                $ "{_configuration.Protocol}://{_configuration.Address}:{_configuration.Port}/nrdp/",
            null, data, null,
            "application/x-www-form-urlencoded");
            return true;
        }
    }

One of the things, that you need to be aware when implementing this. Is that, if you send any message, valid or invalid to the nrdp endpoint, you will get a 200 back regardless, with a small description that either denote success, or what went wrong.

so as it is obvious the only thing you need to do with just to send a message.

_monitor?.SendPassiveCheck(NagiosWarningLevel.Critical,
                           _nagiosConfiguration.HostName,
                           _nagiosConfiguration.ServiceName,
                           ex.Message);

by selecting one of the warning levels

public enum NagiosWarningLevel
    {
        OK,
        Warning,
        Critical,
        Unknown
    }

But the only thing you managed to achieve so far is send one message to nagios. Which is not an achievement of great importance.

What you need is periodic messages, and recoverability. So when something goes wrong, nagios can alert you, that something is wrong, or a service managed to recover.

    public class PassiveMonitor : IDisposable
    {
        private readonly Monitor _monitor;
        private readonly Configuration _configuration = new Configuration ();
        private Timer _timer;

        public NagiosWarningLevel NagiosWarningLevel { private get; set; }
        public string Message { private get; set; }

        public PassiveMonitor(Monitor monitor)
        {
            _monitor = monitor;
            NagiosWarningLevel = NagiosWarningLevel.OK;
            Message =
                $"HeartBeat at {DateTime.UtcNow.ToString(CultureInfo.InvariantCulture)}";
        }

        private void OnTimedEvent(object source, ElapsedEventArgs e)
        {
            // ... do checks here and set the NagiosWarningLevel and Message
            _monitor.SendPassiveCheck(NagiosWarningLevel,
                                      _configuration.HostName,
                                      _configuration.ServiceName,
                                      Message).Wait();
        }

        public void SetTimer()
        {
            CreateTimer();
        }

        public void PauseTimer()
        {
            if (_timer == null) return;
            _timer.Enabled = false;
        }

        public void ResumeTimer()
        {
            if (_timer == null) return;
            _timer.Enabled = true;
        }

        private void CreateTimer()
        {
            if (_timer != null) return;

            // Create a timer with a X minute interval.
            _timer = new Timer(_configuration.RepeatTestEveryXMinutes * 60000);
            // Hook up the Elapsed event for the timer. 
            _timer.Elapsed += OnTimedEvent;
            _timer.AutoReset = true;
            _timer.Enabled = true;
        }

        public void Dispose()
        {
            _timer?.Dispose();
        }
    }

Happy Monitoring