Project of the month: A check plugin to check checks

July 2009: With over 200 services checks on each of their many hosts, T-Mobile admins were receiving more than the average barrage of alerts. As with any large and complex environment, too much detail can be counter productive.
So a plugin was needed, which could gauge a critical level of alerts for a specific service type. In effect it would summarise checks at Critical/Unknown, Warning, OK levels, and send an alert only once a significant level was reached. Check_cluster was unworkable, as it required each of the 200+ service names to be configured.  Check_multi was also inappropriate for the very same reason – too much unnecessary configuration work.
Beyond evaluating a critical level, the plugin needed to prioritise groups of service groups between hosts, sending a higher alert for group fallouts. Flexibility was naturally also important; it had to allow T-Mobile to arbitrarily choose the monitored service checks by service group or by regular expression on the service name. To top it off, the plugin would also need to integrate into their new NETWAYSGrapherv2.
Within 10 days, Birger presented check_ndo.pl, to meet all these needs. By calling up the Nagios database, the plugin could not only filter service groups but also retrieve status information, making it an simple, elegant piece of code. And the good news is, he will release it soon for all to download under GPL at www.MonitoringExchange.org. We’ll keep you posted.