Send events to Zenoss from scripts

Working at Zenoss the company, our IT group uses Zenoss the application to monitor our infrastructure. Initially, we setup the needed monitoring for various resources, added thresholds and received email alerts if there were problems. That was all well and good, but in scripts written to perform various maintenance tasks on our systems, problems encountered were instead logged and emailed.

One of my clever co-workers noted it would make more sense to send such problems to our Zenoss instance, in order to incorporate them into our existing alerting schemes, as well as quickly see if the problem encountered had been caused by other problems on the same machine, etc. As it turns out, it’s actually quite easy to do this! This howto explains the process fairly well, giving basic examples in several languages. I’ll provide a more robust Python example here, and explain how we used a non-documented feature to great effect.

I found it best to create a function to try to send the event to Zenoss, falling back to sending an email if problems were encountered. This would then be called whenever an event needs to be sent:

You might want to handle problems sending the event differently of course xmlrpclib is part of the standard library, so you don’t need to install anything. send_alert() is a function I commonly include in my sysadmin scripts. It simply logs the message passed (on how to setup logging, see this post), then emails it to the appropriate party. This way you can ensure that even if something is up with the Zenoss instance, someone still finds out there’s a problem. You could also use this same function as a way to exit your problem cleanly if problems are encountered that can’t be resolved, or connect it to another system that handles this.

On the event attributes:

• The value for ‘device’ should match a device name in Zenoss.
• Severity is a numeric value mapped to what sort of event it is. The following options are available:
  • Severity: Description
  • 5: Critical
  • 4: Error
  • 3: Warning
  • 2: Info
  • 1: Debug
  • 0: Clear

  It might be useful to change severity in the above function to a parameter, so the full range of event types would be available.

• Component is an object contained by a device, such as CPU or hard drive space. It’s basically free form, although there are a number setup in Zenoss by default. You might decide to utilize and augment this structure, or you might create your own custom one.
• Event classes exist to categorize events. A number of these are available in Zenoss by default, and you can also add your own.

There is an additional attribute that turned out to be quite useful: timeout. When a timeout is passed, it is added to the events.heartbeat table in the Zenoss database. Once the specified time has passed, an alert will be raised for that device. This is why the device name passed needs to match the device name in Zenoss, otherwise nothing happens after the timeout. To send it, simply add these two lines to the evt dictionary defined in the above function:

We took advantage of timeout to keep track of our backup processes. When the backup process for a device started, it sent an event to Zenoss, the timeout being the time recorded for the last backup, plus some buffer. Then if the backup didn’t complete in the proper period of time, an alert was raised for that device, with the component of “/Backup”. In addition, when the backup completed, another event was sent to Zenoss, with a new timeout of the time between backups of the current server minus the time taken for the current backup, again plus some buffer. Then if the next backup didn’t start (and thus reset the timer) in time, an alert was also sent. This system combined with email fall back means that we always know if our backups don’t start and/or don’t complete.

Be Sociable, Share!

• 

No related posts.

Related posts brought to you by Yet Another Related Posts Plugin.