=cut

TITLE=alert-monitor
DESCRIPTION=decide on status of rrd variables, and optionally generate alarms
KEYWORDS=alert-monitor
DOCTOP=index
DOCPREV=monitors
DOCNEXT=alerter

=pod

=head1 alert-monitor - a status evaluator and alert trigger

=head2 Usage:

=exec ../alert-monitor -h 2>&1

=head2 Description:

The C<alert-monitor> compares the current value of variables specified in
the L<alerts file|configfile-alerts> in the configuration directory with
threshold values and sets the status of those variables accordingly.  
It saves the current status of variables in C<@@DATADIR@@/ALERTS>.

What value corresponds to what status level is set in the 
L<rrd definition|configfile-rrds> or sometimes the 
L<host definition|configfile-hosts>.  This way an rrd
definition will specify generally reasonable levels, but they can be 
overridden for hosts where they aren't reasonable.

For an rrd definition, an alert line looks like:

	alert varname relation oklevel [warnlevel [errorlevel]]

or

	alert varname nodata status

[The latter says that missing data for variable C<varname> will cause its status
to be level C<status>.]

For a host-specified alert level, the line looks like:

	alert rrdname varname relation oklevel [warnlevel [errorlevel]]

or

	alert rrdname varname nodata status

and the interpretation is the same, except that you're having to say
which rrd this alert refers to.


The available relations are:

	< (value is less than threshold)
	> (value is greater than threshold)
	= (value is equal to threshold)
	|< (absolute value of value is less than threshold)
	|> (absolute value of value is greater than threshold)
	delta< (difference between last two values is less than threshold)
	delta> (difference between last two values is greater than threshold)
	<daystddev (value is outside threshold * the past day's standard-deviation)
	<weekstddev (value is outside threshold * the past day's standard-deviation)
	<monthstddev (value is outside threshold * the past day's standard-deviation)

=head2 Example

To make things more concrete for the first (normal) case, here's a real example, 
from the C<load> rrd supplied in C<config-base>:

	alert load5 < 3 7 10

This means that if the C<load5> variable is less than 3, the status is set to OK.
If it's less than 7, it's WARN, less than 10 it's ERROR and more than that, it's
CRITICAL.

Since the first match is taken, it's possible to leave out the upper levels if
you don't want them to ocurr.  For example if you only wanted C<load5> to ever
go to WARN level, never above, you could use:

	alert load5 < 3

and then the only possible status levels are OK and WARN.

The possible C<relation>s are: <, =, >, |<, |>, delta<, delta>.  The first
three should be obvious.  The next two allow comparisons to the absolute value of
the variable's current value.  The last two allow comparisons to the change in
value.

=head2 Causing alerts

Depending on the lines in the L<alerts file|configfile-alerts>, the status may also
trigger alerts.  A matching line in the L<alerts config-file|configfile-alerts> will cause
C<alert-monitor> to run the L<alerter> for each of the specified 
recipients.  It will also be passed, in order:

=over 4

=item B<recipient> - the recipient; for L<alert-email> it
will be an email address

=item B<hostname> - the name of the host that the alert applies to

=item B<ip> - the IP number for that host, in case it's not in DNS

=item B<rrdname> - the name of the RRD

=item B<wildpart> - the wild part of a wildcard RRD.  E.G, for an
RRD of C<port-ftp> (using the wildcard RRD C<port-*>) the wildpart
would be C<ftp>.

=item B<variable> - the name of the variable

=item B<status> - the current status, as decided by alert-monitor

=item B<old_status> - the previous status

=item B<value> - the current value of the variable

=item B<relation> - the relation used to compare the variable to the
threshold, mostly for creating informative messages

=item B<threshold> - the threshold value that was exceeded

=item B<start> - timestamp of when the alert started

=item B<duration> - number of seconds that the alert has been active

=item B<host-description> - the description field from the host config-file

=item B<rrd-description> - the description tag on this rrd (desc="xxx")

=item B<webmaster> - the email address of the remstats person

=item B<template> - the name of the template file to generate the message from.

=back 4
