monit - utility for monitoring services on a Unix system
monit [options] {arguments}
monit is a utility for managing and monitoring processes, files, directories and devices on a Unix system. Monit conducts automatic maintenance and repair and can execute meaningful causal actions in error situations. E.g. monit can start a process if it does not run, restart a process if it does not respond and stop a process if it uses to much resources. You may use monit to monitor files, directories and devices for changes, such as timestamps changes, checksum changes or size changes.
Monit is controlled via an easy to configure control file based
on a free-format, token-oriented syntax. Monit logs to syslog or
to its own log file and notifies you about error conditions via
customizable alert messages. Monit can perform various TCP/IP
network checks, protocol checks and can utilize SSL for such
checks. Monit provides a http(s) interface and you may use a
browser to access the monit program.
The behavior of monit is controlled by command-line options and a run control file, ~/.monitrc, the syntax of which we describe in a later section. Command-line options override .monitrc declarations.
The following options are recognized by monit. However, it is recommended that you set options (when applicable) directly in the .monitrc control file.
-c file Use this control file
-d n Run as a daemon once per n seconds
-g Set group name for start, stop, restart and status
-l logfile Print log information to this file
-p pidfile Use this lock file in daemon mode
-s statefile Write state information to this file
-I Do not run in background (needed for run from init)
-t Run syntax check for the control file
-v Verbose mode, work noisy (diagnostic output)
-H [filename] Print MD5 and SHA1 hashes of the file or of stdin if the filename is omitted; monit will exit afterwards
-V Print version number and patch level
-h Print a help text
In addition to the options above, monit can be started with one of the following action arguments; monit will then execute the action and exit without transforming itself to a daemon.
start all Start all services listed in the control file and enable monitoring for them. If the group option is set, only start and enable monitoring of services in the named group.
start name Start the named service and enable monitoring for it. The name is a service entry name from the monitrc file.
stop all Stop all services listed in the control file and disable their monitoring. If the group option is set, only stop and disable monitoring of the services in the named group.
stop name Stop the named service and disable its monitoring. The name is a service entry name from the monitrc file.
restart all Stop and start all services. If the group option is set, only restart the services in the named group.
restart name Restart the named service. The name is a service entry name from the monitrc file.
monitor all Enable monitoring of all services listed in the control file. If the group option is set, only start monitoring of services in the named group.
monitor name Enable monitoring of the named service. The name is a service entry name from the monitrc file. Monit will also enable monitoring of all services this service depends on.
unmonitor all Disable monitoring of all services listed in the control file. If the group option is set, only disable monitoring of services in the named group.
unmonitor name Disable monitoring of the named service. The name is a service entry name from the monitrc file. Monit will also disable monitoring of all services that depends on this service.
status Print full status information for each service.
summary Print short status information for each service.
reload Reinitialize a running monit daemon, the daemon will reread its configuration, close and reopen log files.
quit Kill a monit daemon process
validate Check all services listed in the control file. This action is also the default behavior when monit runs in daemon mode.
You may use monit to monitor daemon processes or similar programs running on localhost. Monit is particular useful for monitoring daemon processes, such as those started at system boot time from /etc/init.d/. For instance sendmail, sshd, apache and mysql. In difference to many monitoring systems, monit can act if an error situation should occur, e.g.; if sendmail is not running, monit can start sendmail or if apache is using to much system resources (e.g. if a DoS attack is in progress) monit can stop or restart apache and send you an alert message. Monit does also monitor process characteristics, such as; if a process has become a zombie and how much memory or cpu cycles a process is using.
You may also use monit to monitor files, directories and devices on localhost. Monit can monitor these items for changes, such as timestamps changes, checksum changes or size changes. This is also useful for security reasons - you can monitor the md5 checksum of files that should not change.
You may even use monit to monitor remote hosts. First and foremost monit is a utility for monitoring and mending services on localhost, but if a service depends on a remote service, e.g. a database server or an application server, it might by useful to be able to test a remote host as well.
You may monitor the general system-wide resources such as cpu usage, memory and load average.
monit is configured and controlled via a control file called monitrc. The default location for this file is ~/.monitrc. If this file does not exist, monit will try /etc/monitrc, then /usr/local/etc/monitrc and finally ./monitrc.
A monit control file consists of a series of service entries and global option statements in a free-format, token-oriented syntax. Comments begin with a # and extend through the end of the line. There are three kinds of tokens in the control file: grammar keywords, numbers and strings.
On a semantic level, the control file consists of three types of statements:
This is the hello galaxy version of a monit control file:
# # monit control file #
set daemon 120 # Poll at 2-minute intervals
set logfile syslog facility log_daemon
set alert foo@bar.baz
set httpd port 2812 and use address localhost
allow localhost # Allow localhost to connect
allow admin:monit # Allow Basic Auth
check system myhost.mydomain.tld
if loadavg (1min) > 4 then alert
if loadavg (5min) > 2 then alert
if memory usage > 75% then alert
if cpu usage (user) > 70% then alert
if cpu usage (system) > 30% then alert
if cpu usage (wait) > 20% then alert
check process apache
with pidfile "/usr/local/apache/logs/httpd.pid"
start program = "/etc/init.d/httpd start"
stop program = "/etc/init.d/httpd stop"
if 2 restarts within 3 cycles then timeout
if totalmem > 100 Mb then alert
if children > 255 for 5 cycles then stop
if cpu usage > 95% for 3 cycles then restart
if failed port 80 protocol http then restart
group server
depends on httpd.conf, httpd.bin
check file httpd.conf
with path /usr/local/apache/conf/httpd.conf
# Reload apache if the httpd.conf file was changed
if changed checksum
then exec "/usr/local/apache/bin/apachectl graceful"
check file httpd.bin
with path /usr/local/apache/bin/httpd
# Run /watch/dog in the case that the binary was changed
# and alert in the case that the checksum value recovered
# later
if failed checksum then exec "/watch/dog"
else if recovered then alert
include /etc/monit/mysql.monitrc include /etc/monit/mail/*.monitrc
This example illustrate a service entry for monitoring the apache web server process as well as related files. The meaning of the various statements will be explained in the following sections.
monit will log status and error messages to a log file. Use the set logfile statement in the monitrc control file. To setup monit to log to its own logfile, use e.g. set logfile /var/log/monit.log. If syslog is given as a value for the -l command-line switch (or the keyword set logfile syslog is found in the control file) monit will use the syslog system daemon to log messages. To turn off logging, simply do not set the logfile in the control file (and of course, do not use the -l switch)
The -d interval command-line switch runs monit in daemon mode. You must specify a numeric argument which is a polling interval in seconds.
In daemon mode, monit detaches from the console, puts itself in the background and runs continuously, monitoring each specified service and then goes to sleep for the given poll interval.
Simply invoking
monit -d 300
will poll all services described in your ~/.monitrc file every 5 minutes.
It is strongly recommended to set the poll interval in your ~/.monitrc file instead, by using set daemon n, where n is an integer number of seconds. If you do this, monit will always start in daemon mode (as long as no action arguments are given).
Monit makes a per-instance lockfile in daemon mode. If you need more monit instances, you will need more configuration files, each pointing to its own lockfile.
Calling monit with a monit daemon running in the background sends a wakeup signal to the daemon, forcing it to check services immediately.
The quit argument will kill a running daemon process instead of waking it up.
Monit can run and be controlled from init. If monit should crash, init will re-spawn a new monit process. Using init to start monit is probably the best way to run monit if you want to be certain that you always have a running monit daemon on your system. (It's obvious, but never the less worth to stress; Make sure that the control file does not have any syntax errors before you start monit from init. Also, make sure that if you run monit from init, that you do not start monit from a startup scripts as well).
To setup monit to run from init, you can either use the 'set init' statement in monit's control file or use the -I option from the command line and here is what you must add to /etc/inittab:
# Run monit in standard run-levels mo:2345:respawn:/usr/local/bin/monit -Ic /etc/monitrc
After you have modified init's configuration file, you can run the following command to re-examine /etc/inittab and start monit:
telinit q For systems without telinit:
kill -1 1
If monit is used to monitor services that are also started at boot time (e.g. services started via SYSV init rc scripts or via inittab) then, in some cases, a race condition could occur. That is; if a service is slow to start, monit can assume that the service is not running and possibly try to start it and raise an alert, while, in fact the service is already about to start or already in its startup sequence. Please see the FAQ for solutions to this problem.
The monit control file, monitrc, can include additional configuration files. This feature helps to maintain a certain structure or to place repeating settings into one file. Include statements can be placed at virtually any spot. The syntax is the following:
INCLUDE globstring
The globstring is any kind of string as defined in glob(7). Thus, you can refer to a single file or you can load several files at once. In case you want to use whitespace in your string the globstring need to be embedded into quotes (') or double quotes (``). For example,
INCLUDE "/etc/monit/monit configuration files/printer.*.monitrc"
loads any file matching the single globstring. If the globstring matches a directory instead of a file, it is silently ignored.
INCLUDE statements in included files are parsed as in the main control file.
If the globstring matches several results, the files are included in a non sorted manner. If you need to rely on a certain order, you might need to use single include statements.
Service entries in the control file, monitrc, can be grouped together by the group statement. The syntax is simply (keyword in capital):
GROUP groupname
With this statement it is possible to group similar service entries together and manage them as a whole. Monit provides functions to start, stop and restart a group of services, like so:
To start a group of services from the console:
monit -g <groupname> start
To stop a group of services:
monit -g <groupname> stop
To restart a group of services:
monit -g <groupname> restart
Monit supports three monitoring modes per service: active, passive and manual. See also the example section below for usage of the mode statement.
In active mode, monit will monitor a service and in case of problems monit will act and raise alerts, start, stop or restart the service. Active mode is the default mode.
In passive mode, monit will passively monitor a service and specifically not try to fix a problem, but it will still raise alerts in case of a problem.
For use in clustered environments there is also a manual mode. In this mode, monit will enter active mode only if a service was brought under monit's control, for example by executing the following command in the console:
monit start sybase (monit will call sybase's start method and enable monitoring)
If a service was not started by monit or was stopped or disabled for example by:
monit stop sybase (monit will call sybase's stop method and disable monitoring)
monit will not monitor the service. This allows for having services configured in monitrc and start it with monit only if it should run. This feature can be used to build a simple failsafe cluster. To see how, read more about how to setup a cluster with monit using the heartbeat system in the examples sections below.
Monit will raise an email alert in the following situations:
o A service timed out o A service does not exist o A service related data access problem o A service related program execution problem o A service is of invalid object type o A icmp problem o A port connection problem o A resource statement match o A file checksum problem o A file size problem o A file/directory timestamp problem o A file/directory/device permission problem o A file/directory/device uid problem o A file/directory/device gid problem
Monit will send an alert each time a monitored object changed. This involves:
o Monit started, stopped or reloaded o A file checksum changed o A file size changed o A file content match o A file/directory timestamp changed
You use the alert statement to notify monit that you want alert messages sent to an email address. If you do not specify an alert statement, monit will not send alert messages.
There are two forms of alert statement:
o Global - common for all services o Local - per service
In both cases you can use more than one alert statement. In other words, you can send many different emails to many different addresses. (in case you now got a new business idea; monit is not really suitable for sending spam).
Recipients in the global and in the local lists are alerted when a service failed, recovered or changed. If the same email address is in the global and in the local list, monit will send only one alert. Local (per service) defined alert email addresses override global addresses in case of a conflict. Finally, you may choose to only use a global alert list (recommended), a local per service list or both.
If a change occured on a monitored services, monit will send an alert to all recipients in the global list who have registred interest for the event type. Here is the syntax for the global alert statement:
Simply using the following in the global section of monitrc:
set alert foo@bar
will send a default email to the address foo@bar whenever an event occurred on any service. Such an event may be that a service timed out, a service was doesn't exist or a service does exist (on recovery) and so on. If you want to send alert messages to more email addresses, add a set alert 'email' statement for each address.
For explanations of the events, MAIL-FORMAT and REMINDER keywords above, please see below.
Each service can also have its own recipient list.
If you only want an alert message sent for certain events for certain service(s), for example only for timeout events or only if a service died, then postfix the alert-statement with a filter block:
check process myproc with pidfile /var/run/my.pid
alert foo@bar only on { timeout, nonexist }
...
(only and on are noise keywords, ignored by monit. As a side note; Noise keywords are used in the control file grammar to make an entry resemble English and thus make it easier to read (or, so goes the philosophy). The full set of available noise keywords are listed below in the Control File section).
Event filtering can be used to send a mail to different email addresses depending on the events that occured. For instance:
alert foo@bar { nonexist, timeout, resource, icmp, connection }
alert security@bar on { checksum, permission, uid, gid }
alert manager@bar
This will send an alert message to foo@bar whenever a nonexist, timeout, resource or connection problem occurs and a message to security@bar if a checksum, permission, uid or gid problem occurs. And finally, a message to manager@bar whenever any error event occurs.
This is the list of events you can use in a mail-filter: uid, gid, size, nonexist, data, icmp, instance, invalid, exec, changed, timeout, resource, checksum, match, timestamp, connection, permission
The following example will alert foo@bar on all events on all services by default, except the service mybar which will send an alert only on timeout. The trick is based on the fact that local definition of the same recipient overrides the global setting (including registered events and mail format):
set alert foo@bar
check process myfoo with pidfile /var/run/myfoo.pid
...
check process mybar with pidfile /var/run/mybar.pid
alert foo@bar only on { timeout }
The 'instance' alert type report events related to monit internals, such as when a monit instance was started, stopped or reloaded.
If the MTA (mailserver) for sending alerts is not available, monit can queue events on the local file-system until the MTA recover. Monit will then post queued events in order with theire original timestamp so the events are not lost. This feature is most useful if monit is used together with e.g. m/monit and when event history is important.
monit provides a default mail message layout that is short and to the point. Here's an example of a standard alert mail sent by monit:
From: monit@tildeslash.com Subject: monit alert -- Does not exist apache To: hauk@tildeslash.com Date: Thu, 04 Sep 2003 02:33:03 +0200
Does not exist Service apache
Date: Thu, 04 Sep 2003 02:33:03 +0200
Action: restart
Host: www.tildeslash.com
Your faithful employee, monit
If you want to, you can change the format of this message with the optional mail-format statement. The syntax for this statement is as follows:
mail-format {
from: monit@localhost
subject: $SERVICE $EVENT at $DATE
message: Monit $ACTION $SERVICE at $DATE on $HOST: $DESCRIPTION.
Yours sincerely,
monit
}
Where the keyword from: is the email address monit should pretend it is sending from. It does not have to be a real mail address, but it must be a proper formated mail address, on the form: name@domain. The keyword subject: is for the email subject line. The subject must be on only one line. The message: keyword denotes the mail body. If used, this keyword should always be the last in a mail-format statement. The mail body can be as long as you want and must not contain the '}' character.
All of these format keywords are optional but you must provide at least one. Thus if you only want to change the from address monit is using you can do:
set alert foo@bar with mail-format { from: bofh@bar.baz }
From the previous example you will notice that some special $XXX variables was used. If used, they will be substituted and expanded into the text with these values:
A string describing the event that occurred. The values are fixed and are:
Event: | Failure state: | Recovery state: --------------------------------------------------------------- CHANGED | "Changed" | "Changed back" CHECKSUM | "Checksum failed" | "Checksum passed" CONNECTION| "Connection failed" | "Connection passed" DATA | "Data access error" | "Data access succeeded" EXEC | "Execution failed" | "Execution succeeded" GID | "GID failed" | "GID passed" ICMP | "ICMP failed" | "ICMP passed" INSTANCE | "Monit instance changed"| "Monit instance changed not" INVALID | "Invalid type" | "Type passed" MATCH | "Regex match" | "No regex match" NONEXIST | "Does not exist" | "Exists" PERMISSION| "Permission failed" | "Permission passed" RESOURCE | "Resource limit matched"| "Resource limit passed" SIZE | "Size failed" | "Size passed" TIMEOUT | "Timeout" | "Timeout recovery" TIMESTAMP | "Timestamp failed" | "Timestamp passed" UID | "UID failed" | "UID passed"
The service entry name in monitrc
The current time and date (RFC 822 date style).
The name of the host monit is running on
The name of the action which was done. Action names are fixed and are:
Action: | Name: -------------------- ALERT | "alert" EXEC | "exec" MONITOR | "monitor" RESTART | "restart" START | "start" STOP | "stop" UNMONITOR| "unmonitor"
The description of the error condition
It is possible to set a standard mail format with the following global set-statement (keywords are in capital):
Format set with this statement will apply to every alert statement that does not have its own specified mail-format. This statement is most useful for setting a default from address for messages sent by monit, like so:
set mail-format { from: monit@foo.bar.no }
Monit by default sends just one error notification when the service failed and another one when it has recovered. If you want to be notified more then once in the case that the service remains failed, you can use the reminder option of alert statement (keywords are in capital):
For example if you want to be notified each tenth cycle when the service remains failed, you can use:
alert foo@bar with reminder on 10 cycles
If you want to be notified on each failed cycle, you can use:
alert foo@bar with reminder on 1 cycle
The mail server monit should use to send alert messages is defined with a global set statement (keywords are in capital and optional statements in [brackets]):
SET MAILSERVER {host name [PORT port]|ip-address [PORT port]}+
[with TIMEOUT X SECONDS]
The port statement allows to use SMTP servers other then those listening on port 25. If omitted, port 25 is used for the connection.
As you can see, it is possible to set several SMTP servers. If monit cannot connect to the first server in the list it will try the second server and so on. Monit has a default 5 seconds connection timeout and if the SMTP server is slow, monit could timeout when connecting or reading from the server. You can use the optional timeout statement to explicit set the timeout to a higher value if needed. Here is an example for setting several mail servers:
set mailserver mail.tildeslash.com, mail.foo.bar port 10025,
localhost with timeout 15 seconds
Here monit will first try to connect to the server ``mail.tildeslash.com'', if this server is down monit will try ``mail.foo.bar'' on port 10025 and finally ``localhost''. We do also set an explicit connect and read timeout; If monit cannot connect to the first SMTP server in the list within 15 seconds it will try the next server and so on. The set mailserver .. statement is optional and if not defined monit defaults to use localhost as the SMTP server.
Monit provide optionally queueing of event alerts that cannot be sent. For example, if no mail-server is available at the moment, monit can store events in a queue and try to reprocess them at the next cycle. As soon as the mail-server recover, monit will post the queued events. The queue is persistent across monit restarts and provided that the back-end filesystem is persistent too, across system restart as well.
By default, the queue is disabled and if the alert handler fails, monit will simply drop the alert message. To enable the event queue, add the following statement to the monit control file:
SET EVENTQUEUE BASEDIR <path> [SLOTS <number>]
The <path> is the path to the directory where events will be stored. Optionaly if you want to limit the queue size (maximum events count), use the slots option. If the slots option is not used, monit will store as many events as the backend filesystem allows.
Example:
set eventqueue
basedir /var/monit
slots 5000
The events are stored in binary format, one file per event. The file size is ca. 130 bytes or a bit more (depending on the message length). The file name is composed of the unix timestamp, underscore and the service name, for example:
/var/monit/1131269471_apache
If you are running more then one monit instance on the same machine, you must use separated event queue directories to avoid sending wrong alerts to the wrong addresses.
If you want to purge the queue by hand (remove queued event-files), monit should be stopped before the removal.
monit provides a service timeout mechanism for situations where a service simply refuses to start or respond over a longer period. In cases like this, and particularly if monit's poll-cycle is low, monit will simply increase the machine load by trying to restart the service.
The timeout mechanism monit provides is based on two variables, i.e. the number the service has been started and the number of poll-cycles. For example, if a service had x restarts within y poll-cycles (where x <= y) then monit will timeout and not (re)start the service on the next cycle. If a timeout occurs monit will send you an alert message if you have register interest for this event.
The syntax for the timestamp statement is as follows (keywords are in capital):
CYCLE(S) THEN TIMEOUTWhere the first number is the number of service restarts and the second, the number of poll-cycles. If the number of cycles was reached without a timeout, the service start-counter is reset to zero. This provides some granularity to catch exceptional cases and do a service timeout, but let occasional service start and restarts happen without having an accumulated timeout.
Here is an example where monit will timeout (not check the service) if the service was restarted 2 times within 3 cycles:
if 2 restarts within 3 cycles then timeout
To have monit check the service again after a timeout, run 'monit monitor service' from the command line. This will remove the timeout lock in the daemon and make the daemon start and check the service again.
Monit provides several tests you may utilize in a service entry to test a service. Basicaly here are two classes of tests: variable and constant object tests.
Constant object tests are related to failed/passed state. In the case of error, monit will watch whether the failed parameter will recover - in such case it will handle recovery related action. General format:
For constant object tests if the <TEST> should validate to true, then the selected action is executed each cycle the condition remains true. The value for comparision is constant. Recovery action is evalueated only once (on failed->passed state change only). The 'ELSE IF PASSED' part is optional - if omitted, monit will do alert action on recovery by default. The alert is delivered only once on each state change unless overriden by 'reminder' alert option.
Variable object tests beggins with 'IF CHANGED' statement and serves for monitoring of object, which property can change legaly - monit watches whether the value will change again. You can use it just for alert or to involve some automatic action, as for example to reload monitored process after its configuration file was changed. Variable thin_5d_">">Setting a global alert statement
If a change occured on a monitored services, monit will send an alert to all recipients in the global list who have registred interest for the event type. Here is the syntax for the global alert statement:
Simply using the following in the global section of monitrc:
set alert foo@bar
will send a default email to the address foo@bar whenever an event occurred on any service. Such an event may be that a service timed out, a service was doesn't exist or a service does exist (on recovery) and so on. If you want to send alert messages to more email addresses, add a set alert 'email' statement for each address.
For explanations of the events, MAIL-FORMAT and REMINDER keywords above, please see below.
Each service can also have its own recipient list.
If you only want an alert message sent for certain events for certain service(s), for example only for timeout events or only if a service died, then postfix the alert-statement with a filter block:
check process myproc with pidfile /var/run/my.pid
alert foo@bar only on { timeout, nonexist }
...
(only and on are noise keywords, ignored by monit. As a side note; Noise keywords are used in the control file grammar to make an entry resemble English and thus make it easier to read (or, so goes the philosophy). The full set of available noise keywords are listed below in the Control File section).
Event filtering can be used to send a mail to different email addresses depending on the events that occured. For instance:
alert foo@bar { nonexist, timeout, resource, icmp, connection }
alert security@bar on { checksum, permission, uid, gid }
alert manager@bar
This will send an alert message to foo@bar whenever a nonexist, timeout, resource or connection problem occurs and a message to security@bar if a checksum, permission, uid or gid problem occurs. And finally, a message to manager@bar whenever any error event occurs.
This is the list of events you can use in a mail-filter: uid, gid, size, nonexist, data, icmp, instance, invalid, exec, changed, timeout, resource, checksum, match, timestamp, connection, permission
The following example will alert foo@bar on all events on all services by default, except the service mybar which will send an alert only on timeout. The trick is based on the fact that local definition of the same recipient overrides the global setting (including registered events and mail format):
set alert foo@bar
check process myfoo with pidfile /var/run/myfoo.pid
...
check process mybar with pidfile /var/run/mybar.pid
alert foo@bar only on { timeout }
The 'instance' alert type report events related to monit internals, such as when a monit instance was started, stopped or reloaded.
If the MTA (mailserver) for sending alerts is not available, monit can queue events on the local file-system until the MTA recover. Monit will then post queued events in order with theire original timestamp so the events are not lost. This feature is most useful if monit is used together with e.g. m/monit and when event history is important.
monit provides a default mail message layout that is short and to the point. Here's an example of a standard alert mail sent by monit:
From: monit@tildeslash.com Subject: monit alert -- Does not exist apache To: hauk@tildeslash.com Date: Thu, 04 Sep 2003 02:33:03 +0200
Does not exist Service apache
Date: Thu, 04 Sep 2003 02:33:03 +0200
Action: restart
Host: www.tildeslash.com
Your faithful employee, monit
If you want to, you can change the format of this message with the optional mail-format statement. The syntax for this statement is as follows:
mail-format {
from: monit@localhost
subject: $SERVICE $EVENT at $DATE
message: Monit $ACTION $SERVICE at $DATE on $HOST: $DESCRIPTION.
Yours sincerely,
monit
}
Where the keyword from: is the email address monit should pretend it is sending from. It does not have to be a real mail address, but it must be a proper formated mail address, on the form: name@domain. The keyword subject: is for the email subject line. The subject must be on only one line. The message: keyword denotes the mail body. If used, this keyword should always be the last in a mail-format statement. The mail body can be as long as you want and must not contain the '}' character.
All of these format keywords are optional but you must provide at least one. Thus if you only want to change the from address monit is using you can do:
set alert foo@bar with mail-format { from: bofh@bar.baz }
From the previous example you will notice that some special $XXX variables was used. If used, they will be substituted and expanded into the text with these values:
A string describing the event that occurred. The values are fixed and are:
Event: | Failure state: | Recovery state: --------------------------------------------------------------- CHANGED | "Changed" | "Changed back" CHECKSUM | "Checksum failed" | "Checksum passed" CONNECTION| "Connection failed" | "Connection passed" DATA | "Data access error" | "Data access succeeded" EXEC | "Execution failed" | "Execution succeeded" GID | "GID failed" | "GID passed" ICMP | "ICMP failed" | "ICMP passed" INSTANCE | "Monit instance changed"| "Monit instance changed not" INVALID | "Invalid type" | "Type passed" MATCH | "Regex match" | "No regex match" NONEXIST | "Does not exist" | "Exists" PERMISSION| "Permission failed" | "Permission passed" RESOURCE | "Resource limit matched"| "Resource limit passed" SIZE | "Size failed" | "Size passed" TIMEOUT | "Timeout" | "Timeout recovery" TIMESTAMP | "Timestamp failed" | "Timestamp passed" UID | "UID failed" | "UID passed"
The service entry name in monitrc
The current time and date (RFC 822 date style).
The name of the host monit is running on
The name of the action which was done. Action names are fixed and are:
Action: | Name: -------------------- ALERT | "alert" EXEC | "exec" MONITOR | "monitor" RESTART | "restart" START | "start" STOP | "stop" UNMONITOR| "unmonitor"
The description of the error condition
It is possible to set a standard mail format with the following global set-statement (keywords are in capital):
Format set with this statement will apply to every alert statement that does not have its own specified mail-format. This statement is most useful for setting a default from address for messages sent by monit, like so:
set mail-format { from: monit@foo.bar.no }
Monit by default sends just one error notification when the service failed and another one when it has recovered. If you want to be notified more then once in the case that the service remains failed, you can use the reminder option of alert statement (keywords are in capital):
For example if you want to be notified each tenth cycle when the service remains failed, you can use:
alert foo@bar with reminder on 10 cycles
If you want to be notified on each failed cycle, you can use:
alert foo@bar with reminder on 1 cycle
The mail server monit should use to send alert messages is defined with a global set statement (keywords are in capital and optional statements in [brackets]):
SET MAILSERVER {host name [PORT port]|ip-address [PORT port]}+
[with TIMEOUT X SECONDS]
The port statement allows to use SMTP servers other then those listening on port 25. If omitted, port 25 is used for the connection.
As you can see, it is possible to set several SMTP servers. If monit cannot connect to the first server in the list it will try the second server and so on. Monit has a default 5 seconds connection timeout and if the SMTP server is slow, monit could timeout when connecting or reading from the server. You can use the optional timeout statement to explicit set the timeout to a higher value if needed. Here is an example for setting several mail servers:
set mailserver mail.tildeslash.com, mail.foo.bar port 10025,
localhost with timeout 15 seconds
Here monit will first try to connect to the server ``mail.tildeslash.com'', if this server is down monit will try ``mail.foo.bar'' on port 10025 and finally ``localhost''. We do also set an explicit connect and read timeout; If monit cannot connect to the first SMTP server in the list within 15 seconds it will try the next server and so on. The set mailserver .. statement is optional and if not defined monit defaults to use localhost as the SMTP server.
Monit provide optionally queueing of event alerts that cannot be sent. For example, if no mail-server is available at the moment, monit can store events in a queue and try to reprocess them at the next cycle. As soon as the mail-server recover, monit will post the queued events. The queue is persistent across monit restarts and provided that the back-end filesystem is persistent too, across system restart as well.
By default, the queue is disabled and if the alert handler fails, monit will simply drop the alert message. To enable the event queue, add the following statement to the monit control file:
SET EVENTQUEUE BASEDIR <path> [SLOTS <number>]
The <path> is the path to the directory where events will be stored. Optionaly if you want to limit the queue size (maximum events count), use the slots option. If the slots option is not used, monit will store as many events as the backend filesystem allows.
Example:
set eventqueue
basedir /var/monit
slots 5000
The events are stored in binary format, one file per event. The file size is ca. 130 bytes or a bit more (depending on the message length). The file name is composed of the unix timestamp, underscore and the service name, for example:
/var/monit/1131269471_apache
If you are running more then one monit instance on the same machine, you must use separated event queue directories to avoid sending wrong alerts to the wrong addresses.
If you want to purge the queue by hand (remove queued event-files), monit should be stopped before the removal.
monit provides a service timeout mechanism for situations where a service simply refuses to start or respond over a longer period. In cases like this, and particularly if monit's poll-cycle is low, monit will simply increase the machine load by trying to restart the service.
The timeout mechanism monit provides is based on two variables, i.e. the number the service has been started and the number of poll-cycles. For example, if a service had x restarts within y poll-cycles (where x <= y) then monit will timeout and not (re)start the service on the next cycle. If a timeout occurs monit will send you an alert message if you have register interest for this event.
The syntax for the timestamp statement is as follows (keywords are in capital):
CYCLE(S) THEN TIMEOUTWhere the first number is the number of service restarts and the second, the number of poll-cycles. If the number of cycles was reached without a timeout, the service start-counter is reset to zero. This provides some granularity to catch exceptional cases and do a service timeout, but let occasional service start and restarts happen without having an accumulated timeout.
Here is an example where monit will timeout (not check the service) if the service was restarted 2 times within 3 cycles:
if 2 restarts within 3 cycles then timeout
To have monit check the service again after a timeout, run 'monit monitor service' from the command line. This will remove the timeout lock in the daemon and make the daemon start and check the service again.
Monit provides several tests you may utilize in a service entry to test a service. Basicaly here are two classes of tests: variable and constant object tests.
Constant object tests are related to failed/passed state. In the case of error, monit will watch whether the failed parameter will recover - in such case it will handle recovery related action. General format:
For constant object tests if the <TEST> should validate to true, then the selected action is executed each cycle the condition remains true. The value for comparision is constant. Recovery action is evalueated only once (on failed->passed state change only). The 'ELSE IF PASSED' part is optional - if omitted, monit will do alert action on recovery by default. The alert is delivered only once on each state change unless overriden by 'reminder' alert option.
Variable object tests beggins with 'IF CHANGED' statement and serves for monitoring of object, which property can change legaly - monit watches whether the value will change again. You can use it just for alert or to involve some automatic action, as for example to reload monitored process after its configuration file was changed. Variable thin_5d_">">Setting a global alert statement
If a change occured on a monitored services, monit will send an alert to all recipients in the global list who have registred interest for the event type. Here is the syntax for the global alert statement:
Simply using the following in the global section of monitrc:
set alert foo@bar
will send a default email to the address foo@bar whenever an event occurred on any service. Such an event may be that a service timed out, a service was doesn't exist or a service does exist (on recovery) and so on. If you want to send alert messages to more email addresses, add a set alert 'email' statement for each address.
For explanations of the events, MAIL-FORMAT and REMINDER keywords above, please see below.
Each service can also have its own recipient list.
If you only want an alert message sent for certain events for certain service(s), for example only for timeout events or only if a service died, then postfix the alert-statement with a filter block:
check process myproc with pidfile /var/run/my.pid
alert foo@bar only on { timeout, nonexist }
...
(only and on are noise keywords, ignored by monit. As a side note; Noise keywords are used in the control file grammar to make an entry resemble English and thus make it easier to read (or, so goes the philosophy). The full set of available noise keywords are listed below in the Control File section).
Event filtering can be used to send a mail to different email addresses depending on the events that occured. For instance:
alert foo@bar { nonexist, timeout, resource, icmp, connection }
alert security@bar on { checksum, permission, uid, gid }
alert manager@bar
This will send an alert message to foo@bar whenever a nonexist, timeout, resource or connection problem occurs and a message to security@bar if a checksum, permission, uid or gid problem occurs. And finally, a message to manager@bar whenever any error event occurs.
This is the list of events you can use in a mail-filter: uid, gid, size, nonexist, data, icmp, instance, invalid, exec, changed, timeout, resource, checksum, match, timestamp, connection, permission
The following example will alert foo@bar on all events on all services by default, except the service mybar which will send an alert only on timeout. The trick is based on the fact that local definition of the same recipient overrides the global setting (including registered events and mail format):
set alert foo@bar
check process myfoo with pidfile /var/run/myfoo.pid
...
check process mybar with pidfile /var/run/mybar.pid
alert foo@bar only on { timeout }
The 'instance' alert type report events related to monit internals, such as when a monit instance was started, stopped or reloaded.
If the MTA (mailserver) for sending alerts is not available, monit can queue events on the local file-system until the MTA recover. Monit will then post queued events in order with theire original timestamp so the events are not lost. This feature is most useful if monit is used together with e.g. m/monit and when event history is important.
monit provides a default mail message layout that is short and to the point. Here's an example of a standard alert mail sent by monit:
From: monit@tildeslash.com Subject: monit alert -- Does not exist apache To: hauk@tildeslash.com Date: Thu, 04 Sep 2003 02:33:03 +0200
Does not exist Service apache
Date: Thu, 04 Sep 2003 02:33:03 +0200
Action: restart
Host: www.tildeslash.com
Your faithful employee, monit
If you want to, you can change the format of this message with the optional mail-format statement. The syntax for this statement is as follows:
mail-format {
from: monit@localhost
subject: $SERVICE $EVENT at $DATE
message: Monit $ACTION $SERVICE at $DATE on $HOST: $DESCRIPTION.
Yours sincerely,
monit
}
Where the keyword from: is the email address monit should pretend it is sending from. It does not have to be a real mail address, but it must be a proper formated mail address, on the form: name@domain. The keyword subject: is for the email subject line. The subject must be on only one line. The message: keyword denotes the mail body. If used, this keyword should always be the last in a mail-format statement. The mail body can be as long as you want and must not contain the '}' character.
All of these format keywords are optional but you must provide at least one. Thus if you only want to change the from address monit is using you can do:
set alert foo@bar with mail-format { from: bofh@bar.baz }
From the previous example you will notice that some special $XXX variables was used. If used, they will be substituted and expanded into the text with these values:
A string describing the event that occurred. The values are fixed and are:
Event: | Failure state: | Recovery state: --------------------------------------------------------------- CHANGED | "Changed" | "Changed back" CHECKSUM | "Checksum failed" | "Checksum passed" CONNECTION| "Connection failed" | "Connection passed" DATA | "Data access error" | "Data access succeeded" EXEC | "Execution failed" | "Execution succeeded" GID | "GID failed" | "GID passed" ICMP | "ICMP failed" | "ICMP passed" INSTANCE | "Monit instance changed"| "Monit instance changed not" INVALID | "Invalid type" | "Type passed" MATCH | "Regex match" | "No regex match" NONEXIST | "Does not exist" | "Exists" PERMISSION| "Permission failed" | "Permission passed" RESOURCE | "Resource limit matched"| "Resource limit passed" SIZE | "Size failed" | "Size passed" TIMEOUT | "Timeout" | "Timeout recovery" TIMESTAMP | "Timestamp failed" | "Timestamp passed" UID | "UID failed" | "UID passed"
The service entry name in monitrc
The current time and date (RFC 822 date style).
The name of the host monit is running on
The name of the action which was done. Action names are fixed and are:
Action: | Name: -------------------- ALERT | "alert" EXEC | "exec" MONITOR | "monitor" RESTART | "restart" START | "start" STOP | "stop" UNMONITOR| "unmonitor"
The description of the error condition
It is possible to set a standard mail format with the following global set-statement (keywords are in capital):
Format set with this statement will apply to every alert statement that does not have its own specified mail-format. This statement is most useful for setting a default from address for messages sent by monit, like so:
set mail-format { from: monit@foo.bar.no }
Monit by default sends just one error notification when the service failed and another one when it has recovered. If you want to be notified more then once in the case that the service remains failed, you can use the reminder option of alert statement (keywords are in capital):
For example if you want to be notified each tenth cycle when the service remains failed, you can use:
alert foo@bar with reminder on 10 cycles
If you want to be notified on each failed cycle, you can use:
alert foo@bar with reminder on 1 cycle
The mail server monit should use to send alert messages is defined with a global set statement (keywords are in capital and optional statements in [brackets]):
SET MAILSERVER {host name [PORT port]|ip-address [PORT port]}+
[with TIMEOUT X SECONDS]
The port statement allows to use SMTP servers other then those listening on port 25. If omitted, port 25 is used for the connection.
As you can see, it is possible to set several SMTP servers. If monit cannot connect to the first server in the list it will try the second server and so on. Monit has a default 5 seconds connection timeout and if the SMTP server is slow, monit could timeout when connecting or reading from the server. You can use the optional timeout statement to explicit set the timeout to a higher value if needed. Here is an example for setting several mail servers:
set mailserver mail.tildeslash.com, mail.foo.bar port 10025,
localhost with timeout 15 seconds
Here monit will first try to connect to the server ``mail.tildeslash.com'', if this server is down monit will try ``mail.foo.bar'' on port 10025 and finally ``localhost''. We do also set an explicit connect and read timeout; If monit cannot connect to the first SMTP server in the list within 15 seconds it will try the next server and so on. The set mailserver .. statement is optional and if not defined monit defaults to use localhost as the SMTP server.
Monit provide optionally queueing of event alerts that cannot be sent. For example, if no mail-server is available at the moment, monit can store events in a queue and try to reprocess them at the next cycle. As soon as the mail-server recover, monit will post the queued events. The queue is persistent across monit restarts and provided that the back-end filesystem is persistent too, across system restart as well.
By default, the queue is disabled and if the alert handler fails, monit will simply drop the alert message. To enable the event queue, add the following statement to the monit control file:
SET EVENTQUEUE BASEDIR <path> [SLOTS <number>]
The <path> is the path to the directory where events will be stored. Optionaly if you want to limit the queue size (maximum events count), use the slots option. If the slots option is not used, monit will store as many events as the backend filesystem allows.
Example:
set eventqueue
basedir /var/monit
slots 5000
The events are stored in binary format, one file per event. The file size is ca. 130 bytes or a bit more (depending on the message length). The file name is composed of the unix timestamp, underscore and the service name, for example:
/var/monit/1131269471_apache
If you are running more then one monit instance on the same machine, you must use separated event queue directories to avoid sending wrong alerts to the wrong addresses.
If you want to purge the queue by hand (remove queued event-files), monit should be stopped before the removal.
monit provides a service timeout mechanism for situations where a service simply refuses to start or respond over a longer period. In cases like this, and particularly if monit's poll-cycle is low, monit will simply increase the machine load by trying to restart the service.
The timeout mechanism monit provides is based on two variables, i.e. the number the service has been started and the number of poll-cycles. For example, if a service had x restarts within y poll-cycles (where x <= y) then monit will timeout and not (re)start the service on the next cycle. If a timeout occurs monit will send you an alert message if you have register interest for this event.
The syntax for the timestamp statement is as follows (keywords are in capital):
CYCLE(S) THEN TIMEOUTWhere the first number is the number of service restarts and the second, the number of poll-cycles. If the number of cycles was reached without a timeout, the service start-counter is reset to zero. This provides some granularity to catch exceptional cases and do a service timeout, but let occasional service start and restarts happen without having an accumulated timeout.
Here is an example where monit will timeout (not check the service) if the service was restarted 2 times within 3 cycles:
if 2 restarts within 3 cycles then timeout
To have monit check the service again after a timeout, run 'monit monitor service' from the command line. This will remove the timeout lock in the daemon and make the daemon start and check the service again.
Monit provides several tests you may utilize in a service entry to test a service. Basicaly here are two classes of tests: variable and constant object tests.
Constant object tests are related to failed/passed state. In the case of error, monit will watch whether the failed parameter will recover - in such case it will handle recovery related action. General format:
For constant object tests if the <TEST> should validate to true, then the selected action is executed each cycle the condition remains true. The value for comparision is constant. Recovery action is evalueated only once (on failed->passed state change only). The 'ELSE IF PASSED' part is optional - if omitted, monit will do alert action on recovery by default. The alert is delivered only once on each state change unless overriden by 'reminder' alert option.
Variable object tests beggins with 'IF CHANGED' statement and serves for monitoring of object, which property can change legaly - monit watches whether the value will change again. You can use it just for alert or to involve some automatic action, as for example to reload monitored process after its configuration file was changed. Variable thin_5d_">">Setting a global alert statement
If a change occured on a monitored services, monit will send an alert to all recipients in the global list who have registred interest for the event type. Here is the syntax for the global alert statement:
Simply using the following in the global section of monitrc:
set alert foo@bar
will send a default email to the address foo@bar whenever an event occurred on any service. Such an event may be that a service timed out, a service was doesn't exist or a service does exist (on recovery) and so on. If you want to send alert messages to more email addresses, add a set alert 'email' statement for each address.
For explanations of the events, MAIL-FORMAT and REMINDER keywords above, please see below.
Each service can also have its own recipient list.
If you only want an alert message sent for certain events for certain service(s), for example only for timeout events or only if a service died, then postfix the alert-statement with a filter block:
check process myproc with pidfile /var/run/my.pid
alert foo@bar only on { timeout, nonexist }
...
(only and on are noise keywords, ignored by monit. As a side note; Noise keywords are used in the control file grammar to make an entry resemble English and thus make it easier to read (or, so goes the philosophy). The full set of available noise keywords are listed below in the Control File section).
Event filtering can be used to send a mail to different email addresses depending on the events that occured. For instance:
alert foo@bar { nonexist, timeout, resource, icmp, connection }
alert security@bar on { checksum, permission, uid, gid }
alert manager@bar
This will send an alert message to foo@bar whenever a nonexist, timeout, resource or connection problem occurs and a message to security@bar if a checksum, permission, uid or gid problem occurs. And finally, a message to manager@bar whenever any error event occurs.
This is the list of events you can use in a mail-filter: uid, gid, size, nonexist, data, icmp, instance, invalid, exec, changed, timeout, resource, checksum, match, timestamp, connection, permission
The following example will alert foo@bar on all events on all services by default, except the service mybar which will send an alert only on timeout. The trick is based on the fact that local definition of the same recipient overrides the global setting (including registered events and mail format):
set alert foo@bar
check process myfoo with pidfile /var/run/myfoo.pid
...
check process mybar with pidfile /var/run/mybar.pid
alert foo@bar only on { timeout }
The 'instance' alert type report events related to monit internals, such as when a monit instance was started, stopped or reloaded.
If the MTA (mailserver) for sending alerts is not available, monit can queue events on the local file-system until the MTA recover. Monit will then post queued events in order with theire original timestamp so the events are not lost. This feature is most useful if monit is used together with e.g. m/monit and when event history is important.
monit provides a default mail message layout that is short and to the point. Here's an example of a standard alert mail sent by monit:
From: monit@tildeslash.com Subject: monit alert -- Does not exist apache To: hauk@tildeslash.com Date: Thu, 04 Sep 2003 02:33:03 +0200
Does not exist Service apache
Date: Thu, 04 Sep 2003 02:33:03 +0200
Action: restart
Host: www.tildeslash.com
Your faithful employee, monit
If you want to, you can change the format of this message with the optional mail-format statement. The syntax for this statement is as follows:
mail-format {
from: monit@localhost
subject: $SERVICE $EVENT at $DATE
message: Monit $ACTION $SERVICE at $DATE on $HOST: $DESCRIPTION.
Yours sincerely,
monit
}
Where the keyword from: is the email address monit should pretend it is sending from. It does not have to be a real mail address, but it must be a proper formated mail address, on the form: name@domain. The keyword subject: is for the email subject line. The subject must be on only one line. The message: keyword denotes the mail body. If used, this keyword should always be the last in a mail-format statement. The mail body can be as long as you want and must not