monit - utility for monitoring services on a Unix system
monit [options] {arguments}
monit is a utility for managing and monitoring processes, files, directories and devices on a Unix system. Monit conducts automatic maintenance and repair and can execute meaningful causal actions in error situations. E.g. monit can start a process if it does not run, restart a process if it does not respond and stop a process if it uses to much resources. You may use monit to monitor files, directories and devices for changes, such as timestamps changes, checksum changes or size changes.
Monit is controlled via an easy to configure control file based
on a free-format, token-oriented syntax. Monit logs to syslog or
to its own log file and notifies you about error conditions via
customizable alert messages. Monit can perform various TCP/IP
network checks, protocol checks and can utilize SSL for such
checks. Monit provides a http(s) interface and you may use a
browser to access the monit program.
The behavior of monit is controlled by command-line options and a run control file, ~/.monitrc, the syntax of which we describe in a later section. Command-line options override .monitrc declarations.
The following options are recognized by monit. However, it is recommended that you set options (when applicable) directly in the .monitrc control file.
-c file Use this control file
-d n Run as a daemon once per n seconds
-g Set group name for start, stop, restart and status
-l logfile Print log information to this file
-p pidfile Use this lock file in daemon mode
-s statefile Write state information to this file
-I Do not run in background (needed for run from init)
-t Run syntax check for the control file
-v Verbose mode, work noisy (diagnostic output)
-H [filename] Print MD5 and SHA1 hashes of the file or of stdin if the filename is omitted; monit will exit afterwards
-V Print version number and patch level
-h Print a help text
In addition to the options above, monit can be started with one of the following action arguments; monit will then execute the action and exit without transforming itself to a daemon.
start all Start all services listed in the control file and enable monitoring for them. If the group option is set, only start and enable monitoring of services in the named group.
start name Start the named service and enable monitoring for it. The name is a service entry name from the monitrc file.
stop all Stop all services listed in the control file and disable their monitoring. If the group option is set, only stop and disable monitoring of the services in the named group.
stop name Stop the named service and disable its monitoring. The name is a service entry name from the monitrc file.
restart all Stop and start all services. If the group option is set, only restart the services in the named group.
restart name Restart the named service. The name is a service entry name from the monitrc file.
monitor all Enable monitoring of all services listed in the control file. If the group option is set, only start monitoring of services in the named group.
monitor name Enable monitoring of the named service. The name is a service entry name from the monitrc file. Monit will also enable monitoring of all services this service depends on.
unmonitor all Disable monitoring of all services listed in the control file. If the group option is set, only disable monitoring of services in the named group.
unmonitor name Disable monitoring of the named service. The name is a service entry name from the monitrc file. Monit will also disable monitoring of all services that depends on this service.
status Print full status information for each service.
summary Print short status information for each service.
reload Reinitialize a running monit daemon, the daemon will reread its configuration, close and reopen log files.
quit Kill a monit daemon process
validate Check all services listed in the control file. This action is also the default behavior when monit runs in daemon mode.
You may use monit to monitor daemon processes or similar programs running on localhost. Monit is particular useful for monitoring daemon processes, such as those started at system boot time from /etc/init.d/. For instance sendmail, sshd, apache and mysql. In difference to many monitoring systems, monit can act if an error situation should occur, e.g.; if sendmail is not running, monit can start sendmail or if apache is using to much system resources (e.g. if a DoS attack is in progress) monit can stop or restart apache and send you an alert message. Monit does also monitor process characteristics, such as; if a process has become a zombie and how much memory or cpu cycles a process is using.
You may also use monit to monitor files, directories and devices on localhost. Monit can monitor these items for changes, such as timestamps changes, checksum changes or size changes. This is also useful for security reasons - you can monitor the md5 checksum of files that should not change.
You may even use monit to monitor remote hosts. First and foremost monit is a utility for monitoring and mending services on localhost, but if a service depends on a remote service, e.g. a database server or an application server, it might by useful to be able to test a remote host as well.
You may monitor the general system-wide resources such as cpu usage, memory and load average.
monit is configured and controlled via a control file called monitrc. The default location for this file is ~/.monitrc. If this file does not exist, monit will try /etc/monitrc, then /usr/local/etc/monitrc and finally ./monitrc.
A monit control file consists of a series of service entries and global option statements in a free-format, token-oriented syntax. Comments begin with a # and extend through the end of the line. There are three kinds of tokens in the control file: grammar keywords, numbers and strings.
On a semantic level, the control file consists of three types of statements:
This is the hello galaxy version of a monit control file:
# # monit control file #
set daemon 120 # Poll at 2-minute intervals
set logfile syslog facility log_daemon
set alert foo@bar.baz
set httpd port 2812 and use address localhost
allow localhost # Allow localhost to connect
allow admin:monit # Allow Basic Auth
check system myhost.mydomain.tld
if loadavg (1min) > 4 then alert
if loadavg (5min) > 2 then alert
if memory usage > 75% then alert
if cpu usage (user) > 70% then alert
if cpu usage (system) > 30% then alert
if cpu usage (wait) > 20% then alert
check process apache
with pidfile "/usr/local/apache/logs/httpd.pid"
start program = "/etc/init.d/httpd start"
stop program = "/etc/init.d/httpd stop"
if 2 restarts within 3 cycles then timeout
if totalmem > 100 Mb then alert
if children > 255 for 5 cycles then stop
if cpu usage > 95% for 3 cycles then restart
if failed port 80 protocol http then restart
group server
depends on httpd.conf, httpd.bin
check file httpd.conf
with path /usr/local/apache/conf/httpd.conf
# Reload apache if the httpd.conf file was changed
if changed checksum
then exec "/usr/local/apache/bin/apachectl graceful"
check file httpd.bin
with path /usr/local/apache/bin/httpd
# Run /watch/dog in the case that the binary was changed
# and alert in the case that the checksum value recovered
# later
if failed checksum then exec "/watch/dog"
else if recovered then alert
include /etc/monit/mysql.monitrc include /etc/monit/mail/*.monitrc
This example illustrate a service entry for monitoring the apache web server process as well as related files. The meaning of the various statements will be explained in the following sections.
monit will log status and error messages to a log file. Use the set logfile statement in the monitrc control file. To setup monit to log to its own logfile, use e.g. set logfile /var/log/monit.log. If syslog is given as a value for the -l command-line switch (or the keyword set logfile syslog is found in the control file) monit will use the syslog system daemon to log messages. To turn off logging, simply do not set the logfile in the control file (and of course, do not use the -l switch)
The -d interval command-line switch runs monit in daemon mode. You must specify a numeric argument which is a polling interval in seconds.
In daemon mode, monit detaches from the console, puts itself in the background and runs continuously, monitoring each specified service and then goes to sleep for the given poll interval.
Simply invoking
monit -d 300
will poll all services described in your ~/.monitrc file every 5 minutes.
It is strongly recommended to set the poll interval in your ~/.monitrc file instead, by using set daemon n, where n is an integer number of seconds. If you do this, monit will always start in daemon mode (as long as no action arguments are given).
Monit makes a per-instance lockfile in daemon mode. If you need more monit instances, you will need more configuration files, each pointing to its own lockfile.
Calling monit with a monit daemon running in the background sends a wakeup signal to the daemon, forcing it to check services immediately.
The quit argument will kill a running daemon process instead of waking it up.
Monit can run and be controlled from init. If monit should crash, init will re-spawn a new monit process. Using init to start monit is probably the best way to run monit if you want to be certain that you always have a running monit daemon on your system. (It's obvious, but never the less worth to stress; Make sure that the control file does not have any syntax errors before you start monit from init. Also, make sure that if you run monit from init, that you do not start monit from a startup scripts as well).
To setup monit to run from init, you can either use the 'set init' statement in monit's control file or use the -I option from the command line and here is what you must add to /etc/inittab:
# Run monit in standard run-levels mo:2345:respawn:/usr/local/bin/monit -Ic /etc/monitrc
After you have modified init's configuration file, you can run the following command to re-examine /etc/inittab and start monit:
telinit q For systems without telinit:
kill -1 1
If monit is used to monitor services that are also started at boot time (e.g. services started via SYSV init rc scripts or via inittab) then, in some cases, a race condition could occur. That is; if a service is slow to start, monit can assume that the service is not running and possibly try to start it and raise an alert, while, in fact the service is already about to start or already in its startup sequence. Please see the FAQ for solutions to this problem.
The monit control file, monitrc, can include additional configuration files. This feature helps to maintain a certain structure or to place repeating settings into one file. Include statements can be placed at virtually any spot. The syntax is the following:
INCLUDE globstring
The globstring is any kind of string as defined in glob(7). Thus, you can refer to a single file or you can load several files at once. In case you want to use whitespace in your string the globstring need to be embedded into quotes (') or double quotes (``). For example,
INCLUDE "/etc/monit/monit configuration files/printer.*.monitrc"
loads any file matching the single globstring. If the globstring matches a directory instead of a file, it is silently ignored.
INCLUDE statements in included files are parsed as in the main control file.
If the globstring matches several results, the files are included in a non sorted manner. If you need to rely on a certain order, you might need to use single include statements.
Service entries in the control file, monitrc, can be grouped together by the group statement. The syntax is simply (keyword in capital):
GROUP groupname
With this statement it is possible to group similar service entries together and manage them as a whole. Monit provides functions to start, stop and restart a group of services, like so:
To start a group of services from the console:
monit -g <groupname> start
To stop a group of services:
monit -g <groupname> stop
To restart a group of services:
monit -g <groupname> restart
Monit supports three monitoring modes per service: active, passive and manual. See also the example section below for usage of the mode statement.
In active mode, monit will monitor a service and in case of problems monit will act and raise alerts, start, stop or restart the service. Active mode is the default mode.
In passive mode, monit will passively monitor a service and specifically not try to fix a problem, but it will still raise alerts in case of a problem.
For use in clustered environments there is also a manual mode. In this mode, monit will enter active mode only if a service was brought under monit's control, for example by executing the following command in the console:
monit start sybase (monit will call sybase's start method and enable monitoring)
If a service was not started by monit or was stopped or disabled for example by:
monit stop sybase (monit will call sybase's stop method and disable monitoring)
monit will not monitor the service. This allows for having services configured in monitrc and start it with monit only if it should run. This feature can be used to build a simple failsafe cluster. To see how, read more about how to setup a cluster with monit using the heartbeat system in the examples sections below.
Monit will raise an email alert in the following situations:
o A service timed out o A service does not exist o A service related data access problem o A service related program execution problem o A service is of invalid object type o A icmp problem o A port connection problem o A resource statement match o A file checksum problem o A file size problem o A file/directory timestamp problem o A file/directory/device permission problem o A file/directory/device uid problem o A file/directory/device gid problem
Monit will send an alert each time a monitored object changed. This involves:
o Monit started, stopped or reloaded o A file checksum changed o A file size changed o A file content match o A file/directory timestamp changed
You use the alert statement to notify monit that you want alert messages sent to an email address. If you do not specify an alert statement, monit will not send alert messages.
There are two forms of alert statement:
o Global - common for all services o Local - per service
In both cases you can use more than one alert statement. In other words, you can send many different emails to many different addresses. (in case you now got a new business idea; monit is not really suitable for sending spam).
Recipients in the global and in the local lists are alerted when a service failed, recovered or changed. If the same email address is in the global and in the local list, monit will send only one alert. Local (per service) defined alert email addresses override global addresses in case of a conflict. Finally, you may choose to only use a global alert list (recommended), a local per service list or both.
If a change occured on a monitored services, monit will send an alert to all recipients in the global list who have registred interest for the event type. Here is the syntax for the global alert statement:
Simply using the following in the global section of monitrc:
set alert foo@bar
will send a default email to the address foo@bar whenever an event occurred on any service. Such an event may be that a service timed out, a service was doesn't exist or a service does exist (on recovery) and so on. If you want to send alert messages to more email addresses, add a set alert 'email' statement for each address.
For explanations of the events, MAIL-FORMAT and REMINDER keywords above, please see below.
Each service can also have its own recipient list.
If you only want an alert message sent for certain events for certain service(s), for example only for timeout events or only if a service died, then postfix the alert-statement with a filter block:
check process myproc with pidfile /var/run/my.pid
alert foo@bar only on { timeout, nonexist }
...
(only and on are noise keywords, ignored by monit. As a side note; Noise keywords are used in the control file grammar to make an entry resemble English and thus make it easier to read (or, so goes the philosophy). The full set of available noise keywords are listed below in the Control File section).
Event filtering can be used to send a mail to different email addresses depending on the events that occured. For instance:
alert foo@bar { nonexist, timeout, resource, icmp, connection }
alert security@bar on { checksum, permission, uid, gid }
alert manager@bar
This will send an alert message to foo@bar whenever a nonexist, timeout, resource or connection problem occurs and a message to security@bar if a checksum, permission, uid or gid problem occurs. And finally, a message to manager@bar whenever any error event occurs.
This is the list of events you can use in a mail-filter: uid, gid, size, nonexist, data, icmp, instance, invalid, exec, changed, timeout, resource, checksum, match, timestamp, connection, permission
The following example will alert foo@bar on all events on all services by default, except the service mybar which will send an alert only on timeout. The trick is based on the fact that local definition of the same recipient overrides the global setting (including registered events and mail format):
set alert foo@bar check process myfoo with pidfile /var/run/myfoo.pid ... check pro