\documentclass{book}
\usepackage{index}
\makeindex
\begin{document}
\newcommand{\unknowncharacter}[1]{}
\author{Thomas Erskine}
\title{Remstats 1.0.12b}
\date{Thu Jan 23 11:07:51 EST 2003}
\maketitle
\tableofcontents
%------------------------------------ index.pod ---

\chapter{About Remstats}
\index{About Remstats}

Remstats is a system of programs to:

\begin{itemize}


\item  gather data from servers and routers,

\item  store and maintain the data for long periods,

\item  produce graphs and web-pages tieing them together, and

\item  monitor the data for anomalous behavious and issue alerts

\end{itemize}
It's built on
RRDtool (see \textbf{http://ee-staff.ethz.ch/\~{}oetiker/webtools/rrdtool/}).
There is a proto-FAQ (see the proto-FAQ section) ; feel free to contribute.  There's
also a to-do list (see the to-do list section)  to give some idea what might be coming.

\subsection{Where to get it}%
\index{Where to get it}

The best available version is 1.0.9b.  You can get it at 
http://remstats.sourceforge.net/release/src/remstats-1.0.9b.tar.gz (see \textbf{http://remstats.sourceforge.net/release/src/remstats-1.0.9b.tar.gz}).
The current version is 1.0.13b.  (This version may not be
available yet, as I will push out new documentation before the
new release, to make additions/corrections available as soon 
as possible.)  You can get other versions of remstats from the 
source archive (see \textbf{http://remstats.sourceforge.net/release/src}).  Since almost all of it is 
written in perl (see \textbf{http://www.perl.org/}) scripts, there is no binary version.  

\subsection{How to get started}%
\index{How to get started}

First, you should make sure that you have all the requirements (see the requirements section) .
Then read the installation docs (see the installation docs section) .  Then read the 
server installation docs (see the server installation docs section) .  

Then check out run-remstats (see the run-remstats section)  which runs almost everything else and documents
how all the pieces work together.

Thank-you (see the Thank-you section) .
%------------------------------------ architecture.pod ---

\section{Architecture }%
\index{Architecture }

This will tell how the major pieces of remstats work together, when I finish it.

\subsection{A Bit of History}%
\index{A Bit of History}

There were several things which decided how remstats would work.  One of the
major ones was, of course, RRDtool (see \textbf{http://people.ee.ethz.ch/\~{}oetiker/webtools/rrdtool/}).  I wanted to have good
graphs, without having to build them from scratch.  I liked the side benefit
that RRDs do not grow, lessening the file-management required.

I've dealt with various monitoring packages with long-running daemons which are
always needing to be restarted, or checked via cron jobs and I decided that
remstats wouldn't have that kind of problem.

Back in the bad old days, you couldn't count on having an SNMP agent available
on every platform, so I decided that remstats wouldn't require SNMP to work.
Now, of course, there is the wonderful 
Net-SNMP project (see \textbf{http://net-snmp.sourceforge.net/}) to provide a portable
SNMP implementation.

I wanted it to be easily extendable, without having to completely understand
the guts of remstats.  This goal has not been met very successfully, though
I hope to improve this in the 2.x releases.  However, some folks have managed
to create new collectors, so I guess it wasn't a complete failure.

\subsection{The Collection Process}%
\index{The Collection Process}

The process of data collection in remstats is begun by the run-remstats2 (see the run-remstats2 section) 
program.  (In older versions, run-remstats (see the run-remstats section) , a more primitive program,
filled the same niche.)  It is started by cron at the minimum collection
interval.  It passes through several stages in the order listed in the
run config-file (see the run config-file section) .  Some of these stages may have conditions
on them.  E.G. the standard stage \texttt{pagemakers} will only be run if the
remstats configuration has changed.

Each stage is described by a file in the
run-stages config-dir (see the run-stages config-dir section) , which lists all the programs
to run.  These are all started and run in parallel.  When all the programs
(or pipelines) are finished, run-remstats2 moves to the next stage.

The standard stages are:

\begin{itemize}


\item check - checks the configuration to make sure it's good, and as a side-effect
creates any required new/missing RRDs, other files or directories.

\item ping - runs the ping-collector (see the ping-collector section)  to see which hosts are up, so as to avoid
wasting time on hosts which are down.  Hosts which aren't pinged are considered
to be "up" for collection purposes, to deal with un-pingable hosts.

\item collectors - runs all the other collectors (see the collectors section)  in parallel

\item post-collector - runs the error-collector (see the error-collector section) , which collects info on the collectors.

\item monitors - runs the remstats monitors (see the monitors section)  which evaluate the current status of the
monitored systems and issue alerts, if warranted.

\item pagemakers - runs a motley crew of programs to create various kinds of web pages, but
only if the configuration has changed.  One of the standard cron jobs will
touch the man configuration directory to force page generation once a day.

\end{itemize}
\subsection{Collectors }%
\index{Collectors }

At the centre of remstats are the collectors (see the collectors section) , and their servers (see the servers section) .  The
collectors do whatever they need to do to acquire data, reformat it to the
remstats standard collector form and pass it to an instance of the updater (see the updater section) 
which does the RRD updating.  Some collectors have a corresponding server
which they will contact on remote hosts, and which will provide information
on request.  The servers do not know about the configuration of remstats and
are completely stand-alone; the collector will tell them what kind of data it
requires and the server will provide it in a standard form.

XXX
%------------------------------------ releasenotes.pod ---

\chapter{Release Notes}
\index{Release Notes}

\textbf{Note:} These release notes do not cover the usual install, just differences
from previous version, new features and special procedures involved in
upgrading from one version to another.  In other words, this isn't a short-cut,
it's \textbf{more} stuff you've got to read.  Sorry.

\chapter{Release Notes}
\index{Release Notes}

\section{Release Notes for Remstats version 1.0.13b}%
\index{Release Notes for Remstats version 1.0.13b}

\subsection{Bug Fixes}%
\index{Bug Fixes}

\subsection{Significant changes}%
\index{Significant changes}

\begin{itemize}


\item Switched all uses of localtime to gmtime, hence all timestamps are
now in UTC.  Sorry if you don't like it, but it's the only way I could figure
out how to deal with multiple time-zones in any civilized way.

\item Dropped graph-writer (see the graph-writer section)  from distribution.  If you've already moved
to using run\_remstats2 (see the run\_remstats2 section) , you'll already be using page-writer (see the page-writer section)  instead.
If not, switch now.

\end{itemize}
\chapter{Release Notes}
\index{Release Notes}

\section{Release Notes for Remstats version 1.0.12b}%
\index{Release Notes for Remstats version 1.0.12b}

\subsection{Bug Fixes}%
\index{Bug Fixes}

\begin{itemize}


\item  Problem with time-sync for remstats servers fixed, see item 198
in done (see the done section) .  Unless all your hosts running remstats servers are time
synchronized, you need this release.  I guess that's why I never ran into
this before. :-)

\end{itemize}
\subsection{Changes }%
\index{Changes }

\begin{itemize}


\item  If you're upgrading, you should switch from run-remstats (see the run-remstats section)  to
run-remstats2 (see the run-remstats2 section) .  The former will be going away soon and the latter
is much more flexible.

\item  moved the default location of the datapage directory.  If you don't
change yours, then all the pagemakers will run every time, which is bad.
Edit your \texttt{general} config-file and change \texttt{datapagedir} to something
outside /home/remstats/etc/config.  The new default is \texttt{/home/remstats/data/DATAPAGES}, which
you should probably use unless you have reasons to do otherwise.  Then 
move /home/remstats/etc/config/datapages to /home/remstats/data/DATAPAGES, or wherever you
decided to put it.

\item  replaced the remoteping rrd with remoteping-*, allowing one RRD per
remoteping-server (see the remoteping-server section)  host.

\item  renamed \texttt{cleanup} to remstats-cleanup (see the remstats-cleanup section)  to avoid name collisions.
Make sure that you change the remstats user's crontab.

\end{itemize}
\subsection{New Features}%
\index{New Features}

\begin{itemize}


\item  new docs section on troubleshooting (see the troubleshooting section) .  Check it out and suggest
additions.

\item  check-rrdlast (see the check-rrdlast section)  will show the last-update times for all RRDs and 
optionally the data for that last update.

\item  new error-collector (see the error-collector section)  to count errors/aborts in the other collectors
and thus allow alerts based on this.

\item  new alert-syslog (see the alert-syslog section)  alert-sender to log alerts to syslog

\end{itemize}
\chapter{Release Notes}
\index{Release Notes}

\section{Release Notes for Remstats version 1.0.11b}%
\index{Release Notes for Remstats version 1.0.11b}

\begin{itemize}


\item all configuration files/directories will be checked to see if the 
configuration has changed, to determine if run-remstats (see the run-remstats section)  needs to run
the pagemakers (see the pagemakers section) .

\item New rt-updater (see the rt-updater section)  program to watch info on a specified host+RRD+graph
in near real-time.  It's not well integrated with everything else, but it's
still usefull for testing.

\item New program-collector (see the program-collector section)  to run external programs to supply values
for RRDs, written by Arnauld Michelizza.

\item All the collectors (see the collectors section)  now honor a stop file.  If there is a file in the
configuration-specified tempdir called \texttt{STOP-programname}, then the collector
called \texttt{programname} will exit after reading its configuration.  This is more
for debugging, but there may be other reasons to use it.  This is similar to
the feature in run-remstats (see the run-remstats section)  and run-remstats2 (see the run-remstats2 section) .

\item New show-config.cgi (see the show-config.cgi section)  to allow configuration browsing.
No editing.  Also, it's incomplete, but it does do the important pieces:
hosts, and rrds and what they refer to. However, it's not secure, in that 
it may reveal too much, such as SNMP community strings.  Make \textbf{very} sure 
that you control access to it.

\item \texttt{Make install} now will make default htpasswd and htgroup files for
use with apache's (see \textbf{http://httpd.apache.org/}) htpasswd feature, if it can
find an \texttt{htpasswd} program in the path.  These will also be used by the new
access\_control (see the access\_control section)  routine which will be used to control
access to all the cgis (see the cgis section) .

\item \texttt{configure} will now do proper checking for required and
optional perl modules.  Missing required modules will abort \texttt{configure};
missing optional modules will mark the files which require them so that
they won't be syntax checked (which would fail).

\end{itemize}
\chapter{Release Notes}
\index{Release Notes}

\section{Release Notes for Remstats version 1.0.10b}%
\index{Release Notes for Remstats version 1.0.10b}

If you see this version, you're really fast.  Please remove it and take
1.0.11b instead.

\chapter{Release Notes}
\index{Release Notes}

\section{Release Notes for Remstats version 1.0.9b}%
\index{Release Notes for Remstats version 1.0.9b}

\textbf{Note:} There is a new target in the Makefile: \texttt{upgrade}.  It's supposed
to do the required changes for folks who are upgrading from previous
versions.  It won't work for upgrading from versions before 1.0.4a.  It has
\textbf{NOT} been exhaustively tested, especially for earlier versions.  If
the install target notices that the configuration directory exists, it will try to do the upgrade for you.  To use it manually:

\textbf{BACKUP YOUR CURRENT CONFIGURATION!}

\begin{verbatim}%
	make all
	make upgrade
	make install
	...\end{verbatim}

\subsection{New Features}%
\index{New Features}

\begin{itemize}


\item The new run-remstats2 (see the run-remstats2 section)  program replaces run-remstats (see the run-remstats section) .
It also brings two more config-files run (see the run section)  and
run-stages (see the run-stages section) .  This permits \texttt{run-remstats2} to be
completely configurable in what programs get run.  This can be used to run
more processes in parallel, but intelligently divided, to permit the
run to complete more quickly.
[UPGRADE] If you are upgrading, this will require you to do the following,
before doing the install:

\begin{verbatim}%
	mkdir /home/remstats/etc/config-base/run-stages
	ln -s /home/remstats/etc/config-base/run /home/remstats/etc/config/run
	ln -s /home/remstats/etc/config-base/run-stages /home/remstats/etc/config/run-stages\end{verbatim}


\item The new \texttt{headerfile} directive for the
hosts config-file (see the hosts config-file section)  permits you in inject arbitrary
information into the host header.

\item The new \texttt{currentvalue} directive for the
rrds config-dir (see the rrds config-dir section)  permits you to store the current value
of variables in status files in the host's data directory.  Particularly
useful in conjunction with the \texttt{headerfile} directive.  Can also be used
to permit other processes to act on values collected by the collectors.

\item A new section, \texttt{sar}, for the unix-status-server (see the unix-status-server section) .  This runs
"\texttt{sar -A}" and parses the result.  Unfortunately, there is no consistency
about what data is available or when the various items should be called, so
the RRDs are OS-specific.

\end{itemize}
\chapter{Release Notes}
\index{Release Notes}

\section{Release Notes for Remstats version 1.0.8a}%
\index{Release Notes for Remstats version 1.0.8a}

Another minor release.

\subsection{New Features}%
\index{New Features}

\begin{itemize}


\item Added \texttt{--with-perl=xxx} option to \texttt{configure} in case you need a 
specific version of perl for some reason.

\item Added \texttt{HTMLDIR}, \texttt{CGIDIR} and \texttt{CGIURL} to the list of 
magic cookies (see the magic cookies section)  substituted.

\item New programs port-query (see the port-query section)  and pattern-query (see the pattern-query section)  for testing scripts
used by the port-collector (see the port-collector section) .

\end{itemize}
\chapter{Release Notes}
\index{Release Notes}

\section{Release Notes for Remstats version 1.0.7a}%
\index{Release Notes for Remstats version 1.0.7a}

Another minor release.  This one gets rid of GIFs and replaces them by png 
files.

Also some minor bug-fixes.

\chapter{Release Notes}
\index{Release Notes}

\section{Release Notes for Remstats version 1.0.6a}%
\index{Release Notes for Remstats version 1.0.6a}

Oops.  There was a small glitch in the install for 1.0.5a.  Here's a fixed 
version.

\chapter{Release Notes}
\index{Release Notes}

\section{Release Notes for Remstats version 1.0.5a}%
\index{Release Notes for Remstats version 1.0.5a}

\begin{itemize}


\item  [UPGRADE] If you're upgrading from a previous version of remstats, 
you'll have to:
\begin{verbatim}%
	cp /home/remstats/etc/config-base/ntops /home/remstats/etc/config\end{verbatim}

after the install of the new version, even though it's effectively empty 
and you're not using the new ntop-collector (see the ntop-collector section)  (yet).  See below for 
more pointers on it.


\item  [UPGRADE] You'll need to do the following to accomodate the new
page-writer (see the page-writer section) :
\begin{verbatim}%
	cp /home/remstats/etc/config-base/pages /home/remstats/etc/config/pages
	mkdir /home/remstats/etc/config/page-templates
	cp /home/remstats/etc/config-base/page-templates/* /home/remstats/etc/config/page-templates\end{verbatim}

after the install of the new version.


\item  [UPGRADE] The host \texttt{tags} directive, has been renamed to \texttt{keys}
to avoid completely confusing me.

\item  [UPGRADE] The new \texttt{environment} config-file requires you to:
\begin{verbatim}%
	cp /home/remstats/etc/config-base/environment /home/remstats/etc/config\end{verbatim}

after the install of the new version.


\item  The collectors now all will use configuration-supplied IP numbers,
if they are there, in preference to looking the host-name up in DNS.  If
you find an instance of this not happening, it's a bug.

\end{itemize}
\subsection{New Features}%
\index{New Features}

\begin{itemize}


\item  RRD config-files can now (like host config-files) have \texttt{keys}
associated with them.  This was mainly added for the new page-writer (see the page-writer section) ,
but could be of other use.

\item  graph-writer (see the graph-writer section)  has been replaced by page-writer (see the page-writer section) , which writes
directory trees of pages from simple templates.  The original look of
the pages graph-writer created has been re-created with 8 templates 
totalling 94 lines.  Other completely different interfaces can now be
created.

\item  macinfo (see the macinfo section)  a script (and cgi (see the cgi section) ) to look up MAC addresses
from an ethernet switch via SNMP and show, for each port on the switch, which
host is on that port (by MAC address or IP number or domain-name, or a 
combination).  The related script update-switch-ports (see the update-switch-ports section)  will also update the 
description of each interface on the switch, with the host which is connected 
to it.

\item  The unix-status-server (see the unix-status-server section)  has some new sections to make available 
various data from /proc under linux, including disk I/O, memory, ...

\item  alert-monitor (see the alert-monitor section)  get some new relations: $<$daystddev, $<$weekstddev 
$<$monthstddev (to compare standard-deviations),
|$>$ |$<$ (absolute value comparisons), delta$>$, delta$<$ (change comparisons).

\item  the links config-file (see the links config-file section)  now has a more flexible
definition and all the links are now configurable.

\item  the lockfile (see the lockfile section)  script and remstats-backup (see the remstats-backup section)  which uses it.

\item  the hosts config-file (see the hosts config-file section)  has a new directive \texttt{tags}
to allow processing of sub-sets of all hosts.

\item  The new ntop-collector (see the ntop-collector section) , configured by the 
ntops config-file (see the ntops config-file section) ,
allows you to collect stats on protocol usage distribution.

\item  The graph.cgi (see the graph.cgi section)  script now allows the time-span to be
user-specified, and also the graph width and height.

\item  All the collectors now accept \texttt{-H}, \texttt{-G} and \texttt{-T} flags to allow
them to be run on subsets of all hosts, for an explicit list of hosts (\texttt{-H}),
all hosts in a list of groups (\texttt{-G}), or hosts which have a particular tag
(\texttt{-T}).

\item  A new dbi-collector (see the dbi-collector section)  to collect information from remote databases.
Look at the rrd \texttt{dbitest1} for an example of how to create your own.

\item  A new config-file environment (see the environment section)  to set up
the environment for children of run-remstats.

\item  The log-server (see the log-server section)  implements a new function \texttt{lastof} to permit it 
to match patterns, while maintaining state from previous runs.  See the
RRD \texttt{nfsproblems} for an example.

\item  A new utility remstats-rrdtune (see the remstats-rrdtune section)  to tune all the RRD files to match
the remstats RRD definitions.  Or as much tuning as rrdtool will do.  The
big one that you probably wanted, mucking with archives, it doesn't do.

\end{itemize}
\chapter{Release Notes}
\index{Release Notes}

\section{Release Notes for Remstats version 1.00a4}%
\index{Release Notes for Remstats version 1.00a4}

[There is no version 1.00a3.]

It's always a good idea to run check-config (see the check-config section)  after changing any of the
config-files, but it's also a good idea to run it after doing an upgrade,
especially when, as in this version, there are changes to the config-files.

Mostly small new features and bug-fixes, except:

\begin{itemize}


\item  \textbf{Incompatible:} re-written alert-sending mechanism.  Permits
easily written new methods of sending alerts, by separating the 
alert-text generation (see alerter (see the alerter section) ) from the alert-sending (see 
alert-email (see the alert-email section)  and alert-winpopup (see the alert-winpopup section) ).  The new 
alert-destination-map (see the alert-destination-map section)  config-file permits
mapping an alert-destination to different addresses depending on the 
time-of-day, day-of-week, ... and its alias facility permits sending to a 
list of addresses which may use different methods of sending the alert.

\item  \textbf{Incompatible:} The unix-status-server (see the unix-status-server section) 's do\_df now returns 
\textbf{bytes} not \textbf{K-bytes}.  This avoids silliness in the graphs saying that 
you've got 20k kbytes free.  It may have been correct, but it wasn't
intuitive.  You can multiply all the old numbers by 1000 to convert RRDs.

\item  \textbf{Warning:} new-config (see the new-config section)  now copies the configuration files which 
you are likely to change, so that your changes won't be overwritten by an 
update to remstats.  Unfortunately, as all updates (including this one) 
will overwrite config-base, so people upgrading from a previous version 
of remstats should convert the following files from symlinks to config-base
into copies of those files:
\begin{itemize}


\item alerts alert-destination-map general html links tools

\end{itemize}
You can do this by running the supplied convert-config-links (see the convert-config-links section)  script
\textbf{BEFORE INSTALLING THIS VERSION}


\end{itemize}
\subsection{New Features}%
\index{New Features}

\begin{itemize}


\item  the new datapage-status (see the datapage-status section)  script generates datapages for each host, 
which will show the current values of all rrd variables.  There is a new 
tool in the default tools config-file (see the tools config-file section)  which will show 
this page, and it's been added to the defaults generated by the 
new-xxx-hosts (see the new-xxx-hosts section)  programs.

\item  Host templates (see the Host templates section) .  So that you can configure
similar hosts like:
\begin{verbatim}%
	desc	whatever
	template their-template\end{verbatim}

Can also be used to make changing some things for many hosts easier.  E.G., you
could have a template, say \texttt{default-nt-status-server} which contained:

\begin{verbatim}%
	nt-status-server some-host\end{verbatim}

and configure all hosts which use c$<$some-host$>$ like:

\begin{verbatim}%
	template default-nt-status-server\end{verbatim}


\item  New availability-report (see the availability-report section)  and 
availability-report.cgi (see the availability-report.cgi section)  and config-file
availability (see the availability section)  for reporting on "availability".

\item  New nt-status-server (see the nt-status-server section)  and nt-status-collector (see the nt-status-collector section)  and RRDs for them 
(ntactivity, ntmemory, ntpaging, ntnetwork and ntlogicaldisk-*).

\item  New cleanup (see the cleanup section)  program to remove stale files.

\item  New new-snmp-hosts (see the new-snmp-hosts section)  now adds other rrds than snmpif-*

\item  New new-unix-hosts (see the new-unix-hosts section)  program to add hosts which are running the
unix-status-server (see the unix-status-server section)  with the apropriate rrds.

\item  ought to work with perl 5.6 now.  I'm not using perl 5.6 on the main 
collector yet, but it seems to install correctly on a test system.

\item  run-remstats (see the run-remstats section)  now checks all configuration sub-directories to figure
out if anything has changed, so you ought to be able to just edit files and
the changes will get caught on the next run.

\item  remstats internal instrumentation allows monitoring remstats collectors,
for now.  More later.  Look at the pseudo-host \_remstats\_.

\item  removed old Overall Index, since I never looked at it, and wrote a new
RRD Index, which I was wanting.

\item  new nt-discover (see the nt-discover section)  program to discover and add NT systems
\textbf{Note}: this adds the new discovery config-file (see the discovery config-file section)  which
must be locally configured.  I won't even attempt to guess at values here.


\item  The old \texttt{alertflag} entry in the html config-file (see the html config-file section)  
has been replaced by three new entries: \texttt{alertflagcritical}, 
\texttt{alertflagerror} and \texttt{alertflagwarn}, allowing e.g. different colors for 
the different levels of alert.

\item  datapage.cgi (see the datapage.cgi section)  now does variable substitution 
properly for HTML macros.  See the example datapage upss.page under 
/home/remstats/etc/config/datapages.

\item  datapage.cgi (see the datapage.cgi section)  and dataimage.cgi (see the dataimage.cgi section)  
have two new commands: \texttt{alertstatus} and \texttt{alertvalue} to fetch alert 
statuses and values.  To be used on forthcoming status pages.

\end{itemize}
\chapter{Release Notes}
\index{Release Notes}

\section{Release Notes for Remstats version 0.13.1}%
\index{Release Notes for Remstats version 0.13.1}

I fixed a minor buglet in 0.13.0 which was noticed shortly after release.
I was annoyed enough with it that I made 0.13.1.

\chapter{Release Notes}
\index{Release Notes}

\section{Release Notes for Remstats version 0.13.0 (AKA 0.12.2)}%
\index{Release Notes for Remstats version 0.13.0 (AKA 0.12.2)}

There are lots of little improvements, which are detailed in the
Change History (see the Change History section) , which I'm not going into here.  The main
incompatible changes are:

\begin{itemize}


\item  The configuration structure (\$main::config) now has the graphs stored
under \$main::config\{RRD\}\{\$wildrrd\}\{GRAPH\} instead of \$main::config\{GRAPH\},
so there won't be problems with having the same graph-name defined under two
different rrds.  This will only affect you if you've been writing your own
code for remstats, like a new page-maker.  I thought that the bug was
annoying enough and difficult to figure out when it was triggered that
the incompatibility was worth the change.

\item  After typing \$main::config\{CUSTOMGRAPH\} instead of \$main::config\{CUSTOM\}
one too many times, I renamed \$main::config\{CUSTOM\} to \$main::config\{CUSTOMGRAPH\}
which is what it should have been all along.  Again, this should only affect
you if you've been writing your own remstats code, like a new page-maker.

\item  Changed default location for datapages to /home/remstats/etc/config/datapages, so that
all the configuration, including the datapages are together.

\item  Removed the general config-file directive \texttt{pagesas} since all the
generated pages are CGIs now.  Check-config will abort if you still have it.
Just delete that line in the general config-file.

\item  \texttt{use strict} in all the scripts (unless I missed some) in preparation
for perl 5.6, which doesn't like \texttt{use vars}.  Shouldn't bother you unless
you've been writing remstats code, in which case, you probably know what to
do.

\item  To deal with alert templates (see below), you'll need to manually fix
your config-dirs.  For each one, you need to:
\begin{verbatim}%
	su remstats
	cd your-config-dir
	cp /home/remstats/etc/config-base/alert-template-map .
	mkdir alert-templates
	cp /home/remstats/etc/config-base/alert-templates/* alert-templates\end{verbatim}


\item  remoteping-collector has been modified to return the server-name
instead of a number to differentiate the data from different servers.
There's also a new remoteping-* wildcard RRD to make it more usefull.

\end{itemize}
\subsection{Customgraphs on host-index pages}%
\index{Customgraphs on host-index pages}

The new \texttt{customgraph graphname} directive for host config-files
permits you to add a customgraph to a host-index page. (Thanks Marek.)

\subsection{graph.cgi - remstats graphs anywhere}%
\index{graph.cgi - remstats graphs anywhere}

Like it says, using the new graph.cgi (see the graph.cgi section) , you can put
remstats graphs on any page you want.

\subsection{Views }%
\index{Views }

You can now define your own pages with page-layout of your choice
using views (see the views section) . (Thanks to Marek and Thorsten and Matt and
anyone else I've forgotten.)  Don't forget to add view-writer (see the view-writer section)  to the
list of pagemakers if you've changed the default.

\subsection{ping-* rrd}%
\index{ping-* rrd}

You can now ping different interfaces on a host separately (Thanks Steve)

\subsection{fileage section for unix-status-server}%
\index{fileage section for unix-status-server}

This allows you to fetch the last-modification-time for specified files.  It
was written to allow remstats to monitor lock-files to check for stale locks.
There is no included rrd using it as lock-files are all over the place.

\subsection{port-collector can collect data from results}%
\index{port-collector can collect data from results}

The port-collector (see the port-collector section)  has always been able to send a string to remote services
to that it could tell if they were working correctly.  Now it can pull values
for RRDs and status-files from the results as well.  I've included a sample
rrd (weathernetwork) and script (weathernetwork) to collect current weather data
for Ottawa.  Look at the updated docs for 
scripts config-files (see the scripts config-files section) .

\subsection{new script - snmpif-description-updater}%
\index{new script - snmpif-description-updater}

The snmpif-description-updater (see the snmpif-description-updater section)  will keep the descriptions on snmpif-*
RRDs up-to-date with whatever you've set as ifAlias for that interface.
(Thanks Steve Francis.)

\subsection{Alert Templates}%
\index{Alert Templates}

This feature allows you to customize the alert messages by addressee
or by RRD.  Look at the docs in 
alert-template-map (see the alert-template-map section)  and
alert-templates (see the alert-templates section) .

\subsection{Autoconf-like configure}%
\index{Autoconf-like configure}

You can now do:

\begin{verbatim}%
	./configure
	make
	make install\end{verbatim}

for the beginning of of the install (see the install section) .

\chapter{Release Notes}
\index{Release Notes}

\section{Release Notes for Remstats version 0.12.1}%
\index{Release Notes for Remstats version 0.12.1}

Ideally, this document will only have to tell you about the
great new features of remstats in this version.

Not this time.

In addition, due to various stuff (read the Change History (see the Change History section) ),
this covers changes since version 0.11.1.

\subsection{Configuration File Replaced by Configuration Directory}%
\index{Configuration File Replaced by Configuration Directory}

The old "one huge configuration file" has been replaced by a directory
of files and sub-directories.  (See the new configuration docs (see the new configuration docs section) 
for details.)  This means that most programs don't need to read and parse
everything, including stuff that they're not going to use.  It also
makes it easier to find things, as you can go directly to the file that
has what you want, e.g. details on a particular host.  It also made
possible the newly revamped replacements for \texttt{make-ping-hosts},
\texttt{make-port-hosts} and \texttt{make-snmp-hosts}, which will insert their
additions directly into the appropriate configuration files.

There is a new script, split-config (see the split-config section) , which will take your old
config-file and a new name and generate a new config-dir from it.

On a related note, I broke the groups (see the groups section)  line 
out of the \texttt{general} config-file
into its own file.  It's easier to see what you've got.  \texttt{split-config}
will do this for you.  Also the (undocumented) [html] section will
absorb large portions of the [general] section which really belong to
wep-page generation.

If you've made your own collector, you'd better look at the new
skeleton-collector for the required changes.  Just change \texttt{read\_config}
to \texttt{read\_config\_dir}, with extra args.  There's also documentation
on how to write your own collector (see the collector section) .

\subsection{do-remstats replaced by run-remstats}%
\index{do-remstats replaced by run-remstats}

The old \texttt{do-remstats} shell-script and all the kludgy shell-scripts
that went with it and the \texttt{watchdog} and \texttt{lockfile} scripts have
all gone away.  The replacement run-remstats (see the run-remstats section)  does everything they
did and does it correctly.  It's also configurable, so you don't need
to modify the scripts to change which collectors you want to run, e.g.

A new feature of \texttt{run-remstats} configurability is that you can have
it run the \texttt{ping-collector} before everything else and not bother
trying hosts that didn't answer it.  You can also choose which
collectors (see the collectors section) , monitors (see the monitors section)  and pagemakers (see the pagemakers section)  to run.

\subsection{CGI scripts and non-default config-dirs}%
\index{CGI scripts and non-default config-dirs}

At the moment, the supplied CGI scripts don't deal with non-default
config-dirs.  I do consider this to be a problem, but I need to get this
release out to deal with other serious installation problems.

You can work-around this by editing the installed CGI scripts and 
putting in the correct definition for \$config\_dir, near the top.

\subsection{plugin-collector is gone}%
\index{plugin-collector is gone}

It was an inefficient, difficult-to-configure,  kludge and isn't 
needed anymore with the new run-remstats.

\subsection{pre-release testing automated}%
\index{pre-release testing automated}

You won't see it, but I hope you'll all notice the improvement
in release quality.
%------------------------------------ bugs.pod ---

\section{New Known Bugs for version 1.0.12b}%
\index{New Known Bugs for version 1.0.12b}

\begin{itemize}


\item All the collectors are skipping some of alternate updates.  This is
because the code to tell if it's time to collect isn't quite right.  For 
now, I've enabled the \texttt{-F} for all collectors in
config-base/run-stages/collectors (see the config-base/run-stages/collectors section) .  If you've made changes or copies of
this, you need to make this update, for now.

\end{itemize}
\section{New Known Bugs for version 1.0.9b}%
\index{New Known Bugs for version 1.0.9b}

\begin{itemize}


\item Alert quench (in alert.cgi (see the alert.cgi section) ) doesn't work, on purpose now.
It's related to splitting access for the \texttt{webgroup} group.  I'll make it 
work again later.

\item Similarly, the log-event.cgi (see the log-event.cgi section)  CGI script doesn't
work now.

\item Every now and then, I get error messages from the alert-monitor (see the alert-monitor section) 
saying that there were no destinations for an alert.  The message needs to 
be improved so that people (myself included) can figure out what's wrong
and how to fix it.

\end{itemize}
\section{New Known Bugs for version 1.00a}%
\index{New Known Bugs for version 1.00a}

\begin{itemize}


\item Alerts in general - The alerts shown by the alerts.cgi (see the alerts.cgi section)  
page never get expired. [FIXED $<$1.0.12b]

\item If a hub is down, then you'll get alerts for everything behind it.  
Ought to only get the alert for the hub.

\end{itemize}
\section{New Known Bugs for version 0.12.2}%
\index{New Known Bugs for version 0.12.2}

\begin{itemize}


\item Neither new-snmp-hosts, nor snmp-collector use get\_ifname, 
with the consequence that neither copes well with "oddly" named interfaces, 
say with spaces in them.  [FIXED in 0.12.3]

\end{itemize}
\section{New Known Bugs for version 0.12.1}%
\index{New Known Bugs for version 0.12.1}

\begin{itemize}


\item CGI scripts don't work with non-default config-dirs.  I consider
this a bug, but I need to get this release out now to deal with
serious installation problems with previous releases.  For now, do the
following for each config-dir:
\begin{verbatim}%
	\% make install-cgis CONFIGDIR=/wherever/you/put/it\end{verbatim}

Unless someone else suggests an acceptable fix, or I have a an inspiration,
this isn't going to be fixed.  With both the new page-writer (see the page-writer section)  and
run-remstats2 (see the run-remstats2 section) , there are fewer reasons to need this.
[ACCEPTED 1.0.12b]


\item run-remstats only checks the config-dir for change, not the 
subdirectories.  For now, just \texttt{touch config-dir} whenever you make a change.
[FIXED $<$1.0.12b]

\item customgraphs are completely broken.  Urgh.  Upgrade.
[FIXED in 0.12.2]

\item You can't have two graphs of the same name, even in different rrd 
definitions.  This is just flat-out wrong and will be fixed.  Unfortunately, 
the fix will mean even longer file-names, so I hope nobody has some old system
with the 14-character limit. [FIXED in 0.12.2]

\item graphs with descriptions can't have quotes in the description.  
[FIXED in 0.12.2]

\end{itemize}%------------------------------------ todo.pod ---

\section{To-Do List for Remstats}%
\index{To-Do List for Remstats}

\subsection{Unclassified Priority}%
\index{Unclassified Priority}

\textbf{214 20030516 [HIGH]}- finish architecture docs

\textbf{213 20030515 [HIGH]} - change localtime to gmtime througout

\textbf{212 20030509 [MED]} - fix dns-collector (see the dns-collector section)  to have the rrd definitions
specify the record-types instead of the rrd instances.

\subsection{High Priority}%
\index{High Priority}

\textbf{210 20030508 [HIGH]} - allow configurable wait between items in a run-stage
to allow slow machines to cope better (thanks Marek).

\textbf{211 20030508 [HIGH]} - iptables collector or section for unix-status-server
Allow configuration of which tables and chains to look at.

\textbf{208 20030508 [HIGH]} - fix docs for new-xxx-hosts to make it clearer that
"hostsfile" is just a list of hosts, not a host configuration file.

\textbf{206 20030328 [HIGH]} - fix showlog.cgi - it doesn't like March 29, 30 or 31.
Probably related to February.

\textbf{204 20030128 [HIGH]} - only log alerts on transitions
Right now, it logs every time after the initial waiting period.  It ought to
log right away, and not again until there is another change.

\textbf{202 20020821 [HIGH]} - add walk capability to snmp-collector (see the snmp-collector section) ,
so that it can collect things with multiple instances.  The problem is
figuring out how to identify the instances.  Could be as simple as a
wildcard RRD with an enhancement on the \texttt{oid} line to allow the wildcard
to be propagated to there, if the number is sufficient as an ID.  Ideally,
I'd prefer a name, but that would require some way of specifying a
number-to-name mapping which wouldn't get out-of-date.  Many oid trees have
a name for the instance as one of the oids. [Ecaroh] See also \textbf{42}.

\textbf{200 20020819 [HIGH]} - monitoring of remstats errors
- While cleaning up stderr files, count errors and aborts and log that too.

- [DONE 20020820] - make sure it works.

\textbf{193 20020726 [HIGH,BUG]} - look at datastuff.pl
RRDNODATA causes all sorts of error messages in the web error log if
it gets RRDNODATA.

\textbf{191 20020716 [HIGH,BUG]} - alert-monitor (see the alert-monitor section)  is logging too much
It needs to only log state transitions.

\textbf{187) 20020627 [HIGH]} - some df's give blocks instead of k

\textbf{185) 20020625 [HIGH,BUG]} - host-templates are broken
In some cases they wipe out all RRDs from \$main::config.  Unfortunately,
I can't duplicate this any more.  Double-plus-ungood.

\textbf{99) 20000619 [HIGH]} make unix-status-collector 
send the directories that we want df for and make unix-status-server 
do "df /dir1 /dir2" to get them, and pull them off one line at at time.  
This is to deal with things like disconnected NFS-mounted directories 
hanging df when we do just a bare "df".

\textbf{86) 20000419 [HIGH]} trends analysis

- [20020621] consider Holt-Winters smoothing to create temporary RRD
files to show smoothed data and projections.  Store it under GRAPHS/TMP

\textbf{87) 20000419 [HIGH]} alerts based on trends analysis and historical
data, like one-week average and standard-deviation, ... (for Steve)

- [20020621] new kind of alert using Holt-Winters to extrapolate trends
and base the alert on some value in the future.  Definition like:

\begin{verbatim}%
	alert trend(xyzzy,7d) $<$ 20 40 60\end{verbatim}

or maybe
	trend-alert xyzzy 7d $<$ 20 40 60

to trigger an alert if the trend for variable \texttt{xyzzy} at \texttt{7 days from now}
is ... (the rest interpreted in the usual alert manner.)

Could even generalize this by making it \texttt{\&trend} instead of \texttt{trend}, 
which would apply the named function to the data.  Probably not worth
implementing, as I can't think of what I'd do with it.

\textbf{146) 20011220 [HIGH]} - deal with broken html in release/src directory

\subsection{Medium Priority}%
\index{Medium Priority}

\textbf{207 20030404 [MEDIUM]} - alert-fixup - run command on detecting condition
The command is found in \texttt{/home/remstats/etc/config/fixups} and is named either
\texttt{address/hostname/rrdname/varname}, \texttt{address/ANY/rrdname/varname},
\texttt{address/ANY}, \texttt{ANY/hostname/rrdname/varname}, \texttt{ANY/ANY/rrdname/varname},
\texttt{NOMATCH}, or \texttt{ALL}looked for in that order.  The default, \texttt{NOMATCH} will be
supplied with remstats distribution and will do nothing, but may be altered to
act on all non-matching fixups.  The others are site-specific.  \texttt{ALL}, if
present, will be run after any specific script is found.  Scripts will be run
with an environment consisting of all the magic cookies which can be
substituted into a regular alert message.

\textbf{203) 20030122 [MEDIUM]} - move host data to data/HOSTS/$<$hostname$>$

\textbf{188) 20020628 [MEDIUM]} - new show-alert-thresholds.cgi
to extract the levels from rrds and host (separately) and format them prettily.

\textbf{180) 20020613 [MEDIUM]} - development hints
E.G. use skeleton-collector as template to make new collectors.

\textbf{168) 20020508 [MEDIUM]} - check that we're not lying with 
/o in pattern matches.

\textbf{127) 20010622 [MEDIUM]} - graph data together with historical data.  This
will probably mean either populating another rrd with historical averages, 
temporarily or permanently, or modifying rrdtool.  The former is certainly
simpler to do, given my knowledge of the internals of rrdtool.  However,
it needs to have another rrd for each period?  Need to keep the same data
over some longer period, a multiple of the period of interest, as well as
the averages, from period to period.

- see also 86 and 87.

\textbf{138) 20011002 [MEDIUM]} - make new-config symlink all rrds from config-base 
instead of just the directory.  That will make it easier for people to 
have their own rrds which don't get overwritten.  

\textbf{137) 20011002 [MEDIUM]} - make datapage-interfaces include if-* and 
procnetdev-* collected interfaces as similar info is available.

\textbf{131) 20010824 [MEDIUM]} - make status pages for each host, 
group and for all hosts using the new alertstatus and possibly alertvalue.

- [DONE before 20011220] see datapage-alert-writer (see the datapage-alert-writer section) 

\textbf{109) 20001212 [MEDIUM]} \texttt{nt-log-collector}, with modules for event-logs
and ordinary log-files.  Note that this implies a new \texttt{nt-log-server}.
Or it could be added to the nt-status-server (see the nt-status-server section) .  Probably should be a new
service as the \texttt{nt-status-server} can already be quite slow.

\textbf{106) 20000922 [MEDIUM]} make a file-collector.
Similar to the log-collector, only for small, local files.  Slurp the 
file into memory, match patterns and pull out values.  The data line in an
rrd definition would be like:

\begin{verbatim}%
	source file
	data VARNAME	GAUGE:600:0:U FUNCTION PATTERN(WITH)PARENS\end{verbatim}

In fact, this would share so much code with the log-collector that it
might be worth combining the two.  This allows collection from things
like Linux's /proc.

\textbf{103) 20000915 [MEDIUM]} make-path doesn't 
work with non fqdn hosts.  Make it read the configuration, so it can 
look up the IP number in the host config and use that if it's defined.  
Otherwise, default to gethostbyaddr.

\textbf{45) 20000121 [MEDIUM]} make snmp-collector 
send only one packet per host

- test and make sure that we do get back whatever succeeded.  I vaguely
remember that it didn't work.  [Later: at least under UCD snmp under linux,
if an item isn't implemented in the MIB, you get back NOTHING.  Specifically,
look for the non-unicast packet counters as well as something else; you get 
nothing back.  This isn't good.]

- have to re-write snmp-collector completely, which isn't that bad an idea.
This means a two-pass structure.  On pass one, we construct the complete query
and then send it.  On pass two, we examine all the results and format them.

\textbf{9) ???????? [MEDIUM,TESTING]} make alerts take 
connectivity dependence into account

- add "via" line to host section to deal with hubs and switches [DONE]

- I think it's done.  See what happens next outage.

\subsection{Low Priority}%
\index{Low Priority}

\textbf{134) 20010829 [LOW]} - make header\_bar (in htmlstuff) 
do the link making, if available and fix whatever uses it not to.

\textbf{133) 20010829 [LOW]} - add an option to make 
nt-discover update old hosts with a standard set of RRDs, even if the 
hosts are already known.

\textbf{102) 20000912 [LOW]} add see-also to host config, which will
materialize links in the host header.  Config line like:

\begin{verbatim}%
	seealso	host:xyzzy http://www.somewhere ftp://ftphost\end{verbatim}

the special "host:" pseudo-URL gets changed to a link to the
remstats page for that host.

\textbf{51) 20000216 [LOW]} need a way to specify URL for port-http. The root page 
doesn't always exist.

\textbf{37) 19991216 [LOW]} traceroute sometimes shows incorrect routing, which 
confuses the topology-monitor, causing false positives

\textbf{50) 20000215 [LOW]} make inventory script.  Runs uname 
(for hardware and software), \texttt{ifconfig -a}, \texttt{netstat -nr}, \texttt{hostname}
and any others I can think of to collect configuration info.  
Then figures out the versions of important software, e.g. run \texttt{perl -v},
\texttt{gcc -v ...}  Make a subdir to put it in and make a tool definition to get it
onto the host pages.

- looks like the beginning of a discovery script.

\textbf{69) 20000406 [LOW]} is there any use for write\_environment in 
check-config?

\section{On Hold}%
\index{On Hold}

Usually waiting for next major release, or trapped by something else.
(in priority order)

\subsection{High Priority}%
\index{High Priority}

\textbf{164) 20020409 [HIGH,HOLDRELEASE,NEEDS=165]} - document perl libraries

\textbf{165) 20020409 [HIGH,HOLDRELEASE]} - convert perl libraries to modules 
that can be \texttt{use}'d.  Consider OO, if there's a benefit which outweighs
having to re-write almost everything.

\textbf{92) 20000518 [HIGH,HOLD]} collect traffic info from cflowd (artsportms).
Make it flexible enough that it can let you choose which ports you
want (one per rrd?).  Make a loader to load historical data.

- [DONE 20000524] artsportms-loader done

- [HOLD] I no longer have access to devices with this feature

\textbf{70) 20000407 [HIGH,HOLD]} CGI scripts need to have a way to deal with
alternate config-files, and graph-writer needs to tell them if they
can't work it out themselves.  Otherwise, people need to be told to
do multiple installs of the CGI scripts, which might be the best way.

\begin{verbatim}%
	make install-cgis CONFIGDIR=config-xxx\end{verbatim}

Not that painfull, but wastefull and makes upgrade messier.

- I don't like the multiple-install method, but any other method needs 
a way of getting configuration information into the CGI scripts.  Any
method which passes info in via the URL or form fields is out: too unsafe.
The only other method I can think of is to read a configuration file in the
same directory as the CGI script.  This ought to be safe from modification,
or your web-site is waiting to be mutilated.  The other part to consider is
whether any part of the info in the CGI config-file is sensitive.  I.E. do
we have to protect it in some way.

- Configuration file in the same directory won't work either, you'd still
have to install the cgi's multiple times.  I'm starting to think that
multiple installations may be the only safe thing to do.

\subsection{Medium Priority}%
\index{Medium Priority}

\textbf{60) 20000328 [MEDIUM,HOLD]} replace route-collector with something which
scales.  SNMPwalking bgp4PathAttrBest doesn't scale to large Internet
routers with 400 peers, taking over an hour to complete. (see also 61)

- look at a script to follow the output of zebra.  That's a lot of
overhead though.  Easy if zebra is solid.

- How difficult can it be to make a native BGP listener?  I'm not clear on
the protocol, but it doesn't look too bad.

- [HOLD] As I don't need it, and have no access to anything which does.

\textbf{42) 20000114 [MEDIUM,HOLD]} snmp-collector mod to 
allow summary data collected from a walk and then filtered as a single 
data-point.  E.G. specify a rrd "oid" like:

\begin{verbatim}%
	walk    count ifOperStatus = 1\end{verbatim}

would produce a count of the number of interfaces on that device that
were active (i.e. had a live device plugged into them).  Or a similar one
would let you count BGP routes, or arp addresses, ...  

- Unfortunately, from experience with the snmp-route-collector, this is
going to be slow for anything with a large number of items.

- [HOLD] Until I think of something to use it for.

\textbf{171) 20020510 [MEDIUM,HOLDRELEASE]} - have collectors log how many 
hosts, unique rrds and rrd instances they collected from for \_remstats\_.

\textbf{121) 20010202 [MEDIUM,HOLD]} - how about an discovery program, to 
find and identify hosts and then run the appropriate new-xxx-hosts 
scripts to add them?

- DONE 20010608 - nt-discover to find and add NT boxen

\textbf{128) 20010629 [MEDIUM,HOLD]} - custom, configuration-supplied info 
per rrd which is simply available wherever it makes sense, e.g. in alerts.

- first make sure someone has a use for it.

\textbf{40) 20000104 [MEDIUM,HOLD]} consider some form of access-control for servers

- hash-based "password"

- ssl tunneling ought to work for everything except SNMP

- what does this buy?  With the various servers run under tcp\_wrappers
an attacker must either gain access to the remstats collector 
machine or spoof a tcp session from them.  If you've been "owned" 
you've got bigger problems.  If the attacker spoofs a session with 
a remstats server, tcp-wrappers will insist that it must come from 
one of the allowed hosts, so that's where the stolen output will go.  
This is only usefull to the attacker if they have access to the 
remstats collector machine or if they can sniff the traffic between 
the collector and the server.  The only data loss possible is with
the log-server (see the log-server section)  which keeps state.  (Ignoring DOS attacks which are
always a problem.)

- unless someone needs this, it's on hold

\subsection{Low Priority}%
\index{Low Priority}

\textbf{10) ???????? [LOW,INPROGRESS,HOLD]} make graph of connectivity 

\textbf{13) ???????? [LOW,INPROGRESS,HOLD]} snmp trap listener to update status files

- needs filter to be usefull [DONE]

- I haven't seen any useful traps so this is on hold.

\textbf{39) ???????? [LOW,HOLD]} make RRD dumper, to put data out in a form that can 
be loaded into a database

- I don't need it, per se, but it might be easier than writing the 
availability report generator.

\textbf{52) 20000215 [LOW,HOLD]} make a makegraph.cgi, or whatever, that will let you 
make a somewhat custom graph on the fly.  makegraph.cgi by itself will list 
all the hosts and let you choose one.  makegraph.cgi?host=xxx will list 
all the RRDs for this host and let you choose ?one?.  
makegraph.cgi?host=xxx\&rrd=yyy will list the various DSs for this RRD and 
let you choose the ones you want.  Then you get to define any CDEFs needed 
and then LINEn/AREA/STACK for each DEF or CDEF desired.  And size, title, 
legends...

- On hold since graph.cgi (see the graph.cgi section)  will let you get at any existing 
graph you want.  If I find a use or need for this, I'll re-activate it.

- see XXX

\textbf{112) 20001212 [LOW,HOLD]} - web-based remstats configurator.  Needs
to consider security, at least from the point of view that you don't 
want to lose your configuration.  The most important part is hosts.
A lot of the rest doesn't have to be changed, or only once.

\textbf{111) 20001212 [LOW,HOLD]} consider grafting on (at least links to) 
some kind of system configuration interface. For configuring the
mmonitored entities, not remstats.

\textbf{110) 20001212 [LOW,HOLD]} consider problem-fixing 
interface.  It'd be nice to try to fix things if there is a known way 
to do so.  A simple kludge would be to add another method to the 
alert-destination-map which deals with problems that it knows about, 
possibly invoking plugins for specific alerts.

\textbf{130 20010823 [LOW,HOLD]} - add an $<$RRD::EXEC ...$>$ tag to rrgcgi.

- [HOLD] I thought I had a use for it, but I can't think of one now.

\textbf{162) 20020327 [LOW,HOLD]} - allow logging as a possible type of alert 
"notification".

- [HOLD] Why?

\textbf{147) 20011220 [LOW,HOLD]}  - get list of links to users

- [HOLD] I don't know what this is.

I've also kept the stuff that used to be here, but has already been
done (see the done section) .
%------------------------------------ faq.pod ---

\chapter{FAQ}
\index{FAQ}

\section{FAQ for remstats}%
\index{FAQ for remstats}

\begin{enumerate}


\item  \textbf{What is remstats?}
Remstats is a system of programs to:

\begin{itemize}


\item gather network accessible data via SNMP and its own servers,

\item store and maintain the data for long periods,

\item produce graphs and web-pages tieing them together

\item monitor the data for anomalous behaviour and issue alerts

\end{itemize}

\item  \textbf{Where can I get it?}
The latest stable version is 1.0.9b.  You can get it at 
http://remstats.sourceforge.net/release/src/remstats-1.0.9b.tar.gz (see \textbf{http://remstats.sourceforge.net/release/src/remstats-1.0.9b.tar.gz}).  The current 
development version is 1.0.13b.  You can find it with all the other versions
in the source archive (see \textbf{http://remstats.sourceforge.net/release/src}).  As most of it is perl scripts, there
is no binary version.


\item  \textbf{How do I ...?}
Just a sec, there's lots of documentation online on how to set this up.  
You found the FAQ and that's good, but in order to get remstats installed 
and configured, you're going to have to read at least the 
prerequisites (see the prerequisites section) , and installation (see the installation section)  sections and all 
their sub-sections.  Go do that now before bothering to read the rest 
of this FAQ.  You'll need to do it anyway and it ought to answer most 
of the questions about getting going.  Complain (see the Complain section) 
if it doesn't.

If you'd prefer to print something and
read it offline, you can look at book.ps (see the book.ps section)  or book.pdf (see the book.pdf section) .  
Note: The pdf translation is \textbf{really} slow loading.  Be patient.

We now return you to your regularly scheduled FAQ.


\end{enumerate}
\subsection{Miscellaneous }%
\index{Miscellaneous }

This is only a proto-faq, covering the things I couldn't figure out where else to
put.

\begin{enumerate}


\item  \textbf{snmp-collector complains that I don't have SNMP\_Session installed, but
I don't want SNMP.  Do I have to collect and install it?}
No you don't.  If you're still using the old run-remstats (see the run-remstats section) , modify the
\texttt{collectors} line in the general (see the general section)  config-file to
exclude snmp.  If you're using run-remstats2 (see the run-remstats2 section) , just change the
run-stages (see the run-stages section)  collectors config-file to comment out the 
the lines for the various SNMP-based collectors, primarily snmp-collector (see the snmp-collector section) .


\item  \textbf{I modified the RRD definition, adding a new RRA, but 
remstats is ignoring it.  How do I do this?}
Sorry.  At the moment, remstats won't propagate any changes to the
RRD structure after creation.  Some changes, like extending RRAs
and changing min/max for DSs can be done manually with rrdtool.
If you do use rrdtool manually, I recommend that you modify the
rrd definition as well, to keep them in sync.  At some point in the
future, I'd like to try to do this kind of update, and it's more
likely to succeed if remstats' rrd definition matches what's in
the actual RRD.


\item  \textbf{How do I move the remstats installation to another directory or
machine?}
The best way to do this is to a complete new installation in the new location
and then copy the \texttt{config} and \texttt{data} directories to the new installation.
As usual, make sure that the files and directories end up owned by the
remstats user (see the remstats user section) .
=back 4
%------------------------------------ conventions.pod ---

\section{Documentation Conventions}%
\index{Documentation Conventions}

The only documentation conventions the reader has to know about are:

\begin{itemize}


\item things inside [square brackets] are optional

\item parenthesized lists with the items separated by vertical bars,
(like | this | one) require that you choose one and only one of the 
alternatives.

\end{itemize}
Everything else ought to be explicit.  If it isn't, or if you don't 
understand it, please bring it to the author's attention, stating
which part you don't understand.  There's not a lot of point in my
writing documentation which no-one else can understand.  I'd rather
do it right.
%------------------------------------ required.pod ---

\chapter{Prerequisites}
\index{Prerequisites}

\section{What you need to install remstats}%
\index{What you need to install remstats}

\begin{enumerate}


\item  You'll need perl (see \textbf{http://www.perl.org/}), at least version 5.005\_03.  
If you don't already have it you can get it from 
CPAN (see \textbf{ftp://ftp.crc.ca/pub/packages/lang/perl/CPAN/src/stable.tar.gz})
(the Comprehensive Perl Archive Network).

\item  You'll need a \texttt{C} compiler that works. :-)  gcc or 
egcs (see \textbf{ftp://ftp.crc.ca/pub/packages/egcs/}) will do fine and 
you can find them easily in many different places.

\item  Make sure you have the following perl modules installed 
(most of which you can find at CPAN (see \textbf{http://www.cpan.org/modules/by-module})).
The versions are the versions I'm using, but more recent versions should work 
too, unless there have been radical changes.  They should be installed in 
the listed order to avoid dependency problems:
\begin{itemize}


\item RRDs 1.0.33 (see \textbf{http://remstats.sourceforge.net/release//src/rrdtool-1.0.33.tar.gz})
- the key piece.  It comes with RRDtool and does the database and graphs.
Originally from 
http://ee-staff.ethz.ch/\~{}oetiker/webtools/rrdtool (see \textbf{http://ee-staff.ethz.ch/\~{}oetiker/webtools/rrdtool}).

\item Socket - should be standard if you've got the required perl

\item IO::Socket - should also be standard in the required version of perl

\item Time::HiRes 01.20 (see \textbf{http:src/Time-HiRes-01.20.tar.gz})
- used by the port-collector (see the port-collector section)  and the dbi-collector (see the dbi-collector section)  and probably a few
others, to determine response time.  You can comment out the 
\texttt{"use Time::HiRes"} line and the programs will still work, but the 
response-time resolution will be one second instead of one milli-second.  
Originally from 
CPAN (see \textbf{ftp://ftp.crc.ca/pub/packages/lang/perl/CPAN/modules/by-module/Time}).

\item SNMP\_Session 0.83 (see \textbf{http:src/SNMP\_Session-0.83.tar.gz})
- used by the snmp-collector and a few other SNMP-related tools.  If you 
don't need SNMP, you can leave it out, but you'll have to change the 
general config file (see the general config file section) .  (Modify the 
\texttt{collectors} line to leave out \texttt{snmp}.)  Or, if you're using
run-remstats2 (see the run-remstats2 section) , you'll have to change the 
run-stages (see the run-stages section) /collectors configfile.  Originally from 
ftp://ftp.switch.ch/software/sources/network/snmp/perl/ (see \textbf{ftp://ftp.switch.ch/software/sources/network/snmp/perl/}).

\item GD (see \textbf{http:src/GD-1.30.pm.tar.gz}) - used only by dataimage.cgi (see the dataimage.cgi section)  to create images on the fly.
Originally from http://stein.cshl.org/WWW/software/GD/GD.html (see \textbf{http://stein.cshl.org/WWW/software/GD/GD.html}).

\item Win32::Daemon (see \textbf{http://www.roth.net/perl/packages/}) - Used only by the nt-status-server (see the nt-status-server section)  so that it can run as a Win32
service.

\item Win32::PerfLib (see the Win32::PerfLib section)  - Used only by the nt-status-server (see the nt-status-server section)  to obtain performance counters.

\item \texttt{DBI} - used only by the dbi-collector (see the dbi-collector section) 

\item \texttt{Pod::Pdf} - used only by podpdf (see the podpdf section)  to create book.pdf.  Only needed by people who are
changing the remstats documentation, not just using it.

\item \texttt{Time::Local} - used by availability-report (see the availability-report section) , 
availability-report.cgi (see the availability-report.cgi section) , nt-status-server (see the nt-status-server section) ,
and showlog.cgi (see the showlog.cgi section) , but it ought to be part of the required version of perl.

\item \texttt{Net::YahooMessenger} and \texttt{Crypt::PasswdMD5}
- used by the alert-yahoo (see the alert-yahoo section)  alerting method.

\end{itemize}

\item  You'll also need the following programs for the unix-status-server (see the unix-status-server section) .
(You can change the locations at the top of it.)  You almost certainly
have most of these and can ignore any that you don't tell the 
\texttt{unix-status-collector} to query.  For details, look in the
unix-status-server (see the unix-status-server section)  docs:
\begin{verbatim}%
	uname vmstat df uptime netstat ps ftpcount qmail-qstat\end{verbatim}


\end{itemize}

\end{enumerate}%------------------------------------ install.pod ---

\chapter{Installation}
\index{Installation}

\section{How to install remstats}%
\index{How to install remstats}

READ THE RELEASE NOTES (see the RELEASE NOTES section)  FIRST.  This page is generic 
and does \textbf{NOT} include version-specific instructions.

I know that this is not simple.  I do plan to make it simpler, but it'll 
\textbf{never} be "\texttt{./configure; make; make install}" because I don't know what 
you want to monitor.  

The two C programs (multiping (see the multiping section)  and traceroute (see the traceroute section) ) now use autoconf, and the
main configure script works (from the outside) simlarly to an 
autoconf-generated configure.  I haven't seen a need to convert it to 
autoconf yet.  It's mostly perl scripts and if you have the right version 
of perl properly installed, it shouldn't need
anything special.  The \texttt{unix-status-server} is a slight exception to this,
but the only configuration needed so far is done dynamically and is only the 
location of the various required utilities.

\begin{enumerate}


\item  Unpack the distribution tarball:
\begin{verbatim}%
	gunzip -dc remstats.tar.gz | tar xf -\end{verbatim}


\item  Create the remstats user and group, if you haven't already,
(by default \texttt{remstats} and \texttt{remstats} respectively.)  (See also 
the remstats user (see the the remstats user section) .)  Unless you plan to run a separate
instance of the webserver for remstats, you'll also want to create the
webgroup (see the webgroup section)  group.

\item  Build and install the software.  If you're upgrading, you
might want to take a copy of fixup.config from the old version:
\begin{verbatim}%
	sh configure\end{verbatim}

If you want to override the defaults, then run 

\begin{verbatim}%
	sh configure --help\end{verbatim}

for a list of what can be overridden.  

[Check fixup.config to make sure it is properly setup.]

\begin{verbatim}%
	make all
	make upgrade
	make install
	su - 'make install-suid'\end{verbatim}

\textbf{Note:} this step also customizes the programs and documentation 
with your choice of directories, owner, ...  so this documentation
should refer to your setup after you've done the install.

The \texttt{make install-suid} simply makes traceroute and multiping suid root.
They won't work most places unless run as root, one way or another.  Since I
don't like to run all of remstats as root, this was the best compromise I
could come up with.


\item  Fix the config-base for site-specific things.  Edit the following
files in /home/remstats/etc/config-base, looking the the string "FIXME", without the "quotes".
\begin{verbatim}%
	alerts general html scripts/http-proxy\end{verbatim}

I'll try to keep this list up to date, but you can make sure by doing:

\begin{verbatim}%
	grep -l FIXME /home/remstats/etc/config-base/* /home/remstats/etc/config-base/*/*\end{verbatim}


\item  Make a config-dir (see the config-dir section)  to describe what you 
want to monitor.  You can do this by hand, or using the configuration 
building tools.  To use the tools, you'll have to make a few files 
listing various kinds of hosts:
\begin{verbatim}%
	cd /home/remstats/etc
	/home/remstats/bin/new-config config
	/home/remstats/bin/new-ping-hosts groupname1 group1-hosts-file
	/home/remstats/bin/new-ping-hosts groupname2 group2-hosts-file
	...
	/home/remstats/bin/new-port-hosts groupname3 port-hosts-file
	/home/remstats/bin/new-snmp-hosts groupname4 SNMP-community-string snmp-hosts-file\end{verbatim}

After you've installed the unix-status-server (see the unix-status-server section)  on some hosts, you can also
use:

\begin{verbatim}%
	/home/remstats/bin/new-unix-hosts groupname5 unix-hosts-file\end{verbatim}

If you have any Windows NT hosts that you want to monitor, after you
have installed the nt-status-server (see the nt-status-server section) , you can run nt-discover (see the nt-discover section)  to
find and add the NT hosts for a given NT domain.

If you're going to use the log-collector, you'll have to build the 
rrd entries for each by hand.  There doesn't seem to be much standard 
in where log-files go, let alone what's in them.


\item  Arrange for cron to run run-remstats2 (see the run-remstats2 section)  at an appropriate interval.  
For a five-minute interval, something like the following will do:
\begin{verbatim}%
	0,5,10,15,20,25,30,35,40,45,50,55 * * * * /home/remstats/bin/run-remstats2\end{verbatim}

This checks the configuration, collects the new data, updates the rrds, 
runs the monitors to compute statuses and updates the web-pages.  Note:  it
does \textbf{not} re-write the web-pages for every iteration; it only does so when
configuration files have changed, as the web-pages will show new data by
themselves.  This and more are in the sample \texttt{crontab.new} file under
\texttt{/home/remstats/etc}.


\item  [optional] Arrange for cron to run \texttt{do-traceroutes} at an appropriate
interval.  You could run it in the wee hours of each morning like:
\begin{verbatim}%
	5 3 * * * /home/remstats/bin/do-traceroutes\end{verbatim}

This information isn't currently used, but I'm planning to make use of it.


\item  [optional] Arrange for cron to run snmpif-description-updater (see the snmpif-description-updater section) 
periodically, if you have any  snmpif-* RRDs, which you're likely to change
the descriptions on.  Say every day, like:
\begin{verbatim}%
	0 3 * * * /home/remstats/bin/snmpif-description-updater\end{verbatim}


\item  Arrange for cron to run remstats-cleanup (see the remstats-cleanup section)  every now and then to
remove old un-needed files, like:
\begin{verbatim}%
	0 3 * * * /home/remstats/bin/remstats-cleanup\end{verbatim}

This removes no-longer-needed files, like old host graphs, traceroute results,
log-files, ...  It does \textbf{not} remove rrd and status files, on the principle
that you might want them and they can't be re-created.


\item  You'll need to set up your web-server (see the web-server section)  to allow 
CGI scripts in the remstats html tree and make sure that you're not 
allowing everyone in.

\item  Make a symlink in the html directory from whichever index 
you prefer to index.cgi.

\item  You'll want to look at the server installation docs (see the server installation docs section) 
if you're going to be running any of the remote servers (
log-server (see the log-server section) , nt-status-server (see the nt-status-server section) , remoteping-server (see the remoteping-server section) , and 
unix-status-server (see the unix-status-server section) ).

\end{enumerate}
Enjoy your pretty pictures and I hope that you find them usefull.
%------------------------------------ install-user.pod ---

\section{The remstats user}%
\index{The remstats user}

You \textbf{must} choose a userid to run the remstats processes under.
By default, it will be the user \texttt{remstats}, but you'll have to
create it manually, as I'm not going to risk damaging someone's
/etc/passwd file.  Many operating-systems have a script called
\texttt{useradd} or \texttt{adduser} or some variant on that.

\textbf{NOTE:} Don't run the remstats programs except as the remstats users.
Many of the programs write extra files you won't know about unless
you read the source, and when you do run them as the remstats user,
it won't be able to modify the files that were created by the other
user.  This will probably cause the program to die, with a meaningful
error message I hope, and you'll have to modify the owner by hand, as
root.  If you need to do this, go back to the source directory and 
do:

\begin{verbatim}%
	\% su - 'make install-owner'\end{verbatim}

The remstats user must be able to write files within the remstats
directory trees rooted at /home/remstats, /home/remstats/data and /home/remstats/html.
The collection/update processes will also create files under 
/home/remstats/tmp and /home/remstats/data.  The pagemakers write files under
/home/remstats/html.  It's simplest to have all the remstats files and 
directories(except multiping (see the multiping section)  and traceroute (see the traceroute section) ) owned by 
the remstats user.

You must also ensure that the CGI scripts (and almost every web-page
remstats creates is a CGI script) get run by the remstats user.  The
CGI scripts read files under /home/remstats/data and /home/remstats/datapage.
(See also the web-server installation (see the the web-server installation section) .

There is a recently added feature in \texttt{configure} which permits you to specify
a \texttt{webgroup}, which is the group the web-server runs as.  If you are using
this feature, you need to add the remstats user to the \texttt{webgroup} group and
logout and back in so that the remstats user has access to this group to
\texttt{chgroup} appropriate directory trees, primarily images, that get written
as the web-server.
%------------------------------------ webgroup.pod ---

\section{The Webgroup Group}%
\index{The Webgroup Group}

This group isn't required if you're going to run a separate instance of the
web-server, running under the remstats user id.  However, if you want to
use an existing web-server and just tell it to serve the remstats pages
as well, it's the easiest clean way to set things up.

The reason it is required is that the graphs images are written to files 
at run-time.  Run-time means here that they are written by whichever user
the web-server is running as, often something like 'httpd'.  The remstats
user needs to be able to re-write the web-pages and create directory trees
for the pages and images.  In fact, it needs to be able to create the
directories that the images go into.

So we create a 'webgroup' group.  Let's call it 'xxx' for this explanation.
This group has two members: the web-server user and the remstats user, so the
line in /etc/groups might look like:

\begin{verbatim}%
	xxx:x:219:httpd,remstats\end{verbatim}

When the page-writer (see the page-writer section)  creates the directory tree for the graph images
(called GRAPHS), it creates the new directories with mode 2775 and group 
'webgroup' ('xxx' in our example).  This makes the directories writable by
the CGI scripts running under the web-server, and the set-group-id bit will
cause the group to be propagated to files created under this directory on 
most unix variants.
%------------------------------------ cookies.pod ---

\section{Magic Cookies}%
\index{Magic Cookies}

There are various places in the configuration file where you can have 
text substituted for you.  It's a (very-limited) macro facility.  Currently,
it only works in graphs, scripts and tools.  The cookies are always 
UPPERCASE and the name is surrounded by "\#\#", so that a request for 
"color1" would look like "\#\#COLOR1\#\#", without the quotes.

Here they are:

\subsection{Colours: }%
\index{Colours: }

You're not required to use these, but you'll really regret not doing so
when you decide to change colors later.

\begin{itemize}


\item COLOR1, COLOR2, ... COLOR6 and also COLOR1a, ...
- generic colours for graphs

\item PROBLEMCOLOR - an alarming colour

\item TOTALCOLOR - the full amount of something

\item USEDCOLOR - how much is used of something

\end{itemize}
\subsection{Other stuff:}%
\index{Other stuff:}

\begin{itemize}


\item DB - the full path and file-name of the current rrd

\item CGIDIR - where the CGI programs live

\item CGIURL - where the CGI programs live in web-space

\item DATADIR - where the data files live

\item GRAPH - the name of the current graph

\item GRAPHTIME - the name of the current time-span

\item HOST - the name of this host

\item HOSTDATADIR - where the current host's data files live

\item HOSTDESC - the description of this host

\item HTMLDIR - where the html and graphs live

\item HTMLURL - where the html and graphs live in web-space

\item IP - the IP number of the host

\item IPORHOST - the IP number of the host, if it's defined in the host's
config-file, or else its name.

\item RRD - the name of the current RRD, without the .rrd extension or
file-name fixing

\item RRDDESC - the description of the current RRD

\item SHORTHOST - the name of the host before the first dot, 
unless it is a generic name like www or ftp or mail, in which case we use
the next part unconditionally.

\item THUMBHEIGHT, THUMBWIDTH - how large to ask for the thumbnail 
to be.  Not how large the resulting gif is.

\item WEBMASTER - who is responsible for this web presense

\item WILDPART - the instance part of a wild rrd.  If if-* 
is being used by if-le0, then WILDPART will be le0.

\end{itemize}%------------------------------------ private.pod ---

\section{private.pl - configuration-supplied functions}%
\index{private.pl - configuration-supplied functions}

There are several places where you can have functions you choose invoked
by remstats:

\begin{itemize}


\item  updater (see the updater section)  permits functions to be applied to incoming data via the
rrd definition (see the rrd definition section) .

\item  datapage.cgi (see the datapage.cgi section)  and 
dataimage.cgi (see the dataimage.cgi section)  permit functions to be applied on
any \texttt{eval} line.

\end{itemize}
Rather than me continually adding functions or you having to hand-modify
your copy of remstats whenever I make new releases, I've supplied an
almost empty perl file called \texttt{private.pl} which will be installed in the
/home/remstats/lib if you don't have one, but will never be modified by me after
that.
%------------------------------------ install-servers.pod ---

\section{Server Installation}%
\index{Server Installation}

One of the interesting things about remstats (I think), is 
the remote servers (see the remote servers section) .  To install them, you'll need to do a few
things on each host which will run the servers:

\begin{itemize}


\item Add entries for the servers in \texttt{/etc/services}, like
this:
\begin{verbatim}%
	unix-status	1957/tcp	\# remstats unix-status server
	log-server	1958/tcp	\# remstats log server\end{verbatim}

You can run them on different ports, but these are the 
defaults and you'd have to change run-remstats (see the run-remstats section) 
to add the appropriate switches.


\item [Optional] Unless you're going to run the servers as 
root (unnecessary), you'll need to create the user that 
the servers will run as.  
The only reasons I can think of for running the servers as 
root are if you need to run them on a low-numbered port
($<$=1024), or if you need to read a log-file which isn't 
readable by the remstats user, or if you want to run
multiping non-suid.  On Linux and Solaris, you can do:
\begin{verbatim}%
	groupadd remstats
	useradd -g remstats -d /home/remstats remstats\end{verbatim}


\item Modify \texttt{/etc/inetd.conf} to get the servers invoked,
like this:
\begin{verbatim}%
	unix-status	stream tcp nowait remstats /home/remstats/unix-status-server unix-status-server
	log-server	stream tcp nowait remstats /home/remstats/log-server log-server logfile1 logfile2\end{verbatim}

Or if you're using 
tcp\_wrappers (see \textbf{ftp://ftp.porcupine.org/pub/security/tcp\_wrappers\_7.6.BLURB}), 
which you should be:

\begin{verbatim}%
	unix-status	stream tcp nowait remstats /path/to/tcpd /home/remstats/unix-status-server
	log-server	stream tcp nowait remstats /path/to/tcpd /home/remstats/log-server logfile1 logfile2\end{verbatim}

And remember to update \texttt{/etc/hosts.allow} to allow your remstats host access.

\begin{verbatim}%
	unix-status:remstats-host-name-here:allow
	log-server:remstats-host-name-here:allow\end{verbatim}


\item Tell inetd to re-read it's config-file:
\begin{verbatim}%
	kill -HUP pid-of-inetd\end{verbatim}


\item copy the remstats servers to the machines which will run them
\begin{verbatim}%
	scp unix-status-server log-server remoteping-server multiping host:/home/remstats\end{verbatim}


\end{itemize}
\section{The nt-status-server}%
\index{The nt-status-server}

This one is a bit different to install.  I've only done it under the
ActiveState Perl (see \textbf{http://www.activestate.com/Products/ActivePerl/index.html}) under
Windows NT 4.0.  Installing the ActiveState Perl is straightforward; if you got here,
you'll have no trouble with that.  Installing it as a service is not as simple as I
intended.  You'll have to get the SRVANY and INSTSRV programs from the NT Resource Kit,
and follow their instructions.  The program that SRVANY will be running is, of course,
perl (usually \texttt{C:$\backslash$Perl$\backslash$bin$\backslash$perl.exe}), and the argument string something like:

\begin{verbatim}%
	c:$\backslash$wherever$\backslash$you$\backslash$put$\backslash$nt-status-server -s -t 10.111.12.13\end{verbatim}

You'll have to replace \texttt{c:$\backslash$wherever$\backslash$you$\backslash$put} with the path to nt-status-server (see the nt-status-server section) ,
and \texttt{10.111.12.13} with the IP number of the host running the nt-status-collector (see the nt-status-collector section) .

Or you can run it not as a service like:

\begin{verbatim}%
	c:$\backslash$wherever$\backslash$you$\backslash$put$\backslash$nt-status-server -s -t 10.111.12.13 -s -T\end{verbatim}

but you'll have to start and stop it by hand at the console.

It also needs the \texttt{Win32::Daemon} module, as noted on the required (see the required section)  
page.  The simplest way to install this (under ActiveState), is to use the 
\texttt{PPM} program that came with ActiveState Perl.  Like this:

\begin{verbatim}%
	c:$\backslash$$>$ ppm
	ppm$>$ set repository roth http://www.roth.net/perl/packages/
	ppm$>$ install Win32::Daemon
	ppm$>$ exit\end{verbatim}

Currently, the code needs this module, even if it's invoked with the \texttt{-s} flag, which
doesn't use the \texttt{Win32::Daemon} code at all.
%------------------------------------ install-webserver.pod ---

\section{Getting your web-server ready for remstats}%
\index{Getting your web-server ready for remstats}

\subsection{Choosing userid for remstats}%
\index{Choosing userid for remstats}

Almost all the remstats web-pages are generated by some kind of CGI
script.  Many of them will read additional files not available under the
html directory tree.  In order to provide access to these files, 
the simplest way is to make sure that the scripts get run as the 
remstats user.  You could also accomplish this by making the web-server
user a member of the remstats group.  The simplest way to make sure that
the scripts can read all the files they need to, is 
to run a separate instance of the web-server software
as the remstats user.  You may have other methods of accomplishing this,
depending on the web-server you're using.  (See also 
remstats user (see the remstats user section) .)  Note that some scripts also require
write access, specifically datapage.cgi (see the datapage.cgi section)  (sometimes),
alert.cgi (see the alert.cgi section)  (sometimes) and 
log-event.cgi (see the log-event.cgi section)  (always).

\subsection{Running CGI scripts under the remstats tree}%
\index{Running CGI scripts under the remstats tree}

You also may need to tell your web-server that \texttt{xxx.cgi} means that this
file is a CGI script and needs to be run, instead of just displayed.  With
the apache (see \textbf{http://httpd.apache.org}) web-server, you could add the following
lines to the \texttt{httpd.conf} file:

\begin{verbatim}%
	$<$Directory /home/remstats/html$>$
	Options FollowSymlinks ExecCGI
	AddHandler cgi-script .cgi
	$<$/Directory$>$\end{verbatim}

and

\begin{verbatim}%
	ScriptAlias /remstats/cgi-bin /home/remstats/cgi-bin
	$<$Directory /home/remstats/cgi-bin$>$
	Options FollowSymlinks ExecCGI
	$<$/Directory$>$\end{verbatim}

\subsection{Restricting access to CGI scripts}%
\index{Restricting access to CGI scripts}

[You should also look at the access config-file (see the access config-file section) .]

There are a few things you should do before telling others about remstats.
Remstats comes with a few CGI scripts which you probably don't want to make
publicly available and two that you certainly don't.  \texttt{ping.cgi}, 
\texttt{traceroute.cgi} and \texttt{whois.cgi} should probably be restricted to your 
own organization, unless you don't mind letting anyone on the Internet run 
pings, traceroutes and whois queries from your domain.  Rectricted to your 
domain, you only have to worry about your own people.

However, \texttt{alert.cgi} and \texttt{log-event.cgi} are a different kettle of fish.  
They will permit anyone who can run it to quench alerts and log comments 
about them.  You will probably want to be a bit more restrictive about 
who you let run this.

Using the apache (see \textbf{http://httpd.apache.org/}) web-server, you can restrict 
the use of these CGIs using a \texttt{.htaccess} file something like this:

\begin{verbatim}%
	\# Note that this example uses the private network 192.168.0.0.
	\# Stuff to make Apache expire the files to get them refreshed
	ExpiresActive on
	\# images every 5 minutes, when the data gets updated
	ExpiresByType image/gif M300
	ExpiresByType image/png M300
	\# html every day
	ExpiresByType text/html M300\end{verbatim}

\begin{verbatim}%
	\# What to allow
	Options ExecCGI FollowSymlinks Indexes\end{verbatim}

\begin{verbatim}%
	$<$Files "\^(whois.cgi|traceroute.cgi|ping.cgi)\$"$>$
	order deny,allow
	deny from all
	allow from 192.168. 127.0.0.1
	$<$/Files$>$\end{verbatim}

\begin{verbatim}%
	$<$Files "\^(alert.cgi|log-event.cgi)\$"$>$
	order deny,allow
	deny from all
	allow from 192.168.20.1 192.168.23.3
	$<$/Files$>$\end{verbatim}

\begin{verbatim}%
	\# How they're allowed in
	order deny,allow
	allow from all\end{verbatim}

I won't claim the IP\#-based access-control is completely safe, but it's 
easy and keeps out casual browsers.  If you \textbf{really} need to keep
this information safe, use a secure web-server, say apache with mod\_ssl.
If that's not good enough, you ought to consider whether this stuff 
really belongs on a network at all.
%------------------------------------ configuration.pod ---

\chapter{The Configuration Directory}
\index{The Configuration Directory}

The run-time configuration of remstats is done through a directory-tree of files.
The current tree structure is:

\begin{verbatim}%
    configdir
        +--- access (see the access section) 
        +--- alert-template-map (see the alert-template-map section) 
        +--- alert-destination-map (see the alert-destination-map section) 
        +--- alert-templates (see the alert-templates section) 
        |       +--- template1
        |       +--- template2
        |       ...
        +--- alerts (see the alerts section) 
        +--- archives (see the archives section) 
        +--- availability (see the availability section) 
        +--- colors (see the colors section) 
        +--- customgraphs (see the customgraphs section) 
        |       +--- graph1
        |       +--- graph2
        |       ...
        +--- dbi-connects (see the dbi-connects section) 
        |       +--- connect1
        |       +--- connect2
        |       ...
        +--- dbi-selects (see the dbi-selects section) 
        |       +--- select1
        |       +--- select2
        |       ...
        +--- discovery (see the discovery section) 
        +--- environment (see the environment section) 
        +--- general (see the general section) 
        +--- groups (see the groups section) 
        +--- host-templates (see the host-templates section) 
        |       +--- host-template1
        |       +--- host-template2
        |       ...
        +--- hosts (see the hosts section) 
        |       +--- host1
        |       +--- host2
        |       ...
        +--- html (see the html section) 
        +--- links (see the links section) 
        +--- ntops (see the ntops section) 
        +--- oids (see the oids section) 
        +--- page-templates (see the page-templates section) 
        |       +--- page-template1
        |       +--- page-template2
        |       ...
        +--- pages (see the pages section) 
        +--- remotepings (see the remotepings section) 
        +--- rrds (see the rrds section) 
        |       +--- rrd1
        |       +--- rrd2
        |       ...
        +--- run (see the run section) 
		+--- run-stages (see the run-stages section) 
		|       +--- stage1
		|       +--- stage2
		|       ...
        +--- scripts (see the scripts section) 
        |       +--- script1
        |       +--- script2
        |       ...
        +--- times (see the times section) 
        +--- tools (see the tools section) 
        +--- view-templates (see the view-templates section) 
        |       +--- template1
        |       +--- template2
        |       ...
        +--- views (see the views section) 
        |       +--- view1
        |       +--- view1
        |       ...\end{verbatim}

(You can look at the base configuration directory if 
you want, but you should also read through this so you know the 
significance of what you see.

Almost all the configuration files allow both blank lines and comment-lines.
A comment-line \textbf{begins} with a \textbf{\#} and the whole line is ignored by 
remstats.  Inline comments are \textbf{not} permitted as the '\textbf{\#}' is used 
in some places for other purposes.  The only files which don't permit comments
are the \texttt{view-templates}, which are html.

\texttt{alert-templates}, \texttt{customgraphs}, \texttt{dbi-connects}, \texttt{dbi-selects},
\texttt{hosts}, \texttt{host-templates}, \texttt{rrds}, \texttt{run-stages}, \texttt{scripts},
\texttt{views} and \texttt{view-templates} are sub-directories with the files within
describing one of that kind of entity.  E.G. a file in the \texttt{hosts}
sub-directory is named for the host and contains that host's configuration.

\textbf{NOTE}: within the sub-directories, files with names beginning with 'IGNORE-' 
or ending with '\~{}', will be ignored.

There are also a few tools (see the tools section)  to help
you make and update your config-dir, although not all parts of it.
%------------------------------------ configfile-access.pod ---

\section{Configuration - access}%
\index{Configuration - access}

The first matching record, (all fields match or are "*", which matches
anything), is used and grants the access specified.

Any of the fields may be "*" except the "access" field.  Any fields which
are not set to "*", must match the correcponding data.  E.G. if the 
"program" is set to "xyzzy", then the record only applies to the 
"xyzzy" program and if all the other fields match, then the user is 
granted the access specified.

The fields are: access program user group ip/host.

\begin{itemize}


\item "access" is either "allow" or "deny", with obvious meaning.

\item "program" is the name of the program (whatever is set in \$main::prog).

\item "user" is from the CGI variable REMOTE\_USER

\item "group" is a group that the "user" belongs to, as
implemented in the htpasswd program supplied by the apache web-server.

\item "ip/host" is either a domain-name, a domain-name suffix (indicated
by a leading "."), an IP-number, or an IP-number prefix (indicated by
a trailing ".").  A domain-name suffix matches if the name that REMOTE\_ADDR
resolves to ends with the suffix.  Similarly, an IP-number prefix matches
if REMOTE\_ADDR begins with the prefix.

\end{itemize}%------------------------------------ configfile-alerts.pod ---

\section{Configuration - Alerts}%
\index{Configuration - Alerts}

The alerts config-file is used by the alert-monitor (see the alert-monitor section)  to decide who to 
send alerts for problems.  The rrds (see the rrds section)  and hosts (see the hosts section) 
config-files decide \texttt{when} an alert needs to be raised, and these lines tell \texttt{who}
gets the alert.

Each line is in seven parts, most of which are patterns, e.g.:

\begin{verbatim}%
	warn	* MISC UPTIME 0 0 uptime-alerts
	error   silverlock.dgim.crc.ca  * * 600 900 test-alerts
	error   news.crc.ca * * 600 900 news-alerts
	critical * * * 600 900 critical-alerts\end{verbatim}

The first "word" is the status, as decided by the alert-monitor (see the alert-monitor section) .
The second word is a regex to match against the hostname.  The third is
a regex to match against the rrdname.  The fourth is a regex for the 
variablename.  The fifth is the minimum time for the alert condition
to be present before an alert can be triggered.  The sixth is the interval
after sending an alert before another will be sent.  The seventh is
an \texttt{alert-destination} as specified in the 
alert-destination-map (see the alert-destination-map section) .

Note: The seventh used to be an alert program and there was an eighth which was
an address, of a form appropriate to the alert program.  This has been
rolled into the \texttt{alert-destination-map} to make it more flexible.

If the current condition matches the status, host, rrd, and variable,
then alert-monitor has to look at the times.  If this is a new 
condition (i.e. it was in OK status previously), then an alert won't
be triggered until after the minimum time has passed.  This avoids transient
problems being reported.  If you want these to be reported, then set it to zero.
If this is an old alert, then an alert won't be
triggered until the interval time has passed since the previous alert.
If the interval is 0 (zero) then there will only be one alert at the
start-time.
%------------------------------------ configfile-alert-destination-map.pod ---

\section{Configuration - alert-destination-map}%
\index{Configuration - alert-destination-map}

This config-file specifies the mapping from an abstract alert destination,
specified in the alerts config-file (see the alerts config-file section) , and the address(es) to
send it to.  The alerter (see the alerter section)  attempts to match the abstract alert destination
against each of the \texttt{map} lines and sends alerts to any which match.

There are three kinds of lines in this config-file: \texttt{map}, \texttt{alias}, and \texttt{method}.
\texttt{Map} lines map from an abstract alert destination, listed in the alerts
config-file, to a less abstract alias, listed here.  The \texttt{alias} lines allow a
crude list capability and also permit the use of different methods to deliver the
alert.  \texttt{Method} lines tell what program to run with what arguments in order to
deliver to that type of address.

\subsection{Map Lines }%
\index{Map Lines }

A map line looks like:

\begin{verbatim}%
	map DEST TIME DOW DOM MON ALIAS\end{verbatim}

Where:

\begin{itemize}


\item DEST is an abstract alert destination, listed in the alerts config-file

\item TIME is a time-of-day specification, comma-separated time-ranges or '*'
meaning all times.  A range looks like HHMM-HHMM.

\item DOW is a day-of-the-week spec, a comma-separated list of weekdays, in
numeric form (0=sunday, 1=monday, ...) or '*' for all weekdays.

\item DOM is a day-of-the-month spec.  It's a comma-separated list of day-ranges,
where a range is a day or DD-DD, or '*' ofr all days.

\item MON is a month spec.  It's a comma-separated list of month-ranges, in
numeric form, like MM or MM-MM or '*' for all months.

\item ALIAS is the alias that this DESTination maps to during the specified
time-period.  It's defined in an alias line.

\end{itemize}
This permits different DESTinations to be sent to different people at different times,
depending on who's on duty.

\subsection{Alias Lines}%
\index{Alias Lines}

An alias line looks like:

\begin{verbatim}%
	alias ALIAS METHOD:ADDR ...\end{verbatim}

Where:

\begin{itemize}


\item ALIAS is the alias being defined

\item METHOD is an alert-delivery method (see methods below)

\item ADDR is an address which is valid for that method

\end{itemize}
This indirection permits delivery of the same alert via multiple methods, in case
one or more of the methods isn't available, as well as to different people.

\subsection{Method Lines}%
\index{Method Lines}

A method line looks like:

\begin{verbatim}%
	method METHOD COMMAND-LINE\end{verbatim}

Where:

\begin{itemize}


\item METHOD is the method being defined

\item COMMAND-LINE is the program to run with any arguments it requires.
It will be passed the alert message on stdin and the address
to send it to at the end of the COMMAND-LINE.

\end{itemize}
\subsection{An Example}%
\index{An Example}

We have three guys on different shifts who manage network operations (Tom, Dick and Harry)
during the week.  On the week-end Frank is on call.  We also want to email a copy to
an email address which collects all the alerts.
We want to send the alerts to whoever is working at the time.  Say the abstract destination
specified in the alerts config-file (see the alerts config-file section)  is \texttt{alerts}.  We might use lines
like those below:

\begin{verbatim}%
	map alerts 0600-1359 1,2,3,4,5 * * tom
	map alerts 1400-2159 1,2,3,4,5 * * dick
	map alerts 0000-0559,2200-2359 1,2,3,4,5 * * harry
	map alerts * 0,6 * * frank\end{verbatim}

\begin{verbatim}%
	alias tom email:tom@our.com email:alert-history@our.com winpopup:console
	alias dick email:dick@our.com email:alert-history@our.com winpopup:console
	alias harry email:harry@our.com email:alert-history@our.com winpopup:console
	alias frank page:555-1234 email:alert-history@our.com winpopup:console\end{verbatim}

\begin{verbatim}%
	method email /home/remstats/bin/alert-email
	method winpopup /home/remstats/bin/alert-winpopup
	method page /home/remstats/binalert-page\end{verbatim}

Note: the hypothetical \texttt{page} method isn't provided with remstats.  There are lots of different
programs to send pages.  Look at alerter (see the alerter section)  if you want to add your own methods; it's easy.
%------------------------------------ configfile-alert-template-map.pod ---

\section{Configuration - alert-template-map}%
\index{Configuration - alert-template-map}

The \texttt{alert-template-map} tells which 
alert-template (see the alert-template section)  to use for which addressee or
RRD.  The lines look like:

\begin{verbatim}%
	address	regex template\end{verbatim}

or

\begin{verbatim}%
	rrd rrd template\end{verbatim}

or

\begin{verbatim}%
	rrd rrd:variable template\end{verbatim}

Addresses are checked first.  This is to allow special mapping for devices
like pagers which can't display a lot of information.  If none of the 
special addresses match, then RRDs are checked, first with variables
then without.  An RRD can be an RRD instance, like \texttt{port-ftp}, or the 
wild RRD, e.g. \texttt{port-*}.

The \texttt{template} is the name of the template file in the \texttt{alert-templates}
config-dir.
%------------------------------------ configfile-alert-templates.pod ---

\section{Configuration - alert-templates}%
\index{Configuration - alert-templates}

The \texttt{alert-templates} directory contains the alert message templates.  They
are just text files with magic cookies (see the magic cookies section)  in them.  The cookies
available are slightly different than the standard list, but they work the 
same way.  You put \#\#COOKIENAME\#\# wherever you want to see the value of 
the 'cookiename' variable.  The ones available for alerts are:

\begin{itemize}


\item HOST - the host for the alert

\item REALRRD - the RRD instance for the alert

\item FIXEDRRD - the RRD instance with the character-set translated a bit
for file-names and message-id's

\item VAR - the variable name

\item STATUS - the alert status (OK, WARN, ERROR, CRITICAL)

\item VALUE - the value of the variable that caused this alert

\item RELATION and THRESHOLD - the alert is triggered when the VALUE is
no longer in RELATION to the THRESHOLD value.

\item START - when the alert was first noticed

\item DURATION - how long the alert has been in this STATUS

\item HOSTDESC - the description line for this host

\item RRDDESC - the description for this instance of the RRD

\item NOW - the current time as a unix timestamp

\item NOWTEXT - the current time for email headers

\item ALERTHOST - the hostname of the host sending the alert

\item TOWHO - the addressee for this alert

\end{itemize}
There are three special files in the \texttt{alert-templates} directory, 
which \textbf{must} exist:

\begin{itemize}


\item DEFAULT - which contains the default template to be used when no
other matches the alert-template-map (see the alert-template-map section) .

\item HEADERS - which supplies the headers for each message, with
the same substitutions as the rest of the template files.  Make very sure
that the HEADERS file ends with or contains an empty line or your 
message will be interpreted as part of the headers and will 
undoubtedly look wrong.  The alert-email (see the alert-email section)  script does not check this.

\item FOOTER - supplies a standard ending for each message.

\end{itemize}%------------------------------------ configfile-archives.pod ---

\section{Configuration - Archives}%
\index{Configuration - Archives}

The archives file names various data-retention periods.  RRDtool|http://people.ee.ethz.ch/\~{}oetiker/webtools/rrdtool/
calls them RRAs.  Each line is in two pieces: an archive name and an RRA
specification, exactly as documented in the rrdcreate manpage.  Unfortunately, 
modifying this after the rrd has been created isn't one of the things that RRDtool
does.  I've got an rrd munger on my L$<$to-do list|todo (see the RRDtool|http://people.ee.ethz.ch/\~{}oetiker/webtools/rrdtool/
calls them RRAs.  Each line is in two pieces: an archive name and an RRA
specification, exactly as documented in the rrdcreate manpage.  Unfortunately, 
modifying this after the rrd has been created isn't one of the things that RRDtool
does.  I've got an rrd munger on my L$<$to-do list|todo section) , but it's still
not done.

For example:

\begin{verbatim}%
	day-avg         AVERAGE:0.1:1:600
	week-avg        AVERAGE:0.1:7:300
	month-avg       AVERAGE:0.1:30:300
	3month-avg      AVERAGE:0.1:90:300
	year-avg        AVERAGE:0.1:365:300\end{verbatim}

\begin{verbatim}%
	day-min         MIN:0.1:1:600
	week-min        MIN:0.1:7:300
	month-min       MIN:0.1:30:300
	3month-min      MIN:0.1:90:300
	year-min        MIN:0.1:365:300\end{verbatim}
%------------------------------------ configfile-availability.pod ---

\section{Configuration - Availability}%
\index{Configuration - Availability}

There are two types of availability definitions: for an RRD or for an 
RRD on a particular host.  The RRD may also be a wildcard RRD, like
"df-*" or an instance of an RRD, like "df-/home".  The definitions
look like:

\begin{verbatim}%
	rrd RRDNAME VARNAME CF RELATION THRESHOLD
and
	host HOSTNAME RRDNAME VARNAME CF RELATION THRESHOLD\end{verbatim}

The CF is one of LAST, MIN, MAX or AVERAGE, with rrdtool's usual
meaning.  The RELATION can be any one of: $<$, $<$=, $>$, $>$=, or =.  
The THRESHOLD is the number to which the value of VARNAME must be 
in the correct RELATION.  (Clear as mud.)  

As an example, take the following definition:

\begin{verbatim}%
	rrd ping rcvd MINIMUM $>$ 0\end{verbatim}

This means that the variable 'rcvd' in the ping rrd must be greater 
than zero for it to be considered "available".  All time intervals 
where it isn't, or for which no data is available, are considered 
"unavailable".

There are also two other record types: colors and thresholds.  A colors
record looks like:

\begin{verbatim}%
	colors COLOR1 ...\end{verbatim}

A thresholds line looks like:

\begin{verbatim}%
	thresholds NUMBER ...\end{verbatim}

and must have the same number of values as the colors line.  Only one of
each.  Here's an example to make the use clear (I hope):

\begin{verbatim}%
	colors avail1 avail2 avail3 avail4
	thresholds 99 98 95 90\end{verbatim}

The \texttt{colors} line above requires that the colors 'avail1', ... be defined in the
colors (see the colors section)  config-file.  The \texttt{thresholds} line above specifies 
that if an availability is 99\% or above, it should be colored 'avail1' color, 
98\% to 99\%, use 'avail2' color, etc.
%------------------------------------ configfile-colors.pod ---

\section{Configuration - Colors}%
\index{Configuration - Colors}

The \texttt{colors} config-file just gives names to colors so you don't have 
to remember or decipher hex numbers.  Each line is: a colour 
name and a six-digit hex number giving the RGB values.  The color 
"down" is special and is used in the ping index to color the 
background of hosts which are down.

For example:

\begin{verbatim}%
	totalcolor      00a000
	usedcolor       a00000
	downcolor       a07777\end{verbatim}
%------------------------------------ configfile-customgraphs.pod ---

\section{Configuration - Customgraphs}%
\index{Configuration - Customgraphs}

The \texttt{customgraphs} directory contains files which define graphs which 
aren't associated with a specific host.  They are very like a graph 
defined on an rrd (see the rrd section) .  They do have a times
line preceeding the graph definition, to set the time-periods for that
graph.  They are linked in under the \texttt{Custom Index}.

Note that the graph definition itself must be indented, like rrd graphs.

As an example:

\texttt{times   day yesterday week 3month year}

\begin{verbatim}%
        --upper-limit 100 --lower-limit 0 --rigid
        --vertical-label 'CPU \%'
        --title 'CPU Usage (\#\#GRAPHTIME\#\#)'
        DEF:silverlockuser=\#\#DATADIR\#\#/silverlock.dgim.crc.ca/cpu.rrd:user:AVERAGE
        DEF:silverlocksys=\#\#DATADIR\#\#/silverlock.dgim.crc.ca/cpu.rrd:system:AVERAGE
        DEF:loisuser=\#\#DATADIR\#\#/lois.dgim.crc.ca/cpu.rrd:user:AVERAGE
        DEF:loissys=\#\#DATADIR\#\#/lois.dgim.crc.ca/cpu.rrd:system:AVERAGE
        CDEF:silverlock=silverlockuser,silverlocksys,+
        CDEF:lois=loisuser,loissys,+
        'LINE2:silverlock\#\#\#COLOR1\#\#:silverlock'
        'LINE2:lois\#\#\#COLOR2\#\#:lois'\end{verbatim}
%------------------------------------ configfile-dbi-connects.pod ---

\section{dbi-connects }%
\index{dbi-connects }

The \texttt{dbi-connects} config-dir is used by the dbi-collector (see the dbi-collector section)  to specify
how to connect to a database.  Each file in this directory is named for the
connection, which is used in the RRD definition.  There are three directives 
which may occur in these files:

\begin{itemize}


\item \textbf{dbisource} - specifies a DBI source string.  There are three parts to
it, separated by colons (':').  The first part is always '\texttt{dbi}'.  The
second part is the DBD (DataBase Driver) name, e.g. 'Oracle' for an Oracle
database.  The final part is database-specific, so you'll have to look it
up under \texttt{DBD::Whatever}.  As an example, for Oracle, you might have a
\texttt{dbisource} like:
\begin{verbatim}%
	dbisource dbi:Oracle:MyDbName\end{verbatim}

or

\begin{verbatim}%
	dbisource dbi:Oracle:host=my.host.com;sid=MYSID\end{verbatim}


\item \textbf{user} - specifies the username to use to connect to this database

\item \textbf{password} - specifies the password for that user

\end{itemize}
The \texttt{user} and \texttt{password} are optional, and can be specified on the RRD
directive of the appropriate host's config-file.  In addition, the string
can be parameterized using remstats usual magic cookies (see the magic cookies section) 
mechanism, in this case restricted to USER, PASSWORD, SELECT, DATABASE and HOST.
%------------------------------------ configfile-dbi-selects.pod ---

\section{dbi-select }%
\index{dbi-select }

The \texttt{dbi-select} config-dir contains files which each contain a single
SQL select statement, formatted however you like.  An rrd which is
collected by the \texttt{dbi-collector} will specify both a \texttt{connect} directive
(dbi-connects config-dir (see the dbi-connects config-dir section) 
directive.  The \texttt{select} directive specifies the name of the select, which
is defined in a file of the same name.  All names are lower-cased, so do
not make file-names of mixed or upper case.
%------------------------------------ configfile-environment.pod ---

\section{environment config-file}%
\index{environment config-file}

\subsection{Description: }%
\index{Description: }

It contains environment variables and values to be supplied by run-remstats (see the run-remstats section) 
to its children.  Each line looks like a name, whitespace and everything else
on the line is the value:

\begin{verbatim}%
	XYZZY	Everything else on the line\end{verbatim}
%------------------------------------ configfile-general.pod ---

\section{Configuration - General}%
\index{Configuration - General}

This is the miscellaneous config-file, but there are some critical pieces here:

\begin{itemize}


\item \texttt{datadir} (REQUIRED) - 
The data for a given host is stored under \texttt{datadir/hostname}.  
There are also other status files stored in this directory.

\item \texttt{staletime} (UNUSED) - 
How long before we count a status as stale. (seconds)

\item \texttt{minuptime} -
How long a host must be up before it stops being flagged as recently up.
(seconds)

\item \texttt{keepalerts} (UNUSED) -
How long to keep records of alerts after the condition no longer exists.
(seconds)

\item \texttt{uptimealert} - 
If this is set, the alert-monitor (see the alert-monitor section)  will cause a warning level condition on the
fake rrd MISC for the fake variable UPTIME, for any host whose uptime
is less than this value.  Whether this will trigger an alert depends on
the alerts (see the alerts section)  file.

\item \texttt{pinger} -
If defined, this names the ping-collector (see the ping-collector section)  to be used before all the 
other collectors (see the collectors section) .  (Unless you write your own, you put \texttt{ping-collector}
here.)  If you don't include this line, then you'll want to make sure
the the \texttt{ping-collector} is listed in the \texttt{collectors} line (below).

\item \texttt{collectors} -
This line tells run-remstats (see the run-remstats section)  which collectors to run.  The default
list is all of them, so you can gain some benefit by pareing this line
down to those you are using, but remember it if you add new 
rrds (see the rrds section)  that need other collectors later. \textbf{Note:} you 
list the names of the collectors without the '\texttt{-collector}' on the
end.  E.G. the \texttt{ping-collector} would be included as just '\texttt{ping}'.

\item \texttt{monitors} -
This line tells run-remstats (see the run-remstats section)  which monitors (see the monitors section)  to run.  The default is
all of them.

\item \texttt{pagemakers} -
This tells run-remstats (see the run-remstats section)  which pagemakers (see the pagemakers section)  to run at the end, if the 
config-dir has changed.  The default is all of them.

\item \texttt{max-port-patterns} -
This tells the port-collector (see the port-collector section)  how many parenthesised patterns there can
be, at most, in \texttt{valuepattern}s or \texttt{infopattern}s.  The default is 10.

\item \texttt{watchdogtimer} -
This sets the limit that run-remstats (see the run-remstats section)  will apply to each of the programs
that it runs, so that, e.g., a hanging collector will not hang the whole
remstats cycle.

\item \texttt{keeplogs} -
This tells how long cleanup (see the cleanup section)  will permit old files to hang around, in seconds.

\item \texttt{pagetemplatedir -
This tells where the page templates for L$<$page-writer} are kept.  This
allows you to change the look of pages with a one-line change here.  Assuming
that you have a different set of page templates in another directory.

\end{itemize}%------------------------------------ configfile-groups.pod ---

\section{Configuration - Groups}%
\index{Configuration - Groups}

If any name has spaces, it must be 'quoted'
or "quoted".  Any groups not listed here \textsl{will not} be linked into the HTML 
index pages generated by page-writer (see the page-writer section) , but data will still be collected.
If there is a file in the \texttt{htmldir} with the same name as a group-name, but
with the spaces replaced by '\_' and all the letters lower-cased, followed by
'.html', then the group-names in html pages will be linked to those pages.
E.G. for a group named \texttt{"Local Routers"}, if there is a file called 
\texttt{local\_routers.html}, the name of the group will be made into a link to that
file.
%------------------------------------ configfile-hosts.pod ---

\section{Configuration - Hosts}%
\index{Configuration - Hosts}

The \texttt{hosts} files are what the whole configuration has been working 
toward.  Here we tell which hosts we're interested in and what we want to 
monitor.  Here's a sample host file called \texttt{clark.dgim.crc.ca}:

\begin{verbatim}%
	desc    DNS and Web
	ip      142.92.39.18
	aliases	ns1.crc.ca
	via	142.92.32.10
	keys	server sun
	group   Servers
	contact	Thomas Erskine $<$thomas.erskine@crc.ca$>$
	tools   ping traceroute telnet http clark-special:special
	rrd     ping 
	rrd     cpu 
	noalert	cpu user
	community	xyzzy
	rrd     load 
	nograph load users
	rrd     if-le0 
	alert	if-le0 ierr $<$ 1000 5000 10000
	alert	if-le0 in WARN
	rrd     df-/var 
	rrd     df-/tmp 
	rrd     port-http critical
	rrd     port-ssh 
	rrd     port-whois
	noavailability port-whois status
	noavailability port-whois response
	rrd     port-domain critical\end{verbatim}

The name of the file (\texttt{clark.dgim.crc.ca]}) is the host that you're
interested in.  The name should be a fully-qualified-domain-name, but anything
which perl's getaddrbyname can resolve should work.

The \texttt{ip} line saves the IP number from having to be looked up and
could be used to deal with hosts which aren't in the DNS.  If you want the
IP number to be looked up each time, you can leave this line out.

The \texttt{desc} line gives this host a description page-writer (see the page-writer section) 
will put on pages about this host.

The \texttt{alias} line tells remstats about other names for this host.  This
is mainly for the \texttt{ping-collector} to allow it to tell for sure when 
it has got a response from this host.

The \texttt{via} line is used by the topology-monitor (see the topology-monitor section)  to specify networking
gear (like hubs and switches) which are in the path to the host, but won't
show up in a traceroute.

The \texttt{keys} line is used to attach arbitrary attributes to hosts, for
selection purposes.  This is to make it easier to write scripts which
act on specified collections of hosts.  For an example, see the
update-switch-ports (see the update-switch-ports section)  script.

The \texttt{group} line is required and tells which group this host
belongs to.  Remember, you defined all the groups back in the 
groups (see the groups section)  file?

The \texttt{contact} line tells who to contact for this host.  If a line in
the alerts config-file (see the alerts config-file section)  refers to a 
recipient called \texttt{CONTACT}, the value of the host's contact line 
will be substituted.

The \texttt{tools} line tells which tools (defined in the 
tools config-file (see the tools config-file section) )
you want to appear for this host.  E.G. if a host doesn't have a 
web-server, there's no point in providing a link to connect to it.
To accomodate host-specific tools, a toolname can be given as 
\texttt{real-tool-name:display-name}.  This means that the tool will be defined
in the \texttt{tools} config-file as \texttt{real-tool-name}, but will be displayed
as \texttt{display-name}.

The rrd (see the rrd section)  lines tell which rrds to collect for this host.  
If the rrd was defined as a wildcard, it will have the instance 
specified here.  In the example there are three wildcard lines, referring 
to \texttt{if-le0}, \texttt{df-/var} and \texttt{df-/mail}.  The first is looking at the data 
for network interface hme0 and the others are getting data on the /var and 
/mail file-systems, respectively.  

[Alert definitions are fully specified in alert-monitor (see the alert-monitor section) .]

The first \texttt{alert} line is setting the alert threshold for \texttt{if-le0}
to 50.  If this host file was from the same configuration as the previous
\texttt{rrd} sample, the alert here would override the one in the \texttt{rrd} file.
There is also a \texttt{noalert} line, which cancels an alert set in the
rrd without setting a replacement alert.  The alert line for a host
must specify the rrd as well, but is otherwise the same as an alert on an
rrd.

The second \texttt{alert} line is specifying the status (\texttt{WARN}) for missing data
for the \texttt{in} variable.

There can also be descriptions for rrds.  If you append to an
\texttt{rrd} line something like \texttt{desc='xyzzy'}, then you'll see
that description on pages dealing with it.  I added this for labelling
network interfaces, but you can use if for anything you want.

The \texttt{community} specifies the SNMP community string to use for this 
host to fectch SNMP data.  If the host config-file doesn't specify any RRDs 
collected by the snmp-collector, you don't need to specify a community.

If this host uses any rrds collected by the snmp-collector, it can also
specify a port to use like:

\begin{verbatim}%
	snmpport	3401\end{verbatim}

If the RRD itself specifies a port, then the RRD-specified port will be
used instead, for that RRD.

If you don't want a particular graph for this host, you can include a 
\texttt{nograph} line.  It looks like:

\begin{verbatim}%
	nograph rrdname graphname\end{verbatim}

There can also be a \texttt{statusfile} line, looking like:

\begin{verbatim}%
	statusfile NNN\end{verbatim}

with \texttt{NNN} replaced by the name of a status file from that hosts's 
data directory.  This permits the main index pages to show the status 
of an un-pingable host as the status of something else, like the 
reachability of it's web-server (STATUS-port-http).

Related to that, there can be multiple \texttt{headerfile} directives, which look
like:

\begin{verbatim}%
	headerfile FFF LLL\end{verbatim}

with \texttt{FFF} being the name of a file in that host's data directory, and
\texttt{LLL} being a label for the header.  This can be used to inject arbitrary
information into the header.

The \texttt{noavailability} lines tell the availability-report (see the availability-report section)  program 
not to report on certain rrd/variable combinations.  In this case, 
we don't want to see availability stats on the whois server.  Maybe 
it's too embarassing?

\subsection{RRD-specific Information}%
\index{RRD-specific Information}

There may also be RRD-specific information on an \texttt{rrd} directive, anything
after the rrd-name.

Any of them can include "\texttt{desc="whatever"}" as part of the extra information.
This description will be included in the headers of graph pages for that RRD.

For RRDs collected by the port-collector (see the port-collector section) , there may optionally be the
string \texttt{critical}, which will cause a status of ERROR to be elevated to
CRITICAL.  This is a relic, which may go away, so don't count on it.  Instead,
use the \texttt{alert} directive to override the RRDs alert specifications.

For RRDs collected by the log-collector (see the log-collector section) , the extra information is not
optional, and consists of the full path to the log-file from which that
RRd's information will be collected.

For RRDs collected by the L$<$dbi-collector, there may be several different
pieces of extra information, like:

\begin{verbatim}%
  rrd rrdname CONNECT="cc" SELECT="ss" USER="uu" PASSWORD="pp" DATABASE="dd"\end{verbatim}

This permits using an RRD definition while overriding many parts of that
definition.
%------------------------------------ configfile-host-templates.pod ---

\section{Configuration - Host Templates}%
\index{Configuration - Host Templates}

These config-files simply hold the same kind of lines as a
host config-file (see the host config-file section) .  By adding a line like:

\begin{verbatim}%
	template some-host-template\end{verbatim}

to a host config-file, you achieve the same effect as adding all
the lines contained within the template file.  If you have many
hosts which are similar, this can be a usefull way of keeping the
configuration consistent.

It can also be used to parameterize things.  As an example, if you
are using the nt-status-server (see the nt-status-server section)  and are only running it on a single
host which is providing information on various other NT hosts, you
might make a template, say \texttt{default-nt-status-server} like:

\begin{verbatim}%
	nt-status-server my.nt.status.server\end{verbatim}

and replace the \texttt{nt-status-server} lines in those hosts with:

\begin{verbatim}%
	template default-nt-status-server\end{verbatim}

Then if you want to change which machine is running the \texttt{nt-status-server},
you'd just have to change the template.
%------------------------------------ configfile-html.pod ---

\section{Configuration - HTML}%
\index{Configuration - HTML}

The \texttt{html} file defines stuff related to web-page generation.  There
are several different kinds of information.

\subsection{Locations }%
\index{Locations }

These things define where things are, like URLs.  They are:

\begin{itemize}


\item \texttt{htmldir} (REQUIRED) - 
The html stuff for a given host is stored under \texttt{htmldir/hostname}.

\item \texttt{htmlurl} (REQUIRED) -
How to refer to the \texttt{htmldir} in a URL.

\item \texttt{cgidir} (REQUIRED) - 
The static cgi scripts, like ping.cgi (see the ping.cgi section) , live here.

\item \texttt{cgiurl} (REQUIRED) -
How to refer to the \texttt{cgidir} in a URL.

\item \texttt{viewdir} -
Where to store the views, in case you don't want them under the \texttt{htmldir}.

\item \texttt{viewurl} -
How to refer to the \texttt{viewdir} in a URL.

\item \texttt{webmaster} (REQUIRED) -
Who's in charge of these web-pages, an email address to get stuffed into
mailto URLs.

\item \texttt{logourl} -
Where is the logo for this site

\item \texttt{homeurl} - 
ehere is home for this site

\item \texttt{topurl} -
where top goes for this site

\item \texttt{rrdcgi} -
where to find the rrdcgi program, I like to link it and rrdtool into 
\texttt{/usr/local/bin}, for ease of use.

\item \texttt{motdfile} -
where to find the Message-Of-The-Day file.  This is used to add in 
announcements at the top of the index pages, except the host index.

\end{itemize}
\subsection{"Look and Feel"}%
\index{"Look and Feel"}

\begin{itemize}


\item \texttt{thumbnail} -
How big the graph portion of a thumbnail image is to be (WIDTHxHEIGHT)

\item \texttt{metadata} -
Where to store CERN-style meta-data, to set expiry times for the gifs.
(METADIR METASUFFIX)

\item \texttt{background} -
what should the background look like.  It's mostly obsolete, because 
you can get the most of the same effects by editing the \texttt{default.css} 
style file instead.

\item \texttt{htmlrefresh} - 
How often to cause the web pages to refresh themselves. (seconds)

\item \texttt{upstatus}, \texttt{upunstablestatus}, \texttt{downunstablestatus}, \texttt{downstatus}, 
\texttt{okstatus}, \texttt{warnstatus}, \texttt{errorstatus}, \texttt{criticalstatus} - 
HTML to display for various statuses.  The defaults use $<$span style="xxx"$>$ tags.

\item \texttt{viewindices} -
Should view-writer (see the view-writer section)  write the index links at the top of view pages? 
(yes or no)

\item \texttt{showinterfaces} -
Should graph-writer (see the graph-writer section)  show interfaces on a host page?
(yes or no)

\item \texttt{keepimages} -
How long cleanup (see the cleanup section)  will permit old images to hang around, in seconds.

\item \texttt{default-tools} -
What tools to show for a host which doesn't specify any.

\end{itemize}
\subsection{Markers }%
\index{Markers }

This group supplies html to wrap various things in the generated web-pages.

\begin{itemize}


\item indexprefix, indexsuffix - for the items on the \texttt{Indices} line
of the header

\item groupprefix, groupsuffix - for the group names on the various indices

\item hostprefix, hostsuffix - for the host names on the various indices

\item toolprefix, toolsuffix - for the tool names on the toolbar

\item linkprefix, linksuffix - for the links in the footer

\item outofrangeprefix, outofrangesuffix - for the current value on the
availability pages when it has gone outside the specified bounds.  (See
availability-report (see the availability-report section) .)

\end{itemize}
\subsection{Labels }%
\index{Labels }

If you translate the labels, the web-pages should be translated.  It
doesn't include error-messages or debugging messages.

The currently available ones, with their defaults, are:

\begin{verbatim}%
	alertreport	Alert Report
	comment		Comment
	contact		Contact
	customindex	Custom Index
	description	Description
	error		Error
	groupindex	Group Index
	hardware	Hardware
	hostindex	Host Index
	indices		Indices
	ipnumber	IP \#
	lastupdateon	This page last updated on
	links		Links
	logreport	Log Report
	operatingsystem	Operating System
	overallindex	Overall Index
	pingindex	Ping Index
	quickindex	Quick Index
	status		Status
	tools		Tools
	uptime		Uptime
	viewindex	View Index\end{verbatim}

And also the:

\begin{itemize}


\item \texttt{uptimeflag} - shows on some index pages when a host has
been up for less than \texttt{mintime} (defined in the 
general (see the general section)  file.)

\item \texttt{alertflagwarn}, \texttt{alertflagerror} and \texttt{alertflagcritical} - 
give HTML to be inserted in the quick index for hosts which have
alerts active.

\end{itemize}%------------------------------------ configfile-links.pod ---

\section{Configuration - Links}%
\index{Configuration - Links}

The links config-file supplies links that you want to put with the standard
links at the bottom of the web pages.  They're in two or three pieces.  The
simple (two-part) form is text and a URL to link to, like:

\begin{verbatim}%
	SourceWorks http://www.sourceworks.com/\end{verbatim}

The third part, if present, is assumed to be a URL for an image if it
begins with "http:", "ftp:", '/', or '../'.  Otherwise it is taken as
literal HTML and inserted without addition.

So, a simple link definition, like:

\begin{verbatim}%
	name url\end{verbatim}

produces HTML like:

\begin{verbatim}%
	$<$A HREF="url"$>$name$<$/A$>$\end{verbatim}

A three-part link with an image, like:

\begin{verbatim}%
	name url imageurl\end{verbatim}

produces HTML like:

\begin{verbatim}%
	$<$A HREF="url"$>$$<$IMG SRC="imageurl" ALT="name"$>$$<$/A$>$\end{verbatim}

Otherwise, a link definition like:

\begin{verbatim}%
	name url html\end{verbatim}

produces:

\begin{verbatim}%
	$<$A HREF="url"$>$html$<$/A$>$\end{verbatim}
%------------------------------------ configfile-ntops.pod ---

\section{ntops config-file}%
\index{ntops config-file}

This simple config-file tells the ntop-collector (see the ntop-collector section)  where the hosts running
an ntop server are.  Each line of the file contains a hostname and the port 
number on that host where the ntop server is running.
%------------------------------------ configfile-oids.pod ---

\section{Configuration - OIDs}%
\index{Configuration - OIDs}

[These are for SNMP and you can ignore this config-file if you're not interested.]

The SNMP implementation in the snmp-collector is primitive and only knows
about OIDs (Object IDs) by their number.  Since I'm not interested in bringing
in a full MIB compiler to deal with the MIBs, this section lets you specify
names for the OID numbers you're interested in using later.  The lines look
like:

\begin{verbatim}%
	CiscoCpuLoad    1.3.6.1.4.1.9.2.1.58.0\end{verbatim}

for a non-hypothetical example, if you happen to have Cisco routers.  If you
have the ucd-snmp package, their snmptranslate program comes in handy for
pulling out the appropriate numbers without the bother of tracking through
the wretched MIBs.
%------------------------------------ configfile-pages.pod ---

\section{the pages config-file}%
\index{the pages config-file}

This file specifies the pages to be generated by page-writer (see the page-writer section)  and 
requires knowledge of it.  It is 
processed sequentially from the beginning.  Each line has one directive.
The directives are taken from the tags processed by \texttt{page-writer}:

\begin{itemize}


\item var VARNAME VALUE
- this creates a variable called \texttt{VARNAME} and gives it the value \texttt{VALUE}

\item var::config VARNAME CONFIGVALUE ...
- this creates a variable from the values in the configuration hash.  In 
theory, any value can be obtained, but in practice two levels seems to be
sufficient.

\item page TEMPLATENAME FILENAME
- This causes the generation of a web-page using the \texttt{TEMPLATENAME} template
from the page-templates config-dir (see the page-templates config-dir section)  which will
be stored in the file \texttt{FILENAME}

\item chdir DIRNAME
- Changes to the directory \texttt{DIRNAME}, creating it if necessary.

\end{itemize}
Blank lines and lines beginning with a \texttt{\#} are treated as comments and 
ignored.
%------------------------------------ configfile-page-templates.pod ---

\section{The page-templates config-dir}%
\index{The page-templates config-dir}

This directory contains the templates referred to in the
pages config-file (see the pages config-file section)  and used by the page-writer (see the page-writer section) .  You
need to read the documentation for all three of these together.

Each file contains a single template.  The template is HTML with the
addition of some special $<$REMSTATS::xxx$>$ tags.  The tags are documented 
in the docs for \texttt{page-writer} as it implements them.
%------------------------------------ configfile-remotepings.pod ---

\section{Configuration - remotepings}%
\index{Configuration - remotepings}

The \texttt{remotepings} file simply lists all the machines which are
running the remoteping-server.  Or at least all the machines
that you want to query.
%------------------------------------ configfile-run.pod ---

\section{Configuration - Run}%
\index{Configuration - Run}

This config-file lists the run-stages (see the run-stages section)  which
run-remstats2 (see the run-remstats2 section)  will run, giving the order to run them in.  It also
specifies special conditions required to run them.  Each line looks
like:

\begin{itemize}


\item NAME - the name of the stage to be run

\item WHEN - the names of conditions required to run this stage.  Currently, there is
only one: \texttt{CONFIGCHANGE}

\end{itemize}%------------------------------------ configfile-run-stages.pod ---

\section{Configuration - Run-Stages}%
\index{Configuration - Run-Stages}

This sub-directory contains files, each defining the programs to be run
for a given run stage.  [The order of the run stages is listed in the
run config-file (see the run config-file section) .]  Each line contains to following
fields, whitespace separated:

\begin{itemize}


\item NAME - This gives this line a name, which will be used for error-messages and
file-names.  It's used instead of the program-name to enable multiple
instances of a given program, primarily the collectors.

\item ASYNC - If this flag is set (1, y, yes or true), then run-remstats2 (see the run-remstats2 section)  will not
wait for it to complete before moving on the the next stage.  This permits
long-running programs, on which nothing depends, to run through several
stages.

\item FREQUENCY - This is the minimum interval between runs.  \texttt{run-remstats2} will not
run this line if it has been run within this time.

\item COMMAND - This is the command-line to run.  It may be a pipeline.  The command will be augmented by the addition of output redirection for stdout and stderr to
\texttt{@@TMPDIR@@/NAME.PID}, where PID is the process-id of the instance of 
\texttt{run-remstats2} which is running it.

\end{itemize}
\subsection{Notes: }%
\index{Notes: }

Any command can have the following \texttt{\#\#MAGICCOOKIES\#\#}:

\begin{itemize}


\item RUNPID - is the process-id of the run-remstats2 (see the run-remstats2 section)  process which is
invoking the program.

\end{itemize}%------------------------------------ configfile-rrds.pod ---

\section{Configuration - RRDs}%
\index{Configuration - RRDs}

[Note: there is a list of supplied rrds (see the list of supplied rrds section)  with descriptions.]

These files are the most complicated.  Here's an example, again
taken from the \texttt{if-} rrd supplied with remstats.

\begin{verbatim}%
	collector       unix-status
	step            300
	keys			xyzzy
	data            in=interface\_packets\_in:* COUNTER:600:0:U
	data            ierr=interface\_errors\_in:* COUNTER:600:0:U
	data            out=interface\_packets\_out:* COUNTER:600:0:U
	data            oerr=interface\_errors\_out:* COUNTER:600:0:U
	data            coll=interface\_collisions:* COUNTER:600:0:U
	alert           in $<$ 100
	alert           out $<$ 100
	alert		in nodata WARN
	archives        day-avg week-avg month-avg 3month-avg year-avg
	times           day yesterday week month 3month year
	graph           if-* desc='Interface data for \#\#RRD\#\#'
			--title 'Interface \#\#RRD\#\# \#\#GRAPHTIME\#\#'
			--lower-limit 0
			--vertical-label 'packets'
			DEF:in=\#\#DB\#\#:in:AVERAGE
			DEF:out=\#\#DB\#\#:out:AVERAGE
			DEF:ierr=\#\#DB\#\#:ierr:AVERAGE
			DEF:oerr=\#\#DB\#\#:oerr:AVERAGE
			'LINE1:in\#\#\#COLOR1\#\#:Input Packets'
			'LINE1:out\#\#\#COLOR2\#\#:Output Packets'
			'LINE1:ierr\#\#\#COLOR3\#\#:Input Errors'
			'LINE1:oerr\#\#\#COLOR4\#\#:Output Errors'\end{verbatim}

This example shows most things that can be done, except multiple graphs on
the same rrd, which is as simple as adding another graph line and its
definition.

First, the rrd name is special, in this case.  Any rrd definition file 
which ends in a '-' is assumed to be for a wildcard rrd, in this 
case \texttt{if-*}.  This avoids problems with file-systems which are overly 
fussy about which characters can be in file-names.

This rrd definition will match any rrd beginning with 'if-' specified 
in a host config-file.  Wildcard rrds are necessary when a given host 
may have more than one of whatever the rrd is referring to, in this 
case network interfaces.  The network interface name will replace the 
'*' in the rrd line in the host config-file.  It will also be available 
in the \texttt{\#\#WILDPART\#\#} magic cookie (see the magic cookie section) .

The \texttt{collector XXX} tells which collector (see the collector section)  supplies the data
for this RRD.  \texttt{collector unix-status} means that this RRD gets its data 
from the unix-status-collector (see the unix-status-collector section) .  [This used to be called \texttt{source}.]

The \texttt{step} line sets the step value for the rrd.  This is the expected
frequency of data updates.  [See the manpage for rrdcreate.]
N.B. Setting this is required, but changing some RRDs won't 
change how often the collectors run.  You'll have to change the crontab
entries which run run-remstats2 (see the run-remstats2 section)  to specify the shortest interval.
However, in the run-stages config-files (see the run-stages config-files section)  you can
specify how often certain lines should be run.

The \texttt{keys} line attaches an arbitrary attribute to this rrd.  This
can be used by various programs to select subsets to operate on.  E.G.
page-writer (see the page-writer section)  can be told to use this to select only some RRDs.

(FIXME - the writing stinks)

The \texttt{data} lines define various DS elements for this RRD.  [See the
manpage for rrdcreate.]  The first part is the DS name, with an extension.
The collectors produce long names and may have instance-names added to the 
variable name, in this case to tell which interface this data is for.  So
the first part looks like \texttt{dsname=variable:instance}.  The 
\texttt{dsname} is used for the RRD DS name and the \texttt{variable:instance}
part is used to tell updater which collector information applies to this DS.
Most of the rest of the line is straight from rrdcreate's description of DS.

It's also possible to invoke remstats functions (see the functions section)  or configuration-supplied 
private functions (see the private functions section)  on the incoming raw data.  The \texttt{data} line 
would look like:

\begin{verbatim}%
	data	xyzzy=\&function(variablename) ...\end{verbatim}

It's your responsibility to make sure that \texttt{function} is available and that it
returns a valid number.

There is a pseudo-function builtin to the updater (see the updater section)  called \texttt{FAKE}.  Its
purpose is to allow testing of alerts, but it could also be used to allow
some foreign system to inject data, by putting its values there.  It allows
you to replace collected values with values of your choice.

It will look for a file in the host's data directory called 
\texttt{FAKE-rrdname-varname} with \texttt{rrdname} and \texttt{varname} munged in the usual 
fashion to make an acceptable file-name.  If the file is there, then its 
content will be returned (with leading and trailing whitespace removed) 
as the value.  If the file is not there, then the collected value will be 
returned.

The \texttt{alert} lines are setting the thresholds for alerts, in this
case for the variables \texttt{in} and \texttt{out}.  They must
specify, in order: the variable-name, the relation ($<$, =, $>$, delta$<$ and delta$>$)
and a space-separated list of thresholds.  Since these examples only 
supply one number each, they will only set OK or WARN statuses.  If the 
variables \texttt{in} or \texttt{out} have values less than ($<$) 100, they
are considered to be OK.  Otherwise they're elevated to WARN status.
What will happen when they go into WARN status depends on the 
alerts (see the alerts section)  file.  These alerts will apply to any host 
which uses this rrd, unless the host overrides it.

The last alert specifies that missing data for the variable \texttt{in} will be
considered to be status \texttt{WARN}, for purposes of generating alerts.
The full description of the alerts is kept in the docs for alert-monitor (see the alert-monitor section) 
as it is the program which implements them.

The \texttt{archives} line tells how to keep the data for this rrd, using
the names defined in the archives (see the archives section)  file.

There can be multiple \texttt{graph} lines describing as many graphs from 
the data in this rrd as you want.  The graph-name must be wildcarded if the
rrd is.  A \texttt{graph} line is followed by its definition which must be
indented.  The definition is straight from rrdgraph with the
magic cookie (see the magic cookie section)  substitution.  If you want a 
description , you can add:

\begin{verbatim}%
	desc='whatever you want'\end{verbatim}

or

\begin{verbatim}%
	desc="whatever you want"\end{verbatim}

to the \texttt{graph} line.  This is used to set the alt text on the web-page.

\subsection{The \texttt{currentvalue} Directive}%
\index{The \texttt{currentvalue} Directive}

It looks like:

\begin{verbatim}%
	currentvalue  STATUSFILENAME VARIABLENAME\end{verbatim}

or

\begin{verbatim}%
	currentvalue  STATUSFILENAME \&FUNCTION(VARIABLENAME)\end{verbatim}

This permits the RRD definer to have the current value of an RRD variable
stored externally in a status file in the host's data directory.  As an
example, the supplied \texttt{apcupsreplacebattery} RRD definition arranges to
store the current value of the indicator in a status file.  A host using
this RRD can then use the \texttt{headerfile} directive to arrange for this
indicator to be displayed in the "host" (in this case a UPS) headers.

\subsection{Collector-specific Stuff}%
\index{Collector-specific Stuff}

Any \texttt{data} line can have extra stuff added to the end of the line.  What
this extra stuff means depends on the RRD.

Any data collected by the unix-status-collector (see the unix-status-collector section)  will have a \texttt{section-name}
added to the end, to be passed on to the unix-status-server (see the unix-status-server section)  to tell it
which sections need to be run for this host.  Some sections will also
specify extra information.  E.G. the \texttt{procname} section will need the name
of the process to be counted.  [See the unix-status-server (see the unix-status-server section)  for complete
specifications.]

An rrd collected by the log-collector (see the log-collector section)  will have extra stuff
on each \texttt{data} line after the DS information.  The extra stuff will be
the function and pattern needed by log-collector to pass to the
log-server (see the log-server section)  to get that variable's data.

An rrd collected by the port-collector (see the port-collector section)  may specify that this particular
service is critical, by simply including the word "critical" at the end of
line.  This will cause the status to be elevated to CRITICAL status  if 
the status ever reaches ERROR level.

An RRD collected by the snmp-collector (see the snmp-collector section)  needs to specify which OIDs to fetch.
They are specified by name in the RRD with a line like:

\begin{verbatim}%
	oid APCUpsAdvInputLineVoltage\end{verbatim}

which refers to a name defined earlier in the 
oids config-file (see the oids config-file section) .

An RRD collected by the \texttt{snmp-collector} may also specify an SNMP port to use
with a line like:

\begin{verbatim}%
	port	3401\end{verbatim}

For an RRD collected by the \texttt{dbi-collector}, the \texttt{data} directive will
require the column number in the select that will provide its data.  This
will be appended to the \texttt{data} directive like:

\begin{verbatim}%
        data xxx=yyy GAUGE:600:0:U COL=12\end{verbatim}

There are also pseudo-columns called \texttt{STATUS} and \texttt{RESPONSE} to enable the
rrd definition to reference the connection-status and response-time.
%------------------------------------ rrds.pod ---

\section{List of RRDs}%
\index{List of RRDs}

\subsection{Collected by the cisco-access-collector (see the cisco-access-collector section) }%
\index{Collected by the cisco-access-collector (see the cisco-access-collector section) }

\begin{itemize}


\item \textbf{ciscolinespeeds} - distribution of line speeds used on a Cisco 
Access Server.  It worked on a 5300 and ought to work on the 52xx series.

\item \textbf{collector-cisco-access} - status of the cisco-access-collector itself

\end{itemize}
\subsection{Collected by the log-collector (see the log-collector section) }%
\index{Collected by the log-collector (see the log-collector section) }

\begin{itemize}


\item \textbf{collector-log} - status of the log-collector itself

\item \textbf{dnscachelog} - counts of access to dnscache (from djbdns)

\item \textbf{ftpxferlog} - file and byte counts from xferlog produced by wu-ftpd

\item \textbf{httpdlog} - from a standard httpd log-file, counts accesse, successes and bytes

\item \textbf{namedstats} - stats from a BIND8 server, triggered by signalling it 
every 5 minutes.  Only possible on a lightly used server.

\item \textbf{newshistory} - counts articles by hierarchy, from an INN server's history file.
You'll have to change this to include hierarchies which are of interest to you.

\item \textbf{newslog} - from the newslog of INN, counts articles by news-neighbor.  Again,
you'll have to change this to include your neighbours.

\item \textbf{qmaillog} - from qmail-send's log, counts messages in, deliveries (ok, 
failed or deferred), and concurrency of local and remote deliveries.

\item \textbf{qmail-poplog} - from a tcpserver log of running qmail-pop3d, counts sessions
and concurrency.

\item \textbf{qmail-smtplog} - from a tcpserver log of running qmail-smtpd, counts sessions
and concurrency.

\item \textbf{sendmaillog} - from a sendmail log, counts messages, bytes and deliveries.
Not tested recently; let me know if it still works, or not.

\end{itemize}
\subsection{Collected by the nt-status-collector (see the nt-status-collector section) }%
\index{Collected by the nt-status-collector (see the nt-status-collector section) }

\begin{itemize}


\item \textbf{collector-nt-status} - shows the status of the nt-status-collector itself

\item \textbf{ntactivity} - from performance counters, gives interrupts, context-switches,
system-calls, sessions and open-files counts.

\item \textbf{ntcpu} - from performance counters, gives cpu usage.  Unfortunately,
I have yet to figure out what these numbers actually mean.  Explanations
and/or calculations are welcome.

\item \textbf{ntlogicaldisk-*} - from performance counters, gives reads, writes, 
bytes in and out and disk free.

\item \textbf{ntmemory} - from performance counters, gives memory free and committed.

\item \textbf{ntnetwork} - from performance counters, gives network bytes and packets
in and out, for the host as a whole.  Not very usefull for a host with 
more than one interface.

\item \textbf{ntnetworkinterface-*} - from performance counters, gives, for input 
and output, bytes, unicast packets, non-unicast packets, errors and discards.

\item \textbf{ntpaging} - from performance counters, gives page reads and writes, 
count of pages in and out and page fount counts.

\item \textbf{ntservice-*} - from SRVINFO (in the NT Resource Kit), gives status
of the named NT service.

\item \textbf{nttime} - from time, gives time difference between NT host and remstats
collector host.

\item \textbf{ntuptime} - from SRVINFO (in the NT Resource Kit), gives uptime

\end{itemize}
\subsection{Collected by the ping-collector (see the ping-collector section) }%
\index{Collected by the ping-collector (see the ping-collector section) }

\begin{itemize}


\item \textbf{collector-ping} - the status of the ping-collector itself

\item \textbf{ping} - gives ping-echo's received and min, max and average of response
time for them.

\item \textbf{ping-*} - as ping, but allows you to ping a specified interface on this
host.

\end{itemize}
\subsection{Collected by the port-collector (see the port-collector section) }%
\index{Collected by the port-collector (see the port-collector section) }

\begin{itemize}


\item \textbf{collector-port} - gives the status of the port-collector itself.

\item \textbf{airqualityontario-*} - collects air quality info from the Ontario Ministry of
the environment web-site, giving a number which indicates how bad the air is.

\item \textbf{environmentcanada-*} - collects weather info from the Environment Canada
web-site, by airport code, and gives status and response-time, temperature, pressure, 
windspeed, windchill, dewpoint, humidity and visibility.  An example of a wildcard port
collector.

\item \textbf{port-*} - gives status and response-time for the specified port

\item \textbf{weathernetwork} -  collects weather for Ottawa from The Weather Network's web-site
and gives status and response-time, temperature, humidity, humidex, pressure, windspeed, 
and dewpoint.

\end{itemize}
\subsection{Collected by the snmp-collector (see the snmp-collector section) }%
\index{Collected by the snmp-collector (see the snmp-collector section) }

\begin{itemize}


\item \textbf{apcenv} - collects info from an APCC UPS's Environmental Monitoring unit, giving
temperature and humidity.

\item \textbf{apcups} - collects ups-related info from an APCC UPS's SNMP unit 
giving time on battery, battery capacity, battery temperature, battery
run-time remaining, load, output current, voltage and frequency and input
voltage, current, frequency and phase.

\item \textbf{bgppeer-*} - collects info about BGP4 peers via SNMP giving current BGP
state, counts of updates and sessions, peer start time and last update time.

\item \textbf{ciscotemperatures} - collects info from high-end Cisco routers giving the
inlet and outlet temperatures for the chassis.

\item \textbf{collector-snmp} - shows the status of the snmp-collector (see the snmp-collector section)  itself.

\item \textbf{dsinuse} - collects the counts of lines in use from a Cisco 5300 Access
Server via SNMP (ought to work on a 5200 too).

\item \textbf{frif-*} -  collects info on Frame-Relay interfaces, similar to snmpif-*,
giving input and output bytes and frames, speed and operational status.

\item \textbf{netappcpu} - collects CPU usage from a Network Appliance Filer.

\item \textbf{pixmem} - collects memory usage info from a Cisco Pix Firewall.

\item \textbf{pixsessions} - collects session counts from a Cisco Pix Firewall.

\item \textbf{snmpcpu} - collects CPU usage from Cisco Routers.

\item \textbf{snmpif-*} - collects interface data giving input and output bytes, 
unicast and non-unicast packets, errors and interface status.

\item \textbf{snmpif2-*} - collects interface data giving input and output bytes, 
unicast and non-unicast packets, errors and interface status.  This
rrd is intended to replace snmpif-* when it's working correctly, as it
doesn't require special-casing in the snmp-collector.

\item \textbf{snmpmem} - collects memory usage from Cisco routers

\item \textbf{snmpuptime} - collects uptime

\item \textbf{squid} - collects usage info from te squid web-cache giving memory,
CPU and disk usage, number of objects in the cache, cache hits, bytes
in and out and current number of clients.

\end{itemize}
\subsection{Collected by the snmp-route-collector (see the snmp-route-collector section) }%
\index{Collected by the snmp-route-collector (see the snmp-route-collector section) }

\begin{itemize}


\item \textbf{collector-snmp-route} -  shows the status of the snmp-route-collector (see the snmp-route-collector section) 
itself.  Note that this collector is too slow to be of interest except
for routers with a small number of routes, not Internet gateway routers.

\item \textbf{snmp-routes-*} -  collects counts of routes and best routes from a given
peer.

\end{itemize}
\subsection{Collected by the unix-status-collector (see the unix-status-collector section) }%
\index{Collected by the unix-status-collector (see the unix-status-collector section) }

\begin{itemize}


\item \textbf{afpdprocs} - (obsolete) collects count of AppleTalk File 
Protocol Daemons running (from netatalk).

\item \textbf{atalkdprocs} - (obsolete) collects count of AppleTalk Daemons running
(from netatalk).

\item \textbf{collector-unix-status} - shows the status of the
unix-status-collector (see the unix-status-collector section)  itself.

\item \textbf{cpu} - collects user and system cpu usage.

\item \textbf{df-*} - collects disk (file-system actually) size and free 
space in bytes and (if applicable) inode totals and free counts.

\item \textbf{ftpcount} - collects count of wu-ftpd daemons running, by access class.
Note that you'll have to modify this one to specify the access classes of 
interest.

\item \textbf{ftpusers} - collects counts of wu-ftpd daemons running.

\item \textbf{if-*} - collects info from "netstat -i" interface packets and errors, 
both input and output, plus collisions and interface status.

\item \textbf{inndsize} - collects the size of the INN News Daemon

\item \textbf{load} - collects the load average for 1, 5 and 15 minutes.

\item \textbf{memory} - collects the memory usage from vmstat, including scanrate
on Solaris.

\item \textbf{namedxfers} - collects the count of BIND's zone transfers running.

\item \textbf{newsreaders} - collects the count of news readers, actually the
count of nnrpd processes.

\item \textbf{nmbdprocs} - (obsolete, replaced by  procname-*) collects counts
of nmbd processes running (part of the samba package).

\item \textbf{proc-diskio} - collects disk I/O info from /proc/stat on linux giving
read and write operations and block counts.

\item \textbf{procname-*} - collects a count of the number of the * named process.

\item \textbf{qmail-qstat} - collects info about the qmail queue giving size and
backlog.

\item \textbf{qmail-qstat2} - collects info about the qmail queue giving size and
backlog.  This version operates without calling \texttt{qmail-stat}, to avoid a
bug in it when dealing with large queues.

\item \textbf{qmail-qtypes} - (obsolete, use qmail-qstat)

\item \textbf{sarlin-paging} - SAR stats for Linux paging, showing both activity
and occupancy type.

\item \textbf{smbdprocs} - (obsolete, use procname-*) collects a count of the 
number of smbd processes running.

\item \textbf{tcpstates} - collects counts of tcp sessions in their various
states from netstat giving established, syn-received, listen, close-wait,
last-ack, fin-wait-1, fin-wait-2, closing, time-wait, and syn-sent.

\item \textbf{unixtime} - collects the difference between the collector's clock and
the collected system's clock.

\item \textbf{uptime} - collects uptime (number of seconds).

\end{itemize}%------------------------------------ configfile-scripts.pod ---

\section{Configuration - Scripts}%
\index{Configuration - Scripts}

The \texttt{script XXX} files are describing how to query a given port for 
its status and are used by the port-collector (see the port-collector section) .  They look like:

\begin{verbatim}%
	send	GET / HTTP/1$\backslash$.0$\backslash$n$\backslash$n
	timeout	5
	port	80
	infopattern \^Server:$\backslash$s+(.*)\$
	valuepattern \^Content-length:$\backslash$s*($\backslash$d+)
	ok	\^HTTP/$\backslash$d$\backslash$.$\backslash$d 200
	warn	\^HTTP/$\backslash$d$\backslash$.$\backslash$d [45]$\backslash$d$\backslash$d\end{verbatim}

This example is taken from the supplied \texttt{config-base} and queries an 
HTTP server for its root page.  First, it sends the "send" text, which 
in this case is a minimal HTTP request, and waits no more than 5 seconds.  
After the port is closed from the remote end, or the timeout expires, 
any text which was returned is examined by the various tests.  In this 
case, if the web-server sends back a line beginning something like 
"HTTP/1.1 200", the port will be marked as "OK".  Similarly, there are 
"warn", "error" and "critical" statuses possible.

The \texttt{port} is optional and \texttt{getservbyname} will be called on the 
script name, if port isn't specified.  This also lets you have multiple
scripts for the same port, using different names for the script.

The \texttt{infopattern} is optional, and supplies a pattern which will be matched
against each line in the result.  If there is a match, files will be created
in the data directory for that host called \texttt{INFOn-rrdname}, where \texttt{n} will
be in the range 1..9 and \texttt{rrdname} will be the name of this rrd, converted to
a file-name.  The files will contain matches for parenthesised items in the
regular expression.  E.G. in the example above, a file will be created called
\texttt{INFO1-http} which will contain whatever the web-server said its 
type and version was.

Similarly, the \texttt{valuepattern} is also optional, but the matches will be returned
as collected items called \texttt{value1} through \texttt{value9}.  In the example, this
would cause the collector to return a line like:

\begin{verbatim}%
	hostname timestamp value1 1022\end{verbatim}

An RRD definition could use this by including a line like:

\begin{verbatim}%
	data pagesize=value1 GAUGE:600:0:10000\end{verbatim}

For a working example, look at the RRD definition for weathernetwork.
%------------------------------------ configfile-times.pod ---

\section{Configuration - Times}%
\index{Configuration - Times}

The \texttt{times} file specifies time intervals for which graphs will be made.
I suppose it should be renamed graphtimes or something, but I've got other
things to do.  Each line is in three pieces: a time name, a start time and 
and end time.  The times are relative to the current time and so will always
be non-positive.

The currently defined times are:

\begin{verbatim}%
	thumb           -60*60*2 0
	day             -60*60*24 0
	week            -60*60*24*7 0
	month           -60*60*24*30 0
	3month          -60*60*24*30*3 0
	year            -60*60*24*365 0
	yesterday       -60*60*24*2 -60*60*24
	lastweek        -60*60*24*7*2 -60*60*24*7\end{verbatim}

\section{Note: }%
\index{Note: }

The times \texttt{thumb} and \texttt{day} are special.  The graph-writer (see the graph-writer section)  expects
them to exist and to have certain meanings.  The \texttt{thumb} time is a short
interval which is used to make the ping thumbnail graphs for the ping index.
The \texttt{day} time is the default time interval.  The higher-level pages will
use the \texttt{day} graph as a link to the other time intervals.
%------------------------------------ configfile-tools.pod ---

\section{Configuration - Tools}%
\index{Configuration - Tools}

The \texttt{tools} file is only used by graph-writer (see the graph-writer section)  to create toolbars.  Each 
line is in two pieces: a tool-name and a URL to link to for this tool.  

The URL can have magic cookies (see the magic cookies section)  in it to substitute
in things like hostname.  Currently, the only cookies which will get
substituted here are \texttt{HOST}, \texttt{IP} and \texttt{HTMLURL}.  If you think of 
other usefull ones, please tell me (see the tell me section) .
%------------------------------------ configfile-views.pod ---

\section{Views - your own selection of graphs on one page}%
\index{Views - your own selection of graphs on one page}

The views config-dir contains files, one per view, describing a collection
of things that you want to see on one page.  There are three kinds:

\begin{itemize}


\item \textbf{simple} - you specify which graphs or customgraphs you want, using 
lines like:
\begin{verbatim}%
	graph        hostname rrdname graphname
	customgraph  customgraphname\end{verbatim}

You can have as many lines as you want, and can mix graphs and customgraphs.
The order you list them in is the order they will appear in the resultant
page.


\item \textbf{template} - you specify a view template to use to generate the page.  
See the docs on view template (see the view template section)  for explanations.

\item \textbf{datapage} - you specify a datapage to use to generate the page.
See the docs on datapage.cgi (see the datapage.cgi section)  for explanations.

\end{itemize}
You can also specify a \texttt{description} line, like:

\begin{verbatim}%
	desc	This is what I'm taking about\end{verbatim}

Where the pages are generated is dependant on the \texttt{viewdir} and \texttt{viewurl}
directives in the html config-file (see the html config-file section) .  The view pages may
have the usual indices on them, if the html config-file (see the html config-file section) 
includes:

\begin{verbatim}%
	viewindices	yes\end{verbatim}

but by default leave them off.
%------------------------------------ configfile-view-templates.pod ---

\section{View Templates}%
\index{View Templates}

View templates give you complete control over page layout in a view.  They are
complete HTML pages with embedded magic cookies (see the magic cookies section) , which are 
substituted for during view generation (by view-writer (see the view-writer section) ).  The resulting 
page will be an \texttt{rrdcgi} CGI script.  The magic cookies are:

\begin{itemize}


\item $<$VIEW::GRAPH hostname rrdname graphname [graphtime]$>$ -
This inserts a graph definition for \texttt{rrdcgi}. The graphtime is from the 
times config-file (see the times config-file section) , and is optional.

\item $<$VIEW::CUSTOMGRAPH customgraphname [graphtime]$>$ - This inserts a customgraph 
definition for \texttt{rrdcgi}. The graphtime is from the 
times config-file (see the times config-file section) , and is optional.

\item $<$VIEW::INCLUDE filename$>$ - This inserts the contents of the named file when
the view-page is displayed. (Using the \texttt{rrdcgi} cookie $<$RRD::INCLUDE filename$>$.)

\item $<$VIEW::HEADER title here$>$ - Inserts a standard remstats header.

\item $<$VIEW::FOOTER$>$ - Inserts a standard remstats footer.

\item $<$VIEW::STATUS host status-file$>$ - inserts the contents of the named status-file when
the view-page is displayed. (Using the \texttt{rrdcgi} cookie $<$RRD::INCLUDE filename$>$.)

\end{itemize}
You can also include \texttt{rrdcgi} magic cookies.
%------------------------------------ config-tools.pod ---

\chapter{Configuration Tools}
\index{Configuration Tools}

These tools are intended to help you build the hosts part of your 
configuration file.  They take a file (or files) of hostnames and
emit host config-files for them.  There are currently no config generators
for the log-collector, the remoteping-collector or the unix-status-collector.

\begin{itemize}


\item split-config (see the split-config section)  - converts old config-files to new config-dirs (see the config-dirs section) 

\item macinfo (see the macinfo section)  - show what is connected to which ports of an ethernet switch

\item new-config (see the new-config section)  - makes a new config-dir populated by symlinks to config-base

\item new-ping-hosts (see the new-ping-hosts section)  - adds a hosts with a ping rrd

\item new-port-hosts (see the new-port-hosts section)  - adds hosts which are collected by the port-collector (see the port-collector section) 

\item new-snmp-hosts (see the new-snmp-hosts section)  - adds hosts which are collected by the snmp-collector (see the snmp-collector section) 

\item new-unix-hosts (see the new-unix-hosts section)  - adds hosts which are running the unix-status-server (see the unix-status-server section) 

\item nt-discover (see the nt-discover section)  - finds and adds Windows NT hosts

\item snmp-get (see the snmp-get section)  - for testing if you can get a particular OID

\item snmp-showif (see the snmp-showif section)  - shows interfaces from SNMP

\item snmpif-description-updater (see the snmpif-description-updater section)  - like it says

\item update-switch-ports (see the update-switch-ports section)  - like it says

\end{itemize}%------------------------------------ macinfo.pod ---

\section{macinfo }%
\index{macinfo }

\subsection{Usage: }%
\index{Usage: }

\begin{verbatim}%
../macinfo version 1.10 from remstats 1.0.13b
usage: ../macinfo [options] [community@]host
where options are:
  -d ddd  enable debugging output at level 'ddd'
  -h      show this help
  -i      show IP numbers, where available
  -m      show MAC addresses
  -n      show domain-names
  -N      show short domain-names (before the first ".")
  -u uuu  show unknown name/ip/mac as 'uuu' [?]

Note: if community isn't supplied on the command-line, then it must be
supplied in the environment in COMMUNITY.
\end{verbatim}

\subsection{Description: }%
\index{Description: }

This utility shows what is connected to which ports of an ethernet switch.
It uses SNMP to query the bridge MIB to get MAC addresses and map them to
ports.  It then gets mappings from port number to ifIndex and thence to 
interface names (either ifName or ifDescr).  Then it looks in the data
from arpwatch, if available, and in /proc/net/dev to get MAC to IP\# mappings.
Finally, it shows what is on which port.

It's used by macinfo.cgi (see the macinfo.cgi section)  to make this information available
on a web-page, and update-switch-ports (see the update-switch-ports section)  to update the descriptions of switch
ports with the name of the host on that port.
%------------------------------------ new-config.pod ---

\section{new-config - make a new config-dir}%
\index{new-config - make a new config-dir}

\subsection{Usage: }%
\index{Usage: }

\begin{verbatim}%
new-config version 1.17 from remstats 1.0.13b
usage: new-config [options] new-config-dir
where options are:
    -d ddd  set debug level to 'ddd'
    -f fff  use 'fff' as config-base [/home/remstats/etc/config-base]
    -h      show this text
\end{verbatim}

\subsection{Description: }%
\index{Description: }

\texttt{new-config} makes a new config-dir and populates it.  Things which you're
less likely to want to change are symlink'ed into the new config-dir.  Things
which you are expected to change, are copied.  Look at your new config-dir
to see which things are symlinks and which aren't.

It's also a good idea to check everything in the new config-dir for the
string \textbf{FIXME}, which is used to indicate something which can't be 
figured out automatically by configure, but still needs to be changed.  You
could use a shell command like the following to get a list of files:

\begin{verbatim}%
	\% find /new/config/dir -type f -exec grep -l FIXME \{\} /dev/null $\backslash$;\end{verbatim}
%------------------------------------ new-ping-hosts.pod ---

\section{new-ping-hosts - add ping RRD to host definition}%
\index{new-ping-hosts - add ping RRD to host definition}

\subsection{Usage: }%
\index{Usage: }

\begin{verbatim}%
new-ping-hosts version 1.11 from remstats 1.0.13b
usage: new-ping-hosts [options] group [hostfile ...]
where options are:
    -d nnn  enable debugging output at level 'nnn'
    -f fff  use 'fff' as config-dir [/home/remstats/etc/config]
    -h      show this help
\end{verbatim}

\subsection{Description: }%
\index{Description: }

You supply a list of files containing hostnames, one per line.  
If there is no hostfile supplied, then it will read from stdin.
If there is no host config-file for that host, \texttt{make-ping-hosts} 
will write entries like:

\begin{verbatim}%
	\# hosts/www.example.com
	desc	gggg host
	group	gggg
	ip	123.456.789.123
	tools	ping traceroute
	rrd	ping\end{verbatim}

for that host.  If that host already exists, it will simply add:

\begin{verbatim}%
	rrd   ping\end{verbatim}

to the end of the hosts file.  It doesn't check if the host already
has a ping rrd.
%------------------------------------ new-port-hosts.pod ---

\section{new-port-hosts - add RRDs for services}%
\index{new-port-hosts - add RRDs for services}

\subsection{Usage: }%
\index{Usage: }

\begin{verbatim}%
new-port-hosts version 1.12 from remstats 1.0.13b
usage: new-port-hosts [options] group [hostfile ...]
where options are:
    -d      enable debugging output
    -f fff  use 'fff' for config-dir [/home/remstats/etc/config]
    -h      show this help
\end{verbatim}

\subsection{Description: }%
\index{Description: }

You supply a file or files of hostnames, one per line, or let
it read from stdin.

You can use this to add RRDs to a host describing
the various services running on a server, or at least the ones that the 
port-collector (see the port-collector section)  knows how to talk to.  It's actually a very limited
port-scanner, and will attempt to connect to each of the 
services.  For each one that answers, \texttt{new-port-hosts}
will write an entry for the corresponding rrd.

If the host has no file, \texttt{new-port-hosts} will add one with appropriate
header info and a ping rrd.  Otherwise, it will just add the port-based
rrd's to the end of the host file.
%------------------------------------ new-snmp-hosts.pod ---

\section{new-snmp-hosts - add RRDs collected by snmp-collector}%
\index{new-snmp-hosts - add RRDs collected by snmp-collector}

\subsection{Usage: }%
\index{Usage: }

\begin{verbatim}%
new-snmp-hosts version 1.14 from remstats 1.0.13b
usage: new-snmp-hosts [options] group community-string [hostfile ...]
where options are:
    -d      enable debugging output
    -f fff  use 'fff' for config-dir [/home/remstats/etc/config]
    -h     	show this help
\end{verbatim}

\subsection{Description: }%
\index{Description: }

if you don't supply a file of hostnames, then it will read from stdin.

You can use \texttt{new-snmp-hosts} to add RRDs collected by the snmp-collector.
It works by looking for a single OID for each RRD.  If the OID exists
on a given host (i.e. it returns data), then the RRD which uses that 
data is added to the host.  There is currently no way to configure it
except for modifying the code.  Complain if you'd actually use such a thing.

It's up to you to discard any you don't want.

If there is no hosts file, it will create one with default
header info.  If there is one, it will just append the rrds.
%------------------------------------ new-unix-hosts.pod ---

\section{new-unix-hosts - add rrds collected by the unix-status-collector}%
\index{new-unix-hosts - add rrds collected by the unix-status-collector}

\subsection{Usage: }%
\index{Usage: }

\begin{verbatim}%
new-unix-hosts version 1.11 from remstats 1.0.13b
usage: new-unix-hosts [options] group [hostfile ...]
where options are:
    -d      enable debugging output
    -f fff  use 'fff' for config-dir [/home/remstats/etc/config]
    -h     	show this help
    -p ppp  use port 'ppp' [1957]
    -t ttt  use 'ttt' for timeout [10]
\end{verbatim}

\subsection{Description: }%
\index{Description: }

If you don't supply a file of hostnames, then it will read from stdin.

This will add all the remstats distributed rrds collected by the
unix-status-collector (see the unix-status-collector section) , except for those using the \texttt{PS} section
of the collector.

It's up to you to discard any you don't want.

If there is no existing host file for a given host, it will create one 
with default header info.  If there is one, it will just append the
new rrds.
%------------------------------------ nt-discover.pod ---

\section{nt-discover - find and add new NT hosts}%
\index{nt-discover - find and add new NT hosts}

\subsection{Usage: }%
\index{Usage: }

\begin{verbatim}%
nt-discover version 1.8 from remstats 1.0.13b
usage: ../nt-discover [options]
where options are:
    -d nnn  enable debugging output at level 'nnn'
    -f fff  use 'fff' for the config-dir [/home/remstats/etc/config]
    -h      show this help
    -s      update status files even if host data-dir found
    -t ttt  use 'ttt' as timeout (in seconds) [10]
\end{verbatim}

\subsection{Description: }%
\index{Description: }

Note: \texttt{Nt-discover} is not called \texttt{new-nt-hosts} because it is a different kind of
program.  Instead of you providing a list of hosts to add, it finds them itself.

Using information supplied in the discovery config-file (see the discovery config-file section) , \texttt{nt-discover}
will contact a host running the nt-status-server (see the nt-status-server section) , and run three separate queries:

\begin{itemize}


\item \texttt{NET-VIEW} will give a list of hosts to check

\item \texttt{USRSTAT} will give a list of NT users (currently unused, but may be interesting)

\item \texttt{SRVINFO} (for each of the hosts found in the first step) will give some more details
on each host and write new host config-files for each new one.  It will not update an existing
host config-file.

\end{itemize}
It will create files under /home/remstats/data/NT to store the discovered hosts and users.
%------------------------------------ pattern-query.pod ---

\section{pattern-query }%
\index{pattern-query }

\subsection{Usage: }%
\index{Usage: }

\begin{verbatim}%
pattern-query  version 1.2 from remstats 1.0.13b
usage: ../pattern-query [options] pattern-file [file-to-search]
where options are:
  -h      show this help
  -m mmm  check 'mmm' matches at most [10]
\end{verbatim}

\subsection{Description: }%
\index{Description: }

This is for testing the \texttt{infopattern} and \texttt{valuepattern} regex's used in
the port-collector (see the port-collector section)  scripts.  Usually it's used in conjunction with the
port-query (see the port-query section)  program.  The \texttt{port-query} program fetches data and the 
\texttt{pattern-query} program tests that data against various patterns.
%------------------------------------ port-query.pod ---

\section{port-query }%
\index{port-query }

\subsection{Usage: }%
\index{Usage: }

\begin{verbatim}%
port-query version 1.6 from remstats 1.0.13b
usage: ../port-query [options] host port [arg] ...
where the optional args are to be substituted for ##1##, ##2##, ...
where options are:
    -d nnn  enable debugging output at level 'nnn'
    -f fff  read query from file 'fff' [stdin]
    -h      show this help
    -s      do magic-cookie substitutions on query
    -t ttt  use 'ttt' for timeout for response [10]
\end{verbatim}

\subsection{Description: }%
\index{Description: }

Sort of like \texttt{telnet host port}, the \texttt{port-query} program connects to the
named host on the specified port, sends the contents of the query file and
returns whatever the service on the other end says.  Can be used with the
pattern-query (see the pattern-query section)  program to test scripts used by the port-collector (see the port-collector section) .
%------------------------------------ remstats-rrdtune.pod ---

\section{remstats-rrdtune }%
\index{remstats-rrdtune }

\subsection{Usage: }%
\index{Usage: }

\begin{verbatim}%
remstats-rrdtune version 1.4 from remstats 1.0.13b
usage: ../remstats-rrdtune [options]
where options are:
    -d nnn  enable debugging output at level 'nnn'
    -h      show this help
	-G GGG  only apply to hosts in groups 'GGG' (a comma-separated list)
	-H HHH  only apply to hosts 'HHH' (a comma-separated list)
	-K KKK  only apply to hosts with a key in 'KKK' (a comma-separated list)
Note:  Only one of -G and -H can be used simultaneously.  If -K us used with
-H or -G, the keys will select a subset of the hosts selected by -H or -G.
\end{verbatim}

\subsection{Description: }%
\index{Description: }

The purpose of \texttt{remstats-rrdtune} is to synchronize the remstats RRD 
definitions from the rrds config-dir (see the rrds config-dir section)  with the actual 
RRD files.  It can only apply changes which \texttt{rrdtool tune} would let
you make, so changing RRAs cannot be done, e.g.  You can change the ds-type,
heartbeat, minimum and maximum.
%------------------------------------ snmp-get.pod ---

\section{snmp-get }%
\index{snmp-get }

\subsection{Usage: }%
\index{Usage: }

\begin{verbatim}%
snmp-get version 1.11 from remstats 1.0.13b
usage: snmp-get [options] {-c ccc | -r rrr} host oid
where options are:
    -d ddd  set debugging output to level 'ddd'
    -h      show this help
    -f fff  use 'fff' for config-dir [/home/remstats/etc/config]
    -p ppp  use port 'ppp' instead of the usual port

required args are only one of:
    -c ccc  use 'ccc' as SNMP community-string
    -r rrr  get SNMP community-string from RRD 'rrr'

- 'host' is the host to query
- 'oid' is the Object ID to return the value of, either fully numeric or
  as defined in the 'oids' config-file
\end{verbatim}

\subsection{Description: }%
\index{Description: }

Gets a value via SNMP.  In case you don't have another snmp package.
%------------------------------------ snmp-showif.pod ---

\section{snmp-showif - display interfaces from SNMP}%
\index{snmp-showif - display interfaces from SNMP}

\subsection{Usage: }%
\index{Usage: }

\begin{verbatim}%
snmp-showif version 1.8 from remstats 1.0.13b
usage: ../snmp-showif [options] host community
where options are:
	-d	enable debugging output
	-h	show this help
\end{verbatim}

\subsection{Description: }%
\index{Description: }

This utility can be used to get a list of interfaces on an SNMP queryable 
host.  You can use it to figure out which interfaces you want to add,
and what names remstats uses for them.  

It will show ifIndex, ifDescr, ifSpeed, ifType, ifName, ifInOctets, 
ifOutOctets, ifOperStatus and ifAlias (if it exists).
%------------------------------------ snmpif-description-updater.pod ---

\section{snmpif-description-updater }%
\index{snmpif-description-updater }

\subsection{Usage: }%
\index{Usage: }

\begin{verbatim}%
snmpif-description-updater version 1.11 from remstats 1.0.13b
usage: snmpif-description-updater [options]
where options are:
	-d ddd  set debugging level to 'ddd'
	-f fff  use 'fff' for config-dir [/home/remstats/etc/config]
	-h      show this help message
\end{verbatim}

\subsection{Description: }%
\index{Description: }

This script updates host config-files, for those which contain \texttt{snmpif-*}
RRDs, by fetching the ifAlias OID via SNMP and re-writing the configuration
file if any of the descriptions have changed.  I'd suggest running it every
now and then, say once a day, unless you're making very frequent changes to
the descriptions.

If somebody has gear which uses an OID other than ifAlias to store the 
description, then I'll have to consider making this more general, but
it'll do for now.
%------------------------------------ update-switch-ports.pod ---

\section{update-switch-ports }%
\index{update-switch-ports }

\subsection{Usage: }%
\index{Usage: }

\begin{verbatim}%
update-switch-ports version 1.5 from remstats 1.0.13b
usage: ../update-switch-ports [options]
where options are:
  -d nnn  enable debugging output at level 'nnn'
  -f fff  use 'fff' as config-dir [/home/remstats/etc/config]
  -h      show this help
\end{verbatim}

\subsection{Description: }%
\index{Description: }

This runs macinfo (see the macinfo section)  on each host with a \texttt{switch} tag.  From the reported
names it updates the description for each port rrd on the switch to contain
the name of the host on that port, shown as \texttt{\{hostname\}}.  It will replace
any previous host noted by an earlier run and will append to an existing
description.
%------------------------------------ servers.pod ---

\chapter{Servers}
\index{Servers}

\section{Servers }%
\index{Servers }

There are a four servers currently:

\begin{itemize}


\item unix-status-server (see the unix-status-server section)  is queried by the unix-status-collector (see the unix-status-collector section) 
for various information it obtains by running various 
commonly-available programs: \texttt{df}, \texttt{vmstat}, \texttt{uptime}, \texttt{netstat},
\texttt{uname}, \texttt{ps} ...

\item log-server (see the log-server section)  (queried by the log-collector (see the log-collector section) )
reads the unread portion of the specified log-file and returns the requested
statistics.

\item remoteping-server (see the remoteping-server section)  is contacted by the remoteping-collector (see the remoteping-collector section)  which
supplies a list of hosts.  The server runs multiping (see the multiping section)  against them and
returns results for each, similar to the results obtained from the 
ping-collector (see the ping-collector section) .

\item nt-status-server (see the nt-status-server section)  - provides access to information from Windows NT workstations and servers.

\end{itemize}%------------------------------------ log-server.pod ---

\section{log-server - providing remote access to log information}%
\index{log-server - providing remote access to log information}

\subsection{Usage: }%
\index{Usage: }

\begin{verbatim}%
log-server version 1.15 from remstats 1.0.13b
usage: ../log-server [options] logfile ...
where options are:
	-d nnn	enable debugging output at level 'nnn'
	-p ppp	set the prefix for context-files to 'ppp' [log-server-]
	-h	show this help
\end{verbatim}

The log-server must be supplied with at least one log-file
to serve.

\subsection{Description: }%
\index{Description: }

The log-server is queried by the log-collector (see the log-collector section)  using a
"protocol" described in the log-collector documentation.
It will provide information from any of the log-files on it's 
command-line, but no others.  It is recommended that you use
the tcp\_wrappers or some other form of access-control to 
limit access to this server.  The information may or may not 
be sensitive, according to which log-files you are serving,
but letting anyone query it will mean that you will lose some
data, unless you're sure that they will only query it in test
mode.

The log-server will store context for each log-file
that is served, by default in \texttt{/var/tmp/log-server-XXX},
where \texttt{XXX} is replaced by a munged version of the log-file
name.  If you want this stored somewhere else, use the \texttt{-p}
switch or change the program.

\subsection{Notes: }%
\index{Notes: }

Don't forget to list all the log-files that you want to serve on
the command-line.  If there are too many for your inetd, make a
tiny shell script with the \texttt{log-server} invocation and run that
from inetd.

For details on installation, you'd better look at the 
server installation docs (see the server installation docs section) .
%------------------------------------ nt-status-server.pod ---

\section{nt-status-server - allow remote gathering of Windows NT data}%
\index{nt-status-server - allow remote gathering of Windows NT data}

\subsection{Usage: }%
\index{Usage: }

\begin{verbatim}%
	usage: nt-status-server [options]
	where options are:
	    -a      show all available performance counters
	    -d ddd  enable debugging at level 'ddd' [\$main::debug]
	    -h      show this help message
	    -i [ppp sss] install this as an NT service, using 'ppp' as
		    as perl and 'sss' as this script.  Defaults to
			perl=C:$\backslash$Perl$\backslash$bin$\backslash$perl.exe
			script=wherever-it's-invoked-from
	    -p ppp  run server on port 'ppp' [1957]
	    -s      run stand-alone, i.e. not as a service
		-t ttt  trust hosts 'ttt' (comma-separated) [127.0.0.1]
		-T      show status in title-bar of window
	    -u      un-install this service\end{verbatim}

\subsection{Description: }%
\index{Description: }

The nt-status-server allows the nt-status-collector (see the nt-status-collector section) 
to get data from a remote machine running some flavour of NT and possibly 
Windows 2000.  It runs various programs from the NT Resource Kit, for some
information and examines the NT performance counters for other information.

For details on installation, look at the server installation docs (see the server installation docs section) .

\subsection{Protocol: }%
\index{Protocol: }

The nt-status-collector (see the nt-status-collector section)  connects to the \texttt{nt-status-server} and sends
a series of commands, ending with 'GO'.  Then the server sends back the data
it obtained and closes the connection.
The commands are often the names of programs to run (in UPPERCASE) and
the ones known currently are:

\begin{itemize}


\item SRVINFO - runs SRVINFO and returns the version of NT

\item PERFCOUNTERS - examines the NT performance counters and returns 
information about memory, disk, processes, ...

\item PULIST - runs PULIST (from the NT ResKit) and shows counts for all
the running processes.

\item MSDRPT - runs WINMSDP to show (currently) memory total and free.

\item USRSTAT - runs USRSTAT (from the NT ResKit) and shows when the various
users in the specified NT domain last logged in, and which domain-controller
authorized them.

\item NETVIEW - runs "NET VIEW" to list the computers currently visible.

\item TIME - compares local and remote times

\end{itemize}
If you want to see what it returns, you can simply start it up and telnet to it.

\subsection{Bugs }%
\index{Bugs }

\begin{itemize}


\item It is intended that it will eventually install itself as an NT service, and most
of the code is there, but it doesn't currently work.  Patches gratefully accepted.
For now you have to invoke it with the \texttt{-s} switch to have it run stand-alone or use
SRVANY to provide the NT service stuff, (which also requires the \texttt{-s} flag.

\item Not only is is currently single-threaded (i.e. won't accept more than one connection
at a time), but if a second connection comes in the server won't answer any more requests
and will have to be re-started.  I've added code to nt-status-collector (see the nt-status-collector section)  so that it
won't run if there is another instance running already.  This won't help if you're using
telnet to test the nt-status-server, so be prepared to restart nt-status-server if it
gets wedged because of this.

\end{itemize}%------------------------------------ remoteping-server.pod ---

\section{remoteping-server - allow remote collection of ping data}%
\index{remoteping-server - allow remote collection of ping data}

\subsection{Usage: }%
\index{Usage: }

\begin{verbatim}%
remoteping-server version 1.7 from remstats 1.0.13b
usage: ../remoteping-server [options]
where options are:
	-d nnn	enable debugging output at level 'nnn'
	-h	show this help
\end{verbatim}

\subsection{Description: }%
\index{Description: }

The remoteping-server allows the remoteping-collector (see the remoteping-collector section)  to obtain ping 
data remotely.  Like the ping-collector (see the ping-collector section) , it uses multiping (see the multiping section) 
to get ping data, but it can be queried from a remote site.

I'm looking for volunteers; please look at
this note (see the this note section) .
%------------------------------------ unix-status-server.pod ---

\section{unix-status-server - allow remote gathering of unix data}%
\index{unix-status-server - allow remote gathering of unix data}

\subsection{Usage: }%
\index{Usage: }

\begin{verbatim}%
unix-status-server version 1.42 from remstats 1.0.13b
usage: ../unix-status-server [options]
where options are:
  -d nnn  enable debugging output at level 'nnn'
  -h      show this help
  -r      include remotely-mounted file-systems
  -t tst  do tests 'tst', a comma-separated list of:
          VMSTAT, DF, UPTIME, NETSTAT, UNAME, PS, PROC, MASQCONN,
          FTPCOUNT, NETSTAT-TCPSTATES, FILEAGE, QMAILQSTAT,
          QMAILQREAD, PROCNAME, PROCDISKIO, PROCMEMINFO, SAR
\end{verbatim}

\subsection{Description: }%
\index{Description: }

The unix-status-server allows the unix-status-collector (see the unix-status-collector section) 
to get data from a remote machine running some flavour of unix.  It
runs several different programs on request (uname, vmstat, df, uptime, 
netstat, ps, ftpcount, qmail-qstat, and qmail-qread).  

\subsection{Protocol: }%
\index{Protocol: }

The unix-status-collector (see the unix-status-collector section)  connects to the \texttt{unix-status-server} and sends
a series of commands, ending with 'GO'.  Then the server sends back the data
it obtained by running the requested programs and closes the connection.
The commands are usually the names of programs to run (in UPPERCASE) and
the ones known currently are:

\begin{itemize}


\item UNAME runs uname and returns:
machine, os\_name, os\_release, os\_version

\item VMSTAT runs vmstat and returns variables relating to memory usage
depending on the operating system

\item DF runs df and for each file-system returns:
dfsize:FSNAME, dfused:FSNAME, dfpercent:FSNAME
inodessize:FSNAME, inodesused:FSNAME, inodespercent:FSNAME

\item UPTIME runs uptime and returns:
uptime (a timestamp in seconds), users, load1, load5, load15

\item NETSTAT runs netstat and, for each interface, returns:
interface\_packets\_in:IFNAME, interface\_errors\_in:IFNAME,
interface\_packets\_out:IFNAME, interface\_errors\_out:IFNAME,
interface\_collisions:IFNAME

\item PS runs ps and returns various numbers pulled out of the output
(see below)

\item FTPCOUNT runs ftpcount (from wuftpd) to find out which groups the
ftp-server's users fall into

\item QMAILQSTAT runs qmail-qstat and returns: qmail\_qsize, qmail\_qbacklog

\item QMAILQREAD runs qmail-qread and returns FIXME

\item FILEAGE returns timestamps for the ages of specified files
(see below)

\item TIME return the difference in time-stamps between the host running the unix-status-server
and the querying host.  It must be given the querying host's timestamp following
the \texttt{TIME} directive.  The two variables returned are time and timediff.

\item NETSTAT-TCPSTATES runs netstat and produces counts of the number of tcp
sessions in the various TCP states.

\item SAR runs "\texttt{sar -A}" and produces a lot of operating-system specific 
info.

\end{itemize}
If you want to see what it will return, you can simply invoke the 
\texttt{unix-status-server} as a local script and type commands at it.

\subsection{Programs: }%
\index{Programs: }

\begin{itemize}


\item \texttt{/usr/local/bin/uname} or \texttt{/usr/bin/uname}
It's mostly, cosmetic, for web-page header info, but sometimes it's
really usefull too.  It's required for some of the other sections, in order
to figure out how to adapt to various operating-system sepcific quirks.

\item \texttt{/usr/bin/vmstat} or \texttt{/usr/ucb/vmstat}
for scanrate, interrupts, context-switches and cpu-time, and 
of course free memory and swap

\item \texttt{/usr/local/bin/df} or \texttt{/usr/xpg4/bin/df} or \texttt{/bin/df}
the gnu df.  I need the -P flag (for Posix, but it makes df put
its info on one line), and the -i flag for inode info.

\item \texttt{/usr/local/bin/uptime} or \texttt{/usr/bin/uptime} or 
\texttt{/usr/ucb/uptime}
the gnu uptime has a format I know how to parse.  Others often
invent new ways to be cute about the display that I don't always
recognise.  Please show me examples of formats that the 
unix-status-server doesn't know, and which os (uname -a) they
come from.

\item \texttt{/usr/bin/netstat} or \texttt{/usr/ucb/netstat}
to get network interface info

\item \texttt{/usr/bin/ps} or \texttt{/bin/ps}
for counting the number of running instances of a named process.

\item \texttt{/usr/local/bin/ftpcount} (part of wu-ftpd distribution) shows the number of ftp clients of wu-ftpd
from each access group.

\item \texttt{/var/qmail/bin/qmail-qstat} and \texttt{/var/qmail/bin/qmail-qread}
If you have qmail (see \textbf{http://www.qmail.org/}), these will
let you get information about the queue size, which you can't
find from the logs.  Otherwise, ignore them.

\end{itemize}
\subsection{PS Usage:}%
\index{PS Usage:}

With extended commands, of which \texttt{PS} is the first, you also specify
what you want to look for with extra commands, in addition to the \texttt{PS}
command.  A command looks like:

\begin{verbatim}%
	varname PS func pattern\end{verbatim}

The \texttt{varname} is used to create a variable-name for the returned data.
The name will be \texttt{ps:varname}.  \texttt{Func} is one of \texttt{count}, \texttt{sum},
\texttt{last}, \texttt{average}, \texttt{min}, \texttt{max}.  \texttt{Pattern} is a perl-style 
regular-expression, the simplest form of which is just a string.

For an example, if we wanted to know how many web-servers were running
over time, we might use (very sloppily):

\begin{verbatim}%
	webservers PS count httpd\end{verbatim}

[You probably want a better regular expression.]

\subsection{FILEAGE Usage:}%
\index{FILEAGE Usage:}

For the \texttt{FILEAGE} command, you have to specify an extended command
that looks like:

\begin{verbatim}%
	varname FILEAGE /path/to/file\end{verbatim}

This will produce a timestamp for the last-modification time of \texttt{/path/to/file}.

\subsection{PROCNAME Usage:}%
\index{PROCNAME Usage:}

PROCNAME examines /proc (only under linux) and produces counts of
specified named processes

\subsection{PROCDISKIO Usage:}%
\index{PROCDISKIO Usage:}

PROCDISKIO examines /proc/stat (only under linux) and returns disk
I/O operation and bytes counts.

\subsection{PROCMEMINFO Usage:}%
\index{PROCMEMINFO Usage:}

PROCMEMINFO examines /proc/meminfo (only under linux) and returns
returns the memory stats.  Some-one please tell me 
terskine@users.sourceforge.net (see the terskine@users.sourceforge.net section)  if you can interpret these to tell when
a host is \textbf{really} running out of memory.

\subsection{PROCNETDEV Usage:}%
\index{PROCNETDEV Usage:}

PROCNETDEV examines /proc/net/dev (only under linux) and returns usefull 
network counters

\subsection{Notes: }%
\index{Notes: }

With older versions of \texttt{vmstat}
(ones that mash fields together), it will give up on \texttt{vmstat} and not return
memory and CPU info.  It also requires a version of \texttt{df} that will accept the
\texttt{-P} and \texttt{-i} flags.  The \texttt{-P} flag forces the output 
for a file-system to stay on one line (easier for me to parse) and the
\texttt{-i} returns info about inodes.  If the \texttt{-i} flag is missing,
you won't get any inode data.  You also won't get any inode data if
the file-system doesn't have inodes.  (Duh :-).

For details on installation, look at the 
server installation docs (see the server installation docs section) .
%------------------------------------ collectors.pod ---

\chapter{Collectors}
\index{Collectors}

\section{Collectors Data Format}%
\index{Collectors Data Format}

All the collectors produce data on stdout in the same standard form:

\begin{verbatim}%
	host timestamp variable value\end{verbatim}

If the \texttt{variable} is for something like network interfaces,
where the host can have several of them, the data will look like:

\begin{verbatim}%
	host timestamp variable:instance value\end{verbatim}

Having all the collectors using a standard form permits a single 
updating program, updater (see the updater section) , to process the data from them all, and 
also means that I can write a new collector without needing to change 
the updater.

\chapter{Collectors}
\index{Collectors}

\section{Remstats supplied collectors}%
\index{Remstats supplied collectors}

\begin{itemize}


\item log-collector (see the log-collector section)  - gets info from remote 
log-files

\item ping-collector (see the ping-collector section)  - pings hosts

\item port-collector (see the port-collector section)  - checks on remote services

\item program-collector (see the program-collector section)  - runs arbitrary commands to collect info

\item remoteping-collector (see the remoteping-collector section)  - pings hosts from somewhere else

\item unix-status-collector (see the unix-status-collector section)  - gets info from unix hosts

\item snmp-collector (see the snmp-collector section)  - gets info via SNMP

\item snmp-route-collector (see the snmp-route-collector section)  - counts routes available from BGP peers

\end{itemize}
The usual invocation of a collector (via run-remstats (see the run-remstats section) ) is:

\begin{verbatim}%
	xxx-collector | updater xxx\end{verbatim}

\chapter{Collectors}
\index{Collectors}

\section{How to write your own collector}%
\index{How to write your own collector}

There are a few requirements:

\begin{enumerate}


\item  it must write its results to stdout in 
standard form (see the top of this file.)

\item  it must be placed in the same directory with the rest of
the collectors, and must be called XXX-collector, replacing "XXX"
with whatever it's collecting.

\item  it must take (or at least ignore), the same arguments that 
the other collectors do, specificly, the \texttt{-f}, \texttt{-u} and \texttt{-F} flags, 
and the \texttt{-G} and \texttt{-H} flags, if I ever get around to implementing 
them (see todo (see the todo section) ).

\item  you must add it to the list of collectors (the \texttt{collectors}
line in the general config-file  (see the general config-file  section) ).

\item  you must define rrd(s) specifying "source XXX" to use the data
from this collector.

\item  you must add "rrd YYY" to the appropriate 
host config-files (see the host config-files section) .

\end{enumerate}
There is a supplied \texttt{skeleton-collector.pl} file supplied with the
distribution, which does everything except collect data.  You should
be able to plug your code into its \texttt{collect\_host} routine and have a
collector, if you don't mind writing in perl.
%------------------------------------ cisco-access-server-collector.pod ---

\section{cisco-access-server-collector }%
\index{cisco-access-server-collector }

\subsection{Usage: }%
\index{Usage: }

\begin{verbatim}%
_remstats_ 1053112182 cisco-access-server-collector:requests 0
_remstats_ 1053112182 cisco-access-server-collector:collected 0
_remstats_ 1053112182 cisco-access-server-collector:used 0
_remstats_ 1053112182 cisco-access-server-collector:runtime 0
\end{verbatim}

\subsection{Description: }%
\index{Description: }

Collects, via SNMP, totals from a Cisco Access Server 52xx or 53xx
because the snmp-collector (see the snmp-collector section)  doesn't do roll-ups.
%------------------------------------ dbi-collector.pod ---

\section{dbi-collector - data from remote databases}%
\index{dbi-collector - data from remote databases}

\subsection{Usage: }%
\index{Usage: }

\begin{verbatim}%
dbi-collector version 1.8 from remstats 1.0.13b
usage: ../dbi-collector [options]
where options are:
    -d nnn  enable debugging output at level 'nnn'
    -f fff  use 'fff' for config-dir [/home/remstats/etc/config]
    -F      force collection even if it's not time
   	-G GGG  only try hosts from group 'GGG', a comma-separated list
    -h      show this help
    -H HHH  only try hosts from 'HHH', a comma-separated list
    -K KKK  only try hosts with key(s) 'KKK', a comma-separated list
    -u      ignore uphosts file
\end{verbatim}

\subsection{Description: }%
\index{Description: }

The \texttt{dbi-collector} uses perl's DBI interface to access remote databases to
collect data.  If you have a DBD driver for your database, and can write
an SQL select statement to select the data you're interested in, the 
\texttt{dbi-collector} can import the data into rrdtool and remstats.

The \texttt{dbi-collector} adds three new directives to RRD definitions which use it:

\begin{itemize}


\item \textbf{connect CONNECTNAME} - specifies the connection to use, documented
under the dbi-connects config-dir (see the dbi-connects config-dir section) .

\item \textbf{select SELECTNAME} - specifies the select statement to use, by name
as documented in dbi-selects config-dir (see the dbi-selects config-dir section) .

\item \textbf{multirowid COL\#} - specifies the column number to use to construct
the variable name for variables from this row.  This permits the use of
select statements which return more than one row.

\end{itemize}
The \texttt{connect} directive tells how to connect to the appropriate server 
and database; the \texttt{select} tells how to select the data.

For an RRD collected by the \texttt{dbi-collector}, the \texttt{data} directive will
require the column number in the select that will provide its data.  This
will be appended to the \texttt{data} directive like:

\begin{verbatim}%
	data xxx=yyy GAUGE:600:0:U COL=12\end{verbatim}

There are also pseudo-columns called \texttt{STATUS} and \texttt{RESPONSE} to enable the
rrd definition to reference the connection-status and response-time.

There are also some possible additions to the \texttt{rrd} directive in the host
file.  The rrd directive can look like:

\begin{verbatim}%
	rrd	rrdname CONNECT="cc" SELECT="ss" USER="uu" PASSWORD="pp" DATABASE="dd"\end{verbatim}

All of the extras on the end are optional, and are provided in order to 
be able to use a single connect file with multiple databases, users, ...

If the \texttt{multirowid} is specified, then the variable returned is named
\texttt{RRDNAME:DATANAME:ROWNAME}.  Otherwise, it is named \texttt{RRDNAME:DATANAME}.

\subsection{Notes }%
\index{Notes }

The \texttt{dbi-collector}, when querying an Oracle database, needs to know where
Oracle was installed, specifically, it needs the environment to contain
\texttt{ORACLE\_HOME} pointing at where Oracle was installed.  This can be simply
done using the new environment config-file (see the environment config-file section) .
%------------------------------------ dns-collector.pod ---

\section{dns-collector }%
\index{dns-collector }

\section{Usage: }%
\index{Usage: }

\begin{verbatim}%
dns-collector version 1.2 from remstats 1.0.13b
usage: ../dns-collector [options]
where options are:
    -d nnn  enable debugging output at level 'nnn'
    -f fff  use 'fff' for config-dir [/home/remstats/etc/config]
    -F      force collection even if it's not time
   	-G GGG  only try hosts from group 'GGG', a comma-separated list
    -h      show this help
    -H HHH  only try hosts from 'HHH', a comma-separated list
    -K KKK  only try hosts with key(s) 'KKK', a comma-separated list
    -u      ignore uphosts file
\end{verbatim}

\section{Description: }%
\index{Description: }

The \texttt{dns-collector} does DNS queries to test availability of specified records
and response time of the server providing them.

The queries are made to the host that the DNS rrds are defined on.  The
records are specified in the rrd instance.  E.G.

\begin{verbatim}%
	rrd dnssite-example.net SOA A NS MX\end{verbatim}

This example will query the host that it belongs to for \texttt{SOA}, \texttt{A}, \texttt{NS},
and \texttt{MX} records for \texttt{example.net}.
%------------------------------------ error-collector.pod ---

\section{error-collector }%
\index{error-collector }

\section{Usage: }%
\index{Usage: }

\begin{verbatim}%
error-collector version 1.1 from remstats 1.0.13b
usage: ../error-collector [options] run-pid
where options are:
    -d nnn  enable debugging output at level 'nnn'
    -f fff  use 'fff' for config-dir [/home/remstats/etc/config]
    -F      force collection even if it is not time
   	-G GGG  only try hosts from group 'GGG', a comma-separated list
    -h      show this help
    -H HHH  only try hosts from 'HHH', a comma-separated list
    -K KKK  only try hosts with key(s) 'KKK', a comma-separated list
    -u      ignore uphosts file
\end{verbatim}

\section{Description: }%
\index{Description: }

More remstats self-monitoring.  It counts the errors and aborts produced by
the other collectors.
%------------------------------------ log-collector.pod ---

\section{log-collector - get stats from remote log-files}%
\index{log-collector - get stats from remote log-files}

\subsection{Usage: }%
\index{Usage: }

\begin{verbatim}%
log-collector version 1.21 from remstats 1.0.13b
usage: ../log-collector [options]
where options are:
   -d nnn  enable debugging output at level 'nnn'
   -f fff  use config-dir 'fff'[/home/remstats/etc/config]
   -F      force collection, even if it is not time
   -G GGG  only do hosts from group 'GGG', a comma-separated list
   -h      show this help
   -H HHH  only do hosts from 'HHH', a comma-separated list
   -K KKK  only try hosts with key(s) 'KKK', a comma-separated list
   -p ppp  contact log-server on port 'ppp' [1958]
   -t ttt  timeout each port attempt after 'ttt' seconds [10]
   -u      ignore uphosts file
\end{verbatim}

\subsection{Description: }%
\index{Description: }

The log-collector gets data from remote log-server (see the log-server section) 's.  This way the
whole log-file doesn't have to be transfered.  The "protocol",
if it deserves that name, is very simple.  The collector sends a request,
which looks like (you can type it in via telnet):

\begin{verbatim}%
	LOGFILE /wherever/the/logfile.is
	varname	function pattern
	...
	GO\end{verbatim}

The directives are all in \textbf{UPPERCASE}.  They are \texttt{LOGFILE}, 
\texttt{GO}, \texttt{DEBUG} and \texttt{TEST}.  The \texttt{LOGFILE} directive 
tells the \texttt{log-server} which file to read.  The \texttt{GO} 
directive starts the request.  \texttt{DEBUG}
causes some extra remote debugging output, and \texttt{TEST} makes the
\texttt{log-server} operate in test mode.  In test mode it doesn't update the
last-read position for that log-file, so you won't lose any data when testing.

The other lines are telling the \texttt{log-server} what data to collect.
The first "word" is the variable name to be returned.  The next is the function
to be applied (from \texttt{count}, \texttt{sum}, \texttt{average}, \texttt{min}, \texttt{max}, 
\texttt{first}, \texttt{last}) and \texttt{lastof}.  The rest of the line is a perl-style regex.  

\textbf{Note1:} The \texttt{lastof} function is different from the others, in several
respects.  First, it takes multiple regex's.  Second, it saves the current
value of the variable in a context file with the same name as the context file
for the log-file with '-' and the variable-name appended.  The way it works is
to return the number of the pattern which was last matched (starting with 1).
If none of the patterns have ever matched, it will return zero, until one
of them matches.

\textbf{Note2:} Except for the count function, the regex must contain a 
(parenthesized) expression which will match a number, to which the function 
will be applied.

For example, the line:

\begin{verbatim}%
	rootlogins count ROOT LOGIN\end{verbatim}

would return data for a variable called rootlogins.  The value would be 
the count of the records in the specified logfile which had the string 
'ROOT LOGIN' in them.

The pattern can be much more complicated, for example (from the httpdlog rrd):

\begin{verbatim}%
	bytes sum $\backslash$sHTTP/$\backslash$d$\backslash$.$\backslash$d"$\backslash$s+2$\backslash$d$\backslash$d$\backslash$s+($\backslash$d+)\end{verbatim}

This looks through a standard web-server log-file and extracts the bytes 
transferred and adds them up to produce the total number of bytes 
transferred for successes in that sample period.

\subsection{How to make RRDs that use the log-collector}%
\index{How to make RRDs that use the log-collector}

It's easiest to explain by example.  Look at the beginning of the 
rrd \texttt{httpdlog}, copied here:

\begin{verbatim}%
	source  log
	step    300
	data    requests        GAUGE:600:0:U   count (GET|POST)
	data    success         GAUGE:600:0:U   count $\backslash$sHTTP/$\backslash$d$\backslash$.$\backslash$d"$\backslash$s+2$\backslash$d$\backslash$d
	data    bytes           GAUGE:600:0:U   sum $\backslash$sHTTP/$\backslash$d$\backslash$.$\backslash$d"$\backslash$s+2$\backslash$d$\backslash$d$\backslash$s+($\backslash$d+)\end{verbatim}

To form the requests to be sent to the log-server, the log-collector takes the 
DS name, e.g. \texttt{success}, and the last part of the line after all the 
DS definition \texttt{count $\backslash$sHTTP/$\backslash$d$\backslash$.$\backslash$d"$\backslash$s+2$\backslash$d$\backslash$d}, combines the two and sends:

\begin{verbatim}%
	success count $\backslash$sHTTP/$\backslash$d$\backslash$.$\backslash$d"$\backslash$s+2$\backslash$d$\backslash$d\end{verbatim}

Note that the pattern can include magic cookies (see the magic cookies section)  as of remstats 
version 0.12.2.

The line in the host config-file must specify the log-file name, like:

\begin{verbatim}%
	httpdlog /var/log/httpd/access\_log\end{verbatim}

This permits the same code to get information from the same log format, no
matter where it's stored on various machines.  Since log-file locations
seem to vary a lot by host, I put the log-file specification in the host
config-file, rather than in the rrd config-file.
%------------------------------------ nt-status-collector.pod ---

\section{nt-status-collector - stats from Windows NT hosts}%
\index{nt-status-collector - stats from Windows NT hosts}

\subsection{Usage: }%
\index{Usage: }

\begin{verbatim}%
nt-status-collector version 1.40 from remstats 1.0.13b
usage: ../nt-status-collector [options]
where options are:
    -d nnn  enable debugging output at level 'nnn'
    -f fff  use 'fff' for config-dir [/home/remstats/etc/config]
    -F      force collection, even if it is not time
    -G GGG  only try hosts from group 'GGG', a comma-separated list
    -h      show this help
    -H HHH  only try hosts from 'HHH', a comma-separated list
    -p ppp  connect to server on port 'ppp' [1957]
    -t ttt  set timeout to 'ttt' [25]
    -K KKK  only try hosts with key(s) 'KKK', a comma-separated list
    -u      ignore uphosts file
\end{verbatim}

\subsection{Description: }%
\index{Description: }

The \texttt{nt-status-collector} gets its data from the nt-status-server (see the nt-status-server section) .
It sends a query consisting mostly of the sections of the server
that it wants to run.  It can also request that processes with
specific names be counted and that information returned too.
A query for all the sections might look like:

\begin{verbatim}%
	SRVINFO
	PERFCOUNTERS
	PULIST
	MSDRPT
	USRSTAT ntdomain
	NET-VIEW
	GO\end{verbatim}

Note that \texttt{PULIST}, \texttt{MSDRPT}, \texttt{USRSTAT} and \texttt{NET-VIEW} aren't currently
used by anything, but they may be usefull for something.  Also, USRSTAT wants
an NT domain-name with the query.
%------------------------------------ ntop-collector.pod ---

\section{ntop-collector }%
\index{ntop-collector }

\subsection{Usage: }%
\index{Usage: }

\begin{verbatim}%
ntop-collector version 1.9 from remstats 1.0.13b
usage: ../ntop-collector [options]
where options are:
  -d nnn  enable debugging output at level 'nnn'
  -f fff  use 'fff' for config-dir [/home/remstats/etc/config]
  -F      force collection even if it is not time
  -G GGG  only try hosts from group 'GGG', a comma-separated list
  -h      show this help
  -H HHH  only try hosts from 'HHH', a comma-separated list
  -K KKK  only try hosts with key(s) 'KKK', a comma-separated list
  -u      ignore uphosts file
\end{verbatim}

\subsection{Description: }%
\index{Description: }

To be written.

FIXME
%------------------------------------ ping-collector.pod ---

\section{ping-collector - get reachability of hosts}%
\index{ping-collector - get reachability of hosts}

\subsection{Usage: }%
\index{Usage: }

\begin{verbatim}%
ping-collector version 1.25 from remstats 1.0.13b
usage: ping-collector [options]
where options are:
    -d nnn  enable debugging output at level 'nnn'
    -f fff  use 'fff' for config-dir [/home/remstats/etc/config]
    -F      force collection even if it is not time
    -G GGG  only try hosts from group 'GGG', a comma-separated list
    -h      show this help
    -H HHH  only try hosts from 'HHH', a comma-separated list
    -K KKK  only try hosts with key(s) 'KKK', a comma-separated list
    -u      for compatibility with run-remstats; ignored
\end{verbatim}

\subsection{Description: }%
\index{Description: }

ping-collector uses multiping (see the multiping section)  to get numbers on how reachable the 
hosts are.  Each host is sent 10 pings (ICMP echo-request) and the 
number of responses and the min, max and average RTT (Return
Trip Time) is logged, giving the variables ping-sent, ping-rcvd,
pingrtt-min, pingrtt-avg and pingrtt-max.
%------------------------------------ multiping.pod ---

\section{multiping }%
\index{multiping }

\subsection{Usage: }%
\index{Usage: }

\begin{verbatim}%
usage: ../multiping/multiping [-Rdfnqrtv] [-c count] [-i wait] [-l preload]
            [-p pattern] [-s packetsize] host

       Options:
       R: ICMP record route
       d: set SO_DEBUG socket option
       f: 'flood' mode
       n: force addresses to be displayed in numeric format
       r: set SO_DONTROUTE socket option
       t: show results in tabular form
       v: verbose mode for ICMP stuff
\end{verbatim}

\subsection{Description: }%
\index{Description: }

Multiping runs pings in parallel which permits you to ping lots of hosts 
quickly.  I wrote a program using Net::Ping to try to how bad a simple-minded
approach was.  With .025s between pings in my programs and 1s in multiping,
my program took 17 minutes and multiping took 37 seconds.  Maybe someone
will not be able to get multiping going and will re-write it in perl.  Not
me.  I'd be happy to include it if some-one wants to send me a copy.
%------------------------------------ port-collector.pod ---

\section{port-collector - get service status}%
\index{port-collector - get service status}

\subsection{Usage: }%
\index{Usage: }

\begin{verbatim}%
port-collector version 1.26 from remstats 1.0.13b
usage: ../port-collector [options]
where options are:
    -d nnn  enable debugging output at level 'nnn'
    -f fff  use 'fff' for config-dir [/home/remstats/etc/config]
    -F      force collection even if it is not time
    -G GGG  only try hosts from group 'GGG', a comma-separated list
    -h      show this help
    -H HHH  only try hosts from 'HHH', a comma-separated list
    -t ttt  set default timeout to 'ttt' [5]
    -K KKK  only try hosts with key(s) 'KKK', a comma-separated list
    -u      ignore uphosts file
\end{verbatim}

\subsection{Description: }%
\index{Description: }

The \texttt{port-collector} gets data about services running on
specified TCP ports.  It will attempt to connect to the specified
port on the host, and optionally send a string to it.  It will
then examine the response, if any, for certain patterns.  This
permits it to query almost any text-based protocol.  For information
on how to set up a new service, look in the 
scripts (see the scripts section)  directory in the configuration directory.

The rrd specification can also override the port specified by the script,
by appending a colon and the port-number.  E.G.  for a web-server
on port 8000, you could specify the rrd like:

\begin{verbatim}%
	rrd port-http:8000\end{verbatim}

According to whether it can connect to the service and what response
it gets back, it sets a status to one of OK, WARN, ERROR, CRITICAL.
These are arbitrary levels, except that OK means normal, and their
meanings are determined by the configuration file.  The 
\texttt{port-collector} will also log how long it took the service to
respond.  These numbers are not intended for benchmarking, but only
for determining the health of the service.

\subsection{Returned Data}%
\index{Returned Data}

The variables returned by the port-collector are:

\begin{verbatim}%
	port-PORTNAME - containing the status of the port, calculated by 
		the script for this port
	port-PORTNAME-response - containing the response-time for the query\end{verbatim}

\subsection{Other data from the port-collector}%
\index{Other data from the port-collector}

The main RRD for the port-collector is \texttt{port-*}, but it is possible to 
define other RRDs.  If you want to collect information from the results
elicited by the send string, you can provide either or both of the
\texttt{infopattern} or \texttt{valuepattern} in the script associated with the RRD.
The script must be named the same as the rrd.  Look in the
scripts configfile docs (see the scripts configfile docs section)  for details on scripts.

A matching \texttt{valuepattern} will cause the port-collector to return
variables named \texttt{RRDNAME:value\#} with "\texttt{\#}" replaced by a single digit,
corresponding to the number of the parenthesized part of the pattern that
was matched.  

A matching \texttt{infopattern} will cause the port-collector to create status
files for this host called INFO1-RRDNAME.  None of the existing pagemakers (see the pagemakers section) 
use these status files, but the view-writer (see the view-writer section)  could do so if a 
view template (see the view template section)  refered to them.
%------------------------------------ program-collector.pod ---

\section{program-collector }%
\index{program-collector }

\subsection{Usage: }%
\index{Usage: }

\begin{verbatim}%
program-collector version 1.5 from remstats 1.0.9b
usage: ../program-collector [options]
where options are:
    -d nnn  enable debugging output at level 'nnn'
    -f fff  use 'fff' for config-dir [/home/remstats/etc/config]
    -F      force collection even if it is not time
   	-G GGG  only try hosts from group 'GGG', a comma-separated list
    -h      show this help
    -H HHH  only try hosts from 'HHH', a comma-separated list
    -K KKK  only try hosts with key(s) 'KKK', a comma-separated list
    -u      ignore uphosts file
\end{verbatim}

\subsection{Description: }%
\index{Description: }

The \texttt{program-collector} runs a configuration-specified program to get the
values for its RRD.  The program must print its output in one of two forms.

The simple form is just simple values, one per line.  The collector will
invent names for each value, based on the RRD name.  For the first value,
the name for the value will be the name of the RRD.  Any subsequent values
will be named the RRD name followed by a hyphen and a number, giving names
like \texttt{rrdname-2}, \texttt{rrdname-3}, ...

In the other form, the program must provide names itself, so each line is
the name of the value followed by white-space and the value itself.  If the
values produced by the program contain white-space, the program \textbf{must}
use this form.

In either case, the collector will materialize values for the response-time
in named the same as the value, with "\texttt{-response}" appended.  The
response-time is how long the program took to produce that value, cumulative
since the beginning of the running of the program.

There is supplied an RRD definition, \texttt{program-*}, to make this easier to
use for simple cases.  All it needs is a "program" which prints a single value.
The "program" could be anything, including a pipeline.  The RRD has two
variables, \texttt{response} and \texttt{response-time}, the former being the value
supplied by the program, and the latter the run-time of the program.  E.G.
to get a count of th enumber of processes running, you could include an
RRD directive like:

\begin{verbatim}%
	rrd	program-processes /usr/bin/ps -e | wc -l\end{verbatim}

or, if you've got a BSD version of ps:

\begin{verbatim}%
	rrd program-processes /usr/bin/ps ax | wc -l\end{verbatim}
%------------------------------------ remoteping-collector.pod ---

\section{remoteping-collector - reachability from other sites}%
\index{remoteping-collector - reachability from other sites}

\subsection{Usage: }%
\index{Usage: }

\begin{verbatim}%
remoteping-collector version 1.18 from remstats 1.0.13b
usage: ../remoteping-collector [options]
where options are:
    -d nnn  enable debugging output at level 'nnn'
    -f fff  use 'fff' for config-dir [/home/remstats/etc/config]
    -h      show this help
    -p ppp  use port 'ppp' instead of the default [1959]
    -t ttt  use 'ttt' for timeout [60]
    -u      for run-remstats compatibility
\end{verbatim}

\subsection{Description: }%
\index{Description: }

The \texttt{remoteping-collector} is intended to gather ping statistics
(see ping-collector (see the ping-collector section) ) from remote sites.  It works by contacting
remoteping-server (see the remoteping-server section) s running on other machines.  This way it will be 
possible to monitor the same list of hosts from multiple points 
on the network and determine several things:

\begin{itemize}


\item general health of parts of the network of interest to
the co-operating parties, and 

\item if certain parts of the network are performing better or
worse than others.

\end{itemize}
\subsection{Note: }%
\index{Note: }

Note the use of the future tense in the previous paragraph.  I'm
looking for volunteers to run the remoteping-server (see the remoteping-server section)  and let me 
have access to it.  In return, I'll let you look at the
stats that this process gathers.  I'm planning to monitor ISPs and
other sites across Canada, as well as commonly accessed sites around
the world, to determine how we're doing in network 
performance, here in Canada.  If you want to volunteer, please hit
the mailto URL below and ignore the bounce from 
my spam protection (see \textbf{http://silverlock.dgim.crc.ca/\~{}terskine/qmail/tms.html}).
%------------------------------------ snmp-collector.pod ---

\section{snmp-collector - get data via SNMP}%
\index{snmp-collector - get data via SNMP}

\subsection{Usage: }%
\index{Usage: }

\begin{verbatim}%
snmp-collector version 1.28 from remstats 1.0.13b
usage: snmp-collector [options]
where options are:
    -c ccc  use 'ccc' for the read community string; overrides host
    -d nnn  enable debugging output at level 'nnn'
    -f fff  use 'fff' for config-dir [/home/remstats/etc/config]
    -F      force collection even if it's not time
	-G GGG  only try hosts from group 'GGG', a comma-separated list
    -h      show this help
    -H HHH  only try hosts from 'HHH', a comma-separated list
   	-K KKK  only try hosts with key(s) 'KKK', a comma-separated list
    -u      ignore uphosts file
\end{verbatim}

\subsection{Description: }%
\index{Description: }

The snmp-collector collects data available via SNMP.  There are
some things that are hardcoded in, but it's mostly configurable.
It will attempt to query for the following, if available:

\begin{itemize}


\item \textbf{sysDescr} - tells what kind of a device this is

\item \textbf{sysUptime} - how long it has been up

\end{itemize}
and use them in the host index page.

The \texttt{snmpif-*} rrd is wired into the snmp-collector (for now) and
causes it to fetch the following for each interface:

\begin{itemize}


\item \textbf{ifType} - interface type

\item \textbf{ifOperStatus} - operational status

\item \textbf{ifSpeed} - interface speed

\item \textbf{ifInErrors} - input errors

\item \textbf{ifOutErrors} - output errors

\item \textbf{ifInOctets} - input octets (aka bytes)

\item \textbf{ifOutOctets} - output octets (aka bytes)

\item \textbf{ifInUcastPkts} - input unicast packets

\item \textbf{ifOutUcastPkts} - output unicast packets

\item \textbf{ifInNUcastPkts} - input non-unicast 
(broadcast and multicast) packets

\item \textbf{ifOutNUcastPkts} - output non-unicast 
(broadcast and multicast) packets

\end{itemize}
The sysDescr and sysUptime are saved for the host display and the
ifType and ifSpeed are combined to give a hardware description
for the interface.  Crude, but portable.

For other SNMP data, you'll need to look at the 
[oids] (see the [oids] section)  file in the 
configuration directory.  The rrd will need to contain \texttt{oid}
lines specifying names assigned in the oids section (see the oids section section) .
If the host doesn't have one, the rrd will also need to specify
a \texttt{community}.  Here's an example:

\begin{verbatim}%
	[rrd snmpmem]
	source          snmp
	step            300
	data            freemem=ciscofreemem GAUGE:600:0:U
	data            totalmem=ciscototalmem GAUGE:600:0:U
	archives        day-avg week-avg month-avg year-avg
	times           day yesterday week month year
	oid             CiscoFreeMem
	oid             CiscoTotalMem\end{verbatim}

This rrd definition will fetch the amount of free memory and total memory
available on a Cisco router.  Since it's querying a Cisco-specific MIB,
it's not usefull on other gear.
%------------------------------------ snmp-route-collector.pod ---

\section{snmp-route-collector }%
\index{snmp-route-collector }

\subsection{Usage: }%
\index{Usage: }

\begin{verbatim}%
snmp-route-collector version 1.22 from remstats 1.0.13b
usage: snmp-route-collector [options]
where options are:
    -c ccc  use 'ccc' for the read community string; overrides host
    -d nnn  enable debugging output at level 'nnn'
    -f fff  use 'fff' for config-dir [/home/remstats/etc/config]
    -F      force collection even if it's not time
	-G GGG  only try hosts from group 'GGG', a comma-separated list
    -h      show this help
    -H HHH  only try hosts from 'HHH', a comma-separated list
	-K KKK  only try hosts with key(s) 'KKK', a comma-separated list
    -u      ignore uphosts file
\end{verbatim}

\subsection{Description: }%
\index{Description: }

This is a specialized form of SNMP collector which walks a part of the BGP4
MIB, specificly \texttt{bgp4PathAttrBest}, to count routes available from a given
BGP peer host.  It notes both the total number of routes and how many of them
are the best route to that destination.

Unfortunately, it doesn't scale.  On routers with large numbers of peers 
it can take a \textbf{long} time to troll through all the routes.  This needs to 
be replaced.
%------------------------------------ unix-status-collector.pod ---

\section{unix-status-collector - stats from unix hosts}%
\index{unix-status-collector - stats from unix hosts}

\subsection{Usage: }%
\index{Usage: }

\begin{verbatim}%
unix-status-collector version 1.31 from remstats 1.0.13b
usage: unix-status-collector [options]
where options are:
    -d nnn  enable debugging output at level 'nnn'
    -f fff  use 'fff' for config-dir [/home/remstats/etc/config]
    -F      force collection even it it is not time yet
    -G GGG  only try hosts from group 'GGG', a comma-separated list
    -h      show this help
    -H HHH  only try hosts from 'HHH', a comma-separated list
    -K KKK  only try hosts with key(s) 'KKK', a comma-separated list
    -p ppp  connect to server on port 'ppp' [1957]
    -t ttt  set timeout to 'ttt' [10]
    -u      ignore uphosts file
\end{verbatim}

\subsection{Description: }%
\index{Description: }

The unix-status-collector gets its data from the unix-status-server (see the unix-status-server section) .
For a detailed explanation of what the various directives mean,
see its documentation, as it's the implementor.
It sends a query consisting mostly of the sections of the server
that it wants to run.  It can also request that processes with
specific names be counted and that information returned too.
A query for all the sections might look like:

\begin{verbatim}%
	UNAME
	UPTIME
	TIME 986485967
	VMSTAT
	DF
	NETSTAT
	QMAILQ
	PS
	webservers ps count httpd
	FILEAGE
	test fileage /var/spool/locks/lockfile
	PROC
	swaptot proc /proc/meminfo \^SwapTotal:$\backslash$s+($\backslash$d+)
	GO\end{verbatim}

The \texttt{webservers ps count httpd} line requests that the ps section
count the number of processes called \texttt{httpd} and return
that as a variable called \texttt{webservers}.

The \texttt{test fileage /var/spool/locks/lockfile} line requests the last 
modification time of the file \texttt{/var/spool/locks/lockfile}, which is 
returned in seconds.

The \texttt{swaptot ...} line looks for the total swap size in \texttt{/proc/meminfo}.

The best way to see what it will produce is to run it manually.
%------------------------------------ updater.pod ---

\chapter{updater}
\index{updater}

\section{updater - add new data to RRDs}%
\index{updater - add new data to RRDs}

\subsection{Usage: }%
\index{Usage: }

\begin{verbatim}%
updater version 1.23 from remstats 1.0.13b
usage: updater [options] collector
where options are:
  -d nnn  enable debugging output at level 'nnn'
  -f fff  use 'fff' for config-dir [/home/remstats/etc/config]
  -h      show this help
  -H hhh  only process host 'hhh'
  -r rrr  update rrd file 'rrr' (see note1)
  -R RRR  update only rrd 'RRR'

Note1: The -r option must be combined with the -H and -R options and is
intended only to be used by rt-updater.  It will take data only for the
specified host (-H) and RRD (-R) and update the specified (-r) RRD file,
instead of the usual calculated one.
\end{verbatim}

\subsection{Description: }%
\index{Description: }

It reads collector output in standard form from stdin and updates the 
appropriate RRDs.  It wants to know which collector the information came
from to avoid looking for information that won't be there.

It is responsible for figuring out which value belongs to which RRD and 
variable.  It is also responsible for applying any functions to that
value.  It also implements the \texttt{FAKE} function, see the docs for the
rrds config-dir (see the rrds config-dir section) .
%------------------------------------ monitors.pod ---

\chapter{Monitors}
\index{Monitors}

\section{Monitors }%
\index{Monitors }

Currently, there are three monitors:

\begin{itemize}


\item ping-monitor (see the ping-monitor section)  - determines reachability of hosts

\item alert-monitor (see the alert-monitor section)  - figures out status of various values
specified in the rrds (see the rrds section)  and hosts (see the hosts section)  config-files

\item topology-monitor (see the topology-monitor section)  - to analyze changing routes to your monitored hosts

\end{itemize}%------------------------------------ alert-monitor.pod ---

\section{alert-monitor - a status evaluator and alert trigger}%
\index{alert-monitor - a status evaluator and alert trigger}

\subsection{Usage: }%
\index{Usage: }

\begin{verbatim}%
alert-monitor version 1.27 from remstats 1.0.13b
usage: ../alert-monitor [options]
where options are:
  -d nnn  enable debugging output at level 'nnn'
  -f fff  use config-dir 'fff'[/home/remstats/etc/config]
  -h      show this help
  -G GGG  only do hosts in groups 'GGG' (a comma-separated list)
  -H HHH  only do hosts 'HHH' (a comma-separated list)
  -K KKK  only do hosts with keys 'KKK' (a comma-separated list)
  -s sss  search 'sss' data samples for values [5]
  -u      generate alerts for hosts unreachable via a down host
\end{verbatim}

\subsection{Description: }%
\index{Description: }

The \texttt{alert-monitor} compares the current value of variables specified in
the alerts file (see the alerts file section)  in the configuration directory with
threshold values and sets the status of those variables accordingly.  
It saves the current status of variables in \texttt{/home/remstats/data/ALERTS}.

What value corresponds to what status level is set in the 
rrd definition (see the rrd definition section)  or sometimes the 
host definition (see the host definition section) .  This way an rrd
definition will specify generally reasonable levels, but they can be 
overridden for hosts where they aren't reasonable.

For an rrd definition, an alert line looks like:

\begin{verbatim}%
	alert varname relation oklevel [warnlevel [errorlevel]]\end{verbatim}

or

\begin{verbatim}%
	alert varname nodata status\end{verbatim}

[The latter says that missing data for variable \texttt{varname} will cause its status
to be level \texttt{status}.]

For a host-specified alert level, the line looks like:

\begin{verbatim}%
	alert rrdname varname relation oklevel [warnlevel [errorlevel]]\end{verbatim}

or

\begin{verbatim}%
	alert rrdname varname nodata status\end{verbatim}

and the interpretation is the same, except that you're having to say
which rrd this alert refers to.

The available relations are:

\begin{verbatim}%
	$<$ (value is less than threshold)
	$>$ (value is greater than threshold)
	= (value is equal to threshold)
	|$<$ (absolute value of value is less than threshold)
	|$>$ (absolute value of value is greater than threshold)
	delta$<$ (difference between last two values is less than threshold)
	delta$>$ (difference between last two values is greater than threshold)
	$<$daystddev (value is outside threshold * the past day's standard-deviation)
	$<$weekstddev (value is outside threshold * the past day's standard-deviation)
	$<$monthstddev (value is outside threshold * the past day's standard-deviation)\end{verbatim}

\subsection{Example }%
\index{Example }

To make things more concrete for the first (normal) case, here's a real example, 
from the \texttt{load} rrd supplied in \texttt{config-base}:

\begin{verbatim}%
	alert load5 $<$ 3 7 10\end{verbatim}

This means that if the \texttt{load5} variable is less than 3, the status is set to OK.
If it's less than 7, it's WARN, less than 10 it's ERROR and more than that, it's
CRITICAL.

Since the first match is taken, it's possible to leave out the upper levels if
you don't want them to ocurr.  For example if you only wanted \texttt{load5} to ever
go to WARN level, never above, you could use:

\begin{verbatim}%
	alert load5 $<$ 3\end{verbatim}

and then the only possible status levels are OK and WARN.

The possible \texttt{relation}s are: $<$, =, $>$, |$<$, |$>$, delta$<$, delta$>$.  The first
three should be obvious.  The next two allow comparisons to the absolute value of
the variable's current value.  The last two allow comparisons to the change in
value.

\subsection{Causing alerts}%
\index{Causing alerts}

Depending on the lines in the alerts file (see the alerts file section) , the status may also
trigger alerts.  A matching line in the alerts config-file (see the alerts config-file section)  will cause
\texttt{alert-monitor} to run the alerter (see the alerter section)  for each of the specified 
recipients.  It will also be passed, in order:

\begin{itemize}


\item \textbf{recipient} - the recipient; for alert-email (see the alert-email section)  it
will be an email address

\item \textbf{hostname} - the name of the host that the alert applies to

\item \textbf{ip} - the IP number for that host, in case it's not in DNS

\item \textbf{rrdname} - the name of the RRD

\item \textbf{wildpart} - the wild part of a wildcard RRD.  E.G, for an
RRD of \texttt{port-ftp} (using the wildcard RRD \texttt{port-*}) the wildpart
would be \texttt{ftp}.

\item \textbf{variable} - the name of the variable

\item \textbf{status} - the current status, as decided by alert-monitor

\item \textbf{old\_status} - the previous status

\item \textbf{value} - the current value of the variable

\item \textbf{relation} - the relation used to compare the variable to the
threshold, mostly for creating informative messages

\item \textbf{threshold} - the threshold value that was exceeded

\item \textbf{start} - timestamp of when the alert started

\item \textbf{duration} - number of seconds that the alert has been active

\item \textbf{host-description} - the description field from the host config-file

\item \textbf{rrd-description} - the description tag on this rrd (desc="xxx")

\item \textbf{webmaster} - the email address of the remstats person

\item \textbf{template} - the name of the template file to generate the message from.

\end{itemize}%------------------------------------ alerter.pod ---

\section{alerter - construct and send alert text}%
\index{alerter - construct and send alert text}

\subsection{Usage: }%
\index{Usage: }

\begin{verbatim}%
alerter version 1.14 from remstats 1.0.13b
usage: alerter [options] args
where options are:
    -d ddd  set debugging output to level 'ddd'
    -h      show this help
    -f fff  use 'fff' as configuration directory [/home/remstats/etc/config]
The args are documented in alert-monitor; a quick list:
    towho host ip realrrd wildpart var status old_status value relation 
    threshold alertstart duration hostdesc rrddesc webmaster template
\end{verbatim}

\subsection{Description: }%
\index{Description: }

[\texttt{Alerter} may be rolled into the \texttt{alert-monitor} at some point in
the future.  It was easier to test as a separate program, and the performance
hasn't been an issue for me.]

\texttt{Alerter} is passed its parameters (specified above) by the alert-monitor (see the alert-monitor section) .
Most of them are used to fill in information in the text of the alert.  The
interesting ones are \texttt{towho} and \texttt{template}.

It also reads the alert-destination-map config-file (see the alert-destination-map config-file section) 
to decide where the alert needs to go.  This will give it a list of (method, address)
pairs.

For a given template-name, say \texttt{xxx}, and method, say \texttt{method}, it will 
look for files in /home/remstats/etc/config/alert-templates, called:

\begin{verbatim}%
	method-xxx
	method-DEFAULT
	xxx
	DEFAULT\end{verbatim}

and take the first one it finds.  Similarly, it will look for a header to add to the
top of the template called:

\begin{verbatim}%
	method-HEADER
	HEADER\end{verbatim}

and a footer in one of:

\begin{verbatim}%
	method-FOOTER
	FOOTER\end{verbatim}

The three pieces will be concatenated giving the template text.  Then substitutions
will be done for the following magic cookies (see the magic cookies section) :

\begin{verbatim}%
	HOST IP REALRRD WILDPART FIXEDRRD VAR STATUS OLDSTATUS
	VALUE RELATION THRESHOLD START DURATION HOSTDESC RRDDESC 
	NOW TEXTNOW ALERTHOST TOWHO WEBMASTER HTMLURL CGIURL HOSTINDEXURL\end{verbatim}

This gives the alert text.  From the method definition in the
alert-destination-map config-file (see the alert-destination-map config-file section) 
\texttt{alerter} knows which program to run to send the alert text to the
appropriate address, and it does it.

\subsection{Alert-Sending Scripts}%
\index{Alert-Sending Scripts}

These are now easy to write, and in many cases you won't even have to
write one.  There are two requirements for an alert-sending script:

\begin{enumerate}


\item  It must take an address to send to on the command-line, and

\item  It must accept the text on stdin.

\end{enumerate}
E.G. you could use sendmail with no wrapper.
%------------------------------------ alert-email.pod ---

\section{alert-email - an alert sending script}%
\index{alert-email - an alert sending script}

This is a simple script intended to be run by the alerter (see the alerter section) .  
Like all alert-senders, it takes one argument on the command-line:
the "address" to send the alert to.  The text of the alert is
fed to this script on stdin.  

It sends the alert text to "address" via email, by invoking sendmail,
though there's no reason that it couldn't be re-written to do an SMTP
injection directly if some-one wanted to.  With no error-checking, it
could be re-written as:

\begin{verbatim}%
	\#!/bin/sh
	/usr/lib/sendmail "\$1"\end{verbatim}
%------------------------------------ alert-syslog.pod ---

\section{alert-syslog - an alert sending script}%
\index{alert-syslog - an alert sending script}

\begin{verbatim}%
alert-syslog version 1.1 from remstats 1.0.13b
usage: alert-syslog [options] facility:severity
Where options are:
  -d ddd  set debugging output to level 'ddd'
  -f fff  set config-dir to 'fff' [/home/remstats/etc/config]
  -h      show this help
  -l lll  use 'sss' for logger program

Facility comes from the list:
	kern user mail daemon auth lpr news uucp cron local0 - local7
And severity from:
	emerg alert crit err warning notice info debug
\end{verbatim}

This simply raises an alert by writing it to syslog.  Useful if you want to
keep everything in syslog.
%------------------------------------ alert-winpopup.pod ---

\section{alert-winpopup - an alert sending script}%
\index{alert-winpopup - an alert sending script}

This is a simple script intended to be run by the alerter (see the alerter section) .  
Like all alert-senders, it takes one argument on the command-line:
the "address" to send the alert to.  The text of the alert is
fed to this script on stdin.  

It sends the alert to use "address", in this case a windows NetBIOS
machine name, via a Windows popup message.
%------------------------------------ alert-yahoo.pod ---

\section{alert-yahoo - an alert sending script}%
\index{alert-yahoo - an alert sending script}

This is a simple script intended to be run by the alerter (see the alerter section) .  
Like all alert-senders, it takes one argument on the command-line:
the "address" to send the alert to.  The text of the alert is
fed to this script on stdin.  

It sends the alert to "address", in this case a yahoo ID.
%------------------------------------ ping-monitor.pod ---

\section{ping-monitor - determine reachability status}%
\index{ping-monitor - determine reachability status}

\subsection{Usage: }%
\index{Usage: }

\begin{verbatim}%
ping-monitor version 1.10 from remstats 1.0.13b
usage: ping-monitor [options]
where options are:
    -d nnn  enable debugging output at level 'nnn'
    -f fff  use 'fff' for config-dir [/home/remstats/etc/config]
    -h      show this help
    -s sss  examine 'sss' samples [5]
\end{verbatim}

\subsection{Description: }%
\index{Description: }

The \texttt{ping-monitor} looks at the last 5 samples (by default)
of ping data and determines the status of the host.  It will choose
one of the following four statuses:

\begin{itemize}


\item \textbf{UP} - the host is up now and has always (throughout
the sample period) responded to pings.

\item \textbf{UPUNSTABLE} - the host is up now, but on at least
one of the samples, it did not respond. 

\item \textbf{DOWNUNSTABLE} - the host is not responding now, but
it has responded within the sample period.

\item \textbf{DOWN} - the host is down now and has not responded
within the sample period.

\end{itemize}
It also writes coloring information for the ping-index web-page.
%------------------------------------ topology-monitor.pod ---

\section{topology-monitor }%
\index{topology-monitor }

\subsection{Usage: }%
\index{Usage: }

\begin{verbatim}%
topology-monitor version 1.9 from remstats 1.0.13b
usage: topology-monitor [options] oldfile newfile
where options are:
	-d	enable debugging output
	-f fff use 'fff' for config-dir [/home/remstats/etc/config]
	-h	show this help
\end{verbatim}

\subsection{Description: }%
\index{Description: }

The \texttt{topology-monitor} runs out of do-traceroutes (see the do-traceroutes section)  to find changes in 
network paths to the monitored hosts.  After \texttt{do-traceroutes} has run 
traceroute for each monitored host, the \texttt{topology-monitor} compares the
current network path to that host with the previous path.  Currently,
all that is done with this information is to log when it changes.
%------------------------------------ pagemakers.pod ---

\section{Pagemakers }%
\index{Pagemakers }

The remstats pagemakers make web-pages, and update other information used
in making web-pages.  They only need to be run when the configuration file
has changed and run-remstats (see the run-remstats section)  is smart enough to do that.

\begin{itemize}


\item graph-writer (see the graph-writer section)  - makes web-pages with the graphs and links them together

\item snmpif-setspeed (see the snmpif-setspeed section)  - sets maximums on snmpif-* rrds

\item datapage-interfaces (see the datapage-interfaces section)  - makes datapages for every snmpif-* rrd

\item datapage-inventory (see the datapage-inventory section)  - lists all monitored hosts, uptime, software and hardware

\item snmpif-description-updater (see the snmpif-description-updater section)  - updates the descriptions for snmpif-* rrds
from the SNMP descriptions

\end{itemize}%------------------------------------ graph-writer.pod ---

\section{graph-writer }%
\index{graph-writer }

\textbf{Note:} \texttt{graph-writer} has been dropped in favour of the new page-writer (see the page-writer section) 
which is much more flexible.
%------------------------------------ snmpif-setspeed.pod ---

\section{snmpif-setspeed }%
\index{snmpif-setspeed }

This is a gross hack which modifies all the \texttt{snmpif-*} rrds to set the
maximum limits on all monitored interfaces.  The input and output bps 
variables have their maximum set to the the maximum for that interface.
The various packet counters have their maximums set to the maximum possible
packets-per-second assuming minimum-length packets, for that interface
speed.

You may not want to run this if you have better knowledge of what real
maximums will be encountered for particular interfaces.
%------------------------------------ datapage-alert-writer.pod ---

\section{datapage-alert-writer }%
\index{datapage-alert-writer }

\subsection{Usage: }%
\index{Usage: }

\begin{verbatim}%
datapage-alert-writer version 1.4 from remstats 1.0.13b
usage: ../datapage-alert-writer [options]
where options are:
    -d nnn  enable debugging output at level 'nnn'
    -h      show this help
\end{verbatim}

\subsection{Description: }%
\index{Description: }

This writes a datapage (see datapage.cgi (see the datapage.cgi section) ) which gets linked into the header of
each page as the "Alert Index".  It gives a quick overview of alert statuses and values for
each host.  It ought to be self-explanatory.
%------------------------------------ datapage-interfaces.pod ---

\section{datapage-interfaces }%
\index{datapage-interfaces }

This makes a datapage (see the datapage section)  for each host with \texttt{snmpif-*}
rrds, showing all those interfaces.
%------------------------------------ datapage-inventory.pod ---

\section{datapage-inventory }%
\index{datapage-inventory }

This makes a single page showing all the monitored hosts.  For each host,
it shows:

\begin{itemize}


\item the uptime (from the \texttt{uptime} program or SNMP uptime)

\item the hardware type (from the \texttt{uname} program), if available

\item the software version (from the \texttt{uname} program or 
the SNMP system.sysDescr)

\end{itemize}%------------------------------------ datapage-status.pod ---

\section{datapage-status }%
\index{datapage-status }

\subsection{Usage }%
\index{Usage }

\begin{verbatim}%
datapage-status version 1.8 from remstats 1.0.13b
usage: datapage-status [options] file ...
where options are:
	-d	enable debugging output
	-e      show run-time errors in generated pages
	-f fff use 'fff' for config-dir [/home/remstats/etc/config]
	-h	show this help
\end{verbatim}

\subsection{Description }%
\index{Description }

The \texttt{datapage-status} program creates a datapage (to be interpreted by
datapage.cgi (see the datapage.cgi section) ) showing the current values of all
variables in all RRDs for that host, in addition to the usual remstats
headers.
%------------------------------------ page-writer.pod ---

\section{page-writer }%
\index{page-writer }

\textbf{Note:} I intend to completely replace graph-writer (see the graph-writer section)  with page-writer
as soon as I'm comfortable with it.  If you've made changes to \texttt{graph-writer},
then you will need to look at \texttt{page-writer} to see how to incorporate your
changes.  Luckily, \texttt{page-writer} is intended to be configurable, unlike
\texttt{graph-writer}, so you oughtn't to need to change the program, only the page
templates.

\subsection{Usage: }%
\index{Usage: }

\begin{verbatim}%
page-writer version 1.14 from remstats 1.0.13b
usage: ../page-writer [options] collector
where options are:
  -d nnn  enable debugging output at level 'nnn'
  -f fff  use 'fff' for config-dir [/home/remstats/etc/config]
  -h      show this help
  -p ppp  use 'ppp' as the pages file instead
\end{verbatim}

\subsection{Description: }%
\index{Description: }

Like view-writer (see the view-writer section) 's templates, it generates pages from templates with
special mark-up.  It is triggered as a pagemaker (see the pagemaker section)  and attempts to
generate the pages specified in the pages config-file (see the pages config-file section) .

\subsection{Simple Tags}%
\index{Simple Tags}

The first lot are the most important as they produce the graphs:

\begin{itemize}


\item $<$REMSTATS::GRAPH imagedir host rrd graph graphtime$>$
- which gets translated to a $<$RRD::GRAPH...$>$ tag for \texttt{rrdcgi} to generate 
the requested graph.

\textbf{Note:} don't get attached to the form of the tag as I'm considering
modifying it to something like:

\begin{verbatim}%
	$<$REMSTATS::GRAPH HOST="host" RRD="rrdname" GRAPH="graphname" 
	[TIME="timename"] [START="starttime"] [END="endtime"] [SIZE="XxY"]$>$\end{verbatim}

to accomodate the need for extra optional things, like overriding the size.


\item $<$REMSTATS::CUSTOMGRAPH imagedir customname graphtime$>$
- which get translated to a $<$RRD::GRAPH ...$>$ tag for rrdcgi.

\end{itemize}
The next lot make it easy to produce the original look of remstats:

\begin{itemize}


\item $<$REMSTATS::HEADER title$>$
- produces a remstats standard page header

\item $<$REMSTATS::STATUSHEADER$>$ (for hosts status headers) and

\item $<$REMSTATS::RRDHEADER$>$ (for graph headers) and

\item $<$REMSTATS::BAREHEADER$>$ - for page-top HTML up to the begining of the body.

\item $<$REMSTATS::TOOLBAR hostname$>$
- produces a remstats host toolbar

\item $<$REMSTATS::FOOTER$>$ - produces a remstats page footer

\end{itemize}
The next group are to allow you to design your own look:

\begin{itemize}


\item $<$REMSTATS::INCLUDE filename$>$
- translated to $<$RRD::INCLUDE ...$>$ for rrdcgi

\item $<$REMSTATS::TEMPLATE templatename$>$
- inserts a copy of the named template at this point, substituted as
usual.  (See $<$REMSTATS::PAGE...$>$ below.)

\item $<$REMSTATS::VAR varname$>$
- inserts the value of the variable "varname" (usually one of: NAME, FILE,
GROUP, HOST, RRD, GRAPH or GRAPHTIME)

\item $<$REMSTATS::VAR::WILD varname$>$
- inserts the value of the variable "varname" (usually one of: NAME, FILE,
GROUP, HOST, RRD, GRAPH or GRAPHTIME), but doesn't substitute wild graph-names.

\item $<$REMSTATS::VAR::FIXED varname$>$
- as $<$REMSTATS::VAR...$>$ but applies the to\_filename function to the value 
so that it can be used in a path (file or URL) without worrying about, 
e.g., "/" characters screwing things up.

\item $<$REMSTATS::VAR::FIXED::WILD varname$>$
- as $<$REMSTATS::VAR...$>$ but applies the to\_filename function to the value 
so that it can be used in a path (file or URL) without worrying about, 
e.g., "/" characters screwing things up, but doesn't substitute wild 
graph-names.

\item $<$REMSTATS::SET::VAR varname the value$>$
- define a variable for later use.

\item $<$REMSTATS::SET::VAR::FIXED varname the value$>$
- as $<$REMSTATS::SET::VAR...$>$, but applies the to\_filename function to
the value as does $<$REMSTATS::VAR::FIXED...$>$.

\item $<$REMSTATS::SET::VAR::CONFIG varname key1 ...$>$
- sets a variable to a value from the configuration specified by the keys

\end{itemize}
\subsection{Iterator Tags}%
\index{Iterator Tags}

Thes tags have begin and end markers and replicate the text between them.  For
example, a $<$REMSTATS::FOR::GROUP ...$>$ tag will replicate the enclosed text
once for each selected group.

\begin{itemize}


\item $<$REMSTATS::FOR::GROUP [NAME="groupname"] [LIST="g1,g2,..."]$>$ ...
$<$/REMSTATS::FOR::GROUP$>$

\item $<$REMSTATS::FOR::HOST [NAME="hostname"] [LIST="h1,h2,..."] 
[WITHIN="GROUP"] [TAG="TAG"]$>$ ... $<$/REMSTATS::FOR::HOST$>$

\item $<$REMSTATS::FOR::RRD [NAME="rrdname"] [LIST="r1,r2,..."]
[WITHIN="HOST"] [TAG="TAG"]$>$ ... $<$/REMSTATS::FOR::RRD$>$

\item $<$REMSTATS::FOR::GRAPH [NAME="rrdname"] [LIST="g1,g2,..."]
[WITHIN="RRD|CUSTOMGRAPH"]$>$ ... $<$/REMSTATS::FOR::GRAPH$>$

\item $<$REMSTATS::FOR::GRAPHTIME [NAME="rrdname"] [LIST="t1,t2,..."]
[WITHIN="GRAPH"]$>$ ... $<$/REMSTATS::FOR::GRAPHTIME$>$

\item $<$REMSTATS::FOR::CUSTOMGRAPH [NAME="customgraphname"] [LIST="c1,c2,..."]
[WITHIN="HOST"]$>$ ... $<$/REMSTATS::FOR::CUSTOMGRAPH$>$

\end{itemize}
All of these group everything inside them and evaluate the contents 
with the appropriate variables set as specified.  There are four cases:

\begin{enumerate}


\item  The tag has no parameters.  This means that the block will be 
replicated once for each instance of that kind of thing.  E.G. the 
\texttt{$<$REMSTATS::FOR::HOST}$>$ tag with no parameters will be replicated as 
many times as there are hosts, and each copy will have HOST set to 
one of the possible hosts, sorted alphabetically.

\item  The tag supplies a variable the same as its type.  E.G. 
\texttt{$<$REMSTATS::FOR::HOST HOST="xxx"}$>$ This will produce one copy of the block, 
with the variable set to the specified value (in the example, HOST 
will be "xxx").

\item  The tag supplies a parameter \texttt{LIST="x,y,..."}.  This kind of tag 
will be copied as many times as there are elements of the list 
(comma-separated, no spaces).  The copies will be ordered as they are 
in the list.

\item  The tag supplies a parameter \texttt{WITHIN="xxx"}.  This will select a
subset of the items:
\begin{itemize}


\item \texttt{HOST WITHIN="GROUP"} - gives all the hosts in the current group,
as specified in the \texttt{GROUP} variable, sorted by host-name.

\item \texttt{RRD WITHIN="HOST"} - gives all the rrds that the current \texttt{HOST}
has on it, in the order they were specified in the 
hosts config-file (see the hosts config-file section) .

\item \texttt{GRAPH WITHIN="RRD"} - gives all the graphs defined for the current
\texttt{RRD}, in the order they were defined.  If \texttt{HOST} is defined, then it 
will obey \texttt{nograph} directives from the hosts config-file.

\item \texttt{GRAPH WITHIN="CUSTOMGRAPH"} - gives all the graphs defined on the
current \texttt{CUSTOMGRAPH}, in the order they were defined.

\item \texttt{GRAPHTIME WITHIN="GRAPH"} - gives all the graphtimes specified on
the current \texttt{GRAPH}, in the order they were specified.

\item \texttt{GRAPHTIME WITHIN="CUSTOMGRAPH"} - similarly for customgraphs.

\item \texttt{CUSTOMGRAPH WITHIN="HOST"} - gives all the customgraphs specified
for the current \texttt{HOST}, in the order they were specified.

\end{itemize}

\end{enumerate}
\subsection{The $<$REMSTATS::PAGE ...$>$ Tag}%
\index{The $<$REMSTATS::PAGE ...$>$ Tag}

The powerful one which allows a page to cause the generation of other pages.

\begin{itemize}


\item $<$REMSTATS::PAGE pagename filename$>$

\end{itemize}
This will cause page-writer to generate the specified page at this point.  
If it's within one of the iterators, then it will be triggered multiple 
times with various values for the variables.  This way, specifying a 
single page in the pages config-file (see the pages config-file section) , can trigger 
the generation of a whole family of pages.

\subsection{The $<$REMSTATS::CHDIR ...$>$ Tag}%
\index{The $<$REMSTATS::CHDIR ...$>$ Tag}

This changes the current directory for page generation.  This allows a PAGE
directive to generate a full sub-directory tree of pages, with the correct
templates.  Note that \textbf{it will create missing directories, but not
missing hierarchies}.  This is intentional; it would be tedious and 
error-prone to have to create them by hand.  It will, however, note as a
non-fatal error when it creates directories.

\subsection{The $<$REMSTATS::CHMOD mode file$>$ Tag}%
\index{The $<$REMSTATS::CHMOD mode file$>$ Tag}

Changes the mode (an octal number) of the \texttt{file}.  Mainly to make the
GRAPHS sub-directories writable by the apache group.

\subsection{The $<$REMSTATS::CHGRP group file$>$ Tag}%
\index{The $<$REMSTATS::CHGRP group file$>$ Tag}

Changes the group of the file \texttt{file} to \texttt{group}.  Mainly to make the GRAPHS
sub-directories writable by the apache group.

\subsection{The $<$REMSTATS::MKDIR ...$>$ Tag}%
\index{The $<$REMSTATS::MKDIR ...$>$ Tag}

This is identical to the $<$REMSTATS::CHDIR$>$ tag, except that it doesn't
change the current directory.  Note that it won't complain if the
directory already exists.

\subsection{Variable substitution within tags}%
\index{Variable substitution within tags}

Since I don't want to get into the hairy parsing required, you can't 
embed tags within other tags.  I've found, so far, that the only thing
that I've needed to insert into a tag was the value of a variable, so
that's the only non-simple-text that you can use inside a tag.

To insert the value of a variable, include a string like "\texttt{\#\#XXX\#\#}"
where "\texttt{XXX}" is the name of the variable whose value you want.

\subsection{Avoiding extra whitespace}%
\index{Avoiding extra whitespace}

End a line in a page template with a backslash "$\backslash$" and the newline that
would normally be present will not be passed on to the generated page.
%------------------------------------ view-writer.pod ---

\section{view-writer }%
\index{view-writer }

\textbf{Note:} The \texttt{view-writer} is obsolete and replaced by the new page-writer (see the page-writer section) .
It is capable of doing everything \texttt{view-writer} can, and much more.  If there's
anything you're using of \texttt{view-writer} that you think \texttt{page-writer} doesn't
do, please let me know.  If it still works for you, fine, but I'm not 
interested in it any more.

\subsection{Description: }%
\index{Description: }

The \texttt{view-writer} makes web-pages from the view definitions which are
contained in the views config-dir (see the views config-dir section) .  All the documentation
relating to \texttt{view-writer} is there too, as it only implements what the views
request.
%------------------------------------ run-remstats2.pod ---

\chapter{run-remstats2}
\index{run-remstats2}

\section{run-remstats2 }%
\index{run-remstats2 }

\subsection{Usage: }%
\index{Usage: }

\begin{verbatim}%
run-remstats2 version 1.10 from remstats 1.0.13b
usage: ../run-remstats2 [options] [-- options-to-be-passed]
where options are:
    --debug=nnn       enable debugging output at level 'nnn'
    --config_dir=fff  use 'fff' for config-dir [/home/remstats/etc/config]
    --help            show this help
\end{verbatim}

\subsection{Description: }%
\index{Description: }

[\texttt{Run-remstats2} is a replacement for run-remstats (see the run-remstats section) .  If you're converting
to use \texttt{run-remstats2}, then you need to know that the old \texttt{collectors},
\texttt{monitors} and \texttt{pagemakers} lines in the
general config-file (see the general config-file section)  are ignored by \texttt{run-remstats2}.]

\texttt{Run-remstats2} is the main process part of remstats.  It looks at the
run config-file (see the run config-file section)  to get a list of the stages it is to
go through, and their order.  These stages are defined in the
run-stages config-dir (see the run-stages config-dir section) .  Each stage consists of
one or more named command-lines (from a single program, to a pipeline with
many options.  All the requested command-lines for a stage are run in
parallel.  \texttt{Run-remstats2} (like \texttt{run-remstats}) will then wait for all
the commands to finish, before proceeding with the next stage.  There are
more details to controlling this process, but they are documented with
the \texttt{run} and \texttt{run-stages} documentation.

Each command will have stderr redirected to a separate file, and if the
command is a pipeline, each part of the pipeline will have stderr directed to
it's own file, thus keeping each program's error output separate.  These
files are stores in the \texttt{tmp/run-stages} directory, with names like \texttt{[stage]-[id]-out-[pid].[pipeline-part]}.

\subsection{Notes }%
\index{Notes }

When starting, it will look for a file in the tmp directory called 
\texttt{STOP-run-remstats2} and if it exists, will exit immediately.
%------------------------------------ run-remstats.pod ---

\section{run-remstats - run a complete cycle}%
\index{run-remstats - run a complete cycle}

\section{Usage: }%
\index{Usage: }

\begin{verbatim}%
run-remstats version 1.20 from remstats 1.0.13b
usage: ../run-remstats [options] [-- options-to-be-passed]
where options are:
    --debug=nnn       enable debugging output at level 'nnn'
    --config_dir=fff  use 'fff' for config-dir [/home/remstats/etc/config]
	--locking         create a lock-file to prevent two instances running
    --help            show this help
\end{verbatim}

\subsection{Description: }%
\index{Description: }

\textbf{This is obsolete.  You should be running L$<$run-remstats2} instead.

\texttt{run-remstats} is the main script for a remstats collection machine.
As a simplified overview:

\begin{itemize}


\item check-config (see the check-config section)  is run first.

\item In parallel, all the collectors (see the collectors section)  are run, each feeding it's own
updater (see the updater section)  process.  Some of them query remstats servers (see the servers section) , some get their
information in other ways.  So you have a bunch of pipelines like:
\begin{verbatim}%
	xxx-collector | updater xxx\end{verbatim}


\item When all the collectors have finished, the monitors (see the monitors section)  get run in
parallel to figure out what's happening.

\item Afterwards, if the configuration directory has changed, run the pagemakers (see the pagemakers section) ,
to re-do the web-pages.

\item Finally, it prints all the stderr output of all the various programs,
separated by program.

\end{itemize}
For each of these programs, \texttt{run-remstats} will set a timer (see \texttt{watchdogtimer}
in the general config-file (see the general config-file section) ).  If the timer expires and the
program is still running, \texttt{run-remstats} will kill that process.  This avoids the
problem of a hanging collector hanging the whole remstats cycle.

It also manages a lock-file to make sure that two instances don't run 
concurrently.  The lock-file's name is based on the name of the \texttt{run-remstats}
script.  (See \textbf{Running multiple copies of run-remstats} below.)

It keeps a status file in the configured temp directory (\texttt{/home/remstats/tmp} 
by default) which is used by monitor (see the monitor section)  to show where the \texttt{run-remstats} 
process has gotten to.

When starting, it will also look for a file in the tmp directory called 
\texttt{STOP-run-remstats} (default), and if it exists, will refuse to run at all.

\subsection{Running multiple copies of run-remstats}%
\index{Running multiple copies of run-remstats}

If you symlink \texttt{run-remstats} to \texttt{run-remstats-XXX}, then the 
default configuration directory for \texttt{run-remstats-XXX} will be 
\texttt{/home/remstats/etc/config-XXX}.  Since the lock-file is named for the script 
which invokes it, you won't have collisions between
the two instances, as long as your configuration files don't 
conflict.  You can have multiple collector-only instances collecting 
data which is formatted by a single pagemaker instance, (in theory) 
but this will require at least three config-dirs which must be 
closely co-ordinated.  If you want to do this for performance 
reasons, I do plan to address this in future.

\subsection{Configuration: }%
\index{Configuration: }

See the general (see the general section)  config-file.  The lines
to configure run-remstats are:

\begin{verbatim}%
	pinger, collectors, monitors, pagemakers, watchdogtimer\end{verbatim}
%------------------------------------ check-config.pod ---

\section{check-config }%
\index{check-config }

\subsection{Usage: }%
\index{Usage: }

\begin{verbatim}%
check-config version 1.25 from remstats 1.0.13b
usage: check-config [options]
where options are:
	-c      write SNMP communities
	-C CCC  set configuration-debugging output to level 'CCC'
	-d ddd  enable debugging output at level 'ddd'
	-D      dump configuration (must have DEBUG enabled in fixup.config)
	-e      print environment variables to stdout
	-f fff  use config-dir 'fff' [/home/remstats/etc/config]
	-h      show this help
	-l lll	list 'lll' on stdout (comma-separated list of: host, ip, rrd)
	-s sss  shell type to use for -e [sh]
	-t      test-mode; do not make any changes
\end{verbatim}

\subsection{Description: }%
\index{Description: }

\texttt{check-config} uses the common \texttt{read\_config} routine to
read the configuration file which will confirm that it can be
read successfully by other programs.  It also makes sure that
the data directories for all hosts exist, and creates new RRDs.

run-remstats (see the run-remstats section)  also uses the \texttt{-e} option to make basic configuration
information avaiable to the shell.
%------------------------------------ remstats-monitor.pod ---

\section{remstats-monitor - watch remstats processes}%
\index{remstats-monitor - watch remstats processes}

\subsection{Usage: }%
\index{Usage: }

\texttt{remstats-monitor [sleeptime]}

where sleeptime is the time (in seconds) to wait between polls

\subsection{Description: }%
\index{Description: }

This is primarily a development tool.  It loops doing a \texttt{ps} command,
weeding out everything except the remstats processes and cleaning up the results,
to make it easier to read.  It also shows the process-id from run-remstats (see the run-remstats section) 
lock-file, and the status from its status file.

It's not written portably and will probably have to be tweaked by hand
if you want to run it.  If you find it of interest,
please let me (see the me section)  know.
%------------------------------------ cgis.pod ---

\section{CGI Scripts}%
\index{CGI Scripts}

These are intended to be invoked via the html-writer created toolbars, to
do the supplied functions to the host in question.

\begin{itemize}


\item alert.cgi (see the alert.cgi section)  - Shows the current alert status of selected rrd variables.

\item availability-report.cgi (see the availability-report.cgi section)  - Shows availability of RRD variables.

\item dataimage.cgi (see the dataimage.cgi section)  - Generates images based on live data.

\item datapage.cgi (see the datapage.cgi section)  - Generates web-pages containing dynamic data.

\item graph.cgi (see the graph.cgi section)  - Allows non-remstats web-pages to show remstats graphs.

\item log-event.cgi (see the log-event.cgi section)  - log a manual event.

\item ping.cgi (see the ping.cgi section)  - Ping the host.

\item showlog.cgi (see the showlog.cgi section)  -  Display selected portions of the remstats log files.

\item traceroute.cgi (see the traceroute.cgi section)  - find network path to a host

\item whois.cgi (see the whois.cgi section)  - look up information about hosts, IP\#s, ...

\end{itemize}
\textbf{Note:} Access to the various CGIs is controlled by the
access config-file (see the access config-file section) .
%------------------------------------ alert-cgi.pod ---

\section{alert.cgi - Alert Reporting and Updating}%
\index{alert.cgi - Alert Reporting and Updating}

This CGI script will generate the Alert Report and also let you modify it.
You can turn alerts off for a specific line (by checking the \texttt{quench}
check-box).  You can also attach comments to a specific alert, for example
to let other people know that you're already working on it, or when a
service will be available again.
%------------------------------------ availability-report-cgi.pod ---

\section{availability-report.cgi }%
\index{availability-report.cgi }

The \texttt{availability-report.cgi|availability-report-cgi} calls availability-report (see the availability-report section)  to produce
a report of "availability" according to the definitions in the
availability (see the availability section)  config-file.
%------------------------------------ dataimage-cgi.pod ---

\section{dataimage.cgi - create images driven by live data}%
\index{dataimage.cgi - create images driven by live data}

\subsection{Usage: }%
\index{Usage: }

\begin{verbatim}%
	$<$IMG SRC="/remstats/dataimage.cgi?imagename"$>$\end{verbatim}

\subsection{Description: }%
\index{Description: }

\texttt{Dataimage.cgi} reads an image definition from 
\texttt{/home/remstats/datapage/imagename.image}.  Some of the commands are in common
with datapage.cgi (see the datapage.cgi section)  and are documented
there:

\begin{verbatim}%
	oid, rrd, status, eval, debug, macro, macroend and *EOD*\end{verbatim}

These retrieve and manipulate data.  There are also commands to create
images:

\begin{verbatim}%
	image, colordef, color, linewidth, line, rectangle, circle, 
	fill, font, text, out, flow\end{verbatim}

\section{Image Commands}%
\index{Image Commands}

\subsection{image }%
\index{image }

The image command has two formats.  The first looks like:

\begin{verbatim}%
	image WIDTH HEIGHT\end{verbatim}

This creates a blank image of the size specified.  Sometimes you'll want a
background for the image, and you can use the second form to specify
a file to read for the background:

\begin{verbatim}%
	image BGFILE\end{verbatim}

This will create the new image the same size as the one in \texttt{BGFILE}, 
by reading \texttt{BGFILE} and using its contents as the background.  N.B., the
image must be a PNG graphic.

The \texttt{image} command also defines a few colors (see \texttt{colordef} below):
\texttt{black}, \texttt{white} and \texttt{transparent}, sets the current color to
\texttt{black}, fills the image with \texttt{white} and sets the \texttt{linewidth} to 1.

\subsection{colordef }%
\index{colordef }

[It can also be spelled \texttt{colourdef}.]

This defines a new colour and names it.  The command looks like:

\begin{verbatim}%
	colordef COLORNAME RED GREEN BLUE\end{verbatim}

where \texttt{RED}, \texttt{GREEN} and \texttt{BLUE} specify the level of each of those colours
to be mixed to define the colour referred to in the script as \texttt{COLORNAME}.

\subsection{color }%
\index{color }

[It can also be spelled \texttt{colour}.]

This sets the current colour, to be used by those commands that don't specify 
a colour.  It is used as simply:

\begin{verbatim}%
	color COLORNAME\end{verbatim}

\subsection{linewidth }%
\index{linewidth }

This sets the width of lines.  It isn't honoured by all other commands,
unfortunately, but so far this hasn't been a problem for me.  It looks like:

\begin{verbatim}%
	linewidth WIDTH\end{verbatim}

\subsection{line }%
\index{line }

This just draws a line in the current \texttt{color} and \texttt{linewidth}:

\begin{verbatim}%
	line X1 Y1 X2 Y2\end{verbatim}

\subsection{rectangle }%
\index{rectangle }

This is a way to draw a rectangle, without useing \texttt{line} 4 times:

\begin{verbatim}%
	rectangle X1 Y1 X2 Y2 [filled]\end{verbatim}

The co-ordinates (\texttt{X1}, \texttt{Y1}) and (\texttt{X2}, \texttt{Y2}) define oposite corners
of the rectangle.  If the keyword \texttt{filled} is added to the end, the 
rectangle will be filled with the current colour as well.

\subsection{circle }%
\index{circle }

Here you get a circle:

\begin{verbatim}%
	circle X Y RADIUS [filled]\end{verbatim}

The circle will be centered on (\texttt{X}, \texttt{Y}) with a radius of \texttt{RADIUS}.
If the keyword \texttt{filled} is added to the end, the 
circle will be filled with the current colour as well.

\subsection{fill }%
\index{fill }

This command permits you to fill arbitrary regions:

\begin{verbatim}%
	fill X Y [COLORNAME]\end{verbatim}

The \texttt{COLORNAME} is optional.

\subsection{text }%
\index{text }

The \texttt{text} command sets text into the image, for labelling things:

\begin{verbatim}%
	text X Y TEXT\end{verbatim}

\subsection{font }%
\index{font }

This changes the font for the \texttt{text} command:

\begin{verbatim}%
	font (giant|large|mediumbold|medium|small|tiny)\end{verbatim}

\subsection{out }%
\index{out }

This permits the script to output additional information to an auxiliary
file.  I added this for doing image-maps automatically, which can be
automatically loaded by a datapage.cgi (see the datapage.cgi section)  web-page.

The syntax is:

\begin{verbatim}%
	out TEXT\end{verbatim}

\subsection{flow }%
\index{flow }

This draws a strange double-headed, bi-coloured arrow.  Think of it as two
half arrows, split lengthwise, one in each direction.  The colour and width
of each half arrow indicates the flow in that direction.  I use it for
indicating network traffic flow, which usually isn't the same in both 
directions.  It looks like:

\begin{verbatim}%
	flow X1 Y1 X2 Y2 INFLOW OUTFLOW\end{verbatim}

The co-ordinates (\texttt{X1}, \texttt{Y1}), (\texttt{X2}, \texttt{Y2}) indicate the ends of the 
flow.  \texttt{INFLOW} and \texttt{OUTFLOW} indicate the level in each direction,
relative to (\texttt{X1}, \texttt{Y1}).
%------------------------------------ datapage-cgi.pod ---

\section{datapage.cgi - dynamic data in web-pages}%
\index{datapage.cgi - dynamic data in web-pages}

\subsection{Usage: }%
\index{Usage: }

\begin{verbatim}%
	$<$A HREF="/remstats/datapage.cgi?pagename"$>$whatever$<$/A$>$\end{verbatim}

\subsection{Data Collection}%
\index{Data Collection}

\texttt{Datapage.cgi} looks for the page definition in the file 
\texttt{/home/remstats/data/DATAPAGES/pagename.page} The page definition is in two parts, 
separated by a line like:

\begin{verbatim}%
	BEGIN-PAGE\end{verbatim}

The first part's purpose is to define variables to be included in the 
second part, which is an HTML template, with magic tags.

All lines in the first or definition part are subject to variable interpolation.
Any ocurrance of \texttt{\$\{variablename\}} will be replaced by the current
contents of the variable \texttt{variablename}.  This will be done up to
five levels, permitting expansion of \$\{\$\{h\}\_\$\{interface\}\}, providing that
you've got values for the variables \texttt{h} and \texttt{interface}.
N.B. variable names must be lower case.

In addition, within a macro expansion, macro-arguments will also be
interpolated, before variable interpolation, for all ocurrances of
\texttt{\$\{ARGNAME\}}, assuming that there is an argument for the 
current macro called \texttt{ARGNAME}.  N.B. macro argument names must 
be UPPER case.

\subsection{Common Commands}%
\index{Common Commands}

The commands permitted in the first part are:

\begin{verbatim}%
	oid, rrd, status, eval, debug, macro, macroend,
	alertstatus, alertvalue and *EOD*\end{verbatim}

These commands are in common with the dataimage.cgi (see the dataimage.cgi section) 
script, but are only documented here.

\begin{itemize}


\item oid 
This fetches an SNMP value into a datapage variable.  The command looks like:

\begin{verbatim}%
	oid VARNAME HOSTNAME OIDNAME\end{verbatim}

The \texttt{VARNAME} is the name of the datapage variable (let's just call 
them variables from now on).

The \texttt{HOSTNAME} is the name of the host
to query.  The SNMP community-string is usually supplied in
that host's config-file, but can be supplied in usual MRTG fashion by
giving \texttt{COMMUNITY@HOSTNAME} or even \texttt{COMMUNITY@HOSTNAME:PORTNUMBER}
instead of the \texttt{HOSTNAME}.

The \texttt{OIDNAME} must be defined in the
oids config-file (see the oids config-file section) , but can be suffixed by the usual
numbers.  E.G. you can use ifName.4 to get the ifName for interface 4.


\item rrd 
This fetches a value from an RRD database into a variable.  It looks like:

\begin{verbatim}%
	rrd VARNAME HOSTNAME RRDNAME DSNAME CF\end{verbatim}

The \texttt{RRDNAME} is the name of the rrd, as remstats knows it, not fully
qualified.  I.E. it will be under the config-file defined \texttt{datadir},
and under the host's directory under that.

The \texttt{DSNAME} is the ds-name within that RRD file and the \texttt{CF}
is the usual RRD consolidation-function to be applied.


\item status 
This is so-named because it fetches remstats status files, usually written
by the various collectors (see the collectors section)  and monitors (see the monitors section) .  It looks like:

\begin{verbatim}%
	status VARNAME HOSTNAME STATUSNAME\end{verbatim}

The \texttt{STATUSNAME} is the name of the status file, as named in the
host's data directory.  There is a standard mapping applied by the
function \texttt{to\_filename} from the \texttt{remstats.pl$<$/t} file to 
munge the filename so that it won't conflict with the filesystem.  Either
look for the name in the data directory, use the function (see eval) or
look at the code.  I \textbf{am} planning on changing the mapping when I
figure out the best way to do it.


\item eval 
The eval command lets you modify the values fetched by previous \texttt{oid},
\texttt{rrd}, \texttt{status} and \texttt{eval} commands with arbitrary perl code.  
It looks like:

\begin{verbatim}%
	eval VARNAME PERLEXPRESSION\end{verbatim}

The \texttt{PERLEXPRESSION} is a perl expression and can be arbitrarily complex, 
but gets messy quickly with the \texttt{datapage.cgi} and perl both doing variable
interpolation.

Note: \texttt{datapage.cgi} uses private.pl (see the private.pl section) , so you can include commonly
used functions here to make your datapage creation easier.


\item debug 
The \texttt{debug} command takes a number which is the level to set debugging to.
It causes extra output which may be helpful in figuring out why your page
isn't working the way you expected.


\item alertstatus 
This lets you fetch the alert level for a given (host, rrd, dsname, cf)
combination.  The command looks like:

\begin{verbatim}%
	alertstatus VARNAME HOSTNAME RRDNAME DSNAME [CF]\end{verbatim}

This will fetch the alert status and put it in the datapage variable \texttt{VARNAME}.
The status will be the same set of values shown on the alerts report (see the alerts report section) 
for status.
The \texttt{CF} parameter is optional and is rrdtool's consolidation function.
It will be set to \texttt{AVERAGE} if it's not supplied.


\item alertvalue 
This is the same as \texttt{alertstatus} except that it sets the variable to the
current value of the (host, rrd, variable, cf) combination.


\end{itemize}
\subsection{The HTML template}%
\index{The HTML template}

This is almost just HTML with a few magic tags inserted.  The difference is
that the beginning must include HTTP headers.  If you don't want anything
fancy, just begin like:

\begin{verbatim}%
	------ cut here ------
	BEGIN-PAGE
	content-type: text/html\end{verbatim}

\begin{verbatim}%
	------ cut here ------\end{verbatim}

Note: the empty line after \texttt{content-type:} is \textbf{not} optional.  It's
necessary to end the HTTP headers.

The magic tags are:

\begin{itemize}


\item $<$DATAPAGE::STATUS host statusfile$>$
inserts a specified status file


\item $<$DATAPAGE::VAR varname$>$
interpolates the value of a datapage variable


\item $<$DATAPAGE::HEADER title$>$
generates a standard remstats header


\item $<$DATAPAGE::STATUSHEADER hostname$>$
generates the status headers for the named host


\item $<$DATAPAGE::TOOLBAR hostname$>$
generates the toolbar for the named host


\item $<$DATAPAGE::FOOTER$>$ 
generates a standard remstats footer


\item $<$DATAPAGE::INCLUDE filename$>$
include the contents of a file from the datapage directory, for imagemaps ...


\item $<$DATAPAGE::PATHINCLUDE filename-with-path$>$
include contents of a file specified with a complete path


\item $<$DATAPAGE::MACRO macroname [argvalue] ...$>$
include boilerplate HTML with substitutions


\item $<$DATAPAGE::GRAPH host rrd graph time$>$
generate the specified remstats graph


\item $<$DATAPAGE::CUSTOMGRAPH graph time$>$
generate the specified remstats customgraph


\item $<$DATAPAGE::ERRORS$>$ 
inserts the text of errors encountered in generating the page.  Without
this one, you won't see any errors.  That way you include the errors and
debugging output (see next item), which you're creating/debugging the 
datapage and afterwards turn them off.  The errors and debugging output
may include information you don't want to reveal to outsiders.  Also,
collecting all the error output together avoids spoiling the formatting
of the page.


\item $<$DATAPAGE::DEBUG$>$ 
inserts debugging output.  Without it, you won't see any debugging output.


\end{itemize}%------------------------------------ graph-cgi.pod ---

\section{graph.cgi - exporting remstats graphs}%
\index{graph.cgi - exporting remstats graphs}

The purpose of \texttt{graph.cgi} is to allow remstats graphs
to appear on external (not part of remstats) web-pages.
It's \textbf{not} efficient, with the graphs being generated whenever
the page is reloaded, but it is portable.  All you do is to
create an $<$IMG SRC...$>$ tag with the appropriate values, like:

\begin{verbatim}%
	$<$IMG SRC="/remstats/graph.cgi?host=aaa\&rrd=bbb\&graph=ccc\&graphtime=ddd"$>$\end{verbatim}

and replace \texttt{aaa} with the name of the host, as remstats knows it, \texttt{bbb}
with the name of the RRD, \texttt{ccc} with the name of the graph within that RRD,
and \texttt{ddd} with the name of the timespan, from the 
times config-file (see the times config-file section) .  If the RRD is a wildcard RRD, e.g. 
\texttt{snmpif-*}, then you must use the specific instance, e.g. \texttt{snmpif-eth0}.

The \texttt{graphtime} specifier may be replaced by \texttt{start=xxx} and \texttt{end=yyy} 
or omitted altogether.  If \texttt{start} and \texttt{end} are specified, the permitted
formats are documented in the manpage for rrdfetch.  If you use the form:

\begin{verbatim}%
	hh:mm dd.mm.yyy OFFSET\end{verbatim}

with OFFSET being a string composed of multiple ocurrences of plus or minus,
a number and the units (s|min|h|d|mon|y).  E.G. +2h+40min or -20d+5h

The graph's defined size may also be overridden using \texttt{width=www} and 
\texttt{height=hhh}.  These specify the size, in pixels, of the graph portion of
the image, not the whole image, as usual with rrdtool.

That's all there is to it.
%------------------------------------ log-event-cgi.pod ---

\section{log-event.cgi - log events from a web-page}%
\index{log-event.cgi - log events from a web-page}

This shows up on the showlog.cgi (see the showlog.cgi section)  as a link to permit
you to manually enter events into the log.  Don't feel
obliged to enter all the fields.  The data isn't checked
for meaning, just for syntax.  In other words, a host-name
must look like a host-name, but it doesn't have to be a 
real host-name.

This cgi-script ought to be protected.  See the 
web-server installation (see the web-server installation section)  docs.
%------------------------------------ macinfo-cgi.pod ---

\section{macinfo.cgi }%
\index{macinfo.cgi }

\subsection{Usage: }%
\index{Usage: }

In a URL like:

\begin{verbatim}%
	$<$A HREF="/remstatsmacinfo.cgi?host=10.11.12.13"$>$macinfo for 10.11.12.13$<$/A$>$\end{verbatim}

\subsection{Description: }%
\index{Description: }

It simply re-formats the output of macinfo (see the macinfo section)  into HTML and wraps remstats
headers and footers around it.
%------------------------------------ ping-cgi.pod ---

\section{ping.cgi }%
\index{ping.cgi }

The \texttt{ping.cgi} script allows you to ping a host.  It's intended to be
called off a host's toolbar, but that's not required.  Simply provide
the hostname or IP number and it'll ping it.  

As an example, you could ping ftp.uu.net with a URL like:

\begin{verbatim}%
	/remstats/ping.cgi?host=ftp.uu.net (see the /remstats/ping.cgi?host=ftp.uu.net section) \end{verbatim}
%------------------------------------ show-config-cgi.pod ---

\section{show-config.cgi }%
\index{show-config.cgi }

\subsection{Usage: }%
\index{Usage: }

where \texttt{xxx} is one of host, rrd, tool, time, archive and \texttt{yyy} is the name
of one of whatever \texttt{xxx} refers to.  E.G. of \texttt{xxx} is \texttt{host}, then \texttt{yyy}
must be the name of one of the defined hosts.

The result has links to \texttt{show-config.cgi|show-config-cgi} for other kinds of
configuration entry where it makes sense and links to the documentation where
that makes sense.

Try it.
%------------------------------------ showlog-cgi.pod ---

\section{showlog.cgi }%
\index{showlog.cgi }

Any alert is also logged to the remstats log files (one file per day).  Other
information is also logged, for example, the topology-monitor (see the topology-monitor section)  logs network
topology changes.  The \texttt{showlog.cgi} script allows you to display selected
portions of the log files, by time-period, by host, ...
%------------------------------------ traceroute-cgi.pod ---

\section{traceroute.cgi }%
\index{traceroute.cgi }

This script uses traceroute (see the traceroute section)  to find the path from the remstats host to 
some other specified host.  It's intended to be called off a host's
toolbar, but that's not required. For example, you could trace the path from
here (trevelyan.sourceworks.com) to ftp.uu.net with a URL like:

\begin{verbatim}%
	/remstats/traceroute.cgi?host=ftp.uu.net (see the /remstats/traceroute.cgi?host=ftp.uu.net section) \end{verbatim}

There are other options that you can specify:

\begin{itemize}


\item \texttt{no\_names} - just shows IP numbers instead of looking up the domain-names

\item \texttt{ASNs} - look up the Autonomous System Numbers (ASNs) for the IP number of each hop.  It
can be useful for figuring out which networks you are traversing.

\item \texttt{owners} - look up the "owner" via SOA records

\item \texttt{fast} - continues on to the next hop as soon as the current one answers

\end{itemize}%------------------------------------ whois-cgi.pod ---

\section{whois.cgi }%
\index{whois.cgi }

This script talks to the ARIN whois database (by default) to look up network
names, IP numbers and AS numbers.  It's usually linked into the results of
traceroute.cgi (see the traceroute.cgi section)  so that you can look up what your
traceroute results actually mean.

Try \texttt{traceroute.cgi} if you want to see how it works.
%------------------------------------ troubleshooting.pod ---

\section{Troubleshooting }%
\index{Troubleshooting }

[Almost all of the various programs accept a \texttt{-h} or \texttt{--help} flag to ask
for help, if you've forgotten or don't know it.  Most also accept a \texttt{-d}
or \texttt{--debug} flag to set the level of debugging output.]

\subsection{Tools }%
\index{Tools }

There are two tools which are specifically intended for finding problems:

\begin{itemize}


\item check-config (see the check-config section)  - You should \textbf{always} run this after making any configuration change.  It
won't find all possible problems with the configuration files, but if it
doesn't like your configuration, you can be sure that nothing else will,
If it prints nothing, all is well.  It will also note when it is creating
new RRDs and directories, but this is not an error.

\item check-rrdlast (see the check-rrdlast section)  - checks the last-update time on all RRD files.  Make sure that you have
no \texttt{TIMEWARP} marked RRDs.  These are RRDs with a last-update time in the
future.  They won't be updated until after what they think is the last-update
time.  Anything that's \texttt{STALE} marked is also a concern
if it's long overdue.  These would be RRDs which aren't getting updated, for
some reason.

\end{itemize}
\subsection{Are things running?}%
\index{Are things running?}

The first thing to check is whether run-remstats2 (see the run-remstats2 section)  is being started by cron.
Look in \texttt{tmp} for a file called \texttt{LOG-run-remstats2}.  If it's there and
doesn't have anything marked ERROR or ABORT, this is a good thing.  Check
that the modified time for the file is within the last 5 minutes, to make
sure that it's being run at appropriate times.

The next thing to check is whether run-remstats2 (see the run-remstats2 section)  or run-remstats (see the run-remstats section)  has
been told to run the collector (see the collector section)  at all. 

\begin{itemize}


\item For run-remstats (see the run-remstats section) , look at the \texttt{collectors} line in the \texttt{general}
config-fileebet

\item For run-remstats2 (see the run-remstats2 section) , look at the \texttt{run-stages/collectors} file and
make sure that the appropriate line is uncommented.

\end{itemize}
Once it's running, you can check what data it found in the last run.
Look in data/LAST/$<$collectorname$>$.  The lines are all the data which that
collector found on the last run.

\subsection{Remstats Services}%
\index{Remstats Services}

First of all, make sure that the perl specified in the shebang line exists
and is the right perl to use.  If you're getting a message like "\texttt{No such file or directory}" and you can see that the script exists, check the first line
of the script, which will look something like:

\begin{verbatim}%
	\#!/usr/bin/perl -w\end{verbatim}

And make sure that the specified perl (in the example \texttt{"/usr/bin/perl"}
exists, is executable by the user specified in the inetd configuration, and
is the correct perl to use.

If you're having difficulty getting data from remstats servers (see the servers section) , you should
check them out without the collectors.  For the unix-status-server (see the unix-status-server section) , you
can make sure that you have access, by running the following on the server
host:

\begin{verbatim}%
	\% telnet localhost 1957
	UNAME
	GO\end{verbatim}

If you don't get connected, you'll have to look at your inetd configuration,
and the output it logs.  You might have to run inetd with debugging flags to
get it to log enough to be usefull.

If you get connected and then shortly afterwards disconnected, this is the 
signature of the tcp\_wrappers refusing a connection: check out
\texttt{/etc/hosts.allow}.  You ought to have lines for the remstats servers which
you are running on that host:

\begin{verbatim}%
	unix-status-server : collector-host localhost
	log-server : collector-host localhost\end{verbatim}

Note that inetd will refer to \texttt{unix-status} because that's the name of the
service from \texttt{/etc/services} (I had to shorten the name to \texttt{unix-status}
from \texttt{unix-status-server}, as I ran into a length limit on service names
somewhere.)  However, \texttt{hosts.allow} refers to program names.

\subsection{Debugging Output}%
\index{Debugging Output}

Almost all of the programs will take a \texttt{-d ddd} flag to set the debugging
output level.  Unless you're willing to look at the code, you probably won't
want more than level 1, but it can be helpful.

If you're trying to figure out what's happening, it's helpful to run
run-remstats2 (see the run-remstats2 section)  interactively with \texttt{-d 1}, after disabling it from crontab.
This will tell you which processes are being run and how long they're taking.

Follow that by running a collector interactively with the \texttt{-d 1} flag, and
possibly a \texttt{-H xxx} flag to restrict it to host \texttt{xxx}.

It's also good to know what the output of a collector looks like.  You can
get a general description under collectors (see the collectors section) , or you can just run a collector
interactively and see what it comes up with.  Remember that collectors won't
even try to get information that they don't need yet, so if you want to see
everything that the collector would get in a normal run, you might want to
use the \texttt{-F} flag to force collection and/or \texttt{-u} to attempt even hosts
which are down.

\subsection{Usefull Files}%
\index{Usefull Files}

There are some useful files to look at:

\begin{itemize}


\item \texttt{data/ALERTS} contains all the alert statuses for all variables,
whether they have an alert defined or not.  It is maintained by the
alert-monitor (see the alert-monitor section) .

\item \texttt{data/IP\_CACHE} contains IP numbers which were looked up by the
ping-collector (see the ping-collector section)  during the current run of run-remstats2 (see the run-remstats2 section) .  This is
to save extra DNS lookups for large sites.  If you've re-numbered any
hosts, you might want to look here to see what remstats was using for an
IP number for a host.

\item \texttt{data/LOGS} is a directory containing log-files, one per day.
Currently, they are kept for a bit less than a year.  You can grep them to
do searching in them that showlog.cgi (see the showlog.cgi section)  won't let you do.
These show all events which remstats thought might be interesting.

\item \texttt{data/NT} is a directory with information about Windows NT hosts/domains
collected by nt-discover (see the nt-discover section) .

\item \texttt{data/TRACEROUTES} contains all the aggregate traceroute information
collected by do-traceroutes (see the do-traceroutes section) .

\item \texttt{data/LAST} has the data the last run of the various collectors
collected, in files named after the collector.  Check here to make sure that
your data is being collected, if the graphs are getting no new data.  Note
that if you have RRDs with different step times, it is normal that the longer
step times won't always show up.

\item under \texttt{tmp} are several interesting things.  You'll find a file called
\texttt{LOG-run-remstats2} (or \texttt{LOG-run-remstats} if you're still running it
instead).  This file tells which processes were run during the last run of
run-remstats2 (see the run-remstats2 section)  (or run-remstats (see the run-remstats section) ).  There is also a file called
\texttt{LOG-run-remstats2.old} which has the same stuff for the run before.
The file \texttt{STATUS-run-remstats2} contains info on the progress of a running
instance of \texttt{run-remstats2}.

Under \texttt{tmp/run-stages} are the files containing the stderr output from the
processes started by \texttt{run-remstats2}.  In an ideal world, you'll only see
files in here while \texttt{run-remstats2} is running, but if it dies or you kill it,
you may have files left in here which may give you an idea of what went wrong
before it died.


\item the pseudo-host \texttt{\_remstats\_} maintains information about the
functioning of remstats itself, currently, it's mostly about the
performance of the various collectors.

\end{itemize}
\section{Trouble reporting}%
\index{Trouble reporting}

Send your report to the mailing-list $<$remstats-list@lists.sourceforge.net$>$.

Always include the remstats version.  You can find it in the \texttt{VERSION} file
(just the version number) or the \texttt{release} file (status, number and date).
Unless you've got a pre-release version, the \texttt{VERSION} file has what you
need.

Run check-config (see the check-config section)  interactively and include the output.

Describe what you ran to produce the error, including command-line args.

Include any error messages you received from cron, or found in any of the 
above files, even if you don't understand them.

Complain, in detail, about documentation or error messages which you don't
understand.  They're intended to be comprehensible.

All remstats error messages should be prefixed by \texttt{ERROR:} or \texttt{ABORT:}.
(The only difference is that an abort is an error the remstats can't or
shouldn't continue from.)  Nothing should trigger any perl errors or warnings;
complain about them.
%------------------------------------ check-rrdlast.pod ---

\section{check-rrdlast }%
\index{check-rrdlast }

\subsection{Usage: }%
\index{Usage: }

\begin{verbatim}%
check-rrdlast version 1.3 from remstats 1.0.13b
usage: ../check-rrdlast [options]
where options are:
  -d nnn  enable debugging output at level 'nnn'
  -D      show the data for the last update
  -f fff  use 'fff' for config-dir [/home/remstats/etc/config]
  -h      show this help
Note: the output is one line for each rrd showing: host, rrd, last-update,
a pretty-printed version of last-update, (now - last-update), step, and
status( OK is ok, STALE means that it is overdue for an update and TIMEWARP
means that the last-update time is in the future.  The data will be on its
own line like: 
  timestamp dsname=value ...
\end{verbatim}

\subsection{Description: Shows the last update time for all RRDs and notes if it is in the future or
more than the RRD's step time in the past.}%
\index{Description: Shows the last update time for all RRDs and notes if it is in the future or
more than the RRD's step time in the past.}
%------------------------------------ do-traceroutes.pod ---

\chapter{do-traceroutes}
\index{do-traceroutes}

\section{do-traceroutes - find the path to each host}%
\index{do-traceroutes - find the path to each host}

This runs traceroute against each host being monitored.  After
they've all finished, it runs the topology-monitor (see the topology-monitor section) .  

I'm planning to make graphical representations of how you are
connected to the hosts you're monitoring, but that's not working yet.
%------------------------------------ traceroute.pod ---

\section{traceroute }%
\index{traceroute }

\subsection{Usage: }%
\index{Usage: }

\begin{verbatim}%
2.9.2tee0.0: Usage: traceroute [-adnruvAMOQ] [-w wait] [-S start_ttl] [-m max_ttl] [-p port#] [-q nqueries] [-g gateway] [-t tos] [-s src_addr] [-g router] host [data size]
      -a: Abort after 10 consecutive drops
      -d: Socket level debugging
      -g: Use this gateway as an intermediate hop (uses LSRR)
      -S: Set start TTL (default 1)
      -m: Set maximum TTL (default 30)
      -n: Report IP addresses only (not hostnames)
      -p: Use an alternate UDP port
      -q: Set the number of queries at each TTL (default 3)
      -r: Set Dont Route option
      -s: Set your source address
      -t: Set the IP TOS field (default 0)
      -u: Use microsecond timestamps
      -v: Verbose
      -w: Set timeout for replies (default 5 sec)
      -A: Report AS# at each hop (from GRR)
      -M: Do RFC1191 path MTU discovery
      -O: Report owner at each hop (from DNS)
      -P: Parallel probing
      -Q: Report delay statistics at each hop (min/avg+-stddev/max) (ms)
      -T: Terminator (line end terminator)
      -U: Go to next hop on any success
\end{verbatim}

\subsection{Description: }%
\index{Description: }

Hmm.  I think that describes its use pretty well.  What does it do?  
Oh.  Well it sends UDP packets with the time-to-live set to 1, then 2 
then 3 and so on.  This causes the routers that these packets are sent 
through to complain after the requisite number of hops.  I.E. the first router 
complains about the first packets, with TTL set to one, the second about 
the packets with TTL set to two etc.  Traceroute catches the complaints 
and times how long it took.  This not only shows you how your packets 
are getting to the destination, but sometimes, where the congestion
is as well.  There's a lots better explanation in the source, so 
if you want more, 
UTSL (see \textbf{http://www.tuxedo.org/\~{}esr/jargon/html/entry/UTSL.html}).

This version of traceroute is used in traceroute.cgi (see the traceroute.cgi section) ,
which isn't required,
just handy on occasion, and in do-traceroute (see the do-traceroute section) , which you don't need unless
you're curious about your routing and how it's changing over time.  The only
extra options that do-traceroute uses are the \texttt{-A} option to look up the
ASN (Autonomous System Number) and the \texttt{-O} option to look up the DNS owner.
%------------------------------------ misc.pod ---

\chapter{Miscellany}
\index{Miscellany}

\section{Miscellaneous Scripts}%
\index{Miscellaneous Scripts}

These are scripts that don't really fit in anywhere.

\begin{itemize}


\item alerter (see the alerter section)  sends an alert

\item availability-report (see the availability-report section)  shows availability of RRD variables

\item genindex (see the genindex section)  makes an index

\item genmenu (see the genmenu section)  makes the vertical menu-bars used in these docs.

\item htmlpod (see the htmlpod section)  makes pod files from html files (roughly).

\item lockfile (see the lockfile section)  makes lock-files

\item podhtml (see the podhtml section)  makes html files from pod files.

\item podlatex (see the podlatex section)  makes LaTeX files from pod files.

\item podpdf (see the podpdf section)  makes PDF files from pod files.

\item remstast-backup (see the remstast-backup section)  makes a backup of your data and configuration.

\item rrd-report (see the rrd-report section)  produces reports from a raw rrd.

\end{itemize}
These are release-related scripts:

\begin{itemize}


\item convert-config-links (see the convert-config-links section)  - copies links to files (just read it)

\end{itemize}%------------------------------------ alerter.pod ---

\section{alerter - construct and send alert text}%
\index{alerter - construct and send alert text}

\subsection{Usage: }%
\index{Usage: }

\begin{verbatim}%
alerter version 1.14 from remstats 1.0.13b
usage: alerter [options] args
where options are:
    -d ddd  set debugging output to level 'ddd'
    -h      show this help
    -f fff  use 'fff' as configuration directory [/home/remstats/etc/config]
The args are documented in alert-monitor; a quick list:
    towho host ip realrrd wildpart var status old_status value relation 
    threshold alertstart duration hostdesc rrddesc webmaster template
\end{verbatim}

\subsection{Description: }%
\index{Description: }

[\texttt{Alerter} may be rolled into the \texttt{alert-monitor} at some point in
the future.  It was easier to test as a separate program, and the performance
hasn't been an issue for me.]

\texttt{Alerter} is passed its parameters (specified above) by the alert-monitor (see the alert-monitor section) .
Most of them are used to fill in information in the text of the alert.  The
interesting ones are \texttt{towho} and \texttt{template}.

It also reads the alert-destination-map config-file (see the alert-destination-map config-file section) 
to decide where the alert needs to go.  This will give it a list of (method, address)
pairs.

For a given template-name, say \texttt{xxx}, and method, say \texttt{method}, it will 
look for files in /home/remstats/etc/config/alert-templates, called:

\begin{verbatim}%
	method-xxx
	method-DEFAULT
	xxx
	DEFAULT\end{verbatim}

and take the first one it finds.  Similarly, it will look for a header to add to the
top of the template called:

\begin{verbatim}%
	method-HEADER
	HEADER\end{verbatim}

and a footer in one of:

\begin{verbatim}%
	method-FOOTER
	FOOTER\end{verbatim}

The three pieces will be concatenated giving the template text.  Then substitutions
will be done for the following magic cookies (see the magic cookies section) :

\begin{verbatim}%
	HOST IP REALRRD WILDPART FIXEDRRD VAR STATUS OLDSTATUS
	VALUE RELATION THRESHOLD START DURATION HOSTDESC RRDDESC 
	NOW TEXTNOW ALERTHOST TOWHO WEBMASTER HTMLURL CGIURL HOSTINDEXURL\end{verbatim}

This gives the alert text.  From the method definition in the
alert-destination-map config-file (see the alert-destination-map config-file section) 
\texttt{alerter} knows which program to run to send the alert text to the
appropriate address, and it does it.

\subsection{Alert-Sending Scripts}%
\index{Alert-Sending Scripts}

These are now easy to write, and in many cases you won't even have to
write one.  There are two requirements for an alert-sending script:

\begin{enumerate}


\item  It must take an address to send to on the command-line, and

\item  It must accept the text on stdin.

\end{enumerate}
E.G. you could use sendmail with no wrapper.
%------------------------------------ availability-report.pod ---

\section{availability-report }%
\index{availability-report }

\subsection{Usage: }%
\index{Usage: }

\begin{verbatim}%
availability-report version 1.27 from remstats 1.0.13b
usage: availability-report [options]
where options are:
    -c      use colors in the output
    -d ddd  set debugging output to level 'ddd'
    -f fff  set config-dir to 'fff' [/home/remstats/etc/config]
    -h      show this help
    -H HHH  show only hosts HHH (comma-separated list) [all]
    -G GGG  show only groups GGG (comma-separated list) [all]
    -R RRR  show only rrds RRR (comma-separated list) [all]
    -g      show group summary
    -t ttt  availability for time-period ttt (start,finish)
\end{verbatim}

\subsection{Description: }%
\index{Description: }

This is mainly intended to be called from 
availability-report.cgi (see the availability-report.cgi section) .  It provided
a report on "availability" of specified RRD variables, by default, all
that have definitions in the availability (see the availability section) 
config-file.  Exactly what it means for a variable to be "available"
is up to you.  It's intended to give some measure of when a host or
service isn't useable, so, e.g. the default definition of availability
for the \texttt{ping} RRD variable \texttt{rcvd} (number of ping responses received)
is:

\begin{verbatim}%
	ping rcvd MINIMUM $>$ 0\end{verbatim}

In english, the \texttt{rcvd} variable is considered unavailable if:

\begin{verbatim}%
	- it is less than or equal to zero (I.E. it didn't respond to ping)
	- there is no data available for that time period\end{verbatim}

\textbf{N.B.:} The interaction between rrd archive consolidation and the xff value,
(see rrdcreate), can result in longer periods of unavailability or conversely,
masking periods of unavailability.  Choose the consolidation function carefully
to make sure you're getting the best data possible.
%------------------------------------ convert-config-links.pod ---

\section{convert-config-links }%
\index{convert-config-links }

\subsection{Usage: }%
\index{Usage: }

\begin{verbatim}%
usage: ../convert-config-links [-h]
\end{verbatim}

\subsection{Description: }%
\index{Description: }

The problem is that an upgrade installation of remstats will overwrite 
the config-base directory, but previous installations of remstats 
created new configuration directories as symlinks to config-base.  
Some of these files need to be changed and some are commonly changed, 
specificly:

\begin{verbatim}%
	alerts alert-destination-map general html links tools\end{verbatim}

Installing a new version of remstats will overwrite config-base, 
including these files.

\texttt{convert-config-links} is a conversion tool for upgrading from 
remstats versions before 1.0A.  It will convert the commonly 
changed config-files from symlinks to copies of the appropriate 
files from config-base.  In remstats versions after 1.0A the 
new-config (see the new-config section)  program will "Do the Right Thing" (TM) and make 
copies by itself, so you'll only have to run this once.

If you're installing remstats for the first time, you can ignore 
this program.
%------------------------------------ files.pod ---

\section{Remstats Files}%
\index{Remstats Files}

This is a reference to the various kinds of files remstats uses and
creates.

The simple ones really don't need explanation, and are included here
for completeness:

\begin{itemize}


\item \texttt{/home/remstats/bin} - contains executables provided with remstats

\item \texttt{/home/remstats/lib} - contains "libraries" required at run-time by the
executables.  So far, it's only \texttt{.pl} files.

\item \texttt{/home/remstats/etc} - contains configuration information.  Immediately after
a new installation, it will only contain a single sub-directory - 
\texttt{config-base}.  new-config (see the new-config section)  will create configuration directories 
under \texttt{/home/remstats/etc} containing a mix of symlinks to files and sub-directories
of \texttt{config-base} and copies of those files and sub-directories, according
to whether an installation is likely to want to change them.

\item \texttt{/home/remstats/html} - (\textbf{FIXME} this will have to be re-written for 
page-maker)
contains the created web-pages to display status and
graph information.  At the top level, there are the index pages and a 
sub-directory for each host, named with that host's name.  The host 
sub-directories contain all the index pages and graph images for that host.
In addition, there are a few special files and sub-directories
of \texttt{/home/remstats/html}:
\begin{itemize}


\item \texttt{MOTD.html} - The graph-writer (see the graph-writer section)  will create this file if it doesn't
exist, but no remstats programs will place anything there.  It's purpose it
to contain site-specific information of importance, similar to the
unix \texttt{/etc/motd} file.

\item \texttt{IMAGES} - This contains remstats-supplied static images for 
various things, mostly logos.

\item \texttt{CUSTOM} - This subdirectory contains the web-pages and images
associated with customgraphs (see the customgraphs section) , graphs of
information which don't relate to a single host.  There is a sub-directory
for each customgraph, to hold its index pages and images and a top-level
index page.

\item \texttt{VIEWS} - This directory contains view information created by the
view-writer (see the view-writer section) .  There will be an overall index page and a sub-directory
for each view to conain its index page and images.

\item \texttt{MOVIES} - This directory is used by collect-movie-image (see the collect-movie-image section)  to 
take snapshots of the netflow dataimage (see the dataimage section) , to be assembled
into a "movie".  There will be files there called \texttt{snap-DDHHMM.png}.
This will not be usefull to you unless you write your own netflow datapage.

\end{itemize}

\item \texttt{backup} - This directory is used by remstats-backup (see the remstats-backup section)  to contain
two tar-files: \texttt{data.tar.gz} and $<$config.tar.gz$>$.

\item \texttt{data} - This directory contains the data collected and maintained
by remstats.  There is a sub-directory for each host and a few other
special-purpose files and sub-directories:
\begin{itemize}


\item \texttt{ALERTS} - This file contains the alert status of all variables
which have an alert associated with them, for all RRDs and hosts.  It is
maintained by the alert-monitor (see the alert-monitor section)  and will be re-created if it is missing.

\item \texttt{IP\_CACHE} - This is a cached mapping between IP numbers and host-names
created by the routine \texttt{get\_ip} via \texttt{read\_ip\_cache} and \texttt{write\_ip\_cache}.
In normall use of remstats, it will be created by ping-collector and used by 
subsequent programs invoked during a run of run-remstats (see the run-remstats section) .  It exists
to reduce DNS queries and to provide consistency during a run-remstats run.

\item \texttt{LAST} - This directory contains a file for each collector, named
for the collector, containing the data collected on it's last invocation.
It's mainly intended for debugging, but could be used for other purposes.
It also contains the file \texttt{CONFIGCHANGE} which tracks when the configuration
directory tree last changed.

\item \texttt{LOGS} - This directory contains the event-logs, updated by various
programs which call the routine \texttt{logit}, and displayed by 
showlog.cgi (see the showlog.cgi section) .

\item \texttt{NT} - This directory contains information collected by nt-discover (see the nt-discover section) ,
specifically: nthostinfo (which contains SRVINFO data from the
nt-status-collector (see the nt-status-collector section) ), nthosts (which contains a list of discovered NT 
hosts), and ntusers (which contains login information about NT users).

\item \texttt{TRACEROUTES} - This contains information collected by 
do-traceroutes (see the do-traceroutes section) .  Each file (PATH-yyyymmdd-hhmmss) within it is the 
output of make-path (see the make-path section)  run on the output of traceroute (see the traceroute section)  for all the 
hosts.  There are also two other files: \texttt{PATHS} contains the latest 
available path for each host, and \texttt{NAME} contains the name of the latest 
PATH-* file.

\end{itemize}

\item \texttt{tmp} - Temporary files not needed over the long term.  There are
lock-files (\texttt{LOCK-programname}) and files used to create the lock-files
(\texttt{LOCK-programname.pid}).  There are logs from the current and previous
runs of run-remstats (\texttt{LOG-run-remstats} and \texttt{LOG-run-remstats.old}).  
\texttt{STATUS-run-remstats} is updated by run-remstats to show its status while
running, to be displayed by remstats-monitor (see the remstats-monitor section) .  Nt-discover creates
some files here prefixed with its name.  The most important file here is
\texttt{uphosts}, created by the ping-collector to contain a list of hosts which
were responding to a ping on this run.  This is used by the various
collectors unless they are invoked without the "\texttt{-u}" flag, to allow
them to skip querying hosts which aren't up.

\end{itemize}%------------------------------------ functions.pod ---

\section{Functions }%
\index{Functions }

FIXME

There are a number of remstats-supplied functions which may be useful in
munging collected data for user-defined RRDs.  The \texttt{data} directive in an
RRD definition allows you to specify a function like:

\begin{verbatim}%
	data myvar=\&myfunc(collector-var-name) ...\end{verbatim}

The functions which might be of use are:

\begin{itemize}


\item siunits - converts large numbers to short strings.  E.G. "\texttt{1200000}" 
would be rendered as "\texttt{1.2M}".

\item timestamp - converts a unix timestamp (seconds since Jan 1 1970) into
a more comprehensible form "\texttt{YYYY-MM-DD hh:mm:ss}".

\item timestamp2 - converts a unix timestamp (seconds since Jan 1 1970) into
a different, more comprehensible form "\texttt{YYYYMMDD-hhmmss}".

\item si\_to\_number - converts SI units (as produced by \texttt{siunits} above
to a bare number.

\item m\_or\_km\_to\_km - converts a number of meters or kilometers to kilometers.
Specifically, "\texttt{1.2km}" is converted to "\texttt{1.2}", but "\texttt{2500m}" is
converted to "\texttt{2.5}".

\item sec\_to\_dhms - converts a number of seconds into a string like
"\texttt{2d12:10:14}", meaning "2 days 12 hours 10 minutes and 14 seconds".

\item to\_filename - the remstats file-name numger, it converts a string to
one containing no characters which will give problems in file-names, by
removing such characters.

\item cisco\_modem\_protocol - converts protocol names from a Cisco Access
Server to numbers.

\item cisco\_modem\_modulation - converts modulation names from a Cisco Access
Server to numbers.

\item cisco\_modem\_state - converts modem state names from a Cisco Access
Server to numbers.

\item apcups\_battery\_status - converts battery status numbers from an
APCC UPS (from SNMP) to a string indicating the status.

\item snmpiftype - convert an SNMP interface type to a short name.

\item snmpifstatus - convert an SNMP interface status to a short string.

\item to\_ifname - munge an interface name to lower-case and remove all
characters except letters, digits, colon (:), underscore (\_), hyphen (-)
and period (.).  This is not intended to make a safe file-name (note the
colon), but rather to get names that can be dealt with more easily.

\end{itemize}
If you want to add your own functions, use the private.pl (see the private.pl section)  file.
%------------------------------------ genindex.pod ---

\section{genindex - make an index from output of podhtml}%
\index{genindex - make an index from output of podhtml}

\subsection{Usage: }%
\index{Usage: }

\begin{verbatim}%
genindex version 1.7 from remstats 1.0.13b
usage: genindex [options] file ...
where options are:
	-d	enable debugging output
	-f fff  use 'fff' format for output (html, pod or text)[pod]
	-h	show this help
\end{verbatim}

\subsection{Description: }%
\index{Description: }

\texttt{genindex} reads the index output of podhtml (see the podhtml section)  and builds a crude index.
It's mainly for using your browser's \texttt{find} command on and it's not pretty.
Show me something simple and better and I'll use it.
%------------------------------------ genmenu.pod ---

\section{genmenu - generate a collapsing menu}%
\index{genmenu - generate a collapsing menu}

\subsection{Usage: }%
\index{Usage: }

\begin{verbatim}%
genmenu version 1.7 from remstats 1.0.13b
usage: ../genmenu [options] pagename menufile
where options are:
	-d	enable debugging output
	-h	show this help
\end{verbatim}

\subsection{Description: }%
\index{Description: }

\texttt{genmenu} reads the menu description file and generates a vertical
menu-bar, collapsed according to which pagename you gave it.  This
requires all the documentation to be rebuilt whenever you change the
menu definition, but avoids having to use JavaScript.  

I couldn't find a simple stand-alone program that did this, so here you are.
It doesn't require remstats.

\subsection{The Menu Definition File}%
\index{The Menu Definition File}

The file has a simple format.  Blank lines and lines beginning with '\#'
are ignored.  The other lines look like:

\begin{verbatim}%
	[tabs]pagename [page title]\end{verbatim}

The number of \texttt{tabs} shows the level of sub-menus, making the definition
file easy to grasp at a glance.  Note that the \texttt{tabs} are actual tab characters.
The \texttt{page title} (optional) is what shows up in the menu, while the 
\texttt{pagename} is used to make the URL to link to.  If the page name is 
\texttt{xyz}, then a link to \texttt{xyz.html} is produced.  If the \texttt{page title} is missing,
then the \texttt{pagename} is used instead.
%------------------------------------ htmlpod.pod ---

\section{podhtml - convert HTML to POD}%
\index{podhtml - convert HTML to POD}

\subsection{Usage: }%
\index{Usage: }

\begin{verbatim}%
	htmlpod htmlfile $>$podfile\end{verbatim}

\subsection{Description: }%
\index{Description: }

Quick and Dirty.

I only wrote it to do the first cut for converting my old html-based
documentation to pod format.  It's by no means complete or even
correct.  However, it did convert over 90\% of the HTML markup that
I was using.

Use it if you want, but don't complain about it without providing a
patch to fix your complaint.
%------------------------------------ lockfile.pod ---

\section{lockfile }%
\index{lockfile }

\subsection{Usage: }%
\index{Usage: }

\begin{verbatim}%
lockfile version 1.6 from remstats 1.0.13b
usage: ../lockfile [options]
where options are:
  -b bbb  break locks that are more than 'bbb' seconds old [-1]
  -d nnn  enable debugging output at level 'nnn'
  -f fff  use 'fff' for config-dir [/home/remstats/etc/config]
  -F      remove the lock-file and try to lock it (force it)
  -h      show this help
  -r rrr  retry the lock 'rrr' times (always at least once) [0]
  -s sss  sleep for 'sss' seconds between tries [10]
  -u      remove the lock-file and exit
\end{verbatim}

\subsection{Description: }%
\index{Description: }

This makes remstats generic \texttt{make\_lockfile} function available to shell
scripts so that external things (like backups) can lock the same way
remstats does internally, for the few things which need locking.

If the specified lockfile has no '/' in it, then it will be augmented to

\begin{verbatim}%
	$<$tempdir$>$/LOCK-$<$whatever$>$\end{verbatim}

where $<$tempdir$>$ is the remstats configuration-supplied tempdir and $<$whatever$>$
is what you specified as a lock-file.  The remstats programs which use
the \texttt{make\_lockfile} routine, which is the heart of \texttt{lockfile}, all use
their name as the lock-file.
%------------------------------------ podhtml.pod ---

\section{podhtml - translate a POD file to HTML}%
\index{podhtml - translate a POD file to HTML}

\subsection{Usage: }%
\index{Usage: }

\begin{verbatim}%
podhtml version 1.9 from remstats 1.0.13b
usage: podhtml [options] podfile
where options are:
	-d ddd  enable debugging output
	-h      show this help
	-s sss  use 'sss' as the suffix for html files [.html]
	-u uuu  use 'uuu' as a URL prefix
\end{verbatim}

\subsection{Description: }%
\index{Description: }

See the docs for \texttt{pod2html}.  The only changes that I made
intentionally from how \texttt{pod2html} does things are:

\begin{itemize}


\item if a line looks blank it's treated as blank.  I prefer
to avoid surprises.

\item I added a new \texttt{=exec} which executes a command line and
inserts the output of stdout into the resulting HTML as a $<$PRE$>$
section.  This was so that I could get the latest usage message
from programs inserted without having to run each program separately,
save its output in a file, and manually insert the file into 
the POD file.

\item I also caused it to append to a file called \texttt{podhtml--rawindex}
for each =head1 and =head2, a URL for that page and section and the
contents of that =headN.  This is used by genindex (see the genindex section)  to make an index.

\end{itemize}
I wrote this version in frustration with the way pod2html does
links.  Or doesn't.  I could never tell without trying whether
it would generate a link or not.
%------------------------------------ podlatex.pod ---

\section{podlatex - translate a POD file to LaTeX}%
\index{podlatex - translate a POD file to LaTeX}

\subsection{Usage: }%
\index{Usage: }

\begin{verbatim}%
podlatex version 1.6 from remstats 1.0.13b
usage: podlatex [options] podfile
where options are:
	-d ddd  enable debugging output
	-h      show this help
	-s sss  use 'sss' as the suffix for html files [.html]
	-u uuu  use 'uuu' as a URL prefix
\end{verbatim}

\subsection{Description: }%
\index{Description: }

See the docs for \texttt{pod2latex}.  The only changes that I made
intentionally from how \texttt{pod2latex} does things are:

\begin{itemize}


\item if a line looks blank it's treated as blank.  I prefer
to avoid surprises.

\item I added a new \texttt{=exec} which executes a command line and
inserts the output of stdout into the resulting HTML as a $<$PRE$>$
section.  This was so that I could get the latest usage message
from programs inserted without having to run each program separately,
save its output in a file, and manually insert the file into 
the POD file.

\item I changed the text wrapping links.

\end{itemize}
I wrote this version in frustration with the way pod2latex does
links.
%------------------------------------ podpdf.pod ---

\section{podpdf - translate a POD file to pdf}%
\index{podpdf - translate a POD file to pdf}

\subsection{Usage: }%
\index{Usage: }

\begin{verbatim}%

	../podpdf 
	[ --help --verbose <1|2 --paper <usletter> --podfile <file> ] <file>
	
	--help
	    displays this explanation of correct usage
	    
	--vebose <1|2>
	    regulates the volume of progress comments: argument must be 1 or 2
	    
	--podfile <file>
	    supplies the input file to process as an explicit parameter. The
	    input file may also be supplied from STDIN or from the command
	    line as the array element --paper.
	    
	Further information can be found in the POD section of Pod.pm. Enter:
	    perl -e "use Pod::Pdf; pod2pdf('<your_library_path>/Pod/Pdf.pm')"
	to get the POD in PDF format :)

	\end{verbatim}

\subsection{Description: }%
\index{Description: }

See the docs for \texttt{Pod::PDF}.  This is only a tiny wrapper for it.
%------------------------------------ remstats-backup.pod ---

\section{remstats-backup }%
\index{remstats-backup }

\subsection{Usage: }%
\index{Usage: }

remstats-backup

\subsection{Description: }%
\index{Description: }

It locks out (see the locks out section)  run-remstats and backs up the data sub-tree 
(/home/remstats/data), unlocks it and then backs up the config 
sub-tree (/home/remstats/etc/config).  It simply runs tar on the 
sub-trees and puts the resultant tarballs in
/home/remstats/backup as data.tar.gz and config.tar.gz.
%------------------------------------ remstats-cleanup.pod ---

\section{remstats-cleanup - removes stale, old files}%
\index{remstats-cleanup - removes stale, old files}

\subsection{Usage: }%
\index{Usage: }

\begin{verbatim}%
remstats-cleanup version 1.2 from remstats 1.0.13b
usage: ../remstats-cleanup [options]
where options are:
    -d nnn  enable debugging output at level 'nnn'
    -f fff  use 'fff' for config-dir [/home/remstats/etc/config]
    -h      show this help
\end{verbatim}

\subsection{Description: }%
\index{Description: }

It removes old collector data from /home/remstats/data/LAST, old logs from 
/home/remstats/data/LOGS, old traceroute data from /home/remstats/data/TRACEROUTES
and old images from all the host subdirectories of /home/remstats/html.

Run it out of cron every now and then, say once a day, with a line
like:

\begin{verbatim}%
	0 2 * * * /home/remstats/bin/remstats-cleanup\end{verbatim}
%------------------------------------ remstats-version.pod ---

\section{remstats-version }%
\index{remstats-version }

\subsection{Usage: }%
\index{Usage: }

remstats-version

\subsection{Description: }%
\index{Description: }

It just shows the version of the installed remstats.  (Which should be
1.0.13b for this installation.)
%------------------------------------ rename-host.pod ---

\section{rename-host }%
\index{rename-host }

\subsection{Usage: }%
\index{Usage: }

\begin{verbatim}%
rename-host version 1.1 from remstats 1.0.13b
usage: ../rename-host [options]
where options are:
  -d nnn  enable debugging output at level 'nnn'
  -h      show this help
  -i iii  change the IP number in the new config-file to 'iii'
\end{verbatim}

\subsection{Description: }%
\index{Description: }

Does what's necessary to rename a monitored host, specifically:

\begin{itemize}


\item  renames the host config-file to \texttt{oldname\~{}}, disabling it

\item  renames the host's data directory to \texttt{newname}

\item  renames the host's html directories to \texttt{newname} ( both generated 
CGI pages and images directories)

\item  copies the host config-file to \texttt{newname\~{}}, optionally changing
the IP number of the host

\item  renames the host configfile to \texttt{newname}

\end{itemize}%------------------------------------ rrd-report.pod ---

\section{rrd-report - display summaries of an RRD file}%
\index{rrd-report - display summaries of an RRD file}

\subsection{Usage: }%
\index{Usage: }

\begin{verbatim}%
rrd-report version 1.11 from remstats 1.0.13b
usage: rrd-report [options] rrd-file
where options are:
	-b bbb	begin at time 'bbb' (see Note 3)
	-c ccc  select data from consolidation-function 'ccc' [AVERAGE]
	-d ddd  enable debugging output at level 'ddd'
	-D DDD  show the dates as 'DDD' [both,pretty]
			(none|both|start|finish),(raw|simple|pretty)
	-e eee  end at time 'eee' (see Note 3)
	-f fff  use report format 'fff' [simple]
			(from 'simple', 'label', 'html')
	-h      show this help
	-i iii  report on intervals 'iii' (see Note 2) [1d]
	-l      list the DS names in this rrd, no report
	-n nnn  use 'nnn' as the format to print the numbers [%lf]
	-s sss  summary on interval 'sss' (see Note 2) [1w]
	-v vvv  show variables 'vvv' (var:cf comma-separated) [ALL]
Note: if report interval (-i) and summary interval (-s) are equal,
  no summary reporting is done.
Note 2: intervals are numbers of seconds, minutes, hours, days, weeks, Months
  or years.  E.G. "4w" for 4 weeks, "1M" for one month.
Note 3: Begin and End times are a unix timestamp (seconds since Jan 1, 1970) or
  plus-or-minus an interval, as in Note 2.  E.G. "-1w" means "one week ago".
\end{verbatim}

\subsection{Examples: }%
\index{Examples: }

I hope that the above is enough to use it after seeing a few examples.
Here's the equivalent of the command that created the RRD for the example.

rrdtool create ping.rrd $\backslash$
	DS:sent:GAUGE:600:0:10 $\backslash$
	DS:rcvd:GAUGE:600:0:10 $\backslash$
	DS:min:GAUGE:600:U:U $\backslash$
	DS:avg:GAUGE:600:U:U $\backslash$
	DS:max:GAUGE:600:U:U $\backslash$
	RRA:AVERAGE:0.1:1:600 $\backslash$
	RRA:AVERAGE:0.1:7:300 $\backslash$
	RRA:AVERAGE:0.1:30:300 $\backslash$
	RRA:AVERAGE:0.1:90:300 $\backslash$
	RRA:AVERAGE:0.1:365:300 $\backslash$
	RRA:MIN:0.1:1:600 $\backslash$
	RRA:MIN:0.1:7:300 $\backslash$
	RRA:MIN:0.1:30:300 $\backslash$
	RRA:MIN:0.1:90:300 $\backslash$
	RRA:MIN:0.1:365:300 $\backslash$
	RRA:MAX:0.1:1:600 $\backslash$
	RRA:MAX:0.1:7:300 $\backslash$
	RRA:MAX:0.1:30:300 $\backslash$
	RRA:MAX:0.1:90:300 $\backslash$
	RRA:MAX:0.1:365:300

See "man rrdcreate" for an explanation for the command itself.  The
fields are:

\begin{itemize}


\item sent/rcvd - number of ping packets sent/received

\item min/avg/max - the round-trip-time (min, average and max) for the pings

\end{itemize}
Here's a default report from one of my ping RRDs:

\begin{verbatim}%
	\%rrd-report ping.rrd
	[snip]
	data 1999-10-25 17:20:49 1999-10-26 17:20:49 10.000000 10.000000 10.000000 9.864444 9.899444 9.934444 41.569556 42.210222 42.850889 45.626889 45.758833 45.890778 50.955111 51.253056 51.551000
	data 1999-10-26 17:20:49 1999-10-27 17:20:49 10.000000 10.000000 10.000000 9.934444 9.987819 10.000000 39.536556 41.498103 46.124938 42.166222 45.386889 49.370000 50.955111 52.435532 54.635926
	summary 1999-10-19 17:20:49 1999-10-26 17:20:49 10.000000 10.000000 10.000000 9.331111 9.932391 10.000000 38.317778 42.750146 48.318444 41.500556 46.736223 50.265778 49.122444 52.261605 59.867778
	[snip]
	data 1999-11-17 16:20:49 1999-11-18 16:20:49 10.000000 10.000000 10.000000 8.000000 9.934245 10.000000 36.400000 46.585421 50.000000 41.256667 49.592941 58.640000 49.036667 53.837716 117.240000
	summary 1999-11-16 16:20:49 1999-11-18 16:20:49 10.000000 10.000000 10.000000 9.142857 9.929788 10.000000 38.285714 46.482773 50.000000 47.148095 49.503659 51.294286 50.000000 53.124615 65.880952
	overall 1999-10-19 17:20:49 1999-11-18 16:20:49 9.978889 9.999918 10.000000 1.323333 9.876767 10.000000 6.194333 45.631272 76.394778 6.516556 48.842746 107.286556 6.971111 54.319770 179.554222\end{verbatim}

Each "data" line is a report for the interval covered by the two 
timestamps, (by default one day).  The values are the requested 
(or in this case all) DS:CF combinations. The "summary" lines are 
just reports over a longer interval (by default one week).
The "overall" line is for the whole selected time-period.

Hmm.  There's much too much there.  What I'd really like to see is 
just the interesting stuff.  I know how many pings I'm sending 
during this period (10), so drop that and just show the minimum min
average avg and maximum max:

\begin{verbatim}%
	\% rrd-report -v rcvd:AVERAGE,min:MIN,avg:AVERAGE,max:MAX
	data 1999-10-19 17:54:57 1999-10-20 17:54:57 9.820267 38.317778 43.948411 55.716667
	data 1999-10-20 17:54:57 1999-10-21 17:54:57 9.966716 39.303333 46.180111 59.867778
	data 1999-10-21 17:54:57 1999-10-22 17:54:57 9.907440 40.469000 48.496274 56.022222
	data 1999-10-22 17:54:57 1999-10-23 17:54:57 9.977827 40.232333 47.571133 54.475062
	[snip]
	summary 1999-11-09 16:54:57 1999-11-16 16:54:57 9.950836 39.310056 52.578943 179.554222
	data 1999-11-17 16:54:57 1999-11-18 16:54:57 9.934164 36.400000 49.606736 117.240000
	summary 1999-11-16 16:54:57 1999-11-18 16:54:57 9.928672 38.285714 49.489729 65.880952
	overall 1999-10-19 17:54:57 1999-11-18 16:54:57 9.876767 6.194333 48.842746 179.554222\end{verbatim}

Well, I can figure out when the period ended, so leave out the end-time, and
I don't like seeing all those meaningless (in this case) decimal places, so
how about:

\begin{verbatim}%
	\% rrd-report -D start,pretty -n \%.1lf -v rcvd:AVERAGE,min:MIN,avg:AVERAGE,max:MAX
	[snip]
	data 1999-11-14 17:27:04 10.0 40.0 49.4 88.7
	data 1999-11-15 17:27:04 9.7 21.7 48.0 63.4
	data 1999-11-16 17:27:04 9.9 38.3 49.4 61.3
	summary 1999-11-09 17:27:04 10.0 39.3 52.6 179.6
	data 1999-11-17 17:27:04 9.9 36.4 49.6 117.2
	summary 1999-11-16 17:27:04 9.9 38.3 49.5 65.9
	overall 1999-10-19 18:27:04 9.9 6.2 48.8 179.6\end{verbatim}

OK.  I'd like to see the last year with a one-week interval, with no summaries.
(Setting the report-interval to the same as the summary-interval 
drops summaries.  You still get an overall line.)

\begin{verbatim}%
	\% rrd-report -D start,pretty -n \%.1lf -v rcvd:AVERAGE,min:MIN,avg:AVERAGE,ma
	x:MAX -i 1w -s 1w
	data 1998-11-19 09:04:43 NODATA NODATA NODATA NODATA
	[snip]
	data 1999-02-25 09:04:43 9.9 45.0 55.0 64.2
	data 1999-03-04 09:04:43 10.0 43.9 54.5 64.3
	[snip]
	data 1999-11-11 09:04:43 10.0 39.3 51.7 179.6
	data 1999-11-18 09:04:43 9.9 37.4 49.7 169.7
	overall 1998-11-19 09:04:43 9.6 0.0 45.3 103.7\end{verbatim}

And for those of us who like to see it on the web:
[You'll just have to look in the web version of this documentation
to see what it looks like.]
%------------------------------------ rt-updater.pod ---

\section{rt-updater }%
\index{rt-updater }

\subsection{Usage: }%
\index{Usage: }

\begin{verbatim}%
rt-updater version 1.1 from remstats 1.0.13b
usage: ../rt-updater [options] host rrd graph temp-rrdfile rrdcgi-file
where options are:
  -a aaa  keep 'aaa' samples in the archive [720]
  -A      auto-set step from max collector test run-time
  -c ccc  use consolidation functions 'ccc' (comma-separated list) in
          creating the RRAs [AVERAGE,MIN,MAX]
  -d nnn  enable debugging output at level 'nnn'
  -e      use existing rrd file
  -f fff  use 'fff' for config-dir [/home/remstats/etc/config]
  -F      run forever; usually runs until the archive is full
  -h      show this help
  -r rrr  make the rrdcgi page refresh every 'rrr' seconds [step]
  -s sss  update step time of 'sss' seconds [5]
  -t ttt  how many times to test the collector [3]
  -u      unlink rrd-file, rrdcgi-file and image files after run

Note: 'temp-rrdfile' must be a full path to the file or the generated
'rrdcgi-file' will not work.
\end{verbatim}

\subsection{Description: }%
\index{Description: }

[This is not well integrated with the rest of remstats.  If I figure out a
good way to do it I will, but so far ...]

This program creates an RRD file, using the specified remstats RRD definition,
but with a different set of RRAs and step time.  It creates a command pipeline
to run the required (RRD-specified) collector (see the collector section)  into the
updater (see the updater section) .  It then runs this a number of times, to make sure that it can
obtain data fast enough to meet the user-specified step time.  It writes
an \texttt{rrdcgi} web-page to show the user-specified graph from that RRD.  Finally,
it loops, running the collector-to-updater pipeline.  This process will run
until it fills the archive, or forever, if requested.

The actual image files are stored in \texttt{/home/remstats/html/GRAPHS/TMP}, which should
be created as writable by both the \texttt{@@USER@@} user and the \texttt{apache}
group, and set-group-id to the \texttt{apache} group.  (This should be
done automatically as part of the install and running the page-writer (see the page-writer section) .)
This will allow the rrdcgi web-page to write the image files there while
running as the web-server's user.

This is intended for monitoring a few things during testing of something, to
create very fine-grained data not to replace remstats usual processes, as it's
much more CPU intensive.
%------------------------------------ thanks.pod ---

\chapter{Thank-you's}
\index{Thank-you's}

\section{Thank-you's }%
\index{Thank-you's }

\begin{verbatim}%
	Tobias Oetiker 
	Larry Wall and the rest of the perl hackers 
	Vikas Aggarwal
	Ehud Gavron
	Will Maton
	Adam Kennedy
	Ken Filipps
	Andrew Cochran
	Marek Snowarski
	Matt Duggan
	Steve Francis
	Alexander Reelsen
	Jon Villarreal 
	Robert Jordens
	Vincent Forgues
	Anik Richard
	Arnauld Michelizza\end{verbatim}

I've probably forgotten some others.  Please remind me if I've forgotten 
your contribution.

\printindex{default}
\end{document}
