Figure 1. An example of Cricket in action. These graphs show an OC3 outage.
Jeff R. Allen
WebTV Networks, Inc.
Note: This paper was first published in the proceedings of the 1st Conference on Network Administration, which was held April 6-10, 1999. This paper is copyright 1999 by Jeff Allen. Redistribution of this paper is allowed under the GPL.
Cricket is a tool that lets users visualize a set of measurements over time. It was designed to assist network administrators by letting them see and respond to patterns in their network. In this paper, I will describe the need we saw and attempted to resolve by writing Cricket, then describe the solution we came up with. Finally, I will describe some future work we expect to do to make Cricket a more proactive monitoring tool.
When running a complicated network, there are a multitude of things one needs to keep an eye on. Clearly, immediate concerns like connectivity, link status, and routing stability are top in most administrator's minds. Long-term issues like architecture and technology decisions float in the back of our minds. Often, that leaves no room for the most interesting questions, which help with both short term issues and long term ones. Some of the questions we found ourselves asking about various network components were:
Taking the time to ask these questions allows us to step up a level in our monitoring, and move from a reactive realm (i.e. Is the link to Europe up?) to a more contemplative and predictive realm (i.e. Are we seeing an unexpected burst of traffic to Europe?) It's obviously not possible to definitively say that this makes our job easier and our network run more smoothly, but it certainly seems that way. Simply having the data available to be able to say ``things look OK'' gives us a peace of mind that simply reacting to the network would never give us.
Figure 1 shows a screenshot from Cricket, showing the normal way that data is displayed about a target. In this case, we are looking at the traffic on one of our OC3's, which was operating normally until an outage started a little after noon. From this same view, we can tell that in the last week, the peak bandwidth in use on this link was 60 Mb/sec, on Wednesday evening. By glancing at one page, we can answer many of the questions posed above. By looking at longer time scales (accessed via the links in the upper right) we could answer still other questions about the long-term behavior of the traffic on this link.
Figure 1. An example of Cricket in action. These graphs show an OC3 outage.
One of the reasons it's hard to monitor a network is that there are a lot of different components, each with different operating characteristics and monitoring needs. For instance, in our network, some of the components that we use Cricket to monitor are:
Note that we monitor some things (DNS, hosts) that are not directly related to the health of the physical network. In our shop, the ``host folks'' and the ``network folks'' work together very closely. There is no artificial separation between the two. I will focus on monitoring the networking components, however, keep in mind that Cricket is capable of monitoring many types of components.
Just as the components we wish to monitor are varied, so are the ways in which we talk to them. Sometimes, we can simply talk SNMP, then either record the data directly or do a bit of post-processing to derive a rate of some kind. Other times, we fetch data via SNMP, then post-process it in some more complicated way to get a final data point. For instance, to monitor modem bank usage, we fetch the current state of all the modems, and count the number that are off-hook. Sometimes we simply run a shell command on another data-gathering system. For instance, we can use Unix tools like 'wc' or simple Perl scripts to derive a data point from the data already collected by syslogd. Finally, sometimes we want to measure and observe an aggregate of other data sources. In a site with multiple Internet links, it would be interesting, for instance, to plot the total Internet bandwidth across all links.
Around the time we1
were bringing up the WebTV network, we fetched and
installed a tool called
MRTG2
to let us start answering the kinds of questions mentioned above about
our WAN links. In fact, MRTG was in use at WebTV Networks before we
had brought in any commercial network management tools at all.
It was only by using, and coming to depend on the MRTG graphs,
that we were able to take a step back and figure out why they were so
useful to us. After getting ourselves thoroughly addicted to MRTG,
even while attempting to roll out another network management system,
we were forced to ask the question,
``What is it that MRTG does that we can't get elsewhere.''
The answer to that question makes up the bulk of the first section:
MRTG let us manage our network better than reactive systems.
So, there we were, addicted to MRTG, using it in new situations and in
ways it was never designed to be used. MRTG started showing signs
that it would not scale to handle the new jobs we wanted to throw at
it. Something needed to be done, and quickly. A true addict will not
wait patiently for a new drug!
Before we threw out MRTG and started over, we thought it might be a
good idea to figure out what it does right, and make certain that
whatever we got to replace MRTG could at least do those things. Each
view of the data has just enough detail to tell the story, but not
enough to hide the good bits. For instance, the weekly views show
today and this day last week. There is an enforced data density so
that graphs have just enough data to tell an accurate story, but not
enough to overwhelm the viewer. In addition, the auto-scaling feature
consistently produces readable graphs (though there is a bit of room
for improvement on this front). It uses a fixed size database which
grows linearly with the number of monitored devices, not time. It
needs no extra
``cleanup''
work. Because MRTG is web-based, we have a platform independent tool:
all it takes is a web browser to check the status of the network. You
can even use a WebTV®-based Internet Unit to see graphs of
your network on your home television!
Perhaps more important than all that, though, MRTG passes the
``can it solve real problems''
test:
To see how we use graphs to identify and understand real problems,
examine figures 2 and 3. These two graphs are selected snapshots of
the graphs available to us during backbone routing instability on one
of our peer's networks. During this event, it's clear that about 10
Mb/sec of traffic shifted over to the second peer. We were able to use
Cricket to find the nature of the problem, see that the outbound
traffic had failed over to non-optimal, but functional link, and
finally verify that the traffic returned to normal after the event.
Figure 2. Traffic on a Fast Ethernet handoff to a peer of ours.
Their backbone was suffering routing instability at the time.
Figure 3. Traffic on a DS3 to another peer during the
routing instability shown in Figure 2.
Finally, Cricket passes the
``manager test''
with flying colors.
Managers immediately understand the need for a bandwidth
upgrade once they see a graph produced with MRTG. Even more
astoundingly, it even passes the
``executive test''
which is very
important since executives tend to need to sign orders for OC3's and
above. The simple web-based user interface makes it easy to print and
share the graphs with management, making it easier for everyone to do
their job.
Regardless of all these great features, we identified problems we
wanted to address in the next generation system. First, MRTG has a
narrow view of the world: all targets have exactly two data sources
(used by MRTG for bandwidth in and bandwidth out). We wanted to be
able to measure other things about components. MRTG also has severe
performance problems, resulting from the way it reads and writes
data. MRTG was incrementally developed and the performance problem
only became apparent after it was too late to fix it. Usually, it's
not an issue, because MRTG is mostly used in small installations.
However, with all the different ways we were trying to
use MRTG, performance became a problem for us.
There are ways to overcome these performance problems by arranging to
run multiple instances of MRTG in parallel. However, the configuration
needed to run it in parallel is even more complicated than the usual
configuration. For our varied uses, the configuration was already
unwieldy and inefficient. The last thing we needed was to make it even
more complicated!
Some might recommend commercial products as a replacement for MRTG,
since commercial products are supposed to be designed to scale well
and be easy to configure. Our experience is that commercial tools
introduce as many problems as they purport to solve, and often leave
the original problems (scalable performance and configuration)
unsolved. Here's an undoubtedly one-sided list of the problems
inherent in most commercial tools that could have helped us.
First and foremost is cost. It's expensive to make good software. No
one will argue that point. In fact, it has been more expensive than we
expected to develop Cricket. But when buying an expensive network
management tool, things get ugly. It takes too much time, paperwork,
patience, and persuading management, even if you do have the budget
for a commercial tool. Evaluating and rolling out a free software
package can happen in an afternoon, if you put your mind to
it. There's simply no downside risk - install it and if it solves
your problem, keep it. If it doesn't meet your expectations, delete it
and move on. You get what you pay for, but sometimes it's what you are
not getting (support hassles and licensing nightmares) that really
counts.
With commercial tools, database formats are often secret, API's
(usually only available at extra cost) are either a pain to work with,
or are major performance pigs. Customer support varies, but the simple
fact is that even stellar customer support won't do for you what
getting your own two hands dirty in the code will do: give you a deep
understanding of your NMS and the confidence to trust the system and
your ability to operate it. Then, there's the version tango. The NMS
doesn't find out about new router models until 6 months after you've
already deployed them because the router company and the NMS company
are feuding. The trouble-tracking system you are using is only
certified to talk to the next version of the NMS that you cannot use
yet because that version introduces a critical bug that no other
customers are seeing, and thus, will not get repaired until you
threaten to revoke your support license - which, by the way, does not
expire until next December.
Finally, let's be honest: Free Software rules. Who would you rather
trust with the smooth operation of your network, a bunch of folks who
are in the same boat as you, or a giant vendor with 100-page
implementation plans and slick salesmen? The choice is obvious, to me
at least: network managers design better tools to do their
job3.
Sometimes commercial tools are the right tool for the job, but in this
case, it made more sense to go at it on our own. It will make even
more sense for your organization, since WebTV Networks has
done much of the hard
work. All you need to do is download, install and configure
Cricket.
The solution comes as two pieces, a new database and a new
user-visible front-end. By changing out the back-end, we addressed the
flexibility and performance goals. By changing out the front-end, we
improved the user interface, both for configuring the system and
viewing the graphs. The two pieces are developed semi-independently,
one by me (in California) and one by the original author of MRTG,
Tobias Oetiker (in Switzerland). Keeping the two pieces separate makes
it easier to develop them, and it also means the database component
can be reused in other contexts. RRD, the database backend,
is distributed separately from
Cricket4.
We are deeply indebted to Tobias for his hard work on the database
back-end. Without it, Cricket would be nothing. By engineering RRD
correctly from the start, and by improving it's reliability over the
last year, Tobias solved over half our problems with MRTG himself. I
simply had the easy job of passing the right data into his code. All
the seeming magic of Cricket comes from Tobias's RRD code.
The next generation back-end is called RRD, the Round Robin Database.
It takes over exactly the same jobs the back end provided with MRTG
did: storing and rolling up the data and generating graphs from the
stored data. RRD is written in C, and comes in both a Perl module and
a command line version, which can be used interactively or across a
pipe from scripts. It can be used either directly by simple
scripts, or via frontends like Cricket. Other frontends are available
from the RRD website.
RRD achieves high performance by using binary files to minimize I/O
during the common update operation. The
``round robin''
in RRD refers to
the fact that RRD uses circular buffers to minimize I/O. RRD is at
least one, and possibly two orders of magnitude faster than MRTG's
backend, depending on how you measure it. It's truly incredible to
watch it chew through data if you have ever seen MRTG trying to handle
the same job.
RRD is more flexible than MRTG in at least two dimensions. First, it
can take data from an arbitrary number of data sources. MRTG was
limited to two datasources. Originally they were reserved for input
bandwidth and output bandwidth, though they got co-opted for other
measurements by many MRTG hackers. RRD can also keep an arbitrary
number of data arrays, each fed at a different rate. For instance, you
can keep 600 samples taken every 5 minutes alongside 600 samples
taken every 30 minutes. Thus, you can have 5 minute data stretching back
50 hours into the past, and 30 minute data stretching back 12
days. When you draw a graph of this data, you can see recent data in
high resolution, and older data in lower resolution. This is one of
the original features of MRTG that made us so happy. In RRD this
feature is completely configurable.
RRD stores data with higher precision than ever before (float versus
integer) which allows us to measure bigger things (OC3's) and smaller
things (Unix load averages) without relying on goofy scaling hacks, as
we were required to do with MRTG. The data file where this data lives
on disk is still a fixed size over time, and scales with the product
of the number of datasources and number and sizes of the data
arrays. The data storage in use for an entire installation still
scales linearly with the number of targets under observation. At WebTV
Networks, we have Cricket configured to store 6 variables for every
router and switch interface. Each interface requires 174 kilobytes on
disk, which is enough room to store all the data for 600 days. Of
course, your mileage may vary; Cricket is configurable, so you can
choose to keep more or less data, depending on your needs.
RRD still provides the same simple, sparse graphs we grew to love
while using MRTG. There is now much more flexibility at the time the
graph is generated. For instance, we can choose to show some data
sources, but not others. We can choose to scale a data source using an
arbitrary mathematical expression. For instance, we can fetch the
temperature from a router in Celsius, and present it to the user in
Fahrenheit. Since almost all data seems to arrive in exactly the wrong
units, it's quite helpful to have this flexibility. Do you think
about Ethernet capacity in
``megabytes/sec''
The scaling feature
lets us turn that measurement into
``megabits/sec''
It is also possible
to integrate data from multiple sources into a single graph. For
instance, you could make a summary graph showing the sum of the
traffic across all of your Internet links.
Figure 4. A simplified example of a config tree.
Recall that part of the problem was flexibility and performance of the
database, both of which RRD solved. The other part of the problem was
scalability of the configuration. Our solution
is something called the config tree. A config tree is a set of
configuration files organized into a tree. This hierarchical
configuration structure allows us to use inheritance to reduce
repeated information in the configuration. To simplify implementation
of the config tree, the inheritance strictly follows the directory
structure; complicated concepts like multiple inheritance are not
supported. The attributes that are present in a leaf node are the
union of all the attributes in all the nodes on a path leading from
the root of the config tree to the leaf node. Lower nodes can override
higher nodes, so it's possible to make exceptions in certain subtrees,
and do other clever sleight of hand.
It's easier to understand the config tree by looking at an example
(see figure 4). Attributes that all of the system will share are
located at the root of the config tree. For instance, the length of
the polling interval is set there. At the next level, we set
attributes that will be restricted to the current subtree. At this
level, typically you will find the target type. Finally at the lowest
level we set things that will vary on a per-target basis. For
instance, we set the interface name that we are trying to measure
here. By using the power of inheritance, you can avoid repeating the
information at the top of the config tree many times near the bottom
of the config tree. The three level config tree in the example is the
simplest in common use. The one that ships with Cricket in the
sample-config directory works like this. Config trees can and do have
many more levels. At WebTV Networks, for instance, we add levels to
break apart routers in different data centers to make the directory
listings more manageable. There are no built-in limits on the shape of
the config tree, only practical ones.
Figure 5 shows the power of the string expansion feature in
Cricket. When it comes across strings in the config tree with special
markers in them, it expands the strings in much the same way a
scripting language expands variables. The sample config tree that
ships with Cricket uses this feature in several places to dynamically
build strings from settings already available. In the example, the
short and long descriptions for the target are set based on some data
inherited from high in the tree, and other data related to the target
itsbase.
thus, will not get repaired until you
threaten to revoke your support license - which, by the way, does not
expire until next December.
Finally, let's be honest: Free Software rules. Who would you rather
trust with the smooth operation of your network, a bunch of folks who
are in the same boat as you, or a giant vendor with 100-page
implementation plans and slick salesmen? The choice is obvious, to me
at least: network managers design better tools to do their
job3.
Sometimes commercial tools are the right tool for the job, but in this
case, it made more sense to go at it on our own. It will make even
more sense for your organization, since WebTV Networks has
done much of the hard
work. All you need to do is download, install and configure
Cricket.
The solution comes as two pieces, a new database and a new
user-visible front-end. By changing out the back-end, we addressed the
flexibility and performance goals. By changing out the front-end, we
improved the user interface, both for configuring the system and
viewing the graphs. The two pieces are developed semi-independently,
one by me (in California) and one by the original author of MRTG,
Tobias Oetiker (in Switzerland). Keeping the two pieces separate makes
it easier to develop them, and it also means the database component
can be reused in other contexts. RRD, the database backend,
is distributed separately from
Cricket4.
We are deeply indebted to Tobias for his hard work on the database
back-end. Without it, Cricket would be nothing. By engineering RRD
correctly from the start, and by improving it's reliability over the
last year, Tobias solved over half our problems with MRTG himself. I
simply had the easy job of passing the right data into his code. All
the seeming magic of Cricket comes from Tobias's RRD code.
The next generation back-end is called RRD, the Round Robin Database.
It takes over exactly the same jobs the back end provided with MRTG
did: storing and rolling up the data and generating graphs from the
stored data. RRD is written in C, and comes in both a Perl module and
a command line version, which can be used interactively or across a
pipe from scripts. It can be used either directly by simple
scripts, or via frontends like Cricket. Other frontends are available
from the RRD website.
RRD achieves high performance by using binary files to minimize I/O
during the common update operation. The
``round robin''
in RRD refers to
the fact that RRD uses circular buffers to minimize I/O. RRD is at
least one, and possibly two orders of magnitude faster than MRTG's
backend, depending on how you measure it. It's truly incredible to
watch it chew through data if you have ever seen MRTG trying to handle
the same job.
RRD is more flexible than MRTG in at least two dimensions. First, it
can take data from an arbitrary number of data sources. MRTG was
limited to two datasources. Originally they were reserved for input
bandwidth and output bandwidth, though they got co-opted for other
measurements by many MRTG hackers. RRD can also keep an arbitrary
number of data arrays, each fed at a different rate. For instance, you
can keep 600 samples taken every 5 minutes alongside 600 samples
taken every 30 minutes. Thus, you can have 5 minute data stretching back
50 hours into the past, and 30 minute data stretching back 12
days. When you draw a graph of this data, you can see recent data in
high resolution, and older data in lower resolution. This is one of
the original features of MRTG that made us so happy. In RRD this
feature is completely configurable.
RRD stores data with higher precision than ever before (float versus
integer) which allows us to measure bigger things (OC3's) and smaller
things (Unix load averages) without relying on goofy scaling hacks, as
we were required to do with MRTG. The data file where this data lives
on disk is still a fixed size over time, and scales with the product
of the number of datasources and number and sizes of the data
arrays. The data storage in use for an entire installation still
scales linearly with the number of targets under observation. At WebTV
Networks, we have Cricket configured to store 6 variables for every
router and switch interface. Each interface requires 174 kilobytes on
disk, which is enough room to store all the data for 600 days. Of
course, your mileage may vary; Cricket is configurable, so you can
choose to keep more or less data, depending on your needs.
RRD still provides the same simple, sparse graphs we grew to love
while using MRTG. There is now much more flexibility at the time the
graph is generated. For instance, we can choose to show some data
sources, but not others. We can choose to scale a data source using an
arbitrary mathematical expression. For instance, we can fetch the
temperature from a router in Celsius, and present it to the user in
Fahrenheit. Since almost all data seems to arrive in exactly the wrong
units, it's quite helpful to have this flexibility. Do you think
about Ethernet capacity in
``megabytes/sec''
The scaling feature
lets us turn that measurement into
``megabits/sec''
It is also possible
to integrate data from multiple sources into a single graph. For
instance, you could make a summary graph showing the sum of the
traffic across all of your Internet links.
Figure 4. A simplified example of a config tree.
Recall that part of the problem was flexibility and performance of the
database, both of which RRD solved. The other part of the problem was
scalability of the configuration. Our solution
is something called the config tree. A config tree is a set of
configuration files organized into a tree. This hierarchical
configuration structure allows us to use inheritance to reduce
repeated information in the configuration. To simplify implementation
of the config tree, the inheritance strictly follows the directory
structure; complicated concepts like multiple inheritance are not
supported. The attributes that are present in a leaf node are the
union of all the attributes in all the nodes on a path leading from
the root of the config tree to the leaf node. Lower nodes can override
higher nodes, so it's possible to make exceptions in certain subtrees,
and do other clever sleight of hand.
It's easier to understand the config tree by looking at an example
(see figure 4). Attributes that all of the system will share are
located at the root of the config tree. For instance, the length of
the polling interval is set there. At the next level, we set
attributes that will be restricted to the current subtree. At this
level, typically you will find the target type. Finally at the lowest
level we set things that will vary on a per-target basis. For
instance, we set the interface name that we are trying to measure
here. By using the power of inheritance, you can avoid repeating the
information at the top of the config tree many times near the bottom
of the config tree. The three level config tree in the example is the
simplest in common use. The one that ships with Cricket in the
sample-config directory works like this. Config trees can and do have
many more levels. At WebTV Networks, for instance, we add levels to
break apart routers in different data centers to make the directory
listings more manageable. There are no built-in limits on the shape of
the config tree, only practical ones.
Figure 5 shows the power of the string expansion feature in
Cricket. When it comes across strings in the config tree with special
markers in them, it expands the strings in much the same way a
scripting language expands variables. The sample config tree that
ships with Cricket uses this feature in several places to dynamically
build strings from settings already available. In the example, the
short and long descriptions for the target are set based on some data
inherited from high in the tree, and other data related to the target
itsbase.
thus, will not get repaired until you
threaten to revoke your support license - which, by the way, does not
expire until next December.
Finally, let's be honest: Free Software rules. Who would you rather
trust with the smooth operation of your network, a bunch of folks who
are in the same boat as you, or a giant vendor with 100-page
implementation plans and slick salesmen? The choice is obvious, to me
at least: network managers design better tools to do their
job3.
Sometimes commercial tools are the right tool for the job, but in this
case, it made more sense to go at it on our own. It will make even
more sense for your organization, since WebTV Networks has
done much of the hard
work. All you need to do is download, install and configure
Cricket.
The solution comes as two pieces, a new database and a new
user-visible front-end. By changing out the back-end, we addressed the
flexibility and performance goals. By changing out the front-end, we
improved the user interface, both for configuring the system and
viewing the graphs. The two pieces are developed semi-independently,
one by me (in California) and one by the original author of MRTG,
Tobias Oetiker (in Switzerland). Keeping the two pieces separate makes
it easier to develop them, and it also means the database component
can be reused in other contexts. RRD, the database backend,
is distributed separately from
Cricket4.
We are deeply indebted to Tobias for his hard work on the database
back-end. Without it, Cricket would be nothing. By engineering RRD
correctly from the start, and by improving it's reliability over the
last year, Tobias solved over half our problems with MRTG himself. I
simply had the easy job of passing the right data into his code. All
the seeming magic of Cricket comes from Tobias's RRD code.
The next generation back-end is called RRD, the Round Robin Database.
It takes over exactly the same jobs the back end provided with MRTG
did: storing and rolling up the data and generating graphs from the
stored data. RRD is written in C, and comes in both a Perl module and
a command line version, which can be used interactively or across a
pipe from scripts. It can be used either directly by simple
scripts, or via frontends like Cricket. Other frontends are available
from the RRD website.
RRD achieves high performance by using binary files to minimize I/O
during the common update operation. The
``round robin''
in RRD refers to
the fact that RRD uses circular buffers to minimize I/O. RRD is at
least one, and possibly two orders of magnitude faster than MRTG's
backend, depending on how you measure it. It's truly incredible to
watch it chew through data if you have ever seen MRTG trying to handle
the same job.
RRD is more flexible than MRTG in at least two dimensions. First, it
can take data from an arbitrary number of data sources. MRTG was
limited to two datasources. Originally they were reserved for input
bandwidth and output bandwidth, though they got co-opted for other
measurements by many MRTG hackers. RRD can also keep an arbitrary
number of data arrays, each fed at a different rate. For instance, you
can keep 600 samples taken every 5 minutes alongside 600 samples
taken every 30 minutes. Thus, you can have 5 minute data stretching back
50 hours into the past, and 30 minute data stretching back 12
days. When you draw a graph of this data, you can see recent data in
high resolution, and older data in lower resolution. This is one of
the original features of MRTG that made us so happy. In RRD this
feature is completely configurable.
RRD stores data with higher precision than ever before (float versus
integer) which allows us to measure bigger things (OC3's) and smaller
things (Unix load averages) without relying on goofy scaling hacks, as
we were required to do with MRTG. The data file where this data lives
on disk is still a fixed size over time, and scales with the product
of the number of datasources and number and sizes of the data
arrays. The data storage in use for an entire installation still
scales linearly with the number of targets under observation. At WebTV
Networks, we have Cricket configured to store 6 variables for every
router and switch interface. Each interface requires 174 kilobytes on
disk, which is enough room to store all the data for 600 days. Of
course, your mileage may vary; Cricket is configurable, so you can
choose to keep more or less data, depending on your needs.
RRD still provides the same simple, sparse graphs we grew to love
while using MRTG. There is now much more flexibility at the time the
graph is generated. For instance, we can choose to show some data
sources, but not others. We can choose to scale a data source using an
arbitrary mathematical expression. For instance, we can fetch the
temperature from a router in Celsius, and present it to the user in
Fahrenheit. Since almost all data seems to arrive in exactly the wrong
units, it's quite helpful to have this flexibility. Do you think
about Ethernet capacity in
``megabytes/sec''
The scaling feature
lets us turn that measurement into
``megabits/sec''
It is also possible
to integrate data from multiple sources into a single graph. For
instance, you could make a summary graph showing the sum of the
traffic across all of your Internet links.
Figure 4. A simplified example of a config tree.
Recall that part of the problem was flexibility and performance of the
database, both of which RRD solved. The other part of the problem was
scalability of the configuration. Our solution
is something called the config tree. A config tree is a set of
configuration files organized into a tree. This hierarchical
configuration structure allows us to use inheritance to reduce
repeated information in the configuration. To simplify implementation
of the config tree, the inheritance strictly follows the directory
structure; complicated concepts like multiple inheritance are not
supported. The attributes that are present in a leaf node are the
union of all the attributes in all the nodes on a path leading from
the root of the config tree to the leaf node. Lower nodes can override
higher nodes, so it's possible to make exceptions in certain subtrees,
and do other clever sleight of hand.
It's easier to understand the config tree by looking at an example
(see figure 4). Attributes that all of the system will share are
located at the root of the config tree. For instance, the length of
the polling interval is set there. At the next level, we set
attributes that will be restricted to the current subtree. At this
level, typically you will find the target type. Finally at the lowest
level we set things that will vary on a per-target basis. For
instance, we set the interface name that we are trying to measure
here. By using the power of inheritance, you can avoid repeating the
information at the top of the config tree many times near the bottom
of the config tree. The three level config tree in the example is the
simplest in common use. The one that ships with Cricket in the
sample-config directory works like this. Config trees can and do have
many more levels. At WebTV Networks, for instance, we add levels to
break apart routers in different data centers to make the directory
listings more manageable. There are no built-in limits on the shape of
the config tree, only practical ones.
Figure 5 shows the power of the string expansion feature in
Cricket. When it comes across strings in the config tree with special
markers in them, it expands the strings in much the same way a
scripting language expands variables. The sample config tree that
ships with Cricket uses this feature in several places to dynamically
build strings from settings already available. In the example, the
short and long descriptions for the target are set based on some data
inherited from high in the tree, and other data related to the target
itsbase.
thus, will not get repaired until you
threaten to revoke your support license - which, by the way, does not
expire until next December.
Finally, let's be hone
Why Not Commercial Tools?
The Solution
RRD: The Round Robin Database
The Config Tree
The Solution
RRD: The Round Robin Database
The Config Tree
The Solution
RRD: The Round Robin Database
The Config Tree