This manual is for the System NodeBrain Kit, used to create a sample NodeBrain application. Releases prior to 1.0 are prototypes intended to motivate additional development, but are expected to be complete enough to support a working application that demonstrates how the NodeBrain rule engine can be used.
Release 0.8.17, July 2014
Copyright © 2014 Ed Trettevik <eat@nodebrain.org>
Permission is granted to copy, distribute and/or modify this document under the terms of either the MIT License (Expat) or NodeBrain License. See the Licenses section at the end of this document for details.
Short Table of Contents
This chapter introduces basic concepts of the System NodeBrain Kit. The reader should be familiar with the concepts introduced by the Caboodle NodeBrain Kit, upon which the System Kit depends.
The System NodeBrain Kit provides a NodeBrain agent called the System agent, that monitors various elements of a server or application and issues alarms to the Caboodle agent provided by the Caboodle NodeBrain Kit. A server may run any number of System agents, each dedicated to monitoring the server, a shared application, or an application managed by a single user.
A small minimal set of monitoring plans is provided by the System Kit. Users are encouraged to develop additional kits that build on the System Kit and make them available to others. If you develop general monitors that seem like logical additions to the System NodeBrain Kit, let us know and your recommendation will be considered.
This kit can be used to monitor a server while running as root or as a dedicated user.
A startup/shutdown script called "sysmon" is provided for this use case on Linux servers.
The NodeBrain Open Source Project does not provide platform specific kits for monitoring servers, instead only
a skeletal kit is provided. Users of the System Kit are encouraged to develop additional kits built on top
of the System Kit, and to share those kits with others. For example, a robust set of log monitoring plans for
a given platform would make a very useful kit.
For security reasons, when running NodeBrain as root or any account with special systemr-wide permissions,
you should avoid using node modules that enable remote control.
Applications often have unique monitoring requirements that are not satisifed by server level monitoring. The System Kit may be used to implement application sepecific monitoring. When developing monitoring plans for generic applications used by many, consider reusability so others can benefit from your work.
The System Kit is released as a GNU style source distribution file, and an architecture independent RPM file, both of which can be downloaded at http://nodebrain.org, with SourceForge.net providing the download service.
When installing from a GNU source distribution release file, follow these steps, adjusting the release number as needed to match the file you download.
# tar -xf nbkit-system-0.8.14.tar.gz
# cd nbkit-system-0.8.14
# ./configure
# make
# make check
# make install
|
When using this method, the kit is installed to /usr/local/share/nbkit/system-0.8.14.tar.gz.
You can override this using ./configure --prefix=/home/foo, which would cause the kit to be
installed as /home/foo/share/nbkit/system-0.8.14.tar.gz.
If on a Linux platform that supports RPM files, you may issue the following command to create a source RPM file and a noarch RPM file to enable native package management.
# make rpm |
When installing from an RPM file, use the following rpm command. The RPM is not architecture dependent because it contains no compiled code, only Perl and NodeBrain scripts, and a few web pages and associated image files.
# rpm --install nbkit-system-0.8.14-1.noarch.rpm |
When using this method, the kit is installed to /usr/share/nbkit/system-0.8.14.tar.gz.
This chapter walks the reader through the initial setup process as performed on a development system.
The nbkit commands used in this chapter are explained in the Caboodle NodeBrain Kit manual and nbkit man page.
To use the System Kit, you must already have a caboodle that uses the Caboodle NodeBrain Kit.
To add the System Kit to your caboodle, issue the use command of nbkit as shown below.
The nbkit -k command will display the kits installed on the development server.
Specify the kit you want to use for the caboodle.
The last line of the this example would be used for System Kit release 0.8.14 installed using an RPM file.
However, the kit could just as easily come from a location within your home directory.
$ nbkit -k
$ nbkit caboodle use kit-directory
$ nbkit caboodle use /usr/share/nbkit/system-0.8.14.tar.gz
|
Plans provided by the System Kit are placed in the SystemFolder. To provide visibility of the SystemFolder when using the web interface, include it in the _Admin folder as follows.
$ nbkit caboodle enable SystemFolder._Admin.parent |
The System agent runs with a NodeBrain identity, which must be established before starting the agent. Issue these commands.
$ nb
> peer.identify System
> quit
$ echo "declare System identity;" >> ~/.nb/user.nb
|
Use the following command to start the System agent. This agent is responsible for monitoring the application provided by the caboodle.
$ nbkit caboodle start System |
To display plans provided by the System Kit, available for setup, issue the following nbkit command.
$ nbkit caboodle setup |
Plans starting with "System" that are not already setup, may be setup as follows, where SystemProcess is an example.
$ nbkit caboodle setup plan
$ nbkit caboodle setup SystemProcess
|
System plans are designed as rules to be processed by the System agent. However, the System agent will only process them when they are enabled as children of the System agent.
$ nbkit caboodle enable plan.System.parent
$ nbkit caboodle enable SystemProcess.System.parent
|
To see the current relationships for a plan, use the show command.
$ nbkit caboodle show plan
$ nbkit caboodle show System
$ nbkit caboodle show SystemProcess
|
If your caboodle is intended as a system service, you can use a couple files provided by the System Kit in the setup directory to help on a Linux server. Here the steps are descibed as if you are performing the setup on a development server with a caboodle that uses the System Kit. Later you will automate these steps as part of your application package, perhaps an RPM file.
Let's say you are calling your service "foobar", and the caboodle is /var/foobar.
Edit /var/foobar/setup/nodebrain.sysconfig and change CABOODLE to
/var/foobar as illustrated below.
# Distributed as setup/nodebrain.sysconfig
CABOODLE=/var/foobar
AGENT=System
|
The value for AGENT will remain System if you are using the System agent
within your caboodle to provide the foobar service. If the foobar service is provided
by a different agent within our caboodle, then modify AGENT as well.
Perform the following commands to complete the setup.
# cp /var/foobar/setup/nodebrain.service /etc/init.d/foobar
# cp /var/foobar/setup/nodebrain.sysconfig /etc/sysconfig/foobar
# chkconfig add foobar
# service start foobar
|
Before converting a NodeBrain agent into a system service, issue
a stop command using nbkit, even if the agent is not running. This establishes an
intent for the service to be down with respect to the nbkit command.
$ nbkit foobar stop System |
After converting a NodeBrain agent into a system service, you should perform all start,
stop, and restart operations using the service command as root, and avoid
using the start, stop, check and bounce options of the nbkit command
on the agent.
# service foobar start
# service foobar stop
|
The problem with using options of nbkit to start or stop the agent is distruption of the service command's use
of a pid file to identify a running service. (This is an opportunity for a future release of nbkit to include awareness
of agents running as services, and to use the service command when appropriate.)
You can still use other nbkit commands like edit and compile on the agent.
When the agent is compiled, either directly or as a result of a recursive compile started
by a child plan change, restart the service.
# nbkit foobar compile System
# service foobar restart
|
The System Kit does not provide any commands. The commands provided by the Caboodle Kit are sufficient.
The System Kit provides only one agent called "System". This agent, in addition to the Caboodle agent provided by the Caboodle Kit, is often sufficient to monitor a server or application. However, you are free to create additional agents like the System agent to distribute monitoring tasks for any reason. Performance and availability are two reasons one might want to isolate one set of monitoring rules from another by using multiple agents.
Adapers are small scripts that adapt NodeBrain to an application environment based on a model of interaction not defined by NodeBrain, but taking advantage of more general types of interaction defined by NodeBrain. In other words, adapters can fit into models defined by NodeBrain Kits, about which NodeBrain has no awareness.
The System Kit provides the following plan compilers to implement additional rule generation schemes. These schemes are described a bit under Model Plans later in this document. However, be best we to understand them is to study plans that use these schemes.
SystemAudit.compiler
SystemThreshold.compiler
SystemThresholds.compiler
|
The System Kit does not provide any alarm adapters.
Servants connect NodeBrain rules to an application environment using a method of interaction prescribed by, and fully support by, NodeBrain. This sets them apart from adapters, which interact with NodeBrain indirectly, based on a model unknown to NodeBrain.
The System Kit provides a small set of servants, intended primarily to demonstrate a concept you can extend, but also to provide an application you can use without too much effort.
SystemCpu.probes
SystemFilesys.probe
SystemProcess.probe
SystemThreshold.node
|
These servants are used by related plans described in the next chapter, and it isn't necessary
to understand these servants before using the plans. However, if you intend to create additional
servants for use with the System Kit, it may be helpful to know that the System Kit defines, by example,
types of servants identified by a file extension of probes, probe, and node.
By modeling a servant after a probes servant, it can be used with the SystemThresholds compiler,
or a compatible compiler. By modeling a servant after a probe servant, it can be used with the
SystemThreshold compiler, or a compatible compiler. The SystemThreshold.node servant is a helper
to both the SystemThreshold and SystemThresholds compiler.
When creating compilers and servants of your own, after first just getting something to work, consider how your might conform to an existing model or API for reusability, or how you might define a new one that others can follow. The idea is to build up a set of reusable compilers and model plans, so others can focus on the creation of plans built from the model, and servants that plug into those models.
Plans are XML documents that represent a set of NodeBrain rules, or configuration file, as a table with options. This kit provides just a few to get you started with monitoring a server or application.
Folder are provided to aid in navigation when managing plans using the NodeBrain Planner.
| Plan | Purpose
|
|---|---|
| SystemFolder | Enables navigation to all application plans provided by this kit. Parent is _Admin.
|
Plans in this section are used as a model for creating new plans. They are children of the _Model folder.
| Plan | Purpose
|
|---|---|
| SystemAudit | Audit a log file, watching for patterns matching regular expressions.
|
| SystemThreshold | Create a new theshold plan to monitor values returned from one or more types of "probe" servant.
|
| SystemThresholds | Create a new theshold plan to monitor multiple values returned by a single "probes" servant.
|
Plans in this section provide a minimal application that can be used to monitor the health of a server or application.
| Plan | Purpose
|
|---|---|
| System | Agent responsible for monitoring a server or application. If monitoring a NodeBrain application, additional
agents are often used to implement the application.
|
| SystemCpu | SystemThresholds plan that uses the SystemCpu.probes servant to monitor CPU utilization.
|
| SystemFilesys | SystemThreshold plan that uses the SystemFilesys.probe servant to monitor file system space.
|
| SystemProcess | SystemThreshold plan that used the SystemProcess.probe servant to monitor process to ensure required
processes are running, and forbidden processes are not.
|
NodeBrain is free software; you can modify and/or redistribute it, including this document, under the terms of either the MIT License (Expat) or the NodeBrain License.
MIT License
Copyright © 2014 Ed Trettevik <eat@nodebrain.org>
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
NodeBrain License
Copyright © 2014 Ed Trettevik <eat@nodebrain.org>
Permission to use and redistribute with or without fee, in source and binary forms, with or without modification, is granted free of charge to any person obtaining a copy of this software and included documentation, provided that the above copyright notice, this permission notice, and the following disclaimer are retained with source files and reproduced in documention included with source and binary distributions.
Unless required by applicable law or agreed to in writing, this software is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.