NodeBrain

A Rule Engine for State and Event Monitoring

Home

News

Packages

Controlled Automated Regression Tests

The goal of controlled automated regression tests is to exercise a wide range of possible input streams and measure success or failure quickly. We want to be able to repeat these tests any number of times until all exposed defects are eliminated. This requires tools for feeding sequences of commands (input) to nb and comparing it's reaction (output) to stored reaction sequences. For many tests, input and output sequence is important, but timing is not. For tests involving time conditions, timed input and output sequences are required.

While there are automated software testing tools available, the automation requirements for testing nb seem relatively simple---a Perl script should do it for a common and portable test set. If individual testers want to use their favorite tools, all the better. But we need to develop a basic set of tests that can be distributed with the source code and easily executed to verify a build.

            
                   
  Test Script ------>[ Automation ]-------> Exceptions
                      |          ^
                      V          |
                  input -> nb -> output

Cell Expression Tests

It is very important that each release be tested to verify that cell expressions are interpreted as documented and that syntax valid under the prior release is still valid. This testing is a great place to start because there aren't many "moving parts". This simply requires test scripts that cover a wide variety of expressions with known solutions---even when known to be unknown. :)

Rule Engine Tests

Provided the interpreter evaluates cell expressions properly, the next step is to verify that rules fire as expected for given sequences of events and state changes. Here we are interested in testing the response of each operator or function to various state changes. For example, does a flip-flop operator respond to left and right operand state changes as expected? These tests can get progressively more complicated. Simple tests can be performed on rules that have no dependence on time. When event caches are used with time interval parameters, the test script must specify time intervals from one input command to the next, as well as time intervals between expected output lines. This is also true when time conditions are specified in cell expressions.

Time Expression Tests

NodeBrain has a rich set of time functions and operators that require special attention. This is not different from Cell Expression Testing or Rule Engine Testing, it is more of the same with special emphasis on time conditions. The FORECAST command can be used to quickly test syntax and interpretation of time conditions. This is a form of Cell Expression Testing, where we have "no moving parts." However, we must still perform Rule Engine Testing on time conditions because it is a different mode of operation that could fail to interpret a time expression as forecasted.

Command Tests

It is necessary to include tests to exercise every type of command prefix or verb supported by NodeBrain. Tests that focus on an individual command and test a wide variety of parameter combinations are good. Testing combinations of closely related interacting commands is also good.

Performance Tests

Tests are required to measure the rate at which commands can be handled, and to measure associated resource utilization---at a minimum, cpu, disk IO, and memory. Tests are also needed to do repetitive operations looking for memory leaks and system handle leaks.

Security Tests

There are two levels of security tests. First we need to verify that the documented security features work as advertised. Secondly we need to look for ways to circumvent these features. For example, do any of the listeners use unchecked buffers enabling a possible buffer overflow attack? Does the command parser copy tokens into fixed length buffers without testing the token size? These types of tests should be included in the regression test suite.

New Feature Tests

When new features are introduced, they must be considered with respect to each of the prior test areas and tests must be added or modified as required to exercise them. These modifications to the regression test suite become a deliverable of an enhancement release. These tests are designed based on the documentation. Exceptions will reveal errors in the code and/or the documentation.