Main Page Modules Class Hierarchy Alphabetical List Compound List File List Namespace Members Compound Members File Members

Test tools

This document describes one way of creating a regression test using the log file of an IDEA agent. What we propose is using the log file, extracting what we refer to as events, possible combining events from multiple agents and check them against a 'gold standard' event file.

For new models the 'gold standard' file needs to be constructed from the initial log file and verified by the domain-expert as being correct. See 'Building a "Gold Standard" events file'.

Processing an IDEA-agent log file

Given an IDEA-agent run which produced a log file we can process the file to extract events.

Events

Events are being extracted from the log file using the script 'build-summary-from-log'. The following events are being extracted:

The agent relay is being invoked to send a start message, in the log file this is a line containing the string "SEND_START"
The agent relay is being invoked to notify the agent received a start message, in the log file this is a line containing the string "RECV_START"
The agent relay is being invoked to send a stop message, in the log file this is a line containing the string "SEND_STOP"
The agent relay is being invoked to notify the agent received a stop message, in the log file this is a line containing the string "RECV_STOP"
The agent relay is being invoked to send a value message, in the log file this is a line containing the string "SEND_VALUE"
The agent relay is being invoked to notify the agent received a value message, in the log file this is a line containing the string "RECV_VALUE"
The agent relay is being invoked to send a status message, in the log file this is a line containing the string "SEND_STATUS"
The agent relay is being invoked to notify the agent received a status message, in the log file this is a line containing the string "RECV_STATUS"
The agent relay is being invoked to send a telemetry message, in the log file this is a line containing the string "SEND_TELEMETRY"
The agent relay is being invoked to notify the agent received a telemetry message, in the log file this is a line containing the string "RECV_TELEMETRY"
The agent relay is being invoked to send a user-defined message, in the log file this is a line containing the string "SEND_USERDEF"
The agent relay is being invoked to notify the agent received a user-defined message, in the log file this is a line containing the string "RECV_USERDEF"

When an agent relay is invoked to send a message the next step by the agent is to retrieve a communication relay and notify the communication relay to send the message. This means that seeing the event associated with the agent relay invocation for sending a message does not guarantee the actual sending of the message.

If the message can not be send by the communication relay (application specific code) we expect the relay to raise an error which should be picked up by the 'report-errors-in-log' script (if the agent did not end).

Integrating multiple summaries in one file

In case you run a scenario involving more than one IDEA agent we provide a script to integrate multiple summaries of IDEA log files into one sorted integrated summary file. The events in the integrated file will be sorted on the time they occurred.

Building a "Gold Standard" Events File

The test tools are build on the assumption that there exists a "gold standard" events file that defines the correct behavior of the agent with respect to the "events" therein.

The output produced by the agent contains just such set of events which can be easily extracted from a log file for comparison against such a "gold standard" set of events. This output can also be used to create the "gold standard" events file, and in practice, this is what developers and users appear to want to do.

The tool which does this is build-summary-from-log. So, to create a so-called "gold standard", simply run build-summary-from-log on an output log, and copy/move the <log-file>.summary to wherever you keep your "gold standard" events files, and rename it to an appropriate name. For example (assuming that build-summary-from-log is in your path):

% cd Logs/nominal
% build-summary-from-log k9Runner_IPC.log
% mv k9Runner_IPC.log.summary ../../Scenarios/nominal.events

You now have a "gold standard" events file in your Scenarios directory.

Script files

build-summary-from-log <log-file>

This script extracts the execution events from a log file, and places them in <log-file>.summary.

report-errors-in-log <log-file>

This script reports all line in <log-file>.log which match a known set of error/warning signatures, and places them in <log-file>.error.log.

build-integrated-summary <summary-file> [<summary-file>+]

Sort the named <summary-file>s and place them in "integrated.summary".

verify-events-in-summary <events_file> <summary_or_log_file> <report_file> [<format>]

This script reads in a set of expected "gold standard" events from the named <events_file>, and the execution events summary (or full log) <summary_or_log_file> containing the events and "verifies" that each expected event is (or is not) in fact in the summary (or log), and places the result in <report_file>.

Generated on Fri Feb 3 17:09:44 2006 for IDEA.