// -*- Mode: text -*-
// File: $CVSROOT/IDEA-DEV/Documentation/Tools/Test/test.txt
//  CVS: $Id: test.txt,v 1.4 2006/01/21 01:03:04 plaunt Exp $

/*!

\page test Test tools

<hr> 

This document describes one way of creating a regression test using the log file
of an IDEA agent. What we propose is using the log file, extracting what we refer 
to as events, possible combining events from multiple agents and check them against
a 'gold standard' event file.

For new models the 'gold standard' file needs to be constructed from the initial log
file and verified by the domain-expert as being correct. See 'Building a "Gold Standard"
events file'.

\section testtools Processing an IDEA-agent log file

Given an IDEA-agent run which produced a log file we can process the file to extract events.

<img src="../Images/extract.png">

\subsection events Events

Events are being extracted from the log file using the script 'build-summary-from-log'. 
The following events are being extracted:

\li The agent relay is being invoked to send a start message, 
in the log file this is a line containing the string "SEND_START"
\li The agent relay is being invoked to notify the agent received a start message, 
in the log file this is a line containing the string "RECV_START"
\li The agent relay is being invoked to send a stop message, 
in the log file this is a line containing the string "SEND_STOP"
\li The agent relay is being invoked to notify the agent received a stop message, 
in the log file this is a line containing the string "RECV_STOP"
\li The agent relay is being invoked to send a value message, 
in the log file this is a line containing the string "SEND_VALUE"
\li The agent relay is being invoked to notify the agent received a value message, 
in the log file this is a line containing the string "RECV_VALUE"
\li The agent relay is being invoked to send a status message, 
in the log file this is a line containing the string "SEND_STATUS"
\li The agent relay is being invoked to notify the agent received a status message, 
in the log file this is a line containing the string "RECV_STATUS"
\li The agent relay is being invoked to send a telemetry message, 
in the log file this is a line containing the string "SEND_TELEMETRY"
\li The agent relay is being invoked to notify the agent received a telemetry message, 
in the log file this is a line containing the string "RECV_TELEMETRY"
\li The agent relay is being invoked to send a user-defined message, 
in the log file this is a line containing the string "SEND_USERDEF"
\li The agent relay is being invoked to notify the agent received a user-defined message, 
in the log file this is a line containing the string "RECV_USERDEF"

When an agent relay is invoked to send a message the next step by the agent is to retrieve a 
communication relay and notify the communication relay to send the message. This means that
seeing the event associated with the agent relay invocation for sending a message does 
not guarantee the actual sending of the message.
 
If the message can not be send by the communication relay (application specific code) we expect 
the relay to raise an error which should be picked up by the 'report-errors-in-log' script 
(if the agent did not end).

\subsection integrating Integrating multiple summaries in one file

In case you run a scenario involving more than one IDEA agent we provide a script to integrate
multiple summaries of IDEA log files into one sorted integrated summary file. The events in 
the integrated file will be sorted on the time they occurred.

<img src="../Images/integrate.png">

\subsection buildinggold Building a "Gold Standard" Events File 

The test tools are build on the assumption that there exists a "gold
standard" events file that defines the correct behavior of the agent
with respect to the "events" therein.

The output produced by the agent contains just such set of events
which can be easily extracted from a log file for comparison against
such a "gold standard" set of events.  This output can
also be used to create the "gold standard" events file, and in
practice, this is what developers and users appear to want to do.

The tool which does this is build-summary-from-log.  So, to create a
so-called "gold standard", simply run build-summary-from-log on an
output log, and copy/move the <log-file>.summary to wherever you
keep your "gold standard" events files, and rename it to an
appropriate name.  For example (assuming that build-summary-from-log
is in your path):

\li  % cd Logs/nominal
\li  % build-summary-from-log k9Runner_IPC.log
\li  % mv k9Runner_IPC.log.summary ../../Scenarios/nominal.events

You now have a "gold standard" events file in your Scenarios directory.

\subsection scriptfiles Script files

\li <tt>build-summary-from-log <log-file></tt>

This script extracts the execution events from a log file, and
places them in <log-file>.summary.

\li <tt>report-errors-in-log <log-file></tt>

This script reports all line in <log-file>.log which match a known set
of error/warning signatures, and places them in <log-file>.error.log. 

\li <tt>build-integrated-summary <summary-file> [<summary-file>+]</tt>

Sort the named <summary-file>s and place them in "integrated.summary".

\li <tt>verify-events-in-summary <events_file> <summary_or_log_file> <report_file> [<format>]</tt>

This script reads in a set of expected "gold standard" events from the
named <tt><events_file></tt>, and the execution events summary (or
full log) <tt><summary_or_log_file></tt> containing the events and
"verifies" that each expected event is (or is not) in fact in the
summary (or log), and places the result in <tt><report_file></tt>.

*/
