Data Storage Requirements

Data must must be self-describing or tightly coupled to a metadata descriptor/schema file

• Should include units, description of data.
• In the current implementation, a file named "slate.dir" is written along with the binary output log. This "slate.dir" is managed by the Slate class.
• Every time a reference to a named code unit is made via Slate::GetName( const Str& name ) (called by Component and Logger constructors), if the name is new and unique, then Slate::WriteNameDirectory is called to add the name to slate.dir
• Every time Slate::NewInputReader Slate::NewReader or Slate::NewWriter or Slate::NewUniversalReader or Slate::NewUniversalWriter is called a new DataElement (aka slate variable) is constructed. The DataElement constructor calls Slate::GetElementURI( const ElementURI& elementURI ), which calls Slate::WriteElementURIDirectory to add the element name to slate.dir.
• Also, calls to create new instances of DataReader or DataWriter call the DataAccessor constuctor, which calls Slate::GetDataAccess – which may call Slate::WriteDataAccessDirectory if the DataAccess is new. The DataAccess contains metadata about the variable.

Data must be logged in a space-efficient format.

• This is a subjective requirement, but efforts have been made to keep the log as compact as possible. For example:
  1. Bit-packing flags
  2. The creation of DataValue types such as Float, Float3, and Float2 to allow for low-resolution logging of floating-point values.
  3. Logging of compressed 16 and 64 bit integers (for ids and timestamps) using OutStream::writeShortCompact OutStream::writeLongLongCompact – and logging only timestamp differences within a cycle.

Lose no more than one compute cycle of data in the event of a catastrophic failure

• i.e. kernel crash or sudden power disconnect.
• LogEngine runs at the end of computations in each compute cycle.
• Each cycle starts with a "0" in the log file. Due to compression of values in the log, the value "0" is surprisingly unique. However, testing may encourage us to select a more unique (i.e., multi-byte) value.

• Todo:

  Test ability to recover from corrupted files.

Data file format must minimize data loss in the event of file corruption

• i.e. to minimize the amount of data lost in the case of one or more byte-errors.
• Also, a new log file will be started each hour during a mission, so that a completely corrupted log file will destroy at most one hour of data.

• Todo:

  Implement new file every hour.

Data logging system must be optimized for power- and processor efficiency

• Data is logged at the end of each cycle. Achieves requirement Lose no more than one compute cycle of data in the event of a catastrophic failure, while avoiding complications of running logging in its own thread (and there appears to be a significant cost for context-switching on the arm9 processor).

New variables (from control and science sensors) must be handled without recompilation.

• As described under requirement req_storage_metadata, the system can accommodate new variables appearing on the fly.

Data files must include format versioning information

• to address cross- and backwards compatibility between writers and readers.
• The first write to slate.dir is a version packet, written by Slate::WriteVersion

Data sets must include information on the currently running missions and the current state within that mission.

• Currently, such transitions are only stored as free-text SyslogEntry instances.

• Todo:

  Log mission transitions as EventEntry instances.

Double-precision floating point must be loggable.

• The Double class extends the abstract DataValue class to provide logging of double-precision floats.

Less than double-precision floating point must be loggable for space efficiency.

• The Float, Float3, and Float2 classes extend the abstract DataValue class to provide logging of 4-byte, 3-byte, and 3-byte lower precision floats (internally, they all use 4-byte floats).
• The Location class logs latitude and longitude pairs as 6-byte floats (but uses a full 8-byte float internally).

32-bit, and 8-bit integers must be loggable.

• The Int and UChar classes extends the abstract DataValue class to provide logging of 4-byte, 1-byte integers (internally, they both use 4-byte integers).

Scaling values for storing floating values as integers must be available.

• Not implemented

• Todo:

  Implement Scaling values for storing floating values as integers must be available.

It must be possible to log variable-length strings

• The StrValue class DataValue the abstract DataValue class to provide logging of Str "string" values.