ESP Drifter Telemetry
brent@mbari.org
5/10/11

Overview

The ESP drifter's telemetry gathering software consists of two pair of independent executables.  On the mooring controller, the raw logger configures sensors to report information at rates determined by an ASCII configuration file.  These reports are captured verbatim in the unstructured raw log.  In parallel, the same raw logger executable also writes the salient information from the unstructured raw log in a separate structured log file.  The structured log file consists of a copy of the ASCII configuration followed by an array of fixed length sensor reports.  The summary server, running on the mooring, compiles slices of this structured log into compact, binary summaries suitable for uploading via low-bandwidth network links.

Aboard ship, the summary client  will be run from a script  with a standard query every few minutes to append the latest summary data from the mooring to the shipboard summary log.  The shipboard summary log is simply a historical record of summary data previously uploaded from the mooring's summary server.  I.E.  The queries and results of each summary query on the mooring's structured log in the order in which they where issued.

The shipboard summary log server tries to satisfy structured queries on telemetry from the shipboard summary log first, issuing new queries on the mooring only as necessary.  If the query is for data in the past, the entire shipboard log is scanned to extract as much data as possible before issuing a new query to retrieve the rest of the required data.  The shipboard summary log server always returns data in comma delimited ASCII strings.

Raw Logger

The Raw Logger is a daemon running on the mooring.  It is typically started at the beginning of the deployment.  Each time the daemon starts, it either creates two new raw log files – the unstructured and structured raw logs or appends to existing ones.  These files will be written somewhere under /var/log/$USER.  Their names will be typically be drifter{timestamp}.raw and drifter{timestamp}.data.  

If the daemon is given just a configuration file on its command line, new log files are created.  In this case, it is an error if either file already exists.  If the daemon is given a an existing .raw file, a new .data file is created and populated by reading the .raw file, then logging begins as usual.  The resulting .data file will likely be “sparse” in this case.  There may be a special flag to “trust” that the .data and .raw files are in sync to save the time of regenerating the .data from the .raw file.

Raw Logger Configuration File

The configuration file consists of a sequence of key/value pairs.

Global keys:		Values

RawFile:	path name	#which is passed through 'C's strftime() function
DataFile:	path name	#ditto for the structured log
ticRate:	numeric tic rate in hz  [float]
ticPeriod:	numeric tic period  [float] (hz and period are mutually exclusive)
days:		duration days [float] (additive)
hours:		duration hours [float] (additive)
minutes:	duration minutes [float]  (additive)
seconds:	duration seconds [float] (additive)
tics:		duration expressed as number of tic periods [integer >= 0] (additive)
endTime:	end time string  (mutually exclusive with all additive duration keys)
Device:	device name  (subsequent keys are processed by named device driver)
: device name	exits current scope (exits current Device)

Device Specific keys:	Values

divisor:	all devices support sampling every nth tic  [integer >= 1] (defaults to 1)

port:		path to I/O port to which device is connected
IP:		IP address (for a networked device)

item:		[identifier string]
  channel:	[integer]  (must be unique)
  offset:	[float]
  gain:		[float]
  units:		[string]   e.g.  Volts
  suffix:	[string]   e.g.  V
  format:	[string]   e.g.  “.2f”
: item id		optional explicit ending of item definition

item :		{item name2}
…
: item name2		optional explicit ending of item name2 definition

: device name	optional explicit ending of a specific device's definition

After global parsing loop recognizes a Device key/value, parsing control passes to the named device driver which starts looking for its device specific keys.  Typically, within a driver, the “item” key defines a data channel, which starts another context for processing the driver's data channel specific keys.

When the driver sees the outer null key “:”, it must return the total number of bytes required by each report and the report's rate divisor for the structured log.

GM Comments

In the interest of minimizing the potential for mistakes, I’d like to see if we can simplify a few things and make a few other things really specific to our mission.

1)	If I understand correctly, you will be logging the raw sensor messages in the vendor’s format from the accelerometers, GPS and Labjack in the .raw log.  Sounds great.
2)	Don’t make the user specify the .raw and .data filenames.  Just make the daemon start new files every time it starts with the file format you described drifter{timestamp}.raw and drifter{timestamp}.data
3)	I think we can modify the configuration file you described to make it specific to our mission and our system and make it a little simpler a less prone to user mistakes.  Something like the following 

Mission Keys
endTime:	end time string  

Labjack Keys:
rawSampleRate (Each channel Samples/second):	20
item:		Tension
  channel: 		0
  summaryRate(samples/second): 0.5 
  min/max/mean:  yes 	
  offset (volts): 	[float]
  gain:		[float]
  units:		Pounds force
  suffix:		lbf
  format:		“.0f”
: item id		

item:		Battery Voltage
  channel:	1
  summaryPeriod (seconds): 300 	
	or
  summaryRate (samples/second): 0.00333 	
  min/max/mean:  yes 	
  offset:	[float]
  gain:	[float]
  units:	Volts
  suffix:	V
  format:	“.02f”
: item id		

item:		Battery Current
  channel:	2
  summaryPeriod:300 (seconds) 	
  min/max/mean:  yes 	
  offset:	[float]
  gain:	[float]
  units:	Amperes
  suffix:	A
  format:	“.2f”
: item id		 

ESP Accelerometer Keys
	rawSampleRate (samples/second):	20
item:		Z Acceleration
  command:C2
  summaryRate (samples/second) :0.5  	
  min/max/mean:  yes 	
  format:	“.3f”
: item id		

item:		X Acceleration
  command:C2
  summaryRate (samples/second): 0.1  	
  min/max/mean:  yes 	
  format:	“.3f”
: item id		

item:		Y Acceleration
  command:C2
  summaryRate (samples/second) :0.1  	
  min/max/mean:  yes 	
  format:	“.3f”
: item id		 

item:		Euler Angles (pitch, roll, yaw  If we can combine these into one item that will be great since they will always be treated the same, otherwise we can split them out)
  command:CE
  summaryRate (samples/second) :0.1	
  min/max/mean:  yes 	
  format:	3X “.1f”
: item id		

item:		Orientation Matrix (9 values in a 3x3 matrix  If we can combine these into one item that will be great since they will always be treated the same, otherwise we can split them out)
  command:C5
  summaryRate (samples/second) :0.0 (don’t send these, just log them.  This is       the item we would like to use to mathematically rotate the raw acceleration values into an Earth Centered reference frame.  	
  min/max/mean:  no 	
  format:	same as raw
: item id		

Buoy Accelerometer Keys
	Same format as ESP accelerometer

Buoy GPS
	rawSamplePeriod (seconds): 5
	summaryPeriod (seconds): 300
	NMEAMessage: GGA