Navproc and Logging on the Rachel Carson

This document details the software components that read in data from various data providers on the Rachel Carson. It also provides links to more detailed documentation about each of the separate pieces and how to develop and deploy them.

Below is a high level flow diagram of the different components in the system. Basically, there are a bunch of data producing instruments and software deployed both on the ship and on the ROV (and the ROV itself) that can output their data as serial data streams (either RS-232 or RS-485). When the system is set up, wiring is run all over the ships to connect all of these serial streams to a Digi Terminal Server that allows many serial ports to be connected to a single computer. The software proccesses known as 'navproc' are then deployed on the computer that is connected to these serial ports (Ubuntu linux machine) and is responsible for reading in data from the serial ports, parsing each stream in a manner appropriate to the data producer, and then publishing the parsed data as LCM messages on the ROV subnet. Then, there are several processes known as 'logr' processes that are consumers of those LCM messages. The logr processes then do two things. First, they log those messages to log files that are then shipped off to shore at a later date. Second, they make the data available over UDP. There is another set of processes known as 'bcserver's that then consumers UDP messages from the logrs and provides another UDP service that is exposed outside the ROV subnet. The design was done to keep the traffic on the ROV subnet as light as possible. From there, any client can make UDP requests to the UDP server to get the latest data cached on the bcserver.

Component Overview

Each of the components in the data flow is made up of a series of software stored in multiple repositories. These pieces are:

Core Nav
navproc-commmon: This repository contains C++ code that is used in the various other navproc- projects. It contains things like LCM message formats, utlities, LCM publishers and subscribers, etc.
navproc-logger: This contains C++ code that is compiled into a utility logr which is installed in /usr/local/bin. This is a configuration driven utility for logging data items into log files and publishing on a UDP port.
navproc-packages: These are code packages that needed to be tweaked to work with the codebase defined in the other navproc repositories. They may be temporary fixes for things that are waiting for kernel or other package updates and may disappear if they are no longer needed.
navproc-process: This repo contains code and scripts that are used to manage the processes that run to read in serial port data, convert to LCM and publish.
Core Logging
bcserver: This software is a bridge for data coming from the navproc system. During logging, the logr processes push data out over UDP messages, but in an effort to not flood the network with UDP messages, the bcserver software was written to listen for the UDP messages and cache data. Then when clients want the data, they make requests from the bcserver which returns the most recent data.
corelogging: This repository contains all the scripts and configuration files that are used to set up the logr instances which listen for LCM messages, convert them to the legacy style data manager items, log them, and then respond to UDP requests to provide the most recent data.
Clients
Ship position grabber (part of corelogging repo): This is a Python script that is actually part of the corelogging project and is run using cron. It's queries the bcserver for navigation data and then publishes the ships latitude and longitude to a RabbitMQ server on shore.

Components Details

This section goes into more details about each of the components that make up this data system.

Core Nav

This section describes the components that make up what is known as the Core Nav section of the data system. Consult the following diagram as you read the sections about 'navproc-common' and 'navproc-process'

graph LR subgraph BitBucket navproc-common-repo["navproc-common"] navproc-process-repo["navproc-process"] end subgraph rcnavproc1.rc.mbari.org:/home/ops subgraph CoreNav subgraph navproc-common lcm-types["lcm-types/(publishers|subscribers)/*.hpp"] end navproc-common-repo --> lcm-types subgraph navproc-process subgraph src subgraph digitquartz_pub lcm-types --> digiquartz_pub_exe["digiquartz_pub(executable)"] end subgraph nmeaGPS_pub lcm-types --> nmeaGPS_pub_exe["digiquartz_pub(executable)"] end subgraph *_pub lcm-types --> \*_pub_exe["*_pub(executable)"] end end subgraph bin subgraph rc bot-proc-cfg-rc["bot_proc.cfg"] core-nav-rc-ini["core-nav-rc.ini"] navproc-start-rc["navproc-start"] end end subgraph processes digiquartz_pub_exe["digiquartz_pub(executable)"] --> digiquartz_pub_proc["digiquartz_pub(process)"] nmeaGPS_pub_exe["digiquartz_pub(executable)"] --> nmeaGPS_pub_proc["nmeaGPS_pub(process)"] \*_pub_exe["*_pub(executable)"] --> \*_pub_proc["*_pub(process)"] end bot-proc-cfg-rc --> digiquartz_pub_proc core-nav-rc-ini --> digiquartz_pub_proc bot-proc-cfg-rc --> nmeaGPS_pub_proc core-nav-rc-ini --> nmeaGPS_pub_proc bot-proc-cfg-rc --> \*_pub_proc core-nav-rc-ini --> \*_pub_proc subgraph serial-ports /dev/ttyaXX --> digiquartz_pub_proc /dev/ttyaYY --> nmeaGPS_pub_proc /dev/ttyaZZ --> \*_pub_proc end subgraph lcm-messages["LCM Messages"] digiquartz_pub_proc --> digiquartz-lcm["Digiquartz LCM"] nmeaGPS_pub_proc --> nmeaGPS-lcm["NMEA GPS LCM"] \*_pub_proc --> \*-lcm["* LCM"] end end navproc-process-repo --> digiquartz_pub_exe navproc-process-repo --> nmeaGPS_pub_exe navproc-process-repo --> \*_pub_exe end end

Navproc-common

This component contains LCM definition files and C++ header files that are used across the navproc-process code and the navproc-logger code (see section under Data Logging). It appears that the .hpp files are not just header files, but contain implementation details as well. There is an lcm_types folder which contains the .lcm message structure as well as subscribers and publishers which are really meant to be the interface to and from LCM. They contain channel management, message buffer construction, etc.

Navproc-process

This repo contains folders for each of the LCM publishers that is specific to an instrument and/or data stream. Each folder contains a _frame.hpp file which is reponsible for the desctructuring (i.e. parsing) of the message that will then be pushed out to LCM. The _pub.cpp file is responsible for working with the serial port to get the message from the instrument or stream. It then hands the message off to the _frame.hpp methods which parse them out into fields that make up the LCM data message and then the _pub.cpp file hands the publishing.

When make is run in navproc-process, it goes into each subfolder and runs make there. Each folder ends up with corenav subfolder which contains _t.hpp files and an _pub executable that will process the data streams and publish messages over LCM.

Now, under navproc-process, there is also a bin directory which has a subdirectory for each location where navproc will run. In that directory are two configuration files bot_proc.cfg and core_nav_xxx.ini where xxx is the name of the location where it is running. It also contains scripts that are used to start and stop navproc. When navproc is to be run in a specific location (rc|sim|wf), the PATH of the account where it is to be run is set to include the proper bin subdirectory. Then, when navproc is to be started, the navproc_start command is issued which reads in the bot_proc.cfg and core_nav_xxx.ini files to start up all the various executables that were built in the src subdirectories.

Core Logging

This section describes the components that make up what is known as the Core Logging part of the data system. Consult the following diagram as you read the sections on 'navproc-logger' and 'corelogging'.

graph LR subgraph BitBucket navproc-common-repo["navproc-common"] navproc-logger-repo["navproc-logger"] end subgraph rcnavproc1.rc.mbari.org:/home/ops subgraph CoreNav subgraph navproc-common lcm-types["lcm-types/(publishers|subscribers)/*.hpp"] end navproc-common-repo --> lcm-types subgraph navproc-processes digiquartz_pub_proc["digiquartz_pub(process)"] nmeaGPS_pub_proc["nmeaGPS_pub(process)"] \*_pub_proc["*_pub(process)"] digiquartz_pub_proc --> digiquartz-lcm["Digiquartz LCM"] nmeaGPS_pub_proc --> nmeaGPS-lcm["NMEA GPS LCM"] \*_pub_proc --> \*-lcm["* LCM"] end subgraph rc-navproc-logger["navproc-logger"] subgraph src logr.cpp --> logr end lcm-types --> logr.cpp end navproc-logger-repo --> logr.cpp subgraph datalogger-processes logr-dq logr-gps logr-* end digiquartz-lcm --> logr-dq nmeaGPS-lcm --> logr-gps \*-lcm --> logr-* logr --> logr-dq logr --> logr-gps logr --> logr-* end end

Navproc-logger

The first component of the core logging is the navproc-logger source repo which contains the source code for an executable named logr. There is a Makefile in the src/datalogger directory that pulls in the lcm-types from the navproc-common project and first generates the .hpp files from the .lcm files and then compiles the logr.cpp file into the logr executable. This executable is then copied into the /usr/local/bin directory which puts it on the path for any user. One thing to note here is that while the navproc-logger project is built alongside the navproc-process, it is not actually started by anthing in the navproc-logger repo source. The management of the processes occurs in code that is under another BitBucket project named 'corelogging' (see below).

Corelogging

The BitBucket project corelogging contains three subdirectories which are each setup to run the core data logging (logr) scripts for each of the three environments (rc|sim|wf). The key subdirectories in each of these are the cfgs and the scripts directories. There are two other subdirectories, data where the data files will be written, and logs directory where logs of the logr processes will be written.

cfgs

In the cfgs directory, there is one main configuration file named core_logging.ini. This file contains 'sections' that are delimited by a name contained in square brackets (e.g. [videologr]). When the logr executable is run, two arguments are passed. The first is the location of this core_logging.ini file and the second is the 'name' of the section to be used out of that configuration file. For example, to start a logr instance for the videologr running on the simulator, you would run:

/usr/local/bin/logr /home/ops/corelogging/sim/cfgs/core_logging.ini videologr

In each section of the core_logging.ini file, it tells the logr executable where to read the logging configuration from and where to write both resulting logs and data files to. It also includes a coredataBroadcastPort which is the UDP port the logger will publish messages to that are in the old datamanager format. Here is an example section of the core_logging.ini file:

##############################################################################
[videologr]
##############################################################################

#name of coredata root directory
coredataRoot = /home/ops/corelogging/sim

#coredata console output logs dirctory
coredataLogsDir = logs

#coredata cfg files directory
coredataCfgsDir = cfgs

#coredata data files directory
coredataDataDir = data

# name of logr items config file (in the cfgs dirctory)
coredataLogItemsCfg = videologr.cfg

#platform  may be used in forming filenames or tagging datafiles etc
#typically wfly or rcsn
coredataPlatform = sim

#Broadcast and client UDP port assignments
coredataBroadcastPort = 54004
clientConnectPort = 54005

The coredataLogItemsCfg property points to a .cfg file that is a simple text file which directs the logr executable on which items to subscribe to from LCM and how to publish those messages over UDP. Here is an example of the videolgr.cfg file:

################################################################################
# Data Logging Configuration
# Data logging is controlled by one of two modes:
#
# 1) UPDATE with a positive rate in seconds
#    and specifying all DM items on the list to be logged as 'Slave'
#
# 2) UPDATE with a positive rate in seconds
#    and specifying ** ONE ** a DM item on the
#    list to be logged as 'Trigger'
#
# 3) UPDATE may be followed by a positive
#    timeout value in seconds.  This can be used to force a read of all
#    of the DM items on the list at some interval if the trigger item
#    does not update withing the specified interval.  A value of 0.0
#    will be interpreted as disabling the timeout feature (default).
#

# ItemTypes must be one of: BOOL, INT, DOUBLE or STRING 

# MAX_AGE (in seconds): if payload timestamp exceeds MAX_AGE then set
# item values on output to NO_PUB  MaxAge = 0 skips the test.  Generally
# MAX_AGE should be set fairly large, its useful to detect a hung or crashed
# publisher

# Time = yes then pre-pend the item's timestamp
# Set# = yes the pre-pend the item's sequence number

# Allow_no_prov If the subscriber status is 'stale' we normally 
# set the item's fields to NO_PROV, you can overide this by setting Allow_no_provider
#

UPDATE 2

#    Channel       Type                getItemName     getItemType  Alias              Format   Time  Seq# UpdateMode MaxAge Allow_No_Prov
# ------------------------------------------------------------------------------------------------------------------------------------------
LOG  NMEA_GPS     lcm_nmeaGPS_sub      ZDA_USECS      Int          SHIP.GPS.TIME        %d      No    No    Slave      10     No
LOG  SEABIRD_CTD  lcm_seabirdctd_sub   INWATER        BOOL         ROV.CTD.INWATER      %d      No    No    Slave      30     Yes
LOG  LAPBOX_HD    lcm_lapbox_hd_sub    ZOOM           Double       HD_LAPBOX.ZOOM       %3.4lf  No    No    Slave      30     No
LOG  LAPBOX_HD    lcm_lapbox_hd_sub    FOCUS          Double       HD_LAPBOX.FOCUS      %3.4lf  No    No    Slave      30     No
LOG  LAPBOX_HD    lcm_lapbox_hd_sub    IRIS           Double       HD_LAPBOX.IRIS       %3.4lf  No    No    Slave      30     No
LOG  VENTANA_CSP  lcm_ventanaCSP_sub   PAN            Double       ROV.CAMERA.PAN       %lf     No    No    Slave      30     No
LOG  VENTANA_CSP  lcm_ventanaCSP_sub   TILT           Double       ROV.CAMERA.TILT      %lf     No    No    Slave      30     No
LOG  VENTANA_CSP  lcm_ventanaCSP_sub   SHOULDER       Double       ROV.CAMERA.SHOULDER  %lf     No    No    Slave      30     No

This file tells the logr executable to look for LCM messages on a specific channel, of a specific type, what the getItem name is (and it's data type) and which data manager alias to use when publishing to UDP.

scripts

In the scripts directory, there are several start and stop scripts:

startBdfServer: This is a script that looks for an existing process named bdfserver and if it's not found, it runs the bdfserver.pl script. This script starts up a UDP server listening on port 54022. When a message is received from a client on that port, the script runs the df command to check on disk space and then sends that result back to the client.
startLoggers: This script removes the current crontab and then installs the crontab specified in the my_cron_with_logging file. This crontab starts all the logr executables as well as the bdf server and ship position grabber
startLogr: This script can be used to start an individual logr executable.
startServer: This is a script that can be used to start a bcserver for a specific logr.
stopLoggers: This runs the stopLogr script on each logger that might be running and then installs the my_cron_no_logging crontab to replace the one that automatically restarts logrs.
stopLogr: Is a script that can be used to stop an individual logr instance.

End-to-End Message Flow

Another helpful perspective on this data management system is to look at it from the perspective of message and data flow through the system. In order to understand the flow of data items through the entire system, it's good to know where each item is defined and how it travels through the system all the way to the log files and Dataprobe. Below is an example of this for Ventana CSP messages.

Navproc To Logging/UDP

graph LR subgraph navproc-common subgraph lcm-types ventanaCSP_t.lcm["ventanaCSP_t.lcm
heading"] subgraph publishers lcm_ventanaCSP_pub.hpp end subgraph subscribers lcm_ventanaCSP_sub.hpp end end end subgraph navproc-process subgraph src subgraph ventanaCSP_pub ventanaCSP_frame.hpp ventanaCSP_pub.cpp end end end subgraph serial-ports ventanaCSP-serial-port["Ventana CSP Message"] --> ventanaCSP_pub.cpp end ventanaCSP_t.lcm --> ventanaCSP_frame.hpp ventanaCSP_pub.cpp --> ventanaCSP_frame.hpp ventanaCSP_pub.cpp --> ventanaCSP-lcm["Ventana CSP LCM
heading"] subgraph corelogging core_logging.ini --> logr dataprobelogr.cfg --> logr ventanaCSP-lcm --> logr logr --> 2021077dataprobelogr.dat logr --> translate["LCM:VENTANA_CSP:lcm_ventanaCSP_sub:HEADING->ROV.HEADING"] translate --> coredata-broadcast-port["coredataBroadcastPort(54008)
ROV.HEADING"] end

UDP to Dataprobe

graph LR subgraph bcserver-process coredata-broadcast-port["coredataBroadcastPort(54008)
ROV.HEADING"] --> bcserver bcserver --> client-connect-port["clientConnectPort(54009)
ROV.HEADING"] end subgraph dataprobe-client dataprobe-pref["~/Dataprobe/Resources/Dataprobe.pref"] --> sockShipNav["sockShipNav
ROV.HEADING->RovHeading.Value"] sockShipNav --> Dataprobe client-connect-port --> sockShipNav end

Logr Files to Shore

Please look at the documentation on ship and ROV data workflow for more details on this portion of the system.

Common Clients

Clients
Dataprobe: This is a XOJO application that allows the pilots to monitor data from the system by connecting to the bcserver and requesting data messages. The documentation for Dataprobe can be found here
DynamicUDPClient_XOJO: This is a XOJO application that can be used to monitor UDP messages in a dynamic way. Please consult the Dynamic UDP Client Documentation for more details.
SerialDataSimulator_XOJO: This is a XOJO application that can be used to simulate data that is pushed out over serials ports for the purposes of testing the navproc and datamanger systems.
SimpleUDPClient_XOJO: This is a bare-bones XOJO application that can be used to grab UDP messages and show them to the user. Very helpful for troubleshooting. Please consult the Simple UDP Client Documentation for more details.
DMtoAMQP (part of corelogging repo)
ArcMapNav: For more details on the ArcMapNav components, look at this documentation.