David Packard

Warning

This document is under construction

This page documents the Navproc, Logging and API systems on the David Packard

Overview

First, let's look at a high level diagram of the various components. There are basically four main components and they are:

Two Digi EZ32 terminal servers that take in physical connections from various ship and ROV signals
An Ubuntu 22 NUC that is running the current version of navproc/logr software that has the ROV control room TS32 ports mounted virtually as /dev/tty devices
An Ubuntu 24 virtual machine that is running the data distribution side of things (ZeroMQ, NATS, TimescaleDB, Grafana etc.)

--- title: Logical Components --- flowchart LR subgraph David Packard direction LR ship-signals[Various Ship and ROV Signals] dpkd-ts2["Digi EZ32
134.89.217.35
ROV Control Room"] navproc-dpkd["Navproc NUC
navproc-dpkd.dp.mbari.org
134.89.217.34"] coredata-dpkd["Navproc API
coredata-dpkd.dp.mbari.org
134.89.217.34"] end ship-signals --> dpkd-ts2 dpkd-ts2 -- TCP and RealPort --> navproc-dpkd navproc-dpkd --> coredata-dpkd

Data Flow

The data flow goes through several stages of processing as it travels through the system.

It is first generated by the data source (instrument)
The data is consumed by a navproc process through either virtual serial port, TCP socket or UDP datagram
The raw data is published as an LCM message
Most of the time, that message is then consumed by another navproc process which parses the data from the message and puts it into a navproc_t formatted LCM message and publishes it.
There are two consumers of these LCM messages: loggers and the LCM bridge
1. Logrs
  1. There are logr processes that are configured with the logr-config.json file
  2. Each logr can subscribe to multiple LCM channels and it builds log files in the /data directory using items from the various LCM channels
  3. In addition, the logr process creates it's own LCM message that has the 'header' item and 'log_line' item that is publishes to
  4. There is a Python process that lives on Navproc called bcserver, that reads in the logr header and log_line, caches the most recent entries and then listens on UDP ports (different port for each logr).
2. LCM Bridge
  1. There is a process named lcm-bridge-controller that reads configuration from lcm-bridge-controller.json
  2. The configuration file can be used to send JSON formatted LCM messages to various 'bridges'. The currently supported bridges are Sockets, ZMQ, MQTT, Redis, and NATS.
API
1. NATS to TSDB: This is a configurable process that subscribes to NATS messages and then inserts the data into a TimescaleDB table.
Navproc
1. LCM Channel Name (this is defined in the navproc.ini file)
2. Item Name
3. Item Type
Logr (logr-config.json file)
1. Logr alias
ZMQ topic is a three part message
1. Part one: LCM channel name
2. Part two: message type (navproc/msg_t)
3. Part three: LCM message converted to JSON
NATS Message Subject: Payload is message converted to JSON
NATS Store Key Alias
Redis has three databases
1. ?
2. ?
3. Aliased data (logr aliases)
TimescaleDB Table: Name = LCM Channel (this is defined in the navproc-api codebase in the nats-to-tsdb/config.json file)
TimescaleDB Column: item name

AUX_CAMERA_CMD

Currently, this channel is not logged or sent out, it's just part of the control system

Item Name	Type	Logr Item (logr-config.json)	NATS Bucket Key (lcm-bridge-config.json)	TimescaleDB Column (nats-to-tsdb/config.json)
pan_cmd	real	NA	NA	NA
pan_deg	real	NA	NA	NA
tilt_cmd	real	NA	NA	NA
tilt_deg	real	NA	NA	NA

CAMERA_4K_MSG

Full JSON Payload

ZMQ topic: CAMERA_4K_MSG:navproc/msg_t
NATS subject: navproc.CAMERA_4K_MSG
TimescaleDB Table: navproc_camera_4k_msg

Individual Items

Item Name	Type	Logr Item (logr-config.json)	NATS Bucket Key (lcm-bridge-config.json)	TimescaleDB Column (nats-to-tsdb/config.json)
focus_volts	real	CAMERA_4K.FOCUS	CAMERA.4K.FOCUS	focus_volts
humidity	real	CAMERA_4K.HUMIDITY	CAMERA.4K.HUMIDITY	humidity
iris_volts	real	CAMERA_4K.IRIS	CAMERA.4K.IRIS	iris_volts
leak_detect	boolean	CAMERA_4K.LEAK	CAMERA.4K.LEAK	leak_detect
pressure	real	CAMERA_4K.PRESSURE	CAMERA.4K.PRESSURE	pressure
sentence	string	NA	CAMERA.4K.MSG.RAW	NA
serial_port_name	string	NA	CAMERA.4K.SERIAL_PORT_NAME	NA
temperature	real	CAMERA_4K.TEMPERATURE	CAMERA.4K.TEMP	temperature
zoom_volts	real	CAMERA_4K.ZOOM	CAMERA.4K.ZOOM	zoom_volts

CTD_DEPTH_MSG

Full JSON Payload: Currently only being logged, not sent out yet.

Individual Items

Item Name	Type	Logr Item (logr-config.json)
depth	real	ROV.CTD.DEPTH
depth_rate	real	NA
latitude_age	real	NA

DIGIQUARTZ_MSG

Full JSON Payload

ZMQ topic: DIGIQUARTZ_MSG:navproc/msg_t
NATS subject: navproc.DIGQUARTZ_MSG
TimescaleDB Table: N/A

Individual Items

Item Name	Type	Logr Item (logr-config.json)	NATS Bucket Key (lcm-bridge-config.json)	TimescaleDB Column (nats-to-tsdb/config.json)
pressure_dbar	real	ROV.DIGIQ.SCALED_PRESSURE	ROV.DIGIQ.SCALED_PRESSURE	NA
pressure_raw	real	NA	ROV.DIGIQ.RAW_PRESSURE	NA
sentence	string	NA	ROV.DIGIQ.MSG.RAW	NA

DIGIQUARTZ_SERIN

This is currently not logged or sent out.

Individual Items

Item Name	Type	Logr Item (logr-config.json)	NATS Bucket Key (lcm-bridge-config.json)	TimescaleDB Column (nats-to-tsdb/config.json)	Notes
sentence	string	NA	NA	NA

IMU_OCTANS_MSG

Full JSON Payload

ZMQ topic: IMU_OCTANS_MSG:navproc/msg_t
NATS subject: navproc.IMU_OCTANS_MSG
TimescaleDB Table: N/A

This is currently not being logged (we get data from the DVECS messages from this, but we don't currently log the raw)

Individual Items

Item Name	Type	Logr Item (logr-config.json)	NATS Bucket Key (lcm-bridge-config.json)
data_frame	blob	NA	NA
heading_deg	real	NA	ROV.OCTANS.HEADING_DEG
heading_deg_sec	real	NA	ROV.OCTANS.HEADING_DEG_SEC
parse_err_cnt	integer	NA	ROV.OCTANS.PARSE_ERR_CNT
pitch_deg	real	NA	ROV.OCTANS.PITCH_DEG
pitch_deg_sec	real	NA	ROV.OCTANS.PITCH_DEG_SEC
roll_deg	real	NA	ROV.OCTANS.ROLL_DEG
roll_deg_sec	real	NA	ROV.OCTANS.ROLL_DEG_SEC
serial_port_name	string	NA	NA
status	integer	NA	ROV.OCTANS.STATUS
x_meters_sec2	real	NA	ROV.OCTANS.X_METERS_SEC2
y_meters_sec2	real	NA	ROV.OCTANS.Y_METERS_SEC2
z_meters_sec2	real	NA	ROV.OCTANS.Z_METERS_SEC2

LAPBOX_4K_MSG

Note: Not being logged with logrs.

Full JSON Payload:

ZMQ topic: LAPBOX_4K_MSG:navproc/msg_t
NATS subject: navproc.LAPBOX_4K_MSG
TimescaleDB Table: navproc_lapbox_4k_msg

Individual Items

Item Name	Type	NATS Bucket Key (lcm-bridge-config.json)	TimescaleDB Column (nats-to-tsdb/config.json)
aux_cam_btn_cnt	integer	LAPBOX.4K.AUX_CAM_BTN_CNT	aux_cam_btn_cnt
aux_cam_btn_val	boolean	LAPBOX.4K.AUX_CAM_BTN_VAL	aux_cam_btn_val
main_cam_btn_cnt	integer	LAPBOX.4K.MAIN_CAM_BTN_CNT	main_cam_btn_cnt
main_cam_btn_val	boolean	LAPBOX.4K.MAIN_CAM_BTN_VAL	main_cam_btn_val
pan_cmd	real	LAPBOX.4K.PAN_CMD	pan_cmd
pan_raw	real	LAPBOX.4K.PAN_RAW	pan_raw
port_light_btn_cnt	integer	LAPBOX.4K.PORT_LIGHT_BTN_CNT	port_light_btn_cnt
port_light_btn_val	boolean	LAPBOX.4K.PORT_LIGHT_BTN_VAL	port_light_btn_val
serial_port_name	string	LAPBOX.4K.SERIAL_PORT_NAME
stbd_light_btn_cnt	integer	LAPBOX.4K.STBD_LIGHT_BTN_CNT	stbd_light_btn_cnt
stbd_light_btn_val	boolean	LAPBOX.4K.STBD_LIGHT_BTN_VAL	stbd_light_btn_val
tilt_cmd	real	LAPBOX.4K.TILT_CMD	tilt_cmd
tilt_raw	real	LAPBOX.4K.TILT_RAW	tilt_raw
tilt_swap_btn_cnt	integer	LAPBOX.4K.TILT_SWAP_BTN_CNT	tilt_swap_btn_cnt
tilt_swap_btn_val	boolean	LAPBOX.4K.TILT_SWAP_BTN_VAL	tilt_swap_btn_val
tilt_swap_enabled	boolean	LAPBOX.4K.TILT_SWAP_ENABLED	tilt_swap_enabled

LAPBOX_MIO_UDPOUT

Not being logged or sent out

Individual Items

Item Name	Type	Logr Item (logr-config.json)	NATS Bucket Key (lcm-bridge-config.json)	TimescaleDB Column (nats-to-tsdb/config.json)	Notes
sentence	string	NA	NA	NA

MAIN_CAMERA_CMD

Currently, this channel is not logged or sent out, it's just part of the control system

Item Name	Type	Logr Item (logr-config.json)	NATS Bucket Key (lcm-bridge-config.json)	TimescaleDB Column (nats-to-tsdb/config.json)
pan_cmd	real	NA	NA	NA
pan_deg	real	NA	NA	NA
tilt_cmd	real	NA	NA	NA
tilt_deg	real	NA	NA	NA

MBARI_DEPTH_MSG

Full JSON Payload

ZMQ Topic: MBARI_DEPTH_MSG:navproc/msg_t
NATS subject: navproc.MBARI_DEPTH_MSG
TimescaleDB Table: N/A

Individual Items

Item Name	Type	Logr Item (logr-config.json)	NATS Bucket Key (lcm-bridge-config.json)	TimescaleDB Column (nats-to-tsdb/config.json)
depth	real	ROV.MBARI.DEPTH	ROV.MBARI.DEPTH	NA
depth_rate	real	NA	ROV.MBARI.DEPTH_RATE	NA
latitude_age	real	NA	ROV.MBARI.LATITUDE_AGE	NA

NAV_4D_MSG

Full JSON Payload

ZMQ Topic: NAV_4D_MSG:navproc/msg_t
NATS subject: navproc.NAV_4D_MSG
TimescaleDB Table: navproc_nav_4d_msg

Individual Items

Item Name	Type	Logr Item (logr-config.json)	NATS Bucket Key (lcm-bridge-config.json)	TimescaleDB Column (nats-to-tsdb/config.json)
fix_time	string	NA	NA	fix_time
gga_diff_mode	boolean	NA	ROV.GPS.DIFFERENTIAL	gga_diff_mode
gga_fix_age_seconds	real	NA	ROV.GPS.GGS.FIX_AGE	gga_fix_age_seconds
gga_hdop	real	NA	ROV.GPS.HDOP	gga_hdop
gga_nsats	integer	NA	ROV.GPS.NSATS	gga_nsats
gga_quality	integer	NA	ROV.GPS.QUALITY	gga_quality
gll_data_valid	boolean	NA	ROV.GLL.DATA_VALID	gll_data_valid
gll_fix_age_seconds	real	NA	ROV.GPS.GLL.FIX_AGE	gll_fix_age_seconds
latitude	real	ROV.POSITION.LAT	ROV.POSITION.LAT	latitude
longitude	real	ROV.POSITION.LON	ROV.POSITION.LON	longitude
sentence	string	NA		sentence
zda_date_time	string	NA	NA	zda_date_time
zda_fix_age_seconds	real	NA	ROV.GPS.ZDA.FIX_AGE	zda_fix_age_seconds
zda_utc_seconds	real	ROV.POSITION.USECS	ROV.GPS.TIME	zda_utc_seconds

NAV4D_UDPIN

Full JSON Payload

ZMQ Topic: Not current sending
NATS Subject: navproc.NAV4D_UDPIN

Individual Items

Item Name	Type	Logr Item (logr-config.json)	NATS Bucket Key (lcm-bridge-config.json)	TimescaleDB Column (nats-to-tsdb/config.json)	Notes
sentence	string	NAV4D.GPS.RAW and ROV.POSITION.NAV4D.STRING	NAV4D.GPS.RAW	NA

NMEA_GPS_MSG

Full JSON Payload

ZMQ Topic: NMEA_GPS_MSG:navproc/msg_t
NATS subject: navproc.NMEA_GPS_MSG
TimescaleDB Table: navproc_nmea_gps_msg

Individual Items

Item Name	Type	Logr Item (logr-config.json)	NATS Bucket Key (lcm-bridge-config.json)	TimescaleDB Column (nats-to-tsdb/config.json)
fix_time	string	NA	NA	fix_time
gga_diff_mode	boolean	NA	ROV.GPS.DIFFERENTIAL	gga_diff_mode
gga_fix_age_seconds	real	NA	ROV.GPS.GGS.FIX_AGE	gga_fix_age_seconds
gga_hdop	real	NA	ROV.GPS.HDOP	gga_hdop
gga_nsats	integer	NA	SHIP.GPS.NSATS	gga_nsats
gga_quality	integer	SHIP.GPS.QUALITY	SHIP.GPS.QUALITY	gga_quality
gll_data_valid	boolean	NA	ROV.GLL.DATA_VALID	gll_data_valid
gll_fix_age_seconds	real	NA	ROV.GPS.GLL.FIX_AGE	gll_fix_age_seconds
latitude	real	ROV.POSITION.LAT	ROV.POSITION.LAT	latitude
longitude	real	ROV.POSITION.LON	ROV.POSITION.LON	longitude
sentence	string	NA	NA	sentence
zda_date_time	string	NA	NA	zda_date_time
zda_fix_age_seconds	real	NA	ROV.GPS.ZDA.FIX_AGE	zda_fix_age_seconds
zda_utc_seconds	real	ROV.POSITION.USECS	ROV.GPS.TIME	zda_utc_seconds

NMEA_GPS_TCPIN

Full JSON Payload

ZMQ Topic: Not current sending
NATS Subject: navproc.NMEA_GPS_TCPIN

Individual Items

Item Name	Type	Logr Item (logr-config.json)	NATS Bucket Key (lcm-bridge-config.json)	TimescaleDB Column (nats-to-tsdb/config.json)	Notes
sentence	string	SHIP.GPS.RAW	SHIP.GPS.RAW

ROV_UHS_MSG

Full JSON Payload (currently not encoding the raw binary data as blob. This needs to be done still)

ZMQ Topic: ROV_UHS_MSG:navproc/msg_t
NATS subject: navproc.ROV_UHS_MSG

Individual Items

Item Name	Type	Logr Item (logr-config.json)	NATS Bucket Key (lcm-bridge-config.json)	TimescaleDB Column (nats-to-tsdb/config.json)
alarm_flags	integer	ROV.UHS.ALARM_FLAGS	ROV.UHS.ALARM_FLAGS	alarm_flags
ctd_set_length	real	ROV.UHS.CTD.SET_LENGTH	ROV.UHS.CTD_SET_LENGTH	ctd_set_length
ctd_tension	real	ROV.UHS.CTD.TENSION	ROV.UHS.CTD_TENSION	ctd_tension
ctd_wire_length	real	ROV.UHS.CTD.WIRE_LENGTH	ROV.UHS.CTD_WIRE_LENGTH	ctd_wire_length
ctd_wire_speed	real	ROV.UHS.CTD.WIRE_SPEED	ROV.UHS.CTD_WIRE_SPEED	ctd_wire_speed
data_frame	blob	NA
drive_alarm	integer	ROV.UHS.DRIVE_ALARM	ROV.UHS.DRIVE_ALARM	drive_alarm
rov_set_length	real	ROV.UHS.ROV.SET_LENGTH	ROV.UHS.ROV_SET_LENGTH	rov_set_length
rov_tension	real	ROV.UHS.ROV.TENSION	ROV.UHS.ROV_TENSION	rov_tension
rov_wire_length	real	ROV.UHS.ROV.WIRE_LENGTH	ROV.UHS.ROV_WIRE_LENGTH	rov_wire_length
rov_wire_speed	real	ROV.UHS.ROV.WIRE_SPEED	ROV.UHS.ROV_WIRE_SPEED	rov_wire_speed
winch_mode	integer	ROV.UHS.WINCH_MODE	ROV.UHS.WINCH_MODE	winch_mode

SEABIRD_CTD_MSG

Full JSON Payload

ZMQ Topic: SEABIRD_CTD_MSG:navproc/msg_t
NATS subject: navproc.SEABIRD_CTD_MSG
TimescaleDB Table: navproc_seabird_ctd_msg

Individual Items

Item Name	Type	Logr Item (logr-config.json)	NATS Bucket Key (lcm-bridge-config.json)	TimescaleDB Column (nats-to-tsdb/config.json)	Notes
analog1	real	ROV.CTD.ANALOG1	ROV.CTD.ANALOG1	analog1
analog2	real	ROV.CTD.ANALOG2	ROV.CTD.ANALOG2	analog2
analog3	real	ROV.CTD.ANALOG3	ROV.CTD.ANALOG3	analog3
analog4	real	ROV.CTD.ANALOG4	ROV.CTD.ANALOG4	analog4
conductivity	real	ROV.CTD.CONDUCTIVITY	ROV.CTD.CONDUCTIVITY	conductivity
in_water	boolean	ROV.CTD.INWATER	ROV.CTD.INWATER	in_water
pressure	real	ROV.CTD.PRESSURE	ROV.CTD.PRESSURE	pressure
receiving_data	boolean	ROV.CTD.RECEIVING_DATA	ROV.CTD.RECEIVING_DATA	receiving_data	This is also available in SEABIRD_CTD_STAT
sound_velocity	real	NA	ROV.CTD.SOUND_VELOCITY	sound_velocity
temperature	real	ROV.CTD.TEMPERATURE	ROV.CTD.TEMPERATURE	temperature

SEABIRD_CTD_SERIN

Full JSON Payload

ZMQ Topic: Not being sent currently
NATS subject: navproc.SEABIRD_CTD_SERIN
TimescaleDB Table: N/A

Item Name	Type	Logr Item (logr-config.json)	NATS Bucket Key (lcm-bridge-config.json)	TimescaleDB Column (nats-to-tsdb/config.json)	Notes
sentence	string	ROV.CTD.RAW	SEABIRD.CTD.RAW	NA	This will likely change to SEABIRD_CTD_TCPIN

SEABIRD_CTD_STAT

Full JSON Payload

ZMQ Topic: SEABIRD_CTD_STAT:navproc/msg_t
NATS subject: navproc.SEABIRD_CTD_STAT
TimescaleDB Table: N/A

Item Name	Type	Logr Item (logr-config.json)	NATS Bucket Key (lcm-bridge-config.json)	TimescaleDB Column (nats-to-tsdb/config.json)	Notes
receiving_data	boolean	ROV.CTD.RECEIVING_DATA	ROV.CTD.RECEIVING_DATA	NA	This is also available in SEABIRD_CTD_MSG

SEAPATH_MSG

Full JSON Payload

ZMQ Topic: SEAPATH_MSG:navproc/msg_t
NATS subject: navproc.SEAPATH
TimescaleDB Table: navproc_seapath_msg

Individual Items

Item Name	Type	Logr Item (logr-config.json)	NATS Bucket Key (lcm-bridge-config.json)	TimescaleDB Column (nats-to-tsdb/config.json)
horiz_qual	integer	SHIP.SEAPATH.HORIZ_QUAL	SHIP.SEAPATH.HORIZ_QUAL	horiz_qual
hgt_qual	integer	SHIP.SEAPATH.HGT_QUAL	SHIP.SEAPATH.HGT_QUAL	hgt_qual
head_qual	integer	SHIP.SEAPATH.HEAD_QUAL SHIP.SEAPATH.HEAD_QUAL	head_qual
rp_qual	integer	SHIP.SEAPATH.RP_QUAL	SHIP.SEAPATH.RP_QUAL	rp_qual
roll_deg	real	SHIP.SEAPATH.ROLL_DEG	SHIP.SEAPATH.ROLL_DEG	roll_deg
pitch_deg	real	SHIP.SEAPATH.PITCH_DEG	SHIP.SEAPATH.PITCH_DEG	pitch_deg
heading_deg	real	SHIP.SEAPATH.HEADING_DEG	SHIP.SEAPATH.HEADING_DEG	heading_deg
heave_meters	real	SHIP.SEAPATH.HEAVE_METERS	SHIP.SEAPATH.HEAVE_METERS	heave_meters
psxn20_cs_valid	boolean	SHIP.SEAPATH.PSXN20_CS_VALID	SHIP.SEAPATH.PSXN20_CS_VALID	psxn20_cs_valid
psxn23_cs_valid	boolean	SHIP.SEAPATH.PSXN23_CS_VALID	SHIP.SEAPATH.PSXN23_CS_VALID	psxn23_cs_valid

SEAPATH_TCPIN

Full JSON Payload

ZMQ Topic: Not current sending
NATS Subject: Not current sending
TimescaleDB: N/A

Individual Items

Item Name	Type	Logr Item (logr-config.json)	NATS Bucket Key (lcm-bridge-config.json)	TimescaleDB Column (nats-to-tsdb/config.json)	Notes
sentence	string	NA	NA

SHIP_GYRO_MSG

Full JSON Payload

ZMQ Topic: SHIP_GYRO_MSG:navproc/msg_t
NATS subject: navproc.SHIP_GYRO_MSG
TimescaleDB Table: navproc_ship_gyro_msg

Individual Items

Item Name	Type	Logr Item (logr-config.json)	NATS Bucket Key (lcm-bridge-config.json)	TimescaleDB Column (nats-to-tsdb/config.json)
heading	real	SHIP.GYRO.DEGREES	SHIP.GYRO.DEGREES	heading
heading_age	real	NA	SHIP.GYRO.HEADING_AGE	heading_age
heading_rate	real	NA	SHIP.GYRO.HEADING_RATE	heading_rate
heading_rate_age	real	NA	SHIP.GYRO.HEADING_RATE_AGE	heading_rate_age
sentence	string	NA	NA

SHIP_GYRO_TCPIN

Full JSON Payload

ZMQ Topic: Not currently sending
NATS Subject: Not currently sending

Individual Items

Item Name	Type	Logr Item (logr-config.json)	NATS Bucket Key (lcm-bridge-config.json)	TimescaleDB Column (nats-to-tsdb/config.json)	Notes
sentence	string	NA	NA

TO BE DOCUMENTED

LCM Channel	Item Name	Type	Logr Item	Notes
OVERLAY_MSG	sentence	string
PORT_LIGHT_CMD	pan_cmd	real
PORT_LIGHT_CMD	tilt_cmd	real
PT_CONTROL_LAPBOX_4K	aux_cam_selected	boolean
PT_CONTROL_LAPBOX_4K	main_cam_selected	boolean
PT_CONTROL_LAPBOX_4K	port_light_selected	boolean
PT_CONTROL_LAPBOX_4K	stbd_light_selected	boolean
PT_CONTROL_LAPBOX_MIO	aux_cam_selected	boolean
PT_CONTROL_LAPBOX_MIO	main_cam_selected	boolean
PT_CONTROL_LAPBOX_MIO	port_light_selected	boolean
PT_CONTROL_LAPBOX_MIO	stbd_light_selected	boolean
PT_CONTROL_PILOT	aux_cam_selected	boolean
PT_CONTROL_PILOT	main_cam_selected	boolean
PT_CONTROL_PILOT	port_light_selected	boolean
PT_CONTROL_PILOT	stbd_light_selected	boolean
PT_CONTROL_STAT	aux_cam_src	string
PT_CONTROL_STAT	main_cam_src	string
PT_CONTROL_STAT	port_light_src	string
PT_CONTROL_STAT	stbd_light_src	string
PMBAAC_SENTENCE	comms_watchdog	integer
PMBAAC_SENTENCE	auto_mode_flags	integer
PMBAAC_SENTENCE	x_effort	real
PMBAAC_SENTENCE	y_effort	real
PMBAAC_SENTENCE	z_effort	real
PMBAAC_SENTENCE	yaw_effort	real
PMBAAC_SENTENCE	sentence	string
PMBANP_SENTENCE	sentence	string
PMBAPT_SENTENCE	sentence	string
PSMDPT_MSG	remote_control_flags	integer
PSMDPT_MSG	chair_select_flags	integer
PSMDPT_MSG	aux_cam_btn_cnt	integer
PSMDPT_MSG	main_cam_btn_cnt	integer
PSMDPT_MSG	port_light_btn_cnt	integer
PSMDPT_MSG	stbd_light_btn_cnt	integer
PSMDPT_MSG	port_light_pan	real
PSMDPT_MSG	port_light_tilt	real
PSMDPT_MSG	stbd_light_pan	real
PSMDPT_MSG	stbd_light_tilt	real
PSMDPT_MSG	science_cam_pan	real
PSMDPT_MSG	science_cam_tilt	real
PSMDPT_MSG	pilot_cam_pan	real
PSMDPT_MSG	pilot_cam_tilt	real
PSMDPT_MSG	joystick_left_right	real
PSMDPT_MSG	joystick_up_down	real
PSMDPT_MSG	pan_cmd	real
PSMDPT_MSG	tilt_cmd	real
PSMDPT_MSG	checksum_valid	boolean
PSMDPT_MSG	aux_cam_btn_val	boolean
PSMDPT_MSG	main_cam_btn_val	boolean
PSMDPT_MSG	port_light_btn_val	boolean
PSMDPT_MSG	stbd_light_btn_val	boolean
PSMDPT_MSG	sentence	string
PSMDNP_MSG	depth_source	integer
PSMDNP_MSG	altitude_source	integer
PSMDNP_MSG	attitude_source	integer
PSMDNP_MSG	auto_mode_flags	integer
PSMDNP_MSG	dive_number	integer
PSMDNP_MSG	roll_degs	real
PSMDNP_MSG	pitch_degs	real
PSMDNP_MSG	heading_degs	real	ROV.DVECS.HEADING
PSMDNP_MSG	altitude_meters	real
PSMDNP_MSG	depth_pressure_psi	real
PSMDNP_MSG	hydraulic_pressure_bar	real
PSMDNP_MSG	turns_count	real
PSMDNP_MSG	auto_head_setpoint	real
PSMDNP_MSG	auto_altitude_setpoint	real
PSMDNP_MSG	auto_depth_setpoint	real
PSMDNP_MSG	observation_mode	real
PSMDNP_MSG	checksum_valid	boolean
PSMDNP_MSG	date	string
PSMDNP_MSG	sentence	string
PSMDAC_MSG	comms_watchdog	integer
PSMDAC_MSG	depth_source	integer
PSMDAC_MSG	altitude_source	integer
PSMDAC_MSG	attitude_source	integer
PSMDAC_MSG	auto_mode_flags	integer
PSMDAC_MSG	roll_degs	real
PSMDAC_MSG	pitch_degs	real
PSMDAC_MSG	heading_degs	real
PSMDAC_MSG	altitude_meters	real
PSMDAC_MSG	depth_meters	real
PSMDAC_MSG	turns_count	real
PSMDAC_MSG	auto_head_setpoint	real
PSMDAC_MSG	auto_altitude_setpoint	real
PSMDAC_MSG	auto_depth_setpoint	real
PSMDAC_MSG	observation_mode	real
PSMDAC_MSG	pilot_effort_x	real
PSMDAC_MSG	pilot_effort_y	real
PSMDAC_MSG	pilot_effort_z	real
PSMDAC_MSG	pilot_effort_yaw	real
PSMDAC_MSG	summed_effort_x	real
PSMDAC_MSG	summed_effort_y	real
PSMDAC_MSG	summed_effort_z	real
PSMDAC_MSG	summed_effort_yaw	real
PSMDAC_MSG	checksum_valid	boolean
PSMDAC_MSG	sentence	string
STBD_LIGHT_CMD	pan_cmd	real
STBD_LIGHT_CMD	tilt_cmd	real
WIND_SENSOR_MSG	direction	real	SHIP.WIND.TRUE_DIRECTION	Looks like the old item name was true_direction and new is just direction. This may be a problem
WIND_SENSOR_MSG	reference	string	NA
WIND_SENSOR_MSG	speed	real	SHIP.WIND.TRUE_SPEED	Looks like the old item name was true_speed and new is just speed. This may be a problem
WIND_SENSOR_MSG	speed_units	string	NA
WIND_SENSOR_MSG	status	string	NA
WIND_SENSOR_MSG	sentence	string	NA
WIND_SENSOR_TCPIN	sentence	string	SHIP.WIND.RAW
dataprobelogr	header	string
dataprobelogr	log_line	string
gpsrawlogr	header	string
gpsrawlogr	log_line	string
gyrorawlogr	header	string
gyrorawlogr	log_line	string
m3rslogr	header	string
m3rslogr	log_line	string
nav4dlogr	header	string
nav4dlogr	log_line	string
nav4drawlogr	header	string
nav4drawlogr	log_line	string
pantiltlogr	header	string
pantiltlogr	log_line	string
rovctdlogr	header	string
rovctdlogr	log_line	string
shipnavlogr	header	string
shipnavlogr	log_line	string
videologr	header	string
videologr	log_line	string

Installation and Setup

Before starting the installation writeup, it's important to know that for the Packard, IS set up a dedicated subnet for the navproc related systems. Navproc owns the .217.XXX subnet and we have about 30 dedicated addresses to use in the network.

Server Room Terminal Server

Doug Conlin set up and configured this terminal server and wired the signals from the signal distribution box to the terminal server.

ROV Control Room Terminal Server

Drew installed the terminal server in the rack and powered it up.
It was connected to a network switch and Adriana configured the switch port to be on the .217.XXX subnet
I used the Windows Digi Navigator to find it on the network.
We first set the IP address and gave it a static IP of 134.89.217.35
I used the Navigator utility to first push the configuration to enable RealPort on all 32 ports just to get started. I think this can be done through the web interface, but it was just easy to do it here.
I logged into the web interface using the 'admin' login and the password that is printed on the label.
I then changed the password (talk to Kevin Gomes or Mike Risi to get the password)
I then needed to connect the signals coming from the signal distribution box to the terminal server. There is a small DB9 panel on the rear side of one of the ROV control racks, at the very bottom. From left to right, the first 9 ports are labeled 40438, 40458, 40431, 40441, 40446, 40451, 40434, 40454, 40416.
1. From what I could gather, the signals were as follows:
  1. 40438: Wind: 4800 baud (This is currently not wired correctly, they are not wind, look like water velocity)
  2. 40458: Wind: 4800 baud (This is currently not wired correctly, they are not wind, look like water velocity)
  3. 40431: Ship GPS: 9600 baud, RS 422
  4. 40441: Nothing that I could see
  5. 40446: Ship GPS: 9600 baud, RS 422
  6. 40451: Ship GPS: 9600 baud, RS 422
  7. 40434: Ship Gyro: 38400 baud
  8. 40454: Ship Gyro: 38400 baud
  9. 40416: Ship GPS: 9600 baud, RS 422 (I think, did not verify)
2. So, I connected the first DB9 (40438) to Port 8 on the terminal server which is where navproc expects it.
3. Then, I connected the third DB9 (40431) to Port 9 on the terminal server
4. Then, I connected the 7th DB9 (40434) to Port 10 on the terminal server
5. Lastly, I connected the last open serial port on the Sonardyne to Port 11 on the terminal server
6. Then through the Digi web interface, I configured the ports the following way:
  1. For Port 9 (Ship GPS)
    1. I changed the mode from RealPort to Remote Access
    2. Changed the signal type from RS-232 to RS-422
    3. Under Service Settings, I enabled TCP on port 4009
    4. Under Service Settings, I changed the connection type to Raw TCP Connection
    5. Under Session Settings, I enabled Serial Port Sharing
  2. For Port 10 (Ship Gyro)
    1. I changed the mode from RealPort to Remote Access
    2. Under Serial Settings, I changed the baud rate to 38400
    3. Under Service Settings, I enabled TCP on port 4010
    4. Under Service Settings, I changed the connection type to Raw TCP Connection
    5. Under Session Settings, I enabled Serial Port Sharing
  3. For Port 11 (Sonardyne)
    1. I changed the mode from RealPort to Remote Access
    2. Under Service Settings, I enabled TCP on port 4011
    3. Under Service Settings, I changed the connection type to Raw TCP Connection
    4. Under Session Settings, I enabled Serial Port Sharing

navproc-dpkd

See the computer setup instructions for specific details on this setup. One thing that was different than the stock setup instructions is that I ran the terminal port server add a little differently since it has 32 ports. The command I ran was sudo dgrp_cfg_node init -v -e none a 134.89.217.35 32 which created virtual serial ports /dev/ttya00 -> /dev/ttya31.

coredata-dpkd

This is the main server for the Navproc API (and other applications). This is a server in the VM cluster that will act as an API server for the data coming from navproc. This API consists of a ZMQ proxy, a NATS cluster, a UDP proxy that replicates the old bcserver, a TimescaleDB server and a Grafana server. Here is a basic diagram of the API services and data flow.

--- title: Navproc API --- flowchart LR subgraph David Packard Navproc API Server direction LR subgraph navproc navproc-process-1 navproc-process-2 logr-process-1 logr-process-2 lcm-bridge subgraph logger files navproc-process-1-log-file[Logger File 1] navproc-process-2-log-file[Logger File 2] end bcserver(bcserver.py) np-port54003(54003) np-port54005(54005) np-port54007(54007) np-port54009(54009) np-port54010(54010
DiskUsage) np-port54017(54017) np-port54019(54019) end subgraph navproc-api port5555(5555) zmq-proxy[ZMQ Proxy
Python] port5556(5556) port4222in(4222) subgraph nats-cluster nats-server-1 nats-server-2 nats-server-3 end port4222out(4222) port8222html(8222-http) port8080ws(8080-web-sockets) nats-udp-proxy[NATS UDP Proxy
Python] port54003(54003) port54005(54005) port54007(54007) port54009(54009) port54010(54010
DiskUsage) port54017(54017) port54019(54019) grafana nats-to-tsdb[NATS to TimescaleDB
Python] port5432in(5432) timescaledb[TimescaleDB] position-to-odss[Position To ODSS
Python] pull-logr-files[Pull Logr File Process] subgraph "/data/logr/YYYY" navproc-process-1-log-file-gzip[Logger File 1 GZip] navproc-process-2-log-file-gzip[Logger File 2 GZip] end end end subgraph Shore messaging[AMQP Server
messaging.shore.mbari.org] subgraph coredata8 navprocessing-directory["~/shipandrov/navprocessing/rcsn-sim/todo/YYYYDDDshipnavlogr.dat.gz"] mbari-directory["/mbari/ShipData/logger/YYYY/rcsn-sim/YYYYDDDDxxxxxlogr.dat.gz"] end subgraph Thalassa ship-data-share["ShipData://logger/YYYY/rcsn-sim/YYYYDDDxxxxxlogr.dat.gz"] end end navproc-process-1 --> logr-process-1 navproc-process-1 --> lcm-bridge logr-process-1 --> lcm-bridge logr-process-1 --> navproc-process-1-log-file navproc-process-2 --> logr-process-2 navproc-process-2 --> lcm-bridge navproc-process-1 --> bcserver navproc-process-2 --> bcserver bcserver --> np-port54003 bcserver --> np-port54005 bcserver --> np-port54007 bcserver --> np-port54009 bcserver --> np-port54010 bcserver --> np-port54017 bcserver --> np-port54019 logr-process-2 --> lcm-bridge logr-process-2 --> navproc-process-2-log-file lcm-bridge --> port5555 --> zmq-proxy --> port5556 lcm-bridge --> port4222in --> nats-cluster --> port4222out nats-cluster --> port8222html nats-cluster --> port8080ws port4222out --> nats-udp-proxy timescaledb --> grafana nats-udp-proxy --> port54003 nats-udp-proxy --> port54005 nats-udp-proxy --> port54007 nats-udp-proxy --> port54009 nats-udp-proxy --> port54010 nats-udp-proxy --> port54017 nats-udp-proxy --> port54019 port4222out --> position-to-odss --> messaging port4222out --> nats-to-tsdb --> port5432in --> timescaledb navproc-process-1-log-file --> pull-logr-files --> navproc-process-1-log-file-gzip --> mbari-directory --> ship-data-share navproc-process-2-log-file --> pull-logr-files --> navproc-process-2-log-file-gzip --> mbari-directory --> ship-data-share navproc-process-1-log-file-gzip --> navprocessing-directory

I submitted a ticket to IS to create a new VM coredata-dpkd.dp.mbari.organd here is the original request

Details
- Request_Submitted_By: kgomes@mbari.org
- VM_Name: see comments
- VM_Purpose: This will be the coredata server on the David Packard
- VM_Expiration: 5 years
- VM_Support: IS_Supported
- VM_Support_Alt_Adm: 
- VM_OS: Ubuntu 24.04 LTS
- CPU: 2
- CPU_Addl_Reason: 
- RAM: 4
- RAM_Addl_Reason: 
- GPU_Extra: 
- GPU: NO
- Disk_Extra: I know this still isn't defined exactly, but just like on the Carson, it would be great to add a large, fast section of storage for the database. Should be a similar set up like we have on coredata-rcsn.rc.mbari.org
- Network: David Packard
- Resource_Priority: High
- Resource_Priority_Reason: 
- Conf_Firewall: 4222, 5432, 5555, 5556, 8080, 8222, 54000->54050
- Conf_Desktop_Extra: 
- Conf_Desktop: NO
- Conf_Logins: local
- Conf_Docker: YES
- Conf_Docker_Extra: Should be usable by both 'kgomes' and 'ops' accounts
- Conf_WebServer: nginx
- Conf_sudo: Like on other systems, kgomes and ops should have sudo.
- Conf_vCenter_Access: None
- VM_Comments: This VM will be doing the exact same thing as 'coredata-rcsn.rc.mbari.org' on the Carson so it should be set up very similarly. I don't think I need remote desktop on this, but if it's something that is hard to add later, you can put it on there. Thanks much.

I needed to ssh into the VM first to make sure that the system prompted for a new password by running ssh ops@coredata-dpkd.dp.mbari.org
I changed the password to the normal ops password.
Now I need to add the ssh key to BitBucket. Since, I had already setup an ssh key on the coredata-rcsn-sim.shore.mbari.org machine, I just copied that over to the ship-board system and set the permissions correctly (along with the config file). See the directions here if you need to create a key from scratch. In the config file, I changed the name of the navproc machine to navproc-dpkd.dp.mbari.org and added an entry for the spare navproc-dpkd-spare.dp.mbari.org.
I neede to change the owner of the /data directory to ops so I ran sudo chown ops /data.
Next, it was time to install the API stack. One small thing with this stack is that the data being written by docker in the timescale DB has to be done as user with UID 1000. So I had to pre-emptively create the /data/timescaledb and change the owner to UUID 1000.
```
cd /data
mkdir timescaledb
mkdir timescaledb/data
sudo chown -R 1000 timescaledb/
cd /opt
sudo mkdir corenav
sudo chown ops corenav
cd corenav
git clone git@bitbucket.org:mbari/navproc-api.git
cd navproc-api/
cp .env.template .env
```
I then edited the .env file to set up all the usernames and passwords for the various connected services.
Also, there is a ssh key that you need to allow the pull-logr-file container to grab log files from navproc. I copied it into the project by running cp ~/ssh/id_navproc /opt/corenav/navproc-api/pull-logr-files/ssh-key. Also, you need to install the public part of the key on the navproc machine. If you ssh into the navproc machine and either create or edit the ~/.ssh/authorized_keys file and add the contents of the ~/.ssh/id_navproc.pub file to that file, you can then ssh without password which is what the pull-logr-file proccess needs.
I then started the API stack by running docker compose up -d
You can verify it's running by using docker ps and you should see several containers running. You can check the following to see if they are running:
The non-https NATS Web Site should be available here
The non-https Grafana Web Site should be available here

Now, in order for things to be more straightforward, I wanted to set up proxy passes in Nginx so that we can map clean https URLs to the ports of the underlying services. The host is already running Nginx and we just need to set up the proxy passes. Peter had already configured an example proxy pass and it was located at /etc/nginx/sites-available/mbari-revproxy and looked like this:

server {
    listen 80;
    server_name  coredata-dpkd.dp.mbari.org;
    return 301   https://$host$request_uri;
}

server {
    listen 443 ssl;
    server_name                 coredata-dpkd.dp.mbari.org;
    ssl_certificate             /etc/nginx/ssl/wcbundle.crt;
    ssl_certificate_key         /etc/nginx/ssl/wcprivate.key;
    ssl_prefer_server_ciphers   on;

    location / {
        proxy_pass  http://localhost:8080;

        proxy_set_header   Host $host;
        proxy_set_header   X-Real-IP $remote_addr;
        proxy_set_header   X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header   X-Forwarded-Proto $scheme;
    }
}

In order to configure Grafana to run through reverse proxy, I added the following properties to .env file and restarted the grafana container:
```
GF_SERVER_ROOT_URL="https://coredata-dpkd.dp.mbari.org/grafana/"
```
which tells the grafana app it will be running proxied on a sub path.

Then I added the following section to the mbari-revproxy file:

location /grafana/ {
    proxy_pass  http://localhost:3000/;
    proxy_set_header   Host $host;
    proxy_set_header   X-Real-IP $remote_addr;
    proxy_set_header   X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header   X-Forwarded-Proto $scheme;

    # Required for WebSocket support
    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection "upgrade";
}

Note

The ending / on both the location and proxy_pass are VERY important and it won't work without it.

Next, I wanted to set up the reverse proxy for the NATS web page and so I added the following configuruation:

location /nats/ {
    rewrite ^/nats(/.*)$ $1 break;
    proxy_pass http://127.0.0.1:8222;
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;

    proxy_http_version 1.1;
}

Lastly, I wanted to make all the log files available to download over https. I added the following location clause:

location /logs/ {
    alias /data/logr/;
    autoindex on;
    autoindex_exact_size on;     # Optional: show human-readable file sizes
    autoindex_localtime on;       # Optional: show local time
}

After making these changes, restarting the docker containers and the nginx service, you can visit the following URLs:
1. Grafana https://coredata-dpkd.dp.mbari.org/grafana/
2. NATS https://coredata-dpkd.dp.mbari.org/nats/

Notes

We had some very strange behavior on the EZ32 when we were trying different wiring configuration to connect the terminal server to the pan and tilts. Something about the wiring or different adapters we were trying we causing the terminal server to reboot itself. Very strange and somewhat concerning.