Skip to content

ROV CTD Data

This document details the collection, processing and access methods for the ROV CTD data for the ROV Ventana on the Rachel Carson.

CTD Data Flow

flowchart TB subgraph MBARI direction TB subgraph rcsn["RV Rachel Carson"] direction TB subgraph ROV Ventana rovctd[CTD] end subgraph ROV Control Room terminal-server[Digi Terminal Server] subgraph navproc[Navproc NUC] ctd-serin["Navproc Serial In
(CTD)"] seabird-ctd["Navproc Seabird CTD"] ctd-logr["CTD Logr Process"] ctd-logr-files@{ shape: docs, label: "/data/YYYYDDDrovcdtlogr.dat"} lcm-bridge[LCM Bridge] bcserver["BCServer:54007"] end ctd-tech([CTD Technician]) subgraph rovctd-pc["ROV CTD Computer"] ctd-coeff["CTDCoeff.ini"] ctd-gui["LabVIEW CTD GUI"] ctd-cfg-monitor["CTD Coeff Monitor"] end end subgraph coredata-rcsn[RCSN Coredata] pull-logr-files["Log File Copy"] navproc-api-ctd-logr-files@{ shape: docs, label: "/data/logr/YYYY/YYYYDDDrovcdtlogr.dat.gz"} apache-httpd["Apache Web Server"] zmq-proxy["ZMQ Proxy"] nats[NATS Server] nats-udp["NATS UDP Proxy:5007"] nats-to-tsdb["NATS to TimescaleDB"] timescale-db[(TimescaleDB)] grafana[Grafana] end subgraph glacierwind["Glacierwind SQL Server"] vnta-ctd-cfg-db[("VNTA_CTD_Config")] end end subgraph Shore subgraph coredata8[Coredata on Shore] coredata-ship-data["/mbari/ShipData/logger/YYYY/rcsn"] end subgraph draco["draco.shore.mbari.org"] ventana-ctd-cfg-to-perseus["Ventana CTD Cfg to Perseus"] end subgraph thalassa[Thalassa] thalassa-share["ShipData/logger/YYYY/rcsn"] end end end %% Connections rovctd -- focal/serial --> terminal-server -- realport/serial --> ctd-serin -- LCM --> seabird-ctd -- LCM --> ctd-logr --> ctd-logr-files ctd-logr-files -- scp --> pull-logr-files --> navproc-api-ctd-logr-files --> apache-httpd pull-logr-files -- scp --> coredata-ship-data --> thalassa-share seabird-ctd -- LCM --> lcm-bridge seabird-ctd -- LCM --> bcserver -- UDP --> ctd-gui ctd-tech -- edit --> ctd-coeff ctd-coeff --> ctd-gui ctd-coeff --> ctd-cfg-monitor --> vnta-ctd-cfg-db --> ventana-ctd-cfg-to-perseus lcm-bridge --> zmq-proxy lcm-bridge --> nats --> nats-udp nats --> nats-to-tsdb --> timescale-db --> grafana %% These connectors are invsible but are need to make the diagram render properly, don't remove them apache-httpd ~~~ coredata-ship-data ctd-gui ~~~ nats grafana ~~~ coredata-ship-data %% Click links click terminal-server "https://hub.digi.com/support/products/infrastructure-management/digi-connectport-ts-8-16-terminal-server/" click ctd-serin "https://bitbucket.org/mbari/navproc-process/src/master/serial-input/" click seabird-ctd "https://bitbucket.org/mbari/navproc-process/src/master/seabird-ctd/" click ctd-logr "https://bitbucket.org/mbari/navproc-logger/src/master/src/" click lcm-bridge "https://bitbucket.org/mbari/lcm_bridge/src/master/" click bcserver "https://bitbucket.org/mbari/navproc-logger/src/master/src/bcserver/" click ctd-gui "https://bitbucket.org/mbari/ctd-gui/src/main/" click ctd-cfg-monitor "https://bitbucket.org/mbari/rovctdcfgfilemonitor/src/master/" click pull-logr-files "https://bitbucket.org/mbari/navproc-api/src/master/pull-logr-files/" click zmq-proxy "https://bitbucket.org/mbari/navproc-api/src/master/zmq-proxy/" click nats "https://bitbucket.org/mbari/navproc-api/src/master/nats-server/" click nats-udp "https://bitbucket.org/mbari/navproc-api/src/master/nats-udp-proxy/" click apache-httpd "https://coredata-rcsn.rc.mbari.org/logs/"

Connectivity and Data Flow Description

  1. Navproc
    1. The CTD on the ROV is connected to the topside control system through the Focal 903 Digital Multiplexer and is presented in the control room as an RS-232 port.
    2. The signal from the Focal RS-232 is then routed to a Digi TS-16 Terminal Server.
    3. On the navproc computer, there are Digi RealPort drivers that connect to the terminal server and present the serial port as a local /dev/tty device on the navproc computer.
    4. A navproc process then reads the raw serial data from the terminal server and publishes the raw data as LCM messages.
    5. Another navproc process reads in the raw serial LCM message, parses the raw data into data fields and then publishes that as a new LCM message in a generic navproc\msg_t formatted message.
  2. Logging
    1. There are logger processes running that can then subscribe to those LCM message and build logger files using the latest values from the LCM messages. The main logger file is the YYYYDDDrovctdlogr.dat file stored in the /data directory on the navproc computer
    2. There is an hourly process that runs on coredata-rcsn.rc.mbari.org that secure copies (SCP) the logr files to the coredata machine and then creates a zipped version of it to save space. In addition, it runs a check to see if the ship is connected to the shore network and if so, pushes a copy of the log files to the ShipData share on Thalassa using scp through the coredata.shore.mbari.org where the ShipData share is mounted.
  3. UDP Server
    1. There is a Python process running on the navproc machine that listens for UDP requests on specific ports. The ports are mapped to specific loggers and if you send messages to those UDP ports, you can retrieve the headers being written to the logging files as well as the last line that was written to the log file. The port mapping is listed below so for ROV CTD data you would want to use port 54007:
      1. 54009: dataprobelogr
      2. 54019: m3rslogr
      3. 54017: nav4dlogr
      4. 54007: rovctdlogr
      5. 54003: shipnavlogr
      6. 54005: videologr
  4. LCM Bridge
    1. There is a process known as the LCM Bridge that is running on the navproc machine and it can be configured using a JSON config file to subscribe to LCM messages and then 'bridge' them over to other types of systems. You can find the current mapping file here. The process uses this config file to subscribe to specific LCM messages and then reformat the message into JSON and then forwards them to other systems.
    2. For the Rachel Carson, the bridge is configured to send the messages to a ZeroMQ proxy running on coredata-rcsn.rc.mbari.org and also to a NATS messaging server also running on coredata-rcsn.rc.mbari.og.
  5. There is a Windows PC in the ROV control room named rc-rctd.rc.mbari.org that is used to visualize the CTD data coming from ROV Ventana. There is a LabVIEW application that was custom built that reads data from the navproc system and plots the CTD data using calibration coefficients defined in a CTDCoeff.ini file. It uses the UDP server on the navproc machine to read the latest logger data and then it plots the CTD in the application.
  6. On the same Windows PC (rc-rctd.rc.mbari.org) there is a Python process running as a Windows service that watches the CTDCoeff.ini file and if it is edited, it reads the contents of the file and inserts a record into the VNTA_CTD_Config database on the SQL Server glacierwind.rc.mbari.org on the ship.
  7. NATS Clients
    1. Downstream of the NATS server, there are two clients. The first is a python script running on coredata-rcsn.rc.mbari.org that opens up UDP ports and listens for messages exactly as the UDP Server on navproc does. It uses the same port mapping and replies with the same reponses that can either be the header from the logger or the latest log file entry. The second client subscribes to several NATS channels and then creates a table in a TimescaleDB server and stores all the streaming data from that channel in the table.

Data Access

Onboard - Logger Files

You can get easy access to the ROV CTD logger files from a web server hosted on the Carson. You can find the files organized by year located at the link below. Click on that link and then select the year you are interested in. The raw CTD data is logged in files with names YYYYDDDseabirdctdfulllogr.dat.gz and the parsed CTD data can be found in the YYYYDDDrovctdlogr.dat.gz files.

Carson Core Data Logs

Onboard - UDP

If you want to retrive messages using UDP, you can get the data from the logging system from two different locations. The directions are detailed here

  1. Python Example

Onboard - ZMQ Proxy

If you are onboard the Carson and you want to subscribe to the ZeroMQ messages for the CTD, you an use any of the client examples listed below:

  1. Python Example

Onboard - NATS Messaging

If you want to access data using the NATS messaging system, check out the documentation for how to do this here

  1. Python Example

Onboard - TimescaleDB

The data from navproc is being streamed to a TimescaleDB onboard for historical purposes and can be accessed from the database directly (although this isn't really recommended). If you want information on how to do this, consult the docs here.

  1. Python Example

Onboard - Grafana

There is a Grafana server onboard the Carson that is connected to the TimescaleDB and can be used to create dashboard to look at data from the CTD. The server can be found at https://coredata-rcsn.rc.mbari.org/grafana/. Underneath the 'ops' account in Grafana, there is a CTD Dashboard that shows the data from the last 6 hours.

Shore - Thalassa Data Share

If you need to get to the ShipData share directly, you can work with IS on how to get read-only permissions to the share and mount it on your local machine. The logger data is stored in the logger/YYYY/rcsn directory.