Rachel Carson

This page documents the Navproc and Logging system on the Rachel Carson

Overview

First, let's look at a high level diagram of the various components. There are basically four main components and they are:

Two Digi Connect TS16s that takes in physical connections from various ship board signals
An Ubuntu 22 NUC that is running the current version of navproc/logr software that has the TS16 ports mounted virtually as /dev/tty devices
An Ubuntu 24 virtual machine that is running the data distribution side of things (ZeroMQ, NATS, TimescaleDB, Grafana, etc.)

--- title: Logical Components --- flowchart LR subgraph Rachel Carson direction LR ship-signals[Various Ship and ROV Signals] rcsn-ts1["Digi TS16
134.89.22.61
rctermserva"] rcsn-ts2["Digi TS16
134.89.22.62
rctermservb"] navproc-rcsn["Navproc NUC
navproc-rcsn.rc.mbari.org"] coredata-rcsn["Navproc API
coredata-rcsn.rc.mbari.org"] end ship-signals --> rcsn-ts1 ship-signals --> rcsn-ts2 rcsn-ts1 --> navproc-rcsn rcsn-ts2 --> navproc-rcsn navproc-rcsn --> coredata-rcsn

--- title: Signal Connections --- flowchart LR subgraph signal-connections[Signal Connections] direction LR subgraph signals lapbox-4k-sig[4K Lapbox] lapbox-hd-sig[HD Lapbox
134.89.22.21] nav4d-sig[4D Nav] ship-gps-sig[Ship GPS] seabird-ctd-sig[Seabird CTD] ship-gyro-sig[Ship Gyro] ventana-sig[Ventana] vorne-display-sig[Vorne Display] gtdpro-sig[GTDPro] lodestar-sig[LodeStar] uhs-sig[UHS] udp[UDP
134.89.23.23] end subgraph rctermserva direction TB porta1[Port 1] porta2[Port 2] porta3[Port 3] porta4[Port 4] porta5[Port 5] porta6[Port 6] porta7[Port 7] porta8[Port 8] porta9[Port 9] porta10[Port 10] porta11[Port 11] porta12[Port 12] porta13[Port 13] porta14[Port 14] porta15[Port 15] porta16[Port 16] end subgraph rctermservb direction TB portb1[Port 1] portb2[Port 2] portb3[Port 3] portb4[Port 4] portb5[Port 5] portb6[Port 6] portb7[Port 7] portb8[Port 8] portb9[Port 9] portb10[Port 10] portb11[Port 11] portb12[Port 12] portb13[Port 13] portb14[Port 14] portb15[Port 15] portb16[Port 16] end subgraph navproc-rscn subgraph ttys direction TB ttya00["/dev/ttya00
9600"] ttya01 ttya02["/dev/ttya02
4800"] ttya03["/dev/ttya03
9600"] ttya04 ttya05["/dev/ttya05
9600"] ttya06["/dev/ttya06
9600"] ttya07["/dev/ttya07
9600"] ttya08["/dev/ttya08
9600"] ttya09["/dev/ttya09
9600"] ttya10["/dev/ttya10
9600"] ttya11 ttya12 ttya13 ttya14 ttya15 ttyb00["/dev/ttyb00
9600"] ttyb01 ttyb02["/dev/ttyb02
19200"] ttyb03["/dev/ttyb03
19200"] ttyb04 ttyb05["/dev/ttyb05
9600"] ttyb06["/dev/ttyb06
9600"] ttyb07["/dev/ttyb07
9600"] ttyb08["/dev/ttyb08
115200"] ttyb09 ttyb10["/dev/ttyb10
19200"] ttyb11 ttyb12 ttyb13 ttyb14 ttyb15 end subgraph navproc-processes direction TB lapbox-4k[lapbox_4k] lapbox-hd[lapbox_hd] nav4d-gps-serin[nav4d_gps_serin] nav4d-gps[nav4d_gps] nav4d-output[nav4d_output] nav4d-output-serout[nav4d_output_serout] nav4d-general-serout[nav4d_general_serout] nmea-gps-serin[nmea_gps_serin] nmea-gps[nmea_gps] seabird-ctd-serin[seabird_ctd_serin] seabird-ctd[seabird_ctd] ship-gyro-serin[ship_gyro_serin] ship-gyro[ship_gyro] ventana-csp-serin[ventana_csp_serin] ventana-csp[ventana_csp] vorne-display[vorne_display] gtdpro-serin[gtdpro_serin] lodestar-serin[lodestar_serin] uhsmsg-serin[uhsmsg_serin] ctd-gss-serout[ctd_gss_serout] gps-lodestar-serout[gps_lodestar_serout] rovdepth-mesotech-serout[rovdepth_mesotech_serout] rovdepth-sonardyne-serout[rovdepth_sonardyne_serout] soundv-sprintins-serout[soundv-sprintins-serout] uhsmsg-gss-serout[uhsmsg_gss_serout] gps-octans-udpout[gps_octans_udpout] end subgraph lcm-messages direction TB pt-control-lapbox-4k-lcm[PT_CONTROL_LAPBOX_4K] lapbox-4k-lcm[LAPBOX_4K_MSG] camera-4k-lcm[CAMERA_4K_MSG] pt-control-lapbox-hd-lcm[PT_CONTROL_LAPBOX_HD] lapbox-hd-lcm[LAPBOX_HD_MSG] camera-hd-lcm[CAMERA_HD_MSG] nav4d-gps-serin-lcm[NAV4D_GPS_SERIN] nav4d-gps-msg-lcm[NAV4D_GPS_MSG] nav4d-output-msg-lcm[NAV4D_OUTPUT_MSG] nmea-gps-serin-lcm[NMEA_GPS_SERIN] nmea-gps-msg-lcm[NMEA_GPS_MSG] seabird-ctd-serin-lcm[SEABIRD_CTD_SERIN] seabird-ctd-msg-lcm[SEABIRD_CTD_MSG] ship-gyro-serin-lcm[SHIP_GYRO_SERIN] ship-gyro-msg-lcm[SHIP_GYRO_MSG] ventana-csp-serin-lcm[VENTANA_CSP_SERIN] ventana-csp-msg-lcm[VENTANA_CSP_MSG] nav4d-output-msg-lcm[NAV4D_OUTPUT_MSG] gtdpro-serin-lcm[GTDPRO_SERIN] lodestar-serin-lcm[LODESTAR_SERIN] uhsmsg-serin-lcm[UHS_SERIN] end end end porta1 --- ttya00 porta2 --- ttya01 porta3 --- ttya02 porta4 --- ttya03 porta5 --- ttya04 porta6 --- ttya05 porta7 --- ttya06 porta8 --- ttya07 porta9 --- ttya08 porta10 --- ttya09 porta11 --- ttya10 porta12 --- ttya11 porta13 --- ttya12 porta14 --- ttya13 porta15 --- ttya14 porta16 --- ttya15 portb1 --- ttyb00 portb2 --- ttyb01 portb3 --- ttyb02 portb4 --- ttyb03 portb5 --- ttyb04 portb6 --- ttyb05 portb7 --- ttyb06 portb8 --- ttyb07 portb9 --- ttyb08 portb10 --- ttyb09 portb11 --- ttyb10 portb12 --- ttyb11 portb13 --- ttyb12 portb14 --- ttyb13 portb15 --- ttyb14 portb16 --- ttyb15 lapbox-4k-sig --> portb9 ttyb08 --> lapbox-4k --> lapbox-4k-lcm lapbox-4k --> camera-4k-lcm pt-control-lapbox-4k-lcm --> lapbox-4k lapbox-hd-sig --> lapbox-hd --> lapbox-hd-lcm lapbox-hd --> camera-hd-lcm pt-control-lapbox-hd-lcm --> lapbox-hd nav4d-sig --> porta11 ttya10 --> nav4d-gps-serin --> nav4d-gps-serin-lcm nav4d-gps-serin-lcm --> nav4d-gps --> nav4d-gps-msg-lcm ventana-csp-msg-lcm --> nav4d-output --> nav4d-output-msg-lcm nav4d-output-msg-lcm --> nav4d-output-serout --> ttya07 nav4d-gps-serin-lcm --> nav4d-general-serout --> ttya06 ship-gps-sig --> porta4 ttya03 --> nmea-gps-serin --> nmea-gps-serin-lcm nmea-gps-serin-lcm --> nmea-gps --> nmea-gps-msg-lcm seabird-ctd-sig --> porta5 ttya04 --> seabird-ctd-serin --> seabird-ctd-serin-lcm seabird-ctd-serin-lcm --> seabird-ctd --> seabird-ctd-msg-lcm ship-gyro-sig --> porta3 ttya02 --> ship-gyro-serin --> ship-gyro-serin-lcm ship-gyro-serin-lcm --> ship-gyro --> ship-gyro-msg-lcm ventana-sig --> porta10 ttya09 --> ventana-csp-serin --> ventana-csp-serin-lcm ventana-csp-serin-lcm --> ventana-csp --> ventana-csp-msg-lcm ventana-csp-msg-lcm --> vorne-display --> ttya00 nav4d-gps-msg-lcm --> vorne-display ttya00 --> vorne-display-sig gtdpro-sig --> portb11 ttyb10 --> gtdpro-serin --> gtdpro-serin-lcm lodestar-sig --> portb7 ttyb06 --> lodestar-serin --> lodestar-serin-lcm uhs-sig --> portb3 ttyb02 --> uhsmsg-serin --> uhsmsg-serin-lcm seabird-ctd-serin-lcm --> ctd-gss-serout ctd-gss-serout --> ttya05 nmea-gps-serin-lcm --> gps-lodestar-serout --> ttyb05 ventana-csp-msg-lcm --> rovdepth-mesotech-serout --> ttyb00 ventana-csp-msg-lcm --> rovdepth-sonardyne-serout --> ttya08 seabird-ctd-msg-lcm --> soundv-sprintins-serout --> ttyb07 uhsmsg-serin-lcm --> uhsmsg-gss-serout --> ttyb03 nmea-gps-serin-lcm --> gps-octans-udpout --> udp

Installation and Setup

rctermserva

This was largely left untouched when we moved from the old navproc to new so nothing was changed

rctermservb

This was largely left untouched when we moved from the old navproc to new so nothing was changed

navproc-rcsn

See the computer setup instructions for specific details on this setup

coredata-rcsn

This is the main server for the Navproc API (and other applications). This is a server in the VM cluster that will act as an API server for the data coming from navproc. This API consists of a ZMQ proxy, a NATS cluster, a UDP proxy that replicates the old bcserver, a TimescaleDB server and a Grafana server. Here is a basic diagram of the API services and data flow.

--- title: Navproc API --- flowchart LR subgraph Rachel Carson Navproc API Server direction LR subgraph navproc navproc-process-1 navproc-process-2 logr-process-1 logr-process-2 lcm-bridge subgraph logger files navproc-process-1-log-file[Logger File 1] navproc-process-2-log-file[Logger File 2] end bcserver(bcserver.py) np-port54003(54003) np-port54005(54005) np-port54007(54007) np-port54009(54009) np-port54010(54010
DiskUsage) np-port54017(54017) np-port54019(54019) end subgraph navproc-api port5555(5555) zmq-proxy[ZMQ Proxy
Python] port5556(5556) port4222in(4222) subgraph nats-cluster nats-server-1 nats-server-2 nats-server-3 end port4222out(4222) port8222html(8222-http) port8080ws(8080-web-sockets) nats-udp-proxy[NATS UDP Proxy
Python] port54003(54003) port54005(54005) port54007(54007) port54009(54009) port54010(54010
DiskUsage) port54017(54017) port54019(54019) grafana nats-to-tsdb[NATS to TimescaleDB
Python] port5432in(5432) timescaledb[TimescaleDB] position-to-odss[Position To ODSS
Python] pull-logr-files[Pull Logr File Process] subgraph "/data/logr/YYYY" navproc-process-1-log-file-gzip[Logger File 1 GZip] navproc-process-2-log-file-gzip[Logger File 2 GZip] end end end subgraph Shore messaging[AMQP Server
messaging.shore.mbari.org] subgraph coredata8 navprocessing-directory["~/shipandrov/navprocessing/rcsn-sim/todo/YYYYDDDshipnavlogr.dat.gz"] mbari-directory["/mbari/ShipData/logger/YYYY/rcsn-sim/YYYYDDDDxxxxxlogr.dat.gz"] end subgraph Thalassa ship-data-share["ShipData://logger/YYYY/rcsn-sim/YYYYDDDxxxxxlogr.dat.gz"] end end navproc-process-1 --> logr-process-1 navproc-process-1 --> lcm-bridge logr-process-1 --> lcm-bridge logr-process-1 --> navproc-process-1-log-file navproc-process-2 --> logr-process-2 navproc-process-2 --> lcm-bridge navproc-process-1 --> bcserver navproc-process-2 --> bcserver bcserver --> np-port54003 bcserver --> np-port54005 bcserver --> np-port54007 bcserver --> np-port54009 bcserver --> np-port54010 bcserver --> np-port54017 bcserver --> np-port54019 logr-process-2 --> lcm-bridge logr-process-2 --> navproc-process-2-log-file lcm-bridge --> port5555 --> zmq-proxy --> port5556 lcm-bridge --> port4222in --> nats-cluster --> port4222out nats-cluster --> port8222html nats-cluster --> port8080ws port4222out --> nats-udp-proxy timescaledb --> grafana nats-udp-proxy --> port54003 nats-udp-proxy --> port54005 nats-udp-proxy --> port54007 nats-udp-proxy --> port54009 nats-udp-proxy --> port54010 nats-udp-proxy --> port54017 nats-udp-proxy --> port54019 port4222out --> position-to-odss --> messaging port4222out --> nats-to-tsdb --> port5432in --> timescaledb navproc-process-1-log-file --> pull-logr-files --> navproc-process-1-log-file-gzip --> mbari-directory --> ship-data-share navproc-process-2-log-file --> pull-logr-files --> navproc-process-2-log-file-gzip --> mbari-directory --> ship-data-share navproc-process-1-log-file-gzip --> navprocessing-directory

I submitted a ticket to IS to create a new VM coredata-rcsn.rc.mbari.organd here is the original request

  - Request_Submitted_By: kgomes@mbari.org
  - VM_Name: coredata-rcsn
  - VM_Purpose: This will be the API server for the Rachel Carson simulator
  - VM_Expiration: 5 years
  - VM_Support: IS_Supported
  - VM_Support_Alt_Adm: 
  - VM_OS: Ubuntu 24.04 LTS
  - CPU: 2
  - CPU_Addl_Reason: 
  - RAM: 8
  - RAM_Addl_Reason: 
  - GPU_Extra: 
  - GPU: NO
  - Disk_Extra: 50G
  - Network: Rachel Carson
  - Resource_Priority: Low
  - Resource_Priority_Reason: 
  - Conf_Firewall: The ports allowed should be 54000->54050; 80/443; 6379; 5556; 5555; 8086; 8094; 1883; 9001; 5432
  - Conf_Desktop: YES
  - Conf_Desktop_Extra: 
  - Conf_Logins: local
  - Conf_Docker: YES
  - Conf_Docker_Extra: 
  - Conf_WebServer: apache
  - Conf_sudo: There should be two accounts, 'kgomes' and 'ops' and they both should have sudo privs.
  - Conf_vCenter_Access: None
  - VM_Comments: None

I needed to ssh into the VM first to make sure that the system prompted for a new password by running ssh ops@coredata-rcsn.rc.mbari.org
I changed the password to the normal ops password.
Now I need to add the ssh key to BitBucket. Since, I had already setup an ssh key on the coredata-rcsn-sim.shore.mbari.org machine, I just copied that over to the ship-board system and set the permissions correctly (along with the config file). See the directions here if you need to create a key from scratch.

Next, it was time to install the API stack. One small thing with this stack is that the data being written by docker in the timescale DB has to be done as user with UID 1000. So I had to pre-emptively create the /data/timescaledb and change the owner to UUID 1000.

cd /
sudo mkdir data
sudo chown ops data
cd data
mkdir timescaledb
mkdir timescaledb/data
sudo chown -R 1000 timescaledb/
cd /opt
sudo mkdir corenav
sudo chown ops corenav
cd corenav
git clone git@bitbucket.org:mbari/navproc-api.git
cd navproc-api/
cp .env.template .env

I then edited the .env file to set up all the usernames and passwords for the various connected services
I then started the API stack by running docker compose up -d
You can verify it's running by using docker ps and you should see several containers running. You can check the following to see if they are running:
The non-https NATS Web Site should be available here
The non-https Grafana Web Site should be available here

Now, in order for things to be more straightforward, I wanted to set up proxy passes in Nginx so that we can map clean https URLs to the ports of the underlying services. The host is already running Nginx and we just need to set up the proxy passes. Peter had already configured an example proxy pass and it was located at /etc/nginx/sites-available/mbari-revproxy and looked like this:

server {
    listen 80;
    server_name  coredata-rcsn.rc.mbari.org;
    return 301   https://$host$request_uri;
}

server {
    listen 443 ssl;
    server_name                 coredata-rcsn.rc.mbari.org;
    ssl_certificate             /etc/nginx/ssl/wcbundle.crt;
    ssl_certificate_key         /etc/nginx/ssl/wcprivate.key;
    ssl_prefer_server_ciphers   on;

    location / {
        proxy_pass  http://localhost:8080;

        proxy_set_header   Host $host;
        proxy_set_header   X-Real-IP $remote_addr;
        proxy_set_header   X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header   X-Forwarded-Proto $scheme;
    }
}

In order to configure Grafana to run through reverse proxy, I added the following properties to .env file and restarted the grafana container:
```
GF_SERVER_ROOT_URL="https://coredata-rcsn.rc.mbari.org/grafana/"
```
which tells the grafana app it will be running proxied on a sub path.

Then I added the following section to the mbari-revproxy file:

location /grafana/ {
    proxy_pass  http://localhost:3000/;
    proxy_set_header   Host $host;
    proxy_set_header   X-Real-IP $remote_addr;
    proxy_set_header   X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header   X-Forwarded-Proto $scheme;

    # Required for WebSocket support
    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection "upgrade";
}

Note

The ending / on both the location and proxy_pass are VERY important and it won't work without it.

Next, I wanted to set up the reverse proxy for the NATS web page and so I added the following configuruation:

location /nats/ {
    rewrite ^/nats(/.*)$ $1 break;
    proxy_pass http://127.0.0.1:8222;
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;

    proxy_http_version 1.1;
}

Lastly, I wanted to make all the log files available to download over https. I added the following location clause:

location /logs/ {
    alias /data/logr/;
    autoindex on;
    autoindex_exact_size on;     # Optional: show human-readable file sizes
    autoindex_localtime on;       # Optional: show local time
}

After making these changes, restarting the docker containers and the nginx service, you can visit the following URLs:
1. Grafana https://coredata-rcsn.rc.mbari.org/grafana/
2. NATS https://coredata-rcsn.rc.mbari.org/nats/