Skip to content

Custom Logger

Rich Schramm wrote a utility that the pilot's can use to start and stop a custom LCM logger to capture log files on bubinga. The idea is that if something strange is going on, the pilots can ssh into bubinga from infernalis and start the logger which will log all LCM messages. They can then stop the logger when they feel as if they have captured the event.

Users Guide

It's pretty simple to use. From whatever machine you are on, you can ssh into bubinga as ventana. You can use the command line from Unix or Putty from windows. Once you are logged in as ventana, you can start the logger using:

startlog

This will run a logger in the background and capture log files to the /home/ventana/log_files directory.

Warning

These log files are enormous so make sure you stop the logger when you don't need it anymore

To stop the logger, simply issue:

stoplog

Note

There is a cron job that runs nightly that takes the log files that are in the log_files directory and copies them off to a Network Attached Storage (NAS) device and then the logs are deleted. This is to make sure bubinga does not fill it's hard drive. You can find out more information about the NAS here

Developer's Guide

The custom logging script is a small C program located in ventana's dev/custom_lcm_logger directory. If you want to rebuild the application, you need to ssh into bubinga, then issue the following:

cd dev/custom_lcm_logger
make
sudo make install

This will copy the freshly built mylcm-logger to /usr/local/bin. The executable is not run directly, but there are two shell scripts that start and stop the loggers are located at:

/home/ventana/startlog.sh
/home/ventana/stoplog.sh

The scripts are simple and are repeated here for record keeping:

startlog.sh

#!/bin/bash

### run lcm-logger with log rotation and console output to
### a timestamed file.
### 27Oct2015 r.schramm
### 27Apr2016 r.schramm  Changed split size from 600Mb to 300Mb
### 09Jun2022 kgomes Rebuilt and deployed on the new bubinga that is running Ubuntu instead of Debian

echo "Starting lcm-logger"

pkill -f lcm-logger
pkill -f mylcm-logger
#sleep 1

cd /home/ventana/log_files

DATE=`date +%Y-%m-%d-%H%M%S`
fname=console-$DATE

/usr/local/bin/mylcm-logger --split-mb=300 --flush-interval=75 --max-unwritten-mb=500 &> $fname.log &

exit 0

stoplog.sh

#!/bin/bash

echo "Stopping ALL lcm-loggers running on this system"

pkill -f lcm-logger
pkill -f mylcm-logger

exit 0

Custom GSS tags

In August of 2022, Greensea added a button on the pilot interface that would allow pilots to insert LCM 'tags' in the log files collecting on bubinga to help identify sections of the logs of interest for further exploration. The log files can be played back on the GSS Playback interface, but in order to find the log files of interest, grep can be used. The log files for the custom logger are located in /home/ventana/log_files and are broken up into 300MB file sizes which is about 5 minutes of data. In order to find the log files that have the GSS_PCOMMS_LOG_MARK tag in them, simply ssh into bubinga, cd to the /home/ventana/log_files directory and run grep GSS_PCOMMS_LOG_MARK lcmlog* and after minute or so, it will spit out log files that contained that particular tag.

When an event happens that the pilots want GSS to look at, we have to make the appropriate logs easily accessible for them to get to. A common way is to make them available is via the https://data.mbari.org server. When looking for the logs to share, note that there are two spots you might have to look. The first one is on bubinga.rc.mbari.org since that is where the logs are captured during the dive and is likely the ones they would be interested in sharing. Every night a script runs to copy those logs to the Network Attached Storage device on the carson so if you can't find the logs on bubinga, you should look on the NAS. So, the process might look like:

  1. If the files are still on bubinga
    1. ssh into bubinga using ssh ventana@bubinga.rc.mbari.org.
    2. The change into the directory with the log files using cd log_files
    3. Then run a grep to look for files with the right tags in them by running grep GSS_PCOMMS_LOG_MARK lcmlog* (this can take a long time as the files are quite large).
    4. Jot down the list of files that have the tag of interest as those will be the ones you want to copy.
    5. The way I do this is mount the Data_Repository atlas share on my local machine (ask IS for permissions if you can't mount it as read/write).
    6. From a command line I cd into the Data_Repository/platforms/ships/rachel-carson/gss/logs directory and then copy over each file using scp. It might looks something like scp ventana@bubinga.rc.mbari.org:/home/ventana/log_files/lcmlog-2022-09-14.12 ..
    7. You can then share those log files by pointing the GSS folks to https://data.mbari.org/platforms/ships/rachel-carson/gss/logs/
  2. If the files have been moved to the NAS
    1. Mount the NAS on your local machine using SMB and connect to smb://134.89.23.21/VentanaLogs. All of the log files are copied to the mbari_logs/YYYY directories.
    2. If you know the date of the event, in this case, let's say it was on September 14, 2022. You can open a command prompt on your machine, navigate to the VentanaLogs/mbari_logs/2022 directory and run grep GSS_PCOMMS_LOG_MARK lcmlog-2022-09-14.* and that should show you any log files that have the tag of interest (this can take quite awhile since the files are huge).
    3. Then I would mount the Data_Repository volume as in the steps in the above section and simply copy/paste the files of interest into the Data_Repository/platforms/ships/rachel-carson/gss/logs directory and share the link above with GreenSea.