Skip to content

Ship and ROV Data Workflow

Introduction

This document details the workflow for all the data coming off our research vessels and into the Expedition Database. Consult the documentation for the Navproc and Logging Systems for detailed information on how the data is collected on the ships and delivered to shore.

Collection to Shore

flowchart TD subgraph ship[Ship] direction LR subgraph ship-sources[Data Sources] rov[ROV] instruments[Shipboard
Instruments] end subgraph navproc-logging[Navproc Logging] navproc[Navproc] loggers[Logger] lcm-bridge[LCM Bridge] logs-dir["/data/YYYYDDDsource.dat"] end subgraph navproc-api[Navproc API] timescaledb[TimescaleDB] nats[NATS Messaging] zeromq[ZeroMQ] udp-proxy[UDP Proxy] log-puller[Log Transfer] log-files["/data/logr/YYYY/YYYYDDDsource.dat.gz"] end end subgraph shore[Shore] subgraph coredata8["coredata8.shore"] shore-logs["/mbari/ShipData/logger/YYYY/[ship]"] coredata-todo["/u/coredata/shipandrov/
navprocessing/[ship]/todo"] end end rov --> navproc instruments --> navproc navproc --> loggers navproc --> lcm-bridge loggers --> logs-dir lcm-bridge --> zeromq lcm-bridge --> nats nats --> udp-proxy nats --> timescaledb logs-dir --> log-puller --> log-files log-puller --> shore-logs log-puller -- if shipnavlogr
(copy) --> coredata-todo udp-proxy ~~~ coredata-todo
  1. Data is logged on ships 24/7 into year-day files on the Navproc NUC PCs running in the ROV control room.
  2. On the navproc machine, the logr processes write files to the /data directory in the form of YYYYDDDlogr-type.dat.
  3. On the Navproc API machine, there is a pull-logr-files docker container that reaches out to the navproc machine over ssh and copies any complete log files from the /data directory and stores them in the /data/logr/YYYY directory and then gzips the file. After successfully copying, the pull-logr-files process then checks to see if it's looks like it is connected to shore by checking the latency pings of a shore side machine. If it's fast enough, it will assume it's connected to the dock and it will push any new log files that haven't been copied to shore to the /mbari/ShipData/logger directory on coredata8.shore which is the ShipData file share on Thalassa. Also, if it's a shipnavlogr file, it will create a copy of that file and place it in the /u/coredata/shipandrov/navprocessing/ship/todo directory to be processed. You can see the code here.

Warning

Note that the below documentation for nav data processing into the Expedition Database has two distinct sections. One for the current production data processing and one for the new processes being created by the Expedition Database Modernization project.

Current Production Workflow

Processing to Archives and RovNavEdit

Scripts below run the nightly to kick off the first step in processing the ship and ROV navigation data.

20 22 * * * /u/coredata/shipandrov/navprocessing/bin/NightlyRC > /u/coredata/.log_LateNightlyRC 2>&1
25 22 * * * /u/coredata/shipandrov/navprocessing/bin/NightlyDP > /u/coredata/.log_LateNightlyDP 2>&1

NightlyRC and NightlyDP are shell scripts that each call one perl program as described below and then emails the results of the script. The diagram below is for the Rachel Carson, but the same process runs for the David Packard.

Overview

Converts gzipped logger data files into cleaned, formatted navigation text files for both the ship and the ROV. Supports three ship configurations: wfly (Western Flyer), dpkd (David Packard), and rcsn (Rachel Carson), with corresponding ROVs docr (Doc Ricketts) or vnta (Ventana).

Usage: processNav.perl rcsn|wfly|dpkd fromYYYYJJJ toYYYYJJJ [GPSTime]

flowchart TD CLI["CLI Args\nshipName rcsn|wfly|dpkd\nfromYYYYJJJ toYYYYJJJ\n[GPSTime]"] subgraph INPUT["Input Files"] RAW["~/shipandrov/navprocessing/{ship}/todo/\nYYYYDDD{ship}navlogr.dat.gz"] end subgraph INIT["Phase 1: initialize()"] ARGS["Validate args &\nmap ship→ROV name"] PATHS["Set rawPath / prcPath / archPath"] COLS["Map datamanager field names\nto CSV column variables\n(per ship config: wfly/dpkd/rcsn)"] SCAN["Scan rawPath for *.dat.gz\nbuild sorted @navlgs list"] end subgraph LOOP["Phase 2: Main File Loop"] FILTER["Skip if date outside range"] MKDIRS["Create archive subdirs if needed\n/mbari/RovNavEdit/YYYY/{ship|rov}/"] CREATETXT["createTxt()"] end subgraph PARSE["Phase 3: createTxt() — Parse & Collect"] ZCAT["Open via zcat pipe"] HDRCONST["#CONST lines → read in-water flag"] HDRLOG["#LOG lines → map column indices"] DATALINE["Data lines (CSV)"] DOCDIST["docDist() — skip if within 50m of dock\n(MBARI Moss Landing)"] CALCTIME["calcTime() — epoch sec → YYYY/DDD/HH:MM:SS UTC"] WRITESHP["writeShpDat() → stream-write each record to SHIP file"] INWATER{"ROV in water?\n(inwater flag OR\nprimary pres >2.5db OR\nalt pres >2.5db OR\ndepth >2.5db)"} SPECIAL["Special overrides: docr\n2012-296–334 & 2013-086–087\n→ force alternate pressure"] PUSHROV["pushRovDat() → accumulate ROV records\nin memory arrays"] end subgraph CLEAN["Phase 4: ROV Data Cleaning"] MARK["markMissingData()\nNaN/NO_ITEM/NO_PROV → value=0, flag=1"] ALTI["cleanAltitude()\nalt ≤0 or ≥60m → flag=2"] PRES["cleanPressure()\noutlier vs median of ±5 neighbors\n(±20 dbar threshold) → flag=2\nuses Statistics::Descriptive"] POS1["cleanPosition(4000 pts, 2σ)"] POS2["cleanPosition(180 pts, 2σ)"] POS3["cleanPosition(4000 pts, 3σ)"] WRITEROV["writeRovDat() → write all ROV records with flags"] end subgraph OUTPUT["Output Files"] TMPSHIP["tmp/navYYYYDDD{ship}.txt\n(Year,Day,Time,Usec,Lat,Lon,Qual,Mode)"] TMPROV["tmp/navYYYYDDD{rov}.txt\n(Year,Day,Time,Usec,Lat,Lon,\nEast,North,Pres,Head,Alti,Pitch,Roll,\nPosFlag,PresFlag,HeadFlag,AltiFlag,AttitFlag)"] ARCSHIP["/mbari/RovNavEdit/YYYY/{ship}/\nnavYYYYDDD{ship}.txt\n(read-only)"] ARCROV["/mbari/RovNavEdit/YYYY/{rov}/\nnavYYYYDDD{rov}.txt\n(read-only)"] LOGFILE["/mbari/RovNavEdit/YYYY/{rov}/\nnavYYYYDDD{rov}_log.txt\n(filtering stats & rules)"] end CLI --> INIT RAW --> LOOP ARGS --> PATHS --> COLS --> SCAN SCAN --> LOOP FILTER --> MKDIRS --> CREATETXT CREATETXT --> PARSE ZCAT --> HDRCONST & HDRLOG & DATALINE DATALINE --> DOCDIST --> CALCTIME --> WRITESHP CALCTIME --> INWATER SPECIAL --> INWATER INWATER -->|yes| PUSHROV INWATER -->|no| DATALINE WRITESHP --> TMPSHIP PUSHROV --> CLEAN MARK --> ALTI --> PRES --> POS1 --> POS2 --> POS3 --> WRITEROV WRITEROV --> TMPROV TMPSHIP -->|cp| ARCSHIP TMPROV -->|cp| ARCROV CLEAN --> LOGFILE ARCSHIP & ARCROV & LOGFILE --> DONE["Archive complete\nTemp files deleted"]
  1. Phase 1 — Initialization (initialize())
    1. Perl library path setup — Appends ~/shipandrov/navprocessing/lib/ to @INC so Statistics::Descriptive can be found.
    2. Command-line validation — Requires exactly: ship name (rcsn, wfly, or dpkd), start date (YYYYJJJ), optional end date, optional GPSTime flag. Exits with usage message on invalid input.
    3. ROV name mapping:
      • wfly or dpkd → ROV docr
      • rcsn → ROV vnta
    4. Path setup:
      • rawPath = ~/shipandrov/navprocessing/{shipName}/todo/ — input .dat.gz files placed here by ship data processing push
      • prcPath = ~/shipandrov/navprocessing/{shipName}/tmp/ — temporary output
      • archPath = /mbari/RovNavEdit/ — final archive
    5. Constants setup:
      • NaN = 1e36, NoItem = "NO_ITEM", NoPvdr = "NO_PROV" — sentinel values for missing data
      • MBARI dock coordinates: lat=36.80213, lon=-121.78714 (Moss Landing)
      • Distance conversion constants
    6. Data column name mapping — Based on ship name, assigns the datamanager field names to look for in logger headers (e.g. SHIP.GPS.LAT, ROV.DIGIQ.SCALED_PRESSURE, etc.). These differ by vessel/era.
    7. Input file discovery — Scans rawPath for all *.dat.gz files, builds a sorted list (@navlgs). Defaults date range to first/last file if not specified.
  2. Phase 2 — Main File Loop (per .dat.gz file), for each logger file found in rawPath:
    1. Date range filter — Skips files whose YYYYDDD prefix falls outside startYday–endYday.
    2. Output filename construction:
      • Ship temp file: {prcPath}/nav{YYYYDDD}{shipName}.txt
      • ROV temp file: {prcPath}/nav{YYYYDDD}{rovName}.txt
      • Ship archive: /mbari/RovNavEdit/{YYYY}/{shipName}/nav{YYYYDDD}{shipName}.txt
      • ROV archive: /mbari/RovNavEdit/{YYYY}/{rovName}/nav{YYYYDDD}{rovName}.txt
      • ROV log: /mbari/RovNavEdit/{YYYY}/{rovName}/nav{YYYYDDD}{rovName}_log.txt
    3. Archive directory creation — Creates year subdirectories for wfly, dpkd, docr, rcsn, and vnta under /mbari/RovNavEdit/ if they don't exist (mode 0775).
    4. Core processing — Calls createTxt() (see Phase 3).
    5. Console summary — Prints ship and ROV record counts, start/end time ranges for each navigation segment.
    6. Archival copy — Copies temp ship and ROV files to their respective archive paths; sets archive files read-only (chmod 0444).
    7. Cleanup — Deletes the temp files in prcPath and the .dat.gz from rawPath.
  3. Phase 3 — createTxt() — The Core Processing
    1. Opens input file via zcat pipe to decompress on the fly.
    2. Header line parsing — Lines starting with # are metadata:
      • #CONST lines: Checks for ROV.CTD.INWATER (or vessel equivalent) — captures the in-water flag value and its timestamp.
      • #LOG lines: Maps each data field name to its column index. This is how the script knows which CSV column holds ship lat, ROV pressure, heading, etc.
    3. Data line processing — For each comma-separated data record:
      • Dock proximity filter — Calls docDist(). Skips the record entirely if the ship is within 0.05 km (50 m) of the MBARI dock.
      • Time calculation (calcTime()) — Converts epoch seconds from the designated time column to year, day-of-year, hour, minute, second in UTC. Skips record if time is zero/invalid.
      • Ship data write (writeShpDat()) — Immediately writes the ship record (year, day, time, usec, lat, lon, GPS quality, differential mode) to the SHIP file.
      • ROV in-water determination — Checks in priority order:
        1. The #CONST ROV.CTD.INWATER flag read from the header
        2. Primary pressure sensor > 2.5 decibars (normally ROV.DIGIQ.SCALED_PRESSURE)
        3. Alternate pressure sensor > 2.5 decibars (e.g. ROV.CTD.PRESSURE)
        4. Depth channel > 2.5 decibars
      • Special case overrides — Hard-coded date ranges for docr (2012-296 through 2012-334, and parts of 2013-086/087) force use of alternate pressure due to faulty primary sensor.
      • ROV data accumulation (pushRovDat()) — If in water, appends lat, lon, pressure, heading, altitude, pitch, roll, plus all-zero flag arrays to in-memory arrays. Tracks segment start/end times.
  4. Phase 4 — ROV Data Post-Processing. After all records are read from a file:
    1. markMissingData() — Iterates all ROV arrays. Any value equal to 1e36 (NaN sentinel) is set to 0 and its flag array entry set to 1 (= "missing").
    2. cleanAltitude() — Flags any altitude value ≤ 0 or ≥ 60 meters with flag 2 (= "suspect/bad").
    3. cleanPressure() — For each un-flagged pressure value, computes the median of up to 10 surrounding good points (5 before, 5 after). If the current value is more than 20 dbar from the median, sets flag to 2. Uses Statistics::Descriptive.
    4. cleanPosition() — Three sequential passes of statistical position outlier detection using Statistics::Descriptive:
      • Pass 1: groups of 4000 points, flag if outside 2× standard deviation of group mean
      • Pass 2: groups of 180 points, flag if outside 2× standard deviation
      • Pass 3: groups of 4000 points, flag if outside 3× standard deviation
    5. writeRovDat() — Opens the ROV output file, writes header, then writes all accumulated records with their computed flag values.
    6. printRovLog() — Counts flagged records per category, writes a log .txt file to the archive documenting the filtering rules applied and statistics (rows processed, records flagged by type).

  5. Output File Formats

    1. Ship file (navYYYYDDDshipname.txt):

      Year,Day,Time,Usec,Lat,Lon,Qual,Mode

    2. ROV file (navYYYYDDDrovname.txt):

      Year,Day,Time,Usec,Lat,Lon,East,North,Pres,Head,Alti,Pitch,Roll,PosFlag,PresFlag,HeadFlag,AltiFlag,AttitFlag

    (East/North are always 0 — eastings/northings code was removed but columns kept for backward compatibility.)

    1. Log file (navYYYYDDDrovname_log.txt): Human-readable filtering report with rules and flagging statistics.

New Workflow

The .txt files in the /mbari/RovNavEdit directories that were created above are the files that will be loaded into Expd database 'raw nav' data tables.

  1. Rov nav .txt files will also be hand-edited by VideoLab personel using the MBSystem or other editing tools.
  2. In the past the year-day organized .txt files were hand-edited, but this produced artifacts at the year-day boundaries
  3. This step will produce a file of the same format with the 'edited' appended to the filename.
  4. Ship nav files are not currently edited.

Note

This completes the processing workflow that occurs on machine coredata8. Everything is now considered archived and staged for database loading. The loads are performed by Windows TaskManager jobs run on the Windows server "Draco"

RovCtd Configuration and Calibration Data

The labview ctd gui running on rc_rctd and wf_rctd computers is where the CTD calibrations info is maintained by the CTD Tech. There is an entire workflow that gets that information to shore and into the Expd database. It is described HERE

Database Loads on Draco

Preliminaries

  1. Load scripts and direct SQL update queries are run via the Windows TaskScheduler on Draco
  2. User login is SHORE\DB_EXPD_TASKEXEC
  3. All scripts and queries reside on drive D: in directory EXPD_Data_Loads
  4. They are all in the bitbucket repository at: Expd_Data_Loads

Tasks on the scheduler typically either:

  1. Execute a .bat file that then calls perl or python programs with cmd line arguments that perform a load
  2. Directly execute a windows executable file (.exe) to perform some task.
  3. Directly execute some SQL command on the SqlServer via SQLCMD.EXE

All jobs are scheduled to run at least nightly. They can also be run at any time by right-click on the job name and selecting 'Run'.

Figure 5 - TaskScheduler on Draco

Expd Data Loads

The common repeated pattern we use to manage all of the various coredata streams from the ships and rov's is use of a 'load table' which is specific for each data stream.

  1. Coredata are logged in 'yearday' data files. Data files are organized in the archives by year and platform name subdirectories
  2. Individual data file loads are managed and tracked by a "load table"
  3. Scripts typically are passed a platform and year cmd line arguments to load.
    1. Scripts create a list of all yearday files for a platform/year
    2. Check each filename against the 'load table' to if exists and if it has been previousy loaded.
    3. if 'isLoaded' or 'isBlocked' the script just skips to next file.
    4. if file is not loaded yet, script takes an entry in the load table and then attempts to load.
    5. if load fails, leave the 'isLoaded' flag = 0 else set it to 1.
  4. If more than one year's worth of data needs to be checked for load/reload, the script simply needs to be called again with different year value.

There may be other fields and flags in a data stream's 'load table' that are specific to the type of data. These may be flags to other downstream processing tasks that also ustilize the 'load table' to manage the workflow.

  1. isLoaded flags typically have a trigger associated which can be used to back-out dependent rows in the database.
  2. isProcessed flag in some tables such as rovctdload has a trigger to back out dependents 'processed' rows in the database.
  3. isBlocked flags can be manually set by the dba to prevent the back-out or reload of data. This flag is rarely used such as when abnormal steps have been taken to manually correct/modify data in the database base tables (such as substituting CTD pressure for Digiquartz pressure when the sensor failed).

Note

Below will describe the load for each major coredata stream. You will notice the above pattern repeated with slight variations.

Shipnav Ingest to Expd

After the extracted navYYYYDDDSSSS.txt files appear in the ATLAS RovNavEdit/YYYY/rcsn or wfly directories they are will be loaded into Expd database.Unlike ROV nav, there is no manual editing step performed on ship nav data, They will be loaded by the next TaskScheduler run of the job EXPD_NightlyNavLoadToPerseus_Early .. or '_Late' - we run twice as some people want the nav data soon after the ship returns. (See also Figure 4 above for how the files get placed here.)

The Draco taskScheduler job runs the large loadAllNightlyNavdata_perseus.bat script which does many things - including invoking the perl program D:\EXPD_Data_Loads\Navdata\LoadShipNavdata_perseus.pl for each ship, by year/yearday.

This perl program scans the Atlas share RovNavEdit/YYYY/rcsn or wfly directory for ship nav files that need loading.

It uses the EXPD database RawShipNavLoad table to determine this.

  1. If there is no entry for the file it attempts a load and makes an entry
  2. If there is an entry and the isLoaded flag false (and isBlocked is not set) it attempts a load.

Note

The perl program has several hard-coded lat/lon locations that MBARI vessels have commonly docked at. This is to prevent loading unnecessary data when the ship is in port. In addition to Moss Landing dock, there are locations for ports of call in Hawaii, Gulf of Mexico, Eureka, Astoria etc.

  1. Should a data reload be needed - using a tool like Aquadata Studio to set the isLoaded flag to 0 will cause a trigger to fire and back out all dependent data rows.

Rovnav Ingest to Expd

Similar to how ship nav data are loaded as described above, ROV nav data is handled much the same way. Raw rov nav files are loaded automatically from the RovNavEdit share on atlas. After raw nav edits are performed by VideoLab personel the raw file, there is an edited file created in the same directory as the raw file. (eg nav2021062vntaedited.txt). This procedure is now deprecated in favor of editing by_dive data files as described in expd_mapping_workflow.

The two 'load tables' for rov are RawRovNavLoad and CleanRovNavLoad. They operate in the same way as the Shipnav Ingest described above.

During the Draco TaskScheduler pass that ran the loadAllNightlyNavdata_perseus.bat script - it invokes several perl calls to load rov nav.

  1. First it calls perl D:\EXPD_Data_Loads\Navdata\UpdateRovNavdata_perseus.pl for each ROV. This step examines timestamps for the edited rov nav files to detect any nav that has been re-edited. If found, that entry's isLoaded flag is set to 0 to cause the previously loaded data to be deleted.

Warning

the .bat script only contains calls to UpdateRovNavdata_perseus.pl for the current year and several years in the past. Every new year it needs to be edited to make entries for the year just ended. See New Year Rollover

1.The .bat script the calls D:\EXPD_Data_Loads\Navdata\LoadRovNavdata_perseus.pl with arguments of vehicle, year vnta|docr raw|clean

  1. The bat script lastly invoke perl program D:\EXPD_Data_Loads\Navdata\MergeNavdata_perseus.pl which will add ship lat/lon/heading back into the raw rov nav tables so it is basically 'pre-joined' for more cconvenient query writing.

  2. Should a data reload be needed - using a tool like Aquadata Studio to set the isLoaded flag to 0 will cause a trigger to fire and back out all dependent data rows.

The loadAllNightlyNavdata_perseus.bat script looks back only a few years for newly edited nav files. Another script, updateAllNightlyNavdata_perseus.bat, loops through all years of ROV data and is scheduled to run nightly by the task scheduler. In order for downtream databases to know when re-edited ROV nav data has been reloaded the start_datetime and end_datetime columns in the CleanRovNavLoad table can be used for the time bounds of data in the reloaded nav file.

Camera Ingest to Expd

Similar to how ship nav data are loaded as described above, ROV camera log data are handled much the same way.

Raw camlog files are loaded automatically. These data are loaded directly from the raw logr files found in the the ShipData/logger share on Atlas. No intermediate file is needed.

The Draco taskScheduler job runs the large loadNightlyCamlog_perseus.bat script which invokes the perl program D:\EXPD_Data_Loads\Camlog\LoadCamlogdata_perseus.pl for each rov, by year/yearday.

This perl program scans the Atlas share ShipData/logger/YYYY/vnta or docr directory for camlog files that need loading

The load table for camlog is CamlogLoad. It operates in the same way as the Shipnav Ingest described above.

Prior to early 2018, either a good tape timecode was required for a camlog record to be inserted. Camlog data was critical for match tape timecode to gmt clock time. When we no longer recorded to tape the perl script was modified to use the ROV.CTD.INWATER flag to only load data while the sub was in use as data such as zoom, focus, iris are still important.

Note

Database load of main camera Pan and Tilt has never been needed. Starting in April 2021 pan and tile are now logged in the camlog file for Ventana. For DocRickets and earlier Ventana, Pan and Tilt were logged in a seperate file - but only for most recent few years.

RovCtd Ingest to Expd

Similar to how ship nav data are loaded as described above, ROVCTD log data are currently handled much the same way.

Raw files are loaded automatically by TaskScheduler job on Draco. These data are loaded directly from the raw logr gzip files found in the ShipData/logger share on Atlas. No intermediate file is needed.

The current load table for managing rovctd loads AND PROCESSING is RawCtdLoad. From late 1999 thru mid-2008 load table is RovCtdLoad. Loads and processing before then was handled quite differently (see Some Improtant History below)

The rovctd load sequence is more complex than for the nav and camlog loads due to the need to apply calibrations, filter and despike the data and store both raw and bin averaged data.

The Draco taskSchedulerjob runs the large loadNightlyRovctd_perseus.bat script which invokes a sequence of perl programs.

  1. Processing is run directly against the raw data in the database using calibration data in the RovCtdCfg table. The .bat script first invokes the perl program D:\EXPD_Data_Loads\RovCtd\updateRovCtdCfg_perseus.pl to make sure the latest/greatest calibrations are in the system. (see: Calibration Workflow

  2. The perl program D:\EXPD_Data_Loads\RovCtd\loadRawRovctdLogrFiles_perseus.pl is run for each rov, by year/yearday. It loads raw (P,T,C and analog data) into 'raw tables' partitioned by vehicle and year (eg tables VentanaRawCtdData_YYYY or DocRickettsRawCtdData_YYYY). This perl program scans the Atlas share ShipData/logger/YYYY/vnta or docr directory for rovctdlogr files that need loading

  3. after running the raw loads for the current year, both vehicles, the bat script runs the raw load for years past that may have been unloaded

Warning

the .bat script only invokes loads for the current year and several years in the past. Every new year it needs to be edited to make entries for the year just ended. See New Year Rollover

  1. The bat script then invokes D:\EXPD_Data_Loads\RovCtd\processRawRovctdLogrData_perseus.pl several times. This is the program that
    1. retrieves raw seabird CTD data from database.
    2. retrieves the correct calibration from the RovCtdCfg table
    3. Despikes and applies calibration, computes derived variables like salinity and O2.
    4. Uploads results to the processed high-frequency data tables
    5. Sends sql cmds necessary to bin the data into the '15sec bin date tables on the server

Note

isLoaded is used to signal the raw data load and trigger the unload if needed. isProcessed is used to signal the processing script or trigger the processed data unload. raw data very seldom needs to be unloaded. isProcessed is often used to cause a reprocessing when calibrations change etc.

Warning

before triggering an unload of raw data via the isLoaded flag, first trigger the unload of the processed records via the isProcessed flag.

Dives and DiveSummary table

Dives and Expeditions are created thru the Expd web pages. Scientists and pilots often think sets of data in terms of dive numbers with start and end times. This is most often the way the database is queried. But core data loads described above have no knowledge of the concept of dive, start, end. Until a Dive is created (usually thru the post-cruise reports) the data cant be shown or queried 'by dive'

There is an important table called DiveSummary which contains certain important statistics related to the dive. It gets a new record inserted whenever a dive is created (insert trigger on table Dive) It also gets updates via Dive table triggers when start/end times are changed. It has a 'dirtybit' that is set whenever these trigger fire to indicate that the dive needs to be re-summarized.

The TaskScheduler job (Expd_UpdateDirtyDiveSummaries) is run nightly that executes the sql contained in the file D:\EXPD_Data_Loads\SqlExecScripts\EXPD_UpdateDirtyDiveSummaries.sql

That query checks the dirtybit for any dives needing attention and computes new statistics then clears then dirtybit.

Some Important History

Rovctd data handling has changed numerous times since we started in 1988.

At that time we only had Ventana. Data were collected on low capacity floppy disks using Seabird CTD PC software. (earliest Ctd data may have been recorded by the MOS system and later moved to Seabird PC software - I cant quite recall 30 years later!)

Because of storage limitations on the hard drives and floppys of the time the data were bin averaged to 15 seconds.

Floppy disks were hand carried from the ship to Pacific Grove by the scientists along with nav data on magnetic tape cartridges from the old 'meridian ocean systems' (MOS) system and VCR tapes.

The data Sybase database "MODB" was implemented to hold the data, and 'load paths' were written in 'C' to process and load the data. CTD data loaded into MODB on the HP850 after being despiked and bin-averaged.

In 1999 we left Sybase for MSSql server and the current Expd database. Legacy data were bulk-copied from MODB to Expd. New programs were implemented in linux to log and process the data. This was due to the need handle the sensors chosen for rov Tiburon which Seabird did not support. (Falmouth sensors, and later the Aanderra O2 Optode)

During the "Tiburon years" (1197-2007/8) the load paths and processing for the two vehicles also diverged due to the differences in the shipboard computing systems. Western Flyer ran the datamanager/navproc architecture that was developed for the Tiburon control system. The Point Lobos continued to run the old MOS for nav. Later ( ~8/1999 ) the Pt Lobos was upgraded to the datamanager/navproc architecture for core datastreams (although not for the subsea control system as with the datamanager/navproc on Tiburon).

The older data 1988 thru approximately 2009 have a very different archive layout and processing history. The source raw and processed files are archived on the Atlas ShipData share in the subdirectory "rovctd". The directory layout is rovname/year/yrday. The critical files for the database load scripts of the time were the "der" (derived) files (eg. 2008316vntaCtdDer.txt). These could be either 15 second average, later 2 second average, and eventually 1 second was used (and the bin averaging was eventually done directly in the database via SQL cmds as described above in RovCtd Ingest.)

Warning

From 1988 thru the end of 1997 we only archived 15-second bin data. Raw data starts in 1998 in Expd. We continue to compute 15 sec bin data as users want to be able to query or compute statistics accross the entire history. Raw high frequency (1 or 2 sec) started to be loaded directly into base tables in 1988. Prior to that we only stored 15-second processed data in 15 second bin averages.

NO_ITEM, NO_PROV, NO_PUB, NO_SUB In Log Files and Usage In Processing

When the log files are generated on the ship, there is logic to decide if there should be a special flag written in place of the data itself for certain situations. There are four possible flags that can be written and they are NO_ITEM, NO_PROV, NO_PUB, and NO_SUB. The meaning behind each of this is as follows:

  1. NO_ITEM: This means that in the configuration for the logger on the ship, the item name for this column was not available in the data. This can happen when there just isn't any data available in the LCM message coming from the publisher.
  2. NO_PROV: There are subtle differences depending on the instrument providing the data and setting this flag, but mainly it indicates if data timed out, was not received correctly or not parsed correctly.
  3. NO_PUB: The logger looks to see if the timestamp when the data was last parsed successfully is past a pre-defined 'maxAge' in the logger configuration file, if so, this flag gets set
  4. NO_SUB: This indicates that while an item was specified in the configuration file for the logger, no data subscriber was actually instantiated and usually indicates a logger mis-configuration.

In order to understand how those flags are used in data processing, the table below describes if and how each flag is used in the processing scripts.

Status processNav.pl loadRawRovctdLogrFiles_perseus.pl LoadCamlogdata_perseus.pl LoadLegacyCamlogdata_perseus.pl LoadShipNavdata_perseus.pl
NO_ITEM Equivalent to NO_PROV and indicates no data available for the field. Usually replaced missing value Equivalent to NO_PROV and indicates no data available for the field. Usually replaced missing value Equivalent to NO_PROV and indicates no data available for the field. Usually replaced missing value Equivalent to NO_PROV and indicates no data available for the field. Usually replaced missing value Equivalent to NO_PROV and indicates no data available for the field. Usually replaced missing value but is also used to check for valid GPS data (return 1 or 0 depending)
NO_PROV Equivalent to NO_ITEM and indicates no data available for the field. Usually replaced missing value Equivalent to NO_ITEM and indicates no data available for the field. Usually replaced missing value Equivalent to NO_ITEM and indicates no data available for the field. Usually replaced missing value Equivalent to NO_ITEM and indicates no data available for the field. Usually replaced missing value Equivalent to NO_ITEM and indicates no data available for the field. Usually replaced missing value but is also used to check for valid GPS data (return 1 or 0 depending)
NO_PUB Not used Not used Not used Not used Not used
NO_SUB Not used Not used Not used Not used Not used

Data Workflow Diagram

graph LR subgraph Carson subgraph GlacierWind subgraph VNTA_CTD_Config sourcetable[VNTA_CTD_Config] end subgraph VNTA_DiveLog divesourcetable[Dive_Ventana] end end end subgraph Shore subgraph Perseus subgraph EXPD RovCtdCfg_Ventana VentanaPilotsDive end end end subgraph Perseus subgraph StoredProcedures sp_expireWaypointsMsg AdminUpdateDiveSummaryByDiveID end end subgraph Draco subgraph 3DReplay subgraph geo3DreplayJob_perseus.bat makegeo3DReplayFiles_perseus.pl makeKmlFiles_perseus.pl end end subgraph DocRickettsCtdCfgToPerseus cpyRovCtdCfg2Perseus.exe[cpyRovCtdCfg2Perseus.exe
Note this is a XOJO App built to an executable] end subgraph DocRickettsDivelogToPerseus cpyDivelog2Perseus.exe[cpyDivelog2Perseus.exe
Note this is a XOJO App built to an executable] end subgraph EXPD_ExpeditionDataLoads subgraph ExpeditionDataLoads_perseus.bat popExpeditionData_perseus.pl popEDwithDBqueries_perseus.pl end end subgraph EXPD_ExpireAndDeleteWaypoints_0 EXPD_ExpireAndDeleteWaypoints_0.sql --call--> sp_expireWaypointsMsg end subgraph EXPD_ExpireAndDeleteWaypoints_7 EXPD_ExpireAndDeleteWaypoints_7.sql --call--> sp_expireWaypointsMsg end subgraph EXPD_NightlyCamlogLoadToPerseus_Late subgraph loadNightlyCamlog_perseus.bat LoadCamlogdata_perseus.pl end end subgraph EXPD_NightlyMinirovNavLoadToPerseus subgraph load_minirov_nav.bat urn1[UpdateRovNavdata_perseus.pl] lrn1[LoadRovNavdata_perseus.pl] end end subgraph EXPD_NightlyNavLoadToPerseus_Early subgraph loadALLNightlyNavdata_perseus.bat urn2[UpdateRovNavdata_perseus.pl] lrn2[LoadRovNavdata_perseus.pl] end end subgraph EXPD_NightlyNavLoadToPerseus_Late note3[Same as EXPD_NightlyNavLoadToPerseus_Early] end subgraph EXPD_NightlyNavUpdateToPerseus subgraph updateALLNightlyNavdata_perseus.bat urn3[UpdateRovNavdata_perseus.pl] lrn3[LoadRovNavdata_perseus.pl] end end subgraph EXPD_NightlyProcessMiniRovctdToPerseus subgraph processRawMinirovCtd.bat processMinirovCtdData_perseus.pl end end subgraph EXPD_NightlyRovctdLoadToPerseus_Early subgraph loadNightlyRovctd_perseus.bat updateRovCtdCfg_perseus.pl loadRawRovctdLogrFiles_perseus.pl processRawRovctdLogrData_perseus.pl end end subgraph EXPD_NightlyRovctdLoadToPerseus_Late note4[EXPD_NightlyRovctdLoadToPerseus_Early] end subgraph EXPD_UpdateDirtyDiveSummaries EXPD_UpdateDirtyDiveSummaries.sql --call--> AdminUpdateDiveSummaryByDiveID end subgraph VentanaCtdCfgToPerseus sourcetable --> cprovctd2[cpyRovCtdCfg2Perseus.exe
Note this is a XOJO App built to an executable] --> RovCtdCfg_Ventana end subgraph VentanaDivelogToPerseus divesourcetable --> cpdive2[cpyDivelog2Perseus.exe
Note this is a XOJO App built to an executable] --> VentanaPilotsDive end end click cpyRovCtdCfg2Perseus.exe href "https://bitbucket.org/mbari/cpyrovctdcfg2perseus/src/master/" click cpyDivelog2Perseus.exe href "https://bitbucket.org/mbari/cpydivelog2perseus/src/master/" click popExpeditionData_perseus.pl href "https://bitbucket.org/mbari/expd_data_loads/src/master/Expedition/popExpeditionData_perseus.pl" click popEDwithDBqueries_perseus.pl href "https://bitbucket.org/mbari/expd_data_loads/src/master/Expedition/popEDwithDBqueries_perseus.pl" click EXPD_ExpireAndDeleteWaypoints_0.sql href "https://bitbucket.org/mbari/expd_data_loads/src/master/SqlExecScripts/EXPD_ExpireAndDeleteWaypoints_0.sql" click EXPD_ExpireAndDeleteWaypoints_7.sql href "https://bitbucket.org/mbari/expd_data_loads/src/master/SqlExecScripts/EXPD_ExpireAndDeleteWaypoints_7.sql" click LoadCamlogdata_perseus.pl href "https://bitbucket.org/mbari/expd_data_loads/src/master/Camlog/LoadCamlogdata_perseus.pl" click urn1 href "https://bitbucket.org/mbari/expd_data_loads/src/master/Navdata/UpdateRovNavdata_perseus.pl" click lrn1 href "https://bitbucket.org/mbari/expd_data_loads/src/master/Navdata/LoadRovNavdata_perseus.pl" click urn2 href "https://bitbucket.org/mbari/expd_data_loads/src/master/Navdata/UpdateRovNavdata_perseus.pl" click lrn2 href "https://bitbucket.org/mbari/expd_data_loads/src/master/Navdata/LoadRovNavdata_perseus.pl" click urn3 href "https://bitbucket.org/mbari/expd_data_loads/src/master/Navdata/UpdateRovNavdata_perseus.pl" click lrn3 href "https://bitbucket.org/mbari/expd_data_loads/src/master/Navdata/LoadRovNavdata_perseus.pl" click processMinirovCtdData_perseus.pl href "https://bitbucket.org/mbari/expd_data_loads/src/master/RovCtd/processMinirovCtdData_perseus.pl" click updateRovCtdCfg_perseus.pl href "https://bitbucket.org/mbari/expd_data_loads/src/master/RovCtd/updateRovCtdCfg_perseus.pl" click loadRawRovctdLogrFiles_perseus.pl href "https://bitbucket.org/mbari/expd_data_loads/src/master/RovCtd/loadRawRovctdLogrFiles_perseus.pl" click processRawRovctdLogrData_perseus.pl href "https://bitbucket.org/mbari/expd_data_loads/src/master/RovCtd/processRawRovctdLogrData_perseus.pl" click cprovctd2 href "https://bitbucket.org/mbari/cpyrovctdcfg2perseus/src/master/" click cpdive2 href "https://bitbucket.org/mbari/cpydivelog2perseus/src/master/"

TODOS

  1. Find the source code and get into repository for makegeo3DReplayFiles_perseus.pl
  2. Find the source code and get into repository for makeKmlFiles_perseus.pl
  3. Link the Draco processes to what source log files they pull from in the diagram above.