MBARI Mooring Data Pipeline

Background: The Legacy Pipeline

Before mooring-ssds, mooring data were processed by a suite of Perl scripts that fetched raw records from SSDS, Ferret scripts that produced NetCDF files and GIF time-series plots, and Matlab routines for QC and derived products. The canonical NetCDF files produced by that pipeline follow this naming pattern:

OS_M1_<YYYYMMDD><res>_CMSTV.nc          # combined all
OS_M1_<YYYYMMDD><res>_TS.nc             # CTD temperature/salinity profiles
OS_M1_<YYYYMMDD><res>_M.nc              # meteorological

Those files and their associated GIF plot galleries live under:

<ssdsdata>/deployments/<mooring>/<YYYYMM>/
<ssdsdata>/deployments/<mooring>/<YYYYMM>/gifs/

What mooring-ssds Replaces

mooring-ssds is a Python/Django reimplementation of that pipeline. Each management command is a drop-in replacement for one legacy script:

mooring-ssds command Legacy script(s) replaced
ds_to_netcdf Perl SSDS fetch + Ferret per-instrument NetCDF
combine_ts combineTS.pl + Ferret gridding
combine_m combineM.pl + Ferret gridding
combine_all combineAll.pl + Ferret CMSTV merge
plot_netcdf Ferret WindTstring.jnl and per-variable GIF generation

Parallel Operation Period

To validate accuracy before fully switching over, mooring-ssds is currently run in parallel with the legacy pipeline. Both sets of products are written to the same deployment directories on ssdsdata.

The _py suffix distinguishes mooring-ssds products from legacy ones:

OS_MBARI-M1_<YYYYMMDD>hourly_R_CMSTV_py.nc     ← mooring-ssds (this project)
OS_M1_<YYYYMMDD>hourly_CMSTV.nc                 ← legacy Perl/Ferret

The _py suffix is embedded in the filename (before .nc) so legacy tools that glob for *.nc without the suffix can ignore the new files, and vice versa. Code in this project always operates exclusively on *_py.nc files.

Plots follow the same co-location strategy. plot_netcdf writes PNG files into the existing gifs/ subdirectory alongside the legacy GIF files:

gifs/
  OS_MBARI-M1_20250724hourly_R_CMSTV_last30_nc/   ← mooring-ssds PNGs
    wind_TEMP.png
    wind_PSAL.png
  OS_M1_20250724hourly_CMSTV_last30_nc/            ← legacy GIFs
    wind_SEA_WATER_TEMPERATURE_HR_last30.gif

Placing new and old plots side-by-side in gifs/ lets operators compare them visually without any filesystem reorganisation. The gallery web view (views_plots.py) renders both sets from the same directory.

OceanSITES File Naming

mooring-ssds follows the OceanSITES v1.4 file naming convention (see the OceanSITES Data Format Reference Manual) for its combined outputs:

OS_MBARI-<MOORING>_<YYYYMMDD><res>_R_<TYPE>_py.nc
  • <MOORING> — uppercase mooring ID: M0, M1, M2
  • <YYYYMMDD> — deployment start date (from the SSDS root deployment record)
  • <res> — resolution token, directly appended (no separator): hourly or tenMinute
  • R — data mode: Realtime (all SSDS streams are realtime)
  • <TYPE> — OceanSITES data type: TS (temperature/salinity), M (met), CMSTV (all combined)
  • _py — distinguishes this file from the legacy product

Per-instrument files (written by ds_to_netcdf) use a shorter pattern:

<mooring>_<devicetype>_<YYYYMMDD>_original_py.nc

Transition Plan

The parallel period ends when operators are confident that mooring-ssds products are accurate. At that point:

  1. Legacy Perl/Ferret/Matlab scripts are retired.
  2. The _py.nc suffix may be dropped (or kept for provenance).
  3. The gifs/ directory continues to be the plot output location; it just no longer contains legacy GIF files.

Until then, any script or view in this project that globs for files must use *_py.nc patterns to avoid accidentally consuming legacy files.