Adding Delayed Mode QC'd Data File to the Archive

A delayed-mode (DM) file is a QC'd NetCDF file produced after a mooring deployment is recovered — as opposed to a realtime file, which is produced automatically while the deployment is still in the water. Each closed deployment needs three DM file kinds before it's considered complete: M (met), TS (Tstring temperature/salinity), and adcp. See MBARI Mooring Data Pipeline for how DM and realtime data fit together in the pipeline as a whole.

This guide covers what to do once a new DM file has been QC'd and is ready to add to the archive.

Step 1 — Test the New File

Open in Ferret and Plot Some Data

Before archiving the file, open it in Ferret and spot-check it:

yes? use "m0_D_M.nc"
yes? show data
yes? plot/vs TIME AIRT

I like to use Ferret because it will report a problem if the time axis has any out of order values, it also uses the dataset's metadata for labeling the plot. It's a quick way to evaluate the file, but other tools (e.g R, Xarray, Pandas, Matlab) may also be used.

Confirm the time range matches the deployment, that values are in a sane physical range, and that there are no obvious gaps or spikes that QC should have caught.

Examine Metadata and Naming Convention

Check that the QC flag variable is present and populated (flag 0 = good, 1 = bad, 2 = probably bad, 3 = missing — only flag 0 data makes it into the combined archive files, see combine_delayed.py), and that the global attributes (deployment name, mooring, date range) match the deployment being processed.

The archive recognizes each DM file kind by a fragment of its file name, not by inspecting its contents, so getting the name right matters:

Kind	Recognized by	Example
M (met)	`*_D_M.nc`	`m0_D_M.nc`
TS (Tstring)	a `D` token immediately followed by a `TS` token	`OS_M0_20250601_D_TMTS.nc`, `TM_D_TS.nc`
adcp	`_D_ADCP.nc`	`m0_D_ADCP_20250601.nc`

The D marks delayed mode, matching OceanSITES' R/D data-mode codes. The ideal OceanSITES ordering is {data mode}_{measurement type}_{location}_ {frequency} (e.g. D_TS_TM.nc for a delayed-mode, 10-minute TS file, where TM is MBARI's own 10-minute marker, not an OceanSITES code) — but the archive also has two non-conforming Tstring variants in the wild, TM_D_TS.nc and D_TMTS.nc, which are still recognized as TS files. If you're replacing a file that's already in the archive, name the new one consistently with what's already there rather than introducing a fourth variant.

If you're superseding an existing file rather than replacing it outright, rename the old one to include old in its name (e.g. m0_D_M_old.nc) instead of deleting it — file discovery skips any name containing old, so this keeps the earlier version around without it being picked up again.

One more naming detail matters for a genuinely new file (not just replacing one already in the archive): combine_delayed — the command that rebuilds the mooring's DM record from all 3 file kinds — only globs OS_*.nc directly under the netcdf/ directory. A file without the OS_ prefix may still show up as a link on the deployments dashboard, but it won't be picked up when combine_delayed runs in Step 3. Give the file an OS_-prefixed name unless you're intentionally matching an existing non-conforming file already in the archive.

Step 2 — Copy the File to the Archive

DM files live under the oasis_coredata deployments tree, in a directory named for the mooring and the deployment's start year and month:

<coredata-root>/<mooring>/<YYYYMM>/<mooring>/netcdf/

For example, M1's June 2025 deployment archives to:

/mbari/oasis_coredata/deployments/m1/202506/m1/netcdf/

<mooring> is lower case and repeated (once for the deployment directory, once for the per-mooring subdirectory beneath it — this mirrors the layout of the realtime archive under ssdsdata/deployments/<mooring>/<YYYYMM>/, see MBARI Mooring Data Pipeline). <YYYYMM> is the deployment's start date, not the date the file happens to be QC'd or copied.

Copy the file into that directory alongside the mooring's other DM files for that deployment.

Step 3 — Execute Scripts

Run the delayed-mode processing script for the mooring:

bin/process_mooring_delayed.sh <MOORING>

This runs combine_delayed → daily_averaged → plot_netcdf, then combine_realtime → plot_netcdf (see the script for the exact commands). It's safe to run any time after copying in a new DM file — combine_realtime checks each deployment for itself and only stitches in realtime data for a deployment that's still missing one of its 3 DM file kinds.

Once the deployment you just added a file for has all 3 kinds present, combine_realtime stops filling in realtime data for it, and its data comes from the QC'd DM archive from then on. You can confirm this on the deployments dashboard (/deployments/<mooring>/), which flags each deployment as complete once all 3 DM kinds are found in its directory.

Each command skips its work when its output is newer than its inputs. To bypass these freshness checks, pass --force:

bin/process_mooring_delayed.sh --force <MOORING>

This is useful, for example, when the software has been updated but the input files have not — the outputs would otherwise be considered up to date and left unchanged. --force is passed to every command in the pipeline, so expect the full run time.

The final hourly combined file is about 2 GB in size and takes about an hour to generate on the MBARI network linux machine mooring-ssds.shore.mbari.org.

Notes

Data mode matters for trust: DM data has been through QC and is the authoritative record for a closed deployment; realtime data is a best-effort stand-in used only until DM data catches up.
Order of archiving doesn't matter: the 3 DM kinds can be added in any order — completeness is just "have all 3 been seen for this deployment directory," not "were they added in a particular sequence."