Skip to content

ROVCTD Calibration Coefficients Workflow

Preliminaries

The LabVIEW CTD GUI should be running on rc_rctd and (TBD) computers. These computers are where the CTD calibrations info is maintained by the CTD Tech. Note that only raw ctd data is logged and loaded into the the EXPD database. Furthermore, the on-shore data archive can be subject to reprocessing anytime issues are discovered post-cruise. On-shore processes need access to the full history of ctd configuration and calibrations.

Also, there should be a Python-based Windows service running on the rc-rctd and (TBD) computers that monitors the LabVIEW CTDCoeff file and pushes any changes to a database on the ship. This service is documented here.

The CTDCoeff File

All things start with the CTD Tech or pilots modifying the CTDCoeff.ini file on the shipboard rc-rctd or (TBD) windows computers where the Labview GUI is run. The Python script mentioned in the previous paragraph makes sure any time the coeff file gets updated, the contents of the file get inserted into a database on the ship.

sequenceDiagram autonumber participant CTDTech participant CTDCoeff.ini participant CfgFileMonitorService participant alaskanwind.DOCR_CTD_Config CTDTech->> CTDCoeff.ini:updates CfgFileMonitorService-->>CTDCoeff.ini:detects change CfgFileMonitorService-->>alaskanwind.DOCR_CTD_Config:inserts file into database record
  1. The CTD tech edits the file using calibrations from Seabird (or others) and saves it.
  2. The Python file monitoring service running on the same computer detects the change
  3. The Python file monitoring service will update the shipboard database on Glacierwind or Alaskanwind (typically takes < 60 seconds ).

Check the CTDcoeff.ini file

Note

I encourage you to check the date stamp of the CTDCoeff.ini on each system periodically and compare it to the datetime stamp in the calibration tables in Expd.

Warning

If you want to Remote Desktop into rc-rctd or (TBD), ask first as you will bump off whoever is on that !!!

Check the file in use on the ship walking over to the boat or use remote desktop. Below is a sample screengrab of the directory from the WesternFlyer's wf-rctd.

Figure 1 - Screengrab Figure 1: Screengrab of CTD Coefficient files on WesternFlyer

Examine the results (see Figure 1):

  • Note the timestamp of the current CTDcoeff.ini (8/6/2020 10:25 in this example) this should be the file with the ‘new/current’ sensors.
  • Also note the CTDTech has carefully made a safety copy of the prior file.

Verify Cfg Loaded to Shipboard Database

Next, we need to verify any new cfgs came thru to the shipboard database. On glacierwind.rc (Carson) or alaskanwind.dp (Packard) (remotely via Aquadata Studio or some other client), run the following query against the VNTA_CTD_Config table of hte VNTA_CTD_Config database (Carson) or the DOCR_CTD_Config table of the DOCR_CTD_Config database (Packard). Sorry, probably should have given them different names.

SELECT top 10 * FROM [dbo].[DOCR_CTD_Config] ORDER BY eventtime desc
Here is the database and table.

Figure 2a - Screengrab

Figure 2a: The on-ship database

Here are the query results.

Figure 2b - Screengrab

Figure 2b: Query run against on-ship database

Examine the results in Figure 2b above.

  • Note in this case the ‘localtime’ column shows two updates – the most recent one matches the most recent file in Figure 1. The second-most-recent, while not exact match - is consistent with CTDTech having modified the file and letting it post, then renaming and (saving) it as a backup. It is not unusual for there to be multiple 'saves'.
  • Note the ‘id’ column shows the database assigned key for the entries – jot those down, we will use those values next.

Verify Cfg Loaded to On-shore Database

Next we can verify the workflow that gets the config changes from shipboard databases to on-shore database.

sequenceDiagram autonumber participant DracoTaskScheduler participant DocRickettsCtdCfgToPerseus participant cpyRovCtdCfg2Perseus.exe participant alaskanwind.DOCR_CTD_Config participant perseus.RovCtdCfg_DocRicketts DracoTaskScheduler->> DocRickettsCtdCfgToPerseus:runs task DocRickettsCtdCfgToPerseus->>cpyRovCtdCfg2Perseus.exe:executes cpyRovCtdCfg2Perseus.exe-->>perseus.RovCtdCfg_DocRicketts:deletes all records cpyRovCtdCfg2Perseus.exe-->>alaskanwind.DOCR_CTD_Config: reads all records from cpyRovCtdCfg2Perseus.exe-->>perseus.RovCtdCfg_DocRicketts: inserts all records to

We can verify any new configurations came through to the on-shore database using Aquadata Studio (or other) to query the EXPD database on perseus (as user=expddba). You can check that they made it from GlacierWind/Alaskanwind to the temporary tables on Perseus EXPD database by running:

SELECT top 10 * FROM [dbo].[RovCtdCfg_DocRicketts] ORDER BY eventtime desc

Figure 3 - Screengrab

Figure 3 Checking if the cfgs made it from ship to the staging table in Expd on perseus

Examine the results (See Figure 3):

  • Make sure all events have come across using the ‘id’ keys obtain from Fig 2.
  • If not ... check using the task scheduler on Draco. Run the DocRickettsCtdCfgToPerseus job (or Ventana’s) and then re-run the SQL query above.

Cfg Cascades to RovCtdCfg Table

The configuration files are then parsed and the information is cascaded up to the RovCtdCfg table on perseus.

sequenceDiagram autonumber participant DracoTaskScheduler participant EXPD_NightlyRovctdLoadToPerseus participant loadNightlyRovctd_perseus.bat participant updateRovCtdCfg_perseus.pl %%participant loadRawRovctdLogrFiles_perseus.pl participant table.RovCtdCfg_DocRicketts participant table.RovCtdCfg participant loadRawRovctdLogrFiles_perseus.pl DracoTaskScheduler-->>EXPD_NightlyRovctdLoadToPerseus: Runs task EXPD_NightlyRovctdLoadToPerseus-->>loadNightlyRovctd_perseus.bat: Calls script loadNightlyRovctd_perseus.bat-->>updateRovCtdCfg_perseus.pl: Executes updateRovCtdCfg_perseus.pl-->>table.RovCtdCfg_DocRicketts:New configs ? updateRovCtdCfg_perseus.pl-->>table.RovCtdCfg:Parse each new config updateRovCtdCfg_perseus.pl-->>table.RovCtdCfg:Insert each parsed config loadNightlyRovctd_perseus.bat-->>loadRawRovctdLogrFiles_perseus.pl: Execute for each year/vehicle

To check the results of the above workflow, you can run the following on EXPD database on perseus.

SELECT top 10 * FROM [dbo].[RovCtdCfg] ORDER BY startdtg desc

Figure 4 - Screengrab

Figure 4

Examine the results (see Figure 4):

  • You’ll notice combined data for both vehicles are in this table.
  • Don’t expect start/end dtg’s to match the CTDCoeff.ini file timestamps... aren’t supposed to... (startdtg, enddtg is when a calibration is to be applied over a range of dives)
  • What is important is that cfgfileid_fk fields match the ‘id’ fields of what came off of the ships.
  • You will notice that when compared to Figure 3 above, the most recent configs from the DocRicketts haven’t cascaded up yet.
  • In this case, I happened to be working on this the ‘same day’ as Erich has made his updates. Since the cascade is set to run in the evening, they haven't sync'd up yet. (task EXPD_NightlyRovCtdLoadToPerseus had not run yet).
  • Also note how the old, last-known docr config (cfgfileid_fk = 342) is valid beginning startdtg in October 2019, with no enddtg specified (this is the sign to the CTD processing it is the ‘most current’ calibration)

Force a Cascade If needed

So - based on results above – we can either wait for the nightly job to run, or we can go to Draco and force the cascade to happen immediately as follows.

  1. Using RemoteDesktop, log in to draco as SHORE\DB_EXPD_TASKEXEC
  2. Open/Run TaskScheduler
  3. Execute job: EXPD_NightlyRovCtdLoadToPerseus_early (it includes a step that runs the calibration cascade, (specifically the script updateRovCtdCfg_perseus.pl)
  4. Now re-run the above check, again on perseus (as expddba)
SELECT top 10 * FROM [dbo].[RovCtdCfg] ORDER BY startdtg desc

Figure 5 - Screengrab

Figure 5

Examine the results again carefully (see figure 5)

  • Ok, now there are entries for the two new cfgfileid_fk’s.
  • The cascade update program also closed-out (by setting the ‘enddtg’ the old October 2019 config (cfgfileid_fk=342).

Note that the new calibration has its enddtg is null – indicating to downstream processing that it is the ‘most current’ calibration.

Warning

The startdtg and enddtg fields of the RovCtdCfg table are LOCAL time. There is a trigger on the RovCtdCfg table startdtg and enddtg fields that does a conversion back to GMT fields (further to the right in the table, out of view in the figure).

Reprocess data or add/modify a configuration

There are scenarios where you have to add or force a new calibration record. Most often this is in response to new slope and offset to O2 calibration from winkler titration analysis.

That process is described here