Skip to content

Expedition Database Staging Environment

Staging Setup

--- title: High Level View of System --- flowchart LR subgraph simnavproc direction LR navproc --> logr --> logr-data-file[YYYYDDDlogr.dat] --> nightly-job end subgraph MBARI FTP direction LR ftp-dir[incoming/corenav_sim] end subgraph sim-coredata direction LR local-logr-files[~/shipandrov/datatransfer/sim] end subgraph Atlas-ShipData direction LR logger-files[logger/YYYY/sim] end subgraph Atlas-RovNavEdit direction LR ship-edit-logs[YYYY/sim] rov-edit-logs[YYYY/sim-rov] end subgraph Glacierwind ctd-config[(VNTA_CTD_Config)] dive-log-db[(Dive_Ventana)] end subgraph Perseus expd[(EXPD)] end subgraph Staging DB Server sim-expd[(SIM_EXPD)] sim-ctd-config[(SIM_CTD_Config)] sim-dive-log-db[(Dive_Sim)] end subgraph sim-draco tasks[All Tasks] end subgraph SIM EXPD Web Server app[IIS and Scripts] end nightly-job --> ftp-dir ftp-dir --> local-logr-files local-logr-files --> logger-files local-logr-files --> ship-edit-logs local-logr-files --> rov-edit-logs logger-files --> tasks ship-edit-logs --> tasks rov-edit-logs --> tasks ctd-config -- IS --> sim-ctd-config dive-log-db -- IS --> sim-dive-log-db expd -- IS --> sim-expd tasks --> sim-expd sim-expd --> app schema-change-scripts[Schema Migration Scripts] --> sim-expd note-1>Manual Edit Happens Here] --- rov-edit-logs

Tasks

Note

For the database development, we should have the databases overwritten every night and then as we develop changes, we should document and write DDL/Scripts to make the schema changes as we develop them and apply those nightly after the databases are overwritten. This ensures we will be ready when we want to move the production DB to the new schema

  1. Have IS set up nightly jobs to replicate (overwrite) the following databases to a development MSSQL server (foggy?)
    1. EXPD on Perseus to EXPD on dev MSSQL
    2. VNTA_CTD_Config on Glacierwind to SIM_CTD_Config on dev MSSQL
    3. Dive_Ventana on Glacierwind to Dive_Sim on dev MSSQL
  2. Create job to FTP files off simnavproc to incoming/corenav_sim on ftp.mbari.org
  3. Ask IS to spin up VM named sim-coredata.shore.mbari.org
    1. Add accounts for Brian, Kevin, Mike, and Karen
    2. Allow for all users to 'sudo -u coredata -i' (everything should run as coredata user)
  4. Set up scripts and cron on sim-coredata to copy files from FTP to local storage /u/coredata/shipandrov/datatransfer/sim
  5. Set up scripts and cron on sim-coredata to copy files from local storage to Atlas share ShipData under the logger/YYYY/sim directory.
  6. Set up scripts (processNav.perl) and cron on sim-coredata to copy files from local storage to Atlas share RovNavEdit under the YYYY/sim directory.
  7. NOTE This is where manual edit process would happen
  8. Have IS duplicate Draco to another VM (sim-draco?), but disable all tasks in task scheduler
  9. Go through each task on sim-draco, migrate to the simulation locations, document them and then re-enable them.
  10. Build new XOJO application for copy CTD data from CTD database to EXPD database
  11. Build new XOJO application for copy Dive log data from Dive log DB to EXPD database.
  12. Have IS duplicate Windows box with web application(?)

TODO kgomes - document how to add mini rov pipeline to staging environment