Skip to content

Expedition Mapping Database Workflow

Overview

Please see the Expedition Mapping Database page on BitBucket for an overview of the Expedition Mapping Database, which also includes details of running a local instance of the database and deploying the prodcution database on https://emdb.shore.mbari.org.

Crontab jobs on emdb.shore.mbari.org

These jobs run each night starting around 4 pm PM local time. They create new raw nav.txt files and load data from EXPD into EMDB:

  1. Add any new dives to EMDB that have been entered into EXPD
  2. Create new _by_dive raw nav.txt files files
  3. Load newly edited ROV navigation data into EMDB making them available on the dive detail page as the rov_by_dive_edited_file layer
  4. Load newly edited ROV navigation data into EXPD's CleanRovnavData tables, also updating the CleanRovNavLoad tracking table
  5. Load updated records from EXPD's CleanRovnavData tables into EMDB's CleanRovnavData layer
  6. Update Expd’s RovctdBinData tables (elon & elat columns) with newly edited and loaded ROV nav data 
    10 16 * * * time /usr/local/bin/docker-compose -f /opt/expd_mapping/emdb/production.yml run --rm --name add_new_dives -u 11903 -e DOCKER_USER=11903 django scripts/load.py -v --add_new_dives >> /tmp/load_add_new_dives.out 2>&1
20 16 * * * time /usr/local/bin/docker-compose -f /opt/expd_mapping/emdb/production.yml run --rm --name gen_new_raw_nav -u 11903 -e DOCKER_USER=11903 django scripts/gen_nav_by_dive.py -v --new_rawnav >> /tmp/gen_new_rawnav.out 2>&1
00 17 * * * time /usr/local/bin/docker-compose -f /opt/expd_mapping/emdb/production.yml run --rm --name load_by_dive_edited -u 11903 -e DOCKER_USER=11903 django scripts/load.py -v --load_by_dive_edited --lookback_days 1 >> /tmp/edited_navfiles.out 2>&1
10 17 * * * time /usr/local/bin/docker-compose -f /opt/expd_mapping/emdb/production.yml run --rm --name load_edited_into_expd -u 11903 -e DOCKER_USER=11903 django scripts/load_edited_into_expd.py -v --last_n_days 1 >> /tmp/load_edited_into_expd.out 2>&1
50 17 * * * time /usr/local/bin/docker-compose -f /opt/expd_mapping/emdb/production.yml run --rm --name updated_navfiles -u 11903 -e DOCKER_USER=11903 django scripts/load.py -v --updated_navfiles --lookback_days 1 >> /tmp/load_updated_navfiles.out 2>&1
10 18 * * * time /usr/local/bin/docker-compose -f /opt/expd_mapping/emdb/production.yml run --rm --name load_cleanrovnav_into_rovctdbin -u 11903 -e DOCKER_USER=11903 django scripts/load_cleanrovnav_into_rovctdbin.py -v --db_alias expd_perseus --last_n_days 1 >> /tmp/load_cleanrovnav_into_rovctdbin.out 2>&1

Workflow for "_by_dive" organized ROV Navigation Data

The legacy data processing workflow as outlined in Figure 4 of ship_and_rov_core_data_workflow.md is used to populate the "RawRovnavData" tables from which raw data are loaded into EMDB and raw _by_dive .txt files are created. Dive numbers and times need to be entered into EXPD before this can be done by expd_mapping's gen_nav_by_dive.py script. These Figure shows the general workflow:

graph TD RawRovnavData[("EXPD RawRovnavData tables")] by_dive_file[("Atlas /mbari/RovNavEdit/YYYY/*_by_dive/ .txt files")] RRD["RawRovnavData layer"] emdb[("emdb.shore.mbari.org web app")] subgraph "Expedition Mapping Database RawRovnavData Data Flow" RawRovnavData-- gen_nav_by_dive.py -v -->by_dive_file RawRovnavData-- "load.py --load_rawnav -v" -->RRD RRD-->emdb end

FIGURE 1. Raw ROV navigation data is loaded into EXPD's RawRovnavData tables from year-day organized rovnav files as depicted in Figure 4 of ship_and_rov_core_data_workflow.md. EMDB's gen_nav_by_dive.py script creates _by_dive .txt files from data in the RawRovnavData tables provided that dive numbers and times have been entered into EXPD.

In 2022 we determined that editing the ROV navigation data form "_by_dive" organized .txt files would be better than editing from "year-day" organized files. A collection of these _by_dive files was created by running gen_nav_by_dive.py --edited -v. These files are then hand edited using mbnavedit, QGIS, and any other tool that's desired. The _by_dive navedited.txt files are then loaded into EMDB's rov_by_dive_edited_file layer with load.py --load_by_dive_edited -v (this is so that the edits can be validated using the Dive Detail view of the EMDB web app). The navedited.txt files are also loaded into EXPD's CleanRovnavData tables using load_edited_into_expd.py -v. The load_edited_into_expd.py script also updates the CleanRovNavLoad tracking table. The following Figure shows the general workflow:

The following Figure shows the general workflow:

graph TD by_dive_file[("Atlas /mbari/RovNavEdit/YYYY/*_by_dive/ navedited.txt files")] CleanRovnavData[("EXPD CleanRovnavData tables")] rbde["rov_by_dive_edited_file layer"] emdb[("emdb.shore.mbari.org web app")] CRD["CleanRovnavData layer"] subgraph "Expedition Mapping Database _by_dive edited and CleanRovnavData Data Flow" CleanRovnavData -- "gen_nav_by_dive.py -v --edited" -->by_dive_file by_dive_file--Human edits-->by_dive_file by_dive_file-- load_edited_into_expd.py -v -->CleanRovnavData by_dive_file-- "load.py --load_by_dive_edited -v" --> rbde rbde-->emdb CleanRovnavData-- "load.py --updated_navfiles -v" --> CRD CRD-->emdb end

FIGURE 2. Data flow for "_by_dive" organized ROV navigation data. The _by_dive navedited.txt files are loaded into EMDB's rov_by_dive_edited_file layer with load.py --load_by_dive_edited -v (this is so that the edits can be validated using the Dive Detail view of the EMDB web app). The navedited.txt files are also loaded into EXPD's CleanRovnavData tables using load_edited_into_expd.py -v. The load_edited_into_expd.py script also updates the CleanRovNavLoad tracking table. The CleanRovnavData tables are then loaded into EMDB's CleanRovnavData layer with load.py --updated_navfiles -v.