Tape To Video Ingest Instructions¶

Overview:

Neil will stage files to smb://titan.shore.mbari.org/M3/IMES and the team will receive an email notification when the files are ready.
Brian will generate initial mv script and video metadata csv file.
The Team will review the mv scripts and address problems.
Kris will review the final mv scripts.
Brian will run the final mv scripts.
Brian will archive the ancillary metadata.

Setup¶

Install a Java Development Kit (JDK) version 11 or later.
Copy m3-merge.zip and tapemigration.zip to your computer and unzip them.
Mount smb://titan.shore.mbari.org/m3 on your computer

Workflow¶

If you've mounted titan:/m3 on a Mac, the contents of an LTO tape are located in a directory like /Volumes/m3/IMES/<LTO ID> (e.g. /Volumes/m3/IMES/S99506L8). The steps are as follows:

Step 1 - Generate the `mv` script and video metadata¶

cd tapemigration/bin
./titan-mv-script <path/to/videodir> <path/to/m3> -o <script_name>

For example, to process the directory /Volumes/m3/IMES/S99506L8

./titan-mv-script /Volumes/m3/IMES/S99506L8/ /Volumes/m3 -o S99506L8.sh

This will create a script in the same directory as the titan-mv-script named S99506L9.sh

Brian will generate the video metadata and share it with the group.

Step 2 - Note problem videos¶

If a video cannot be processed correctly, the script generated in step 1 will have a line that describes the problem with the video and possible a guess at what the correct move command should be. Note, to make analysis easier, sort the lines in the mv script generated in step 1.

If a video had an issue, there will be 1 or more lines starting with the video path followed by a description of the problem. Examples of the most common problems are:

The video was recorded before the start of the dive¶

# /Volumes/m3/IMES/S99505L8/V1207_SD01_Seq02_00-01-24-11_T470245924_prores.mov - 00:01:24:11 < 00:01:55:06 [1997-01-27T17:55:56Z] - File timecode before min EXPD timecode for Ventana 1207
# /Volumes/m3/IMES/S99505L8/V1207_SD01_Seq02_00-01-24-11_T470245924_prores.mov guess: mv -f /Volumes/m3/IMES/S99505L8/V1207_SD01_Seq02_00-01-24-11_T470245924_prores.mov /Volumes/m3/mezzanine/Ventana/1997/01/1207/V1207_19970127T175519.614Z_t1s2_sd_tc00012411_estimatedtime_prores.mov

The video was recorded after the end of the dive¶

# /Volumes/m3/IMES/S99505L8/D03_HD04_Seq01_02-25-01-00_T1054339137_h264.mp4 - 02:25:01:00 > 02:24:52:12 [2009-02-11T21:48:38Z] - File timecode after max EXPD timecode for Doc Ricketts 0003
# /Volumes/m3/IMES/S99505L8/D03_HD04_Seq01_02-25-01-00_T1054339137_h264.mp4 guess: mv -f /Volumes/m3/IMES/S99505L8/D03_HD04_Seq01_02-25-01-00_T1054339137_h264.mp4 /Volumes/m3/mezzanine/DocRicketts/2009/02/3/D3_20090211T214846.599Z_t4s1_hd_tc02250100_estimatedtime_h264.mp4

There are no camera logs for the dive¶

# /Volumes/m3/IMES/S99505L8/V1298_SD01_Seq01_00-00-10-00_T470557008_h264.mp4 - No camera logs were found for Ventana 1298

Step 3 - Investigate problem videos¶

If there are no camera logs, the mv script generator will provide an estimate based on the videos timecode and the start time of the dive as reported in EXPD. If there are no camera logs please check that there were not two dives in one day; the problem files might be associated with the wrong dive.

Other than missing camera logs, the most common reasons a video can not be correctly renamed are:

The timecode was reset during the dive.
The dive start and/or end time is entered incorrectly in the expedition (EXPD) database.
There were two or more dives in one day, but all the tapes for that day are labeled as belonging to only one dive. (e.g Dives V1222 and V1223 were both done on the same day, but the tapes for both dives are labeled as V1222 only)

To determine which of the above cases is occurring, do the following:

# Using V1222 and V1223 as an example
cd m3-merge/bin
dive_timecode_report "Ventana 1222" -e 60
dive_timecode_report "Ventana 1223" -e 60

This will produce a report for each dive showing the start/end times for a dive. If the reports show the same date and the same chief scientist, then there were two dives in one day. Note that in the move script by adding a line like:

# TODO - V1222 tapes 6-9 are dive V1223

inspect-tapes¶

There is a reporting tool to help look at what tapes already exist in VARS and if they have been annotated. For example running the following. This tools is also useful for determining if there are two dives in one day.

inspect-tapes "Ventana 1031"

produces this report:

-- Tapes before, during, and after Ventana 1031
   Tape       | StartTime                | URN                         | AnnotationCount
   V1030-01   | 1996-03-19T20:09:41Z     | urn:tid:mbari.org:V1030-01  |   6
   V1030-02   | 1996-03-19T20:05:59.117Z | urn:tid:mbari.org:V1030-02  |   10
   V1030-03   | 1996-03-19T20:42:51.280Z | urn:tid:mbari.org:V1030-03  |   1
   V1030-04   | 1996-03-19T21:05:06.047Z | urn:tid:mbari.org:V1030-04  |   7
   V1030-05   | 1996-03-19T21:30:43.827Z | urn:tid:mbari.org:V1030-05  |   4
   V1030-06   | 1996-03-19T22:08:58.867Z | urn:tid:mbari.org:V1030-06  |   1
   V1030-07   | 1996-03-19T22:27:13.677Z | urn:tid:mbari.org:V1030-07  |   2
-> V1031-01   | 1996-03-25T16:22:54Z     | urn:tid:mbari.org:V1031-01  |   5
-> V1031-02   | 1996-03-25T16:53:35Z     | urn:tid:mbari.org:V1031-02  |   48
-> V1031-03   | 1996-03-25T17:26:46Z     | urn:tid:mbari.org:V1031-03  |   15
-> V1031-04   | 1996-03-25T18:04:32Z     | urn:tid:mbari.org:V1031-04  |   5
   V1032-04   | 1970-01-01T00:00:00Z     | urn:tid:mbari.org:V1032-04  |   0
   V1032-05   | 1996-03-25T20:29:46Z     | urn:tid:mbari.org:V1032-05  |   35
   V1032-06   | 1996-03-25T20:42:56Z     | urn:tid:mbari.org:V1032-06  |   49

tc-estimator¶

Given a IMES file name it will look up the cam logs from EXPD and provide guesses on what the video should be named in M3. If there are timecode resets, it will provide a guess for each unbrokedn segment of the cam log.

tc-estimator D903_HD05_Seq03_06-50-39-06_T1056705998_prores.mov

Start: 2016-11-11T14:30:38Z   Timecode: 02:11:43:25
End:   2016-11-12T00:38:59Z   Timecode: 12:12:21:14
-- Extrapolated: D903_20161111T191159.366Z_t5s3_hd_tc06503906_prores.mov
-- Guess:        D903_20161111T191159.366Z_t5s3_hd_tc06503906_prores.mov

Step 4 - Make Changes¶

Change start and end times¶

Video lab can update start/end times of dives and rename dive number in file names. Also note which guess is correct. Then archive the script in smb://atlas.shore.mbari.org/Volumes/VideoLab/DigitizingProject/Digitized_video_docs/LTO_scripts. Be sure to append your name so Brian knows it's a temporary script. e.g. S99506L8_kris.sh

Rename videos and ancillary data files¶

Lonny, Kris, Diana, and Brian can rename video files to match the correct dive. Be sure to change the names of the corresponding QC files too.

Step 5 - Rerun `titan-mv-script` to generate cleaned script¶

Merge your comments from the original mv script into the new one.

Step 6 - Hand off scripts to Kris to review¶

Step 7 - Kris will hand off scripts to Brian¶

Brian will:

Archive the ancillary data files under
Update the mv script to be run on cuttlefish.shore.mbari.org
Copy the mv script to cuttlefish and run it.
Archive the mv script on