Skip to content

Sync

Sync Push

This will sync data in a local directory to a remote S3 bucket. Data sync is throttled to a maximum of 62 MB/sec to avoid saturating the MBARI network.

Note

This recursively syncs only core raw or decimated MARS files, i.e. screenshots, high-frequency experiment data are skipped. Data and the underlying directory organization is mirrored in the bucket.

Usage

Run as user pam on cetacean.shore.mbari.org

python opendata/run.py sync_push [bucket name] [options]

Options can be seen with the --help option, e.g. opendata sync_pull --help

Usage: run.py sync_push [OPTIONS] BUCKET

  Syncs all data to a bucket

Options:
  -d, --wav_path TEXT  Absolute path to data
  -p, --profile TEXT    AWS profile to use. Default is 'soudscapeopendata'
  -e, --exclude TEXT   Exclude directory or file. Excludes any directory or
                       file that contains the given string
  -h, --help           Show this message and exit.

Example output

export HOME=/opt/pam/deployed
python opendata/run.py sync_push pacific-sound-16khz -d /tmp/decimated_16kHz/


Setting maximum bandwidth to 62 MB/sec
Fetching files to upload :  52% (557 of 1069) |#####################################                                   | Elapsed Time: 0:00:16 ETA:   0:00:13
...
Uploading : 100% (2 of 2) |############################################################################################| Elapsed Time: 0:00:28 Time:  0:00:28

Sync Pull

This will sync data from an AWS bucket to a directory. Data sync is throttled to a maximum of 62 MB/sec to avoid saturating the MBARI network.

Usage

python opendata/run.py sync_pull [bucket name] [options]

Options can be seen with the --help option, e.g. opendata sync_push --help

Usage: run.py sync_pull [OPTIONS] BUCKET

  Syncs all data to a bucket

Options:
  -d, --data_path TEXT  Absolute path to store the data  [required]
  -p, --profile TEXT    AWS profile to use. Default is 'soudscapeopendata'
  -h, --help            Show this message and exit.

Example output

export HOME=/opt/pam/deployed
opendata sync_pull pacific-sound-logs -d /PAM_Analysis/pacific-sound-logs