Usage

The m3_download package provides the m3-download command-line tool, which is the main entrypoint. To see the usage, including the available subcommands, use the following command:

m3-download --help

Available Commands

The m3-download tool provides the following subcommands:

Command	Description
auth	Authenticate with Raziel and display endpoint info
generate	Download images & extract localizations as Pascal VOC
split	Assign a generated dataset to leakage-safe train/val/test splits
filter	Exclude specified concepts from Pascal VOC annotations
taxonomy	Add taxonomic information to Pascal VOC annotations
count	Count localizations in Pascal VOC annotations
find-overlaps	Find localizations with IoU above a threshold
fix-dimensions	Correct image dimensions in Pascal VOC annotations
dedup	Deduplicate Pascal VOC annotations
gallery	Generate ROI sample galleries per class from a Pascal VOC dataset
convert voc-yolo	Convert Pascal VOC annotations to YOLO format
convert yolo-voc	Convert YOLO annotations to Pascal VOC format
convert yolo-json	Convert YOLO annotations to JSON format
remap voc	Remap concepts in Pascal VOC annotations
remap yolo	Remap class IDs in YOLO annotations

For more detailed information on each command, see the dedicated documentation page or use:

m3-download COMMAND --help

Workflow Examples

Complete Workflow: Download to YOLO Format

This is the canonical end-to-end pipeline: acquire, manually QA/QC, split, then convert to a training format. split runs last, since it needs a stable, QA'd set of images/annotations to partition, and every conversion/training tool downstream expects per-split directories.

# 1. Download data for Sebastes and its descendants (see generate.md for the spec file format)
m3-download generate sebastes.yaml dataset/

# 2. Manual QA/QC: deduplicate annotations, then filter out unwanted concepts
#    (these only touch the XML -- dataset/JPEGImages/ is unchanged throughout)
m3-download dedup dataset/Annotations/ --output-dir deduped_voc/
m3-download filter deduped_voc/ --exclude "unidentified rockfish" --output-dir filtered_voc/

# 3. Remap similar concepts, writing straight into a new dataset dir's Annotations/ folder,
#    and symlink the (unchanged) images in alongside it so `split` sees one dataset dir
mkdir qa_dataset
ln -s "$(realpath dataset/JPEGImages)" qa_dataset/JPEGImages
m3-download remap voc remapping.csv filtered_voc/ --output-dir qa_dataset/Annotations

# 4. Assign leakage-safe train/val/test splits, using the manifest generate wrote in step 1
m3-download split qa_dataset/ split_dataset/ --manifest-path dataset/manifest.json

# 5. Convert each split to YOLO format for training
m3-download convert voc-yolo split_dataset/train/Annotations --output-dir yolo_dataset/train
m3-download convert voc-yolo split_dataset/val/Annotations --output-dir yolo_dataset/val
m3-download convert voc-yolo split_dataset/test/Annotations --output-dir yolo_dataset/test

# 6. Optionally remap YOLO class IDs (repeat per split)
m3-download remap yolo yolo_mapping.csv yolo_dataset/train --output-dir final_yolo_dataset/train

Quality Control Workflow

Run any of these against generate's flat output before splitting (see above) -- they all expect a flat directory of VOC XML/image pairs, not per-split subdirectories.

# 1. Correct image dimensions in annotations
m3-download fix-dimensions dataset/

# 2. Deduplicate near-identical annotations
m3-download dedup dataset/Annotations/

# 3. Find overlapping annotations
m3-download find-overlaps --threshold 0.5 dataset/Annotations/

# 4. Visually spot-check samples per class
m3-download gallery dataset/ --output-dir gallery/

# 5. Count annotations per class
m3-download count dataset/Annotations/ > class_distribution.txt

Dataset Enhancement Workflow

# 1. Download base dataset (spec.yaml: concepts.include_file: species_list.txt)
m3-download generate spec.yaml dataset/

# 2. Add taxonomic information
m3-download taxonomy dataset/Annotations/ --output-dir voc_with_taxonomy/

# 3. Convert to multiple formats
m3-download convert voc-yolo voc_with_taxonomy/ --output-dir yolo_dataset/
m3-download convert yolo-json yolo_dataset/ yolo_dataset/yolo.names 1920 1080 annotations.json

Custom Filtering Pipeline

# 1. Download all data (empty spec file = no filters)
m3-download generate all.yaml dataset/

# 2. Filter by concept
m3-download filter dataset/Annotations/ --exclude "artifact" --output-dir no_artifacts/

# 3. Filter by overlapping annotations (create list first)
m3-download find-overlaps no_artifacts/ 0.7 > overlapping.txt

# 4. Process the overlapping files manually or with other tools

YOLO Dataset Manipulation Workflow

# 1. Convert VOC annotations to YOLO format
m3-download convert voc-yolo voc_annotations/ --output-dir yolo_dataset/

# 2. Remap YOLO class IDs (merge similar classes)
m3-download remap yolo class_merge.csv yolo_dataset/ --output-dir merged_classes/

# 3. Convert back to VOC format (if needed)
m3-download convert yolo-voc merged_classes/ images/ voc_final/