`generate`: Download Images & Extract Localizations from M3¶

The generate command queries M3's annotation database to download images and extract bounding box localizations in Pascal VOC format. This is the primary data acquisition tool for creating object detection datasets from MBARI's VARS.

v0.17.0

Breaking change

As of v0.17.0, generate is driven by a YAML dataset spec file instead of a large number of CLI flags. See Migrating from flags below if you have existing scripts using the old interface.

Breaking change: operational flags and split moved out of the spec

The dataset spec file (below) now only controls what dataset gets pulled. Run-operational concerns that don't affect the resulting dataset -- download parallelism, overwrite behavior, XML pretty-printing -- moved from download/output spec sections to CLI flags, and train/val/test splitting moved out of generate entirely into its own downstream command, m3-download split. A spec file still containing download:, output:, or split: sections will now fail validation immediately.

Breaking change: single DATASET_DIR instead of IMAGE_DIR/XML_DIR

generate now takes a single DATASET_DIR instead of separate IMAGE_DIR/XML_DIR arguments, writing images to DATASET_DIR/JPEGImages/ and Pascal VOC XML to DATASET_DIR/Annotations/ -- the standard Pascal VOC devkit layout, for maximum interoperability with other VOC tooling. m3-download split, fix-dimensions, and gallery all read/write this same directory shape.

Usage¶

m3-download generate SPEC_FILE DATASET_DIR [OPTIONS]

Required Parameters¶

SPEC_FILE: Path to a YAML dataset spec file describing what to pull (see below)
DATASET_DIR: Directory to write the dataset to -- images to DATASET_DIR/JPEGImages/, Pascal VOC XML to DATASET_DIR/Annotations/

Options¶

--config-url: Raziel config URL (default: https://m3.shore.mbari.org/config)
--manifest-path: Path to write the run manifest to (default: DATASET_DIR/manifest.json)
--download-images / --no-download-images: Actually download the images; if --no-download-images, only XML annotation files are generated (default: download)
--max-workers: Maximum number of concurrent image/frame downloads; 1 runs serially (default: 8)
--overwrite / --no-overwrite: Overwrite files that already exist in DATASET_DIR/JPEGImages (default: --no-overwrite)
--pretty-print / --no-pretty-print: Pretty-print the Pascal VOC XML output (default: --pretty-print)
--verbose, -v: Display additional debugging information

These are deliberately CLI flags, not spec fields: re-running the same spec file with a different --max-workers must produce the identical dataset, so nothing that only affects how the pull executes belongs in the file that defines what it pulls.

Authentication

You'll be prompted to enter your username and password for M3 access, unless credentials are already stored on disk (see m3-download auth). The tool does not store your credentials anywhere itself.

Dry Run

Use --no-download-images to generate only annotation files for planning purposes. This will help you assess the dataset size before downloading all the images.

The dataset spec file¶

Everything about what dataset to build lives in the spec file — concept/group/activity/tag filters, video filters, and sampling behavior. Only where to write output and how the tool talks to M3 stay as CLI options, since those are machine/run-specific rather than portable parts of the dataset definition.

Editor autocomplete/validation

A JSON Schema for this file is checked in at schemas/generate-spec.schema.json (repo root). Point your editor at it for inline validation and autocomplete -- e.g. with the VS Code YAML extension, add a comment at the top of your spec file:

# yaml-language-server: $schema=../path/to/schemas/generate-spec.schema.json

or configure it globally via the extension's yaml.schemas setting. The schema is regenerated from m3_download/lib/generate/spec.py with uv run python -m m3_download.lib.generate.schema (a test asserts it's kept in sync).

A minimal spec:

concepts:
  include:
    - concept: Sebastes
      descendants: true

Concept entries¶

Each entry in concepts.include/concepts.exclude is either a bare concept name (shorthand for "just this concept, not its descendants") or a {concept, descendants} mapping. The descendant-expansion toggle is per-entry, not per-list, so a spec can include a whole subtree, exclude a narrower subtree within it, and separately exclude one specific concept without touching its own descendants. For example, pulling in the whole rockfish/thornyhead order while dropping thornyheads entirely and one specific rockfish species:

concepts:
  include:
    - concept: Scorpaeniformes
      descendants: true            # every genus/species in the order, e.g. Sebastes, Sebastolobus, ...
  exclude:
    - concept: Sebastolobus
      descendants: true            # drops the whole thornyhead genus and its species
    - Sebastes paucispinis         # drops just bocaccio -- species are leaf taxa, so there's
                                    # no descendant subtree to worry about excluding here

concepts.include_file/concepts.exclude_file still take a plain newline-delimited concept list (one name per line, no per-line descendant control).

A fuller example showing every section (all fields are optional; defaults shown):

version: 1

concepts:
  include: []               # list of concept names, or {concept, descendants} mappings;
                             # empty = all concepts
  include_file: null        # path (relative to this spec file) to a newline-delimited concept list
  exclude: []
  exclude_file: null

video:
  names: []                 # video names to include
  sequence_names: []        # video sequence names to include
  after: null                # ISO 8601 minimum video start timestamp, e.g. "2021-01-01"
  before: null                # ISO 8601 maximum video start timestamp

annotations:
  include_groups: []         # if set, only these observation groups are included
  exclude_groups: []
  exclude_activities: []
  exclude_projects: []
  with_tags: []              # only bounding boxes with ALL of these tags are included
  observers: []               # if set, only annotations from these observers are included
  include_imaged_moment: true # include all annotations from an imaged moment with a matching
                               # concept, not just the matching-concept annotations themselves
  merge_image_video_moments: off # "off", "prefer_image", or "prefer_video" -- see below

sampling:
  max_per_concept: null      # cap the number of localizations kept per concept; null = no cap
  strategy: random           # "random" or "diversity" (spreads samples across dives/observers/
                             # depth bins rather than picking uniformly at random)
  seed: 0
  diversity:
    by: [video_sequence_name, observer, depth_bin]
    depth_bin_size_m: 50.0

dedup:
  enabled: false             # reserved for a future temporal/redundancy-dedup pass; no-op today

Unknown fields in the spec file are a validation error (not silently ignored) — a typo like includee: will fail immediately with a message telling you exactly which field is wrong.

How It Works¶

The command authenticates with the Raziel service to get endpoint information
It loads and validates the spec file
For concept entries with descendants: true, it uses the VARS KB Server to expand the query
It sends a query to the Annosaurus service to retrieve bounding box annotations
It associates each annotation with either:
- An image reference (a direct URL to an image)
- A video frame (from which an image can be extracted)
Images are downloaded and saved to DATASET_DIR/JPEGImages//DATASET_DIR/Annotations/ as a flat directory (not split into train/val/test -- see Splitting below)
Pascal VOC XML annotations are generated and saved

Network Requirements

This command requires internet access to the MBARI endpoints. The download process can be bandwidth and time intensive for large datasets.

Splitting¶

generate always writes a single flat DATASET_DIR, matching every other tool in this package (filter, dedup, gallery, fix-dimensions, etc.), all of which expect a flat directory of VOC XML/image pairs. Assigning train/val/test splits happens downstream, as its own step, with m3-download split -- run it after any manual QA/QC on generate's output, since splitting is a training-prep decision, not part of what dataset gets pulled. split reads manifest.json (below) to recover each image's dive (video_sequence_name) so it can keep whole dives together in one split without re-deriving that from filenames.

The manifest¶

Every run writes a manifest.json (to DATASET_DIR/manifest.json by default, or --manifest-path) recording:

resolved_spec: the spec actually used, with concept filters fully expanded (descendants resolved to concrete concept names) and video timestamps filled in as concrete ISO strings. This is a valid dataset spec on its own — useful for confirming exactly what a past run pulled, or as a starting point for reproducing it.
results: concrete counts (images, video frames, localizations, distinct annotations, a per-concept breakdown), the query's start/end time, whether extended contextual metadata (depth, position, platform, etc.) was available on the server for this run, and a per-image images list that m3-download split uses downstream. Each entry has the filename, dive (video_sequence_name, None if not tied to a video sequence), imaged_moment_uuid, source ("image" for a captured still, "video" for a frame annotated directly on the video), and the imaged moment's timestamp -- both resolved (timestamp) and the raw columns it was resolved from (recorded_timestamp -> elapsed_time_millis -> timecode, in fallback order).

Every output filename is {imaged_moment_uuid}_img.{ext} (stills) or {imaged_moment_uuid}_vid.jpg (video frames).

Image/video co-occurrence¶

An imaged moment can have bounding boxes of both kinds: one drawn on a captured still, one drawn directly on the video. The still is usually, but not always, the same frame as the video at that moment. annotations.merge_image_video_moments controls what happens when both occur on the same imaged moment:

off (default): keep them as two separate output images, tagged source: "image"/"video" in the manifest. Safe default when the two might genuinely differ.
prefer_image / prefer_video: merge onto the chosen source -- all of that imaged moment's bounding boxes end up on a single output image, and the other source is never fetched at all. Use this when you know the still and the video frame are equivalent for your purposes and want to avoid downloading two near-duplicate images per co-occurring moment.

results.co_occurring_imaged_moments_detected reports how many imaged moments were found with both kinds, independent of the merge setting; results.co_occurring_imaged_moments reports how many are still two separate images in this run's actual output (equal to _detected when merging is off, ~0 when a merge preference is set and applied cleanly).

Examples¶

Download a dataset for Sebastes rockfish including descendant species:

# sebastes.yaml
concepts:
  include:
    - concept: Sebastes
      descendants: true

m3-download generate sebastes.yaml Sebastes_dataset/

Download rockfish and thornyheads, but exclude the thornyhead genus entirely and one specific rockfish species:

concepts:
  include:
    - concept: Scorpaeniformes
      descendants: true
  exclude:
    - concept: Sebastolobus
      descendants: true      # excludes the whole genus and its species
    - Sebastes paucispinis   # excludes just this one species

Download data using a species list file (path resolved relative to the spec file):

concepts:
  include_file: species_list.txt

Download data excluding problematic groups and activities:

annotations:
  exclude_groups: ["ROV:pending-verifications"]
  exclude_activities: ["unspecified"]
  exclude_projects: ["ML-Tracking"]

Download data with specific tags (e.g., only training examples):

annotations:
  with_tags: [training]

Filter by video sequence name and start timestamp range:

video:
  sequence_names: ["Ventana 4361"]
  after: "2021-01-01"
  before: "2021-01-31"

Cap each concept at 500 localizations, sampled for diversity across dives and observers:

concepts:
  include:
    - concept: Sebastes
      descendants: true
sampling:
  max_per_concept: 500
  strategy: diversity

Generate annotations without downloading images (useful for planning):

m3-download generate sebastes.yaml dataset/ --no-download-images

Pull with 32 concurrent downloads instead of the default 8 (this is a run-operational choice -- it doesn't change the resulting dataset, so it's a flag, not a spec field):

m3-download generate sebastes.yaml dataset/ --max-workers 32

After any manual QA/QC on dataset/Annotations/, assign a leakage-safe train/val/test split with m3-download split.

Migrating from flags¶

Every pre-v0.17.0 CLI flag maps directly onto a spec file field:

Old flag	New spec field
`--include-concept`	`concepts.include` (or `concepts.include_file` for a file)
`--include-descendants`	`descendants: true` on the relevant `concepts.include` entry
`--exclude-concept`	`concepts.exclude` / `concepts.exclude_file`
`--exclude-descendants`	`descendants: true` on the relevant `concepts.exclude` entry
`--include-group`	`annotations.include_groups`
`--exclude-group`	`annotations.exclude_groups`
`--exclude-activity`	`annotations.exclude_activities`
`--exclude-project`	`annotations.exclude_projects`
`--with-tag`	`annotations.with_tags`
`--observer`	`annotations.observers`
`--include-imaged-moment`	`annotations.include_imaged_moment`
`--video-name`	`video.names`
`--video-sequence-name`	`video.sequence_names`
`--video-timestamp-min`	`video.after`
`--video-timestamp-max`	`video.before`
`--pretty-print`	unchanged — still a CLI flag (`--pretty-print` / `--no-pretty-print`)
`--download-images`	unchanged — still a CLI flag
`--config-url`	unchanged — still a CLI flag
(new)	`--max-workers`, `--overwrite`/`--no-overwrite` -- run-operational, CLI flags only

Train/val/test splitting, previously a split.* spec section, is no longer part of generate at all -- see m3-download split, a separate downstream command.

IMAGE_DIR/XML_DIR as two separate positional arguments are also gone -- generate now takes one DATASET_DIR, writing to its JPEGImages//Annotations/ subfolders.

For example, this old invocation:

m3-download generate sebastes_filtered/ sebastes_voc/ --include-concept Sebastes --include-descendants --exclude-concept "Sebastes mystinus" --exclude-concept "Sebastes serranoides"

becomes:

# sebastes.yaml
concepts:
  include:
    - concept: Sebastes
      descendants: true
  exclude: ["Sebastes mystinus", "Sebastes serranoides"]

m3-download generate sebastes.yaml sebastes_dataset/

generate: Download Images & Extract Localizations from M3¶