Workflows¶

Argo Workflows in this repository fall into two groups: Tator-integrated (download media, run inference, upload localizations) and direct Argo (read host-mounted data, write files). Manifests live in workflows/. Apply RBAC once before submitting jobs:

kubectl apply -f workflows/rbac.yml

Select a workflow below for description, steps, parameters, and how to submit.

ONNX 3-stageTrack loadUltralyticsExpedition metadata loadSDCATVideo reportSingle-class reportGPU CUDA testRetinaNetCluster RBAC (Role Based Access Controls)

ONNX detect, classify, track, and load to Tator¶

Generic Tator pipeline for video or stills. The same YAML is reused across projects (i2MAP, UAV, Triton) by changing models, thresholds, and export paths at submit time. This can run two stage: detect+classify, or three stage: detect+classify+track processing pipelines on images or video. Uses ONNX models in /mnt/DeepSea-AI/models


Manifest	`workflows/tator-3stage.yml`
Image	`registry:5000/onnx:latest` (`docker/onnx`)
Compute	GPU for detection; CPU for Tator load and export
Entry points	`track.tator_onnx_3stage`, then `utils/load_tracks_tator.py`

Steps

generate-detections — connect to Tator, download (or resolve) media, run ONNX detection, optional classifier, optional BoostTrack; write *-tracks.tar.gz with metadata.json provenance
load-detections — upload box localizations (and tracks) to a Tator Version
export-results — copy /work/output to export_dir on the host share

Detection is always required. Classification is skipped when classify_model is empty. Tracking is skipped when disable_tracking=true (pseudo-tracks are still exported so the loader can create boxes). SAHI tiled inference is used when slice_height / slice_width are non-zero.

Tator workflows that upload localizations require an explicit version_id. Create the Version in Tator first; there is no Baseline fallback.

Key parameters

Parameter	Role
`project_id`, `media_ids`, `host`, `rest_token`, `version_id`	Tator connection (required)
`detection_model`	ONNX detector path (cluster-visible, e.g. under `/mnt/DeepSea-AI`)
`classify_model`	Optional ONNX classifier; labels from `config.json` / `labels.json` beside the model
`disable_tracking`	`true` = detect (+ classify) only
`slice_height`, `slice_width`	SAHI tile size; `0` = full-frame
`detector_score_threshold`	Min ONNX detector confidence (boxes below this are dropped)
`tracker_score_threshold`	Min detector score BoostTrack will associate; ignored if `disable_tracking=true`
`nms_iou_threshold`	Detection NMS
`letterbox`	`true` = YOLO26 LetterBox (aspect pad 114); default stretch-resize. End-to-end YOLO26 `(1,N,6)` is parsed as xyxy.
`stride`, `start_seconds`, `max_seconds`	Video sampling window
`export_dir`	Host path for `*-tracks.tar.gz`

Submit

just submit-i2map-detect-track PROJECT_ID MEDIA_IDS HOST VERSION_ID TOKEN
just submit-i2map-detect-classify-track PROJECT_ID MEDIA_IDS HOST VERSION_ID TOKEN
just submit-i2map-detect-classify PROJECT_ID MEDIA_IDS HOST VERSION_ID TOKEN   # disable_tracking
just submit-uav-detect-classify PROJECT_ID MEDIA_IDS VERSION_ID TOKEN          # SAHI tiles, drone.mbari.org
just submit-triton-detect-classify-track PROJECT_ID MEDIA_IDS HOST VERSION_ID TOKEN

Filesystem-only (no Tator download): just onnx-3stage MEDIA_ROOT=/path/to/media runs src/track/onnx_3stage.py.

Load existing track archives into Tator¶

CPU-only follow-on for archives already produced by the 3-stage workflow. Reads a text file of *-tracks.tar.gz paths (one per line) that must be visible inside the pod (typically under /mnt/DeepSea-AI).


Manifest	`workflows/tator-track-load-only.yml`
Image	`registry:5000/onnx:latest`
Compute	CPU
Entry point	`utils/load_tracks_tator.py`

Uses the same loader as 3-stage: time-decay label averaging, optional --best-label-box-only, IoU merge (iou_threshold, frame_gap), and edge_ignore_percent to drop boxes near frame edges. Requires version_id.

Key parameters

Parameter	Role
`project_id`, `host`, `rest_token`, `version_id`	Tator connection (required)
`archive_list_file`	Path to a `.txt` list of archives (default `/mnt/DeepSea-AI/data/track_archives/archives.txt`)
`iou_threshold`, `frame_gap`	Track merge
`best_label_box_only`	Keep the highest-scoring label per box
`edge_ignore_percent`	Ignore detections near frame edges

Submit

just submit-tator-3stage-load PROJECT_ID HOST VERSION_ID TOKEN
# optional: ARCHIVE_LIST_FILE=/mnt/DeepSea-AI/data/track_archives/archives.txt

YOLOv11 object detection¶

Direct Argo inference on images in /data/input. Writes boxes and a results summary under the workflow output directory. Does not talk to Tator.


Manifest	`workflows/ultralytics-inference.yml`
Image	`registry:5000/ultralytics:latest` (`docker/ultralytics`)
Compute	GPU
Entry point	`inference/ultralytics_inference.py`

Steps: setup workspace → run inference → post-process summary JSON → notify.

Key parameters

Parameter	Default	Role
`model-name`	`yolo11x.pt`	Ultralytics weights
`confidence-threshold`	`0.35`	Detection confidence
`iou-threshold`	`0.45`	NMS IoU
`output_dir`	`/data/output`	Results directory

Submit

just submit-ultralytics

Attach expedition CTD metadata to detections¶

Unpacks a *-detections.tar.gz, joins latitude, depth, oxygen, salinity, and related fields via Java get-expd-data.jar, and writes *-detections-expd.tar.gz (CSV + metadata.json). Useful for workflows that need to add metadata to the detections.


Manifest	`workflows/expd-metadata.yml`
Image	`registry:5000/expd:latest` (`docker/get-expd`)
Compute	CPU
Entry point	`utils/add_expd_metadata.py`

Steps: validate that the detections archive exists → run the JAR-backed Python driver.

Key parameters

Parameter	Role
`detections`	Path to a `*-detections.tar.gz`
`output_dir`	Where the expd archive is written (default `/data/output`)

Submit

just submit-expd-metadata
just submit-expd-metadata DETECTIONS="/data/rfdetr/your-archive-detections.tar.gz"

Sliced detection (SDCAT)¶

Runs SDCAT detect on a directory of images. Optional post-process step summarizes detection counts. Image mbari/sdcat:latest.


Manifest	`workflows/sdcat-detection.yml`
Image	`mbari/sdcat:latest`
Compute	CPU (raise resources in the YAML if needed)

Steps: setup workspace → sdcat detect → post-process → notify.

Key parameters

Parameter	Default	Role
`model-name`	`MBARI-org/uav-yolov5`	Detector
`image-dir`	`/data/images`	Input images
`output_dir`	`/data/detections`	SDCAT save directory

Submit

just submit-sdcat

Aggregate detection archives to CSV¶

Walks a directory for *-detections.tar.gz (or another pattern), parses each archive, and writes a summary CSV (per-class counts and mean confidence). CPU-only; no GPU.


Manifest	`workflows/video-report.yml`
Image	`python:3.10-slim`
Compute	CPU

Steps: setup (create export dir, verify search path) → generate report → display results.

Key parameters

Parameter	Default	Role
`output_dir`	`/data/output`	Directory to search
`pattern`	`*-detections.tar.gz`	Archive glob
`min-confidence`	`0.0`	Score floor
`recursive`	`true`	Recurse into subdirs
`output`	`/data/export/detection_report.csv`	CSV path

Submit

just submit-report
just submit-report-docricketts-2018

Detection report filtered to one class¶

Same pipeline as the video report, with an extra class filter (for example Ctenophora sp. A). Only matching labels are counted in the CSV.


Manifest	`workflows/report-single-class.yml`
Image	`python:3.10-slim`
Compute	CPU

Key parameters — same as the video report, plus class (default Ctenophora sp. A). Output default: /data/export/detection_report_single_class.csv.

Submit

just submit-report-single-class CLASS=Ctenophora sp. A

Five-minute PyTorch CUDA smoke test¶

Operational check that the node can see CUDA and sustain GPU work. Allocates random 1024×1024 matrices and multiplies them for five minutes, printing device name and periodic memory stats. Not a science workflow.


Manifest	`workflows/gpu-5min-test.yml`
Image	`pytorch/pytorch:2.1.0-cuda12.1-cudnn8-runtime`
Compute	1× GPU

Submit

argo submit -n default workflows/gpu-5min-test.yml --watch

Detectron2 RetinaNet on Tator images¶

Tator algorithm for images (not video). The container downloads selected media, runs Detectron2 RetinaNet, and uploads bounding boxes back to Tator. Not a science workflow. Useful for exercising the integration with Tator


Manifest	`workflows/tator-retinanet.yml`
Image	`registry:5000/retinanet:latest` (`docker/retinanet`)
Compute	CPU by default; uncomment `nvidia.com/gpu` in the YAML for GPU
Entry point	`/infer.py` (`src/inference/detectron_infer_tator.py`)

Register this YAML as a Tator algorithm so the platform injects project_id, media_ids, host, and rest_token. See Adding a Tator algorithm.

Key parameters

Parameter	Role
`project_id`, `media_ids`, `host`, `rest_token`	Provided by Tator
`output_dir`	Working directory (default `/work`)

Submit (without the Tator UI)

just submit-retinanet-tator PROJECT_ID MEDIA_IDS HOST TOKEN

workflows/rbac.yml is a Kubernetes Role and RoleBinding, not an Argo Workflow. It grants the default service account in default permission to manage Argo workflow resources, pods/logs, PVCs, and related reads (needed by executors and Tator-style submissions). This generally is run during setup and does not need to be run otherwise.

kubectl apply -f workflows/rbac.yml

Monitoring¶

just argo-ui          # http://localhost:2746
just argo-list
just argo-logs        # latest workflow
just argo-get name

See Viewing workflow outputs.

🗓️ Updated: 2026-08-25