DeepSeaAI Changelog
1.25.0 (2024-02-29)Features
added support for specifying the volume size (BlockDeviceVolumeGB) and timeout (TimeoutHours) in the ECS config.yml file to support larger video/model processing (be93068 )
Bug Fixes
update link to preparing data doc (3a1a358 )
remove scale down on tasks, bump up warmup time to 5 minutes to allow for longer model spin-up; bump to 180 minutes for complex videos (a32c60f )
1.24.9 (2023-11-08)Bug Fixes
change scale in to the longest expected processing time of 2 hours for scale-in protection in cases with just one instance, one video (886e17b )
1.24.8 (2023-11-08)Bug Fixes
pass through the job name to the process sagemaker job for consistency with the ecsprocess command (9ffd71b )
1.24.7 (2023-11-08)
CDK slow the spin-up to 5 minute average and set daemon false explicitly on service (2455e9d )
1.24.6 (2023-10-31)Bug Fixes
update media timestamp only with newer message (5c02b49 )
1.24.5 (2023-10-31)Bug Fixes
update status for repeat jobs and force database commit (24126b4 )
Reverts
to single GPU task per instance (de53f9c )
1.24.4 (2023-10-26)
bump to latest cdk 2.103.0 and adjust license (7255757 )
1.24.3 (2023-10-26)Bug Fixes
correct pass through of database path and some refactoring of the config.ini (e9c688c )
1.24.2 (2023-10-26)Bug Fixes
handle second arg to save cdk output and run ecs container as docker_user to be consistent with mbari/deepsea-ai container (10451ec )
1.24.1 (2023-10-25)Bug Fixes
1.24.0 (2023-10-25)Features
added option to pass in reporting directory (be5b290 )
bump ecs storage from 10 to 20 GB and run 2 tasks per instance for improved throughput. relax the 1 GPU requirement to support running two videos per GPU as there is no fractional support for GPUs matching in AWS. this effectively doubles the number of videos per instance (9dce062 )
1.23.1 (2023-10-20)Bug Fixes
handle missing log directory (67acae0 )
1.23.0 (2023-10-19)Features
added support for LOG_PATH env (1251ae7 )
1.22.3 (2023-10-03)Bug Fixes
improved error handling on NFS m3 disconnect (816745a )
1.22.2 (2023-09-22)Bug Fixes
added missing SUCCESS status (4908fbe )
1.22.1 (2023-09-20)Bug Fixes
better sql deletion behavior with PRAGMA FOREIGN_KEYS=ON; (f58b601 )
1.22.0 (2023-09-19)Bug Fixes
correct pass through of job db (cb633d1 )
handle empty SQS message (b9568e6 )
handle no credential errors in config (25df156 )
handle resuming from checkpoint out of space rrors and dryrun ecsprocess (89104d4 )
handle single videos correctly (f1b2618 )
id is autoassigned in db commit (89fb48c )
no quoted args needed for sqs messages (d854787 )
remove args completely (11bca59 )
Features
add process command test (f2d6af3 )
added in cdk stack (needs to be moved to python) (9b08879 )
added more granular config sections for docker, aws, aws_public, and added default buckets for video/track/track-config with setup command, and other enhancements to support dry-run and testing (9ae7aff )
added new logging group called dsai/ecs and explicit iam for log actions (2833efe )
remove default arguments in CDK to simplify (5172a8f )
removed deepsort, added dry-run option for ecsprocess/process, and pass through --args for processing (9e39b50 )
test refactoring and sqlite logging for local/remote (51794a4 )
try to continue if no permission on bucket (11e5d9d )
replace pickledb with sqlalchemy and monitor test (31f8179 )
scale out faster every 60 sec (2405c46 )
Reverts
1.21.3 (2023-05-31)Bug Fixes
correct imports on monitor test (c9b4d95 )
1.21.2 (2023-05-31)Bug Fixes
1.21.1 (2023-04-11)Bug Fixes
added missing img2label_paths for split (a1147a3 )
1.21.0 (2023-03-25)Features
added dry-run and fix to video check (removed quotes) (9bf5509 )
added simple job caching and job report to monitor command (d048211 )
initial commit of log; more work to add reporting. Includes more detail on monitoring command and four logging levels (617c8a2 )
add tagging check (2f5a4ea )
autodump and improved monitor state machine logic (8bab569 )
1.20.0 (2023-03-18)Features
add tagging check, retry 10 times every minute for upload failures, and add support for single file processing (b49eeda )
1.19.2 (2023-03-14)Bug Fixes
minor fix to capture resource name which is now capitalized and graphql ini endpoint missing s (176b9b6 )
1.19.1 (2023-02-08)Bug Fixes
updating doc with process commands (467e21a )
updating doc with process commands (e5341e5 )
updating README.md with ecsprocess example and with process example (06ce655 )
1.19.0 (2023-02-03)Bug Fixes
allow buckets already created to be reused (91479ba )
Features
1.18.0 (2022-12-09)Features
handle empty prefix paths defined as inputs in processing jobs (64a9e4e )
Reverts
add back in default input/outputs (ff596cc )
1.17.1 (2022-12-09)Bug Fixes
add correct path to processingjobconfig.json (0379447 )
1.17.0 (2022-12-09)Features
add VideoName to job output which is less fragile than using the tar prefix (f71e64c )
1.16.0 (2022-12-04)Features
add suppport for Users/ and home/ prefixes (e605e4c )
1.15.0 (2022-11-20)Features
added support for passing in custom trained ReID weights for testing and add yolo.txt file back into tar.gz (a2728f8 )
1.14.1 (2022-11-18)
get rid of empty / prefixes in s3 calls and add iou pass through to process (0c381a0 )
1.14.0 (2022-11-11)Features
bump strong sort docker images (c98e428 )
1.13.1 (2022-11-11)Bug Fixes
strip rightmost / which errors process command out the with "bucket does not exist" error (53fa17e )
1.13.0 (2022-11-11)Features
add instance-type option to process for Duane (e60b2de )
1.12.0 (2022-11-11)Features
pass in iou and conf for batch (d8b4258 )
1.11.0 (2022-11-10)Features
1.10.0 (2022-11-09)Features
added mirror as optional command (01c7fdf )
1.9.0 (2022-11-09)Features
better formatting of scaling output and force cluster name as default (adace33 )
1.8.3 (2022-11-09)Bug Fixes
improved parsing to fetch the username in both subaccounts and a root leve account by the Arn (a1b4b6d )
1.8.2 (2022-11-08)
require cluster and job description for batch process (22591b3 )
1.8.1 (2022-11-08)Bug Fixes
add GetAndPassRolePolicy to allow getting and passing of roles and scope boto3 to the profile set by AWS_PROFILE (425b3ce )
correct fetching of username (346cfdd )
exit on error if any missing data during upload (ff966d6 )
1.8.0 (2022-11-02)Bug Fixes
correct path for default .ini store (23c1b0f )
Features
make track queue strong and deep sort docker images the default (ce47580 )
1.7.1 (2022-10-31)Bug Fixes
added -j option back in and handle small < 1 GB videos (fb192fa )
1.7.0 (2022-10-28)Features
beginnings of monitoring of ECS; added ASG pring (02c6328 )
1.6.3 (2022-10-27)Bug Fixes
correct args for video upload in batch (349e79f )
1.6.1 (2022-10-26)Bug Fixes
correct logic for bucket check (451a153 )
bump default role session duration (fae4a4a )
1.6.0 (2022-10-24)Features
exit training on bucket failure (28dd6a0 )
1.5.1 (2022-10-24)Bug Fixes
correct policy arn, print SageMaker role for clarity and quiet down the config parse print (542af3a )
1.5.0 (2022-10-17)Bug Fixes
correct uri for training/processing (2b13fa6 )
downgrade unlink for python3.7 support (fbba59e )
non static method and return missing arn (8dd846d )
Features
1.4.0 (2022-09-27)Features
switched to the latest public ecr images as default (ac332e1 )
1.3.3 (2022-09-26)Bug Fixes
remove , from tags - these throw InvalidTag exceptions in boto3 and fix video and job in ecs args (20a859f )
1.3.2 (2022-09-26)Bug Fixes
1.3.1 (2022-09-26)Bug Fixes
remove any leading / for video upload (aa419f7 )
1.3.0 (2022-09-26)Bug Fixes
correct instance type var which broke after refactor (9631304 )
put back in check_videos which was accidentally removed (36829dc )
Features
added config.ini support to support modifying tags, docker uris, and allocate process disk based on video size and command (701623e )
1.2.0 (2022-09-19)Features
bumped training docker image to deepsea-yolov5:1.0.1 (f0c499c )
1.1.1 (2022-09-18)Bug Fixes
correction in training volume size estimate and some minor documentation fixes (0593eef )
1.1.0 (2022-09-13)Features
added test of expected path for split command (19c5291 )
1.0.1 (2022-09-10)
bumped the boto3 and added awscli to default install to simplify use (40e946a )
DeepSea AI PyPi Changelog
1.0.0 (2022-09-02)