Examples on interacting with Jobs through GraphQL¶

Overview¶

A job refers to a collection of processed data. This could be processed either in AWS SageMaker Script Processor, AWS Elastic Container Service, or on a local GPU. It represents a collection of data that was processed the same way, e.g. by the same model with the same parameters.

In [1]:

import requests
# This is the endpoint for the GraphQL API. On a production installation it might be http://<yourservername>/api .
API_URL = "http://localhost:4000/graphql"
# Put your auth token from login here
AUTH_TOKEN = "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJ1c2VySWQiOjEsImlhdCI6MTY1OTY1NDMxMywiZXhwIjoxNjYwMjU5MTEzfQ.rbx1CldVG60n8WenpnsAraInGvO0zMgA4dXPIvSi0kk"

import requests # This is the endpoint for the GraphQL API. On a production installation it might be http:///api . API_URL = "http://localhost:4000/graphql" # Put your auth token from login here AUTH_TOKEN = "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJ1c2VySWQiOjEsImlhdCI6MTY1OTY1NDMxMywiZXhwIjoxNjYwMjU5MTEzfQ.rbx1CldVG60n8WenpnsAraInGvO0zMgA4dXPIvSi0kk"

Getting all jobs¶

All available jobs and their associated media can be fetched with the query below.

In [2]:

def get_jobs():
    query = """ {
          jobs {
            id
            medias {
                id
                name
                uuid
            }
          }
        }
    """

    reply = (requests.post(API_URL, json={'query': query}).json())
    return(reply['data']['jobs'])

def get_jobs(): query = """ { jobs { id medias { id name uuid } } } """ reply = (requests.post(API_URL, json={'query': query}).json()) return(reply['data']['jobs'])

In [3]:

jobs = get_jobs()
num_medias = len(jobs[0]['medias'])
job_id = jobs[0]['id']

print(f"Job id {job_id}")
print(f"Job has {num_medias} media {jobs[0]['medias']}")

jobs = get_jobs() num_medias = len(jobs[0]['medias']) job_id = jobs[0]['id'] print(f"Job id {job_id}") print(f"Job has {num_medias} media {jobs[0]['medias']}")

Job id 1
Job has 1 media [{'id': 1, 'name': 'V4244_20191205T165023Z_h264.mp4', 'uuid': '95344a11-eefc-a896-4ce9-074a12d0362a'}]

Delete a job¶

To remove a job use the job id and the deleteJobById mutation. This removes all tracks, events, and medias associated with that job.

Only the user who ran the job can remove the job, but any user can modify the tracks or events in a given job.

In [8]:

def delete_job(job_id: int):
    query = """ mutation
        deleteJob($job_id: Int!) {
            deleteJobById(id: $job_id) {
              success
              message
            }
        }
    """

    variables = {"job_id": job_id}
    reply = (requests.post(API_URL, headers={'Authorization': f'Bearer {AUTH_TOKEN}'},
                             json={'query': query, 'variables': variables}).json())
    print(reply)

def delete_job(job_id: int): query = """ mutation deleteJob($job_id: Int!) { deleteJobById(id: $job_id) { success message } } """ variables = {"job_id": job_id} reply = (requests.post(API_URL, headers={'Authorization': f'Bearer {AUTH_TOKEN}'}, json={'query': query, 'variables': variables}).json()) print(reply)

In [9]:

delete_job(job_id)

delete_job(job_id)

{'data': {'deleteJobById': {'success': True, 'message': 'Deleted job 21'}}}

Job detail¶

Each job has a detail field that has information about what model, arguments, etc. were used in processing. This detail is encoded in a JSON formatted string and varies depending on how the job was run. You can grab that detail with the following query:

In [4]:

def get_job_detail(job_id: int):
    
    query = """ query
        getJobDetail($job_id: Int!) {
          jobs(where: { id: { equals: $job_id  } } ) {
            detail
        }
    }
    """
    
    variables = {"job_id": job_id}
    reply = (requests.post(API_URL, json={'query': query, 'variables': variables}).json())
    return(reply['data']['jobs'])

def get_job_detail(job_id: int): query = """ query getJobDetail($job_id: Int!) { jobs(where: { id: { equals: $job_id } } ) { detail } } """ variables = {"job_id": job_id} reply = (requests.post(API_URL, json={'query': query, 'variables': variables}).json()) return(reply['data']['jobs'])

In [5]:

jobs_detail = get_job_detail(job_id)
print(jobs_detail)

jobs_detail = get_job_detail(job_id) print(jobs_detail)

[{'detail': '{"UserName":"duane","ProcessingJobName":"deepsort-yolov5-tracker-“Testing-ECS-Duane”","AppSpecification":{"ImageUri":"mbari/deepsort-yolov5-tracker:1.4.4","ContainerArguments":["/app/pipeline/run.py","dettrack","--conf-thres","0.01","--model-size","640","-c","s3://902005-video-in/config/lonny_deep_sort_benthic.yaml","--model-location","s3://902005-lonny-33k-model/lonny_model.tar.gz"]}}'}]

Parse the job detail¶

You can parse the json to get a better view

In [6]:

import json
 
detail_json = json.loads(jobs_detail[0]['detail'])

import json detail_json = json.loads(jobs_detail[0]['detail'])

In [7]:

detail_json

detail_json

Out[7]:

{'UserName': 'duane',
 'ProcessingJobName': 'deepsort-yolov5-tracker-“Testing-ECS-Duane”',
 'AppSpecification': {'ImageUri': 'mbari/deepsort-yolov5-tracker:1.4.4',
  'ContainerArguments': ['/app/pipeline/run.py',
   'dettrack',
   '--conf-thres',
   '0.01',
   '--model-size',
   '640',
   '-c',
   's3://902005-video-in/config/lonny_deep_sort_benthic.yaml',
   '--model-location',
   's3://902005-lonny-33k-model/lonny_model.tar.gz']}}

Find if a media was processed in a given job by its ProcessingJobName¶

The job ProcessingJobName can be used to search for a given media as well using the mediaInJob query. This can be useful to find if media was already processed in a given job.

In [22]:

def get_media_in_job(processing_job_name: str, media_name: str):        
            
    query = """ query 
        getMediaInJob($processing_job_name: String!, $media_name: String!) {
            mediaInJob(processing_job_name: $processing_job_name, media_name: $media_name) {
                name
                frame_rate
                uuid
            }
        }
    """
    
    variables = {"processing_job_name": processing_job_name, "media_name": media_name}
    reply = (requests.post(API_URL, json={'query': query, 'variables': variables}).json())
    return(reply)

def get_media_in_job(processing_job_name: str, media_name: str): query = """ query getMediaInJob($processing_job_name: String!, $media_name: String!) { mediaInJob(processing_job_name: $processing_job_name, media_name: $media_name) { name frame_rate uuid } } """ variables = {"processing_job_name": processing_job_name, "media_name": media_name} reply = (requests.post(API_URL, json={'query': query, 'variables': variables}).json()) return(reply)

In [24]:

media = get_media_in_job("deepsort-yolov5-tracker-“Testing-ECS-Duane”", "V4244_20191205T165023Z_h264.mp4")
print(media['data'])

media = get_media_in_job("deepsort-yolov5-tracker-“Testing-ECS-Duane”", "V4244_20191205T165023Z_h264.mp4") print(media['data'])

{'mediaInJob': [{'name': 'V4244_20191205T165023Z_h264.mp4', 'frame_rate': 29.97, 'uuid': '95344a11-eefc-a896-4ce9-074a12d0362a'}]}

Copyright (c) 2022, MBARI