From: Edgington, Duane Sent: Friday, June 17, 2005 3:58 PM To: 'gopalang@coas.oregonstate.edu' Cc: 'mark@coas.oregonstate.edu'; 'Matthew Arrott'; Edgington, Duane Subject: FW: Real-time detection and classification of organisms demo for SuperComputing '05 Attachments: 2344_00_32_40_25.events.xml; 2344_00_32_40_25.results.mpeg; AVED.xsd -------------------------------------------------------------------------------- From: Cline, Danelle Sent: Friday, June 17, 2005 3:41 PM To: Edgington, Duane Subject: RE: Real-time detection and classification of organisms demo for SuperComputing '05 In case Ganesh did not received the XML data, here are the attachments and perhaps a description if you want to sent this to him. 2344_00_32_40_25.results.mpeg - an mpeg of the processed results to help you understand what the XML represents. This is basically the original clip overlayed with bounding boxes. You'll notice each yellow box around each "EventObject" has a number. This number corresponds to the "ObjectID" in the XML file. 2344_00_32_40_25.events.xml. - the AVED processed output in XML format AVED.xsd - the AVED xml schema description . A few words on the XML format. We process frame-by-frame. In the XML file is a frame-by-frame representation of the AVED results. Each "FrameEventSet" represents a physical frame and under this is a collection of all the "EventObjects" that are found in that frame. An event object is simply a representation of an object's features (e.g. CurrSize = square pixel size, CurrX,Y = current centroid, ObjectID=an index number, Saliency = the relative saliency of the object in Volts, and Bounding Box = frames the animal lower left and upper right coordinates). Hope this helps. Danelle -------------------------------------------------------------------------------- From: Edgington, Duane Sent: Friday, June 17, 2005 2:58 PM To: 'Ganesh Gopalan'; Edgington, Duane Cc: mark@coas.oregonstate.edu; 'Matthew Arrott' Subject: RE: Real-time detection and classification of organisms demo for SuperComputing '05 Hi Ganesh, Tell me what you are looking for in the outline of our processing, and I will send you what we have that matches that as close as I can. I guess the information on our output in XML did not tell you what you needed to know. We may not have what you are looking for, but we can make it for you if I know what you need. Duane -------------------------------------------------------------------------------- From: Ganesh Gopalan [mailto:gopalang@coas.oregonstate.edu] Sent: Thursday, June 16, 2005 5:37 PM To: 'Edgington, Duane' Cc: mark@coas.oregonstate.edu; 'Matthew Arrott' Subject: RE: Real-time detection and classification of organisms demo for SuperComputing '05 Duane, Thanks for the update. However, I seem to be missing the attachments (or e-mail) with the outline of your processing. Right now, I’m in the process of identifying elements within the MPEG-7 schema that could incorporate the metadata that is of interest to us, as well as a suitable native XML database to facilitate the searches (this will probably be eXist) with XQuery. There will probably be scalability issues from our side, as well (in spite of eXist’s ability to handle very large documents and document collections), related to the amount of video that will be annotated and searched, but these have not yet been fully investigated. Ganesh. -------------------------------------------------------------------------------- From: Edgington, Duane [mailto:duane@mbari.org] Sent: Wednesday, June 15, 2005 5:27 PM To: Edgington, Duane; 'Ganesh Gopalan' Cc: 'Mark Abbott'; 'Matthew Arrott' Subject: RE: Real-time detection and classification of organisms demo for SuperComputing '05 Hi Ganesh, Just an update. I talked with Matthew this afternoon. I am going to write up a work-plan like you produced. I am going to focus on classification of organisms in this hydrothermal vent ecosystem, with a goal to real-time processing of HD video, but with a fall-back of processing down-converted HD at reduced frame rate. I am going to try to get that draft work-plan out tomorrow. Matthew may explore engaging yet another group in the task of getting our system to run truly real-time on a cluster. I am going to explore getting the HD input stream hardware integrated into our processing system so we can do experiments with full resolution HD. Do you have an update to your proposed set of year one activities (Enabling Content-based Searches in Video Streams using MPEG-7)? The version I have is dated June 1, 2005. Best regards, Duane -------------------------------------------------------------------------------- From: Edgington, Duane Sent: Wednesday, June 15, 2005 3:05 PM To: 'Ganesh Gopalan'; Edgington, Duane Cc: 'Mark Abbott'; 'Matthew Arrott' Subject: RE: Real-time detection and classification of organisms demo for SuperComputing '05 Hi Ganesh, Thanks for the note. I apologize for the delayed response. I have been working on conference deadlines and travel and just could not get back to you earlier. Yesterday I received your proposed set of year one activities (Enabling Content-based Searches in Video Streams using MPEG-7) from Matthew. We believe that interfacing to our system, flowing to MPEG-7, would be pretty straightforward. We output information on our annotations in XML. I have attached an email and the associated files that outline what we are doing. The XML has evolved a little from this information, but it should give you an idea of the info. I am stalled on the following points, though: - We don't do real time processing of video streams (at full frame rate and resolution, anyway) at the present time. I am not sure that is required for the demo in November (which is working from Tape gathered in September). If it is required then our AVED system is not going to be ready to do that this year. - We don't do HDTV at the current time. All of our HD video has been down-sized to digitalBetaCam (NTSC) before we input the data. We can commit to *trying* HDTV native input by September, but it means we need to get hardware integrated into our system at MBARI to input the HDTV streams and we need to allocate the time. It is not on our project roadmap for this year at this time. Again, I don't know if that is required for the demo, or just nice to have. - Finally, and probably most importantly, we do not know what it means to "detect organisms" in the busy scenes coming off of Endeavor. In the one that I have seen, like these http://www.amnh.org/nationalcenter/expeditions/blacksmokers/gallery.html there are often more animals than a human could count. That is not the application space of our system, which is trying to deal with sparseb scenes (where it is expense to have a human look through a lot of "nothing" to get to the interesting events). So for me, the Real Time and HDTV aspects that you lead with are moot if the video is so busy that our system just can't handle it in any case. So, I am eager to get observatory video, and try it in our system. We would be delighted to work with you to integrate our stuff to your MPEG-7 encoder and your search engine work. I am guarded about successfully processing the video that is going to be collected at Endeavor until we have a much better idea of what that is going to look like. Duane -------------------------------------------------------------------------------- From: Ganesh Gopalan [mailto:gopalang@coas.oregonstate.edu] Sent: Thursday, May 26, 2005 5:18 PM To: duane@mbari.org Cc: 'Mark Abbott' Subject: Real-time detection and classification of organisms demo for SuperComputing '05 Duane, It was good talking to you at the Skamania meeting earlier this month. With regard to a teleconference with Matthew Arrott today, a scenario that was discussed for the demo in November was that your AVED system would be coupled with the live HD feed from the bottom of the Pacific to detect organisms in real time, and we thought we'd encode the annotations in MPEG-7 (which is basically XML) and provide a retrospective search engine (web-based or otherwise) for these creatures, where one could type in the name of a creature, the depth and/or latitude/longitude and times and get a set of matching frames that contain the organisms. Would it be possible for you to give us an overview of your current system (or pointers to white papers) to give us an idea of the interface required to enable the MPEG-7 encoding? At some point we were also wondering if we could have access to processed video samples with annotations in whatever format available. Thanks, Ganesh.