Good afternoon,

My name is Marco Moreira, I came from the Federal University of Minas Gerais - Brazil
The project I worked in this summer is the "Automatic Visual Classification of Events in Underwater Video"
And my mentors are Duane Edgington and Danelle Cline

MBARI ROVs collected lots of video from the ocean in the past years.
These videos bring lots of information, but usually the interesting events are time sparse.
As MBARI increases the number of video sources, we are able to improve our ocean statistics 
but it arises the need to create an automatic system to detect the interesting events and separate them from
the rest of the video...

Thinking about that, MBARI engineers created the AVED system, Automated Video Event Detector. 
Separating the interesting events from the rest of the video is a hard task. Besides detecting the events, 
you must track them along the video to ensure that a single event is not counted twice. However, this system is 
already in a mature stage and currently works well.

The next step is to classify the events into their categories. The AVED extracts the interesting events from videos 
and the classifier has to assign the correct class to them: is it a Rathbunaster, a leukothele, ?...
This is the goal of this work. We take the output of the AVED system and try to say the correct
class of each event. The classifier that I'm presenting here is based on a previous one tested at MBARI, which uses 
grayscale images. The contribution of this work is to add color information to that. First I will describe
the old classifier and then explain how color was inserted in it.

Looking at this image, we can say that it is similar to this second one and also that it is very different from the 
animal in the third image. However, when we present these images to a computer, the first image is just a bunch of 
numbers, different from the second bunch of numbers and also different from the third bunch of numbers. Thus, we 
have to think about strategies to extract meaninful features from these bunch of numbers. These features must be 
independent of the animal position in the image.

Schmid and Mohr (in 1997) present some important features of a point which are invariant to translation and 
rotation of the image. One of the invariants is the gray level at a point. The gray level at a point p in the 
animal surface should be the same regardless of the animal position. Another invariant is the magnitude
of the gradient vector. At each point, the gradient vector points to the direction where the greylevel is
increasing with the maximum rate and its magnitude is that rate.  
Altough it has different directions depending on the animal position, the magnitude is the same. 
Schmid and Mohr define 9 of these invariants. Because we don't know where the point p is, we get rid of the
background (the background is estimated in this region close to the borders) and then take the average of the invariants.

So we could build our classifier this way: We have an input image (let's say 100 pixels by 100 pixels, 10000 
pixels), we compute the invariants, 9 values per pixel, then we take the average over the animal and we have 9 
values per image. These 9 values might be enough to describe the animal. 

In 2004 Marc'Aurelio Ranzato observed that if we discount the background and then take the positive, negative and 
absolute part of the invariants, we improve the performance of the classifier. Adding nonlinearities to 
a classifier is a common practice in computer vision.
So our new attempt to build the classifier is: we compute the invariants, 9 values per pixel, then we separate them 
in their positive, negative and absolute parts, 3 times more numbers, then we take the average and we have 27 values 
per image.

Take a look at this image. Can you read what is written in it? Maybe if I blur it you'll be able. This is because we have 
information embedded in low frequencies and when we blur the image, we highlight that information.

Thus we can improve our classifier by computing the invariants in different blurred versions of the image. We have 
the original image, we compute the invariants, then we blur it, compute the invariants again, blur again, compute 
the invariants again and so on. In the end, we have 108 values to describe the image.

108 numbers per image. This is a lot of numbers. We have to think about strategies to reduce it. Then we come up 
with the Fisher's Linear Discriminant. Let's suppose that we don't have 108 numbers. Let's suppose that we have only 
2 numbers. They are the feature 1 and feature 2. 
We have two classes. The class A, these 2 blue clusters and 
class B, which is the red cluster. We could project these cluster in one dimension like this and use just the values 
along this line. Then we could use a probabilistic 
model to describe each class like this.
In our case we are reducing the dimensionality not from 2 to 1 dimensions but from 108 dimensions to a number D, 
that we choose and that is much smaller than 108.

And this is MBARI original classifier: the input is a grayscale image, we use the monochromatic feature extractor, 
then we have 108 values per image, then we use Fisher's Linear Discriminant to reduce the dimensionality to D values 
per image.

Then we added color to that. Instead of using grayscale images, we want to use true color images. 
Color images usually come in the RGB color space. The first attempt is to run the feature extractor 
in each layer: R, G and B.
We have 324 values per image in the end. Again we use Fisher's Linear Discriminant to reduce the dimensionality to a 
chosen number D of dimensions.

Another approach is to convert the images from RGB to YCrCb color space. 
The YCrCb color space is just another representation in which luminance (the Y channel) is separated from
chrominance (Cr and Cb channels). Why should we convert to YCrCb?
The first channel, the Y channel, is similar to the grayscale channel. Then we have in the end of the process 
the same information we had before. We apply the same feature extractor to the Cr and to the Cb color channels and 
we have 324 values per image in the end. Again we use Fisher's Linear Discriminant to reduce the dimensionality to a 
chosen number D of dimensions.

Here are the results:
First we have the grayscale classifier. we have 3 classes: the Rathbunaster, which is this animal; leukothele, this 
one; and other, which can be everything else. When the true class is a Rathbunaster, in 91.7 percent of the cases we 
do say that it's a Rathbunaster; when the true class is a leukothele, in 68 percent of the cases we do the correct 
classification; and when it's "other", we say that it's "other" 79 percent of the cases.

When we use the YCrCb color space, for Rathbunaster, we do the correct classification in 95 percent of the cases; 
for leukothele we get it right in 92 percent of the cases; and for "other" we do the correct thing 96 percent of the 
times. Here we use only 3 features, instead of 8.

Comparing YCrCb to RGb we see that using RGB can give us even better results. We always get it right for the 
Rathbunaster class; we do the correct classification in 94 percent of the cases for leukothele and in 96 percent of 
the cases for the class "other".

How many features should we use? When plotting the individual probability distribution function for 8 features it 
seems that only two of them are important. Off course, we can't visualize these features in 8 dimensions, but when 
we use only 2 numbers for classification, the results are not so much different. 
Here I'm using the RGB color space, but it's also true for grayscale and YCrCb. 

When we use only two features, we see that they have an overlap for the grayscale case (you can see here between 
class Leukothele and class "other"). But when we use the RGB color space, the features are much more well separated.
That's the reason why we got better results when adding color information.

Conclusions

Color information is important for classification
The AVED system + classifier have good perspectives
We can also improve the classification results using other features,
such as the Hu moments, Fourier-Mellin Transform, SIFT descriptors... techniques that were proven to be translation, rotation and scale invariant.