\chapter{Whole system}	\label{sec:ws}
\thispagestyle{empty}
  

This chapter has for its main aim to combine detector and classifier in order to assess the performance of the whole system.\\
We can identify particles only in the database of pollen captured by spore trap and so, we will refer to this database throughout the following discussion.

We consider the morphological detector because it gave good results and it is able to do a good segmentation of the object of interest. We
assume that for each found pollen there are 10 false alarms (see \tabref{performanceDetMor2}). \\
The first step is to build a database with those patches individuated by this detector. Basically, we run the detector on all images with
pollen grains captured by air sampler machine and each detected patch is put in the proper folder according to the experts' reference list. We
add a new class, called ``\emph{unknown2}'', to gather the false alarm patches. Note that during classification we use also class 
``\emph{unknown1}''. This is the class where we put the images which do not pass the test of eq. (\ref{eq:testrisol}). \\
In \tabref{listaDBDet}, you can find the number of patches in each category and in \figref{esempioDBDet} we show some patches of this database.

\begin{table}[htbp]
\begin{center}
\begin{tabular}{||p{5cm}|c||}
\hline
\scshape{genus}	& \scshape{number of items} 	\\
\hline
\hline
\bfseries ash & 72\\
\hline
\bfseries chin. elm & 119 \\
\hline
\bfseries oak & 67 \\
\hline
\bfseries pecan & 5 \\
\hline
\bfseries cypress & 32 \\
\hline
\bfseries eucalyp. & 3 \\
\hline
\bfseries grass & 1 \\
\hline
\bfseries olive  & 10 \\
\hline
\bfseries walnut & 17 \\
\hline
\bfseries pine & 32 \\
\hline
\bfseries liq. amber & 19 \\
\hline
\bfseries alder & 13 \\
\hline
\bfseries asteraceae & 3 \\
\hline
\bfseries c. myrtle & 3 \\
\hline
\bfseries chenopod & 19 \\
\hline
\bfseries ginkgo & 3 \\
\hline
\bfseries mulberry & 36 \\
\hline
\bfseries palm & 21 \\
\hline
\bfseries plane t. & 7 \\
\hline
\bfseries poplar & 5\\
\hline
\bfseries sycamore & 17 \\
\hline
\bfseries umbellif. & 4 \\
\hline
\bfseries false alarms & 8922 \\
\hline
\hline
\end{tabular}
\end{center}
\caption{Pollen database built by morphological detector. Number of patches in each class.}
\label{tab:listaDBDet}
\end{table}


\begin{table}[htbp]
\begin{center}
\begin{tabular}{||p{5cm}|c|c||}
\hline
\scshape{class}	& \scshape{train.} 	& \scshape{test}\\
\hline
\hline
\bfseries ash & 65	& 7\\
\hline
\bfseries chin. elm & 107	& 12\\
\hline
\bfseries oak & 61	& 6\\
\hline
\bfseries unk. 2 & 2330		& 250\\
\hline
\hline
\bfseries \emph{total} & \emph{2563}	& \emph{275} \\
\hline
\hline
\end{tabular}
\end{center}
\caption{Number of samples used in training and test to assess the performance of the whole system.}
\label{tab:listaWSExp}
\end{table}



\begin{figure}[htbp]
\centering
\includegraphics[width=0.65\textwidth,height=0.4\textwidth]{./figure/esempioDBDet.eps}
\caption{Example of patches in database built by detector.}
\label{figure:esempioDBDet}
\end{figure}

We apply the Bayesian classifier using features based on local jets. This classifier is one of the best we have found but it needs a lot of images
to estimate parameters in the MoG. For this reason, we consider only categories with more than 50 images, namely: ash, elm, oak and obviously
``unknown2''. We extract from the ``unknown2'' set a number of images that is coherent with detector performance. The total number of pollen
patches is 258, thus we consider 2580 patches of unknown particles. From each class set we take randomly the 10\% of images for test, the
rest is for training. To summarize, we test 7 images of ash, 6 of oak, 12 of chinese elm and 250 of ``unknown2'' class, see \tabref{listaWSExp}. 

We can choose to model the ``unknown2'' class or to model only the pollen categories. In our experiments we try both solutions.

Test stage in the classifier is modified to take into account the analysis of ``unknown2'' particles. Precisely, besides the test on 
resolution of eq. (\ref{eq:testrisol}), there is a test on the estimated probability of the class given the image. If this value is too low,
then the classifier puts the image in class ``unknown1'', the same happens if the image resolution is out of the range of every class, see eq.
(\ref{eq:testrisol}).

In the next sections, we will describe the experiments done to tune the classifier and then, we will show the final results.

\section{Tuning}

We consider two systems: the first one models only the three pollen categories, the second one models also the ``unknown2'' class.
For each of these, we have to find:

\begin{itemize}
\item number of components in the mixture and structure of their covariance matrix
\item initial conditions for random number generator used to initialize EM algorithm
\item threshold on probability to decide if an image has to be put in ``unknown1'' class
\end{itemize}

When we make the optimization of parameters for pollen categories we consider only the case of spherical or diagonal structure for the
covariance matrix because we do not have enough data to estimate all parameters of a MoG with full covariance matrix components. 
On the other hand, in the
optimization of ``unknown2'' class we only consider full covariance structure and we compute a more complex model for this class because
we have available a lot of images and because this class has presumably a statistics quite broad and complex. In other words, it is likely
that points belonging to this class ``fill'' all the feature space and do not gather in a single cluster.
\figref{ERvsParamCL3WS} and \figref{ERvsParamCL4UWS} show the proportional error rate as a function of the number of parameters 
when you do not model the ``unknown2'' class and when you do. In the latter case, we changed only the number of Gaussians in the mixture
which model the ``unknown2'' class and the pollen categories are described by mixtures with 2 spherical Gaussians. In \tabref{paramWS}, we
show the chosen parameters.

\begin{figure}[tbp]
\centering
\includegraphics[width=1\textwidth]{./figure/ERvsParamCL3WS.eps}
\caption{Proportional error rate as a function of the number of parameters. The Bayesian classifier using features based on local jets does not
model the ``unknown2'' class.}
\label{figure:ERvsParamCL3WS}
\end{figure}

\begin{figure}[tbp]
\centering
\includegraphics[width=1\textwidth]{./figure/ERvsParamCL4UWS.eps}
\caption{Proportional error rate as a function of the number of parameters. The Bayesian classifier using features based on local jets models the
``unknown2'' class. In these experiments, only the model for this class was changed, namely the number of components in the MoG was varied at
each iteration. The pollen classes were modeled with MoG having 2 spherical Gaussians.}
\label{figure:ERvsParamCL4UWS}
\end{figure}


Note that throughout this chapter, the proportional error rate is our reference point differently from the previous chapters because of the great
difference in the number of test images belonging to different categories. Moreover, we consider that an image
belonging to ``unknown2'' class is correctly classified when it is assigned both to class ``unknown1'' and, if the model exists, to class
``unknown2''.\\
Looking at \figref{ERvsParamCL4UWS} and to the absolute error rate here not reported, we observe that increasing the dimensions of feature
space causes a better classification for the junk class but a worse classification for the pollen categories. Proportional error rate allows
to weight more errors on pollen categories that have less test samples.

Once also the best initial conditions have been found applying the same method described in par. \ref{sec:classexp}, we are ready 
to estimate the optimal threshold of probability. With the previous optimal parameters found, we run 10 experiments for each choice of threshold; the error
rates shown in \figref{ThresProbCL3WS} and \figref{ThresProbCL4UWS} are the averaged values. When the system does not model the junk class,
the pollen classification is not influenced by the threshold of probability (the blue line is nearly constant) while the junk class is
obviously better identified by an high threshold (the red line is strongly related to the classification of this class). 
Indeed in chapter \ref{sec:errana}, we noticed that the classifier 
is most often wrong when the probability assigned to an image is below 90\% and on the other hand, correctly classified images have high probability.\\
Instead, \figref{ThresProbCL4UWS} shows a behavior more independent from the threshold choice.

From this analysis, we deduce that it is better to model the ``unknown2'' class. In \figref{ThresProbCL3WS}, we see that most of junk
particles are misclassified while in \figref{ThresProbCL4UWS} we note that this trend is corrected because the absolute error rate, 
which mainly depends on junk images, is much less than the proportional error rate. This observation will be confirmed by the final results.
\tabref{paramWS} summarize the choice of parameters. 

\begin{table}[htbp]
\begin{center}
\begin{tabular}{||p{3cm}|c|c|c|c|c|c||}
\hline
\scshape{classif.}	& \scshape{Dimens.} 	& \scshape{$ng_1$}	& \scshape{$type_1$} 	 & \scshape{$ng_2$}	& \scshape{$type_2$}
& Thres.	\\
\hline
\hline
\bfseries only 3 pollen classes	& 10	& 2	& spher.	& $\backslash$ 	& $\backslash$ & 0.7\\
\hline
\bfseries 3 pollen classes and unk. 	& 16	& 2	& spher.	& 6	& full & 0.5\\
\hline
\hline
\end{tabular}
\end{center}
\caption{Parameter chosen for the Bayesian classifier using MoG; the first row is referred to the classifier that does not model junk class.
 The dimension of feature space, the number of components and their covariance matrix structure is shown for the pollen models (subscript 1) and for the junk class (subscript 2).}
\label{tab:paramWS}
\end{table}



\begin{figure}[tbp]
\centering
\includegraphics[width=1\textwidth,height=0.6\textwidth]{./figure/ThresProbCL3WS.eps}
\caption{Error rate as a function of the threshold on probability. Plot referred to classifier which does not model junk class.}
\label{figure:ThresProbCL3WS}
\end{figure}

\begin{figure}[tbp]
\centering
\includegraphics[width=1\textwidth,height=0.6\textwidth]{./figure/ThresProbCL4UWS.eps}
\caption{Error rate as a function of the threshold on probability. Plot referred to classifier which models junk class.}
\label{figure:ThresProbCL4UWS}
\end{figure}

\newpage
\section{Final Experiments}

Because of the shortage of images in our database, we could not divide it in training, validation and test sets. Each time we took randomly 
10\% of images for test and the rest for training. 
In order to evaluate the performance of the system, we made an average of the results of 100 experiments. The confusion matrices are shown in
\figref{CM100CL3WS} and \figref{CM100CL4UW}. 

\begin{figure}[htbp]
\centering
\includegraphics[width=0.95\textwidth,width=0.85\textwidth]{./figure/CM100CL3WS.eps}
\caption{Test and training confusion matrix, left and right respectively. The classifier does not model the junk class. Note that almost all junk particles are classified as pollen.}
\label{figure:CM100CL3WS}
\end{figure}

\begin{figure}[htbp]
\centering
\includegraphics[width=0.95\textwidth,width=0.85\textwidth]{./figure/CM100CL4UWS.eps}
\caption{Test and training confusion matrix, left and right respectively. The classifier models the ``unknown2'' class.}
\label{figure:CM100CL4UW}
\end{figure}

The performance of the system which models the junk class is much better. In this system, most of false alarms fall in either ``unknown1''
or ``unknown2'' classes. Note that the error rate has about the same value of the error rate found in par. \ref{sec:classFinExp} when we tested
on air sampled pollen grains with patches selected by the human expert and without the false alarm class. This is possible because our detector is able to do a segmentation
comparable to the expert's one and we have a good model for the false alarm category. 

The variance of the error rate is quite high, around 7\%. For example in the experiments of \figref{CM100CL4UW}, the minimum error rate was
about 12\% while the maximum 48\%. This should be interpreted as a lack of reliability of our result because of the shortage of images used in
test and in training.

You should note also the gap between training and test error rate. This means that we are using too few images for training and we are
overfitting the training data. On the other hand, we are confident that using more images for training this gap will decrease until nearly
zero and the final performance will be around or below 10\% as predicted by the reasoning done for \figref{estimatedERTrainTestAir}.

\section{Conclusion}

In this chapter it was described the performance of the whole system on images of pollen captured by spore trap. The detector is able to detect
with probability of 9/10 all pollen grains but it detects also for each pollen ten other particles. With the spatial resolution of our images this
result is quite good.

The next stage is classification. When the system learns to classify using few images for training and it models also the junk class the
error rate is about 30\%. This result confirms the performance of the same classifier when the expert's patches were used thus, we deduce
that the automatic segmentation is really good. Moreover, this time we deal also with the false alarm class but thanks to a more complex 
model, we achieve on this category the best correct classification rate:  nearly 90\%.\\
Looking at the training error rate and to the estimate done in \figref{estimatedERTrainTestAir}, 
we are confident that this system will be able to achieve an error rate of 10\% when it will be fed with more images because we will be able to estimate more complex models and the system will have a better generalization of training data. 

The experiments are done in conditions which are very close to a real situation. Particularly, we keep the ratio between pollen and junk
particles at one out of ten and the classifier receives objects in the right proportion. In this way, we overcome the difficulty to select
images for training and to classify pollen particles that are not modeled because their number is too low.

To give an idea of the current performance we now give an example. \\
Suppose the system receives 50 images with 100 pollen grains. Then 90 of them will be detected and 1000 false alarms will be generated.
The classifier receives 1090 patches and will correctly identify nearly 70\% of pollen particles. This means that at the end 63 pollen grains 
 will be correctly recognized by our system. 

    
 

 
\clearemptydoublepage
