Activity 12 - Color Image Segmentation

In previous activities, we have dealt with mostly grayscale images. Segmentation of grayscale images is easier, especially if the region of interest is darker or lighter than the background. However, for colored images, segmentation is not as easy as thresholding the image. This is because when converted to grayscale, the region of interest sometimes blends in with the background. The activity protocol provides an example, but here is another example.

In the figure below, suppose we want to segment the green gummy bear in the first panel.


Converting it to grayscale results to the second panel. While the green gummy bear is still a bit darker than the background and three other gummy bears, it is lighter than the red and dark red gummy bears. Thresholding the grayscale image will not isolate the green gummy bear, as seen in the third panel. Because it is lighter than the two other gummy bears, thresholding the image until the green gummy bear is visible results in an image where the other two are also visible.

So we see, thresholding is not the best option. There must be other ways in which we can segment color images.

The activity presents two ways of color image segmentation: parametric and non-parametric probability distribution estimation.

Parametric Probability Distribution Estimation

First we select a patch from the region of interest. In the gummy bear image, suppose our region of interest is again the green gummy bear. A patch is selected from the green gummy bear, as shown below.


We assume that its histogram is Gaussian, in both the r and g channels. The mean and standard deviation of the red Gaussian PDF is the mean and standard deviation of the red channel of the selected patch. Below is the Gaussian PDF for the red channel:


In the equation, r is the red chromaticity coordinate of the original image (first panel in image below). The probability that the particular pixel in the original image is given by the above equation. If the pixel is blue, for example, then value in the red chromaticity coordinate is far from the mean, so the probability is low. If the pixel is red, it is near the mean, so the probability is high. The same idea goes for the green chromaticity coordinate, with probability p(g). The joint probability is the product of p(r) and p(g), and this gives the image of the location of the green gummy bear (shown in the second panel in the image below).



The segmented image in the second panel is in grayscale, and if we are to use this segmented image for, say, area estimation purposes, we then convert it to binary (1's and 0's) by thresholding, as seen in the third panel.

Notice that the segmentation did not recover some of the areas of the green gummy bear, especially in the ears. This may be because of the area covered by the patch, which did not include the bright spots near the ears. Doing the same procedure, this time for the patch (below, left) which now includes the bright spots but with the majority still being the green color of the body, the segmentation is better (below, center). Thresholding the segmented image now gives a better segment of the region of interest which is the green gummy bear.


(Note that the patch above is not scaled with the other two images.)

So far, we have implemented parametric PD estimation and we have seen that is is dependent on the selected patch.

Non-parametric probability distribution estimation (Histogram backprojection)

For the histogram backprojection, we first determine the histogram of the selected patch. I used the second patch from the method above because it gave better results for that method. The histogram is shown below (the origin is the lower left pixel).


In the histogram, the x-axis is the red values and the y-axis is the green value. The values in the histogram are mostly in the green area, in agreement with the perceived color of the selected patch.

Next, we check each pixel in the image. Each pixel in the image has both an r and g chromaticity coordinate. The values of r and g are from 0-1, so we need to match it first with the histogram. The bin size of the histogram is chosen to be 32, so the values of r and g are scaled such that the values are between 1 and 32. Checking each pixel in the image gives us an r and g value. We look this values up in the patch histogram. For example, the first pixel has an r value of 15 and a g value of 20. We look for the probability in the histogram for the r value of 15 (row 15) and the g value of 20 (column 20). This pixel value is the probability that the current image pixel has the color of the patch. Doing this for the whole image gives the result shown in the second panel in the image below. The image is thresholded, giving the third panel in the image below.




The parametric method segments the image better than the non-parametric method, at least for the choice of patch. The thresholded image still has holes for the non-parametric method while for the parametric method, the segment is almost solid.

For comparison, I tested both the methods for the orange gummy bear this time. Below are the grayscale segmented images.



The parametric method again is better because it differentiated the two orange gummy bears while the non-parametric method detected both the orange gummy bears. It is also easier to implement than the non-parametric, at least for me.

I give myself 10 points for this activity. I was able to implement both methods, and determined which method is better. The next blog entry shows an extension to this activity.