Adaptive Skin Color Modeling (based on Dr. Soriano's paper)

This activity is on adaptive skin color tracking. The objective is to be able to track the skin under changing illimination conditions. This means that the skin must be tracked even if the subject in the video changing location, walking from a fluorescent lit room to a sunny outdoors.

SKIN LOCUS

First, all possible colors of skin under different illumination conditions must be determined. Its values in the normalized chromaticity coordinates is called the skin locus. Ideally, this is calculated by taking images with all possible permutation of illumination condition and white balance setting of the camera and adding up all the histograms of the skins.

In our case, we were not able to take images under all possible illumination settings that the camera can white balance. Instead, we took the histograms of the skins under different illumination as captured in our video. Shown below are the cropped portions of the face where the histogram was taken.

The combined histograms of the cropped portions of skin is known as the skin locus, shown below.

ADAPTIVE HISTOGRAM BACKPROJECTION

First, a portion of the face or skin is cropped. This will give the initial histogram of which will be used to locate regions in the image corresponding to the skin. Filter its histogram with the skin locus histogram to remove values which do not correspond to skin.

This filtered histogram is used to segment the current image. The segmented image is then binarized and cleaned using morphological operations. The box enclosing the largest cluster is used to select a portion in the current image. The histogram of this portion is used to segment the next image or frame. But in the next frame, the search region or the region where the histogram backprojection will be applied is limited only to a box slightly larger than the box enclosing the largest cluster. This is done under the assumption that the object or face has not moved too swiftly, moving only a few pixels between frames. This is applied to all the images in the video.


RESULTS

Presented below are slices of the video where the subject is under different illumination conditions. The success of the technique is shown by the correct tracking even if the light illuminating the subject changes.


SOME COMMENTS

Sometimes the largest cluster unusually jumps a large distance, probably because the clusters so disconnected or maybe the wall and the subject becomes similar in color. To address this solution (as suggested by Ma'am Jing), the distance between the centroid of the current largest cluster and the centroid of the previous largest cluster is checked. If it moved more than a set threshold distance, the current largest cluster is rejected. The previous largest cluster is used in the process.

In this activity, I worked with Kaye and Thirdy. Thanks to Kirby for sharing his face for the video hehe :D

I give myself 10 points for this activity. Face/skin tracking was successfully done even with changing illumination. Also, I had to code some functions myself because they were not available in Scilab. :)

Reference:
Soriano, Martinkauppi, Huovinen, Laaksonen, Adaptive skin color modeling modeling using the skin locus for selecting training pixels, Pattern Recognition, Vol. 36, No. 3, March 2003.