Mouth removal method for red-eye detection and correction

BACKGROUND

Field of Invention

Red-eye detection and correction technologies are used in printers, digital cameras, and image editing software to localize and correct the red-eye effects in digital photographs captured using a flash. Though there has been a great deal of progress in red-eye detection and correction in the last few years, many problems remain unsolved. Research in these areas must confront many challenging problems, especially when dealing with varying illumination, low image quality and resolution, eye size and variations in face orientation, and background changes in complex scenes.

In general, early stages of a red-eye detection involve distinguishing between true red-eye objects and a number of incorrectly detected non-red-eye objects (referred to as false positives or falses), which are particularly evident in complex scenes. This false detection processing can be reduced based on the evaluation of the object's color, structural and geometric characteristics, as disclosed in commonly assigned U.S. patent application Ser. No. 12/349,911, filed Jan. 7, 2009, and entitled “Method of detecting red-eye objects in digital images using color, structural, and geometric characteristics,” which application is hereby expressly incorporated herein by reference in its entirety. False detection processing can also be reduced based on luminance-chrominance characteristics, contrast characteristics, region smoothness characteristics, binary pattern characteristics and glasses frame characteristics, as disclosed in commonly assigned U.S. patent application Ser. No. 12/575,321, filed Oct. 7, 2009, and entitled “Automatic red-eye object classification in digital photographic images,” which application is hereby expressly incorporated herein by reference in its entirety. False detection processing can be reduced further based on some trained classifier such as boosting-based framework, as disclosed in commonly assigned U.S. patent application Ser. No. 12/575,298, filed Oct. 7, 2009, and entitled “Automatic red-eye object classification in digital images using a boosting-based framework,” which application is hereby expressly incorporated herein by reference in its entirety.

Although these techniques can significantly suppress false detections, some false positives still remain, in particular mouth patterns. This is due to the fact that mouth patterns usually exhibit similar color and structural characteristics as that of true red-eye objects. Correction of the mouth area as a falsely detected red-eye object will ruin the picture and disappoint the end user. Therefore, suppressing mouth (false red-eye) detections becomes a critical factor in ensuring a successful application in real world products.

SUMMARY OF INVENTION

An object of the present invention is to remove mouth detections (false positives) as red-eye candidates while preserving high-computational speed and eye detection rate.

The present invention assumes that an input image (e.g. a digital RGB color image) has been subjected to an eye classifier that is targeted at discriminating a complete eye pattern from any non-eye patterns. The present invention starts with the red-eye candidate list with associated bounding boxes that are generated by the red-eye classifier.

The bounding rectangles are subjected to object segmentation. As is known, segmentation, generally, is the process of grouping together into a single entity (object) pixels that have something in common. In the present invention, the purpose of object segmentation is to group or segment out red regions (mouth or red retina).

A connected component labeling procedure is then applied to obtain one or more red regions. The largest red region denoted as “newFeaObj” is then chosen for feature extraction.

A number of features are then extracted from this region. These include: feature k_w, defined as the ratio between a width of the segmented red region and a width of the bounding rectangle; feature k_h, defined as the ratio between a height of the segmented red region and a height of the bounding rectangle; feature s, defined as the ratio between the width and height of the segmented red region; feature v, defined as a decision score from the eye classifier; and feature m, defined as a geometrical relationship between the particular candidate red-eye object and others of the plurality of candidate red-eye objects.

The present invention then uses these features k_w, k_h, s, v, and m to determine if the particular candidate red-eye object is a mouth.

Other objects and attainments together with a fuller understanding of the invention will become apparent and appreciated by referring to the following description and claims taken in conjunction with the accompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

In the drawings wherein like reference symbols refer to like parts:

<figref idrefs="DRAWINGS">FIG. 1</figref> is a flowchart of the general method of the present invention;

<figref idrefs="DRAWINGS">FIG. 2</figref> is a general block diagram of an image capture device and system for utilizing the present invention;

<figref idrefs="DRAWINGS">FIG. 3</figref> is an example of a captured image in RGB color space and the bounding boxes output by a typical red-eye classifier;

<figref idrefs="DRAWINGS">FIG. 4</figref> shows the image of <figref idrefs="DRAWINGS">FIG. 3</figref> in normalized LAB color space;

<figref idrefs="DRAWINGS">FIG. 5</figref> illustrates the results of segmentation of the image of <figref idrefs="DRAWINGS">FIG. 3</figref> in LAB color space;

<figref idrefs="DRAWINGS">FIG. 6</figref> illustrates the bounding box of eye candidates output by a typical red-eye classifier, with (a) showing a whole eye, and (b) showing a partial eye;

<figref idrefs="DRAWINGS">FIG. 7</figref> illustrates the bounding box of mouth pattern candidates output by a typical red-eye classifier, with (a) showing a whole mouth, and (b) showing a partial mouth;

<figref idrefs="DRAWINGS">FIG. 8</figref> illustrates the geometrical relationship between objects in an image with (a), (b), and (c) showing three different resultant labels for feature m used in the present invention;

DESCRIPTION OF THE PREFERRED EMBODIMENTS

In the following detailed description, reference is made to the accompanying drawings that show, by way of illustration, example embodiments of the invention. In the drawings, like numerals describe substantially similar components throughout the several views. These embodiments are described in sufficient detail to enable those skilled in the art to practice the invention. Other embodiments may be utilized and structural, logical and electrical changes may be made without departing from the scope of the present invention. Moreover, it is to be understood that the various embodiments of the invention, although different, are not necessarily mutually exclusive. For example, a particular feature, structure, or characteristic described in one embodiment may be included within other embodiments. The following detailed description is, therefore, not to be taken in a limiting sense, and the scope of the present invention is defined only by the appended claims, along with the full scope of equivalents to which such claims are entitled.

In general, example embodiments relate to methods, devices, and computer-readable media for detecting red-eye objects in images. Example embodiments can be used in conjunction with red-eye correction applications to produce images in which red-eye objects are detected and modified to remove or minimize the red-eye effect. Consequently, image quality may be enhanced automatically with little or no user manipulation of the image.

Example embodiments detect a red-eye effect in images by evaluating whether or not a red-eye candidate object is a mouth or not. Methods consistent with the invention may be implemented in image capture devices such as scanners or digital cameras, as well as in software modules including printer drivers or image editing software, among other things.

With reference now to <figref idrefs="DRAWINGS">FIG. 1</figref>, an example method <b>100</b> for red-eye detection is disclosed. More particularly, the example method <b>100</b> identifies objects in the image and determines whether the objects are candidate red-eye objects. Each candidate red-eye object may then be eliminated or retained as a candidate by deciding whether or not the candidate object is a mouth pattern or not.

The example method <b>100</b> and variations thereof disclosed herein can be implemented using computer-readable media for carrying or having computer-executable instructions or data structures stored thereon. Such computer-readable media can be any available media that can be accessed by a processor of a general purpose or special purpose computer. By way of example, and not limitation, such computer-readable media can comprise RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to carry or store program code in the form of computer-executable instructions or data structures and which can be accessed by a processor of a general purpose or special purpose computer. Combinations of the above should also be included within the scope of computer-readable media.

Computer-executable instructions comprise, for example, instructions and data which cause a processor of a general purpose computer or a special purpose computer to perform a certain function or group of functions. Although the subject matter is described herein in language specific to methodological acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific acts described herein. Rather, the specific acts described herein are disclosed as example forms of implementing the claims.

Examples of special purpose computers include image processing devices such as digital cameras (an example of which includes, but is not limited to, the Epson R-D1 digital camera manufactured by Seiko Epson Corporation headquartered in Owa, Suwa, Nagano, Japan), digital camcorders, projectors, printers, scanners, copiers, portable photo viewers (examples of which include, but are not limited to, the Epson P-3000 or P-5000 portable photo viewers manufactured by Seiko Epson Corporation), or portable movie players, or some combination thereof, such as a printer/scanner/copier combination (examples of which include, but are not limited to, the Epson Stylus Photo RX580, RX595, or RX680, the Epson Stylus CX4400, CX7400, CX8400, or CX9400Fax, and the Epson AcuLaser® CX11NF manufactured by Seiko Epson Corporation) or a printer/scanner combination (examples of which include, but are not limited to, the Epson TM-J9000, TM-J9100, TM-J7000, TM-J7100, and TM-H6000III, all manufactured by Seiko Epson Corporation) or a digital camera/camcorder combination. An image processing device may include a red-eye detection capability, for example, to detect red-eye effects in an image. For example, an image capture device, such as a camera or scanner, with this red-eye detection capability may include one or more computer-readable media that implement the example method <b>100</b>. Alternatively, a computer connected to the image capture device may include one or more computer-readable media that implement the example method <b>100</b>.

A schematic representation of an example image capture device <b>200</b> is disclosed in <figref idrefs="DRAWINGS">FIG. 2</figref>. The example image capture device <b>200</b> exchanges data with a host computer <b>250</b> by way of an intervening interface <b>202</b>. Application programs and an image capture device driver may also be stored for access on the host computer <b>250</b>. When an image retrieve command is received from the application program, for example, the image capture device driver controls conversion of the command data to a format suitable for the image capture device <b>200</b> and sends the converted command data to the image capture device <b>200</b>. The driver also receives and interprets various signals and data from the image capture device <b>200</b>, and provides necessary information to the user by way of the host computer <b>250</b>.

When data is sent by the host computer <b>250</b>, the interface <b>202</b> receives the data and stores it in a receive buffer forming part of a RAM <b>204</b>. The RAM <b>204</b> can be divided into a number of sections, for example through addressing, and allocated as different buffers, such as a receive buffer or a send buffer. Data, such as digital image data, can also be obtained by the image capture device <b>200</b> from the capture mechanism(s) <b>212</b>, the flash EEPROM <b>210</b>, or the ROM <b>208</b>. For example, the capture mechanism(s) <b>212</b> can generate a digital photographic image. This digital image can then be stored in the receive buffer or the send buffer of the RAM <b>204</b>.

A processor <b>206</b> uses computer-executable instructions stored on a ROM <b>208</b> or on a flash EEPROM <b>210</b>, for example, to perform a certain function or group of functions, such as the method <b>100</b> for example. Where the data in the receive buffer of the RAM <b>204</b> is a digital image, for example, the processor <b>206</b> can implement the methodological acts of the method <b>100</b> on the digital image to detect red-eye objects in the digital image and thereby remove or minimize red-eye effects. Further processing in an imaging pipeline may then be performed on the digital image before the image is displayed by the image capture device <b>200</b> on a display <b>214</b>, such as an LCD display for example, or transferred to the host computer <b>250</b>, for example.

The example method <b>100</b> for detecting red-eye effects in an image will now be discussed in connection with <figref idrefs="DRAWINGS">FIG. 1</figref>. Prior to performing method <b>100</b>, an input image can be targeted for various image processing operations including red-eye detection. The targeted input image may be a digital color image or a digitized or scanned version of a color image. Various image processing techniques may be applied to the targeted input image before method <b>100</b> is performed.

The present invention assumes that an input image (e.g. a digital RGB color image) has been subjected to an eye classifier that is targeted at discriminating a complete eye pattern from any non-eye patterns. An example of such classifier is the boosting based classifier disclosed in commonly assigned U.S. patent application Ser. No. 12/575,298, filed Oct. 7, 2009, and entitled “Automatic red-eye object classification in digital images using a boosting-based framework,” which application is hereby expressly incorporated herein by reference in its entirety. The present invention is not limited to or concerned with any particular eye classifier and any known classifier such as those based on an AdaBoost algorithm may be used to generate the red eye candidate list and their associated bounding boxes. Other examples include, but are not limited to, support vector machines, neural networks and other training or non-training based classifiers. The output of such classifier is usually a rectangular bounding box which is intended to include a complete eye region, but may indeed include a bounding box of a mouth region.

The present invention starts (START, step <b>102</b>) with the red-eye candidate list with associated bounding boxes that are generated by the red-eye classifier. <figref idrefs="DRAWINGS">FIG. 3</figref> illustrates a typical example of an input image that has been subjected to a red-eye classifier with the result being 3 candidate objects (two true red-eye objects and one mouth, false positive, object) being output from the classifier. The rectangles shown in <figref idrefs="DRAWINGS">FIG. 3</figref> represent the bounding boxes output by the red-eye classifier. Based on the observation that in the detected region (bounded by the rectangular bounding box), the true red-eye pattern usually contains only a relatively few red pixels due to the fact the eye ball region occupies a small part of a complete eye. However, for a mouth pattern, the red area usually occupies a relatively large part of the bounding box. Therefore, the ratio between the bounding box size and red part size can be an effective indicator to help discriminate between an eye pattern and a mouth pattern. This distinction between true red-eye candidates and false mouth pattern candidates is used in the present invention to exclude mouth patterns.

In the present invention, the bounding rectangles, for example those shown in <figref idrefs="DRAWINGS">FIG. 3</figref>, are subjected to object segmentation (<figref idrefs="DRAWINGS">FIG. 1</figref>, step <b>104</b>). As is known, segmentation, generally, is the process of grouping together into a single entity (object) pixels that have something in common. In the present invention, the purpose of object segmentation is to group or segment out red regions (mouth or red retina). In these groups, features are extracted or calculated as will be described later herein.

Object segmentation is used in the present invention to detect red pixels and regions. The normalized LAB space is proved to be effective for segmenting out mouth-like patterns. As is well known, a LAB color space is a color-opponent space with dimension L measuring lightness, and A and B representing color-opponent dimensions. <figref idrefs="DRAWINGS">FIG. 4</figref> depicts the normalized LAB image of <figref idrefs="DRAWINGS">FIG. 3</figref>. In this depiction the whole image is transformed into LAB space for illustration purposes. However, in the actual implementation only pixels in the bounding rectangles are needed to be transformed into LAB space. The values of L, A, and B are first normalized to [0, 1] and then mapped into [0, 255]. It is observed from experiments that in most cases, for a mouth object, values of A and (A−B) of the red pixels are normally larger than those of other pixels. Therefore, each pixel (located at coordinate (x, y)) inside the candidate bounding rectangle can be determined as a red pixel or not according to its A and (A-B) values as follows,

<maths id="MATH-US-00001" num="00001"><math overflow="scroll"><mrow><mrow><mi>b</mi><mo>⁡</mo><mrow><mo>(</mo><mrow><mi>x</mi><mo>,</mo><mi>y</mi></mrow><mo>)</mo></mrow></mrow><mo>=</mo><mrow><mo>{</mo><mtable><mtr><mtd><mn>1</mn></mtd><mtd><mrow><mrow><mi>if</mi><mo>⁢</mo><mstyle><mspace width="0.8em" height="0.8ex" /></mstyle><mo>⁢</mo><msub><mi>A</mi><mrow><mo>(</mo><mrow><mi>x</mi><mo>,</mo><mi>y</mi></mrow><mo>)</mo></mrow></msub></mrow><mo>&gt;</mo><mrow><msub><mi>T</mi><mi>A</mi></msub><mo>⁢</mo><mstyle><mspace width="0.8em" height="0.8ex" /></mstyle><mo>⁢</mo><mi>and</mi><mo>⁢</mo><mstyle><mspace width="0.8em" height="0.8ex" /></mstyle><mo>⁢</mo><mrow><mo>(</mo><mrow><msub><mi>A</mi><mrow><mo>(</mo><mrow><mi>x</mi><mo>,</mo><mi>y</mi></mrow><mo>)</mo></mrow></msub><mo>-</mo><msub><mi>B</mi><mrow><mo>(</mo><mrow><mi>x</mi><mo>,</mo><mi>y</mi></mrow><mo>)</mo></mrow></msub></mrow><mo>)</mo></mrow></mrow><mo>&gt;</mo><msub><mi>T</mi><mrow><mi>A</mi><mo>⁢</mo><mi>_</mi><mo>⁢</mo><mi>B</mi></mrow></msub></mrow></mtd></mtr><mtr><mtd><mn>0</mn></mtd><mtd><mi>otherwise</mi></mtd></mtr></mtable></mrow></mrow></math></maths>

where b(x,y)=1 denotes a red pixel at coordinate (x, y) and b(x,y)=0 denotes a non-red pixel at coordinate (x, y), T_Aand T_A_{<sub2>—</sub2>}_Bare thresholds determined by an automatic threshold selection technique. In a preferred embodiment, Otsu's method is used. As is well known, Otsu's method automatically performs histogram shape-based image thresholding, or, the reduction of a graylevel image to a binary image. A connected component labeling procedure is then applied to obtain one or more red regions. The largest red region denoted as “newFeaObj” is then chosen for feature extraction, discussed later herein. <figref idrefs="DRAWINGS">FIG. 5</figref> depicts the segmentation results for each candidate object in <figref idrefs="DRAWINGS">FIG. 3</figref>, with the largest red region being the mouth pattern.

Due to the complex illumination variations, segmentation in a single color space can not always guarantee optimum results. Therefore, segmentation in different color spaces is also applied. The features described later will be calculated in LAB space and some other color spaces. Features extracted from different color spaces are then combined and utilized in the decision making step <b>108</b>, described later, to give a more reliable result. In one embodiment, similar segmentation is applied in the RGB (red, green, blue) color space to identify red pixels in the bounding rectangle of each candidate object.

The present invention is not limited to, or particularly concerned with any particular segmentation method, thresholding method, or color space. Although Otsu's method is applied in the LAB and RGB color spaces in one exemplary implementation, any type of segmentation technology, redness measurement and color spaces may be utilized as long as the mouth like red pixels can be extracted.

The next step in method <b>100</b> is feature extraction (step <b>106</b>). Before detailing this step, we will first discuss the various features that are of interest in the present invention.

Feature k_w

First, we consider a horizontal scenario where the left eye and right eye are horizontally aligned.

<figref idrefs="DRAWINGS">FIG. 6</figref> depicts a typical example of an eye pattern and its corresponding bounding box (large, exterior rectangle in the figure) output by an eye classifier. <figref idrefs="DRAWINGS">FIG. 6</figref> (a) demonstrates a perfect detection scenario and (b) demonstrates a partial detection scenario, which is due to the imperfectness of the eye classifier. It has been observed that although some partial detection exists in many eye classification methods, the output bounding box of an eye candidate is usually much larger than red retina region, which is bounded by the smaller, interior rectangle in the figure.

<figref idrefs="DRAWINGS">FIG. 7</figref> depicts a typical example of a mouth pattern and its corresponding bounding box (exterior rectangle in the figure) output by an eye classifier. It has been observed that the red region (bounded by the smaller, interior rectangle in the figure) occupies a larger portion of the object bounding box in both perfect (a) and partial (b) detection cases.

Therefore, the first feature k_wis defined as follows: <br /><i>k</i>_w<i>=W</i>_newFeaObj<i>/W</i>_EC,<br /> where W_newFeaObjdenotes the width of the red region, which was segmented out of the image in step <b>104</b> (<figref idrefs="DRAWINGS">FIG. 1</figref>) discussed above, and W_ECdenotes the width of the bounding box output by an eye classifier. In most cases, the k_wvalue of a mouth pattern is larger than that of an eye pattern.

Feature k_h

Feature k_wis effective for horizontal eye/mouth classification as shown in <figref idrefs="DRAWINGS">FIGS. 6 and 7</figref>. For vertical cases where the two eyes of one person are vertically aligned, feature k_his proposed as follows: <br /><i>k</i>_h<i>=H</i>_newFeaObj<i>/H</i>_EC,<br /> where H_newFeaObjdenotes the height of the red region, which was segmented out of the image in step <b>104</b> (<figref idrefs="DRAWINGS">FIG. 1</figref>) discussed above, and H_ECdenotes the height of the bounding box output by an eye classifier. Similarly, in most cases, the k_hvalue of a vertical mouth pattern is larger than that of a vertical eye pattern.

Feature s

Feature s is defined as the ratio between W_newFeaObjand H_newFeaObjas follows, <br /><i>s=W</i>_newFeaObj<i>/H</i>_newFeaObj

Feature v

Feature v is defined as the decision score output by an eye classifier. It can be similarity score, confidence level or vote. In one embodiment, v denotes the vote of a boosting based classifier. The present invention is not limited to or concerned with any particular eye classifier. However, AdaBoost is a good, commonly known and commonly used method that constructs a classifier in an incremental fashion by adding simple classifiers to a pool and using their weighted vote to determine their final classification.

Feature m

Feature m captures the geometric relationship between objects in a given image. <figref idrefs="DRAWINGS">FIG. 8</figref> shows three cases of feature m, assuming a horizontal mouth (as shown in <figref idrefs="DRAWINGS">FIG. 7</figref>) is considered. In <figref idrefs="DRAWINGS">FIG. 8</figref>, “object” represents the candidate object (to be determined as a mouth or not) and the other circles represent other objects in the given image. In case (a), if 1 or 2 objects are found in the top side region within a predetermined distance of the candidate object, feature m of the candidate object is flagged as “Mouth_Eye_Pair”. In case (b), m of the candidate object is flagged as ‘Eye_Eye_Pair” if there is one object found in its left or right side region within a predetermined distance. In other cases, m is flagged as “No_Pair”. <figref idrefs="DRAWINGS">FIG. 3</figref> (c) illustrates an example where m is flagged as “No_Pair”. The “predetermined distances” referenced above must be determined empirically for a particular device, for example by taking test photos of faces with a device, subjecting them to an eye classifier and determining typical distances between candidate red-eye objects that correspond to the actual eyes and mouth of the test photos.

For vertical cases, feature m can be determined in a similar way.

Now that we have outlined the features of interest in the present invention, we will discuss how these features are used in the decision making step, which is step <b>108</b> in <figref idrefs="DRAWINGS">FIG. 1</figref>.

To determine if a candidate object is a mouth pattern or not, the present invention employs a decision making mechanism using the obtained 8 features k^LAB_w, k^LAB_h, s^LAB, k^RGB_w, k^RGB_h, s^RGB, v and m.

Each candidate object's features are extracted as described above, and for a particular candidate, if its feature k^LAB_w&gt;k^LAB_hor k^RGB_w&gt;k^RGB_hand at the same time k^LAB_wis larger than a threshold t_kw, a horizontal mouth scenario is assumed for that particular candidate and m is determined based on the horizontal case as described above.

The decision as to whether or not the particular candidate object is a mouth can then be made by examining if all other features are satisfied with some pre-defined thresholds in a horizontal mouth scenario. In one embodiment, the decision logic can be as follows: the candidate object is determined as a mouth pattern if m is not flagged as “Eye_Eye_Pair” and s^LABis larger than a threshold t_s, where tk_kw=0.7 and t_s=0.79.

For each candidate object, if its feature k^LAB_h&gt;k^LAB_wor k^RGB_h&gt;k^RGB_wand at the same time k^LAB_his larger than a threshold t_kh, a vertical mouth scenario is assumed and m is calculated based on the vertical case. In a similar way, the decision as to whether or not the candidate is a mouth can then be made by examining if all other features are satisfied with some pre-defined thresholds in a vertical mouth scenario. In one embodiment, the decision logic can be as follows: the candidate object is determined as a mouth pattern if v is larger than a threshold t_vand Max(W^LAB_newFeaObj,H^LAB_newFeaObj) is larger than a threshold t_hw, where t_kh=0.6, t_v=6, t_hw=6, and Max(W^LAB_newFeaObj,H^LAB_newFeaObj) represents the larger value of W^LAB_newFeaObjand H^LAB_newFeaObj.

If a red-eye candidate is determined to be a mouth pattern then it is excluded from further processing, i.e. excluded from red-eye correction. Thus, the present invention enhances red-eye correction mechanism currently used by identifying and excluding certain “false positives,” particularly mouth patterns from red-eye correction. This enhances the overall result and provides an improved corrected image to the user.

While the invention has been described in conjunction with several specific embodiments, it is evident to those skilled in the art that many further alternatives, modifications and variations will be apparent in light of the foregoing description. Thus, the invention described herein is intended to embrace all such alternatives, modifications, applications and variations as may fall within the spirit and scope of the appended claims.