Image processing system

BACKGROUND OF THE INVENTION

0001The present invention relates to an image processing system which detects a change in a predetermined region of a facial image and decides a facial expression.

0002Generally, major elements for deciding a good picture of a photographed person include facial expression, color of the face, lighting and shading of the face, and photographic angle of the face. Among these, the most dominant element is the facial expression of the photographed person. However, since the facial expression always changes, it is difficult to catch the optimal photo opportunity. Further, depending on circumstances, a person to be photographed often cannot express the desired expression, due to not only feelings of tension but also other circumstance in which not-smiling person, on a routine bases, cannot control facial muscles for the desired expression.

0003In conventional analog photography, the above-described problem has been solved by the photographer guiding the person to be photographed, and also by photographic technology via which the photographer captures the best photo opportunity. Further, when a photographed image has something wrong or also when it is impossible to capture the image again, the solution is re-touching, in which a skilled person modifies an original picture by hand painting. However, such methods of manual image modification can at most enhance a face with closed eyes or wrinkles, but it is difficult to actually change the facial expression.

0004On the contrary, in an image taken by digital means, or an image which is digitized from an analog image via a scanner, it is relatively easy compared to the analog image to perform image modification after photography. However when major image modification of a facial expression is conducted, the facial image may become quite artificial.

0005Accordingly, it is preferable that the digitally photographed facial expression is initially the best one possible, like that of the analog photography.

0006Often the favored facial expression is a smiling face. With respect to a facial expression captured by photographic means, if it is possible to determine a smiling face, it is then possible to detect the optimal photo opportunity, that is, after reading continually changing facial expression as digital images, these interpreted as smiling expressions can be selected and stored in a memory means. Yet further, among plural facial images stored in the memory means, only the images of the smiling faces can be extracted with relative ease.

0007It is well-known that the facial expression is greatly changed owing to the small changes of shape, position and angle, with respect to an eye, an eyebrow and the mouth, which happen in a time as short as 0.3 sec. Widely proposed are technologies based on the above-described well-known art, by applying digital image processing technology, changing or displacing the fundamental predetermined regions of the facial image, such as an eye, an eyebrow and the mouth, the facial expression of a smile, anger, surprise and sorrow can be formed (See for example, Patent Document 1).

0008However, in order to determine the expression of a facial image, a great number of data must first be processed with respect to the facial image of an individual person, each of course having different facial characteristics. Next, those characteristics must be selected from the data, from which the chosen expression must be determined. However, practical problems, such as excessively long processing time and the high cost of such an apparatus, still exist.

0009In portrait photography, when a smiling face, as seemed to be the best representative expression, is to be automatically captured, the above-described problems occur.

0000[Patent Document 1: JP-A 11-167626 (Pages 1-2)]

0010(JP-A means Japanese Patent Application publication.) [Document 1: Muscle Activity Patterns and Geometrical Features in Attractive Facial Expressions (Sugahara et. al), Proceedings of the 5^thAnnual Conference of JSKE 2003]

SUMMARY OF THE INVENTION

0011The present invention has been achieved based on the above-described conditions. The objective of the present invention is to provide an image processing system wherein a figure construction is performed based on the position of predetermined regions of a facial image captured via a photographic means, and the characteristics of the obtained figure is correlated with the facial expressions, whereby, the smiling face can be quickly chosen with ease.

0012The above described objective is attained by realizing the structures described below.

0000Structure 1

0013An image processing system, including: <ul id="ul0001" list-style="none"><li id="ul0001-0001" num="0000"><ul id="ul0002" list-style="none"><li id="ul0002-0001" num="0014"> a first detecting means for detecting lateral angle points of both eyes (which mean outer corners of the eyes) in a facial image; </li><li id="ul0002-0002" num="0015"> a second detecting means for detecting both oral angle points (which mean outer corners of the mouth); and </li><li id="ul0002-0003" num="0016"> a constructing means for constructing a rectangle using a first line segment drawn between both lateral angle points of the eyes detected by the first detecting means, as well as using a second line segment which passes through both oral angle points detected by the second detecting means and is parallel to the first line segment and the same length as the first line segment. <br /> Structure 2 </li></ul></li></ul>

0017The image processing system in Structure 1, further including a determining means for calculating the ratio of the long side and the short side of the rectangle obtained via the constructing means, and for determining whether the ratio is within a predetermined range.

0000Structure 3

0018The image processing system in Structure 2, further including: <ul id="ul0003" list-style="none"><li id="ul0003-0001" num="0000"><ul id="ul0004" list-style="none"><li id="ul0004-0001" num="0019"> a photographing means for photographing a facial image; </li><li id="ul0004-0002" num="0020"> a sound outputting means for outputting an audible signal; </li><li id="ul0004-0003" num="0021"> a control means for activating a shutter of the photographic means, and controlling the sound outputting means to output an audible signal, when the determining means calculates that the ratio is within the predetermined range. <br /> Structure 4 </li></ul></li></ul>

0022An image processing system, including: <ul id="ul0005" list-style="none"><li id="ul0005-0001" num="0000"><ul id="ul0006" list-style="none"><li id="ul0006-0001" num="0023"> an image display means for displaying a facial image: </li><li id="ul0006-0002" num="0024"> a memory means for storing a facial image: </li><li id="ul0006-0003" num="0025"> a first detecting means for detecting the lateral angle points of both eyes in the facial image; </li><li id="ul0006-0004" num="0026"> a second detecting means for detecting both oral angle points; </li><li id="ul0006-0005" num="0027"> a constructing means for constructing a rectangle using a first line segment drawn between both lateral angle points of the eyes detected by the first detecting means, as well as using a second line segment which passes through both oral angle points detected by the second detecting means and is parallel to the first line segment and the same length as the first line segment; and </li><li id="ul0006-0006" num="0028"> a control means for reading a facial image from plural facial images stored in the memory means and displaying on the image displaying means the facial image which has a rectangle ratio of the long side and the short side within a predetermined range. <br /> Structure 5 </li></ul></li></ul>

0029An image processing system, including: <ul id="ul0007" list-style="none"><li id="ul0007-0001" num="0000"><ul id="ul0008" list-style="none"><li id="ul0008-0001" num="0030"> an image display means for displaying a facial image: </li><li id="ul0008-0002" num="0031"> a memory means for storing a facial image: </li><li id="ul0008-0003" num="0032"> a first detecting means for detecting the lateral angle points of both eyes in the facial image; </li><li id="ul0008-0004" num="0033"> a second detecting means for detecting both oral angle points; </li><li id="ul0008-0005" num="0034"> a constructing means for constructing a rectangle using a first line segment drawn between both lateral angle points of the eyes detected by the first detecting means, as well as using a second line segment which passes through both oral angle points detected by the second detecting means and is parallel to the first line segment and the same length as the first line segment; </li><li id="ul0008-0006" num="0035"> a control means for reading a facial image from plural facial images stored in the memory means, and displaying the facial image which has a rectangle having a ratio of a long side and a short side within a predetermined range on the image displaying means, within which a specific image frame is attached.</li></ul></li></ul>

BRIEF DESCRIPTION OF THE DRAWINGS

0036<figref idref="DRAWINGS">FIG. 1</figref> is a block diagram showing a control system of an image processing system.

0037<figref idref="DRAWINGS">FIG. 2</figref> explains a rectangle constructing process.

0038<figref idref="DRAWINGS">FIG. 3</figref> is flow chart showing the steps of determination of the facial expression.

0039<figref idref="DRAWINGS">FIG. 4</figref> is a histogram showing the frequency of aspect ratios of the rectangle in the smiling face.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT

0040The present invention will now be detailed referring to the drawings.

0041<figref idref="DRAWINGS">FIG. 1</figref> is a block diagram of an image processing system based on the present invention.

0042The present image processing system includes control means C, photographic means I, image displaying means D, sound outputting means S, operation means T, image forming means P, information medium recording means B and communication means E.

0043Control means C is a computer system incorporating memory M as a memory means, operational unit U and various input-output interfaces I/F.

0044Characteristic point detecting means <b>10</b>, figure constructing means <b>20</b> and facial expression determining means <b>30</b> are stored in memory M, all of which are software, and the prescribed processes are conducted by programs which structure each of these means. Additionally, concerning software used for characteristic point detecting means <b>10</b>, an appropriate software can be selected from among the several software products used for facial image recognition.

0045Photographic means I is a digital camera, which captures a picture of a photographic subject based on control signal from control means C. Plural images can be photographed in a predetermined time, or at a predetermined frequency, or a single image in a brief moment can be photographed. The photographed image (which is an original image) is sent to memory M of control means C and stored in a prescribed format. Further in some cases, a prescribed process is conducted onto the original image, which is then stored after a compressing process is carried out.

0046Image displaying means D is a device, such as a CRT or a liquid crystal display, which displays text characters and the images, and allows selection from among them.

0047Sound outputting means S is a small speaker, which can be incorporated into photographic means I or image displaying means D.

0048Operation means T is an input device, such as a key board or a mouth, which is used for a regular computer, and allows selection from among them. Further, in some cases, a touch panel can be used, in which a transparent film having transparent electrodes is adhered onto the screen of image display means D.

0049Image forming means P is a means for outputting hard copies chosen from among the selected images via operation means T, and can be produced from a silver halide photographic printer, an ink-jet printer, an electro-photographic printer, a sublimation type thermal printer, or a fusion type thermal printer.

0050Information medium recording means B can be selected from a floppy disk drive, a CD writer, a DVD writer and an MO disk drive.

0051Communication means E is a communication interface for transferring the image, selected by the selecting operation of operation means T, to other image processing systems via communication lines. Any device having the specification conforming to the communication line connection can be used.

0052<figref idref="DRAWINGS">FIG. 2</figref> shows a rectangle (shown by dashed lines) which is constructed in such a way that characteristic point detecting means <b>10</b> detects lateral angle points of the eyes “a” and “b” and oral angle points “c” and “d”, from which figure constructing means <b>20</b> constructs the rectangle, based on points “a” “b” “c” and “d”. In this case, corner points of the eyes (which are canthus points) mean the intersecting points of an upper lid and a lower lid, and the lateral angle points of the eyes mean the outer points of each eye of the above described intersecting points. Further, the oral angle points mean the intersecting points of the upper and lower lips.

0053Firstly, figure constructing means <b>20</b> constructs line segment {overscore (ab)}. Next, figure constructing means <b>20</b> constructs a straight line parallel to line segment {overscore (ab)} through oral angle points “c” and “d”. If the straight line parallel to line segment {overscore (ab)} passes through only one point of oral angle points “c” and “d”, figure constructing means <b>20</b> constructs a straight line passing through the middle point of line segment {overscore (cd)}. Perpendicular lines are dropped from points “a” and “b” to the straight line passing through points “c” and “d” at “f” and “e”, that is, rectangle “abef” is constructed. In this case, both lengths of line segments {overscore (ab)} and {overscore (ef)} is “x”, and both lengths of line segments {overscore (af)} and {overscore (be)} is “y”.

0054<figref idref="DRAWINGS">FIG. 3</figref> is a flow chart showing the procedures described below.

0055A facial image of a subject is captured by photographing means I.

0056The lateral angle points of the eyes and the oral angle points of the facial image are detected by characteristic point detecting means <b>10</b>.

0057Rectangle “abef” of the facial image is constructed by figure constructing means <b>20</b>.

0058Facial expression determining means <b>30</b> obtains a ratio of long side “x” and short side “y” of rectangle “abef” and determines whether the ratio is within the predetermined range.

0059Memory means M memorizes the facial image when the facial image falls within the ratio of the predetermined range.

0060Photographing means “I” takes the facial image of the subject and sends the facial image as facial image data to control means C (Step <b>1</b>).

0061Characteristic point detecting means <b>10</b> detects the eyes and the mouth of the facial image from among the facial image data, and determines the position of tails of the eyes (which are the lateral angle points of the eyes) as well as the position of both ends of the lips (which are the oral angle points) (Step S<b>2</b>).

0062Figure constructing means <b>20</b> constructs the rectangle, shown by the dashed lines in <figref idref="DRAWINGS">FIG. 2</figref>, within the facial image, by the above-described procedure (Step S<b>3</b>).

0063Facial expression determining means <b>30</b> obtains lengths x and y of the long side (which is {overscore (ab)} or {overscore (ef)}) and short side (which is {overscore (af)} or {overscore (be)}) respectively, and calculates ratio R (R=x/y) of the long side and the short side based on the obtained line segment values (Step S<b>4</b>). Next, facial expression determining means <b>30</b> reads out values R<b>1</b> and R<b>2</b>, previously set in a table of memory means M, and determines whether value R, obtained in step <b>4</b>, satisfies inequalities R<b>1</b>&lt;R&lt;R<b>2</b> (Step <b>5</b>).

0064If R is in the range shown by inequalities R<b>1</b>&lt;R&lt;R<b>2</b>, facial expression determining means <b>30</b> determines as a smiling face (Step S<b>5</b>: Y), and stores the image data in a predetermined file in memory means M (Step <b>6</b>), and further, outputs the shutter sound to sound outputting means S (Step <b>7</b>). However, if R is not in the range shown by inequalities R<b>1</b>&lt;R&lt;R<b>2</b>, (Step S<b>5</b>: N), facial expression determining means <b>30</b> obtains another facial image from photographing means I (Step <b>1</b>), repeating the above-mentioned processes of Steps <b>2</b>-<b>5</b>.

0065Additionally, in the above explanation of the present flow chart, the facial image is captured by photographing means I, however, instead of using photographing means I, it is also possible to sequentially read out facial image information from the files in memory means M in which several facial images are stored.

0066Further, it is possible to employ the structure in which plural photographing means are provided, and the facial expression is determined by the facial image determining means with respect to the facial image captured by a single photographing means, and thereby the other photographing means are energized to operate the shutters. In this case, “the other photographing means” are not limited to digital cameras, but analog cameras may also be employed.

0067<figref idref="DRAWINGS">FIG. 4</figref> is a histogram, in which twenty adult men and women were photographed as they expressed blank faces and when they smiled, after that, the above-mentioned ratios were obtained from their facial images so that the frequency of R are shown as the histogram. In the case of a blank face, R=1.40±0.07 (see Table 1), while in the case of smiling face, R=1.63±0.04. Ratio R of the rectangle of the smiling face is decided by moving of the greater zygomatic muscles which pull up the oral angles.

0068Table 1 shows the range of ratio R of the facial expressions which were obtained via-the experiments by the inventor of the present invention. Since the ranges of various facial expressions are stored in the memory section, which is previously selected by the photographer, the photographic system of the present invention can detect the quick change of the facial expression of the subject, and thereby, the photographic system can automatically capture the image of the subject, when the subject naturally expresses the predetermined facial expression. <tables id="TABLE-US-00001" num="1"><table frame="none" colsep="0" rowsep="0"><tgroup align="left" colsep="0" rowsep="0" cols="2"><colspec colname="OFFSET" colwidth="49PT" align="left" /><colspec colname="1" colwidth="168PT" align="center" /><thead><row><entry /><entry namest="OFFSET" nameend="1" align="center">TABLE 1</entry></row></thead><tbody valign="top"><row><entry /><entry /></row><row><entry /><entry namest="OFFSET" nameend="1" align="center" rowsep="1" /></row><row><entry /><entry>Facial expressions</entry></row></tbody></tgroup><tgroup align="left" colsep="0" rowsep="0" cols="5"><colspec colname="OFFSET" colwidth="49PT" align="left" /><colspec colname="1" colwidth="42PT" align="center" /><colspec colname="2" colwidth="42PT" align="center" /><colspec colname="3" colwidth="42PT" align="center" /><colspec colname="4" colwidth="42PT" align="center" /><tbody valign="top"><row><entry /><entry>blank face</entry><entry>Smile</entry><entry>grin</entry><entry>laugh</entry></row><row><entry /><entry namest="OFFSET" nameend="4" align="center" rowsep="1" /></row></tbody></tgroup><tgroup align="left" colsep="0" rowsep="0" cols="5"><colspec colname="1" colwidth="49PT" align="left" /><colspec colname="2" colwidth="42PT" align="center" /><colspec colname="3" colwidth="42PT" align="center" /><colspec colname="4" colwidth="42PT" align="center" /><colspec colname="5" colwidth="42PT" align="center" /><tbody valign="top"><row><entry>Adult</entry><entry>1.40 ± 0.07</entry><entry>1.53 ± 0.03</entry><entry>1.63 ± 0.04</entry><entry>1.68 ± 0.05</entry></row><row><entry>Preschooler</entry><entry>—</entry><entry>—</entry><entry>1.70 ± 0.04</entry><entry>—</entry></row><row><entry>Baby</entry><entry>—</entry><entry>—</entry><entry>1.78 ± 0.03</entry><entry>—</entry></row><row><entry namest="1" nameend="5" align="center" rowsep="1" /></row></tbody></tgroup></table></tables>

0069From the experimental result described above, it is understood that a facial image having R in a certain range has a high possibility that the facial image is any one of a blank face, a smiling face, a grin face or a laughing face.

0070Concerning advantageous effects of this invention, according to the invention in structure 1, some characteristics of a facial expression can be converted to numerical characteristics.

0071According to structure 2 of the invention, the numerical characteristic of a facial expression can be quickly determined from the ratio of a long side and a short side of the rectangle obtained by structure 1.

0072According to structure 3 of the invention, it is possible to announce the photographed moment to the photographed subject indicating the end of a photo and prepare for the next exposure. Further when the shutter sound occurs, if the subject memorizes the tightness of the zygomatic muscles of a specific facial expression, he/she can more easily express the better face for the next exposure.

0073According to structure 4 of the invention, it is possible to select a desired facial image from among a large number of the captured facial images stored in the memory means.

0074According to structure 5 of the invention, a large number of captured facial images are displayed on the image display means, and distinguishing marks are attached to the facial images having a desired condition, and thereby, it is possible to easily select the faces of the attractive expressions from among the displayed facial images.