Face image processor and processing method therefor

[0001] [Technical field to which the invention belongs] The present invention relates to an image processing device, and particularly deals with a human face image.<u style="single">Face image processing device</u>It is about. [0002] [Conventional technology] Recently, digital image devices such as electronic still cameras have become remarkably widespread and are widely used in various fields. [0003] For example, when shooting a person with an electronic still camera, videophone, or surveillance camera, when trying to shoot the face of one or more people when the face orientation, eyes, mouth, etc. are in the desired state, the picture is taken. Take a method such as having a person adjust the condition of the face to the desired condition, or use a surveillance camera or the like to continuously shoot with a video tape or the like at all times, and then visually observe the optimum image later. The method of choosing is taken. [0004] [Problems to be Solved by the Invention] However, when shooting one or more people, it is necessary to inform the person to be photographed in advance of the desired facial condition in order to obtain the image desired by the photographer. Or, when shooting multiple people, if there is a person who is not suitable for shooting even one person, it is necessary to shoot again. Therefore, it is very difficult to shoot when you do not want the other party to know that you are shooting like surveillance, or when there are multiple people and all of them have different faces at all times. There is a problem. [0005] In view of the above problems, an object of the present invention is to provide a face image processing device capable of automatically determining the state of a face and acquiring a desired image. [0006] [Means for solving problems] The face image processing device of the present invention includes an image input means for continuously inputting images of a plurality of people including a face image, and a plurality of images input by the image input means for each image.<u style="single">The state in which the degree of similarity between the image of the dictionary according to the state of multiple eyes created in advance and the obtained image of the pupil is the highest is set as the current state of the eye, and this is used.</u>It is evaluated whether the facial expression of each of the plurality of persons is the facial expression desired by the photographer, and based on the evaluation result, the facial expression of each person of the plurality of persons is evaluated in the plurality of images. It is provided with an image selection means for selecting and outputting one image having the highest evaluation value for each image obtained by multiplying the value by a coefficient. [0007]<u style="single">According to the present invention, the above configuration</u>By detecting a person's face image from a continuous face image from a camera or the like, the state of the face image intended by the photographer, for example, the face image in which the subject's eyes are open is determined and selected. is there. As a result, the photographer does not have to worry about whether or not the subject's eyes are closed as in the past, but only points the camera or the like at the subject for a certain period of time and does not close his eyes. It is possible to provide a face image processing device that automatically selects a face image. [0009]<u style="single">Further, according to the present invention, the above-described configuration</u>By evaluating each of the facial images of multiple people and automatically selecting the image with the best overall from the accumulated images, for example, the optimum one for so-called group photos is automatically selected. To provide a face image processing device<u style="single">be able to</u>.. [0023] BEST MODE FOR CARRYING OUT THE INVENTION Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings. [0024] First, a device that recognizes the state of one or more human faces contained in a continuous image input from a TV camera or an electronic still camera using this method, and captures the face in the state desired by the photographer. An embodiment is shown. [0025] (1) Processing description of the overall processing outline of the embodiment FIG. 1 is a configuration diagram showing an example of a system according to an embodiment of the present invention. In FIG. 1, the present embodiment is a calculation and storage device similar to a PC inside a portable housing such as a television camera and monitor 1, devices 2 and 3 consisting of a PC (or workstation), or an electronic still camera. Etc., and consists of a device 4 equipped with a small display such as a liquid crystal or plasma. [0026] FIG. 2 is a block diagram according to the processing of the system according to the embodiment of the present invention. In FIG. 2, the system according to the present invention includes an image input unit 11, an image storage unit 12, a face region extraction unit 13, a pupil detection unit 14, a nasal hole detection unit 15, a mouth detection unit 16, and a pupil state. Judgment unit 17, mouth condition determination unit 18, face condition determination unit 19, attribute-specific counting unit 20, optimum image capture unit 21, optimum image composition unit 22, face size correction unit 23, and output unit 24. And have. [0027] In such a system, the image processing of the present invention is performed by the following procedure. That is, the digitized image is input from the image input unit 11, and the contents are continuously stored in the image storage unit 12. By applying the face area extraction unit 13 to the input image, the faces of one or more persons existing in the input image are extracted, and in each extracted face area, the pupil detection unit 14, the nose hole detection unit 15, and the mouth are extracted. The detection unit 16 is used to detect the parts of the eyes, nose and mouth in the face. When each part of the face is detected, the pupil condition determination unit 17 and the mouth condition determination unit 18 obtain the open / closed state of the pupil, the state of the line of sight, the open / closed state of the mouth, etc., and the face condition determination unit 19 uses the results to cover the face. Determine what the condition of each photographer's face is. [0028] The attribute-based counting unit 20 finds the gender, adult / child, and other attributes of each person in the shooting area, and measures each attribute and the total number of people in the shooting area. The optimum image capturing unit 21 determines for each image whether or not the obtained image is in the state desired by the photographer, and outputs the image closest to the optimum state among the plurality of images obtained. Then, when a plurality of people are photographed, the optimum image compositing unit 22 saves the optimum image for each photographed person and synthesizes the final output image. [0029] The obtained result or candidate image is displayed by the output unit 24 while correcting the size by the input image size or the face size correction unit 23, and informs the photographer of the result. [0030] Next, the operation thereof will be described in detail with reference to the respective processing units 11 to 23. [0031] (2) Image input unit 11<u style="single">Processing description</u> An image is digitized and input in color or monochrome using a TV camera for moving image input, an electronic still camera for inputting still images, or the like, which is installed so that one or more people can be captured. The gradation and size of the input image are not particularly limited and follow the input gradation and input resolution of the camera. [0032] (3) Image storage unit 12<u style="single">Processing description</u> The image captured from the image input unit 11 is saved in the memory as it is, and the plurality of images immediately before (up to N frames) are saved in another area. [0033] (4) Face area extraction unit 13<u style="single">Processing description</u> Of the human face areas, the upper and lower ends are located near the eyebrows to the lips, and the left and right ends are located outside both ends of both eyes as the face search area. Create a face dictionary for face search by using the upper component specific vector. In addition, various images are evaluated in advance with a face search dictionary, and if a non-face dictionary is obtained in an area having a high degree of similarity to the face dictionary, the images are collected as a non-face dictionary. In order to eliminate the influence of the size of the face on the input image, create enlarged / reduced images in multiple stages, and use the compound similarity method or template matching method for each image to create a face area. Perform a search. The scanning procedure is shown in the explanatory diagram of FIG. Ideally, the face area should have a high degree of similarity to the face dictionary and a low degree of similarity to the non-face dictionary. Evaluation value = Similarity with face dictionary-Similarity with non-face dictionary The place with the highest evaluation value given in is found and used as the first face detection area. By setting the face detection area for the area that gives an evaluation value equal to or higher than the predetermined evaluation threshold value at a position that does not overlap with the area where the highest value is obtained and is separated by a predetermined distance or more, a plurality of people can input the image. Even if it is included, it is possible to detect everyone and measure the number of people in the area to be photographed. [0034] (5) Eye detector 14<u style="single">Processing description</u> For each face area extracted by the face area extraction unit 13, a circular separation filter with multiple radii (Face recognition system using moving images, Osamu Yamaguchi et al., Shingaku Giho PRMU97-50, PP17- (See 23) is applied to enumerate the circular and darker areas as pupil candidate points. Since the pupil region is assumed to be in the upper region of the face, the search region does not need to be processed for the entire face. In addition, it is possible to increase the speed by calculating the circular separation degree that calculates the ratio of the luminance dispersion in each of the outer region and the inner region shown in Fig. 4 only in the binarized and determined dark areas. is there. Next, for each of the obtained candidate points, the combination of candidate points (one set on the left and right) is narrowed down using the geometric arrangement conditions according to the application. For example, the magnitude threshold of the distance between both pupils is determined by the distance from the camera. Alternatively, when there is only a face in a stationary state in front, the threshold value of the angle is determined so that the line connecting both pupils is close to horizontal. The following evaluation values are calculated for each of the eyes, and the sum of the left and right evaluation values is used as the evaluation value for the combination. [0035] Evaluation value = Similarity with pupil dictionary-Similarity with non-pupil dictionary In addition, each dictionary shall be prepared in advance from the data of a plurality of subjects in the same manner as the face area extraction unit, and in this case, the pupil dictionary is eye-catching, blinding, sideways, half-eyed. It has various pupil states such as, etc. as separate multiple dictionaries, and can stably detect the pupil region even in various states such as blinding and sideways eyes. In addition, the non-pupil dictionary also has multiple dictionaries by dividing the classes such as nostrils, outer corners of the eyes, and eyebrows, which are easily mistaken for the pupil, and when calculating the similarity of the non-pupil dictionary, select the one that gives the highest similarity. Deal with various extraction failures by calculating. This is shown in Fig. 6. [0036] In addition, the accuracy of pupil detection can be improved by defining geometrical constraint conditions as shown in FIG. 5 in combination with the nostril detection unit 15. [0037] (6) Processing explanation of nostril detection unit 15 The nose region is limited by using the positional relationship between the face detection unit 13 and the pupil detection unit 14. In the central part of the face area, below both pupils, the dark and round areas are listed as nostril candidate points by binarization and circular separation filter processing in the same way as the pupil detection part 14, and for each Similar to the face detection unit, the similarity is calculated with the nostril dictionary and the non-nostril dictionary, and the following evaluation values are obtained at each point. [0038] Evaluation value = similarity with nostril dictionary-similarity with non-nostril dictionary In addition, among the combination of all the two candidate points, the set of points (two points on the left and right) that have the highest evaluation value while matching the geometrical arrangement conditions with the pupil given in advance is selected. Find it and detect it as the position of both nostrils. Also, as shown in the pupil detection unit 14, it is possible to improve the accuracy by performing four points of the pupil and the nostril under the geometric arrangement condition. [0039] (7) Processing explanation of mouth detection unit 16 Since the arrangement of the face and eyes and nostrils was obtained by the face region extraction unit 13, the pupil detection unit 14, and the nostril detection unit 15, the centers of both pupils and the centers of both nostrils were obtained, and the mouth was provided using the average geometric arrangement. Do the calculations that you think would be. FIG. 5 is an explanatory diagram for explaining the positional relationship between the pupil, the nostril, and the mouth in the pupil detection unit and the nostril detection unit of the present invention, and refer to FIG. [0040] Further, an explanatory diagram of the process of the mouth detection unit 16 is shown in FIG. 7, which is an explanatory diagram illustrating the detection process of the mouth detection unit 16 in the present invention. In FIG. 7, a binarization process is performed in which pixels having a brightness equal to or lower than a predetermined threshold value such that only the darkest pixels appear in the region are black pixels, and the other pixels are white pixels, and this image is used as a reference. It is an image. Since the area extracted even at this threshold is a dark part or a black part, it is defined as a whiskers area or an open mouth area. From there, the threshold value is gradually raised to binarize, labeling is performed on the difference image from the reference image, and when a horizontally long area (label) appears and becomes large, that area is specified vertically and horizontally. When it reaches the size or larger, it is used as the mouth area. On the other hand, the reason why the size is almost the same as the binarization result of the initial threshold value is that the black area such as a whiskers can be excluded by the difference processing and can be distinguished from the mouth area. [0041] (8) Processing explanation of pupil state determination unit 17 Obtained by creating a dictionary for each of the left and right pupil areas obtained by the pupil detection unit 14 according to various eye states such as "eyes closed", "half eyes", "side eyes", and "upper eyes". The state in which the degree of similarity with the pupil image is highest is determined as the current state of the pupil. [0042] Further, as described in the face condition determination unit 19 described later, when the photographer has previously selected which state is desired, the optimum image is selected by the following method. [0043] FIG. 9 is a flowchart showing a determination process of the pupil state determination unit. By this processing, the optimum image can be selected even when the state of the pupil changes sequentially such as blinking or movement of the line of sight, or even for a subject whose eyes are narrow and it is difficult to determine the opening / closing of the pupil. [0044] The evaluation value is the difference between the similarity with the dictionary indicating the desired state and the highest similarity among other dictionaries. If this value is high, it can be judged that the state is close to the ideal state and can be clearly distinguished from other states. When this evaluation value is judged from a single image, it is not possible to distinguish between the open state of a person with narrow eyes and the half-eye state of a person with large eyes, so it takes more time than the time from the start to the end of blinking. Images are continuously accumulated for a sufficient number of N to take a picture, and the variance and average value of the evaluation values are calculated. [0045] In FIG. 9, when the variance of the evaluation value is small (S31), there is almost no change in the state of the eyes, and when the time is higher than the average value for a long time (S32), the evaluation value is higher than the average value. The state that gives the evaluation value closest to the average is the optimum image (S35), and when the time lower than the average value is long, the state that gives the evaluation value closest to the average among the evaluation values lower than the average is the optimum image. Select as (S33). On the contrary, when the variance is large, it is considered that the state of the eyes fluctuates greatly, and the one giving the highest evaluation value is regarded as the optimum image (S34). [0046] FIG. 10 is an explanatory diagram for explaining the determination process of the pupil state determination unit in the present invention. Taking this as an example, (a) and (b) have less movement, less dispersion, and a longer time than the average. Therefore, the image that is higher than the average and gives the evaluation value closest to the average value is selected. In (c), the variation is large and the variance is large, so select the image that gives the highest value. In (d), since the variance is small and the time lower than the average is long, the image closest to the average value is selected among the evaluation values lower than the average. [0047] (9) Processing explanation of mouth condition determination unit 18 Next, FIG. 11 shows a flowchart of processing of the mouth state determination unit 16. [0048] In FIG. 11, it is determined whether the mouth is open or closed by comparing the vertical and horizontal widths of the mouth, the vertical and horizontal widths, and the threshold values set for each. If the vertical width of the mouth is equal to or greater than the predetermined threshold (S41), it is determined that the mouth is open (S44), and if the width is equal to or less than the predetermined threshold and the width is equal to or greater than the predetermined threshold (S42). Determines that is closed (S45). Furthermore, if it does not belong to either of them, a dictionary of multiple states (ordinary mouth, pointed mouth, clenching) in an image in which the vertical and horizontal widths of the mouth and the vertical and horizontal widths are normalized so as to have a constant size. , Create a dictionary according to each of Akanbe, etc.) (S43) Judge the state of the mouth (S46, S47). [0049] (10) Processing explanation of face condition determination unit 19 Using the outputs of the pupil state determination unit 17 and the mouth state determination unit 18, it is determined whether or not the face condition is desired by the photographer. The desired state is, for example, the state in the case of an ID photo or the like is "a state in which the eyes are open facing the front and the mouth is in a closed state", and in a snapshot or the like, "a state in which the eyes are open". The state of the mouth does not matter. "" The state where the eyes are open and the mouth is laughing. " [0050] For the actual state judgment, a matrix with each of the pupil and mouth states as shown in Fig. 12 on the vertical and horizontal axes is prepared, and whether or not it is the desired state is entered in each cell. .. [0051] (11) Processing explanation of counting unit 20 by attribute Each face area extracted by the face area extraction unit 13 has a dictionary consisting of average faces of men and women, a dictionary consisting of average faces of adults and children, and an average face image dictionary for each nationality, etc., and calculates similarity. The number of people is measured for each attribute depending on which one is closer, and the attributes are labeled for the face area based on the obtained results. In addition, the number of people can be measured by totaling the total number of people existing in the non-shooting area regardless of the attribute. [0052] (12) Processing explanation of the optimum image capturing unit 21 In the time-series continuous images accumulated within a predetermined time, using a matrix as shown by the face condition determination unit 19, whether or not the image is in the state desired by the photographer can be determined one by one and one by one. The evaluation value is calculated by counting and integrating each person and each part. The formula is as follows. [0053] Evaluation value = (Similarity with desired dictionary-Highest similarity among non-desired dictionaries) Here, the "face" indicates the entire face included in the photographing area, and the "site" indicates the eyes and mouth in each face area. The image having the highest evaluation value among the plurality of obtained images is selected as the optimum image. [0054] (13) Processing explanation of the optimum image compositing unit 22 If you are shooting for multiple people and want to take a picture of everyone in the shooting area with their eyes open and laughing (open mouth), etc., you want to shoot in the desired state. Among the images accumulated by repeating the processing up to the face condition determination unit 19 for a predetermined time, the optimum image for each photographed person is saved in the face area and the peripheral image of the face in a predetermined range, and the final output image is optimized. By applying the images and synthesizing them, the photographed person creates the optimum image without the need for adjustment of the shooting timing and surroundings. When compositing, it is assumed that the subject does not move as much as possible, but if it does move, it is unnaturally composited by applying antialiasing processing along the periphery of the storage area that is larger than the face area. Process so that it is no longer an image. [0055] (14) Processing explanation of face size correction unit 23 The image input when outputting to the output unit 24 can be output as it is, but the size of the output image is enlarged or reduced according to the size of the extracted face area of one or more people. The size of the face can be obtained by using the size of the multi-resolution face dictionary used in the face area extraction unit 13, but since the resolution of the size is required, another method is used here. [0056] Using only the brightness distribution in the area extracted as the face area, binarization is performed by the P-Tile method such that the white pixel black pixel ratio is constant, or by a method such as a constant threshold value or discriminant analysis method. Then, the area including the peripheral area of the face is binarized at the threshold value when the face area is binarized. By labeling the binarized image, a connected area including the center of the face is extracted, and the left and right ends of the area are set as the left and right ends of the face, and the width value is used as the face size. However, since there are cases where the ears are protruding and cases where the ears are hidden by the hair, the classification is performed using the pupil positions obtained by the pupil detection unit 14 and the positions of the left and right edges of the face. [0057] An explanatory diagram of the process is shown in FIG. 13, and the description will be given by taking the left side as an example with respect to the center D of both pupils. The left edge of the face is in position A when the ears are exposed, and the threshold value is set in advance so that the AD length / CD length is equal to or greater than the predetermined threshold value. If the ears are hidden by hair, the leftmost position will be the position B (BD length / CD length), so the value will be smaller than when the ears are out, so the ears will come out here. Judge whether or not it is. Similarly, it is determined whether or not the ear on the opposite side has an ear. [0058] [0058] If there are no ears, the position extracted as the left and right edges is regarded as the face area, and if there are ears, the average value of (AD) / (BD) calculated in advance from the data of multiple people is used. Use to calculate the position of B without affecting the ear position . If the photographer has input a desired size based on the face size obtained as described above, the image is output in the desired size by performing enlargement / reduction processing. [0059] (15) Processing description of output unit 24 Finally, the processing of the output unit 24 will be described below. [0060] In the case of a stationary device with a TV camera, the optimum image and the optimum candidate image are arranged side by side on a monitor, and in the case of a portable type, the built-in monitor is output. As shown in FIG. 14, the image judged to be the optimum image is output in a large size, and the ones having high evaluation values are arranged next to them along the time sequence. If the desired image is in the candidate column, the final output image can be changed by allowing the desired image to be selected with the up, down, left, and right buttons, as in the rectangular area H surrounded by the dotted square in Fig. 14. It is also possible to mark the face area of each image and manually synthesize the optimum face from a plurality of images. [0061] [Effect of the invention] As described in detail above, according to the present invention, when shooting the faces of one or more people by shooting with an electronic still camera, a videophone, a surveillance camera, etc., the other party is informed of the desired shooting state and shooting. Without notifying, it is possible to judge whether the face is facing the front, the open / closed state of the eyes, the open / closed state of the mouth, etc. without being affected by the fineness of the eyes and movement, which is suitable for the state required for shooting. While checking the condition of the face, you can automatically select the most suitable one and shoot. [0062] Further, when a plurality of people such as a group photograph are photographed, the optimum images of all the photographed persons can be easily obtained by automatically synthesizing the images in the optimum state of each photographed person. [Simple explanation of drawings] FIG. 1 is a configuration diagram showing an example of a system according to an embodiment of the present invention. FIG. 2 is a block diagram according to the processing of the system according to the embodiment of the present invention. FIG. 3 is an explanatory diagram illustrating the processing of the face region extraction unit of the present invention. FIG. 4 is an explanatory diagram illustrating processing of a circular separation filter of the pupil detection unit of the present invention. FIG. 5 is an explanatory diagram illustrating the positional relationship between the pupil, the nostril, and the mouth in the pupil detection unit and the nostril detection unit of the present invention. FIG. 6 is an explanatory diagram illustrating a detection process of a pupil detection unit in the present invention. FIG. 7 is an explanatory diagram illustrating a detection process of a mouth detection unit in the present invention. FIG. 8 is an explanatory diagram illustrating a determination process of the pupil state determination unit in the present invention. FIG. 9 is a flowchart showing a determination process of the pupil state determination unit in the present invention. FIG. 10 is an explanatory diagram illustrating a determination process of the pupil state determination unit in the present invention. FIG. 11 is a flowchart illustrating a determination process of the mouth state determination unit in the present invention. FIG. 12 is an explanatory diagram illustrating a determination process of the face state determination unit in the present invention. FIG. 13 is an explanatory diagram illustrating a size correction process of the face size correction unit in the present invention. FIG. 14 is a diagram showing a captured image selection screen and an interface according to the present invention. [Explanation of symbols] 1 ... camera 2 ... display 3 ... personal computer or workstation 4 ... Digital camera including PC-equivalent calculation / storage device and internal display device 11 ... Image input section 12 ... Image storage 13 ... Face area extractor 14 ... Eye detector 15 ... Nostril detector 16 ... Mouth detector 17 ... Eye condition detector 18 ... Mouth condition detector 19 ... Face condition judgment unit 20 ... Counting section by attribute 21 ... Optimal imaging section 22 ... Optimal image compositing section 23 ... Face size compensation 24 ... Output