Face detection device and method thereof

The present invention relates to a face detection device and a method for detecting a person's face by using the skin color information of the person photographed by a camera, for example, various image communication such as video conference and video telephone, database search, and the like. It can be used for security systems such as building surveillance, face verification systems, games, education, production of movies and TV programs, human interfaces, and real-time facial expression tracking devices that link faces generated by computer graphics.

For face detection methods that do not use color information, feature based approach by detecting face parts such as eyes and mouth, and pattern recognition such as neural networks and support vector machines (SVM). Various methods have been widely proposed, such as an image-based approach and a method combining both of them.

For example, a method of extracting candidate points for eyes and mouth, determining face candidates based on the combination of the candidate points, comparing the face candidates with a standard face pattern, and performing face / non-face determination, that is, face detection is performed. Yes (see Patent Document 1).

There is also a method of extracting candidate points between the eyes and detecting the face portion (see Patent Document 2).

Further, in the case of a moving image, an edge image is extracted from the time difference between frames, and in the case of a still image, an edge image is extracted from the still image, a face candidate is determined from this edge image, and the face candidate is determined. There is also a method of performing face / non-face determination, that is, face detection using a learning algorithm (see Patent Document 3).

There is also a method of determining a face candidate by matching with a template representing an average face image and performing face / non-face determination, that is, face detection, using a learning algorithm for this face candidate (see Patent Document 4).

And, what can be said in common with the methods described in Patent Documents 1 to 4 above is that since color information such as skin color is not used, there is an advantage that it is not easily affected by the color environment due to lighting conditions, camera conditions, and the like. On the other hand, there is a demerit that it takes time to process because it is basically necessary to scan the entire screen.

On the other hand, since it is known that human skin color exists in a specific region in the color region, a method of performing face detection by using skin color information at the time of face detection has been widely proposed. For example, as follows, a method for identifying a human skin color region in a color space formed by various pigment expression systems such as HSV, RGB, and YUV has been proposed.

That is, a skin color area in RGB space that matches the lighting conditions and camera conditions is manually created in advance, and the skin color is extracted by this to determine the face candidate part, and the eyebrows are detected in this face candidate part. There is a method to realize face detection by doing so (see Non-Patent Document 1).

In addition, color information is converted from RGB space to HI (V) space, skin color is extracted by predetermined thresholds of H and I, and a genetic algorithm is applied to the skin color part to determine face candidates. There is also a method of applying a neural network to a face candidate to perform face / non-face determination, that is, face detection (see Patent Document 5).

Furthermore, a skin color model is created in advance by the Gaussian function, the skin color is extracted by the skin color model to determine the face candidate part, and the distribution of dark parts (parts corresponding to eyes and mouth) in the face candidate part. There is also a method of performing face / non-face determination, that is, face detection by examining (see Patent Document 6).

And, what can be said in common with the methods described in Non-Patent Document 1 and Patent Documents 5 and 6 is that since skin color information is used, efficient face search can be performed and processing speed is high. While it has the advantage of being able to be used, it has the disadvantage of being easily affected by the color environment such as lighting conditions and camera conditions, or requiring manual setting of the skin color area. These are contrary to the characteristics of the methods described in Patent Documents 1 to 4 described above.

That is, although it is expected that the processing speed will be increased by detecting the face using the skin color information, since the skin color is extracted using the fixed area on the color space, the color is always determined by the lighting conditions and the camera conditions. Due to the influence of the environment, there remain problems such as the inability to extract human skin color, or the extraction of objects other than human skin that have a color close to skin color as skin color.

Therefore, in order to solve the problem in the skin color extraction process due to the difference in the color environment conditions, we try to adapt to the difference in the color environment by using the skin color information of the face in the camera installation environment as the skin color standard. A method has been proposed.

For example, a skin color portion having a specific area is extracted from the image using the skin color reference value, and the face is detected by determining whether or not the skin color portion is a moving portion and performing face determination, and the detection is performed. There is a method of updating the skin color information of the face as a skin color reference value and performing the following face detection processing (see Patent Document 7). Here, when the face is not detected, the peak value in the skin color effective region in the color histogram in the image is updated as the skin color reference value.

Further, first, the skin color of a person in the image is specified by an input means such as a mouse under the color environment, and the skin color extraction parameter is determined based on this skin color information. Then, there is also a method of performing face detection processing by extracting the skin color portion using this skin color extraction parameter (see Patent Document 8).

<patcit num="1"><text>Japanese Unexamined Patent Publication No. 7-311833</text></patcit><patcit num="2"><text>Japanese Unexamined Patent Publication No. 2001-216515</text></patcit><patcit num="3"><text>Japanese Patent Application Laid-Open No. 2003-044853</text></patcit><patcit num="4"><text>Japanese Unexamined Patent Publication No. 2003-271933</text></patcit><patcit num="5"><text>Japanese Unexamined Patent Publication No. 2000-048184</text></patcit><patcit num="6"><text>Japanese Unexamined Patent Publication No. 11-015979</text></patcit><patcit num="7"><text>Japanese Unexamined Patent Publication No. 2000-105819</text></patcit><patcit num="8"><text>Japanese Unexamined Patent Publication No. 2003-108980</text></patcit><nplcit num="1"><text>Shinjiro Kawato, Shinji Tetsuya, "Real-time detection of the eyebrows using a ring frequency filter", IEICE Transactions D-II, December 2001, Vol.J84-D-II, No.12, p. 2577-2584</text></nplcit>

<p> However, the method described in Patent Document 7 described above has mainly the following three problems. That is, the first problem is that the human skin color may not be extracted correctly depending on the image conditions. The reason is that the skin color standard is retained / updated using the detected skin color information of the face, and after that, the skin color is extracted using this skin color standard, but the face part is not detected or the skin color standard is retained. If not, a color histogram of pixels having color information in the skin color effective region on the current image is created, and the peak value is set as the skin color reference. For this reason, for example, if the image contains many objects that are not human skin color but have a color included in the skin color effective region, the peak value in the color histogram is used as the skin color standard. Therefore, the skin color standard created by this is used. This is because the value may not be able to correctly extract the human skin color.</p><p> The second problem is that there is no merit in setting the skin color reference value except when the face is always present in the image. The reason is that if there is no skin color area larger than the specified size in the image, the current skin color reference value is discarded and the skin color reference value is set again using the peak value in the color histogram in the skin color effective area. This is because it ends up.</p><p> The third problem is that the skin color reference value is determined only by the detected skin color information of one person's face. The reason is that the skin color varies from person to person, and if the skin color standard value is determined based on the skin color information of one person's face, some people may not be able to correctly extract the skin color with that skin color standard value. is there.</p><p> Further, the method described in Patent Document 8 described above has mainly the following three problems. That is, the first problem is that the skin color reference value under the color environment cannot be automatically acquired. The reason is that when setting the skin color reference value, a method of manually instructing the skin color position in the image or projecting the skin color at a specified position is adopted.</p><p> The second problem is that skin color extraction cannot be performed if the lighting conditions change over time. The reason is that there is no means for setting / updating the skin color reference value other than the manual method.</p><p> The third problem is that the skin color parameter is determined only by the skin color information of one person who first sampled the skin color. The reason is that the skin color varies from person to person, and if the skin color parameter is determined based on the skin color information of one person's face, some people may not be able to correctly extract the skin color with the skin color parameter.</p><p> An object of the present invention is that a skin color model adapted to the color environment can be automatically constructed under various lighting conditions and / or various camera conditions, and by using the skin color model, a high-speed and color environment can be constructed. It is an object of the present invention to provide a face detection device and a method thereof, which can perform face detection regardless of the above, and can automatically make a skin color model follow changes in lighting conditions and / or camera conditions over time.</p>

<p> The present invention is a face detection device that detects the face of a person by using the skin color information of the person taken by the camera, and when performing a process of narrowing down a face candidate portion from the current frame image taken by the camera. In order to narrow down the face candidate part from the current frame image by using the skin color model storage means for storing the skin color model used in the above and the skin color model stored in this skin color model storage means, the skin color part is selected from the current frame image. A skin color portion extraction processing means that performs extraction processing, a face detection processing means that performs processing for detecting a person's face from the skin color portion extracted as a face candidate portion by this skin color portion extraction processing means, and this face detection processing. Using the detected person face skin color information extraction processing means for extracting the skin color information of the person's face detected by the means and the skin color information of the person's face extracted by the detected person face skin color information extracting processing means, It is equipped with a skin color model update processing means that automatically updates the skin color model stored in the skin color model storage means to a skin color model suitable for the current lighting conditions and / or the current camera conditions of the camera installation location. , The skin color model storage means is a color that includes various lighting conditions and / or various camera conditions when performing the process of narrowing down the face candidate part from the first frame image taken by the camera immediately after the device starts operation. The feature is that the skin color region not adapted to the environment is stored as the default skin color model, and the skin color region narrower than the skin color region not adapted to the color environment is stored as the updated skin color model after the update processing by the skin color model update processing means. Is to be.</p><p> Here, the "skin color region narrower than the skin color region not adapted to the color environment" is the skin color region to which the color environment has been applied at that time (the skin color region in the optimum state for the lighting conditions and / or the camera conditions at that time), or the color. It is a skin color region in an intermediate state between the skin color region not adapted to the environment and the skin color region to which the color environment is applied.</p><p> The "lighting condition" is, for example, the sun or shade, the daytime or evening or nighttime, the fluorescent lamp or the incandescent bulb, and the "camera condition" is the camera type, color tone adjustment, and brightness. It is a condition such as setting.</p><p> In such a face detection device of the present invention, immediately after the start of operation of the device, a skin color region not adapted to the color environment including various lighting conditions and / or various camera conditions is initially set as a skin color model, and this skin color model is used. When a person's face is detected by the face detection processing means, the skin color information of the detected person's face extracted by the detected person's face skin color information extraction processing means is obtained. By updating the skin color model by the skin color model updating processing means using the skin color model, a skin color region narrower than the skin color region not adapted to the color environment is set as the updated skin color model. Then, every time a person's face is detected by the face detection processing means, the skin color model update processing means performs the update processing using the skin color information of the face, and the skin color model updated in this way is used. The skin color extraction process and the face detection process are repeated.</p><p> Therefore, it is possible to automatically build a skin color model adapted to the color environment under various lighting conditions and / or various camera conditions, and the lighting conditions and / or camera conditions change over time. It is possible to automatically follow the skin color model. Therefore, it is not necessary to manually adjust the color tone and the like, and it is possible to perform face detection at high speed and regardless of the color environment, thereby achieving the above object.</p><p> Further, the above-mentioned face detection device is provided with a person-specific face skin color information storage means for individually storing the skin color information of the person's face extracted by the detected person face skin color information extraction processing means for each person, and the detected person face skin color information. When the person whose face is detected is a person who has already been detected up to the previous frame, the extraction processing means has the skin color of the face of the person stored in the person-specific face skin color information storage means. If the information is updated and the person is a newly detected person who has not been detected by the previous frame, the process is configured to newly write the skin color information of the person's face to the person-specific face skin color information storage means. It is desirable that the skin color model update processing means is configured to perform processing for updating the skin color model using the skin color information of the face individually stored for each person in the face skin color information storage means for each person.</p><p> Here, when updating the face skin color information for the detected person, the face skin color information obtained from the current (latest) frame image may be used as the updated information, but from the viewpoint of improving the face detection accuracy. From, if there is facial skin color information for the same person obtained from multiple frame images including the current (latest) frame image, the average of the facial skin color information obtained from all those frame images. It is preferable that the value (however, processing such as excluding the protruding value) is used as the updated information. Whether or not the faces of the same person are present depends on the positional relationship of each detected face on the screen (whether or not the amount of change in the position of the face between each frame image is within a range that can be regarded as the movement of the face of the same person, or Whether or not the time change of the face position between each frame image can be regarded as the trajectory of the movement of the face of the same person), or the identity / similarity of the skin color information and facial feature information of each face can be judged. ..</p><p> When the skin color model is updated as needed using the skin color information that is individually updated for each person in this way, a skin color model that includes individual skin color variations is constructed, and skin color extraction and face detection processing are performed. Can be done.</p><p> Further, in the above-mentioned face detection device, the skin color model update processing means stores the skin color model in the skin color model storage means when the cumulative number of persons whose faces are detected by the face detection processing means is less than the specified number. It is desirable that the process is gradually narrowed down from the state initially set in the skin color region to which the color environment is not adapted.</p><p> When the cumulative number of people whose faces are detected is less than the specified number, the skin color model is used as a skin color model storage means when the skin color model is gradually narrowed down from the state of the skin color region not adapted to the color environment. In the stored skin color model, when the cumulative number of people whose faces are detected by the face detection processing means reaches the specified number, the skin color region to which the color environment is applied at that time (lighting conditions at that time and / or the camera) It will be set to the skin color area in the optimum state), and until then it will be set to the skin color area in the intermediate state between the skin color area not adapted to the color environment and the skin color area to which the color environment has been applied. Become. For this reason, when the cumulative number of people whose faces are detected is small, the inconvenience that occurs when narrowing down to the skin color area to which the complete color environment is applied at once, that is, the skin color information of the person whose face is detected is the standard skin color of the person. The inconvenience that the skin color model is not set in the appropriate area when it is out of the information is solved.</p><p> Then, in the above-mentioned face detection device, the skin color model update processing means stores the skin color model in the skin color model storage means when the cumulative number of persons whose faces are detected by the face detection processing means is less than the specified number. Is gradually narrowed down from the initial setting in the skin color area not adapted to the color environment, and the center of gravity is moved. After the specified number is reached, the center of gravity is moved without narrowing down. It is desirable to have.</p><p> When the cumulative number of people whose faces are detected reaches the specified number in this way, the process shifts from the process of narrowing down and moving the center of gravity to the process of moving the center of gravity without narrowing down. In the skin color model stored in the skin color model storage means, when the cumulative number of persons whose faces are detected by the face detection processing means reaches a specified number, the skin color area to which the color environment has been applied at that time (their). It is set to the skin color region in the optimum state for the current lighting conditions and / or camera conditions), and after that, the process of moving the center of gravity of the skin color model but not narrowing down is repeated. For this reason, while avoiding the inconvenience that occurs when narrowing down to the skin color area to which the complete color environment has been applied at once, processing that can respond to temporal changes in lighting conditions and camera conditions before and after the cumulative number of people reaches the specified number. Is realized.</p><p> Further, in the above-mentioned face detection device, the skin color model update processing means uses the skin color information of the face of the person detected within the latest specified time or the skin color information of the face of the person within the specified number of people detected most recently. , It is desirable that the configuration is such that the skin color model is updated.</p><p> In this way, when the skin color model is updated using the skin color information of the faces of people within the latest specified time or the specified number of people, the lighting conditions and / or the camera conditions gradually change over time. Can be automatically handled.</p><p> Further, in the above-mentioned face detection device, the brightness and brightness over the entire screen are compared with the current frame image and the reference screen for detecting the change in lighting / camera conditions set based on at least one frame image taken before that. When / or a change in color information is detected and a change in brightness and / or color information over the entire screen is detected, the skin color model stored in the skin color model storage means is initially set in the skin color area not adapted to the color environment. It is desirable to have a lighting / camera condition change detection processing means that performs processing to return to the desired state.</p><p> When the lighting / camera condition change detection processing means is provided in this way, it is possible to detect a sudden change in the lighting condition and / or the camera condition, and when a sudden change is detected, the current skin color. Illumination conditions and / or lighting conditions and / or by reconstructing the skin color model in a wide color environment unadapted skin color region containing various lighting conditions and / or various camera conditions by discarding the model and restarting the skin color extraction and face detection processing. / Or processing that can automatically respond to sudden changes in camera conditions is realized. Not only can it automatically respond to sudden changes in lighting conditions and / or camera conditions, but the lighting conditions and / or camera conditions change slowly, and the skin color model is not detected during that time. It is also possible to avoid the situation where the skin color model is in an inappropriate state as a result of not being updated.</p><p> Then, in the above-mentioned face detection device, the reference screen for detecting changes in lighting / camera conditions can be the latest frame image among the frame images in which the face of a person is detected by the face detection processing means.</p><p> Further, the reference screen for detecting changes in lighting / camera conditions may be an image in which a moving object is not shown in the screen, that is, a background image. For this background image, for example, a plurality of frame images taken by a camera are analyzed for each pixel, and the background image is in a stable state (a state in which no change in brightness and color information is observed) among the plurality of frame images. When a pixel is detected, it can be formed by processing such that the state is used as a background image for the pixel portion. Such a background image forming process may be realized by a program different from the program that performs the skin color extraction and face detection processes.</p><p> Further, when the above-mentioned face detection device cannot detect a person's face continuously for a certain period of time or longer, or even though a moving object appears on the screen, the face is continuously or accumulated for a certain period of time or more a certain number of times. As described above, when the face of a person cannot be detected, the skin color model defect detection processing means for returning the skin color model stored in the skin color model storage means to the default state in the skin color area not adapted to the color environment is provided. It is desirable to have.</p><p> Here, in "when the face of a person cannot be detected continuously for a certain period of time" and "when a moving object appears on the screen but the face of a person cannot be detected continuously for a certain period of time". The fixed time includes not only the case of grasping by measuring the time but also the case of grasping by measuring the number of frames.</p><p> In addition, the "fixed number of times" in "when a moving object appears on the screen but the face of a person cannot be detected more than a certain number of times" is (1) the total number of times after the device starts operation ( The total number of frames) may be set, that is, even if the state in which the face cannot be detected even though the moving object appears on the screen is interrupted, the total may be made regardless of the interruption. (2) Movement If the face cannot be detected even though the object appears on the screen, but it is interrupted by the frame where the face can be detected is inserted in the middle, it is not reset, whereas the moving object is on the screen. If a frame that is not judged to have appeared inside is interrupted by entering in the middle, it may be reset (returning the cumulative number to zero), and (3) a moving object may appear on the screen. Regardless, if the state where the face cannot be detected is interrupted by a frame in which it is not judged that a moving object has appeared on the screen, it will not be reset, whereas the frame in which the face can be detected is in the middle. If it is interrupted by entering with, it may be reset. (4) If it is possible to determine whether or not the moving objects are the same from consecutive frames, the number of appearances of the moving objects may be set.</p><p> When the skin color model defect detection processing means is provided in this way, the lighting conditions and / or the camera conditions are suddenly changed, as in the case where the lighting / camera condition change detection processing means is provided. By discarding the current skin color model and reconstructing the skin color model in a wide color environment unadapted skin color region including various lighting conditions and / or various camera conditions, and restarting the skin color extraction and face detection processing. , Lighting conditions and / or processing that can automatically respond to sudden changes in camera conditions is realized.</p><p> Then, in the face detection device described above, the skin color model stored in the skin color model storage means is the initially set color environment unadapted skin color region and the narrow skin color region set by the update process by the skin color model update processing means. In each case, the skin color distribution that extends over the color space formed by HSV, RGB, YUV, or a pigment expression system similar to these is not the area to be extracted by the skin color partial extraction processing means and the non-extraction area on this color space. It can be a skin color model constructed by setting a threshold value with the extraction target area.</p><p> Here, the "threshold" may be defined by, for example, one mathematical expression or a combination of a plurality of mathematical expressions that can divide the color space into an extraction target area and a non-extraction target area by the skin color partial extraction processing means, and the color may be defined. It may be defined as a set of points in space. In addition, the boundary position between the extraction target area and the non-extraction target area by the skin color partial extraction processing means (for example, the boundary line in the case of a two-dimensional color space, the boundary surface in the case of a three-dimensional color space, etc. The shape of) is, for example, a polygonal shape such as a triangle or a trapezoid, a circular shape, an elliptical shape, a combination of a straight line and a curved line, a combination of curved lines, a polygonal columnar shape, a polygonal cone shape, a spherical shape, and a plane and a curved surface. Any combination, combination of curved surfaces, etc.</p><p> Further, in the face detection device described above, the skin color model stored in the skin color model storage means is a initially set color environment unadapted skin color area and a narrow skin color area set by the update process by the skin color model update processing means. In each case, a skin color model constructed by training a large number of skin color samples in a color space formed by HSV, RGB, YUV, or a similar pigment expression system by a known learning algorithm is used. be able to.</p><p> Here, when constructing the "initially set color environment unadapted skin color region" as the skin color model, for example, under various lighting conditions and / or various camera conditions, the extraction target by the skin color partial extraction processing means. Many skin color samples to be included in the area (or, together with such skin color samples, many non-skin color samples to be included in the non-extraction target area by the skin color partial extraction processing means under any lighting condition / camera condition) Can be used for learning. Further, when constructing the "narrow skin color area set by the update process by the skin color model update processing means", for example, extraction by the skin color partial extraction processing means under specific lighting conditions and / or specific camera conditions. A large number of skin color samples to be included in the target area (or, in combination with such skin color samples, should be included in the non-extracted target area by the skin color partial extraction processing means under specific lighting conditions and / or specific camera conditions. Learning can be performed using a large number of non-skin color samples). As the skin color model constructed by such learning, for example, a Gaussian mixed model, a support vector machine, a neural network, or the like can be adopted. In addition, the skin color region (skin color model constructed in the process of narrowing down the skin color region) in an intermediate state between the skin color region not adapted to the color environment and the skin color region adapted to the color environment is similar to the skin color region not adapted to the color environment. It may be a model (reduced model), a similar model (enlarged model) of the skin color region adapted to the color environment, or an interpolation model constructed by interpolating the skin color region not adapted to the color environment and the skin color region adapted to the color environment. May be.</p><p> Further, as a face detection method realized by the face detection device of the present invention described above, the following face detection method of the present invention can be mentioned.</p><p> That is, the present invention is a face detection method for detecting the face of a person by using the skin color information of the person photographed by the camera, and immediately after the start of operation of the apparatus, various illuminations are performed by the skin color partial extraction processing means. Face from the first frame image taken by the camera using the skin color model stored in the skin color model storage means with the default settings in the skin color region not adapted to the color environment including conditions and / or various camera conditions. In order to narrow down the candidate parts, a process of extracting a skin color part from the first frame image is performed, and then, a person's skin color part is extracted from the skin color part extracted as a face candidate part by the face detection processing means. The process of detecting the face is performed, and thereafter, the processes of the skin color partial extraction processing means and the face detection processing means are repeated until the person's face is detected. When the person's face is detected, the detected person's face skin color is performed. After performing the process of extracting the skin color information of the face of the person detected by the face detection processing means by the information extraction processing means, the person extracted by the detected person face skin color information extraction processing means by the skin color model update processing means. Using the skin color information of the face, the skin color model stored in the skin color model storage means in the state of being initially set in the skin color area not adapted to the color environment is changed to a skin color area narrower than the skin color area not adapted to the color environment. Automatically updates to a skin color model suitable for the current lighting conditions and / or current camera conditions of the camera installation location, and then skin color partial extraction processing means, face detection processing means, detected person face skin color. While repeating the processing by the information extraction processing means and the skin color model updating processing means, the skin color model stored in the skin color model storage means is suitable for the lighting conditions at each time point and / or the camera conditions at each time point. It is characterized by automatically updating to the skin color model.</p><p> In such a face detection method of the present invention, the actions and effects obtained by the face detection device of the present invention described above can be obtained as they are, thereby achieving the above object.</p><p> Further, in the above-mentioned face detection method, the face skin color information of the person's face extracted by the detected person face skin color information extraction processing means is individually stored in the person-specific face skin color information storage means for each person, and the face is detected. the person, if it is previously detected persons are detected before the frame by detecting human face skin color information extraction processing section, a person by facial skin face of the person stored in the color information storage means On the other hand, if it is a newly detected person who has not been detected by the previous frame, the face of the person is stored in the face skin color information storage means for each person by the detected person face skin color information extraction processing means. When the process of newly writing the skin color information of is performed and the process of updating the skin color model by the skin color model update processing means is performed, the face skin color information individually stored for each person is stored in the face color information storage means for each person. It is desirable to use it.</p><p> Further, in the above-mentioned face detection method, when the skin color model is updated by the skin color model update processing means, the cumulative number of persons whose faces are detected by the face detection processing means is less than the specified number. It is desirable to gradually narrow down the skin color model stored in the skin color model storage means from the state initially set in the skin color region not adapted to the color environment.</p><p> Then, in the above-mentioned face detection method, when the skin color model is updated by the skin color model update processing means, the cumulative number of persons whose faces are detected by the face detection processing means is less than the specified number. The skin color model stored in the skin color model storage means is gradually narrowed down from the state initially set in the skin color environment not adapted to the color environment, and the center of gravity is moved. It is desirable to move the center of gravity without doing.</p><p> Further, in the above-mentioned face detection method, when the skin color model is updated by the skin color model update processing means, the skin color information of the face of the person detected within the latest specified time or the most recently detected specified number of people is performed. It is desirable to perform processing to update the skin color model using the skin color information of the face of the person inside.</p><p> Further, in the above-mentioned face detection method, the illumination / camera condition change detection processing means sets a reference screen for illumination / camera condition change detection based on the current frame image and at least one frame image taken before that. When the change in brightness and / or color information over the entire screen is detected and the change in brightness and / or color information over the entire screen is detected, the skin color model stored in the skin color model storage means is stored. , It is desirable to perform the process of returning to the default state in the skin color area where the color environment is not adapted.</p><p> Then, in the above-mentioned face detection method, the latest frame image among the frame images in which the face of a person is detected by the face detection processing means can be used as the reference screen for detecting the change in lighting / camera conditions.</p><p> In the face detection method described above, an image in which a moving object is not shown in the screen, that is, a background image may be used as a reference screen for detecting a change in lighting / camera conditions.</p><p> Further, in the above-mentioned face detection method, when the face of the person cannot be detected continuously for a certain period of time or longer by the skin color model defect detection processing means, or even though a moving object appears on the screen, for a certain period of time or longer. When the face of a person cannot be detected continuously or cumulatively more than a certain number of times, the process of returning the skin color model stored in the skin color model storage means to the default state in the skin color area not adapted to the color environment is performed. It is desirable to do it.</p><p> Further, in the face detection method described above, the skin color model stored in the skin color model storage means is the initially set color environment unadapted skin color area and the narrow skin color area set by the update process by the skin color model update processing means. In each case, the skin color distribution that extends over the color space formed by HSV, RGB, YUV, or a pigment expression system similar to these is not the area to be extracted by the skin color partial extraction processing means and the non-extraction area on this color space. It can be constructed by setting a threshold with the extraction target area.</p><p> Further, in the face detection method described above, the skin color model stored in the skin color model storage means is the initially set color environment unadapted skin color area and the narrow skin color area set by the update process by the skin color model update processing means. In either case, a large number of skin color samples in a color space formed by HSV, RGB, YUV, or a similar dye expression system may be constructed by training with a known learning algorithm.</p>

<p> As described above, according to the present invention, a skin color region not adapted to the color environment including various lighting conditions and / or various camera conditions is initially set as a skin color model, and the skin color extraction process is performed using this skin color model. After that, when a person's face is detected, the skin color model is updated using the skin color information of the detected person's face, so that the skin color area narrower than the skin color area to which the color environment is not adapted is updated. Since the skin color model is set as the skin color model, and then the skin color model is updated and the skin color extraction process is performed using the updated skin color model, the color is obtained under various lighting conditions and / or various camera conditions. A skin color model adapted to the environment can be automatically constructed, and the skin color model can be automatically followed by changes in lighting conditions and / or camera conditions over time, so manual color adjustment, etc., etc. There is an effect that setting is unnecessary, and face detection can be performed at high speed and is not affected by the color environment.</p>

An embodiment of the present invention will be described below with reference to the drawings. FIG. 1 shows the overall configuration of the face detection device 10 of the present embodiment. FIG. 2 shows an example of a skin color distribution 50 (a distribution corresponding to a skin color region 51 in which the color environment is not adapted) by multiple people under various lighting / camera conditions, and FIG. 3 shows a specific lighting / camera. An example of the skin color distribution 60 (distribution corresponding to the color environment-adapted skin color region 61) by a plurality of people under the conditions is shown. Further, FIG. 4 is a diagram showing an example of a skin color region 51 not adapted to the color environment and a skin color region 61 adapted to the color environment set as the skin color model, and FIG. 5 is a diagram showing the color environment adapted from the skin color region 51 not adapted to the color environment. It is explanatory drawing of narrowing down to finished skin color region 61, and moving the center of gravity. Further, FIG. 6 is an explanatory diagram of the skin color portion extracted by the skin color model in the state of being initially set in the skin color region 51 in which the color environment is not adapted, and FIG. 7 is updated and set in the skin color region 61 in which the color environment is adapted. It is explanatory drawing of the skin color part extracted by the skin color model of a state. Then, FIGS. 8 to 10 show a flowchart of the processing flow by the face detection device 10.

In FIG. 1, the face detection device 10 includes a camera 20 such as a CCD camera or a CMOS camera that captures a person to be detected, a processing means 30 that performs various processes necessary for detecting the face of the person, and the like. Color environment non-adaptive skin color area storage means 40, skin color model storage means 41, frame image storage means 42, person-specific face skin color information storage means 43, color environment-adapted skin color area storage means 44, and It is equipped with a reference screen storage means 45 for detecting changes in lighting / camera conditions.

The processing means 30 includes an initial setting processing means 31, a frame image capture processing means 32, a lighting / camera condition change detection processing means 33, a skin color partial extraction processing means 34, a face detection processing means 35, and a detected person face. The skin color information extraction processing means 36, the skin color model updating processing means 37, and the skin color model defect detection processing means 38 are included.

Immediately after the start of operation of the face detection device 10, the initial setting processing means 31 initially sets the wide color environment unadaptive skin color area 51 (see FIG. 4) stored in the color environment unadaptive skin color area storage means 40. The process of storing the skin color model in the skin color model storage means 41 as the skin color model of the state is performed. Immediately after the start of operation of the device, it is unclear under what conditions the color environment is at that time, so a wide skin color model is set to prevent omission of extraction of human skin parts.

The frame image capture processing means 32 takes in a frame image taken by the camera 20 and sent from the camera 20 into the processing means 30 and stores the frame image in the frame image storage means 42. At this time, the captured frame image is stored in the frame image storage means 42 as the current frame image (the latest frame image to be the target of the face detection process), and is stored as the current frame image up to that point. The frame image is shifted and stored as the previous frame image. In this embodiment, for example, import / storage is performed at intervals of 5 frames per second or the like.

The lighting / camera condition change detection processing means 33 includes the current frame image (the latest frame image that is the target of the face detection processing) and the lighting stored in the lighting / camera condition change detection reference screen storage means 45. -By comparing with the reference screen for detecting changes in camera conditions, changes in brightness and / or color information over the entire screen are detected, and when changes in brightness and / or color information over the entire screen are detected, skin color is detected. This is a process of returning the skin color model stored in the model storage means 41 to the state initialized in the skin color region 51 (see FIG. 4) in which the color environment is not adapted. At this time, not only the background image but also a person may be shown in both the current frame image and the reference screen for detecting the change in lighting / camera conditions to be compared, but even in this case, the background image is also shown. Assuming that there is a part that is not hidden by a person, if you look at the entire screen pixel by pixel and there is no part that matches the screen brightness or color information, the brightness over the entire screen and / Or it can be determined that the color information has changed.

The skin color portion extraction processing means 34 uses the skin color model currently stored in the skin color model storage means 41 to extract the skin color portion from the current frame image in order to narrow down the face candidate portion from the current frame image. It is something to do. At this time, the skin color partial extraction processing means 34 includes whether the color information of each pixel constituting the current frame image is included in the extraction target area classified by the skin color model or in the non-extraction target area. It is done by judging.

The face detection processing means 35 performs a process of detecting a person's face from the skin color portion in the frame image extracted as a face candidate portion by the skin color portion extraction processing means 34. At this time, the face detection processing means 35 determines whether or not the skin-colored portion in the frame image narrowed down as the face candidate portion is finally a face by a known method such as statistical method or pattern matching. ..

The detected person face skin color information extraction processing means 36 performs a process of extracting the skin color information of the face of the person detected by the face detection processing means 35. At this time, the detected person face skin color information extraction processing means 36 performs a process of averaging the skin color information possessed by each pixel constituting the face portion of the detected person, and the processing is performed in the frame image to be processed. Calculate the averaged skin color information of the entire face of the detected person.

Further, when the person whose face is detected is a person who has already been detected up to the previous frame, the detected person face skin color information extraction processing means 36 is stored in the person-specific face skin color information storage means 43. The face skin color information about the person is updated, and if it is a newly detected person that has not been detected by the previous frame, the face skin color information about the person is stored in the person-specific face skin color information storage means 43. Perform a new writing process. At this time, the process of updating the facial skin color information for the detected person is performed on the facial skin color information extracted from the current frame image for that person and the facial skin color information extracted from the previous frame images (for the same person). The skin color information obtained by averaging the skin color information (extracted information from all the frame images determined to be the skin color information of the face) is calculated, and the averaged skin color information is used as the corresponding person in the face skin color information storage means 43 for each person. It is a process to write to the record of. However, in the averaging process, the skin color information that stands out from other skin color information may be excluded.

The process of determining whether the detected person face skin color information extraction processing means 36 is a detected person or a newly detected person is the position of the face grasped when the face is detected by the face detecting processing means 35 in the processing up to the previous frame. This is done by using information, facial skin color information, facial feature information, and the like. That is, whether or not the amount of change in the position of the face between the frame images is within the range that can be regarded as the movement of the face of the same person, or the time change of the position of the face between the frame images is the locus of the movement of the face of the same person. It can be judged by whether or not it can be regarded. The position information of the face of each detected person in each frame image can be stored, for example, in the person-specific face skin color information storage means 43 together with the skin color information for each person. In addition, when a person who once disappeared from the screen reappears on the screen, even if they are actually the same person, they are treated as different persons in the processing by the detected person face skin color information extraction processing means 36. Is done. On the other hand, even if a frame in which the face cannot be detected intervenes in the middle due to the person appearing on the screen turning sideways for a moment, it can be determined that the person is the same person based on the position information of the face of the detected person in the frame images before and after that. In some cases, they are treated as the same person.

The skin color model update processing means 37 uses the skin color information of each person's face extracted by the detection person face skin color information extraction processing means 36 and individually stored in the person-specific face skin color information storage means 43 for each person. , The skin color model stored in the skin color model storage means 41 is automatically updated to the skin color model suitable for the current lighting conditions of the camera installation location and the current camera conditions. At this time, the skin color model updating processing means 37 colors the skin color model stored in the skin color model storage means 41 when the cumulative number of persons whose faces are detected by the face detection processing means 35 is less than the specified number. The process of gradually narrowing down from the initial setting in the environment-unadapted skin color region 51 (see Fig. 4) and moving the center of gravity is performed. Then, when the specified number is reached, the skin color model stored in the skin color model storage means 41 is set in the skin color region 61 adapted to the color environment, and thereafter, only the center of gravity is moved without narrowing down (). See Figure 5).

Even after narrowing down to the state of the skin color region 61 that has been adapted to the color environment, it is known that the human skin color exists in a specific region in the color space to move the center of gravity, but that region is This is because it depends on the lighting conditions and camera conditions of the camera installation location at each time point. In other words, even if the optimum skin color region under a specific color environment at a certain point in time is set as the skin color model, the skin color model can correctly extract the human skin color part due to changes over time in lighting conditions and camera conditions. This is because, on the contrary, an object that is not human skin but has a color close to the skin color may be extracted.

Further, when the skin color model update processing means 37 updates the skin color model, the skin color information of each person's face (skin color information averaged for each person) is further averaged for a plurality of people. The skin color model is calculated and the skin color model is moved so that the center of gravity of the skin color model matches the calculated skin color information. Then, when calculating the skin color information averaged for a plurality of persons in this way, the skin color model update processing means 37 detects the skin color information of the face of the person detected within the latest specified time, or the most recently detected person. Use the skin color information of the faces of people within the specified number of people.

The skin color model defect detection processing means 38 cannot detect a person's face continuously for a certain period of time or longer (a predetermined number of frames or more), or even though a moving object appears on the screen, the skin color model defect detection processing means 38 for a certain period of time or longer (predetermined frame). When the face of a person cannot be detected continuously or cumulatively more than a certain number of times, the skin color model stored in the skin color model storage means 41 is used as the skin color region 51 in which the color environment is not adapted (see FIG. 4). It is a process to return to the default state. In the present embodiment, as an example, when the cumulative number of occurrences of a situation in which a person's face cannot be detected even though a moving object appears on the screen exceeds a certain number of times, the skin color model is set to the initial setting state. The explanation is given assuming that the process of returning to is performed (see Fig. 10).

The color environment unadaptive skin color region storage means 40 stores the color environment unadaptive skin color region 51 (see FIG. 4) including various lighting conditions and various camera conditions. This color environment unadapted skin color region 51 is a skin color model for initial setting constructed to correspond to the skin color distribution 50 (see FIG. 2) by a plurality of people sampled under various lighting and camera conditions. The skin color distribution 50 shown in FIG. 2 is a wider distribution than the skin color distribution 60 (see FIG. 3) described later because it is a skin color sample of a large number of people acquired while changing the lighting conditions and camera conditions. .. In the present embodiment, as an example, an HS (V) system is used as the dye expression system, so that the skin color region 51 not adapted to the color environment defines a threshold value in the HS space formed by this HS dye expression system. It can be constructed by doing.

At this time, the threshold value in the HS space stored in the skin color region storage means 40 not adapted to the color environment is, for example, a straight line forming a boundary line between the extraction target region and the non-extraction target region by the skin color partial extraction processing means 34. It may be specified by giving numerical values such as the coefficient of the mathematical formula showing the curve, the intercept of the H axis, the intercept of the S axis, the slope, the intersection, the start point and the end point, and the length, or a point located on the boundary line (HS space). It may be specified as a set of points) represented as the above coordinate values (H, S). In the present embodiment, as an example, a plurality of (here, four) straight lines of a quadrangular shape (a shape in which the apex portion of the triangle is partially missing) and a color environment unadapted skin color region 51 as shown in FIG. 4 are used. Prescribe. When the threshold value in the HS space is defined as a set of points, for example, the skin color partial extraction processing means 34 rounds off the color information of each pixel constituting the frame image to be processed. By comparing the color information of each pixel in the digit-matched state with the number of digits of the threshold value and the threshold value, whether the color information of each pixel is included in the extraction target area or the non-extraction target area. Can be processed to determine. More specifically, for example, the threshold value in HS space is defined as (H, S) = (-21,0.41) to (75,0.41) for the horizontal line of S = 0.41 and S = 0.42. When the horizontal line is specified as (H, S) = (-22,0.42) to (76,0.42), the color information of a certain pixel is (H, S) = (18.38,0.4124). When this happens, it can be rounded off to (H, S) = (18,0.41), and a process can be performed to determine whether or not this is included between the thresholds of the horizontal line of S = 0.41.

The skin color model storage means 41 is used when performing a process of narrowing down a face candidate portion from the current frame image (the latest frame image targeted for face detection processing) taken by the camera 20, that is, performing an extraction process of the skin color portion. It memorizes the skin color model used. The skin color model storage means 41 does not have a color environment including various lighting conditions and camera conditions when performing a process of narrowing down a face candidate portion from the first frame image taken by the camera 20 immediately after the device starts operating. The adaptive skin color area 51 (see FIG. 4) is stored as the default skin color model, and after the update process by the skin color model update processing means 37, the skin color area narrower than the color environment unadapted skin color area 51 (color environment adapted skin color area). 61, or the skin color region 62) in an intermediate state between the color environment unadapted skin color region 51 and the color environment adapted skin color region 61) is stored as the updated skin color model. The skin color model stored in the skin color model storage means 41 is a skin color region 51 not adapted to the color environment, a skin color region 61 adapted to the color environment, or a skin color region 62 in an intermediate state between them. Since there is only a change in the position of the center of gravity, and there is only a change in the enlargement / reduction and the position of the center of gravity for the region 62, the storage form (holding form of the threshold value) of this skin color model is the skin color not adapted to the color environment described above. This is the same as in the case of region 51.

The frame image storage means 42 stores the latest frame image (current frame image) captured by the frame image capture processing means 32 and the frame image (previous frame image) captured before the latest frame image. is there. It should be noted that the frame image captured earlier may be stored.

The person-specific face skin color information storage means 43 stores the skin color information of the person's face extracted by the detected person face skin color information extraction processing means 36 individually for each person. In this person-specific face skin color information storage means 43, each person B_jPerson B in each frame image of frame number f = 1,2,3, ..., F for each (j = 1 ~ J)_jSkin color information extracted from the face of_j, S_j)^fAnd the average value (H) for all those frames (f = 1 ~ F)_j, S_j) = Σ_f(H_j, S_j)^f/ F is memorized. Here, each frame number f = 1,2,3, ..., F is a number starting from the frame in which the face of a certain person is newly detected, and F is the face of that person so far. Is the total number of frames detected, and J is the total number of people whose faces have been detected. In addition, (H_j, S_j)^fIs the person B in each frame image_jIt is the average value of the skin color information of each pixel that composes the face part of (H)._j, S_j) Further it is person B_jThis is the average value for all the frames in which the face was detected. And this (H_j, S_j) Is averaged for more than one person during the skin color model update process.

The color environment-adapted skin color area storage means 44 stores the color environment-adapted skin color area 61 (see FIG. 4) adapted to a specific lighting condition and camera condition. This color environment-adapted skin color region 61 is a skin color model for update settings constructed to correspond to a skin color distribution 60 (see FIG. 3) by a plurality of people sampled under a specific lighting / camera condition. The skin color distribution 60 shown in FIG. 3 is a narrower distribution than the skin color distribution 50 (see FIG. 2) described above because it is a skin color sample of a large number of people obtained with fixed lighting conditions and camera conditions. .. In the present embodiment, as an example, the HS (V) system is used as the pigment expression system, so that the color environment-adapted skin color region 61 is the same as in the case of the color environment unadapted skin color region 51 (see FIG. 4). , Constructed by defining a threshold in HS space. Therefore, the storage form (threshold value holding form) of the skin color region 61 adapted to the color environment is the same as that of the skin color region 51 not adapted to the color environment described above. Further, when the skin color region 61 to which the color environment has been adapted is made into a similar shape (reduced form) of the skin color region 51 to which the color environment has not been adapted, or when the skin color region 51 to which the color environment has not been adapted is automatically deformed and constructed, the color environment is adapted. It is possible to omit the installation of the finished skin color area storage means 44.

The lighting / camera condition change detection reference screen storage means 45 is a lighting / camera condition change detection reference screen used when the lighting / camera condition change detection processing means 33 detects a change in the lighting condition and / or the camera condition. That is, it stores a reference screen to be compared with the current frame image (the latest frame image that is the target of face detection processing). In the present embodiment, the reference screen for detecting changes in lighting / camera conditions is, for example, the latest frame image among the frame images in which the face of a person is detected by the face detection processing means 35.

In the above, each of the processing means 31 to 38 included in the processing means 30 is not a computer (not only a personal computer but also a higher-end model thereof, or a general-purpose machine) constituting the face detection device 10, but is dedicated to image processing, for example. A central arithmetic processing unit (CPU) installed inside a device and a calculation unit incorporated in various electric products other than a personal computer, and one or more that specify the operation procedure of this CPU. Realized by the program.

The storage means 40 to 45 include, for example, hard disk, ROM, EEPROM, flash memory, RAM, MO, CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RAM, FD, magnetic. A tape or a combination thereof can be used.

In such an embodiment, the face detection process of a person is performed by the face detection device 10 as follows.

First, before the start of operation of the face detection device 10, the wide skin color distribution 50 shown in FIG. 2 is used to initially set the color environment unadapted skin color region 51 shown by the solid line in FIG. 4 so as to correspond to the skin color distribution 50. It is constructed as a skin color model for setting and stored in the skin color region storage means 40 for which the color environment is not adapted, and the relatively narrow skin color distribution 60 in FIG. 3 is used to correspond to this skin color distribution 60 in FIG. The color environment-adapted skin color area 61 indicated by the one-point chain line inside is constructed as a skin color model for update setting, and is stored in the color environment-adapted skin color area storage means 44. When sampling the skin color distribution 50 in FIG. 2, a skin color sample is obtained as an arbitrary method for setting the type of camera 20 and its color tone / brightness. For example, the type of camera 20 to be used and its color tone / brightness are obtained. If the brightness setting method is predetermined, those conditions may be fixed at the time of sampling.

Next, in FIG. 8, after the camera 20 is installed at the shooting location, the power of the face detection device 10 is turned on, a program for face detection processing is started, and the face detection processing is started (step S1). Subsequently, the initial setting processing means 31 reads the color environment unadaptive skin color area 51 from the color environment unadaptive skin color area storage means 40, and stores this in the skin color model storage means 41 as the initial setting skin color model (step S2). .. In addition, the initial setting processing means 31 sets the count number M for detecting a defect of the skin color model (a state in which the face cannot be detected many times even though a moving object appears) to zero. (Step S3).

Then, the frame image capture processing means 32 captures the first frame image sent from the camera 20 and stores it in the frame image storage means 42 as the current frame image (step S4).

Subsequently, it is determined whether or not the skin color model stored in the skin color model storage means 41 is the skin color region 51 in which the color environment is not adapted (step S5). Here, when it is determined that the skin color region 51 is not adapted to the color environment, the current frame image (the latest frame image to be processed) and the illumination / camera condition change detection processing means 33 are used. Reference screen for camera condition change detection By comparing with the lighting / camera condition change detection reference screen stored in the storage means 45, changes in brightness and color information over the entire screen are detected (step S6). Here, when a change in brightness or color information over the entire screen is detected, the lighting / camera condition change detection processing means 33 has a sudden change in the lighting condition or the camera condition, or the lighting condition or the camera condition. There was no face detection during that time (therefore, there was no update of the skin color model during that time, and there was no update setting of the reference screen for detecting changes in lighting / camera conditions. The cumulative amount of changes in lighting conditions and camera conditions has increased), and the skin color model stored in the skin color model storage means 41 is initially set in the skin color region 51 (see Fig. 4) where the color environment is not adapted. After returning to the state (step S7), the lighting / camera condition change detection reference screen stored in the lighting / camera condition change detection reference screen storage means 45 is discarded (step S8).

If the lighting / camera condition change detection reference screen is discarded in step S8, the lighting / camera condition change detection reference screen exists until the next face detection is performed and the setting process of step S15 described later is performed. It will be in a state where it does not. However, the skin color model is not updated until the next face detection, and therefore the skin color model remains in the state of the color environment unadaptive skin color region 51, so that the color environment unadaptive skin color region 51 is obtained in step S5. Therefore, the process of step S6 is not performed. Therefore, when the process of step S6 is performed, the inconvenient situation that the reference screen for detecting the change in lighting / camera conditions to be compared does not exist does not occur. In addition, after the device starts operating, the reference screen for detecting changes in lighting / camera conditions does not exist until the first face is detected and the setting process in step S15 is performed. In this case, the same applies to the above. Since the skin color model is not updated and remains in the state of the color environment unadapted skin color region 51 until the face is first detected, it is determined in step S5 that the skin color region is the color environment unadapted skin color region 51. , The process of step S6 is not performed, and no inconvenient situation occurs.

On the other hand, when no change in brightness or color information over the entire screen is detected in step S6, there is no sudden change in the lighting condition or the camera condition, and the lighting condition or the camera condition gradually changes. It is judged that there is no situation where the face was not detected during that time, and the processing of steps S7 and S8 is not performed. In addition, when it is determined in step S5 that the skin color region is not adapted to the color environment 51, the skin color model is a wide skin color model that can respond to various conditions in the first place, so that the lighting / camera conditions suddenly change. Since it is not necessary to consider whether or not there was a problem, the processes of steps S6 to S8 are not performed. In the case of processing the first frame image after the start of operation of the device, since the skin color model is in the initial setting state, it is determined in step S5 that the skin color region 51 is not adapted to the color environment.

After that, the skin color part extraction processing means 34 performs a process of extracting the skin color part from the current frame image using the skin color model currently stored in the skin color model storage means 41, and narrows down the face candidate parts (step S9). .. At this time, if the skin color model currently stored in the skin color model storage means 41 is a wide color environment unadapted skin color region 51 (see FIG. 4), even if it is not human skin, it depends on the lighting / camera conditions. All colors that can be human skin tones are subject to skin tones extraction. Therefore, as shown in FIG. 6, not only the human skin part such as the face 70 and the hand 71, but also an object having a color close to the skin color such as clothes 72 having a color close to the skin color is used as the skin color part. Be extracted. In this way, when the color environment is unknown (for example, immediately after the start of operation of the device), processing that prioritizes the reliable extraction of human skin rather than the efficiency of narrowing down face candidate parts by skin color extraction. I do.

On the other hand, when the skin color model currently stored in the skin color model storage means 41 is the skin color region 61 (see FIG. 4) adapted to a relatively narrow color environment, the skin color model update process described later (step S14). ) Is repeated, and since the skin color model is suitable for the color environment, it is possible to effectively extract the human skin color under the color environment. Therefore, as shown in FIG. 7, for example, the human skin part such as the face 80 and the hand 81 is extracted as the skin color part, but has a color close to the skin color such as clothes 82 having a color close to the skin color. The object is less likely to be extracted as a flesh-colored portion.

Subsequently, after setting the count number N for grasping the number of detected faces in one frame image to zero (step S10), the face detection processing means 35 is used as the skin color portion extraction processing means 34 as the face candidate portion. From the skin-colored part in the extracted frame image, a process of detecting a person's face is performed using a known method such as a statistical method or template matching (step S11).

Then, it is determined whether or not a face is detected (step S12), and if a face is detected, after adding 1 to the count number N (step S13), the skin color model update processing means 37 is used to determine the skin color model. Perform the update process (step S14). Further, the skin color model update processing means 37 uses the current frame image (frame image in which the face is detected) as a reference screen for detecting changes in lighting / camera conditions after the skin color model is updated, and is used as a reference for detecting changes in lighting / camera conditions. It is stored in the screen storage means 45 (step S15). Then, after that, the processing of steps S11 to S15 is repeated for the frame image currently being processed until no face is detected.

FIG. 9 shows a detailed flow of the skin color model update process (step S14). In FIG. 9, the detected person face skin color information extraction processing means 36 performs a process of extracting the skin color information of the face of the person detected by the face detection processing means 35 (step S1401). Then, the detected person face skin color information extraction processing means 36 determines whether or not the person whose face is detected is the same person as the detected person detected up to the previous frame (step S1402), and has already been detected. When it is determined that the person is a detected person, the face skin color information of the person stored in the person-specific face skin color information storage means 43 is updated (step S1403), while the detection is performed by the previous frame. If it is determined that there is no newly detected person, a process of newly writing the face skin color information about the person in the person-specific face skin color information storage means 43 is performed (step S1404).

After that, the skin color information of each person's face, which is extracted by the detected person's face skin color information extraction processing means 36 by the skin color model updating processing means 37 and stored individually in the person-specific face skin color information storage means 43 for each person, is stored. Using this, the skin color model stored in the skin color model storage means 41 is automatically updated to the skin color model suitable for the current lighting conditions and the current camera conditions of the camera installation location (step S1405).

More specifically, for example, as shown in FIG. 5, when the face is detected for the first time after the device starts operation, the alternate long and short dash line starts from the color environment unadapted skin color region 51 shown by the solid line in the figure. Instead of narrowing down the skin color model to the color environment-adapted skin color region 61 shown in 1 at a stretch, first narrow down to the skin color region 62 in the intermediate state shown by the dotted line in the figure. In the present embodiment, the skin color region 62 has, as an example, a similar shape (enlarged shape) of the skin color region 61 to which the color environment has been adapted, and has a size and shape that fits in the skin color region 51 not adapted to the color environment. However, the present invention is not limited to this, and for example, a similar shape (reduced form) of the skin color region 51 not adapted to the color environment may be used, and the skin color region 51 not adapted to the color environment and the skin color region 61 adapted to the color environment are interpolated and constructed. It may have the same size and shape. In addition, when setting the skin color area 62 as the updated skin color model, if the dispersion of the skin color information of the face of each detected person up to that point in the H axis (horizontal axis) direction is large, the color environment is not yet available. When the width of the updated skin color model is increased in the process of narrowing down from the adaptive skin color area 51 (processing to reduce the narrowing down in the horizontal direction), while the dispersion in the S-axis (vertical axis) direction is large. May be processed so that the vertical width of the updated skin color model becomes large (process to reduce the narrowing down in the vertical direction). Then, it is arbitrary how many stages the skin color region 62 in such an intermediate state is constructed between the skin color region 51 not adapted to the color environment and the skin color region 61 adapted to the color environment, for example, the area is 10%. It is possible to construct the skin color region 61 to which the color environment has been adapted when the cumulative number of detected persons reaches the specified number.

Further, when the skin color region 51 in which the color environment is not adapted is narrowed down to the skin color region 62 in the intermediate state, when the skin color region 62 in the intermediate state is narrowed down to the narrower skin color region 62, and the skin color in the intermediate state is narrowed down. When narrowing down from the area 62 to the skin color area 61 to which the color environment has been adapted, not only the areas are narrowed down but also the center of gravity is moved. Then, after the cumulative number of detected persons reaches the specified number and the skin color model is narrowed down to the state of the skin color region 61 to which the color environment has been adapted, the region is not narrowed down and only the center of gravity is moved.

For example, suppose that the position of the center of gravity of the skin color region 51 in which the color environment is not adapted is G1. The skin color model update processing means 37 is the skin color information of each person's face that is individually stored in the person-specific face skin color information storage means 43 for each person (the average value of all frames for the skin color information of that person's face). Is used to calculate the average value of a plurality of persons (persons detected within the latest specified time or persons within the most recently detected specified number of people). Then, the center of gravity position of the skin color region 62 is moved (determined) so that the calculated average value of the skin color information for the plurality of detected persons and the center of gravity position G2 of the updated skin color model match. Therefore, the region is narrowed down from the skin color region 51 in which the color environment is not adapted to the skin color region 62 in the intermediate state, and the center of gravity moves from G1 to G2. At this time, when the skin color region 62 protrudes from the skin color region 51 to which the color environment is not adapted as a result of moving the center of gravity, the amount of movement of the center of gravity is suppressed so as not to protrude.

Similarly, when narrowing down from the skin color region 62 in the intermediate state to the skin color region 61 (61A) adapted to the color environment, the center of gravity shifts from G2 to G3. After that, the area is not narrowed down, only the area is moved while maintaining the size and shape of the color environment-adapted skin color area 61, and the color environment is adapted from the color environment-adapted skin color area 61 (61A). When moving to the skin color area 61 (61B), the center of gravity moves from G3 to G4, and when moving from the color environment adapted skin color area 61 (61B) to the color environment adapted skin color area 61 (61C), the center of gravity is Move from G4 to G5.

In FIG. 8, if a face is not detected in step S12, whether or not the count number N for grasping the number of detected faces in the frame image to be processed is zero (that is, that frame). It is determined (whether or not even one face is detected from the image) (step S16). Here, when it is determined that no face is detected from the frame image (N = 0), even though the moving object appears on the screen by the skin color model defect detection processing means 38, It detects whether or not the cumulative number of occurrences of the situation where the person's face cannot be detected exceeds a certain number of times, and if it exceeds a certain number of times, the skin color model stored in the skin color model storage means 41 is stored. Performs the process of returning to the default state in the skin color region 51 (see Fig. 4) where the color environment is not adapted (step S17). On the other hand, if it is determined in step S16 that at least one face is detected (not N = 0) from the frame image to be processed, the skin color model defect detection process in step S17 is not performed.

FIG. 10 shows a detailed flow of the skin color model defect detection process (step S17). In FIG. 10, the skin color model defect detection processing means 38 compares the current frame image with the previous frame image (or a frame image before that) stored in the frame image storage means 42, and screens the screen. Perform the process of confirming whether or not there is a moving object inside (step S1701). Then, it is determined whether or not there is a moving object (step S1702), and if it is determined that there is a moving object, there is a defect in the skin color model (a situation in which the face cannot be detected even though the moving object appears). After adding 1 to the count number M for detecting the number of occurrences) (step S1703), it is determined whether or not the count number M exceeds a certain value (step S1704).

Here, when it is determined that the count number M is equal to or higher than a certain value, the skin color model defect detection processing means 38 has a defect in the skin color model due to a sudden change in lighting conditions or camera conditions ( The skin color model is not adapted to the environment), and the skin color area 51 that is not adapted to the color environment is read from the skin color area storage means 40 that is not adapted to the color environment, and this is stored in the skin color model storage means 41 to store the skin color. After returning the model to the default state (step S1705), the reference screen for detecting the change in lighting / camera conditions stored in the lighting / camera condition change detection reference screen storage means 45 is discarded (step S1706). Then, the skin color model is returned to the state of the skin color region 51 in which the color environment is not adapted, and when the reference screen is discarded, the count number M is reset and returned to zero (step S1707) to prepare for the next skin color model defect detection process.

On the other hand, if it is determined in step S1702 that there is no moving object, the processes of steps S1703 to S1707 are not performed. If it is determined in step S1704 that the count number M has not reached a certain value, the processes of steps S1705 to S1707 are not performed.

In FIG. 8, when it is determined in step S16 that at least one face is detected (not N = 0) from the frame image to be processed, or the skin color model defect detection process in step S17 is performed. After that, the frame image capture processing means 32 captures the next frame image (step S18), and thereafter, the processes of steps S5 to S18 are repeated.

According to the present embodiment as described above, there are the following effects. That is, since the face detection device 10 includes the initial setting processing means 31 and the skin color model update processing means 37, a wide color environment unadapted skin color region including various lighting conditions and camera conditions is immediately after the start of operation of the device. 51 (see Fig. 4) is initially set as the skin color model, and when the skin color information of the face of the detected person is obtained, the skin color area narrower than the skin color area 51 that is not adapted to the color environment is updated and set as the skin color model. The skin color model can be updated repeatedly each time a face is detected. Therefore, it is possible to automatically build a skin color model adapted to the color environment under various lighting conditions and camera conditions, and automatically follow the skin color model according to changes in lighting conditions and camera conditions over time. Can be made to. Therefore, it is possible to eliminate the need for manual setting such as color tone adjustment, and it is possible to perform face detection at high speed and regardless of the color environment.

Further, since the face detection device 10 includes the detected person face skin color information extraction processing means 36 and the person-specific face skin color information storage means 43, it is determined whether the person whose face is detected is a detected person or a newly detected person. , The skin color information of the face of the detected person can be retained and updated individually for each person. Therefore, the skin color model update processing means 37 can update the skin color model at any time using the skin color information of the face that is individually held and updated for each person, so that the skin color model including the variation of the skin color depending on the individual can be updated. It can be constructed and processed for skin color extraction and face detection.

Further, the skin color model update processing means 37 uses the skin color model stored in the skin color model storage means 41 in a color environment when the cumulative number of persons whose faces are detected by the face detection processing means 35 is less than the specified number. Since the process is gradually narrowed down from the default state to the unadaptive skin color area 51 (see Fig. 4), the complete color environment applied skin color area 61 (see Fig. 4) at once even when the cumulative number of detected persons is small. It is possible to solve the inconvenience that occurs when narrowing down to, that is, the inconvenience that the skin color model is not set in an appropriate area when the skin color information of the person whose face is detected deviates from the standard person's skin color information.

Then, the skin color model update processing means 37 shifts from the process of performing both narrowing down and moving the center of gravity to the process of moving the center of gravity without narrowing down when the cumulative number of detected persons reaches the specified number. As mentioned above, while avoiding the inconvenience that occurs when narrowing down to the skin color area 61 to which the complete color environment has been applied at once, the lighting conditions and camera conditions change over time before and after the cumulative number of detected persons reaches the specified number. It is possible to realize the processing that can be handled.

Further, the skin color model update processing means 37 updates the skin color model by using the skin color information of the face of the person detected within the latest specified time or the skin color information of the face of the person within the specified number of people detected most recently. Since the processing is performed, it is possible to automatically respond to gradual changes in lighting conditions and camera conditions over time.

Further, since the face detection device 10 includes the lighting / camera condition change detection processing means 33, it is possible to detect changes in brightness and / or color information over the entire screen. Therefore, since it is possible to detect abrupt changes in lighting conditions and camera conditions, the current skin color model is discarded and the skin color model is created in a wide color environment unadapted skin color region 51 including various lighting conditions and camera conditions. By reconstructing and restarting the skin color extraction and face detection processes, it is possible to realize a process that can automatically respond to sudden changes in lighting conditions and camera conditions. In addition, since it is equipped with the lighting / camera condition change detection processing means 33, not only can it automatically respond to sudden changes in lighting conditions and camera conditions, but also the lighting conditions and camera conditions change slowly, and during that time. It is also possible to avoid a situation in which the skin color model becomes inappropriate as a result of the person's face not being detected and the skin color model not being updated.

Since the face detection device 10 includes the skin color model defect detection processing means 38, when there is a sudden change in lighting conditions or camera conditions, as in the case of the lighting / camera condition change detection processing means 33. Illumination conditions by discarding the current skin color model, reconstructing the skin color model in a wide color environment unadapted skin color region 51 including various lighting conditions and camera conditions, and resuming skin color extraction and face detection processing. It is possible to realize processing that can automatically respond to sudden changes in camera conditions.

The present invention is not limited to the above-described embodiment, and modifications and the like within a range in which the object of the present invention can be achieved are included in the present invention.

That is, in the above-described embodiment, the color space using the HS (V) system as the dye expression system was formed (see FIGS. 2 to 4), but the dye expression system forming the color space is limited to this. Instead, for example, RGB, YUV, etc. may be used.

Further, in the above-described embodiment, the skin color model has any of the color environment unadapted skin color region 51, the color environment adapted skin color region 61, and the skin color region 62 in an intermediate state (see FIGS. 4 and 5). , However, the skin color model in the present invention is limited to the skin color model constructed by setting the threshold in the color space by such an HS (V) system or the like. However, for example, it may be a skin color model constructed by training a large number of skin color samples in a color space formed by HSV, RGB, YUV, or a pigment expression system similar thereto by a known learning algorithm.

For example, under various lighting conditions and / or various camera conditions, a large number of skin color samples (colors represented by coordinates (H, S) in HS space) to be included in the extraction target area by the skin color partial extraction processing means. Information) is used to construct a skin color region that is not adapted to the color environment as a skin color model by a Gaussian mixed model, and a large number of non-skin color models that contradict this should be included in the non-extraction target region by the skin color partial extraction processing means. Construct using a non-skin color sample (color information represented by coordinates (H, S) in HS space).

Similarly, under specific lighting conditions and / or specific camera conditions, it is represented by a large number of skin color samples (coordinates (H, S) in HS space) to be included in the extraction target area by the skin color partial extraction processing means. The color environment-adapted skin color region as a skin color model should be constructed by the Gaussian mixed model, and the non-skin color model that contradicts this should be included in the non-extraction target region by the skin color partial extraction processing means. It is constructed using a large number of non-skin color samples (color information represented by coordinates (H, S) in HS space).

Further, when constructing an intermediate skin color region between the skin color region not adapted to the color environment and the skin color region adapted to the color environment, for example, the skin color region adapted to the color environment as a skin color model and the non-skin color contradicting the skin color region are constructed. Use the model to build a model similar to these.

Then, when the skin color portion is extracted from the frame image by the skin color portion extraction processing means, the color information (color information represented by the coordinates (H, S) in the HS space) of each pixel of the frame image is used as the input value. , Using the skin color model and the non-skin color model constructed by the Gaussian mixed model, it is determined whether the color information of each pixel is included in the extraction target area by the skin color partial extraction processing means or the non-extraction target area. ..

Further, in the above embodiment, the color environment-adapted skin color region 61 (see FIG. 4) set as the skin color model when the cumulative number of detected persons reaches the specified number is made to correspond to the skin color distribution 60 in FIG. After the cumulative number of detected persons reaches the specified number, the center of gravity of this one fixed size / shape color environment-adapted skin color area 61 is moved. However, the color environment-adapted skin color region set as the skin color model when the cumulative number of detected persons reaches the specified number is not limited to one fixed size / shape region, and is large. Multiple color environment-adapted skin color regions with different values (area and volume) and / or shape may be set as the skin color model, and therefore, after the cumulative number of detected persons reaches the specified number, Along with moving the center of gravity, a process of changing the size (area or volume) and / or shape may be performed. For example, a skin color distribution corresponding to the skin color distribution 60 in FIG. 3 (skin color distribution by multiple people sampled under a specific lighting / camera condition) is formed by sampling with different lighting / camera conditions. , A plurality of color environment-adapted skin color regions may be prepared corresponding to these plurality of skin color distributions. At this time, for example, the S component value of the center of gravity of the skin color region adapted to the color environment is large in the HS space in accordance with the size and shape of each of the plurality of skin color distributions formed by sampling under each condition. In the case (when it is located at the upper part in Fig. 3), the width of the skin color region adapted to the color environment in the H direction (horizontal direction) is narrowed, while the S component value at the center of gravity is small (in Fig. 3). In the case of being located at the lower part of, the width in the H direction (horizontal direction) may be widened. In this case, not only the shape of the skin color region adapted to the color environment but also the size (area in the HS space) may be widened. ) May also be changed according to the position of the center of gravity. Further, in this case, the size (area and volume) and / or shape of the skin color region in the intermediate state corresponding to the skin color region 62 in FIG. 5 also changes according to the change in the position of the center of gravity. For example, the skin color region adapted to the color environment according to the position of the center of gravity.

Further, in the above-described embodiment, in the update process of the skin color information of the face of the detected person (step 1403 in FIG. 9), every frame in which the face of the person is detected, all frames in which the face of the person is detected are detected. However, the process of calculating the average value of is not limited to such a process. For example, if the face of a certain person is detected and then the face of the person is not detected, at that point in time. For the first time, a process may be performed in which the average value of all frames at the time when the face of the person is detected is calculated, and the skin color information of the face of the person is stored in the face skin color information storage means 43 for each person.

As described above, the face detection device and its method of the present invention include, for example, various image communications such as video conferences and video telephones, security systems such as database search and building monitoring, face matching systems, games, education, and movies. It is suitable for production of TV programs, human interfaces, real-time facial expression tracking devices that link faces generated by computer graphics, and the like.

<figref num="1">The whole block diagram of the face detection apparatus of one Embodiment of this invention.</figref><figref num="2">Illustrative diagram of skin color distribution (distribution corresponding to skin color region not adapted to color environment) by multiple people under various lighting and camera conditions.</figref><figref num="3">An example diagram of a skin color distribution (a distribution corresponding to a skin color region adapted to a color environment) by a plurality of people under a specific lighting / camera condition.</figref><figref num="4">Illustration of the skin color region not adapted to the color environment and the skin color region adapted to the color environment set as the skin color model.</figref><figref num="5">Explanatory drawing of narrowing down and moving the center of gravity from the skin color region not adapted to the color environment to the skin color region adapted to the color environment.</figref><figref num="6">Explanatory drawing of the skin color part extracted by the skin color model in the state of being initially set in the skin color region not adapted to the color environment.</figref><figref num="7">Explanatory drawing of the skin color part extracted by the skin color model in the state of being updated and set in the skin color region adapted to the color environment.</figref><figref num="8">The figure of the flowchart which shows the flow of the process (overall) by the face detection apparatus of the said embodiment.</figref><figref num="9">The figure of the flowchart which shows the flow of the process (update process of a skin color model) by the face detection device of the said embodiment.</figref><figref num="10">The figure of the flowchart which shows the flow of the process (skin color model defect detection process) by the face detection device of the said embodiment.</figref>

Code description

10 Face detection device 33 Lighting / camera condition change detection processing means 34 Skin color partial extraction processing means 35 Face detection processing means 36 Detected person Face skin color information extraction processing means 37 Skin color model update processing means 38 Skin color model defect detection processing means 41 Skin color model memory Means 43 Face skin color information storage means for each person 51 Color environment unadaptive skin color area 61 Color environment unadaptive skin color area Narrower skin color area Color environment adapted skin color area 62 Color environment unadaptive skin color area narrower than skin color area Intermediate skin color area