Image processor and method of processing image

The present invention relates to an image processing apparatus and an image processing method, and more particularly to an image processing apparatus and an image processing method for detecting a subject included in a moving image.

Image processing techniques that detect a specific subject (person, animal, specific object, etc.) from an image are extremely useful. For example, image processing technology for detecting a human face as a subject can be used in many fields such as video conferencing, man-machine interface, security, monitor system for tracking human face, and image compression.

Digital cameras and digital video cameras have already realized exposure control and focus detection control based on face detection results by detecting a person's face from captured images.

Various methods have been proposed as such an image processing technique for detecting a specific subject from an image, but most of them are methods based on pattern matching. For example, there is a method of cutting out a partial image at a plurality of different positions on the image, determining whether or not the partial image is an image of a face region, and detecting the face region on the image. Whether or not the partial image is a face region can be determined by a method using template matching or a method using a classifier in which facial features are learned by a learning method such as a neural network.

In either method, the reliability indicating the certainty that the partial image is an image of the subject area is calculated based on the image pattern of the partial image, and the partial image whose reliability exceeds a predetermined threshold value is the image of the subject area. It is common to detect as.

For example, in Patent Document 1, the reliability is calculated from partial images having a plurality of different resolution patterns, and the image in the subject region is detected (subject detection) based on the sum of the reliabilitys of each partial image.

In Patent Document 2, a person's face is detected as a subject, the position of the face in the image is detected based on the detection result, the face is focused, and the face is automatically focused so as to be photographed with the optimum exposure. It discloses a photographing device that detects and automatically exposes.

<patcit num="1"><text>Japanese Unexamined Patent Publication No. 2008-033424</text></patcit><patcit num="2"><text>Japanese Patent Application Laid-Open No. 2005-318554</text></patcit>

<p> Subject detection by pattern matching is based on the degree of possibility that a subject is included in an image, and does not mean that only a subject to be detected is detected. For example, when detecting a person's face as a subject, if the reliability of face-likeness based on pattern matching satisfies a predetermined threshold value, the face is detected even if it is not a face.</p><p> In order to reduce such false detection (excessive detection), if the reliability threshold value for considering the subject area is strictly set, an area that does not satisfy the threshold value occurs even though the subject area is present, and the detection rate of the subject is lowered. (Omission of detection). Further, if the threshold value is set loosely, erroneous detection frequently occurs in which a region different from the subject to be detected is detected as the subject. That is, there is a trade-off relationship between detection omission and false detection, and it is not easy to set the optimum detection conditions.</p><p> It is conceivable to increase the amount of calculation for pattern matching to improve the reliability of detection, but this is not realistic for devices such as digital video cameras and digital cameras, which have limited computing resources and require real-time detection.</p><p> The present invention has been made in view of such problems of the prior art, and in an image processing apparatus and an image processing method for detecting a subject from an image, the subject detection accuracy can be improved by a simple method while suppressing the amount of calculation. The purpose is to improve.</p>

<p> The above-mentioned purpose is to detect a predetermined subject area from images supplied in time series and to detect the reliability of the detected area, and a region detected by the subject detecting means from a different image. Among them, a specific means for specifying the area of the same subject, a recording means for recording subject data including a history of reliability detected by the subject detection means for the area specified by the specific means as the area of the same subject, and a specific means. Of the areas specified by the same subject, the area in which the reliability history recorded by the recording means meets the determination criteria determined according to the reliability level is determined to be the predetermined subject area. It is achieved by an image processing apparatus characterized by having means.</p><p> Further, the above-mentioned object is a lens for forming an optical image of a subject, an imaging means for sequentially capturing an optical image of a subject formed by the lens, and outputting an image supplied in time series, according to the present invention. It is also achieved by the image processing apparatus, which comprises the image processing apparatus and the control means for controlling the imaging condition by using the information of the region determined as the subject region determined in advance by the determining means.</p><p> Further, the above-mentioned object is an imaging device, which sequentially images a lens for forming an optical image of a subject and an optical image of the subject formed by the lens, and outputs an image supplied in time series. The present invention has a means and a change detecting means for calculating the amount of change of adjacent images in time series and deleting all the subject data recorded by the recording means when the amount of change exceeds a predetermined amount of change. The image processing device includes the image processing device according to the above, and the control means for controlling the imaging conditions using the information of the area determined to be the subject area predetermined by the determination means, and the image processing device is the movement of the imaging device or the lens. A change in which information on changes in the angle of view is acquired, and when at least one of these pieces of information exceeds a predetermined threshold, all subject data recorded by the recording means is deleted assuming that the image output by the imaging means changes significantly. It is also achieved by an imaging device characterized by having detection means.</p><p> Further, the above-mentioned purpose is to detect a predetermined subject area from images supplied in time series, and to detect the reliability of the detected area in a subject detection step and a subject detection step from different images. Of the identified areas, the subject data including the history of the reliability detected in the subject detection step is used as the recording means for the specific step for specifying the region of the same subject and the region identified as the region for the same subject in the specific step. Of the recording step to be recorded and the area specified as the same subject area in the specific step, the area where the reliability history recorded in the recording means satisfies the judgment criteria determined according to the reliability level. It is also achieved by an image processing method characterized by having a determination step of determining a predetermined subject area.</p>

<p> With such a configuration, according to the present invention, in the image processing apparatus and the image processing method for detecting a subject from an image, the subject detection accuracy can be improved by a simple method while suppressing the amount of calculation.</p>

Hereinafter, preferred and exemplary embodiments of the present invention will be described in detail with reference to the drawings. FIG. 1 is a block diagram showing a configuration example of an image pickup apparatus as an example of the image processing apparatus according to the embodiment of the present invention.

The lens 101 forms an optical image of the subject on the image pickup surface of the image pickup device 102 such as a CCD image sensor or a CMOS image sensor. The image sensor 102 outputs an electric signal corresponding to the intensity of the incident light beam in pixel units. This electric signal is a video signal. The video signal output from the image sensor 102 is subjected to analog signal processing such as correlated double sampling (CDS) in the analog signal processing unit 103.

The video signal output from the analog signal processing unit 103 is converted into a digital data format by the A / D conversion unit 104 and input to the photographing control unit 105 and the image processing unit 106. The image processing unit 106 performs image processing such as gamma correction and white balance processing. In addition to these normal image processings, the image processing unit 106 also performs image processing using information on the face region detected in the image, which is supplied from the face detection unit 109, as will be described later.

The video signal output from the image processing unit 106 is sent to the display unit 107. The display unit 107 is, for example, an LCD or an organic EL display, and displays a video signal. The display unit 107 can function as an electronic viewfinder (EVF) by sequentially displaying the images continuously captured in time series on the display unit 107. Further, the video signal is recorded on a recording medium 108, for example, a removable memory card. The recording destination may be the built-in memory of the camera or a connected external device capable of communicating.

The video signal output from the image processing unit 106 is also supplied to the face detection unit 109. The face detection unit 109 detects the face of a person in the image and identifies the number of subjects and the face area. A known face detection method is used as the detection method. For example, there are a method of utilizing knowledge about a face (skin color information, parts such as eyes, nose, and mouth) and a method of constructing a classifier for face detection by a learning algorithm represented by a neural net. In order to improve the recognition rate, it is common to perform face recognition by combining a plurality of methods. Specific examples thereof include a method of detecting a face by using the wavelet transform and the image feature amount described in JP-A-2002-251380.

The face area information output by the face detection unit 109 as a detection result includes the position, size, inclination, reliability, and the like of the face area for the number of detected people. Here, the reliability is a value indicating the certainty of the face detection result, and is determined in the process of face detection.

There are various methods for calculating reliability. For example, the features of the face image stored in advance are compared with the features of the image of the face area detected by the face detection unit 109, and the probability that the detected image of the face area is the image of the subject is obtained, and from this probability. There is a way to calculate the reliability. There is also a method of calculating the difference between the features of the face image stored in advance and the features of the image of the face region detected by the face detection unit 109, and calculating the reliability from the magnitude of the difference. Regardless of the reliability calculated by any method, if the reliability is high, the possibility of false detection is low, and if the reliability is low, the possibility of false detection is high.

Information on the face region, which is the detection result of the face detection unit 109, is sent to the subject identification unit 110. The subject identification unit 110 identifies the same subject from the face detection results that are continuous in time series, and sends the subject information to the face determination unit 111. The face determination unit 111 determines a highly reliable face region as a face based on the time-series face detection results supplied from the face detection unit 109. Information on the detection result determined as a face (position, size, etc. of the face region in the captured image) is supplied to the image processing unit 106 and the photographing control unit 105. Then, it can be used for controlling imaging conditions such as automatic focus detection control and automatic exposure control using information on the face region.

The photographing control unit 105 controls a focus control mechanism and an exposure control mechanism (not shown) of the imaging lens based on the video signal output from the A / D conversion unit 104. The photographing control unit 105 can use the information of the detection result supplied from the face determination unit 111 to control the focus control mechanism and the exposure control mechanism. Therefore, the image pickup apparatus of the present embodiment can realize a function of performing a shooting process in consideration of the information of the face region in the captured image. Specifically, exposure control, focus detection control, flash control, etc. based on the face region can be realized. The photographing control unit 105 controls the output timing and output pixels of the image sensor 102.

(Subject identification processing) Information on the face region (position, size, reliability, etc. in the captured image) is supplied to the subject identification unit 110 as a face detection result from the face detection unit 109. The face detection unit 109 performs face detection processing without holding or using past face detection results. Therefore, the subject identification unit 110 holds the face detection result of the face detection unit 109 in time series, and identifies the face region of the same subject based on the time series face detection result. This enables subject tracking.

The process of identifying the same subject using the position and size of the face region will be described with reference to FIG. FIG. 2 shows two consecutive frames taken in chronological order, in which (b) shows the current captured image and (a) shows the image one frame before (b). Although face detection is performed here for each frame, face detection may be performed every few frames. The face areas 201, 202, and 203 detected in each frame by the face detection unit 109 are indicated by a display (face frame) indicating the face area.

For the face area detected in the previous frame shown in Fig. 2 (a), the position was determined using the coordinates in the image (x)._t-1(i), y_t-1(i)), size s_t-1Expressed as (i). Also, for the face area detected in the current frame, the position is (x)._t(j), y_t(j)), size s_tExpressed as (j). Here, the values of i and j are integers that take values from 1 to n, and the values are assigned to each face area detected in the same frame. n indicates the total number of face regions detected.

The subject identification unit 110 determines a face region whose position and size are 0 or more in the value of Equation 1 below as the same subject in the face detection results that are continuous in time series.

<maths num="1"><img file="JP2010141847A_D0001.tif" /></maths>

In other words, for face areas of the same size between consecutive frames, if the face area of the current frame is included in the range where the area of the same size as the face area is adjacent to the face area of the previous frame, both are combined. Judged as the face area of the same subject.

Further, for face regions having different sizes between consecutive frames, if the coordinate difference between the two face regions is included in the adjacent range set with reference to the smaller face, it is determined that the person is the same person. Based on the result of the current frame, if there are a plurality of detection results of the previous frame in which the value of the expression 1 is 0 or more, the smaller value of the expression is determined as the same person. Further, if there is no result that the detection result of the previous frame is 0 or more with the value of Equation 1 based on the result of the current frame, the subject as a result is regarded as a new subject. That is, in FIG. 2, the face area 201 in the frame t-1 and the face area 202 in the frame t are determined to be the same subject, and the face area 203 in the frame t is determined to be a newly appearing subject.

The subject identification method described here is an example, and the same subject may be identified by using another method. For example, if information on the inclination and orientation of the face can be obtained from the face detection unit 109, they may be used as a condition for identifying the subject in the subject identification unit 110. It is also possible to acquire information on the imaging device such as the zoom ratio (focal length) of the lens 101, the amount of movement of the video camera, and ON / OFF of image stabilization, and use these as conditions for identifying the subject.

The subject identification unit 110 holds the subject list 112 as shown in FIG. 6, and updates the subject list 112 every time the detection result is received from the face detection unit 109. The details of the subject list 112 will be described later, but for each face area, information on the position and size, the history of the reliability level, and the like are stored.

(Face judgment processing) The face determination unit 111 determines information that can be effectively used by the image processing unit 106 and the shooting control unit 105 from the face detection result output by the face detection unit 109, and supplies the information to the image processing unit 106 and the shooting control unit 105. .. That is, the face determination unit 111 extracts a highly reliable detection result from the face detection results output by the face detection unit 109 and supplies it to the image processing unit 106 and the photographing control unit 105.

Specifically, the face determination unit 111 holds in advance the number of continuous detections or the continuous detection time according to the reliability level (height) as a determination criterion. Then, with respect to the face region determined by the subject identification unit 110 to be the same subject, the face region satisfying the determination criteria is finally determined to be a highly reliable face region as a face.

That is, the face determination unit 111 determines the reliability of the face area that is continuously detected for the number of times or time according to the reliability level calculated by the face detection unit 109 among the face areas determined to be the same subject. Judged as a high face area. Here, the reliability level is a value obtained by subjecting the reliability obtained from the face detection unit 109 to a predetermined process such as a normalization process, even if the reliability itself is supplied from the face detection unit 109. You may. Actually, the face determination unit 111 refers to the subject list 112 (FIG. 6) held and managed by the subject identification unit 110, and determines a face region satisfying the determination criteria from the history of the reliability level as a face. Then, the face determination unit 111 reads out information on the position and size of the face region determined to be a face from the subject list 112 and supplies the information to the image processing unit 106 and the photographing control unit 105.

FIG. 3 shows an example of the face region detected by the face detection unit 109 and the corresponding reliability level. In this embodiment, the reliability level has 5 levels from 1 to 5, and 5 is the most reliable. In the example shown in FIG. 3, many of the face areas (401,403,404,406,408,410) that correctly detect the face have a high reliability level, and the face areas (402,405,407,409) that erroneously detect areas other than the face have a low reliability level. There are many results. However, some correctly detected face regions have a low reliability level (face region 403,408), and some falsely detected face regions have a high reliability level (face region 402,407).

Therefore, it is difficult to accurately distinguish between correct detection and false detection based on the reliability level obtained from only one frame. Therefore, in the present embodiment, the reliability level of the face region of the same subject is tracked over time, and it is determined whether or not the face region of each subject is truly a face region. Specifically, as described above, for the face area determined to be the face area of the same subject, a judgment standard is set according to the reliability level, and the face area satisfying the judgment standard is determined to be a true face area. .. The criterion may be, for example, the number of times or time that a certain reliability level is continuously detected. In the present embodiment, the subject identification unit 110 tracks and records the reliability level for each face region, and the face determination unit 111 makes a determination based on the determination criteria.

FIG. 4 is a diagram schematically showing an example in which the judgment criteria are applied in FIG. In FIG. 4, the face area whose face frame is indicated by a solid line is determined to be a face by the face determination unit 111, and the face area whose face frame is indicated by a dotted line is determined to be a face by the face determination unit 111. Indicates that it has not been done.

In FIG. 4, in order to simplify understanding and explanation, it is assumed that a judgment criterion is set in which a face is judged when reliability level 4 or higher continues for 2 frames, and a face is not judged otherwise. At the stage of frame t-2 (Fig. 4 (a)), none of the face areas meet the judgment criteria, so none of the face areas 401 to 403 are judged to be faces, and the face frame is indicated by a dotted line. ing.

In the next frame t-1 (FIG. 4 (b)), the face area 404 has a reliability level of 5 continuously from the face area 401 of the previous frame determined to be the same subject. Is determined. The erroneously detected face area 405 was not determined to be a face because the reliability level of the face area 402 corresponding to the previous frame was 4, but the reliability level was lowered to 1 in the current frame. Further, the correctly detected face area 406 is not determined to be a face at this point because the corresponding face area 403 in the previous frame has a reliability level of 2.

In frame t (FIG. 4 (c)), the reliability level of the face area 408 was lowered to 2, but since it was already determined to be a face, it is still determined to be a face at this point. Since the face area 410 continuously has the reliability level 4 from the face area 406 of the previous frame which is determined to be the same subject, it is determined to be a face at this point. The falsely detected face area 409 is still not determined to be a face because it has a reliability level of 3. The face area 407 erroneously detected in the frame t-1 is not determined to be a face because there is no face area that is determined to be the same subject in the frame t.

The face determination process in the image pickup apparatus of the present embodiment will be further described with reference to the flowchart shown in FIG. First, the face detection unit 109 detects a person's face in the image and obtains the position (coordinates), size, reliability, etc. in the image for each of the detected face areas (S601). In the present embodiment, the face detection unit 109 detects the face for each image obtained continuously in time series.

Next, for one of the detected face regions, the subject identification unit 110 compares the detection result of the current frame with the detection result of the previous frame by the subject identification method described above, and identifies the face region of the same subject ( S602).

Based on this determination, the subject identification unit 110 updates or registers the subject list 112 as shown in FIG. That is, the subject identification unit 110 updates the data registered in the subject list 112 for the face area detected in the current frame that is determined to be the same subject as the face area detected in the previous frame. (S603). Further, the subject identification unit 110 deletes the subject data related to the subject which was detected in the previous frame but was not detected in the current frame from the subject list 112.

The subject identification unit 110 rewrites and updates the position and size of the face region in the subject data based on the detection result in the current frame. Further, the number of continuous detections of the reliability level is rewritten and updated by a method according to the reliability level in the current frame and the judgment standard.

For example, if the criterion is the number of continuous detections of a specific reliability level, the number of continuous detections of the corresponding reliability level is set to 1 only when the reliability level in the current frame is equal to the reliability level in the previous frame. increase. If the reliability level in the current frame is different from the reliability level in the previous frame, the number of continuous detections of the reliability level in the current frame is set to "1", and the number of continuous detections of other reliability levels is set. Set to "0".

On the other hand, when the criterion is the number of continuous detections above a certain reliability level, the number of continuous detections of the reliability level in the current frame and the reliability level lower than that is increased, and the reliability level in the current frame is increased. However, the number of continuous detections of high reliability level is set to 0.

For example, if the reliability level in the current frame is 5, the number of continuous detections of all reliability levels is increased. If the reliability level in the current frame is 3, the number of continuous detections of reliability levels 1 to 3 is increased, and the number of continuous detections of reliability levels 4 to 5 is set to 0. By such an update method, it is possible to record the number of times that the reliability level equal to or higher than the individual reliability level is continuously detected.

On the other hand, the subject identification unit 110 assigns a new subject ID to the subject list 112 for a subject different from the face region detected in the previous frame among the face regions detected in the current frame. Register as subject data (S604).

The subject identification unit 110 updates and registers the data of the subject list 112 of FIG. 6 based on the position / size of the face area, the reliability level, and the subject identification result supplied from the face detection unit 109. ..

In the subject list 112, the position and size of the face area, the number of continuous detections for each reliability level, and the face determination flag indicating whether or not the face is determined are displayed for each subject ID for identifying the subject (face area). , An update flag is associated to indicate that it has been registered or updated in the current frame. The update flag is cleared to 0 every frame, and the update flag becomes 1 by updating S603 or registering S604.

Since there is no data in the subject list 112 at the time of processing in the initial frame, the subject identification unit 110 registers all the detected face area information in the subject list 112 as new subject data. After the second frame, the subject identification unit 110 performs subject determination processing using the subject data in the subject list 112 and the information of the face detection result of the current frame.

The face determination unit 111 of each subject is based on the history of the reliability level of the face area of each subject recorded in the subject list 112, specifically, the number of continuous detections or the continuous detection time according to the reliability level. Of the face areas, the face area that is likely to be a face is determined.

Further, in the present embodiment, for a subject determined as a face by the face determination unit 111, if a face region identified as the same subject is detected regardless of the level of reliability in the subsequent detection results, Determined to be a face.

Therefore, the face determination unit 111 first determines whether or not the face area detected in the current frame is already determined to be a face (S605). The face determination unit 111 determines that if the face determination flag in the subject list 112 shown in FIG. 6 is 1, it is already determined to be a face, and if it is 0, it is not determined to be a face. The value of the face determination flag at the time of registering the subject data is set to 0.

Then, the face determination unit 111 refers to the history of the reliability level with respect to the face region in which the face determination flag is 0, and determines whether or not the predetermined determination criteria are satisfied (S606). Here, a reference value f (L) for the number of continuous detections according to the reliability level L is defined as a determination criterion, and it is determined whether or not this determination criterion is satisfied.

FIG. 7 shows an example of a reference value f (L) of the number of continuous detections required to determine a face, which is set according to the reliability level. In FIG. 7, the horizontal axis shows the reliability level L, and the vertical axis shows the reference value f (L) for the number of continuous detections. The higher the reliability level, the lower the reference value f (L). There is. When the continuous detection time is used instead of the continuous detection number, the higher the reliability level, the shorter the reference value f (L) is set.

The reliability level L and the reference value f (L) for the number of continuous detections may have a linear relationship or a non-linear relationship. Further, the reference value f (L) of the number of continuous detections may be the number of times for each reliability level or the number of times of each reliability level or higher.

The face determination unit 111 determines as a face a face area included in the subject list 112 shown in FIG. 6 in which the number of continuous detections according to the reliability level satisfies the reference value shown in FIG. 7, and determines the subject list. Set the value of the face judgment flag in 112 to 1 (S607). Then, the face determination unit 111 reads out the information of the face region determined to be a face from the subject list 112 and supplies it to the image processing unit 106 and the photographing control unit 105.

On the other hand, the face determination unit 111 does not determine the face region in which the number of continuous detections does not meet the reference value as a face, and leaves the face determination flag at 0. Then, the face determination unit 111 does not supply the information of the face region that is not determined to be a face to the image processing unit 106 and the photographing control unit 105.

In order to perform the above-mentioned processes S602 to S607 for each face region detected by the face detection in S601, whether or not the face determination unit 111 has processed all the face regions detected in the current frame. Is determined (S608).

Then, when the unprocessed face area remains, the face determination unit 111 returns the processing to S602 with one of them as the processing target. On the other hand, when all the face regions detected in the current frame are processed, the face determination unit 111 shifts the processing to S609.

Then, if there is unupdated data in the subject data in the subject list 112, the face determination unit 111 detects the data in the previous frame but not in the current frame. Therefore, the face determination unit 111 starts from the subject list 112. Delete (S609).

As described above, according to the present embodiment, when determining whether the face region detected by the face detection process is a face, it is based not only on the reliability in the detected frame but also on the history of the reliability. By making the determination, the accuracy of face detection can be improved by a simple method.

(Modification example 1) In the present embodiment, the subject data that was detected in the previous frame but not in the current frame was described as being deleted from the subject list 112 in S609.

However, there is a possibility that the subject was not detected due to a problem of the face detection unit 109 even though the subject actually exists. In order to prevent the detection accuracy from being lowered due to such detection omission, the subject data once determined as a face should be retained until it is confirmed that the subject is not continuously detected over a predetermined number of frames. It may be configured.

The face determination process in the image pickup apparatus when such a configuration is adopted will be further described with reference to the flowchart shown in FIG. S901 to S908 in FIG. 8 are the same as S601 to S608 in FIG. 5 described above. By the processing of S901 to S908, the subject identification processing and the face determination processing are performed on the face area detected by the face detection unit 109 in the current frame.

FIG. 9 shows an example of the subject list 112'used in this modified example. The subject list 112'in FIG. 9 has, in addition to the information of the subject list 112 of FIG. 6, information on the number of frames that retain the undetected subject data and the number of frames that have been retained so far in the undetected state. .. The number of retained frames is cleared to 0 when the subject identification unit 110 determines that the same subject has been detected by the face detection unit 109.

When the face determination process for all the face areas detected in the current frame is completed, the face determination unit 111 determines whether or not there is unupdated subject data with the face determination flag set to 1 (determined as a face). Judge (S909).

When the face determination flag is 1 and there is unupdated subject data, the face determination unit 111 refers to the corresponding "number of retained frames" and determines whether or not it is equal to or less than the "number of retained frames" (S910).

If the "number of retained frames" is less than or equal to the "number of retained frames", the face determination unit 111 updates by increasing the corresponding "number of retained frames" without deleting the unupdated subject data (the number of retained frames). S911).

On the other hand, the face determination unit 111 does not update the unupdated subject data whose "number of retained frames" exceeds the "number of retained frames". It is confirmed in S909 that the face determination unit 111 has performed determination processing of whether or not the "number of retained frames" is equal to or less than the "number of retained frames" for all the subject data that has been determined to be a face and has not been updated. Until, the processing of S909 to S911 is repeatedly executed.

After that, the face determination unit 111 deletes the unupdated subject data from the subject list 112'(S912). Therefore, the subject data deleted in S912 is the subject data that has been determined to be a face, and is the subject data for the face area that has not been continuously detected for a predetermined number of frames, and the subject data that has not been determined to be a face and is not determined to be the current frame. This is the subject data for the face area that was not detected in.

The values such as the position and size in the subject data with respect to the face area that were not detected in the current frame, which are retained in this modification, are the values before the face area is not detected even if it is the value immediately before the face area is not detected. It may be an estimate from the value in the frame.

Further, the value of the "number of frames to be retained" can be determined according to the reliability level that was continuously detected immediately before the face area was no longer detected and the number of continuous detections thereof. For example, if a high reliability level is continuously detected many times, the value of "the number of frames to be retained" can be set large. Further, when the high reliability level is continuously detected a small number of times or when the low reliability level is detected, the value of "the number of frames to be retained" can be set small.

Further, the value of the number of frames to be retained can be determined based on other conditions in addition to or instead of the reliability level and the number of continuous detections. Other conditions may be, for example, the position and size of the face area in the subject list. For example, if the position of the face is the edge of the image, it is highly possible that the face protrudes from the angle of view as a factor of not being detected. Therefore, if the position of the face is near the center of the image, the number of frames to be held can be increased, and if the position of the face is near the edge of the image, the number of frames to be held can be reduced.

(Modification example 2) In the present embodiment, a case where the continuous detection number or the continuous detection time set according to the reliability level is used as the judgment standard used in the face judgment in the face judgment unit 111 has been described. However, the number of continuous detections or the continuous detection time may be changed according to the number of detected face areas (subjects), the amount of movement of the face area, and the like.

The face determination process in the imaging device when the number of continuous detections or the continuous detection time is changed according to the number of detected face regions (subjects) will be further described with reference to the flowchart shown in FIG.

First, the face detection unit 109 detects a person's face from the image (S1101). Then, the face detection unit 109 counts the number of detected persons (S1102). The number of detected people may be counted as the cumulative value of the number of detected people in each frame included in a certain period of time, instead of the number of detected people in each frame. For example, if a plurality of people pass by at one time, the number of detected people will temporarily increase and the judgment criteria will fluctuate significantly. Therefore, there is a possibility that the face of the person who has been continuously detected up to that point will not be detected. In addition, the determination criteria may change each time the face detection unit 109 fails to detect. Therefore, by setting the count of the number of detected persons as the cumulative value of the number of detected persons over a certain period, it is possible to suppress fluctuations in the determination criteria.

Next, the subject identification unit 110 compares the detection results of the previous frame with the detection result of the current frame as a reference, and determines the face region of the same subject (S1103). Then, if the same subject face area as the previous frame is detected in the current frame, the subject identification unit 110 updates the corresponding subject data in the subject list (S1104).

On the other hand, when the face area of the subject not detected in the previous frame is detected in the current frame, the subject identification unit 110 registers the information about the face area in the subject list as new subject data (S1105). ..

FIG. 11 shows a subject list 112 used in this modified example. The subject list 112 in this modified example has information on the number of detected persons in addition to the information in the subject list 112 in FIG. The number of detected persons is supplied from the face detection unit 109 to the subject identification unit 110, and when the subject data is registered, the subject identification unit 110 registers the subject list 112 .

The face determination unit 111 determines whether or not the face region detected in the current frame is the face region of the subject already determined to be a face (S1106). Then, the face determination unit 111 determines whether or not the number of continuous detections satisfies the reference value f (L, n) corresponding to the reliability level L and the number of detected persons n with respect to the face region that is not determined as a face ( S1107).

The face determination unit 111 determines a face region in which the number of continuous detections satisfies the reference value as a face, and updates the subject list (S1108). Here, in this modification, if the number of detected persons counted by S1102 is small, the detection frequency should be increased. Therefore, the reference value of the number of continuous detections is not reduced and the face determination standard is loosened. On the other hand, when the number of people to be detected is large, it is desired to suppress erroneous detection rather than increasing the detection frequency. Therefore, the reference value for the number of continuous detections is increased and the judgment criteria are tightened.

Figure 12 shows an example of setting the reference value for the number of continuous detections according to the reliability level and the number of detections. In FIG. 12, the horizontal axis shows the reliability level, the depth axis shows the level of the number of detected people, and the vertical axis shows the reference value of the number of continuous detections corresponding to them. In the example of FIG. 12, the number of detected people is classified into two, large and small, according to a predetermined threshold value, but it may be classified more finely. Further, the relationship between the reliability level and the number of detected persons and the number of continuous detections may be a linear relationship or a non-linear relationship.

The face determination unit 111 determines whether or not the above-described processes S1103 to S1108 have been executed for each of the face regions detected by the face detection in S1101 (S1109). If there is an unprocessed face area, the processing from S1103 is repeatedly executed for one of them. When the processing of all the face areas is completed, the face determination unit 111 deletes the unupdated data among the subject data in the subject list in S1110.

Instead of the number of people detected by the face detection unit 109, the number of face regions determined as faces by the face determination unit 111 may be reflected in the determination criteria. Needless to say, the continuous detection time may be used instead of the continuous detection number.

Next, the face determination process in the imaging device when the number of continuous detections or the continuous detection time used as the determination criterion is changed according to the number of detected face regions (subjects) is further described by using the flowchart shown in FIG. explain.

First, the face detection unit 109 detects a person's face from the image (S1401). Next, the subject identification unit 110 compares the detection results of the previous frame with the detection result of the current frame as a reference, and determines the face region of the same subject (S1402).

When a face region determined to be the same subject as the previous frame is detected in the current frame, the subject identification unit 110 calculates the amount of movement of the subject from the difference in the detection positions of the face region (S1403). The amount of movement of the subject may not be the amount of movement between two adjacent frames, but may be a cumulative value obtained by adding the amount of movement between two adjacent frames for a plurality of frames included within a certain period of time. By setting the cumulative value, even if the amount of movement of the same subject temporarily increases due to an erroneous determination of the same subject, it is possible to prevent the determination criteria from fluctuating significantly. Then, the subject identification unit 110 updates the corresponding subject data in the subject list, including the movement amount (S1404).

On the other hand, when the face area of the subject not detected in the previous frame is detected in the current frame, the subject identification unit 110 registers the information about the face area in the subject list as new subject data (S1405). .. The amount of movement when newly registered is set to 0.

FIG. 14 shows a subject list 113 used in this modified example. The subject list 113 in this modification has information on the amount of movement in addition to the information on the subject list 112 in FIG. The movement amount is, for example, registered (updated) with respect to the subject data corresponding to the face area determined by the subject identification unit 110 to be the same subject.

The face determination unit 111 determines whether or not the face region detected in the current frame is the face region of the subject already determined to be a face (S1406). Then, the face determination unit 111 determines whether or not the number of continuous detections satisfies the reference value f (L, w) corresponding to the reliability level L and the movement amount w of each subject with respect to the face region that is not determined as a face. Judge (S1407).

The face determination unit 111 determines a face region in which the number of continuous detections satisfies the reference value as a face, and updates the subject list (S1408). In normal photography, a subject with a large amount of movement is unlikely to be the main subject, and a subject with a small amount of movement is likely to be the main subject. Therefore, in this modification, the reference value for the number of continuous detections is reduced for the face region where the amount of movement is small, and the face determination standard is loosened. On the other hand, for the face region where the amount of movement is large, the reference value for the number of continuous detections is increased to make the face determination criteria stricter .

FIG. 15 shows an example of setting a reference value for the number of continuous detections according to the reliability level and the amount of movement of the subject. In FIG. 15, the reliability level is shown on the horizontal axis, the amount of movement is shown on the depth axis, and the reference value of the number of continuous detections corresponding to them is shown on the vertical axis. In the example of FIG. 15, the movement amount is classified into two large and small according to a predetermined threshold value, but it may be classified more finely. Further, the relationship between the reliability level and the amount of movement and the number of continuous detections may be a linear relationship or a non-linear relationship.

The face determination unit 111 determines whether or not the above-described processes S1402 to S1408 have been executed for each of the face regions detected by the face detection in S1401 (S1409). If there is an unprocessed face area, the processing from S1402 is repeatedly executed for one of them. When the processing of all the face areas is completed, the face determination unit 111 deletes the unupdated data among the subject data in the subject list in S1410.

The method of reflecting the amount of movement of the subject in the judgment criteria is not limited to the method of changing the number of continuous detections according to the amount of movement between consecutive frames or the added value of the amount of movement of each subject within a certain period. .. For example, when the amount of movement between consecutive frames exceeds a certain threshold value, the reference value of the number of continuous detections may be changed.

(Modification example 3) In the present embodiment, any subject data is kept in the subject list until it is determined that the subject data has not been updated. Therefore, when the angle of view changes significantly due to a change in the imaging mode or the amount of movement of the camera is large between consecutive frames, the subject identification unit 110 specifies the face region of a different subject as the face region of the same subject. There is a risk that it will end up. If the subject is specified incorrectly, the reliability of the final face determination result will be greatly reduced.

In order to deal with this, in this modification, if it is detected that the entire image has changed significantly, such as a large change in the angle of view or shooting direction, all the subject data in the subject list is deleted, and then the next step is taken. Register new subject data after the frame.

FIG. 16 is a flowchart for explaining the face determination process in this modified example. First, for example, the image processing unit 106 as a change detecting means calculates the amount of change in the image between consecutive frames and detects whether or not there is a large change in the entire image (S1701). This amount of change can be calculated from the luminance component, color component, or edge component between consecutive frames. Then, when the amount of change exceeds a predetermined amount of change, the image processing unit 106 determines that the image has changed significantly. When the image processing device according to the embodiment is applied to an image pickup device, a change in a shooting mode (for example, zoom magnification) is detected, or information is acquired from a gyro sensor that the image pickup device has, for example, for camera shake correction. It may be determined whether or not the entire image has changed significantly.

If there is a large change in the entire image, the image processing unit 106 deletes all the subject data in the subject list (S1702). Then, in S1703, the face detection unit 109 performs face detection processing. At this time, all the face detection results are registered in the subject list as new subject data. On the other hand, if there is no big change in the whole image, move to S1703. Since S1703 to S1711 in FIG. 16 are the same as S601 to S609 in FIG. 5, the description thereof will be omitted.

Instead of the image processing unit 106, the face detection unit 109 is configured to supply image data to the subject identification unit 110 without performing face detection, and the subject identification unit 110 detects a large change in the entire image. May be good.

In this modified example, the "major change in the entire image" is, for example, a movement of the camera or a change in the angle of view that changes more than half of the scene, or can be discriminated as a scene change by general scene change detection technology. It may be such a change.

(Other embodiments) The above-described embodiment and its modification have been described by applying the image processing apparatus according to the present invention to the imaging apparatus and focusing on the face determination processing at the time of imaging. However, it will be understood that the image used for face determination is not limited to the image taken in real time, and may be a recorded image. Therefore, the above-described embodiment and its modification can be similarly applied to the face determination process at the time of reproducing a moving image. Moreover, it is also possible to carry out the above-mentioned modification in combination of a plurality.

Further, although face detection and face determination have been described as an example of subject detection and subject determination, the subject is not limited to the face of a person, and any object or organism that can be detected from the image by applying a known image recognition technique. It will also be understood that it may be.

Further, the above-described embodiment (including a modified example; the same applies hereinafter) can also be realized by software by a computer (or CPU, MPU, etc.) of the system or the device. Therefore, in order to realize the above-described embodiment on a computer, the computer program itself supplied to the computer also realizes the present invention. That is, the computer program itself for realizing the functions of the above-described embodiment is also one of the present inventions.

The computer program for realizing the above-described embodiment may be in any form as long as it can be read by a computer. For example, it can be composed of object code, a program executed by an interpreter, script data supplied to the OS, etc., but is not limited to these.

The computer program for realizing the above-described embodiment is supplied to the computer by a storage medium or wired / wireless communication. Examples of the storage medium for supplying the program include a flexible disk, a hard disk, a magnetic storage medium such as a magnetic tape, an optical / magneto-optical storage medium such as MO, CD, and DVD, and a non-volatile semiconductor memory.

As a method of supplying a computer program using wired / wireless communication, there is a method of using a server on a computer network. In this case, a data file (program file) that can be a computer program forming the present invention is stored in the server. The program file may be an executable file or a source code.

Then, it is supplied to the client computer that has accessed this server by downloading the program file. In this case, it is also possible to divide the program file into a plurality of segment files and distribute the segment files to different servers. That is, one of the present inventions is a server device that provides a client computer with a program file for realizing the above-described embodiment.

In addition, a storage medium in which a computer program for realizing the above-described embodiment is encrypted and stored is distributed, and key information for decrypting the encryption is supplied to a user who satisfies a predetermined condition, and the computer owned by the user is provided with the key information. You may allow the installation. The key information can be supplied, for example, by downloading it from a homepage via the Internet.

Further, the computer program for realizing the above-described embodiment may use the function of the OS already running on the computer. Further, the computer program for realizing the above-described embodiment may be partially configured with firmware such as an expansion board mounted on the computer, or may be executed by the CPU provided in the expansion board or the like. Good.

<figref num="1">It is a block diagram which shows the structural example of the image pickup apparatus as an example of the image processing apparatus which concerns on embodiment of this invention.</figref><figref num="2">It is a figure explaining the process of identifying the same subject using the position and size of a face region in the image pickup apparatus which concerns on embodiment of this invention.</figref><figref num="3">It is a figure which shows the example of the face region detected by the face detection part, and the corresponding reliability level in the image pickup apparatus which concerns on embodiment of this invention.</figref><figref num="4">It is the figure which showed typically the example which applied the judgment criteria to the image shown in FIG.</figref><figref num="5">It is a flowchart for demonstrating the face determination process in the image pickup apparatus which concerns on embodiment of this invention.</figref><figref num="6">It is a figure which shows the example of the subject list in the image pickup apparatus which concerns on embodiment of this invention. ..</figref><figref num="7">It is a figure which shows the setting example of the reliability level and the reference value of the continuous detection number of times in the image pickup apparatus which concerns on embodiment of this invention.</figref><figref num="8">It is a flowchart for demonstrating the face determination process in the image pickup apparatus which concerns on modification 1 of embodiment of this invention.</figref><figref num="9">It is a figure which shows the example of the subject list in the image pickup apparatus which concerns on the modification 1 of the Embodiment of this invention.</figref><figref num="10">It is a flowchart for demonstrating the face determination process in the image pickup apparatus which concerns on modification 2 of embodiment of this invention.</figref><figref num="11">It is a figure which shows the example of the subject list in the image pickup apparatus which concerns on the modification 2 of the Embodiment of this invention.</figref><figref num="12">It is a figure which shows the setting example of the reference value of the continuous detection number according to the reliability level and the number of detections in the image pickup apparatus which concerns on the modification 2 of the Embodiment of this invention.</figref><figref num="13">It is a flowchart for demonstrating another face determination processing in the image pickup apparatus which concerns on modification 2 of embodiment of this invention.</figref><figref num="14">It is a figure which shows the example of another subject list in the image pickup apparatus which concerns on the modification 2 of the Embodiment of this invention.</figref><figref num="15">It is a figure which shows the setting example of the reference value of the continuous detection number | times according to a reliability level and a subject movement amount in the image pickup apparatus which concerns on modification 2 of the Embodiment of this invention.</figref><figref num="16">It is a flowchart for demonstrating the face determination processing in the image pickup apparatus which concerns on modification 3 of embodiment of this invention.</figref>