Eye opening/closing determining apparatus

【0001】

[Technical field to which the invention belongs]

The present invention relates to an open / close eye determination device that performs an open / close eye determination process, for example, when notifying a driver who drives a vehicle of inattentiveness or dozing.

【0002】

[Conventional technology]

Conventionally, as described in Patent Document 1, Patent Document 2, and Patent Document 3 below, for example, a face image of a vehicle driver is imaged, and the amount of change in the vertical width of the eye is used from the face image. , It is known that the opening / closing eye is determined by using the amount of change in the distance between the peak point of the upper eyelid and the eyebrows or by using the shape of the upper eyelid.

【0003】

[Patent Document 1]

Japanese Unexamined Patent Publication No. 9-44685 [0004]

[Patent Document 2]

Japanese Unexamined Patent Publication No. 10-143669 [0005]

[Patent Document 3]

Japanese Unexamined Patent Publication No. 2000-123188 [0006]

[Problems to be Solved by the Invention]

However, in the above-mentioned conventional technique, when the light environment condition for taking a facial image changes, or when there are individual differences in facial features such as whether or not glasses are worn and the eyebrows are thin, the determination accuracy of the open / closed eyes is correct. There was a problem that it got worse.

【0007】

[Means for solving problems]

In the present invention, when the image data obtained by imaging the face is processed to determine the open / closed state of the eye, the face is photographed by an image capturing means to generate image data, and the eye position is detected from the entire image data. The position of the eye is detected by the means, and a predetermined area including the eye to be the tracking area of the eye is set by the eye tracking means based on the position of the eye, and the eye is tracked.

【0008】

Then, in the present invention, when a plurality of determination results indicating the open / closed state of the eye are generated by using at least two or more methods for the eyes being tracked in time series by the open / closed state determination means, the self-diagnosis means. Therefore, it is diagnosed whether or not the judgment result judged by each method is used for the comprehensive opening / closing eye judgment, and the judgment result determined to be used for the comprehensive opening / closing eye judgment is targeted for the comprehensive opening / closing eye judgment. The above-mentioned problems are solved by performing a comprehensive open / close eye determination based on the logical product by the determination means.

【0009】

[Effect of the invention]

According to the present invention, it is determined by the self-judgment means whether the determination result of each method of the open / close eye determination means is suitable for use in the open / close eye determination comprehensive determination means, and the determination is determined to be used in the diagnosis. Since the open / close eye determination comprehensive process is performed using only the results, it is possible to realize highly accurate open / close eye determination that allows for various light environments and differences in facial features.

【0010】

BEST MODE FOR CARRYING OUT THE INVENTION

Hereinafter, embodiments of the present invention will be described with reference to the drawings.

【0011】

The present invention is applied to, for example, an open / close eye determination device configured as shown in FIG. This open / close eye determination device is connected to a device that determines a plurality of states such as inattentiveness and dozing of the driver driving the vehicle and notifies the device, and indicates the open / closed state of the driver's eyes to the device. It sends information.

【0012】

[Configuration of open / close eye judgment device]

In this open / close eye determination device, for example, an open / close eye determination program is stored in a ROM (Read Only Memory) or the like (not shown), and the open / close eye determination program is stored in RAM (Central Processing Unit) or the like as needed by a CPU (Central Processing Unit) or the like (not shown). By executing using Random Access Memory etc.), each part as shown in Fig. 1 is realized.

【0013】

This open / close eye determination device has an image pickup device in which the driver's face image is attached to a column module or the like near the steering wheel, and an image capturing unit (image capturing means) 1 that generates a driver's face image (image data). To be equipped. The image capturing unit 1 generates, for example, a face image at predetermined intervals and sends it to the eye position detecting unit (eye position detecting means) 2. When the face image from the image capturing unit 1 is sent, the eye position detection unit 2 extracts the image area of the driver's eye using the face image, detects the eye coordinate value, and detects the eye coordinate value (eye tracking unit (eye tracking unit 2). Eye tracking means) Send to 3. The eye tracking unit 3 accumulates the eye coordinate values from the eye position detecting unit 2 sent in chronological order, tracks the eye position of the driver, and determines the open / closed state (open / closed state) of the image including the eye image. Judgment means) Send to 4.

【0014】

The open / closed state determination unit 4 performs an open / closed eye determination process based on the vertical width of the eye, the distance between the peak point of the upper eyelid and the eyebrow, and the shape change of the upper eyelid using the three-point method or the radius of curvature method. The open / close state determination unit 4 generates a plurality of determination results by performing a plurality of types of open / close eye determination processes and sends them to the open / close eye determination self-determination unit (self-judgment means) 5.

【0015】

Further, the open / close state determination unit 4 may send the determination result by a single method to the open / close eye determination self-determination unit 5 as needed. At this time, the open / close state determination unit 4 compares a plurality of determination results with the reference value for generating the determination result when sending the open / close eye determination self-determination unit 5 to the open / close eye determination self-determination unit 5. When sending to the part 5, the reference value for determining the open / closed state of the eye may be relaxed.

【0016】

The open / close eye judgment self-judgment unit 5 determines whether or not the judgment result of the open / close state judgment unit 4 is suitable as the judgment result used for the comprehensive judgment result of the open / close eye comprehensive judgment unit (open / close eye comprehensive judgment means) 6 in the subsequent stage. Perform self-diagnosis processing. The open / close eye determination self-judgment unit 5 sends the diagnosis result for each determination result to the open / close eye comprehensive determination unit 6.

【0017】

The open / close eye comprehensive determination unit 6 prepares in advance a plurality of determination results from the open / close state determination unit 4 by using the determination result determined by the open / close eye determination self-judgment unit 5 to be suitable for the comprehensive determination result. The open / close eye comprehensive judgment table is compared with the opened / close eye comprehensive judgment table, and the open / close eye comprehensive judgment process for generating the open / close eye comprehensive judgment result that integrates a plurality of judgment results is performed and output to a notification device or the like.

【0018】

Further, when the judgment result obtained by each opening / closing eye judgment processing is used, the opening / closing eye comprehensive judgment unit 6 sets a judgment threshold value (reference value) in each opening / closing eye judgment processing according to the number of judgment results to be used. It is desirable to change. Specifically, the open / close eye comprehensive judgment unit 6 determines that the judgment result used for the comprehensive open / close eye judgment is closed when the number of judgment results decreases due to the diagnosis of the open / close eye judgment self-judgment unit 5. Change the reference value to make it difficult.

【0019】

[Open / close eye determination process in open / close eye determination result generation process]

Next, the processing content of each open / close eye determination process performed by the open / close state determination unit 4 described above will be described.

【0020】

"Open / close eye determination process based on the vertical width of the eye" In the open / close eye determination process based on the vertical width of the eye, the open / close state determination unit 4 first uses the reference coordinates of the eye generated by the eye tracking unit 3 as a reference for the face image. A tracking area corresponding to the size of the eye, which is obtained from the angle of view of the image pickup device for capturing 11 images and the distance between the driver's face and the image pickup device, is set. Next, the open / close state determination unit 4 calculates a binarization threshold value suitable for generating extraction points by creating a histogram for each concentration value in the set tracking area. Next, the open / closed state determination unit 4 uses the calculated binarization threshold value to set the image portion having the maximum vertical width of the binarized eye image as the eye opening value.

【0021】

As a result, the open / closed state determination unit 4 increases the opening value h when the vertical width becomes large when the eyes are opened, as shown in FIG. 2 (a). On the other hand, the open / closed state determination unit 4 reduces the opening value h when the vertical width becomes small when the eyes are closed, as shown in FIG. 2 (b). Then, the open / closed state determination unit 4 compares the opening value h with the preset opening threshold (reference value), and when the opening value h is larger than the opening threshold, the eye is open. When the opening value h is smaller than the opening threshold value, the judgment result that the eye is closed is generated.

【0022】

Here, when determining the open / closed state of the eye, the open / closed state determination unit 4 stores the vertical width of the eye image at the time of opening or closing the eye as personal data and performs learning processing to change the opening threshold. It is possible to deal with cases where the size of the eyes differs for each individual.

【0023】

However, as shown in Table 1 below, the opening / closing eye determination process based on the vertical width of the eye may determine that the eye is open even if the eye is closed, or that the eye is closed just by narrowing the eye. It may not be possible to deal with it simply by changing the opening / closing threshold. As shown in FIG. 3A, the reason why the eye is determined to be open even when the eyes are closed is that the tracking area 12 including the eyes is in the shade or the tracking area 12 including the eyes is at night. It may be under near-infrared light illumination. In such a state, the contrast of the tracking region 12 becomes weak, and it becomes difficult to separate the shadow under the eye and the shape of the eye based on the density, especially when the eyes are closed.

【0024】

[table 1]<img file="JP2004192552A_D0001.tif" />That is, when the image portion corresponding to the shadow 13 under the outer corner of the eye shown in FIG. 3 (b) is binarized, the image portion corresponding to the shadow 13 and the image portion corresponding to the eye are obtained as shown in FIG. 3 (c). Can not be separated and become one. In such a state, the vertical width CL_h of the eye corresponding to the opening value h becomes large even though the eye is closed, and it is determined that the eye is in the open state. In addition, the reason why it is determined that the eyes are closed only by squinting the eyes is when the eyes are squinted with a smile or dazzling. That is, from the vertical width OP_h of the eye in the state of gazing forward as shown in FIG. 4 (a), when the driver smiles, it is wider than the vertical width OP_h of the eye as shown in FIG. 4 (b). The vertical width of the small eye is SMILE_h. As a result, it is determined that the eyes are closed even though the person is gazing at the front.

【0025】

"Open / close eye determination process based on the distance between the peak point of the upper eyelid and the eyebrow" In the open / close eye determination process based on the distance between the peak point of the upper eyelid and the eyebrow, the open / close state determination unit 4 is first included in the tracking area 12. The shape of the eye is cut out in a binarized state of the face image 11, and the peak point of the upper eyelid is obtained from the shape. Further, the open / closed state determination unit 4 obtains an extraction point having the same coordinate value in the X-axis direction as the peak point of the upper eyelid as a representative coordinate point from the data corresponding to the eyebrows. As a result, the open / closed state determination unit 4 subtracts the Y-axis coordinates of the representative coordinate points of the eyebrows from the Y-axis coordinates of the peak points of the upper eyelids to calculate the peak points of the upper eyelids, the eyebrows, and the vertical distance.

【0026】

By performing such an open / closed eye determination process, as shown in FIG. 5, the distance h1 between the upper eyelid and the eyebrow becomes smaller when the eye is opened, and the distance h2 between the upper eyelid and the eyebrow becomes larger when the eye is closed. Then, the open / closed state determination unit 4 compares the obtained distance between the upper eyelid and the eyebrow and the preset opening threshold value, and when the distance between the upper eyelid and the eyebrow is smaller than the opening threshold value, A determination result that the eye is open is generated, and a determination result that the eye is closed is generated when the distance between the upper eyelid and the eyebrow is larger than the opening threshold.

【0027】

However, as shown in Table 1 above, in the opening / closing eye determination process based on the distance between the peak point of the upper eyelid and the eyebrows, it may be determined that the eyes are closed even if the eyes are open. The reason why the eyes are judged to be closed even if the eyes are open is that, for example, when driving with a certain degree of tension, the distance between the upper eyelids and the eyebrows reduces the tension of the driver and loosens the facial muscles. The position of the eyebrows is raised, the distance between the upper eyelid and the eyebrow is widened, and the distance between the upper eyelid and the eyebrow is increased. In such a case, a determination result is generated that the driver is in the closed eye state even though the driver is in the forward gaze state. Here, although there are individual differences in the movement of the eyebrows, in the case of a driver with a large movement, the stroke amount for opening and closing the eyes may be exceeded.

【0028】

Further, as shown in FIG. 6, the reason why the opening / closing eye determination process due to the distance between the upper eyelid and the eyebrow is not stable is that the driver wears the glasses 21. That is, as the data detected from the face image 11, the eyebrow data EB and the frame data EG of the glasses exist, and both data EB and EG exist at relatively close positions, or the eyebrow data EB is the lens of the glasses 21. When it is reflected on the surface, the contrast of the eyebrows may be temporarily weakened and it may not be possible to extract data accurately. In such a case, the eyebrow data EB cannot be tracked stably, and the eyebrow data EB and the frame of the spectacles 21 may be replaced.

【0029】

Therefore, the value calculated as the distance between the upper eyelid and the eyebrows is also the distance between the upper eyelid and the eyebrows EB_d and the distance between the upper eyelid and the frame of the eyebrows 21 EG_d. If the tracking area of the eyebrows moves to the frame of the eyebrows 21 while learning the value in the open eye state, the interval is widened, so that the eyebrow is erroneously determined to be closed even in the open eye state.

【0030】

Further, when the spectacles 21 is a metal frame, the data of the eyebrows may be divided into upper and lower parts by the frame of the spectacles, and even in such a state, the data under the eyebrows divided by the spectacles is stored in the lens portion. Since reflection may occur, there is a problem similar to the above-mentioned spectacle frame and eyebrow data.

【0031】

Further, as a cause of the unstable opening / closing eye determination process due to the distance between the upper eyelid and the eyebrows, there is a case where the face is tilted in the pitch direction in addition to the case where the glasses are worn. That is, as shown in FIG. 7, the camera 31 installed in front of the driver recognizes the distance between the upper eyelid and the eyebrow at a distance of EB_d. At this time, when the face is not tilted, the eye position is E, the eyebrow position is EB1, and the eye position is intentionally adjusted with the face tilted backward, as shown in Fig. 8. Assuming that the position of the eyebrows is EB2, the distance between the upper eyelid and the eyebrows is EB_d1 when the face is not tilted, whereas it is EB_d2 when the face is tilted backward. In this state, the distance between the upper eyelid and the eyebrows when the eyes are opened is further reduced, so even if the eyes are closed, it may not be possible to determine that the eyes are closed.

【0032】

"Open / close eye determination process by changing the shape of the upper eyelid" In the open / close eye determination process by changing the shape of the upper eyelid, the open / close state determination unit 4 determines the angle of view of the imaging device that captures the face image 11 and the driver's face. A tracking area 12 corresponding to the size of the eye, which is obtained from the distance to the image pickup device in terms of market value, is set. Next, the open / closed state determination unit 4 generates data indicating the upper eyelid line by performing edge detection processing within the set tracking area 12.

【0033】

Next, when the three-point method is used, the open / closed state determination unit 4 connects both end points with the left end point of the data indicating the upper eyelid line as the inner corner point 41A and the right end point as the outer corner point 41C, as shown in FIG. Find the perpendicular length of the peak point 41B of the upper eyelid with respect to the straight line. As a result, the open / closed state determination unit 4 obtains a calculation result in which the perpendicular length h3 becomes longer when the eyes are opened as shown in FIG. 9 (a) and the perpendicular length h4 becomes shorter when the eyes are closed as shown in FIG. 9 (b). Then, the open / close state determination unit 4 compares the obtained perpendicular line length with the preset opening threshold value, and determines that the eye is in the open eye state when the perpendicular line length is larger than the opening threshold value. It is generated, and when the vertical line length is smaller than the opening threshold value, a determination result that the eye is closed is generated.

【0034】

On the other hand, when the open / closed state determination unit 4 uses the radius of curvature method, as shown in FIG. 10, the open / closed state determination unit 4 calculates an approximate expression of a quadratic curve for each edge extraction point 51 of the upper eyelid, and calculates the radius of curvature of the upper eyelid. Ask for. As a result, the open / closed state determination unit 4 obtains a calculation result in which the radius of curvature becomes small when the eyes are opened as shown in FIG. 10 (a) and the radius of curvature becomes large when the eyes are closed as shown in FIG. 10 (b). Then, the obtained radius of curvature is compared with the preset opening threshold value, and when the radius of curvature is smaller than the opening threshold value, a determination result that the eye is open is generated, and the radius of curvature is the opening degree. If it is larger than the threshold value, a determination result indicating that the eye is closed is generated.

【0035】

However, as shown in Table 1 above, in the open / closed eye determination process based on the shape change of the upper eyelid using the three-point method and the radius of curvature method, it may be determined that the eyes are closed even if the eyes are open. The reason why it is determined that the eyes are closed even if the eyes are open is that when a part of the eyes is exposed to the direct sunlight, the image parts of the inner and outer corners of the eyes are overexposed. That is, when the inner corner of the eye is overexposed with respect to the normal eye image 61 shown in FIG. 11 (a), the inner corner of the eye image 61 becomes the overexposed portion 62 as shown in FIG. 11 (b), and the outer corner of the eye is formed. When is overexposed, the outer corner of the eye of the eye image 61 becomes the overexposed portion 62 as shown in FIG. 11 (c). As a result, the edge detection process of the overexposed portion 62 becomes difficult, and the accurate inner corner point 41A, peak point 41B and outer corner point 41C cannot be detected, or the edge extraction point 51 cannot be detected.

【0036】

Further, as a limited problem of the radius of curvature method, it may be determined that the eye is open when the eye is closed. The reason why the eyes are judged to be open even if the eyes are closed is due to individual differences in the upper eyelids, and when the driver has double eyelids, the edge line consisting of the edge extraction points 51 is exactly the upper eyelids as shown in FIG. This is because it is difficult to follow the line of, and therefore the correlation between the radius of curvature of the approximate curve and the real line connecting a plurality of extraction points becomes poor.

【0037】

"Advantages of each open / close eye determination process" First, the advantages of the open / close eye determination process based on the vertical width of the eye are as shown in Table 2 below, in situations such as when a part of the eye is exposed to direct sunlight. Further, as shown in FIG. 13, as long as the eyeball portion of the eye image 23 is not overexposed, it is possible to recognize the vertical width OP_h of the eye and obtain a correct open / closed eye determination result.

【0038】

[Table 2]<img file="JP2004192552A_D0002.tif" />In addition, the advantages of the open / closed eye determination process based on the distance between the peak point of the upper eyelid and the eyebrows are as shown in Table 2 and Fig. 14 (a) above, when the eyes are squinting due to smiling or dazzling. As shown in 14 (b), it is possible to obtain a correct open / closed eye determination result by distinguishing from a sleepy state in which the eyes are closed. That is, when you smile or dazzle and squint your eyes, the muscles under your eyes are raised, so the distance D between your upper eyelids and eyebrows does not widen, but when you are sleepy and close your eyes. As the upper eyelid position is lowered, the distance between the upper eyelid and the eyebrows is widened. Therefore, in this open / close eye determination process, it is possible to distinguish whether the state of squinting is smiling, dazzling, or sleepy.

【0039】

In addition, as shown in Table 2 above, the advantage of the open / closed eye determination process due to the change in the shape of the upper eyelid is that the eyes are shaded as shown in FIG. 15 (a) in both the three-point method and the radius of curvature method. Even when the contrast is low with the eyes under near-infrared light at night, the three-point method can be used using the images shown when the eyes are closed (Fig. 15 (b)) and when the eyes are open (Fig. 15 (c)). It is possible to determine the open / closed eye with high accuracy using the radius of curvature method. Here, when the contrast of the eye is low, it becomes difficult to set the binarization threshold value in order to correctly obtain the vertical width of the eye. In particular, when the eyes are closed, the density gradation between the eye portion and the shadow portion under the eye is close to each other, so that the density cannot be separated and the vertical width of the eye does not decrease. However, even in such a situation, the shape of the edge line of the upper eyelid is not easily affected by the shape, so that it is possible to correctly determine the open / closed eye.

【0040】

In each of these open / close eye determination processes, the open / close eye determination self-judgment unit 5 diagnoses whether or not to use in the open / close eye determination comprehensive process of the open / close eye comprehensive determination unit 6 in the subsequent stage so as to make the best use of each advantage.

【0041】

[Open / close eye judgment result generation process by open / close eye judgment device]

Next, the flowcharts of FIGS. 16 to 18 regarding the opening / closing eye determination result generation process for determining the opening / closing state of the driver's eye by the opening / closing eye determination device in which the opening / closing state determination unit 4 performs each opening / closing eye determination process as described above. Will be described with reference to.

【0042】

In the open / closed eye determination device, first, in step S1, a face image is generated in response to input of image data in the image capturing unit 1, and processing proceeds to step S2 and subsequent steps.

【0043】

In steps S2 to S5, the eye position detection unit 2 detects the eye coordinate value from the face image, and the eye tracking unit 3 performs preprocessing until the eye tracking is started. That is, first, in step S2, the eye position detection unit 2 extracts the density feature amount from the face image which is the data continuously generated.

【0044】

At this time, the eye position detection unit 2 detects the density value of the pixel in the vertical direction (Y direction) in the face image 11 as shown by the vertical line in FIG. 19, and the density value becomes a black pole. And, the pixels (extraction points) whose concentration change rate exceeds a predetermined threshold value up to the inflection point are extracted. By executing such processing over the entire face image 11, the eye position detection unit 2 has the right eyebrow portion, the left eyebrow portion, the right eye portion, the left eye portion, and the nose portion as shown in FIG. , Obtain an extraction point corresponding to the mouth part.

【0045】

Then, the eye position detection unit 2 groups the extraction points that are adjacent in the X direction and are close to each other in the Y direction, and obtains continuous data continuous in the X direction showing the density characteristics. .. That is, in the eye position detection unit 2, as shown in FIG. 21, continuous data 71A corresponding to the right eyebrow portion, continuous data 71B corresponding to the left eyebrow portion, continuous data 71C corresponding to the right eye portion, and the left eye portion. The corresponding continuous data 71D, the continuous data 71E corresponding to the nose part, and the continuous data 71F corresponding to the mouth part are obtained.

【0046】

In the next step S3, the eye position detection unit 2 selects continuous data 71 corresponding to the right eye and the left eye to detect the position of the eye. At this time, the eye position detection unit 2 zones the image region in which the continuous data 71 composed of each feature amount appears in the Y direction, and detects the position of the eye from the relative positional relationship of each continuous data 71.

【0047】

Specifically, as shown in FIG. 22, the eye position detection unit 2 sets the zone 72 so as to include the continuous data 71A and 12C, sets the zone 73 so as to include the continuous data 71E and 71F, and continuously sets the zone 73. Set zone 74 to include data 71B, 71D. Then, in the eye position detection unit 2, the median value in the X-axis direction of the continuous data 71C included in the zone 72 having the smallest X-axis coordinate value among the zone 72, the zone 73, and the zone 74 and having the smallest Y-axis coordinate value is used as the right eye. The coordinate value is 75C, and the median value in the X-axis direction of the continuous data 71A whose Y-axis coordinate value is smaller than that of the continuous data 71C is the right eyebrow coordinate value 75A. Further, in the eye position detection unit 2, the median value in the X-axis direction of the continuous data 71E included in the zone 73 whose X-axis coordinate value is larger than that of the zone 72 and whose Y-axis coordinate value is small is set to the nose coordinate value 75E, and the continuous data. Let the median value of continuous data 71F, which has a larger Y-axis coordinate value than 71E, in the X-axis direction be the mouth coordinate value 75F. Further, in the eye position detection unit 2, the median value in the X-axis direction of the continuous data 71D included in the zone 74 having the largest X-axis coordinate value among the zone 72, the zone 73, and the zone 74 and having the large Y-axis coordinate value is the left eye. The coordinate value is 75D, and the median value in the X-axis direction of the continuous data 71B whose Y-axis coordinate value is smaller than that of the continuous data 71D is the left eyebrow coordinate value 75B.

【0048】

In the next step S4, the eye position detection unit 2 stores the right eye coordinate value 75C and the left eye coordinate value 75D obtained in step S3 in a memory (not shown) as the reference coordinate value of the eye, so that the eye tracking unit 3 Can be used by.

【0049】

In the next step S5, the eye tracking unit 3 sets a small area including the right eye coordinate value 75C and a small area including the left eye coordinate value 75D stored in step S4, and sets each small area as the eye tracking area. Set as. As a result, the open / closed eye determination device completes the preprocessing until the transition to eye tracking, and proceeds to the loop process in step S6 to determine the open / closed state of the eye.

【0050】

In step S6, the image capturing unit 1 newly inputs the face image 11 and proceeds to step S7.

【0051】

In step S7, the eye tracking unit 3 performs the same process as the process of extracting continuous data in step S2 in each tracking area set in step S5 using the face image 11 input in step S6. This is limited to extracting continuous data and proceeding to step S8.

【0052】

In this step S7, the eye tracking unit 3 performs a process of detecting the density value of the pixels in the vertical direction (Y direction) and extracting the extraction points for the entire tracking area 12 as shown in FIG. 23 (a). Then, continuous data for the eye image 76A is created. Here, in order to improve the detection accuracy of continuous data, the eye tracking unit 3 has a scanning interval density in the X-axis direction and the Y-axis direction as compared with the case where the extraction point is detected from the face image 11 in step S2. To raise.

【0053】

In the next step S8, the eye tracking unit 3 uses the continuous data extracted in step S7 to obtain the reference coordinates of the eye and proceed to step S9.

【0054】

In step S8, in the eye tracking unit 3, since the tracking area 12 is very small as compared with the case where the position of the eye from the entire face image 11 is detected, only the eye image 76A exists in the tracking area 12. The center coordinate 77A of the eye is specified by finding the representative coordinate point of the continuous data. Further, in the eye tracking unit 3, for example, when a plurality of continuous data appear in the tracking area 12, the center coordinates of the eye are obtained from the average density value of each extraction point constituting the continuous data and the shape feature of the continuous data. Identify 77A.

【0055】

In step S9, the eye tracking unit 3 stores the center coordinate 77A specified in step S8 as the reference coordinate in the memory, and proceeds to step S10.

【0056】

In step S10, the eye tracking unit 3 updates the position of the tracking area 12 so that the center coordinate 77A stored in step S9 becomes the center position of the tracking area 12, and proceeds to step S11.

【0057】

In this step S10, in the eye tracking unit 3, when the tracking area 12 shown in FIG. 23 (a) is set in step S5, the center coordinates 77 and the center position of the tracking area 12 coincide with each other. After that, when a new face image 11 is captured in step S6 and the position changes from the eye image 76A to the eye image 76B as shown in FIG. 23 (b), the eye is changed as shown in FIG. 23 (c). The center coordinate 77B of the image 76B is brought closer to the center position of the tracking area 12, and the center coordinate 77B of the eye image 76B is further brought closer to the center position of the tracking area 12 as shown in FIG. 23 (d). By performing such processing, the position of the tracking area 12 is continuously updated based on the center coordinate 77B detected in the tracking area 12.

【0058】

Here, the eye tracking unit 3 sets the tracking area 12 so as to be the minimum necessary image area according to the amount of movement of the eyes with respect to the capture interval of the face image 11, so that the processing time of steps S7 and S8 can be set. It is desirable to shorten the length and improve the followability of the eye.

【0059】

In step S11, the open / close state determination unit 4 performs the open / close eye determination process based on the vertical width of the eye described above, stores the determination result obtained by the open / close eye determination process, and proceeds to step S12. ..

【0060】

In the next steps S12 and S13, the open / close eye determination self-judgment unit 5 performs a self-diagnosis process for determining whether or not the open / close eye determination process based on the vertical width of the eye performed in step S11 is normally performed. Execute.

【0061】

That is, as a self-diagnosis of the opening / closing eye determination based on the vertical width of the eye, first, in step S12, the opening / closing eye determination self-judgment unit 5 determines the vertical width of the eye used for the determination in step S11 of FIG. It is stored in a plurality of memories, and the process proceeds to step S13.

【0062】

In step S13, the open / close eye determination self-judgment unit 5 determines whether or not the average value and standard deviation of the vertical widths of a predetermined number of eyes satisfy the predetermined conditions. Under this predetermined condition, as long as the tracking object is the eye, the range in which the vertical width changes depending on the opening and closing eyes is limited. Therefore, even if several vertical width values of the closed eye are included due to blinking or the like, the vertical width at the time of substantially opening the eye The value close to the width is the average value, and the standard deviation is small. In addition, even when considering the situation where the arousal level is low, it is possible to perform a self-diagnosis of whether or not the tracked object is an eye simply by setting the average value, which is a predetermined condition, slightly smaller. At the level.

【0063】

In step S13, if it is determined that the mean value and standard deviation of the vertical width of a predetermined number of eyes stored in step S12 do not satisfy the predetermined conditions, at least the process is returned to step S1 assuming that the tracking target is not an eye. In step S1 and subsequent steps, the eye position is detected again from the entire image. On the other hand, in step S13, when it is determined that the average value and standard deviation of the vertical widths of the predetermined number of eyes stored in step S12 satisfy the predetermined conditions, it is determined that the change in the vertical width values of the eyes is the eye. Then, the process proceeds to step S14 of FIG.

【0064】

In step S14, the open / close state determination unit 4 performs the open / close eye determination process based on the distance between the peak point of the upper eyelid and the eyebrows described above, and stores the determination result obtained by the open / close eye determination process. Proceed to step S15.

【0065】

In steps S15 to S21 and S22, the open / close eye determination self-judgment unit 5 determines whether or not the open / close eye determination process based on the distance between the peak point of the upper eyelid and the eyebrows is normally performed. I do. That is, the self-diagnosis of the open / closed eye determination process based on the distance between the peak point of the upper eyelid and the eyebrows is performed when three types of diagnostic processes of step S15, step S17 and step S19 are performed and all the diagnostic processes are satisfied. In step S22, it is assumed that the open / close eye determination process based on the distance between the peak point of the upper eyelid and the eyebrows is used.

【0066】

That is, first, in step S15, how accurately the eyebrow data could be detected for the image frame processed for the tracking area of the eye within a predetermined time by the open / close eye determination self-judgment unit 5. It is calculated as the appearance rate of data, and it is determined whether or not the appearance rate satisfies a predetermined condition. This predetermined condition is set in advance, and a predetermined value for determining that the eyebrow data can be accurately detected is set.

【0067】

Then, when the open / close eye determination self-judgment unit 5 determines that the appearance rate of the eyebrow data is equal to or less than a predetermined value, the driver's eyebrows are thin or the eyebrow data is difficult to detect due to light environment conditions. In this case, it is determined that the eyebrows are hidden by the bangs and cannot be detected, and the process proceeds to step S20.

【0068】

In step S20, the open / close eye determination self-judgment unit 5 is set so that the determination result of the open / close eye determination process based on the distance between the peak point of the upper eyelid and the eyebrows is not used in the open / close eye determination comprehensive process, and step S21. Proceed to processing.

【0069】

In step S21, the open / close eye determination self-judgment unit 5 decided not to use the determination result based on the distance between the peak point of the upper eyelid and the eyebrows in the open / close eye determination comprehensive process in step S20. In the opening / closing eye determination process based on the vertical width and the opening / closing eye determination process based on the shape change of the upper eyelid, the criteria for determining the closed eye are tightened, the degree of determining the closed eye is set to the normal state, and the process proceeds to step S23 in FIG. .. Such processing in step S21 is performed in a plurality of non-interrelated image feature quantities (in this example, the "vertical width of the eye" and the "distance between the upper eyelid and the eyebrow" in the subsequent open / close eye determination comprehensive processing. ) Is used as the calculation target, and by taking the logical product of the judgment result of closing the eyes, it is judged that the eyes are closed comprehensively, and the criteria for judging that the eyes are closed are relaxed in each opening / closing eye judgment process. By assuming that it is set. In addition, in this open / close eye determination comprehensive processing, a plurality of interrelated image feature quantities (in this example, "change in the shape of the upper eyelid using the three-point method" and "change in the shape of the upper eyelid using the radius of curvature method". ") May be used as the calculation target, and a comprehensive judgment may be made by taking the logical sum of the judgment results with the closed eyes.

【0070】

On the other hand, in step S16 when it is determined in step S15 that the appearance rate of the eyebrow data is equal to or higher than a predetermined value, the open / close eye determination self-judgment unit 5 stores the appearance positions of a plurality of eyebrows in time series in the memory. Then, the process proceeds to step S17.

【0071】

In step S17, the open / close eye determination self-judgment unit 5 determines whether or not the average value and standard deviation of the appearance position coordinates (x, y) of a plurality of eyebrows satisfy the predetermined conditions, thereby determining whether or not the eyebrows Determine the stability of the data appearance position. That is, the position of the eyebrows is less affected by the opening and closing eyes, and does not become larger than the variation in the appearance position of the eye data unless the entire face is moved. Therefore, if the stability of the appearance position of the eyebrow data is low, it is judged that the tracking stability of the eyebrows is not stable, and if the driver's eyebrows are thin or the eyebrow data is difficult to detect due to the light environment conditions. If it is, it is recognized that the eyebrows are hidden by the bangs, or that there is data that causes noise such as wrinkles or shadows near the eyebrows.

【0072】

Then, when the open / close eye determination self-judgment unit 5 determines that the appearance position of the eyebrow data is unstable, the processing of steps S20 and S21 is performed as described above to obtain the peak point of the upper eyelid. It is set that the judgment result based on the distance from the eyebrows is not used for the open / close eye judgment comprehensive processing, and the processing proceeds to step S23 in FIG.

【0073】

On the other hand, in step S18 when it is determined that the appearance position of the eyebrow data is stable, the open / close eye determination self-judgment unit 5 stores a plurality of time-series upper eyelid and eyebrow spacing values in the memory. Then, the process proceeds to step S19.

【0074】

In step S19, the open / close eye determination self-judgment unit 5 determines whether or not the average value and standard deviation of the distance between the plurality of upper eyelids and the eyebrows satisfy the predetermined conditions. This predetermined condition is a constant average value even if several eye-closing interval values are included due to blinking or the like, because the range in which the interval changes depending on the opening and closing eyes is limited as long as the tracking objects are the eyes and the eyebrows. And a small standard deviation. Here, to explain using FIG. 6, in step S18, the distance between the upper eyelid and the eyebrow at the time of opening the eye is memorized as EB_d, and if the eyebrow can be traced correctly, the distance between the upper eyelid and the eyebrow is divided. The mean value is almost the same value, and the standard deviation is also a very small value.

【0075】

After that, when the eyebrow tracking target was moved to the frame of the eyebrows 21, the distance between the upper eyelid and the eyebrow became EG_d, the average value became large, and the predetermined condition was not satisfied, so there was an error in tracking the eyebrow data. Can be determined. Further, in the determination based on the standard deviation, it is possible to determine an unstable state in which the frames of the eyebrows and the spectacles 21 are alternately tracked in a short cycle.

【0076】

Also, as shown in Fig. 8, if the distance between the eyelids and the eyebrows when the eyes are opened when the face is straight is memorized as EB_d1, and then the face is tilted backward, the distance between the upper eyelids and the eyebrows is EG_d2, which is the average. Since the value becomes small and the predetermined condition is not satisfied, it can be determined that there was an error in tracking the eyebrow data.

【0077】

Since the average value included in the predetermined conditions is used as a reference for the distance between the upper eyelid and the eyebrows when the eyes are opened, the minimum value is set near the distance between the upper eyelids and the eyebrows when the eyes are opened, and the maximum value is set for the eyes. It is set in consideration of the amount that the distance between the upper eyelid and the eyebrows is widened by closing. In addition, the standard deviation value included in the predetermined condition is set with reference to the value when the eyebrows are stably tracked.

【0078】

If it is determined in step S19 that the average value and standard deviation of the distance between the upper eyelid and the eyebrow do not satisfy the predetermined conditions, the process of steps S20 and S21 is performed as described above to obtain the peak point of the upper eyelid. It is set that the judgment result based on the distance from the eyebrows is not used for the open / close eye judgment comprehensive processing, and the processing proceeds to step S23 in FIG.

【0079】

On the other hand, if it is determined in step S19 that the average value and standard deviation of the distance between the upper eyelid and the eyebrow satisfy the predetermined conditions, the process proceeds to step S22, and the opening and closing eyes are opened and closed depending on the distance between the peak point of the upper eyelid and the eyebrow. It is set that the judgment result of the judgment process is used in the open / close eye judgment comprehensive process, and the process proceeds to step S23 in FIG.

【0080】

In step S23, the open / closed state determination unit 4 performs an open / close eye determination process based on a change in the shape of the upper eyelid using the three-point method, stores the determination result obtained by the open / close eye determination process, and steps. Proceed to S24.

【0081】

In step S24, the open / close eye determination self-judgment unit 5 stores the time-series distances between the inner and outer corners of the eyes in the memory obtained in step S23, and proceeds to step S25.

【0082】

In step S25, the open / close eye determination self-judgment unit 5 calculates the average value and standard deviation of the distances between the inner and outer corners of the eyes stored in step S24, and determines the average value and standard deviation. Determine if the conditions are met. This predetermined condition is determined based on the distance between both points of the inner and outer corners of the eye when the eye is correctly detected as an image feature amount. That is, as shown in FIGS. 11 (b) and 11 (c), when a part of the eye is exposed to the direct sunlight and there is an overexposed portion 62, the average distance between both points of the inner and outer corners of the eye is average. Since the value becomes smaller, the use of the judgment result due to the change in the shape of the upper eyelid is restricted.

【0083】

Therefore, if it is determined in step S25 that the predetermined condition is not satisfied, the process proceeds to step S31, and the open / close eye determination self-judgment unit 5 determines the open / close eye determination result of the open / close eye determination process due to the change in the shape of the upper eyelid. Set not to be used for comprehensive processing and proceed to step S32.

【0084】

In step S32, by applying the same processing as in step S21 by the open / close eye determination self-judgment unit 5, the eye is closed in the open / close eye determination process based on the vertical width of the eye and the open / close eye determination process based on the distance between the upper eyelid and the eyebrow. The judgment criteria are tightened, and the process proceeds to step S35.

【0085】

On the other hand, if it is determined in step S25 that the predetermined condition is satisfied, the process proceeds to step S26. In step S26, the open / close eye determination self-judgment unit 5 sets that the determination result of the open / close eye determination process based on the shape change of the upper eyelid using the three-point method is used, and proceeds to step S27.

【0086】

In step S27, the open / closed state determination unit 4 performs an open / close eye determination process by changing the shape of the upper eyelid using the radius of curvature method, stores the determination result obtained by the open / close eye determination process, and steps. Proceed to S28.

【0087】

In step S28, the open / close eye determination self-judgment unit 5 stores in the memory the correlation coefficients of the plurality of time-series upper eyelid approximation curves and the actual lines obtained in step S27, and processes them in step S29. To proceed.

【0088】

In step S29, the open / close eye determination self-judgment unit 5 calculates the average value and standard deviation of the plurality of correlation coefficients stored in step S28, and whether or not the average value and standard deviation satisfy the predetermined conditions. Is determined.

【0089】

When it is determined in step S29 that the average value of the correlation coefficient is small, or when it is determined that the standard deviation is large and the predetermined condition is not satisfied, the edge line of the upper eyelid is determined as described with reference to FIG. It is determined that the trace cannot be performed accurately due to the influence of the double eyelids, etc., and the process proceeds to step S33.

【0090】

In step S33, the open / close eye determination self-judgment unit 5 is set so that the determination result of the open / close eye determination process due to the shape change of the upper eyelid using the radius of curvature method is not used for the open / close eye determination comprehensive process. Proceed to S34.

【0091】

In step S34, the open / close eye determination self-judgment unit 5 applies the same processing as in step S21 to perform eye closure in the open / close eye determination process based on the vertical width of the eye and the open / close eye determination process based on the distance between the upper eyelid and the eyebrow. The judgment criteria are tightened, and the process proceeds to step S35.

【0092】

On the other hand, in step S30 when it is determined that the predetermined condition is satisfied in step S29, it is set to use the judgment result of the open / closed eye judgment process due to the shape change of the upper eyelid using the radius of curvature method, and the process is performed in step S35. To proceed. In this example, in the diagnostic process of the open / closed eye determination process due to the change in the shape of the upper eyelid, the case of diagnosing using the average value of the correlation coefficient and the standard deviation has been described, but the correlation coefficient for each processing frame is used. It may be diagnosed whether or not to use the determination result of the opening / closing eye determination process due to the change in the shape of the upper eyelid.

【0093】

In step S35, the open / close eye comprehensive determination unit 6 performs the open / close eye determination process based on the distance between the upper eyelid and the eyebrows by the above-mentioned diagnostic process by the open / close eye determination self-determination unit 5, and the open / close eye determination process based on the shape change of the upper eyelid. It is determined whether or not both determination results are set as unused. If both judgment results are set to be unused, the process proceeds to step S1 to detect the position of the eye from the entire face image 11 again.

【0094】

On the other hand, when it is determined that both determination results are not unused, the process proceeds to step S36, and the open / close eye comprehensive determination unit 6 determines that the determination result is usable or unused for each open / close eye determination process. While recognizing whether it is set, the determination result of the open / close eye determination process determined to be usable is acquired from the open / close state determination unit 4, the open / close eye determination comprehensive process is executed, and the open / close eye comprehensive determination process is performed. The obtained comprehensive judgment result is output to a notification device (not shown) or the like, and the process is returned to step S6. The processing content of this open / close eye comprehensive determination process will be described later.

【0095】

[Comprehensive processing of open / close eye judgment by open / close eye comprehensive judgment unit 6]

Next, the processing content of the open / close eye determination comprehensive processing performed in step S36 described above will be described. In this open / close eye comprehensive determination process, the open / close eye comprehensive determination table shown in FIGS. 24 and 25 created in consideration of the problems of each open / close eye determination process described above is used.

【0096】

First, as shown in FIG. 24, when a part of the eye is exposed to direct light when the eye is opened, the edge line is shortened due to overexposure of the inner and outer corners of the eye in the opening / closing eye determination process due to the change in the shape of the upper eyelid. It may be determined that the eyelids are closed, or that the eyelids are unused in the self-diagnosis process. On the other hand, the open / close eye comprehensive determination unit 6 correctly determines that the open / closed eye determination process is based on the vertical width of the eye and the open / close eye determination process is based on the distance between the upper eyelid and the eyebrow. It is judged that the eyes are open by the open / close eye judgment comprehensive processing.

【0097】

In addition, since a part of the eye is exposed to direct light, even if the driver is dazzling and narrows his eyes, the correct judgment can be made by the open / closed eye judgment based on the distance between the upper eyelid and the eyebrows. The result is not wrong.

【0098】

Further, in the open / close eye comprehensive determination unit 6, when a part of the eye is exposed to direct light when the eye is closed, the open / close eye determination process based on the vertical width of the eye, the open / close eye determination process based on the distance between the upper eyelid and the eyebrow, and the open / close eye determination process of the upper eyelid Since the judgment result is unused in the diagnostic processing for all the opening / closing eye judgment processing due to the shape change, the comprehensive judgment result with the closed eye is generated.

【0099】

Furthermore, in the open / closed eye comprehensive determination unit 6, when the contrast of the eyes is weak when the eyes are opened, the criteria for determining the closed eyes in the open / closed eye determination process based on the vertical width of the eyes are to ensure that the eyes are closed when the eyes are closed. To alleviate. As a result, the eyes may be determined to be closed by the opening / closing eye determination process based on the vertical width of the eye, but the open / close eye comprehensive determination unit 6 performs the opening / closing eye determination process based on the distance between the upper eyelid and the eyebrows, and the opening / closing eye due to the shape change of the upper eyelid. Assuming that the correct determination result that the determination process is eyebrow is obtained, the comprehensive determination result that the eyebrow is open is generated.

【0100】

Furthermore, the open / close eye comprehensive determination unit 6 relaxes the criteria for determining that the eye is closed in the open / close eye determination process based on the vertical width of the eye when the contrast of the eye is weak when the eye is closed. As a result, the open / closed eye comprehensive determination unit 6 can correctly generate the determination result of the closed eye by the open / close eye determination process based on the vertical width of the eye, and generates the comprehensive determination result of the closed eye.

【0101】

Furthermore, in the open / closed eye comprehensive judgment unit 6, when the eyes are narrowed with a smile, the open / closed eye is judged by the vertical width of the eye, and the judgment resolution is low even if the shape of the upper eyelid is changed (3-point method). It may be judged that the eyes are closed, but since the opening and closing eyes are judged to be open by the distance between the upper eyelids and the eyebrows and the shape change of the upper eyelids (radius of curvature method), the comprehensive opening and closing eyes are judged. There is no mistake in the judgment.

【0102】

Furthermore, in the open / close eye comprehensive determination unit 6, when the eyes are dazzling and the eyes are narrowed, the opening change of the eyes is similar to the case where the eyes are smiled and the eyes are narrowed. It can be performed.

【0103】

Furthermore, when the facial expression is loosened due to a decrease in the degree of tension, a determination result of closed eyes may be generated by the opening / closing eye determination process based on the distance between the upper eyelid and the eyebrows. On the other hand, the open / close eye comprehensive determination unit 6 correctly determines that both the open / close eye determination process based on the vertical width of the eye and the open / close eye determination process based on the shape change of the upper eyelid are open eyes. There is no mistake in the overall opening / closing eye judgment.

【0104】

Furthermore, as shown in FIG. 25, in the case of a person with thin eyebrows, the open / close eye determination self-judgment unit 5 performs a diagnostic process in which the determination result of the open / close eye determination process based on the distance between the upper eyelid and the eyebrows is unused. , It is possible to eliminate the influence on the open / close eye determination comprehensive process in the open / close eye comprehensive determination unit 6 and maintain the accuracy of the open / close eye determination comprehensive process.

【0105】

Furthermore, when the bangs are a person with eyebrows, the open / close eye determination self-judgment unit 5 performs a diagnostic process in which the determination result of the open / close eye determination process based on the distance between the upper eyelid and the eyebrows is unused. It is possible to eliminate the influence on the open / close eye determination comprehensive process in the comprehensive determination unit 6 and maintain the accuracy of the open / close eye determination comprehensive process.

【0106】

Furthermore, when the contrast is weak and the eyebrow data cannot be recognized, the open / close eye determination self-judgment unit 5 performs diagnostic processing in which the determination result of the open / close eye determination process based on the distance between the upper eyelid and the eyebrow is unused. It is possible to eliminate the influence of the eyebrow comprehensive determination unit 6 on the open / close eye determination comprehensive process and maintain the accuracy of the open / close eye determination comprehensive process.

【0107】

Furthermore, when the driver wears spectacles, the distance between the upper eyelid and the eyebrows when the eyes are opened is accurately detected, or reflection occurs on the lens of the spectacles and the upper eyelid is displayed on the spectacle frame. Depending on whether the distance from the eyebrows is detected, it depends on whether the eye is closed or the judgment result is unused by the self-diagnosis process. On the other hand, in any case, the open / close eye determination self-judgment unit 5 performs a diagnostic process in which the determination result of the open / close eye determination process based on the distance between the upper eyelid and the eyebrows is unused, so that the open / close eye determination is performed. It is possible to determine that the eye is open by the open / close eye comprehensive determination unit 6 without using it for the comprehensive process.

【0108】

Here, as shown in FIG. 13, the open / closed eye determination self-judgment unit 5 can correctly detect the eyebrows at the beginning, correctly detect the distance between the upper eyelid and the eyebrows when the eyes are opened, and in step S19 of FIG. The predetermined conditions are set so that the proportion of non-use is increased when the distance between the upper eyelid and the eyebrows is small, and the proportion of use is increased when the distance between the upper eyelid and the eyebrows is large. This ensures that the eyebrows are only tracked correctly and will not be unused in the diagnostic process while seeking the distance between the upper eyelid and the eyebrows.

【0109】

Further, when the tracking target of the frame of the eyebrows 21 is changed from the eyebrows, and the change amount of the interval is smaller than the change amount of the opening / closing stroke of the eye, it is not determined to be unused in the diagnostic process. If the amount of change in the opening / closing stroke of the eye exceeds the determination threshold value in the opening / closing eye determination process, it may be determined that the eye is closed. In addition, if the eyebrows cannot be detected correctly at the beginning and the upper eyelid and the frame of the spectacles when the eyes are opened are detected as the distance between the upper eyelid and the eyebrows, if the spectacle frame is continuously tracked, it is not used in the diagnostic process. Although the tracking object is replaced with the spectacle frame, the opening / closing eye determination process can be continued correctly. When the tracking target returns to the eyebrows in such a state, the distance between the upper eyelid and the eyebrow becomes smaller than the detected value, so that the determination result of the distance between the upper eyelid and the eyebrow can be unused in the diagnostic process. In such a state, for example, if the diagnostic process is not performed on the judgment result based on the distance between the upper eyelid and the eyebrow, the tracking target is changed from the spectacle frame to the eyebrow, and the distance value between the upper eyelid and the eyebrow becomes smaller. Even if the eyes are closed, the opening / closing eye determination process may not be possible. As described above, the diagnostic processing for the opening / closing eye judgment processing based on the distance between the upper eyelid and the eyebrows eliminates the influence on the opening / closing eye judgment comprehensive processing because the judgment result is unused, and the accuracy of the opening / closing eye judgment comprehensive processing is improved. Can be maintained.

【0110】

Furthermore, when the face is tilted in the pitch direction, the distance between the upper eyelid and the eyebrows when the eyes are opened is learned with the face straight as shown in FIGS. 17 and 18, and the distance between the upper eyelid and the eyebrows is EB_d1. When the face is tilted backward, the distance between the upper eyelid and the eyebrows becomes smaller, so the judgment result is unused in the diagnostic process, and when the face is tilted forward, the distance between the upper eyelid and the eyebrows becomes larger. It is judged that the eyes are closed. Also in this example, the adverse effect of the open / close eye determination process at the distance between the upper eyelid and the eyebrow can be eliminated, and the accuracy of the open / close eye determination comprehensive process can be maintained.

【0111】

Furthermore, in the case of a person with double eyelids, the judgment result of the open / closed eye judgment process due to the shape change of the upper eyelid using the radius of curvature method is regarded as unused by the diagnostic process, so that the influence on the open / close eye judgment comprehensive process is eliminated. The accuracy of the open / close eye judgment comprehensive processing can be maintained.

【0112】

Furthermore, if the eye tracking is incorrect, the process is returned to step S1 in the vertical width diagnosis process for re-detection, or the distance between the upper eyelid and the eyebrow and the opening and closing due to changes in the shape of the upper eyelid. When the eye determination process is unused at the same time, it is determined that the eye tracking is incorrect, and the accuracy of the open / close eye determination comprehensive process is maintained.

【0113】

[Effect of embodiment]

As described in detail above, according to the open / close eye determination device according to the present embodiment, the open / close eye determination self-judgment unit determines whether the determination result of each open / close eye determination process is suitable for use in the open / close eye determination comprehensive process. Since the open / close eye judgment comprehensive processing is performed using only the judgment results judged to be used in the diagnostic processing after diagnosis in 5, the accuracy is high, allowing for various light environments and differences in facial features. It is possible to realize the opening / closing eye determination.

【0114】

Further, according to this open / close eye determination device, the determination threshold value (reference value) of each open / close eye determination process is changed according to the number of determination results used in the open / close eye determination comprehensive process. Even under conditions that are disadvantageous to the open / close eye determination process as a feature amount, the accuracy of the open / close eye determination comprehensive process can be maintained.

【0115】

Further, according to this open / close eye determination device, when the number of determination results used in the open / close eye determination comprehensive process is reduced due to the diagnostic process of the open / close eye determination process, the open / close eye comprehensive determination unit 6 determines the open / close eye. Since the judgment threshold in the open / close eye judgment process that is not used in the comprehensive process is made strict, the accuracy of the open / close eye judgment comprehensive process is maintained even under conditions that are disadvantageous to the open / close eye judgment process as facial features. be able to.

【0116】

Furthermore, according to this open / close eye determination device, the calculation target of the image feature amount in the open / close eye determination process is the vertical width of the eye, so that in a light environment condition where a part of the eye is exposed to direct sunlight. It is also possible to perform highly accurate opening / closing eye determination.

【0117】

Furthermore, according to this open / close eye determination device, as shown in Table 3 below, the effect of the self-diagnosis method is diagnosed from the vertical width values of a plurality of eyes output in chronological order. , When it is judged that the vertical width value of the eye is abnormal, the position of the eye is detected again because there is an eye tracking error, so that the abnormal state of the open / close eye judgment comprehensive processing can be continued for a short time. Can be stopped with.

【0118】

[Table 3]<img file="JP2004192552A_D0003.tif" />Furthermore, according to the open / close eye determination device, the calculation target of the image feature amount in the open / close eye determination process is the distance between the peak point of the upper eyelid and the eyebrows, so that the driver is sleepy and squints. It is possible to perform an open / close eye determination process that distinguishes between a situation in which the eyes are squinting with a smile and a situation in which the eyes are squinting due to dazzling.

【0119】

Furthermore, according to the open / close eye determination processing device, the diagnosis process of the determination result is performed from the appearance rate of the eyebrow data within the previous predetermined time, so that the open / close eye determination process is determined based on the distance between the peak point of the upper eyelid and the eyebrow. It is possible to accurately determine whether or not the result should be used for the open / close eye determination comprehensive process.

【0120】

Furthermore, according to the open / close eye determination processing device, the diagnosis processing of the determination result is performed from the stability of the appearance position of the data of a plurality of eyebrows output in time series, so that the peak point of the upper eyelid and the eyebrow It is possible to accurately determine whether or not the determination result of the open / close eye determination process based on the interval between the eyebrow and the eyebrow should be used for the open / close eye determination comprehensive process.

【0121】

Furthermore, according to the open / close eye determination processing device, the diagnosis processing of the determination result is performed from the interval value between the eyebrows and the peak points of the plurality of upper eyelids output in chronological order, so that the peak points of the upper eyelids are processed. and by the distance between the eyebrows whether or not to use the judgment result of the opening and closing eye determination process that the opening and closing eye judgment overall processing can be determined accurately.

【0122】

Furthermore, according to the open / close eye determination processing device, the calculation target of the image feature amount in the open / close eye determination process is the shape of the upper eyelid, so that the eye is used when the face image 11 is dark or in a near-infrared light environment at night. Even when the contrast is low, it is possible to perform highly accurate opening / closing eye determination processing.

【0123】

Furthermore, according to the open / close eye determination processing device, the diagnosis processing of the determination result is performed from the distances between the two end points of the plurality of inner and outer corners of the eye that are output in time series, so that the calculation target of the image feature amount can be calculated. It is possible to accurately determine whether or not the determination result of the open / close eye determination process based on the shape of the upper eyelid should be used for the open / close eye determination comprehensive process.

【0124】

Furthermore, according to the open / close eye determination processing device, the calculation target of the image feature amount in the open / close eye determination process is the shape of the upper eyelid, and the shape is used for a straight line connecting both end points of the inner and outer corners of the eye obtained by changing the density. Since it is calculated from the vertical distance from the peak point of the upper eyelid, the contrast of the eyes is low and the robustness to individual differences is further improved when the face image 11 is dark or in a near-infrared light environment at night. It is possible to perform highly accurate open / close eye determination processing in the state.

【0125】

Furthermore, according to the open / close eye determination processing device, the calculation target of the image feature amount in the open / close eye determination process is the shape of the upper eyelid, and the shape is the radius of curvature of the approximate expression of the quadratic curve for the upper eyelid edge extraction point. Therefore, it is possible to perform more accurate opening / closing eye determination processing when the face image 11 is dark or in a near-infrared light environment at night when the contrast of the eyes is low.

【0126】

Furthermore, according to the open / close eye determination processing device, the determination result is diagnosed from the correlation between the approximate curve and the actual line obtained at the same time as the calculation of the radius of curvature of the upper eyelid. This shape is used as the target of the upper eyelid edge extraction point, and it is accurately determined whether or not the judgment result of the opening / closing eye judgment processing based on the magnitude of the radius of curvature of the approximate expression of the quadratic function should be used for the opening / closing eye judgment comprehensive processing. can do.

【0127】

Furthermore, according to the open / close eye determination processing device, the determination result is diagnosed from the correlation between a plurality of approximate curves output in time series and the actual line, so that the calculation target of the image feature amount is the upper eyelid. This shape is used as the target of the upper eyelid edge extraction point, and it is accurately determined whether or not the judgment result of the opening / closing eye judgment processing based on the size of the radius of curvature of the approximate expression of the quadratic function should be used for the opening / closing eye judgment comprehensive processing. can do.

【0128】

The above-described embodiment is an example of the present invention. Therefore, the present invention is not limited to the above-described embodiment, and even if it is not the embodiment, it varies depending on the design and the like as long as it does not deviate from the technical idea of the present invention. Of course, it is possible to change.

[Simple explanation of drawings]

FIG. 1 is a block diagram showing a configuration of an open / close eye determination device to which the present invention is applied.

FIG. 2 is a diagram for explaining a change in an opening value between when the eyes are opened (a) and when the eyes are closed (b) in the opening / closing eye determination process based on the vertical width of the eyes.

FIG. 3 is a diagram for explaining that a tracking area is cut out from a face image (a), the presence of a shadow in the tracking area (b), and the vertical width of the eye changes (c).

FIG. 4 is a diagram for explaining a change in the vertical width of the eye between a state of gazing forward (a) and a state of smiling (b).

FIG. 5 is a diagram for explaining a change between when the eyes are opened and when the eyes are closed in the opening / closing eye determination process based on the distance between the peak point of the upper eyelid and the eyebrows.

FIG. 6 is a diagram for explaining an opening / closing eye determination process based on the distance between the upper eyelid and the eyebrows when the driver wears spectacles.

FIG. 7 is a diagram for explaining the distance between the upper eyelid and the eyebrows depending on the positional relationship between the driver's face position and the camera.

FIG. 8 is a diagram for explaining the distance between the upper eyelid and the eyebrows when the driver's face is not tilted and when the driver's face is tilted backward.

FIG. 9 is a diagram for explaining a change between when the eyes are opened (a) and when the eyes are closed (b) in the opening / closing eye determination process due to a change in the shape of the upper eyelid using the three-point method.

FIG. 10 is a diagram for explaining a change between when the eyes are opened (a) and when the eyes are closed (b) in the opening / closing eye determination process due to a change in the shape of the upper eyelid using the radius of curvature method.

FIG. 11 is a diagram showing an eye image (a) in normal times, an eye image (b) when the inner corner of the eye is overexposed, and an eye image (c) when the outer corner of the eye is overexposed.

FIG. 12 is a diagram for explaining an edge line consisting of edge extraction points when the driver has a double eyelid.

FIG. 13 is a diagram for explaining an opening / closing eye determination process based on the vertical width of the eye when a part of the eye image is overexposed.

FIG. 14 shows a state in which the eyes are narrowed due to smiling or dazzling (a) and a state in which the eyes are closed due to sleepiness (b) in the opening / closing eye determination process based on the distance between the peak point of the upper eyelid and the eyebrows. It is a figure for demonstrating the distinction.

FIG. 15: In the opening / closing eye determination process due to a change in the shape of the upper eyelid, the opening / closing eye determination using the three-point method or the radius of curvature method is performed using the images shown when the eyes are closed (b) and when the eyes are opened (c). It is a figure for demonstrating.

FIG. 16 is a flowchart showing an open / close eye determination result generation process for determining an open / closed state of a driver's eye by an open / close eye determination device to which the present invention is applied.

FIG. 17 is a flowchart showing an open / close eye determination result generation process for determining an open / closed state of a driver's eye by an open / close eye determination device to which the present invention is applied.

FIG. 18 is a flowchart showing an open / close eye determination result generation process for determining an open / closed state of a driver's eye by an open / close eye determination device to which the present invention is applied.

FIG. 19 is a diagram for explaining a process of obtaining an extraction point of a face image.

FIG. 20 is a diagram showing extraction points obtained from a face image.

FIG. 21 is a diagram for explaining a process of obtaining continuous data from an extraction point.

FIG. 22 is a diagram for explaining a process of obtaining each coordinate value corresponding to each part of the face from each continuous data.

FIG. 23 is a diagram for explaining a process of setting a tracking area by an eye tracking unit and tracking an eye image and center coordinates.

FIG. 24 is a diagram showing an open / close eye comprehensive determination table used in the open / close eye comprehensive determination unit.

FIG. 25 is a diagram showing an open / close eye comprehensive determination table used in the open / close eye comprehensive determination unit.

[Explanation of symbols]

1 Imaging unit 2 Eye position detection unit 3 Eye tracking unit 4 Open / close state judgment unit 5 Open / close eye judgment self-judgment unit 6 Open / close eye comprehensive judgment unit 11 Face image 12 Tracking area 51 Edge extraction point 61 Eye image 62 Overexposure part