Object detection device

The present invention relates to an object detection device that detects an object by using an optical flow obtained from a plurality of images captured at different times by an imaging means mounted on a moving body.

The optical flow obtains the velocity field of each point (pixel, region) in the image from a time-continuous image sequence, and represents the movement of an object in the image as a vector. A technique for detecting the movement of another vehicle around the own vehicle has been developed by utilizing the magnitude and direction of the vector of this optical flow. However, since the optical flow is easily affected by noise in the image, the optical flow may include an error. Therefore, the apparatus described in Patent Document 1 calculates the optical flow of each region set in the image, and detects the movement of another vehicle based on the optical flow of the optical flow of each region that is equal to or higher than the threshold value. ..<patcit num="1"><text>Japanese Patent Application Laid-Open No. 6-314340</text></patcit><patcit num="2"><text>Japanese Unexamined Patent Publication No. 2007-286724</text></patcit>

<p> When calculating the optical flow using an image captured by a camera mounted on a vehicle, the camera itself moves in the direction of travel of the vehicle together with the vehicle, so that the optical flow (movement of an object) occurs near the center of the direction of travel in the image. It becomes smaller and the optical flow becomes larger toward the periphery. Therefore, when the optical flow is determined by a threshold value of a uniform size as in the above device, the optical flow near the center may be determined as noise even though it indicates the movement of another vehicle. There is. On the other hand, although the optical flow near the peripheral edge shows noise, it may be determined as the optical flow of the object to be detected.</p><p> Therefore, an object of the present invention is to provide an object detection device that detects an object with high accuracy by using an optical flow.</p>

<p> The object detection device according to the present invention is an object detection device that detects an object from a plurality of images captured at different times by an imaging means mounted on a moving body, and is an optical device from a plurality of images captured at different times. Set by the limit range setting means from the optical flow calculation means that calculates the flow, the limit range setting means that sets the limit range of the movement of the optical flow according to the position on the image, and the optical flow calculated by the optical flow calculation means. It is characterized by including an optical flow extracting means for extracting an optical flow having a movement within the limited range, and an object detecting means for detecting an object based on the optical flow extracted by the optical flow extracting means.</p><p> In this object detection device, the optical flow is calculated from a plurality of images captured at different times by the optical flow calculation means. When the optical flow is calculated using the image captured by the imaging means mounted on the moving body, the imaging means itself moves together with the vehicle in the traveling direction of the vehicle, so that a predetermined center in the image (for example, the moving body) is calculated. The closer to the center of the traveling direction (the farther from the imaging position), the smaller the optical flow, and the farther from the predetermined center (the closer to the imaging position), the larger the optical flow. That is, even if the movement of the object is the same, the optical flow differs depending on the position on the image. Therefore, in the object detection device, the limit range of the movement of the optical flow is set according to the position on the image by the limit range setting means. Then, in the object detection device, the optical flow within the limited range is extracted from the optical flows at each position of the image by the optical flow extraction means. Further, the object detection device detects an object from the extracted optical flow by the object detection means. In this way, in the object detection device, the limit range is set according to the position on the image, and the optical flow is limited according to the position on the image, so that the noise is accurately based on the reference according to the position on the image. Can be removed, and the object can be detected with high accuracy by optical flow.</p><p> The limited range is a range in which the optical flow for the object to be detected can be taken in consideration of the range in which the object to be detected moves on the image according to the position on the image (position relative to the imaging means). .. The limit range may be the limit range of the size of the optical flow (upper limit and / lower limit), the limit range of the direction of the optical flow, or the limit range of the size and the limit range of the direction.</p><p> In the object detection device of the present invention, it is preferable that the limiting range setting means sets a larger limiting range as it is closer to the imaging position.</p><p> As described above, the farther the image is from the imaging position, the smaller the optical flow, and the closer to the imaging position, the larger the optical flow. Therefore, in the object detection device, an appropriate limiting range can be provided according to the relative position from the imaging means by setting a larger limiting range closer to the imaging position by the limiting range setting means.</p><p> In the object detection device of the present invention, it is preferable that the limiting range setting means sets a larger limiting range centered on a predetermined position in the image as the distance from the predetermined position increases.</p><p> As described above, the closer to the predetermined center in the image, the smaller the optical flow, and the farther away from the predetermined center, the larger the optical flow. Therefore, in the object detection device, an appropriate limiting range is set according to the position from the predetermined position in the image by setting a larger limiting range centered on the predetermined position in the image by the limiting range setting means. Can be provided.</p><p> In the object detection device of the present invention, the predetermined position may be set based on the object detection target area in the image.</p><p> In this object detection device, by setting a predetermined position (center) in the image in consideration of the object detection target area, an appropriate limiting range can be provided according to the position from the predetermined position in the image. Further, the processing load can be reduced by limiting the object detection target area in the image.</p><p> In the object detection device of the present invention, the predetermined position may be set based on the shape of the traveling path detected from the image.</p><p> The vanishing point of the runway in the image corresponds to a position on the horizon (a predetermined center in the image). Therefore, in this object detection device, by setting a predetermined position (center) in the image using the shape of the traveling path, an appropriate limiting range can be provided according to the position from the predetermined position in the image. As the shape of the traveling road, for example, it may be detected from a pair of white lines indicating lanes, or the traveling road itself may be detected.</p><p> In the object detection device of the present invention, the predetermined position may be set based on the direction of the optical flow of the peripheral region in the image.</p><p> Most of the peripheral area in the image corresponds to the background that exists far from the own vehicle. Therefore, in the peripheral region, the direction of the optical flow is oriented toward a predetermined center in the image, and a stable optical flow is calculated. Therefore, in the object detection device, by setting a predetermined position (center) in the image based on the direction of the optical flow of the peripheral region in the image, an appropriate limiting range is set according to the position from the predetermined position in the image. be able to.</p><p> In the object detection device of the present invention, the predetermined position may be corrected based on the behavior of the moving body.</p><p> When the moving body on which the imaging means is mounted is tilted in a predetermined direction, the imaging direction of the imaging means changes, and the predetermined center position in the image also changes. Therefore, the object detection device can correct the deviation of the predetermined position (center) in the image according to the behavior of the moving body by correcting the predetermined position (center) in the image according to the behavior of the moving body.</p><p> In the object detection device of the present invention, the limit range setting means may be configured to correct the limit range according to the speed of the moving body.</p><p> The higher the speed of the moving body equipped with the imaging means, the larger the optical flow. Therefore, in the object detection device, a more appropriate limiting range can be provided by correcting the limiting range according to the speed of the moving body by the limiting range setting means.</p>

<p> In the present invention, by setting a limiting range according to the position on the image and setting a limit on the optical flow according to the position on the image, noise can be accurately removed according to the reference according to the position on the image, and the optical Objects can be detected with high accuracy by the flow.</p>

Hereinafter, embodiments of the object detection device according to the present invention will be described with reference to the drawings.

In the present embodiment, the object detection device according to the present invention is applied to a peripheral monitoring device mounted on a vehicle. The peripheral monitoring device according to the present embodiment detects an obstacle in front of the own vehicle (for example, a moving object such as another vehicle, a bicycle, or a pedestrian, or a stationary object such as a falling object), and obtains the detected obstacle information. It is provided to the driver by output to a driving support device (collision prevention device, etc.) or by voice or display. The detection direction is forward, but may be another direction such as lateral or backward.

The peripheral monitoring device 1 according to the present embodiment will be described with reference to FIGS. 1 and 2. FIG. 1 is a configuration diagram of a peripheral monitoring device according to the present embodiment. FIG. 2 is an example of setting a divided area for the captured image in the ECU of FIG.

The peripheral monitoring device 1 detects obstacle information based on the optical flow detected from the images between frames that are continuous in time. In particular, the peripheral monitoring device 1 sets a region for calculating the optical flow and a limited range of the size of the optical flow in the region according to the position on the image in order to improve the detection accuracy of the object. The peripheral monitoring device 1 includes a camera 2, a vehicle speed sensor 3, a yaw sensor 4, a pitch sensor 5, and an ECU [Electronic Control Unit] 6.

In the present embodiment, each process in ECU 6 corresponds to the optical flow calculating means, the limiting range setting means, the optical flow extracting means, and the object detecting means described in the claims.

The camera 2 is a camera that captures the front of the own vehicle. The camera 2 is attached to the center of the front side of the own vehicle. The camera 2 images the front of the own vehicle and transmits the captured image information to the ECU 6 as an image signal. This captured image is a frame image at regular time intervals (for example, 1/30 second).

The vehicle speed sensor 3 is a radar sensor that detects the vehicle speed of the own vehicle. The vehicle speed sensor 3 detects the vehicle speed of the own vehicle and transmits a vehicle speed signal indicating the detected vehicle speed to the ECU 6.

The yaw sensor 4 is a sensor that detects the yaw angle of the own vehicle. The yaw sensor 4 detects the yaw angle of the own vehicle and transmits a yaw signal indicating the detected yaw angle to the ECU 6.

The pitch sensor 5 is a sensor that detects the pitch angle of the own vehicle. The pitch sensor 5 detects the pitch angle of the own vehicle and transmits a pitch signal indicating the detected pitch angle to the ECU 6.

The ECU 6 is an electronic control unit composed of a CPU [Central Processing Unit], a ROM [Read Only Memory], a RAM [Random Access Memory], an image processing chip, and the like, and controls the peripheral monitoring device 1 in an integrated manner. The ECU 6 takes in the image signal from the camera 2 and the detection signals from the sensors 3, 4 and 5 at regular time intervals, and stores the captured image information and the detection information in chronological order. Then, the ECU 6 uses this information to perform processing for detecting obstacle information, and outputs the detected obstacle information (for example, position, moving direction, moving speed) to the driving support device or provides the driver with the detected obstacle information. To do.

In ECU6, the optical flow for each pixel of the entire image is calculated using the image of the frame at the previous time (t-1) and the image of the frame at the current time (t). As a method of calculating the optical flow, a conventional method is applied. The optical flow for each pixel may be calculated only for the pixels in the visual field region described below.

In order to divide the entire image captured by the camera 2 into regions, the ECU 6 considers the lens parameters (angle of view, etc.) of the camera 2 and sets a basic division region for obtaining the optical flow for each region. As shown in FIG. 2, in each of the divided regions, a plurality of concentric circles having different radii are provided, and a plurality of lines are provided radially from the center of the circles at predetermined angular intervals. The area is defined by the radial lines of.

The size of the optical flow decreases as it is closer to the center of the vehicle's traveling direction on the image (farther from the image pickup position of camera 2), and larger as it is farther from the center (closer to the image pickup position of camera 2). Become. Therefore, in order to make the division area corresponding to the size of the optical flow according to the position on the image, a very small radius is set for the innermost perfect circle, and the radius increases as the outer peripheral perfect circle increases. Set the radius with a larger ratio. Therefore, the length of the divided region on the outer peripheral side becomes longer in the radial direction. In addition, the spacing angle of the plurality of radial lines is set in consideration of the size of the object to be detected on the image and the processing capacity of the ECU 6. The interval angle may or may not be equal. For example, the closer the lateral position is to the center (the closer the line is to the vertical line), the narrower the angle may be.

In order to provide a visual field area, in ECU6, an area to be detected as an obstacle (area to be processed) is selected from the set basic divided area group, and the visual field area is configured by the selected divided area group. To do. This visual field area is an area where an obstacle to the own vehicle may exist, and an area such as the sky where there is no obstacle is excluded. The center position of the visual field region is a position on the horizon and is the center of the traveling direction of the own vehicle.

In order to obtain the center of the visual field region, the ECU 6 detects a pair of white lines constituting the lane in which the own vehicle is traveling from the image of the frame at the current time (t). The conventional method is applied to this white line detection method. Then, in ECU6, the vanishing point (point on the horizon) of the detected pair of white lines is calculated. Further, in ECU6, the vanishing point of the white line is set at the center of the visual field region, and the divided region group constituting the visual field region is arranged on the image.

However, there are cases where there is no white line on the road or the white line cannot be detected. In that case, in ECU6, the direction of the optical flow of each pixel in the peripheral region on the image is extended toward the center, the intersection is set at the center of the visual field region, and the divided region group constituting the visual field region is arranged on the image. Let me. Most of the peripheral region of the image corresponds to the background existing far from the own vehicle. Therefore, in this peripheral region, since the movement is relative to the forward movement of the own vehicle, the direction of the optical flow is toward the center of the image, and a stable optical flow is calculated.

In order to correct the center of the visual field area, the ECU6 corrects the center of the visual field area according to the yaw angle of the own vehicle and also corrects the center of the visual field area according to the pitch angle. Is rearranged on the image. When the own vehicle on which the camera 2 is mounted yaws or pitches, the imaging direction of the camera 2 changes, and the area where obstacles exist on the image also changes. Therefore, it is necessary to move the field of view region on the image according to the behavior of the own vehicle.

In the case of the example shown in FIG. 2, a pair of white lines WL and WL can be detected on the road R on which the own vehicle is traveling, and the center of the visual field area VA is set at the vanishing point of the white lines WL and WL. The field of view area VA is mainly set on the lower side of the image P and is composed of a large number of divided areas. As can be seen from this example, the division region is smaller toward the center C of the visual field region VA, and the division region is larger toward the periphery.

In order to set the limit range for each of the divided areas constituting the field of view area, the ECU 6 sets the limit range of the size of the optical flow for each divided area according to the position on the image. The size of the optical flow becomes smaller as it is closer to the center of the field of view (farther from the image pickup position of the camera 2), and becomes larger as it is farther from the center of the field of view (closer to the image pickup position of the camera 2). Therefore, the division area closer to the center of the visual field area is set to a smaller range of the optical flow size, and the division area farther from the center is set to a larger range of the optical flow size limitation range. This limit range consists of a lower limit value and an upper limit value of the size. The small limit range is a range in which an optical flow having a relatively small magnitude is judged to be a normal optical flow (not noise or the like). In the large limit range, an optical flow having a relatively large size is judged to be a normal optical flow.

The limit range may be set in consideration of the lens parameter of the camera 2, the size of the optical flow in pixel units at each position on the image, and the like.

In order to correct the limited range for each divided area that constitutes the visual field area, the ECU 6 corrects the limited range for each divided area according to the vehicle speed of the own vehicle. As the vehicle speed of the own vehicle increases, the amount of movement of the own vehicle between the frames increases, so that the optical flow increases. Therefore, as the vehicle speed of the own vehicle increases, the correction is made to a range larger than the reference limit range set according to the position on the image. Instead of correcting the size of the limit range, the limit range may be normalized according to the vehicle speed.

As described above, in the ECU 6, the field of view area is set in the image, and the field of view area is divided into a large number of divided areas. The divided area is set to a smaller area as it is closer to the center of the visual field area, and a larger area as it is farther from the center. Each divided area is provided with a limited range. The limiting range is set to a smaller range as it is closer to the center of the visual field area, and a larger range as it is farther from the center.

If a limiting range is set for each divided area that constitutes the field of view, the ECU 6 averages the optical flow of each pixel included in the divided area for each divided area, and the optical flow (vector) for each area is used. Is calculated. Further, in ECU6, for each divided area, it is determined whether or not the size of the optical flow for each area is within the limit range. If the size of the optical flow is within the limit range, ECU6 determines that the optical flow is normal and adopts the optical flow. If the magnitude of the optical flow is out of the limit range, ECU6 determines that the optical flow is abnormal (noise, etc.) and rejects the optical flow.

Then, in ECU6, the optical flow showing a size and direction different from the optical flow of the surrounding area is extracted from the adopted optical flow of each area. When an object (obstacle) different from the background exists on the image, the optical flow for the object shows a size and direction clearly different from the surroundings (background). Therefore, by extracting a peculiar optical flow different from the surroundings, the optical flow of the obstacle can be found.

Then, in ECU6, obstacle information (movement direction, movement speed, movement amount, position, etc.) is set from the optical flow of the extracted area. Further, the ECU 6 outputs the obstacle information to the driving support device or provides the driver with the obstacle information. As other information on the obstacle, the size of the obstacle may be detected from the image, or the type of the obstacle may be detected by using pattern recognition or the like. In the case of a stereo camera, the relative distance to an obstacle and the relative speed can be detected with high accuracy. Further, the relative distance to the obstacle, the relative velocity, the relative lateral position, and the like may be detected with high accuracy by using a millimeter wave sensor or the like.

The operation of the peripheral monitoring device 1 will be described with reference to FIGS. 1 and 2. In particular, the processing in ECU 6 will be described with reference to the flowchart of FIG. FIG. 3 is a flowchart showing the processing flow in the ECU of FIG.

The camera 2 takes an image of the front of the own vehicle at regular intervals and transmits an image signal consisting of the image information to the ECU 6. The vehicle speed sensor 3 detects the vehicle speed of the own vehicle at regular time intervals and transmits a vehicle speed signal consisting of the vehicle speed to the ECU 6. The yaw sensor 4 detects the yaw angle of the own vehicle at regular time intervals and transmits a yaw signal consisting of the yaw angle to the ECU 6. The pitch sensor 5 detects the pitch angle of the own vehicle at regular time intervals and transmits a pitch signal consisting of the pitch angle to the ECU 6.

The ECU 6 receives an image signal from the camera 2 at regular time intervals and acquires an image of each frame (S1). The images of each frame are temporarily stored in ECU 6 in chronological order. Then, in ECU6, the optical flow of each pixel is calculated using the image of the frame at the current time (t) and the image of the frame at the previous time (t-1) (S2).

In ECU6, in consideration of the characteristics of the camera 2, etc., the image is divided into areas and a large number of divided areas that are the basis for obtaining the optical flow for each area are set (S3). Then, in ECU6, a region to be detected is selected from the divided region group, and a visual field region is constructed from the selected divided region (S4).

ECU6 detects a pair of white lines from the image of the frame at the current time (t) (S5). Then, in ECU6, the vanishing point of a pair of white lines on the image of the frame at the current time (t) is obtained, the vanishing point is set at the center of the visual field region, and the divided region group in the visual field region is arranged on the image. Let (S6). At this time, if a pair of white lines cannot be detected, the center of the visual field region is set based on the direction of the optical flow of each pixel in the peripheral region on the image.

The ECU 6 receives the yaw signal from the yaw sensor 4 at regular time intervals and acquires the yaw angle (S7). Further, the ECU 6 receives the pitch signal from the pitch sensor 5 at regular time intervals and acquires the pitch angle (S7). Then, in ECU6, the center of the visual field region is corrected according to the yaw angle and the pitch angle, and the divided region group in the visual field region is rearranged on the image (S8).

In ECU6, the limit range of the size of the optical flow is set according to the position in the image for each divided area of the field of view (S9).

The ECU 6 receives the vehicle speed signal from the vehicle speed sensor 3 and acquires the vehicle speed (S10). Then, in ECU6, the limitation range of the size of the optical flow is corrected according to the vehicle speed for each divided area of the visual field area (S11).

In ECU6, the optical flow (vector) of each area is calculated by using the optical flow of each pixel in the area for each divided area of the visual field area (S12).

In ECU6, it is determined whether or not the size of the optical flow in each area is within the limit range for each divided area of the visual field area (S13). If it is determined in S13 that it is out of the limited range, ECU6 determines that the optical flow is abnormal (noise, etc.) and rejects the optical flow (S15).

If it is determined in S13 that it is within the limited range, ECU6 determines that the optical flow is normal (S14). Then, in ECU6, the optical flow presumed to be an obstacle is extracted from the optical flows judged to be normal, and the obstacle information is set from the extracted optical flow (S14).

Then, in ECU6, obstacle information is output to the driving support device or provided to the driver by voice or display.

According to this peripheral monitoring device 1, the divided area and the limited range in the divided area are set according to the position on the image, and the size of the optical flow is limited for each divided area according to the position on the image. As a result, noise can be accurately removed according to the reference according to the position on the image, and the object can be detected with high accuracy by the optical flow.

Further, according to the peripheral monitoring device 1, the farther from the center of the field of view (the closer to the camera 2), the larger the limiting range is set, so that the position on the image (the relative position from the camera 2) is set. Appropriate limits can be set.

Further, according to the peripheral monitoring device 1, the processing load can be reduced by setting the visual field area and setting only the visual field area as the target for detecting obstacles.

Further, according to the peripheral monitoring device 1, the center position of the visual field region can be obtained with high accuracy by setting the center of the visual field region using the vanishing point of the pair of white lines or the direction of the optical flow in the peripheral region. It is possible to set an appropriate limiting range according to the position from the center. Further, according to the peripheral monitoring device 1, the center position of the visual field region can be obtained with higher accuracy by correcting the center of the visual field region according to the behavior of the vehicle.

Further, according to the peripheral monitoring device 1, a more appropriate limiting range can be provided by correcting the limiting range according to the vehicle speed of the own vehicle.

Although the embodiments according to the present invention have been described above, the present invention is not limited to the above embodiments and is implemented in various forms.

For example, in the present embodiment, it is mounted on a vehicle and applied to a peripheral monitoring device for detecting an obstacle around the vehicle, but it can be applied to various object detection devices that detect an object using an optical flow. Further, it may be mounted on another moving body such as a robot.

Further, in the present embodiment, the yaw sensor and the pitch sensor are used as means for detecting the vehicle behavior, but the vehicle behavior may be detected by other means. For example, the left and right wheel speed pulses by the wheel speed sensors of each wheel. Find the behavior of your vehicle from such factors.

Further, in the present embodiment, a plurality of concentric circles and a plurality of radial lines from the center are used as the division region, but in consideration of the lens parameters of the camera and the like, the division region is divided into other shapes by using an ellipse or the like. May be set.

Further, in the present embodiment, the limiting range is a limiting range consisting of a lower limit and an upper limit of the size of the optical flow, but only the lower limit or the upper limit of the size may be used, or only the limiting range of the direction of the optical flow may be used. Alternatively, it may be a limited range of size and orientation.

Further, in the present embodiment, the visual field area is set and the object is detected only in the visual field area, but the object may be detected in the entire image.

Further, in the present embodiment, the center of the visual field region is corrected according to the behavior of the own vehicle, but such a correction may not be performed. In particular, when the imaging means is mounted on a moving body whose behavior does not change, the imaging direction does not change, so that such correction is not necessary.

Further, in the present embodiment, the limit range is corrected according to the vehicle speed of the own vehicle, but such a correction may not be performed.

<figref num="1">It is a block diagram of the peripheral monitoring apparatus which concerns on this embodiment.</figref><figref num="2">This is an example of setting the divided area for the captured image in the ECU shown in FIG.</figref><figref num="3">It is a flowchart which shows the processing flow in the ECU of FIG.</figref>

Code description

1 ... peripheral monitoring device, 2 ... camera, 3 ... vehicle speed sensor, 4 ... yaw sensor, 5 ... pitch sensor, 6 ... ECU