Image processing unit, and sight support device and method

The present invention relates to a visibility support device and a visibility support method. The present invention particularly relates to a technique of generating a bird's-eye view image from images captured by a plurality of cameras installed in a vehicle and synthesizing each bird's-eye view image. The present invention also relates to an image processing device used in a field of view support device.

For drivers such as automobiles, it is difficult to check backwards because a blind spot is created when backing up. Therefore, a system has already been developed that is equipped with an in-vehicle camera that monitors the rear of the vehicle, which tends to be a blind spot for the driver, and displays the captured image on a screen such as a car navigation system.

In addition, research is being conducted to show more human-friendly images by using image processing technology instead of simply displaying the images from the camera. One of them is to generate and display a bird's-eye view image as seen from above the ground by converting the coordinates of the captured image. By displaying this bird's-eye view image, the driver can easily grasp the situation behind the vehicle.

Further, a visibility support device has been developed in which images obtained from a plurality of cameras are converted into an all-around bird's-eye view image by geometric transformation and displayed on a display unit (see, for example, Patent Documents 1 and 2 below). This visibility support device has the advantage of being able to cover the periphery of the vehicle without a 360-degree blind spot because it can present the situation around the entire circumference of the vehicle to the driver as an aerial image.

A conventional example of this type of vision support device will be described. FIG. 16 is a plan view of the vehicle 100 as viewed from above, showing the installation status of the camera on the vehicle 100. FIG. 17 is a view of the vehicle 100 viewed diagonally from the front left. In FIG. 17, the field of view (imaging area) of each camera is schematically shown. The vehicle 100 is a truck formed by a driver's cab and a luggage compartment higher than the driver's cab.

As shown in FIG. 16, cameras (imaging devices) 1F, 1B, 1L and 1R are attached to the front part, the rear part, the left side part and the right side part of the vehicle 100, respectively. The cameras 1F, 1B, 1L and 1R correspond to the front camera, the rear camera, the left side camera and the right side camera, respectively.

In the field of view support device, a bird's-eye view image is generated from the captured images obtained from each camera, and each bird's-eye view image is combined to display the all-around bird's-eye view image as shown in FIG. 18 on the display device. On the display screen of the display device, the vehicle is displayed in the center, and bird's-eye view images obtained from the cameras 1F, 1B, 1L and 1R are displayed on the front, rear, left and right sides of the vehicle, respectively.

By the way, as shown in FIG. 17, there are overlapping portions in the field of view (imaging area) of each camera. For example, the field of view of the camera 1F and the field of view of the camera 1L overlap in a predetermined area diagonally to the left of the vehicle 100. This overlapping portion corresponds to the region designated by reference numeral 101 in FIG. Usually, the area 101 displays an image based on the image captured by the camera 1F, an image based on the image captured by the camera 1L, or an averaged image thereof.

In this type of visibility support device, since the composition process is performed so that the images have continuity on the ground surface, the lines, signs, characters, etc. of the parking lot drawn on the ground surface are displayed without any particular problem. This also applies to region 101 in FIG. However, since the appearance of a three-dimensional object placed on the ground surface differs depending on the viewpoint of the camera, it is difficult in principle to accurately and continuously depict it with a bird's-eye view image of the entire circumference.

For example, as shown in FIG. 19, consider a case where a person as a three-dimensional object 102 exists in a portion (space) where the field of view of the camera 1F and the field of view of the camera 1L overlap. In this case, when a bird's-eye view image is generated from the image of the three-dimensional object 102 captured by the camera 1F, the three-dimensional object 102 appears as an image tilted to the left on the bird's-eye view image as shown in FIG. On the other hand, when a bird's-eye view image is generated from an image obtained by capturing the three-dimensional object 102 with the camera 1L, the three-dimensional object 102 appears as an image tilted forward on the bird's-eye view image as shown in FIG.

When synthesizing the bird's-eye view image obtained from the camera 1F and the bird's-eye view image obtained from the camera 1L, a composite boundary 103 as shown in FIGS. 20 and 21 was defined, and both bird's-eye view images were simply pasted together at the composite boundary 103. In this case, there is a problem that the three-dimensional object 102 disappears in the all-around bird's-eye view image obtained by the composition.

In order to solve this problem, it is conceivable to generate an image of a common area where the bird's-eye view image obtained from the camera 1F and the bird's-eye view image obtained from the camera 1L overlap by averaging both bird's-eye view images. However, if such averaging is adopted, a three-dimensional object appears as a double image in the all-around bird's-eye view image. Further, since each image of the double image is averaged with the background image, the three-dimensional object becomes very difficult to see depending on the color of the three-dimensional object and the background.

Further, in the method described in Patent Document 1 below, only the bird's-eye view image obtained from the camera 1F as the image of the common area and the bird's-eye view image obtained from the camera 1L as the image of the common area are used. The adopted all-around bird's-eye view image and the adopted all-around bird's-eye view image are generated separately, and these two types of all-around bird's-eye view images are displayed side by side at the same time. The display image in this case is as shown in the display image 200 of FIG. However, in this method, a plurality of images to be confirmed by the driver are displayed at the same time, which may cause confusion and conversely impair safety.

Further, it is also conceivable to switch manually or by driving operation whether to adopt the bird's-eye view image obtained from the camera 1F or the bird's-eye view image obtained from the camera 1L as the image of the common area. In this case, the display image 201 and the display image 202 shown in FIG. 23 are switched and displayed. However, manual switching is complicated, and the image generated based only on the driving operation is not always an image that makes it easy to grasp a three-dimensional object.

<patcit num="1"><text>Japanese Unexamined Patent Publication No. 2003-125224</text></patcit><patcit num="2"><text>Japanese Unexamined Patent Publication No. 2004-235986</text></patcit>

<p> All of these problems are caused by the difficulty in determining the image to be displayed preferentially in the common area. In order to accurately depict a three-dimensional object in a single image while maintaining the continuity of the bird's-eye view, which camera captures the three-dimensional object more accurately (in other words, which bird's-eye view image captures the three-dimensional object). It is necessary to judge (whether it is expressed more accurately).</p><p> For example, in the situations shown in FIGS. 19 to 21, the camera 1F can capture only the lower half of the person as a three-dimensional object 102, while the camera 1L can capture the entire image of the person. This is mainly due to the fact that the camera 1L installed in the upper part of the luggage compartment can capture the three-dimensional object (person) 102 from a higher position than the camera 1F installed in the upper part of the driver's cab. .. In such a case, it is desirable to display an image that preferentially uses the image taken by the camera 1L in order to grasp the situation around the vehicle. On the contrary, depending on the position of the three-dimensional object 102, it may be better to preferentially use the captured image on the 1st floor of the camera.</p><p> Therefore, an object of the present invention is to provide a view support device and a view support method capable of displaying a composite bird's-eye view image more suitable for grasping a situation such as around a vehicle. Another object of the present invention is to provide an image processing apparatus used for them.</p>

<p> In order to achieve the above object, the image processing device according to the present invention is a viewpoint conversion means for converting each image captured by n image pickup devices (n is an integer of 2 or more) into a bird's-eye view image viewed from a virtual viewpoint. In an image processing device that includes an image synthesizing means that synthesizes the obtained bird's-eye view images to generate a composite bird's-eye view image and generates a video signal for displaying the composite bird's-eye view image on the display device, the image. The synthesizing means identifies the difference area between the plurality of bird's-eye view images in the common area by comparing the images of the common area where the plurality of bird's-eye view images overlap with each other during the composition. One bird's-eye view image is selected as the adopted image from the plurality of bird's-eye view images according to the position of the difference region in the common area, and the image of the common area in the composite bird's-eye view image is the said in the adopted image. It is characterized by adopting an image of a common area.</p><p> For each camera, what position it is easy to capture a three-dimensional object and what position it is difficult to capture a three-dimensional object depends on the installation position of each camera and the like, and they differ between different cameras. On the other hand, due to the principle of generating a bird's-eye view image, a three-dimensional object forms a difference region within the common region. The position of the difference region in the common region depends on the position of the three-dimensional object (real space position). In consideration of this, the above-mentioned adopted image is selected according to the position of the difference area in the common area, which reflects the position of the three-dimensional object (real space position), and the composite bird's-eye view image is generated. As a result, a bird's-eye view image that is expected to more accurately represent a three-dimensional object is selected as the adopted image, and it is possible to display an image suitable for grasping the situation around the vehicle.</p><p> Specifically, for example, a weight value corresponding to each pixel position of the image in the common area is set for each of the plurality of bird's-eye view images, and the weight value set between the plurality of bird's-eye view images is set. Part or all of them are different, and the image synthesizing means calculates a total value by adding up the weight values corresponding to the pixels included in the difference region for each of the plurality of bird's-eye view images, and is based on each total value. Select the adopted image.</p><p> More specifically, for example, the image synthesizing means selects the adopted image by comparing the total value between the plurality of bird's-eye view images.</p><p> Specifically, for example, the n image pickup devices are installed in the vehicle and image the surroundings of the vehicle, and the image synthesizing means adopts the above based on the operating state of the vehicle in addition to each total value. You may choose an image.</p><p> This makes it possible to display an appropriate image in consideration of the driving state.</p><p> Further, for example, the weight value is set based on the installation positions of a plurality of imaging devices corresponding to the plurality of bird's-eye view images.</p><p> Then, it is preferable to configure a field of view support device including the image processing device, the n image pickup devices, and at least one of the display devices.</p><p> Further, in order to achieve the above object, the visibility support method according to the present invention converts each of the images captured by n image pickup devices (n is an integer of 2 or more) into a bird's-eye view image viewed from a virtual viewpoint. In the visibility support method of displaying the composite bird's-eye view image obtained by synthesizing each of the obtained bird's-eye view images on the display device, the images of the common area where the plurality of bird's-eye view images overlap at the time of composition are compared between the plurality of bird's-eye view images. A difference area between the plurality of bird's-eye view images in the common area is specified, and one bird's-eye view image is selected as an adopted image from the plurality of bird's-eye view images according to the position of the difference area in the common area. As the image of the common area in the composite bird's-eye view image, the image of the common area in the adopted image is adopted.</p>

<p> According to the present invention, it is possible to display a composite bird's-eye view image that is more suitable for grasping the situation around the vehicle.</p>

Hereinafter, embodiments of the present invention will be specifically described with reference to the drawings. In each of the referenced figures, the same parts are designated by the same reference numerals, and duplicate explanations regarding the same parts will be omitted in principle.

[How to generate a bird's-eye view image] First, a method of generating a bird's-eye view image from an image captured by one camera will be described. In the following description, it is assumed that the ground is on a horizontal plane, and "height" represents a height with respect to the ground.

As shown in FIG. 1, consider a case where the camera 1 is arranged diagonally backward and downward at the rear of the vehicle 100. Vehicle 100 is, for example, a truck. The angle formed by the horizontal plane and the optical axis of the camera 1 is the angle represented by θ in FIG. 1 and θ.₂There are two types, the angle represented by. Angle θ₂Is commonly referred to as the top-down angle or depression angle. Now, the angle θ is taken as the tilt angle of the camera 1 with respect to the horizontal plane. 90 ° &lt;θ &lt;180 ° holds.

FIG. 2 shows the camera coordinate system XYZ and the coordinate system X of the imaging surface S of the camera 1._buY_buAnd the two-dimensional ground coordinate system X_wZ_wWorld coordinate system X including_wY_wZ_wShows the relationship with. The camera coordinate system XYZ is a three-dimensional coordinate system whose coordinate axes are the X-axis, the Y-axis, and the Z-axis. Coordinate system X_buY_buIs X_buAxis and Y_buIt is a two-dimensional coordinate system with axes as coordinate axes. 2D ground coordinate system X_wZ_wIs X_wAxis and Z_wIt is a two-dimensional coordinate system with axes as coordinate axes. World coordinate system X_wY_wZ_wIs X_wAxis, Y_wAxis and Z_wIt is a three-dimensional coordinate system with axes as coordinate axes.

In the camera coordinate system XYZ, with the optical center of camera 1 as the origin O, the Z axis is orthogonal to the optical axis, the X axis is orthogonal to the Z axis, and the X axis is orthogonal to the Z axis and the X axis. The Y axis is taken. Coordinate system X of imaging surface S_buY_buNow, take the origin at the center of the imaging surface S and X in the lateral direction of the imaging surface S._buThe axis is Y in the vertical direction of the imaging surface S._buThe axis is taken.

World coordinate system X_wY_wZ_wThen, the intersection of the vertical line passing through the origin O of the camera coordinate system XYZ and the ground is the origin O._wAnd Y in the direction perpendicular to the ground_wX in the direction in which the axis is parallel to the X axis of the camera coordinate system XYZ_wThe axis is X_wAxis and Y_wZ in the direction orthogonal to the axis_wThe axis is taken.

X_wThe amount of translation between the axis and the X-axis is h, and the direction of translation is the vertical direction. And Z_wThe obtuse angle formed by the axis and the Z axis coincides with the tilt angle θ.

The coordinates in the camera coordinate system XYZ are expressed as (x, y, z). x, y and z are the X-axis component, the Y-axis component and the Z-axis component in the camera coordinate system XYZ, respectively. World coordinate system X_wY_wZ_wCoordinates in (x_w, y_w, z_w). x_w, Y_wAnd z_wAre, respectively, the world coordinate system X_wY_wZ_wIn, X_wShaft component, Y_wShaft component and Z_wIt is an axial component. Two-dimensional ground coordinate system X_wZ_wCoordinates in (x_w, z_w). x_wAnd z_wAre each two-dimensional ground coordinate system X_wZ_wIn, X_WShaft component and Z_WAxis components, they are the world coordinate system X_wY_wZ_wX in_WShaft component and Z_WConsistent with the axis component. Coordinate system X of imaging surface S_buY_buCoordinates in (x_bu, y_bu). x_buAnd y_buAre the coordinate systems X of the imaging surface S, respectively._buY_buIn, X_buShaft component and Y_buIt is an axial component.

Camera coordinate system XYZ coordinates (x, y, z) and world coordinate system X_wY_wZ_wCoordinates (x)_w, y_w, z_wThe conversion formula between) is expressed by the following formula (1).

<maths num="1"><img file="JP4248570B2_D0001.tif" /></maths>

Here, let f be the focal length of camera 1. Then, the coordinate system X of the imaging surface S_buY_buCoordinates (x)_bu, y_bu) And the coordinates (x, y, z) of the camera coordinate system XYZ are expressed by the following equation (2).

<maths num="2"><img file="JP4248570B2_D0002.tif" /></maths>

From the above equations (1) and (2), the coordinate system X of the imaging surface S_buY_buCoordinates (x)_bu, y_bu) And the two-dimensional ground coordinate system X_wZ_wCoordinates (x)_w, z_w) Is obtained.

<maths num="3"><img file="JP4248570B2_D0003.tif" /></maths>

Also, although not shown in FIG. 2, the bird's-eye view coordinate system X, which is the coordinate system for the bird's-eye view image._auY_auTo define. Bird's-eye view coordinate system X_auY_auIs X_auAxis and Y_auIt is a two-dimensional coordinate system with axes as coordinate axes. Bird's-eye view coordinate system X_auY_auCoordinates in (x_au, y_au). The bird's-eye view image is represented by pixel signals of a plurality of pixels arranged in two dimensions, and the position of each pixel on the bird's-eye view image is the coordinates (x)._au, y_au). x_auAnd y_auAre bird's-eye view coordinate system X_auY_auX in_auShaft component and Y_auIt is an axial component.

The bird's-eye view image is obtained by converting an image captured by an actual camera into an image viewed from the viewpoint of a virtual camera (hereinafter referred to as a virtual viewpoint). More specifically, the bird's-eye view image is obtained by converting an image captured by an actual camera into an image looking down on the ground surface in the vertical direction. The conversion of the viewpoint when generating the bird's-eye view image from the captured image is generally called the viewpoint conversion.

Two-dimensional ground coordinate system X_wZ_wFrom the bird's-eye view of the virtual camera Coordinate system X_auY_auProjection to is done by parallel projection. Assuming that the height of the virtual camera (that is, the height of the virtual viewpoint) is H, the two-dimensional ground coordinate system X_wZ_wCoordinates (x)_w, z_w) And bird's-eye view coordinate system X_auY_auCoordinates (x)_au, y_auThe conversion formula between) is expressed by the following formula (4). The height H of the virtual camera is preset. Further, by modifying the equation (4), the following equation (5) can be obtained.

<maths num="4"><img file="JP4248570B2_D0004.tif" /></maths>

<maths num="5"><img file="JP4248570B2_D0005.tif" /></maths>

Substituting the obtained equation (5) into the above equation (3) gives the following equation (6).

<maths num="6"><img file="JP4248570B2_D0006.tif" /></maths>

From the above equation (6), the coordinate system X of the imaging surface S_buY_buCoordinates (x)_bu, y_bu), Bird's-eye view coordinate system X_auY_auCoordinates (x)_au, y_au) Is obtained by the following equation (7).

<maths num="7"><img file="JP4248570B2_D0007.tif" /></maths>

Coordinate system X of imaging surface S_buY_buCoordinates (x)_bu, y_bu) Represents the coordinates in the image captured by the camera 1, so the image captured by the camera 1 is converted into a bird's-eye view image by using the above equation (7). Actually, the image captured by the camera 1 is appropriately subjected to image processing such as lens distortion correction, and the captured image after the image processing is converted into a bird's-eye view image using the above equation (7).

[Basic concept of how to generate a bird's-eye view image of the entire circumference] In the present embodiment, the plan view showing the installation status of the camera on the vehicle 100 is the same as that in FIG. 16 described above, and therefore the overlapping illustration is omitted. 3 and 4 are views of the vehicle 100 viewed diagonally from the front left.

As shown in FIG. 16, cameras (imaging devices) 1F, 1B, 1L and 1R are attached to the front part, the rear part, the left side part and the right side part of the vehicle 100, respectively. In the following description, the cameras 1F, 1B, 1L and 1R may be referred to as a front camera 1F, a rear camera 1B, a left side camera 1L and a right side camera 1R, respectively.

Further, as shown in FIGS. 3 and 4, the camera 1F is installed on the upper part of the front mirror of the vehicle 100, for example, and the camera 1L is installed on the uppermost part of the left side surface of the vehicle 100, for example. Although not shown in FIGS. 3 and 4, the camera 1B is installed, for example, at the top of the rear of the vehicle 100, and the camera 1R is installed, for example, at the top of the right side of the vehicle 100.

The optical axis of the camera 1F is diagonally downward in the front of the vehicle 100, the optical axis of the camera 1B is diagonally downward in the rear of the vehicle 100, and the optical axis of the camera 1L is diagonally to the left of the vehicle 100. The cameras 1F, 1B, 1L and 1R are installed in the vehicle 100 so as to face downward and the optical axis of the camera 1R faces diagonally downward to the right of the vehicle 100.

The height of cameras 1L and 1R shall be higher than the height of camera 1F. Further, the vehicle 100 shall be located on the ground.

3 and 4 show the field of view of each camera, that is, the imaging region of each camera. Each camera produces a captured image of a subject in its field of view. The fields of view of cameras 1F, 1B, 1L and 1R are represented by 12F, 12B, 12L and 12R, respectively. The fields of view 12R and 12B are only partially shown in FIGS. 3 and 4.

The field of view 12F of the camera 1F includes a three-dimensional object located within a predetermined range in front of the vehicle 100 and the ground in front of the vehicle 100 with reference to the installation position of the camera 1F. The field of view 12B of the camera 1B includes a three-dimensional object located within a predetermined range behind the vehicle 100 and the ground behind the vehicle 100 with respect to the installation position of the camera 1B. The field of view 12L of the camera 1L includes a three-dimensional object located within a predetermined range on the left side of the vehicle 100 and the ground on the left side of the vehicle 100 based on the installation position of the camera 1L. The field of view 12R of the camera 1R includes a three-dimensional object located within a predetermined range on the right side of the vehicle 100 and the ground on the right side of the vehicle 100 with respect to the installation position of the camera 1L.

In this way, the viewpoints of each camera are different, and the subjects that fit within the field of view (angle of view) of each camera are different. A three-dimensional object is a tall object such as a person. The road surface that forms the ground is not a three-dimensional object because it has no height.

The cameras 1F and 1L commonly capture a predetermined area diagonally to the left of the vehicle 100. That is, the fields of view 12F and 12L overlap in a predetermined area diagonally to the left of the vehicle 100. This overlapping portion is referred to as a common field of view (common imaging space).

Similarly, the fields of view 12F and 12R overlap each other in a predetermined area diagonally to the right of the vehicle 100 to form a common field of view, and the fields of view 12B and 12L overlap to form a common field of view in a predetermined area diagonally to the left of the vehicle 100. A field of view is formed, and the fields of view 12B and 12R overlap each other in a predetermined area diagonally to the right and rear of the vehicle 100 to form a common field of view.

The common field of view between the fields of view 12F and 12L is particularly referred to as the common field of view 13, and the following description will be given focusing on this common field of view 13. The same processing is performed for the common visual field other than the common visual field 13.

In FIG. 4, the common field of view 13 is shown using thick lines. The common field of view 13 is a space similar to a cone whose bottom surface is the ground diagonally to the left of the vehicle 100. As shown in FIG. 4, it is assumed that the three-dimensional object 14 exists in the common visual field 13.

In the present embodiment, as shown in FIG. 5, bird's-eye view images 10F, 10B, 10L and 10R are generated from the captured images obtained from the cameras 1F, 1B, 1L and 1R, respectively, using the above equation (7). Next, by rotating and / or translating the other three bird's-eye view images 10F, 10L and 10R with reference to the bird's-eye view image 10B corresponding to the camera 1B, those (10F, 10L and 10R) are moved in the bird's-eye view image 10B. Convert to coordinates. As a result, the coordinates of each bird's-eye view image are converted into the coordinates of the all-around bird's-eye view image. Hereinafter, the coordinates in the all-around bird's-eye view image will be referred to as "all-around bird's-eye view coordinates".

FIG. 6 shows bird's-eye view images 10F, 10B, 10L and 10R represented on the coordinates of the bird's-eye view. When considered in terms of all-around bird's-eye view coordinates, as shown in FIG. 6, there is a portion where the two bird's-eye view images overlap.

In Figure 6, C_FLThe shaded area marked with is the part where the bird's-eye view images 10F and 10L overlap on the coordinates of the bird's-eye view, and this is the common area C._FLCalled. Common area C in bird's-eye view image 10F_FLThe image of the subject in the common field of view 13 (see Fig. 4) seen from the camera 1F appears in the common area C in the bird's-eye view image 10L._FLThe image of the subject in the common field of view 13 seen from the camera 1L appears in. The common area can also be called an overlapping area in which a plurality of bird's-eye view images overlap.

Common area C_FLBesides, the common area C where the bird's-eye view images 10F and 10R overlap._FRAnd the common area C where the bird's-eye view images 10B and 10L overlap_BLAnd the common area C where the bird's-eye view images 10B and 10R overlap_BRHowever, in particular, the common area C corresponding to the common field of view 13._FLThis embodiment will be described with a focus on.

In FIGS. 5 and 6, the XF axis and the YF axis are the coordinate axes of the coordinate system of the bird's-eye view image 10F, and they are X._auAxis and Y_auCorresponds to the axis. Similarly, the XR and YR axes are the coordinate axes of the bird's-eye view image 10R coordinate system, and they are X._auAxis and Y_auCorresponds to the axis. Similarly, the XL and YL axes are the coordinate axes of the bird's-eye view image 10L coordinate system, and they are X._auAxis and Y_auCorresponds to the axis. Similarly, the XB axis and the YB axis are the coordinate axes of the coordinate system of the bird's-eye view image 10B, and they are X._auAxis and Y_auCorresponds to the axis.

Further, in FIG. 6, the common area C is shown for simplification of the illustration._FLIs a rectangle, but in reality, the common area C_FLIs not a rectangle. Also, each bird's-eye view image is not always rectangular. In Fig. 7, the area where each bird's-eye view image appears and the common area C_FLWill be shown more specifically. FIGS. 7 (a) and 7 (b) represent the bird's-eye view images 10L and 10F in the all-around bird's-eye view coordinates, respectively, and in FIG. 7 (c), their common area C._FLIs shown in the shaded area. However, in FIGS. 7 (a) and 7 (c), the illustration of the image from the rear of the vehicle is omitted. This common area C_FLThe feature of this embodiment is how to combine the two bird's-eye view images.

[Specific method for generating a bird's-eye view image of the entire circumference] FIG. 8 shows an overall configuration diagram of the visibility support device (vehicle peripheral visibility support device) according to the present embodiment. The visibility support device according to the present embodiment is an image processing unit 2 that generates an all-around bird's-eye view image from the cameras 1F, 1B, 1L and 1R attached to the vehicle 100 as described above and the captured images obtained by each of these cameras. And a display unit 3 for displaying an all-around bird's-eye view image generated by the image processing unit 2. The all-around bird's-eye view image is basically the same as that shown in FIG. However, this embodiment is characterized by the above-mentioned synthesis method in the common region.

As cameras 1F, 1B, 1L and 1R, for example, a camera using a CCD (Charge Coupled Devices) or a camera using a CMOS (Complementary Metal Oxide Semiconductor) image sensor is used. The image processing unit 2 is formed of, for example, an integrated circuit. The display unit 3 is formed of a liquid crystal display panel or the like. A display device included in a car navigation system or the like may be diverted as a display unit 3 in the visual field support device.

FIG. 9 shows a flowchart showing the processing procedure of the visual field support device of FIG. The processing of steps S2 to S6 shown in FIG. 9 is performed by the image processing unit 2, the processing of step S1 is performed by each camera and the image processing unit 2, and the processing of step S7 is performed by the image processing unit 2 and the display unit 3. Will be done.

First, the image processing unit 2 reads the captured images of each camera 1F, 1B, 1L, and 1R (step S1). Next, each captured image is converted into a bird's-eye view image using conversion table data or the like (step S2). As a result, the above-mentioned bird's-eye view images 10F, 10B, 10L and 10R are generated. The conversion table data is preset according to the above equation (7). At this time, each captured image may be subjected to necessary image processing such as lens distortion correction and then converted into a bird's-eye view image.

Subsequently, in step S3, a difference image in the common area between the two bird's-eye view images is generated, and the difference area in the common area between the two bird's-eye view images is detected based on the difference image. As explained with reference to FIG. 6, there are four common areas, but the common area C_FLThe explanation will be given with particular attention to. The same processing is applied to other common areas.

Common area C_FLThe difference image and the difference area will be described. As mentioned above, common area C_FLIt is assumed that the three-dimensional object 14 exists in the common field of view 13 corresponding to (see FIG. 4).

See Figure 10. In FIG. 10, reference numeral 111 indicates a common area C in the bird's-eye view image 10L corresponding to the left lateral camera 1L._FLThe symbol 112 represents the common area C in the bird's-eye view image 10F corresponding to the front camera 1F._FLRepresents the image of. The shaded area 121 in the image 111 is the drawing area of the three-dimensional object 14 in the image 111, and the shaded area 122 in the image 112 is the drawing area of the three-dimensional object 14 in the image 112. Now, for the sake of simplification of the explanation, it is assumed that the uniform ground (road surface, etc.) is drawn in the image 111 except for the area 121, and the uniform ground (road surface, etc.) is drawn in the image 112 except for the area 122. It is assumed that it has been done.

In step S3, the image processing unit 2 generates the difference image 113 by taking the difference between the image 111 and the image 112. For example, the images 111 and 112 are captured as a shade image, and the difference image 113 between the image 111 and the image 112 is generated as a shade image. The pixel value (pixel signal value) of each pixel of the difference image 113 is represented by the difference (or the absolute value of the difference) between the pixel value of each pixel of the image 111 and the pixel value of each pixel of the image 112. Then, the difference region 123 is specified from the difference image 113. In FIG. 10, the difference region 123 is a shaded region in the difference image 113. For example, in the difference image 113, a region in which the value of the pixel signal of the pixels forming the difference image 113 is equal to or greater than a predetermined threshold value is defined as the difference region 123. The pixel signal is a luminance signal representing the brightness of a pixel or a color signal (color difference signal or the like) representing the color of a pixel.

The difference area 123 is a combination of the area 121 and the area 122. When converting to a bird's-eye view image, a predetermined conversion is performed so that each bird's-eye view image has continuity on the ground surface, so that the difference image 113 has a certain size only in the region involved in the existence of the three-dimensional object. A difference value having is obtained. Therefore, the difference region 123 can be detected by the above-mentioned processing. Further, in the difference image 113, edge detection may be performed to detect the contour surrounding the difference region 123, and the inside of the detected contour may be specified as the difference region 123. In addition, edge detection is performed individually for each of the images 112 of the image 111, the difference in the edge detection result (presence or absence of an edge) between the image 111 and the image 112 is extracted, and the difference region 123 is extracted from the different image portion. It may be detected.

The process of step S4 following step S3 will be described. Common area C_FLWill be described as an example of the process of step S4. The same processing is applied to other common areas.

In step S4, the weight value table data set for each camera is referred to. Common area C_FLWith respect to, the weight value table data corresponding to the front camera 1F and the weight value table data corresponding to the left side camera 1L are set in advance.

Each weight value table data has a common area C_FLThe weight value corresponding to each pixel position of the image of is defined. For example, common area C_FLIs divided into three regions AR1, AR2 and AR3, as shown in FIG. Then, the weight value of the pixel belonging to the area AR1, the weight value of the pixel belonging to the area AR2, and the weight value of the pixel belonging to the area AR3 are stored in each weight value table data.

Common area C_FLThe weight value determined by the relationship with the pixel position of the image is set to be different between different weight value table data. FIG. 12 shows an example of setting the weight value corresponding to the front camera 1F and the weight value corresponding to the left side camera 1L. For example, in the weight value table data corresponding to the front camera 1F, the weight value W of the pixels belonging to the area AR1_F1Is 1, and the weight value W of the pixels belonging to the area AR2_F2Is 2, and the weight value W of the pixels belonging to the area AR3_F3Is set to 3. On the other hand, in the weight value table data corresponding to the left horizontal camera 1L, the weight value W of the pixels belonging to the area AR1_L1Is 2, and the weight value W of the pixels belonging to the area AR2_L2Is 3, and the weight value W of the pixels belonging to the area AR3_L3Is 2.

In step S4, the weight values corresponding to the pixels belonging to the difference region 123 are added up for each camera (in other words, for each bird's-eye view image), and the totaled value is used as the weight evaluation value. For convenience of explanation, as shown in FIG. 13, the difference region 123 is decomposed into a partial difference region 123a corresponding to the region 121 (see FIG. 10) and a partial difference region 123b corresponding to the region 122.

FIG. 14 shows the number of pixels in the partial difference region 123a and the number of pixels in the partial difference region 123b belonging to each region AR1 to AR3. Now, assuming that the number of pixels in the partial difference region 123a belonging to the regions AR1, AR2 and AR3 is 15, 18 and 2, respectively, the number of pixels in the partial difference region 123b belonging to the regions AR1, AR2 and AR3. However, it is assumed that they are 12, 5 and 0, respectively.

In this case, the weight evaluation value for the front camera 1F is "W_F1× (15 + 12) + W_F2× (18 + 5) + W_F3It is represented by "× (2 + 0)" As mentioned above, "W_F1= 1, W_F2= 2, W_F3If = 3 is set, it becomes 1 × (15 + 12) + 2 × (18 + 5) + 3 × (2 + 0) = 79 . On the other hand, the weight evaluation value for the left side camera 1L is "W_L1× (15 + 12) + W_L2× (18 + 5) + W_L3It is represented by "× (2 + 0)" As mentioned above, "W_L1= 2, W_L2= 3, W_L3If = 2 is set, it becomes 2 × (15 + 12) + 3 × (18 + 5) + 2 × (2 + 0) = 127 .

In step S5 following step S4, a priority camera that provides a bird's-eye view image including an image to be adopted as an image of a common area is selected based on the weight evaluation value calculated in step S4. As mentioned above, common area C_FLWhen the weight evaluation value for the front camera 1F is 79 and the weight evaluation value for the left side camera 1L is 127, the left side camera 1L corresponding to the larger weight evaluation value is selected as the priority camera. In other words, the bird's-eye view image 10L corresponding to the left horizontal camera 1L is selected as the adopted bird's-eye view image (adopted image). The priority camera is similarly selected for other common areas.

Then, in step S6, the image processing unit 2 generates an all-around bird's-eye view image based on the selection result of the priority camera. That is, the all-around bird's-eye view image is generated after adopting the image of the common area in the bird's-eye view image based on the image captured by the priority camera as the image of the common area in the all-around bird's-eye view image. For example, common area C_FLWhen the left horizontal camera 1L is selected as the priority camera, the common area C in the bird's-eye view image 10L_FLImage of common area C in the bird's-eye view image_FLThe image is. The same applies to other common areas.

FIG. 15 shows an example of the obtained bird's-eye view image of the entire circumference. The whole image of the three-dimensional object appears in this bird's-eye view image. Bird's-eye view images 10F, 10B, 10L and 10R are arranged as shown in FIG. 6 for images other than the common area in the all-around bird's-eye view image. That is, on the all-around bird's-eye view image, images based on the bird's-eye view images 10F, 10B, 10L, and 10R are drawn on the front, rear, left, and right sides of the drawing area of the vehicle, respectively.

In step S7 following step S6, the image processing unit 2 generates a video signal necessary for displaying the all-around bird's-eye view image generated in step S6 on the display unit 3 and outputs it to the display unit 3. As a result, the bird's-eye view image of the entire circumference as shown in FIG. 15 is displayed on the display unit 3. When step S7 is completed, the process returns to step S1 and the processes of steps S1 to S7 are repeated in order to periodically update the bird's-eye view image of the entire circumference to the latest image.

The above-mentioned weight value is set according to the installation position of each camera with respect to the vehicle 100. At this time, the shape of the vehicle 100 is also taken into consideration. In the present embodiment, as shown in FIG. 3, the vehicle 100 is a truck formed by a driver's cab and a luggage compartment higher than the driver's cab, and the front camera 1F is installed in the upper part of the driver's cab and the left side camera. It is assumed that 1L will be installed at the upper part of the luggage compartment, which is at a higher position.

The front camera 1F can accurately capture a three-dimensional object at a position close to itself, but it is difficult to capture the entire image of a three-dimensional object at a position far from itself due to the low installation position. On the other hand, the left horizontal camera 1L makes it easy to capture the entire image of a three-dimensional object by viewing the three-dimensional object from a higher position, but when the three-dimensional object is near the front camera 1F, it captures the entire image of the three-dimensional object. It becomes difficult.

Taking these factors into consideration, as shown in FIG. 12, the front camera is located near the front camera 1F so that the front camera 1F can be easily selected as the priority camera when a three-dimensional object exists at a position relatively close to the front camera 1F. Set the weight value for 1F relatively large. On the other hand, the weight values for the left side camera 1L are compared in the area away from the front camera 1F so that the left side camera 1L is easily selected as the priority camera when a three-dimensional object exists at a position away from the front camera 1F. Set large.

By setting the weight value according to the installation position of each camera in this way, the entire image of the three-dimensional object can be easily displayed as shown in FIG. In other words, the camera that is expected to be able to acquire an image that makes it easier for the driver to grasp the situation around the vehicle is automatically selected, and an image of the common area in the bird's-eye view image of the entire circumference is formed. It is possible to display an image suitable for grasping.

If the position and shape of the difference region in the common region are different, the priority camera selected may be different. Further, even if the shape of the difference region is the same, the selected priority camera may be different if the position of the difference region in the common region is different. That is, in the present embodiment, the priority camera is selected according to the position of the difference region in the common region (in other words, according to the position of the three-dimensional object in relation to the vehicle 100). The advantages of adopting this selection method are understood from the above description. That is, for example, the common area C_FLRegarding, if a three-dimensional object exists at a position relatively close to the front camera 1F, the front camera 1F is likely to be selected as the priority camera, and if a three-dimensional object exists at a position relatively far from the camera 1F, the three-dimensional object is selected. The horizontal camera 1L viewed from a high position is more likely to be selected as the priority camera. As a result, it is possible to display an image that is more suitable for grasping the situation than before.

Further, in step S5, the above-mentioned weight evaluation values may be corrected according to the driving state of the vehicle 100, and the priority camera (priority bird's-eye view image) may be selected based on the corrected weight evaluation values.

The driving state includes the state of the gear of the vehicle 100, the operating direction of the steering wheel (the traveling direction of the vehicle 100), the speed, and the like. Information on the driving state is given to the image processing unit 2 from a part for detecting the information (the part is provided in, for example, the vehicle 100).

Consider the case where the weight evaluation values of the front camera 1F and the left side camera 1L before the correction are 79 and 127, respectively, as in the above example. For example, when the vehicle 100 is turning left, the weight evaluation value for the front camera 1F is multiplied by a coefficient 1, and the weight evaluation value for the left side camera 1L is multiplied by a coefficient 2. In this case, the weight evaluation values of the front camera 1F and the left side camera 1L after correction are 79 (= 79 × 1) and 254 (= 127 × 2), respectively, and 79 &lt;254, so the left side camera 1L. Is selected as the preferred camera. Further, for example, when the vehicle 100 is moving forward, the weight evaluation value for the front camera 1F is multiplied by a coefficient 2, and the weight evaluation value for the left side camera 1L is multiplied by a coefficient 1. In this case, the weight evaluation values of the front camera 1F and the left side camera 1L after correction are 158 (= 79 × 2) and 127 (= 127 × 1), respectively, and 158&gt; 127, so the front camera 1F is Selected as the preferred camera.

When the vehicle 100 is turning left, the need for the image on the left side of the vehicle 100 is relatively large, while the need for the image in front of the vehicle 100 is relatively small. When the vehicle 100 is moving forward, the need for the image on the left side of the vehicle 100 is relatively small, while the need for the image in front of the vehicle 100 is relatively large. In consideration of this, the coefficient is set as described above to correct each weight evaluation value. This makes it possible to display an appropriate image in consideration of the driving state.

Although examples have been given for turning left and moving forward, the weight evaluation value is corrected in the same way according to the driving conditions such as turning right and moving backward. Further, the degree of correction may be changed according to the state and speed of the gear of the vehicle 100.

The vision support device described above is merely an example of an embodiment of the present invention, and the present invention includes various modifications (or examples). Hereinafter, Modifications 1 to 5 will be illustrated as modifications (or examples) relating to the present invention. The contents described in each modification can be arbitrarily combined as long as there is no contradiction.

[Modification example 1] The specific numerical values shown in the above description are merely examples, and as a matter of course, they can be changed to various numerical values. The above-mentioned weight value is appropriately changed according to the installation position of each camera, the shape of the vehicle in which each camera is installed, and the like.

[Modification 2] Common area C_FLHas been shown as an example of dividing the above into three regions AR1 to AR3, but the number of divided regions may be other than three. In addition, the common area C between the cameras 1F and 1L (bird's-eye view images 10F and 10L)_FLAlthough the case where the method of dividing the area is the same is illustrated, the method of dividing may be different between different cameras.

Moreover, the division of such an area is not indispensable. That is, common area C_FLIt suffices if the weight value is determined corresponding to the pixel position of each pixel of the image, and in the extreme, the weight values corresponding to any two different pixel positions may be different.

Further, in the weight value setting example shown in FIG. 12, the weight values corresponding to the same pixel position are all different between the cameras 1F and 1L (bird's-eye view images 10F and 10L), but only a part is different. It doesn't matter.

[Modification 3] Further, the image processing unit 2 of FIG. 8 can be realized by hardware, software, or a combination of hardware and software. All or a part of the functions realized by the image processing unit 2 may be described as a program, and the program may be executed on a computer to realize all or a part of the functions.

[Modification example 4] The image processing unit 2 generates a composite bird's-eye view image by synthesizing the viewpoint conversion means for converting each of the captured images of each camera into a bird's-eye view image by the viewpoint conversion and each bird's-eye view image obtained by the conversion. It has the means. As a composite bird's-eye view image, an all-around bird's-eye view image is illustrated in the above-described embodiment.

[Modification 5] In the above-described embodiment, the truck is illustrated as the vehicle 100, but the present invention can be applied to ordinary passenger cars and the like regardless of the type of vehicle.

Further, it is possible to install a plurality of cameras (for example, the above-mentioned cameras 1F, 1B, 1L and 1R) other than the vehicle. That is, the present invention can also be applied to a monitoring system installed in a building or the like. Also in this type of monitoring system, as in the above embodiment, a common field of view exists between the plurality of cameras, and a common area (C) is provided between different bird's-eye view images._FLEtc.). When synthesizing each bird's-eye view image to form a composite bird's-eye view image, it is preferable to select a priority camera by the same method as in the above-described embodiment. The present invention functions particularly effectively when the heights of a plurality of cameras forming a common area are different.

<figref num="1">It is a top view of a vehicle in which one camera is installed as viewed from the side, and is a diagram for explaining a method of generating a bird's-eye view image from an image captured by one camera.</figref><figref num="2">Camera coordinate system XYZ and coordinate system X of the imaging surface of the camera_buY_buAnd the two-dimensional ground coordinate system X_wZ_wWorld coordinate system X including_wY_wZ_wIt is a figure which shows the relationship between.</figref><figref num="3">It is a figure which looked at the vehicle which concerns on embodiment of this invention from the diagonally left front.</figref><figref num="4">It is a figure which looked at the vehicle which concerns on embodiment of this invention from the diagonally left front.</figref><figref num="5">It is a figure which shows each bird's-eye view image which concerns on embodiment of this invention.</figref><figref num="6">It is the figure which converted each bird's-eye view image shown in FIG. 5 into the coordinates of the all-around bird's-eye view image.</figref><figref num="7">It is a figure which clearly shows each bird's-eye view image generated from the images taken by the front camera and the horizontal camera shown in FIG. 3, and their common areas.</figref><figref num="8">It is an overall block diagram of the view support device which concerns on embodiment of this invention.</figref><figref num="9">It is a flowchart which shows the processing procedure of the visual field support apparatus of FIG.</figref><figref num="10">It is a figure which shows the difference image and the difference area in the common area of FIG.</figref><figref num="11">It is a figure for demonstrating the process of step S4 of FIG. 9, and is the figure which shows the state which divided the common area of FIG. 7 into three areas.</figref><figref num="12">It is a figure for demonstrating the process of step S4 of FIG. 9, and is the figure which shows the weight value for the front camera and the weight value for a left side camera with respect to the common area of FIG.</figref><figref num="13">It is a figure for demonstrating the process of step S4 of FIG.</figref><figref num="14">It is a figure for demonstrating the process of step S4 of FIG. 9, and is the figure which illustrates the number of pixels belonging to each area of FIG.</figref><figref num="15">It is a figure which shows the example of the all-around bird's-eye view image generated by the image processing part of FIG.</figref><figref num="16">It is a top view which shows the situation where a plurality of cameras are installed in a vehicle.</figref><figref num="17">It is the figure which looked at the vehicle of FIG. 16 from the diagonally left front.</figref><figref num="18">It is a figure which shows the all-around bird's-eye view image generated from the captured image obtained from each camera of FIG.</figref><figref num="19">It is the figure which looked at the vehicle of FIG. 16 from the diagonally left front.</figref><figref num="20">It is a figure for demonstrating the conventional method at the time of generating the all-around bird's-eye view image of FIG.</figref><figref num="21">It is a figure for demonstrating the conventional method at the time of generating the all-around bird's-eye view image of FIG.</figref><figref num="22">It is a figure which shows the example of the display image when the conventional method is used.</figref><figref num="23">It is a figure which shows another example of the display image when the conventional method is used.</figref>

Code description

1,1F, 1B, 1L, 1R cameras 2 Image processing unit 3 Display 10F, 10B, 10L, 10R bird's-eye view image 12F, 12B, 12L, 12R field of view 13 Common field of view 14 Three-dimensional object C_FL Common area 100 vehicles 113 Difference image 123 Difference area