Vehicle peripheral observation device

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a Continuation of U.S. application Ser. No. 13/636,041 filed Sep. 19, 2012, which is a National Stage of International Application No. PCT/JP2011/052136, filed on Feb. 2, 2011, which claims priority from Japanese Patent Application No. 2010-072326, filed Mar. 26, 2010, the contents of all of which are incorporated herein by reference in their entirety.

TECHNICAL FIELD

The present invention relates to a vehicle peripheral observation device, and more specifically relates to a technique for displaying an image captured using a camera on a monitor and enlarging the display of the captured image displayed on the monitor.

BACKGROUND ART

As a vehicle peripheral observation device configured as described above, Patent Document 1 discloses a processing format in which an overhead image of the vehicle periphery is generated from images captured using a plurality of cameras and displayed on a monitor (the display in the document), the overhead image divides the peripheral area of the vehicle into a plurality of areas, and when a user selects any of these areas with a touch panel, an overhead image of the selected area is enlarged and displayed on the monitor.

Patent Document 1 also discloses that the area displaying the overhead image and the area displaying the enlarged image in the monitor are set separately, and it is possible to make a selection for the enlarged image between a mode of enlarging the overhead image and a mode of enlarging the image captured using the camera that images the selected area (the camera-view image).

PRIOR ART DOCUMENTS

Patent Documents

[Patent Document 1] Japanese Laid-open Patent Application No. 2009-239674

DISCLOSURE OF THE INVENTION

Problems that the Invention is Intended to Solve

In a mode where part of the overhead image is enlarged and displayed on the monitor, when the enlarged image is simply displayed on the monitor, it is difficult to distinguish which region the enlarged image is in. In a setup in which a comparatively small area is indicated and the area is enlarged as disclosed in Patent Document 1, the driver will sometimes perform an accidental operation and thereby enlarge the image of an undesired area. When such an accidental operation is performed, not only does the proper operation need to be additionally performed, but it also takes time to notice the accidental operation, leaving room for improvement.

An object of the present invention is to rationally configure a device whereby it is possible to accurately recognize which area is enlarged when part of an overhead image displayed on a monitor is displayed as enlarged.

Means for Solving the Problems

The present invention comprises a plurality of cameras for imaging areas including the road surface of a vehicle periphery, and an image processor for displaying the images captured using the plurality of cameras on a monitor;

the image processor having overhead-image-generating unit for generating an overhead image looking down on the vehicle periphery from above from the images captured using the plurality of cameras, and displaying the overhead image in an overhead image display area of the monitor; and enlarged display unit for generating, when any of a plurality of divided areas set by dividing the overhead image display area is indicated, a plurality of intermediate images of an intermediate size between the size of the indicated divided area and the size of the enlarged display area; executing a process of sequentially displaying the intermediate images beginning with the smallest size; setting a display-position relationship at the start of the process so that the indicated divided area is included in the display area of the next intermediate image to be displayed; and using the image of the indicated divided area as an image to be enlarged, and enlarging the image to the size of the enlarged display area.

According to this configuration, when a divided area is indicated, an image of the divided area is displayed in the enlarged display area. When the enlargement is performed, a process is executed for generating a plurality of intermediate images of an intermediate size between the size of the divided area and the size of the enlarged display area, and sequentially displaying the intermediate images beginning with the smallest intermediate image. The display-position relationship is set at the start of this process so that the indicated divided area is included in the divided area of the next intermediate image to be displayed. The final enlarged image is displayed in the enlarged display area in a format such that the continuous display of intermediate images has resulted in a zooming in from such the display format, with the indicated divided image as the origin point.

Particularly, in addition to being indicated manually, the divided area can also be indicated by automatic control when an obstacle is detected, for example.

As a result, the device is configured such that it is possible to accurately recognize which area has been enlarged when part of the overhead image displayed on the monitor is displayed as enlarged.

In the present invention, the enlarged display unit may set the overhead image display area as the enlarged display area, cause the intermediate images to be laid over the overhead images displayed in the divided areas other than the indicated divided area, and enlarge the displays thereof.

According to this configuration, because the intermediate images are displayed in a positional relationship of being superimposed over the overhead images already displayed, the driver can be informed that the overhead images already displayed have been updated.

In the present invention, at least one of the overhead images displayed in the plurality of divided areas may be generated by combining overhead images generated from the images captured using two cameras, and when the divided area is indicated, the overhead image displayed in the indicated divided area may be set as the image to be enlarged.

According to this configuration, in a state in which a divided area displays an overhead image that is a combination of overhead images generated from the images captured using two cameras, the combined overhead images are enlarged when this overhead image is set as the image to be enlarged.

In the present invention, the monitor may have a touch panel in a display surface, and when touching of any of the plurality of divided areas is sensed by the touch panel, the enlarged display unit may indicate the overhead image of the touched divided area as the image to be enlarged.

According to this configuration, when any divided area is indicated via the touch panel, an enlarge process is implemented in which the overhead image displayed in the indicated divided area is set as the image to be enlarged. Because the image thus enlarged is an overhead image, the conditions of the vehicle periphery can be perceived from a viewpoint of looking down on the vehicle from above.

In the present invention, the monitor may have a touch panel in a display surface, and when touching of any of the plurality of divided areas is sensed by the touch panel, the enlarged display unit may display, in the divided area where touching was sensed, an image captured using the camera that imaged the area displayed in the divided area instead of the overhead image already displayed in the divided area; and may indicate the thus-displayed captured image as the image to be enlarged.

According to this configuration, when any divided area is indicated via the touch panel, an enlarge process is implemented in which instead of the overhead image already displayed in the indicated divided area, the image captured using the camera for imaging the area displayed in this divided area is set as the image to be enlarged. Because the image thus enlarged is the image captured by the camera, the conditions of the vehicle periphery can be perceived from the viewpoint of the camera.

In the present invention, the enlarged display unit may display a borderline clarifying the border in the outer periphery of the intermediate image.

According to this configuration, when intermediate images are sequentially displayed, the border between the intermediate image and the image already displayed becomes distinct, and it is possible to easily assess which divided area is used as a basis for the enlarged display.

BRIEF DESCRIPTION OF THE DRAWINGS

<figref idref="DRAWINGS">FIG. 1</figref> is a plan view showing an overview of the configuration of the vehicle;

<figref idref="DRAWINGS">FIG. 2</figref> is a partially cutaway perspective view showing the configuration of the vehicle;

<figref idref="DRAWINGS">FIG. 3</figref> is a block circuit diagram of the control system;

<figref idref="DRAWINGS">FIG. 4</figref> is a flowchart of the vehicle periphery observation process;

<figref idref="DRAWINGS">FIG. 5</figref> is a continuous depiction of the process of enlarging the overhead image of a divided area to the auxiliary display area;

<figref idref="DRAWINGS">FIG. 6</figref> is a continuous depiction of the process of enlarging the overhead image of a divided area to the auxiliary display area;

<figref idref="DRAWINGS">FIG. 7</figref> is a continuous depiction of the process of enlarging the overhead image of a divided area to the primary display area; and

<figref idref="DRAWINGS">FIG. 8</figref> is a continuous depiction of the process of enlarging the overhead image of a divided area to the entire screen of the monitor.

BEST MODE FOR CARRYING OUT THE INVENTION

An embodiment of the present invention will be described below with reference to the accompanying drawings.

In the present embodiment, provided is a vehicle peripheral observation device for generating an overhead image in the format of looking down on the periphery of a vehicle <b>1</b> from above from images captured using a plurality of cameras provided to the vehicle <b>1</b>, and displaying the overhead image on a monitor <b>21</b>. Particularly, when the vehicle peripheral observation device detects an obstacle X in the vicinity of the vehicle, the overhead image displayed on the monitor <b>21</b> is divided into a plurality of parts, divided areas D<b>1</b> to D<b>4</b> are set, an emphasis display is performed for specifying the divided areas D<b>1</b> to D<b>4</b> including the obstacle X, and control is executed for displaying an enlarged image of the area containing the obstacle X on the monitor <b>21</b> (hereinbelow referred to as an enlarge process). The format and configuration of such control are described hereinbelow.

The vehicle <b>1</b> as a passenger vehicle comprises left and right front wheels <b>2</b> and left and right rear wheels <b>3</b>, as shown in <figref idref="DRAWINGS">FIGS. 1 and 2</figref>. A left front door <b>4</b> and a right front door <b>5</b> that can open and close by hinges are provided to positions in the front of the vehicle <b>1</b>, a left rear door <b>6</b> and a right rear door <b>7</b> that can open and close in a sliding manner are provided to positions in the rear of the vehicle <b>1</b>, and a hatchback type back door <b>8</b> is provided in the rear end of the vehicle <b>1</b>.

A driver seat <b>13</b>, a passenger seat <b>14</b>, and a plurality of rear seats <b>15</b> are provided inside the interior of the vehicle <b>1</b>. A steering wheel <b>16</b> is provided in a position in front of the driver seat <b>13</b>, and a panel having gauges is disposed in the front part of the steering wheel. At the foot of the driver seat <b>13</b> are disposed an acceleration pedal <b>17</b> and a brake pedal <b>18</b> for operating braking devices BK of the front wheels <b>2</b> and rear wheels <b>3</b> and exerting braking force on the front wheels <b>2</b> and rear wheels <b>3</b>, and a shift lever <b>19</b> for implementing a speed change is disposed in the side of the driver seat <b>13</b>.

The monitor <b>21</b>, which has a touch panel <b>21</b>T formed in a surface display, is provided in proximity to the driver seat <b>13</b> and in a position in the top part of a console. The monitor <b>21</b> is a liquid crystal device having a backlight. As shall be apparent, the monitor may also use a plasma display or a CRT. The touch panel <b>21</b>T is configured as a pressure-sensitive or electrostatic command input device for outputting the contact position of a finger or the like as location data. The housing <b>21</b>H of the monitor <b>21</b> is also provided with a speaker <b>22</b>, but the speaker <b>22</b> may also be provided in another location such as the inner side of a door. The monitor <b>21</b> can also serve as a display device of a navigation system, but a monitor that is used solely for a vehicle peripheral observation display may also be used.

The left front door <b>4</b> is provided with a left side mirror <b>23</b>, and the right front door <b>5</b> is provided with a right side mirror <b>24</b>. The left side mirror <b>23</b> is provided with a left side camera <b>25</b>, and the right side mirror <b>24</b> is provided with a right side camera <b>26</b>. The front end of the vehicle <b>1</b> is provided with a front camera <b>27</b>, and the rear end of the vehicle <b>1</b> is provided with a rear camera <b>28</b>.

Particularly, the imaging directions of the left side camera <b>25</b> and right side camera <b>26</b> are oriented downward, and part of the vehicle <b>1</b> and the road surface are included in the imaging area. The imaging directions of the front camera <b>27</b> and rear camera <b>28</b> are oriented at a downward slant, and part of the vehicle <b>1</b> and the road surface are included in the imaging area.

These cameras, used as imaging devices, are digital cameras equipped with charge coupled devices (CODs), CMOS image sensors (CISs), or other imaging elements, and the cameras output imaged information as video information in real time. The cameras are ensured a wide field of vision by being provided with wide-angle lenses.

The monitor <b>21</b> is used to display navigation information during navigation mode control, but images captured by the rear camera <b>28</b> are displayed when the shift lever <b>19</b> has been operated to the reverse position. Particularly, the image captured by the rear camera <b>28</b> is an image in which left and right are interchanged by left-right mirror reflection, and the content displayed in the monitor <b>21</b> can be visually confirmed with the same sensation as that of using the rearview mirror to confirm the rearward view.

The front part of the vehicle is provided with a power steering unit PS for transmitting the rotational operation force of the steering wheel <b>16</b> to the front wheels <b>2</b> and performing drive steering. The front part of the vehicle is also provided with an engine E, and a speed change mechanism T consisting of a torque converter, a CVT, or the like for changing the speed of motive power from the engine E and transmitting it to the front wheels <b>2</b>. Switching between forward and reverse and changing the travel speed of the speed change mechanism T are performed by operating the shift lever <b>19</b>.

The vehicle <b>1</b> is provided with various sensors for detecting the driving operation and the movement state of the vehicle <b>1</b>. More specifically, the operating system of the steering wheel <b>16</b> is provided with a steering sensor S<b>16</b> for measuring the steering operation direction (the steering direction) and the operated amount (steered amount). The operating system of the shift lever <b>19</b> is provided with a shift position sensor S<b>19</b> for distinguishing the shift position. The operating system of the acceleration pedal <b>17</b> is provided with an acceleration sensor S<b>17</b> for measuring the operated amount, and the operating system of the brake pedal <b>18</b> is provided with a brake sensor S<b>18</b> for detecting whether or not the brake pedal is being operated.

In the proximity of the rear wheels <b>3</b>, there is provided a photo interrupter or pickup type movement distance sensor S<b>3</b> so as to measure the movement amount of the vehicle <b>1</b> from the rotation amount of the rear wheels <b>3</b>. The movement distance sensor S<b>3</b> may be one that acquires the movement amount from the rotation amount of the transmission system inside the speed change mechanism T. The movement distance sensor S<b>3</b> may measure the rotation amount of the front wheels <b>2</b>. Furthermore, the movement distance sensor S<b>3</b> may be configured so as to detect the movement amount and steering amount of the vehicle <b>1</b> from image processing of the captured image of the front camera <b>27</b> or the rear camera <b>28</b>.

[Control Configuration]

An ECU <b>30</b> as the vehicle peripheral observation device of the present invention is disposed in the middle of the vehicle <b>1</b>. The ECU <b>30</b> comprises an interface consisting of a sensor input interface <b>31</b> and a communication interface <b>32</b>, and also comprises an image output module <b>33</b> and an audio output module <b>34</b>, as shown in <figref idref="DRAWINGS">FIG. 3</figref>.

The ECU <b>30</b> comprises a microprocessor, a digital signal processor (DSP), or another processing system for processing information obtained via the interface, the processing results are outputted from the image output module <b>33</b> to the monitor <b>21</b>, and audio information is outputted from the audio output module <b>34</b> to the speaker <b>22</b>.

The ECU <b>30</b> comprises a navigation controller <b>35</b> for implementing navigation mode control, a map data preservation part <b>36</b> for sending map information to the navigation controller <b>35</b>, a parking assistance controller <b>37</b> for implementing parking assistance mode control, an obstacle detector <b>38</b> (an example of the obstacle-detecting unit) for sensing an obstacle X in the proximity of the vehicle, and an image processor <b>39</b> for implementing vehicle peripheral observation processing.

The navigation controller <b>35</b> acquires the vehicle position indicated by latitude information and longitude information from a GPS unit <b>41</b> during travel, acquires map data corresponding to the vehicle position, and displays the position and data on the monitor <b>21</b>. The navigation information leading to the destination is displayed on the monitor <b>21</b>, and leading information is outputted in audio from the speaker <b>22</b>. The map data preservation part <b>36</b> performs a process of sending the map data corresponding to the vehicle position to the navigation controller <b>35</b>.

The parking assistance controller <b>37</b> displays an estimated route image Y (see <figref idref="DRAWINGS">FIG. 6(<i>a</i>)</figref>), which leads the vehicle <b>1</b> to a parking position, superimposed over the image displayed on the monitor <b>21</b> by the image processor <b>39</b>. The driver can easily lead the vehicle <b>1</b> into the target parking position by performing the steering operation on the basis of the estimated route image Y displayed on the monitor <b>21</b>.

The image processor <b>39</b> disposes a primary display area <b>21</b>A and an auxiliary display area <b>21</b>B (an example of the overhead image display area) in the layout shown in <figref idref="DRAWINGS">FIG. 5(<i>a</i>)</figref> on the monitor <b>21</b> in the vehicle peripheral observation process, and displays a camera icon <b>21</b>C showing the camera that captured the image of this area in a position in the top of the primary display area <b>21</b>A.

The image processor <b>39</b> comprises overhead-image-generating unit <b>39</b>A for generating an overhead image in the format of looking down on the vehicle <b>1</b> from above from the images captured by the plurality of cameras previously described; divided area management unit <b>39</b>B for dividing and managing the overhead image displayed on the monitor <b>21</b> into a plurality of divided areas D<b>1</b> to D<b>4</b>; specified information output unit <b>39</b>C for displaying an emphasized frame F corresponding to the divided area D containing the obstacle X from among a plurality of frames F<b>1</b> to F<b>4</b>, the frame F being information specifying which area of the plurality of divided areas D<b>1</b> to D<b>4</b> displays the obstacle X; enlarge mode setting unit <b>39</b>D for setting an image to be enlarged and an enlarged display area; and enlarged display unit <b>39</b>E for enlarging the image to be enlarged to the enlarged display area. As shown in <figref idref="DRAWINGS">FIG. 5(<i>a</i>)</figref>, the divided area D<b>1</b> and the divided area D<b>2</b> correspond to the left and right sides of the front part of the vehicle, while the divided area D<b>3</b> and the divided area D<b>4</b> correspond to the left and right sides of the rear part of the vehicle. Similarly, the frames F<b>1</b> to F<b>4</b> correspond to the divided areas D<b>1</b> to D<b>4</b> as shown in the drawings. The divided area D is used as a broader term for the divided areas D<b>1</b> to D<b>4</b>, and the frame F is used as a broader term for the frames F<b>1</b> to F<b>4</b>.

The ECU <b>30</b> is configured from an electronic circuit, and some or the entirety of the interface or output system may be included in this electronic circuit. The ECU <b>30</b> has either an electronic circuit constituting a processing system, or a storage part configured by a memory, register, or the like as a separate component; and information is inputted and outputted by a database, an address bus, a control bus, and other components.

The sensor input interface <b>31</b> receives input of driving operation information and information for detecting the movement state from the steering sensor S<b>16</b>, the shift position sensor S<b>19</b>, the acceleration sensor S<b>17</b>, the brake sensor S<b>18</b>, and the movement distance sensor S<b>3</b>.

The communication interface <b>32</b> conducts communication between the power steering unit PS, the speed change mechanism T, the braking devices BK, and the GPS unit <b>41</b> via communication lines. Furthermore, the communication interface <b>32</b> acquires information from the touch panel <b>21</b>T, the left side camera <b>25</b>, the right side camera <b>26</b>, the front camera <b>27</b>, and the rear camera <b>28</b>.

The image output module <b>33</b> outputs images to the monitor <b>21</b>, and the audio output module <b>34</b> outputs audio to the speaker <b>22</b>.

[Control Format]

A format of the vehicle peripheral observation display process is shown in the flowchart of <figref idref="DRAWINGS">FIG. 4</figref>, and the content displayed in the monitor <b>21</b> when this process is executed is shown in <figref idref="DRAWINGS">FIGS. 5 to 8</figref>.

In other words, when the shift lever <b>19</b> is placed in the forward position in a state in which the vehicle peripheral observation process has been selected, the image captured using the front camera <b>27</b> is displayed on the primary display area <b>21</b>A as shown in <figref idref="DRAWINGS">FIG. 5(<i>a</i>)</figref>. The camera icon <b>21</b>C, which shows that the image is the image captured using the front camera <b>27</b>, is displayed in a position in the top part of the primary display area <b>21</b>A, and an overhead image is displayed in the auxiliary display area <b>21</b>B. Conversely, when the shift lever <b>19</b> is placed in the reverse position, the image captured using the rear camera <b>28</b> is displayed in the primary display area <b>21</b>A as shown in <figref idref="DRAWINGS">FIG. 6(<i>a</i>)</figref>. The camera icon <b>21</b>C, which shows that the image is the image captured using the rear camera <b>28</b>, is displayed in a position in the top part of the primary display area <b>21</b>A, and an overhead image is displayed in the auxiliary display area <b>21</b>B (steps #<b>101</b> and #<b>102</b>).

A specific format of the process is that when the shift lever <b>19</b> has been placed in the forward position, the image captured using the front camera <b>27</b> is displayed in the primary display area <b>21</b>A. When the shift lever <b>19</b> has been placed in the rear position, the image captured using the rear camera <b>28</b> is acquired, a mirror reflection image interchanging the left and right of this captured image is generated, and the image is displayed on the monitor <b>21</b>.

The overhead image displayed in the auxiliary display area <b>21</b>B is generated as a single overhead image disposed on the road surface by the overhead-image-generating unit <b>39</b>A acquiring the images captured using the left side camera <b>25</b>, the right side camera <b>26</b>, the front camera <b>27</b>, and the rear camera <b>28</b>, converting the images to projected images in the format of looking down from above, and mapping the projected images in an imaginary plane in a position encompassing the vehicle image.

The overhead image displayed in this manner is divided and managed as four divided areas D<b>1</b> to D<b>4</b> so as to be sectioned by sectioning lines L shown as dashed lines. The sectioning lines L shown in the drawing are not displayed on the screen, but the process format may be set so that the sectioning lines L are displayed on the screen.

Particularly, the overhead image displayed in the auxiliary display area <b>21</b>B is generated from the images captured using four cameras, but the respective overhead images of the four divided areas D<b>1</b> to D<b>4</b> sectioned by the sectioning lines L are generated by combining the images captured using two cameras. In other words, overlapping areas W (see <figref idref="DRAWINGS">FIG. 1</figref>) in the images captured using two cameras are formed in outer positions at the front and rear corners of the vehicle <b>1</b>, and the images are combined so that the border regions in the overlapping areas W do not stand out. When the overhead image combined in this manner is set as the image to be enlarged, the overhead image is enlarged to the size of the enlarged display area.

When the shift lever <b>19</b> has been placed in the reverse position, it is possible to execute parking assistance control as well, and when parking assistance control is thus executed, an estimated route image Y is displayed superimposed over the primary display area <b>21</b>A and the auxiliary display area <b>21</b>B as shown in <figref idref="DRAWINGS">FIG. 6(<i>a</i>)</figref>.

The estimated route image Y represents the route the vehicle <b>1</b> will traverse when the vehicle <b>1</b> is reversed with the current steering amount, and the estimated route image Y is generated in an imaginary plane coinciding with the road surface with a sense of perspective corresponding to the captured image displayed on the monitor <b>21</b>. The estimated route image is formed in a frame by a pair of vehicle width lines Ya forming a width equal to the width of the vehicle <b>1</b>, and a plurality of distance lines Yb showing the positions the vehicle <b>1</b> will reach upon reversing in the set amount.

The obstacle detector <b>38</b> previously described is configured as software for extracting, as an obstacle X, a three-dimensional object that produces a parallax from the images of the overlapping areas W that overlap in the imaging areas of the left side camera <b>25</b>, the right side camera <b>26</b>, the front camera <b>27</b>, and the rear camera <b>28</b>. A summary of the process for extracting the three-dimensional object can be described as follows. In other words, the image of an overlapping area W in the images captured using two cameras is projected and converted to an imaginary plane having the same background, and the two images that have been projected and converted are superimposed in a positional relationship such that the same backgrounds of the images coincide. The image that does not coincide during this superimposing is the three-dimensional object, the three-dimensional object is estimated to be an obstacle X, and the position of the obstacle X is specified.

The overlapping area W of the imaging area of the two cameras could also be in a blind spot for the driver, and it is rational that only the obstacle X in this area be the object of detection. The vehicle may have four or more cameras in order to enlarge the area of detection for the obstacle X. Furthermore, the obstacle detector <b>38</b> may be configured as software for detecting the obstacle X by an optical-flow technique.

Particularly, the obstacle detector <b>38</b> may comprise an ultrasonic sensor or other obstacle sensors in a plurality of locations in the vehicle <b>1</b>, such as the front end and rear end. Configuring the obstacle detector <b>38</b> using obstacle sensors in order to detect obstacles X in this manner makes it possible to detect obstacles X in the entire periphery around the vehicle <b>1</b>.

When an obstacle X is detected by the obstacle detector <b>38</b>, and specifically when, among the divided areas D<b>1</b> to D<b>4</b>, the obstacle X is in the divided area D<b>2</b> as shown in <figref idref="DRAWINGS">FIG. 5</figref>, the divided area D<b>2</b> (the divided area D that includes the detected obstacle X) is indicated, and the display of the frame F<b>2</b> in the outer periphery of the divided area D<b>2</b> is emphasized (shown in hatching in the drawing). At the same time, the color tone of the divided area D<b>2</b> containing the obstacle X is varied, and a warning sound is outputted from the speaker <b>22</b> (steps #<b>103</b> and <b>104</b>). The color tone of D<b>2</b> need not be varied. In this case, it is acceptable to vary the color tone of only the intermediate image.

In this process, the display is emphasized in the divided area D<b>2</b> containing the obstacle X from among the divided areas D<b>1</b> to D<b>4</b> managed by the divided area management unit <b>39</b>B, by the specified information output unit <b>39</b>C increasing the brightness of the frame F<b>2</b>, increasing the chroma, or varying the frame to a different hue. In addition to this emphasized display, the display of the frame F<b>2</b> may be varied so as to blink. One possibility for the process of varying the color tone of the divided area D<b>2</b> containing the obstacle X to a different color tone is a process of increasing the brightness or reducing the brightness of one of the three primary colors, blue (B), of the pixels of the image in the divided area D<b>2</b>, for example. Furthermore, the warning sound outputted from the speaker <b>22</b> need not be an electronic sound, and may be a worded message.

Next, when there is sensed a touch operation of the driver's fingers or the like touching the auxiliary display area <b>21</b>B, the touch operation is a trigger for the image (the overhead image or image captured using the camera) of the divided area D<b>2</b> already indicated to be used as the image to be enlarged, and an enlarge process is performed for enlarging the image to be enlarged to the enlarged display area (steps #<b>105</b> and #<b>106</b>).

In this process, the enlarged display unit <b>39</b>E executes an enlarge process in accordance with the enlarge mode set by the enlarge mode setting unit <b>39</b>D. An image to be enlarged and an enlarged display area are set in this enlarge mode. Set as the image to be enlarged is either the overhead image displayed in the divided area D<b>2</b> containing the obstacle X, or the image captured by one of the cameras that has imaged the obstacle X (the image as viewed from the camera). One of the following modes is set: a normal enlarge mode for setting the auxiliary display area <b>21</b>B as the enlarged display area, a separate screen enlarge mode for setting the primary display area <b>21</b>A as the enlarged display area, or an entire screen enlarge mode for setting the entire screen of the monitor <b>21</b> as the enlarged display area.

The procedure for setting the enlarge mode may consist of a mode setting screen being displayed on the monitor <b>21</b> and the mode being set in advance manually.

[Control Format: Normal Enlarge Mode]

When the enlarge process is performed in normal enlarge mode, the enlarged display unit <b>39</b>E sets the image to be enlarged (the overhead image of the divided area D in the drawing), sets the auxiliary display area <b>21</b>B shown in <figref idref="DRAWINGS">FIGS. 5 and 6</figref> as the enlarged display area, and sets an enlarged display trajectory (not shown) between the divided area D<b>2</b>·D<b>3</b> (meaning D<b>2</b> in <figref idref="DRAWINGS">FIG. 5</figref> and divided area D<b>3</b> in <figref idref="DRAWINGS">FIG. 6</figref>) and the enlarged display area (the auxiliary display area <b>21</b>B).

Next, the image of the divided area D<b>2</b>·D<b>3</b> (an overhead image) is enlarged as shown in <figref idref="DRAWINGS">FIGS. 5(<i>b</i>) and 6(<i>b</i>)</figref>, thereby generating an intermediate image M slightly larger in size than the divided area D<b>2</b>·D<b>3</b>, and the intermediate image M is displayed superimposed over the overhead image of the auxiliary display area <b>21</b>B in a position covering the entire divided area D<b>2</b>·D<b>3</b> (the divided area D set by the presence of the obstacle X) which acts as a display origin point. The image of the divided area D<b>2</b>·D<b>3</b> (an overhead image) is then immediately further enlarged as shown in <figref idref="DRAWINGS">FIGS. 5(<i>c</i>) and 6(<i>c</i>)</figref>, whereby an intermediate image M of a larger size is formed and displayed in a position conforming to the enlarged display trajectory. In this case, a process is performed for deleting the previously displayed intermediate image M and displaying the next intermediate image M, and in normal enlarge mode, the next intermediate image M is displayed superimposed over the overhead image of the auxiliary display area <b>21</b>B in a positional relationship of covering the entire intermediate image M that was previously displayed.

As the size of the intermediate image M is gradually moved along the enlarged display trajectory, intermediate images M of progressively larger sizes are displayed. The enlarged display is executed for a duration of about 0.5 seconds or 1 second, for example, and ultimately an enlarged image consisting of an enlargement of the image (overhead image) of the divided area D<b>2</b>·D<b>3</b> fills up the area of the auxiliary display area <b>21</b>B as shown in <figref idref="DRAWINGS">FIGS. 5(<i>d</i>) and 6(<i>d</i>)</figref>. An area icon <b>21</b>D is then displayed in the corner of the enlarged image, thereby completing the enlarge process. Particularly, the area icon <b>21</b>D is set to a shape showing the area imaged by the camera and a size that does not overlap the road surface (a size that overlaps the vehicle image), and the original position of the image to be enlarged can be recognized from the area icon <b>21</b>D.

[Control Format: Separate Screen Enlarge Mode]

When the enlarge process is performed in separate screen enlarge mode, the enlarged display unit <b>39</b>E sets the image to be enlarged (the overhead image of the divided area D<b>3</b> in <figref idref="DRAWINGS">FIG. 7</figref>), sets the primary display area <b>21</b>A as the enlarged display area, and sets an enlarged display trajectory (not shown) between the divided area D<b>3</b> and the enlarged display area (the primary display area <b>21</b>A). In this control format as well, the color tone of the divided area D<b>3</b> containing the obstacle X is varied and the display of the frame F<b>3</b> in the outer periphery is emphasized (shown in hatching in the drawing).

Next, the image (an overhead image) of the divided area D<b>3</b> is enlarged as shown in <figref idref="DRAWINGS">FIG. 7(<i>a</i>)</figref>, thereby generating an intermediate image M slightly larger in size than the divided area D<b>3</b>, and the intermediate image M is displayed superimposed in a position covering much of the display area D<b>3</b> (the divided area D set by the presence of the obstacle X) which acts as a display origin point. The image of the divided area D<b>3</b> (an overhead image) is then immediately further enlarged as shown in <figref idref="DRAWINGS">FIG. 7(<i>b</i>)</figref>, whereby an intermediate image M of a larger size is formed and displayed in a position conforming to the enlarged display trajectory.

As the size of the intermediate image M is gradually moved along the enlarged display trajectory, intermediate images M of progressively larger sizes are displayed. The enlarged display is executed for a duration of about 0.5 seconds or 1 second, for example, and ultimately an enlarged image consisting of an enlargement of the image (overhead image) of the divided area D<b>3</b> fills up the area of the primary display area <b>21</b>A as shown in <figref idref="DRAWINGS">FIG. 7(<i>c</i>)</figref>. An area icon <b>21</b>D is then displayed in the corner of the enlarged image, thereby completing the enlarge process. Particularly, the area icon <b>21</b>D has a shape showing the area imaged by the camera, and the original position of the image to be enlarged can be recognized from the area icon <b>21</b>D.

In separate screen enlarge mode, because the enlarged display area (the primary display area <b>21</b>A) and the divided area D<b>3</b> have different aspect ratios, there is a deformation of the enlarged image in which the overhead image of the divided area D<b>3</b> fills up the area of the primary display area <b>21</b>A. The display may also have the top and bottom ends of the enlarged image trimmed off while the area displaying the obstacle X remains.

[Control Format: Entire Screen Enlarge Mode]

When the enlarge process is performed in entire screen enlarge mode, the enlarged display unit <b>39</b>E sets the image to be enlarged (the overhead image of the divided area D<b>3</b> in <figref idref="DRAWINGS">FIG. 8</figref>), sets the display area of the monitor <b>21</b> as the enlarged display area, and sets an enlarged display trajectory (not shown) between the divided area D<b>3</b> and the enlarged display area (the display area of the monitor <b>21</b>). In this control format as well, the color tone of the divided area D<b>3</b> containing the obstacle X is varied and the display of the frame F<b>3</b> in the outer periphery is emphasized (shown in hatching in the drawing).

Next, the image (an overhead image) of the divided area D<b>3</b> is enlarged as shown in <figref idref="DRAWINGS">FIG. 8(<i>a</i>)</figref>, thereby generating an intermediate image M slightly larger in size than the divided area D<b>3</b>, and the intermediate image M is displayed superimposed in a position covering much of the display area D<b>3</b> (the divided area D set by the presence of the obstacle X) which acts as a display origin point. The image of the divided area D<b>3</b> (an overhead image) is then immediately further enlarged as shown in <figref idref="DRAWINGS">FIG. 8(<i>b</i>)</figref>, whereby an intermediate image M of a larger size is formed and displayed in a position conforming to the enlarged display trajectory.

As the size of the intermediate image M is gradually moved along the enlarged display trajectory, intermediate images M of progressively larger sizes are displayed. The enlarged display is executed for a duration of about 0.5 seconds or 1 second, for example, and ultimately an enlarged image consisting of an enlargement of the image (overhead image) of the divided area D<b>3</b> fills up the display area of the monitor <b>21</b> as shown in <figref idref="DRAWINGS">FIG. 8(<i>c</i>)</figref>. An area icon <b>21</b>D is then displayed in the corner of the enlarged image, thereby completing the enlarge process. Particularly, the area icon <b>21</b>D has a shape showing the area imaged by the camera, and the original position of the image to be enlarged can be recognized from the area icon <b>21</b>D.

In entire screen enlarge mode, because the enlarged display area (the display area of the monitor <b>21</b>) and the divided area D<b>3</b> have different aspect ratios, there is a deformation of the enlarged image in which the overhead image of the divided area D<b>3</b> fills up the display area of the monitor <b>21</b>. The display may also have the top and bottom ends of the enlarged image trimmed off while the area displaying the obstacle X remains.

Regardless of which enlarge mode the enlarge process is performed in, the image is enlarged in a format of moving the display position of the intermediate image M along the enlarged display trajectory using the position of the divided area D as an origin point (the position of the corner of the intermediate image M does not move in normal enlarge mode of <figref idref="DRAWINGS">FIGS. 5 and 6</figref>), and enlargement in a zoomed-in format is achieved while the position of the original image being enlarged (the position of the divided area D) is clarified. It is then possible to recognize which area the enlarged image belongs to, from the format of the area icon <b>21</b>D displayed in the corner of the enlarged image or from the display position of the area icon <b>21</b>D.

A superimposed display is implemented by performing the process for forming intermediate images M in a movie layer when the enlarge process is performed in either enlarge mode, and while this superimposed display is in effect, changes in the positional relationship between the vehicle <b>1</b> and the obstacle X can be displayed in real time using either overhead images or captured images. As will be described next, the process is finished by deleting the enlarged image and deleting the video layer image even when the original display is to be restored.

Next, when there is sensed a touch operation of the driver's fingers or the like touching any area of the monitor <b>21</b>, or when a state is reached in which no obstacle X is detected by the obstacle detector <b>38</b>, the enlarged image is deleted and the sequence transitions to the process of step #<b>101</b>, thereby resuming control in which a captured image is displayed in the primary display area <b>21</b>A of the monitor <b>21</b> and an overhead image is displayed in the auxiliary display area <b>21</b>B (steps #<b>107</b> to #<b>109</b>).

The vehicle peripheral observation display process is then continued until a selection is made for ceasing control (step #<b>110</b>).

Thus, in the present invention, when the vehicle <b>1</b> is being led into a parking position, or when the vehicle <b>1</b> is traversing a road surface in which a fence, a parked vehicle, or another obstacle X is present, a image captured by a camera is displayed in the primary display area <b>21</b>A of the monitor <b>21</b>, and an overhead image is displayed in the auxiliary display area <b>21</b>B of the monitor <b>21</b>. The positional relationship between the vehicle <b>1</b> and an obstacle X in proximity to the vehicle can thereby be confirmed even when the obstacle X is in the driver's blind spot.

When the vehicle is in proximity to an obstacle X, an emphasized display is performed, increasing the brightness of the frame F among the frames F<b>1</b> to F<b>4</b> that corresponds to the outer periphery of the divided area D containing the obstacle X among the divided areas D<b>1</b> to D<b>4</b> of the overhead image. At the same time as this display, a display is performed for varying the color tone of the divided area D containing the obstacle X and a warning sound is outputted from the speaker <b>22</b>, whereby the area containing the obstacle X can be perceived.

By touching the auxiliary display area <b>21</b>B with the finger or the like in this state of being able to perceive the position of the obstacle X, the divided area D containing the obstacle X among the divided areas D<b>1</b> to D<b>4</b> is enlarged to the enlarged display area, and the positional relationship between the obstacle X and the vehicle <b>1</b> can be confirmed in the enlarged image.

Particularly, when the image of any of the plurality of divided areas D<b>1</b> to D<b>4</b> is enlarged to the enlarged display area, display is performed in a format such that, using the divided area D containing the obstacle X as an origin point, a plurality of intermediate images M of different sizes are moved sequentially along the enlarged display trajectory so as to transition from smaller sized images to larger sized images. By enlarging in this manner, an enlargement is implemented in a format of zooming in while clarifying the position of the original image being enlarged (the position of the divided area D containing the obstacle X).

As a result of the enlarged display of either the overhead image of the area containing the obstacle X or the image captured using the camera that imaged the obstacle X, the positional relationship between the obstacle X and the vehicle <b>1</b> can be perceived from the enlarged image, and the vehicle <b>1</b> can be moved in a manner that avoids contact with the obstacle X.

[Other Embodiments]

Other than the embodiments described above, the present invention may also be configured in the following manner.

(a) When the enlarged display unit <b>39</b>E forms an intermediate image M, a borderline of high brightness is displayed in order to make the border more distinct between the intermediate image M and the image already displayed. When the borderline is displayed in this manner, the hue of the background image is acquired, and the process format may be set so that a different color such as the complementary color to this hue is set. The outline of the intermediate image M is formed by displaying a borderline in the intermediate image M in this manner, and enlargement is implemented in a zoomed-in format while the position of the originally enlarged image (the position of the divided area D containing the obstacle X) is further clarified.

(b) When the overhead image is divided into and managed as a plurality of divided areas D, it may be divided into and managed as a number of divisions other than four, such as two or six divisions. The configuration may allow the user to set the number of divisions as desired. When the division is a large number, the proximity of the obstacle X to local areas in the periphery of the vehicle <b>1</b> can be confirmed, and the region of the obstacle can be enlarged and displayed.

(c) In a state in which obstacles X are being detected, an enlarged display is performed of the image of the divided area D (an overhead image or an image captured by a camera) where an obstacle X is detected even if any region of the display area of the auxiliary display area <b>21</b>B is touched through the touch panel <b>21</b>T, and in a state in which obstacles X are not being detected, the control format may be set so that an enlarged display is performed of the image of the divided area D (an overhead image or an image captured by a camera) selected through the touch panel <b>21</b>T. Designated enlarge buttons corresponding to the number of divided areas D may be displayed on the monitor <b>21</b>, and a designated button for ending the enlarged display may be displayed on the monitor <b>21</b>.

(d) When an image to be enlarged is set in the enlarge process by the enlarge mode setting unit <b>39</b>D and when a frontal divided area D is indicated by manual operation as described in (c), or when the obstacle X is detected in the front position, an overhead image is set as the image to be enlarged. Conversely, when a rear divided area D is indicated by manual operation as described in (c), or when the obstacle X is detected in the rear position, an image captured by a camera is set as the image to be enlarged.

In other words, when the area is at the front end of the vehicle <b>1</b>, the driver will sometimes lean out of the vehicle to visually confirm the area, and it is effective for this visually confirmable area to display an image that is seen from a different direction than an area that could be seen by the driver, which is made possible by setting an overhead image as the image to be enlarged. Areas at the rear end of the vehicle <b>1</b> the obstacle often cannot be seen by the driver, and it is effective for the image to be enlarged to be a captured image that gives the sense of looking directly at the image captured by a camera.

(e) In the previous other embodiment (d), when an image captured using a camera is used as the image to be enlarged and when the operated direction of the steering wheel <b>16</b> is determined to be the forward direction of the vehicle <b>1</b> approaching the imaging area (the obstacle X), a photographed image of a side camera (the left side camera <b>25</b> or the right side camera <b>26</b>) is set as the display target image. When the forward direction of the vehicle <b>1</b> is determined to be a direction going away from the imaging area (the obstacle X), an image captured using a camera in an end of the vehicle <b>1</b> (the front camera <b>27</b> or the rear camera <b>28</b>) is set as the display target.

By selecting the camera that acquires the captured image on the basis of the steering direction in this manner, changes in the relative positional relationship between the obstacle X or the like and the vehicle <b>1</b> can easily be perceived from the image captured using a side camera (the left side camera <b>25</b> or the right side camera <b>26</b>), even when the vehicle <b>1</b> has moved somewhat in a direction approaching the obstacle X or the like. When the vehicle <b>1</b> moves in a direction going away from the obstacle X or the like, a wide area in the forward direction of the vehicle <b>1</b> can be displayed on the monitor <b>21</b> and the forward direction can be confirmed from an image captured by a camera in an end of the vehicle <b>1</b>.

(f) When it is detected that “the overhead image display area (the auxiliary display area) has been touched,” it may either be a detection of a touching state change from a state of no touching of the touch panel <b>21</b>T (the overhead image area) to a state of touching, or a detection of a state change from a state of a finger or the like touching the touch panel <b>21</b>T (the overhead image area) to a state of the finger or the like taken away.

INDUSTRIAL APPLICABILITY

The present invention can be utilized in an entire vehicle having an image processing system for generating an overhead image from the images captured using a plurality of cameras provided to the vehicle and displaying the overhead image on a monitor.

KEY

<b>1</b> Vehicle

<b>21</b> Monitor

<b>21</b>B Overhead image display area (auxiliary display area)

<b>21</b>T Touch panel

<b>25</b> Camera (left side camera)

<b>26</b> Camera (right side camera)

<b>27</b> Camera (front camera)

<b>28</b> Camera (rear camera)

<b>39</b> Image processor

<b>39</b>A Overhead-image-generating unit

<b>39</b>E Enlarged display unit

D Divided area

M Intermediate image