Method and device for visualising the surroundings of a vehicle

p0001The invention relates to a method for the visualization of a vehicle environment, in particular for the detection and display of the traffic situation in front of a driver, comprising a night vision information system in which largely mobile telephones of different spectral ranges are fused With weighting factors.

p0002The invention also relates to a device for the visualization of a vehicle environment, in particular for the detection and display of the traffic situation in front of a driver, comprising a night vision infusion system in which video cameras of different spectral ranges can be accommodated Fusion device.

p0003Night vision information systems are known for supporting the viewing conditions for the driver of a vehicle at night or in unfavorable weather conditions, for example in the case of rain, fog or snow drifts. These systems take video mobiles from a traffic situation in front of the driver, in order to present them to the driver on a display. Night vision information systems usually operate with an infrared (IR) camera, which detects a thermal image of a section of the environment. The driver is thereby provided with an aid for detecting objects which are not illuminated and therefore can not be perceived directly by the driver, in particular pedestrians, cyclists or animals, in the event of unfavorable visibility, and to avoid accidents.

p0004From DE 102 07 039 A1 a method and a device for the visualization of a vehicle environment is known, in which, in addition to the IR camera for the infrared spectral range, a VIS camera is used for the visible (visual) spectral range in a night vision information system. In a fusion device, in each case a substantially localized and temporally equalized IR image and a VIS image are pixel-superimposed by software. Individual pixel pairs or entire pixel areas are processed by a weighted averaging of the pixel information. In the averaging process, the IR information and the VIS information can be weighted differently, in particular as a function of the ambient brightness, the color in the visual image, or a range of information which can be derived from the information theory in the respective spectral range. The weighting can take place over a whole area or in individual image areas. Due to the superimposition of the IR image with the VIS image, the driver can more easily map the traffic situation before him, which he or she is visually observing through the windshield, and the corresponding traffic situation represented by the night vision information system by means of a video image than with a night vision information system , Which only works with IR images.

p0005A disadvantage of the known method has the effect that the contrast in the fused images is set relatively uniformly over the image area. Although the adjustable weighting of the IR image and of the VIS image in the mean value formation of the fused image generally improves contrasting against non-weighted images, various details within an image area are treated the same. This can lead to the fact that important details, for example persons on the roadway, do not rise to the desired extent on the displayed video image in certain traffic situations.

p0006It is therefore an object of the present invention to improve the known method for the visualization of a vehicle environment with a night vision information system in such a way that the perceptibility of individual picture objects is increased.

p0007The object is achieved according to the invention in conjunction with the preamble of claim 1 in that firstly the video mobiles are weighted with a globally adjusted preliminary weighting factor, secondly with the aid of an object recognition system, an object analysis of the video mobiles is carried out and thirdly on the basis of the object analysis an object- Adjustment of the weighting factors.

p0008The object-selective adaptation of the weighting factors enables detected objects in the vehicle environment to be represented in a contrast-optimized manner for better perception by the driver, since the spectral components which make the greatest contribution to the perception of the objects are weighted more heavily. For this purpose, a preliminary (global) weighting of the spectral components for the image fusing, for example half the intensity of the VIS spectral range and the IR spectral range, is set for the first time. Subfusion is, in principle, the formation of new pixels from a weighted averaging of the intensity of associated pixel pairs, as well as the superimposition with a weighted addition of the intensities. Subsequently, the local adaptation of the weighting factors and their final determination takes place on the basis of the objects detected or recognized by the object recognition system.

p0009Appropriate object recognition systems are in principle available. Their functioning is known, for example, from DE 199 55 919 C1 and the cross references contained therein. They are usually based on an automatic object analysis of digital images or on a pattern recognition and provide a statement about the presence of certain predefined objects. The object recognition system according to the invention performs image analysis in each recorded spectral region. On the basis of the recognition and differentiation of detected objects, for example people, animals, obstacles, the spectral weighting factors can then be adapted in the respective image areas in the visible and in the infrared according to their radiation strength. The weighting factors can be adjusted in such a way that the detected objects are mainly represented in the spectral range in which they are most noticeable for the driver on the fused video image. In particular, persons and game who have no visual parts at a greater distance but are very well detectable in the infrared can be better positioned in the own vehicle environment by this adaptation since a contrast reduction due to the fusion with the dark background is dispensed with. Accordingly, objects which leave a significantly better signature in the visual, such as, for example, road limiting posts due to the provided reflectors, are provided with a stronger weighting in the visual.

p0010According to a preferred embodiment of the invention, an object evaluation of the detected objects is carried out according to their relevance with the aid of the object recognition system and the local weighting factors are adapted to the relevance of the respective object assigned on the basis of the object evaluation so that predetermined detected objects are mainly shown in the spectral range , In which they are most perceptible to the driver on the merged video image.

p0011Through the object evaluation, a targeted amplification or attenuation of local weighting factors can be carried out. In particular, it is thereby possible to make the driver explicitly aware of particularly relevant, in particular of dangerous and / or endangered, objects with an overweighting of a specific spectral component. This allows an increased safety (driver) and other passengers in the dark and poor visibility to be achieved. On the one hand, the overweighting can be achieved by a local elevation in the interesting image area or by a reduction of a spectral component in the object environment.

p0012Particularly suitable for the object evaluation are object recognition systems which operate with object classification. A particularly fast and reliable classification of the relevance of the detected objects can be made on the basis of different object classes, for example people, animals, vehicles, obstacles, etc.

p0013According to a further preferred embodiment of the invention, image areas in which no relevant objects are detected or whose relevance can not be unambiguously assigned are weighted with the globally adjusted weighting factor.

p0014By setting the global weighting factor in image areas without relevant objects or without a clear assignment, it is ensured that the information of these image areas is also displayed to the driver.

p0015According to a further preferred embodiment of the invention, a continuous transition is produced in the fused video image between adjacent relevant and non-relevant image regions, with the aid of a variable characteristic curve of the ratio of the weighting factors of the different spectral regions.

p0016Due to the continuous transition between particularly prominent, because relevant, image areas and their surroundings, the imperceptible soft course produces a harmonic and realistic image impression in the respective fused video image.

p0017According to a further preferred embodiment of the invention, the fusion of the video camera radiation components with wavelengths from the visible spectral range and radiation components with wavelengths from the infrared spectral range are taken into account.

p0018The detection of the visible spectral range (approx. 380nm - 780nm) means that all objects illuminated or illuminated for the human eye are recorded. In the infrared range (about 780nm - 1mm), all objects that emit a heat radiation are detected. In this way, an information content which comprehensively reproduces the current traffic situation is achieved in the fused images. A stronger weighting in the FIR spectral range (approx. 25 μm-1 mm) is particularly effective in people who are not detected by any illumination and thus do not reflect any visual radiation.

p0019According to a further preferred embodiment, color information is processed in the visual video devices.

p0020The color information can further improve the perceptibility of certain objects in the fused images. The effect of object-selective weighting is thereby supported. The driver can, for example, be made even more aware of red brake lights, traffic lights, or signal systems, which means that the current traffic situation is recorded even more quickly and the risk of an accident is further reduced.

p0021According to a further preferred embodiment of the invention, masking and / or masking effects of detected objects are at least partially suppressed with the aid of the adaptation of the local weighting factors.

p0022Due to the local weighting factors, possible masking and masking effects can be controlled. By selectively changing the weighting factors in the concealment areas, the masking can be considerably suppressed. This also improves the interpretability of the displayed images.

p0023The device for the visualization of a vehicle environment known from DE 102 07 039 A1 has the disadvantage that it captures the traffic situation in front of a driver in different spectral ranges, but not in terms of content. This can lead to the fact that relevant objects are relatively inconspicuous to the driver.

p0024It is therefore a further object of the present invention to improve the known apparatus for the visualization of a vehicle environment in such a way that relevant image objects can be displayed more significantly.

p0025The object is achieved according to the invention in conjunction with the preamble of claim 9 in that the fusing device is preceded by an object recognition system with which objects which are located in the vehicle environment can be detected via the camera system and that a control device is provided, Wirkverbi invention with the object recognition system the fusion device can be controlled with an object-selective local Spektralbereichsgewichtung.

p0026Due to the active connection of the night vision information system, which combines image information from at least two different spectral regions via the camera system with the object detection system, the driver can be made particularly relevant to relevant objects, for example persons, because these objects are influenced by the object-selective spectral range weighting in The merged video recorders. On the whole, the contrast in the fused video memories is increased, since in principle all detected objects can be represented in the spectral range, in which they leave the best signature, ie appear the most significant.

p0027According to a preferred embodiment of the invention, the camera system has at least one sensor which is sensitive in the visible spectral range and at least one sensor which is sensitive in the infrared spectral range.

p0028Because the camera system can detect objects in the visible and in the infrared, visible objects, as well as pure heat emitters, can be displayed in the fused video bodies. This achieves a very high perception quality and a simple allocation of the display of the night vision information system to the traffic situation ahead of the driver.

p0029According to a further preferred embodiment of the invention, a head-up display is provided, on which the fused video vehicle can be presented to the driver.

p0030By means of the head-up display: the fused video vehicle can in particular be projected onto a lower part of the windshield of a vehicle. The images thus appear in the immediate field of view of the driver without disturbing the driver's straightforward view, whereby a particularly high level of attention is achieved in the driver and a rapid reaction can be carried out with a perceived danger. In principle, however, it is of course also possible to display the video device on a conventional monitor, for example an LCD display.

p0031Further details of the invention will become apparent from the following detailed description and the appended drawing in which a preferred embodiment of the invention is illustrated by way of example.

p0032It shows:<dl id="dl0001" compact="compact"><dt>FIG. 1:</dt><dd>A block diagram of a device for visualizing a vehicle environment.</dd></dl>

p0033An apparatus for visualizing a vehicle environment consists essentially of a night vision information system 1 installed in a vehicle, which has an object recognition system 5.

p0034The structure of the night vision information system 1 shown as a block diagram in FIG. 1 corresponds in principle to the system known from DE 102 07 039 A1.

p0035A visual camera 3 for recording video mobiles in the visible spectral range as well as an infrared camera 4 for recording video mobiles from the infrared spectral range are arranged in a camera system 2. The cameras are advantageously designed as digital cameras with sensors sensitive in the corresponding spectral regions. In accordance with the invention, the objective recognition system 5 is connected downstream of the camera system 2, ie a fusion device 9, to the camera system 2. A control device 6 is assigned to the object recognition system 5, via which the fusion device 9 can be controlled. Interposed is a normalization device 7 in which imaging errors of the video mobile can be eliminated by software in a manner known in principle. Furthermore, an alignment device 8 is provided in which the VIS images and IR images, taken locally and in time, can be aligned with one another with the aid of a digital image processing so that the respectively associated image pairs come to coincide pixel-wise. The alignment device 8 is followed by the fusion device 9 for the weighted fusion of the VIS images and IR images which have been brought into contact. The fused images can be displayed on a head-up display 10.

p0036A method for the visualization of a vehicle environment is essentially based on a weighted fusion of largely location- and time-equivalent video vehicles from different spectral regions of a traffic situation in front of a driver.

p0037The method is described below with reference to the device shown in FIG. 1:<ul><li>A recording of a vehicle environment located in front of the driver is made over the camera system 2 with the camera 3 sensitive to visible radiation and with the camera 4 which is sensitive to infrared radiation. The resulting VIS digital imaging and the IR digital imaging of the vehicle environment are subjected to an automatic image analysis in the object recognition system. The detected, ie recognized, objects are evaluated according to their relevance in an object classification. For example, people are given the highest relevance level. The video mobiles are subsequently corrected in the normalization device 7 with respect to existing imaging errors, for example distortions, and are then aligned pixel-wise in the alignment device 8. The video mobiles which have been covered are then digitally processed in the fusion device 9.</li></ul>

p0038In this digital image processing, a fused pixel is generated from each pixel pair. The IR component and the VIS component are multiplied by weighting factors. First of all, a weighting is set up over the entire area (global), which depends, for example, on the brightness of the environment (daylight / twilight / darkness). The local weighting factors are then determined in the object recognition system 5 by means of an object analysis. The detected objects thereby receive a stronger weighting in the spectral range in which they radiate more strongly. Furthermore, particularly relevant objects receive a stronger weighting than less relevant objects. The fusion device 9 is then controlled via the control device 6 in such a way that the local image areas which contain the detected objects are weighted according to the local weighting factors, that is to say essentially represented in the spectral range in which these objects leave the best signature. For example, non-illuminated persons are mainly represented in the infrared, while reflective street restrictions are mainly displayed in the visible. Furthermore, fine tuning can be performed to generate soft transitions between relevant and irrelevant image areas and suppression of masking effects. As a result, a contrast-rich fused video image is generated on which relevant objects appear particularly prominent.

p0039Preferably, the visual images are captured with a color VIS camera to generate a fused color video image. As a result, the images appear more realistic and the various detected objects are even more clearly recognizable for the driver and are more easily assigned to the vehicle surroundings lying ahead of him.

p0040The fused images are continuously projected onto the head-up display 10 in accordance with the recording frequency of the camera system 2 and the processing speed in the downstream devices 5 to 9 so that the objects traveling into the vehicle surroundings are immediately displayed to the driver and viewed by the driver with the aid of the camera High reliability even in darkness and poor visibility.

<u>List of references</u>

p0041<dl id="dl0002" compact="compact"><dt>1</dt><dd>night vision information system</dd><dt>2</dt><dd>Camerasystem</dd><dt>3</dt><dd>VIS camera</dd><dt>4</dt><dd>IR camera</dd><dt>5</dt><dd>Object recognition system</dd><dt>6</dt><dd>control device</dd><dt>7</dt><dd>normalization device</dd><dt>8th</dt><dd>aligning device</dd><dt>9</dt><dd>fusion facility</dd><dt>10</dt><dd>Head-Up Display</dd></dl>