Method and device for visualizing a motor vehicle environment with environment-dependent fusion of an infrared image and a visual image

FIELD OF THE INVENTION The present invention relates to a method and a device for displaying the environment of a vehicle according to the preamble of the corresponding independent claim, and a method and device for displaying the environment of a vehicle according to the preamble of the corresponding independent claim. The calibration device to calibrate the device.

BACKGROUND From DE 695 06 174 T2 known night vision system for vehicles (Nachsichtsystem). The system includes an infrared camera that produces an infrared image, the infrared image represents a segment of the vehicle environment, the thermal radiation emitted by people and objects. The infrared image is projected to the lower part of the windshield through a head-up display (HeadupDispIay), and is therefore displayed to the driver.

The driver corresponds the people and objects in the infrared image to the traffic situation that is located in front of the driver and can be obtained visually through the windshield. This is especially difficult in the dark environment where known systems should be targeted, and often even impossible, because infrared images show people and objects that the driver cannot visually recognize. Therefore, accurate positioning of people and objects that are only identifiable in infrared images is usually impossible and distracts the driver in a dangerous way.

SUMMARY OF THE INVENTION The task of the present invention is in particular to provide an improved method and device for displaying the environment of a vehicle, especially an improved night vision system.

The tasks of the present invention are accomplished according to the method or according to the device through the corresponding independent claims. Advantageous embodiments of the present invention are given in the corresponding dependent claims. The first main aspect of the present invention is to represent or display the vehicle environment as much as possible The visible image of the segment of the phase circle, hereinafter referred to as the visible image, and the infrared image, hereinafter referred to as the red

200480003555.6 The combination or superposition of the outer image. By superimposing or combining images of different spectral ranges in the same scene, the aforementioned shortcomings of the prior art are eliminated as much as possible.

The second main aspect is that when combining the visible images and the infrared images, the mutual proportions of the two spectral components (the visible component and the infrared component) are weighted according to the specific driving situation. In the first extreme case of weighting, the image produced by combining includes only visible information as much as possible. Under normal circumstances, the visible and infrared information are superimposed, and in the second extreme case of weighting, the image is produced by combining The image only includes infrared information as much as possible.

If, for example, the vehicle in front is identified by a distance measuring system on a highway, such as an ACC system (automatic cruise control) supported by a known radar, it is stipulated in an embodiment of the present invention that when combining these two spectral components The vehicle is highlighted relative to the driving environment so that it is clearly displayed relative to the environment in the combined image. This can be achieved, for example, by changing the weighting of the two spectral components of the relevant area relative to the driving environment; in this example, it is an image segment that combines the image and represents the vehicle driving immediately in front. For example, in the daytime, visual information can be weighted higher than infrared information, and therefore produce a more significant shape in the combined image, while the weighting is the opposite in the dark.

If the vehicle is traveling in a well-lit area, in another embodiment of the present invention, it is alternatively or additionally provided that the visual component or visual information is more weighted relative to the infrared component or relative to the infrared information. On darkened roads and/or highways, alternatively or in addition, weighting is performed inversely with respect to the foregoing, so that the infrared information or infrared components are more clearly displayed in the combined image relative to the visible components.

Alternatively or in addition, in another embodiment of the present invention, it is provided that, on darkened roads and/or highways, the higher weights are used in the combination to highlight them relative to the visual information of distant objects. Infrared information is different from objects closer to the vehicle. The combined image presented to the driver on the display may include higher component visual information in the near area and higher component infrared information in the far area. In rainy weather, When the road is foggy or wet, the practicality of infrared channel or infrared information is limited relative to the standard conditions. In order to deteriorate the combined image caused by the limited available infrared information under such weather conditions Acting on the contrary, alternatively or in addition, it is provided that the visible component is increased relative to the infrared component, that is, in the direction of the visible information correspondingly

200480003555. 6 The first change is the weighting of the two spectral components in the combined image.

In order to identify the aforementioned weather conditions, a two-drop sensor (Regensensor) or a fog sensor (Nebelsensor) can be specially arranged on the vehicle. Similarly, it is possible to analyze which vehicle lights are turned on, especially automobile fog lights. However, this information can also be provided by a dynamic navigation system installed in the vehicle, and the dynamic traffic or weather conditions at the point where the vehicle is currently located, or especially predicted for the planned driving route, are transmitted to the system. This transmission can be achieved via a mobile radio network (Mobilfimknetz) or via vehicle-to-vehicle communication.

Alternatively or in addition, it is provided in an embodiment of the present invention that the driver can manually adjust the mutual weighting of these two spectral components in the combined image, and thus can optimally match their needs and specific conditions. In one embodiment of the present invention, continuous conversion from visible information to infrared information display and vice versa are specified. This allows the driver to use the relationship between visual and infrared information and the combined image of the driving environment through the movement between the spectral components, which is advantageous for faster and more reliable acquisition of traffic conditions. .

In another embodiment, alternatively or in addition, a plurality of combined images are provided for the driver to select a combined image suitable for display to him, wherein the combined image provided for selection is based on the relationship between the visible information and the infrared information component Is different. In particular, the visible component can be weighted differently with respect to the infrared component when combining.

In a specific embodiment of the night vision system according to the present invention, the visual camera or the first sensor and its optical system include a first optical axis, and the infrared camera or the second optical sensor and its optical system include a second optical axis , They are offset from each other in space. Therefore, the camera or sensor captures at least partially different segments of the vehicle environment, namely the first and second segments. Moreover, they are related to distance. It can be understood that, according to the present invention, more than two infrared cameras or infrared sensors can also be provided, the sensitivity of which covers different wavelengths, and the digital images of the vehicle environment provided by them can be superimposed or combined. In order to obtain as distortion-free as possible Combining images to display to the driver. According to one embodiment of the present invention, it is stipulated that the superimposing or combining device shall superimpose or combine the provided first segment and the provided first segment in a pixel manner and/or a district manner in whole or in part. Two pieces

200480003555.6 Paragraph. In order to provide the combined image, the first segment and/or the second segment and/or the combined image and its direct or processed digital data are matched by at least one matching parameter.

Preferably, when calibrating the camera or sensor system and the night vision system according to the present invention, this or these distances are determined for at least two distance ranges (Entfernungsbereich) or separation ranges (Abstandsbereich) between the camera or sensor and the calibration device. Related parameters. One of the main purposes is to adjust this or these matching parameters so that the combined image of objects within the relevant distance range generated when superimposing or combining images is as undistorted as possible, especially without ghosting (Geisterbilder) or double (Doppelbilder). According to In the present invention, this or these matching parameters particularly relate to at least one recording parameter (Registrierungs parameter) or transformation parameter (Transformations parameter). A similar parameter is known, for example, from the recording of a digital image processing system and the superposition of two images performed by it. This or these matching parameters related to driving conditions or distance are preferably stored in the data storage of the vehicle during calibration.

In a preferred embodiment of the present invention, the first distance range corresponds to a typical driving situation in the city, especially a distance range of about 15 to 75 m.

Alternatively or in addition, the second distance range corresponds to typical driving conditions of road driving (Landstrassenfahrt), in particular a distance range of approximately 30 to 150 m. Alternatively or additionally, the third distance range corresponds to typical driving conditions of a highway , Especially the distance range of about 50 to 250m.

It should be understood that, as an alternative or supplement to the above-mentioned matching parameters related to distance or distance range, other matching parameters related to driving conditions, especially having the aforementioned purpose, can also be determined. Here, it may for example relate to Appropriate matching parameters for driving when the sun is low, in fog, when it is dark (einsetzende Dunkelheit), or in the dark. Another main aspect of the present invention is to automatically determine the current driving condition of the vehicle and provide corresponding Driving conditions or environmental conditions, especially by calibrating the four-part parameters stored in the vehicle by Röder, in order to be queried according to the invention and set

200480003555.6 (Ausricht-vorrichtung) or used by stacking or combined equipment. The superposition or combination device superimposes or combines the visible image and the infrared image in the form of digital data in a pixel and/or area manner, where one or more distance-related matching parameters affect the infrared image and/or visible The image and/or the combined image are such that it is preferable to provide the driver with a combined image that is as distortion-free and/or ghost-free as possible of the environment.

As an alternative to or in addition to the foregoing automatic determination, in one embodiment of the present invention, it is provided that the driver selects matching parameters suitable for display to him, especially obtained through calibration and stored in the vehicle, so as to be determined by the driver according to the present invention. The alignment device is either used by stacking or combined equipment. Therefore, it is possible, for example, in the form of manual selection possibilities or through voice input, to provide the driver with information such as typical urban driving, typical highway driving, typical highway driving, and/or other driving conditions when possible. Matching parameters for selection. As a result, the driver can generate a combined image that is as distortion-free or ghost-free as possible without the need for a car navigation system in the vehicle. Moreover, this provides the driver with the possibility of ignoring automatic selection when possible. Similarly, the driver can be provided with the possibility of choosing to display one or more matching parameters of the surrounding environment of his vehicle in the combined image without distortion, such as the distance within 20m of his vehicle. For example, when the driver approaches his garage in the dark and determines whether there are bushes behind him by combining the infrared information in the image, the driver can make this selection.

In a preferred embodiment of the present invention, the current position of the vehicle is determined by a vehicle navigation system, especially a satellite navigation system. With the help of this position, the navigation system in the vehicle automatically determines the corresponding road category (Strassenkategorie) or driving situation, especially the city road (Stadtstrasse), highway (Landstrasse) or highway. These vehicle navigation systems currently exist in a large number of vehicles for the purpose of route guidance (Routenfuehrung), and can be used for the aforementioned automatic, driving situation and environment-related navigation in conjunction with images of the vehicle environment without great expense. Optimization. Alternatively or in addition, in another embodiment of the invention, it is provided that the driving situation is determined by means of at least one driving dynamics variable, in particular the vehicle speed and/or the short-range light (Abblendlicht) or the high-beam (Fernlicht) Operation and/or vehicle auxiliary acceleration and/or

200480003555.6 No. Brightness and/or Ji, to determine. In current vehicles, this information may be provided in whole or in part by the on-board network (Bordnetz), and greater costs are not necessary.

In a preferred embodiment of the present invention, the color visible image and the infrared image are combined and combined or superimposed. Different from black and white visual images, a combined image containing visual image color information, such as red brake lights, red traffic lights, color traffic signs, etc., of the preceding vehicle is generated. The color information in the combined image makes it easier for the driver of the correspondingly configured vehicle to quickly understand and obtain the traffic conditions represented in the combined image. Moreover, color information reduces the risk of ignoring color warnings (red traffic lights, etc.).

The summary eliminates interference effects, such as infrared or visible image distortion, especially due to image defects of the respective optical systems, etc., environment or scene images of different spectrum ranges. This is preferably based on the software through the known measures of editing digital images to realise. The images whose interference effects are eliminated as much as possible or their digital image data are preferably aligned or overlaid with each other through digital image processing, so that the infrared and visible images or their digital data provide image pairs with the same time and position as possible. According to the invention, this is achieved by using at least one distance-related and/or driving condition-related matching parameter that results in a distortion-free combined image. The software and hardware used for digital image processing preferably allow mutual offset, rotation and scaling of images. Through this editing, it is possible to economically-although the image pairs with the same time and position are processed in real time as much as possible-to minimize the subsequent hardware consumption for superimposing or combining images of different spectral ranges.

According to a preferred embodiment of the present invention, at the same image repetition rate (Bildwiederholrate), preferably one or two cameras or sensors for the spectral range generate infrared images and visible images. As a result, simultaneous image pairs can be generated from the infrared image and the visible image in a particularly simple manner, which significantly reduces the software and software used to superimpose or combine the two images according to the present invention-as real-time as possible. Hardware consumption; the cost of semiconductor memory for intermediate storage of images is reduced.

In a preferred embodiment of the present invention, the infrared image represents infrared radiation or thermal radiation emitted by people and objects, and its wavelength range is about 8 to preferably, using an IR camera that is sensitive to a range of about 8 to 10 recesses. Camera or IR sensor·Therefore, in an advantageous way, there is no need for an infrared radiator or such a lamp for the vehicle environment

200480003555.6 No. (Beleuchtung) (Typical wavelength range is about 800m to 2.5μm). There is no alternating dazzling of relatively moving vehicles each equipped with infrared lamps in a known manner. Likewise, the infrared image according to the present invention is not limited to the range of the infrared lamp.

BRIEF DESCRIPTION OF THE DRAWINGS The invention will be described in detail below with the aid of embodiments.

FIG. 1 shows a block diagram of a device according to the present invention for displaying a segment of a vehicle environment, or a night vision system with the aid of which a display method according to the present invention is introduced.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS The device or night vision system 100 according to the present invention, represented in the form of a block diagram in FIG. 1, comprises an electronic, here so-called visual camera 101, which captures a visible spectral range, such as a CCD sensor, for about 8 to 10 pm An electronic infrared camera 102 that is sensitive to the infrared spectrum range, such as an IR sensor, a first normalization device (Normalisierungsvorrichtung) 103, a second normalization device 104, an alignment device 105, and a superimposing or combining device 106. The visual camera 101 provides a visual image in color.

The optical axes of the cameras 101 and 102 are preferably aligned parallel to each other, thereby reducing parallax (Parallaxenfehler), and are preferably arranged adjacent to each other, thereby reducing dislocation defects (Versetzimgsfehler). The image planes of the two cameras or sensors are preferably aligned parallel to each other and perpendicular to the optical axis, and are arranged adjacent to each other. The photosensitive sensor surfaces of the two cameras or sensors are preferably neither twisted nor inclined to each other, but are arranged as parallel to each other as possible. Furthermore, the two cameras or sensors preferably have the same aperture angle (Oeffnungswinkel). Through these measures, it can be achieved that cameras or sensors provide images of different spectral ranges, which represent as much the same segment of the environment as possible, and do not twist each other and relative to the actual situation. As a result, it is possible to significantly reduce the cost of processing the image to provide a combined image from the two images and thereby significantly reduce the hardware and software cost.

The optical surface of the camera is preferably hydrophob coated, and a high-pressure nozzle (Hochdruckduese) or the like is preferably provided for its cleaning, as used in automobiles

200480003555.6 The headlight cleaning is already very common. The two cameras are preferably arranged in a common housing (easy to install, co-aligned with the vehicle axis, without mutual deviation of the optical parameters of the cameras). On the housing of the camera, a clamp is preferably provided, which ensures vibration-free operation of the vehicle or the internal camera. For the work of the camera, set the connection for the working voltage. The operating voltage of the camera should be flexibly matched to the respective on-board network voltage, such as 12 volts and/or 42 volts. In order to protect the electronic components and sensors of the camera from damage, overload protection and polarity change protection (Verpolungsschutz) are preferably introduced in the operating voltage branch. The output of the video signal generated by the camera or sensor (spectral range separated or The already combined) should follow the standard (such as NTSC, PAL, SECAM or its own standard). Use existing semiconductor components as digital/analog converters. Cameras or sensors can be installed on the front, back, or side of the vehicle to display the vehicle environment.

The calibration of the night vision system or device 100 will be described in more detail below. For calibration, a calibration device (not shown) according to the present invention is used. It comprises a plurality of glow lamps arranged preferably in a staggered pattern. The highlight of the glow lamp is that it emits both thermal radiation and visually visible radiation. Preferably, a board or the like provided with a plurality of glow lamps is continuously arranged in front of the two cameras or sensors 101 and 102 at different distance ranges. These distance ranges relate to typical distance ranges related to the environment or driving conditions, which will be described in detail below.

The calibration device located in front of the cameras 101 and 102, preferably in a dark environment and not adjacent to the heat source, produces a (so-called) visual image in the visual camera 101, which represents a staggered arrangement Glow lamp, as the human eye sees it. Furthermore, the calibration device generates a thermal image (Waermebild) in the infrared camera 102, which also represents the setting of the glow lamp. Typically, the visible image and the so-called infrared image exhibit sharp distortion of the respective image, especially due to optical image defects or the like. In a known manner, the first normalization device 103 eliminates distortion or image defects in the visible image as much as possible. In a known manner, the second normalization device 104 eliminates the infrared image as much as possible. Distortion or image defects. Preferably, use the calibration parameter 107 for the visible image and the calibration parameter 108 for the infrared image, through known, software-based measures to achieve normalization or defect elimination (digital image processing) of the image exclamation data.

200480003555.6 First pass the known recording process in digital image processing, through the alignment device 105, use the recording parameters 109 to align the images or their digital data that are normalized to each other or that eliminate interference as much as possible. During the alignment process, one of the images preferably remains unchanged and serves as a reference for the other image. The size and position of the second image are changed in such a way that an image that is as object-identical as possible (objektgleich) relative to the first image is produced.

The normalized images are also aligned with each other, so that one or the same object appears in the same position as possible and the same size as possible in the combined image. If this preprocessing step is not performed, it will be due to different camera geometry and camera size. Offset, produces a shaded color map (Schattenbilder) or a ghost map (Zwillingsbilder). This means that an object appears in two locations on the combined image and has different sizes. Such an image is not so much to provide positioning for the observer, as it is to confuse it.

In order to optimize the combination of pixel accuracy or area accuracy according to the environment or driving conditions, a recording process for typical driving conditions in the city is first performed. To this end, the distance between the calibration device and the cameras 101 and 102 is varied, for example, within a range between approximately 15 to 75 m, and recording parameters suitable for this distance range are determined and stored. In a corresponding manner, a second recording process for a typical driving situation on a road is performed, that is, for example, a range of about 30 to 150 m. Finally, in a corresponding manner, a third recording process for typical highway driving conditions is performed, that is, for example, in the range of about 50 to 250 m.

Using the current location data provided by the vehicle navigation system (not shown), the system determines the road category or driving condition corresponding to the location of the vehicle by means of the digital card data, especially the city road, highway or highway. Especially in the dark, the records or matching parameters corresponding to the driving conditions and stored in the vehicle during the calibration process are superimposed or used by the alignment device 105 to align the images in accordance with the driving conditions. Therefore, in the combined images of the relevant driving conditions, especially shadow maps or reshaping maps are avoided as much as possible. Alternatively or in addition, the current driving situation of the vehicle is determined by means of at least one driving dynamics variable. According to the present invention, the recording or matching parameters corresponding to the driving dynamics and stored in the vehicle are superimposed or aligned by the device 105 for aligning the image in accordance with the driving situation. By this measure, in the combined image of the relevant driving situation Also avoid as much as possible, especially the shadow map or the reshaping map and the reshaping. The driving dynamics of the vehicle especially relates to its speed, near light

200480003555.6 The first or high beam is turned on, or it involves its positive or negative acceleration.

The above-mentioned alignment of the normalized image can be divided into three steps: moving, rotating and scaling. In fact, it turns out that sequential rotation, scaling, and movement provide the best quality results. Because the sequence of these steps is generally not interchangeable or interchangeable, it should be noted that the sequence of these steps is the same during the calibration of the night vision system according to the present invention and the subsequent operation. Where possible, the calibration and/or operating software of the night vision system shall be constituted accordingly.

Images aligned with each other are superimposed or combined in the superimposing or combining device 106 by processing their digital data according to software. A combined or superimposed image is generated from each image pair of the same time and position or the same object of the visual image and the infrared image, which is preferably provided to the driver of the vehicle on a color monitor in the vehicle.

Preferably, the combination of the image pairs with the same time and position of the two images is realized based on the respective pixel pairs of the visible image and the infrared image corresponding to each other or using multiple pixels of the two images. This may depend in particular on which desired resolutions are provided and/or which computing power is used for digital image processing. The preprocessed image as described is superimposed and displayed by digitally processing its image data. From the results, this process can almost be compared with the overlap of films or slides of the same scene or driving environment. In computing technology or in digital image processing, this is achieved by calculating the average value of pixel information, especially considering the brightness in each image and the color information contained in the visible image and/or infrared image. This does not necessarily have to be done pixel by pixel, but it can also be achieved by calculating the average value of the pixel areas in the two images with the same position and time. Moreover, it may make sense, that is, when calculating the average value. , The pixel information in the infrared image is weighted differently from the pixel information of the same time and position in the visible image. This different weighting can be realized, for example, according to natural light and/or weather and/or according to the headlight of a car and/or according to the color in the visible image; this can be realized, for example, that the red traffic light can be particularly noticeable in the combined image. Recognition. Moreover, the weighting of partial areas of the combined image, such as the difference between the foreground and the background, or the entire combined image, can be manually changed by the vehicle driver. In this way, a single image area can be particularly highlighted. Therefore, for example, the direct adjacent driving area of the vehicle can be highlighted more strongly to guide the driver with certainty. In the case of getting more and more riding, when calculating the average value Can increase infrared information relative to

200480003555.6 The weighting of visual information. When the close-range lights are turned on, compared with turning on the high-beam lights, the weight of the infrared information relative to the visible information can be increased.

Similarly, the information content of a region in each image can be weighted. If the information content in a region of the same time and position of the visual image is, for example, significantly higher than the same region of the infrared region, it makes sense that it is considered by the higher weighting of the visual information when calculating the average value.

As already described, the image generated by the camera or sensor must be preprocessed to correct and reliably align with the object. In order to save expensive memory, the software algorithm preferably accesses the sensor information of the cameras 101 and 102 in a pixel-wise manner.

The apparatus for digital image processing shown in FIG. 1 in whole or in part, preferably involves one or more software-controlled digital processors, which are preferably optimized for real-time digital image processing. But similarly, when the processing speed of one or more software-controlled PC processors allows the image to be processed as real-time as possible to provide a combined image with visible and infrared information, one or more software can also be considered more economical. Controlled PC processor.

200480003555. 6