Method and device for compensating for shadows in digital images

The present invention relates to a method for generating a calculated digital image of a scene that can be illuminated with a different illumination intensity under a first illumination angle by a first illumination source. At this time, a large number of individual images of the scene differ depending on an image sensor. It is recorded under illumination conditions and is cleared to each other pixel by pixel in the data processing device to generate the resulting image.

Furthermore, the present invention relates to a device for generating a calculated digital image, which is illuminated with a first illumination source capable of illuminating a scene to be recorded with different illumination intensities under a first illumination angle. It includes an image sensor for detecting the formation of an image of the scene and for outputting a digital individual image of the scene composed of a large number of pixels, and a control means for cooperatively controlling the illumination source and the image sensor. , Individual images of the scene can be recorded under different lighting conditions, and the apparatus further includes a data processing apparatus for clearing at least several individual images of the individual images pixel by pixel with each other.

This type of method and device is known, for example, in applications for image-assisted monitoring or control of very different processes, in which automatic analysis of the scene is desired and simplification of their application. For this purpose, image preprocessing for removing a predetermined disturbing effect is performed.

From Patent Document 1, for example, a method and a device for monitoring the internal space of an automatic vehicle (motor vehicle) are known, in which an infrared (IR) -sensing CMOS digital camera is used as an individual image of a scene. That is, an individual image of the interior space of the automatic vehicle is recorded. Furthermore, an infrared radiation source for illuminating the scene is provided in the interior space of the automatic vehicle. Two individual images are recorded in quick chronological order each time to generate the resulting image. At this time, during the recording of the first individual image, the scene is illuminated at full output by the infrared radiation source. During the recording of the second individual image, the scene is illuminated with reduced power by an infrared source, or in extreme cases not illuminated at all. Both individual images are then subtracted from each other pixel by pixel for the generation of the resulting image, so that the pixel values of the resulting image are modified for the infrared source independent background radiation portion, the so-called offset portion. ing.

For the purpose of monitoring the internal space of an automatic vehicle, for example, from Patent Document 1, protection against unauthorized access is known. As another purpose, for example, from Patent Document 2, it is known to monitor the position of a person in a vehicle. This dynamic form of surveillance is used for the control of passive safety systems such as airbags. Especially in this form of surveillance system, scene objects of interest, such as those whose position should be detected, are reliably distinguished from their own shadows or shadows cast by other scene objects. It is important to be able to be. This task, which is simple for the human brain, means a great deal of difficulty for automated systems, which requires enormous computational effort, and thus extremely time-consuming computational effort. However, this time factor is very dangerous in dynamic systems, especially passive safety systems.

<patcit num="1"><text>German Patent Application Publication No. 10062977A1</text></patcit><patcit num="2"><text>German Patent Application Publication No. 19908167A1</text></patcit>

<p> Therefore, an object of the present invention is to construct a method of the form listed at the beginning so that the shadow action of a scene object can be easily compensated.</p><p> Another object of the present invention is to construct a device of the type listed at the beginning so as to be suitable for carrying out the method according to the present invention.</p>

<p> The first task is related to the configuration requirements described in the premise part (so-called part) of claim 1, and at least one other lighting source is provided, and this lighting source causes the scene to be the first lighting. It is possible to illuminate with a different illumination intensity under a different illumination angle than the source illumination angle, and when recording individual images, different illumination conditions are the illumination intensity of the first illumination source and at least one other illumination. It is solved by being achieved by a different combination with the illumination intensity of the source.</p><p> The effects of the method according to the present invention are as follows. That is, individual images of the same scene object are generated, however, different lighting conditions cast shadows that are directed differently during each recording of the individual images, that is, the values of different pixels within each individual image are cast by shadow projection. It is to cause being affected. This makes it possible to compensate (offset) the effects of shadow projection within the framework of subsequent mutual clearing of individual images, and to reach a result image that is virtually shadow-free.</p><p> The following definitions are incorporated for further simplification of the description. Value I of pixel n in individual image j_j(n) is simplified and expressed as:</p><p><maths num="1"><img file="JP4399366B2_D0001.tif" /></maths></p><p> Here, ρ (n) is the surface reflectance (surface reflectance) of the scene point imaged in the pixel n, and E (n) is the irradiation intensity provided to this scene point. , Α is a proportional coefficient, and this proportional coefficient α describes the influence such as the brightness of the detector optical system (detector lens) used, the efficiency of the detector, etc., and for the sake of simplicity, here , Is assumed to be the same 1 (α = 1) for all pixels without limiting generality.</p><p> Certainly, various image processing methods can be considered, in which pixels that are affected by shadow projection in individual images are identified as such pixels and their values are shadowed during the generation of the resulting image. Can be modified to compensate for action. However, when all pixel positions are treated equally, at least one illumination, each of which is two pixels in the resulting image, is a scene point that is directly illuminated by a different illumination source or a different number of illumination sources. Clearing each of the individual images so that each of the two pixels in which the scene point illuminated by the source is imaged has substantially the same pixel value, except for a factor that represents the reflectance of each scene point. Is especially advantageous when</p><p> By applying the same mathematical operation to all images, that is, by applying the same mathematical operation to all pixel positions, the time-consuming operation itself in the image, such as an algorithm for recognizing contour lines, can be performed. Can be avoided. Such a method may be necessary, for example, within the framework of the analysis of subsequent scenes. However, in the preparation generation of the input image for analysis, that is, in the preparation generation of the input image for analyzing the result image of the method according to the present invention described herein, the preparation generation should be avoided for time reasons. Is advantageous. By equivalent treatment of all pixels of each pixel of the individual image, the individual images and, optionally, the intermediate images calculated from those individual images are observed as variables of mathematical operation in their respective wholeness. These variables can be applied in parallel to all or at least some of the pixels, for example by a special image processing processor. This achieves a far more advantageous time advantage over interpretations that have actual image content with respect to the object.</p><p> The operation to be applied to the image is such that each value of pixel n in the resulting image is selected to substantially represent the reflectance ρ (n) of the surface of the scene point imaged within pixel n. necessary. This means that the scene areas affected by the shadow action, that is, the scene areas that are not directly illuminated by all the illumination sources, are adjusted with respect to the scene areas that overlap each other and have no shadow by the image processing according to the present invention. Means that. In other words, ideal and shadowless lighting in the scene is simulated. At this time, scene areas that are not illuminated by the illumination source, that is, scene areas that cannot provide important information, can preferably be ignored, or those scene areas can be ignored during other subsequent image processing. It can be set to a value that identifies it. This marking should also preferably be done under equivalent treatment for all pixels. By "substantially" in this regard is meant that no other reasoned difference in lighting intensity, such as a difference based on the distance between the scene point and the lighting source and / or the image sensor, is taken into account. To do.</p><p> Basically, a number of suitable control modes for image recording and lighting can be envisioned. However, a prominent method by recording three individual images to calculate the resulting image has been shown to be particularly advantageous, in which case the two sources of illumination are controlled as follows: That is, when recording the first individual image, the first illumination source illuminates the scene with the first illumination intensity, the second illumination source illuminates the scene with the second illumination intensity different from the first illumination intensity, and the second illumination source. When recording an individual image, the first lighting source illuminates the scene with the second lighting intensity, the second lighting source illuminates the scene with the first lighting intensity, and when recording the third individual image, both lighting sources illuminate the scene. It seems that each illuminates with the second illumination intensity. As a result, it is particularly easy to settle each other of individual images. On the one hand, only a few individual images, i.e. three individual images, are required, and on the other hand, the illumination intensity between both sources in both the first individual image and the second individual image is exchanged according to the present invention. Produces substantially symmetrical illumination patterns, which facilitate further processing of the data.</p><p> When the individual images are cleared from each other, it is preferable to use an operation that can be performed with a small amount of calculation effort and does not lead to an excessive increase in pixel noise. That is, division by noise-loaded values should be avoided as much as possible. On the other hand, the mutual settlement processing of the individual images is the calculation of the pixel-by-pixel difference and / or total of the first individual image and the second individual image, and / or the value of the difference between the first individual image and the second individual image. And it has been shown to be advantageous to include a pixel-by-pixel sum calculation from the sum value. Furthermore, it has been shown to be advantageous to perform pixel-by-pixel subtraction of the third image.</p><p> An embodiment of the method according to the invention, in which a combination of mathematical operations is applied to calculate the resulting image, is particularly advantageous, the mathematical operation being expressed by the following equation:</p><p><maths num="2"><img file="JP4399366B2_D0002.tif" /></maths></p><p> Where I_{out out}Is the pixel value of the resulting image, I₁Is the value of the corresponding pixel of the first individual image, I₂Is the value of the corresponding pixel of the second individual image, I₃Is the value of the corresponding pixel of the third individual image. Not surprisingly, this number 2 is symbolic and also includes additional corrections or coefficients that do not essentially change the image-technical effects of operations that follow this number 2. It should be understood that it has been done.</p><p> The second issue mentioned above is the claim.<u style="single">4</u>In connection with the configuration requirements described in the premise part (so-called part) of, at least one other lighting source is provided, and this lighting source causes the scene to have a second lighting different from the first lighting angle. It is possible to illuminate with different illumination intensities depending on the angle, and those illumination sources have different illumination conditions depending on the control means, when recording individual images, the illumination intensity of the first illumination source and at least one other illumination source. It is solved by being controllable so that it can be achieved by a different combination with the illumination intensity. This is the technical premise of the underlying device for the methods described above to be feasible. The premise for the method according to the present invention to be feasible in the above-mentioned advantageous embodiments is a correspondingly suitable device in the control means and the data processing equipment. The advantages gained from the ability of the control means and preferably the programming technology device in the data processing equipment to implement the method according to the present invention can be seen from the above-mentioned advantages of the method itself.</p><p> The method according to the present invention is basically all kinds of illumination radiation, such as electromagnetic radiation in the visible region, UV region, IR region, X-ray region, microwave region, radio wave region, or within any frequency region. , Especially suitable for acoustic radiation in the ultraviolet region, but has been shown to be particularly reliable and low cost for application in the near infrared region. This frequency domain is particularly suitable for invisibility to the human eye, relatively low risk due to manipulative intensity, and at the same time good spatial resolution based on short wavelengths, especially in applications around people. ..</p><p> A particularly advantageous scope of application of a method according to the present invention and a device according to the present invention is the monitoring of the interior space of an automatic vehicle (motor vehicle); for example, passenger positions for theft prevention or for controlling a passive protection system, for example. This is for detection.</p><p> As mentioned above, the above embodiment ignores the effect of the distance between the scene point and the illumination source. This is often justified, especially if the distance between the scene point and the illumination source is not small compared to the distance between the two scene points. That is, in other words, the illumination source is similarly spaced from virtually all scene points. In this case, the approximation made implicitly in the upper part is correct that the illumination intensity E (n) for all the directly illuminated scene points also depends on the radiation intensity of each illumination source. Is. Moreover, if the shadow effect is certainly not completely compensated, but significantly reduced, it is sufficient for many applications.</p>

Other details of the present invention will be apparent from the following detailed description and accompanying drawings, in which the advantageous embodiments of the present invention are embodied as examples.

FIG. 1 shows an automatic vehicle (motor vehicle) 1 as a bird's-eye view. In the automatic vehicle 1, an image sensor 2 is arranged at a central position in a roof flyer, and the image sensor 2 is a vehicle 1. The interior space is detected at least partially within the near-infrared spectral region, and as is well known, a digital image is generated from it. The image sensor 2 is controlled by a control means (not shown), and provides the image data to a data processing facility (not shown) as well. Both of these devices can be incorporated, for example, into a board computer that is usually provided in modern vehicles anyway. Of course, it is also possible to use your own special unit.

A group of light emitting diodes 3 and 4 or light emitting diodes are arranged on each side in the area of the B pillar of the vehicle, and these light emitting diodes 3 and 4 emit light 5 and 6 in the near infrared region of the vehicle. It radiates into the internal space. The light emitting diodes are controlled with respect to their illumination intensity and their illumination time by control means (not shown), similar to the image sensor 2 that detects the illumination light of the diode reflected by the scene.

FIG. 2 shows a scene 7 having a scene object 8 as an outline, and the scene object 8 has a predetermined scene area 9, 10, with respect to illumination by one or both of the illumination sources (light sources) 3, 4. Make 11 a shadow. For the purpose of making the subsequent calculations as an example easier to understand, in Scene 7 different regions are additionally characterized by uppercase letters due to their shadow characteristics, and these uppercase letters are also used in Figure 3. .. At this time, A indicates an area directly illuminated by both illumination sources 3 and 4. B indicates an area that is directly illuminated by only the first illumination source 3. C indicates an area that is directly illuminated by only the second illumination source 4. D indicates a region located within the main shadow of the scene object 8 and not directly illuminated by any of the illumination sources 3 and 4.

FIG. 3 shows a graph of two intersecting set circles, each of which can be grasped as a set of illuminated scene points, in which the notation of the set is made according to the above description.

Next, the individual steps of a particularly advantageous embodiment of the method according to the invention will be described in detail.

In order to record the first individual image, the scene 7 is illuminated by the first light source 3 at the illumination intensity E1 and by the second light source 4 at the illumination intensity E2 (E2 &lt;E1). Furthermore, it is assumed that all scenes are additionally illuminated by diffuse background radiation E0, which is independent of illumination sources 3 and 4, and this background radiation is the same for all scene points. In this case, the number 1 is used to obtain the following value:

A: For pixels where scene points from scene area A are imaged

<maths num="A1"><img file="JP4399366B2_D0003.tif" /></maths>

B: For pixels where scene points from scene area B are imaged

<maths num="B1"><img file="JP4399366B2_D0004.tif" /></maths>

C: For pixels where scene points from scene area C are imaged

<maths num="C1"><img file="JP4399366B2_D0005.tif" /></maths>

D: For pixels where scene points from scene area D are imaged

<maths num="D1"><img file="JP4399366B2_D0006.tif" /></maths>

In order to record the second individual image, the lighting conditions are symmetrically exchanged according to an advantageous embodiment, that is, in scene 7, the illumination intensity E2 by the first light source 3 and the illumination intensity E1 by the second light source 4. Is irradiated with. In this case, the number 1 is used to obtain the following value:

A: For pixels where scene points from scene area A are imaged

<maths num="A2"><img file="JP4399366B2_D0007.tif" /></maths>

B: For pixels where scene points from scene area B are imaged

<maths num="B2"><img file="JP4399366B2_D0008.tif" /></maths>

C: For pixels where scene points from scene area C are imaged

<maths num="C2"><img file="JP4399366B2_D0009.tif" /></maths>

D: For pixels where scene points from scene area D are imaged

<maths num="D2"><img file="JP4399366B2_D0010.tif" /></maths>

To record the third individual image, scene 7 is illuminated by both light sources 3 and 4 with an illumination intensity of E2. In this case, the number 1 is used to obtain the following value:

A: For pixels where scene points from scene area A are imaged

<maths num="A3"><img file="JP4399366B2_D0011.tif" /></maths>

B: For pixels where scene points from scene area B are imaged

<maths num="B3"><img file="JP4399366B2_D0012.tif" /></maths>

C: For pixels where scene points from scene area C are imaged

<maths num="C3"><img file="JP4399366B2_D0013.tif" /></maths>

D: For pixels where scene points from scene area D are imaged

<maths num="D3"><img file="JP4399366B2_D0014.tif" /></maths>

These three individual images are subjected to pixel-by-pixel mathematical manipulation according to Equation 2, according to an advantageous embodiment. At this time, two intermediate images I in the two intermediate steps₄And I₅Is generated. 1st intermediate image I₄Is the first both individual images I₁And I₂Represents the pixel-by-pixel difference value (absolute value) of:

<maths num="3"><img file="JP4399366B2_D0015.tif" /></maths>

At that time, the pixel values acquire the following formats in different illumination regions.

A: For pixels where scene points from scene area A are imaged

<maths num="A4"><img file="JP4399366B2_D0016.tif" /></maths>

B: For pixels where scene points from scene area B are imaged

<maths num="B4"><img file="JP4399366B2_D0017.tif" /></maths>

C: For pixels where scene points from scene area C are imaged

<maths num="C4"><img file="JP4399366B2_D0018.tif" /></maths>

D: For pixels where scene points from scene area D are imaged

<maths num="D4"><img file="JP4399366B2_D0019.tif" /></maths>

2nd intermediate image I₅Is the first both individual images I₁And I₂Represents the pixel-by-pixel sum of:

<maths num="4"><img file="JP4399366B2_D0020.tif" /></maths>

At that time, the pixel values acquire the following formats in different illumination regions.

A: For pixels where scene points from scene area A are imaged

<maths num="A5"><img file="JP4399366B2_D0021.tif" /></maths>

B: For pixels where scene points from scene area B are imaged

<maths num="B5"><img file="JP4399366B2_D0022.tif" /></maths>

C: For pixels where scene points from scene area C are imaged

<maths num="C5"><img file="JP4399366B2_D0023.tif" /></maths>

D: For pixels where scene points from scene area D are imaged

<maths num="D5"><img file="JP4399366B2_D0024.tif" /></maths>

Result image I obtained according to Equation 2_{out out}Finally both intermediate images I₄And I₅Is written using:

<maths num="5"><img file="JP4399366B2_D0025.tif" /></maths>

At that time, the pixel values acquire the following formats in different illumination regions.

A: For pixels where scene points from scene area A are imaged

<maths num="A6"><img file="JP4399366B2_D0026.tif" /></maths>

B: For pixels where scene points from scene area B are imaged

<maths num="B6"><img file="JP4399366B2_D0027.tif" /></maths>

C: For pixels where scene points from scene area C are imaged

<maths num="C6"><img file="JP4399366B2_D0028.tif" /></maths>

D: For pixels where scene points from scene area D are imaged

<maths num="D6"><img file="JP4399366B2_D0029.tif" /></maths>

This result means that: That is, in the result image, the shadow effect is compensated (offset) in all the image regions representing the scene regions directly illuminated by at least one illumination source 3 and 4, and the pixel values are substantially reduced. , It is distinguished only by the reflectance value of each scene point, that is, only by the information that is actually of interest. This means that, in terms of images, an ideal light source with an illumination intensity of 2 × (E1-E2) and no shadow can be simulated by the method according to the present invention. Only scene areas that are not directly illuminated by the illumination source are set to virtually zero. This corresponds to the offset correction of the image and leads to the rejection of pixels that do not have important information.

Of course, a detailed description of this advantageous embodiment should be understood merely as an example. One of ordinary skill in the art will also recognize that the use of correction factors or correction terms, in particular reflecting the conditions of the geometry, does not touch the core of the invention.

<figref num="1">It is a figure which shows the schematic diagram of the automatic vehicle provided with the internal space monitoring mechanism according to this invention.</figref><figref num="2">It is a figure which shows the schematic diagram of the different shadow action within the frame of application of the method according to this invention.</figref><figref num="3">It is a figure which shows the graph for demonstrating the different illumination area of one scene.</figref>

Code description

1 Automatic vehicle 2 image sensor 3 Light emitting diode (illumination source) 4 Light emitting diode (illumination source) 5 Near infrared light 6 Near infrared light 7 scenes 8 Scene object 9 Predetermined scene area 10 Predetermined scene area 11 Predetermined scene area A Area directly illuminated by both illumination sources 3 and 4 B Area directly illuminated by only the first illumination source 3 C Area directly illuminated by the second illumination source 4 only D Area that is located in the main shadow of scene object 8 and is not directly illuminated by any of the illumination sources 3 and 4.