Method and device for processing image files

p0001The invention relates to a method and apparatus for processing image data at least one image pixel by pixel captured by an image acquisition unit, wherein the imaging unit comprises at least one imaging sensor with a grid-like arranged pixel detection areas.

p0002For classifying objects, in particular for classifying lights, potential objects are detected in captured images. This is especially useful and necessary when image data by means of arranged in motor vehicles image capture units captured images are processed by the environment of the motor vehicle in order to detect at least some of which present in the environment of the motor vehicle objects and classify. Usual in motor vehicles which is arranged and associated with motor vehicles imaging units are monocular cameras or stereo cameras, and color / or detect the grayscale images. Both objects are detected and classified, which differ due to its light reflection properties of their environment as well as objects that have a separate light source. So to be detected as objects especially in the dark taillights and / or brake lights and positioning lights of vehicles. Such objects can be reliably detected when the image is projected onto a sensor surface of the image capture unit to several pixel detection ranges, so that the position and brightness of these objects can be determined with sufficient accuracy. This information may include, for example, be used in a driver assistance system from the traffic ahead adjusts the light distribution of the headlight system own function. For this particular light sources are determined driving ahead and oncoming vehicles and classified as taillights, brake lights, headlights, rear lights, turn signals and / or position lamps based on the detected information. The farther away the object is from the imaging sensor and the smaller the light emitting surface or the light-reflecting surface of the object is, the smaller the object image on the sensor surface of the imaging sensor. The optical systems of conventional image capture units have a point transfer function that causes an intensity distribution of incident light. The planned during the image acquisition unit point transfer function a point-like object is imaged on an area of ​​at least one pixel detection area. The image of a distant object can both have a small size as an image point and carried on a plurality of image point detection areas is effected so that an intensity distribution of the emitted light from this object on at least two adjacent pixel sensing areas. Thus, in such an image of the object, the size, the intensity and the position of the image of the object on the sensor surface can not be determined sufficiently accurately. Currently, the information such pictures are not further processed, since this information for evaluation of the image and the classification of the object are not sufficient.

p0003The object of the invention is to provide a method and apparatus for processing image data, by including information about objects can be identified, their pictures would be without an intensity distribution of the optical system of the imaging unit displayed on an area less than or equal to a pixel detection area.

p0004This object is achieved by a method having the features of patent claim 1 and by an apparatus having the features of patent claim 18. Advantageous developments of the invention are specified in the dependent claims.

p0005By a method having the features of claim 1 is achieved that the actual picture size of the object and / or the actual position of the image of the object without the caused by the optical system of the imaging unit intensity distribution in a simple way can be found. With the help of the determined picture size, the determined intensity and the determined position of the image can then be performed to classify the object. Preferably, more information will be used for classification. The classifier can perform at least a plausible classification of the object and specify a hypothesis that it is the object is an object of a specified object class, with a determined probability, or a hypothesis that it is the object to an object of an object class is set up and / or confirm.

p0006It is advantageous to determine the progress of the color and / or brightness value of at least two adjacent pixel detection areas, preferably adjacent a matrix of 2x2, 3x3, 4x4, 5x5 or 6x6 pixel areas. Further, the intensity distribution of the emitted light beams from the object can be accomplished in at least two adjacent pixel areas, said intensity distribution is caused by the aberration or aberrations of the optical system of the imaging unit when imaging of objects.

p0007In a further development of the invention, at least one comparison amplitude value of the actual brightness value can be preset, wherein the distributed by the intensity distribution on the at least two pixel detection ranges light amount of the emitted by the imaged object and incident upon this pixel detection areas light is determined. Depending on the comparison amplitude value, the actual picture size of the object is determined. For this purpose, the entire determined using the pixel detection ranges light amount is determined, in particular, the quantity of light based on the preset amplitude value comparison and the actual imaging surface of the object is determined on the sensor surface.

p0008The comparative amplitude value can the minimum thickness, the permissible maximum light intensity and / or the normal luminous intensity of a lamp of a motor vehicle sector. The minimum luminous intensity and as a permitted maximum intensity specified values ​​can be used in the ECE Regulations and / or the published by the Society of Automotive Engineers regulations. The usual light intensity may correspond to the average light intensity of light of the current car and / or truck models. The lamp is especially a tail light, brake light, marker light, flashing light, a headlamp and / or another light of a motor vehicle.

p0009Furthermore, it is advantageous to determine due to the light distribution of the emitted from the object incident on the at least two imaging areas light a position of the image of the object on the sensor surface independent of the grid-like arranged pixel detection ranges or by specifying the position of the image using at least one Subbildpunkterfassungsbereichs. This may be the position of the detected image of the object relatively precisely specified and used for further processing.

p0010It is also advantageous to preset multiple comparison amplitude values, each corresponding to the intensity of light and / or a minimum intensity, allowable maximum light intensity and / or conventional light intensity of a motor vehicle light. This can be easily checked, if it is determined the object is such a motor vehicle light. Here, the image information can be processed in particular to each of the comparison amplitude values, thereby in each case a hypothesis that it is the imaged object is such a luminaire, is accepted and verified in the further evaluation of the object. The parallel tracking multiple hypotheses can be carried out in parallel processing operations as well as sequentially or in multitasking mode.

p0011Furthermore, the actual image size of the image of the object on the sensor surface and / or the actual position of the object on the sensor surface can be determined. By means of this size and / or position of the image of the object on the sensor surface a classification procedure may be performed to classify the object. The picture size and / or position relative to the sensor surface can be easily processed.

p0012Furthermore, the actual position of the image of the object on the sensor surface at a can also at least one known aberration of the optical system of the image acquisition unit and the detected course of the color and / or brightness value are determined with the aid.

p0013The pixel detection areas preferably form the sensor surface of the imaging sensor and arranged in an image plane of the image capture unit. The optical system of the imaging unit forms the image of the objects in the image plane from preferably sharp, so that the objects are mapped relatively sharply on the sensor surface. This further processing of the image data by relatively sharp edges of objects is easy.

p0014To determine the actual image size of the image of the object and / or to determine the actual position of the object in advantageous embodiments of the invention, a deconvolution method is used. This particular point spread function of the optical system of the imaging unit may be considering a known aberration, conclusions on the picture size and / or imaging position of the imaging of the object to be drawn.

p0015Preferably the image using a arranged in a motor vehicle imaging unit is detected, especially by means of a monocular front camera or a stereo camera. This can be detected and evaluated in the recorded by the environment of the motor vehicle images just Objects. Using the detected and verified information objects can be further processed through these objects of driver assistance systems of the motor vehicle. The imaging unit may in particular take multiple pictures one after another, the image data is processed repeatedly. The actual position of the image of the object can then be determined in the recorded images and tracked over several successively captured images. For this purpose preferably a known tracking method is used.

p0016The apparatus with the features of claim 18 may be further formed in the same manner as described for the method having the features of claim 1, wherein with this device, the same advantages are obtained as explained in connection with the method. In particular, the device having the features of claim 18 having the features defined for the development of the method according to claim 1 in the dependent claims or corresponding device characteristics can be further developed.

p0017Further features and advantages of the invention will become apparent from the following description, explained in more detail in conjunction with the accompanying figures, the invention using an exemplary embodiment.

p0018Show it:<dl id="dl0001"><dt>figure 1</dt><dd>a block diagram of a system for image acquisition and image analysis of the environment of a motor vehicle,</dd><dt>figure 2</dt><dd>a diagram showing the dependence of the intensity of the radiation energy impinging on the sensor surface of an imaging sensor light from the distance of an object to the sensor surface, </dd><dt>figure 3</dt><dd>a diagram of a dependency of the size of the image of the object on the distance of the object to the sensor surface,</dd><dt>figure 4</dt><dd>a diagram showing a function of the determined by the pixel sensing elements brightness or gray scale value of the distance between the sensor surface and the object,</dd><dt>Figures 5 to 7</dt><dd>the imaging of objects as a function of the distance of the object to the sensor surface of the imaging sensor,</dd><dt>Figures 8 to 11</dt><dd>a chain of transmission members for imaging the object on the sensor surface of the image capture unit, and for generating and outputting of the picture corresponding to the brightness values, and</dd><dt>Figures 12 to 14</dt><dd>a model-based reconstruction of some features of the object.</dd></dl>

p0019In <figref idrefs="f0001">figure 1</figref> is a block diagram of an image recording and analysis system 10 shown, which is integrated into a motor vehicle, especially in a car or truck. The image recording and analysis 10 includes a camera system 12 that captures an image sequence with images of at least a portion of the car, and generates corresponding image data. This image data is transferred 10 to an image processing unit 14 of the system and analyzed by means of control and processing modules 16a to 16d of the image processing unit 14 and processed.

p0020The image processing unit 14 is connected via a vehicle bus 18 of the vehicle with other control and processing units 20 to 26, such as the navigation system, the Geschwindigkeitserfassungs- and / or control system of the vehicle, wherein the image processing unit 14 via the vehicle bus 18 data with these further control units 20 can exchange up to 26th The image processing unit 14 is further connected via a suitable data line with a display and warning unit 28 and with a Geschwindigkeitsregel- and / or limit control 30th Further, the image processing unit 14 is connected via a data line with a permanent memory 32nd Such non-volatile memory 32 may for example be a hard disk, a flash memory or another controller. Further, the nonvolatile memory 32 can be provided by a database system of the motor vehicle. The camera system 12 may comprise a single camera (monocular camera), several mono cameras, a stereo camera system and / or more stereo camera systems, the individual cameras grayscale images or color images produce. As vehicle bus 18, conventionally known systems such as a LIN bus, a CAN bus and / or a FlexRay bus, are used. The optical systems of the camera or the cameras of the camera system 12 can in particular have different fixed focal length or an adjustable particular via a turret lens system or a zoom lens system focal length.

p0021Via the vehicle 18, the image processing unit 14 also communicate with other sensors and receiver units that can in particular an ad hoc communication with other vehicles and / or transport facilities, such as traffic signals, gantries and / or individual road signs record. Furthermore, via the vehicle 18 information on signal states of individual actuators and sensors, in particular on the activation of the direction indicator lamps of the vehicle are transferred.

p0022Using the recorded by the camera system 12 sequence are by analyzing the image data using the image processing unit 14 detects the objects in the detection range of the camera system 12 and classified. By classifying the objects by means of a suitable detection method to traffic signs, traffic control and / or other road users, particularly driving ahead and oncoming vehicles are detected and classified. These objects detected and optionally present additional information about these objects can then be further processed by at least one driver assistance system. It is advantageous to further processing by a brake assist system, a light controlling assistance system, a distance warning and / or control system.

p0023A further processing and control module 16a to 16d of the image processing unit 14 may determine the current lane of the vehicle and / or the driving lane of a detected object as the preceding vehicle or the oncoming vehicle based on the images of the image sequence.

p0024In <figref idrefs="f0002">figure 2</figref> there is shown a diagram showing the dependence of the intensity of the radiation energy or the irradiation intensity of light incident on the sensor surface of an imaging sensor of the camera system 12 is an image of an object generating light from the distance of the object to the sensor surface. Graph 50 shows the trend of decrease in intensity of the radiant energy of a radiation source with the increasing distance from the radiation source and the object to the sensor surface of the imaging sensor. The radiation source is a lamp, in particular a motor vehicle.

p0025In <figref idrefs="f0002">figure 3</figref> there is shown a diagram in which the size of the image of the object is shown as a function of the distance of the object to the sensor surface. The course of the graph 52 shows that the size of the image of the object is, whenever possible, with the aid of the imaging sensor, to the size of the figure is below the surface of a pixel sensing element. At a size of the image of the smaller area of ​​a pixel sensing element, the image of the object is carried out only on part of a pixel sensing element.

p0026In <figref idrefs="f0003">figure 4</figref> is represented by a function of the determined point of the image sensing elements brightness or gray scale value of the distance between the sensor surface and the object is a graph. At a short distance of the object to the sensor surface, the image of the object is performed on multiple imaging areas. This gives each pixel sensing element whose pixel detection area is completely covered by the image of the object, a constant maximum brightness value. If the image of the object with increasing distance mapped only on part of the pixel coverage of a pixel detection element, the incident on this section of the pixel sensing element radiant energy is indeed constant, but from the pixel sensing element only over the entire image capture area of ​​the image point detecting element average brightness or gray scale value is output , The thus caused changes in brightness is shown by the graph 54, in which the brightness value decreases, after the distance between the object and the sensor surface has increased so that the image on the d_{1Pix<sup2>2</sup2>} marked point the image of the object is performed only a part of the imaging area of ​​the pixel sensing element. In the embodiment according to<figref idrefs="f0003">figure 4</figref> is a drop in the brightness displayed when the image of the object is smaller than the pixel detection region of the pixel sensing element. This is the case when the environment of the object is darker than the object itself. In other embodiments, the environment of the object is brighter than the object itself, so that then an increase in the brightness from the point d_{1Pix<sup2>2</sup2>} takes place.

p0027In the <figref idrefs="f0004 f0005">Figures 5 to 7</figref> is shown to the sensor surface of the imaging sensor of the camera system 12, the imaging of objects as a function of the distance of the object. In<figref idrefs="f0004">figure 5</figref> the brightness values ​​of eight neighboring pixel detection areas are represented, the object having a size and such a small distance to the camera system 12 that four adjacent pixel detection areas are completely covered by the image of the object and each outputting a corresponding to the emitted light from the object gray value. This is both the size and the position of the object or the image of the object on the sensor surface, easily determinable.

p0028In <figref idrefs="f0004">figure 6</figref> the object has a greater distance to the camera system 12, so that the image of the object of the eight adjacent pixel detection areas only the fifth pixel detection area completely covered. As a result, the position and the size of the image of the object on the sensor surface be determined with sufficient accuracy.

p0029In <figref idrefs="f0005">figure 7</figref> is the object against the <figref idrefs="f0004">Figures 5 and 6</figref> arranged in an even greater distance to the camera system 12, so that the image of the object is imaged with a relatively low average detected amount of radiation at the adjacent pixel sensing elements. None of the pixel sensing elements can thereby determine the actual radiation intensity of the emitted by the object light, so that neither the correct brightness nor the correct size of the image of the object or the correct position of the image of the object on the sensor surface of the imaging sensor can be determined.

p0030In the <figref idrefs="f0006">Figures 8 to 11</figref> is shown schematically the chain of transmission members for imaging the object on the sensor surface of the image capture unit and generating and outputting the brightness values ​​corresponding to the figure. In<figref idrefs="f0006">figure 8</figref> is shown schematically itself the object. The imaging of the object is changed by the optical system of the camera system 12 or by its point transfer function such that in<figref idrefs="f0006">figure 9</figref> Intensity distribution shown the radiated from the object and the optical system of the camera system 12 supplied light is effected. This intensity distribution a point object is mapped to at least one pixel sensing element or a plurality of adjacent pixel sensing elements. The incident on three adjacent pixel detecting elements in the present embodiment is light in<figref idrefs="f0006">figure 10</figref> shown, wherein according to the in the center pixel sensing element, the largest amount of light <figref idrefs="f0006">figure 9</figref> Light distribution function shown falls. According to another caused by the digitization of the incident light quantity loss of information are as an output or as image data for the three adjacent pixel sensing elements in the<figref idrefs="f0006">figure 11</figref> Brightness values ​​shown schematically issued. It can be seen that for the left pixel sensing element and the right pixel sensing element, the same brightness values ​​are output, although different amounts of light are incident on these pixel sensing elements, as in<figref idrefs="f0006">figure 10</figref> is shown. However, this difference is no longer included as an information in the output signal by digitizing or by the resolution of the digitized signal. Conventional imaging systems and camera systems 12 have a resolution 256-4095 grayscale, so the determined brightness values ​​must be normalized to this grayscale. This normalization leads to a using<figref idrefs="f0006">Figures 10 and 11</figref> Loss of information shown.

p0031By in <figref idrefs="f0006">figure 11</figref> Output is shown in the prior art, a sufficiently accurate determination of the properties of the object in <figref idrefs="f0006">figure 8</figref> depicted object can not be determined. In particular, the position of the image of the object on the sensor surface and the size of the image of the object on the sensor surface can not be determined sufficiently accurately.

p0032The invention is to determine the size of the image of the object preferably from the assumption that the light intensity or the minimum luminous intensity of a lamp or light source is constant and independent of the distance between the light source and sensor surface as a property of the light source. If this assumed light intensity assumes the actual size and / or the actual position of the image of the object due to the whole of the adjacent picture elements incident through the imaging of the object amount of light can be determined. When all the pixels of the image of the object amount of light striking the brightness difference is determined to further adjacent pixel elements preferably on which the image of the object is not shown. The actual size, ie the actual area, the picture on the sensor surface can be easily calculated. Using this information, in particular the size of object itself or the ratio of the size of the object can be determined at a distance or to the size of another object detected. This can be found at the distance of two tail lights of a motor vehicle and checked for plausibility in particular the size of a tail lamp with respect.

p0033It can also be determined by the light distribution in various adjacent pixel areas, the actual position of the image of the object on the sensor surface. This position can be specified as a coordinate or subdivision of the grid-shaped pixel elements in Subbildpunktelemente using the coordinates of these Subbildpunktelemente indicated.

p0034The intensity distribution is caused in particular by an aberration or by a plurality of aberrations of the optical system of the camera system 12th

p0035In the <figref idrefs="f0007">Figures 12 to 14</figref> is a model-based reconstruction of some features of the object in <figref idrefs="f0007">figure 12</figref> shown shown, whose image has been captured by the camera system 12, in which three pixel sensing elements of the camera system 12 record and output caused by the image of the object brightness values.

p0036In <figref idrefs="f0007">Figur13</figref> is shown the point spread function of the optical system of the camera system 12th In reconstructing the brightness values ​​so that the actual picture size in be inverted using the point transfer function,<figref idrefs="f0007">figure 14</figref> Object represented, the actual intensity of the emitted light from the object and / or the actual position of the object can be determined.