Method for detecting and/or tracking objects

The present invention relates to a method for the detection and / or Tracking of objects, the objects in a detection area a sensor for electromagnetic radiation are, as well as a Apparatus for carrying out the method.

Method for detecting and / or tracking of objects are basically known. Typically, while at constant intervals Images of a detection range by a sensor for electromagnetic Radiation, such as a laser scanner or a video camera, detected. The pictures will then search for objects, the objects meet in the detection area. Was in an image found an object for the first time, is in subsequent cycles of the process searched for this object to its location or change in position in be able to follow the course of time. For this purpose, often starting from the Position and velocity of an object in a previous cycle its location for a current cycle or in a current image is predicted, then in the image near the predicted position of the object discovered elements, eg segments, the object Assign the previous cycle and thus detect its current location to.

Such methods are suitable for example for monitoring a range before and / or adjacent to a motor vehicle. One possible advantage Such monitoring can be that sudden Threats automatically detected and appropriate countermeasures can be initiated. For this it is however necessary that objects can be quickly detected and tracked accurately.

In the methods mentioned above is with increasing duration of Tracking an object often the accuracy of the detected object properties increase as in the course of pursuing further information can be collected on the object, a better characterization allow the object or the state of the object. Accordingly can especially occur inaccuracies when in Images new items occur. In the example of the motor vehicle, it can is here for example, pedestrians, the obscured, by another object such as a on a kerbside parked motor vehicle, suddenly into the street and then be detectable by the sensor. The same problem can occur when pedestrians are positioned in front of an object so that the Object tracking method the pedestrian and the object to a summarizes object so that the pedestrian is also detected only if he has far enough away from the object.

The prelim ying invention is therefore based on the object, a method for detecting and / or tracking of objects, the objects in a detection range of a sensor for electromagnetic radiation, meet to provide that and rapid detection Tracking of objects allowed, and a corresponding device to provide for performing the method.

The object is achieved by a method having the features of Claim 1.

In the method for detection and / or tracking according to the invention of objects, the objects in at least a detection region at least one sensor for electromagnetic radiation are, on the basis of repeated images detected by the sensor of the detection area are consecutive in successive cycles Pictures evaluated and there is at least one each in aufgefundenes one cycle on the basis of a corresponding image Object in a later cycle in a corresponding later image wanted to track the object. It is on the basis of the results an at least preliminary evaluation of a current image while at least one current cycle at least one in the current Cycle determined part of a current image or a current in the Cycle detected object at least a part of an earlier image and / or at least an indication relating to an earlier state the object or a name corresponding article, under Using a corresponding earlier image in the current Cycle is determined assigned.

The object is further achieved by a device with the features of claim 11.

The inventive device for detecting and tracking Objects has at least one to capture images of a detection region formed sensor for electromagnetic radiation and means connected to the sensor data processing means, which is designed for carrying out the method according to the invention and in particular means for evaluation of current images of the sensor, Means for determining parts of a current image as a result of at least preliminary evaluation of the current image or for the recognition an object in the current cycle and means for assigning of parts of an earlier image and / or for the determination of at least an indication relating to an earlier state of the object or a this corresponding object using a corresponding earlier image in the current cycle and association of the includes indication to the portion of the current image or the current object.

The method according to the invention, the means of the invention Device is feasible, useful for the identification and / or tracking of objects in a detection area. This can in principle be stationary or moving, which is particularly the case , when the images by means of a sensor attached to a vehicle are recorded.

The images are of at least one sensor for electromagnetic Radiation detected. The sensor may be a purely passive Sensor act which receives only electromagnetic radiation. It can However, a sensor may be used, via a source of particular directed radiated electromagnetic radiation by an illuminated by the radiant point or area on a Object can be thrown back, and at least one corresponding reflected on a subject point or range Radiation-sensitive sensor element has. Furthermore, even used combinations of sensors for electromagnetic radiation, be that detect various types of images. The detection areas the sensors need then not necessarily overlap, but it is preferred that this is at least partially overlap.

Under a picture this is especially a lot of data understood that a respective position of a point or area on or more detected by the sensor in at least one object Direction transverse to a direction of view of the sensor and at least one Property of the object point or reflect -area. The Location of the object point can, for example, upon detection opposite of object points in a scan plane by an angle an arbitrary but fixed predetermined reference direction, for example, the sensor, be given. Upon detection of object points or region in space the layers may by layers of sensor elements the sensor and optionally added imaging properties be. In the property may be, for example, the intensity and / or wavelength or color of the emitted optical radiation, The distance from the sensor or the speed in the radial Direction act relative to the sensor. For example, video images Data intensity radiated optical radiation as a property and with respect to a sensor element that the radiation of an object point or -area has detected, and thus with respect to a containing layer while removing photos as properties of the detected Object points or regions of the distance from the sensor and as the position information, an angle at which the object point may contain has been detected or range. speed pictures finally contain data about the speed of the object points or ranges relative to the sensor.

The images need not directly by a single sensor to be detected. Rather, at least two, it is also possible to temporally mutually assigned images for by at least two sensors electromagnetic radiation were detected, an overall picture to combine, the greater detection range or additional Properties of object points or regions reproduces. For example, each of two images which are spaced by two arranged video cameras were recorded, of taking into account Imaging characteristics and the arrangement of the video cameras range images are generated which in addition to a distance Object point or -area of ​​one relative to the video cameras the stationary point of reference data with respect to the intensity or Color of each of the object point or range emitted have light.

Furthermore, different types of images can an overall picture combined or together or interdependently are processed. Thus, preferably video and range images are processed together, without the narrower sense a set of image data combined to be.

The images used in the inventive process are to recorded consecutive times, which is preferably a constant time interval from one another. The gauge of according to the invention apparatus is therefore for repeated detection of Images of the detection range displayed. If for one cycle uses a plurality of images, these are preferably substantially synchronously detected, ie their sensing times differ by one smaller than the period of time between successive samples by the sensor with the lowest recording rate.

The captured images are then analyzed in consecutive cycles, wherein in a given cycle, at least one corresponding Image is evaluated. If multiple images substantially synchronously recorded, is sufficient that one of the images evaluated at least provisionally becomes.

In each case, at least one in one cycle on the basis of at least a corresponding image aufgefundenes object in a later Cycle wanted in a corresponding later image to trace it. These relate particularly to conventional object recognition and -verfolgungsverfahren be used. In this case, an assignment of pixels of the current cycle in a previous Cycle or image detected objects occur.

In particular, a prediction method may be used, in which on the basis of the position of at least one object in a former, in particular in a preceding cycle and the speed a position in the current cycle is predicted and around the predicted position after the object is searched. New objects can determined from parts of an image, ie pixels or in the cycle Quantities of image points, in particular segments are formed, could be assigned not already known objects.

According to the invention, it is now deviating from conventional object recognition and -verfolgungsverfahren provided that for at least a current cycle on the basis of the results of at least preliminary evaluation of a current image, at least one in the current cycle determined part of a current image or in the current cycle detected object at least part of a previous Image and / or at least an indication relating to an earlier State of the object or of an appropriate object, the is determined on the basis of a corresponding earlier image, associated with becomes.

Are there more substantially synchronously captured images same or different type is used, it is sufficient that a preliminary evaluation one of these images is done, the assignment may be parts of the the same or a different, substantially synchronously captured image done.

This means that in a current cycle is not only a current image is evaluated. Rather, first the least provisional or optionally partially evaluating a current image in the current Cycle performed. Based on these preliminary least Evaluation, ie in particular in dependence on the result of the at least preliminary evaluation, then for object tracking and Determination of data with respect to an object using a previous image, which is in particular an immediately before the current picture captured image associated with a preceding cycle act can.

In the preliminary analysis can be particularly at least one criterion be reviewed, reference to which it is decided whether an earlier Image to be used or not. This then means that for non-compliance the criterion an otherwise known object recognition and can be carried out tracking.

In the at least preliminary evaluation is part of a current determines the image, which is one pixel or in the current cycle determined amount of pixels, for example, at least a segment that can act. Where in a current cycle used more current images, while the picture that preliminary needs was evaluated, not necessarily coincide with the image of the part is determined. However, it may also be an object in the current be image identified and used. Both the determination of the part the image and the recognition of the object can with first Process steps occur to those in the conventional method same.

The figure recorded in the at least preliminary evaluation part of the current Image or the object in the current cycle can then be a part of be assigned to the previous image, which is in turn around one pixel or one determined in the current cycle amount of Pixels, for example, at least one segment, the previous image can act. The parts of the images can each have different have numbers of pixels, so that no one-to-one mapping to be given of pixels needs. For any first and / or second objects and / or parts of the first and / or second Images is thereby under an assignment of a portion of a first image or a first object to a part of a second image or a second object understood that the part of the second image or the second object is specified, and for an appropriate part of a first Part or a suitable first object of optionally several alternatives is selected and assigned.

The portion of the current picture or the object in the current cycle However, an indication can be assigned, based on the object or a name corresponding article refers. This figure can be, and to values ​​of any state variables of an object Article act, for example, the location, speed, shape, Size, allocation to one of several classes of objects in the objects according to their relevance to the object tracking typical properties be classified, or the presence of the object at all. To determine the indication not need the entire previous image to be used, rather, the determination can also only Subregions limit. However, for the determination is always immediately at least one pixel of the previous image, preferably to still may be stored during the current cycle, and at the same time the result of the at least preliminary evaluation of the current Image used.

In any case, it is essential that a part of the earlier image or of a Statement obtained from the previous image to a part of the current image be assigned or an object of the current cycle and not Part of the current image objects or information, the former from the Image obtained.

This means a total of that for object tracking, depending on the preliminary Evaluating information from a current image to a renewed and improved at least partial re-evaluation of the previous Image are used, the results of which in the current Cycle are used. In this way, in the images contained Information to be better evaluated, resulting in a faster and more accurate Object recognition and tracking permits.

The data obtained by the process can then output or are stored in order of the following devices, for example, used to be used to control a vehicle.

To carry out the individual steps in the inventive Device, a data processing device is provided which with the sensor for electromagnetic radiation for the transmission of associated images. The data processing device may be wholly or partially as a non-programmable circuit, what the execution speed increases. Preferably, the Data processing device, however, a for carrying out the invention Process programmable processor and associated with it a memory, an output interface and an input sections to the on sensor.

In particular, means for evaluation of current images of the sensor, Means for determining parts of a current image as a result of at least preliminary evaluation of the current image or for the recognition an object in the current cycle and / or for the assignment of Parts of an earlier image and means for determining at least one An indication relating to an earlier state of the object or of this corresponding object on the basis of a corresponding previous image and assigning the item to the part of the current Image or the current object has to be provided, in whole or partially formed as a non-programmable electric circuit may be, but preferably by an appropriately programmed Processor are formed.

Further developments and preferred embodiments of the invention are , The drawings and the claims described in the specification.

As the sensor, a sensor for any electromagnetic radiation be used. In a preferred embodiment of the invention Device as a sensor, at least one radar sensor, particularly a spatially resolving radar sensor, used. Accordingly be spatially resolved preferably in the inventive process Images of a radar sensor used. Radar sensors can in particular radial velocities of objects relative to the Radar sensor may detect directly.

Preferably, however, at least one sensor for optical radiation, ie, radiation in infrared, visible or ultraviolet range of the electromagnetic spectrum can be used. Such sensors have generally a better spatial resolution than radar sensors.

Sensors are particularly preferred for recording distance images used. For example, a stereo video camera system with at least two video cameras are used, which in a plane in of a distance detected by the stereo video camera system, are spaced apart. Preferably, however, sensors used in which at least one scanning beam of electromagnetic Radiation is used for scanning at least a scan. It allows multiple scanning beams to substantially simultaneous Sampling are used strips in a scan or may, preferably, a scanning pivoted in the scanning plane are being reflected in each of objects with radiation a corresponding detector is received. Specifically, corresponding laser scanners are used, by means of which at least a pulsed laser beam of radiation through a detection area pivoted, from an object reflected radiation of the pulsed laser radiation beam can be received and on the basis of duration a pulse of the laser radiation beam to the object and back to the laser scanner a distance of the object with angular resolution can be determined. Such laser scanners are characterized by a good spatial resolution with high rate of detection. particular preference Laser scanners are used, the more like a fan above the other disposed scan planes scanned.

Furthermore, it is preferred that in the inventive method Video images are used and that the device according to the invention for capturing video images having a video system comprising at least a video camera may comprise an imaging optical system and a sensor system for spatially resolved reception of optical radiation, For example, visible light and / or infrared radiation, comprising. As a video camera may vary by application, in particular a wide-angle or a panoramic camera are used. to monitor the distance zone is a video camera with a telephoto lens particularly suitable. It can in principle also with video cameras Zoom lenses are used. Furthermore, a sensor system as Black and white or grayscale system or even a color-sensitive System used. Here, for example corresponding CCD or CMOS sensor elements are used.

Furthermore, combinations of sensors can for electromagnetic Radiation are employed, the same or different types capture images. The need detection ranges of the sensors not to be identical, it is sufficient that this in itself monitoring area overlap.

The object recognition and / or object tracking in the current cycle to improve, it is preferred that the assignment of the part of the former Image or statement with respect to the previous state of the object for the determination of a further indication with respect to the current state of of an object is used. The determined additional disclosure may be not to refer to the object for which the statement regarding whose previous state was determined. Rather, it may be a act on any object in the current cycle. It must therefore be effectively a least partially recursively working Method in which, starting from the information from at least preliminary Evaluation of a current image first new information on corresponding an object or an object to a previous image Time is obtained that the turn for evaluation the current image is used. Thus, the existing information much more effectively exploited in the images than exclusively linearly operating procedures.

Particularly in the treatment of newly detected objects, it is preferable that when a new object is recognized in a current cycle, a current position of the new object is determined in the current cycle, that based on the current location of the new object a previous location the new object in an earlier cycle is estimated and that using the estimated earlier position of the new object of current value of a state variable of the new object in the current Cycle is determined. In particular, as a state variable speed of the object are estimated, otherwise on the basis of only an image can not be determined.

That an object in the earlier cycle initially not detected was, several reasons may be. In a further development of the Method of the invention it is therefore preferred that on the basis of the current location of the new object in the current cycle and a former location of at least one other object in an earlier cycle it is checked whether the in the evaluated in the previous cycle image corresponding to the new object object by the other be concealed object in the earlier cycle corresponding article could. Under masking is understood to mean that object points the hidden object not of the sensor used were detected as areas of the other object in the propagation the electromagnetic radiation from the object points to were arranged in the sensor. This further facilitates the one occlusion detection in the previous cycle and allowed by Estimate the size and position of the hidden portion of the detection range corresponding estimate the position of the new object Object at the time of the previous image.

It is particularly preferred that for the new object in the current Cycle, a current size and / or shape is determined, and that the current size and / or shape used in testing for concealment will. This can for example be ensured that a concealment is only detected when a detectable by the sensor Silhouette or a detectable by the sensor outline the new object or of the corresponding object from the point of view of the sensor less than the corresponding outline or silhouette of the corresponding other object. Moreover, the size and / or shape serve the new object to this at first in one of several classify object classes with each objects for object recognition and tracking characteristics include. For example, object classes for pedestrians, motor vehicles, Road boundaries or the like provided. Dependent on can of the object classes then assess whether the subject in the present time between the detection of the current image and the previous image with the maximum of objects concerned Object class possible speed from another by the have moved object hidden area in the detected position could.

From the by evaluating the current image in the current Cycle obtained information that a new object in the detection area is present, ie, by the occlusion verification be determined whether an object corresponding to this new Object already in a previous time in the detection area was the sensor available. In particular, in this case, it is preferred that assuming a concealment of the current situation and preferably the current size and / or shape of the masking Object corresponding object an earlier position of the new Object corresponding object in the earlier cycle and from this a current speed of the new object is estimated. In the the current cycle information obtained on the new object in the previous cycle can therefore be used, the speed of as a state variable in the current cycle by both the estimate direction and in terms of size. This is, for example, especially helpful to park cars undercover pedestrians suddenly emerge behind the vehicle, quickly detect and then if necessary, be able to respond.

However, in an actual image newly detected object can also a reason other than a concealment in the previous cycle is not have been recognized. Thus, it is preferred that in a previous image searching for the new object, and that, in finding the object in the previous image in the current cycle, a current value at least a state variable for the object and / or other object using a previous value of the state variable for the new identified object and / or the other object in the earlier cycle becomes. The search can be facilitated on the one hand, that Properties of the object, such as its size and / or outline or shape of the current cycle are already known. On the other hand can be used that in the previous image, the new object must have been such that for the evaluation of the former Image in the previous cycle, if any, uncertainties in the allocation of portions of an image, for example, pixels or segments, can be resolved to objects.

Such uncertainty may occur in particular when in the previous image actually two different objects associated detected object points or regions as an object belonging will. the new object finds in the previous picture again, can so information on both the new object or the corresponding thereto an object and an object corresponding to the other Another, now for the earlier cycle newly given object be won. In particular, for the new object or the other object as a state variable in the previous cycle which plies and, optionally, sizes and / or shapes to be determined. Among Using corresponding data for the current cycle can then the one for the current cycle in the first new object already a speed be determined and, secondly, for the other Object that the object at the previous cycle corresponds to a more accurate speed can be determined.

By using the method according to this invention Training can especially very simple example pedestrian be appreciated from the view of the sensor immediately before a Subject have stopped at the roadside and on the road entered. Immediately with recognition of the pedestrian in a current image can then by the time tracing its Speed ​​both in terms of size as well as their direction be estimated, so that valuable time in detecting the can be obtained pedestrian.

The search for the new object in the earlier image in different Way done. In one development of the invention Method, it is preferred that the images with a predetermined segmentation method and with at least one corresponding predefined segmentation parameters are segmented, and that the finding of the new object in the earlier image at least a corresponding portion of the previous image with a other segmentation method and / or with a changed segmentation parameters is segmented again. As a result, a new assignment of the newly determined segments to corresponding objects done. The new segmentation does not need it for the entire to take place image, but may preferably only on the by the Location of the new object in the current cycle and appropriate, nearby other objects depending on their speeds certain range extending so that the recognition of other Objects is not affected. Both the segmentation process used as well as the possibly changed segmentation parameters can thereby be adjusted, in particular situational, ie in Depending on the positions, types or sizes and / or the Number of pixels corresponding to the new object and the respective other object to be selected.

In a preferred development of the inventive method one at a time associated with removal and video images are used, for example, by a laser scanner for the detection of Distance images and a video camera for capturing video images, or by a video system for generating stereo images with at least two spaced apart arranged, together coupled video cameras can be recorded. For example, the Video image processing to reduce the processing burden by an evaluation corresponding distance images controlled, done that an attention control of the video image processing by the Laser scanner, after only in those areas of the video image Features and / or objects sought, include the portions, in those in the corresponding distance image object points or Hazardous Areas corresponding pixels were detected. Especially , the areas through in a direction orthogonal to a scanning plane extending strips be formed whose section with the detection area the sensor for detecting the distance images in his Position and width by at least one distance picture element or more Distance picture elements is defined. An object is in the Video image only detected when in the detection range of the sensor enters for detecting the distance images. For example, for Detecting the distance images, a laser scanner used by the a pivoted in a scanning laser beam of a plane Detection range scans an object in a video image can be recognizable, although this is not in the range image, for example, due to concealment at the height of the scanning plane, recognizable is.

It is therefore preferred that temporally mutually assigned removal and Video images of at least one joint portion of the detection range be used that in video images for objects and / or features are searched only in sections, the function be determined by temporally associated distance picture elements, that in a current cycle upon detection of a new object, and / or feature the new object and / or feature in the video image is traced back in time to an earlier cycle, and that a location of corresponding to the new object and / or feature of the article in the previous cycle using the information on the one Object corresponding object and / or feature in the earlier Video image is determined. The data used were temporally associated with one another Distance and video images are preferably substantially synchronously detected, ie their sensing times differ by one smaller than the period of time between successive samples by the sensor with the lowest recording rate. The subsections may, in particular through spaces of predetermined size and shape the distance picture elements, preferably the aforementioned strip orthogonal be added to a scanning plane. For distance pixels in the distance image in the corresponding section of the Video image ie after at least a corresponding feature or to search an entire object in the video image. Using only of features is the tracing usually carried out more quickly. Determining the location of the new object or feature corresponding object in the earlier cycle can in particular carried out by an estimate, which is preferably the position of the corresponding Feature and / or object is based in the video image. Furthermore, from the position of corresponding to the new object or feature Object in the earlier cycle and the location of the new object or feature in the current cycle, the speed of which be determined in the current cycle. In this process variant can be particularly exploited that with a video camera often already objects can be appreciated as the same in the Distance images that the usually lower only in height extended, ie reflect shallower detection range, not, for example, by concealment at the height of the sensing range are recognized for the range images. If an object in the Distance image for the first time on, it is possible to provide a corresponding thereto to detect feature in the preceding cycle in the video image, when the article only in the distance image, but not in the Video image was obscured. A velocity of the object can then are determined on the basis of the location of the feature in the video images. The occurrence of "blind spots" of the video image processing, by attention control are limited, so it can easily considerable Expenses are limited.

In another embodiment of the method according to the invention it is preferred that temporally mutually associated distance and Video images of at least one joint portion of the detection range be used that distance picture elements in a previous Distance picture distance picture elements in a current distance image be assigned by at least one of a distance picture element in the earlier distance image corresponding area and / or at least one a distance picture element in the earlier Distance image corresponding feature of the corresponding previous Video image and / or of the current video image, a displacement and / or displacement speed, in particular an optical River, and it is determined that the shift or displacement speed, in particular the optical flow, the allocation of a Distance pixel in the current distance image to the distance picture element is used in the earlier distance image, wherein for objects in the distance image, which is the removal of images approach sensing sensor and / or in a current cycle more distance picture elements have as in a previous cycle, Distance picture elements in the current cycle using the corresponding shift or displacement speed of Region or feature, in particular the corresponding optical assigned flow, distance picture elements in the previous cycle will. In this embodiment, information from distance images are with information from corresponding video images linked, which in addition to at least one distance picture element Data are obtained in the processing of the distance picture elements can be used. In this case, in particular exploited that, for a movement of an object generally in the range image and the video image captured areas on the article be moved. Assign the areas in the video image an intensity profile or corresponding features, the movement expresses the object approximately as displacement of the corresponding Intensity characteristics or features in the video image. Under a shift is this a shift between successive understood cycles, but not necessarily in immediate succession to need follow. The movement information from the video images can then the respective corresponding regions of the article or Distance picture elements are assigned.

Preferably, for the determination of the displacement or displacement velocity the range, a corresponding optical flow determined. In the area used to define the optical flow can on the one image plane are used in the video pixels of then also lie in the plane defined video image. Preferably, however, an area is used, the one in the frame Camera model is used for the treatment of a video image. in principle it is sufficient if a distance picture element for at least one optical flow can be determined. Generally, namely the case occur that the distance image points detected on object surfaces are having no intensity structure and for which therefore also an optical flow can not be determined. The optical flow is, for example, then fully determined when the respective area of Video image into two linearly independent directions a comprises changing the intensity. preferably takes place a corresponding Persecution therefore based on these areas.

Method for calculating the optical flow are basically known and for example in the article "The computation of optical flow "by JL Barren and SS Beauchemin in ACM Computing Survey, . Vol. 27, No. 3, pp 433-467 (1995) and in the article "Performance of optical flow techniques "by JL Barren, DJ Flied, DJ and SS Beauchemin in International Journal of Computer Vision, 12 (1), pp 43-77 (1994), the contents hereby incorporated by reference in the description is received.

The optical flow does not need it necessarily for the entire video image to be calculated. It is sufficient that for the distance picture element corresponding object point or - if necessary - Identify its surroundings. The place in which to define the optical flow area used to calculate the optical flow must be, is of the position coordinates of the distance picture element, the relative position of the range images used for detection Sensor to the video system used for detecting the video images, the imaging geometry of the video system and a video camera in it and a model for this and the predetermined surface on which the optical flow is to be determined, derivable. Here, as a place corresponding video image point are used, however, it is also possible the optical flow with subpixel accuracy at locations between video pixels to identify or pixels of the video image. The pursuit of Distance picture elements with the aid of the optical flow in removing pixels in a previous cycle distance pixels are allocated in a later cycle is in the from of Applicant filed German patent application with the official File number 103 12 249.4, the contents hereby incorporated by reference is included in the description. There are range images called depth-resolved images.

This for moving away from the sensor objects or objects with a not greater number of distance picture elements in the current therefore cycle method used has the advantage that a prediction the object positions does not need to be made, since the use of the displacement or the optical flow tracking individual distance picture elements allowed. approaching the reverse procedure in itself Objects or objects in the current cycle longer distance picture elements have as in a previous cycle, firstly has the Advantage that an allocation of a greater number of distance picture elements to a smaller number of distance picture elements, in particular in situations in which objects in the current cycle close adjacent to each other can be carried out correctly with greater certainty. Secondly, the corresponding object more accurate speed are allocated, as to determine the speed of the Object averaging over respective speeds each other associated distance picture elements can be used and in the current cycle, a larger number of such distance picture elements for Available.

In another development of the method it is preferred that the object detection and tracking on the basis of Video images takes place, and that in the object recognition, the detection of takes place objects on the basis of the filtered video images represented by the Filtering have a reduced resolution, and that after detection an object in the current cycle on the basis of the location and, optionally, Shape of the object in the current filtered video image to one corresponding to the object article in a previous video image is wanted, whose resolution is higher than that of the filtered video image, and that when finding a corresponding object in the previous video image corresponding data about the object in the evaluation be used in the current video image. Object recognition and tracking can optionally by the use of Distance images are supported. In the case of this development, mainly to save computational effort, initially captured video images subjected to reduction in the resolution of a filtering. In the Filtering can be any process in which the Number of pixels of the video image is reduced. In the simplest case for example, can easily be performed only a sub-sampling of the image be in the from square blocks of pixels or Pixels, only one pixel or pixels representative of the whole Arrangement is used. a far object nears Sensor used for detection of the video image, the case may occur, that at the first detection of the object very few pixels or pixels are available in the filtered video image. through analysis the previous image with increased resolution, where relevant, the Object recognizable only because the filtering is not in the previous image was, will now be recognized, so that the newly discovered object in the current cycle despite the only low resolution a trajectory and speed can be assigned without the need for a significant Computational effort would be necessary.

The invention is also a computer program with program code means, to the process of the invention to perform, when the program is run on a computer.

Another subject of the invention is also a computer program product with program code means that on a computer readable medium are stored to perform the inventive method, when the computer program product runs on a computer becomes.

At a computer In this case, any data processing device, in particular a data processing device according to the invention Apparatus understood performed with the method can be. In particular, this may include a digital signal processor and / or microprocessor having, with which the process wholly or is executed in parts.

The invention is preferably used for the monitoring of areas with movable Objects, in particular road traffic, is suitable, the acquisition of images by at least one stationary sensor and / or one held sensor in a vehicle can take place.

The invention will now be exemplified further with reference to the drawings explained. Show it:<dl tsize="16"><dt>Fig. 1</dt><dd>a schematic plan view of a vehicle having a Apparatus for the detection and tracking of objects according to a first preferred embodiment of the Invention and located a front of the vehicle Object,</dd><dt>FIG. 2</dt><dd>a flow diagram in which schematically the flow a method according to a first preferred embodiment of the invention is illustrated,</dd><dt>FIGS. 3A and 3B</dt><dd>Excerpts from at successive times acquired range images of a detection range a laser scanner of the apparatus in Fig. 1 with a only partially shown vehicle and a moving Pedestrian, </dd><dt>Fig. 4</dt><dd>the representation in FIG. 3B with additional guides for explaining a calculation of the position or speed the pedestrian,</dd><dt>Fig. 5</dt><dd>a flow diagram in which schematically the flow a method according to a second preferred embodiment of the invention is illustrated,</dd><dt>Figs. 6A to 6G</dt><dd>Excerpts from time sequentially detected Distance images of scenes with an only partially shown vehicle and at times before Vehicle in front, moving pedestrians,</dd><dt>Fig. 7</dt><dd>a schematic plan view of a vehicle having a Apparatus for the detection and tracking of objects according to a second preferred embodiment the invention and located a front of the vehicle Object,</dd><dt>Fig. 8</dt><dd>a schematic, partial side view of the vehicle and the object in Fig. 7,</dd><dt>Fig. 9</dt><dd>a flow diagram in which schematically the flow a method according to a third preferred embodiment of the invention is illustrated;</dd><dt>FIG. 10A and 10B</dt><dd>Excerpts from a distance image and a corresponding video image with a vehicle and a moving pedestrian at a first detection time,</dd><dt>FIG. 11A and 11B</dt><dd>the images in Figs. 10A and 10B corresponding Of distance or video images at a later time of data,</dd><dt>FIG. 12A and 12B</dt><dd>a flow diagram in which schematically the flow a method according to a fourth preferred embodiment of the invention is illustrated,</dd><dt>Fig. 13</dt><dd>a schematic, perspective view of a Part of a scan with a distance picture element, an imaging geometry to calculate a optical flow and at the end of the process in FIG. 12A and 12B occurring layers of recorded and predicted distance pixels and video pixels appropriate and an optical flow Displacement vector,</dd><dt>Fig. 14</dt><dd>a flow diagram in which schematically the flow a method according to a fifth preferred embodiment of the invention is illustrated,</dd><dt>FIG. 15A, 15B</dt><dd>Excerpts from an unfiltered and a filtered Video image with a pedestrian shown schematically at a first time, and </dd><dt>Fig. 16</dt><dd>an unfiltered image of the pedestrian in Fig. 15 to an earlier time.</dd></dl>

In Fig. 1, a vehicle 10 carries on its front side, a laser scanner 12 and a connected via a data line with the laser scanner 12 Data processing means 14, which together with the laser scanner 12 a device for detecting and / or tracking of objects forms according to a first preferred embodiment of the invention. In Direction ahead of the vehicle is a in Fig. 1 very Person schematically shown 16 that in the invention of simplicity half is regarded as an object.

By means of the laser scanner 12 is in FIG. 1 only in partial detection range 18 scanned, the result of the mounting position of the laser scanner 12 at the front side of the vehicle 10 symmetrically to the longitudinal axis is arranged the vehicle 10 and an angle of slightly more than 180 ° covers. The detection area 18 is in Fig. 1 only schematically and for better representation in particular in the radial direction too small shown.

The laser scanner 12 scans its detection range 18 in principle known manner with a rotating with a constant angular velocity, pulsed laser radiation bundle 20 from which also peripherally τ at constant intervals .DELTA.t at times_i in solid angle regions α to an average angle_i it is detected whether the laser radiation beam 20 from a point 22 or region of an object, for example, the person 16, is reflected. The index i runs from 1 to the case Number of angular regions in the detection area 18. From this angle ranges is shown in FIG. 1, only an angular range shown, the middle of the angle α_i assigned. Here, the angular range is, however, to clearer expression shown exaggeratedly large.

Due to the pivoting of the laser radiation beam 20 is the Detection area 18 to the expansion of the laser radiation beam 20 is substantially two-dimensional and substantially forms, that is to to the diameter of the laser radiation beam 20, a scanning plane.

With the maturity of the votes of the laser scanner 12 laser radiation pulse of the laser scanner 12 to the object point 22 and back to the laser scanner 12, the distance d_i the object point 22 determined by the laser scanner 12th The laser scanner 12 therefore recorded as coordinates in a the subject of point 22 Object or person 16 corresponding distance picture element the angle α_i and those established in this angular distance d_i, ie the position of the object point 22 in polar coordinates. Each detected therefore subject point a distance picture element is assigned.

The amount of in which a scan detected distance image points forming a distance image in the sense of the present application.

The laser scanner 12 scans the detection area 18 respectively in successive Samples at intervals of time .DELTA.t, so that a temporal Series of scans and corresponding distance images arises. Processing takes place the distance images of the laser scanner 12 by the data processing means fourteenth

The data processing device 14 has to, inter alia, for a Carrying out the method according to the invention with a corresponding inventive computer program programmed digital And signal processor means connected to the digital signal processor Storage means. In another embodiment of the invention Apparatus, the data processing device also has a comprise conventional processor to which a in the data processing device stored inventive computer program is executed for performing the inventive method.

The programmed data processing device is doing in terms the invention provides means for evaluation of current images of the sensor means to identify parts of a current image as a result of at least Preliminary evaluation of the current image or for the recognition an object in the current cycle and means for assigning Parts of an earlier image and / or for determining at least one An indication relating to an earlier state of the object or of this corresponding object on the basis of a corresponding previous image and assigning the item to the part of the current Image or the current object ready.

On the basis of the acquired of the laser scanner 12 range images is that illustrated in Fig. 2 according to a first preferred method Embodiment of the invention carried out.

In successive cycles respectively the steps S 10 to S28 are performed.

First, in a current cycle in step S 10 by one scan the detection region 18 a distance image detected and in a Memory is read into the data processing device 14th

In step S 10 it is a pre-processing the distance image data carried out, in which, possibly after correction of the data, Transformation of the position coordinates of the distance picture elements in a Cartesian, firmly connected to the vehicle 10 vehicle coordinate system is carried out.

In step S 12, the distance image is then segmented. Here are Amounts of distance picture elements formed in a conventional manner, which are characterized in that each distance pixel a Amount of at least one other distance pixel thereof Amount has a root mean square distance, which is smaller than a predetermined segmentation distance. A lot thus formed corresponds in each case one segment. In this case, a segment by a be formed individual distance picture element, the other of all Distance picture elements of the current distance image a mean has square distance which is greater than the predetermined segmentation interval.

In step S 14, which is not performed in the very first cycle of the process is, then an allocation of segments already takes place in an earlier cycle of the process detected objects, including itself known methods can be used. In the present process is this for any known object in the previous cycle each a predicted in step S28 of the previous cycle property Location used. In the current cycle is a segment of the current associated with a corresponding object cycle of the previous cycle, if at least one of the distance picture elements of the segment of the object in the predicted object position at a distance, the is less than a of an uncertainty of the prediction and the size and orientation of the object in the previous cycle dependent Maximum distance.

In step S 16 are then segmented, which no preceding from the Cycle known objects could be assigned, new formed objects and constituent of the layers of the segments Distance picture elements a location of the object in the current cycle determined.

The above described steps S 10 to S 16 and to be described yet Steps S26 and S28 are not different from conventional An object recognition and -verfolgungsverfahren.

In contrast to these, however, in step S 18, after examination, whether new objects were found, checks whether in the preceding Cycle an object newly formed corresponding object in a preceding distance image or cycle optionally corresponding one of a detected in the previous cycle object Object could be obscured.

The processing of the current distance image in the steps S 10 to S 16 including the check in step S 18, whether a new object was found, provides an at least preliminary evaluation of the current Image represents.

For occlusion detection in the preceding distance image first checks which in the previous image objects detected in this a corresponding by a compensation curve through the formed distance picture elements comprise contour which is greater than the by the corresponding distance picture elements certain extent the new object in the current cycle.

This is exemplified in Figs. 3A and 3B, in which cutouts in from successive distance images, a vehicle 24, which is only partially represented by the black solid line, and symbolized by a rectangle 26 pedestrian, who with him a speed v relative to the vehicle 24 is moved are shown. The individual dots indicate distance picture elements of the respective Distance pictures again. In Figs. 3A and 3B is the laser scanner 12 at the coordinate origin, that is, in the point (0,0), wherein the Coordinate axes of the Cartesian coordinate system in arbitrary but fixed selected distance units are divided.

In Fig. 3A, a preceding distance image, the pedestrian is 26 concealed from the view of the laser scanner 12 of the vehicle 24 and therefore not detectable in the range image. In Fig. 3B, in which the current shown distance image, the pedestrian has 24 behind the vehicle 24 moved away and is now of the laser scanner 12 first detected. corresponding the contours of the articles 24 and 26 Objects are shown in Fig. 3A and 3B respectively by a dashed line Compensation curve given by the distance picture elements. It is easy to recognize that the corresponding contour of the vehicle 24 in Fig. 3B Object is greater than that of the corresponding pedestrian 26 Object.

The further masking recognition preferably takes situationsadaptiv, ie, depending on the rates of already detected Objects in the previous cycle. In the example in FIGS. 3A and 3B It is known that the vehicle 24 is resting, and that, since no additional object very close to the newly recognized, corresponding to the pedestrian 26 Object can be found or any other object having a smaller distance from it and the pedestrian 26 corresponding object because of the size not very fast moving may correspond to the subject, the object or pedestrian 26 in the previous image with very high probability by the vehicle was 24 concealed.

In step S20, then the location of the preceding distance image corresponding hidden, new objects objects as specified in Reference to a previous state of the respective new objects within the meaning the invention estimated. In order in the example of Fig. 3A and 3B, a minimum to estimate velocity of the object or pedestrian 26, is used to estimate the situation in the preceding distance image . 3A assumed in Fig, that the subject matter or pedestrian 26 for Date and time of the distance image was not visible straight, so that a this object box surrounding in the current distance image, their size dictated pedestrians in the process for objects of type is, with its edge just not from the orbiting laser radiation beam could be detected in 20th In this way, the results in Fig. 4 by the dotted rectangle drawn given estimated Location of the pedestrian 26 in the preceding distance image or Cycle, in order the displacement vector d relative to the position of the current distance image or cycle is postponed.

In step S22, the speed of the objects is then estimated by changing the position of an object in the previous frame and is the current image determined, for the new objects, in so far as they were covered, the estimated position is used. The estimated Speed ​​of the objects is then obtained both in terms of the Direction and the magnitude of the velocity by dividing the given by the displacement vector d change in position by the Period of time At between successive scans of the detection area 18th

While for already in the preceding cycle detected objects determined speeds due to the present measurement data only a small error can be expected to result in new items only afflicted with a larger error estimate. This estimate provides However, faster and much more especially security Information than would be possible without recourse to the previous image.

In step S24, then the determined object positions and sizes and the current distance image in the following cycle previous as then Distance image must be available, stored, the previous range image is deleted.

In step S26, the object positions and speeds to be appropriate further processing devices issued.

In step S28, the end of the current cycle, a prediction of new Object positions for the following cycle performed. The prediction of the Location of the object as well as the determination of the uncertainty of the prediction can, for example in a known manner by means of a Kalman filter done.

Because the new object or object in the current Cycle already a speed can be assigned, the Initialization of the Kalman filter for this object with much more accurate carried out data, which the following object recognition and tracking substantially facilitated.

Then the next cycle with step S 10 can begin.

A method for detecting and tracking objects after a second preferred embodiment of the invention is schematically illustrated in the flowchart in FIG. 5. It differs from the The method according to the first embodiment in that, in a current cycle newly discovered objects are treated differently, so that Steps S 10 to S 16 and S22 through S28 are the same as in the previously described embodiment, and the explanations there correspondingly apply here. Accordingly, the device is for Tracking and recognition of objects by the method of second preferred embodiment of the invention over the corresponding Device in the first embodiment, only the effect changed so that the worked off in the data processing device 14 Program is modified accordingly, including, in particular, that the means for the association of parts of an earlier image and / or for determining at least one indication with respect to an earlier state of the object or of an appropriate object on the basis of a corresponding previous image and assign the item to the part of the are the current image or the current object modified.

In the method, after the first, as in the embodiment running steps S 10 from S to 16 after the formation of new objects previously assigned segments in step S16 in step S30 after Review of the criteria, if at all in the current new Object was found, according to the new objects corresponding objects or objects in the preceding distance image wanted. It is assumed that the new object in the preceding Cycle or distance image could not be found, because it due to the segmentation and object formation or segment-object association could not be recognized as an independent object.

An example of this is shown in Figures 6A. To 6G illustrates in which each shown excerpts from successive distance images are. The laser scanner 12 is in the range images again in (0,0) of the coordinate system arranged that the Cartesian Coordinate system of the previous embodiment corresponds.

In the distance images, a pedestrian 26 moves from the perspective the laser scanner 12 first immediately before a parked vehicle 24 at a speed v in order in FIGS. 6F and 6G Finally, as far removed from the vehicle 24 that he as an independent object detected with the normal segmentation process can be. In the foregoing, however, the range images Segmentation corresponding to the vehicle 24 Distance pixels not from the distance picture elements corresponding to the pedestrian 26 separate.

Based on the size of the object S 26 or of the corresponding new object in the current cycle, which in the example the distance image is evaluated in Fig. 6F, it is assumed that this only may have a low speed, since the object is a pedestrian have to be. To search in step S30, therefore, be for the situation adaptive Resegmentation the objects determined that the pedestrian 26 corresponding new object in the current, the range image are shown in Fig. 6F corresponding cycle closest. In our example, only the object corresponding to the vehicle 24. The steps up to this Stadium correspond to at least preliminary evaluation of current distance image within the meaning of the invention.

It is a part of the preceding distance image is then re-segmented. More specifically, the distance picture elements of the segments in which are preceding distance image or cycle into new segments, the each previously determined for segmentation objects in the preceding form cycles. In the example, these are all distance picture elements of the cutout of the distance image in Fig. 6E.

For this purpose, in the present embodiment, the applied in step S 12 Segmentation method, however, reduced to a segmentation distance, used. The segmentation spacing can be fixed. Preferably, the segmentation distance but at least one of the detected depending on the detected size Objects, preferably the smallest, chosen so that a situationally can be done custom segmentation.

there is a change in the number of segments, in step S32 a new allocation of segments to the re-segmentation of the infected objects made. It is for the previous in the Cycle already known objects whose position and speed in the preceding cycle in conjunction with the current in the Cycle determined orientation, shape and size as well as for the new Object whose determined in the current cycle shape and size used.

In step S34, it is then checked whether the preceding distance image new or changed objects have been found. This is not the case the method is continued with step S22.

Otherwise, in step S36 will be a location of the new object corresponding Subject and the modified object in the preceding Distance image determined. The new object is then as a known treated object, its location in the preceding cycle is known, including appropriate changes in data structures of the program be carried out and a corresponding Kalman filter for the previous Cycle is initiated subsequently. Furthermore, for the modified object a new prediction of the situation for the current Cycle must be to improve the accuracy of the Kalman filter.

There are therefore in the current cycle object positions for the preceding and the current cycle for the previous cycle known and in step S32 newly discovered objects obtained so that in step S22 now carried an estimate of the speed of these objects can, the object positions in the preceding distance image or Cycle may be used.

The subsequent steps S24 to S28 extend as in the first embodiment.

The fact that the object or pedestrian 26 in the current Cycle already a speed can be assigned, the Initialization of the Kalman filter in the current cycle as a result already on the basis of data for this object from the previous cycle and thus overall done with much more accurate data, which the following object recognition and tracking much easier.

In this way it is possible, for example, one directly in front of a vehicle is parked moving pedestrian, who then on the road occurs after processing only a distance image, in which he as single object has been detected, a trajectory and thus speed assigned.

Basically, the method of the first two embodiments may be combined. For example, only an assessment under carried out step S30 and, if found in step S34, no new objects were, a masking recognition according to the first embodiment.

In Figs. 7 and 8 1 is in a situation in Fig. Corresponding Situation a vehicle 10 is shown as a device for detecting and tracking of objects according to a third preferred embodiment, the invention contributes.

The different device for detecting and tracking objects from the apparatus for the detection and tracking of objects in the first embodiment in that in addition to the laser scanner 12, a video system 30 is provided which, as the laser scanner 12 also, via a corresponding data connection with an opposite the data processing means 14 in the first embodiment modified data processing device 14 '. For the others, like parts or features are each the same reference numerals used and the explanations on the first embodiment, apply accordingly.

The video system 28 has a monocular video camera 30, at which one is a conventional black and white video camera with CCD area sensor 32, and an imaging device shown in FIGS. 7 and 8 is shown schematically as a simple lens 34, but in fact consists of a lens system, and a detection range 36 the video system, incident light reflects on the CCD area sensor 32nd An optical axis 38 of the video camera 30 is in a low, in Fig. 8 with an exaggerated size angle shown on the pivot plane the laser radiation beam 20 given scanning 40 of the laser scanner 12 inclined.

The CCD area sensor 32 comprises a matrix disposed photodetection elements in which the formation of video images with video pixels are read cyclically. The video images included for each pixel initially in each case the position of the photodetection elements in the matrix or other identifier for the photodetection elements, and in each case one of the intensity of the corresponding one of the photodetection element received light corresponding intensity value. The video images are in this embodiment substantially in synchronism with the Distance images and thus recorded with the same rate at which also are captured by the laser scanner 12 range images.

From an object in Figs. 7 and 8 of the person 16, emanating Light is imaged by the lens 34 onto the CCD area sensor 32nd This is shown in Figs. 7 and 8 for the outline of the object or of the only schematically shown person 16 by the short dashed lines schematically indicated.

From the distance of the CCD area sensor 32 and lens 34 and from the Location and the imaging properties of the lens 34, for example, their Focal length, may be made of the location of an object point, for example the object point 22 on the person 16, calculated on which Location of the CCD area sensor 32 or that the arranged as a matrix Photodetection elements of the object point is imaged. Vice versa can from the position of a photodetection element by short dashed lines in Fig. 7 and Fig. 8, indicated light cone be determined in the object points or regions must be, from which radiation detected on the photodetection element may fall. For this purpose, a corresponding camera model used will. The example is a per se known pinhole model.

A common detection area 42 is shown in Figs. 7 and 8 schematically approximately represented by a dotted line and by the average of the detection region 18 of the laser scanner 12 and the Detection range 36 of the video system 28 given.

The data processing device 14 'differs from the data processing device 14 on the one hand in that interfaces are provided to substantially synchronous, ie, within a Period, which is significantly smaller than the period between successive .DELTA.t Scans the detection area 18 of the laser scanner 12, respectively, a current distance image and a current video picture Video system 28 read. On the other hand, the processor is provided with a A computer program according to a third preferred embodiment of the Invention for performing a method according to a third preferred programmed embodiment of the invention. The data processing device 14 'therefore includes means for evaluating current Images of the sensor means of identifying parts of a current image as a result of at least preliminary evaluation of the current image or for the detection of an object in the current cycle and means for the association of parts of an earlier image and / or for determining at least one indication with respect to an earlier state of the object or of an appropriate object on the basis of a corresponding previous image and assign the item to the part of the the current image or with the current object in the sense of the invention.

On the basis of of the laser scanner 12 and the video system 28 in Substantially synchronously sensed distance images and video images is that illustrated in Fig. 9 A method according to the third preferred Embodiment of the invention carried out. In the method this embodiment are identified features in the video images and followed, except that the video image processing of computing time, is made on the basis of the distance images by after Features is searched only in those portions of the video image, in which distance picture elements of the distance image and thus corresponding Subject ranges.

First, in step S38 are a current distance image and a read current video image and preprocessed. The reading and the Preprocessing of the images may independently for performed two images in parallel or in any order will.

The distance picture elements in the range image is additionally to the position coordinates in the scanning plane 40 a by the scanning 40, in the corresponding the the distance picture elements object points there are certain location component to form a complete Location coordinates sentence added in three dimensions. With capable of removing pixel is hereinafter represented by these Coordinates referred defined position.

Furthermore, for possible correction of the video image, the transformed data of the video image in the vehicle coordinate system, in which the distance picture elements are defined. For this purpose a Rectification of the video image data, for example, to eliminate distortions, and a transformation of the video pixels on an image plane performed. By means of the camera model for the video camera 30 can the video pixels then layers in the vehicle coordinate system are assigned in a corresponding plane.

The current distance image and the current video image will be to stored for further use.

In step S 12 is then, as in the first embodiment, the Distance image segmented.

In the following step S40 features are detected in the video image, wherein only portions used by the video image, in which the According distance image, a one segment in the distance image corresponding Subject must be detectable. More specifically, the detection is carried out only in substantially perpendicular to the scanning plane of the laser scanner 12 extending, respectively corresponding to a segment of the distance image Stripes of the video image, which in the distance picture elements comprise the segments corresponding video pixels. The Stripes are defined such that it in a direction orthogonal to the Scanning plane 40 of the laser scanner 12 over the entire extent of the Video image and extend in the orthogonal direction, a width exhibit, which is determined so that each of all distance picture elements a segment corresponding video pixels in each Strips lie. This video pixels can by using the be camera model determined. Furthermore, the strips in the Width of each predetermined edge regions on both sides of each extreme video pixels on. Since other parts of the video image first are ignored, done so far an attention control Video image processing.

In this step, continue an assignment in the current video image found dead characteristics to from the previous cycle known characteristics using in step S49 of the preceding Cycle respectively predicted for the current cycle position of the respective characteristics. Here are the same layers of features in the current cycle is determined.

In step S41 it is then checked whether new features have been found, are not known from the previous cycle characteristics could be assigned.

Are there new features found in step S42 according to the corresponding new features features in the preceding Video Pictures wanted by backtracking. Purpose, known A method of video image processing are used.

In step S44, the position of the new feature will be appropriate Feature identified in the preceding video image.

On the basis of the positions of features in the previous video image and the positions of the features in the current video image are then in Step S46 estimated speeds of features.

The determined feature positions and velocities are then output in step S26 after storing for further use.

In step S48, the current video image will be saved after the previous video image has been deleted.

In step S49, new feature layers now be predicted, so that a Assignment of characteristics in step S40 in the following cycle to Objects is facilitated. For this purpose, for example, can now known velocities used or a Kalman filter will.

Starting from the at least preliminary evaluation of the range-and Video image in the steps S38, S12 and S40 thus takes a retrospective or additional evaluation of a previous video image. The results of this analysis, the located feature or its Location in the preceding video image, then in the current Cycle for determining the speed of the current characteristic used.

The use of the method for detection of a sudden behind parked Vehicles protruding pedestrians is shown in FIGS. 10A and 10B and 11A and 11B. FIGS. 10A and 11A each show a section of successive, of the laser scanner 12 detected distance image height on the scanning plane 40, while Fig. 10B and 11B excerpts from corresponding substantially synchronously show with the video system 28 captured video images. In these figures, the scan plane 40 of the laser scanner 12 by a dotted line shown. Furthermore, this corresponds in FIGS. 10A and 10B used Cartesian coordinate system which. in the preceding embodiments

In FIGS. 10A and 11A are the vehicle by a rectangle 44 and the pedestrian represented by a rectangle 46 schematically.

First, is the pedestrian, as shown in FIGS. 10A and 10B, of which Laser scanner 12 can not be detected, because it masked by the vehicle 44 is. However, since the detection area 36 of the video system 28 in the vertical Direction to the scanning plane 40 has a larger opening angle than the detection range 18 of the laser scanner 12, in which already seen corresponding video image of the pedestrian 46th conditioned by the attention control but is not this in the Video image detected, since it in the distance image (see. FIG. 10A) obscured is.

Since the pedestrian 46 in the example with a speed v after move right, he is no longer after some time by the vehicle 44 masked, as shown in FIGS. 11A and 11B. Now the pedestrian may 46 are recorded in the range image so that it in the video image is detected via the attention control. While a conventional video image processing now except for the position of 46 pedestrian nothing would be known about this, in the method the embodiment of the pedestrian 46 in the video images in time traced, whereby it again detected in the video images shown in Fig. 10B can be. The resulting finding made in the previous Video image now allows the speed of the first only in the current video image detected pedestrian 46 even at its first to determine detection or recognition. Since the tracing only Discovery of new features occurs, the execution speed is the process only slightly decreased on average.

by In a method for detecting and tracking objects a fourth preferred embodiment of the invention is substantially the same device for detecting and tracking Objects as used in the third embodiment, except that programming the data processing device 14 'in accordance with is also changed to the changed method. This also means that the means for the association of parts of an earlier image and / or for Determining at least one indication with respect to a previous state of the object or of an appropriate object on the basis of a corresponding earlier image and allocation of the indication to the Part of the current image or the current object have been modified.

In this in Fig. 12 A and B schematically illustrated method are distance picture elements in successively captured range images using the optical flow corresponding in areas covered in substantially synchronously with the distance images Video images tracked.

Here is an assignment of objects and representative thereof distance picture elements used to each other, so that persecution of Objects by following the appropriate, belonging to the object can be carried away pixels. Regarding details of some steps the present method and is in terms of possible variants on the first embodiment and the other embodiments in the German patent application with the assignee of the present referenced Application with the official file number 103 12 249.4, the content of which hereby incorporated by reference into the present application is received.

In step S50, a current cycle are first a current detected distance image and a current video image, and in the data processing device read, in the preprocessing of this Images corresponding to step S38 in the previous embodiment, takes place. The transformation of the video pixels, however, for easier calculation of the optical flow over the step S38 been modified in the previous embodiment, that the transformation of video pixels to the definition or Image plane used an optical flow calculation 48 (see. FIG. 13) is carried out. For S52 and S60 to be carried out in steps Projection of distance picture elements in the image plane is the For simplicity, a principle to the expert known modified Pinhole or ground glass model of the video camera 30 used by the position of an optical center, the 50 and the Image plane 48 is defined as the area for defining or determining the optical flow is used. The location of the optical center 50 is using the imaging geometry of the video camera 30, in particular the position relative to the laser scanner 12 and the focal length of the Lens 34, is determined. The pinhole camera model is the simplified representation modified so that the image plane 48 to a relative to the video system 28 fixed to the vehicle 10 position between the optical center 50 and the object points, in Fig. 13 the Points 52 and 58 is located, and by point reflection of the actual Image plane at the optical center 50 is apparent from this.

In step S52, which is appropriate in the first cycle of the process, are first all distance picture elements of the current cycle immediately determined the preceding cycle extending from the sensor or laser scanner 12 removing objects correspond. For simplicity, these distance picture elements as receding objects corresponding distance picture elements designated. Using the Camera model are then appropriate for this receding objects Distance pixels from the current cycle immediately previous cycle corresponding layers or coordinates corresponding video pixels in the image plane is calculated 48 and Use in the current cycle is stored. geometrically as in Fig. 13 illustrates the situation results in each case by the intersection 54 a through the distance picture element, in Figure the distance picture element 52, and the optical center 50 extending degrees with the Image plane 48th

In step S54, which is also omitted in the first cycle, then for all projected, moving away from the laser scanner 12 objects corresponding distance picture elements of the preceding cycle or corresponding intersections corresponding current optical flow vectors, hereinafter also referred to merely as optical flows, on the basis the video image from the immediately preceding cycle and from determines the current cycle and multiplying by the cycle duration or the reciprocal of the sampling frequency in a displacement vector converted. In Fig. 13 this is for the distance pixel 52 shown, intended for the at the intersection 54, an optical flux vector is located in the image plane 48 and the after scaling with the cycle time a starting position at the intersection 54 displacement vector 56 results. The optical flow is in this case by means of a differential Method determined, in this example the in "Performance of Optical flow techniques "by JL Barren, DJ fleed and SS Beauchemin, International Journal of Computer Vision, 12 (1), pp 43-77 (1994) described A method according to Luke and Canade.

For all receding objects corresponding distance picture elements, for an optical flow was determined, then in step S56 a predicted by the optical flow position of a corresponding Object point in the current cycle in the scan plane 40 determined. Geometric illustrated in FIG. 13 is to the point of intersection 57 a through the optical center 50 and the end point of the Displacement vector 56 extending straight with the scanning plane 40 determined.

In step S58 are current distance picture elements, in Fig. 13, for example, the current distance picture element 58, the current distance image to the predicted positions and corresponding distance picture elements assigned the previous frame or cycle, provided that this is possible. Allocation criteria are, for example, used be that one current distance picture element is assigned, the squared distance between the predicted position and the actual current situation minimal compared with those of other current Distance picture elements is.

In step S60, then the unassigned distance pixels the current cycle, which accordingly is not of the laser scanner must comply with 12 removing objects to which Image plane 48 is projected, which is analogous to the projection in step S52.

For all projected, not yet associated distance picture elements of the current cycle are then in step S62 as in step S54 optical Rivers calculated.

For all not associated distance picture elements of the current Cycle for which an optical flow has been determined, is now at step S64 a retraced position of a corresponding object point determined in the scan 40th The determination of the retraced Location is carried out analogously to the determination of the predicted location in Step S56, but as a displacement vector to the optical River oppositely directed such vector is used.

In step S66, then an assignment of distance picture elements of the previous cycle, the not moving away from the sensor Objects correspond to specified then, retraced documents and corresponding not yet associated distance picture elements the current cycle in step S64. The assignment can be for a corresponding schema effected, as in step S58, but now predetermined distance picture elements of the current cycle and the corresponding retraced layers distance picture elements of the preceding Distance image are assigned.

In this manner, the distance image points are the previous Cycle and the current cycle and the image associated with each other, wherein the mapping for moving away from the laser scanner 12 objects takes place such that current distance picture elements distance picture elements are assigned to the previous image, while the remaining Distance picture elements of the preceding image, which not assigned by the sensor or laser scanner 12 removed objects are, the not yet associated distance picture elements of the current assigned cycle and distance image or their retraced documents will.

In step S68 will now be a segmentation of the current distance image according to the segmentation in step S 12 of the first embodiment.

In step S70 a segment-object association is then carried out, wherein the association of distance picture elements of the preceding Distance image and the current distance image and the assignment between distance picture elements of the preceding distance image and objects of the preceding cycle are used.

In step S72, object properties, particularly their documents and velocities determined and output.

The method may then continue with step S50 in the next cycle will. A prediction, by the use of the optical flow accounts.

In other variants of the method according to the fourth embodiment , the projection in other forms and means, examples these are in the referenced patent application in another described embodiments, which are incorporated herein by reference be included in the description.

By treating other distance picture elements that are not located match of the laser scanner 12 removed objects will achieved a clearer association of distance picture elements. Because are in the current cycle due to the imaging geometry of the laser scanner 12, which scans its detection range 18 radially, such Objects more each distance picture elements correspond as in previous cycle. It can then be a mapping of a plurality of Distance picture elements of the current cycle to for example a avoided predetermined distance image point of the previous cycle are, in themselves the laser scanner approaching objects of case may occur that more distance picture elements of the current image a predicted distance pixel should be assigned, which, however, because of the ambiguity of a complicated assignment process can bring.

In a method according to a fifth preferred embodiment of the Invention there is a object recognition and tracking only on the basis of video images. A corresponding apparatus for object recognition and tracking according to a fifth preferred embodiment of the Invention therefore comprises a as in the previous two embodiments, used video system whose video camera with a Data processing device as in the previous two embodiments, is connected, which, however, for implementing the method is programmed according to the fifth preferred embodiment. This also means that the funds for the association of parts of an earlier Image and / or for determining at least one indication regarding corresponding to an earlier state of the object or of this Object on the basis of a respective previous image, and Assignment of the statement to the part of the current image or the current Object are modified. The laser scanner 12 is omitted.

In this in Fig. 14 illustrated process is in a current Cycle initially recognized in step S74, a current video image and read. In this case, as in the previous two embodiments, appropriate preprocessing of the video image data is carried out.

In step S76, for increasing the processing speed of the captured video image of the image resolution reducing filtering subjected, which in the present embodiment in that of 4x4 blocks of video picture elements, or pixels, only the pixel is used in the lower left corner.

In step S78, then objects in the sub-sampled video image detected. For this purpose, conventional methods of object recognition and Tracking can be used in video images.

In step S80 will now be given an assignment of the detected objects to Objects from the previous cycle. The allocation is in Example, based on the predicted object positions for the current cycle, which have been determined in the preceding cycle in a step S94, by known association methods.

In step S82, it is checked whether new objects were detected, not the objects of the preceding cycle were assigned. were not detects new objects, the method continues with step S90.

Otherwise, first in step S84 in the preceding video image, that is not, or in the example only half as strong as undersampled the image generated in step S76, search for an object that a corresponding new object.

Due to the increased resolution in the preceding video image hence detectable objects, which otherwise, due to the subsampling would not be detected.

This is illustrated in FIGS. 15A and 15B, and 16 again for themselves the Video system approaching person represented. In Fig. 15A, the person is schematically shown in a current video image full resolution, representing black rectangles video pixels and pixels that the person correspond. Through the above-mentioned sub-sampling results in the in Fig. 15B shown undersampled video image in which the person only is represented by four individual pixels. Has now the person from approached a distance of the video camera, so they can in the preceding Video image at full resolution, for example, in Fig. 16 have size shown, wherein after undersampling over would remain no pixels, by means of which a corresponding object could be detectable.

However, in the method used herein in the preceding View full resolution and not in the lowered resolution after wanted the new object, which can be therefore found. There the new object or of the object only in the vicinity of the current in the may occur video image detected object, is nonetheless the computing time for the detection of the object in the previous video image greatly reduced.

In step S88, the amount equivalent to the new objects are objects followed in the preceding video image in the current cycle in, which is easily possible because the object is already in the was detected undersampled current video image.

Due to the determined positions of the objects in the preceding video image and thus in the preceding cycle and the current Cycle can now step through S90 velocities of objects Estimate or difference can be determined.

In step S92, the current video image will be saved, the previous video image is deleted.

In step S94, an output of the object positions and speeds takes place for use in the following applications.

then on the basis of the object positions and speeds in Step S96 new object positions for the following cycle predicted, in the Nearby wanted those for objects in the video image of the next cycle becomes. To this end, a Kalman filter can be reused.

The method according to the fifth embodiment allows through to a the sub-sampling speed of execution. On the other hand however, it is for approaching, new and thus critical for a vehicle Objects possible at its first appearance already using a previous video image to determine their speeds, so that in a dangerous situation, the response time to a cycle time can be shortened.

<b>LIST OF REFERENCE NUMBERS</b>

<dl tsize="7" compact="compact"><dt>10</dt><dd>vehicle</dd><dt>12</dt><dd>laser scanner</dd><dt>14, 14 '</dt><dd>Data processing device</dd><dt>16</dt><dd>object</dd><dt>18</dt><dd>detection range</dd><dt>20</dt><dd>Laser radiation beam</dd><dt>22</dt><dd>Subject point</dd><dt>24</dt><dd>vehicle</dd><dt>26</dt><dd>pedestrian</dd><dt>28</dt><dd>video system</dd><dt>30</dt><dd>video camera</dd><dt>32</dt><dd>CCD area sensor</dd><dt>34</dt><dd>lens</dd><dt>36</dt><dd>detection range</dd><dt>38</dt><dd>optical axis</dd><dt>40</dt><dd>scan</dd><dt>42</dt><dd>common detection area</dd><dt>44</dt><dd>vehicle</dd><dt>46</dt><dd>person</dd><dt>48</dt><dd>image plane</dd><dt>50</dt><dd>optical center</dd><dt>52</dt><dd>Subject point</dd><dt>54</dt><dd>intersection</dd><dt>56</dt><dd>displacement vector </dd><dt>57</dt><dd>intersection</dd><dt>58</dt><dd>Subject point</dd></dl><dl tsize="1" compact="compact"><dt>v</dt><dd>velocity vector</dd><dt>d</dt><dd>displacement vector</dd></dl>